Stock Video Collector

Headless browser crawler with a dark-themed PyQt6 desktop GUI for discovering, cataloging, and downloading stock video clips from multiple sites — with full metadata extraction, FTS5 keyword search, and a concurrent download manager.

Quick Start

git clone https://github.com/SysAdminDoc/VideoScraper.git
cd VideoScraper
python artlist_scraper.py  # Auto-installs all dependencies on first run

That's it. The script bootstraps everything automatically:

1. Installs Python packages (PyQt6, playwright, etc.)
2. Downloads Chromium via Playwright
3. Launches the GUI

Requirements: Python 3.9+ — no other prerequisites. Works on Windows, Linux, and macOS.

Features

Multi-Site Crawling

The Generic profile works on any site — it intercepts all video network requests and extracts whatever metadata is available.

Browser Automation & Anti-Detection

Video Discovery

The crawler uses four complementary strategies to find video URLs on every page:

┌───────────────────────────────────────────────────────────────────┐
│                         Page Load                                 │
├───────────────┬─────────────────┬─────────────┬───────────────────┤
│  XHR/Fetch    │   DOM Observer  │  Response   │   HTML Regex      │
│  Intercept    │   (MutationObs) │  Body Scan  │   Fallback        │
│               │                 │             │                   │
│  Hooks into   │  Watches for    │  Scans all  │  Regex sweep for  │
│  XMLHttpReq & │  <video src>    │  HTTP resp  │  M3U8/MP4/WebM    │
│  fetch() API  │  injections     │  bodies     │  + Canva partner  │
│               │                 │             │  links (Pexels)   │
└───────┬───────┴────────┬────────┴──────┬──────┴────────┬──────────┘
        │                │               │               │
        └────────────────┴───────┬───────┴───────────────┘
                                 ▼
                    ┌─────────────────────┐
                    │  Quality Comparison  │
                    │  UHD > HD > SD       │
                    │  Dedup by clip ID    │
                    └──────────┬──────────┘
                               ▼
                    ┌─────────────────────┐
                    │   SQLite Database    │
                    │   + FTS5 Index       │
                    └─────────────────────┘

Database & Search

Asset Management

Download Manager

Export Formats

GUI

Keyboard Shortcuts

Usage

Basic Workflow

1. Select a site profile — check one or more profiles in the Crawl tab (Artlist, Pexels, Pixabay, Storyblocks, or Generic)
2. Set the start URL — auto-populated per profile, or paste any URL for Generic mode
3. Configure crawl settings — batch size, depth, delays, headless mode
4. Start crawling — the crawler discovers pages, extracts metadata, and intercepts video URLs
5. Browse results — switch to the Library tab to search, filter, rate, tag, and organize clips
6. Download — select clips and download with the built-in manager, or export URL lists for external tools

Configuration

All settings persist automatically in a JSON config file. Key options:

Filename Templates

Customize download filenames using template variables:

{title}                      → Beautiful_Sunset.mp4
{clip_id}_{title}            → abc123_Beautiful_Sunset.mp4
{creator}/{collection}/{title} → JohnDoe/Nature/Beautiful_Sunset.mp4

Available variables: {title}, {clip_id}, {creator}, {collection}, {resolution}

How It Works

┌─────────────────────────────────────────────────────────────────────────┐
│                            PyQt6 GUI                                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐ │
│  │  Crawl   │  │  Library  │  │  Detail   │  │ Download │  │  Export  │ │
│  │  Tab     │  │  Tab      │  │  Panel    │  │  Tab     │  │  Tab    │ │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬────┘ │
└───────┼──────────────┼──────────────┼──────────────┼──────────────┼─────┘
        │              │              │              │              │
        ▼              ▼              ▼              ▼              ▼
┌──────────────┐  ┌──────────────────────────┐  ┌──────────────────────┐
│   Crawler    │  │      SQLite + FTS5       │  │   Download Worker    │
│   Worker     │──│                          │──│                      │
│  (QThread)   │  │  clips, crawl_queue,     │  │  ThreadPoolExecutor  │
│              │  │  crawled_pages,           │  │  + ffmpeg HLS→MP4   │
│  Playwright  │  │  collections,            │  │  + retry backoff     │
│  Chromium    │  │  saved_searches           │  │  + speed tracking    │
└──────────────┘  └──────────────────────────┘  └──────────────────────┘

Crawler Worker — Runs Playwright in an async event loop on a dedicated QThread. Navigates pages, injects JavaScript hooks for XHR/fetch/DOM video interception, extracts metadata via regex selectors + OpenGraph + JSON-LD, and manages the crawl queue with depth/priority.

Database Layer — Thread-safe SQLite with WAL mode and a dedicated threading.Lock. FTS5 external content table indexes title, creator, collection, tags, resolution, camera, and duration. Quality-aware M3U8 URL upgrades prefer UHD over HD over SD.

Download Worker — Persistent queue on a QThread with a ThreadPoolExecutor for concurrent downloads. Handles M3U8→MP4 conversion via ffmpeg, exponential backoff retry, real-time speed/ETA calculation, sidecar JSON metadata, and thumbnail extraction.

Troubleshooting

"Chromium not found" — Click the "Install Browser" button on the Crawl tab. This runs playwright install chromium automatically.

Search results seem wrong or incomplete — Click the "🔄 Rebuild Index" button on the Crawl tab to rebuild the FTS5 search index from scratch.

Bot challenge / CAPTCHA detected — Uncheck "Headless" mode and restart the crawl. The browser will open visibly so you can solve the challenge manually. The crawler pauses and resumes automatically once the challenge clears.

Downloads fail repeatedly — Check that ffmpeg is installed and on your PATH. The scraper auto-detects ffmpeg in common locations, but if it can't find it, downloads that require HLS→MP4 conversion will fail.

Clipboard monitor not working — The clipboard monitor is opt-in. Enable it in your config by adding "clipboard_monitor": true, or toggle it programmatically. On Linux/Wayland, clipboard access may require additional permissions.

License

MIT License — see LICENSE for details.

Contributing

Issues and PRs welcome. If you add support for a new site, submit it as a SiteProfile.register() block with documented selectors and test URLs.