# Design: yt-channel-scraper → Local Content-Mining Platform **Date:** 2026-07-26 **Status:** Approved (design review) **Stack decision:** FastAPI + Alpine.js + Tailwind (CDN), no Node/npm build step **Scope rule:** Data-mining + content focus. **No AI** (excluded: LLM summaries, RAG, NER, speaker diarization). --- ## 1. Goal Turn the existing yt-dlp-based CLI scraper into a **local "intelligent platform"** for scraping and mining content-creator data. Add the high-value CLI features from `OPPORTUNITIES.md` (minus AI), a visually attractive local webapp with drag-and-drop cookie session login, `.bat` launchers with automatic port handling, and an opencode subagent + sub-goal that owns the webapp implementation. Everything runs **100% locally** — no remote deploy, no auth, no rate limits beyond YouTube's own. --- ## 2. Context (current state) - Modular pipeline: `cli → discover → extract → parse → chapters → render → store`, plus `ratelimit` and `config`. - SQLite (`data/state.db`): tables `channels` (2 rows), `videos` (883 rows). Statuses: `pending/done/no_subtitles/error`. - Rich metadata (views, likes, tags, thumbnail, description) currently lives **only in Markdown frontmatter**, not in the DB. - 33 transcripts already rendered as `.md` under `data/markdown//`. - A valid Netscape cookie file exists at `cookies/253506e4-387c-4cda-9f11-e95548f8ca0b.txt`. - **Known bug:** `cli.py:228-229` — `_process_one()` references `cookies` and `cookies_from_browser` that are local to `main()` and therefore out of scope. Cookies are currently non-functional. This design fixes it. --- ## 3. Architecture decision **Approach A — Monorepo, in-package webapp (chosen).** The FastAPI app lives inside the existing package at `src/yt_scraper/webapp/` and reuses `Store` + pipeline modules directly. The same FastAPI process serves the SPA static files, the JSON API, and SSE. One repo, one process, zero query duplication. Rejected alternatives: - B (separate top-level package importing `yt_scraper`): doubles plumbing for no local benefit. - C (separate processes sharing only DB): duplicates query logic, breaks the "one platform" feel. --- ## 4. New repo layout ``` src/yt_scraper/ cli.py +subcommands: search, export, audio, channels, watch, analyze, re-render store.py +schema migration, +segments/metadata/cookies/jobs queries segments.py NEW populate transcript_segments + FTS5; backfill from .md export.py NEW json/csv/srt/html exporters analysis.py NEW word freq, top-N terms, term-over-time timeline (matplotlib) monitor.py NEW watch-mode loop (interval discovery + process) cookies.py NEW Netscape parse/validate/vault; resolve active cookie path webapp/ __init__.py app.py FastAPI factory: static mount, CORS off (local), /healthz, SSE mount api.py REST routers (channels, videos, search, transcript, analysis, scrape, cookies, exports) jobs.py in-process scrape job runner + SSE event bus (queue/stream/cancel) static/ index.html SPA shell (Tailwind CDN, Alpine.js, Chart.js CDN) app.js Alpine app: views, state, API client, SSE consumer styles.css custom dark "command-center" theme pipeline glue: extract.py / cli.py now accept the active cookie path templates/ video.md.j2 unchanged (re-render reuses it) gallery.html.j2 NEW template for `export --format html` docs/superpowers/specs/2026-07-26-platform-design.md this file .opencode/ agent/webapp-builder.md NEW subagent owning Workstream B goals/webapp-build.md NEW sub-goal the subagent executes start-server.bat NEW auto port-scan + launch + open browser (repo root) stop-server.bat NEW graceful shutdown + port cleanup (repo root) .run/ gitignored runtime: server.info (port/PID) data/analysis/ gitignored output of `analyze` (PNG/CSV) ``` `pyproject.toml` gains optional deps: `fastapi`, `uvicorn[standard]`, `matplotlib`, `wordcloud`, and `ffmpeg` is a system requirement for `audio`. --- ## 5. Data model (non-destructive, idempotent migration) Run automatically on every `Store` instantiation via `_init_schema` + `ALTER TABLE ADD COLUMN` guards. ### 5.1 `videos` — added columns ``` view_count INTEGER like_count INTEGER tags TEXT -- JSON array thumbnail TEXT description TEXT chapters_json TEXT -- JSON of chapters (for re-render) segments_json TEXT -- JSON of segments (for re-render, avoids re-download) ``` ### 5.2 New `transcript_segments` + FTS5 ``` CREATE TABLE transcript_segments ( video_id TEXT NOT NULL, idx INTEGER NOT NULL, start_sec REAL NOT NULL, end_sec REAL NOT NULL, text TEXT NOT NULL, PRIMARY KEY (video_id, idx), FOREIGN KEY (video_id) REFERENCES videos(video_id) ); CREATE INDEX idx_segments_video ON transcript_segments(video_id); -- FTS5 external-content table over transcript_segments(text) CREATE VIRTUAL TABLE transcript_fts USING fts5( text, content='transcript_segments', content_rowid='rowid' ); ``` Populated during scrape (after parse) **and** by a one-time backfill that parses the 33 existing `.md` files. Drives `search` (CLI) and webapp search (snippet + timestamp). ### 5.3 New `cookies_meta` ``` CREATE TABLE cookies_meta ( id TEXT PRIMARY KEY, -- uuid (matches filename stem) filename TEXT NOT NULL, label TEXT, added_at TEXT NOT NULL, expires_at TEXT, -- earliest cookie expiry, ISO is_active INTEGER DEFAULT 0, -- exactly one row may be 1 has_session INTEGER DEFAULT 0, -- 1 if SID/SAPISID/LOGIN_INFO present cookie_count INTEGER DEFAULT 0 ); ``` Cookie files are stored in `cookies/`. Exactly **one** cookie may be active globally. `cookies.py` resolves the active path for yt-dlp's `cookiefile`. ### 5.4 New `scrape_jobs` ``` CREATE TABLE scrape_jobs ( id TEXT PRIMARY KEY, channel_id TEXT, opts_json TEXT, status TEXT NOT NULL, -- queued/running/done/cancelled/error started_at TEXT, finished_at TEXT, total INTEGER DEFAULT 0, completed INTEGER DEFAULT 0, last_error TEXT ); ``` The scrape console + SSE consumer read/write this table. ### 5.5 Backfill One-time `segments.backfill_from_markdown()` invoked on first run (or via `re-render --backfill`): parse each done `.md`, extract segments + chapters + frontmatter metadata, write into `transcript_segments`/`transcript_fts` and the new `videos` columns. Idempotent (skip videos already having segments). --- ## 6. Workstream A — Core engine + CLI ### 6.1 CLI subcommands (new) | Command | Behavior | |---|---| | `search "query" [-c CHANNEL] [-l N]` | FTS5 MATCH → prints video_id, start timestamp, snippet. | | `export --format {json,csv,srt,html} [-c CHANNEL] [-o DIR]` | JSON = full dump; CSV = metadata table; SRT = subtitles from segments; HTML = Jinja2 gallery (`gallery.html.j2`). | | `audio [-c CHANNEL] [--limit N] [--from DATE]` | yt-dlp `bestaudio` → mp3 via FFmpegExtractAudio. Skips videos already having audio unless `--force`. Records nothing new in DB beyond existing `done`. | | `re-render [-c CHANNEL]` | Rebuild all `.md` from `segments_json`+`chapters_json`+metadata using the current `video.md.j2`. Errors if segments missing (hint: run backfill). | | `channels add ` / `channels list` / `channels remove ` | Multi-channel management. `list` prints a Rich table. | | `--all-channels` (global flag) | When set, `search`/`export`/`analyze`/`re-render` operate across every channel instead of one. Mutually exclusive with `-c`. | | `watch --interval 6h [-c CHANNEL] [--once]` | Loop: `discover_channel` → upsert → process pending → sleep. `--once` runs a single iteration (useful for Task Scheduler). | | `analyze --wordcloud --top-words N --timeline TERM [-c CHANNEL] [-o DIR]` | Saves `wordcloud.png`, `top_words.csv`, `timeline.png` to `data/analysis/`. Pure frequency mining (regex tokenization, Spanish stopwords). | ### 6.2 Bug fix `_process_one()` and `extract_video()` already accept `cookies_file`/`cookies_from_browser` — the bug is that `main()` doesn't pass its local `cookies`/`cookies_from_browser` into `_process_one()`. Fix: thread the two values through `_process_one()`'s signature. Additionally, when neither flag is given, fall back to the **active cookie from the vault** (`cookies.resolve_active_path()`), so the webapp's drag-and-drop selection works for CLI too. ### 6.3 Module responsibilities - `segments.py`: `store_segments(video_id, segments)`, `populate_fts(video_id)`, `search(query, channel_id, limit) -> list[Hit]`, `backfill_from_markdown(store, md_dir)`. - `export.py`: `export_json/csv/srt/html(store, segments_store, channel_id, out_dir)`. - `analysis.py`: `word_frequency(channel_id)`, `top_words(n)`, `term_timeline(term)` — returns data structures; matplotlib rendering optional behind a `render=True` flag so the webapp can call the same functions for Chart.js data without PNG. - `monitor.py`: `watch_loop(config, interval, once)`. - `cookies.py`: `parse_netscape(path) -> CookieFile`, `validate(meta) -> {valid, has_session, expires_at}`, `import_upload(text_or_path) -> id`, `set_active(id)`, `resolve_active_path() -> str|None`, `list_vault() -> list[CookieMeta]`. --- ## 7. Workstream B — Webapp (owned by the `webapp-builder` subagent) ### 7.1 Backend (FastAPI) - `app.py`: factory mounts `/static` (SPA), includes API routers under `/api`, exposes `/healthz`, and an SSE endpoint `/api/scrape/{job_id}/stream`. - `api.py` routers: - `GET /api/dashboard` → aggregate stats (per-channel counts, total duration, date range, uploads-over-time buckets, top tags, duration histogram, status breakdown). - `GET /api/channels`, `POST /api/channels {url}`, `DELETE /api/channels/{id}`. - `GET /api/videos?channel=&status=&from=&to=&min_dur=&q=&sort=&page=&size=`. - `GET /api/videos/{id}` → full metadata + chapters. - `GET /api/videos/{id}/transcript` → segments list (for the reader). - `GET /api/search?q=&channel=&limit=` → FTS hits with snippet + timestamp. - `GET /api/analysis?channel=&type=wordcloud|topwords|timeline&term=` → JSON for Chart.js. - `POST /api/scrape {channel_id, opts}` → creates a `scrape_jobs` row, enqueues in `jobs.py`, returns `{job_id}`. - `GET /api/scrape/{job_id}/stream` (SSE) → progress + log lines. - `DELETE /api/scrape/{job_id}` → cancel. - `POST /api/cookies/upload` (multipart/text) → import; `GET /api/cookies`; `POST /api/cookies/{id}/activate`; `DELETE /api/cookies/{id}`; `POST /api/cookies/{id}/test`. - `GET /api/export?format=&channel=` → streams the file. - `jobs.py`: a single-threaded `queue.Queue` worker runs scrape jobs sequentially (avoids concurrent yt-dlp hammering). Emits SSE events (`progress`, `log`, `done`, `error`). Persisted in `scrape_jobs`. - The active cookie from the vault is passed to every scrape job automatically. ### 7.2 Frontend (Alpine.js SPA) Design language: **dark "data command center."** Near-black canvas (`zinc-950`), glassy cards (`backdrop-blur`, subtle borders), red→rose accent gradient (creator-content nod), monospace data accents, Chart.js viz, smooth Alpine `x-transition`s. Responsive (works on the laptop you launch it on). No auth. Views (hash routes): 1. **Dashboard** — channel cards + global charts (uploads-over-time line, top-tags bar, duration histogram, status donut). 2. **Channels** — list / add (paste URL) / remove / drill-in to a channel detail. 3. **Videos** — faceted table/grid: filter by channel, status, date range, min duration, free-text; thumbnail grid or list; sort; click → detail. 4. **Video detail** — metadata header, chapters sidebar, full transcript reader with per-segment timestamps, in-page term highlight, "open on YouTube at t", per-video export (srt/json), audio-download trigger. 5. **Search** — FTS across all transcripts → result cards with highlighted snippet + timestamp deep-link into the video detail at that segment. 6. **Analysis** — word cloud (canvas), top-N terms bar, term-over-time timeline. 7. **Scrape console** — launch a job (pick channel + options: limit, since, languages, shorts, which cookie), live progress via SSE, streaming log, cancel button, watch-mode toggle. 8. **Cookie vault** — drag-and-drop zone (see §8), vault list, test-session. 9. **Exports** — pick format + scope → download. ### 7.3 "Intelligent" without AI Smart faceted filters, cross-channel term mining, term-over-time timelines, saved searches (localStorage), live monitoring + auto-processing, auto-suggestions from top terms. Feels like a command center for content-creator data mining. --- ## 8. Cookie drag-and-drop UX - Drop zone accepts **multiple `.txt` files** *or* **pasted Netscape cookie text**. - On upload (`POST /api/cookies/upload`): `cookies.parse_netscape()` validates the format, extracts the earliest expiry, detects session cookies (`SID`, `SAPISID`, `__Secure-3PSID`, `LOGIN_INFO`), and returns a card: `{label, expires_at, has_session, cookie_count, size}`. - Vault list with **radio = active** (exactly one active globally), rename, delete. - The active cookie path is passed to yt-dlp as `cookiefile` for every scrape (members-only, age-gate, etc.). - **"Test session"** → lightweight yt-dlp call (fetch the logged-in user's identity / a known members-only-free check) → reports `OK / expired / weak`. - The existing `cookies/253506e4-...txt` is auto-imported into the vault on first server start. --- ## 9. Launcher `.bat` (automatic port system) PowerShell-assisted `.bat` files at the repo root. - **`start-server.bat`** 1. Resolve project root (script dir). 2. Find a free TCP port: scan from `8000` upward until `bind` succeeds (PowerShell `TcpListener` probe). 3. Write `port`, `PID`, and ISO timestamp to `.run/server.info`. 4. Launch `python -m uvicorn yt_scraper.webapp.app:app --port ` via `start` (minimized). 5. Poll `http://localhost:/healthz` until 200 (or timeout ~30s). 6. Open `http://localhost:` in the default browser. 7. Echo a clear success banner (port, URL) or an error. 8. Idempotent: if `.run/server.info` already describes a live server, just re-open the browser. - **`stop-server.bat`** 1. Read `.run/server.info`. 2. Graceful terminate the uvicorn process (taskkill by PID, then `/T` for children). 3. Confirm the port is freed (re-probe). 4. Remove `.run/server.info`. 5. Echo a clear banner. `.run/` is added to `.gitignore`. --- ## 10. Subagent + sub-goal - **`.opencode/agent/webapp-builder.md`** — an opencode agent specialized for this webapp. It knows: the FastAPI + Alpine + Tailwind (CDN, no-build) stack; the `Store` API surface; the design language (§7.2); and its verification steps (start server, `curl /healthz`, smoke-test each SPA view, run `pytest`). It receives Workstream B scope and is expected to reuse `Store`, `segments`, `analysis`, `cookies` rather than duplicating logic. - **`.opencode/goals/webapp-build.md`** — the sub-goal: "Implement the local webapp (Workstream B) per `docs/superpowers/specs/2026-07-26-platform-design.md`. Build backend routers, SSE job runner, and the 9-view Alpine SPA. Verify with `/healthz`, smoke tests, and `pytest`." --- ## 11. Dependencies Add to `pyproject.toml` `[project.optional-dependencies]`: ``` web = ["fastapi>=0.110", "uvicorn[standard]>=0.27"] analysis = ["matplotlib>=3.8", "wordcloud>=1.9"] ``` System requirement for `audio`: `ffmpeg` (`winget install ffmpeg` or `choco install ffmpeg`). The CLI will detect ffmpeg and print a clear hint if missing. `cookies/` is added to `.gitignore` (real session secrets must never be committed). --- ## 12. Build order A → B → C: 1. **A1** Schema migration + backfill + fix cookies bug. 2. **A2** `segments`/`export`/`analysis`/`monitor`/`cookies` modules. 3. **A3** CLI subcommands (`search`, `export`, `audio`, `channels`, `watch`, `analyze`, `re-render`). 4. **A4** Tests; `pytest` green. 5. **B** Webapp (subagent) — backend routers, job runner, SPA, drag-drop cookies. Verify `/healthz` + smoke. 6. **C** `.bat` launcher + `.opencode` agent/goal files. --- ## 13. Testing & verification - `pytest tests/ -v` covering: schema migration idempotency, FTS search, segment backfill from `.md`, cookie Netscape parse/validate, exporters (json/csv/srt/html), analysis frequency/timeline, multi-channel store ops. - Webapp: `/healthz` 200; representative endpoint per router; SPA loads; SSE streams a fake job. - Manual: run `start-server.bat`, confirm browser opens, drag-drop the real cookie, trigger a small scrape (`--limit 1`) against the existing Nostal Vlad channel, verify a new `.md` + FTS row appear and search returns the new content. - `python -m yt_scraper.cli search "..."` returns hits after backfill. --- ## 14. Out of scope (explicitly excluded) - AI features: LLM summaries, Q&A/RAG, NER (spaCy), speaker diarization (pyannote). - CLI `stats`, `clip`, `thumbnails`, `translation` (dropped per user scope decision — aggregate *queries* still exist internally to feed the dashboard). - Remote deployment, auth, multi-user, cloud sync.