feat: local content-mining platform for YouTube creator scraping
Scraper de canales de YouTube hacia notas Markdown para base de conocimiento (Obsidian-ready), con plataforma web local. Engine + CLI (Workstream A): - Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit - SQLite store con migración idempotente: FTS5 (transcript search), columnas de metadata enriquecida, tablas cookies_meta y scrape_jobs - Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html), analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline - CLI Click group: search, export, audio, channels, watch, analyze, re-render - Fix del bug de scoping de cookies en cli.py Webapp local (Workstream B): - FastAPI backend: dashboard, channels, videos facetado, transcript, search FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports, folders (abrir en OS), tools (re-render, formato) - SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema dark command-center con acento rojo→rosa, cookie vault drag-drop, consola de scrapeo con progreso live vía SSE Launcher + subagentes (Workstream C): - start-server.bat / stop-server.bat con auto port-scan + browser open - .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).
This commit is contained in:
urieljareth
@@ -0,0 +1,122 @@
|
||||
# yt-scraper Platform Implementation Plan
|
||||
|
||||
> **Execution mode:** Subagent-driven + inline, autonomous, self-iterating. User mandated "completa todo el proceso de manera agentica y subagentica creando y resolviendo todos los problemas y dudas en un loop auto iterativo." Proceed without further checkpoints; verify with tests and fix in a loop until green.
|
||||
|
||||
**Goal:** Turn the CLI scraper into a local content-mining platform: CLI features (no AI) + FastAPI/Alpine webapp + .bat launcher + opencode subagent.
|
||||
|
||||
**Architecture:** Monorepo. `Store` extends with non-destructive migration (segments FTS5 + metadata + cookies + jobs). New modules `segments/cookies/export/analysis/monitor`. FastAPI app at `src/yt_scraper/webapp/` serves a no-build Alpine SPA. `.bat` launcher auto-scans ports.
|
||||
|
||||
**Tech:** Python ≥3.10, FastAPI, uvicorn, Alpine.js + Tailwind + Chart.js (CDN), SQLite FTS5, yt-dlp, matplotlib, Jinja2.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-07-26-platform-design.md`
|
||||
|
||||
---
|
||||
|
||||
## File map & responsibilities
|
||||
|
||||
| File | Responsibility | Status |
|
||||
|---|---|---|
|
||||
| `src/yt_scraper/store.py` | Add migration (new cols + 3 new tables), segment/cookie/job queries | modify |
|
||||
| `src/yt_scraper/segments.py` | `store_segments`, `populate_fts`, `search`, `backfill_from_markdown` | new |
|
||||
| `src/yt_scraper/cookies.py` | Netscape parse/validate, vault CRUD, `resolve_active_path` | new |
|
||||
| `src/yt_scraper/export.py` | json/csv/srt/html exporters | new |
|
||||
| `src/yt_scraper/analysis.py` | word freq / top-N / timeline (data + optional matplotlib) | new |
|
||||
| `src/yt_scraper/monitor.py` | `watch_loop(config, interval, once)` | new |
|
||||
| `src/yt_scraper/cli.py` | Click group + subcommands; fix cookies bug | modify |
|
||||
| `src/yt_scraper/extract.py` | unchanged (already accepts cookies) | - |
|
||||
| `src/yt_scraper/webapp/app.py` | FastAPI factory + static + healthz | new |
|
||||
| `src/yt_scraper/webapp/api.py` | REST routers | new |
|
||||
| `src/yt_scraper/webapp/jobs.py` | scrape job runner + SSE bus | new |
|
||||
| `src/yt_scraper/webapp/static/{index.html,app.js,styles.css}` | SPA | new |
|
||||
| `templates/gallery.html.j2` | HTML export gallery | new |
|
||||
| `start-server.bat`, `stop-server.bat` | launcher | new |
|
||||
| `.opencode/agent/webapp-builder.md`, `.opencode/goals/webapp-build.md` | subagent | new |
|
||||
| `tests/` | cover migration, segments, cookies, export, analysis, cli | new/extend |
|
||||
|
||||
---
|
||||
|
||||
## Key interfaces (contract between modules)
|
||||
|
||||
```python
|
||||
# segments.py
|
||||
@dataclass
|
||||
class Hit: video_id: str; channel_id: str; title: str; start_sec: float; end_sec: float; snippet: str; rank: float
|
||||
def store_segments(store, video_id, segments: list[Segment]) -> None
|
||||
def search(store, query: str, channel_id: str|None=None, limit: int=50) -> list[Hit]
|
||||
def backfill_from_markdown(store, md_root: Path, log=print) -> int # returns n videos
|
||||
|
||||
# cookies.py
|
||||
@dataclass
|
||||
class CookieMeta: id:str; filename:str; label:str|None; added_at:str; expires_at:str|None; is_active:bool; has_session:bool; cookie_count:int
|
||||
def parse_netscape(path) -> tuple[bool, dict] # (ok, {expires_at, has_session, count})
|
||||
def import_file(store, path) -> str # id
|
||||
def import_text(store, text, label) -> str
|
||||
def set_active(store, cookie_id) -> None
|
||||
def list_vault(store) -> list[CookieMeta]
|
||||
def delete(store, cookie_id) -> None
|
||||
def resolve_active_path(store) -> str|None
|
||||
def auto_import_dir(store, dir_path) -> None # import loose .txt in cookies/
|
||||
|
||||
# export.py
|
||||
def export_json(store, segments_store_fn, channel_id, out_path) -> Path
|
||||
def export_csv(store, channel_id, out_path) -> Path
|
||||
def export_srt(store, video_id, out_path) -> Path
|
||||
def export_html(store, channel_id, out_path, template_path) -> Path
|
||||
|
||||
# analysis.py
|
||||
def word_frequency(store, channel_id=None) -> dict[str,int]
|
||||
def top_words(store, n=50, channel_id=None) -> list[tuple[str,int]]
|
||||
def term_timeline(store, term, channel_id=None) -> list[tuple[str,int]] # (YYYY-MM, count)
|
||||
def render_wordcloud(freq, out_path) -> None # matplotlib
|
||||
|
||||
# monitor.py
|
||||
def watch_loop(cfg, interval_seconds, channel_id=None, once=False) -> None
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tricky bits (resolved decisions)
|
||||
|
||||
- **FTS5 sync:** manual `populate_fts` (delete+insert) after `store_segments`. No triggers (simpler, idempotent). FTS uses external-content `transcript_fts` with `content='transcript_segments'`; queries via `MATCH` + join.
|
||||
- **Migration:** `ALTER TABLE ADD COLUMN` is guarded by `PRAGMA table_info` checks (idempotent). New tables via `CREATE TABLE IF NOT EXISTS`.
|
||||
- **Backfill parse:** re-use existing `.md` format: lines `**MM:SS** · text` under `### Title (MM:SS)` chapter headers; frontmatter between `---` fences has YAML metadata. Reuse `parse.Segment` + `chapters.Section`.
|
||||
- **Cookie active:** single row `is_active=1`; `set_active` flips all others to 0 in one tx.
|
||||
- **Click structure:** convert top-level `@click.command` to `@click.group` with the original scrape flow under `scrape` command (default invocation preserved via a wrapper), plus `search/export/audio/channels/watch/analyze/re-render`.
|
||||
- **Port scan:** PowerShell `System.Net.Sockets.TcpListener` bind-test 8000..8100.
|
||||
- **SSE:** `EventSourceResponse`-style via `sse-starlette` OR a hand-rolled streaming `Response` with `text/event-stream`. Use `sse-starlette` (add to deps) — cleaner.
|
||||
|
||||
---
|
||||
|
||||
## Task list (build order)
|
||||
|
||||
### A1 — Foundation
|
||||
- T1: Store migration (new cols + `transcript_segments` + `transcript_fts` + `cookies_meta` + `scrape_jobs`) + queries. Test: migration idempotent, columns exist, tables exist.
|
||||
- T2: `segments.py` (`store_segments`, `populate_fts`, `search`). Test: store + search returns Hit.
|
||||
- T3: `cookies.py`. Test: parse real cookie file, validate, active toggling, resolve path.
|
||||
- T4: Fix cli.py cookies bug (thread cookies through `_process_one`; fallback to active vault cookie).
|
||||
- T5: `backfill_from_markdown`. Test: parse one real `.md`, assert segments inserted.
|
||||
|
||||
### A2 — Feature modules
|
||||
- T6: `export.py` (json/csv/srt/html) + `gallery.html.j2`. Test: each format round-trips.
|
||||
- T7: `analysis.py` (freq, top, timeline, wordcloud). Test: counts on synthetic segments.
|
||||
- T8: `monitor.py` (`watch_loop`, `--once`). Test: single iteration with mocked discover.
|
||||
|
||||
### A3 — CLI wiring
|
||||
- T9: Convert `cli.py` to Click group; add `search`, `export`, `audio`, `channels`, `watch`, `analyze`, `re-render`, `--all-channels`. Smoke-test each via `CliRunner`.
|
||||
|
||||
### A4 — Tests green
|
||||
- T10: Full `pytest` green; integration tests marked.
|
||||
|
||||
### B — Webapp (subagent: `webapp-builder`)
|
||||
- T11: `.opencode/agent/webapp-builder.md` + `.opencode/goals/webapp-build.md`.
|
||||
- T12: webapp backend (`app.py`, `api.py`, `jobs.py`) — all routers + SSE.
|
||||
- T13: SPA (`index.html`, `app.js`, `styles.css`) — 9 views, dark command-center, drag-drop cookies, Chart.js, SSE scrape console.
|
||||
- T14: Verify `/healthz` + smoke each endpoint; auto-import real cookie on startup.
|
||||
|
||||
### C — Launcher + deps
|
||||
- T15: `pyproject.toml` optional deps (`web`, `analysis`); `.gitignore` (`cookies/`, `.run/`, `data/analysis/`).
|
||||
- T16: `start-server.bat` + `stop-server.bat` (PowerShell port scan).
|
||||
- T17: End-to-end: `start-server.bat` → browser → `/healthz` 200 → dashboard loads.
|
||||
|
||||
### Verification loop
|
||||
- Run `pytest`, fix until green. Run `start-server.bat`, confirm health + SPA. Iterate.
|
||||
@@ -0,0 +1,296 @@
|
||||
# Design: yt-channel-scraper → Local Content-Mining Platform
|
||||
|
||||
**Date:** 2026-07-26
|
||||
**Status:** Approved (design review)
|
||||
**Stack decision:** FastAPI + Alpine.js + Tailwind (CDN), no Node/npm build step
|
||||
**Scope rule:** Data-mining + content focus. **No AI** (excluded: LLM summaries, RAG, NER, speaker diarization).
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Turn the existing yt-dlp-based CLI scraper into a **local "intelligent platform"** for scraping and mining content-creator data. Add the high-value CLI features from `OPPORTUNITIES.md` (minus AI), a visually attractive local webapp with drag-and-drop cookie session login, `.bat` launchers with automatic port handling, and an opencode subagent + sub-goal that owns the webapp implementation.
|
||||
|
||||
Everything runs **100% locally** — no remote deploy, no auth, no rate limits beyond YouTube's own.
|
||||
|
||||
---
|
||||
|
||||
## 2. Context (current state)
|
||||
|
||||
- Modular pipeline: `cli → discover → extract → parse → chapters → render → store`, plus `ratelimit` and `config`.
|
||||
- SQLite (`data/state.db`): tables `channels` (2 rows), `videos` (883 rows). Statuses: `pending/done/no_subtitles/error`.
|
||||
- Rich metadata (views, likes, tags, thumbnail, description) currently lives **only in Markdown frontmatter**, not in the DB.
|
||||
- 33 transcripts already rendered as `.md` under `data/markdown/<channel>/`.
|
||||
- A valid Netscape cookie file exists at `cookies/253506e4-387c-4cda-9f11-e95548f8ca0b.txt`.
|
||||
- **Known bug:** `cli.py:228-229` — `_process_one()` references `cookies` and `cookies_from_browser` that are local to `main()` and therefore out of scope. Cookies are currently non-functional. This design fixes it.
|
||||
|
||||
---
|
||||
|
||||
## 3. Architecture decision
|
||||
|
||||
**Approach A — Monorepo, in-package webapp (chosen).**
|
||||
|
||||
The FastAPI app lives inside the existing package at `src/yt_scraper/webapp/` and reuses `Store` + pipeline modules directly. The same FastAPI process serves the SPA static files, the JSON API, and SSE. One repo, one process, zero query duplication.
|
||||
|
||||
Rejected alternatives:
|
||||
- B (separate top-level package importing `yt_scraper`): doubles plumbing for no local benefit.
|
||||
- C (separate processes sharing only DB): duplicates query logic, breaks the "one platform" feel.
|
||||
|
||||
---
|
||||
|
||||
## 4. New repo layout
|
||||
|
||||
```
|
||||
src/yt_scraper/
|
||||
cli.py +subcommands: search, export, audio, channels, watch, analyze, re-render
|
||||
store.py +schema migration, +segments/metadata/cookies/jobs queries
|
||||
segments.py NEW populate transcript_segments + FTS5; backfill from .md
|
||||
export.py NEW json/csv/srt/html exporters
|
||||
analysis.py NEW word freq, top-N terms, term-over-time timeline (matplotlib)
|
||||
monitor.py NEW watch-mode loop (interval discovery + process)
|
||||
cookies.py NEW Netscape parse/validate/vault; resolve active cookie path
|
||||
webapp/
|
||||
__init__.py
|
||||
app.py FastAPI factory: static mount, CORS off (local), /healthz, SSE mount
|
||||
api.py REST routers (channels, videos, search, transcript, analysis, scrape, cookies, exports)
|
||||
jobs.py in-process scrape job runner + SSE event bus (queue/stream/cancel)
|
||||
static/
|
||||
index.html SPA shell (Tailwind CDN, Alpine.js, Chart.js CDN)
|
||||
app.js Alpine app: views, state, API client, SSE consumer
|
||||
styles.css custom dark "command-center" theme
|
||||
pipeline glue: extract.py / cli.py now accept the active cookie path
|
||||
templates/
|
||||
video.md.j2 unchanged (re-render reuses it)
|
||||
gallery.html.j2 NEW template for `export --format html`
|
||||
docs/superpowers/specs/2026-07-26-platform-design.md this file
|
||||
.opencode/
|
||||
agent/webapp-builder.md NEW subagent owning Workstream B
|
||||
goals/webapp-build.md NEW sub-goal the subagent executes
|
||||
start-server.bat NEW auto port-scan + launch + open browser (repo root)
|
||||
stop-server.bat NEW graceful shutdown + port cleanup (repo root)
|
||||
.run/ gitignored runtime: server.info (port/PID)
|
||||
data/analysis/ gitignored output of `analyze` (PNG/CSV)
|
||||
```
|
||||
|
||||
`pyproject.toml` gains optional deps: `fastapi`, `uvicorn[standard]`, `matplotlib`, `wordcloud`, and `ffmpeg` is a system requirement for `audio`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Data model (non-destructive, idempotent migration)
|
||||
|
||||
Run automatically on every `Store` instantiation via `_init_schema` + `ALTER TABLE ADD COLUMN` guards.
|
||||
|
||||
### 5.1 `videos` — added columns
|
||||
```
|
||||
view_count INTEGER
|
||||
like_count INTEGER
|
||||
tags TEXT -- JSON array
|
||||
thumbnail TEXT
|
||||
description TEXT
|
||||
chapters_json TEXT -- JSON of chapters (for re-render)
|
||||
segments_json TEXT -- JSON of segments (for re-render, avoids re-download)
|
||||
```
|
||||
|
||||
### 5.2 New `transcript_segments` + FTS5
|
||||
```
|
||||
CREATE TABLE transcript_segments (
|
||||
video_id TEXT NOT NULL,
|
||||
idx INTEGER NOT NULL,
|
||||
start_sec REAL NOT NULL,
|
||||
end_sec REAL NOT NULL,
|
||||
text TEXT NOT NULL,
|
||||
PRIMARY KEY (video_id, idx),
|
||||
FOREIGN KEY (video_id) REFERENCES videos(video_id)
|
||||
);
|
||||
CREATE INDEX idx_segments_video ON transcript_segments(video_id);
|
||||
|
||||
-- FTS5 external-content table over transcript_segments(text)
|
||||
CREATE VIRTUAL TABLE transcript_fts USING fts5(
|
||||
text, content='transcript_segments', content_rowid='rowid'
|
||||
);
|
||||
```
|
||||
Populated during scrape (after parse) **and** by a one-time backfill that parses the 33 existing `.md` files. Drives `search` (CLI) and webapp search (snippet + timestamp).
|
||||
|
||||
### 5.3 New `cookies_meta`
|
||||
```
|
||||
CREATE TABLE cookies_meta (
|
||||
id TEXT PRIMARY KEY, -- uuid (matches filename stem)
|
||||
filename TEXT NOT NULL,
|
||||
label TEXT,
|
||||
added_at TEXT NOT NULL,
|
||||
expires_at TEXT, -- earliest cookie expiry, ISO
|
||||
is_active INTEGER DEFAULT 0, -- exactly one row may be 1
|
||||
has_session INTEGER DEFAULT 0, -- 1 if SID/SAPISID/LOGIN_INFO present
|
||||
cookie_count INTEGER DEFAULT 0
|
||||
);
|
||||
```
|
||||
Cookie files are stored in `cookies/`. Exactly **one** cookie may be active globally. `cookies.py` resolves the active path for yt-dlp's `cookiefile`.
|
||||
|
||||
### 5.4 New `scrape_jobs`
|
||||
```
|
||||
CREATE TABLE scrape_jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
channel_id TEXT,
|
||||
opts_json TEXT,
|
||||
status TEXT NOT NULL, -- queued/running/done/cancelled/error
|
||||
started_at TEXT,
|
||||
finished_at TEXT,
|
||||
total INTEGER DEFAULT 0,
|
||||
completed INTEGER DEFAULT 0,
|
||||
last_error TEXT
|
||||
);
|
||||
```
|
||||
The scrape console + SSE consumer read/write this table.
|
||||
|
||||
### 5.5 Backfill
|
||||
One-time `segments.backfill_from_markdown()` invoked on first run (or via `re-render --backfill`): parse each done `.md`, extract segments + chapters + frontmatter metadata, write into `transcript_segments`/`transcript_fts` and the new `videos` columns. Idempotent (skip videos already having segments).
|
||||
|
||||
---
|
||||
|
||||
## 6. Workstream A — Core engine + CLI
|
||||
|
||||
### 6.1 CLI subcommands (new)
|
||||
|
||||
| Command | Behavior |
|
||||
|---|---|
|
||||
| `search "query" [-c CHANNEL] [-l N]` | FTS5 MATCH → prints video_id, start timestamp, snippet. |
|
||||
| `export --format {json,csv,srt,html} [-c CHANNEL] [-o DIR]` | JSON = full dump; CSV = metadata table; SRT = subtitles from segments; HTML = Jinja2 gallery (`gallery.html.j2`). |
|
||||
| `audio [-c CHANNEL] [--limit N] [--from DATE]` | yt-dlp `bestaudio` → mp3 via FFmpegExtractAudio. Skips videos already having audio unless `--force`. Records nothing new in DB beyond existing `done`. |
|
||||
| `re-render [-c CHANNEL]` | Rebuild all `.md` from `segments_json`+`chapters_json`+metadata using the current `video.md.j2`. Errors if segments missing (hint: run backfill). |
|
||||
| `channels add <URL>` / `channels list` / `channels remove <id|handle>` | Multi-channel management. `list` prints a Rich table. |
|
||||
| `--all-channels` (global flag) | When set, `search`/`export`/`analyze`/`re-render` operate across every channel instead of one. Mutually exclusive with `-c`. |
|
||||
| `watch --interval 6h [-c CHANNEL] [--once]` | Loop: `discover_channel` → upsert → process pending → sleep. `--once` runs a single iteration (useful for Task Scheduler). |
|
||||
| `analyze --wordcloud --top-words N --timeline TERM [-c CHANNEL] [-o DIR]` | Saves `wordcloud.png`, `top_words.csv`, `timeline.png` to `data/analysis/`. Pure frequency mining (regex tokenization, Spanish stopwords). |
|
||||
|
||||
### 6.2 Bug fix
|
||||
`_process_one()` and `extract_video()` already accept `cookies_file`/`cookies_from_browser` — the bug is that `main()` doesn't pass its local `cookies`/`cookies_from_browser` into `_process_one()`. Fix: thread the two values through `_process_one()`'s signature. Additionally, when neither flag is given, fall back to the **active cookie from the vault** (`cookies.resolve_active_path()`), so the webapp's drag-and-drop selection works for CLI too.
|
||||
|
||||
### 6.3 Module responsibilities
|
||||
- `segments.py`: `store_segments(video_id, segments)`, `populate_fts(video_id)`, `search(query, channel_id, limit) -> list[Hit]`, `backfill_from_markdown(store, md_dir)`.
|
||||
- `export.py`: `export_json/csv/srt/html(store, segments_store, channel_id, out_dir)`.
|
||||
- `analysis.py`: `word_frequency(channel_id)`, `top_words(n)`, `term_timeline(term)` — returns data structures; matplotlib rendering optional behind a `render=True` flag so the webapp can call the same functions for Chart.js data without PNG.
|
||||
- `monitor.py`: `watch_loop(config, interval, once)`.
|
||||
- `cookies.py`: `parse_netscape(path) -> CookieFile`, `validate(meta) -> {valid, has_session, expires_at}`, `import_upload(text_or_path) -> id`, `set_active(id)`, `resolve_active_path() -> str|None`, `list_vault() -> list[CookieMeta]`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Workstream B — Webapp (owned by the `webapp-builder` subagent)
|
||||
|
||||
### 7.1 Backend (FastAPI)
|
||||
- `app.py`: factory mounts `/static` (SPA), includes API routers under `/api`, exposes `/healthz`, and an SSE endpoint `/api/scrape/{job_id}/stream`.
|
||||
- `api.py` routers:
|
||||
- `GET /api/dashboard` → aggregate stats (per-channel counts, total duration, date range, uploads-over-time buckets, top tags, duration histogram, status breakdown).
|
||||
- `GET /api/channels`, `POST /api/channels {url}`, `DELETE /api/channels/{id}`.
|
||||
- `GET /api/videos?channel=&status=&from=&to=&min_dur=&q=&sort=&page=&size=`.
|
||||
- `GET /api/videos/{id}` → full metadata + chapters.
|
||||
- `GET /api/videos/{id}/transcript` → segments list (for the reader).
|
||||
- `GET /api/search?q=&channel=&limit=` → FTS hits with snippet + timestamp.
|
||||
- `GET /api/analysis?channel=&type=wordcloud|topwords|timeline&term=` → JSON for Chart.js.
|
||||
- `POST /api/scrape {channel_id, opts}` → creates a `scrape_jobs` row, enqueues in `jobs.py`, returns `{job_id}`.
|
||||
- `GET /api/scrape/{job_id}/stream` (SSE) → progress + log lines.
|
||||
- `DELETE /api/scrape/{job_id}` → cancel.
|
||||
- `POST /api/cookies/upload` (multipart/text) → import; `GET /api/cookies`; `POST /api/cookies/{id}/activate`; `DELETE /api/cookies/{id}`; `POST /api/cookies/{id}/test`.
|
||||
- `GET /api/export?format=&channel=` → streams the file.
|
||||
- `jobs.py`: a single-threaded `queue.Queue` worker runs scrape jobs sequentially (avoids concurrent yt-dlp hammering). Emits SSE events (`progress`, `log`, `done`, `error`). Persisted in `scrape_jobs`.
|
||||
- The active cookie from the vault is passed to every scrape job automatically.
|
||||
|
||||
### 7.2 Frontend (Alpine.js SPA)
|
||||
Design language: **dark "data command center."** Near-black canvas (`zinc-950`), glassy cards (`backdrop-blur`, subtle borders), red→rose accent gradient (creator-content nod), monospace data accents, Chart.js viz, smooth Alpine `x-transition`s. Responsive (works on the laptop you launch it on). No auth.
|
||||
|
||||
Views (hash routes):
|
||||
1. **Dashboard** — channel cards + global charts (uploads-over-time line, top-tags bar, duration histogram, status donut).
|
||||
2. **Channels** — list / add (paste URL) / remove / drill-in to a channel detail.
|
||||
3. **Videos** — faceted table/grid: filter by channel, status, date range, min duration, free-text; thumbnail grid or list; sort; click → detail.
|
||||
4. **Video detail** — metadata header, chapters sidebar, full transcript reader with per-segment timestamps, in-page term highlight, "open on YouTube at t", per-video export (srt/json), audio-download trigger.
|
||||
5. **Search** — FTS across all transcripts → result cards with highlighted snippet + timestamp deep-link into the video detail at that segment.
|
||||
6. **Analysis** — word cloud (canvas), top-N terms bar, term-over-time timeline.
|
||||
7. **Scrape console** — launch a job (pick channel + options: limit, since, languages, shorts, which cookie), live progress via SSE, streaming log, cancel button, watch-mode toggle.
|
||||
8. **Cookie vault** — drag-and-drop zone (see §8), vault list, test-session.
|
||||
9. **Exports** — pick format + scope → download.
|
||||
|
||||
### 7.3 "Intelligent" without AI
|
||||
Smart faceted filters, cross-channel term mining, term-over-time timelines, saved searches (localStorage), live monitoring + auto-processing, auto-suggestions from top terms. Feels like a command center for content-creator data mining.
|
||||
|
||||
---
|
||||
|
||||
## 8. Cookie drag-and-drop UX
|
||||
|
||||
- Drop zone accepts **multiple `.txt` files** *or* **pasted Netscape cookie text**.
|
||||
- On upload (`POST /api/cookies/upload`): `cookies.parse_netscape()` validates the format, extracts the earliest expiry, detects session cookies (`SID`, `SAPISID`, `__Secure-3PSID`, `LOGIN_INFO`), and returns a card: `{label, expires_at, has_session, cookie_count, size}`.
|
||||
- Vault list with **radio = active** (exactly one active globally), rename, delete.
|
||||
- The active cookie path is passed to yt-dlp as `cookiefile` for every scrape (members-only, age-gate, etc.).
|
||||
- **"Test session"** → lightweight yt-dlp call (fetch the logged-in user's identity / a known members-only-free check) → reports `OK / expired / weak`.
|
||||
- The existing `cookies/253506e4-...txt` is auto-imported into the vault on first server start.
|
||||
|
||||
---
|
||||
|
||||
## 9. Launcher `.bat` (automatic port system)
|
||||
|
||||
PowerShell-assisted `.bat` files at the repo root.
|
||||
|
||||
- **`start-server.bat`**
|
||||
1. Resolve project root (script dir).
|
||||
2. Find a free TCP port: scan from `8000` upward until `bind` succeeds (PowerShell `TcpListener` probe).
|
||||
3. Write `port`, `PID`, and ISO timestamp to `.run/server.info`.
|
||||
4. Launch `python -m uvicorn yt_scraper.webapp.app:app --port <port>` via `start` (minimized).
|
||||
5. Poll `http://localhost:<port>/healthz` until 200 (or timeout ~30s).
|
||||
6. Open `http://localhost:<port>` in the default browser.
|
||||
7. Echo a clear success banner (port, URL) or an error.
|
||||
8. Idempotent: if `.run/server.info` already describes a live server, just re-open the browser.
|
||||
- **`stop-server.bat`**
|
||||
1. Read `.run/server.info`.
|
||||
2. Graceful terminate the uvicorn process (taskkill by PID, then `/T` for children).
|
||||
3. Confirm the port is freed (re-probe).
|
||||
4. Remove `.run/server.info`.
|
||||
5. Echo a clear banner.
|
||||
|
||||
`.run/` is added to `.gitignore`.
|
||||
|
||||
---
|
||||
|
||||
## 10. Subagent + sub-goal
|
||||
|
||||
- **`.opencode/agent/webapp-builder.md`** — an opencode agent specialized for this webapp. It knows: the FastAPI + Alpine + Tailwind (CDN, no-build) stack; the `Store` API surface; the design language (§7.2); and its verification steps (start server, `curl /healthz`, smoke-test each SPA view, run `pytest`). It receives Workstream B scope and is expected to reuse `Store`, `segments`, `analysis`, `cookies` rather than duplicating logic.
|
||||
- **`.opencode/goals/webapp-build.md`** — the sub-goal: "Implement the local webapp (Workstream B) per `docs/superpowers/specs/2026-07-26-platform-design.md`. Build backend routers, SSE job runner, and the 9-view Alpine SPA. Verify with `/healthz`, smoke tests, and `pytest`."
|
||||
|
||||
---
|
||||
|
||||
## 11. Dependencies
|
||||
|
||||
Add to `pyproject.toml` `[project.optional-dependencies]`:
|
||||
```
|
||||
web = ["fastapi>=0.110", "uvicorn[standard]>=0.27"]
|
||||
analysis = ["matplotlib>=3.8", "wordcloud>=1.9"]
|
||||
```
|
||||
System requirement for `audio`: `ffmpeg` (`winget install ffmpeg` or `choco install ffmpeg`). The CLI will detect ffmpeg and print a clear hint if missing.
|
||||
|
||||
`cookies/` is added to `.gitignore` (real session secrets must never be committed).
|
||||
|
||||
---
|
||||
|
||||
## 12. Build order
|
||||
|
||||
A → B → C:
|
||||
1. **A1** Schema migration + backfill + fix cookies bug.
|
||||
2. **A2** `segments`/`export`/`analysis`/`monitor`/`cookies` modules.
|
||||
3. **A3** CLI subcommands (`search`, `export`, `audio`, `channels`, `watch`, `analyze`, `re-render`).
|
||||
4. **A4** Tests; `pytest` green.
|
||||
5. **B** Webapp (subagent) — backend routers, job runner, SPA, drag-drop cookies. Verify `/healthz` + smoke.
|
||||
6. **C** `.bat` launcher + `.opencode` agent/goal files.
|
||||
|
||||
---
|
||||
|
||||
## 13. Testing & verification
|
||||
|
||||
- `pytest tests/ -v` covering: schema migration idempotency, FTS search, segment backfill from `.md`, cookie Netscape parse/validate, exporters (json/csv/srt/html), analysis frequency/timeline, multi-channel store ops.
|
||||
- Webapp: `/healthz` 200; representative endpoint per router; SPA loads; SSE streams a fake job.
|
||||
- Manual: run `start-server.bat`, confirm browser opens, drag-drop the real cookie, trigger a small scrape (`--limit 1`) against the existing Nostal Vlad channel, verify a new `.md` + FTS row appear and search returns the new content.
|
||||
- `python -m yt_scraper.cli search "..."` returns hits after backfill.
|
||||
|
||||
---
|
||||
|
||||
## 14. Out of scope (explicitly excluded)
|
||||
|
||||
- AI features: LLM summaries, Q&A/RAG, NER (spaCy), speaker diarization (pyannote).
|
||||
- CLI `stats`, `clip`, `thumbnails`, `translation` (dropped per user scope decision — aggregate *queries* still exist internally to feed the dashboard).
|
||||
- Remote deployment, auth, multi-user, cloud sync.
|
||||
Reference in New Issue
Block a user