urieljareth
621bbc5f5c
Scraper de canales de YouTube hacia notas Markdown para base de conocimiento (Obsidian-ready), con plataforma web local. Engine + CLI (Workstream A): - Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit - SQLite store con migración idempotente: FTS5 (transcript search), columnas de metadata enriquecida, tablas cookies_meta y scrape_jobs - Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html), analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline - CLI Click group: search, export, audio, channels, watch, analyze, re-render - Fix del bug de scoping de cookies en cli.py Webapp local (Workstream B): - FastAPI backend: dashboard, channels, videos facetado, transcript, search FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports, folders (abrir en OS), tools (re-render, formato) - SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema dark command-center con acento rojo→rosa, cookie vault drag-drop, consola de scrapeo con progreso live vía SSE Launcher + subagentes (Workstream C): - start-server.bat / stop-server.bat con auto port-scan + browser open - .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).
123 lines
7.1 KiB
Markdown
123 lines
7.1 KiB
Markdown
# yt-scraper Platform Implementation Plan
|
|
|
|
> **Execution mode:** Subagent-driven + inline, autonomous, self-iterating. User mandated "completa todo el proceso de manera agentica y subagentica creando y resolviendo todos los problemas y dudas en un loop auto iterativo." Proceed without further checkpoints; verify with tests and fix in a loop until green.
|
|
|
|
**Goal:** Turn the CLI scraper into a local content-mining platform: CLI features (no AI) + FastAPI/Alpine webapp + .bat launcher + opencode subagent.
|
|
|
|
**Architecture:** Monorepo. `Store` extends with non-destructive migration (segments FTS5 + metadata + cookies + jobs). New modules `segments/cookies/export/analysis/monitor`. FastAPI app at `src/yt_scraper/webapp/` serves a no-build Alpine SPA. `.bat` launcher auto-scans ports.
|
|
|
|
**Tech:** Python ≥3.10, FastAPI, uvicorn, Alpine.js + Tailwind + Chart.js (CDN), SQLite FTS5, yt-dlp, matplotlib, Jinja2.
|
|
|
|
**Spec:** `docs/superpowers/specs/2026-07-26-platform-design.md`
|
|
|
|
---
|
|
|
|
## File map & responsibilities
|
|
|
|
| File | Responsibility | Status |
|
|
|---|---|---|
|
|
| `src/yt_scraper/store.py` | Add migration (new cols + 3 new tables), segment/cookie/job queries | modify |
|
|
| `src/yt_scraper/segments.py` | `store_segments`, `populate_fts`, `search`, `backfill_from_markdown` | new |
|
|
| `src/yt_scraper/cookies.py` | Netscape parse/validate, vault CRUD, `resolve_active_path` | new |
|
|
| `src/yt_scraper/export.py` | json/csv/srt/html exporters | new |
|
|
| `src/yt_scraper/analysis.py` | word freq / top-N / timeline (data + optional matplotlib) | new |
|
|
| `src/yt_scraper/monitor.py` | `watch_loop(config, interval, once)` | new |
|
|
| `src/yt_scraper/cli.py` | Click group + subcommands; fix cookies bug | modify |
|
|
| `src/yt_scraper/extract.py` | unchanged (already accepts cookies) | - |
|
|
| `src/yt_scraper/webapp/app.py` | FastAPI factory + static + healthz | new |
|
|
| `src/yt_scraper/webapp/api.py` | REST routers | new |
|
|
| `src/yt_scraper/webapp/jobs.py` | scrape job runner + SSE bus | new |
|
|
| `src/yt_scraper/webapp/static/{index.html,app.js,styles.css}` | SPA | new |
|
|
| `templates/gallery.html.j2` | HTML export gallery | new |
|
|
| `start-server.bat`, `stop-server.bat` | launcher | new |
|
|
| `.opencode/agent/webapp-builder.md`, `.opencode/goals/webapp-build.md` | subagent | new |
|
|
| `tests/` | cover migration, segments, cookies, export, analysis, cli | new/extend |
|
|
|
|
---
|
|
|
|
## Key interfaces (contract between modules)
|
|
|
|
```python
|
|
# segments.py
|
|
@dataclass
|
|
class Hit: video_id: str; channel_id: str; title: str; start_sec: float; end_sec: float; snippet: str; rank: float
|
|
def store_segments(store, video_id, segments: list[Segment]) -> None
|
|
def search(store, query: str, channel_id: str|None=None, limit: int=50) -> list[Hit]
|
|
def backfill_from_markdown(store, md_root: Path, log=print) -> int # returns n videos
|
|
|
|
# cookies.py
|
|
@dataclass
|
|
class CookieMeta: id:str; filename:str; label:str|None; added_at:str; expires_at:str|None; is_active:bool; has_session:bool; cookie_count:int
|
|
def parse_netscape(path) -> tuple[bool, dict] # (ok, {expires_at, has_session, count})
|
|
def import_file(store, path) -> str # id
|
|
def import_text(store, text, label) -> str
|
|
def set_active(store, cookie_id) -> None
|
|
def list_vault(store) -> list[CookieMeta]
|
|
def delete(store, cookie_id) -> None
|
|
def resolve_active_path(store) -> str|None
|
|
def auto_import_dir(store, dir_path) -> None # import loose .txt in cookies/
|
|
|
|
# export.py
|
|
def export_json(store, segments_store_fn, channel_id, out_path) -> Path
|
|
def export_csv(store, channel_id, out_path) -> Path
|
|
def export_srt(store, video_id, out_path) -> Path
|
|
def export_html(store, channel_id, out_path, template_path) -> Path
|
|
|
|
# analysis.py
|
|
def word_frequency(store, channel_id=None) -> dict[str,int]
|
|
def top_words(store, n=50, channel_id=None) -> list[tuple[str,int]]
|
|
def term_timeline(store, term, channel_id=None) -> list[tuple[str,int]] # (YYYY-MM, count)
|
|
def render_wordcloud(freq, out_path) -> None # matplotlib
|
|
|
|
# monitor.py
|
|
def watch_loop(cfg, interval_seconds, channel_id=None, once=False) -> None
|
|
```
|
|
|
|
---
|
|
|
|
## Tricky bits (resolved decisions)
|
|
|
|
- **FTS5 sync:** manual `populate_fts` (delete+insert) after `store_segments`. No triggers (simpler, idempotent). FTS uses external-content `transcript_fts` with `content='transcript_segments'`; queries via `MATCH` + join.
|
|
- **Migration:** `ALTER TABLE ADD COLUMN` is guarded by `PRAGMA table_info` checks (idempotent). New tables via `CREATE TABLE IF NOT EXISTS`.
|
|
- **Backfill parse:** re-use existing `.md` format: lines `**MM:SS** · text` under `### Title (MM:SS)` chapter headers; frontmatter between `---` fences has YAML metadata. Reuse `parse.Segment` + `chapters.Section`.
|
|
- **Cookie active:** single row `is_active=1`; `set_active` flips all others to 0 in one tx.
|
|
- **Click structure:** convert top-level `@click.command` to `@click.group` with the original scrape flow under `scrape` command (default invocation preserved via a wrapper), plus `search/export/audio/channels/watch/analyze/re-render`.
|
|
- **Port scan:** PowerShell `System.Net.Sockets.TcpListener` bind-test 8000..8100.
|
|
- **SSE:** `EventSourceResponse`-style via `sse-starlette` OR a hand-rolled streaming `Response` with `text/event-stream`. Use `sse-starlette` (add to deps) — cleaner.
|
|
|
|
---
|
|
|
|
## Task list (build order)
|
|
|
|
### A1 — Foundation
|
|
- T1: Store migration (new cols + `transcript_segments` + `transcript_fts` + `cookies_meta` + `scrape_jobs`) + queries. Test: migration idempotent, columns exist, tables exist.
|
|
- T2: `segments.py` (`store_segments`, `populate_fts`, `search`). Test: store + search returns Hit.
|
|
- T3: `cookies.py`. Test: parse real cookie file, validate, active toggling, resolve path.
|
|
- T4: Fix cli.py cookies bug (thread cookies through `_process_one`; fallback to active vault cookie).
|
|
- T5: `backfill_from_markdown`. Test: parse one real `.md`, assert segments inserted.
|
|
|
|
### A2 — Feature modules
|
|
- T6: `export.py` (json/csv/srt/html) + `gallery.html.j2`. Test: each format round-trips.
|
|
- T7: `analysis.py` (freq, top, timeline, wordcloud). Test: counts on synthetic segments.
|
|
- T8: `monitor.py` (`watch_loop`, `--once`). Test: single iteration with mocked discover.
|
|
|
|
### A3 — CLI wiring
|
|
- T9: Convert `cli.py` to Click group; add `search`, `export`, `audio`, `channels`, `watch`, `analyze`, `re-render`, `--all-channels`. Smoke-test each via `CliRunner`.
|
|
|
|
### A4 — Tests green
|
|
- T10: Full `pytest` green; integration tests marked.
|
|
|
|
### B — Webapp (subagent: `webapp-builder`)
|
|
- T11: `.opencode/agent/webapp-builder.md` + `.opencode/goals/webapp-build.md`.
|
|
- T12: webapp backend (`app.py`, `api.py`, `jobs.py`) — all routers + SSE.
|
|
- T13: SPA (`index.html`, `app.js`, `styles.css`) — 9 views, dark command-center, drag-drop cookies, Chart.js, SSE scrape console.
|
|
- T14: Verify `/healthz` + smoke each endpoint; auto-import real cookie on startup.
|
|
|
|
### C — Launcher + deps
|
|
- T15: `pyproject.toml` optional deps (`web`, `analysis`); `.gitignore` (`cookies/`, `.run/`, `data/analysis/`).
|
|
- T16: `start-server.bat` + `stop-server.bat` (PowerShell port scan).
|
|
- T17: End-to-end: `start-server.bat` → browser → `/healthz` 200 → dashboard loads.
|
|
|
|
### Verification loop
|
|
- Run `pytest`, fix until green. Run `start-server.bat`, confirm health + SPA. Iterate.
|