feat: local content-mining platform for YouTube creator scraping

Scraper de canales de YouTube hacia notas Markdown para base de conocimiento
(Obsidian-ready), con plataforma web local.

Engine + CLI (Workstream A):
- Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit
- SQLite store con migración idempotente: FTS5 (transcript search), columnas
  de metadata enriquecida, tablas cookies_meta y scrape_jobs
- Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html),
  analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline
- CLI Click group: search, export, audio, channels, watch, analyze, re-render
- Fix del bug de scoping de cookies en cli.py

Webapp local (Workstream B):
- FastAPI backend: dashboard, channels, videos facetado, transcript, search
  FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports,
  folders (abrir en OS), tools (re-render, formato)
- SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema
  dark command-center con acento rojo→rosa, cookie vault drag-drop,
  consola de scrapeo con progreso live vía SSE

Launcher + subagentes (Workstream C):
- start-server.bat / stop-server.bat con auto port-scan + browser open
- .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md

Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).
This commit is contained in:
urieljareth
2026-07-26 23:19:34 -06:00
commit 621bbc5f5c
45 changed files with 6541 additions and 0 deletions
+44
View File
@@ -0,0 +1,44 @@
---
description: Builds and maintains the yt-channel-scraper local webapp (FastAPI backend + Alpine.js SPA). Use when implementing, fixing, or extending the webapp at src/yt_scraper/webapp/.
model: zai-coding-plan/glm-5.2
---
# webapp-builder
You are the specialist agent for the **yt-channel-scraper local webapp** (Workstream B of the platform).
## Mission
Implement and maintain the local web application that turns the scraper into a "data-mining command center" for content-creator data. You build on top of the already-working engine modules — you do **not** re-implement them.
## Stack (fixed — do not deviate)
- **Backend:** FastAPI. App factory at `src/yt_scraper/webapp/app.py`. Routers under `/api`. SSE via `sse_starlette`.
- **Frontend:** Single-page app, **no build step**. `index.html` + `app.js` + `styles.css` served as static files by the same FastAPI process. Alpine.js + Tailwind CSS (both via CDN, no npm). Chart.js (CDN) for visualizations.
- **No external services, no auth, no network egress except YouTube via yt-dlp.** This is a 100% local platform.
## What already exists — REUSE, do not duplicate
Read these before writing code. They are your data layer:
- `src/yt_scraper/store.py``Store(db_path)` with all SQL: `dashboard()`, `query_videos(...)`, `get_video(id)`, `get_segments(id)`, `search_segments(q, channel_id, limit)`, `list_channels()`, `list_cookies()`, `set_active_cookie(id)`, `get_active_cookie()`, `create_job/update_job/get_job/list_jobs`, `stats()`.
- `src/yt_scraper/cookies.py``import_text(store, text, label)`, `import_file`, `list_vault`, `set_active`, `delete`, `resolve_active_path`, `parse_netscape`, `auto_import_dir`, `is_expired`.
- `src/yt_scraper/segments.py``search(store, q, channel_id, limit)`, `backfill_from_markdown`, `parse_markdown`.
- `src/yt_scraper/analysis.py``word_frequency`, `top_words`, `term_timeline`, `render_wordcloud`, `render_top_words_chart`, `render_timeline_chart`.
- `src/yt_scraper/export.py``export_json/csv/srt/srt_video/html`.
- `src/yt_scraper/pipeline.py``process_video(row, cfg, store, channel_name, channel_id, channel_url, cookies_file=, cookies_from_browser=, on_log=)` for the scrape job runner.
- `src/yt_scraper/config.py``Config`, `load_config(path)`.
The active cookie (`cookies.resolve_active_path(store)`) MUST be passed as `cookies_file` to every scrape job.
## Design language (non-negotiable visual identity)
Dark "data command center": near-black canvas (`zinc-950` / `#0a0a0f`), glassy cards (`backdrop-blur`, `border-zinc-800`), **red→rose accent gradient** (`from-rose-500 to-red-600`, e.g. `#f43f5e`), monospace accents for data/numbers, smooth Alpine `x-transition`. Responsive. Charts via Chart.js with theme-matched colors (grid `#27272a`, text `#a1a1aa`).
## Required SPA views (hash routes)
Dashboard, Channels, Videos (faceted), Video detail (transcript reader + chapters), Search (FTS + snippet + deep-link), Analysis (wordcloud/topwords/timeline), Scrape console (launch job + SSE progress + cancel + watch toggle), Cookie vault (drag-and-drop + activate + test), Exports.
## Verification you MUST run before declaring done
1. `python -c "from yt_scraper.webapp.app import app; print('app ok')"`
2. Start server: `python -m uvicorn yt_scraper.webapp.app:app --port 8765` and `curl http://localhost:8765/healthz` → expect `{"status":"ok"}`.
3. Smoke each router: `/api/dashboard`, `/api/channels`, `/api/videos?size=5`, `/api/search?q=vaporwave`, `/api/cookies`.
4. Load `/` → SPA renders, navigation works.
5. `python -m pytest tests/ -q` stays green.
6. Confirm the real cookie at `cookies/253506e4-...txt` is auto-imported and shown as active with `has_session=true`.
Reference the full spec at `docs/superpowers/specs/2026-07-26-platform-design.md` and plan at `docs/superpowers/plans/2026-07-26-platform-build.md`.
+31
View File
@@ -0,0 +1,31 @@
# Sub-goal: Build the local webapp (Workstream B)
**Parent goal:** Convert yt-channel-scraper into a local content-mining platform.
**Owner agent:** `webapp-builder` (`.opencode/agent/webapp-builder.md`)
**Spec:** `docs/superpowers/specs/2026-07-26-platform-design.md` (§7 Webapp, §8 Cookies, §7.2 design language)
**Status:** Workstream A (engine + CLI) is DONE and verified — `pytest` green, real DB backfilled, FTS search working, cookie vault auto-imports the real cookie.
## Objective
Implement the FastAPI backend + Alpine.js SPA so the platform is usable from a browser, launched by `start-server.bat`. No Node/npm build step.
## Deliverables
1. `src/yt_scraper/webapp/__init__.py`
2. `src/yt_scraper/webapp/app.py` — FastAPI factory, `/healthz`, static mount, auto-import cookies on startup.
3. `src/yt_scraper/webapp/jobs.py` — single-worker scrape job runner with SSE event bus (queue/progress/log/cancel), persists to `scrape_jobs`.
4. `src/yt_scraper/webapp/api.py` — REST routers documented in spec §7.1 (dashboard, channels, videos, search, analysis, scrape+SSE, cookies with drag-and-drop upload, exports).
5. `src/yt_scraper/webapp/static/index.html`, `app.js`, `styles.css` — the 9-view SPA, dark command-center theme, drag-and-drop cookie vault, Chart.js viz.
## API contract (the SPA consumes exactly this)
- `GET /healthz``{"status":"ok"}`
- `GET /api/dashboard``{channels[], status_breakdown{}, uploads_over_time[], duration_histogram[], top_tags[]}`
- `GET /api/channels` | `POST /api/channels {url}` | `DELETE /api/channels/{id}`
- `GET /api/videos?channel=&status=&from=&to=&min_dur=&q=&sort=&page=&size=``{items[], total, page, size}`
- `GET /api/videos/{id}` | `GET /api/videos/{id}/transcript``[{idx,start_sec,end_sec,text}]`
- `GET /api/search?q=&channel=&limit=``[{video_id,title,start_sec,snippet}]`
- `GET /api/analysis?channel=&type=topwords|wordcloud|timeline&term=` → chart-ready JSON
- `POST /api/scrape {channel_id,opts}``{job_id}` | `GET /api/scrape/{id}/stream` (SSE) | `DELETE /api/scrape/{id}`
- `POST /api/cookies/upload` (multipart file(s) or `{text,label}` JSON) | `GET /api/cookies` | `POST /api/cookies/{id}/activate` | `DELETE /api/cookies/{id}` | `POST /api/cookies/{id}/test`
- `GET /api/export?format=json|csv|srt|html&channel=` → file stream
## Done when
- `/healthz` returns 200; every router smokes green against the real DB; SPA renders all 9 views; SSE streams a scrape job; the real cookie drag-and-drops and activates; `pytest` green.