feat: local content-mining platform for YouTube creator scraping

Scraper de canales de YouTube hacia notas Markdown para base de conocimiento
(Obsidian-ready), con plataforma web local.

Engine + CLI (Workstream A):
- Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit
- SQLite store con migración idempotente: FTS5 (transcript search), columnas
  de metadata enriquecida, tablas cookies_meta y scrape_jobs
- Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html),
  analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline
- CLI Click group: search, export, audio, channels, watch, analyze, re-render
- Fix del bug de scoping de cookies en cli.py

Webapp local (Workstream B):
- FastAPI backend: dashboard, channels, videos facetado, transcript, search
  FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports,
  folders (abrir en OS), tools (re-render, formato)
- SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema
  dark command-center con acento rojo→rosa, cookie vault drag-drop,
  consola de scrapeo con progreso live vía SSE

Launcher + subagentes (Workstream C):
- start-server.bat / stop-server.bat con auto port-scan + browser open
- .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md

Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).
This commit is contained in:
urieljareth
2026-07-26 23:19:34 -06:00
commit 621bbc5f5c
45 changed files with 6541 additions and 0 deletions
+44
View File
@@ -0,0 +1,44 @@
---
description: Builds and maintains the yt-channel-scraper local webapp (FastAPI backend + Alpine.js SPA). Use when implementing, fixing, or extending the webapp at src/yt_scraper/webapp/.
model: zai-coding-plan/glm-5.2
---
# webapp-builder
You are the specialist agent for the **yt-channel-scraper local webapp** (Workstream B of the platform).
## Mission
Implement and maintain the local web application that turns the scraper into a "data-mining command center" for content-creator data. You build on top of the already-working engine modules — you do **not** re-implement them.
## Stack (fixed — do not deviate)
- **Backend:** FastAPI. App factory at `src/yt_scraper/webapp/app.py`. Routers under `/api`. SSE via `sse_starlette`.
- **Frontend:** Single-page app, **no build step**. `index.html` + `app.js` + `styles.css` served as static files by the same FastAPI process. Alpine.js + Tailwind CSS (both via CDN, no npm). Chart.js (CDN) for visualizations.
- **No external services, no auth, no network egress except YouTube via yt-dlp.** This is a 100% local platform.
## What already exists — REUSE, do not duplicate
Read these before writing code. They are your data layer:
- `src/yt_scraper/store.py``Store(db_path)` with all SQL: `dashboard()`, `query_videos(...)`, `get_video(id)`, `get_segments(id)`, `search_segments(q, channel_id, limit)`, `list_channels()`, `list_cookies()`, `set_active_cookie(id)`, `get_active_cookie()`, `create_job/update_job/get_job/list_jobs`, `stats()`.
- `src/yt_scraper/cookies.py``import_text(store, text, label)`, `import_file`, `list_vault`, `set_active`, `delete`, `resolve_active_path`, `parse_netscape`, `auto_import_dir`, `is_expired`.
- `src/yt_scraper/segments.py``search(store, q, channel_id, limit)`, `backfill_from_markdown`, `parse_markdown`.
- `src/yt_scraper/analysis.py``word_frequency`, `top_words`, `term_timeline`, `render_wordcloud`, `render_top_words_chart`, `render_timeline_chart`.
- `src/yt_scraper/export.py``export_json/csv/srt/srt_video/html`.
- `src/yt_scraper/pipeline.py``process_video(row, cfg, store, channel_name, channel_id, channel_url, cookies_file=, cookies_from_browser=, on_log=)` for the scrape job runner.
- `src/yt_scraper/config.py``Config`, `load_config(path)`.
The active cookie (`cookies.resolve_active_path(store)`) MUST be passed as `cookies_file` to every scrape job.
## Design language (non-negotiable visual identity)
Dark "data command center": near-black canvas (`zinc-950` / `#0a0a0f`), glassy cards (`backdrop-blur`, `border-zinc-800`), **red→rose accent gradient** (`from-rose-500 to-red-600`, e.g. `#f43f5e`), monospace accents for data/numbers, smooth Alpine `x-transition`. Responsive. Charts via Chart.js with theme-matched colors (grid `#27272a`, text `#a1a1aa`).
## Required SPA views (hash routes)
Dashboard, Channels, Videos (faceted), Video detail (transcript reader + chapters), Search (FTS + snippet + deep-link), Analysis (wordcloud/topwords/timeline), Scrape console (launch job + SSE progress + cancel + watch toggle), Cookie vault (drag-and-drop + activate + test), Exports.
## Verification you MUST run before declaring done
1. `python -c "from yt_scraper.webapp.app import app; print('app ok')"`
2. Start server: `python -m uvicorn yt_scraper.webapp.app:app --port 8765` and `curl http://localhost:8765/healthz` → expect `{"status":"ok"}`.
3. Smoke each router: `/api/dashboard`, `/api/channels`, `/api/videos?size=5`, `/api/search?q=vaporwave`, `/api/cookies`.
4. Load `/` → SPA renders, navigation works.
5. `python -m pytest tests/ -q` stays green.
6. Confirm the real cookie at `cookies/253506e4-...txt` is auto-imported and shown as active with `has_session=true`.
Reference the full spec at `docs/superpowers/specs/2026-07-26-platform-design.md` and plan at `docs/superpowers/plans/2026-07-26-platform-build.md`.