urieljareth
621bbc5f5c
Scraper de canales de YouTube hacia notas Markdown para base de conocimiento (Obsidian-ready), con plataforma web local. Engine + CLI (Workstream A): - Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit - SQLite store con migración idempotente: FTS5 (transcript search), columnas de metadata enriquecida, tablas cookies_meta y scrape_jobs - Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html), analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline - CLI Click group: search, export, audio, channels, watch, analyze, re-render - Fix del bug de scoping de cookies en cli.py Webapp local (Workstream B): - FastAPI backend: dashboard, channels, videos facetado, transcript, search FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports, folders (abrir en OS), tools (re-render, formato) - SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema dark command-center con acento rojo→rosa, cookie vault drag-drop, consola de scrapeo con progreso live vía SSE Launcher + subagentes (Workstream C): - start-server.bat / stop-server.bat con auto port-scan + browser open - .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).
32 lines
2.5 KiB
Markdown
32 lines
2.5 KiB
Markdown
# Sub-goal: Build the local webapp (Workstream B)
|
|
|
|
**Parent goal:** Convert yt-channel-scraper into a local content-mining platform.
|
|
**Owner agent:** `webapp-builder` (`.opencode/agent/webapp-builder.md`)
|
|
**Spec:** `docs/superpowers/specs/2026-07-26-platform-design.md` (§7 Webapp, §8 Cookies, §7.2 design language)
|
|
**Status:** Workstream A (engine + CLI) is DONE and verified — `pytest` green, real DB backfilled, FTS search working, cookie vault auto-imports the real cookie.
|
|
|
|
## Objective
|
|
Implement the FastAPI backend + Alpine.js SPA so the platform is usable from a browser, launched by `start-server.bat`. No Node/npm build step.
|
|
|
|
## Deliverables
|
|
1. `src/yt_scraper/webapp/__init__.py`
|
|
2. `src/yt_scraper/webapp/app.py` — FastAPI factory, `/healthz`, static mount, auto-import cookies on startup.
|
|
3. `src/yt_scraper/webapp/jobs.py` — single-worker scrape job runner with SSE event bus (queue/progress/log/cancel), persists to `scrape_jobs`.
|
|
4. `src/yt_scraper/webapp/api.py` — REST routers documented in spec §7.1 (dashboard, channels, videos, search, analysis, scrape+SSE, cookies with drag-and-drop upload, exports).
|
|
5. `src/yt_scraper/webapp/static/index.html`, `app.js`, `styles.css` — the 9-view SPA, dark command-center theme, drag-and-drop cookie vault, Chart.js viz.
|
|
|
|
## API contract (the SPA consumes exactly this)
|
|
- `GET /healthz` → `{"status":"ok"}`
|
|
- `GET /api/dashboard` → `{channels[], status_breakdown{}, uploads_over_time[], duration_histogram[], top_tags[]}`
|
|
- `GET /api/channels` | `POST /api/channels {url}` | `DELETE /api/channels/{id}`
|
|
- `GET /api/videos?channel=&status=&from=&to=&min_dur=&q=&sort=&page=&size=` → `{items[], total, page, size}`
|
|
- `GET /api/videos/{id}` | `GET /api/videos/{id}/transcript` → `[{idx,start_sec,end_sec,text}]`
|
|
- `GET /api/search?q=&channel=&limit=` → `[{video_id,title,start_sec,snippet}]`
|
|
- `GET /api/analysis?channel=&type=topwords|wordcloud|timeline&term=` → chart-ready JSON
|
|
- `POST /api/scrape {channel_id,opts}` → `{job_id}` | `GET /api/scrape/{id}/stream` (SSE) | `DELETE /api/scrape/{id}`
|
|
- `POST /api/cookies/upload` (multipart file(s) or `{text,label}` JSON) | `GET /api/cookies` | `POST /api/cookies/{id}/activate` | `DELETE /api/cookies/{id}` | `POST /api/cookies/{id}/test`
|
|
- `GET /api/export?format=json|csv|srt|html&channel=` → file stream
|
|
|
|
## Done when
|
|
- `/healthz` returns 200; every router smokes green against the real DB; SPA renders all 9 views; SSE streams a scrape job; the real cookie drag-and-drops and activates; `pytest` green.
|