feat: local content-mining platform for YouTube creator scraping
Scraper de canales de YouTube hacia notas Markdown para base de conocimiento (Obsidian-ready), con plataforma web local. Engine + CLI (Workstream A): - Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit - SQLite store con migración idempotente: FTS5 (transcript search), columnas de metadata enriquecida, tablas cookies_meta y scrape_jobs - Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html), analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline - CLI Click group: search, export, audio, channels, watch, analyze, re-render - Fix del bug de scoping de cookies en cli.py Webapp local (Workstream B): - FastAPI backend: dashboard, channels, videos facetado, transcript, search FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports, folders (abrir en OS), tools (re-render, formato) - SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema dark command-center con acento rojo→rosa, cookie vault drag-drop, consola de scrapeo con progreso live vía SSE Launcher + subagentes (Workstream C): - start-server.bat / stop-server.bat con auto port-scan + browser open - .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).
This commit is contained in:
@@ -0,0 +1,44 @@
|
||||
---
|
||||
description: Builds and maintains the yt-channel-scraper local webapp (FastAPI backend + Alpine.js SPA). Use when implementing, fixing, or extending the webapp at src/yt_scraper/webapp/.
|
||||
model: zai-coding-plan/glm-5.2
|
||||
---
|
||||
|
||||
# webapp-builder
|
||||
|
||||
You are the specialist agent for the **yt-channel-scraper local webapp** (Workstream B of the platform).
|
||||
|
||||
## Mission
|
||||
Implement and maintain the local web application that turns the scraper into a "data-mining command center" for content-creator data. You build on top of the already-working engine modules — you do **not** re-implement them.
|
||||
|
||||
## Stack (fixed — do not deviate)
|
||||
- **Backend:** FastAPI. App factory at `src/yt_scraper/webapp/app.py`. Routers under `/api`. SSE via `sse_starlette`.
|
||||
- **Frontend:** Single-page app, **no build step**. `index.html` + `app.js` + `styles.css` served as static files by the same FastAPI process. Alpine.js + Tailwind CSS (both via CDN, no npm). Chart.js (CDN) for visualizations.
|
||||
- **No external services, no auth, no network egress except YouTube via yt-dlp.** This is a 100% local platform.
|
||||
|
||||
## What already exists — REUSE, do not duplicate
|
||||
Read these before writing code. They are your data layer:
|
||||
- `src/yt_scraper/store.py` — `Store(db_path)` with all SQL: `dashboard()`, `query_videos(...)`, `get_video(id)`, `get_segments(id)`, `search_segments(q, channel_id, limit)`, `list_channels()`, `list_cookies()`, `set_active_cookie(id)`, `get_active_cookie()`, `create_job/update_job/get_job/list_jobs`, `stats()`.
|
||||
- `src/yt_scraper/cookies.py` — `import_text(store, text, label)`, `import_file`, `list_vault`, `set_active`, `delete`, `resolve_active_path`, `parse_netscape`, `auto_import_dir`, `is_expired`.
|
||||
- `src/yt_scraper/segments.py` — `search(store, q, channel_id, limit)`, `backfill_from_markdown`, `parse_markdown`.
|
||||
- `src/yt_scraper/analysis.py` — `word_frequency`, `top_words`, `term_timeline`, `render_wordcloud`, `render_top_words_chart`, `render_timeline_chart`.
|
||||
- `src/yt_scraper/export.py` — `export_json/csv/srt/srt_video/html`.
|
||||
- `src/yt_scraper/pipeline.py` — `process_video(row, cfg, store, channel_name, channel_id, channel_url, cookies_file=, cookies_from_browser=, on_log=)` for the scrape job runner.
|
||||
- `src/yt_scraper/config.py` — `Config`, `load_config(path)`.
|
||||
|
||||
The active cookie (`cookies.resolve_active_path(store)`) MUST be passed as `cookies_file` to every scrape job.
|
||||
|
||||
## Design language (non-negotiable visual identity)
|
||||
Dark "data command center": near-black canvas (`zinc-950` / `#0a0a0f`), glassy cards (`backdrop-blur`, `border-zinc-800`), **red→rose accent gradient** (`from-rose-500 to-red-600`, e.g. `#f43f5e`), monospace accents for data/numbers, smooth Alpine `x-transition`. Responsive. Charts via Chart.js with theme-matched colors (grid `#27272a`, text `#a1a1aa`).
|
||||
|
||||
## Required SPA views (hash routes)
|
||||
Dashboard, Channels, Videos (faceted), Video detail (transcript reader + chapters), Search (FTS + snippet + deep-link), Analysis (wordcloud/topwords/timeline), Scrape console (launch job + SSE progress + cancel + watch toggle), Cookie vault (drag-and-drop + activate + test), Exports.
|
||||
|
||||
## Verification you MUST run before declaring done
|
||||
1. `python -c "from yt_scraper.webapp.app import app; print('app ok')"`
|
||||
2. Start server: `python -m uvicorn yt_scraper.webapp.app:app --port 8765` and `curl http://localhost:8765/healthz` → expect `{"status":"ok"}`.
|
||||
3. Smoke each router: `/api/dashboard`, `/api/channels`, `/api/videos?size=5`, `/api/search?q=vaporwave`, `/api/cookies`.
|
||||
4. Load `/` → SPA renders, navigation works.
|
||||
5. `python -m pytest tests/ -q` stays green.
|
||||
6. Confirm the real cookie at `cookies/253506e4-...txt` is auto-imported and shown as active with `has_session=true`.
|
||||
|
||||
Reference the full spec at `docs/superpowers/specs/2026-07-26-platform-design.md` and plan at `docs/superpowers/plans/2026-07-26-platform-build.md`.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Sub-goal: Build the local webapp (Workstream B)
|
||||
|
||||
**Parent goal:** Convert yt-channel-scraper into a local content-mining platform.
|
||||
**Owner agent:** `webapp-builder` (`.opencode/agent/webapp-builder.md`)
|
||||
**Spec:** `docs/superpowers/specs/2026-07-26-platform-design.md` (§7 Webapp, §8 Cookies, §7.2 design language)
|
||||
**Status:** Workstream A (engine + CLI) is DONE and verified — `pytest` green, real DB backfilled, FTS search working, cookie vault auto-imports the real cookie.
|
||||
|
||||
## Objective
|
||||
Implement the FastAPI backend + Alpine.js SPA so the platform is usable from a browser, launched by `start-server.bat`. No Node/npm build step.
|
||||
|
||||
## Deliverables
|
||||
1. `src/yt_scraper/webapp/__init__.py`
|
||||
2. `src/yt_scraper/webapp/app.py` — FastAPI factory, `/healthz`, static mount, auto-import cookies on startup.
|
||||
3. `src/yt_scraper/webapp/jobs.py` — single-worker scrape job runner with SSE event bus (queue/progress/log/cancel), persists to `scrape_jobs`.
|
||||
4. `src/yt_scraper/webapp/api.py` — REST routers documented in spec §7.1 (dashboard, channels, videos, search, analysis, scrape+SSE, cookies with drag-and-drop upload, exports).
|
||||
5. `src/yt_scraper/webapp/static/index.html`, `app.js`, `styles.css` — the 9-view SPA, dark command-center theme, drag-and-drop cookie vault, Chart.js viz.
|
||||
|
||||
## API contract (the SPA consumes exactly this)
|
||||
- `GET /healthz` → `{"status":"ok"}`
|
||||
- `GET /api/dashboard` → `{channels[], status_breakdown{}, uploads_over_time[], duration_histogram[], top_tags[]}`
|
||||
- `GET /api/channels` | `POST /api/channels {url}` | `DELETE /api/channels/{id}`
|
||||
- `GET /api/videos?channel=&status=&from=&to=&min_dur=&q=&sort=&page=&size=` → `{items[], total, page, size}`
|
||||
- `GET /api/videos/{id}` | `GET /api/videos/{id}/transcript` → `[{idx,start_sec,end_sec,text}]`
|
||||
- `GET /api/search?q=&channel=&limit=` → `[{video_id,title,start_sec,snippet}]`
|
||||
- `GET /api/analysis?channel=&type=topwords|wordcloud|timeline&term=` → chart-ready JSON
|
||||
- `POST /api/scrape {channel_id,opts}` → `{job_id}` | `GET /api/scrape/{id}/stream` (SSE) | `DELETE /api/scrape/{id}`
|
||||
- `POST /api/cookies/upload` (multipart file(s) or `{text,label}` JSON) | `GET /api/cookies` | `POST /api/cookies/{id}/activate` | `DELETE /api/cookies/{id}` | `POST /api/cookies/{id}/test`
|
||||
- `GET /api/export?format=json|csv|srt|html&channel=` → file stream
|
||||
|
||||
## Done when
|
||||
- `/healthz` returns 200; every router smokes green against the real DB; SPA renders all 9 views; SSE streams a scrape job; the real cookie drag-and-drops and activates; `pytest` green.
|
||||
Reference in New Issue
Block a user