Files

T

urieljareth 621bbc5f5c feat: local content-mining platform for YouTube creator scraping

Scraper de canales de YouTube hacia notas Markdown para base de conocimiento
(Obsidian-ready), con plataforma web local.

Engine + CLI (Workstream A):
- Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit
- SQLite store con migración idempotente: FTS5 (transcript search), columnas
  de metadata enriquecida, tablas cookies_meta y scrape_jobs
- Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html),
  analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline
- CLI Click group: search, export, audio, channels, watch, analyze, re-render
- Fix del bug de scoping de cookies en cli.py

Webapp local (Workstream B):
- FastAPI backend: dashboard, channels, videos facetado, transcript, search
  FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports,
  folders (abrir en OS), tools (re-render, formato)
- SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema
  dark command-center con acento rojo→rosa, cookie vault drag-drop,
  consola de scrapeo con progreso live vía SSE

Launcher + subagentes (Workstream C):
- start-server.bat / stop-server.bat con auto port-scan + browser open
- .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md

Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).

2026-07-26 23:19:34 -06:00

2.5 KiB

Raw Permalink Blame History

Sub-goal: Build the local webapp (Workstream B)

Parent goal: Convert yt-channel-scraper into a local content-mining platform. Owner agent: webapp-builder (.opencode/agent/webapp-builder.md) Spec: docs/superpowers/specs/2026-07-26-platform-design.md (§7 Webapp, §8 Cookies, §7.2 design language) Status: Workstream A (engine + CLI) is DONE and verified — pytest green, real DB backfilled, FTS search working, cookie vault auto-imports the real cookie.

Objective

Implement the FastAPI backend + Alpine.js SPA so the platform is usable from a browser, launched by start-server.bat. No Node/npm build step.

Deliverables

src/yt_scraper/webapp/__init__.py
src/yt_scraper/webapp/app.py — FastAPI factory, /healthz, static mount, auto-import cookies on startup.
src/yt_scraper/webapp/jobs.py — single-worker scrape job runner with SSE event bus (queue/progress/log/cancel), persists to scrape_jobs.
src/yt_scraper/webapp/api.py — REST routers documented in spec §7.1 (dashboard, channels, videos, search, analysis, scrape+SSE, cookies with drag-and-drop upload, exports).
src/yt_scraper/webapp/static/index.html, app.js, styles.css — the 9-view SPA, dark command-center theme, drag-and-drop cookie vault, Chart.js viz.

API contract (the SPA consumes exactly this)

GET /healthz → {"status":"ok"}
GET /api/dashboard → {channels[], status_breakdown{}, uploads_over_time[], duration_histogram[], top_tags[]}
GET /api/channels | POST /api/channels {url} | DELETE /api/channels/{id}
GET /api/videos?channel=&status=&from=&to=&min_dur=&q=&sort=&page=&size= → {items[], total, page, size}
GET /api/videos/{id} | GET /api/videos/{id}/transcript → [{idx,start_sec,end_sec,text}]
GET /api/search?q=&channel=&limit= → [{video_id,title,start_sec,snippet}]
GET /api/analysis?channel=&type=topwords|wordcloud|timeline&term= → chart-ready JSON
POST /api/scrape {channel_id,opts} → {job_id} | GET /api/scrape/{id}/stream (SSE) | DELETE /api/scrape/{id}
POST /api/cookies/upload (multipart file(s) or {text,label} JSON) | GET /api/cookies | POST /api/cookies/{id}/activate | DELETE /api/cookies/{id} | POST /api/cookies/{id}/test
GET /api/export?format=json|csv|srt|html&channel= → file stream

Done when

/healthz returns 200; every router smokes green against the real DB; SPA renders all 9 views; SSE streams a scrape job; the real cookie drag-and-drops and activates; pytest green.

2.5 KiB Raw Permalink Blame History