urieljareth
621bbc5f5c
Scraper de canales de YouTube hacia notas Markdown para base de conocimiento (Obsidian-ready), con plataforma web local. Engine + CLI (Workstream A): - Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit - SQLite store con migración idempotente: FTS5 (transcript search), columnas de metadata enriquecida, tablas cookies_meta y scrape_jobs - Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html), analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline - CLI Click group: search, export, audio, channels, watch, analyze, re-render - Fix del bug de scoping de cookies en cli.py Webapp local (Workstream B): - FastAPI backend: dashboard, channels, videos facetado, transcript, search FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports, folders (abrir en OS), tools (re-render, formato) - SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema dark command-center con acento rojo→rosa, cookie vault drag-drop, consola de scrapeo con progreso live vía SSE Launcher + subagentes (Workstream C): - start-server.bat / stop-server.bat con auto port-scan + browser open - .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).
60 lines
1.6 KiB
Python
60 lines
1.6 KiB
Python
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
from yt_scraper.parse import Segment, parse_json3, parse_vtt, merge_adjacent
|
|
|
|
FIXTURES = Path(__file__).parent / "fixtures"
|
|
|
|
|
|
def test_parse_json3_basic():
|
|
raw = (FIXTURES / "sample.json3").read_text(encoding="utf-8")
|
|
segs = parse_json3(raw)
|
|
assert len(segs) >= 2
|
|
assert segs[0].start == 0.0
|
|
assert "Hola" in segs[0].text
|
|
assert "segunda" in segs[0].text
|
|
assert segs[-1].start == 6.0
|
|
assert "Tercera" in segs[-1].text
|
|
|
|
|
|
def test_parse_json3_skips_empty_events():
|
|
raw = (FIXTURES / "sample.json3").read_text(encoding="utf-8")
|
|
segs = parse_json3(raw)
|
|
texts = [s.text for s in segs]
|
|
assert all(t.strip() for t in texts)
|
|
|
|
|
|
def test_parse_vtt_basic():
|
|
raw = (FIXTURES / "sample.vtt").read_text(encoding="utf-8")
|
|
segs = parse_vtt(raw)
|
|
assert len(segs) == 2
|
|
assert segs[0].start == 0.0
|
|
assert "Hola" in segs[0].text
|
|
assert "tag" in segs[0].text
|
|
assert segs[1].start == 6.0
|
|
assert "Tercer" in segs[1].text
|
|
|
|
|
|
def test_parse_vtt_strips_tags():
|
|
raw = (FIXTURES / "sample.vtt").read_text(encoding="utf-8")
|
|
segs = parse_vtt(raw)
|
|
assert "<c." not in segs[1].text
|
|
|
|
|
|
def test_merge_adjacent():
|
|
segs = [
|
|
Segment(start=0.0, end=1.0, text="a"),
|
|
Segment(start=1.0, end=2.0, text="b"),
|
|
Segment(start=5.0, end=6.0, text="c"),
|
|
]
|
|
merged = merge_adjacent(segs, max_gap=0.0)
|
|
assert len(merged) == 2
|
|
assert "a b" in merged[0].text
|
|
assert merged[1].text == "c"
|
|
|
|
|
|
def test_parse_json3_invalid_json():
|
|
assert parse_json3("not json") == []
|
|
assert parse_json3('{"events": []}') == []
|