feat: local content-mining platform for YouTube creator scraping
Scraper de canales de YouTube hacia notas Markdown para base de conocimiento (Obsidian-ready), con plataforma web local. Engine + CLI (Workstream A): - Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit - SQLite store con migración idempotente: FTS5 (transcript search), columnas de metadata enriquecida, tablas cookies_meta y scrape_jobs - Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html), analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline - CLI Click group: search, export, audio, channels, watch, analyze, re-render - Fix del bug de scoping de cookies en cli.py Webapp local (Workstream B): - FastAPI backend: dashboard, channels, videos facetado, transcript, search FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports, folders (abrir en OS), tools (re-render, formato) - SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema dark command-center con acento rojo→rosa, cookie vault drag-drop, consola de scrapeo con progreso live vía SSE Launcher + subagentes (Workstream C): - start-server.bat / stop-server.bat con auto port-scan + browser open - .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).
This commit is contained in:
@@ -0,0 +1,48 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68.0", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "yt-channel-scraper"
|
||||
version = "0.1.0"
|
||||
description = "Scraper de canales de YouTube hacia notas Markdown para Obsidian"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"yt-dlp>=2024.10.7",
|
||||
"click>=8.1",
|
||||
"rich>=13.7",
|
||||
"jinja2>=3.1",
|
||||
"pyyaml>=6.0",
|
||||
"python-slugify>=8.0",
|
||||
"requests>=2.31",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8.0",
|
||||
"pytest-cov>=4.1",
|
||||
]
|
||||
web = [
|
||||
"fastapi>=0.110",
|
||||
"uvicorn[standard]>=0.27",
|
||||
"sse-starlette>=2.0",
|
||||
]
|
||||
analysis = [
|
||||
"matplotlib>=3.8",
|
||||
"wordcloud>=1.9",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
yt-scraper = "yt_scraper.cli:main"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
yt_scraper = ["../templates/*.j2"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
markers = [
|
||||
"integration: tests que requieren red (deselect with -m 'not integration')",
|
||||
]
|
||||
Reference in New Issue
Block a user