feat: local content-mining platform for YouTube creator scraping
Scraper de canales de YouTube hacia notas Markdown para base de conocimiento (Obsidian-ready), con plataforma web local. Engine + CLI (Workstream A): - Modular pipeline: discover/extract/parse/chapters/render/store + ratelimit - SQLite store con migración idempotente: FTS5 (transcript search), columnas de metadata enriquecida, tablas cookies_meta y scrape_jobs - Módulos: segments, cookies (Netscape vault), export (json/csv/srt/html), analysis (word freq/timeline/wordcloud), monitor (watch loop), pipeline - CLI Click group: search, export, audio, channels, watch, analyze, re-render - Fix del bug de scoping de cookies en cli.py Webapp local (Workstream B): - FastAPI backend: dashboard, channels, videos facetado, transcript, search FTS, analysis, scrape jobs con SSE, cookies drag-and-drop, exports, folders (abrir en OS), tools (re-render, formato) - SPA no-build (Alpine.js + Tailwind + Chart.js por CDN): 9 vistas, tema dark command-center con acento rojo→rosa, cookie vault drag-drop, consola de scrapeo con progreso live vía SSE Launcher + subagentes (Workstream C): - start-server.bat / stop-server.bat con auto port-scan + browser open - .opencode/agent/webapp-builder.md + .opencode/goals/webapp-build.md Tests: 38 pytest verdes. Sin funcionalidad de IA (enfoque data-mining).
This commit is contained in:
urieljareth
@@ -0,0 +1,47 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="es">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>yt-scraper · Gallery</title>
|
||||
<style>
|
||||
:root { color-scheme: dark; }
|
||||
body { background: #0a0a0f; color: #e5e7eb; font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; margin: 0; padding: 2rem; }
|
||||
h1 { font-size: 1.6rem; margin: 0 0 .25rem; }
|
||||
h1 span { color: #f43f5e; }
|
||||
.meta { color: #71717a; margin-bottom: 2rem; font-size: .9rem; }
|
||||
.grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); gap: 1.25rem; }
|
||||
.card { background: #18181b; border: 1px solid #27272a; border-radius: 14px; overflow: hidden; transition: transform .15s, border-color .15s; }
|
||||
.card:hover { transform: translateY(-3px); border-color: #f43f5e; }
|
||||
.card img { width: 100%; aspect-ratio: 16/9; object-fit: cover; background: #0a0a0f; display: block; }
|
||||
.body { padding: .85rem 1rem 1rem; }
|
||||
.title { font-weight: 600; font-size: .95rem; line-height: 1.3; margin: 0 0 .5rem; display: -webkit-box; -webkit-line-clamp: 2; -webkit-box-orient: vertical; overflow: hidden; }
|
||||
.title a { color: inherit; text-decoration: none; }
|
||||
.title a:hover { color: #fb7185; }
|
||||
.tags { display: flex; flex-wrap: wrap; gap: .3rem; margin-top: .5rem; }
|
||||
.tag { background: #27272a; color: #a1a1aa; font-size: .72rem; padding: .15rem .5rem; border-radius: 999px; }
|
||||
.row { font-size: .78rem; color: #71717a; display: flex; gap: 1rem; }
|
||||
.dur { color: #f43f5e; font-variant-numeric: tabular-nums; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>yt-scraper <span>·</span> Gallery</h1>
|
||||
<div class="meta">{{ videos | length }} videos{% if channel_id %} · canal {{ channel_id }}{% endif %}</div>
|
||||
<div class="grid">
|
||||
{% for v in videos %}
|
||||
<div class="card">
|
||||
{% if v.thumbnail %}<img src="{{ v.thumbnail }}" alt="" loading="lazy">{% endif %}
|
||||
<div class="body">
|
||||
<div class="title"><a href="{{ v.url }}" target="_blank" rel="noopener">{{ v.title or v.video_id }}</a></div>
|
||||
<div class="row">
|
||||
<span>{{ (v.upload_date[:4] ~ '-' ~ v.upload_date[4:6] ~ '-' ~ v.upload_date[6:8]) if v.upload_date and v.upload_date|length == 8 else (v.upload_date or '') }}</span>
|
||||
{% if v.duration %}<span class="dur">{{ (v.duration // 3600) ~ ':' ~ '%02d' % ((v.duration // 60) % 60) ~ ':' ~ '%02d' % (v.duration % 60) if v.duration >= 3600 else ((v.duration // 60) ~ ':' ~ '%02d' % (v.duration % 60)) }}</span>{% endif %}
|
||||
{% if v.view_count %}<span>{{ "{:,}".format(v.view_count) }} views</span>{% endif %}
|
||||
</div>
|
||||
{% if v.tags %}<div class="tags">{% for t in v.tags[:5] %}<span class="tag">{{ t }}</span>{% endfor %}</div>{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
video_id: {{ video_id }}
|
||||
title: {{ title | quote_yaml }}
|
||||
channel: {{ channel_name | quote_yaml }}
|
||||
channel_id: {{ channel_id | default('') }}
|
||||
channel_url: {{ channel_url | default('') }}
|
||||
upload_date: {{ upload_date | default('') }}
|
||||
duration: {{ duration | default(0) }}
|
||||
url: {{ url }}
|
||||
transcript_lang: {{ transcript_lang | default('') }}
|
||||
transcript_src: {{ transcript_src | default('') }}
|
||||
views: {{ view_count | default('') }}
|
||||
likes: {{ like_count | default('') }}
|
||||
tags: {{ tags | to_json if tags else '[]' }}
|
||||
thumbnail: {{ thumbnail | default('') }}
|
||||
---
|
||||
|
||||
# {{ title }}
|
||||
|
||||
> [Ver en YouTube]({{ url }})
|
||||
|
||||
{% if description %}{{ description }}
|
||||
|
||||
---
|
||||
|
||||
{% endif %}
|
||||
## Transcripcion
|
||||
|
||||
{% for section in sections if section.segments %}
|
||||
### {{ section.title }} ({{ section.start | format_timestamp }})
|
||||
|
||||
{% for seg in section.segments %}
|
||||
**{{ seg.start | format_timestamp }}** · {{ seg.text }}
|
||||
{% endfor %}
|
||||
|
||||
{% endfor %}
|
||||
Reference in New Issue
Block a user