# Discovery-Only Scrape Design ## Goal Add manual discovery controls for one channel or all tracked channels. Discovery checks the complete recent video list returned by YouTube, registers unknown videos as `pending`, and does not download transcripts, Markdown, audio, thumbnails, or avatars. ## Existing Context The webapp already has a single-worker `JobManager`, SSE job events, and a `POST /api/scrape` endpoint. The current channel job combines discovery with `pipeline.process_video()`, so every pending video is immediately extracted and rendered. The channels table has per-channel actions, while the videos view can be filtered to one channel. ## Architecture - Extend the existing job dispatch with `opts.mode == "discover"`. - `channel_id` selects one tracked channel. A missing `channel_id` means all tracked channels for discovery jobs only. - Discovery uses the existing `discover_channel()` call without a playlist cap. It compares the returned IDs with the SQLite catalog and upserts only catalog metadata. It never calls `process_video()`. - The existing scrape mode remains unchanged for users who want discovery plus transcript/Markdown processing. - All discovery jobs stay in the existing sequential queue and use the existing SSE stream, cancellation, progress, and history. ## Persistence and Results - New `VideoRef` rows are inserted with the existing default status `pending`. - Existing video rows retain their status and processed data; rediscovery only refreshes title, upload date, and duration through the existing upsert behavior. - The channel record is refreshed with its name, handle, catalog count, and `last_scraped`; no avatar fetch/cache is performed by discovery-only jobs. - A per-channel progress event reports `new_videos`, `known_videos`, and the channel ID. - The terminal event reports total channels, discovered entries, new videos, known videos, and errors. - A one-channel discovery failure marks the job `error`. An all-channel job continues after individual failures and finishes with a summary so one broken channel does not prevent other channels from being scanned. ## UI - Channels view: add `Investigar todos` in the header and `Investigar` in each channel row. - Videos view: add an investigation button in the header. It investigates the selected channel when `filters.channel` is set, otherwise all channels. - Buttons use the existing job widget/SSE stream, are disabled while an active job is being followed, and show the number of new videos on completion. - Completion refreshes channels, videos, and dashboard data. Existing `.md`, `Process`, and `Audio` controls remain separate. ## Error Handling - Empty channel catalogs complete successfully with zero channels scanned. - Discovery errors are emitted in the job log and do not invoke video processing. - A cancellation checks the existing cancellation set between channels and leaves already inserted catalog rows intact. - Duplicate IDs from a discovery response are counted once by the catalog comparison. ## Testing - Store tests verify that upserting an existing and a new reference reports only the new reference. - Job tests fake `discover_channel()`, verify new rows are `pending`, existing rows retain their status, and `process_video()` is never called. - Job tests verify all-channel discovery continues after one channel fails and reports the successful channel. - The full existing Python test suite remains the regression check. The static UI is verified by code inspection and the existing webapp smoke path; no frontend build step exists.