Compare commits

...
12 Commits
Author SHA1 Message Date
urieljareth 239a7510c6 OCR Mistral: modelo mistral-ocr-4-0, batch + fallback con reintentos y errores etiquetados
- Modelo por defecto mistral-ocr-4-0 (.env.example, config.py, docker-compose)
- MistralClient: OCR en lote via API Batch (/v1/ocr) subiendo JSONL, polling y descarga de resultados
- Reintentos con backoff exponencial en llamadas a Mistral (429/5xx/red); sin reintento en 401/400
- PdfProcessor: lote cuando >= umbral de paginas, fallback sincrono pagina a pagina, placeholder rastreable por pagina fallida
- Errores etiquetados ([AUTH_MISTRAL], [BATCH], [NETWORK], [OCR_EMPTY], [PDF_PARSE]...) en el mensaje del job
- Nuevas variables: MISTRAL_OCR_BATCH_THRESHOLD/POLL_SECONDS/TIMEOUT_SECONDS/RETRY_ATTEMPTS/RETRY_BACKOFF_BASE
2026-07-26 19:58:38 -06:00
urieljareth 379f563d04 Create CLAUDE.md 2026-06-10 02:01:45 -06:00
urieljareth 2160d4ff70 Commit inicial 2026-05-31 20:23:27 -06:00
urieljarethbusiness-cpu 7361f7ced1 1 2026-05-19 14:21:56 -06:00
urieljarethbusiness-cpu 4f3bc01dcb Update README.md 2026-05-19 13:21:06 -06:00
urieljarethbusiness-cpu 0df056445e 1 2026-05-19 11:23:47 -06:00
urieljarethbusiness-cpu d18ab1f618 Update Dockerfile.coolify 2026-05-19 10:11:36 -06:00
urieljarethbusiness-cpu b04079eaaf Update Dockerfile.coolify 2026-05-19 09:41:43 -06:00
urieljarethbusiness-cpu 7dc0f65045 1 2026-05-17 14:30:00 -06:00
urieljarethbusiness-cpu 8e2a7eadd2 Update docker-compose.yml 2026-05-17 13:18:03 -06:00
urieljarethbusiness-cpu 052580b75b 2 2026-05-17 12:57:23 -06:00
urieljarethbusiness-cpu 306d131136 Update docker-compose.yaml 2026-05-17 12:32:23 -06:00
17 changed files with 702 additions and 91 deletions
+1 -1
View File
@@ -14,6 +14,6 @@ frontend/.env.local
frontend/.env.*.local frontend/.env.*.local
node_modules node_modules
npm-debug.log* npm-debug.log*
Dockerfile* Dockerfile
docker-compose*.yml docker-compose*.yml
docker-compose*.yaml docker-compose*.yaml
+6 -1
View File
@@ -2,5 +2,10 @@ APP_ENV=development
APP_DATA_DIR=./data APP_DATA_DIR=./data
DATABASE_PATH=./data/app.db DATABASE_PATH=./data/app.db
MISTRAL_API_KEY= MISTRAL_API_KEY=
MISTRAL_OCR_MODEL=mistral-ocr-latest MISTRAL_OCR_MODEL=mistral-ocr-4-0
MISTRAL_OCR_BATCH_THRESHOLD=3
MISTRAL_OCR_BATCH_POLL_SECONDS=10
MISTRAL_OCR_BATCH_TIMEOUT_SECONDS=1800
MISTRAL_OCR_RETRY_ATTEMPTS=3
MISTRAL_OCR_RETRY_BACKOFF_BASE=2.0
DEEPGRAM_API_KEY= DEEPGRAM_API_KEY=
+46
View File
@@ -0,0 +1,46 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
> `AGENTS.md` contains the canonical command list, platform behavior notes, and deployment/secrets rules. Read it too — this file focuses on the cross-file architecture that AGENTS.md does not spell out.
## Commands
- **Backend (run from repo root, not `backend/`):** `uvicorn backend.app.main:app --reload`. Imports are absolute from the `backend.` package, so the working directory must be the repo root.
- **Frontend:** `cd frontend && npm run dev` (dev), `npm run build`, `npm run lint`.
- **Syntax check:** `python -m compileall backend` (there is no test suite in the repo).
- **Windows one-click:** `abrir_estacion.bat``scripts/start-platform.ps1` picks free ports, starts both servers, opens the browser; `cerrar_estacion.bat` stops only this project.
- **Container/prod:** `docker compose up --build`, or Nixpacks/`Dockerfile.coolify` → both run `scripts/start-coolify.sh`.
## Architecture
A two-process app for turning class materials (PDF/DOCX/text/audio/video) into LLM-ready Markdown, organized as **Materia → Semana → documentos**.
### Request → background-job flow
1. `backend/app/main.py` is the entire HTTP API (flat FastAPI app, no routers). Every endpoint is a thin try/except that delegates to `KnowledgeManager` and maps `ValueError` → 404/400, `DuplicateSourceError` → 409.
2. File upload saves the raw file, creates a **source** row + a **job** row, then schedules `process_job` via FastAPI `BackgroundTasks` (in-process — no external queue/worker). Job status transitions: `queued → processing → completed|failed`.
3. `process_job` calls `IngestionService.process_source`, which picks a processor by `source_type`, runs it, wraps the result in Markdown frontmatter, and persists a **document**.
### Key components
- **`KnowledgeManager`** (`managers/knowledge_manager.py`) — the single source of truth. Owns **both** the SQLite schema (`subjects`, `weeks`, `sources`, `documents`, `jobs`) and the on-disk layout under `data/subjects/<slug>/week-NN/{raw,processed,exports}/`. Filesystem and DB are kept in sync here; processors and endpoints must go through it, never write final Markdown or touch the DB directly. Schema migrations are done idempotently in `_init_db` via `_ensure_columns` (add-column-if-missing) — there is no migration framework.
- **`IngestionService`** (`services/ingestion_service.py`) — maps `source_type` → processor and assembles frontmatter metadata.
- **Processors** (`services/processors/`) — each returns a `ProcessedContent(title, body, processor)` dataclass (`base.py`). `text`, `docx` (python-docx), `pdf` (PyMuPDF text, or Mistral OCR when `use_ocr=true`), `audio` (Deepgram), `video` (ffmpeg → audio → Deepgram). PDFs accept `page_ranges` like `1-5,8,10-12`.
- **Providers** (`services/providers/`) — thin HTTP clients for Mistral OCR and Deepgram, keyed off `MISTRAL_API_KEY` / `DEEPGRAM_API_KEY`.
- **`markdown_builder.build_markdown`** — the only place that emits the YAML frontmatter block prepended to every processed document.
### Important invariants
- **One document per source.** `create_document` deletes any prior document/jobs for that source before inserting, then deletes the old Markdown file. Reprocessing replaces, never appends.
- **Dedup by hash.** `save_upload` rejects a file whose `sha256` already exists in the same week (`DuplicateSourceError` → 409).
- **Deletes cascade manually** (no FK cascade): deleting a subject/week/source/document removes child DB rows *and* unlinks files. Renaming a week (`update_week`) renames the directory and rewrites stored paths in the DB with SQL `replace()`.
- **Markdown is sanitized** (`_sanitize_markdown` strips control chars) on every read and write.
- **Week exports** (`export_week_markdown`) concatenate documents, stripping each doc's frontmatter and inserting `<!-- document_id: N -->` markers; the `.zip` variant ships individual files.
### Frontend
Next.js 14 App Router (`frontend/app/`), Tailwind, `react-markdown` + `remark-gfm`. All backend calls go through the single `api` object in `frontend/lib/api.ts`. `API_URL` resolves to `http://127.0.0.1:8000` in dev (via `NEXT_PUBLIC_API_URL`) or `/api` in prod, where `next.config.mjs` `rewrites()` proxy `/api/*` to `BACKEND_INTERNAL_URL`. Pages mirror the data model: `/`, `/subjects/[subjectId]`, `/subjects/[subjectId]/weeks/[weekNumber]`, `/documents/[documentId]`, `/settings`.
### Deployment shape
Single container runs **both** processes: `start-coolify.sh` launches uvicorn on `127.0.0.1:8000` (internal only) and Next.js standalone on `PORT` (default 3000, public). `/app/data` **must** be a mounted volume or all subjects/documents/SQLite state are lost on redeploy. `ffmpeg` is required at runtime for video.
## Conventions
- UX, error messages, and generated content are in **Spanish** — preserve this. Source comments/identifiers are mixed Spanish/English; match the surrounding file.
- The standalone `estacion documentos.html` is a browser-only prototype kept for reference behavior; the real app is `backend/` + `frontend/`.
+42
View File
@@ -0,0 +1,42 @@
FROM node:22-bookworm-slim AS frontend-builder
WORKDIR /app/frontend
COPY frontend/package.json frontend/package-lock.json* ./
RUN npm ci
COPY frontend ./
ARG NEXT_PUBLIC_API_URL=
ARG BACKEND_INTERNAL_URL=http://127.0.0.1:8000
ENV NEXT_PUBLIC_API_URL=$NEXT_PUBLIC_API_URL
ENV BACKEND_INTERNAL_URL=$BACKEND_INTERNAL_URL
RUN npm run build
FROM python:3.12-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV NODE_ENV=production
ENV HOSTNAME=0.0.0.0
ENV DEBIAN_FRONTEND=noninteractive
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates ffmpeg \
&& rm -rf /var/lib/apt/lists/*
COPY --from=frontend-builder /usr/local/bin/node /usr/local/bin/node
COPY --from=frontend-builder /usr/local/bin/npm /usr/local/bin/npm
COPY --from=frontend-builder /usr/local/bin/npx /usr/local/bin/npx
COPY --from=frontend-builder /usr/local/lib/node_modules /usr/local/lib/node_modules
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY backend ./backend
COPY scripts ./scripts
COPY --from=frontend-builder /app/frontend/.next ./frontend/.next
COPY --from=frontend-builder /app/frontend/package.json ./frontend/package.json
COPY --from=frontend-builder /app/frontend/node_modules ./frontend/node_modules
EXPOSE 3000
CMD ["bash", "scripts/start-coolify.sh"]
BIN
View File
Binary file not shown.
+6 -1
View File
@@ -8,7 +8,12 @@ class Settings(BaseSettings):
app_data_dir: Path = Path("./data") app_data_dir: Path = Path("./data")
database_path: Path = Path("./data/app.db") database_path: Path = Path("./data/app.db")
mistral_api_key: str = "" mistral_api_key: str = ""
mistral_ocr_model: str = "mistral-ocr-latest" mistral_ocr_model: str = "mistral-ocr-4-0"
mistral_ocr_batch_threshold: int = 3
mistral_ocr_batch_poll_seconds: int = 10
mistral_ocr_batch_timeout_seconds: int = 1800
mistral_ocr_retry_attempts: int = 3
mistral_ocr_retry_backoff_base: float = 2.0
deepgram_api_key: str = "" deepgram_api_key: str = ""
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
+15 -2
View File
@@ -8,7 +8,8 @@ from fastapi.responses import FileResponse, PlainTextResponse
from backend.app.config import settings from backend.app.config import settings
from backend.app.managers.knowledge_manager import DuplicateSourceError, KnowledgeManager from backend.app.managers.knowledge_manager import DuplicateSourceError, KnowledgeManager
from backend.app.models.schemas import MarkdownUpdate, SubjectCreate, WeekCreate, WeekUpdate from backend.app.models.schemas import MarkdownUpdate, SubjectCreate, SubjectUpdate, WeekCreate, WeekUpdate
from backend.app.services.errors import ProcessingError, classify_unknown_error
from backend.app.services.ingestion_service import IngestionService from backend.app.services.ingestion_service import IngestionService
app = FastAPI(title="Knowledge Station", version="0.1.0") app = FastAPI(title="Knowledge Station", version="0.1.0")
@@ -60,6 +61,14 @@ def get_subject(subject_id: int) -> dict:
raise HTTPException(status_code=404, detail=str(exc)) from exc raise HTTPException(status_code=404, detail=str(exc)) from exc
@app.put("/subjects/{subject_id}")
def update_subject(subject_id: int, payload: SubjectUpdate) -> dict:
try:
return manager.update_subject(subject_id, payload.name)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
@app.delete("/subjects/{subject_id}") @app.delete("/subjects/{subject_id}")
def delete_subject(subject_id: int) -> dict: def delete_subject(subject_id: int) -> dict:
try: try:
@@ -276,6 +285,10 @@ def process_job(job_id: int, source_id: int, use_ocr: bool, page_ranges: str | N
try: try:
document = ingestion.process_source(source_id, use_ocr=use_ocr, page_ranges=page_ranges) document = ingestion.process_source(source_id, use_ocr=use_ocr, page_ranges=page_ranges)
manager.update_job(job_id, "completed", "Documento procesado", document_id=document["id"]) manager.update_job(job_id, "completed", "Documento procesado", document_id=document["id"])
except ProcessingError as exc:
manager.update_source_status(source_id, "failed")
manager.update_job(job_id, "failed", exc.labeled)
except Exception as exc: except Exception as exc:
manager.update_source_status(source_id, "failed") manager.update_source_status(source_id, "failed")
manager.update_job(job_id, "failed", str(exc)) code = classify_unknown_error(exc)
manager.update_job(job_id, "failed", f"[{code}] {exc}")
+26
View File
@@ -154,6 +154,14 @@ class KnowledgeManager:
raise ValueError("Materia no encontrada") raise ValueError("Materia no encontrada")
return dict(row) return dict(row)
def update_subject(self, subject_id: int, name: str) -> dict[str, Any]:
if not name.strip():
raise ValueError("El nombre de la materia no puede estar vacio")
self.get_subject(subject_id)
with self._connect() as conn:
conn.execute("update subjects set name = ? where id = ?", (name.strip(), subject_id))
return self.get_subject(subject_id)
def delete_subject(self, subject_id: int) -> None: def delete_subject(self, subject_id: int) -> None:
subject = self.get_subject(subject_id) subject = self.get_subject(subject_id)
with self._connect() as conn: with self._connect() as conn:
@@ -224,6 +232,8 @@ class KnowledgeManager:
def update_week(self, subject_id: int, current_number: int, new_number: int, title: str | None = None) -> dict[str, Any]: def update_week(self, subject_id: int, current_number: int, new_number: int, title: str | None = None) -> dict[str, Any]:
subject = self.get_subject(subject_id) subject = self.get_subject(subject_id)
week = self.get_week(subject_id, current_number) week = self.get_week(subject_id, current_number)
old_dir: Path | None = None
new_dir: Path | None = None
if new_number != current_number: if new_number != current_number:
with self._connect() as conn: with self._connect() as conn:
existing = conn.execute( existing = conn.execute(
@@ -248,6 +258,22 @@ class KnowledgeManager:
"update weeks set number = ?, title = ? where id = ?", "update weeks set number = ?, title = ? where id = ?",
(new_number, title, week["id"]), (new_number, title, week["id"]),
) )
if old_dir and new_dir:
old_prefix = str(old_dir)
new_prefix = str(new_dir)
conn.execute(
"update sources set stored_path = replace(stored_path, ?, ?) where week_id = ? and stored_path like ?",
(old_prefix, new_prefix, week["id"], f"{old_prefix}%"),
)
conn.execute(
"""
update documents
set markdown_path = replace(markdown_path, ?, ?)
where source_id in (select id from sources where week_id = ?)
and markdown_path like ?
""",
(old_prefix, new_prefix, week["id"], f"{old_prefix}%"),
)
return self.get_week(subject_id, new_number) return self.get_week(subject_id, new_number)
def week_dir(self, subject_slug: str, week_number: int) -> Path: def week_dir(self, subject_slug: str, week_number: int) -> Path:
+4
View File
@@ -8,6 +8,10 @@ class SubjectCreate(BaseModel):
name: str = Field(min_length=1) name: str = Field(min_length=1)
class SubjectUpdate(BaseModel):
name: str = Field(min_length=1)
class Subject(BaseModel): class Subject(BaseModel):
id: int id: int
name: str name: str
+85
View File
@@ -0,0 +1,85 @@
from __future__ import annotations
import requests
class ProcessingError(Exception):
"""Error de procesamiento con un codigo corto y rastreable.
El atributo ``code`` es un identificador estable en MAYUSCULAS que se
antepone al mensaje del job (p.ej. ``[AUTH_MISTRAL] API Key invalida``)
para que el usuario pueda buscar y diagnosticar rapidamente el tipo de
fallo desde la UI.
"""
def __init__(self, code: str, detail: str) -> None:
super().__init__(f"[{code}] {detail}")
self.code = code
self.detail = detail
@property
def labeled(self) -> str:
return f"[{self.code}] {self.detail}"
class MistralAuthError(ProcessingError):
def __init__(self, detail: str = "API Key de Mistral invalida o no autorizada (401)") -> None:
super().__init__("AUTH_MISTRAL", detail)
class MistralRateLimitError(ProcessingError):
def __init__(self, detail: str = "Limite de uso de Mistral alcanzado (429)") -> None:
super().__init__("RATE_LIMIT", detail)
class MistralServerError(ProcessingError):
def __init__(self, status: int, body: str) -> None:
super().__init__("MISTRAL_SERVER", f"Mistral HTTP {status}: {body[:200]}")
class MistralBatchError(ProcessingError):
def __init__(self, detail: str) -> None:
super().__init__("BATCH", detail)
class MistralOCREmptyError(ProcessingError):
def __init__(self, detail: str = "OCR no devolvio contenido") -> None:
super().__init__("OCR_EMPTY", detail)
class NetworkError(ProcessingError):
def __init__(self, detail: str) -> None:
super().__init__("NETWORK", detail)
NON_RETRYABLE_CODES = {
"AUTH_MISTRAL",
"CONFIG",
"OCR_EMPTY",
"MISTRAL_CLIENT",
"BAD_PDF",
"FILE_IO",
}
def classify_requests_error(exc: requests.RequestException) -> ProcessingError:
if isinstance(exc, requests.Timeout):
return NetworkError(f"timeout de red con Mistral: {exc}")
if isinstance(exc, requests.ConnectionError):
return NetworkError(f"sin conexion a Mistral: {exc}")
return NetworkError(f"error de red con Mistral: {exc}")
def classify_unknown_error(exc: Exception) -> str:
"""Etiqueta breve para excepciones no envueltas en ProcessingError."""
name = type(exc).__name__.lower()
message = str(exc).lower()
if "fitz" in name or "pdf" in name or "pdf" in message or "muPDF" in message:
return "PDF_PARSE"
if isinstance(exc, FileNotFoundError):
return "FILE_IO"
if isinstance(exc, (PermissionError, OSError)):
return "FILE_IO"
if isinstance(exc, ValueError):
return "VALIDATION"
return "UNKNOWN"
@@ -4,8 +4,10 @@ from pathlib import Path
import fitz import fitz
from backend.app.config import settings
from backend.app.services.markdown_builder import title_from_path from backend.app.services.markdown_builder import title_from_path
from backend.app.services.processors.base import ProcessedContent from backend.app.services.processors.base import ProcessedContent
from backend.app.services.errors import ProcessingError
from backend.app.services.providers.mistral_client import MistralClient from backend.app.services.providers.mistral_client import MistralClient
@@ -30,22 +32,77 @@ class PdfProcessor:
return ProcessedContent(title=title_from_path(path), body=body, processor="pdf-text") return ProcessedContent(title=title_from_path(path), body=body, processor="pdf-text")
def _process_ocr(self, path: Path) -> ProcessedContent: def _process_ocr(self, path: Path) -> ProcessedContent:
doc = fitz.open(path)
client = MistralClient()
parts: list[str] = []
temp_paths: list[Path] = []
try: try:
for index in self._selected_pages(len(doc)): doc = fitz.open(path)
except Exception as exc:
raise ProcessingError("BAD_PDF", f"No se pudo abrir el PDF: {exc}") from exc
selected = self._selected_pages(len(doc))
client = MistralClient()
page_images: dict[int, Path] = {}
try:
for index in selected:
page = doc[index - 1] page = doc[index - 1]
pix = page.get_pixmap(matrix=fitz.Matrix(2, 2), alpha=False) pix = page.get_pixmap(matrix=fitz.Matrix(2, 2), alpha=False)
image_path = path.with_name(f"{path.stem}-page-{index}.jpg") image_path = path.with_name(f"{path.stem}-page-{index}.jpg")
pix.save(image_path) pix.save(image_path)
temp_paths.append(image_path) page_images[index] = image_path
parts.append(f"## Pagina {index}\n\n{client.ocr_image(image_path)}")
results = self._ocr_pages(client, page_images)
finally: finally:
for temp_path in temp_paths: for image_path in page_images.values():
temp_path.unlink(missing_ok=True) image_path.unlink(missing_ok=True)
return ProcessedContent(title=title_from_path(path), body="\n\n".join(parts), processor="mistral-ocr")
parts: list[str] = []
for index in selected:
markdown = results.get(index, "")
parts.append(f"## Pagina {index}\n\n{markdown}")
return ProcessedContent(
title=title_from_path(path),
body="\n\n".join(parts),
processor="mistral-ocr",
)
def _ocr_pages(self, client: MistralClient, page_images: dict[int, Path]) -> dict[int, str]:
"""OCR de las paginas con lote + fallback sincrono.
Estrategia:
1. Si hay >= ``batch_threshold`` paginas, intenta el job en lote.
2. Cualquier pagina no resuelta (o si el lote entero falla) cae a
OCR sincrono pagina a pagina con reintentos.
3. Las paginas que fallen de forma definitiva quedan como un
placeholder breve con el codigo de error rastreable.
"""
indices = list(page_images.keys())
results: dict[int, str] = {}
page_errors: dict[int, str] = {}
if len(indices) >= settings.mistral_ocr_batch_threshold:
ordered = sorted(indices)
paths = [page_images[i] for i in ordered]
try:
batch_results = client.ocr_images_batch(paths)
for pos, index in enumerate(ordered):
if pos in batch_results:
results[index] = batch_results[pos]
except ProcessingError:
# El lote fallo; continua al fallback sincrono para todas
# las paginas no resueltas.
pass
for index in indices:
if index in results:
continue
try:
results[index] = client.ocr_image(page_images[index])
except ProcessingError as exc:
page_errors[index] = exc.labeled
except Exception as exc:
page_errors[index] = f"[OCR_ERROR] {exc}"
for index in indices:
if index not in results:
results[index] = f"> ⚠️ No se pudo procesar esta pagina: {page_errors.get(index, '[UNKNOWN]')}"
return results
def _selected_pages(self, total_pages: int) -> list[int]: def _selected_pages(self, total_pages: int) -> list[int]:
if total_pages < 1: if total_pages < 1:
+261 -19
View File
@@ -1,45 +1,287 @@
from __future__ import annotations from __future__ import annotations
import base64 import base64
import json
import time
from pathlib import Path from pathlib import Path
import requests import requests
from backend.app.config import settings from backend.app.config import settings
from backend.app.services.errors import (
MistralAuthError,
MistralBatchError,
MistralOCREmptyError,
MistralRateLimitError,
MistralServerError,
NetworkError,
NON_RETRYABLE_CODES,
ProcessingError,
classify_requests_error,
)
class MistralClient: class MistralClient:
endpoint = "https://api.mistral.ai/v1/ocr" """Cliente Mistral para OCR.
- ``ocr_image``: OCR sincrono de una pagina, con reintentos y backoff.
- ``ocr_images_batch``: OCR en lote via la API de Batch de Mistral
(endpoint ``/v1/ocr``), subiendo un fichero JSONL. Devuelve un dict
``{page_index: markdown}``. Si el job falla o caduca se lanza
``MistralBatchError`` para que el llamador caiga al modo sincrono.
"""
base_url = "https://api.mistral.ai/v1"
ocr_url = f"{base_url}/ocr"
files_url = f"{base_url}/files"
batch_url = f"{base_url}/batch/jobs"
def __init__(self) -> None:
self._api_key = settings.mistral_api_key
self._model = settings.mistral_ocr_model
# ------------------------------------------------------------------
# API publica
# ------------------------------------------------------------------
def ocr_image(self, image_path: Path) -> str: def ocr_image(self, image_path: Path) -> str:
if not settings.mistral_api_key: self._require_key()
raise RuntimeError("MISTRAL_API_KEY no esta configurada")
data_url = self._image_data_url(image_path) data_url = self._image_data_url(image_path)
return self._retry(lambda: self._ocr_sync(data_url))
def ocr_images_batch(self, image_paths: list[Path]) -> dict[int, str]:
"""OCR en lote. Devuelve ``{posicion_en_lista: markdown}``.
``posicion_en_lista`` es el indice (0-based) dentro de ``image_paths``,
NO el numero de pagina del PDF; el llamador hace esa traduccion.
"""
self._require_key()
if not image_paths:
return {}
payload = self._build_batch_payload(image_paths)
upload = self._retry(lambda: self._upload_batch_file(payload))
file_id = upload.get("id")
if not file_id:
raise MistralBatchError("la subida del fichero batch no devolvio id")
job = self._retry(lambda: self._create_batch_job(file_id))
job_id = job.get("id")
if not job_id:
raise MistralBatchError("la creacion del job batch no devolvio id")
final = self._poll_batch_job(job_id)
if final.get("status") != "SUCCESS":
reason = final.get("errors") or final.get("status") or "sin detalle"
raise MistralBatchError(f"job {job_id} finalizo como {final.get('status')}: {reason}")
output_file_id = final.get("output_file")
if not output_file_id:
raise MistralBatchError(f"job {job_id} sin output_file")
content = self._retry(lambda: self._download_file(output_file_id))
return self._parse_batch_output(content)
# ------------------------------------------------------------------
# OCR sincrono
# ------------------------------------------------------------------
def _ocr_sync(self, data_url: str) -> str:
try:
response = requests.post( response = requests.post(
self.endpoint, self.ocr_url,
headers={ headers=self._json_headers(),
"Content-Type": "application/json",
"Authorization": f"Bearer {settings.mistral_api_key}",
},
json={ json={
"model": settings.mistral_ocr_model, "model": self._model,
"document": {"type": "image_url", "image_url": data_url}, "document": {"type": "image_url", "image_url": data_url},
"include_image_base64": False, "include_image_base64": False,
}, },
timeout=180, timeout=180,
) )
if response.status_code == 401: except requests.RequestException as exc:
raise RuntimeError("API Key de Mistral invalida") raise classify_requests_error(exc) from exc
if not response.ok: self._raise_for_status(response)
raise RuntimeError(f"Mistral error {response.status_code}: {response.text}") markdown = self._extract_markdown(response.json())
data = response.json() if not markdown:
raise MistralOCREmptyError()
return markdown
# ------------------------------------------------------------------
# Batch: construccion, subida, job, polling, descarga, parseo
# ------------------------------------------------------------------
def _build_batch_payload(self, image_paths: list[Path]) -> bytes:
lines: list[str] = []
for idx, path in enumerate(image_paths):
data_url = self._image_data_url(path)
body = {
"document": {"type": "image_url", "image_url": data_url},
"include_image_base64": False,
}
lines.append(
json.dumps(
{"custom_id": str(idx), "body": body},
separators=(",", ":"),
ensure_ascii=False,
)
)
return ("\n".join(lines)).encode("utf-8")
def _upload_batch_file(self, payload: bytes) -> dict:
try:
response = requests.post(
self.files_url,
headers={"Authorization": f"Bearer {self._api_key}"},
data={"purpose": "batch"},
files={"file": ("ocr-batch.jsonl", payload, "application/jsonl")},
timeout=300,
)
except requests.RequestException as exc:
raise classify_requests_error(exc) from exc
self._raise_for_status(response)
return response.json()
def _create_batch_job(self, file_id: str) -> dict:
try:
response = requests.post(
self.batch_url,
headers=self._json_headers(),
json={
"input_files": [file_id],
"model": self._model,
"endpoint": "/v1/ocr",
},
timeout=120,
)
except requests.RequestException as exc:
raise classify_requests_error(exc) from exc
self._raise_for_status(response)
return response.json()
def _poll_batch_job(self, job_id: str) -> dict:
terminal = {"SUCCESS", "FAILED", "TIMEOUT_EXCEEDED", "CANCELLED"}
deadline = time.time() + max(60, settings.mistral_ocr_batch_timeout_seconds)
delay = max(2, settings.mistral_ocr_batch_poll_seconds)
last: dict = {}
while time.time() < deadline:
try:
response = requests.get(
f"{self.batch_url}/{job_id}",
headers=self._json_headers(),
timeout=60,
)
except requests.RequestException:
time.sleep(delay)
continue
if response.ok:
last = response.json()
if last.get("status") in terminal:
return last
time.sleep(delay)
raise MistralBatchError(
f"job {job_id} excedio el timeout de polling "
f"({settings.mistral_ocr_batch_timeout_seconds}s); ultimo estado: "
f"{last.get('status') if last else 'desconocido'}"
)
def _download_file(self, file_id: str) -> bytes:
try:
response = requests.get(
f"{self.files_url}/{file_id}/content",
headers=self._json_headers(),
timeout=300,
)
except requests.RequestException as exc:
raise classify_requests_error(exc) from exc
self._raise_for_status(response)
return response.content
def _parse_batch_output(self, content: bytes) -> dict[int, str]:
results: dict[int, str] = {}
for raw in content.decode("utf-8", errors="replace").splitlines():
line = raw.strip()
if not line:
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
custom_id = obj.get("custom_id")
if custom_id is None:
continue
try:
idx = int(custom_id)
except (TypeError, ValueError):
continue
response = obj.get("response") or {}
status_code = response.get("status_code", 200)
body = response.get("body") or {}
if status_code >= 400 or obj.get("error"):
# Pagina fallida en lote: se omite y el llamador cae a sync.
continue
markdown = self._extract_markdown(body)
if markdown:
results[idx] = markdown
return results
# ------------------------------------------------------------------
# Helpers compartidos
# ------------------------------------------------------------------
def _require_key(self) -> None:
if not self._api_key:
raise ProcessingError("CONFIG", "MISTRAL_API_KEY no esta configurada")
def _json_headers(self) -> dict[str, str]:
return {
"Authorization": f"Bearer {self._api_key}",
"Content-Type": "application/json",
"Accept": "application/json",
}
def _raise_for_status(self, response: requests.Response) -> None:
if response.ok:
return
try:
body = response.text
except Exception:
body = ""
status = response.status_code
if status == 401:
raise MistralAuthError()
if status == 429:
raise MistralRateLimitError()
if status >= 500:
raise MistralServerError(status, body)
raise ProcessingError("MISTRAL_CLIENT", f"HTTP {status}: {body[:200]}")
def _retry(self, fn):
attempts = max(1, settings.mistral_ocr_retry_attempts)
base_delay = max(0.5, settings.mistral_ocr_retry_backoff_base)
last_exc: Exception | None = None
for attempt in range(attempts):
try:
return fn()
except ProcessingError as exc:
if exc.code in NON_RETRYABLE_CODES:
raise
last_exc = exc
except Exception as exc:
last_exc = exc
if attempt < attempts - 1:
time.sleep(base_delay * (2 ** attempt))
if last_exc is not None:
raise last_exc
raise MistralBatchError("reintentos agotados sin excepcion conocida")
def _extract_markdown(self, data: dict) -> str:
pages = data.get("pages") or [] pages = data.get("pages") or []
markdown_pages = [page.get("markdown", "").strip() for page in pages if page.get("markdown")] chunks = [page.get("markdown", "").strip() for page in pages if page.get("markdown")]
if markdown_pages: if chunks:
return "\n\n".join(markdown_pages) return "\n\n".join(chunks)
if data.get("markdown"): if data.get("markdown"):
return data["markdown"].strip() return str(data["markdown"]).strip()
raise RuntimeError("Mistral OCR no devolvio Markdown") return ""
def _image_data_url(self, image_path: Path) -> str: def _image_data_url(self, image_path: Path) -> str:
encoded = base64.b64encode(image_path.read_bytes()).decode("ascii") encoded = base64.b64encode(image_path.read_bytes()).decode("ascii")
+9 -28
View File
@@ -1,8 +1,11 @@
services: services:
backend: app:
build: build:
context: . context: .
dockerfile: Dockerfile dockerfile: Dockerfile.coolify
args:
NEXT_PUBLIC_API_URL: ""
BACKEND_INTERNAL_URL: http://127.0.0.1:8000
environment: environment:
APP_ENV: production APP_ENV: production
APP_DATA_DIR: /app/data APP_DATA_DIR: /app/data
@@ -10,37 +13,15 @@ services:
MISTRAL_API_KEY: ${MISTRAL_API_KEY:-} MISTRAL_API_KEY: ${MISTRAL_API_KEY:-}
MISTRAL_OCR_MODEL: ${MISTRAL_OCR_MODEL:-mistral-ocr-latest} MISTRAL_OCR_MODEL: ${MISTRAL_OCR_MODEL:-mistral-ocr-latest}
DEEPGRAM_API_KEY: ${DEEPGRAM_API_KEY:-} DEEPGRAM_API_KEY: ${DEEPGRAM_API_KEY:-}
volumes:
- station-data:/app/data
expose:
- "8000"
ports:
- "8000"
restart: unless-stopped
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
frontend:
build:
context: ./frontend
dockerfile: Dockerfile
args:
NEXT_PUBLIC_API_URL: ""
BACKEND_INTERNAL_URL: http://backend:8000
environment:
NODE_ENV: production NODE_ENV: production
PORT: 3000 PORT: 3000
HOSTNAME: 0.0.0.0 HOSTNAME: 0.0.0.0
SERVICE_FQDN_APP: ${SERVICE_FQDN_APP:-}
SERVICE_URL_APP: ${SERVICE_URL_APP:-}
volumes:
- station-data:/app/data
expose: expose:
- "3000" - "3000"
ports:
- "3000"
depends_on:
- backend
restart: unless-stopped restart: unless-stopped
healthcheck: healthcheck:
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"] test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
+36 -15
View File
@@ -1,18 +1,39 @@
services: services:
backend: app:
build: .
ports:
- "8000:8000"
env_file:
- .env
volumes:
- ./data:/app/data
frontend:
build: build:
context: ./frontend context: .
ports: dockerfile: Dockerfile.coolify
- "3000:3000" args:
NEXT_PUBLIC_API_URL: ""
BACKEND_INTERNAL_URL: http://127.0.0.1:8000
environment: environment:
NEXT_PUBLIC_API_URL: http://127.0.0.1:8000 APP_ENV: production
depends_on: APP_DATA_DIR: /app/data
- backend DATABASE_PATH: /app/data/app.db
MISTRAL_API_KEY: ${MISTRAL_API_KEY:-}
MISTRAL_OCR_MODEL: ${MISTRAL_OCR_MODEL:-mistral-ocr-4-0}
MISTRAL_OCR_BATCH_THRESHOLD: ${MISTRAL_OCR_BATCH_THRESHOLD:-3}
MISTRAL_OCR_BATCH_POLL_SECONDS: ${MISTRAL_OCR_BATCH_POLL_SECONDS:-10}
MISTRAL_OCR_BATCH_TIMEOUT_SECONDS: ${MISTRAL_OCR_BATCH_TIMEOUT_SECONDS:-1800}
MISTRAL_OCR_RETRY_ATTEMPTS: ${MISTRAL_OCR_RETRY_ATTEMPTS:-3}
MISTRAL_OCR_RETRY_BACKOFF_BASE: ${MISTRAL_OCR_RETRY_BACKOFF_BASE:-2.0}
DEEPGRAM_API_KEY: ${DEEPGRAM_API_KEY:-}
NODE_ENV: production
PORT: 3000
HOSTNAME: 0.0.0.0
SERVICE_FQDN_APP: ${SERVICE_FQDN_APP:-}
SERVICE_URL_APP: ${SERVICE_URL_APP:-}
volumes:
- station-data:/app/data
expose:
- "3000"
restart: unless-stopped
healthcheck:
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
volumes:
station-data:
+61 -4
View File
@@ -10,6 +10,8 @@ export default function DashboardPage() {
const [subjects, setSubjects] = useState<Subject[]>([]); const [subjects, setSubjects] = useState<Subject[]>([]);
const [settings, setSettings] = useState<SettingsStatus | null>(null); const [settings, setSettings] = useState<SettingsStatus | null>(null);
const [name, setName] = useState(""); const [name, setName] = useState("");
const [editingSubject, setEditingSubject] = useState<Subject | null>(null);
const [editName, setEditName] = useState("");
const [loading, setLoading] = useState(true); const [loading, setLoading] = useState(true);
const [error, setError] = useState(""); const [error, setError] = useState("");
@@ -31,9 +33,47 @@ export default function DashboardPage() {
async function createSubject(event: FormEvent) { async function createSubject(event: FormEvent) {
event.preventDefault(); event.preventDefault();
if (!name.trim()) return; if (!name.trim()) return;
try {
setError("");
await api.createSubject(name.trim()); await api.createSubject(name.trim());
setName(""); setName("");
await load(); await load();
} catch (err) {
setError(err instanceof Error ? err.message : "No se pudo crear la materia");
}
}
function startEdit(subject: Subject) {
setEditingSubject(subject);
setEditName(subject.name);
setError("");
}
async function saveEdit(event: FormEvent) {
event.preventDefault();
if (!editingSubject || !editName.trim()) return;
try {
setError("");
await api.updateSubject(editingSubject.id, editName.trim());
setEditingSubject(null);
setEditName("");
await load();
} catch (err) {
setError(err instanceof Error ? err.message : "No se pudo editar la materia");
}
}
async function deleteSubject(subject: Subject) {
const confirmed = window.confirm(`Eliminar la materia "${subject.name}" tambien borrara todas sus semanas, archivos subidos, Markdown procesado y exports. Esta accion no se puede deshacer.`);
if (!confirmed) return;
try {
setError("");
await api.deleteSubject(subject.id);
if (editingSubject?.id === subject.id) setEditingSubject(null);
await load();
} catch (err) {
setError(err instanceof Error ? err.message : "No se pudo eliminar la materia");
}
} }
return ( return (
@@ -50,13 +90,15 @@ export default function DashboardPage() {
{loading ? <Card>Cargando...</Card> : ( {loading ? <Card>Cargando...</Card> : (
<div className="grid gap-4 md:grid-cols-2"> <div className="grid gap-4 md:grid-cols-2">
{subjects.map((subject) => ( {subjects.map((subject) => (
<Link key={subject.id} href={`/subjects/${subject.id}`}> <Card key={subject.id} className="transition hover:border-blue-400/40 hover:bg-blue-500/10">
<Card className="transition hover:border-blue-400/40 hover:bg-blue-500/10">
<div className="text-xl font-semibold">{subject.name}</div> <div className="text-xl font-semibold">{subject.name}</div>
<div className="mt-2 text-sm text-slate-400">{subject.slug}</div> <div className="mt-2 text-sm text-slate-400">{subject.slug}</div>
<div className="mt-5 text-sm text-blue-300">Abrir materia</div> <div className="mt-5 flex flex-wrap gap-2 text-sm">
<Link className="rounded-lg bg-blue-600 px-3 py-2 font-medium text-white hover:bg-blue-500" href={`/subjects/${subject.id}`}>Abrir materia</Link>
<button className="rounded-lg border border-white/10 px-3 py-2 text-slate-200 hover:bg-white/10" type="button" onClick={() => startEdit(subject)}>Editar</button>
<button className="rounded-lg border border-red-400/30 px-3 py-2 text-red-200 hover:bg-red-500/10" type="button" onClick={() => deleteSubject(subject)}>Eliminar</button>
</div>
</Card> </Card>
</Link>
))} ))}
{subjects.length === 0 && <Card className="md:col-span-2">No hay materias todavia. Crea la primera para empezar.</Card>} {subjects.length === 0 && <Card className="md:col-span-2">No hay materias todavia. Crea la primera para empezar.</Card>}
</div> </div>
@@ -64,6 +106,21 @@ export default function DashboardPage() {
</section> </section>
<aside className="space-y-4"> <aside className="space-y-4">
{editingSubject && (
<Card>
<h2 className="text-lg font-semibold">Editar materia</h2>
<p className="mt-1 text-sm text-slate-400">Solo cambia el nombre visible. La carpeta interna queda igual para mantener rutas estables.</p>
<form className="mt-4 space-y-3" onSubmit={saveEdit}>
<Label>Nombre</Label>
<Input value={editName} onChange={(event) => setEditName(event.target.value)} />
<div className="flex gap-2">
<Button className="flex-1" disabled={!editName.trim()}>Guardar</Button>
<button className="flex-1 rounded-xl border border-white/10 px-4 py-2 hover:bg-white/10" type="button" onClick={() => setEditingSubject(null)}>Cancelar</button>
</div>
</form>
</Card>
)}
<Card> <Card>
<h2 className="text-lg font-semibold">Nueva materia</h2> <h2 className="text-lg font-semibold">Nueva materia</h2>
<form className="mt-4 space-y-3" onSubmit={createSubject}> <form className="mt-4 space-y-3" onSubmit={createSubject}>
@@ -62,6 +62,19 @@ export default function SubjectPage({ params }: { params: { subjectId: string }
} }
} }
async function deleteWeek(week: Week) {
const confirmed = window.confirm(`Eliminar la semana ${week.number} tambien borrara sus archivos subidos, Markdown procesado y exports. Esta accion no se puede deshacer.`);
if (!confirmed) return;
try {
setError("");
await api.deleteWeek(subjectId, week.number);
if (editingWeek?.id === week.id) setEditingWeek(null);
await load();
} catch (err) {
setError(err instanceof Error ? err.message : "No se pudo eliminar la semana");
}
}
return ( return (
<Shell> <Shell>
<Link href="/" className="text-sm text-blue-300">Volver a materias</Link> <Link href="/" className="text-sm text-blue-300">Volver a materias</Link>
@@ -78,6 +91,7 @@ export default function SubjectPage({ params }: { params: { subjectId: string }
<div className="mt-5 flex flex-wrap gap-2 text-sm"> <div className="mt-5 flex flex-wrap gap-2 text-sm">
<Link className="rounded-lg bg-blue-600 px-3 py-2 font-medium text-white hover:bg-blue-500" href={`/subjects/${subjectId}/weeks/${week.number}`}>Abrir semana</Link> <Link className="rounded-lg bg-blue-600 px-3 py-2 font-medium text-white hover:bg-blue-500" href={`/subjects/${subjectId}/weeks/${week.number}`}>Abrir semana</Link>
<button className="rounded-lg border border-white/10 px-3 py-2 text-slate-200 hover:bg-white/10" type="button" onClick={() => startEdit(week)}>Editar</button> <button className="rounded-lg border border-white/10 px-3 py-2 text-slate-200 hover:bg-white/10" type="button" onClick={() => startEdit(week)}>Editar</button>
<button className="rounded-lg border border-red-400/30 px-3 py-2 text-red-200 hover:bg-red-500/10" type="button" onClick={() => deleteWeek(week)}>Eliminar</button>
</div> </div>
</Card> </Card>
))} ))}
+13
View File
@@ -65,6 +65,16 @@ export const api = {
body: JSON.stringify({ name }), body: JSON.stringify({ name }),
})); }));
}, },
async updateSubject(id: number, name: string) {
return parseResponse<Subject>(await fetch(`${API_URL}/subjects/${id}`, {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ name }),
}));
},
async deleteSubject(id: number) {
return parseResponse<{ status: string }>(await fetch(`${API_URL}/subjects/${id}`, { method: "DELETE" }));
},
async weeks(subjectId: number) { async weeks(subjectId: number) {
return parseResponse<Week[]>(await fetch(`${API_URL}/subjects/${subjectId}/weeks`, { cache: "no-store" })); return parseResponse<Week[]>(await fetch(`${API_URL}/subjects/${subjectId}/weeks`, { cache: "no-store" }));
}, },
@@ -85,6 +95,9 @@ export const api = {
body: JSON.stringify({ number, title: title || null }), body: JSON.stringify({ number, title: title || null }),
})); }));
}, },
async deleteWeek(subjectId: number, weekNumber: number) {
return parseResponse<{ status: string }>(await fetch(`${API_URL}/subjects/${subjectId}/weeks/${weekNumber}`, { method: "DELETE" }));
},
async sources(subjectId: number, weekNumber: number) { async sources(subjectId: number, weekNumber: number) {
return parseResponse<Source[]>(await fetch(`${API_URL}/subjects/${subjectId}/weeks/${weekNumber}/sources`, { cache: "no-store" })); return parseResponse<Source[]>(await fetch(`${API_URL}/subjects/${subjectId}/weeks/${weekNumber}/sources`, { cache: "no-store" }));
}, },