Compare commits

..
12 Commits
Author SHA1 Message Date
urieljareth 239a7510c6 OCR Mistral: modelo mistral-ocr-4-0, batch + fallback con reintentos y errores etiquetados
- Modelo por defecto mistral-ocr-4-0 (.env.example, config.py, docker-compose)
- MistralClient: OCR en lote via API Batch (/v1/ocr) subiendo JSONL, polling y descarga de resultados
- Reintentos con backoff exponencial en llamadas a Mistral (429/5xx/red); sin reintento en 401/400
- PdfProcessor: lote cuando >= umbral de paginas, fallback sincrono pagina a pagina, placeholder rastreable por pagina fallida
- Errores etiquetados ([AUTH_MISTRAL], [BATCH], [NETWORK], [OCR_EMPTY], [PDF_PARSE]...) en el mensaje del job
- Nuevas variables: MISTRAL_OCR_BATCH_THRESHOLD/POLL_SECONDS/TIMEOUT_SECONDS/RETRY_ATTEMPTS/RETRY_BACKOFF_BASE
2026-07-26 19:58:38 -06:00
urieljareth 379f563d04 Create CLAUDE.md 2026-06-10 02:01:45 -06:00
urieljareth 2160d4ff70 Commit inicial 2026-05-31 20:23:27 -06:00
urieljarethbusiness-cpu 7361f7ced1 1 2026-05-19 14:21:56 -06:00
urieljarethbusiness-cpu 4f3bc01dcb Update README.md 2026-05-19 13:21:06 -06:00
urieljarethbusiness-cpu 0df056445e 1 2026-05-19 11:23:47 -06:00
urieljarethbusiness-cpu d18ab1f618 Update Dockerfile.coolify 2026-05-19 10:11:36 -06:00
urieljarethbusiness-cpu b04079eaaf Update Dockerfile.coolify 2026-05-19 09:41:43 -06:00
urieljarethbusiness-cpu 7dc0f65045 1 2026-05-17 14:30:00 -06:00
urieljarethbusiness-cpu 8e2a7eadd2 Update docker-compose.yml 2026-05-17 13:18:03 -06:00
urieljarethbusiness-cpu 052580b75b 2 2026-05-17 12:57:23 -06:00
urieljarethbusiness-cpu 306d131136 Update docker-compose.yaml 2026-05-17 12:32:23 -06:00
17 changed files with 702 additions and 91 deletions
+1 -1
View File
@@ -14,6 +14,6 @@ frontend/.env.local
frontend/.env.*.local
node_modules
npm-debug.log*
Dockerfile*
Dockerfile
docker-compose*.yml
docker-compose*.yaml
+6 -1
View File
@@ -2,5 +2,10 @@ APP_ENV=development
APP_DATA_DIR=./data
DATABASE_PATH=./data/app.db
MISTRAL_API_KEY=
MISTRAL_OCR_MODEL=mistral-ocr-latest
MISTRAL_OCR_MODEL=mistral-ocr-4-0
MISTRAL_OCR_BATCH_THRESHOLD=3
MISTRAL_OCR_BATCH_POLL_SECONDS=10
MISTRAL_OCR_BATCH_TIMEOUT_SECONDS=1800
MISTRAL_OCR_RETRY_ATTEMPTS=3
MISTRAL_OCR_RETRY_BACKOFF_BASE=2.0
DEEPGRAM_API_KEY=
+46
View File
@@ -0,0 +1,46 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
> `AGENTS.md` contains the canonical command list, platform behavior notes, and deployment/secrets rules. Read it too — this file focuses on the cross-file architecture that AGENTS.md does not spell out.
## Commands
- **Backend (run from repo root, not `backend/`):** `uvicorn backend.app.main:app --reload`. Imports are absolute from the `backend.` package, so the working directory must be the repo root.
- **Frontend:** `cd frontend && npm run dev` (dev), `npm run build`, `npm run lint`.
- **Syntax check:** `python -m compileall backend` (there is no test suite in the repo).
- **Windows one-click:** `abrir_estacion.bat``scripts/start-platform.ps1` picks free ports, starts both servers, opens the browser; `cerrar_estacion.bat` stops only this project.
- **Container/prod:** `docker compose up --build`, or Nixpacks/`Dockerfile.coolify` → both run `scripts/start-coolify.sh`.
## Architecture
A two-process app for turning class materials (PDF/DOCX/text/audio/video) into LLM-ready Markdown, organized as **Materia → Semana → documentos**.
### Request → background-job flow
1. `backend/app/main.py` is the entire HTTP API (flat FastAPI app, no routers). Every endpoint is a thin try/except that delegates to `KnowledgeManager` and maps `ValueError` → 404/400, `DuplicateSourceError` → 409.
2. File upload saves the raw file, creates a **source** row + a **job** row, then schedules `process_job` via FastAPI `BackgroundTasks` (in-process — no external queue/worker). Job status transitions: `queued → processing → completed|failed`.
3. `process_job` calls `IngestionService.process_source`, which picks a processor by `source_type`, runs it, wraps the result in Markdown frontmatter, and persists a **document**.
### Key components
- **`KnowledgeManager`** (`managers/knowledge_manager.py`) — the single source of truth. Owns **both** the SQLite schema (`subjects`, `weeks`, `sources`, `documents`, `jobs`) and the on-disk layout under `data/subjects/<slug>/week-NN/{raw,processed,exports}/`. Filesystem and DB are kept in sync here; processors and endpoints must go through it, never write final Markdown or touch the DB directly. Schema migrations are done idempotently in `_init_db` via `_ensure_columns` (add-column-if-missing) — there is no migration framework.
- **`IngestionService`** (`services/ingestion_service.py`) — maps `source_type` → processor and assembles frontmatter metadata.
- **Processors** (`services/processors/`) — each returns a `ProcessedContent(title, body, processor)` dataclass (`base.py`). `text`, `docx` (python-docx), `pdf` (PyMuPDF text, or Mistral OCR when `use_ocr=true`), `audio` (Deepgram), `video` (ffmpeg → audio → Deepgram). PDFs accept `page_ranges` like `1-5,8,10-12`.
- **Providers** (`services/providers/`) — thin HTTP clients for Mistral OCR and Deepgram, keyed off `MISTRAL_API_KEY` / `DEEPGRAM_API_KEY`.
- **`markdown_builder.build_markdown`** — the only place that emits the YAML frontmatter block prepended to every processed document.
### Important invariants
- **One document per source.** `create_document` deletes any prior document/jobs for that source before inserting, then deletes the old Markdown file. Reprocessing replaces, never appends.
- **Dedup by hash.** `save_upload` rejects a file whose `sha256` already exists in the same week (`DuplicateSourceError` → 409).
- **Deletes cascade manually** (no FK cascade): deleting a subject/week/source/document removes child DB rows *and* unlinks files. Renaming a week (`update_week`) renames the directory and rewrites stored paths in the DB with SQL `replace()`.
- **Markdown is sanitized** (`_sanitize_markdown` strips control chars) on every read and write.
- **Week exports** (`export_week_markdown`) concatenate documents, stripping each doc's frontmatter and inserting `<!-- document_id: N -->` markers; the `.zip` variant ships individual files.
### Frontend
Next.js 14 App Router (`frontend/app/`), Tailwind, `react-markdown` + `remark-gfm`. All backend calls go through the single `api` object in `frontend/lib/api.ts`. `API_URL` resolves to `http://127.0.0.1:8000` in dev (via `NEXT_PUBLIC_API_URL`) or `/api` in prod, where `next.config.mjs` `rewrites()` proxy `/api/*` to `BACKEND_INTERNAL_URL`. Pages mirror the data model: `/`, `/subjects/[subjectId]`, `/subjects/[subjectId]/weeks/[weekNumber]`, `/documents/[documentId]`, `/settings`.
### Deployment shape
Single container runs **both** processes: `start-coolify.sh` launches uvicorn on `127.0.0.1:8000` (internal only) and Next.js standalone on `PORT` (default 3000, public). `/app/data` **must** be a mounted volume or all subjects/documents/SQLite state are lost on redeploy. `ffmpeg` is required at runtime for video.
## Conventions
- UX, error messages, and generated content are in **Spanish** — preserve this. Source comments/identifiers are mixed Spanish/English; match the surrounding file.
- The standalone `estacion documentos.html` is a browser-only prototype kept for reference behavior; the real app is `backend/` + `frontend/`.
+42
View File
@@ -0,0 +1,42 @@
FROM node:22-bookworm-slim AS frontend-builder
WORKDIR /app/frontend
COPY frontend/package.json frontend/package-lock.json* ./
RUN npm ci
COPY frontend ./
ARG NEXT_PUBLIC_API_URL=
ARG BACKEND_INTERNAL_URL=http://127.0.0.1:8000
ENV NEXT_PUBLIC_API_URL=$NEXT_PUBLIC_API_URL
ENV BACKEND_INTERNAL_URL=$BACKEND_INTERNAL_URL
RUN npm run build
FROM python:3.12-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV NODE_ENV=production
ENV HOSTNAME=0.0.0.0
ENV DEBIAN_FRONTEND=noninteractive
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates ffmpeg \
&& rm -rf /var/lib/apt/lists/*
COPY --from=frontend-builder /usr/local/bin/node /usr/local/bin/node
COPY --from=frontend-builder /usr/local/bin/npm /usr/local/bin/npm
COPY --from=frontend-builder /usr/local/bin/npx /usr/local/bin/npx
COPY --from=frontend-builder /usr/local/lib/node_modules /usr/local/lib/node_modules
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY backend ./backend
COPY scripts ./scripts
COPY --from=frontend-builder /app/frontend/.next ./frontend/.next
COPY --from=frontend-builder /app/frontend/package.json ./frontend/package.json
COPY --from=frontend-builder /app/frontend/node_modules ./frontend/node_modules
EXPOSE 3000
CMD ["bash", "scripts/start-coolify.sh"]
BIN
View File
Binary file not shown.
+6 -1
View File
@@ -8,7 +8,12 @@ class Settings(BaseSettings):
app_data_dir: Path = Path("./data")
database_path: Path = Path("./data/app.db")
mistral_api_key: str = ""
mistral_ocr_model: str = "mistral-ocr-latest"
mistral_ocr_model: str = "mistral-ocr-4-0"
mistral_ocr_batch_threshold: int = 3
mistral_ocr_batch_poll_seconds: int = 10
mistral_ocr_batch_timeout_seconds: int = 1800
mistral_ocr_retry_attempts: int = 3
mistral_ocr_retry_backoff_base: float = 2.0
deepgram_api_key: str = ""
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
+15 -2
View File
@@ -8,7 +8,8 @@ from fastapi.responses import FileResponse, PlainTextResponse
from backend.app.config import settings
from backend.app.managers.knowledge_manager import DuplicateSourceError, KnowledgeManager
from backend.app.models.schemas import MarkdownUpdate, SubjectCreate, WeekCreate, WeekUpdate
from backend.app.models.schemas import MarkdownUpdate, SubjectCreate, SubjectUpdate, WeekCreate, WeekUpdate
from backend.app.services.errors import ProcessingError, classify_unknown_error
from backend.app.services.ingestion_service import IngestionService
app = FastAPI(title="Knowledge Station", version="0.1.0")
@@ -60,6 +61,14 @@ def get_subject(subject_id: int) -> dict:
raise HTTPException(status_code=404, detail=str(exc)) from exc
@app.put("/subjects/{subject_id}")
def update_subject(subject_id: int, payload: SubjectUpdate) -> dict:
try:
return manager.update_subject(subject_id, payload.name)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
@app.delete("/subjects/{subject_id}")
def delete_subject(subject_id: int) -> dict:
try:
@@ -276,6 +285,10 @@ def process_job(job_id: int, source_id: int, use_ocr: bool, page_ranges: str | N
try:
document = ingestion.process_source(source_id, use_ocr=use_ocr, page_ranges=page_ranges)
manager.update_job(job_id, "completed", "Documento procesado", document_id=document["id"])
except ProcessingError as exc:
manager.update_source_status(source_id, "failed")
manager.update_job(job_id, "failed", exc.labeled)
except Exception as exc:
manager.update_source_status(source_id, "failed")
manager.update_job(job_id, "failed", str(exc))
code = classify_unknown_error(exc)
manager.update_job(job_id, "failed", f"[{code}] {exc}")
+26
View File
@@ -154,6 +154,14 @@ class KnowledgeManager:
raise ValueError("Materia no encontrada")
return dict(row)
def update_subject(self, subject_id: int, name: str) -> dict[str, Any]:
if not name.strip():
raise ValueError("El nombre de la materia no puede estar vacio")
self.get_subject(subject_id)
with self._connect() as conn:
conn.execute("update subjects set name = ? where id = ?", (name.strip(), subject_id))
return self.get_subject(subject_id)
def delete_subject(self, subject_id: int) -> None:
subject = self.get_subject(subject_id)
with self._connect() as conn:
@@ -224,6 +232,8 @@ class KnowledgeManager:
def update_week(self, subject_id: int, current_number: int, new_number: int, title: str | None = None) -> dict[str, Any]:
subject = self.get_subject(subject_id)
week = self.get_week(subject_id, current_number)
old_dir: Path | None = None
new_dir: Path | None = None
if new_number != current_number:
with self._connect() as conn:
existing = conn.execute(
@@ -248,6 +258,22 @@ class KnowledgeManager:
"update weeks set number = ?, title = ? where id = ?",
(new_number, title, week["id"]),
)
if old_dir and new_dir:
old_prefix = str(old_dir)
new_prefix = str(new_dir)
conn.execute(
"update sources set stored_path = replace(stored_path, ?, ?) where week_id = ? and stored_path like ?",
(old_prefix, new_prefix, week["id"], f"{old_prefix}%"),
)
conn.execute(
"""
update documents
set markdown_path = replace(markdown_path, ?, ?)
where source_id in (select id from sources where week_id = ?)
and markdown_path like ?
""",
(old_prefix, new_prefix, week["id"], f"{old_prefix}%"),
)
return self.get_week(subject_id, new_number)
def week_dir(self, subject_slug: str, week_number: int) -> Path:
+4
View File
@@ -8,6 +8,10 @@ class SubjectCreate(BaseModel):
name: str = Field(min_length=1)
class SubjectUpdate(BaseModel):
name: str = Field(min_length=1)
class Subject(BaseModel):
id: int
name: str
+85
View File
@@ -0,0 +1,85 @@
from __future__ import annotations
import requests
class ProcessingError(Exception):
"""Error de procesamiento con un codigo corto y rastreable.
El atributo ``code`` es un identificador estable en MAYUSCULAS que se
antepone al mensaje del job (p.ej. ``[AUTH_MISTRAL] API Key invalida``)
para que el usuario pueda buscar y diagnosticar rapidamente el tipo de
fallo desde la UI.
"""
def __init__(self, code: str, detail: str) -> None:
super().__init__(f"[{code}] {detail}")
self.code = code
self.detail = detail
@property
def labeled(self) -> str:
return f"[{self.code}] {self.detail}"
class MistralAuthError(ProcessingError):
def __init__(self, detail: str = "API Key de Mistral invalida o no autorizada (401)") -> None:
super().__init__("AUTH_MISTRAL", detail)
class MistralRateLimitError(ProcessingError):
def __init__(self, detail: str = "Limite de uso de Mistral alcanzado (429)") -> None:
super().__init__("RATE_LIMIT", detail)
class MistralServerError(ProcessingError):
def __init__(self, status: int, body: str) -> None:
super().__init__("MISTRAL_SERVER", f"Mistral HTTP {status}: {body[:200]}")
class MistralBatchError(ProcessingError):
def __init__(self, detail: str) -> None:
super().__init__("BATCH", detail)
class MistralOCREmptyError(ProcessingError):
def __init__(self, detail: str = "OCR no devolvio contenido") -> None:
super().__init__("OCR_EMPTY", detail)
class NetworkError(ProcessingError):
def __init__(self, detail: str) -> None:
super().__init__("NETWORK", detail)
NON_RETRYABLE_CODES = {
"AUTH_MISTRAL",
"CONFIG",
"OCR_EMPTY",
"MISTRAL_CLIENT",
"BAD_PDF",
"FILE_IO",
}
def classify_requests_error(exc: requests.RequestException) -> ProcessingError:
if isinstance(exc, requests.Timeout):
return NetworkError(f"timeout de red con Mistral: {exc}")
if isinstance(exc, requests.ConnectionError):
return NetworkError(f"sin conexion a Mistral: {exc}")
return NetworkError(f"error de red con Mistral: {exc}")
def classify_unknown_error(exc: Exception) -> str:
"""Etiqueta breve para excepciones no envueltas en ProcessingError."""
name = type(exc).__name__.lower()
message = str(exc).lower()
if "fitz" in name or "pdf" in name or "pdf" in message or "muPDF" in message:
return "PDF_PARSE"
if isinstance(exc, FileNotFoundError):
return "FILE_IO"
if isinstance(exc, (PermissionError, OSError)):
return "FILE_IO"
if isinstance(exc, ValueError):
return "VALIDATION"
return "UNKNOWN"
@@ -4,8 +4,10 @@ from pathlib import Path
import fitz
from backend.app.config import settings
from backend.app.services.markdown_builder import title_from_path
from backend.app.services.processors.base import ProcessedContent
from backend.app.services.errors import ProcessingError
from backend.app.services.providers.mistral_client import MistralClient
@@ -30,22 +32,77 @@ class PdfProcessor:
return ProcessedContent(title=title_from_path(path), body=body, processor="pdf-text")
def _process_ocr(self, path: Path) -> ProcessedContent:
doc = fitz.open(path)
client = MistralClient()
parts: list[str] = []
temp_paths: list[Path] = []
try:
for index in self._selected_pages(len(doc)):
doc = fitz.open(path)
except Exception as exc:
raise ProcessingError("BAD_PDF", f"No se pudo abrir el PDF: {exc}") from exc
selected = self._selected_pages(len(doc))
client = MistralClient()
page_images: dict[int, Path] = {}
try:
for index in selected:
page = doc[index - 1]
pix = page.get_pixmap(matrix=fitz.Matrix(2, 2), alpha=False)
image_path = path.with_name(f"{path.stem}-page-{index}.jpg")
pix.save(image_path)
temp_paths.append(image_path)
parts.append(f"## Pagina {index}\n\n{client.ocr_image(image_path)}")
page_images[index] = image_path
results = self._ocr_pages(client, page_images)
finally:
for temp_path in temp_paths:
temp_path.unlink(missing_ok=True)
return ProcessedContent(title=title_from_path(path), body="\n\n".join(parts), processor="mistral-ocr")
for image_path in page_images.values():
image_path.unlink(missing_ok=True)
parts: list[str] = []
for index in selected:
markdown = results.get(index, "")
parts.append(f"## Pagina {index}\n\n{markdown}")
return ProcessedContent(
title=title_from_path(path),
body="\n\n".join(parts),
processor="mistral-ocr",
)
def _ocr_pages(self, client: MistralClient, page_images: dict[int, Path]) -> dict[int, str]:
"""OCR de las paginas con lote + fallback sincrono.
Estrategia:
1. Si hay >= ``batch_threshold`` paginas, intenta el job en lote.
2. Cualquier pagina no resuelta (o si el lote entero falla) cae a
OCR sincrono pagina a pagina con reintentos.
3. Las paginas que fallen de forma definitiva quedan como un
placeholder breve con el codigo de error rastreable.
"""
indices = list(page_images.keys())
results: dict[int, str] = {}
page_errors: dict[int, str] = {}
if len(indices) >= settings.mistral_ocr_batch_threshold:
ordered = sorted(indices)
paths = [page_images[i] for i in ordered]
try:
batch_results = client.ocr_images_batch(paths)
for pos, index in enumerate(ordered):
if pos in batch_results:
results[index] = batch_results[pos]
except ProcessingError:
# El lote fallo; continua al fallback sincrono para todas
# las paginas no resueltas.
pass
for index in indices:
if index in results:
continue
try:
results[index] = client.ocr_image(page_images[index])
except ProcessingError as exc:
page_errors[index] = exc.labeled
except Exception as exc:
page_errors[index] = f"[OCR_ERROR] {exc}"
for index in indices:
if index not in results:
results[index] = f"> ⚠️ No se pudo procesar esta pagina: {page_errors.get(index, '[UNKNOWN]')}"
return results
def _selected_pages(self, total_pages: int) -> list[int]:
if total_pages < 1:
+261 -19
View File
@@ -1,45 +1,287 @@
from __future__ import annotations
import base64
import json
import time
from pathlib import Path
import requests
from backend.app.config import settings
from backend.app.services.errors import (
MistralAuthError,
MistralBatchError,
MistralOCREmptyError,
MistralRateLimitError,
MistralServerError,
NetworkError,
NON_RETRYABLE_CODES,
ProcessingError,
classify_requests_error,
)
class MistralClient:
endpoint = "https://api.mistral.ai/v1/ocr"
"""Cliente Mistral para OCR.
- ``ocr_image``: OCR sincrono de una pagina, con reintentos y backoff.
- ``ocr_images_batch``: OCR en lote via la API de Batch de Mistral
(endpoint ``/v1/ocr``), subiendo un fichero JSONL. Devuelve un dict
``{page_index: markdown}``. Si el job falla o caduca se lanza
``MistralBatchError`` para que el llamador caiga al modo sincrono.
"""
base_url = "https://api.mistral.ai/v1"
ocr_url = f"{base_url}/ocr"
files_url = f"{base_url}/files"
batch_url = f"{base_url}/batch/jobs"
def __init__(self) -> None:
self._api_key = settings.mistral_api_key
self._model = settings.mistral_ocr_model
# ------------------------------------------------------------------
# API publica
# ------------------------------------------------------------------
def ocr_image(self, image_path: Path) -> str:
if not settings.mistral_api_key:
raise RuntimeError("MISTRAL_API_KEY no esta configurada")
self._require_key()
data_url = self._image_data_url(image_path)
return self._retry(lambda: self._ocr_sync(data_url))
def ocr_images_batch(self, image_paths: list[Path]) -> dict[int, str]:
"""OCR en lote. Devuelve ``{posicion_en_lista: markdown}``.
``posicion_en_lista`` es el indice (0-based) dentro de ``image_paths``,
NO el numero de pagina del PDF; el llamador hace esa traduccion.
"""
self._require_key()
if not image_paths:
return {}
payload = self._build_batch_payload(image_paths)
upload = self._retry(lambda: self._upload_batch_file(payload))
file_id = upload.get("id")
if not file_id:
raise MistralBatchError("la subida del fichero batch no devolvio id")
job = self._retry(lambda: self._create_batch_job(file_id))
job_id = job.get("id")
if not job_id:
raise MistralBatchError("la creacion del job batch no devolvio id")
final = self._poll_batch_job(job_id)
if final.get("status") != "SUCCESS":
reason = final.get("errors") or final.get("status") or "sin detalle"
raise MistralBatchError(f"job {job_id} finalizo como {final.get('status')}: {reason}")
output_file_id = final.get("output_file")
if not output_file_id:
raise MistralBatchError(f"job {job_id} sin output_file")
content = self._retry(lambda: self._download_file(output_file_id))
return self._parse_batch_output(content)
# ------------------------------------------------------------------
# OCR sincrono
# ------------------------------------------------------------------
def _ocr_sync(self, data_url: str) -> str:
try:
response = requests.post(
self.endpoint,
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {settings.mistral_api_key}",
},
self.ocr_url,
headers=self._json_headers(),
json={
"model": settings.mistral_ocr_model,
"model": self._model,
"document": {"type": "image_url", "image_url": data_url},
"include_image_base64": False,
},
timeout=180,
)
if response.status_code == 401:
raise RuntimeError("API Key de Mistral invalida")
if not response.ok:
raise RuntimeError(f"Mistral error {response.status_code}: {response.text}")
data = response.json()
except requests.RequestException as exc:
raise classify_requests_error(exc) from exc
self._raise_for_status(response)
markdown = self._extract_markdown(response.json())
if not markdown:
raise MistralOCREmptyError()
return markdown
# ------------------------------------------------------------------
# Batch: construccion, subida, job, polling, descarga, parseo
# ------------------------------------------------------------------
def _build_batch_payload(self, image_paths: list[Path]) -> bytes:
lines: list[str] = []
for idx, path in enumerate(image_paths):
data_url = self._image_data_url(path)
body = {
"document": {"type": "image_url", "image_url": data_url},
"include_image_base64": False,
}
lines.append(
json.dumps(
{"custom_id": str(idx), "body": body},
separators=(",", ":"),
ensure_ascii=False,
)
)
return ("\n".join(lines)).encode("utf-8")
def _upload_batch_file(self, payload: bytes) -> dict:
try:
response = requests.post(
self.files_url,
headers={"Authorization": f"Bearer {self._api_key}"},
data={"purpose": "batch"},
files={"file": ("ocr-batch.jsonl", payload, "application/jsonl")},
timeout=300,
)
except requests.RequestException as exc:
raise classify_requests_error(exc) from exc
self._raise_for_status(response)
return response.json()
def _create_batch_job(self, file_id: str) -> dict:
try:
response = requests.post(
self.batch_url,
headers=self._json_headers(),
json={
"input_files": [file_id],
"model": self._model,
"endpoint": "/v1/ocr",
},
timeout=120,
)
except requests.RequestException as exc:
raise classify_requests_error(exc) from exc
self._raise_for_status(response)
return response.json()
def _poll_batch_job(self, job_id: str) -> dict:
terminal = {"SUCCESS", "FAILED", "TIMEOUT_EXCEEDED", "CANCELLED"}
deadline = time.time() + max(60, settings.mistral_ocr_batch_timeout_seconds)
delay = max(2, settings.mistral_ocr_batch_poll_seconds)
last: dict = {}
while time.time() < deadline:
try:
response = requests.get(
f"{self.batch_url}/{job_id}",
headers=self._json_headers(),
timeout=60,
)
except requests.RequestException:
time.sleep(delay)
continue
if response.ok:
last = response.json()
if last.get("status") in terminal:
return last
time.sleep(delay)
raise MistralBatchError(
f"job {job_id} excedio el timeout de polling "
f"({settings.mistral_ocr_batch_timeout_seconds}s); ultimo estado: "
f"{last.get('status') if last else 'desconocido'}"
)
def _download_file(self, file_id: str) -> bytes:
try:
response = requests.get(
f"{self.files_url}/{file_id}/content",
headers=self._json_headers(),
timeout=300,
)
except requests.RequestException as exc:
raise classify_requests_error(exc) from exc
self._raise_for_status(response)
return response.content
def _parse_batch_output(self, content: bytes) -> dict[int, str]:
results: dict[int, str] = {}
for raw in content.decode("utf-8", errors="replace").splitlines():
line = raw.strip()
if not line:
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
custom_id = obj.get("custom_id")
if custom_id is None:
continue
try:
idx = int(custom_id)
except (TypeError, ValueError):
continue
response = obj.get("response") or {}
status_code = response.get("status_code", 200)
body = response.get("body") or {}
if status_code >= 400 or obj.get("error"):
# Pagina fallida en lote: se omite y el llamador cae a sync.
continue
markdown = self._extract_markdown(body)
if markdown:
results[idx] = markdown
return results
# ------------------------------------------------------------------
# Helpers compartidos
# ------------------------------------------------------------------
def _require_key(self) -> None:
if not self._api_key:
raise ProcessingError("CONFIG", "MISTRAL_API_KEY no esta configurada")
def _json_headers(self) -> dict[str, str]:
return {
"Authorization": f"Bearer {self._api_key}",
"Content-Type": "application/json",
"Accept": "application/json",
}
def _raise_for_status(self, response: requests.Response) -> None:
if response.ok:
return
try:
body = response.text
except Exception:
body = ""
status = response.status_code
if status == 401:
raise MistralAuthError()
if status == 429:
raise MistralRateLimitError()
if status >= 500:
raise MistralServerError(status, body)
raise ProcessingError("MISTRAL_CLIENT", f"HTTP {status}: {body[:200]}")
def _retry(self, fn):
attempts = max(1, settings.mistral_ocr_retry_attempts)
base_delay = max(0.5, settings.mistral_ocr_retry_backoff_base)
last_exc: Exception | None = None
for attempt in range(attempts):
try:
return fn()
except ProcessingError as exc:
if exc.code in NON_RETRYABLE_CODES:
raise
last_exc = exc
except Exception as exc:
last_exc = exc
if attempt < attempts - 1:
time.sleep(base_delay * (2 ** attempt))
if last_exc is not None:
raise last_exc
raise MistralBatchError("reintentos agotados sin excepcion conocida")
def _extract_markdown(self, data: dict) -> str:
pages = data.get("pages") or []
markdown_pages = [page.get("markdown", "").strip() for page in pages if page.get("markdown")]
if markdown_pages:
return "\n\n".join(markdown_pages)
chunks = [page.get("markdown", "").strip() for page in pages if page.get("markdown")]
if chunks:
return "\n\n".join(chunks)
if data.get("markdown"):
return data["markdown"].strip()
raise RuntimeError("Mistral OCR no devolvio Markdown")
return str(data["markdown"]).strip()
return ""
def _image_data_url(self, image_path: Path) -> str:
encoded = base64.b64encode(image_path.read_bytes()).decode("ascii")
+9 -28
View File
@@ -1,8 +1,11 @@
services:
backend:
app:
build:
context: .
dockerfile: Dockerfile
dockerfile: Dockerfile.coolify
args:
NEXT_PUBLIC_API_URL: ""
BACKEND_INTERNAL_URL: http://127.0.0.1:8000
environment:
APP_ENV: production
APP_DATA_DIR: /app/data
@@ -10,37 +13,15 @@ services:
MISTRAL_API_KEY: ${MISTRAL_API_KEY:-}
MISTRAL_OCR_MODEL: ${MISTRAL_OCR_MODEL:-mistral-ocr-latest}
DEEPGRAM_API_KEY: ${DEEPGRAM_API_KEY:-}
volumes:
- station-data:/app/data
expose:
- "8000"
ports:
- "8000"
restart: unless-stopped
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
frontend:
build:
context: ./frontend
dockerfile: Dockerfile
args:
NEXT_PUBLIC_API_URL: ""
BACKEND_INTERNAL_URL: http://backend:8000
environment:
NODE_ENV: production
PORT: 3000
HOSTNAME: 0.0.0.0
SERVICE_FQDN_APP: ${SERVICE_FQDN_APP:-}
SERVICE_URL_APP: ${SERVICE_URL_APP:-}
volumes:
- station-data:/app/data
expose:
- "3000"
ports:
- "3000"
depends_on:
- backend
restart: unless-stopped
healthcheck:
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
+36 -15
View File
@@ -1,18 +1,39 @@
services:
backend:
build: .
ports:
- "8000:8000"
env_file:
- .env
volumes:
- ./data:/app/data
frontend:
app:
build:
context: ./frontend
ports:
- "3000:3000"
context: .
dockerfile: Dockerfile.coolify
args:
NEXT_PUBLIC_API_URL: ""
BACKEND_INTERNAL_URL: http://127.0.0.1:8000
environment:
NEXT_PUBLIC_API_URL: http://127.0.0.1:8000
depends_on:
- backend
APP_ENV: production
APP_DATA_DIR: /app/data
DATABASE_PATH: /app/data/app.db
MISTRAL_API_KEY: ${MISTRAL_API_KEY:-}
MISTRAL_OCR_MODEL: ${MISTRAL_OCR_MODEL:-mistral-ocr-4-0}
MISTRAL_OCR_BATCH_THRESHOLD: ${MISTRAL_OCR_BATCH_THRESHOLD:-3}
MISTRAL_OCR_BATCH_POLL_SECONDS: ${MISTRAL_OCR_BATCH_POLL_SECONDS:-10}
MISTRAL_OCR_BATCH_TIMEOUT_SECONDS: ${MISTRAL_OCR_BATCH_TIMEOUT_SECONDS:-1800}
MISTRAL_OCR_RETRY_ATTEMPTS: ${MISTRAL_OCR_RETRY_ATTEMPTS:-3}
MISTRAL_OCR_RETRY_BACKOFF_BASE: ${MISTRAL_OCR_RETRY_BACKOFF_BASE:-2.0}
DEEPGRAM_API_KEY: ${DEEPGRAM_API_KEY:-}
NODE_ENV: production
PORT: 3000
HOSTNAME: 0.0.0.0
SERVICE_FQDN_APP: ${SERVICE_FQDN_APP:-}
SERVICE_URL_APP: ${SERVICE_URL_APP:-}
volumes:
- station-data:/app/data
expose:
- "3000"
restart: unless-stopped
healthcheck:
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
volumes:
station-data:
+61 -4
View File
@@ -10,6 +10,8 @@ export default function DashboardPage() {
const [subjects, setSubjects] = useState<Subject[]>([]);
const [settings, setSettings] = useState<SettingsStatus | null>(null);
const [name, setName] = useState("");
const [editingSubject, setEditingSubject] = useState<Subject | null>(null);
const [editName, setEditName] = useState("");
const [loading, setLoading] = useState(true);
const [error, setError] = useState("");
@@ -31,9 +33,47 @@ export default function DashboardPage() {
async function createSubject(event: FormEvent) {
event.preventDefault();
if (!name.trim()) return;
try {
setError("");
await api.createSubject(name.trim());
setName("");
await load();
} catch (err) {
setError(err instanceof Error ? err.message : "No se pudo crear la materia");
}
}
function startEdit(subject: Subject) {
setEditingSubject(subject);
setEditName(subject.name);
setError("");
}
async function saveEdit(event: FormEvent) {
event.preventDefault();
if (!editingSubject || !editName.trim()) return;
try {
setError("");
await api.updateSubject(editingSubject.id, editName.trim());
setEditingSubject(null);
setEditName("");
await load();
} catch (err) {
setError(err instanceof Error ? err.message : "No se pudo editar la materia");
}
}
async function deleteSubject(subject: Subject) {
const confirmed = window.confirm(`Eliminar la materia "${subject.name}" tambien borrara todas sus semanas, archivos subidos, Markdown procesado y exports. Esta accion no se puede deshacer.`);
if (!confirmed) return;
try {
setError("");
await api.deleteSubject(subject.id);
if (editingSubject?.id === subject.id) setEditingSubject(null);
await load();
} catch (err) {
setError(err instanceof Error ? err.message : "No se pudo eliminar la materia");
}
}
return (
@@ -50,13 +90,15 @@ export default function DashboardPage() {
{loading ? <Card>Cargando...</Card> : (
<div className="grid gap-4 md:grid-cols-2">
{subjects.map((subject) => (
<Link key={subject.id} href={`/subjects/${subject.id}`}>
<Card className="transition hover:border-blue-400/40 hover:bg-blue-500/10">
<Card key={subject.id} className="transition hover:border-blue-400/40 hover:bg-blue-500/10">
<div className="text-xl font-semibold">{subject.name}</div>
<div className="mt-2 text-sm text-slate-400">{subject.slug}</div>
<div className="mt-5 text-sm text-blue-300">Abrir materia</div>
<div className="mt-5 flex flex-wrap gap-2 text-sm">
<Link className="rounded-lg bg-blue-600 px-3 py-2 font-medium text-white hover:bg-blue-500" href={`/subjects/${subject.id}`}>Abrir materia</Link>
<button className="rounded-lg border border-white/10 px-3 py-2 text-slate-200 hover:bg-white/10" type="button" onClick={() => startEdit(subject)}>Editar</button>
<button className="rounded-lg border border-red-400/30 px-3 py-2 text-red-200 hover:bg-red-500/10" type="button" onClick={() => deleteSubject(subject)}>Eliminar</button>
</div>
</Card>
</Link>
))}
{subjects.length === 0 && <Card className="md:col-span-2">No hay materias todavia. Crea la primera para empezar.</Card>}
</div>
@@ -64,6 +106,21 @@ export default function DashboardPage() {
</section>
<aside className="space-y-4">
{editingSubject && (
<Card>
<h2 className="text-lg font-semibold">Editar materia</h2>
<p className="mt-1 text-sm text-slate-400">Solo cambia el nombre visible. La carpeta interna queda igual para mantener rutas estables.</p>
<form className="mt-4 space-y-3" onSubmit={saveEdit}>
<Label>Nombre</Label>
<Input value={editName} onChange={(event) => setEditName(event.target.value)} />
<div className="flex gap-2">
<Button className="flex-1" disabled={!editName.trim()}>Guardar</Button>
<button className="flex-1 rounded-xl border border-white/10 px-4 py-2 hover:bg-white/10" type="button" onClick={() => setEditingSubject(null)}>Cancelar</button>
</div>
</form>
</Card>
)}
<Card>
<h2 className="text-lg font-semibold">Nueva materia</h2>
<form className="mt-4 space-y-3" onSubmit={createSubject}>
@@ -62,6 +62,19 @@ export default function SubjectPage({ params }: { params: { subjectId: string }
}
}
async function deleteWeek(week: Week) {
const confirmed = window.confirm(`Eliminar la semana ${week.number} tambien borrara sus archivos subidos, Markdown procesado y exports. Esta accion no se puede deshacer.`);
if (!confirmed) return;
try {
setError("");
await api.deleteWeek(subjectId, week.number);
if (editingWeek?.id === week.id) setEditingWeek(null);
await load();
} catch (err) {
setError(err instanceof Error ? err.message : "No se pudo eliminar la semana");
}
}
return (
<Shell>
<Link href="/" className="text-sm text-blue-300">Volver a materias</Link>
@@ -78,6 +91,7 @@ export default function SubjectPage({ params }: { params: { subjectId: string }
<div className="mt-5 flex flex-wrap gap-2 text-sm">
<Link className="rounded-lg bg-blue-600 px-3 py-2 font-medium text-white hover:bg-blue-500" href={`/subjects/${subjectId}/weeks/${week.number}`}>Abrir semana</Link>
<button className="rounded-lg border border-white/10 px-3 py-2 text-slate-200 hover:bg-white/10" type="button" onClick={() => startEdit(week)}>Editar</button>
<button className="rounded-lg border border-red-400/30 px-3 py-2 text-red-200 hover:bg-red-500/10" type="button" onClick={() => deleteWeek(week)}>Eliminar</button>
</div>
</Card>
))}
+13
View File
@@ -65,6 +65,16 @@ export const api = {
body: JSON.stringify({ name }),
}));
},
async updateSubject(id: number, name: string) {
return parseResponse<Subject>(await fetch(`${API_URL}/subjects/${id}`, {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ name }),
}));
},
async deleteSubject(id: number) {
return parseResponse<{ status: string }>(await fetch(`${API_URL}/subjects/${id}`, { method: "DELETE" }));
},
async weeks(subjectId: number) {
return parseResponse<Week[]>(await fetch(`${API_URL}/subjects/${subjectId}/weeks`, { cache: "no-store" }));
},
@@ -85,6 +95,9 @@ export const api = {
body: JSON.stringify({ number, title: title || null }),
}));
},
async deleteWeek(subjectId: number, weekNumber: number) {
return parseResponse<{ status: string }>(await fetch(`${API_URL}/subjects/${subjectId}/weeks/${weekNumber}`, { method: "DELETE" }));
},
async sources(subjectId: number, weekNumber: number) {
return parseResponse<Source[]>(await fetch(`${API_URL}/subjects/${subjectId}/weeks/${weekNumber}/sources`, { cache: "no-store" }));
},