Compare commits
2
Commits
2160d4ff70
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
239a7510c6 | ||
|
|
379f563d04 |
+6
-1
@@ -2,5 +2,10 @@ APP_ENV=development
|
|||||||
APP_DATA_DIR=./data
|
APP_DATA_DIR=./data
|
||||||
DATABASE_PATH=./data/app.db
|
DATABASE_PATH=./data/app.db
|
||||||
MISTRAL_API_KEY=
|
MISTRAL_API_KEY=
|
||||||
MISTRAL_OCR_MODEL=mistral-ocr-latest
|
MISTRAL_OCR_MODEL=mistral-ocr-4-0
|
||||||
|
MISTRAL_OCR_BATCH_THRESHOLD=3
|
||||||
|
MISTRAL_OCR_BATCH_POLL_SECONDS=10
|
||||||
|
MISTRAL_OCR_BATCH_TIMEOUT_SECONDS=1800
|
||||||
|
MISTRAL_OCR_RETRY_ATTEMPTS=3
|
||||||
|
MISTRAL_OCR_RETRY_BACKOFF_BASE=2.0
|
||||||
DEEPGRAM_API_KEY=
|
DEEPGRAM_API_KEY=
|
||||||
|
|||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||||
|
|
||||||
|
> `AGENTS.md` contains the canonical command list, platform behavior notes, and deployment/secrets rules. Read it too — this file focuses on the cross-file architecture that AGENTS.md does not spell out.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
- **Backend (run from repo root, not `backend/`):** `uvicorn backend.app.main:app --reload`. Imports are absolute from the `backend.` package, so the working directory must be the repo root.
|
||||||
|
- **Frontend:** `cd frontend && npm run dev` (dev), `npm run build`, `npm run lint`.
|
||||||
|
- **Syntax check:** `python -m compileall backend` (there is no test suite in the repo).
|
||||||
|
- **Windows one-click:** `abrir_estacion.bat` → `scripts/start-platform.ps1` picks free ports, starts both servers, opens the browser; `cerrar_estacion.bat` stops only this project.
|
||||||
|
- **Container/prod:** `docker compose up --build`, or Nixpacks/`Dockerfile.coolify` → both run `scripts/start-coolify.sh`.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
A two-process app for turning class materials (PDF/DOCX/text/audio/video) into LLM-ready Markdown, organized as **Materia → Semana → documentos**.
|
||||||
|
|
||||||
|
### Request → background-job flow
|
||||||
|
1. `backend/app/main.py` is the entire HTTP API (flat FastAPI app, no routers). Every endpoint is a thin try/except that delegates to `KnowledgeManager` and maps `ValueError` → 404/400, `DuplicateSourceError` → 409.
|
||||||
|
2. File upload saves the raw file, creates a **source** row + a **job** row, then schedules `process_job` via FastAPI `BackgroundTasks` (in-process — no external queue/worker). Job status transitions: `queued → processing → completed|failed`.
|
||||||
|
3. `process_job` calls `IngestionService.process_source`, which picks a processor by `source_type`, runs it, wraps the result in Markdown frontmatter, and persists a **document**.
|
||||||
|
|
||||||
|
### Key components
|
||||||
|
- **`KnowledgeManager`** (`managers/knowledge_manager.py`) — the single source of truth. Owns **both** the SQLite schema (`subjects`, `weeks`, `sources`, `documents`, `jobs`) and the on-disk layout under `data/subjects/<slug>/week-NN/{raw,processed,exports}/`. Filesystem and DB are kept in sync here; processors and endpoints must go through it, never write final Markdown or touch the DB directly. Schema migrations are done idempotently in `_init_db` via `_ensure_columns` (add-column-if-missing) — there is no migration framework.
|
||||||
|
- **`IngestionService`** (`services/ingestion_service.py`) — maps `source_type` → processor and assembles frontmatter metadata.
|
||||||
|
- **Processors** (`services/processors/`) — each returns a `ProcessedContent(title, body, processor)` dataclass (`base.py`). `text`, `docx` (python-docx), `pdf` (PyMuPDF text, or Mistral OCR when `use_ocr=true`), `audio` (Deepgram), `video` (ffmpeg → audio → Deepgram). PDFs accept `page_ranges` like `1-5,8,10-12`.
|
||||||
|
- **Providers** (`services/providers/`) — thin HTTP clients for Mistral OCR and Deepgram, keyed off `MISTRAL_API_KEY` / `DEEPGRAM_API_KEY`.
|
||||||
|
- **`markdown_builder.build_markdown`** — the only place that emits the YAML frontmatter block prepended to every processed document.
|
||||||
|
|
||||||
|
### Important invariants
|
||||||
|
- **One document per source.** `create_document` deletes any prior document/jobs for that source before inserting, then deletes the old Markdown file. Reprocessing replaces, never appends.
|
||||||
|
- **Dedup by hash.** `save_upload` rejects a file whose `sha256` already exists in the same week (`DuplicateSourceError` → 409).
|
||||||
|
- **Deletes cascade manually** (no FK cascade): deleting a subject/week/source/document removes child DB rows *and* unlinks files. Renaming a week (`update_week`) renames the directory and rewrites stored paths in the DB with SQL `replace()`.
|
||||||
|
- **Markdown is sanitized** (`_sanitize_markdown` strips control chars) on every read and write.
|
||||||
|
- **Week exports** (`export_week_markdown`) concatenate documents, stripping each doc's frontmatter and inserting `<!-- document_id: N -->` markers; the `.zip` variant ships individual files.
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
Next.js 14 App Router (`frontend/app/`), Tailwind, `react-markdown` + `remark-gfm`. All backend calls go through the single `api` object in `frontend/lib/api.ts`. `API_URL` resolves to `http://127.0.0.1:8000` in dev (via `NEXT_PUBLIC_API_URL`) or `/api` in prod, where `next.config.mjs` `rewrites()` proxy `/api/*` to `BACKEND_INTERNAL_URL`. Pages mirror the data model: `/`, `/subjects/[subjectId]`, `/subjects/[subjectId]/weeks/[weekNumber]`, `/documents/[documentId]`, `/settings`.
|
||||||
|
|
||||||
|
### Deployment shape
|
||||||
|
Single container runs **both** processes: `start-coolify.sh` launches uvicorn on `127.0.0.1:8000` (internal only) and Next.js standalone on `PORT` (default 3000, public). `/app/data` **must** be a mounted volume or all subjects/documents/SQLite state are lost on redeploy. `ffmpeg` is required at runtime for video.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
- UX, error messages, and generated content are in **Spanish** — preserve this. Source comments/identifiers are mixed Spanish/English; match the surrounding file.
|
||||||
|
- The standalone `estacion documentos.html` is a browser-only prototype kept for reference behavior; the real app is `backend/` + `frontend/`.
|
||||||
@@ -8,7 +8,12 @@ class Settings(BaseSettings):
|
|||||||
app_data_dir: Path = Path("./data")
|
app_data_dir: Path = Path("./data")
|
||||||
database_path: Path = Path("./data/app.db")
|
database_path: Path = Path("./data/app.db")
|
||||||
mistral_api_key: str = ""
|
mistral_api_key: str = ""
|
||||||
mistral_ocr_model: str = "mistral-ocr-latest"
|
mistral_ocr_model: str = "mistral-ocr-4-0"
|
||||||
|
mistral_ocr_batch_threshold: int = 3
|
||||||
|
mistral_ocr_batch_poll_seconds: int = 10
|
||||||
|
mistral_ocr_batch_timeout_seconds: int = 1800
|
||||||
|
mistral_ocr_retry_attempts: int = 3
|
||||||
|
mistral_ocr_retry_backoff_base: float = 2.0
|
||||||
deepgram_api_key: str = ""
|
deepgram_api_key: str = ""
|
||||||
|
|
||||||
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
|
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
|
||||||
|
|||||||
+6
-1
@@ -9,6 +9,7 @@ from fastapi.responses import FileResponse, PlainTextResponse
|
|||||||
from backend.app.config import settings
|
from backend.app.config import settings
|
||||||
from backend.app.managers.knowledge_manager import DuplicateSourceError, KnowledgeManager
|
from backend.app.managers.knowledge_manager import DuplicateSourceError, KnowledgeManager
|
||||||
from backend.app.models.schemas import MarkdownUpdate, SubjectCreate, SubjectUpdate, WeekCreate, WeekUpdate
|
from backend.app.models.schemas import MarkdownUpdate, SubjectCreate, SubjectUpdate, WeekCreate, WeekUpdate
|
||||||
|
from backend.app.services.errors import ProcessingError, classify_unknown_error
|
||||||
from backend.app.services.ingestion_service import IngestionService
|
from backend.app.services.ingestion_service import IngestionService
|
||||||
|
|
||||||
app = FastAPI(title="Knowledge Station", version="0.1.0")
|
app = FastAPI(title="Knowledge Station", version="0.1.0")
|
||||||
@@ -284,6 +285,10 @@ def process_job(job_id: int, source_id: int, use_ocr: bool, page_ranges: str | N
|
|||||||
try:
|
try:
|
||||||
document = ingestion.process_source(source_id, use_ocr=use_ocr, page_ranges=page_ranges)
|
document = ingestion.process_source(source_id, use_ocr=use_ocr, page_ranges=page_ranges)
|
||||||
manager.update_job(job_id, "completed", "Documento procesado", document_id=document["id"])
|
manager.update_job(job_id, "completed", "Documento procesado", document_id=document["id"])
|
||||||
|
except ProcessingError as exc:
|
||||||
|
manager.update_source_status(source_id, "failed")
|
||||||
|
manager.update_job(job_id, "failed", exc.labeled)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
manager.update_source_status(source_id, "failed")
|
manager.update_source_status(source_id, "failed")
|
||||||
manager.update_job(job_id, "failed", str(exc))
|
code = classify_unknown_error(exc)
|
||||||
|
manager.update_job(job_id, "failed", f"[{code}] {exc}")
|
||||||
|
|||||||
@@ -0,0 +1,85 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
|
||||||
|
class ProcessingError(Exception):
|
||||||
|
"""Error de procesamiento con un codigo corto y rastreable.
|
||||||
|
|
||||||
|
El atributo ``code`` es un identificador estable en MAYUSCULAS que se
|
||||||
|
antepone al mensaje del job (p.ej. ``[AUTH_MISTRAL] API Key invalida``)
|
||||||
|
para que el usuario pueda buscar y diagnosticar rapidamente el tipo de
|
||||||
|
fallo desde la UI.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, code: str, detail: str) -> None:
|
||||||
|
super().__init__(f"[{code}] {detail}")
|
||||||
|
self.code = code
|
||||||
|
self.detail = detail
|
||||||
|
|
||||||
|
@property
|
||||||
|
def labeled(self) -> str:
|
||||||
|
return f"[{self.code}] {self.detail}"
|
||||||
|
|
||||||
|
|
||||||
|
class MistralAuthError(ProcessingError):
|
||||||
|
def __init__(self, detail: str = "API Key de Mistral invalida o no autorizada (401)") -> None:
|
||||||
|
super().__init__("AUTH_MISTRAL", detail)
|
||||||
|
|
||||||
|
|
||||||
|
class MistralRateLimitError(ProcessingError):
|
||||||
|
def __init__(self, detail: str = "Limite de uso de Mistral alcanzado (429)") -> None:
|
||||||
|
super().__init__("RATE_LIMIT", detail)
|
||||||
|
|
||||||
|
|
||||||
|
class MistralServerError(ProcessingError):
|
||||||
|
def __init__(self, status: int, body: str) -> None:
|
||||||
|
super().__init__("MISTRAL_SERVER", f"Mistral HTTP {status}: {body[:200]}")
|
||||||
|
|
||||||
|
|
||||||
|
class MistralBatchError(ProcessingError):
|
||||||
|
def __init__(self, detail: str) -> None:
|
||||||
|
super().__init__("BATCH", detail)
|
||||||
|
|
||||||
|
|
||||||
|
class MistralOCREmptyError(ProcessingError):
|
||||||
|
def __init__(self, detail: str = "OCR no devolvio contenido") -> None:
|
||||||
|
super().__init__("OCR_EMPTY", detail)
|
||||||
|
|
||||||
|
|
||||||
|
class NetworkError(ProcessingError):
|
||||||
|
def __init__(self, detail: str) -> None:
|
||||||
|
super().__init__("NETWORK", detail)
|
||||||
|
|
||||||
|
|
||||||
|
NON_RETRYABLE_CODES = {
|
||||||
|
"AUTH_MISTRAL",
|
||||||
|
"CONFIG",
|
||||||
|
"OCR_EMPTY",
|
||||||
|
"MISTRAL_CLIENT",
|
||||||
|
"BAD_PDF",
|
||||||
|
"FILE_IO",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def classify_requests_error(exc: requests.RequestException) -> ProcessingError:
|
||||||
|
if isinstance(exc, requests.Timeout):
|
||||||
|
return NetworkError(f"timeout de red con Mistral: {exc}")
|
||||||
|
if isinstance(exc, requests.ConnectionError):
|
||||||
|
return NetworkError(f"sin conexion a Mistral: {exc}")
|
||||||
|
return NetworkError(f"error de red con Mistral: {exc}")
|
||||||
|
|
||||||
|
|
||||||
|
def classify_unknown_error(exc: Exception) -> str:
|
||||||
|
"""Etiqueta breve para excepciones no envueltas en ProcessingError."""
|
||||||
|
name = type(exc).__name__.lower()
|
||||||
|
message = str(exc).lower()
|
||||||
|
if "fitz" in name or "pdf" in name or "pdf" in message or "muPDF" in message:
|
||||||
|
return "PDF_PARSE"
|
||||||
|
if isinstance(exc, FileNotFoundError):
|
||||||
|
return "FILE_IO"
|
||||||
|
if isinstance(exc, (PermissionError, OSError)):
|
||||||
|
return "FILE_IO"
|
||||||
|
if isinstance(exc, ValueError):
|
||||||
|
return "VALIDATION"
|
||||||
|
return "UNKNOWN"
|
||||||
@@ -4,8 +4,10 @@ from pathlib import Path
|
|||||||
|
|
||||||
import fitz
|
import fitz
|
||||||
|
|
||||||
|
from backend.app.config import settings
|
||||||
from backend.app.services.markdown_builder import title_from_path
|
from backend.app.services.markdown_builder import title_from_path
|
||||||
from backend.app.services.processors.base import ProcessedContent
|
from backend.app.services.processors.base import ProcessedContent
|
||||||
|
from backend.app.services.errors import ProcessingError
|
||||||
from backend.app.services.providers.mistral_client import MistralClient
|
from backend.app.services.providers.mistral_client import MistralClient
|
||||||
|
|
||||||
|
|
||||||
@@ -30,22 +32,77 @@ class PdfProcessor:
|
|||||||
return ProcessedContent(title=title_from_path(path), body=body, processor="pdf-text")
|
return ProcessedContent(title=title_from_path(path), body=body, processor="pdf-text")
|
||||||
|
|
||||||
def _process_ocr(self, path: Path) -> ProcessedContent:
|
def _process_ocr(self, path: Path) -> ProcessedContent:
|
||||||
doc = fitz.open(path)
|
|
||||||
client = MistralClient()
|
|
||||||
parts: list[str] = []
|
|
||||||
temp_paths: list[Path] = []
|
|
||||||
try:
|
try:
|
||||||
for index in self._selected_pages(len(doc)):
|
doc = fitz.open(path)
|
||||||
|
except Exception as exc:
|
||||||
|
raise ProcessingError("BAD_PDF", f"No se pudo abrir el PDF: {exc}") from exc
|
||||||
|
selected = self._selected_pages(len(doc))
|
||||||
|
client = MistralClient()
|
||||||
|
page_images: dict[int, Path] = {}
|
||||||
|
try:
|
||||||
|
for index in selected:
|
||||||
page = doc[index - 1]
|
page = doc[index - 1]
|
||||||
pix = page.get_pixmap(matrix=fitz.Matrix(2, 2), alpha=False)
|
pix = page.get_pixmap(matrix=fitz.Matrix(2, 2), alpha=False)
|
||||||
image_path = path.with_name(f"{path.stem}-page-{index}.jpg")
|
image_path = path.with_name(f"{path.stem}-page-{index}.jpg")
|
||||||
pix.save(image_path)
|
pix.save(image_path)
|
||||||
temp_paths.append(image_path)
|
page_images[index] = image_path
|
||||||
parts.append(f"## Pagina {index}\n\n{client.ocr_image(image_path)}")
|
|
||||||
|
results = self._ocr_pages(client, page_images)
|
||||||
finally:
|
finally:
|
||||||
for temp_path in temp_paths:
|
for image_path in page_images.values():
|
||||||
temp_path.unlink(missing_ok=True)
|
image_path.unlink(missing_ok=True)
|
||||||
return ProcessedContent(title=title_from_path(path), body="\n\n".join(parts), processor="mistral-ocr")
|
|
||||||
|
parts: list[str] = []
|
||||||
|
for index in selected:
|
||||||
|
markdown = results.get(index, "")
|
||||||
|
parts.append(f"## Pagina {index}\n\n{markdown}")
|
||||||
|
return ProcessedContent(
|
||||||
|
title=title_from_path(path),
|
||||||
|
body="\n\n".join(parts),
|
||||||
|
processor="mistral-ocr",
|
||||||
|
)
|
||||||
|
|
||||||
|
def _ocr_pages(self, client: MistralClient, page_images: dict[int, Path]) -> dict[int, str]:
|
||||||
|
"""OCR de las paginas con lote + fallback sincrono.
|
||||||
|
|
||||||
|
Estrategia:
|
||||||
|
1. Si hay >= ``batch_threshold`` paginas, intenta el job en lote.
|
||||||
|
2. Cualquier pagina no resuelta (o si el lote entero falla) cae a
|
||||||
|
OCR sincrono pagina a pagina con reintentos.
|
||||||
|
3. Las paginas que fallen de forma definitiva quedan como un
|
||||||
|
placeholder breve con el codigo de error rastreable.
|
||||||
|
"""
|
||||||
|
indices = list(page_images.keys())
|
||||||
|
results: dict[int, str] = {}
|
||||||
|
page_errors: dict[int, str] = {}
|
||||||
|
|
||||||
|
if len(indices) >= settings.mistral_ocr_batch_threshold:
|
||||||
|
ordered = sorted(indices)
|
||||||
|
paths = [page_images[i] for i in ordered]
|
||||||
|
try:
|
||||||
|
batch_results = client.ocr_images_batch(paths)
|
||||||
|
for pos, index in enumerate(ordered):
|
||||||
|
if pos in batch_results:
|
||||||
|
results[index] = batch_results[pos]
|
||||||
|
except ProcessingError:
|
||||||
|
# El lote fallo; continua al fallback sincrono para todas
|
||||||
|
# las paginas no resueltas.
|
||||||
|
pass
|
||||||
|
|
||||||
|
for index in indices:
|
||||||
|
if index in results:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
results[index] = client.ocr_image(page_images[index])
|
||||||
|
except ProcessingError as exc:
|
||||||
|
page_errors[index] = exc.labeled
|
||||||
|
except Exception as exc:
|
||||||
|
page_errors[index] = f"[OCR_ERROR] {exc}"
|
||||||
|
|
||||||
|
for index in indices:
|
||||||
|
if index not in results:
|
||||||
|
results[index] = f"> ⚠️ No se pudo procesar esta pagina: {page_errors.get(index, '[UNKNOWN]')}"
|
||||||
|
return results
|
||||||
|
|
||||||
def _selected_pages(self, total_pages: int) -> list[int]:
|
def _selected_pages(self, total_pages: int) -> list[int]:
|
||||||
if total_pages < 1:
|
if total_pages < 1:
|
||||||
|
|||||||
@@ -1,45 +1,287 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import base64
|
import base64
|
||||||
|
import json
|
||||||
|
import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import requests
|
import requests
|
||||||
|
|
||||||
from backend.app.config import settings
|
from backend.app.config import settings
|
||||||
|
from backend.app.services.errors import (
|
||||||
|
MistralAuthError,
|
||||||
|
MistralBatchError,
|
||||||
|
MistralOCREmptyError,
|
||||||
|
MistralRateLimitError,
|
||||||
|
MistralServerError,
|
||||||
|
NetworkError,
|
||||||
|
NON_RETRYABLE_CODES,
|
||||||
|
ProcessingError,
|
||||||
|
classify_requests_error,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class MistralClient:
|
class MistralClient:
|
||||||
endpoint = "https://api.mistral.ai/v1/ocr"
|
"""Cliente Mistral para OCR.
|
||||||
|
|
||||||
|
- ``ocr_image``: OCR sincrono de una pagina, con reintentos y backoff.
|
||||||
|
- ``ocr_images_batch``: OCR en lote via la API de Batch de Mistral
|
||||||
|
(endpoint ``/v1/ocr``), subiendo un fichero JSONL. Devuelve un dict
|
||||||
|
``{page_index: markdown}``. Si el job falla o caduca se lanza
|
||||||
|
``MistralBatchError`` para que el llamador caiga al modo sincrono.
|
||||||
|
"""
|
||||||
|
|
||||||
|
base_url = "https://api.mistral.ai/v1"
|
||||||
|
ocr_url = f"{base_url}/ocr"
|
||||||
|
files_url = f"{base_url}/files"
|
||||||
|
batch_url = f"{base_url}/batch/jobs"
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._api_key = settings.mistral_api_key
|
||||||
|
self._model = settings.mistral_ocr_model
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# API publica
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
def ocr_image(self, image_path: Path) -> str:
|
def ocr_image(self, image_path: Path) -> str:
|
||||||
if not settings.mistral_api_key:
|
self._require_key()
|
||||||
raise RuntimeError("MISTRAL_API_KEY no esta configurada")
|
|
||||||
data_url = self._image_data_url(image_path)
|
data_url = self._image_data_url(image_path)
|
||||||
|
return self._retry(lambda: self._ocr_sync(data_url))
|
||||||
|
|
||||||
|
def ocr_images_batch(self, image_paths: list[Path]) -> dict[int, str]:
|
||||||
|
"""OCR en lote. Devuelve ``{posicion_en_lista: markdown}``.
|
||||||
|
|
||||||
|
``posicion_en_lista`` es el indice (0-based) dentro de ``image_paths``,
|
||||||
|
NO el numero de pagina del PDF; el llamador hace esa traduccion.
|
||||||
|
"""
|
||||||
|
self._require_key()
|
||||||
|
if not image_paths:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
payload = self._build_batch_payload(image_paths)
|
||||||
|
upload = self._retry(lambda: self._upload_batch_file(payload))
|
||||||
|
file_id = upload.get("id")
|
||||||
|
if not file_id:
|
||||||
|
raise MistralBatchError("la subida del fichero batch no devolvio id")
|
||||||
|
|
||||||
|
job = self._retry(lambda: self._create_batch_job(file_id))
|
||||||
|
job_id = job.get("id")
|
||||||
|
if not job_id:
|
||||||
|
raise MistralBatchError("la creacion del job batch no devolvio id")
|
||||||
|
|
||||||
|
final = self._poll_batch_job(job_id)
|
||||||
|
if final.get("status") != "SUCCESS":
|
||||||
|
reason = final.get("errors") or final.get("status") or "sin detalle"
|
||||||
|
raise MistralBatchError(f"job {job_id} finalizo como {final.get('status')}: {reason}")
|
||||||
|
|
||||||
|
output_file_id = final.get("output_file")
|
||||||
|
if not output_file_id:
|
||||||
|
raise MistralBatchError(f"job {job_id} sin output_file")
|
||||||
|
|
||||||
|
content = self._retry(lambda: self._download_file(output_file_id))
|
||||||
|
return self._parse_batch_output(content)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# OCR sincrono
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _ocr_sync(self, data_url: str) -> str:
|
||||||
|
try:
|
||||||
response = requests.post(
|
response = requests.post(
|
||||||
self.endpoint,
|
self.ocr_url,
|
||||||
headers={
|
headers=self._json_headers(),
|
||||||
"Content-Type": "application/json",
|
|
||||||
"Authorization": f"Bearer {settings.mistral_api_key}",
|
|
||||||
},
|
|
||||||
json={
|
json={
|
||||||
"model": settings.mistral_ocr_model,
|
"model": self._model,
|
||||||
"document": {"type": "image_url", "image_url": data_url},
|
"document": {"type": "image_url", "image_url": data_url},
|
||||||
"include_image_base64": False,
|
"include_image_base64": False,
|
||||||
},
|
},
|
||||||
timeout=180,
|
timeout=180,
|
||||||
)
|
)
|
||||||
if response.status_code == 401:
|
except requests.RequestException as exc:
|
||||||
raise RuntimeError("API Key de Mistral invalida")
|
raise classify_requests_error(exc) from exc
|
||||||
if not response.ok:
|
self._raise_for_status(response)
|
||||||
raise RuntimeError(f"Mistral error {response.status_code}: {response.text}")
|
markdown = self._extract_markdown(response.json())
|
||||||
data = response.json()
|
if not markdown:
|
||||||
|
raise MistralOCREmptyError()
|
||||||
|
return markdown
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Batch: construccion, subida, job, polling, descarga, parseo
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _build_batch_payload(self, image_paths: list[Path]) -> bytes:
|
||||||
|
lines: list[str] = []
|
||||||
|
for idx, path in enumerate(image_paths):
|
||||||
|
data_url = self._image_data_url(path)
|
||||||
|
body = {
|
||||||
|
"document": {"type": "image_url", "image_url": data_url},
|
||||||
|
"include_image_base64": False,
|
||||||
|
}
|
||||||
|
lines.append(
|
||||||
|
json.dumps(
|
||||||
|
{"custom_id": str(idx), "body": body},
|
||||||
|
separators=(",", ":"),
|
||||||
|
ensure_ascii=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return ("\n".join(lines)).encode("utf-8")
|
||||||
|
|
||||||
|
def _upload_batch_file(self, payload: bytes) -> dict:
|
||||||
|
try:
|
||||||
|
response = requests.post(
|
||||||
|
self.files_url,
|
||||||
|
headers={"Authorization": f"Bearer {self._api_key}"},
|
||||||
|
data={"purpose": "batch"},
|
||||||
|
files={"file": ("ocr-batch.jsonl", payload, "application/jsonl")},
|
||||||
|
timeout=300,
|
||||||
|
)
|
||||||
|
except requests.RequestException as exc:
|
||||||
|
raise classify_requests_error(exc) from exc
|
||||||
|
self._raise_for_status(response)
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
def _create_batch_job(self, file_id: str) -> dict:
|
||||||
|
try:
|
||||||
|
response = requests.post(
|
||||||
|
self.batch_url,
|
||||||
|
headers=self._json_headers(),
|
||||||
|
json={
|
||||||
|
"input_files": [file_id],
|
||||||
|
"model": self._model,
|
||||||
|
"endpoint": "/v1/ocr",
|
||||||
|
},
|
||||||
|
timeout=120,
|
||||||
|
)
|
||||||
|
except requests.RequestException as exc:
|
||||||
|
raise classify_requests_error(exc) from exc
|
||||||
|
self._raise_for_status(response)
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
def _poll_batch_job(self, job_id: str) -> dict:
|
||||||
|
terminal = {"SUCCESS", "FAILED", "TIMEOUT_EXCEEDED", "CANCELLED"}
|
||||||
|
deadline = time.time() + max(60, settings.mistral_ocr_batch_timeout_seconds)
|
||||||
|
delay = max(2, settings.mistral_ocr_batch_poll_seconds)
|
||||||
|
last: dict = {}
|
||||||
|
while time.time() < deadline:
|
||||||
|
try:
|
||||||
|
response = requests.get(
|
||||||
|
f"{self.batch_url}/{job_id}",
|
||||||
|
headers=self._json_headers(),
|
||||||
|
timeout=60,
|
||||||
|
)
|
||||||
|
except requests.RequestException:
|
||||||
|
time.sleep(delay)
|
||||||
|
continue
|
||||||
|
if response.ok:
|
||||||
|
last = response.json()
|
||||||
|
if last.get("status") in terminal:
|
||||||
|
return last
|
||||||
|
time.sleep(delay)
|
||||||
|
raise MistralBatchError(
|
||||||
|
f"job {job_id} excedio el timeout de polling "
|
||||||
|
f"({settings.mistral_ocr_batch_timeout_seconds}s); ultimo estado: "
|
||||||
|
f"{last.get('status') if last else 'desconocido'}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _download_file(self, file_id: str) -> bytes:
|
||||||
|
try:
|
||||||
|
response = requests.get(
|
||||||
|
f"{self.files_url}/{file_id}/content",
|
||||||
|
headers=self._json_headers(),
|
||||||
|
timeout=300,
|
||||||
|
)
|
||||||
|
except requests.RequestException as exc:
|
||||||
|
raise classify_requests_error(exc) from exc
|
||||||
|
self._raise_for_status(response)
|
||||||
|
return response.content
|
||||||
|
|
||||||
|
def _parse_batch_output(self, content: bytes) -> dict[int, str]:
|
||||||
|
results: dict[int, str] = {}
|
||||||
|
for raw in content.decode("utf-8", errors="replace").splitlines():
|
||||||
|
line = raw.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
obj = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue
|
||||||
|
custom_id = obj.get("custom_id")
|
||||||
|
if custom_id is None:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
idx = int(custom_id)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
response = obj.get("response") or {}
|
||||||
|
status_code = response.get("status_code", 200)
|
||||||
|
body = response.get("body") or {}
|
||||||
|
if status_code >= 400 or obj.get("error"):
|
||||||
|
# Pagina fallida en lote: se omite y el llamador cae a sync.
|
||||||
|
continue
|
||||||
|
markdown = self._extract_markdown(body)
|
||||||
|
if markdown:
|
||||||
|
results[idx] = markdown
|
||||||
|
return results
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Helpers compartidos
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _require_key(self) -> None:
|
||||||
|
if not self._api_key:
|
||||||
|
raise ProcessingError("CONFIG", "MISTRAL_API_KEY no esta configurada")
|
||||||
|
|
||||||
|
def _json_headers(self) -> dict[str, str]:
|
||||||
|
return {
|
||||||
|
"Authorization": f"Bearer {self._api_key}",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"Accept": "application/json",
|
||||||
|
}
|
||||||
|
|
||||||
|
def _raise_for_status(self, response: requests.Response) -> None:
|
||||||
|
if response.ok:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
body = response.text
|
||||||
|
except Exception:
|
||||||
|
body = ""
|
||||||
|
status = response.status_code
|
||||||
|
if status == 401:
|
||||||
|
raise MistralAuthError()
|
||||||
|
if status == 429:
|
||||||
|
raise MistralRateLimitError()
|
||||||
|
if status >= 500:
|
||||||
|
raise MistralServerError(status, body)
|
||||||
|
raise ProcessingError("MISTRAL_CLIENT", f"HTTP {status}: {body[:200]}")
|
||||||
|
|
||||||
|
def _retry(self, fn):
|
||||||
|
attempts = max(1, settings.mistral_ocr_retry_attempts)
|
||||||
|
base_delay = max(0.5, settings.mistral_ocr_retry_backoff_base)
|
||||||
|
last_exc: Exception | None = None
|
||||||
|
for attempt in range(attempts):
|
||||||
|
try:
|
||||||
|
return fn()
|
||||||
|
except ProcessingError as exc:
|
||||||
|
if exc.code in NON_RETRYABLE_CODES:
|
||||||
|
raise
|
||||||
|
last_exc = exc
|
||||||
|
except Exception as exc:
|
||||||
|
last_exc = exc
|
||||||
|
if attempt < attempts - 1:
|
||||||
|
time.sleep(base_delay * (2 ** attempt))
|
||||||
|
if last_exc is not None:
|
||||||
|
raise last_exc
|
||||||
|
raise MistralBatchError("reintentos agotados sin excepcion conocida")
|
||||||
|
|
||||||
|
def _extract_markdown(self, data: dict) -> str:
|
||||||
pages = data.get("pages") or []
|
pages = data.get("pages") or []
|
||||||
markdown_pages = [page.get("markdown", "").strip() for page in pages if page.get("markdown")]
|
chunks = [page.get("markdown", "").strip() for page in pages if page.get("markdown")]
|
||||||
if markdown_pages:
|
if chunks:
|
||||||
return "\n\n".join(markdown_pages)
|
return "\n\n".join(chunks)
|
||||||
if data.get("markdown"):
|
if data.get("markdown"):
|
||||||
return data["markdown"].strip()
|
return str(data["markdown"]).strip()
|
||||||
raise RuntimeError("Mistral OCR no devolvio Markdown")
|
return ""
|
||||||
|
|
||||||
def _image_data_url(self, image_path: Path) -> str:
|
def _image_data_url(self, image_path: Path) -> str:
|
||||||
encoded = base64.b64encode(image_path.read_bytes()).decode("ascii")
|
encoded = base64.b64encode(image_path.read_bytes()).decode("ascii")
|
||||||
|
|||||||
+6
-1
@@ -11,7 +11,12 @@ services:
|
|||||||
APP_DATA_DIR: /app/data
|
APP_DATA_DIR: /app/data
|
||||||
DATABASE_PATH: /app/data/app.db
|
DATABASE_PATH: /app/data/app.db
|
||||||
MISTRAL_API_KEY: ${MISTRAL_API_KEY:-}
|
MISTRAL_API_KEY: ${MISTRAL_API_KEY:-}
|
||||||
MISTRAL_OCR_MODEL: ${MISTRAL_OCR_MODEL:-mistral-ocr-latest}
|
MISTRAL_OCR_MODEL: ${MISTRAL_OCR_MODEL:-mistral-ocr-4-0}
|
||||||
|
MISTRAL_OCR_BATCH_THRESHOLD: ${MISTRAL_OCR_BATCH_THRESHOLD:-3}
|
||||||
|
MISTRAL_OCR_BATCH_POLL_SECONDS: ${MISTRAL_OCR_BATCH_POLL_SECONDS:-10}
|
||||||
|
MISTRAL_OCR_BATCH_TIMEOUT_SECONDS: ${MISTRAL_OCR_BATCH_TIMEOUT_SECONDS:-1800}
|
||||||
|
MISTRAL_OCR_RETRY_ATTEMPTS: ${MISTRAL_OCR_RETRY_ATTEMPTS:-3}
|
||||||
|
MISTRAL_OCR_RETRY_BACKOFF_BASE: ${MISTRAL_OCR_RETRY_BACKOFF_BASE:-2.0}
|
||||||
DEEPGRAM_API_KEY: ${DEEPGRAM_API_KEY:-}
|
DEEPGRAM_API_KEY: ${DEEPGRAM_API_KEY:-}
|
||||||
NODE_ENV: production
|
NODE_ENV: production
|
||||||
PORT: 3000
|
PORT: 3000
|
||||||
|
|||||||
Reference in New Issue
Block a user