Skip to content

development.md

Latest commit

 

History

History
138 lines (109 loc) · 5.99 KB

development.md

File metadata and controls

138 lines (109 loc) · 5.99 KB
Raw

Development guide

Architecture

Two processes, one contract.

The FastAPI backend (server/) serves the API, the built UI, and the per-run output files. It never runs OCR itself: server/runs.py launches one subprocess per run and tracks it through output/<run_id>/meta.json and log.txt.

Each OCR engine is a command the backend can spawn. marker is called directly (marker_single); every other engine is a small driver script in scripts/engines/ that wraps a tool or API. All of them write marker's output contract, defined in scripts/engines/common.py:

output/<run_id>/<pdf stem>/<pdf stem>.md

one markdown file with {N} + 48-dash separators before each page, plus any extracted images next to it. Because every engine writes this, the Viewer, Review, Benchmark, Models, and PDF export don't know or care which engine produced a run.

The Svelte 5 + Vite frontend (ui/) is a plain SPA (no SvelteKit). It talks to the API with same-origin relative paths (ui/src/lib/api.ts), so it works both served by FastAPI in production and behind the Vite dev proxy.

Module map, server side:

  • server/main.py — all HTTP routes, static mounts (/output, /docs, UI).
  • server/runs.py — run lifecycle: spawn, track, parse paginated markdown.
  • server/engines.py — engine registry: commands and availability probes.
  • server/cleanup.py — the Review workflow's state machine (resumable, atomic writes, one JSON file per page).
  • server/corrections.py — pure helpers: deterministic cleanup, anomaly detection, proposal validation, and the Ollama client.
  • server/benchmark.py — normalization and CER/WER/diff metrics.
  • server/export.py — markdown → A4 PDF via WeasyPrint.
  • server/config.py — env-driven settings (see the table below).
  • server/ollama_service.py — marker plugin for the LLM engine; imported by the marker subprocess, not by the server (so it needs the full install).

Setup for development

uv sync                    # light install: server + tests, no torch (~1 min)
uv sync --extra ocr        # full install: adds marker-pdf and docling (several GB)
cd ui && npm ci

The light install runs the server and the whole test suite; OCR runs themselves need the full install. Dev servers with hot reload (uvicorn :8000 + Vite :5173):

scripts/dev.sh      # macOS/Linux
scripts/dev.ps1     # Windows

Tests

uv run pytest              # Python: server, engines, cleanup, export, docs consistency
cd ui && npm run check     # svelte-check + tsc

The suite mocks Ollama and OCR subprocesses, so it needs no models, no GPU, and passes on the light install. tests/test_docs.py fails if the docs mention a file that doesn't exist or drift out of sync with the actual API routes — if you add or change an endpoint, update the list below in the same commit.

Configuration

All optional, read from the environment or .env (see .env.example):

Variable Default Purpose
OLLAMA_HOST http://localhost:11434 where Ollama listens; bare host:port accepted
OCR_CLEANUP_MODEL qwen3.5:27b vision model for LLM corrections and cleanup
OCR_DEEPSEEK_MODEL deepseek-ocr model for the deepseek-ocr engine
TORCH_DEVICE auto (cuda > mps > cpu) force marker's torch device
MISTRAL_API_KEY unset enables the Mistral cloud engine
WEASYPRINT_DLL_DIRECTORIES auto (C:\msys64\mingw64\bin) Windows: where Pango's DLLs live

API

Interactive docs (Swagger UI) at /api/docs on a running server.

  • GET /api/health — feature availability: PDF export, Ollama.
  • GET /api/pdfs — PDFs in input/ with page counts.
  • GET /api/pdfs/{name}/pages/{page}.png — a page of the original, rendered.
  • GET /api/engines — engines with availability.
  • POST /api/runs — start a run (pdf, engine, force_ocr, page_range).
  • GET /api/runs — all runs, newest first.
  • GET /api/runs/{run_id} — one run's status plus a log tail.
  • GET /api/runs/{run_id}/result — parsed pages, images, full markdown.
  • GET /api/runs/{run_id}/pdf — typeset PDF (?variant=raw for untouched text).
  • POST /api/runs/{run_id}/cleanup — start or resume the Review pass.
  • GET /api/runs/{run_id}/cleanup — progress and page summaries.
  • GET /api/runs/{run_id}/cleanup/pages/{page} — page text, diff, diagnostics.
  • PATCH /api/runs/{run_id}/cleanup/pages/{page} — accept, reject, or edit.
  • POST /api/runs/{run_id}/cleanup/pages/{page}/retry — new proposal.
  • GET /api/references — files in reference/.
  • POST /api/benchmark — CER/WER of a run vs a reference or another run.
  • POST /api/compare — one page across several runs, diffed against the first.

Adding an OCR engine

  1. Write a driver in scripts/engines/ that takes pdf out_dir [--page_range], OCRs the pages however it likes, and hands [(page_number, markdown), ...] to write_output() from scripts/engines/common.py. Look at scripts/engines/tesseract_ocr.py for the minimal version.
  2. Register it in ENGINES in server/engines.py with a build_cmd and an available() probe that answers fast without side effects.
  3. That's it — the UI picks it up from GET /api/engines.

The in-app help

The Help tab renders docs/user-guide.md, which FastAPI serves at /docs (the same file GitHub renders). Keep image paths in that file relative (docs/images/viewer.png is written as images/viewer.png there) so both renderers resolve them; ui/src/lib/Help.svelte rewrites them to /docs/images/... at display time.

CI

.github/workflows/ci.yml, three jobs on push and PR:

  • test — pytest on Ubuntu, macOS, and Windows, on the light install (marker/docling are never installed in CI; server/ollama_service.py is the one module that escapes CI cover because of that). Pango is installed on all three so the WeasyPrint probe stays true.
  • ui — npm run check and the production build, same three platforms.
  • docs — link check plus the docs-consistency tests.