From image drop to searchable archive.

Use collection.sh for every manual or automated import. It is the stable boundary that protects file layout and database integrity.

./collection.sh add ~/Downloads/page.jpg \
  --year 2026

./collection.sh add ~/Downloads/scans/ \
  --year 2026 --source manual

./collection.sh add /tmp/page.jpg \
  --year 2026 \
  --source messenger \
  --captured-at '2026-08-29T06:14:00+08:00'

Local webapp

./webapp/run.sh
# http://127.0.0.1:8000

The wrapper sets the collection root and database path, then runs Uvicorn with reload enabled.

1. Validate.

   Pillow opens and verifies each supported image. One bad input does not stop the batch.

2. Resolve capture time.

   Precedence is explicit --captured-at → EXIF DateTime tags → filesystem mtime.

3. Deduplicate.

   A streaming SHA-256 identifies content. Existing live content is skipped even when its filename differs.

4. Publish atomically.

   A hidden partial file is copied into media-YYYY/, then exposed with an atomic rename. Name collisions receive a digest suffix.

5. OCR once.

   PaddleOCR retains non-empty OCR lines in reading order, up to 12 title candidates by default, and stores every non-empty line in full-text OCR.

6. Select the title.

   Deterministic chord-line and section-label filtering removes notation candidates. Remaining candidates are heuristic-ranked so multi-word natural-language titles outrank single-letter OCR fragments, then Ollama chooses exactly one. When no remaining candidate looks like a real title, a vision-capable Ollama model reads the title straight from the image and adds it as a candidate. Disabled or failed requests fall back to the top heuristic candidate without rewriting text.

7. Upsert.

   The canonical metadata and OCR output are committed to SQLite; FTS triggers synchronize the derived text index.

Each layer has one owner. Collection tools own ingestion, the shared database module owns schema and FTS synchronization, indexing owns OCR persistence, the API owns search semantics, and the browser owns presentation.

Canonical schema

Every image produces many OCR lines. The title algorithm selects one line as the display title through a four-stage pipeline: candidate extraction, chord-line filtering, heuristic ranking, and LLM selection with a vision fallback.

Candidate extraction

PaddleOCR produces ordered rec_texts and rec_scores for each prediction pass. Every non-empty recognized line becomes a candidate with its OCR confidence as the ranking score. The top n candidates (default 24) are retained in reading order; all non-empty lines are preserved separately as ocr_text for full-text search.

This separation is deliberate: candidates are ranked and truncated for title quality, while ocr_text captures every recognized line for token retrieval.

The importer always supplements the full-page pass with temporary full-image views that are enlarged 2×, contrast-enhanced, sharpened, and tested at 0°, 90°, and 270°. Preprocessing is mandatory rather than conditional, so a small handwritten title is considered even when the full-page pass already produces plausible candidates. The full image is retained in every production view, so title text cannot be cut off by a crop boundary. The original image is never modified, and temporary preprocessing files are removed automatically.

Chord-line filter

Music chord sheets produce OCR lines like |D|F#m|G|A| and /F#m-CHm|D-A/CH/B/DH-E∥A that are chord progressions, not titles. Sheets also contain standalone labels such as CODA and BRIDGE. Although the LLM prompt explicitly rejects both chord notation and section labels, small models can select them anyway. Deterministic filters remove them before the LLM sees the candidate list.

A line is classified as a chord line when both conditions hold:

1. Separator density. The line contains at least two pipe, slash, or backslash characters. Real titles rarely use these as delimiters; chord charts use them to separate short fragments like |Am7|, G/B.
2. No space-bounded words. The line contains no word of 3+ consecutive non-chord letters (excludes A–G) or 4+ consecutive letters of any kind, where a word is bounded by whitespace, string boundaries, or common punctuation (:;,.!?). OCR artifacts like CHm or olEm that appear inside pipe/slash fragments are not space-bounded, so they do not prevent chord-line classification.

Formally, define the separator count and two regular-expression predicates over a candidate text $t$:

Then:

The non-chord word pattern excludes letters A–G because those are common note names. A 4-letter word of any kind is strong evidence of natural language regardless of note names—INTRO in INTRO:|A|Al prevents that line from being filtered.

Heuristic ranking

Before the LLM sees the candidates, they are sorted by a cheap heuristic so the no-LLM fallback picks a real title instead of high-confidence OCR garbage. A candidate is title-like when it has at least two whitespace-separated tokens that are each purely alphabetic words of 2+ letters, after stripping trailing punctuation. Trailing punctuation is tolerated so real titles such as HARK THE HERALD ANGELS SING; count; leading punctuation is deliberately kept so slash-prefixed chord tokens such as /GUITAR stay non-title-like.

Candidates are ranked: a vision candidate (present only when OCR had no title-like candidate) first, then title-like OCR candidates, then the rest—each group by descending OCR confidence, ties broken by original reading order. The LLM still chooses semantically from the full reordered list; ranking only decides the fallback.

LLM selection and fallback

After chord-line filtering and heuristic ranking, the remaining candidates are sent to the configured Ollama model (default: glm-5.3-flash:cloud). The system prompt instructs the model to prefer actual song, work, or document titles and reject lyrics, section labels, chord notation, page numbers, and OCR garbage. The model selects exactly one candidate_id from the supplied list; it may never rewrite, combine, or invent text.

The selector falls back gracefully:

1. Successful selection.

   The model returns a valid candidate_id. That candidate's text becomes the title and its OCR confidence becomes title_confidence.

2. Invalid or malformed response.

   The selector uses the top heuristic candidate (a vision candidate if present, else the highest-confidence title-like candidate) without opening the circuit breaker. Subsequent requests continue normally.

3. Service failure.

   A connection, HTTP, or timeout error opens a selector-local circuit breaker. All remaining images in the batch fall back to the top heuristic candidate. --verbose logs identify the failure.

4. Disabled.

   --no-ollama skips the request entirely and uses the top heuristic candidate.

Vision fallback

When none of the selectable OCR candidates is title-like—OCR garbled the title into fragments such as RTUTITFION or produced only labels like ACAPELLA /GUITAR ONLY—a vision-capable Ollama model (default: glm-5.3-flash:cloud) reads the title directly from the image and returns it as a zero-confidence candidate. The text selector then picks among the OCR candidates plus the vision candidate, so the selection layer still owns the final choice and the common OCR-success path never spends vision tokens.

The vision candidate is tagged vision: true in candidates_json for auditing. Vision has its own selector-local circuit breaker, independent of the text-LLM circuit. Disable it with --no-vision-fallback (the OCR-only behavior); override the model with --vision-model. A vision call costs roughly 2k image tokens, so on Ollama Cloud it draws down subscription quota and is negligible on any per-token provider.

API

GET api/search?q=amazing&scope=title&limit=20
GET api/search?q=sweet%20sound&scope=text&limit=20
GET api/titles?letter=F

scope defaults to title. Only title and text are accepted; invalid values return 400. Limits are bounded to 1–100.

Alphabetical browse

The /titles page lists every indexed title grouped by first character, paginated by letter. Each entry links straight to its scan image. GET api/titles?letter=F returns the titles for one bucket plus groups (every populated first character with its count) for the navigation. Letters map to A–Z, digits map to their own bucket, and everything else collapses to #. The homepage links to it as “All titles”.

Safe text queries

User input is never exposed as raw SQLite MATCH syntax. Ordinary terms are quoted, made prefix-searchable, and joined with AND:

sweet sou  →  "sweet"* AND "sou"*

This makes punctuation and quotes harmless while requiring every useful term. Excerpts returned by the API are plain text and must still be escaped by browser rendering.

Back up first

sqlite3 titles.sqlite3 \
  ".backup 'titles.sqlite3.$(date +%Y%m%d%H%M%S).bak'"

Reconcile and migrate

./collection.sh sync
./collection.sh sync media-2026

Normal sync OCRs new files, size/mtime changes, and—during the rollout—legacy rows where ocr_text IS NULL. Ollama selects one of each image's ranked candidates during that OCR pass. A collection-specific checkpoint is committed after each image, so rerunning an interrupted command resumes after the last committed path and safely retries an in-flight image. Existing rows require --force to be reselected.

To explicitly resume from a displayed one-based job number, rerun the same collection selection with --resume-from. For example, ./collection.sh sync --resume-from 30 starts with the job logged as [30/349 …].

Install the default models with ollama pull glm-5.3-flash:cloud (used for both text selection and vision). Use --no-ollama when Ollama is intentionally unavailable; --ollama-model, --ollama-url, and --ollama-timeout are available on add, sync, and reindex. The vision fallback shares --ollama-url/--ollama-timeout and adds --vision-model and --no-vision-fallback.

Use --verbose to replace OCR progress output with filename-scoped emoji logs. Top-level image events remain flush-left; candidate, filter, and LLM decision statements are indented beneath the image event. Each trace shows every retained candidate and score, the complete LLM prompt and raw response, fallback decisions, the selected title and method, and the database save.

./collection.sh sync media-2026 --force --verbose

Reindex by title

When a few stored titles are wrong (OCR garbage, single letters, or pre-vision misses), target just those images instead of re-OCRing the whole collection:

./collection.sh reindex "S" --verbose
./collection.sh reindex "RTUTITFION" --verbose
./collection.sh reindex "amazing" --substring --dry-run
./collection.sh reindex "grace" --substring --limit 5

The command searches the index by title and re-OCRs every matching image through the full pipeline (chord filter, heuristic rank, LLM, vision fallback). Matching is exact and case-insensitive by default; pass --substring for a fragment search. --dry-run lists matches without reindexing; --limit N caps how many are reindexed when a broad query matches too many. Multiple matches (duplicates sharing a title, or a substring hitting several pages) are all listed and all reindexed. index_image preserves captured_at and source and recomputes content_sha256, so reindexing only changes OCR-derived fields.

Exit codes: 0 success, 1 if any image errored, 2 for no matches, an empty title, or no matching files present.

Reset the catalog

To rebuild the database from every existing media-* file, stop the webapp and other SQLite writers, then run the reversible reset helper:

./reset_database.sh --yes

The script creates a timestamped SQLite backup, moves the old catalog and any sidecars aside, creates a fresh schema through sync, and preserves all media files. The --yes flag is required. Use ordinary ./collection.sh sync --force instead when you only need to rerun OCR.

Health checks

sqlite3 titles.sqlite3 "PRAGMA quick_check;"
sqlite3 titles.sqlite3 \
  "SELECT count(*) FROM image_titles WHERE ocr_text IS NULL;"

curl -fsS \
  'http://127.0.0.1:8000/api/search?q=forever&scope=title&limit=1'

After migration, test text search with words known to occur outside the selected title and try punctuation or quotes to verify safe query construction.

Nginx terminates HTTPS at /titles/, strips that prefix, and proxies to loopback Uvicorn. TITLE_BROWSER_PUBLIC_PREFIX=/titles keeps generated media URLs correct. This page uses relative links so it works both at /docs locally and /titles/docs in production.

Deploy script behavior

1. Validate the local database and verify the remote tree.
2. Back up the remote webapp, pipeline, and database under a timestamped root backup.
3. Sync webapp, collection pipeline, documentation, and the staged local database.
4. Remove the obsolete classifier, restore ownership, and restart systemd.
5. Verify public title search, full-text search, and documentation.

curl -fsS \
  'http://127.0.0.1:8001/api/search?q=forever&scope=title&limit=1'

curl -fsS \
  'https://rioges.xyz/titles/api/search?q=forever&scope=title&limit=1'