Last week I finished an MLOps risk platform: a FastAPI service serving calibrated transaction-risk scores, trained on a 1M-row Kaggle dataset, with drift reporting and a release pipeline that publishes security-scanned Docker images. The full build story lives in the repo’s walkthrough doc.
The interesting part isn’t the green checkmarks. It’s the red ones. Here are five things that broke, in the order they broke, and what each taught me.
1. My linter wasn’t my CI’s linter
Local ruff check passed. CI failed with 15 import-sorting errors I’d never seen. Reason: pyproject.toml said ruff>=0.6, so CI resolved 0.16 while I ran 0.15 — and 0.16 enforces import organization that 0.15 didn’t.
Lesson: a lint version range is an unpinned dependency pretending to be a tool. ruff==0.16.5 now, same as any runtime pin. Anything with >= in your dev dependencies is a future CI failure with a date on it.
2. Tests ran before the model existed
The CI workflow ran pytest, then trained the model. On my machine the model file existed from earlier work, so everything passed locally and failed remotely with model artifact is not available.
Lesson: step order is part of correctness. Local state hides ordering bugs that clean runners expose instantly. If a test needs an artifact, the pipeline must create it first — every time, from nothing.
3. scikit-learn 1.9 refused to unpickle my 1.8 artifact
The API worked locally (sklearn 1.8.0) and died in CI with ModuleNotFoundError: No module named '_loss'. The model file was fine; the runtime had moved.
Lesson: a serialized model is a version contract. scikit-learn==1.8.0 is now pinned in pyproject.toml and the Dockerfile, and the reason lives in a comment. If you commit a model artifact, you’re committing to the library version that created it.
4. The image I shipped wasn’t the image I tested
This one stung. CI was green: it trained the v1 model, ran tests, built the container, smoke-tested it. Then I tagged a release, pulled the published image from GHCR anonymously, and /healthz said degraded, model_loaded: false.
Why: the v1 model is gitignored (it’s a trained artifact), CI trains it before its own docker build — but the release workflow built from a clean checkout where that file doesn’t exist. Every published image was missing the v1 model. v2 worked because its ~200KB calibrated artifact is committed deliberately.
The fix trains the deterministic v1 model inside the Dockerfile, so any clean checkout produces a self-sufficient image. Then I shipped v0.1.1 and verified by pulling the released image and hitting both endpoints.
Lesson: CI tests one path; releases take another. The artifact that ships must be verified the way a stranger receives it — pull the published image, hit the endpoints, check the health endpoint. “It worked in CI” is not a deployment claim.
5. A pinned action that didn’t exist, then a broken installer
The release workflow pinned aquasecurity/trivy-action@0.24.0. That version doesn’t exist — the real tags are v0.24.0 and up. Then, after fixing the pin, the action’s trivy-installer step failed on GitHub runners twice in a row trying to download the binary.
Lesson: pin to versions you’ve verified exist, and when an action’s setup step is the flaky part, take a different path entirely. The release now scans with a pinned aquasec/trivy container image mounting the Docker socket — one less moving part, deterministic by construction.
What the whole thing is actually about
The models are deliberately honest: at the default operating point the system catches ~41% of fraud at ~6.5 alerts per confirmed catch, and the high-recall alternative (81% recall, ~22 alerts per catch) ships in the same report. The dataset is synthetic. The value isn’t the model — it’s the machinery around it: a contract that survived a dataset swap, calibration with the economics written down, provenance hashes, and a pipeline that rebuilds everything from a clean checkout on every push.
The failures above are the portfolio. Any tutorial can show you the happy path; the delta between “works on my machine” and “verifiably shipped” is exactly the job.
If you want to read the code: github.com/riogesulgon/fraud-detect — MIT licensed, with CI badges on the README and both v0.1.0 and v0.1.1 on GHCR.