The Anatomy of an MLOps Risk Platform: Architecture and Flow

Two posts ago I wrote about what broke while building this platform. This one is the positive inverse: how the whole thing is designed, how data and requests actually flow through it, and where each piece of standard MLOps practice sits in the architecture.

The system is fraud-detect, a FastAPI service serving calibrated transaction-risk scores trained on a 1M-row Kaggle dataset. Small enough to understand in an afternoon, complete enough to exercise every layer of a real MLOps loop.

The 10,000-foot view

Four layers: data flows into training, training produces artifacts, serving consumes artifacts, and monitoring watches both data and service. Nothing bypasses a layer; every arrow is a file, an HTTP call, or a git push.

flowchart LR
    subgraph DATA["Data layer"]
        RAW["Kaggle dataset<br/>1M rows, 27 features<br/>gitignored, SHA-256 tracked"]
    end
    subgraph TRAIN["Training layer"]
        CAL["scripts/calibrate.py<br/>split, train, calibrate,<br/>choose threshold"]
        BENCH["scripts/benchmark_full.py<br/>model comparison"]
    end
    subgraph SERVE["Serving layer"]
        ART["models/<br/>full-feature-calibrated.joblib<br/>committed, ~200KB"]
        API["FastAPI app<br/>v1 + v2 endpoints"]
    end
    subgraph WATCH["Monitoring layer"]
        DRIFT["Evidently drift report"]
        METRICS["/metrics<br/>Prometheus counters"]
        LOGS["Structured JSON logs"]
    end
    RAW --> CAL
    RAW --> DRIFT
    RAW --> BENCH
    CAL --> ART
    CAL --> REP["reports/calibration.json"]
    ART --> API
    REP --> API
    API --> METRICS
    API --> LOGS
    DRIFT -.-> RETRAIN["retrain decision"]
    RETRAIN -.-> CAL

Two details worth pausing on. First, the raw data never enters git: it is downloaded by script and fingerprinted by SHA-256, and that fingerprint travels with every model into /v1/model-info. Second, the dotted loop at the bottom is the only path from monitoring back to training: drift reports inform a retrain decision, but nothing retrains automatically. In a demonstrator, a human in that loop is a feature, not a gap.

How a request flows

Every scoring request crosses five boundaries: transport, validation, contract enforcement, model inference, and decision logic. Here is the life of a POST /v2/risk-score call:

sequenceDiagram
    participant C as Client
    participant RL as Rate limiter
    participant API as FastAPI handler
    participant V as Pydantic + Pandera
    participant M as Calibrated model
    C->>RL: POST /v2/risk-score (27 fields)
    RL->>RL: per-client window check
    RL->>API: forward (or 429 + Retry-After)
    API->>V: validate types, ranges, contract
    V-->>API: 422 if any field invalid
    API->>API: DataFrame assembly,<br/>one-hot reindex to persisted columns
    API->>M: predict_proba (base model)
    M-->>API: raw probability
    API->>M: Platt sigmoid calibrator
    M-->>API: calibrated probability
    API->>API: threshold + band decision
    API-->>C: risk_score, risk_band, version, threshold

The steps worth explaining:

Validation happens twice, on purpose. Pydantic rejects wrong types and out-of-range values at the edge (a negative amount never reaches the model). Pandera then enforces the shared feature contract that training and inference both import, so the two sides cannot silently drift apart. This is the contract-first idea: src/contracts.py is the single source of truth for what a feature means.

Feature alignment is mechanical, not hopeful. The request is one-hot encoded and then reindexed against feature_columns, persisted inside the artifact at training time. If the training data had a merchant_category = "luxury" that the request lacks, the column is filled with zero rather than crashing. If a future request has a category training never saw, it is dropped rather than inventing a column. Shape is guaranteed; that is what makes serving boring, and boring serving is good serving.

Calibration is a pipeline stage, not a model property. The artifact contains three things: the base gradient-boosted model, a Platt sigmoid calibrator fitted on a held-out slice, and the decision threshold. The API composes them at inference time. This is why switching operating points (high-recall vs cost-based) is a config value rather than a retrain: the probability itself is untouched.

The response is honest. Every response carries model_version and the exact decision_threshold used. A client can always answer “which model said this, and what rule turned the score into a band?” That traceability is what makes a review-queue system auditable.

The training pipeline

Training is a script, not a notebook, and its steps are ordered so that no decision leaks information it should not have:

flowchart TD
    LOAD["Load CSV, sort by timestamp"] --> HASH["Record dataset SHA-256"]
    HASH --> SAMPLE["Downsample deterministically"]
    SAMPLE --> SPLIT["Chronological 60/20/20<br/>train / calibration / test"]
    SPLIT --> FIT["HistGradientBoosting<br/>+ class weights on train only"]
    FIT --> CALFIT["Fit Platt sigmoid<br/>on calibration slice only"]
    CALFIT --> THR["Choose threshold<br/>on validation metrics only"]
    THR --> EVAL["Evaluate both candidates<br/>on untouched test set"]
    EVAL --> SAVE["Save artifact:<br/>base + calibrator + threshold<br/>+ feature columns"]
    SAVE --> REPORT["Write calibration.json:<br/>metrics, costs, provenance"]

Three decisions carry most of the weight:

Chronological splitting. Transaction data has time structure; a random split would let the model learn from next month to predict last month. Every split in this repo is chronological.

Calibration data is distinct from test data. The calibrator sees the 20% slice after train; the test set sees nothing until the very end. That is why the reported Brier and ECE numbers can be believed.

The threshold is chosen against an explicit cost model. Missing a fraud is weighted 20x a wasted review, on validation data. Both the cost-based point (recall 41%, precision 15%) and the high-recall alternative (recall 81%, precision 4.6%) are evaluated on test and written to the report side by side. The economics are data in the repo, not folklore in someone’s head.

Where the MLOps practice lives

This is the outline I wish I had before building: the standard MLOps capabilities, and the concrete artifact or code path that implements each one here.

flowchart LR
    subgraph LOOP["The MLOps loop"]
        D["Data<br/>versioned, fingerprinted"] --> T["Train<br/>deterministic, split-honest"]
        T --> E["Evaluate<br/>calibration + cost metrics"]
        E --> PKG["Package<br/>pinned artifact + container"]
        PKG --> REL["Release<br/>scanned, verified image"]
        REL --> SRV["Serve<br/>versioned API + provenance"]
        SRV --> MON["Monitor<br/>drift, latency, alert volume"]
        MON --> D
    end

Two entries in that table deserve defense. The model registry is not MLflow, and that is deliberate: for one model with one artifact, a committed file with a JSON metadata sidecar is easier to audit than a tracking server. The design principle is that registry needs are proportional to the number of models, not to ambition. Drift detection is offline, run by script rather than streamed: the platform detects drift the way a bank examines statements, periodically and in batches, which matches a system with no live traffic.

The release pipeline

Two pipelines, one repo. CI proves the code; release proves the artifact.

flowchart LR
    PUSH["git push"] --> CI["CI: lint, train, test,<br/>reports, container smoke test"]
    CI --> MERGE["merge to main"]
    TAG["git tag v0.1.x"] --> REL["Release: docker build"]
    REL --> SCAN["Trivy scan<br/>fails on unfixed CRITICAL"]
    SCAN --> GHCR["Push image to GHCR<br/>tagged + by commit SHA"]
    GHCR --> VERIFY["Anonymous pull +<br/>endpoint verification"]

The verify step is the one that paid for itself. After v0.1.0 shipped, I pulled the published image anonymously and found it degraded: CI had trained the v1 model before its own build, but the release built from a clean checkout where that file never existed. The fix trains the deterministic model inside the Dockerfile. Now every image is self-sufficient by construction, and verifying a release means behaving like a stranger: pull, run, hit the endpoints, read the health.

What is deliberately absent

An architecture is also defined by what it refuses to include, and each omission maps to a scale where it would become necessary:

• No authentication or multi-tenancy: one demo consumer. Production would add OAuth2 plus audit logging.
• No distributed serving or autoscaling: single uvicorn process. The rate limiter is per-process by design and documented as such; horizontal scale would need a shared limiter like Redis.
• No online retraining or champion/challenger: the monitoring-to-training arrow is a human decision. Automation there is valuable at real traffic volume and dangerous in a demo.
• No feature store: features arrive with the request. A real platform with many consumers would centralize feature computation to prevent training/serving skew.

Each of these is one RFC away, and the architecture has a seam where each would bolt on. That is the actual test of a demonstrator: not whether it has every capability, but whether the shape accepts them without rearrangement.

Takeaway

The architecture follows one rule end to end: make every handoff explicit and every decision inspectable. Data hands off to training with a hash. Training hands off to serving with a self-describing artifact. Serving hands off to clients with version and threshold in the response. CI hands off to release with a scan gate. Nothing is implicit, so nothing is unauditable.

The code is at github.com/riogesulgon/fraud-detect (MIT), and the failure log from building it is in the previous post.