Designing an MLOps System
A complete worked example — a content-moderation pipeline for a mid-size e-commerce company, sized and justified end to end
Designing an MLOps System
TL;DR
This page takes one realistic brief — a content-moderation model for user-submitted product listings — and makes every decision with a reason attached: the serving pattern, the scaling policy, the CI/CD gate, the registry flow, and what's actually monitored. It's a single coherent system, not a menu of options.
| Property | Value |
|---|---|
| Level | Advanced — brings together every previous page |
| Reading time | ~28 minutes |
| Prerequisites | Failure Modes & Debugging |
| You will understand | How to go from a vague production brief to a sized, justified MLOps system |
Part 1 — The Brief
We're a mid-size e-commerce marketplace. Sellers submit product listings — title, description, images — and we need to screen them for policy violations (counterfeit claims, prohibited items, scam patterns) before they go live. We get roughly 40,000 new listings a day, fairly evenly spread across the day. A missed violation can mean a scam listing reaching buyers before a human catches it; a false positive delays a legitimate seller's listing and costs us support tickets.
| Requirement | Value |
|---|---|
| Volume | ~40,000 listings/day ≈ 0.46/second average, bursty around peak selling hours |
| Latency target | A few seconds is fine — nothing blocks a live user-facing action |
| Cost of a false negative | High — real harm can reach buyers |
| Cost of a false positive | Moderate — seller friction, support load |
| Action on a flagged listing | Hold for human review, never auto-reject or auto-publish |
| Team | A dedicated ML platform team, existing Kubernetes infrastructure |
This is a real-scale, real-stakes problem, and the design should look like one — proper orchestration, a real CI/CD gate, and governance around promotion, unlike a hobby-scale project that could get away with a single container and a cron job.
Part 2 — Serving Pattern
Choosing how requests are served
Synchronous REST, per-listing
Simple, but 0.46 requests/second average with real bursts around peak hours means either over-provisioning for the burst or accepting slower response during it.
Request batching behind a queue
RecommendedNew listings are queued and processed in small batches (e.g. every 2-3 seconds or once 20 accumulate) — a few seconds of latency is acceptable per the brief, and batching gets meaningfully better throughput-per-GPU than one-at-a-time inference, which matters at this volume.
This follows the decision framework from Model Serving Patterns: the brief's explicit latency tolerance is what unlocks batching as the better choice here.
Part 3 — Containerization and Image Strategy
A multi-stage build (per Containerization & Packaging) keeps the shipped image lean, with a separate GPU-enabled base image for the inference workers versus a lightweight CPU image for the queue-consumer/orchestration layer that doesn't need GPU access — no reason to ship CUDA libraries into a component that never touches the model.
Part 4 — Orchestration and Scaling
Scaling policy by component
GPU inference workers
Queue-consumer / API layer
Per the Orchestration & Scaling warning about GPU cold-start cost: the inference workers keep a warm minimum rather than scaling to zero between bursts, because reloading model weights on every scale-up would add real latency exactly when a burst is already pushing the queue.
Part 5 — CI/CD and the Validation Gate
A new model's path to production
The validation gate optimizes for recall on the harmful class specifically, not overall accuracy. A missed violation is far costlier than a false positive here, so the gate has to reflect that asymmetry directly — an overall-accuracy gate would happily pass a model that traded harmful-class recall for looking better in aggregate, per the reasoning in CI/CD for ML.
Part 6 — Registry and Promotion
content-moderation-model
v34 → Staging (passed automated gate, awaiting governance sign-off)
v33 → Production (currently serving, canary-verified)
v32 → Archived (previous production version, retained for rollback)Promotion from Staging to Production requires both the automated gate and the governance sign-off from Security & Governance — this is a case where "passed CI" is necessary but explicitly not sufficient, given the brief's real-harm stakes.
Part 7 — Monitoring Stack
What's actually monitored and alerted on
System metrics
Standard: request latency, error rate, queue depth, GPU utilization — catches infrastructure problems.
Model quality metrics
RecommendedHarmful-class recall on a continuously-sampled, human-reviewed subset of flagged and unflagged listings — this is the metric that actually matters for the brief, and it needs a live feedback signal, not just the training-time number.
Data drift
Listing text/image distribution tracked against the training distribution — new product categories or a shift in how bad actors phrase listings would show up here before it shows up as a recall drop.
Per Monitoring & Observability: "the service is up" and "the model is still catching violations" are different claims, and this brief's stakes make the second one the one that actually matters operationally.
Part 8 — Caching
Caching is a poor fit here — each listing is unique content, so there's essentially no repeat-request structure for an exact-match or semantic cache to exploit. This is a deliberate non-decision, included to show that Caching & Cost Optimization isn't a mandatory layer for every system — it earns its place only where request patterns actually repeat.
The Resulting Design
Content-moderation pipeline, end to end
Ingest
Serve
Decide
Ship changes
Monitor
Concept Checks
Check yourself
Why does this design choose request batching over synchronous per-listing serving?
Because the brief explicitly tolerates a few seconds of latency and the traffic is bursty around peak hours — batching trades a small, acceptable latency cost for meaningfully better GPU throughput at this volume. A latency-sensitive brief would have made the same trade-off wrong; the serving pattern follows directly from what the brief actually requires.
Why does the validation gate check harmful-class recall specifically, instead of overall model accuracy?
Because the brief states a missed violation is far more costly than a false positive, and overall accuracy treats both error types as equally bad — a gate built on overall accuracy could pass a model that quietly traded away harmful-class recall as long as its other predictions improved enough to compensate. The gate has to measure the thing that actually matters for the stated cost structure, not a generic proxy for it.
Why does this design deliberately decide against adding a caching layer?
Because each listing is essentially unique content, so there's no meaningful repeat-request pattern for exact-match or semantic caching to exploit — caching only pays off where requests genuinely repeat. Adding a cache layer here would be complexity with no measurable benefit, which is why the design explicitly calls it out as a considered non-decision rather than an oversight.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Serving pattern follows the latency tolerance | Batching earns its place once a brief explicitly allows a few seconds |
| GPU workers scale conservatively | A warm minimum avoids cold-start cost during bursts |
| The gate measures what actually matters | Harmful-class recall, not generic accuracy, given the stated cost asymmetry |
| Governance beyond the automated gate | Required sign-off for a real-harm-stakes model change |
| Monitor the metric that reflects real stakes | Live, human-reviewed recall — not just system uptime |
| Not every layer is mandatory | Caching was considered and deliberately skipped — it has to earn its place |
Next
That completes the worked example. Glossary closes out the track as a reference for every term used along the way.
Failure Modes & Debugging
The MLOps-stack bugs that show up again and again — shallow health checks, silent drift, skew, loose gates — and the fix for each
Glossary
Every MLOps term and abbreviation used across the fundamentals track, grouped by theme — from health checks and HPA to drift, gateways, and governance