Failure Modes & Debugging
The MLOps-stack bugs that show up again and again — shallow health checks, silent drift, skew, loose gates — and the fix for each
Failure Modes & Debugging
TL;DR
Most confusing MLOps incidents trace back to a missing check, not a mysterious failure — a health check that doesn't exercise the real path, a validation gate with too loose a threshold, a metric nobody was monitoring. This page is the symptom-to-cause table: find the symptom, jump to the mechanism and the fix.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~19 minutes |
| Prerequisites | Security & Governance |
| You will understand | The specific mechanism behind each common MLOps failure, and which layer it actually points to |
Triage First: Which Layer Does This Point To?
Before debugging a specific symptom, ask which layer it belongs to. A production problem can originate in serving, data, model quality, or infrastructure — and the fix for each is completely different. Diagnosing the wrong layer wastes time and sometimes makes the real problem worse.
Is the API returning errors, or wrong-but-200 responses?
│
├── Errors → serving/infrastructure layer
│
└── 200 OK, but wrong → is it wrong for everyone, or drifting slowly?
│
├── Wrong for everyone, suddenly → likely a bad deploy or a skew-introducing change
│
└── Drifting slowly → data or concept drift, or a slowly-changing skewThe Table
| Symptom | Usual cause |
|---|---|
| Service returns 200 but predictions are garbage | Model failed to load correctly, or a dependency (cache, vector store, feature store) silently degraded — a shallow health check missed it |
| Accuracy drops slowly over weeks | Data or concept drift — check the live input distribution against training data, per Data & Model Drift |
| Costs spike unexpectedly | Cache hit rate dropped, or a routing rule sends more traffic to an expensive model than intended — check gateway observability logs first |
| "Works in staging, fails in production" | Training-serving skew, or an environment/config difference the CI pipeline didn't catch |
| A new model deploy regresses quality | Missing or too-loose a validation gate — the pipeline let a worse model through, per CI/CD for ML |
| Autoscaling doesn't respond to real load | Scaling on the wrong metric (CPU) for a workload that's actually GPU- or queue-bound, per Orchestration & Scaling |
| Rollback takes too long during an incident | Nobody could confirm which model version was actually live — a registry gap, per Model Registry & Versioning |
| An A/B test "winner" doesn't hold up after full rollout | The observed difference was noise from too small a sample, not a real effect — see A/B Testing & Experimentation |
Deeper on the Trickiest Two
The health check that lied
Mechanism: A health check that only confirms the process is alive says nothing about whether the model actually loaded, whether a GPU has memory available, or whether a dependency (vector store, feature store, cache) is reachable. The service reports healthy right up until the first real request, which fails or returns garbage.
# Lies: only confirms the process can respond at all
@app.get("/health")
def health():
return {"status": "ok"}
# Tells the truth: exercises the actual dependencies
@app.get("/health")
def health():
if not model_loaded:
raise HTTPException(503, "model not loaded")
if not feature_store.ping():
raise HTTPException(503, "feature store unreachable")
return {"status": "ok"}Fix: A real health check confirms the model and every hard dependency are actually ready, not just that the web server started.
The rollback nobody could execute confidently
Mechanism: Without a registry recording exactly which model version is currently serving, an incident response starts with "which model do we even roll back to?" — a question that should have a one-command answer and instead becomes archaeology through deploy logs and Slack history, while the incident continues.
Fix: The registry (see Model Registry & Versioning) exists specifically to make "what's in Production right now, and what was it before" an instant, reliable query — treat a registry gap as a rollback-time-to-recovery risk, not just a bookkeeping nicety.
Concept Checks
Check yourself
A service returns 200 OK on every request but the predictions are consistently garbage. What's the first thing to check?
Whether the health check actually exercises the model and its dependencies, or only confirms the process is running. A shallow health check reports healthy right up until a real request hits a model that failed to load or a dependency that's unreachable — the fix is checking those specific things, not assuming the model itself is broken.
Why does 'works in staging, fails in production' point toward training-serving skew or a config difference rather than a code bug the CI pipeline should have caught?
Because CI pipelines test code correctness, not whether the data a model sees in each environment is actually identical — a feature computed slightly differently between staging and production data pipelines, or an environment variable that differs, produces exactly this symptom without any code defect for tests to catch. The fix is checking the data path and environment config, not re-reviewing the application code.
An A/B test declares a winning variant, but after full rollout the expected improvement doesn't materialize. What's the likely explanation?
The observed difference during the test was statistical noise from too small a sample, not a real effect — a result that looked significant by chance rather than because the variant was actually better. This is why A/B testing needs a properly sized sample and a real significance check, not just picking whichever arm had a higher number at the end of the test window.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Triage by layer first | Serving/infrastructure vs data vs model quality — each has a different fix |
| Shallow health checks lie | Confirm the model and dependencies are ready, not just that the process runs |
| Cost spikes | Check cache hit rate and routing rules before assuming traffic grew |
| Staging/production mismatch | Usually a data or config difference, not a code bug |
| Missing validation gate | The most common cause of a deploy that shouldn't have shipped |
| Registry gap | Turns a rollback into archaeology during an incident |
| A/B "winner" that doesn't hold | Often a too-small sample producing a noisy result, not a real effect |
Next
With the failure modes catalogued, the next page walks through one complete, worked production design that gets these decisions right from the start: Designing an MLOps System.