Monitoring & Observability
System metrics, model quality signals, and drift — the three layers of watching a production model, and why only some of them catch a silent failure
Monitoring & Observability
TL;DR
Watching a production ML system means three distinct layers — system health, model quality, and data drift — and they catch entirely different failures. A service can look perfectly healthy on system metrics alone while its predictions have quietly become useless, which is exactly why "the service is up" and "the model is still good" have to be monitored separately.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | Model Registry & Versioning |
| You will understand | The three monitoring layers, how to instrument them concretely, and why they need to be watched separately |
The Three Layers
What to watch, and why each layer is necessary
System metrics
Model quality metrics
Data drift signals
Each layer catches failures the others miss. A model can be fast and error-free (system metrics: green) while its predictions have drifted into uselessness (model quality: red) because the world it predicts about changed — this is the central warning of this whole page, worth internalizing before the mechanics below.
Why Percentiles, Not Averages
A good mean latency can hide a bad experience for a meaningful fraction of users. A service with a 120ms mean latency sounds fine — until you learn its p99 is 4 seconds, meaning 1 in 100 requests waits four seconds. Averages get pulled toward the bulk of fast, easy requests; percentiles reveal what the slowest real requests actually experience.
| Metric | What it tells you |
|---|---|
| p50 (median) | The typical request |
| p95 | The slower end of normal — a meaningful chunk of real users see this |
| p99 | The tail — often where infra problems (GC pauses, cold caches, contention) actually live |
Instrumenting It: Prometheus + Grafana
from prometheus_client import Histogram, Counter, make_asgi_app
from fastapi import FastAPI
import time
app = FastAPI()
app.mount("/metrics", make_asgi_app())
REQUEST_LATENCY = Histogram("predict_latency_seconds", "Prediction latency")
PREDICTIONS = Counter("predictions_total", "Predictions served", ["predicted_class"])
@app.post("/predict")
def predict(payload: PredictRequest):
start = time.perf_counter()
result = model.predict(payload.features)
REQUEST_LATENCY.observe(time.perf_counter() - start)
PREDICTIONS.labels(predicted_class=result.label).inc()
return {"prediction": result.label}A Histogram gives Prometheus enough data to compute latency percentiles later, not just an average — that's the point of using one instead of a plain gauge. The Counter labeled by predicted class is a cheap, early model-quality signal: a sudden shift in the distribution of predicted classes (everything suddenly classified as one category) is often visible here before anyone notices a business impact.
A useful Grafana dashboard for this service puts, at minimum: a latency percentile panel (p50/p95/p99 as separate lines, not just one average), a request-rate and error-rate panel, and a prediction-distribution panel — so a shift in what the model is outputting is visible at a glance, not buried in raw logs.
Structured Logging and Tracing
Plain text logs vs structured logs
Plain text: 'Prediction made for user 4821, took 340ms, class=refund'
Readable by a human scanning one line at a time. Nearly impossible to reliably query, aggregate, or alert on at scale — "count how many predictions were class=refund and took over 300ms today" means parsing free text.
Structured: {'user_id': 4821, 'latency_ms': 340, 'class': 'refund'}
RecommendedThe same information, but every field is queryable directly. Aggregation, alerting, and dashboards become straightforward because the log itself is already structured data.
Distributed tracing extends this across services: one request that passes through a gateway, a cache lookup, and a model server gets one trace ID carried through all three, so when latency is high you can see exactly which hop it accumulated in — instead of guessing between "the gateway is slow" and "the model server is slow" from three separate, unlinked log streams.
The Failure Only Model-Quality Monitoring Catches
Concrete scenario: A content-moderation model returns 200 OK on every request, with stable p95 latency and zero errors for weeks — system monitoring shows a perfectly healthy service throughout. Meanwhile, a slow shift in the kind of content being posted has made its predictions steadily less accurate, and nobody notices until a spike in user complaints forces a manual audit. Every system-level signal looked fine the entire time, because uptime and latency have no opinion on whether a prediction is correct.
This is exactly why the third layer — drift, covered in depth next — and model-quality signals like the prediction-distribution panel above have to exist as their own monitored layer, not an assumed byproduct of "the service is up."
Concept Checks
Check yourself
A service has a 120ms mean latency and a 4-second p99. Why is the mean alone misleading here?
Because the mean is pulled toward the large number of fast, typical requests and can look fine even when a real fraction of requests — here, 1 in 100 — experience a dramatically worse latency. Percentiles like p99 reveal what the slowest real requests actually experience, which the mean hides by averaging it away.
Why use a Prometheus Histogram for latency instead of just logging the average latency periodically?
A Histogram records the distribution of observed values, which lets Prometheus compute percentiles (p50/p95/p99) after the fact — an average alone throws away exactly the tail information percentiles need. Logging only a periodic average would make it impossible to later ask "what did the slowest 1% of requests experience."
A service has zero errors, stable latency, and full uptime for weeks, yet its predictions have become substantially less accurate. Why would system monitoring alone miss this entirely?
Because uptime, latency, and error rate measure whether the service is running and responding — they have no signal about whether the predictions themselves are correct. Model quality requires its own monitoring layer (prediction distributions, agreement with ground truth, drift signals), since a system can be perfectly "healthy" by every infrastructure metric while quietly producing wrong answers.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Three monitoring layers | System metrics, model quality metrics, data drift — each catches different failures |
| Percentiles over averages | p95/p99 reveal tail experience an average hides |
| Histogram vs Counter | Histograms enable percentile computation; counters give cheap distribution signals |
| Structured logging | JSON fields are queryable and alertable; plain text mostly isn't |
| Distributed tracing | Follows one request across services to find where latency actually accumulates |
| The core failure mode | A service can be "up" by every system metric while its predictions are silently wrong |
Next
System and quality monitoring both point at the same underlying risk — the next page covers it directly: Data & Model Drift.
Model Registry & Versioning
What a model registry actually stores, how models move through stages, and why "which model is in production" must be a queryable fact
Data & Model Drift
Data drift vs concept drift, how each is actually detected, and designing a retraining trigger that responds to the right signal