Data & Model Drift
Data drift vs concept drift, how each is actually detected, and designing a retraining trigger that responds to the right signal
Data & Model Drift
TL;DR
Data drift is the live input distribution diverging from what the model was trained on — detectable by watching the inputs alone. Concept drift is subtler and more dangerous: the relationship between inputs and the correct output changes while the inputs themselves look normal, which means catching it requires a ground-truth feedback signal, not just input monitoring.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | Monitoring & Observability |
| You will understand | How to detect both kinds of drift, and how to design a retraining trigger around them |
Data Drift: The Inputs Changed
Data drift means the statistical distribution of live input features has measurably diverged from the distribution the model was trained on. It's detected by directly comparing distributions — no ground truth needed, since you're only asking "do the inputs still look like training data," not "are the predictions still correct."
| Technique | What it measures |
|---|---|
| KS-test (Kolmogorov-Smirnov) | Whether two samples (training feature values vs. a recent window of live values) plausibly come from the same distribution — outputs a p-value; a small p-value flags likely drift |
| Population Stability Index (PSI) | Buckets a feature's values and compares the proportion of data in each bucket between training and live windows — a single interpretable number, with common rule-of-thumb thresholds (e.g. PSI > 0.2 as a warning) |
from scipy.stats import ks_2samp
result = ks_2samp(training_feature_values, live_window_feature_values)
if result.pvalue < 0.05:
alert("feature 'session_length' may be drifting")Neither test tells you whether drift matters for model quality — only whether the input distribution changed. A feature drifting that the model barely relies on is a non-event; a feature drifting that the model weights heavily is worth real attention. Prioritize drift alerts by feature importance, not just statistical significance.
Concept Drift: The Relationship Changed
Concept drift is the harder failure, because it's invisible to input monitoring. The inputs can look statistically identical to training data while the correct output for those inputs has genuinely changed — a word that used to signal a complaint now gets used sarcastically to mean praise, and a model trained before that shift keeps confidently getting it backward, with nothing in the input distribution flagging a problem.
Because concept drift doesn't show up in the inputs, detecting it requires some signal about actual outcomes:
| Feedback source | Example |
|---|---|
| Delayed ground truth | A support ticket's real resolution eventually confirms whether the model's triage was right |
| User correction signals | A user editing or reversing a model's suggested action |
| Downstream business metrics | A proxy metric (conversion rate, escalation rate) moving in a way consistent with worse predictions |
A Worked Example: Fraud Detection Over Time
How drift actually plays out, start to finish
Model deployed, performing well
Validated against a held-out set representative of fraud patterns at launch time
Fraud tactics evolve gradually
New patterns emerge that don't resemble training-time fraud — genuine concept drift, not just noisier inputs
Accuracy declines quietly
Caught late by input-only monitoring, since nothing there looks wrong — this is where delayed ground truth (confirmed fraud cases, weeks later) becomes essential
Detected via feedback loop
Confirmed-fraud outcomes start disagreeing with the model's original predictions at a rising rate
Response
Retraining on recent data, or in the worst case a rollback to a simpler, more conservative model while investigating
This is the realistic shape of concept drift: a quiet decline that data-only monitoring is structurally blind to, caught only once a feedback signal exists to compare predictions against real outcomes.
Designing the Retraining Trigger
When to retrain
Scheduled / periodic
Simple to implement and reason about — retrain monthly, say — but wastes effort when nothing has changed and reacts too slowly when something has changed suddenly.
Metric-triggered
RecommendedRetrain when a drift statistic or a quality metric crosses a defined threshold. Responds to actual conditions rather than the calendar, but needs a metric that's actually trustworthy and low-noise enough not to trigger constantly.
Human-reviewed alert
A metric crossing a threshold pages a person to decide, rather than retraining automatically — appropriate when a bad automatic retrain (on drifted, possibly mislabeled recent data) could make things worse rather than better.
An automatic retraining trigger can make a bad situation worse if the "recent data" it retrains on is itself the problem — e.g. retraining on a window that includes a labeling outage or an adversarial spike. A human-reviewed alert is often the safer default for anything higher-stakes than a low-risk, easily-reversible model.
Concept Checks
Check yourself
Why can concept drift occur while every input-distribution drift test reports nothing unusual?
Because concept drift is a change in the relationship between inputs and the correct output, not a change in the inputs themselves — the incoming data can look statistically identical to training data while what it actually means has shifted. Input-distribution tests like KS-test or PSI only compare feature distributions, so they have no way to detect a change that doesn't show up there.
Why does detecting concept drift require a ground-truth or feedback signal, when data drift doesn't?
Because concept drift is about whether predictions are still correct, and correctness can only be assessed by comparing predictions against actual outcomes — delayed ground truth, user corrections, or a downstream proxy metric. Data drift only asks whether the inputs still resemble training data, which can be checked by comparing distributions directly with no outcome information needed.
Why might a human-reviewed alert be safer than a fully automatic retraining trigger for a higher-stakes model?
Because an automatic retrain assumes the recent data it trains on is trustworthy, and if the "recent data" reflects the very problem causing the drift alert — a labeling outage, an adversarial spike, a temporary anomaly — retraining on it can bake the problem in rather than fix it. A human reviewing the alert can distinguish a genuine, retraining-worthy shift from a data-quality issue that retraining would make worse.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Data drift | The live input distribution diverges from training — detectable from inputs alone |
| Concept drift | The input-output relationship changes — invisible to input monitoring, needs a feedback signal |
| KS-test / PSI | Concrete statistical techniques for detecting data drift |
| Feedback sources | Delayed ground truth, user corrections, downstream business metrics |
| Scheduled vs metric-triggered vs human-reviewed | Three retraining-trigger designs with different risk/effort tradeoffs |
| The risk of automatic retraining | Can bake in a problem if the "recent data" is itself the issue |
Next
Drift is one production risk that costs quality. The next page covers a different, more immediate one: cost itself. Caching & Cost Optimization.
Monitoring & Observability
System metrics, model quality signals, and drift — the three layers of watching a production model, and why only some of them catch a silent failure
Caching & Cost Optimization
Exact-match and semantic caching mechanics, why threshold tuning is the whole game, and what to actually track to catch a cost regression