Feature Stores & Data Pipelines
Training-serving skew explained in full, and the online/offline feature store architecture that exists specifically to prevent it
Feature Stores & Data Pipelines
TL;DR
Training-serving skew — a feature computed one way offline during training and a subtly different way online during serving — is one of the most common and most silent causes of production ML failure, because nothing errors; the model just quietly sees different inputs than it was trained on. A feature store exists to compute each feature once, from one definition, and serve identical values to both paths.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | LLM Gateway & Routing |
| You will understand | Exactly how training-serving skew happens, and the architecture that prevents it |
A Worked Example of Skew
Consider a feature: "average purchase amount over the last 30 days." It sounds unambiguous. It usually isn't computed identically in two different pipelines.
The same feature, computed two different ways
Offline (training pipeline)
A nightly batch job computes this over a cleanly closed 30-day window — midnight to midnight, using the full historical purchase table, with a well-defined answer for a customer with zero purchases (feature = 0, or null, decided once).
Online (serving path)
A rolling 30-day window computed at request time, against a cache that refreshes every few hours rather than continuously, with a slightly different null-handling rule written by a different engineer months later who didn't know the offline convention.
Nothing about this errors. Both pipelines return a plausible-looking number. The model was trained on the offline pipeline's numbers and is now scored in production on the online pipeline's subtly different numbers — and the gap between them is invisible unless someone specifically goes looking for it, usually only after accuracy has already quietly dropped.
This is training-serving skew, and it's considered the classic silent production ML bug precisely because of that invisibility — a model failure that looks, from every normal monitoring signal, like the model itself getting slightly worse over time (which invites drift-hunting) when the actual cause is a data pipeline inconsistency that was there from day one.
What a Feature Store Actually Does
The fix is architectural, not procedural: define each feature once, and derive both the training data and the serving-time value from that single definition.
Online/offline feature store architecture
Feature definition
Offline store
Online store
Both stores are derived from the same feature definition, so "average purchase amount over 30 days" means the same computation whether it's being pulled for a training dataset covering the last two years or for a live request scoring a customer right now.
# One definition, used to populate both stores
@feature_definition
def avg_purchase_30d(customer_id: str, as_of: datetime) -> float:
purchases = get_purchases(customer_id, window_days=30, as_of=as_of)
return sum(p.amount for p in purchases) / max(len(purchases), 1)"Point-in-time correct" is worth calling out specifically for the offline store. Training data has to reflect what the feature's value actually was at the historical moment each training example occurred — not its current value. Computing training features from today's data (which includes purchases that happened after the training example's timestamp) leaks future information into training, a distinct but related bug sometimes called "label leakage through features."
Feature Versioning
A feature's definition changes over time — someone adjusts the window from 30 to 14 days, or fixes a null-handling edge case. When that happens, a model trained against the old definition and served against the new one has the exact same skew problem as two independently-written pipelines did.
avg_purchase_30d v1 → 30-day window, nulls treated as 0
avg_purchase_30d v2 → 30-day window, nulls excluded from the average (a real behavior change)
Model trained against v1 must keep being served v1 values,
until it's retrained against v2 and explicitly promoted.A feature store without versioning just moves the skew problem from "two pipelines" to "one pipeline that changed underneath a deployed model." Treat a feature definition change with the same discipline as a model version change — track which version each deployed model depends on.
When This Doesn't Apply
Pure LLM-backed systems mostly sidestep this problem. A prompt built from raw text and retrieved context doesn't go through the kind of numeric feature engineering that creates offline/online skew — see RAG for the retrieval-side equivalent concerns. Feature stores matter most for traditional ML — classification, regression, ranking, recommendation — built on structured, engineered features, which is still the majority of production ML systems even in an LLM-heavy landscape.
Concept Checks
Check yourself
Why is training-serving skew considered the 'silent' bug, rather than something that shows up immediately in testing?
Because neither the offline nor the online computation errors — both return plausible-looking numbers, just subtly different ones for the same underlying feature. The mismatch only shows up as a gradual, unexplained accuracy drop in production, which often gets misdiagnosed as model drift rather than traced back to the actual cause: two pipelines computing the "same" feature differently.
Why do a feature store's online and offline stores need to derive from the same feature definition, rather than each team implementing the logic independently but carefully?
Because independent implementations, however careful, drift apart over time as different engineers touch each pipeline for different reasons — exactly the scenario that produced the 30-day-average example. Deriving both stores from one shared definition makes it structurally impossible for them to diverge, rather than relying on two teams staying in sync through discipline alone.
A feature's definition changes from including to excluding null values in an average. Why isn't updating the pipeline immediately safe for an already-deployed model?
Because the deployed model was trained against the old definition's values, and immediately serving it the new definition's values reintroduces training-serving skew — just from a single pipeline changing underneath the model instead of two pipelines disagreeing. The model needs to keep receiving the version it was trained on until it's retrained against the new definition and explicitly promoted to use it.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Training-serving skew | A feature computed differently offline vs online — invisible until accuracy quietly drops |
| Why it's silent | Neither computation errors; both return plausible values |
| Feature store's job | Compute each feature once, serve identical values to training and serving |
| Online/offline duality | A low-latency online store and a historical offline store, both derived from one definition |
| Point-in-time correctness | Training features must reflect their value at the historical moment, not today's value |
| Feature versioning | A changed definition needs the same version discipline as a model change |
| Where this applies least | Pure LLM/RAG systems built on raw text and retrieval, not structured engineered features |
Next
With data pipelines covered, the next page addresses who's allowed to touch any of this — models, data, and deployments — and how that gets audited: Security & Governance.