Features, Feature Stores, and Training-Serving Skew

A model scores well offline. It ships. Live accuracy is materially worse, and nothing in the code has changed.

Google's MLOps guidance names the cause: training-serving skew, which occurs when the features used for training are different from the ones used during serving.

What makes this the hardest class of ML bug is that every component is behaving correctly. The training code is right. The serving code is right. The model is right. They simply disagree about what a feature means, and no test that examines one component in isolation can see a disagreement between two.

It is also silent. The model does not error on a skewed feature, it just returns a worse answer, and worse answers do not page anyone.

The classic source of skew is writing the same feature twice.

Training runs offline over historical data, usually in Python or SQL against a warehouse. Serving runs online with a latency budget, often in a different language, against a different store. So average_order_value_30d gets implemented once by the data scientist and once by the backend engineer.

# training, over a warehouse table
feat = (orders
        .filter(orders.ts > now - timedelta(days=30))
        .groupby("user_id")
        .agg(avg("amount")))

-- serving, against the transactional store
SELECT AVG(amount) FROM orders
WHERE user_id = ? AND ts > NOW() - INTERVAL '30 days';

These look identical. They disagree if the warehouse excludes cancelled orders and the transactional store does not, if one counts in UTC and the other in local time, or if the warehouse lags by six hours.

Skew is rarely one dramatic mistake. It is an accumulation of small definitional gaps, and these are the recurring ones.

The null and type rows cause the most damage per incident. If training imputed the column mean for missing values and serving sends 0, the model reads a missing value as a genuine extreme, and does so confidently.

There is a second, subtler failure that is not a mismatch between two systems but a mistake inside one.

When you build a training set, each row pairs a label with the features as they were at the moment of the prediction. It is easy to instead join the feature values as they are today. A user who churned last March gets a total_lifetime_orders computed now, which includes everything up to the present.

That is label leakage. The feature carries information from after the label, and the model learns a relationship that cannot exist at serving time. Offline accuracy looks superb, because you handed the model the answer. Live accuracy collapses.

This is why correct feature retrieval requires an as-of join against a full history, not a lookup of the current value.

The structural fix is not better discipline across two implementations. It is having one implementation that all three consumers read from.

Google's guidance introduces the feature store at MLOps level 1, and describes its purpose directly: it avoids training-serving skew by acting as the data source for experimentation, continuous training and online serving alike.

The mechanism is that one feature definition materialises into two stores with different shapes.

• The offline store holds full history and is optimised for large scans with an as-of join, which is what training and backfills need.
• The online store holds only the latest value per entity and is optimised for a single-key lookup in single-digit milliseconds, which is what a prediction request needs.

Both are generated from the same definition, so the two cannot drift apart the way two hand-written implementations do. A secondary benefit is reuse: a feature built for one model becomes discoverable rather than being rebuilt slightly differently for the next.

A feature store makes the two sides agree. It does not tell you when the input itself has gone wrong, and that is a separate gate.

Data validation compares incoming data against an expected schema: the columns present, their types, the categorical values allowed, and the plausible range or distribution of each numeric field. At MLOps level 1 this runs automatically before training, and it can halt the pipeline.

Halting is the point. Without it, an upstream team renaming a category or a sensor returning zeros produces a model trained on corrupted input, which then passes model validation if the corruption is present in the evaluation set too.

The schema is also a contract you can diff. When it changes, someone made a decision, and now there is a record of it.

Feature stores are infrastructure, and infrastructure has a running cost. Several situations do not justify one.

• Batch-only prediction. If you score once a night with the same code that trains, there is no second implementation, so there is no skew to prevent.
• One model, few features. The coordination problem a feature store solves is mostly a problem of many teams and many models sharing definitions.
• Features computed entirely from the request. If everything the model needs arrives in the prediction call, there is nothing to look up.

The cheaper intervention that works at any scale is to write each feature transformation once, as a shared library imported by both the training job and the serving path. That removes the duplicate implementation, which is the actual disease. A feature store is the version of that idea with storage and history attached.

If a model is live and you suspect skew, you do not need new infrastructure to check.

Log the exact feature vector the serving path constructs, for a sample of real requests. Then take those same entities and timestamps, run them through the training feature pipeline, and compare the two vectors field by field.

Compare distributions, not just individual rows. A single row matching proves little, whereas a per-feature comparison of mean, null rate and category frequency between the logged serving values and the training values will surface a mismatched filter or a null convention immediately.

This logged-vector comparison is worth building before a feature store, because it also tells you whether you have a skew problem at all, and therefore whether the larger investment is justified.