Today's concept: feature stores, what they solve and when you skip one
Most "it worked in my notebook, it's broken in production" bugs are not model bugs. They are the same input computed two different ways in two different places.
A "feature" is just an input to a model: "user's average order value over the last 30 days." A feature store is a shared fridge for those numbers instead of every team keeping its own groceries.
1) The bug it kills: training-serving skew The data scientist computes that number as batch SQL over a year of history. The backend engineer recomputes it as a live lookup, in another language, under a millisecond budget. Same intent, two implementations, and they drift. Nothing crashes. The model just quietly gets worse and the blame points nowhere.
2) The shape of the fix Feast (open source, Apache 2.0) splits storage in two: an offline store of historical values for building training sets, and an online store that serves the latest value fast. One feature definition feeds both, and a materialization job pushes values from offline into online. Databricks and Google Cloud ship their own managed feature stores with that same offline/online split, not hosted versions of Feast itself.
3) The part people underrate: point-in-time correctness A training set is not "join the feature table." You need each value as it stood at that past moment. Join today's value onto a year-old event and the model gets to peek at the future: offline metrics look great, production disagrees. Point-in-time correct joins are most of what you are buying.
You can also skip the infra. Google's Rules of ML has a cheaper move: log the features you actually served, then train on those logs, so serving becomes the source of truth. And skip it for prototypes, one model, one team, or when your only "features" are the prompt and context you hand an LLM. The coordination overhead pays off only once several models or teams share the same computed signal.
Same bug, new hats: re-embed your index with a newer embedding model while queries still run through the old one, the two vector spaces stop lining up, and retrieval quietly degrades. For agents it shows up as offline evals disagreeing with live runs because context is not assembled the same way in both. Telling that Google's managed feature store now sits inside its agent platform and handles embeddings.
Quick check before you scroll: Why can't a data scientist just compute features fresh in both the training script and the production API and call it a day?
Full breakdown + the answer: frankduah.me/learnings/2026-08-24-feature-stores-what-they-solve-and-when-you-skip-one
New here? I post a bite-size AI / ML concept like this every day, follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#FeatureStores #AI #LLM #AIAgents #MachineLearning
The answer
Because two independently-written implementations of "the same" feature logic tend to drift apart over time (different libraries, edge cases, timing), causing training-serving skew, the model was trained on one version of reality and scores on another.