Feature Stores and Historical Retrieval

Confusing these components leads to unrealistic expectations about the platform. The split is not architectural taste. At a certain size a single store cannot serve both workloads, and the systems that got there first say so in public.

Uber built a Feature Store inside its Michelangelo platform, and in 2017 Uber's engineering blog said how large it had already become: “At the moment, we have approximately 10,000 features in Feature Store that are used to accelerate machine learning projects, and teams across the company are adding new ones all the time”. One catalogue, approximately ten thousand shared definitions. Two different physical stores underneath it.

Uber's Palette feature store splits storage by workload. Hive/HDFS keeps daily historical snapshots for training jobs to consume in bulk. A Cassandra key-value store keeps the latest known values for online reads, and features are expected back in single-digit P99 latencies. Amit Nene, an engineering lead on Michelangelo, described the online half in a 2019 talk: “The purpose of the online store is to serve those same features in real time in a low latency way, today we use a KV store there, Cassandra in particular.” Online models cannot read HDFS. So the values they need are precomputed into Cassandra, ready for a low-latency read at prediction time.

Notice what is shared and what is not. The definition, the entity key and the point-in-time join rule are shared. The engine, the access pattern and the latency budget are not. That is why the registry is the third component. Something has to assert that a daily Hive snapshot and a Cassandra row are the same feature. That assertion is metadata, not evidence.

Teams adopt feature stores to discover, reuse, retrieve, and serve feature definitions consistently. The platform may manage metadata, historical joins, batch materialization, and low-latency online values.

It cannot decide whether a feature is meaningful, fair, leakage-free, or appropriate for a new task. A shared defect can spread faster than a notebook defect.

KDD Cup 2008 is the case to keep in view, because the defective feature there was immaculate by every check a registry performs. The competition released 1,712 training patients, 118 of them with cancer, and 102,294 candidate lesions described by 117 features. The winning team, from IBM Research and Tel Aviv University, opened its 2008 report with what it had found: “The presence of leakage, whereby patient IDs turned out to carry significant information about a patient's likelihood to be malignant.”

The anonymised patient ID was strongly predictive. Their hypothesis was that ID ranges tracked the contributing medical institution. The numbers are the argument. Of the 254 patients with IDs below 20,000, 36% were malignant. Of the 414 patients with IDs between 100,000 and 500,000, 1%. Of the 1,044 above 4,000,000, 1.7%. Binning the ID on those boundaries generalised perfectly to the test set. A 2011 paper on leakage in data mining returned to the same competition: most competitors ignored the “Patient ID” feature, the leakage was facilitated by assigning consecutive patient IDs per source, and the remedy it prescribes is time-stamped “legitimacy tags” with a learn-predict separation.

Nothing a registry stores would have failed that feature. It had a schema, an owner, a source, a stable key, perfect availability and outstanding measured signal. What it encoded was its own provenance.

The central challenge is dual use. Training needs historical values aligned to many past prediction times. Serving needs the latest eligible value for one or several entities, under a latency budget.

Consistency requires shared semantics, keys, time rules, transformation versions, defaults, freshness, and lineage. Identical feature names are insufficient.

Google's Rules of Machine Learning states the governance version of this. Rule #11 is to “Give feature columns owners and documentation”, a rule written for large systems with many features. Rule #22 is to “Clean up features you are no longer using”. An unused feature is treated there as technical debt, not as harmless clutter.

The metadata layer connects historical and live values without making them physically identical systems. What connects them is a rule about time. In a real store that rule is written down, with parameters you can read and set.

Feast, the open source feature store, specifies historical retrieval as a backward scan. For each row of the entity dataframe it looks back from that row's own timestamp, for at most the feature view's TTL. Its documentation flags the parameter that is misread most often: “Please note that the TTL time is relative to each timestamp within the entity dataframe. TTL is not relative to the current point in time (when you run the query).” TTL bounds how stale a value may be relative to the training example, not relative to the moment someone ran the job. That is why re-running the same query next quarter must return the same rows.

The default has a second edge, and it is the one that quietly breaks step 3 and step 5. Feast by default constrains only the feature's event timestamp. So a value that was backfilled or corrected after an entity row's timestamp can still be joined into that row's training data. Enabling created-timestamp filtering adds a created_timestamp <= entity_timestamp condition to the join, in Feast's own words “to keep backfilled values from leaking into training data, and to reproduce what the online store would have served at each event time”.

An independently built store states the same retrieval rule. Google Cloud's Vertex AI Feature Store says that at a label timestamp T1 it “returns the latest feature values up to time T1 for Feature 1, Feature 2, and Feature 3 and doesn't leak any values past T1”. A null value at that timestamp falls back to the previous non-null value.

So two timestamps decide whether the historical path reproduces the live path. One bounds the backward scan. The other bounds eligibility. Both belong in the feature definition, not in whichever query a consumer wrote last week.

Each case needs the data itself compared, not only the configuration reviewed. The failure class has a published taxonomy, a named mechanism, and a specific detection method.

Google's TFX data-validation system runs at production scale: hundreds of product teams, several petabytes of production data per day, with results reported from a sample of more than 700 ML pipelines. The 2019 paper describing it divides training-serving skew into three kinds: feature skew, distribution skew, and scoring/serving skew. Adrian Colyer's read of the paper puts the first one plainly: “Feature skew occurs when a particular feature assumes different values in training versus serving time”.

One mechanism gets a name of its own: “A more interesting mechanism through which feature skew occurs is termed time travel. This happens when the feature value is determined by querying a non-static source of data.” That is the offline path recomputing today what the online path read months ago, from a source that has moved in between. A store with a registry, a schedule and a shared name is no protection against it.

The detection method the paper gives is what every bullet below has to end in: a key-join between corresponding batches of training and serving data, then a feature-wise comparison. Not a configuration review. A join.

The feature should pass semantic, temporal, operational, and consumer checks before broad reuse. Step 4 is the one teams treat as optional. There is a documented case in which exactly that comparison found the defect, and the fix was measured.

When Google moved the Google Play recommender onto TFX: “By comparing the statistics of serving logs and training data on the same day, Google Play discovered a few features that were always missing from the logs, but always present in training. The results of an online A/B experiment showed that removing this skew improved the app install rate on the main landing page of the app store by 2%.” That is Google's own 2017 paper on TFX, in its Google Play case study. The abstract of the 21-author paper carries the same result at the top level, as “a 2% increase in app installs resulting from improved data and model analysis”. Google's data-validation paper reports the case again two years later, under the heading “Missing features in Google Play recommender pipeline”.

Read what the defect actually was. Not a wrong value — an absent one, on the serving side only, for features the training side always had. Each side was internally valid. No schema check and no null-rate threshold on either path alone would have raised it. Only the comparison of the two, on the same day, on matched statistics, could.

That is what step 4 buys. It is why the evidence a feature needs before broad reuse is value-level rather than descriptive.

Built for one decision, a feature may leak future information, encode a prohibited attribute, or use an unsuitable time window for another. Reuse should begin with semantic review, not name matching.

For high-risk AI systems that review is no longer only good practice. Article 10 of Regulation (EU) 2024/1689, the EU Artificial Intelligence Act, turns it into numbered obligations. Paragraph 2(b) requires documented “data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection”. Paragraph 2(d) requires “the formulation of assumptions, in particular with respect to the information that the data are supposed to measure and represent”. Paragraph 3 judges the data sets themselves against the use: “Training, validation and testing data sets shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose.” A registry entry that cannot say where the values came from, what they were originally collected for, and what they are assumed to measure is not merely thin documentation. For these systems it fails a legal requirement. Note which field that is. The original purpose of collection is the one that would have marked a patient ID as a record-keeping artefact rather than a clinical signal.

Track consumers and models so a definition change has an impact map. Deprecation should tell consumers how to migrate, and leave both versions running for a period.

Measure online retrieval coverage by entity and slice. A high average hit rate can hide severe cold-start or regional gaps.

Finally, remove or archive features that have no active consumers. Unused materializations create cost, security exposure, and confusing alternatives with similar names.

The worst case is a consumer nobody has written down. Without access controls, Sculley and colleagues warned in 2015, some consumers “may be undeclared, silently using the output of a given model as an input to another system”. That coupling is hidden and tight. A change to the model reaches parts of the stack nobody listed. The remedy they name is access restrictions or strict service-level agreements.