Feature Transformations and Stateful Pipelines

Promoting a feature should settle what it means, not merely copy code into a repository. Specify the semantics first: entity, cutoff, source, formula, units, null behavior, and intended consumers. Then classify the state. Fitted parameters, historical windows, external versions, and learned dependencies each impose a different obligation on training and on serving. Implement the logic once. Where two implementations are unavoidable, establish golden tests across runtimes. Add value and invariant tests covering normal, missing, boundary, late, duplicate, and adversarial cases. Then version the feature and watch it: usage, distributions, freshness, online parity, and deprecation status. A feature that nobody observes is a feature nobody can retire.

Unit tests verify formulas on small fixtures. Property tests check invariants, such as counts never being negative or recency increasing for a fixed event as the cutoff moves forward.

Golden-record tests compare expected outputs for realistic timelines. Offline–online comparison replays serving requests and measures exact or tolerance-based equality.

Distribution tests can detect broad regressions. They should not replace value-level evidence. Two wrong implementations can produce similar histograms.

Finally, check slice by slice whether each feature is useful and steady. A feature that helps globally but fails for cold-start entities or one region may need fallback logic or narrower use.

The strongest form of the parity test is a logging rule. Google’s Rules of Machine Learning states it as Rule #29: “save the set of features used at serving time, and then pipe those features to a log to use them at training time”. A small fraction of examples will do. The guide reports that the YouTube home page switched to logging features at serving time, with “significant quality improvements”. Skew stops being an argument and becomes a measurement.

Offline and online values may differ because logic, state, source, or timing diverges. Two of those divergences have standards and papers behind them. Both are worth reading before you write a window or a tokenizer.

The clock first. A stream carries two times, not one. Google’s Dataflow Model paper separated them in 2015: event time, “the time at which the event itself actually occurred”, and processing time, “the time at which an event is observed at any given point during processing within the pipeline”. The gap between the two is not a bug waiting to be fixed. Akidau and colleagues write: “During processing, the realities of the systems in use (communication delays, scheduling algorithms, time spent processing, pipeline serialization, etc.) result in an inherent and dynamically changing amount of skew between the two domains.” Their figure for it is titled “Time Domain Skew”. The watermark that bounds the skew is only a heuristic: “for most real-world distributed data sets, the system lacks sufficient knowledge to establish a 100% correct watermark”.

Apache Flink 1.20 defines the same two clocks independently. “Event time is the time that each individual event occurred on its producing device”. “Processing time refers to the system time of the machine that is executing the respective operation”. And “A Watermark(t) declares that event time has reached time t in that stream”. A window that does not say which clock it counts on has not been specified.

Then the text. “Normalize the same way” is not an instruction until it names a form. Unicode Standard Annex #15 defines four. NFD is canonical decomposition. NFC is canonical decomposition followed by canonical composition. NFKD and NFKC are the compatibility versions of each. The payoff is in the annex’s summary: “When implementations keep strings in a normalized form, they can be assured that equivalent strings have a unique binary representation.” And: “Two equivalent strings will have precisely the same normalized form.” The same visible text held in two different forms is two different code point sequences. Two different code point sequences are two different vocabulary lookups.

Applying a form does not settle it either. The W3C warns, in its string-matching character model, that “Many users are surprised to find that two identical-looking strings—including those that have had a specific Unicode normalization form applied—might not in fact use the same underlying Unicode code points.” Its recommendation is still explicit: “Content authors SHOULD use Unicode Normalization Form C (NFC) wherever possible for content.” Choose a form, record it with the vocabulary, apply it on both sides.

A feature transformation can be simple arithmetic, a time-window aggregation, a category vocabulary, a scaler, a tokenizer, or a learned embedding. These operations define what evidence the model sees.

Some transformations are stateless. Converting meters to kilometers always applies the same rule. Others learn parameters from data: means, quantiles, category frequencies, vocabularies.

Stateful transformations must respect training boundaries and carry their fitted state into evaluation and serving. Recomputing them independently creates leakage or skew. The scikit-learn documentation gives the definition plainly: “Data leakage occurs when information that would not be available at prediction time is used when building the model.”

Fit a data-dependent step on the whole dataset before the evaluation split and you do not shade the estimate. You manufacture it. Ambroise and McLachlan measured that in PNAS in 2002, on microarray gene-expression data. With gene selection performed on the whole leukemia set before cross-validation, “the leave-one-out error CV1IE is zero for only three selected genes” — a perfect classifier from three genes. Move the selection outside the cross-validation and the error rises to about 5%, by external 10-fold CV and by the .632+ bootstrap. On the colon data the same contrast runs 6.5% internal against roughly 15% external. Same data, same genes. Only the position of the selection changed.

scikit-learn reproduces the effect from nothing at all. Its page on common pitfalls builds 200 samples, 10,000 randomly generated features, and randomly assigned binary targets. With SelectKBest(k=25) fitted before the split, accuracy_score comes out at 0.76. Fitted after the split, it comes out at 0.5, which is what a coin does. The scikit-learn developers state the conclusion: “Using all the data to perform feature selection results in an accuracy score much higher than chance, even though our targets are completely random.”

This is not a niche failure. Kapoor and Narayanan, surveying leakage across machine-learning-based science in Patterns, found “17 fields where leakage has been found, collectively affecting 294 papers”. Their taxonomy names both of the failures above. [L1.2] is pre-processing on training and test set: “Using the entire dataset for any pre-processing steps, such as imputation or over/under sampling, results in leakage.” [L1.3] is feature selection on training and test set.

A feature pipeline should expose dependencies, types, time semantics, fitted artifacts, defaults, and version. Reuse is valuable only while the reused definition still matches the decision context.

Google’s TFX team put a price on one such gap. Comparing serving logs against training data on the same day, Google Play found features that were always missing from the logs but always present in training. An online A/B experiment followed. Removing the skew “improved the app install rate on the main landing page of the app store by 2%”. Two percent of a store front, from a mismatch that every column name agreed about.

The category determines how a transformation must be trained, versioned, and served. Five categories cover most of what a pipeline contains.

A pure stateless transform applies fixed logic to each value or record, independently. A fitted transform learns parameters — mean, bins, vocabulary, encoder state — from training data, and therefore carries a dataset version inside it. A historical aggregation summarizes events in a bounded window before a prediction cutoff, which makes it hostage to the clock that window is counted on. An external enrichment adds reference data or model outputs, each with its own version and availability. A learned representation produces embeddings or latent features through a separately trained model, and inherits every boundary that model was trained under.

Only the first is free. The other four each drag something along behind them.

The best choice makes critical semantics reviewable and executable across environments.

A notebook transformation keeps feature logic beside one experiment. It is fast for exploration, difficult to test and reuse consistently, and it often hides fitted state and source assumptions. It is appropriate only before the feature becomes shared.

A shared code library has training and serving import the same implementation. It reduces duplicated logic and supports unit and property tests. It still needs data and time contracts, and it is most useful for complex language-specific transformations.

A declarative feature definition has a platform execute versioned feature metadata and transformations. It makes lineage and reuse easier to discover and can generate offline and online materializations. The cost is limited room for custom algorithms, and coupling to the platform. It earns its keep when many teams share common entities and windows.

That third column is not a category invented for a comparison table. Feast is a dated, documented instance of it. Tim Sell and Willem Pienaar announced it on the Google Cloud Blog on 19 January 2019: “Developed jointly by GO-JEK and Google Cloud, Feast aims to solve a set of common challenges facing machine learning engineering teams by becoming an open, extensible, unified platform for feature storage”. The consistency mechanism is the specific thing worth copying: “Feast provides consistency by managing and unifying the ingestion of data from batch and streaming sources, using Apache Beam, into both the feature warehouse and feature serving stores. Users can query features in the warehouse and the serving API using the same set of feature identifiers.” One ingestion, two materializations, one set of identifiers.

The project documentation describes Feast today as “an open-source feature store that helps teams operate production ML systems at scale by allowing them to define, manage, validate, and serve features for production AI/ML”. It also says what its point-in-time correct retrieval is for: “Avoid data leakage by generating point-in-time correct feature sets so data scientists can focus on feature engineering rather than debugging error-prone dataset joining logic.” A platform that generates the training join for you is a platform that can enforce the cutoff you would otherwise have to remember.