Feature Engineering and Interaction Design

A model cannot learn a threshold, a ratio, a seasonality, or an interaction that its representation makes inaccessible or unnecessarily difficult. Classical algorithms expose this plainly, because their capacity is often tied directly to the columns supplied.

Feature engineering should encode stable domain structure and availability. It should not mine the full dataset for accidental correlations. The difference between those two activities is invisible in a validation score. That is why most of this lesson is spent on cases where the score looked fine and the column did not.

Each family changes the function the downstream estimator can express. Scale transformations — logs, roots, rates, normalizations — reshape numeric relationships. Temporal summaries — lags, windows, recency, seasonality, trend — describe history before the cutoff. Interactions — products, ratios, crosses, conditional terms — encode joint mechanisms. Group context — peer baselines, entity history, hierarchical aggregates — adds reference information. Domain states — rule-derived flags or stages — translate operational knowledge into model inputs.

The last three families are where every leakage failure in this lesson originates. Each of them reaches outside the current row for its value.

An interaction says the effect of one feature changes with another. A main effect says it does not.

An additive model combines separate feature contributions. It is easy to inspect, often a strong baseline, and able to carry smooth or linear terms, but it cannot express conditional slopes by default. An explicit interaction adds a designed product, cross, or conditional term. It tests one mechanism directly and can stay interpretable, at the cost of more features and a requirement for hierarchy and support checks. A tree interaction appears through nested split paths. It is learned automatically, usually axis-aligned, sometimes unstable, and harder to summarize globally. A kernel interaction is encoded implicitly in the kernel feature space. It can be very rich. It is also less transparent, sensitive to scaling and hyperparameters, and computationally expensive.

Useful features respect time, units, and the process that produces the target.

A historical average must include only events available before the row cutoff. A category target mean must be cross-fitted, so that a row and its validation fold do not teach their own encoding.

Even unsupervised aggregates can leak, when they use future membership, revised records, or the full evaluation population. Leakage does not require a label. The most expensive version is not a leaked label at all. It is a leaked action: a column that records something a person did because the outcome was already suspected.

That failure has been measured on a system in daily hospital use. Kamran and colleagues evaluated the Epic Sepsis Model in 2024 on 77,582 University of Michigan hospitalizations from 2018 to 2020. Sepsis was involved in 3,766 of them, 4.9%. Scored the ordinary way, with predictions issued after clinicians had already recognized the case allowed to count, the model reached an AUROC of 0.62 (95% CI 0.61–0.63). Then they enforced the cutoff a deployed early-warning model actually has to meet. Predictions made after the treatment indicators appeared were excluded: antibiotics, fluids, blood culture, lactate. “When excluding predictions after clinical recognition, the AUROC dropped to 0.47 (95% CI, 0.46 to 0.48).” That is below chance, at exactly the moment the model was built to be useful.

Wong and colleagues had already reported the gap from the other side, in JAMA Internal Medicine. At the same site the model scored AUC 0.63 (95% CI 0.62–0.64) against the 0.76–0.83 its developer reported, with 33% sensitivity and alerts on 18% of hospitalizations. Neither finding required a column holding the label. It required columns that only exist once someone has suspected the diagnosis.

A transformation should be reproducible for both historical training and live inference. Write the hypothesis, stating why the transformed evidence should relate to the outcome. Define the time and unit semantics: window, cutoff, denominator, missing behavior. Implement one transformation, deterministic and reusable across offline and online paths. Test the edge cases — zero denominators, empty histories, unknown categories, boundaries. Validate incrementally against the current baseline for gain, stability, and slice effects. Then document lineage.

That last step is not housekeeping. In regulated modelling it has not been optional for a long time. The Federal Reserve Board and the Office of the Comptroller of the Currency issued interagency Supervisory Guidance on Model Risk Management on 4 April 2011, and it says: “The data and other information used to develop a model are of critical importance; there should be rigorous assessment of data quality and relevance, and appropriate documentation.” Adjustments to data had to be tracked. And any proxy standing in for the real quantity had to be “carefully identified, justified, and documented”.

Almost every feature in this lesson is a proxy. Recency stands in for wear, incidents per operating hour for stress, a peer deviation for local normality. That guidance was rescinded and replaced on 17 April 2026 by revised joint Federal Reserve, FDIC and OCC guidance. The documentation expectation it set is not a passing fashion.

Filtering columns by target correlation, mutual information, univariate tests, or model importance uses outcome information. Performing it once on the full dataset contaminates validation. Put selection inside the pipeline, and report stability across folds rather than one final subset.

What happens when it sits outside the loop was measured in 2002, on microarray gene-expression data. The target was a then-common report: a prediction rule built from a handful of genes, with a negligible error rate. In those reports, Ambroise and McLachlan found, “the test error or the leave-one-out cross-validated error is calculated without allowance for the selection bias”. The reason is that “the cross-validation of the rule is not external to the selection process; that is, gene selection is not performed in training the rule at each stage of the cross-validation process”. They redid two published datasets with gene selection moved inside the loop, and found that “when correction is made for the selection bias, the cross-validated error is no longer zero for a subset of only a few genes”. Their recommendation is specific. Use ten-fold rather than leave-one-out cross-validation. And use the .632+ bootstrap estimate when the rule is heavily overfitted.

The size of that inflation has since been measured directly. Rosenblatt and colleagues tested five forms of leakage across four neuroimaging datasets and three phenotypes, in Nature Communications in 2024. Feature selection performed on the combined training and test data was among the two most damaging of the five. “Feature leakage is widely accepted as a bad practice, and as expected, it severely inflated prediction performance.” The inflation ran from Δr = 0.03 to Δr = 0.52 depending on dataset and phenotype. In one case a prediction of attention problems that sat at chance level, r = 0.01, came back as r = 0.48. Nothing about the second number looks wrong on the page. Kapoor and Narayanan put a prevalence on the habit: leakage in 17 fields, collectively affecting 294 papers, sorted into a taxonomy of eight leakage types.

The damage scales with the number of candidates screened. Google Flu Trends is the case where the selection step was effectively the whole model. It was built by matching 50 million candidate search terms against 1,152 CDC data points, Lazer and colleagues reported in Science in 2014. With that many candidates and that few points to score them on, terms tracking the calendar rather than the disease are certain to survive the screen: “In short, the initial version of GFT was part flu detector, part winter detector.” The consequences were not subtle. It overshot the actual 2011–2012 level by more than 50%. From 21 August 2011 to 1 September 2013 it reported overly high flu prevalence in 100 of 108 weeks. Olson and colleagues had independently documented that the model completely missed the first wave of the 2009 A/H1N1 pandemic and greatly overestimated the 2012/2013 A/H3N2 season.

Where the stakes are high enough, independence has stopped being a matter of taste. The US FDA, Health Canada and the UK MHRA jointly issued Good Machine Learning Practice for Medical Device Development: Guiding Principles in October 2021. Guiding principle 4 is titled “Training Data Sets Are Independent of Test Sets”, and it reads: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.” The three named sources of dependence map straight onto the engineered features in this lesson. An entity-history feature makes rows dependent through the patient. A site baseline or peer deviation makes them dependent through the site. An acquisition-normalized measurement makes them dependent through the instrument. A random row split addresses none of the three.

Hundreds of weak transformations can increase maintenance, drift surfaces, and false discoveries. Durable gain does not follow. Prefer a compact feature set whose provenance and failure modes are understood.

When feature work becomes an attempt to imitate an unrestricted learner manually, compare a model family that learns the structure directly.

A large redundant feature table also does something to the selection step itself, and Guyon and Elisseeff named it in March 2003, introducing a Journal of Machine Learning Research special issue on the subject. “Many methods of variable subset selection are sensitive to small perturbations of the experimental conditions. If the data has redundant variables, different subsets of variables with identical predictive power may be obtained according to initial conditions of the algorithm, removal or addition of a few variables or training examples, or addition of noise.” They also say why that variance is a defect rather than a curiosity. “(i) variance is often the symptom of a ‘bad’ model that does not generalize well; (ii) results are not reproducible; and (iii) one subset fails to capture the ‘whole picture’.” Add redundant columns and the subset you end up with stops being a property of the data.