Categorical, Missing, and Sparse Data

A robust pipeline begins by classifying what each field means. Accepting the database type is not enough. Every field below can be stored as a short integer or a string, and each one behaves as a different statistical object the moment it reaches an estimator.

Four of them are the subject of the rest of this lesson. An outcome statistic that separates a training set perfectly and predicts nothing. A code set that gained a new category on a date somebody else chose. A matrix whose zeros are the only reason it fits on a disk. An identifier that a model can read straight off the raw input. Each of those has already cost someone a published result.

The correct choice depends on cardinality, model family, support, and inference behavior. Two of the four options have been measured at scale, and the measurements are what the bullets rest on.

Hashing was tested on a live spam filter. Weinberger and colleagues ran it in 2009 on “a proprietary email spam-classification task of n = 3.2 million emails, properly anonymized, collected from |U| = 433167 users”. And “after tokenization, the data set consists of 40 million unique words”. Giving every user a personalised copy of that vocabulary is what makes the width impossible. It is not merely inconvenient. The cross product of “u = 400K users and n = 40M tokens results in 16 trillion possible unique personalized features”. Hashing into 2^22 buckets — roughly four million columns — was enough. “Personalization results in a 30% spam reduction once the hash table is indexed by 22 bits”. The decision threshold was fixed so that exactly 1% of legitimate test mail was flagged as spam. The collisions were not free. They were cheaper than the dimension.

Target encoding fails in the opposite direction. It is the compact option, and it looks best in training. The 2018 CatBoost paper names the defect in seven words: “The problem of such greedy approach is target leakage”. Then it builds the extreme case. Take a categorical feature whose values are all unique and which carries no signal at all, so that P(y=1|x^i=A)=0.5 for every category A. Encoded greedily, a single split at threshold (0.5+ap)/(1+a) classifies every training example correctly. Every test example receives the prior p, and the model scores accuracy 0.5. Perfect training separation, coin-flip deployment, from a column that knows nothing. The obvious repair does not help: of leave-one-out the authors write “Surprisingly, it does not prevent target leakage”. Nor is the retreat to one-hot open, since on a “user ID” feature it “leads to infeasibly large number of new features”. The method is older than the paper, which credits Micci-Barreca in 2001 — the same reference scikit-learn cites as [MIC] behind its own TargetEncoder.

scikit-learn's runnable example puts a price on the fix. The data is 50,000 synthetic samples: one informative feature, plus two uninformative categoricals, a medium-cardinality “shuffled” column and an extreme “near_unique” one. Fit the encoder with fit_transform, which cross-fits over five folds by default, and a Ridge model scores 0.8000184677460299 on train and 0.7927845601690924 on test. Fit the same encoder with fit and apply it with transform, and the same model scores 0.858486250088675 on train and 0.6338211367110548 on test. The training score went up. The test score fell by about a quarter. That is a train/test gap of roughly 0.22 where cross-fitting leaves roughly 0.01, produced by nothing except the fold-awareness of the encoder. The example says where the weight went: “The ridge model overfits because it assigns much more weight to the uninformative extremely high cardinality (“near_unique”) and medium cardinality (“shuffled”) features than when the model used cross fitting to encode the features.” For scale, the raw unencoded features score 0.0050 on train and 0.0046 on test. The user guide is blunter about the non-cross-fitted path: “It is discouraged to use this method because it can introduce data leakage”.

Unknown values are the other half of the encoding contract, and the same class writes its policy down. TargetEncoder “considers missing values, such as np.nan or None, as another category”, and “Categories that are not seen during fit are encoded with the target mean, i.e. target_mean_”. That is a decision, documented. It is the kind of decision every encoder in a pipeline owes the reader.

Statistical labels such as MCAR, MAR, and MNAR are useful, but operational provenance is equally important.

That vocabulary has a first source and a precise meaning. The conditions were set out in Biometrika in 1976. For direct-likelihood or Bayesian inference, “it is appropriate to ignore the process that causes missing data if the missing data are missing at random and the parameter of the missing data process is ‘distinct’ from θ”. Sampling-distribution inference additionally requires that the observed data be “observed at random”. Even then the conclusions are “generally conditional on the observed pattern of missing data”. Rubin calls these “the weakest general conditions under which ignoring the process that causes missing data always leads to correct inferences”. Ignorability is therefore a property of the process that produced the gaps. It is not a property of the table that contains them.

The three labels are not academic furniture either. A regulator defines them operationally, in a document that governs what evidence will be accepted in a drug trial. The Committee for Medicinal Products for Human Use (CHMP) set out MCAR, MAR and MNAR in its 2010 guideline on missing data in confirmatory clinical trials. Which is why the provenance categories below are doing statistical work, and not clerical work.

Mean or median imputation creates a conventional placeholder; a missingness indicator lets the model distinguish imputed from observed rows. Model-based imputation adds stronger assumptions and can leak if fitted outside folds.

Imputation does not recreate the unobserved truth. It creates a representation that must support the downstream decision.

The careful version of this has a reference implementation, the R package mice. Van Buuren and Groothuis-Oudshoorn documented it at length in 2011, and three details from that paper bear on the paragraphs above. The output is not one completed table. “The default number of multiple imputations is equal to m = 5”, a default the package's own current reference manual still carries. The placeholder is named for what it is: of mice.impute.mean() the paper says it “simply imputes the mean of the observed data. Mean imputation is known to be a bad strategy, and the user should be aware of the implications.” And categorical columns bring a failure of their own. The paper reports that “Imputation of categorical data is improved in order to bypass problems caused by perfect prediction”. That failure made earlier versions emit “fitted probabilities numerically 0 or 1 occurred and algorithm did not converge”. Version 2.9 answers it by “augmenting the rows prior to imputation”, inside mice.impute.logreg() and mice.impute.polyreg().

Careful machinery does not make the assumptions checkable, and two institutions have said so in documents that bind other people's work. The CHMP guideline states in its executive summary: “Unfortunately, when there are missing data, all approaches to analysis rely on assumptions that cannot be verified. It should be noted that the strategy employed to handle missing values might in itself be a source of bias.” The same guideline records that “there is no universally applicable method that adjusts the analysis to take into account that some values are missing, and different approaches may lead to different conclusions”. It therefore requires the handling method to be pre-specified in the protocol — chosen before the data can show which choice flatters the result. The National Research Council panel convened for the FDA reached the same verdict in 2010: “all of these methods ultimately rely on untestable assumptions concerning the factors leading to the missing values”, and “There is no “foolproof” way to analyze data subject to substantial amounts of missing data”. A pipeline that picks an imputer after seeing which one scores best has inverted that order. No validation split will report the inversion.

One-hot categories and token counts can contain millions of logical zeros, and centering a sparse matrix can turn it dense and exhaust memory, while some solvers exploit sparse storage directly.

Choose scaling, regularization, and estimators that preserve sparsity when zero is meaningful.

The standard benchmark for this shape is worth carrying as a number. RCV1-v2 is 804,414 Reuters newswire stories, assembled in 2004 and described in the Journal of Machine Learning Research. Its “text representation approach produced a set of 47,236 features (stemmed words)”. The chronological LYRL2004 split gives 23,149 training and 781,265 test documents. That is a matrix of roughly 38 billion cells. Scikit-learn ships the same collection as fetch_rcv1, and records what is actually stored in it. “The array has 0.16% of non zero values.” About sixty million numbers instead of thirty-eight billion is the entire difference between a file and a cluster. Subtracting a column mean destroys it in a single pass.

Two widely used libraries have written that hazard into their APIs, and they differ only in how they enforce it. scikit-learn's StandardScaler defaults to with_mean=True for dense input; on sparse input it refuses outright. “This does not work (and will raise an exception) when attempted on sparse matrices, because centering them entails building a dense matrix which in common use cases is likely to be too large to fit in memory.” Its documented remedy is to pass with_mean=False “to avoid breaking the sparsity structure of the data”. Apache Spark MLlib makes the opposite default choice for the same reason: withMean is False by default, and switching it on “will build a dense output”. One library raises an error, the other flips the default. Neither treats centering a sparse matrix as an ordinary option.

The industrial instance of this shape has a name and a size, and it is sized by its categorical columns. Criteo AI Lab distributes the 1TB Click Logs release as 24 files, one per day of Criteo traffic, and describes a row exactly: “There are 13 features taking integer values (mostly count features) and 26 categorical features. The values of the categorical features have been hashed onto 32 bits for anonymization purposes.” Missingness gets an explicit representation as well — “When a value is missing, the field is just empty”. The twenty-six code columns are what make the collection heavy. MLCommons builds the MLPerf DLRM-v2 (DCNv2) recommendation benchmark on the multi-hot version of the same data, about 343 GB across day_0.gz to day_23.gz, where “day_23 contains 178274637 rows in total”. Thirteen numbers and twenty-six codes per row, and it is the codes that decide what the deployment costs.

Inference behavior should be specified for every unusual value before launch. Semantic roles first, then documented missingness mechanisms, then fold-aware encoders fitted inside training folds, then an explicit unknown-value policy, then a cardinality and memory stress test, then monitoring of new levels, missing rates, collisions and source-system changes.

The fifth of those is where the arithmetic of the previous sections turns into a configuration flag that somebody has to type. MLCommons runs the official MLPerf DLRM-v2 (DCNv2) configuration with --max-ind-range=40000000, which its reference implementation defines as “the maximum number of vectors allowed in an embedding table”. With that cap in place the reference PyTorch model is 97.31 GB in fp32, described there as “the largest model on the order of 100GB”. Nothing in the data chose forty million. A person chose a width, and the width is the difference between a model that can be built and one that cannot — exactly the trade the hashing paper made, one order of magnitude of engineering further along.

A model may memorize customer, clinician, device, or location IDs and perform well when the same entities appear in validation, but that success can vanish on new entities or encode undesirable profiling. The identifier does not even have to be a column in the table.

A chest radiograph carries its hospital of origin in the pixels, and a network will read it. Zech and colleagues trained convolutional networks on 158,323 chest radiographs from three institutions — NIH 112,120, Mount Sinai 42,396, Indiana University 3,807 — and published the result in PLOS Medicine in November 2018. First they asked whether the pixels carried the institution. “A CNN trained to identify hospital systems accurately identified 22,050 / 22,062 (99.95%, 95% CI 0.9991–0.9997) of NIH, 8,386 / 8,388 (99.98%, 95% CI 0.9991–1.0000) of MSH, and 737 / 771 (95.59%, 95% CI 0.9389–0.9693) of IU test radiographs”. A network trained on Mount Sinai departments went further. It separated inpatient from emergency radiographs at 100%: 5,805 of 5,805 and 449 of 449.

Then the size of the illusion that identity buys. Pneumonia prevalence was 34.2% at MSH against 1.2% and 1.0% elsewhere, so “a trivial model that ranked cases based only on the average pneumonia prevalence in each hospital system achieved AUC 0.861 (95% CI 0.855–0.866) on the joint MSH–NIH test set”. Knowing only where a radiograph came from, and nothing whatever about the patient, scores 0.861. The jointly trained network scored 0.931 internally and 0.815 externally. Most of the internal number was available for free from the site. A shortcut-learning review in Nature Machine Intelligence in 2020 reports the same case as canonical, names the hospital-specific metal token in the corner of the image, and concludes that the model reached a reasonably good prediction “without learning much about pneumonia at all”.

Use grouped splits, provenance review, and purposeful aggregate features instead of raw identity by default. A split that lets the same hospital, clinician or customer appear on both sides is measuring recall of the entity, not skill at the task.

After preprocessing, verify column count, density, names, scale, category support, and missing indicators. Check that no row becomes all zeros unexpectedly. Confirm that train and inference transformations produce compatible schemas.

Each of the failures above is visible in that matrix and invisible in the dataframe. A target-encoded column that separates the training set perfectly looks like an ordinary float. A centering step that took a 0.16%-dense matrix to fully dense is a memory event, not a validation error. A vocabulary that gained a code last quarter shows up as a column count that no longer matches the fitted encoder. A model audit should be able to trace a transformed coefficient or split back to its source meaning, and to say which fold the statistic in that column was computed on.