Categorical, Sparse, and High-Cardinality Data

Even a simple encoder has a lifecycle, and that lifecycle has to be reproducible at training and at serving. Observe category frequencies, aliases, hierarchy and first-seen times. Approve valid, restricted, deprecated and fallback states. Encode under a version. Serve unseen and retired values consistently. Migrate by dual-running versions and measuring the affected population.

Federal statistics run exactly that loop, on a published clock. In a Federal Register notice of 21 December 2021 the U.S. Office of Management and Budget adopted the 2022 revision of the North American Industry Classification System. The notice states the approval cadence in one line: “Revisions are considered every five years in calendar years ending with 2 and 7.” The same notice directs that federal statistical establishment data for reference years beginning on or after 1 January 2022 be published using 2022 NAICS codes. That is the Approve and Encode steps written as policy. A named body, a fixed cycle, an effective date.

The Migrate step is where the diagram usually stays abstract. The NAICS revision is not abstract. The Bureau of Labor Statistics published the change as a measured population. Fourteen six-digit 2017 codes convert directly: about 196,000 establishments and 2.4 million employment. Forty-seven are “Direct+” conversions: about 610,000 establishments, 8.1 million employment. Seventy-three codes consolidate into 30 new codes: about 456,000 establishments, 4.1 million employment. Seven split. And 139 six-digit 2017 codes are removed outright — about 1.4 million establishments, 15.5 million employment. BLS introduced the revision on 7 September 2022, with the first-quarter 2022 QCEW release. A model trained on 2017 codes and served against 2022 codes is not facing a data-quality incident. It is facing a scheduled migration. Somebody already counted who it hits.

The audit exists to settle one question: is the field evidence, identity, or an unstable shortcut? Profile the tail — unique values, frequency concentration, churn, first-seen rates. Inspect the semantics. Test generalization on unseen-category and new-entity slices rather than random rows. Probe for memorised outcomes and post-decision routing. Define unknown, rare, retired and redacted behaviour for serving. Then monitor vocabulary growth, bucket collisions, support and drift after release.

Step one is not a formality, because a categorical field can arrive already rewritten by whatever last touched it. Ziemann and his co-authors screened 35,175 supplementary Excel files for a paper in Genome Biology on 23 August 2016. They found 7,467 gene lists attached to 3,597 publications. In 987 files, from 704 articles, they confirmed erroneous gene name conversions: symbols turned by Excel's default settings into dates, or into floating-point numbers. That is 19.6% of the papers. Their abstract puts it as a rate rather than an anecdote: “A programmatic scan of leading genomics journals reveals that approximately one-fifth of papers with supplementary Excel gene lists contain erroneous gene name conversions.”

The follow-up is the part that should change how you schedule the audit. Abeysooriya and Ziemann, with two co-authors, repeated the scan for PLOS Computational Biology on 30 July 2021. The corpus was wider, and the scanner also caught conversion to Excel's internal five-digit date serials. They found gene name errors in 30.9% of articles with supplementary Excel gene lists — 3,436 of 11,117. Five years after the problem was published in a major journal, the measured error rate had gone up, not down. Nobody in that pipeline was profiling the tail.

A serving system can reproduce the same column names while assigning a different category index, hash salt, bucket count, or default state. The resulting vector is structurally valid and semantically wrong. Version the vocabulary and the hash configuration, preserve feature ordering, and test golden records through the offline and online paths.

The production answer is to make the domain itself an artefact the pipeline checks, not a fact hidden in encoder code. Google published such a system at MLSys in 2019: the schema states the expected domain of each feature, and it is versioned alongside the data. An unseen category is then neither a crash nor a silent default. It is a named anomaly — “Unexpected string values” — reported with the share of rows affected. TensorFlow Data Validation's Get started guide shows that report for the categorical feature payment_type, “indicating that an out of domain value was found in the stats in < 1% of the feature values.” The engineer either fixes the producer upstream or accepts the value by appending it to the schema domain. That is a one-line, reviewable change to the contract.

The scale is what makes this more than a linting idea. The system runs inside TFX, and the paper reports hundreds of product teams using it to monitor and validate several petabytes of production data per day. Where privacy or scale prevents retaining raw values, document the irreversible transformation and the operational limits it creates. A hashed coordinate cannot be reported as an unexpected string.

A customer-support model uses product codes, queue names, issue tags, and agent groups. All are categorical. But they behave differently over time, and every one of them is a string some other system owns and can change without telling you.

Genomics shows how literal that is. An approved human gene symbol such as SEPT1 or MARCH1 was converted into a date by a spreadsheet's default settings. The string that reached the analysis was no longer the category anyone had written. Ziemann and his co-authors measured that substitution in 987 files from 704 articles in 2016. The encoder never sees the difference. It sees a new value.

The contract should state who creates values, whether the vocabulary is closed or open, how hierarchy and aliases work, and what happens when codes are retired.

Missing, unknown, other, not applicable, and redacted are distinct states. Collapsing them into one token hides both operational and social differences.

For high-cardinality identifiers, ask whether the field represents a stable entity, a transient session, or an accidental surrogate for the target.

Google Play's Wide & Deep system shows what that contract looks like when it is a shipped pipeline stage. Cheng and his co-authors describe vocabulary generation, in 2016, as its own step: tables mapping category strings to integer IDs, with an explicit minimum-support rule. “The system computes the ID space for all the string features that occurred more than a minimum number of times.” Then: “A 32-dimensional embedding vector is learned for each categorical feature.” Concatenated with the dense features, the result is “approximately 1200 dimensions”. The model trains on over 500 billion examples and warm-starts each retrain from the previous model's embeddings. The vocabulary and the model version are one artefact.

The payoff was measured online, in a three-week A/B test with 1% of users per arm. The model “improved the app acquisition rate on the main landing page of the app store by +3.9% relative to the control group”, against an offline AUC of 0.728 versus 0.726 for the wide-only control. Every one of those 1,200 dimensions began as a string that somebody's minimum-count rule admitted or dropped.

The right choice depends on cardinality, model family, data volume, cold start, and historical correctness. Each option relocates the state and the risk rather than removing them.

One-hot and multi-hot indicators are transparent and work well with many linear models. They can also be very wide, and they need an explicit unknown state. Hashing handles open vocabularies in fixed dimension and needs a stable hash and salt policy. Its collisions are more tolerable than they look: on Reuters RCV1 a 2^18 dimension with almost 40% collisions matched a 2^24 dimension that “causes almost no collisions”, 5.655% error against 5.586%. But a hashed coordinate cannot be read back, or reported as an unexpected string. Target and count encodings are compact, and they demand fold and time boundaries plus smoothing. CatBoost's counter-example is what their absence buys you: a perfectly separable training set and 0.5 test accuracy. Learned embeddings capture interactions and bind the vocabulary to the model version. Wide & Deep learns 32 dimensions per categorical feature, and still has to decide what happens to a string seen fewer than the minimum number of times.