Classical Models as Structured Biases

A classical machine-learning model is not merely an old alternative to a neural network. It is a compact set of assumptions. The assumptions concern how inputs relate to an outcome.

A linear model assumes effects can be summarized through weighted features. A tree assumes useful decisions can be built from recursive partitions. A nearest-neighbor method assumes nearby cases should behave similarly.

That is a testable claim rather than a slogan. The sections that follow test it against studies in which the assumption, not the accuracy, decided which model was deployed. A pneumonia benchmark whose eight methods finished within a point of each other. A proprietary risk score matched by two coefficients. A 179-classifier ranking that did not survive its own significance tests.

The families differ in how they represent structure, where they spend computation, and what they expose to inspection. Those differences have been measured, not merely described.

A 45-dataset tabular benchmark settled part of the question in 2022. Grinsztajn and two colleagues built it and spent 20,000 compute hours of hyperparameter search per learner, and published the result at NeurIPS. Their abstract states it: “Results show that tree-based models remain state-of-the-art on medium-sized data (~10K samples) even without accounting for their superior speed. To understand this gap, we conduct an empirical investigation into the differing inductive biases of tree-based models and neural networks.”

They traced the gap to three inductive biases the neural nets lacked: robustness to uninformative features, preservation of the data's orientation, and the ability to learn irregular functions. Not one of the three is a claim about a method's age. Each is a property of the box a family draws around the space of functions it is willing to express. That box is what the map below maps.

Shwartz-Ziv and Armon, at Intel AI, got to the same place independently. Their paper is called Tabular Data: Deep Learning is Not All You Need. The conclusion held even on the datasets the deep-model authors had chosen themselves, and XGBoost needed far less tuning.

These questions reveal more than the label attached to an algorithm. Notice what is missing from them: accuracy.

In the pneumonia benchmark discussed in the next section, eight statistical and machine-learning methods were compared. Every model's error rate landed within 1% of every other model's, over survival fractions from 0.1 to 0.6. A single aggregate score separated none of them. Shape, data regime, output, operations and evidence separated them completely. One of those five axes decided which model was allowed near a patient.

Fashion rarely drives the choice. Evidence, latency, sample size, and operating constraints often do. Two of the five cases below are documented rather than illustrative, and in both the family was chosen for what it exposed. The pneumonia data was revisited at KDD in 2015, by Caruana and five colleagues. Their paper recounts the asthma rule and the decision not to deploy the neural nets. A GA2M model fitted to the same 14,199-patient dataset rediscovered the identical asthma pattern.

Before tuning, establish whether the family matches the problem.

Step 5 is not only an engineering concern. For a supervised bank, complexity is now a priced quantity. On 17 April 2026 the Federal Reserve, the OCC and the FDIC jointly issued Revised Guidance on Model Risk Management — Fed SR 26-2, OCC Bulletin 2026-13. It supersedes SR 11-7 of 4 April 2011 and SR 21-8. The guidance reads: “A model’s inherent risk reflects several fundamental factors, such as the assumptions made in developing the model, the model’s complexity, the quality of inputs for the model, and data constraints. Inherent risk increases with model complexity and the criticality or number of assumptions necessary, among other factors.”

An added assumption is therefore not free, even when it is correct. It raises the model's inherent risk. The validation, documentation and monitoring that follow from that rating are exactly the operational costs step 5 asks you to put beside the accuracy gain.

A support-vector machine with a nonlinear kernel can be difficult to scale and interpret, while a generalized additive model may be mathematically richer than a shallow neural network used as a baseline.

The axis was tested directly on 7,214 Broward County defendants. Dressel and Farid ran COMPAS against deliberately plain models and published the comparison in Science Advances on 17 January 2018. COMPAS, drawing on 137 features, reached 65.4% overall accuracy. A logistic regression given seven features reached 66.6%. A logistic regression given two features — age and total number of prior convictions — reached 66.8%. A nonlinear SVM handed the same seven features reached 65.2%, below both linear models. Their Results section states it plainly: “Despite using only 7 features as input, a standard linear predictor yields similar results to COMPAS’s predictor with 137 features.”

The kernel bought nothing. The additional features bought nothing. The ceiling sat in the data rather than in the model class. Angelino and four co-authors reached the same conclusion on the same Broward County data by a different route, with certifiably optimal rule lists.

The useful boundary is not old versus new. It is whether the model's assumptions, complexity, and evidence fit the decision being made.