Interpreting Classical Models Without Fooling Yourself

A request to explain a model can mean several things. You might ask why one row received a score, which inputs the fitted predictor relies on, how predictions vary with a feature, or whether the model is stable.

You might instead ask what intervention would change the outcome. These are different questions. A method that answers one can mislead when presented as another.

Start with the one that sounds least ambiguous: which inputs does the model rely on? COMPAS, a commercial recidivism score, is built on 137 features. Two are enough. Dressel and Farid tested it in 2018 on ProPublica's 7,214 Broward County defendants — the ones followed for two years, out of 18,610 people scored in 2013-2014 and 11,757 pretrial cases. Their finding, in one line: “We further show that a simple linear predictor provided with only two features is nearly equivalent to COMPAS with its 137 features.” The two features were age and the total number of previous convictions. On a fixed held-out subset of 1,000 of those defendants, untrained crowdworkers pooled by majority vote reached 67.0% accuracy. COMPAS reached 65.2%. Two answers to “which inputs matter”, 137 features apart, and nothing in the outcome separates them. Reliance is not a property of the phenomenon. It is a property of one fitted model and one measurement.

The popular answer to that question fails in a specific, reproducible way, and a 2021 paper in Statistics and Computing set the failure out. Its title is the finding: unrestricted permutation forces extrapolation, so variable importance requires at least one more model. There is no free variable importance. The setup was ten features, uniform on [0, 1]. Only x1 and x2 were made dependent, through a Gaussian copula. A random forest, a neural network and a linear model were each fitted to 2,000 observations, in each of fifty simulations. With the features independent, the three agreed on the importance ordering. At correlation 0.9 they did not. Permutation importance applied to the random forest and the neural network “rank x1 and x2 as more important than (x3, x4, x5) and frequently more important than even x9, which has a larger coefficient”. The linear model kept the ordering implied by the true coefficients. The cause is mechanical. Permuting one member of a correlated pair asks the model about points “which, by construction, have no data nearby”. The model answers there anyway.

The method should follow the question, not the popularity of a visualization. Five questions get asked in nearly the same words and need different evidence.

Global structure asks what functional form the model fitted. You read it off coefficients and smooth curves, tree partitions and support vectors, and interaction diagnostics — always in the context of the preprocessing. Global reliance asks which inputs affect predictive performance. It comes from permutation or drop-column tests. It is metric- and dataset-dependent, it redistributes credit among correlated features, and it is not a causal effect. Local attribution asks how one prediction is allocated among inputs. It is a case-specific decomposition, it depends on the background and baseline, it can be unstable, and it explains the model output rather than the world. Counterfactual recourse asks what feasible input change could alter the model decision. It needs actionability constraints, it may reveal brittle boundaries, it can conflict with policy, and it guarantees no real-world outcome change. Causal effect asks what would happen under an intervention in the world. It needs identification assumptions and usually an experiment or a causal design, and a predictive explanation does not supply it at all.

The first of those, global structure, is the one people assume must be paid for in accuracy. It has been measured. Angelino, Larus-Stone, Alabi, Seltzer and Rudin ran CORELS, a certifiably optimal rule-list learner, on the 6,907 complete ProPublica COMPAS records over 13 binary features. The structure it fitted was three to four rules — three rules plus a default in 7 of the 10 cross-validation folds. Mean test accuracy across the ten folds was 0.665, SD 0.018. The proprietary COMPAS score, scored on the same ten test sets, reached 0.660, SD 0.019. In their own words: “Our results indicate that it is possible to construct optimal sparse rule lists that are approximately as accurate as the COMPAS proprietary risk prediction tool on data from Broward County, Florida, but that are completely interpretable.” Two independent groups, one with a two-feature linear predictor and one with a three-rule list, put the same ceiling in the same place. Where the trade between structure and accuracy is real, it is a number you can go and measure. It is not an assumption you inherit.

Interpretation inherits choices made before the estimator is fitted. The source process decides who and what generated the raw evidence. The feature pipeline imputes, encodes, scales, aggregates and selects. The fitted estimator supplies coefficients, rules, neighbors, support vectors or kernels. The explanation method perturbs, attributes, draws a curve, a path or a contrast. Then a human draws the conclusion. Every layer can change the answer, and the size of that change has been measured with the data and the question held fixed.

One dataset, one hypothesis: do football referees give more red cards to dark-skin-toned players? Twenty-nine teams of 61 analysts took it on in 2018, and they did not even agree on what to condition on. The odds ratios ran from 0.89 to 2.93, median 1.31. Twenty teams (69%) reported a significant positive effect. Nine (31%) did not.

A later many-analysts study raised the scale and changed the hypothesis: does immigration reduce public support for social policy? It put 161 researchers in 73 teams on one dataset. The result, reported in PNAS: “More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow.” The pipeline layer is not overhead beneath the interesting part. On these numbers it is most of the result.

Classical models are inspectable. Their outputs remain conditional on data and design — and an explanation that reads cleanly can be built to read that way on purpose.

A classifier that decides purely on race can be dressed up to look innocent. Slack, Hilgard, Jia, Singh and Lakkaraju built the wrapper and demonstrated it in 2020: scaffolding designed to detect the synthetic points an explainer generates. On the held-out COMPAS points LIME never ranked race first — 100% of them — and SHAP failed to on 84%. The wrapper still reproduced the biased classifier's predictions on 100% (LIME) and 91% (SHAP) of real COMPAS instances. Their own summary: “Using extensive evaluation with multiple real world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases.” The mechanism is the one the opening already named for permutation importance. Perturbation-based explainers interrogate points off the data distribution, and what a model does out there need not resemble what it does on real cases. Nothing in the resulting chart looks wrong.

The ordinary versions of the same mismatch are quieter.

Refit the pipeline across folds, seeds, bootstrap samples, time windows, and plausible preprocessing choices. Then track sign, rank, curve shape, split, neighbor identity, and local attribution.

Stable predictions with unstable explanations mean the system can act consistently while the story about why remains uncertain.

This has been done to a whole literature. In February 2005 The Lancet published a reanalysis of “data from the seven largest published studies that have attempted to predict prognosis of cancer patients on the basis of DNA microarray analysis”. In place of one training and validation split, Michiels and colleagues used many random ones. Then they reported the model and the explanation separately.

The gene lists moved. “The list of genes identified as predictors of prognosis was highly unstable; molecular signatures strongly depended on the selection of patients in the training sets.” The performance verdict was its own finding: “Five of the seven studies did not classify patients better than chance”. So was the size effect: “For all but one study, the proportion misclassified decreased as the number of patients in the training set increased.” One refitting procedure, three separate measurements. The gene list — the part everyone quoted as the biological explanation — was the least stable of them.

A useful package combines scope, support, stability, and limitations. Six steps, in order.

First, state the explanation question: local prediction, global reliance, model behavior, recourse, or causal effect. The COMPAS comparison and the recourse audit in this lesson answer different ones, and neither substitutes for the other. Second, trace the input — map encoded features back to sources, units, and availability. Third, select complementary methods. Use at least one structural and one perturbation view where both apply, because the two fail in different directions. Fourth, test stability by repeating the explanation across refits and nearby cases, the way multiple random validation splits were used on the gene signatures. Fifth, show support: data density, leaf counts, neighbor distances, extrapolation flags. A reader should be able to see where the model was interrogated in regions with no data nearby. Sixth, write the limitation, naming what the evidence does not establish.

An algorithm can find a small feature change that crosses the decision boundary. The feature may be immutable, dependent on other fields, or manipulable only through an unsafe action. Repeated applications can also lead to gaming.

Recourse design needs feasibility, cost, robustness, and institutional review. In credit it is also an enforceable obligation rather than a courtesy. A creditor using an uninterpretable “black-box” model must still give the applicant the specific principal reasons for an adverse action, under ECOA and Regulation B. The US Consumer Financial Protection Bureau said so in Circular 2022-03, issued on 26 May 2022, and closed the obvious escape route in one sentence: “A creditor's lack of understanding of its own methods is therefore not a cognizable defense against liability for violating ECOA and Regulation B's requirements.” The duty to produce a reason a person can act on does not weaken as the model gets harder to read.

An explanation depends on the model version, preprocessing, background sample, thresholds, and software settings. Changing any of these can alter the result. That is not a general caution. For the most widely deployed attribution family it is a published result.

“The SHAP value” of a feature is not a single quantity. Two independent papers established that at ICML in 2020. Sundararajan and Najmi set out the incompatible operationalisations that travel under the one name — Baseline Shapley, Integrated Gradients, Conditional Expectation Shapley: “There are, however, a multiplicity of ways in which the Shapley value is operationalized for model explanation. These differ in how they reference the model, the training data, and the explanation context. Hence they differ in output, rendering the uniqueness result inapplicable.” The second paper, from four authors at the same conference, showed that the resulting numbers do not answer feature-importance questions without causal assumptions. The background sample is therefore not a setting to be recorded for tidiness. It selects which quantity was computed.

Store explanation configuration and representative artifacts alongside model lineage so audits can reconstruct what users saw.