Production Explainability, Documentation, and Decision Evidence

A lending service uses feature attributions for adverse-action review. The reason codes it prints are not a property of the model alone. They are a property of the model together with a reference baseline that somebody chose, and the choice is usually made once and never written down.

Shapley values are attractive because of a uniqueness theorem: one provably right answer. Sundararajan and Najmi showed at ICML 2020 that the theorem does not survive the trip into model explanation. Their abstract says it in three sentences. “There are, however, a multiplicity of ways in which the Shapley value is operationalized for model explanation. These differ in how they reference the model, the training data, and the explanation context. Hence they differ in output, rendering the uniqueness result inapplicable.”

The same point had been made in pictures earlier that year. A Distill article of January 2020 runs integrated gradients on one model and one input against four baselines — constant, uniform, gaussian and blur — and gets a different saliency map from each. Sturmfels and colleagues noted at the time that there was “little existing research about the impact of this baseline”. Two research groups, two methods, one finding. The baseline is part of the answer, and an unrecorded baseline makes the answer unreproducible.

An engineer needs to know whether a feature pipeline changed. A reviewer needs evidence for one decision. An affected user needs a comprehensible reason and a way to challenge it. An auditor needs scope, controls, and traceability.

“Add explainability” is not a requirement. The system must define who is asking, what decision they face, and which evidence can answer it honestly.

What the affected user is owed is no longer a matter of product taste. A mobile contract costing €10 a month was refused on the strength of an automated credit assessment, and the refusal ended up in Luxembourg. On 27 February 2025 the Court of Justice of the European Union decided Case C-203/22, CK v Dun & Bradstreet Austria. Under Article 15(1)(h) GDPR, the Court held, “the controller must describe the procedure and principles actually applied in such a way that the data subject can understand which of his or her personal data have been used, and how they have been used, in the automated decision-making”. Handing over the algorithm does not discharge that duty. The audience is named and the question is fixed. The format that satisfies a data scientist is not the one that satisfies the person refused the contract.

The method must match the question. Collapsing the four is how a system ends up answering an auditor with a global bar chart and an applicant with a coefficient.

Notice where the legal duties actually land. Article 15(1)(h) GDPR, as read in Case C-203/22, asks for the procedure and principles actually applied to one data subject — the fourth question, not the first. Regulation B’s principal reasons are likewise a claim about a single decision, not a description of overall model behaviour. And neither is answered by an attribution method whose output changes with the reference baseline.

Feature attribution describes how a model output changes under a particular method, baseline, and representation. It does not prove that changing the real-world feature would cause the outcome to change.

Correlated features, unstable baselines, preprocessing, and model interactions can alter the explanation. Validate stability and connect the method to the user question.

The objection is neither new nor fringe. Kumar and three colleagues argued at ICML 2020 that the justification for Shapley-value feature importance “rests on two pillars: their desirable mathematical properties, and their applicability to specific motivations for explanations”. They then take both away. They show that “mathematical problems arise when Shapley values are used for feature importance and that the solutions to mitigate these necessarily induce further complexity, such as the need for causal reasoning”. Their conclusion is sharper still. Shapley values “do not provide explanations which suit human-centric goals of explainability”.

Different users require different levels of abstraction and recourse. A system can serve one of them completely while failing another in the same month, on the same decisions.

The New York State Department of Financial Services documented precisely that split. Its nineteen-page report on the Apple Card investigation, published on 23 March 2021, examined Goldman Sachs Bank USA’s underwriting across nearly 400,000 New York applicants. It found no violation of fair lending law. It did find deficiencies in customer service and transparency: “Although the Bank was able to explain, at the request of the Department, the credit decisions for all of the individuals who filed complaints, lack of transparency to the complainants themselves in this case seemed to produce confusion that could have been mitigated.”

Read the two halves of that sentence as two rows of the map below. The regulator asked and received an explanation for every complainant’s decision. The complainants, holding those same decisions, could not obtain one. The evidence existed; the product for the affected user did not.

Specify the user and decision before selecting a method. The five steps are the minimum that would have caught the three failures already in this lesson: a reference baseline nobody versioned, a reason code the regulator could obtain and the complainant could not, and an explanation that raised acceptance without raising accuracy.

A launch document is incomplete if it omits monitoring, fallback, incident ownership, change history, and retirement conditions. Documentation should support the next operator, the next reviewer, and the next user dispute — not only the original model review.

Three regulators have now written that down together. On 13 June 2024 the U.S. FDA, Health Canada and the UK MHRA jointly published four pages of guiding principles on transparency for machine learning-enabled medical devices, building on the ten Good Machine Learning Practice principles the same three issued in 2021. It begins by scoping its own subject: “In this document, "transparency" describes the degree to which appropriate information about a MLMD (including its intended use, development, performance and, when available, logic) is clearly communicated to relevant audiences.”

Two things in that text are worth copying outside medical devices. Transparency is organised as who, why, what, where, when and how — an audience question before a content question. And the information to be communicated includes ongoing updates on model and dataset characteristics, performance monitoring, and change management strategies. Those are properties of a system in service, not of a system at launch.

Review documents after changes and incidents; stale documentation is misleading operational evidence.