Rule of Thumb: Explaining Artificial Intelligence Systems using Partial Information

AI summary

Overview

• Research area: Explainable AI (XAI), specifically post-hoc local feature importance explanations, applied to large language models, black-box auditing, and AI-driven scientific discovery.
• Technical level: Intermediate. The formulation is expressed with formal notation (additive feature functions, a marginalized loss over feature subsets), but the core idea can be understood without it. Familiarity with SHAP, LIME, and feature importance concepts helps.
• Scope: The paper introduces "Rule of Thumb" (RoT), an XAI formulation that defines a feature's importance as how predictive it is of an AI system's output, and evaluates it across four experiment families plus a regulatory analysis.

What This Paper Is About

Existing feature-importance explainers like SHAP and LIME work by perturbing inputs and measuring how much the model's output changes. This requires the ability to query the model with new, synthetic datapoints, which is expensive or impossible for models reachable only through an API. The paper proposes an alternative: rank features by how much knowing their value should change our prediction of the model's behaviour, rather than by how sensitive the output is to changing them. The goal is an explainer that works without model weights, without extra API calls, and without the assumptions sensitivity analysis relies on.

Key Contributions

1. A new formulation of feature importance. RoT learns additive per-feature functions whose sum estimates the model output, trained so the estimate stays reasonable for any subset of known features. A feature's importance is its learned contribution to that estimate.
2. An efficient fitting procedure. The objective in Equation 2 requires summing over all subsets of features, but the authors show it can be optimized efficiently using dropout applied to feature importances, making training time nearly the same as fitting a simple additive model that ignores subsets.
3. Demonstration across three emerging XAI application areas: zero-shot classification with LLMs, auditing proprietary black-box AI without model access or mimic models, and scientific discovery from AI predictions.
4. Regulatory alignment. The paper argues RoT meets specific requirements of leading AI regulations, including robustness against adversarial attacks and independence from mimic models, and notes that a common mimic-based workaround can be undermined by the Rashomon effect.

Main Findings

• Best alignment with human annotations on legal text: In the judicial case outcome prediction task using a fine-tuned RoBERTa LLM, average weighted AUROC against human segment annotations was 0.77 for RoT, 0.74 for SHAP (default 500 samples), 0.65 for Integrated Gradients, 0.62 for LIME (default 5000 samples), and 0.47 for a random baseline.
• Works where other explainers cannot be applied: For zero-shot movie review sentiment classification via the GPT-4.1-nano API, RoT achieved a weighted AUROC of 0.72 versus 0.50 for a random baseline. The authors state that gradient-based explainers cannot be compared because GPT-4.1-nano weights are restricted, and that perturbation-based explainers (LIME, SHAP) are infeasible even at this modest scale.
• Negligible impact of demographic attributes in resume filtering: Using the GPT-4.1-nano API to filter resumes for an IT role, RoT indicated negligible importance for race, gender, and political orientation, consistent with findings from the original study being replicated. RoT's ability to use non-input features was exercised here.
• Mimic-based audits are contestable: In a replication of The Markup's audit of Amazon's recommendation system, different mimic models with similar accuracies produced different SHAP explanations. Test accuracies were 0.69 (Random Forest, Markup hyperparameters), 0.69 (Random Forest, scikit-learn defaults), 0.71 (LogReg), 0.72 (LogReg, L1), and 0.69 (LogReg, L2). Mimic-based explanations highlighted "brand is amazon" and "product reviews"; RoT highlighted "brand is amazon" and "sold by amazon," and found "product reviews" unimportant.
• Correctly ignores an uninformative feature: On the Pima diabetes dataset transformed with FairPCA so that age is statistically uncorrelated with the diabetes target, RoT treated age as negligible while SHAP and LIME both found it somewhat important.
• Robust to adversarial "fairwashing": Across 10 adversarial experiments, the rate at which LIME or SHAP recovered the sensitive feature as most important did not exceed 5% except once, and in that case adversarial failures still outnumbered successes. RoT's success rate was always greater than 89%, with a perfect 100% recovery rate in 6 of the 10 experiments. The sensitive features were race for recidivism prediction and gender for loan approvals; the misleading "foil" features were simulated extraneous features.
• Computational efficiency: In the judicial case outcome prediction experiment, the first RoT explanation took 1160 seconds and each additional explanation took less than 0.1 milliseconds. The average SHAP explanation took 1320 seconds, which the authors describe as 13 million times longer. Asymptotically, 13 million new RoT explanations can be computed in the time of a single additional SHAP explanation, and even the 7 fastest SHAP explanations took as long as all 101 RoT explanations.
• Survey of XAI in science: The authors analyzed 1151 research articles at the intersection of XAI and science and found that 4.9% used explanations of AI systems to propose novel scientific hypotheses, of which 70% used SHAP.
• Example prediction case: For a Random Forest trained on the Pima dataset, a datapoint with glucose level 99, age 30, BMI 34, etc., yields a predicted diabetes probability of 0.235. RoT and SHAP explain this differently, but glucose is the most important input feature for both; RoT had glucose and age reducing the predicted probability, while SHAP had age increasing it.
• Visualization compatibility: RoT outputs standard feature importance vectors and can be displayed with familiar plots such as force-plots, swarmplots, pixel-level saliency maps, token-level highlights, and importance-weighted word clouds.
• On regulations, the retrieved text ends mid-sentence in Section 4. The available content mentions EU AI Act Article 74(12) (regulators can observe outputs on fixed input data but not obtain new predictions on arbitrary inputs), Article 15, Article 15(4)–(5), Article 13(1) and (3)(b)(ii), the EU Digital Markets Act, and joint EMA/FDA guidance noting transparency, reliability, and accountability as key to acceptable deployment. Conclusions, limitations, and any future-work discussion are not present in the retrieved content.

Methodology in Plain English

The researchers reframe the explainability question. Instead of asking "how much does the output change if I alter this feature?", they ask "how much should my prediction of the model's behaviour change if I learn this feature's value?"

They build a model consisting of one small learned function per input feature plus a global bias term. The sum of these functions approximates the AI system's output. Crucially, they train it so that the sum remains a sensible estimate no matter which subset of features happens to be known — the estimate is never required to use all features. A feature's importance is the value its function contributes for that datapoint.

Because summing over all possible feature subsets is intractable, they use dropout on the feature importances during training as an efficient approximation, so training time stays close to that of a plain additive model. After fitting, the features with the largest absolute contributions for a given datapoint are the most important ones.

They then evaluate this approach in four ways: (1) three zero-shot LLM classification tasks (resume filtering and movie review sentiment via APIs; judicial appeal prediction via a fine-tuned RoBERTa model with accessible weights), (2) a replication of The Markup's Amazon recommendation audit comparing mimic-based SHAP/LIME explanations against RoT computed directly, (3) scientific-discovery scenarios on diabetes, recidivism, and lending data, including a FairPCA-transformed diabetes dataset and adversarially poisoned models, and (4) runtime measurements without GPU acceleration to simulate consumer hardware.

Why This Matters

Research impact. The paper reframes feature importance away from sensitivity analysis toward predictiveness, and shows that the standard "perturb and re-query" paradigm carries assumptions — that the system can be queried with new datapoints, that it responds to synthetic perturbations the way it does to real data, and that outputs vary continuously — which RoT does not make. It also documents how mimic-based auditing can be contested via the Rashomon effect, which has implications for how audits are conducted and evaluated.

Real-world applications:

• Hiring and resume screening with commercial LLM APIs: RoT can explain which parts of a resume drove a selection decision without exposing model weights or paying for hundreds of extra API calls per candidate.
• Regulatory auditing of proprietary platforms: Under arrangements like EU AI Act Article 74(12), where regulators can observe outputs on fixed data but cannot obtain new predictions, RoT can produce explanations where SHAP and LIME cannot. The paper reports a potentially self-preferencing insight ("sold by amazon") missed by mimic-based analysis.
• Legal and judicial decision support: Token-level explanations of appeal outcome predictions aligned better with human annotator judgments than SHAP, LIME, or Integrated Gradients.
• Scientific hypothesis generation: In settings where XAI is used to learn about a phenomenon rather than a model, the paper argues RoT avoids generating spurious hypotheses from physically implausible synthetic datapoints.

Industry relevance. The paper claims RoT is the first explainer practically deployable for LLM feature-importance XAI at scale, given the reported runtime gap of 13 million to one relative to SHAP. It also argues efficiency gains translate into reduced environmental impact when explaining computationally intensive models, and that RoT supports transparency and robustness requirements under the EU AI Act and aligns with EMA/FDA guidance on interpretability in model development.

Future Directions

The retrieved content does not include a conclusion or future work section, so the following are open questions the paper's content naturally raises rather than stated plans:

• Extending beyond the reported task set. Evaluation covered resume filtering, movie review sentiment, judicial appeal outcomes, diabetes prediction, recidivism, lending, and e-commerce ranking. Whether the approach holds on other modalities and domains is not reported.
• Choosing and validating non-input features. RoT can fit on extended datasets and use external models such as MobileNet or BERT to generate non-input features. How to select those features well is not resolved in the retrieved text.
• Formalizing regulatory claims. The paper argues RoT meets specific requirements of leading AI regulations; the retrieved content is cut off mid-sentence in Section 4, so the full argument and any concrete compliance criteria are not available.
• Comparing explainers further. The paper notes that SHAP is a natural comparison basis because RoT explanations can be visualized with the same plots. Whether RoT's predictiveness framing and SHAP's sensitivity framing diverge systematically in other settings is left open.

Target Audience

• XAI and interpretability researchers, who will care about the alternative formulation of feature importance and its theoretical contrast with sensitivity analysis.
• AI auditors, regulators, and policy analysts, particularly those working with models accessible only through APIs or under fixed-input observation constraints.
• Machine learning practitioners deploying LLMs via commercial APIs, for whom explanation cost and API-call volume are practical constraints.
• Scientists using AI for hypothesis generation in medicine, chemistry, engineering, and related fields, who need explanations that do not depend on out-of-distribution behaviour.
• Readers with some familiarity with SHAP, LIME, and feature importance plots will get the most from the experiments and formulation; the paper is less suited to complete beginners than to those with intermediate grounding in XAI.

Authors’ abstract

Explainable Artificial Intelligence (XAI) seeks to explain how an Artificial Intelligence (AI) system arrived at a particular decision. We propose ''Rule of Thumb'' (RoT) explanations, a new approach to XAI based upon a novel formulation that identifies the most relevant features for predicting the behaviour of an AI system, for a particular datapoint. We show how RoT is well-suited to enable XAI in: (a) zero-shot classification using large language models (LLMs), (b) auditing of opaque AI systems without model access, and (c) the use of AI in scientific discovery. Additionally, RoT meets specific requirements from leading AI regulations, provides a familiar interface and visualisations for XAI practitioners, is model-agnostic, and is substantially faster than alternatives. Code available at: https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial-Information