Measuring What LLMs Think They Do: SHAP Faithfulness and Deployability on Financial Tabular Classification

AI summary

Overview

Research area: Explainable AI (XAI) for large language models, applied to financial tabular classification — specifically, whether an LLM's stated explanation of a feature's effect matches the effect measured by SHAP.

Technical level: Intermediate (assumes familiarity with classification metrics, tree-based models such as LightGBM, and the general idea of feature attribution).

Scope: The paper evaluates four open-source LLMs zero-shot on three imbalanced financial binary classification datasets, computing SHAP values for the LLMs themselves, comparing those SHAP values against the LLMs' own self-explanations and against LightGBM's SHAP values, and drawing deployability conclusions for regulated finance.

What This Paper Is About

Financial institutions must justify model decisions under governance frameworks such as Basel III and GDPR, yet LLMs are opaque black boxes with billions of parameters. The authors ask a specific question: when an LLM is prompted to say whether a feature has a positive, neutral, or negative impact on a prediction, does that self-reported explanation actually match what the model is doing? They answer it by computing SHAP attributions directly on the LLM inference function and measuring agreement with the LLM's own verbal claims and with a trusted LightGBM baseline.

Key Contributions

1. First SHAP-based feature importance computed on LLMs used as probabilistic classifiers on structured financial tabular data. The authors state this is the first study to compute SHAP-based feature importance on LLMs prompted to predict a probabilistic outcome on structured financial classification datasets.

2. An efficient PermutationExplainer SHAP implementation for LLM inference. Because each SHAP evaluation requires an LLM call, the authors use a model-agnostic PermutationExplainer with a k-means background, a max_evals budget of 200, and T = 4 permutations — a roughly 5x reduction in model calls per instance relative to KernelExplainer (about 440 versus about 2205 evaluations per instance at M = 21, C = 5).

3. A systematic faithfulness comparison across three axes: (a) LLM self-explanation versus LLM SHAP values, (b) LLM SHAP values versus LightGBM SHAP values, and (c) the effect of adding a rationale request to the prompt.

4. Deployability guidance for risk-sensitive finance, organized around validity, explainability, calibration, robustness, governance, and cost, with explicit green-light (pilot-only) and red-flag (no deployment without mitigation) categories.

Main Findings

• Zero-shot LLM classification is modest but above chance. On bankruptcy (3.9% positive rate), ROC-AUC values are 0.641 (Gemma-2-9B), 0.524 (Llama-3.2-3B), 0.630 (Qwen-2.5-7B), and 0.624 (Mistral-7B-v0.3); PR-AUC values are 0.059, 0.041, 0.054, and 0.060. Averages across the three datasets are ROC-AUC 0.637 / PR-AUC 0.322 for Gemma-2-9B, 0.526 / 0.304 for Llama-3.2-3B, 0.551 / 0.304 for Qwen-2.5-7B, and 0.614 / 0.320 for Mistral-7B-v0.3.

• PR-AUC lift over baseline is weak. Bankruptcy PR-AUCs correspond to lifts of 1.51x, 1.05x, 1.39x, and 1.54x (average 1.37x) over the 3.9% baseline. Loan repayment lifts are 1.09x, 1.06x, 1.05x, and 1.09x (average 1.07x) against the 80.3% baseline. License expiration lifts are 1.38x, 0.91x, 0.86x, and 1.24x (average 1.10x) over the 2.1% baseline — one model falls below baseline.

• LLMs do not correctly identify their own feature impacts. Agreement between LLM self-explanations and LLM SHAP values sits only slightly above the 33% three-class baseline. Best average is Gemma-2-9B at 57.2% with a rationale prompt and 50.0% without; Qwen-2.5-7B is lowest at 29.4% with rationale and 25.0% without.

• Asking for a rationale helps only modestly and not uniformly. Gemma-2-9B improves from 50.0% to 57.2% average agreement when asked to explain, and Mistral-7B-v0.3 improves from 28.9% to 37.2%, but Llama-3.2-3B moves only from 45.6% to 47.2% and Qwen-2.5-7B from 25.0% to 29.4%.

• LLM SHAP values disagree with LightGBM SHAP values. Kendall's tau on full feature ordering averages 0.042 (bankruptcy), 0.148 (loan), and -0.186 (license expiration). Sign agreement (Dir%) averages 55.0%, 60.4%, and 58.3% respectively — only slightly above random.

• Disagreements persist even for the most important features. Figure 3 shows that for the top three most important features, there are still many cases where the LLM predicts the incorrect impact direction.

• SHAP dependence plots show outright sign inversions. For the highest-importance feature on the bankruptcy dataset, both Qwen-2.5-7B and Mistral-7B-v0.3 label the feature impact "positive" while SHAP values are strongly negatively correlated with the feature values.

• The proposed explanation is lexical priors in feature names. The authors attribute the disagreement to LLMs keying on tokens such as "cash," "profit," or "liabilities" rather than dataset-specific conditional relationships.

• Debiasing a feature also reinterprets a feature: the paper's own caption for Figure 3 carries a caveat. (The paper reports this as an observation rather than a result.)

• Explainability is computationally expensive at audit scale. Using K = 250 rows per dataset, T = 4, and B = 5 produces roughly 110k model calls per dataset-model pair, and roughly 1.32 million evaluations across three datasets and four LLMs.

Methodology in Plain English

The authors take three financial binary classification tasks and treat each as a simple prediction problem: given a row of features, estimate the probability of the positive class.

Data. Bankruptcy uses the Polish Companies Bankruptcy dataset (1-year horizon, 7,027 companies, 271 bankrupt, 3.9% positives), reduced to the top 20 of 64 features by LightGBM importance, all numeric. Loan repayment uses a Kaggle dataset of 79,206 applications with 63,629 fully paid (80.3% positive), 21 features of which 12 are numeric. License expiration uses a Hong Kong SFC dataset restricted to January 2024, with 23,001 employees and 478 non-renewals (2.1% positives), 15 numeric features.

Models. Four open-source instruct LLMs — gemma-2-9b-instruct, llama-3.2-3b-instruct, qwen-2.5-7b-instruct, and mistral-7b-instruct-v0.3 — run locally through vLLM on two NVIDIA A10G 24GB GPUs. Each is prompted with the feature names and values and asked to return a probability in JSON, without performing calculations.

Measuring what the model actually does. Rather than explain the LLM by reading its weights (impossible), the authors treat the entire LLM inference as a black-box function and run SHAP on it. They use the model-agnostic PermutationExplainer, sample 250 instances per dataset, build a background distribution from k-means clustering with C = 5 centroids, and cap max_evals at 200, which works out to T = 4 permutations.

Measuring what the model says it does. For each feature, the LLM is asked to label the feature's impact as positive, neutral, or negative — optionally with a written rationale for why.

Comparing the two. For every feature, SHAP values are correlated with feature values using the Pearson correlation coefficient, and the result is binned: below -0.1 = negative, between -0.1 and 0.1 = neutral, above 0.1 = positive. This binned SHAP label is then compared against the LLM's self-reported label. Separately, the LLM's SHAP values are compared against LightGBM's SHAP values using Kendall's tau and the percentage of features with matching sign.

Why This Matters

Impact on research. The paper provides a concrete, reproducible recipe for auditing whether LLM rationales are faithful, and it demonstrates that plausible-sounding explanations can be decoupled from actual model behavior. It also supplies a practical efficiency argument for PermutationExplainer over KernelExplainer when the prediction function is an expensive LLM call.

Real-world applications:

• Credit and lending decisions — where the loan repayment task mirrors real underwriting and where LLM explanations cannot currently be trusted for adverse-action justifications.
• Bankruptcy and counterparty risk monitoring — one of the three datasets, with weak but detectable signal.
• Regulatory and licensing oversight — the license expiration task models renewal decisions about individuals, where the paper notes the risk of unfair or discriminatory outcomes.
• AML and HR screening — cited in the paper's red-flag list as applications with regulatory or legal consequences that should not deploy without mitigation.

Industry relevance. The authors include a researcher from the Abu Dhabi Investment Authority, and the deployability framing is written for practitioners under Basel III and GDPR. The paper's bottom line for industry is that zero-shot LLMs should be treated as fallback options in small-data settings where fine-tuning is infeasible, and should not be trusted without rigorous auditing.

Future Directions

1. Few-shot and many-shot benchmarking. The authors state that to take advantage of the potential signal shown in Table 1, few-shot performance will need to be assessed, and that future work should benchmark few-shot and many-shot LLMs on structured and unstructured data.

2. Mitigation without retraining. Feature-name neutralization (anonymizing features as f1, …, fM in prompts, then mapping back for human consumption), serialization robustness testing across feature order and delimiters, and reporting beyond-chance agreement measures such as Cohen's kappa or Matthews correlation between SHAP signs and LLM labels.

3. Calibration and decision-theoretic rigor. Reporting reliability diagrams and Brier scores, providing 95% confidence intervals for AUC and PR-AUC, and translating metrics into cost-sensitive operating points, optionally with Platt scaling or isotonic regression trained on validation sets.

4. Hybridization with tabular models. Model augmentation, ensembling, and LLM + LightGBM hybrids, plus domain-specific fine-tuning, are named as the likely prerequisites before LLMs are viable in high-stakes finance.

5. Ablations on the explanation pipeline itself. The authors note that a small ablation varying the SHAP background distribution and the max_evals parameter would strengthen their interpretability claims, and that explanation outputs must be independently audited through sanity checks or falsification tests.

Target Audience

Risk modelers, model validation and governance teams at financial institutions, and applied machine learning researchers working on tabular data or LLM interpretability. It is also relevant to regulators and auditors who need to know what kinds of evidence would be required before accepting an LLM-based classifier's explanations. Readers should already be comfortable with classification metrics under class imbalance and with the general mechanics of feature attribution methods.

Authors’ abstract

Large Language Models (LLMs) have attracted significant attention for classification tasks, offering a flexible alternative to trusted classical machine learning models like LightGBM through zero-shot prompting. However, their reliability for structured tabular data remains unclear, particularly in high stakes applications like financial risk assessment. Our study systematically evaluates LLMs and generates their SHAP values on financial classification tasks. Our analysis shows a divergence between LLMs self-explanation of feature impact and their SHAP values, as well as notable differences between LLMs and LightGBM SHAP values. These findings highlight the limitations of LLMs as standalone classifiers for structured financial modeling, but also instill optimism that improved explainability mechanisms coupled with few-shot prompting will make LLMs usable in risk-sensitive domains.