Compact Example-Based Explanations for Language Models

AI summary

Overview

• Research area: Natural Language Processing — specifically interpretability and example-based explanation of language model predictions, and training data influence estimation.
• Technical level: Advanced (familiarity with gradients, LoRA fine-tuning, and influence functions is assumed by the paper, though the core idea is explainable in plain terms).
• Scope: The paper proposes and validates a retraining-free metric — the selection relevance score — for judging how useful a chosen set of training examples is when used as an explanation for a language model's output, and uses it to benchmark influence estimators and selection strategies.

What This Paper Is About

Methods that estimate training data influence can rank every training document by how much it contributed to a model's output, but no human or retrieval-augmented generation system can digest thousands of documents. Systems therefore show only a small subset, yet prior evaluations of example-based explanations focused on the correctness of the raw influence estimates or on user studies, largely ignoring how examples are chosen. This paper isolates the selection step, proposes a score that measures how well a selected set of examples explains a specific test instance, and shows that a good selection strategy matters as much as the underlying influence estimator.

Key Contributions

1. The selection relevance score (ξ^SR): a retraining-free metric that quantifies how well the gradients of a selected set of training examples can reconstruct the loss gradient of the test instance they are meant to explain, evaluating examples in combination rather than in isolation and remaining agnostic to task and estimation method.
2. A benchmark of 3 influence estimation methods and 4 selection strategies, showing that common selection strategies — in particular naively taking the highest-ranked examples — often underperform random selection.
3. A new coverage-aware selection strategy based on facility location that balances influence and representativeness, and that uses small selection budgets better than naively picking the top-ranked examples.
4. A validation study correlating the selection relevance score against two fine-tuning-based notions of relevance (prediction support and prediction shift), establishing that the score predicts whether training on a selected set will support or undermine the original prediction. Code is released at doi.org/10.5281/zenodo.19483839.

Main Findings

• Gradient-based selections rarely clear the baseline. At a budget of k = 10, every selection derived from the gradient-based influence estimates has an average selection relevance below 0 dB, meaning it fails to convey relevant information relative to a trivial approximation. Only strategies selecting the least influential examples beat the random baseline (for example, -0.14 dB for the least-influential DataInf and LESS selections versus -2.57 dB for random, while the most-influential DataInf selection scores -22.87 dB).
• BM25 is the exception. Selections based on BM25 rankings achieve scores above 0 dB only for the most influential strategies and for AIDE (for example, 22.24 dB for most influential BM25 and 22.08 dB for AIDE with BM25); random, DIVINE and least-influential selections do not.
• Coverage-aware re-ranking helps. Across budgets of k = {1, 5, 10, 25}, the facility location strategy improves selection relevance on average for the two gradient-based estimators for all selection methods except DIVINE at k = 1. For BM25, selection relevance decreases when λ = 1, as expected, since selection becomes purely coverage-based in gradient space rather than token overlap.
• Least-influential examples are the strongest strategy. Aggregating results over k = {1, 5, 10, 25} as an AUC score, selecting the least influential examples remains the most effective strategy. The authors explain this by noting these examples have the smallest absolute influence scores and the weakest impact on model parameters, so removing them is least likely to push the model away from its original prediction.
• Redundancy is visible in the case study. For one DataInf/Olmo2 test instance with k = 5 and the 100 most influential documents, naive selection and AIDE include examples with highly similar prompts; only two of these redundant examples appear at λ = 0.25, only one at λ = {0.5, 0.75}, and the purely coverage-based λ = 1 retains none of the naively selected examples. DIVINE prioritizes diversity more aggressively than AIDE in this setup, and for this instance AIDE produces the same selection as the naive strategy.
• The score predicts fine-tuning effects only above 0 dB. Correlations over the full score range are negligible (ρ = 0.09 with prediction support, ρ = 0.07 with prediction shift). Restricted to selections with ξ^SR > 0 dB, correlations are substantially stronger (0.58 and 0.37 respectively), and a simple rule-based estimator — predict fine-tuning success when ξ^SR > 0 dB — is confirmed to be indicative of fine-tuning success.
• The validation parameters behave sensibly. In 98.49% of cases, the log-likelihood of a test instance increases more when fine-tuning on the instance itself than when fine-tuning on a random example; in 96.28% of cases, fine-tuning on the instance produces a larger Jensen-Shannon divergence in the model's full predicted distribution.
• Re-ranking does not sacrifice prediction constraints. Facility location selection significantly increases correlation for LESS at λ = {0.25, 0.5, 0.75, 1.0} and significantly decreases it only for the purely coverage-based BM25 λ = {1.0}; all other settings change insignificantly.
• Influence and similarity are not aligned. The ineffectiveness of selecting the most helpful examples (those with the most negative influence scores) shows that data influence and similarity-based relevance (BM25) diverge, because a reduction in loss can also come from training on contrastive or dissimilar examples.

Methodology in Plain English

The authors reframe example selection as a reconstruction problem. Given a test instance the model has already answered, they take the loss gradient of that instance and ask: can this gradient be rebuilt as a weighted combination of the gradients of the selected training examples? The better the reconstruction, the more relevant the selection. They formalize this as the ratio of the expected squared gradient norm to the expected squared reconstruction error, and report it in decibels (10 log₁₀ ξ^SR) for readability. A score below 0 dB means the selection does worse than simply predicting zero; above 0 dB suggests non-redundant information.

The combination weights are constrained to be non-negative and to sum to one, so no example can be cancelled out by another and the weights read as relative importance within the selection. This is achieved by computing an unconstrained least-squares solution and projecting it onto the unit simplex.

To check that this notion of relevance means something, the authors compare it against a second, training-based notion. For each test instance, model and estimator pairing, they fine-tune for one step (learning rate 1e-5, one batch) on the selected documents and record two things: whether the likelihood of the originally generated output rises more than it does after training on a random subset of the same size (prediction support), and whether the full predicted distribution shifts more than under the random subset, measured with Jensen-Shannon divergence (prediction shift). Because neither notion is a ground truth, the two are compared by correlation.

Experiments use three estimators — DataInf (which approximates influence functions), LESS (gradient similarity) and a BM25 baseline that retrieves examples by token overlap — with the gradient-based methods restricted to LoRA layers. Selection strategies include the four naive logics (most helpful, most harmful, most influential, least influential), five random selections averaged per k, a reimplementation of DIVINE, an AIDE variant with classification-only elements removed, and the new facility location strategy. That strategy greedily selects the example with the highest marginal gain among the top-100 examples in the naive ranking, where the gain combines coverage with a normalized cost derived from influence: (Δ(j|S)+1)^λ / c_j^(1-λ). Setting λ = 1 is pure coverage, λ = 0 reduces to naive influence-based selection.

Models come from three families — Olmo2 (allenai/OLMO-2-0425-1B), Llama 3.2 (meta-llama/Llama-3.2-1B) and Qwen 2.5 (Qwen/Qwen2.5-0.5B) — each fine-tuned for one epoch with LoRA on the full Tülu3 instruction fine-tuning dataset. For attribution, the authors randomly sample 10% (86.6k) of examples as the set to attribute to, plus a disjoint set of 1,000 test instances to explain, and evaluate selection budgets of k = {1, 5, 10, 25}.

Why This Matters

Impact on research. Evaluations of example-based explanations have often measured either the correctness of raw influence estimates or the end-to-end usefulness of a system, both of which can hide flaws in the selection logic — a system can appear helpful even when the examples it surfaces are not informative. This work separates the selection step and gives it a retraining-free score, showing that the selection strategy can substantially change explanation quality and should be reported alongside estimator correctness. It also argues that faithfulness evaluations alone do not guarantee that selected examples are sufficiently informative from a user's perspective.

Real-world applications:

• Building example-based explanation interfaces for deployed language models, where only a handful of training examples can be shown to a user.
• Designing retrieval-augmented generation systems that must choose which retrieved documents to present or feed forward under a fixed budget.
• Auditing and debugging training data — for example, understanding which documents support a specific model behavior or prediction.
• Data-centric model development, where teams need to decide which examples to surface to annotators, domain experts or stakeholders rather than inspecting entire corpora.

Industry relevance. The finding that naively surfacing the highest-influence documents often underperforms random selection is directly actionable for teams building explanation dashboards, data curation tools or retrieval components. The proposed score is retraining-free and task- and estimation-method agnostic, which makes it practical for pipelines that cannot afford leave-one-out retraining of large models. The authors' caution is equally relevant: relevance is a necessary but not sufficient condition for faithfulness, so the score should not be the sole evaluation metric shipped with a product.

Future Directions

• Investigate why gradient-based selections underperform. The paper proposes increasing relevance but explicitly leaves the cause of the overall low performance of gradient-based selections unexamined. One possibility raised is that instruction fine-tuning data has limited feature redundancy, leaving few truly influential examples to retrieve; another is that relevant examples had little or no influence during training, for example through saturation effects.
• Evaluate explanation faithfulness directly at human-interpretable budgets. The separation between selection relevance and faithfulness means the low relevance scores for gradient-based estimators cannot currently be attributed to low faithfulness versus other factors; dedicated faithfulness evaluations are needed.
• Extend beyond prediction-constrained influence. The authors suggest adapting the framework to other paradigms, such as gradient-tracing methods that leverage multiple model checkpoints.
• Explore alternative definitions of a good explanation. The current setup assumes explanations that support the prediction; future work could target selections that oppose the prediction (counterfactual explanations), for which examples with large absolute influence scores are likely more useful.
• Scale up. Experiments were limited to models in the 0.5–1B parameter range with gradients restricted to LoRA layers for computational feasibility; the scoring framework itself is described as general and applicable to larger model gradients given sufficient resources.

Target Audience

Researchers and practitioners in NLP interpretability and explainable AI who work with training data attribution, influence functions or example-based explanations; machine learning engineers building explanation, retrieval or data curation tooling for language models; and evaluation-focused researchers who need retraining-free metrics for assessing example selection. The paper assumes comfort with gradient-based reasoning and LoRA fine-tuning, so readers without that background will find the conceptual framing accessible but the experimental details dense.

Authors’ abstract

Training data influence estimation methods quantify the contribution of training documents to a model's output, making them a promising source of information for example-based explanations. As humans cannot interpret thousands of documents, only a small subset of the training data can be presented as an explanation. Although the choice of which documents to include directly affects explanation quality, previous evaluations of such systems have largely ignored any selection strategies. To address this, we propose a novel selection relevance score, a retraining-free metric that quantifies how useful a set of examples is for explaining a model's output. We validate this score through fine-tuning experiments, confirming that it can predict whether a set of examples supports or undermines the model's predictions. Using this metric, we further show that common selection strategies often underperform random selection. Motivated by this finding, we propose a strategy that balances influence and representativeness, enabling better use of selection budgets than naively selecting the highest-ranking examples.