From Reweighting to Rewriting: Unlocking the Intervention Effects of Influential Samples in Training Data Attribution

AI summary

Overview

Research area: Training data attribution (TDA) and data-centric intervention for large language models — specifically, whether influence-function (IF) scores identify training examples that are actually useful intervention targets during supervised fine-tuning (SFT).

Technical level: Intermediate. The paper uses influence functions and EK-FAC curvature approximations, but its central claims rest on controlled retraining experiments that can be understood without following the derivation.

Scope (one sentence): A controlled comparison of weight-based reweighting versus response rewriting applied to the same influence-selected SFT examples, evaluated across four open-weight LLMs primarily on epistemic abstention and extended to safety refusal on OLMo2-7B.

What This Paper Is About

Prior work found that training examples selected by influence functions often give little advantage over random selection when they are upweighted or deleted during LLM training. The authors argue this negative result conflates two separate questions: whether influence functions pick the right examples, and whether reweighting is a good way to intervene on them. Their goal is to test whether the same influence-selected examples become effective intervention targets when their responses are rewritten — encouraging or discouraging a target behavior — while their instructions stay fixed.

Key Contributions

1. Influence-guided response rewriting. The authors introduce a supervision-level intervention framework for SFT that separates example selection from intervention design: influence functions determine where to intervene, while response rewriting determines what behavioral signal those examples provide.

2. Revealing hidden behavioral leverage of influential examples. Across four open-weight LLMs, they show that response rewriting of influence-selected examples produces stronger, more stable, and bidirectional behavioral shifts than conventional reweighting or matched controls, primarily on abstention and with similar trends for safety refusal.

3. Characterizing the source and scope of intervention leverage. Through controlled analyses — projection onto an internal unanswerability direction, comparison against projection-based and TRAK selection, fixed-reference and Bayesian influence diagnostics, and scenario-wise evaluation — they argue that rewriting redirects the local supervision signal of influential examples while keeping effects concentrated on target-relevant behaviors and without substantial capability degradation.

Main Findings

• Reweighting does not reliably follow influence predictions. Both upweighting (α = 2) and deletion (α = 0) produced unstable effects that failed to outperform the baseline and sometimes moved in the opposite direction. Varying the reweighting coefficient α did not recover a consistent dose–response pattern or the expected bidirectional behavior.

• Response rewriting produces substantially stronger and more consistent effects. Behavior-aligned rewriting consistently increased abstention recall, while behavior-opposed rewriting decreased it. These effects were larger and more persistent than rewriting randomly selected examples.

• The two ends of the influence ranking show different dynamics. Rewriting supposedly helpful examples often induced a large early shift that gradually decayed, whereas rewriting supposedly harmful examples could emerge more gradually and continue growing at later checkpoints. This pattern was not universal: for Gemma3-4B, harmful-example rewriting did not produce the largest final shift.

• Influential examples are behaviorally related to unanswerability. Both ends of the influence ranking had substantially higher projections onto a linear internal unanswerability direction than the overall training distribution, but were not the most extreme examples.

• Influence selection beats representation-based selection under aligned rewriting. Selecting examples with the highest unanswerability projections was comparable to influence-guided selection under behavior-opposed rewriting, but fell substantially short under behavior-aligned rewriting — plausibly because many high-projection examples already carry abstention-consistent supervision. TRAK and other gradient-based selectors also yielded smaller and less sustained effects than influence-based selection.

• Rewriting changes the predicted influence of the same examples. With the prompt held fixed and curvature fixed at the original checkpoint, aligned rewriting shifted influence scores positively; originally harmful samples reversed sign, and helpful samples became more aligned with the target-improving direction. A symmetric checkpoint-local check using Bayesian influence functions showed aligned responses consistently scoring above their original counterparts throughout SFT.

• Behavioral changes remain concentrated on target scenarios. The largest abstention gains occurred on answer unknown and false premise, the two scenarios used for attribution, with substantially smaller effects on subjective and underspecified context. Random-aligned rewriting produced relatively broader gains on non-target scenarios. No substantial evidence of systematic degradation was found on precision, accuracy, or other general capabilities.

• The contrast transfers to safety refusal on OLMo2-7B. Aligned rewriting strengthened refusal on multiple safety benchmarks while opposed rewriting degraded it. For example, on WJB-Harmful the baseline was 0.792, aligned rewriting on helpful-ranked examples reached 0.880, and opposed rewriting on helpful-ranked examples fell to 0.334. Deletion and upweighting remained less systematic, occasionally producing opposite effects — upweighting supposedly harmful samples led to better performance.

• Safety gains carry an over-refusal cost. Aligned rewriting produced a noticeable drop on XSTest (baseline 0.516; aligned rewriting on harmful-ranked examples 0.368; on helpful-ranked examples 0.252; random 0.504), a trade-off the authors attribute to the broad, aggregate nature of their safety-attribution target set.

Methodology in Plain English

The authors pick a target behavior — here, epistemic abstention, where a model should decline to answer questions that cannot be reliably resolved. They build a small held-out set of 300 abstention queries (drawn mainly from answer unknown and false premise scenarios, following the AbstentionBench taxonomy) and define a differentiable score: the model's average log-likelihood of the abstention responses on those queries.

They then use influence functions, approximated with EK-FAC (via the Kronfluence package), to rank every SFT training example by how much it locally affects that score. The top-ranked examples are called "supposedly helpful," the bottom-ranked "supposedly harmful" — labels that describe only the predicted local effect of their original supervision.

The key move is to hold selection fixed and vary only the intervention. The first family changes how much the selected examples count: upweighting (α = 2) or deletion (α = 0). The second family keeps each selected example's instruction but replaces its response with either a behavior-aligned abstention response (drawn from a diverse pool of semantically equivalent templates, to avoid dependence on a single refusal phrase) or a behavior-opposed response that encourages answering. The same procedures are applied to matched random sets as controls.

Everything is evaluated by retraining from the same base model with the same configuration and a fixed data order, which removes reshuffling as an explanation. Rather than reporting only final checkpoints, they track the change in abstention recall relative to the unmodified baseline at every training step, smoothing with a moving average. Interventions were applied to 2.5% of the SFT data by default. The four models are OLMo2-1B, Qwen3.5-2B, Gemma3-4B, and OLMo2-7B. The safety-refusal extension uses OLMo2-7B with safety-specific aligned and opposed response templates.

Why This Matters

Impact on research. The paper argues that TDA methods should be evaluated not only by whether their rankings predict infinitesimal reweighting effects, but by whether the examples they identify have actionable leverage under realistic interventions. This reframes a negative result about influence functions — that IF-selected examples beat random under reweighting only weakly — as a limitation of the intervention rather than of the selection.

Real-world applications (implied by the paper's settings):

• Curation of abstention-aware instruction data, so that models decline to answer unanswerable queries instead of guessing.
• Safety-refusal tuning, where specific training examples are rewritten to strengthen refusal behavior.
• Auditing and steering datasets by locating the individual training examples that drive a target behavior, rather than retraining blind.
• Evaluating data-attribution tooling by whether it supports effective interventions, not just retrospective attributions.

Industry relevance. Teams that fine-tune models on large instruction datasets often need to change a specific behavior without degrading others. This work suggests a practical route: use attribution to find a small number of examples — 2.5% of the SFT data here — and rewrite their responses rather than reweighting or deleting them. The reported over-refusal cost on XSTest also gives a concrete caution: broad target sets can push a desirable behavior too far.

Future Directions

• Extending beyond behaviors with clear rewriting targets. The authors state their framework is most natural for behaviors with clear behavior-aligned or behavior-opposed targets, such as abstention and safety refusal, and that extending it beyond such settings remains future work.

• More granular safety targeting. Because aligned rewriting caused a substantial over-refusal risk on benign prompts (the XSTest drop), the authors state that more granular refinement is needed before the approach is fully suited to safety refusal.

• Explaining model-dependent dynamics. Rewriting supposedly helpful vs. harmful examples produced different trajectory shapes, and Gemma3-4B did not follow the general pattern. The authors suggest pretrained data mixtures may be responsible and investigate cross-model ranking overlap and ranking transfer, but the underlying causes remain open.

• Intervention-aware evaluation of TDA methods. The paper motivates evaluating attribution methods by the leverage of the examples they select under different interventions, which raises the question of which interventions should become the standard testbed.

Target Audience

Researchers and practitioners working on training data attribution, data-centric fine-tuning, and LLM behavior steering — especially those who have tried influence-based data selection and found weak effects under upweighting or filtering. It is also useful for teams designing abstention or safety-refusal tuning pipelines, and for anyone evaluating whether attribution scores translate into real changes in model behavior. Readers seeking a purely theoretical treatment of influence functions will find the contribution empirical rather than mathematical.

Authors’ abstract

Training data attribution (TDA) aims to identify training examples that shape model behavior, but its intervention value depends on both which examples are selected and how they are modified. Influence functions (IF) estimate behavioral changes under infinitesimal reweighting, yet IF-selected examples often show limited advantages over random selection under conventional weight-based interventions. This raises the question of whether influential examples lack intervention value or whether reweighting fails to realize their behavioral leverage.We introduce influence-guided response rewriting, which uses IF to identify intervention targets and replaces their responses with behavior-aligned or behavior-opposed supervision while keeping instructions fixed. Across four open-weight LLMs, we compare rewriting and reweighting on the same influence-selected examples using epistemic abstention as our primary testbed. Response rewriting produces stronger, more persistent, and bidirectional behavioral shifts, while reweighting the same examples yields weak and inconsistent effects. Further analyses show that influence-selected examples provide greater rewriting leverage than alternative selectors, with changes remaining concentrated on target-relevant behaviors. The same qualitative contrast extends to safety refusal. These results distinguish the local reweighting effects captured by influence estimates from the broader intervention leverage of the examples they identify, motivating intervention-aware evaluation of TDA methods.