Beyond Transfer Accuracy: Mechanism-Guided Controlled Adaptation for Low-Resource Languages

AI summary

Overview

Research area: Natural Language Processing / mechanistic interpretability, specifically cross-lingual transfer to low-resource languages.

Technical level: Intermediate. The paper assumes familiarity with transformer internals (attention heads, LayerNorm, residual streams), fine-tuning, and parameter-efficient adaptation, though the core argument is accessible.

Scope: The paper adapts a circuit-discovery method (Contextual Decomposition for Transformers, CD-T) to unstructured natural text, uses the discovered circuits as the sole target of supervised fine-tuning, and evaluates the result on NusaX sentiment transfer, XNLI, Llama-3.2-1B/3B-Instruct, and Qwen2.5-0.5B.

What This Paper Is About

Circuit discovery methods in mechanistic interpretability usually rely on templated tasks with clean counterfactuals, which makes them hard to apply to messy natural text and to low-resource languages where labeled data is scarce and fine-tuning risks catastrophic forgetting. The authors modify CD-T so it works without counterfactuals, then use the discovered source-language circuit as the restricted set of parameters to update when adapting to a new target language (Circuit-Targeted Supervised Fine-Tuning, or CT-SFT). The goal is not to maximize target-language accuracy alone, but to show that a mechanism-guided update location gives a more controlled transfer-and-preservation tradeoff than global fine-tuning.

Key Contributions

1. Counterfactual-free CD-T for natural text. The authors replace counterfactual-based activation means with label-balanced activation means computed from natural source-language examples, and replace CD-T's unsigned magnitude relevance score with a task-directional relevance score that measures whether a head's contribution supports the correct label rather than merely being large.

2. Circuit-Targeted Supervised Fine-Tuning (CT-SFT). A controlled low-resource adaptation strategy that restricts optimization to discovered task-relevant attention heads plus LayerNorm, freezing MLPs, embeddings, the unembedding matrix, and all unselected heads via head-level gradient masking. This updates 2.355% of Llama-1B parameters and 2.410% of Llama-3B parameters.

3. An editing-versus-recruitment analysis. Using matched sparse controls (least-relevant heads, near-zero-score heads, and random non-circuit heads), the paper shows that circuit tuning edits an existing mechanism, while non-circuit sparse updates can also improve transfer by recruiting capacity outside the discovered circuit.

4. Evaluation beyond target accuracy. The paper adds source-language retention, related-task retention (IndoNLU EmoT), less-related retention (SIB200 topic classification), perturbation consistency (label-preserving and label-flipping pairs), and faithfulness/knockout intervention metrics, then extends the scope to Llama-3.2-3B on NusaX and to Llama-1B plus Qwen2.5-0.5B on XNLI.

Main Findings

• Circuit heads are genuinely load-bearing. In the intervention evaluation, the discovered circuit scored much higher faithfulness and stronger knockout than matched sparse controls. On Llama-3B Indonesian, for example, Circuit reached 53.91% knockout versus 4.30% for Random NC. Sparse controls retained non-zero faithfulness (consistent with distributed, self-repairing behavior) but had weaker knockout, indicating they are less task-specific. This held under a zero-ablation sensitivity check as well, where Circuit had the strongest average knockout for both Llama-1B and Llama-3B.

• Circuit tuning is competitive, but not always dominant, on transfer. Averaged across training sizes on NusaX, Circuit reached 54.40% average transfer versus 49.81% zero-shot, 53.20% for Least-relevant, 50.50% for Near-zero, and 53.60% for Random NC. Non-circuit sparse updates can sometimes match or exceed Circuit, which the authors interpret as capacity recruitment rather than mechanism editing.

• CT-SFT most consistently avoids forgetting. Circuit gave the strongest or near-strongest profile across preservation metrics: source retention 69.50% (versus 65.50% for Near-zero and 73.50% pre-adaptation), Emotion 40.20%, Topic 69.30% (versus 58.70% for Near-zero and 51.25% pre-adaptation), label-preserving pairs 64.70%, and label-flipping pairs 79.50%. Random NC achieved the best average source retention (70.30%), which the authors suggest is because it does not directly disturb the original sentiment circuit.

• Non-circuit heads behave differently on perturbations. Near-zero scored highest on label-flipping pairs (81.90% versus 79.50% for Circuit), suggesting weakly tied heads are more plastic when sentiment cues are flipped. But Near-zero was also weakest on transfer, source retention, topic retention, and label-preserving perturbations, so the authors treat it as more responsive rather than a better sentiment mechanism.

• Full fine-tuning drifts severely; LoRA is safer but narrower. Full FT slightly improved average transfer over Circuit (55.30% versus 54.41%) but collapsed Emotion retention to 6.09% and Topic retention to 12.17%. LoRA preserved source retention (72.98%), Emotion (41.07%), and label-flipping consistency (82.54%), but was weaker on target transfer (50.32%), SIB200 Topic (48.51%), and label-preserving pairs (63.04%).

• LayerNorm is auxiliary, not the main driver. LN-only stayed near the pre-adaptation baseline (transfer 50.70% versus 49.81% pre-adaptation), while heads-only already recovered most of CT-SFT's gains (54.22% transfer, 67.48% Topic). The selected heads carry the primary adaptive capacity.

• Findings extend to 3B scale. On Llama-3.2-3B-Instruct, Circuit reached 58.68% transfer with 76.99% source retention and 68.54% Topic retention, while Full FT reached 60.08% transfer but only 1.01% Emotion and 4.99% Topic retention. Near-zero lost 23.2 points and Least-relevant lost 10.4 points on SIB200 Topic retention relative to Circuit.

• Perplexity confirms the drift story. On Indonesian Wikipedia perplexity, Full FT degraded drastically at 1B (135.74 versus 14.16 pre-adaptation), while Circuit (14.10) and LoRA (14.08) remained close to the zero-shot model. At 3B the gap was smaller but still present (Full FT 16.10 versus 9.60 pre-adaptation).

• XNLI supports preservation and intervention, not a full replication. On XNLI, Full FT caused severe English forgetting (Qwen-0.5B: 41.98 English retention and 39.48 transfer, versus 74.38 and 62.56 for the competence-tuned checkpoint), while Circuit and LoRA stayed closer to the starting checkpoint. Circuit had the strongest knockout effects across both model families, but faithfulness was lower than in NusaX, suggesting NLI behavior is more distributed. On Qwen, the average non-circuit control slightly exceeded Circuit on transfer (64.49 versus 63.52), echoing the recruitment pattern.

Methodology in Plain English

The authors start from CD-T, a technique that splits each attention head's activation into a "background" stream and an "input-induced" stream, then traces how much each head contributes to the model's output. The original requires counterfactual examples to estimate the background level, which is impractical for real sentences.

Their first fix: estimate the background from an equal number of examples per class in the source language, so the baseline is not tilted toward one label.

Their second fix: instead of ranking heads by how large their contribution is, rank them by whether that contribution points toward the correct label. At the output, they compare the head's contribution to the correct label against the average over the other labels. At intermediate layers, they build a "task direction" from the unembedding matrix and project the head's contribution onto it. This is related to direct logit attribution, but applied after CD-T decomposition.

They then expand backward from the output, select top-scoring heads at each depth, and prune the lowest-scoring ones to get a compact circuit. For adaptation, they fine-tune only those heads plus LayerNorm, freezing everything else with head-level gradient masking.

The evaluation uses NusaX-Senti for transfer (Indonesian to Acehnese, Buginese, Javanese, Minangkabau), with 400 examples per language for transfer and source retention, 400 examples each for IndoNLU EmoT and SIB200, and 200 label-preserving plus 200 label-flipping pairs. Each target language is fine-tuned with 25, 50, 75, or 100 examples across four seeds. Sparse controls are matched to the same post-pruning head budget per seed, so the comparison isolates where the update happens rather than how much is updated. XNLI subsets use 500 training, 100 validation, and 400 test examples per language, with 250 English examples for competence tuning before circuit discovery.

Why This Matters

Research impact. The paper reframes circuit discovery from a diagnostic tool into an adaptation strategy, and it introduces a clean editing-versus-recruitment framing for interpreting why sparse fine-tuning works. It also extends interpretability research beyond English and beyond templated tasks, which has been an acknowledged gap in the field, and it argues that target accuracy alone is an insufficient evaluation signal for low-resource adaptation.

Real-world applications:

• Deploying NLP for underserved languages. Sentiment and classification tools for Indonesian local languages such as Acehnese, Buginese, Javanese, and Minangkabau, where labeled data is scarce and full fine-tuning is risky.

• Preserving existing capabilities when adding new ones. Systems that must serve a new language without degrading performance on the languages and tasks they already handle, which is the central failure mode the paper targets.

• Safer model customization. Customers or teams that need to adapt a shared base model to a narrow task while limiting collateral behavioral drift, particularly relevant where serving small models on constrained hardware.

• Auditable adaptation. Because updates are confined to a named, knockout-verified set of heads, the adaptation is easier to describe and inspect than a full fine-tune or an opaque adapter.

Industry relevance. The results are directly relevant to anyone fine-tuning small decoder-only models (Llama-3.2-1B/3B, Qwen2.5-0.5B) under limited data and limited compute, and to teams that care about not regressing a model's existing multilingual or multi-task behavior. The finding that Full FT can crater related-task retention (Emotion dropping to 6.09%) while keeping target accuracy high is a concrete warning about standard transfer evaluation practice.

Future Directions

• Reducing dependence on checkpoint competence. CT-SFT requires the model to already have usable source-task behavior. The authors note that when the Llama-1B XNLI checkpoint stayed near chance, sparse adaptation could not build the capability from scratch. Extending the approach to bootstrapping rather than editing is an open problem.

• Broadening architecture and scale coverage. Experiments are limited to Llama-3.2-1B, Llama-3.2-3B, and Qwen2.5-0.5B. The authors flag that encoder-only models and deployment-scale models are untested, and that larger models may show different redundancy and self-repair dynamics under knockout.

• Extending beyond discrete-label tasks. Task-directional relevance scoring is most natural for tasks with discrete label tokens. Applying it to open-ended generation requires defining a suitable task direction over less constrained outputs.

• Finer-grained circuits and richer evaluation. The current circuit granularity is limited to attention heads, potentially missing computation in MLPs, residual interactions, or edges. The authors also call for evaluation instrumentation that is closer to downstream generation than single-forward-pass label-token accuracy, and for broader replication of the NusaX multi-axis preservation suite.

Target Audience

This paper is most useful to mechanistic interpretability researchers interested in moving beyond templated tasks, to NLP practitioners working on low-resource and multilingual adaptation, and to engineers who fine-tune small language models and need to avoid regressions on existing behavior. It will also interest researchers studying catastrophic forgetting and parameter-efficient transfer, since it provides a mechanism-level account of why some sparse updates preserve behavior and others do not.

Authors’ abstract

Existing circuit discovery methods rely on templated tasks with clean counterfactuals, limiting their use on diverse natural text. We adapt Contextual Decomposition for Transformers (CD-T) for unstructured settings via label-balanced activation means and task-directional relevance scoring, enabling counterfactual-free circuit discovery. We leverage the discovered circuits for Circuit-Targeted Supervised Fine-Tuning (CT-SFT), restricting parameter updates to task-relevant heads and LayerNorm. Experiments on NusaX cross-lingual sentiment transfer show that CT-SFT is highly competitive for low-resource adaptation. While non-circuit sparse updates and full fine-tuning sometimes match target accuracy through capacity recruitment, CT-SFT most consistently avoids catastrophic forgetting, preserving source-language and related-task performance. Extensions to XNLI support the source-retention and intervention findings on a harder task and two model families, showing that circuit-targeted adaptation provides a more controlled, intervention-supported alternative to global fine-tuning.