Finding the Translation Switch: Discovering and Exploiting the Task-Initiation Features in LLMs

AI summary

Overview

Research area: Natural Language Processing / mechanistic interpretability of large language models, specifically the internal mechanisms of machine translation.

Technical level: Intermediate. The high-level question (why can LLMs translate without translation training?) is accessible, but the methods (sparse autoencoders, causal interventions, PCA-based consistency scoring) assume familiarity with how transformer hidden states work.

Scope: This paper uses sparse autoencoders and causal interventions on Gemma-2 and LLaMA models to locate a small set of "translation initiation" features, then repurposes those features as a data-selection signal for more efficient fine-tuning.

What This Paper Is About

LLMs can translate well even when they were never explicitly trained on translation, and nobody has a clear explanation of how that ability is implemented inside the model. The authors look inside the model's hidden states rather than at its training data, searching for a small group of internal features that switch the model into "translation mode." They then use that discovery to pick better fine-tuning data, showing that a mechanistic insight can produce both a scientific explanation and a practical training efficiency gain.

Key Contributions

1. A multi-stage framework for finding task-specific features. The method recalls features that fire frequently at translation-relevant prompt positions, characterizes each feature's directional influence on the residual stream, then filters for a functionally coherent group using a PCA-based consistency score.

2. Discovery and causal validation of "translation initiation" features. The authors isolate a small cluster of features in Gemma-2-2B-IT (45 features distilled from 1,004 candidates), show that amplifying them improves translation and reduces hallucination, and show that ablating them causes hallucinations and off-task output.

3. A mechanistic data-selection strategy for fine-tuning. Samples that fail to naturally activate the initiation features are treated as "mechanistically hard," and training preferentially on them improves data efficiency and suppresses hallucination compared with random, high-quality, and high-loss selection baselines.

4. Evidence about transferability. The identified mechanism transfers across model scale within the same family (Gemma-2B to Gemma-9B) but does not transfer across architectures (Gemma to LLaMA).

Main Findings

• Co-activation alone is not enough. Of the candidate features recalled in Stage 1, most showed poor directional alignment, with median individual alignment scores often below 0.4, while a small number of outliers exceeded 0.95. This motivated the consistency filter that reduced 1,004 candidates in Gemma-2-2B-IT to 45 highly consistent features.

• Task-correlated feature density rises with depth. Plotting recalled feature proportion per layer showed that density increases with model depth, and the distributional shape was strikingly similar between Gemma-2-2B-IT (26 layers) and Gemma-2-9B-IT (42 layers), despite the scale difference.

• Causal impact scales with consistency score. On the en-zh test set, ablating high-consistency features in layer 25 increased the hallucination rate by 47.99% and degraded COMET by 8.49%, while low-consistency features had negligible effect.

• Features generalize across languages. Two features discovered only on en-zh data, l12-f2291 and l13-f3517, improved COMET and cut hallucination across en-zh, en-ja, en-ru, and en-ar. For example, the baseline en-zh hallucination rate of 19.15% fell to 10.42% with l12-f2291 and 10.22% with l13-f3517; baseline COMET rose from 73.62 to 77.98 and 77.83 respectively.

• The features act by triggering translation-framing tokens. The model tends to emit a recurring set of tokens that preface translations. Amplifying the feature increased emission of these tokens in all four languages; for Arabic the rate rose from 46.4% to 77.1%.

• Mechanistic selection beats other data-selection heuristics. Fine-tuning Gemma-2-2B-IT on 20k mechanistically selected samples gave COMET 83.37 and a 0.90% hallucination rate, versus 82.49/3.62% for random, 83.32/2.12% for high-quality, and 82.14/4.32% for high-loss, from a 73.62/19.15% baseline. For LLaMA-3.1-1B-IT, mechanistic selection reached COMET 77.92 with a 2.39% hallucination rate, versus 75.88/4.62% (random), 76.35/4.21% (high-quality), and 73.41/6.57% (high-loss), from a 57.61/32.24% baseline.

• Intra-family transfer works, cross-family does not. Using Gemma-2-2B features to select 50k samples for Gemma-2-9B-IT gave COMET 86.48 and a 0.60% hallucination rate, the best among the compared strategies (random 85.36/4.21%, high-quality 84.17/4.97%, high-loss 83.38/8.51%), against a 79.50/12.50% baseline. For LLaMA-3.2-8B-IT, the high-quality baseline performed best (86.69/0.10%), and feature-based selection offered no clear advantage (86.34/0.30%), against a baseline of 80.60/10.00%.

• A selected subset can beat the full dataset. On Gemma-2-2B-IT, the "skyline" COMET from fine-tuning on the full 100k examples was 83.58. With only 50% of the data, mechanistic selection reached 83.68, surpassing the skyline.

Methodology in Plain English

The authors start from a standard interpretability tool: a sparse autoencoder (SAE), a network trained to break a model's dense internal hidden state into a much larger set of sparsely active features. They use Google's publicly released, pre-trained SAEs for the Gemma-2 family, trained on over 4 billion tokens; the Gemma-2-2B-IT SAE expands the hidden state from 2304 dimensions to 16384 feature dimensions, and uses JumpReLU activations for feature stability.

Their search proceeds in three stages. First, they run many translation prompts through the model and record which features are active at three important positions in the prompt: the last token of the source text, the target-language token, and the last token of the query. A feature is kept as a candidate if it appears at any of those positions in more than 60% of samples. Second, for each candidate they compute a "feature influence vector" by forcing that single feature to a high activation and measuring how the SAE's reconstructed output changes — this captures the direction in which the feature pushes the model's internal state. Third, they ask whether the vectors for a group of features point the same way, using a PCA consistency score: the fraction of variance explained by the first principal component. Groups scoring above 0.95 are treated as a single coherent circuit.

To check that the surviving features actually matter, they multiply feature activations by 0 (ablation) or by 2.0 (amplification) during generation and measure changes in COMET and hallucination rate. For the practical application, they rank training examples by how weakly the initiation features activate on them, keep the high-COMET examples among the weakest, and fine-tune on that subset using a learning rate of 1.0e-5, one epoch, FP16 precision, and an effective batch size of 512.

The setup: WMT24++ supplies four translation directions (en-zh, en-ar, en-ru, en-ja) at roughly 1,000 sentence pairs each, split into a 98-pair identification set and roughly 900 held-out test pairs; translation quality is measured with COMET, and hallucination rates are computed with an LLM-as-a-Judge protocol covering irrelevant translation, untranslated content, repetition, and wrong-language output. The fine-tuning experiments draw 100,000 en-zh pairs from WMT24.

Why This Matters

Impact on research. The paper moves the explanation of emergent translation ability away from "there must be parallel data in the pretraining corpus" and toward a specific, testable internal circuit. It also extends sparse-autoencoder interpretability from small synthetic tasks to a complex, real-world capability, and it shows a feedback loop in which an interpretability finding directly changes training practice.

Real-world applications:

• Cheaper multilingual specialization: selecting a fraction of training data that outperforms the full pool for a target language pair.
• Hallucination mitigation for deployed translation systems, since the interventions and the selection strategy both cut unfaithful outputs.
• Inference-time steering for low-resource language pairs, where amplifying a small number of features improves quality without retraining.
• Data-budget planning: teams can use feature activation as an internal difficulty signal to decide which examples are worth annotating or training on.

Industry relevance. The cross-lingual results, the gains on en-zh, en-ja, en-ru and en-ar, and the transfer from Gemma-2B to Gemma-9B suggest a practical recipe for model families: identify the task-initiation features once on a small model, then reuse them to curate data for larger siblings. The finding that this does not transfer across architectures is a caution for teams mixing model families.

Future Directions

• Why the mechanism is family-specific. The paper reports that Gemma-derived features do not help LLaMA-3.2-8B, but does not explain why the circuits differ; identifying what makes them architecture-dependent is open.
• Extending the framework beyond translation. The three-stage recall, influence-vector, and consistency-filtering pipeline is presented as general for task-specific features, but the paper only demonstrates it on translation.
• Alternative interventions. The causal work uses multiplicative scaling (0 and 2.0); other forms of intervention, and their effect on the same features, are not reported.
• Interaction with other hallucination sources. The work does not report whether the initiation features address hallucinations that arise after the model has already committed to translating.

Target Audience

Interpretability researchers studying how LLMs implement specific capabilities, machine translation researchers working on LLM-based translation and hallucination, and ML engineers who build or fine-tune multilingual models and want a data-selection signal grounded in model internals rather than external heuristics. Readers without background in sparse autoencoders will find the intervention results and the fine-tuning tables the most immediately usable parts.

Authors’ abstract

Large Language Models (LLMs) frequently exhibit strong translation abilities, even without task-specific fine-tuning. However, the internal mechanisms governing this innate capability remain largely opaque. To demystify this process, we leverage Sparse Autoencoders (SAEs) and introduce a novel framework for identifying task-specific features. Our method first recalls features that are frequently co-activated on translation inputs and then filters them for functional coherence using a PCA-based consistency metric. This framework successfully isolates a small set of **translation initiation** features. Causal interventions demonstrate that amplifying these features steers the model towards correct translation, while ablating them induces hallucinations and off-task outputs, confirming they represent a core component of the model's innate translation competency. Moving from analysis to application, we leverage this mechanistic insight to propose a new data selection strategy for efficient fine-tuning. Specifically, we prioritize training on **mechanistically hard** samples-those that fail to naturally activate the translation initiation features. Experiments show this approach significantly improves data efficiency and suppresses hallucinations. Furthermore, we find these mechanisms are transferable to larger models of the same family. Our work not only decodes a core component of the translation mechanism in LLMs but also provides a blueprint for using internal model mechanism to create more robust and efficient models. The codes are available at https://github.com/flamewei123/AAAI26-translation-Initiation-Features.