Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs

AI summary

Overview

Research area: Mechanistic interpretability and multilingual natural language processing, specifically the internal computation of machine translation in multilingual large language models (mLLMs).

Technical level: Intermediate. The paper uses established interpretability tooling (LogitLens, activation patching) but assumes the reader is comfortable with transformer internals such as layers, attention heads, and next-token probability distributions.

Scope: The paper argues, using causal interventions and probing across three multilingual models and three controlled datasets, that machine translation in mLLMs resolves target-side word order (syntax) before target surface language, and that this syntactic commitment is localized to specific layers and, in most models, to individual attention heads that are largely language-invariant.

What This Paper Is About

Prior work suggested that translation in multilingual models separates conceptual content from output language, but it left open how models handle grammatical structure when source and target languages order words differently. This paper asks whether commitment to a grammatical structure (specifically word order) is a distinct, independently trackable stage of translation, and where in the network it happens. The authors use controlled multilingual datasets isolating word-order contrasts, together with activation patching and LogitLens probing, to track how word order, surface language, and lexical content emerge layer by layer.

Key Contributions

1. Commitment to syntactic structure is an independent translation stage. The authors introduce word order as a third independently trackable component alongside lexical content and meaning, yielding a three-way decomposition that extends prior work on conceptual content/language separation.

2. Syntactic structure precedes surface language realization. Intermediate representations encode target-side syntax before resolving the target language, even while remaining aligned with an English-adjacent conceptual space.

3. Commitment to syntactic structure is partially localized to language-agnostic components. Syntax commitment is attributable to specific layers and, in most models, to individual attention heads that are more sensitive to grammatical structure than to language identity, suggesting partially shared mechanisms across languages and constructions.

Main Findings

• Three-stage order S → L → C. Across models and datasets, syntactic structure (S) is resolved before surface language (L), which is resolved before lexical content (C). The maximum-probability-difference layers increase monotonically in that order in eight of the nine model-dataset combinations.

• The single exception is Llama 3 on the SVO dataset. There, syntax and language switches occur at the same layer (14), yielding a collapsed strategy S,L → C rather than S → L → C.

• Layer-level evidence in Llama 3 (English → French base, English → Dutch plant). Patching early layers changes nothing and the prediction stays voiture. From layer 14 only the word order persists (verte), the language persists from layer 17 (groene), and the concept from layer 19 (rode).

• English-like intermediate tokens already follow target word order. For the prompt Deutsch: "Ich sah den roten Apfel" – Français: "J'ai vu la [pomme rouge]" from the NP dataset, intermediate predictions favor the English noun apple rather than the adjective red, consistent with the French noun-first order (pomme rouge). The inverse holds for French → German, where the adjective (circles, roter Apfel) is preferred.

• mGPT does not show the English signal. Its representations do not evoke any English signal in the inner layers, unlike Aya Expanse and Llama 3.

• Identified syntax-switch layers. mGPT: layer 11. Aya Expanse: layer 15. Llama 3: layer 14.

• Single syntax-sensitive heads in two models. In Llama 3 and Aya Expanse the S-switch layer contains a single attention head (H14.25 and H15.0, respectively) with R(h,t) > 1 for all tokens with S_plant. mGPT instead distributes the effect across multiple heads, with H11.2 (R ≈ 1.44) and H11.7 (R ≈ 1.27) strongest.

• mGPT entangles syntax and language. In mGPT the highest-scoring head also affects surface language, making syntax hard to disentangle from surface language for that model.

• Head effects are stronger in the larger models. The identified heads primarily influence syntactic structure while weakly biasing toward the patch language, with the effect substantially stronger in Aya Expanse and Llama 3 (up to R ≈ 3–5) than in mGPT (below R ≈ 2).

• Mean activation patching separates word order from language. KL divergence is consistently higher when base and patch languages differ in word order, and low when they share the same order; distributions isolating surface language show no comparable structure. This pattern is most pronounced in mGPT and Llama 3, while Aya Expanse shows weaker and less symmetric effects.

• A German-specific exception. When German is in the base target language position, the syntax effect is absent in Aya Expanse and Llama 3. The authors note this cannot be attributed to similarity to English alone (no comparable effect in Dutch) and leave it for future investigation.

• Syntactic role is functional, not geometric. Cosine similarity between mean activations of the S-sensitive head does not cluster by word order across target languages, even though the heads are functionally sensitive to word order.

• Reported Δmax values (Table 3). NP: mGPT 0.02 (layer 7), 0.12 (12), 0.23 (19); Aya Expanse 0.10 (11), 0.52 (19), 0.65 (31); Llama 3 0.09 (14), 0.21 (16), 0.55 (25). SVO: mGPT 0.00 (7), 0.04 (12), 0.09 (19); Aya Expanse 0.07 (15), 0.27 (16), 0.41 (29); Llama 3 0.08 (14), 0.11 (14), 0.31 (27). MV: mGPT 0.00 (7), 0.07 (12), 0.35 (23); Aya Expanse 0.06 (18), 0.37 (19), 0.63 (29); Llama 3 0.05 (15), 0.19 (16), 0.57 (31).

• Entropy check. Nouns and adjectives appear to be treated similarly at every stage in Llama 3, and there is a difference between legitimate words and noisy input (a random sequence of characters of length between 3 and 8).

• English as a conceptual anchor. The authors conclude that English functions as a conceptual anchor rather than a syntactic one, since intermediate English-like representations already reflect target word order.

Methodology in Plain English

The authors build three small, controlled parallel datasets in which word order varies systematically between languages:

• Noun Phrase (NP): 197 samples covering Chinese, Dutch, English, German, Russian, French, Italian, and Vietnamese, targeting adjective–noun ordering (contrasting 5 adjective-first languages with 3 noun-first languages; for French, adjectives that canonically follow the noun).
• Subject Verb Object (SVO): 154 samples covering Chinese, English, Russian, Japanese, and Turkish, contrasting SVO and SOV clause-level order.
• Modal Verb (MV): 31 samples covering Chinese, English, German, and Turkish, contrasting languages that place the modal before versus after the infinitive.

All datasets are generated from English source expressions using a list of 200 simple concrete nouns for the NP and SVO datasets; adjectives were generated using ChatGPT to ensure simple natural combinations, while the SVO and MV datasets were constructed manually. Translations into target languages used DeepL and were checked manually, with deviating cases removed or corrected.

For the experiments, each item is expressed as a one-shot prompt of the form [Source language]: [Source sentence] - [Target language]: [Incomplete target], constructed so the next token to be predicted corresponds to a syntactic position of interest. The paper reports 90 combinations and 90 × 197 = 17730 translation prompts for the NP dataset, 36 and 5698 for SVO, and 16 and 496 for MV. (Table 4, by contrast, lists 36 SVO triplets and 12 MV triplets.)

A key design choice is factoring each target token along three binary dimensions: surface language L, syntactic structure S (word order), and lexical content C. Because word orders are language-dependent, S is fully determined by L. This yields 2³ = 8 token combinations, for example voiture (French, car), verte (French, green), auto (Dutch, car), and rode (Dutch, red), with a base prompt in French and a plant prompt in Dutch.

Two analysis techniques are used. LogitLens projects intermediate hidden states through the unembedding layer to approximate next-token predictions at each layer, revealing how predictions evolve. Activation patching replaces hidden states in a base prompt with those from a plant prompt at selected layers, positions, or modules (attention heads, MLP layers), allowing causal localization of where information relevant to a prediction is represented. Interventions were set up using the Python module pyvene, extended by the authors to adapt to Aya Expanse.

To find syntax-sensitive heads, the authors patch individual attention heads within the S-switch layer and compute a ratio R(h,t) comparing the effect of a single patched head to the average across all heads in that layer. They then perform mean activation patching on the most S-sensitive head, replacing its activation with the average over samples from a different target language, varying whether the patch language shares the base word order. Effects are measured with normalized probabilities over part-of-speech contrasts and with Kullback–Leibler divergence, with separate distributions isolating surface language. Finally, they check whether mean activations of the S-sensitive head cluster by word order using cosine similarity, and they add a small check on 27 sentences from the FLORES-200 dataset.

The three models analysed were chosen to differ in size and degree of English bias: mGPT 1.3B (balanced multilingual corpus with no dominant language), Aya Expanse 8B (pretraining language distribution not publicly disclosed), and LLaMA 3 8B (predominantly English-trained).

Why This Matters

For research, the paper extends the two-way decomposition of translation into concept and language with a third, causally separable component: word order. It also reframes the role of English in multilingual models — intermediate English-like representations appear to be a conceptual anchor rather than the vehicle through which syntax is transferred. The finding that syntax commitment is functionally shared across languages without a shared activation geometry is a distinctive result for interpretability work on multilingual models.

Real-world applications implied by the work:

• Debugging and auditing machine translation systems, by identifying which internal components drive word order errors versus wrong-language or wrong-content errors.
• Targeted model editing or steering, since the S-switch layers and syntax-sensitive heads are candidate intervention points for correcting word-order behavior.
• Diagnosing failures in non-English-centric prompting pipelines, because the paper shows syntax is resolved before surface language rather than through an English pivot.
• Interpretability tooling for multilingual models, where the L/S/C decomposition provides a reusable experimental template for probing new languages and constructions.

Industry relevance: developers of multilingual assistants, translation products, and low-resource language tooling could use the layer- and head-level account to decide where to intervene when a model produces correct content in the wrong order, or correct order in a language other than the requested one. The authors' observation that prompt phrasing strongly affects model accuracy is also directly relevant to practitioners who design translation prompts.

Future Directions

• Broaden the grammatical coverage. The study focuses on noun phrase structure and the order of verb and subject; phenomena such as grammatical case or long-distance dependencies may rely on different internal mechanisms and cannot be assumed to follow the same patterns.
• Test with more natural data. The datasets are synthesized and prompts include a one-shot example pooled from the same dataset; more naturally diverse parallel data might lead models to implement different translation strategies. The authors' own FLORES-200 check used only 27 sentences and is explicitly left for future work.
• Move beyond the linear-interchangeability assumption. Activation patching, despite giving clearer causal evidence than LogitLens, still assumes that the computations needed for translation and grammar mapping are easily interchangeable and linear.
• Analyse more models. Due to scarcity of multilingual data, the one model the authors can confirm was trained on non-English-biased data (mGPT 1.3B) is small by current standards of LLMs.
• Explain the German exception and Aya Expanse's asymmetry. The absent syntax effect when German is the base target language, and Aya Expanse's weaker, less symmetric patching effects, remain unexplained and may reflect an interaction between language relatedness and resource effects.

Target Audience

Researchers and graduate students in mechanistic interpretability, multilingual NLP, and machine translation who want to understand how translation is implemented inside transformer language models. It will also be useful to practitioners building or auditing multilingual and translation systems who need layer- and head-level evidence about where word order, language, and content are decided, and to linguists interested in how typological word-order differences are reflected in model internals. A background in transformer architecture and causal intervention methods is helpful; the paper is not an introductory survey.

Authors’ abstract

Multilingual large language models (mLLMs) achieve strong performance in machine translation, yet our understanding of the mechanisms by which they transform representations from one language to another remains incomplete. Prior work suggests that translation decomposes into separable processes within an mLLM, where conceptual content is first represented independently, followed by a production into language-specific form. In this work, we show that translation is even more modular than previously assumed and that the output language production in translation processes is actually further separable into a syntax and a surface language process. We construct controlled multilingual datasets that isolate cross-linguistic differences in word-order and use causal interventions and probing to track how representations are transformed during translation. We find that models first construct target-side word-order before realizing the target language surface form. We identify individual attention heads that are selectively sensitive to syntactic transformations while remaining largely invariant to language identity. These results establish the commitment to a syntactic structure as an independent stage in translation, extending prior decompositions and showing how translation is implemented by functionally different components within mLLMs.