Causal Syntax Circuits in CodeLMs: Budget and Probe Decide What Looks Localised

AI summary

Overview

Research area: Mechanistic interpretability of large language models for code (codeLMs), specifically causal circuit analysis and activation steering of programming-language preference.

Technical level: Intermediate. The paper assumes familiarity with activation patching, mean ablation, circuit extraction and steering vectors, but every operation is defined in the text and the argument is largely about experimental design choices rather than new mathematics.

Scope: A 55-setting empirical grid over ten open code models (five base, five instruction-tuned) and three target languages (Rust, C#, PHP), testing whether the tendency of code models to default to Python is carried by a small, findable, steerable set of components — and showing that the answer depends on the ablation budget and the prompt probe.

What This Paper Is About

Code models asked to "sort a list" usually write Python, even in a Rust repository, unless the prompt names the language. The authors call this programming-language defaultness and ask a mechanistic question: is that preference carried by a small group of internal components that an inference-time edit could redirect, the way refusal and natural-language identity have been reported to be? The paper is a systematic re-examination of an earlier negative answer to that question, showing that the earlier conclusion was an artefact of two choices normally treated as implementation details — how many components the circuit search may keep, and which prompts the preference is measured on.

Key Contributions

1. A 55-setting grid instead of 15. The earlier study covered 15 model–language settings on base models only. The authors add five instruction-tuned checkpoints, a second component budget of 40 (matching the scale used in sparse-feature circuit work), and a second prompt suite drawn from 128 XLCoST parallel functions, giving 55 runs in total.
2. A demonstration that the budget decides the localisation verdict. Running the identical pipeline at k = 12 and k = 40 shows that the count of statistically specific circuits rises from 1 of 15 to 6 of 15 purely as a function of how many components the search may keep.
3. A demonstration that the probe decides the measured size. The same metric on the same models reads about 1.6 on the hand-built aligned prompt suite and about 0.01 on naturally occurring XLCoST pairs — two orders of magnitude apart.
4. A diagnostic for failed circuit searches. Every statistically significant circuit whose effect points away from the target language comes from a setting where greedy extraction failed to find a compact subset, making such results a signal about the search rather than a phenomenon in the model. The full grid, pipeline, prompts and result files are released.

Main Findings

• The preference is large, grows with model size, and survives instruction tuning. The sign of the metric recovers the intended language in all 45 aligned-prompt settings without exception. Separation grows within each family, from a mean of 1.51 at DeepSeek-Coder-1.3B to 1.61 at 6.7B, and from 1.73 at Qwen2.5-Coder-7B to 1.81 at 14B. Instruction-tuned checkpoints have the larger separation in all 15 paired settings, by 12% on average and by as much as 30% (DeepSeek-Coder-1.3B on Rust, 1.49 → 1.93).
• Vocabulary size does not drive the metric. C# and PHP target sets are twice the Rust set after id resolution (12 ids vs. 6; the Python set is 6). Mean separation over the five base models is 1.603 for Rust, 1.650 for C#, and 1.663 for PHP — a gap of 0.06, or 3.7% of the separation, against a 2× difference in set size. Across the 15 aligned base settings, set size and separation are not associated (Spearman ρ = +0.20, p = 0.48).
• The measured size belongs to the probe. On XLCoST, separation collapses from about 1.6 to below 0.012 in all ten settings run (C# and PHP only). Both terms of the metric are near zero: at the position where a naturally occurring aligned pair happens to end, the model is not about to emit a language-defining keyword at all. No XLCoST setting yields a specific circuit and no intervention moves anything.
• The untagged default is real but faint. On ten bare instruction prompts with no language tag, the default is Python-leaning in 12 of 15 settings, with all three exceptions on PHP. Magnitudes are one to two orders below the tagged separation. Unlike the tagged measure, the untagged effect does not grow monotonically with scale — Qwen2.5-Coder-7B shows a stronger untagged default than the 14B.
• Localisation depends on the component budget. At k = 12, exactly 1 of 15 settings (CodeLlama-7B on Rust) beats size-matched random controls after Benjamini–Hochberg correction. At k = 40, 6 of 15 do, including DeepSeek-Coder-1.3B/C# (+0.61) and Qwen2.5-Coder-7B/PHP (+1.25), against random-control means below 0.03. On the instruction-tuned grid at k = 12, 2 of 15 are specific, one at +1.27. Only CodeLlama-7B/Rust is specific at both budgets.
• The surviving circuits share no structure across models. They span the full depth range — L9MLP in CodeLlama-7B, L12MLP in Qwen2.5-Coder-14B, L22–L24MLP in Qwen2.5-Coder-7B, L29MLP in DeepSeek-Coder-6.7B — and are MLP-mediated: attention heads appear in ablation rankings but rarely survive extraction. At k = 40, four settings return the full candidate set and four others return a single component.
• Patching and steering are not uniformly inert. Patching moves the metric across zero in 2 of 45 aligned and instruction-tuned settings: Qwen2.5-Coder-7B base on PHP at k = 40 (−0.91 → +0.50) and the instruction-tuned model on Rust at k = 12 (−0.99 → +0.50). Steering changes the most likely next token to target syntax in 9 of 45 settings (6 at k = 40 on base models, 3 on instruction-tuned). At k = 12 on base models it never does, which is what the earlier version reported.
• Successful steering needs model-specific coefficients. Three of the nine steering successes switch at α = 1; the remaining six need α ≥ 2 and four need α ≥ 4. The upper range is where output degrades, and on Qwen2.5-Coder-14B/C# a coefficient of 4 or more injects 126–128 C# markers into control prompts that should contain none. The safe coefficient for Qwen2.5-Coder-7B is 8 on Rust, 4 on PHP, and 0 on C# — C# cannot be steered safely at any tested strength.
• Away-pointing circuits are extraction failures. Five significant results across three settings point away from the target, the largest being CodeLlama-7B on C# at −0.43, reproduced at both budgets and on the instruction-tuned checkpoint. Late-layer transplant is damaging in every language (summed shifts of −0.46, −0.43 and −0.46 for Rust, C#, PHP over all 40 components), so depth explains the damage but not which setting flips. What separates them is that all five come from settings where extraction returned the full candidate set; at k = 40, full-set circuits have a median shift of −0.04 against +0.05 for compact ones.
• No single component is behaviourally load-bearing. Removing one component at a time from the CodeLlama-7B/Rust top-12 set at α = 1 — including dropping the one specific circuit — leaves marker accuracy and F1 at 1.000, with mean next-token metric moving by less than 0.02. Two CodeXGLUE tasks stay pinned at chance under steering, so the direction encodes surface syntax, not task competence.

Methodology in Plain English

The authors build pairs of prompts that do the same thing in two languages, with a language tag on the final line so the model's very next token must be the keyword that opens a function body. They then measure, in a single forward pass, how much probability mass sits on target-language opening tokens versus Python opening tokens; the difference is a continuous score in [−1, 1], and the gap between the target and Python members of a pair is called the separation. Overlapping tokens are removed so no language gets credit for shared syntax.

Four interventions act on that score. Mean ablation removes each component (an attention head or an MLP block) and ranks it by how much the score drops. Patching transplants target-run activations into the clean Python run to test sufficiency. Greedy extraction adds components by individual patched shift until 90% of the full-set effect is reached, giving the "circuit." Every circuit is then compared against 30 size-matched random component sets with a Welch t-test, corrected for multiple comparisons with Benjamini–Hochberg at q < 0.05 — so a component set only counts as specific if it beats random sets of the same size.

Steering builds a difference-in-means direction per circuit component and adds it during the forward pass at coefficients 0, 1, 2, 4 and 8, recording both the metric and whether the most likely next token is target syntax. Because a rising metric can mean either mass moving onto target keywords or the distribution flattening, they add an argmax check and a contamination check that asks whether the same coefficient injects target markers into unrelated control prompts. The whole grid runs on one A100-80GB in a few GPU-hours per run, in fp16 (bf16 for Qwen, which produced NaN logits in fp16 on the bridge backend).

Why This Matters

Impact on research. The paper's central claim is methodological and portable: two knobs normally reported as setup — the component budget and the prompt probe — moved the headline conclusions from "not localisable, never steerable" to "localisable in 6 of 15 settings, steerable in 9 of 45," and from "a large, robust bias" to "a bias two orders of magnitude smaller." The authors argue circuit papers should report a budget sweep the way ablation papers report a seed sweep, and state localisation claims as a curve over budgets rather than a single number. The finding that away-pointing significant results diagnose a failed search rather than an anti-circuit gives reviewers a concrete check: report extracted circuit size alongside significance.

Real-world applications:

• Codebase language control. Teams whose repositories are not Python want a cheap way to move a model off its default; this paper bounds how reliable that dial currently is (steering works in 9 of 45 settings, with a per-setting coefficient search and a fluency cliff).
• Model monitoring. The preference metric is one forward pass, so it can be logged cheaply as a diagnostic for whether a deployed model is drifting toward Python on underspecified prompts.
• Evaluation design for code benchmarks. The XLCoST result warns that a model's measured language bias depends heavily on whether the evaluation constructs a decision point or uses naturally occurring code.
• Interpreting interpretability claims. Practitioners reading circuit papers can use the budget-sweep and size-matched-random-control conventions to judge whether a reported "localised behaviour" would survive a different search budget.

Industry relevance. The practical lesson is deflationary but useful: prompting and fine-tuning remain the dependable route to language control, while steering trades syntax for fluency past a narrow, model-specific coefficient window. The paper also documents that no single component is behaviourally load-bearing in this setting, which argues against neuron-level editing products for code language control and toward sparse-feature or edge-level analyses.

Future Directions

• A size-matched syntax vocabulary. The authors flag the current metric as unfinished: the C# and PHP sets are twice the size of the Rust set, which §4.2 bounds but does not remove, and the C# and PHP sets share markers with other C-family languages.
• Budget sweeps as standard practice. The paper tests only two budgets (12 and 40) and argues the localisation claim should be reported as a curve over budgets; extending to more budgets and other conventions is left open.
• Finer-grained units of analysis. Because attention heads appear in ablation rankings but rarely survive extraction, and no component is a bottleneck, the authors point to sparse-autoencoder decompositions and edge-level attribution patching as the natural next step.
• Closing the gap between probe and deployment. The aligned suite constructs a forced decision point while XLCoST reads naturally occurring code where no language keyword is imminent; a probe that measures defaultness in the wild without the two-orders-of-magnitude collapse is not yet available. The scope is also limited to three target languages, one base language, and two sizes per family — enough to show budget matters, not enough for a scaling law.

Target Audience

Mechanistic interpretability researchers working on circuit discovery and steering, who will find the budget-sweep and random-control arguments directly applicable to their own claims. Also useful for code-model practitioners choosing between prompting, fine-tuning and activation-level control for language enforcement, for evaluation designers who need to know how much a benchmark's prompt construction determines its reported bias, and for reviewers who want a concrete checklist (extracted circuit size, budget sweep, size-matched controls, argmax and contamination checks) for localisation claims. Readers without prior exposure to activation patching will need to consult the cited methodological work first.

Authors’ abstract

Code models write Python when the prompt does not say which language to use. We ask a mechanistic question about that habit: is it produced by a small, findable group of components inside the model? We run one pipeline---mean ablation, activation patching, greedy circuit extraction, a random-control significance test, and activation steering---over 55 model-language settings spanning ten open code models (five base and five instruction-tuned), three target languages (Rust, C#, PHP), two prompt suites, and two component budgets. We report five findings. First, the preference is large, grows with model size, and survives instruction tuning: it is larger on every instruction-tuned checkpoint than on its base version, by 12% on average. Second, its measured size is mostly a property of the probe; replacing our aligned prompts with XLCoST parallel functions takes the same measurement on the same models from about 1.6 to about 0.01. Third, whether the preference looks localised depends on how many components the search may keep: with a budget of 12, 1 of 15 settings yields a circuit that beats size-matched random controls after multiple-comparison correction, and with a budget of 40, 6 of 15 do. Fourth, interventions that a smaller-budget study reports as uniformly failing do sometimes succeed: patching makes a Python prompt prefer target syntax in 2 of 45 settings, and steering changes the most likely next token in 9 of 45. Fifth, every circuit whose effect is significant but points away from the target language comes from a setting where circuit extraction failed, which makes such results a diagnostic rather than a phenomenon. The practical lesson is that a claim about whether a behaviour is localised can turn on analysis choices that papers rarely report. We release the full grid so those choices can be varied.