What, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations

AI summary

Overview

Research Area: Natural Language Processing Technical Level: Beginner-Friendly Reading Time: ~3 minutes Published: July 23, 2026

What This Paper Is About

Do independently trained language models come to represent the same thing in the same way. We answer for code, extending a recently introduced concept-circuit extraction method to a 2x2 design -- Python and Rust crossed with Qwen2.

Key Contributions

1. Do independently trained language models come to represent the same thing in the same way
2. We answer for code, extending a recently introduced concept-circuit extraction method to a 2x2 design -- Python and Rust crossed with Qwen2
3. 7B -- and measuring a complete inventory of grammatical concepts (58 Python, 57 Rust) identically in all four cells: the smallest design that separates what depends on the task, the language, and the model

Main Findings

• Finding 1: 94x more than Qwen -- a direction no prior result predicts

Methodology in Plain English

We answer for code, extending a recently introduced concept-circuit extraction method to a 2x2 design -- Python and Rust crossed with Qwen2

Why This Matters

Impact on Research: This work opens new directions for researchers in Natural Language Processing.

Real-World Applications:

• Improving AI system performance and reliability
• Developing more efficient and scalable solutions
• Addressing practical challenges in Natural Language Processing
• Enabling new capabilities in AI-powered applications

Industry Relevance: Companies working with AI can leverage these findings to enhance their products and reduce development friction.

Future Directions

• Extending the approach to related domains and use cases
• Improving scalability and efficiency of the proposed methods
• Investigating practical deployment considerations
• Exploring combinations with complementary techniques

Target Audience

• Researchers in Natural Language Processing and related fields
• Engineers building AI systems and applications
• Students studying advanced topics in artificial intelligence
• Industry practitioners seeking to apply cutting-edge research

This summary was generated from the paper abstract. For full technical details, refer to the original paper.

Authors’ abstract

Do independently trained language models come to represent the same thing in the same way? We answer for code, extending a recently introduced concept-circuit extraction method to a 2x2 design -- Python and Rust crossed with Qwen2.5-Coder-7B and DeepSeek-Coder-V1-6.7B -- and measuring a complete inventory of grammatical concepts (58 Python, 57 Rust) identically in all four cells: the smallest design that separates what depends on the task, the language, and the model. The answer splits into three parts. What earns dedicated circuitry is set by the task: the models agree on which concepts receive circuits (Spearman $ρ$ = 0.638 for Python, 0.673 for Rust, both p < $10^{-7}$). Where those circuits sit is set by the model: Qwen processes concepts in a late band (~L17-19), DeepSeek at L6-7, for both languages. How circuits grow across layers is also set by the model: Qwen gives its atomic concepts an early spike that DeepSeek does not. "Are circuits universal?" thus has no single answer: yes for What, no for Where and How -- universality is a property of representational content, not of computational organisation. None of this structure was fixed in advance. The agreement could have landed anywhere between independence and identity; it lands at $ρ\approx 0.65$. Rust constructs receive 2-3x more concept-specific circuitry than their Python equivalents, in both models. Both models share neurons between the languages (6/7 and 7/7 paired constructs), DeepSeek 1.94x more than Qwen -- a direction no prior result predicts. And Qwen binds nine keywords of Rust's type-and-trait machinery into one tight neuron cluster (Jaccard 0.535 vs null 0.112, p < 0.001), a semantic dimension invisible in surface syntax. Ablation and linear probes confirm the circuits are functional. All claims are scoped to this 2x2; whether the per-model profile predicts a third model is the designed next test.