Differential syntactic and semantic encoding in LLMs

AI summary

Overview

• Research area: Natural language processing and large language model interpretability, at the intersection of computational linguistics (syntax and semantics).
• Technical level: Intermediate. The core ideas are intuitive (averaging representations of related sentences and subtracting them), while the similarity metric and the ablation algebra require some comfort with vector representations.
• One-sentence scope: The paper uses averaging-based "syntactic" and "semantic" centroid vectors to test whether, and to what degree, syntax and meaning are linearly and separately encoded in the inner layers of large language models.

What This Paper Is About

The paper asks how two core components of linguistic competence, syntax (the structural scaffolding of a sentence) and semantics (its meaning), are stored in the hidden representations of LLMs. To probe this, the authors build a synthetic dataset of sentences matched either in syntactic structure (same part-of-speech template) or in meaning (paraphrases and translations), then check how much the similarity between matched sentences changes when they subtract a "syntax centroid" or a "semantic centroid" from each sentence representation. The goal is to determine whether these two kinds of information are linearly encoded and whether they can be separated from one another.

Key Contributions

1. A linear centroid method for isolating syntax and semantics. The authors show that averaging the representations of sentences that share a syntactic template yields a vector (the syntactic centroid) that carries a substantial share of the syntactic information in a sentence, and that averaging representations of a sentence's translations across 6 languages yields a semantic centroid that carries a substantial share of its meaning. Subtracting these centroids from sentence vectors strongly reduces the corresponding similarity signals.

2. Evidence for differential cross-layer encoding profiles. Syntax similarity is high from the earliest layers and stays roughly constant across depth, while semantic similarity is low early, peaks in the central layers, and drops only at the final layer. Quantifying the norm decomposition confirms syntax dominates early layers and semantics dominates central layers.

3. A partial decoupling of syntax and semantics, with an asymmetry. Removing the semantic centroid barely affects syntactic similarity between syntax twins, whereas removing the syntactic centroid does reduce the semantic similarity between paraphrases (though less strongly than removing the semantic centroid itself), suggesting it is easier to capture syntax independently than semantics.

4. Confirmation through downstream probes. POS-template classification and paraphrase recall@3 degrade in the expected directions under matched ablations and are largely unaffected by permuted (random) centroid ablations.

Main Findings

• Syntax is nearly depth-invariant: Similarity between sentences sharing a POS template exceeds 0.7 in all layers when token representations are concatenated, and stays above 0.4 at all depths when they are averaged. Randomly paired sentences yield a similarity of 0 at every layer as a control.
• Averaging hurts syntax but helps semantics: Averaging over tokens lowers the syntactic signal, which the authors attribute to loss of positional information, while for paraphrases averaging increases similarity relative to concatenation, plausibly because paraphrases share meaning under different word orders.
• Semantics has a central "core": Paraphrase similarity is low in early layers, rises through the middle layers, and falls again only at the very last layer. This is consistent with what was previously observed for translations across languages.
• Both signals are at least partially linear: Subtracting a sentence's projection along its syntax centroid removes a significant fraction of syntax-twin similarity; subtracting its projection along its semantic centroid strongly reduces paraphrase similarity, mostly in the central layers. Matched ablations beat randomly permuted centroid ablations in both cases.
• Asymmetric interaction: Ablating the semantic centroid from syntactically matched pairs largely preserves their syntactic similarity (comparable to a random-centroid control), while ablating the syntactic centroid from paraphrase pairs significantly reduces semantic similarity, especially in central layers.
• Centroids explain part of the norm, not all: In the central layers the syntax and semantic centroids together explain at most roughly 40% of the squared norm of sentence representations, with a large residual unexplained fraction.
• Training dynamics differ (Pythia-6.9b): Syntax centroids alone capture up to roughly half of the total squared norm very early in training, while the norm explained by semantic centroids builds up progressively at later stages.
• Probe results: On the best-performing layer across all layers, baseline performance is 0.85 POS-template classification accuracy and 0.85 paraphrase recall@3. Semantic ablation lowers recall@3 to 0.66 while leaving syntax accuracy at 0.85; syntactic ablation drops syntax accuracy to 0.10 while raising recall@3 to 0.90 (roughly a 5% increase). Random-centroid controls leave performance near baseline (0.85/0.83 for random semantic ablation, 0.81/0.85 for random syntactic ablation).

Methodology in Plain English

The authors compiled roughly 2,000 pairs of English sentences that share a part-of-speech template but express unrelated meanings, plus English paraphrases of the original sentences and translations of them into 6 languages (Arabic, Chinese, German, Italian, Spanish, Turkish). Sentences were generated with Gemini and ChatGPT, and, for Chinese only, DeepSeek; none of these generation models was used for the analysis. The stimuli are short: between 6 and 10 words, with an average word overlap of 0.1 ± 0.1 between originals and their syntax twins, and at least 5 sentences per template required so a centroid can be built.

For each sentence, they collect its hidden representation on every layer of the network, represented either by concatenating the last N tokens (N = 6 for syntax, 3 for semantics) or by averaging them. They then build a syntactic centroid by averaging the representations of all the sentences sharing that sentence's POS template (excluding the sentence itself), and a semantic centroid by averaging the representations of its translations (excluding the English original and its paraphrase). Because the sentences averaged into each centroid differ in the other dimension, the irrelevant information is intended to average out.

Similarity between two sets of representations is measured not by linear metrics such as CKA, which gave very weak signals in high dimensions, but by comparing high-dimensional neighborhoods: specifically the average rank of a point's nearest neighbors across the two spaces, a quantity that is 0 for independent representations and 1 when nearest neighbors coincide, and that is invariant to global translation, rotation and scaling.

To test whether a signal is linearly encoded, they subtract each sentence's projection along the relevant centroid direction, making the sentence vector orthogonal to that centroid, and measure how much similarity drops. Controls include subtracting the projection along a mismatched or randomly permuted centroid. Finally, they measure what fraction of the squared norm of each sentence vector lies along each centroid direction, and they test the behavioral effect of ablations with a linear POS-template classifier and a paraphrase recall@3 probe. Main results use DeepSeek-V3 with 671b parameters; results are qualitatively reproduced with Qwen2-7b and Gemma3-12b, and Pythia6.9b checkpoints are used to study training dynamics.

Why This Matters

Impact on research. The findings support linear superposition as a general encoding mechanism in deep networks, extending it from word-level analogies to abstract features such as sentence structure and meaning. They also provide geometric, ablation-based evidence that a "semantic core" exists in the central layers of LLMs, complementing probing studies that suggest a syntax-then-semantics processing pipeline. For linguistics, the results show that a model trained without explicit linguistic priors develops an imperfect but clear separation of syntax and semantics, which the authors suggest could point to the autonomy of syntax as a property emerging in very different cognitive systems.

Real-world applications:

• Steering model outputs toward or away from specific meanings or syntactic structures by adding or subtracting centroid directions.
• Diagnosing and mitigating spurious correlations between syntactic form and semantic content in training data.
• Auditing and controlling model behavior by locating which layers carry which kind of linguistic information.
• Comparing representations across languages, since the semantic centroid is built from parallel translations.

Industry relevance. If syntactic and semantic information can be manipulated by simple vector arithmetic, developers gain a cheap, layer-targeted tool for model editing, debiasing and controllable generation, rather than resorting to retraining or opaque fine-tuning. The finding that the two signals live at partly different depths also informs where to intervene in a network for a given goal.

Future Directions

• Determine how much of the residual unexplained norm (in central layers, at least roughly 60% of the squared norm is not captured by the centroids) reflects limitations of the linear centroid approach versus non-linguistic "functional" information stored in representations.
• Scale the study beyond roughly 2,000 samples and sentences of at most 10 words, to test dependence on sentence length and on the token aggregation method.
• Test whether centroid directions can steer generation, including interventions during text production rather than only measuring representational similarity.
• Extend the analysis to non-English target languages, to more models of different sizes, training data volumes and objectives, and combine the approach with decomposition of neural activity into frequency bands, under the hypothesis that syntax and semantics are predominantly encoded in high- and low-frequency bands respectively.
• Explore richer, non-linear ways of constructing centroids, and add deeper probing tests following established probing-control protocols.

Target Audience

Researchers and graduate students in NLP interpretability, computational linguistics and cognitive science who are interested in how linguistic structure is represented inside LLMs. It is also useful for machine learning engineers who want practical, layer-specific handles for controlling model behavior.

Authors’ abstract

We study how syntactic and semantic information is encoded in inner layer representations of Large Language Models (LLMs), focusing on the very large DeepSeek-V3. We find that, by averaging hidden-representation vectors of sentences sharing syntactic structure or meaning, we obtain vectors that capture a significant proportion of the syntactic and semantic information contained in the representations. In particular, subtracting these syntactic and semantic ``centroids'' from sentence vectors strongly affects their similarity with syntactically and semantically matched sentences, respectively, suggesting that syntax and semantics are, at least partially, linearly encoded. We also find that the cross-layer encoding profiles of syntax and semantics are different, and that the two signals can to some extent be decoupled, suggesting differential encoding of these two types of linguistic information in LLM representations.