Parts-of-Speech as Emergent Categories in SAE Latent Space

AI summary

Overview

Research area: Natural Language Processing / mechanistic interpretability of large language models, specifically sparse autoencoder (SAE) analysis of morpho-syntactic representation.

Technical level: Intermediate. The paper assumes familiarity with probing classifiers, sparse autoencoders, and Universal Dependencies part-of-speech tags, but the core question and the three-stage pipeline are explained in accessible terms.

Scope in one sentence: The paper uses part-of-speech (PoS) categories as a controlled test case to determine whether morpho-syntactic information in LLaMA-3-8B's SAE latent space is carried by individual latents or by structured groups of sparse features.

What This Paper Is About

Sparse autoencoders are widely used to break dense language-model activations into sparse, supposedly interpretable units called latents, but it remains unclear whether human linguistic categories such as part-of-speech map onto individual latents. The authors ask whether PoS distinctions are recoverable from SAE activations and, more importantly, how those distinctions are organized: as atomic grammatical features, or as distributed groups of features. They test this on token-level SAE activations from LLaMA-3-8B over the UD English GUM treebank, plus a small controlled dataset of 180 lexical items built for targeted validation.

Key Contributions

1. PoS categories align with structured groups of sparse features, not single latents. Through feature-salience and coverage analysis, the authors quantify the size and organization of the latent groups supporting each PoS category and show that the size varies substantially across tags, with a clear difference between Open- and Closed-class categories.

2. These latent groups are compact yet effective. The union of the minimal coverage sets contains 498 latents, of which 12% are shared between two or more PoS categories, and this compact set supports multi-class PoS classification with performance comparable to using the full SAE representation.

3. The identified groups are stable on held-out data while only partially category-specific. Validation on the held-out GUM test split and on the controlled dataset confirms that the selected latents remain active on unseen tokens of their target category, but also reveals cross-category overlap, with a mean Distinctiveness of D = 0.27 (σ = 0.07) against a chance baseline of 1/|C| = 0.06.

4. A methodological argument for evaluating latents against linguistic category inventories. The authors argue that interpretability claims at the latent level should be tested against theoretically grounded category inventories rather than top-activating examples alone.

Main Findings

• PoS information is highly recoverable. One-vs-rest L1-regularized logistic regression probes over SAE activations achieve high F1 scores for most categories across 5-fold cross-validation on the GUM training split (per-tag values are shown in Figure 2 and not reported numerically in the text).

• Recoverability is uneven across tags. Closed-class categories and low-variability labels such as punctuation are easier to recover, while more lexically heterogeneous or less frequent categories score lower. Nouns and verbs are the best performing open-class PoS.

• Recoverability is not reducible to lexical memorisation. A control probe with randomly assigned labels reaches 0.54 Accuracy and 0.42 Macro F1, against 0.88 and 0.97 for the real probe; the 34–37 point gap supports the conclusion that lexical identity plays only a minor role.

• Probes rely on concentrated, not uniform, latent subsets. For each PoS category only a subset of latents receives positive weight, and Open-class PoS generally have more non-zero coefficients than Closed-class and Other-class categories (Figure 3).

• Coverage size is category-dependent. The number of salient latents needed to reach the 95% coverage threshold (k⁹⁵_c) varies across PoS categories: some tags reach it with a small number of latents, others need broader groups (per-tag values in Figure 4). The authors note that this compactness gradient co-varies with the size and formal variability of each category's type inventory, so it should be read primarily as a gradient in lexical variability rather than direct evidence of graded abstraction.

• A compact latent union preserves classification performance. The compact feature set L* contains 498 latents, 12% shared across two or more PoS, and a multinomial classifier trained only on these performs comparably to one trained on the full SAE representation.

• Held-out recall is consistently high. On the held-out treebank test set, most diagonal co-activation values are ≥ 0.95, meaning latents identified as salient for a PoS remain active on unseen tokens of that category.

• Category specificity is only partial. Off-diagonal co-activation values vary considerably. Adjectives are mainly confused with nouns, adverbs and proper nouns; adverbs show diffuse co-activation with adpositions, nouns, adjectives, subordinating conjunctions and verbs; proper nouns and nouns co-activate; and verbs overlap with nouns in homograph pairs such as to drink / the drink. Interjections show high spurious activation rates across several categories, plausibly due to annotation conventions assigning heterogeneous forms such as like, well, or God to intj in pragmatic contexts.

• Comparable to dense probes with far fewer features. The SAE-based probe scores 0.88 Accuracy / 0.78 Macro F1, versus 0.92 / 0.84 for raw layer-30 activations and 0.88 / 0.78 for raw layer-0 embeddings, using roughly 8 times fewer features. The authors explicitly do not claim superiority of SAEs as a probing tool, arguing instead that the SAE's contribution is decomposition and localisation.

• Randomly replacing L* features degrades performance sharply. With 0% overlap with L* the classifier reaches 0.24 Accuracy / 0.16 Macro F1 under cross-validation and 0.23 / 0.16 in train/test; at 25% overlap 0.49 / 0.41 for both; at 50% overlap 0.69 / 0.60 for both; at 100% (original) 0.87 / 0.76 under cross-validation and 0.89 / 0.81 in train/test.

• Controlled-data results show high recall but limited precision. Pseudo-multilabel evaluation on the controlled dataset gives an average of 0.192 precision, 0.937 recall, and 0.318 F1 across templates. Per-template F1 ranges from 0.270 (There is/was ([DET]) [NOUN]) to 0.365 (I see/saw [DET] [NOUN] [PUNCT]).

• A "first-token" latent set confounds the controlled results. The authors identify latents associated with a first-token concept and verify that prefixing each sentence with a different PoS shifts all these activations onto the new PoS.

• Activations are largely additive. On the controlled data, introducing a new token activates latents characteristic of its PoS; adding loyal to There is a dog triggers more adjective-associated latents. Some latents of related PoS are already present without the corresponding words, but their count grows when the word is included.

• Layer choice has limited effect on recoverability. Binary probing F1 is generally stable across layers, with higher variability for Other-class PoS except punctuation and a systematic slight dip in the last few layers. The number of non-zero activations per PoS increases modestly across layers (e.g., NOUN: 9024 → 9935 → 10043; ADP: 4955 → 5742 → 6511) while staying within the same order of magnitude.

Methodology in Plain English

The authors work with LLaMA-3-8B and the pre-trained EleutherAI/sae-llama-3-8b-32x sparse autoencoder, accessed through the Sparsify library.

Data. For naturally occurring text they use the UD English GUM treebank: 14,353 sentences and 252,284 tokens, covering all 17 Universal PoS tags across genres including academic, blog, legal, news, social and wiki. The training split is the discovery set; the test split is held out. They also build a small controlled dataset of 180 lexical items, focused on nouns and verbs: 160 nouns spanning multiple semantic categories and evenly split between animate and inanimate referents, plus 20 high-frequency verbs compatible with an intransitive template, balanced between 10 regular and 10 irregular past-tense forms. Sentences are instantiated in singular and plural, with and without an added adjective or adverb, and with and without trailing punctuation, and in present and past tense.

Extracting activations. Raw sentence text is fed to the model with its original subword tokenizer; hidden states are taken from the residual stream at layer 30 (the last layer before output), specifically after the MLP. The SAE encodes these into sparse vectors. Subword tokens are aligned to UD surface forms using character-span overlap, and the leftmost overlapping subword is designated the anchor, since that is where the UD token's identity first becomes available to the model; averaging over subwords was avoided because continuation pieces could dilute category-bearing activations.

Building the feature matrix. The SAE latent space spans 4096 × 32 = 131,072 dimensions (LLM hidden size times SAE expansion factor). Projecting onto latents attested at least once in the treebank yields a matrix X of shape N × D, where D = 130,246 attested latents.

Probing. For each PoS category, an L1-regularized logistic regression classifier (C = 0.1, liblinear solver, balanced class weighting) distinguishes that category from all others, evaluated with 5-fold cross-validation on the GUM training split. The L1 penalty produces sparse coefficient vectors.

Localization. The same classifiers double as feature-salience mechanisms: latents are ranked per category by their positive coefficients. Coverage is defined as the proportion of gold tokens of category c on which at least one of the top-k salient latents fires; the effective group size k*_c is the smallest k reaching a threshold τ = 0.95. A per-class threshold is used rather than a global top-k to avoid bias from imbalance.

Compact-feature classification. The union of all minimal coverage sets, L*, is used to train a multinomial logistic regression classifier, testing whether localized groups suffice for joint PoS prediction.

Validation. The selected groups are tested on the held-out GUM test split and on the controlled dataset. On held-out data the authors build a cross-PoS activation matrix M(c, c′) giving the probability that at least one latent in the group for c′ is active on gold tokens of c, and derive a Distinctiveness score D(c) = M(c,c) / Σ_c′ M(c,c′).

Controls and baselines. A random-label probe tests lexical memorisation. Raw embeddings at layer 0 and raw layer-30 activations serve as dense baselines. A random-subset baseline re-runs compact-feature classification keeping 0%, 25% or 50% of L* and randomly choosing the rest.

Why This Matters

Impact on research. The paper challenges the implicit assumption that SAE latents correspond one-to-one with interpretable linguistic categories. It shows that classical morpho-syntactic categories act more like emergent sets of localizable but distributed features, and it argues for evaluating latent interpretability against linguistically motivated category inventories rather than cherry-picked top-activating examples. It also sits alongside work such as Kantamneni et al. (2025), who found that probes trained on SAE latents do not consistently outperform simple baselines across 113 binary classification tasks, by clarifying what SAEs actually contribute: decomposition and localisation rather than raw accuracy gains.

Real-world applications:

• Model auditing and safety review. If regulators or auditors need to inspect what a model represents, knowing that linguistic categories are distributed across groups of latents rather than single features changes how inspection tooling must be built.
• Grammar-aware downstream tools. Part-of-speech tagging, parsing and morphological analysis pipelines built on interpretable internal features could benefit from knowing which latent groups carry syntactic information.
• Targeted model editing. If a specific syntactic behaviour needs to be altered or suppressed, group-level rather than single-latent interventions may be the appropriate unit of operation.
• Error diagnosis in low-resource or noisy settings. The finding that categories with small, invariant type inventories need few latents while open classes need broad groups gives a concrete diagnostic for why some grammatical categories are harder for models than others.

Industry relevance. Companies deploying LLMs in legal, medical or financial settings need interpretability methods that deliver reliable, non-misleading explanations. This paper's central caution — that a single latent rarely equals a single linguistic concept — directly affects how such tools should be scoped and marketed, and the compactness results suggest a practical route to leaner, feature-selective probes for internal monitoring.

Future Directions

• Extend beyond PoS to richer linguistic levels. The authors call for analysis of finer-grained morphological features, dependency relations, semantic roles and discourse-level phenomena, where latent organization may work differently.

• Test generalization across models, layers and SAE variants. The study covers one base model (LLaMA-3-8B) and one publicly available SAE; patterns may depend on the underlying model, the extraction layer, the SAE training procedure and the sparsity regime.

• Move from prediction to causal intervention. The localization procedure relies on linear probe coefficients as a salience signal, which the authors stress is not causal evidence. Ablation or activation-patching experiments would be needed to establish whether the identified latents are actually used by the model for morpho-syntactic processing.

• Broaden the controlled dataset. The current controlled set is intentionally small, targets mainly nouns and verbs, and does not cover the full syntactic and lexical variability of English. Larger controlled sets would allow systematic study of how lexical ambiguity, word order, morphology and sentence complexity affect PoS-related latent activations.

• Investigate typologically diverse languages. The authors note this as motivation for future work, since the findings so far rest on English.

Target Audience

This paper is most useful to mechanistic interpretability researchers working on sparse autoencoders and feature discovery; NLP researchers studying probing and the encoding of linguistic structure in LLMs; computational linguists interested in whether formal grammatical categories have correlates in neural representations; and applied practitioners who build or consume interpretability tooling and need to understand its limits. Readers should be comfortable with terms such as sparse autoencoder, latent, probing classifier and Universal Dependencies PoS tags. Code and data are available at https://github.com/colinglab/pos-sae-latents, with the derived latent dataset at HuggingFace (colinglab/UD_English-GUM-Latents_Meta-Llama-3-8B_L30).

Authors’ abstract

Sparse AutoEncoders (SAEs) offer a promising way to inspect language model representations, but it is still unclear what kind of linguistic structure their latents expose. We use part-of-speech (PoS) categories as a controlled test case to study whether morpho-syntactic information is encoded by individual latents or by structured groups of features. We find that PoS distinctions are highly recoverable from SAE activations, but do not align with one-to-one latent / category mappings. This recoverability is not reducible to lexical memorisation, and Open and Closed PoS classes differ substantially. Categories are supported by compact groups of sparse latents, with substantial variation across tags. These groups remain stable on held-out data, while also showing overlap between related categories. Our results show that SAEs localise morpho-syntactic information in a distributed and category-dependent form rather than through atomic grammatical features.