Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability

AI summary

Overview

• Research area: Mechanistic interpretability of large language models, specifically dictionary learning with Sparse Autoencoders (SAEs) for unsupervised concept discovery.
• Technical level: Intermediate. The core idea is intuitive, but Sections 3.1–3.2 define a formal data-generating process, two assumptions, and an InfoNCE-style contrastive loss that assume familiarity with SAE training objectives.
• Scope: The paper introduces Temporal Sparse Autoencoders (T-SAEs), a modification to SAEs that adds a temporal contrastive loss over adjacent tokens so that high-level features capture stable semantic and contextual content while low-level features absorb local syntactic detail.

What This Paper Is About

Sparse Autoencoders are a popular tool for turning the dense internal activations of language models into human-interpretable features, but the features they recover tend to be token-specific, local, noisy, and syntactic — for example, a feature for "the phrase 'The' at the start of sentences" (Neuronpedia feature 11795 of Gemmascope's Gemma2-2b 16k SAE) rather than a coherent semantic concept. The authors argue this failure comes from treating tokens as independent and ignoring the sequential nature of language, in which semantics evolve smoothly while syntax depends on short-range dependencies. Their goal is a self-supervised SAE variant that separates semantic from syntactic features without any explicit semantic labels.

Key Contributions

1. A data-generating framework for language. The paper formalizes language production as τ_t = φ(τ^{t−1}, h_t, l_t), separating high-level, temporally consistent latent variables h_t (semantics, intent, context) from low-level, local variables l_t (grammatical gender, word choice), and states two assumptions: Temporal Consistency (h_t is approximately invariant across tokens of the same sequence) and a Hierarchical Representation of Features (h_t reconstructs x_t up to ε, while l_t explains the unexplained residual).
2. Temporal Sparse Autoencoders. A modification that partitions the SAE feature space into a high-level block and a low-level block (20%–80% split in the main experiments) and adds a contrastive loss encouraging cosine similarity between the high-level latents of adjacent tokens within a batch, applied on top of a Matryoshka-style reconstruction objective so that low-level features reconstruct the residual.
3. Empirical validation across models and datasets. Experiments on Pythia-160m and Gemma2-2b, trained on the Pile and probed on MMLU, Wikipedia, and FineFineWeb, showing improved semantic and contextual recovery, better disentanglement, competitive standard SAE metrics, and improved steering.
4. Practical case studies and released artifacts. A dataset-understanding study on Anthropic's Helpfulness Harmfulness RLHF dataset and a steering evaluation across 30 features, plus released code, trained T-SAEs, and interpreted latents (github.com/AI4LIFE-GROUP/temporal-saes).

Main Findings

• Semantic and contextual recovery improves. Probing SAE decompositions for Gemma2-2b with k-sparse probes (k = 1, 5, 10, 20) and dense logistic regression probes, T-SAEs "outperform the baseline SAEs significantly for semantic and contextual labels, with little-to-no performance drop for syntactic information." TSNE visualizations show T-SAE high-level features clustering by MMLU question category and by question ID, while Matryoshka SAE embeddings cluster mainly by part-of-speech.
• Features are disentangled by level. High-level T-SAE features show semantic and contextual structure; low-level T-SAE features show syntactic structure and can recover syntactic information on their own despite only being trained to reconstruct the residual left by the high-level split. For Matryoshka SAEs, performance across all tasks is attributed almost entirely to the high-level split.
• Reconstruction quality is preserved. In Table 1, Temporal SAEs perform nearly equivalently to Matryoshka and BatchTopK SAEs on Fraction Variance Explained, Cosine Similarity, Fraction Alive, and Autointerp score.

• Temporal consistency is visible in activation traces. Over a sequence concatenating a biology MMLU question, a Charles Darwin letter (Project Gutenberg), a Wikipedia article on Animal Farm, and a mathematics MMLU question, the top-8 T-SAE features show clear phase transitions and smooth activations, with automated explanations matching the content ("Historical literature and academic writing" for the Animal Farm article; "Worship and spiritual practices" for a Bhagavat Gita excerpt). By contrast, top Matryoshka SAE features activate across all three sequences in Figure 1 without differentiating them and fluctuate per token.
• Dataset understanding on preference data. T-SAEs applied to the HH-RLHF dataset surface safety-relevant features for rejected responses such as "physical touch and intimacy", "etiquette and social behavior guidelines", "crime and malicious activities", and "violent or aggressive behavior descriptions", plus features that expose a spurious correlation: because raters often prefer an unhelpful response over a harmful one, chosen responses tend to be incoherent and rejected responses longer and more formal, producing features like "legal and formal language", "quotes and speech", and "transition words and phrases". Matryoshka SAEs instead highlight "specific bicycle components", "terms related to data management", and "references to ecosystem dynamics and environmental conditions".
• Steering improves. Following the evaluation scheme of Bhalla et al. (2025) across 30 features, with Llama3.3-70b grading success and coherence and manual verification of grading, T-SAEs Pareto-dominate existing SAEs, including the best SAE available for the same model on Neuronpedia. The authors report that baseline steering fails via token repetition at aggressive strengths, while high-level feature steering changes the semantics and is more forgiving of strength tuning.
• Ablations show the split and contrastive design matter. Relative to the default Pythia-160m T-SAE, a 50:50 split changes semantics by +0.02, context by +0.09, and syntax by −0.08; a 10:90 split changes semantics by −0.01, context by +0.01, and syntax by +0.01; contrasting against a randomly sampled earlier token (t−r with r < 25) changes context by +0.11 and syntax by −0.10; and removing the contrastive loss entirely (a naive sample-wise similarity loss) changes FVE by +0.01, Fraction Alive by +0.06, smoothness (high) by +0.07, semantics by −0.07, and context by −0.1.

Methodology in Plain English

The authors start from a linguistic intuition: when someone speaks, some factors stay roughly constant across a sentence or passage (the topic, intent, or "state" the speaker is in), while others change from word to word (grammar requirements, specific word choice). They formalize this by saying a language model's internal activation at each token is an invertible function of a high-level variable and a low-level variable, and they assume the high-level variable barely changes between nearby tokens.

They then build that assumption into SAE training. The SAE's features are split into a high-level block and a low-level block. The high-level block is trained to reconstruct the activation on its own, and the full set of features is trained to reconstruct the activation exactly — the same multi-level objective used by Matryoshka SAEs. On top of that, they load activations in pairs of adjacent tokens and shuffle the pairs within a batch, then add a contrastive loss (in the style of InfoNCE) that rewards the high-level latent of a token for being similar to that of its own predecessor and penalizes similarity to the predecessors of other sequences in the batch. This cross-sequence penalty is what prevents the trivial solution where high-level features simply stay constant for everything.

To test whether this works, they train on the Pile and then check three things. First, they probe the resulting features with sparse probes (k = 1, 5, 10, 20) and dense logistic regression to see whether semantic, contextual, or syntactic labels are recoverable, using MMLU question categories as semantic proxies, question IDs as context proxies, and spaCy part-of-speech tags as syntactic proxies. Second, they measure standard SAE health metrics, including a "smoothness" score defined as the average maximum absolute change of a feature across tokens, normalized by the change in model latents. Third, they run two downstream case studies — analyzing chosen versus rejected responses in an RLHF preference dataset, and steering model generations by activating features at inference time.

Why This Matters

• Impact on research. The paper reframes the well-known weakness of SAEs — a bias toward shallow syntactic features, dense activation behavior, and limited utility for steering — as a consequence of ignoring linguistic structure rather than a property of the models themselves. It also connects SAE research to a longer tradition of putting structural priors into dictionary learning (temporal smoothness for video, low-rank and multi-scale priors for images) and to work arguing that semantics exhibit long-range behavior while syntax depends on short-range interactions.
• Real-world applications:
  • Safety monitoring, where users may care primarily about the semantics of an output (the paper's example: whether it contains sexually-explicit content) rather than surface form.
  • Poetry and code generation, where users may instead care about local features such as vowel sounds or closing brackets — the low-level split that T-SAEs keep separate.
  • Auditing and curating human preference and alignment datasets, illustrated by the HH-RLHF study that surfaced both expected safety concepts and an unexpected length/verbosity spurious correlation.
  • Inference-time intervention and model control, where the reported steering gains make SAE-based interventions more practical than previously.
• Industry relevance. Teams that use SAEs as an off-the-shelf tool for monitoring, red-teaming, or steering deployed models would benefit from features that remain stable across a passage instead of flickering per token, since stable features are far easier to aggregate, audit, and turn into interventions. The authors note that SAEs are especially useful for surfacing failure modes that were not anticipated and for computationally efficient inference-time intervention, where supervised alternatives like probing or finetuning offer less leverage.

Future Directions

• Whether long-range contrastive targets help for particular goals. The paper already shows that contrasting against a random earlier token (t−r with r < 25) instead of the immediately preceding token trades syntax performance for context performance, so the right pairing scheme appears application-dependent and is not resolved.
• Extension beyond the two models tested. All results are on Pythia-160m (layer 8) and Gemma2-2b (layer 12), leaving open how the temporal loss behaves at other layers and in larger models.
• Explaining feature rollover. The authors observe "leakage" in which features keep firing into later, semantically unrelated sections of a spliced sequence, suggesting models retain past context — an effect the paper documents but does not fully characterize.
• Broader downstream evaluation. The paper notes that standard SAE metrics often do not predict usefulness for downstream applications, and its evidence for practical benefit rests on two case studies (preference data analysis and steering across 30 features); more tasks and more robust evaluation would strengthen the case.

Target Audience

Interpretability researchers working on SAEs, dictionary learning, and feature disentanglement; mechanistic interpretability engineers who train or select SAEs for monitoring, auditing, and steering; and NLP researchers interested in how linguistic assumptions about syntax, semantics, and locality can be injected as inductive biases into unsupervised representation learning. Readers needing the full theoretical framing should note that Section 3 defines the assumptions formally, and that the truncated content provided ends mid-sentence in the Discussion section.

Authors’ abstract

Translating the internal representations and computations of models into concepts that humans can understand is a key goal of interpretability. While recent dictionary learning methods such as Sparse Autoencoders (SAEs) provide a promising route to discover human-interpretable features, they often only recover token-specific, noisy, or highly local concepts. We argue that this limitation stems from neglecting the temporal structure of language, where semantic content typically evolves smoothly over sequences. Building on this insight, we introduce Temporal Sparse Autoencoders (T-SAEs), which incorporate a novel contrastive loss encouraging consistent activations of high-level features over adjacent tokens. This simple yet powerful modification enables SAEs to disentangle semantic from syntactic features in a self-supervised manner. Across multiple datasets and models, T-SAEs recover smoother, more coherent semantic concepts without sacrificing reconstruction quality. Strikingly, they exhibit clear semantic structure despite being trained without explicit semantic signal, offering a new pathway for unsupervised interpretability in language models.