The Deleuzian Representation Hypothesis

AI summary

Overview

Research area: Mechanistic interpretability of neural networks — specifically unsupervised extraction of human-interpretable concepts from model activations. Adjacent to sparse autoencoder (SAE) research, linear probing, and representational analysis.

Technical level: Intermediate. The method itself is conceptually simple (sampling activation differences + weighted KMeans), but the paper assumes familiarity with sparse autoencoders, linear discriminant analysis, probe-based evaluation, and the activation-space steering literature.

Scope (one sentence): The paper proposes and evaluates an unsupervised concept-extraction method that clusters pairwise differences between activations rather than reconstructing activations, and tests it across five models, five datasets, three modalities, and 874 attributes.

What This Paper Is About

Sparse autoencoders are the dominant tool for pulling interpretable features out of neural network activations, but they are hard to train, can still produce polysemantic features, and rely on sparsity as a proxy for interpretability — a choice the authors note has been criticized. This paper asks whether a simpler, clustering-based alternative can recover concepts of comparable or better quality without any reconstruction objective. The authors ground the idea in Gilles Deleuze's philosophical view of concepts as differences rather than universals, and formalize it as an unsupervised form of discriminant analysis.

Key Contributions

1. An unsupervised concept-extraction method based on activation differences. Instead of reconstructing activations, the method samples pairwise differences between sample activations, clusters them with KMeans, and treats the resulting centroids as concept vectors. The number of concepts k is the only hyperparameter, and it is itself interpretable.

2. Skewness-based reweighting for diversity. The authors weight the KMeans clustering by the inverse skewness of each difference direction, turning it into a variant of Feature-Weighted KMeans. This is intended to prevent highly skewed, spiky directions from dominating Euclidean distance and producing redundant clusters.

3. A theoretical connection to discriminant analysis. The paper shows that treating activation differences as concept directions is equivalent to assuming isotropic distributions in activation space, and that this framing avoids the homoscedasticity and Gaussianity assumptions of standard LDA.

4. A large-scale, multi-modal evaluation plus lossless steering. The method is benchmarked against six SAE variants, ICA, pretrained SAEs, and a supervised LDA baseline across three modalities, and the concepts are shown to causally affect model outputs via steering — including a steering procedure that avoids the reconstruction error introduced by SAE-style projection.

Main Findings

• Concept quality surpasses unsupervised SAE variants. In Table 1, the method achieves the lowest probe loss on 13 of the 20 tested tasks, and the best average rank among unsupervised methods: 1.65 ± 0.85, ahead of Tk-SAE (2.65 ± 1.01), A-SAE (3.20 ± 1.72), Van-SAE (4.65 ± 1.56), Mat-SAE (5.70 ± 1.90), JR-SAE (5.75 ± 0.94), Gat-SAE (6.65 ± 1.42), and ICA (6.85 ± 2.29).

• It approaches supervised LDA in several settings. The paper reports that probe loss is in some cases midway between supervised LDA and the next-best unsupervised method. Notably, LDA performs poorly on BART over CoNLL-2003, which the authors attribute to LDA's extra assumptions (normal distribution and homoscedasticity) not holding in that case.

• Statistical significance of the quality gain. On AST/AudioSet, the method better recovers at least 366/527 attributes (69.4%) than competing methods, with a Wilcoxon signed-rank statistic of 106584 and p-value 1.7 × 10⁻²⁶. On CLIP/WikiArt, it beats TopKSAE on 140/167 attributes (83.8%), with statistic 12671 and p-value 7.9 × 10⁻²⁰.

• Consistency is competitive. Using Maximum Pairwise Pearson Correlation (MPPC) averaged over 10 runs with different seeds, the method is generally more consistent than the alternatives except VanillaSAE (which the authors note reaches much lower concept quality and diversity). Reported values include CLIP/ImageNet 0.821, CLIP/WikiArt 0.856, DinoV2/ImageNet 0.789, DinoV2/WikiArt 0.843, DeBERTa/IMDB 0.980, DeBERTa/CoNLL 0.588, BART/IMDB 1.0, BART/CoNLL 0.768, and AST/AudioSet 0.830.

• Concepts have causal influence. Steering CLIP toward zero on a Romanticism concept while boosting an Abstract paintings concept shifts a painting's nearest neighbors accordingly; steering a country-name concept in BART with α = +5 and α = −5 replaces "Rio de Janeiro" with "February" and "country" with "city", and positive steering evokes country names (frequently the United States) even without geographic context — which the authors flag as a potential bias.

• Skewness weighting matters for diversity. In the ablation (CLIP/WikiArt, DeBERTa/CoNLL NER), the full method reaches probe loss 0.0119 / 0.0665, effective rank 124.4 / 182.0, and maximum pairwise cosine 0.5677 / 0.3908 — versus unweighted KMeans on differences at 0.0128 / 0.0841, effective rank 17.9 / 5.65, and cosine 0.6504 / 0.8357. Representing differences (rather than activations) and using clustering (rather than an SAE) both contributed.

• Compactness. Only 2000 concepts are needed to outperform every concurrent method on the CLIP/WikiArt artist task, versus the 6144 used in the main experiments.

Methodology in Plain English

The researchers take a trained encoder, feed it data, and record the activations at a chosen layer.

1. Sample differences. Computing every pairwise difference is quadratic in dataset size, so they instead randomly sample N pairs while ensuring each data point appears once on each side of the subtraction. This gives N difference vectors.

2. Cluster the differences. They run KMeans on these difference vectors to find recurring patterns. The resulting cluster centroids become the concept vectors — directions in activation space that consistently separate samples from one another.

3. Reweight by skewness. Some difference directions sit near zero for most samples and spike occasionally. Those spikey directions dominate Euclidean distance and cause redundant clusters. They compute the skewness (normalized third central moment) of each direction's projection onto the samples, flip the sign of negatively skewed directions, and weight the clustering inversely to skewness.

4. Evaluate. They measure probe loss: for each ground-truth attribute, train a one-dimensional logistic probe on each extracted concept and record the lowest cross-entropy loss. They also measure cross-run consistency (MPPC over 10 seeds) and effective rank and maximum pairwise cosine for diversity. Steering is tested qualitatively on CLIP and BART.

All experiments use concept spaces of 6144 dimensions (8× the activation dimension), except ICA, which is limited to 768. TopKSAEs use k = 32 and learning rate 10⁻⁵; L1 coefficients were selected from a 7-value sweep between 10⁻⁹ and 10⁻³ (10⁻⁸ for VanillaSAE, 10⁻⁶ for GatedSAE, 10⁻⁵ for JumpReLUSAE). MatryoshkaSAE used group sizes [512, 1024, 1536, 3072]. WikiArt, which has no official split, uses even samples for training and the rest for testing.

Why This Matters

Impact on research. The paper challenges the premise that sparse reconstruction is the right objective for concept extraction. If simple clustering of activation differences can match or beat SAE variants on probe loss while using a single interpretable hyperparameter, then a substantial amount of SAE training complexity may be avoidable. The connection to discriminant analysis also gives the method a cleaner theoretical footing than reconstruction-plus-sparsity heuristics, and the lossless steering property avoids the information loss inherent in projecting into and out of a concept basis.

Real-world applications.

• Style transfer and image editing: steering image-encoder representations toward a target artistic style, as demonstrated with the Romanticism-to-Abstract example in CLIP.
• Bias detection and auditing: the BART country-name experiment surfaced a tendency to mention the United States, an example of how steering probes can expose model biases.
• Controllable text generation: suppressing or amplifying semantic attributes in an encoder before decoding, shown by replacing "Rio de Janeiro" with "February".
• Domain adaptation at scale: because steering is lossless and directly reversible, the authors suggest interventions could be applied broadly to shift a model toward a specific domain.

Industry relevance. Teams deploying or auditing large models need interpretability tools that are cheap, stable across runs, and scalable. This method is fully unsupervised, runs in linear time and memory with respect to dataset size and activation dimension, and is governed by one hyperparameter — all attractive properties for production interpretability pipelines. The open code is available at the authors' GitHub repository.

Future Directions

1. Automatic concept naming. The method finds repeated directions but does not label them. The authors state that a method to automatically name or interpret the extracted concepts would greatly enhance the scope and applicability of the findings, enabling analyses across more datasets, modalities, and models.

2. Label-free evaluation. Current evaluation depends on labeled datasets, so meaningful but subtle concepts that do not align with available labels incur high probe loss. The authors note that a theoretically justified proxy for interpretability without labels is still lacking, and that sparsity alone does not satisfy this criterion.

3. Higher-dimensional concept spaces. All main evaluations use 6144 dimensions (8× activation dimension), except ICA and an ablation. The authors note that increasing dimensionality further could bias evaluation given the limited number of attributes and samples, but could also reveal additional characteristics of concept extraction methods.

4. Beyond linear concepts. The approach assumes concepts are linear projections. This is empirically validated across the five studied models, but the authors acknowledge a model with representations violating this assumption could exist and would require adapting the method. They also derive a quadratic extension accounting for anisotropic concept distributions in Appendix G, though it did not improve experimental results.

Target Audience

Researchers working on mechanistic interpretability, sparse autoencoders, and feature extraction from neural networks; machine learning practitioners who need unsupervised, scalable tools for auditing and steering model representations; and readers interested in the intersection of continental philosophy (Deleuze's account of concepts as differences) and representation learning. It is also relevant to NLP and computer vision researchers who evaluate linear probes over hidden states, since the paper offers a direct comparison against LDA, ICA, and six SAE variants across text, image, and audio encoders.

Authors’ abstract

We propose an alternative to sparse autoencoders (SAEs) as a simple and effective unsupervised method for extracting interpretable concepts from neural networks. The core idea is to cluster differences in activations, which we formally justify within a discriminant analysis framework. To enhance the diversity of extracted concepts, we refine the approach by weighting the clustering using the skewness of activations. The method aligns with Deleuze's modern view of concepts as differences. We evaluate the approach across five models and three modalities (vision, language, and audio), measuring concept quality, diversity, and consistency. Our results show that the proposed method achieves concept quality surpassing prior unsupervised SAE variants while approaching supervised baselines, and that the extracted concepts enable steering of a model's inner representations, demonstrating their causal influence on downstream behavior.