"World Knowledge" in the Weights: Reading Concept Circuits of Vision Transformers

AI summary

Overview

• Research area: Mechanistic interpretability of vision transformers (computer vision / interpretable machine learning)
• Technical level: Advanced
• Scope: The paper adapts Cross-Layer Transcoders (CLTs) to vision transformers to extract "concept circuits" — interpretable graphs of features and their cross-layer interactions — and demonstrates their use for auditing spurious correlations, steering model predictions, and comparing foundation models.

What This Paper Is About

Vision transformers (ViTs) generalize well across visual tasks, but researchers still do not know what internal concepts these models encode or how those concepts interact across layers. Existing interpretability tools mostly highlight where a model looks in an image rather than how it reasons about what it sees. This paper proposes a method for reading structured "concept circuits" out of a ViT's weights, exposing a reusable, input-independent body of "world knowledge" as well as the specific concept pathways used for individual predictions.

Key Contributions

1. Adaptation of CLTs to vision transformers. The authors modify Cross-Layer Transcoders — originally designed for language models — to handle vision-specific challenges such as CLS tokens, the large number of image tokens, and hyperparameter selection, and validate the adaptation on ImageNet.
2. Two-level reading of concept circuits. They introduce both global concept circuits (input-invariant graphs read directly from learned encoder/decoder weights) and instance concept circuits (input-dependent subgraphs showing which concepts actually drove a specific prediction), the former being the first of its kind.
3. Automatic spurious correlation discovery. Using the joint statistics of global edge weights and feature co-activations, the method identifies shortcut concepts the model relies on without requiring human annotation or domain knowledge.
4. Downstream applications. The framework is demonstrated on spurious-correlation removal (outperforming prior methods by 11.0% on Waterbird's worst group) and on structural comparison of foundation models trained under different supervision paradigms (CLIP vs. DINO vs. supervised ViT).

Main Findings

• Global circuits are causally faithful. Mean-ablating the top-10 concepts of a class's global concept circuit causes large accuracy drops on ImageNet (CLIP falls from 0.60 to 0.22, ViT from 0.70 to 0.44, DINO from 0.76 to 0.62), while ablating randomly selected concepts barely affects performance.
• Co-activation reweighting is necessary. Raw CLT weights alone produce interference between features that rarely co-activate; reweighting by empirical co-activation statistics yields circuits whose top edges are far more functionally important.
• CLT circuits beat SAE circuits. Across CLIP, DINO, and supervised ViT, CLT-based instance concept circuits achieve higher faithfulness and lower completeness than Sparse Autoencoder baselines — by 24.4% and 22.1% respectively on CLIP.
• Concept abstraction follows layer depth. Instance circuits for a "School Bus" image show features evolving from colors and textures (blue, grid) to object parts (wheels, windows) to whole objects (working vehicle, school bus).
• Spurious correlations can be found automatically. On ImageNet ViT-B/16, the method recovers shortcuts previously reported in human-inspection studies ("graffiti" for Freight Car, "bird feeder" for Hummingbird), and separates three concept types: spurious (high global weight, low co-activation), non-discriminative (high co-activation, low weight), and causal (both high).
• Steering works better with instance circuits. Ablating background-specific concepts on Waterbird raises worst-group accuracy from 0.48 to 0.71, versus 0.60 for SpLiCE, 0.53 for SAE, and 0.56 for BatchTopKSAE.
• Supervision paradigm shapes circuit topology. CLIP shows dense deep-layer interactions (explaining its strong generalization), DINO concentrates connections in shallow and middle layers (explaining its strength as a dense-prediction backbone), and supervised ViT relies on adjacent-layer flow so abstract class concepts emerge early (e.g., "cat" by layer 8).

Methodology in Plain English

The researchers train a small auxiliary network called a Cross-Layer Transcoder on top of a frozen ViT. This transcoder compresses each layer's internal representation into a sparse set of activations — only a few dozen "features" fire for any given image — and then tries to reconstruct the outputs of every layer from those features. Because the reconstruction is trained to be accurate while keeping activations sparse, each feature tends to correspond to a single human-interpretable concept (a color, a texture, an object part, a whole object). Crucially, the transcoder's encoder and decoder weights are fixed after training, so they describe how concepts connect across layers independently of any particular image. Summing these weight paths gives the global concept circuit, and reweighting each edge by how often its two features actually co-activate removes spurious weight artifacts. For a single image, the same machinery is run forward and gradient attribution is used to compute how strongly each active feature influences the next, producing a much smaller instance circuit after aggregating across patch positions and pruning low-importance nodes and edges.

Why This Matters

The work moves interpretability from "which pixels mattered" toward "which concepts the model uses and how it chains them together," giving a mechanistic substrate for auditing, debugging, and comparing vision models. It also reframes spurious correlation detection as a structural property of internal weights rather than something requiring human inspection of saliency maps, which is a meaningful scalability improvement.

Real-world applications:

• Medical imaging and autonomous driving. Safety-critical domains need interpretable failure diagnosis when a model latches onto scanner artifacts or road-surface shortcuts instead of the actual clinical or driving signal.
• Bias auditing and fairness compliance. Automated shortcut discovery supports regulatory review of deployed vision systems without requiring per-domain expert annotation.
• Model selection and procurement. Practitioners can compare candidate foundation models by their internal concept structure, not just benchmark accuracy, to pick the right backbone for a dense-prediction or classification task.
• Dataset and training-pipeline debugging. Spurious concepts identified in the circuit point directly at dataset artifacts, informing data curation or augmentation strategies.

Industry relevance centers on the growing deployment of pretrained ViT backbones (CLIP, DINO, and their derivatives) in products where robustness and accountability matter. A weight-level readout of "what the model believes" is a more tractable audit target than black-box probing.

Future Directions

• Extending beyond MLP pathways. CLTs primarily approximate MLP computation and reflect attention structure only indirectly; a transcoder that also decomposes attention would give a more complete circuit.
• Reducing reliance on human inspection. Concept labeling currently requires looking at top-activating images; automated labeling via captions or text encoders would make large-scale auditing practical.
• Strengthening causal claims. The authors note that global graphs do not fully capture causal relations, so integrating intervention-based validation for global edges is a natural follow-up.
• Broader architecture and modality coverage. Applying the framework to convolution-based models, video transformers, and multimodal encoders would test whether the observed supervision-dependent topology patterns generalize.

Target Audience

Mechanistic interpretability researchers, computer vision scientists studying vision transformer internals, and machine learning engineers working on robustness, bias mitigation, or model selection for deployed vision systems. Readers should be comfortable with transformer architecture, sparse dictionary learning, and attribution methods; the paper assumes familiarity with SAEs, residual streams, and circuit-faithfulness metrics.

Authors’ abstract

Vision transformers (ViTs) have achieved remarkable generalization across visual domains, yet little is known about how they internally represent the structure of the world. To address this gap, we use Cross-Layer Transcoders (CLTs) to read concept circuits from ViTs: directed graphs whose nodes correspond to sparse, interpretable concepts and edges capture concept interactions across layers. Our method yields two complementary views of model behavior. The global concept circuit is input-invariant and can be recovered directly from learned cross-layer weights, exposing the reusable "world knowledge" encoded in the model. The instance concept circuit is input-dependent and identifies the concepts and pathways actually used for a specific prediction, enabling faithful example-level explanations. We demonstrate the utility of concept circuits in three ways: (1) Automatic spurious correlation discovery: leveraging the statistics of our global concept circuits to identify shortcut dependencies within the model. (2) Spurious correlation removal: intervening on the instance concept circuit to steer the model towards correct predictions. Empirical results show that our method outperforms existing counterparts by 11.0% on the Waterbird dataset. (3) Model comparison: contrasting the global concept circuits of different foundation models (e.g., CLIP vs. DINO) to reveal how supervision paradigms shape representational structure. Our code is available at https://github.com/deep-real/VisionCLT