LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs

AI summary

Overview

Research area: Multimodal interpretability — specifically, understanding what visual tokens look like inside a large language model after a vision encoder's outputs have been projected into it.

Technical level: Advanced. The paper assumes familiarity with transformer internals, embedding and unembedding matrices, vision-language model connectors, and existing interpretability tools such as LogitLens.

Scope: The paper introduces LatentLens, a training-free method that describes visual token representations by retrieving their nearest neighbors from a large precomputed pool of contextualized text representations, and uses it to re-examine how interpretable visual tokens are across 15 vision-language models.

What This Paper Is About

Transforming a frozen LLM into a vision-language model can be as simple as training a shallow MLP or linear layer that maps visual features into the LLM's embedding space, but it is unclear why this works so easily or what the resulting visual tokens actually represent inside the model. Prior interpretability methods (EmbeddingLens, which compares representations to the embedding matrix, and LogitLens, which decodes through the unembedding matrix) suggest visual tokens are rarely interpretable, because they only offer single sub-word tokens as candidate descriptions. This paper argues those methods are the wrong lens and proposes comparing visual token representations to contextualized token representations taken from full sentences at intermediate LLM layers.

Key Contributions

1. LatentLens, a new training-free interpretability method that describes a visual token representation at any layer by retrieving its top-k nearest neighbors from a precomputed index of contextualized token representations drawn from 2.99M Visual Genome captions, encoding each LLM once and storing representations for layers {1, 2, 4, 8, 16, 24, L-2, L-1}.
2. A unified framework showing EmbeddingLens and LogitLens are special cases of the same three-step procedure (score, select top-k, return descriptions), differing only in their candidate description set — the model vocabulary — which exposes two limitations: descriptions are limited to sub-word tokens, and all layers are compared against the same fixed reference vectors.
3. A large-scale empirical comparison across 15 VLMs (9 controlled model combinations plus 6 off-the-shelf models), using GPT-5 as an automated judge, showing that standard lenses substantially underestimate visual token interpretability.
4. The "Mid-Layer Leap" finding: visual token representations at early layers and even at the input align most strongly with contextualized text representations from middle layers (for example layers 8–16), rather than with representations at the same layer.

Main Findings

• Interpretability is much higher than previously reported: Averaged across all 15 VLMs and all layers, LatentLens renders 68% of visual tokens interpretable according to the VLM judge, versus 32% for EmbeddingLens and 24% for LogitLens.
• Consistency across layers and models: In the controlled setup, LatentLens gives roughly 60–85% interpretable visual tokens across all layers, with only minor variation between model combinations; some OLMo variants drop to around 53% at the final layer.
• EmbeddingLens is model-dependent: In the controlled setup, 34–62% of visual tokens are interpretable from the input layer onward for OLMo-based variants, less than 20% for Qwen2-based models, and 20–50% for Llama3-based models.
• LogitLens fails early but improves late: Fewer than 25% of visual token representations are labeled interpretable at lower layers for all models; at later layers most models reach 60–80% (for example all OLMo-based models at layers 24 and 30), with exceptions being Llama3 + DINOv2, Llama3 + SigLIP, and Qwen2 + SigLIP.
• TunedLens does not help: A version of LogitLens with learned per-layer affine probes (Tuned Lens) does not improve visual token interpretability.
• Ablations hold up: Reducing the projection to a linear mapping, or training on much shorter captions, does not substantially change LatentLens interpretability.
• DINOv2 is interpretable despite no language supervision: Visual tokens from DINOv2, which was pre-trained without textual supervision, show consistently high interpretability under all three lenses — though DINOv2-based models score lower on captioning quality.
• The Mid-Layer Leap: With OLMo-7B + SigLIP, the majority of nearest-neighbor contextualized representations for layer-0 visual tokens come from layer 8; for Qwen2-7B + SigLIP, the most similar representations for visual tokens from any layer are always from layer 16. A diagonal (same-layer) pattern only appears once visual tokens reach mid-layers such as layer 8.
• Visual token representations change little across layers, and rogue dimensions are not systematically responsible: Additional analyses found visual token representations change very little throughout layers, and no systematic evidence for rogue dimensions dominating cosine similarity.
• Results transfer to off-the-shelf VLMs: LatentLens achieves the highest interpretability on all six off-the-shelf models. Molmo-7B-D reaches 86% and Molmo-72B 78% on average (consistent with their OLMo backbone matching the controlled setup); Qwen2-VL-7B-Instruct and LLaVA-1.5-7B reach 55–62% on average; the larger Qwen2.5-VL-32B-Instruct and LLaVA-NeXT-34B are lower at 33–35%. EmbeddingLens and LogitLens typically reach 11–35% average interpretability, with the exception of EmbeddingLens on Molmo-72B (70%).
• Modest corpus needs: Using 1% of the contextual embedding pool yields 67.3% average interpretable tokens versus 71.6% for the full corpus; 0.1% drops to 58.4%; 10% gives 72.5%. Below 1%, interpretability drops meaningfully, and the difference between 1% and the full corpus is small (67% vs. 72% on average, as stated in the text).
• Qualitative descriptions are richer: LatentLens returns full-word and sentence-level descriptions, for example "large tower with clocks" and reconstructing "b + elf + ry" into "belfry". LogitLens tends to return subwords, punctuation, unrelated non-English (Chinese) tokens, and next-token predictions — for example predicting "es" or "potato" where LatentLens predicts "couch", or "Tomato" on rendered text. For LatentLens, the magnitude of cosine similarity typically increases at deeper layers; for LogitLens, higher logit similarity does not necessarily mean a more interpretable description.

Methodology in Plain English

The idea. Instead of asking "which vocabulary word is this visual token closest to?", the authors ask "which word as it actually appears in a sentence is this visual token closest to?" Because a token's representation inside a layer already incorporates context, the natural comparison is other in-context representations.

Building the reference pool. They take 2.99M Visual Genome captions, run each caption through the LLM, and save every token's contextualized representation at eight layers, capped at 20 different contextual representations per vocabulary token due to memory limits. This is a one-time cost per LLM: the forward pass over approximately 3M sentences takes about 2 hours of GPU compute and around 13 hours of total wall time including index building, with roughly 26 GB of storage across 8 layers in float8. Once loaded, nearest-neighbor search takes about 29 ms per image.

Retrieving descriptions. For a visual token at a given layer, they compute cosine similarity against every stored reference vector — which may come from a different layer than the visual token — and return the sentences containing the top-k matches.

Controlled training setup. To avoid confounds, they train projections themselves: three LLMs (OLMo-7B, Qwen2-7B, LLaMA3-8B) × three vision encoders (CLIP-ViT-L/14-336, DINOv2-L-336, SigLIP-so400M-patch14-384) = 9 combinations. The projector is a 3-layer MLP; all other weights stay frozen. Training uses the PixMo-Cap dataset (captions averaging 167 words and 9 sentences each), 12K steps, effective batch size 8, with the patchified image fed directly to each model. Caption quality is verified with DCScore, a GPT-4o-based judge rating captions 1–10: their models average 6.0/10, with CLIP and SigLIP encoders averaging 6.8 and DINOv2 models averaging 4.4; off-the-shelf Qwen2-VL-7B-Instruct scores 8.5/10.

Judging interpretability. GPT-5 is shown an image with a red bounding box on the target token plus the 8 surrounding visual tokens, along with the top-5 descriptions, and asked to label each as concrete (directly visible), abstract (conceptually related), or global (present elsewhere in the image). A visual token is interpretable if at least one top-5 description is interpretable. The authors validated the judge against human annotations across all three methods on 1,020 instances and found substantial agreement with Cohen's κ = 0.68. Although LatentLens retrieves full sentences, only full words from the top-5 contextualized tokens are fed to the judge — for fairness against EmbeddingLens and LogitLens, and because the judge was found to be distracted by sentence-level context.

Evaluation scale. For the main controlled comparison, they sample 100 image patches from 100 PixMo-Cap validation images, giving 3 methods × 100 patches × 9 models × 9 layers = 24.3K judge calls. They then repeat the analysis on six off-the-shelf VLMs (Molmo-7B-D, Molmo-72B, LLaVA-1.5-7B, LLaVA-NeXT-34B, Qwen2-VL-7B-Instruct, Qwen2.5-VL-32B-Instruct), plotting against normalized layer depth.

Why This Matters

The paper reframes a negative result — "visual tokens aren't interpretable" — as a measurement artifact. If the majority of visual tokens are in fact interpretable across layers, then the barrier to understanding and controlling multimodal models is partly a matter of choosing the right comparison space, not of fundamental opacity.

• Debugging and auditing VLMs: Higher-fidelity per-layer descriptions of what a visual token encodes can support inspection of where a model is picking up the wrong visual evidence.
• Hallucination mitigation: Prior work already uses LogitLens-style decoding to reduce hallucinations; this paper's evidence that LogitLens returns plausible next-token predictions rather than what a token intrinsically encodes suggests those pipelines may be reading the wrong signal.
• Efficient multimodal training: The finding that a linear projection is sufficient for interpretable visual tokens, and that 1% of the reference corpus (roughly 250 MB of storage) yields comparably strong interpretability, lowers the practical cost of building such tooling.
• Cross-modal representation research: The Mid-Layer Leap is direct evidence that the learned projection targets semantic rather than lexical representations, informing debates about shared structure between vision and language spaces (for example the Platonic Representation Hypothesis).

Industry relevance: The method is training-free and ships with a pip package and an interactive demo, making per-layer visual token inspection feasible as part of model development and evaluation workflows rather than only as an academic analysis.

Future Directions

• Explain the layer-depth trend in larger models: The authors note it will be interesting for future work to explore why larger models (Qwen2.5-VL-32B-Instruct and LLaVA-NeXT-34B at 33–35%) have less consistent interpretability trends across layers.
• Quantify the sentence-level advantage: The current judge setup deliberately withholds sentence context to keep the comparison fair; the authors state that this may underestimate LatentLens's interpretability and quantify the full-sentence benefit separately.
• Scale and diversify the description corpus: The method is described as trivially extendable from sentences to phrases or paragraphs, and corpus-size sensitivity is only ablated at three model pairs and four layers.
• Understand the Mid-Layer Leap mechanistically: The paper documents the phenomenon and rules out two explanations (large representation drift across layers and rogue dimensions) but does not identify the mechanism that makes early visual tokens resemble mid-layer text representations.

Target Audience

Interpretability researchers working on transformers and multimodal models; VLM engineers who need to inspect what a model's visual tokens encode at each layer; and researchers studying cross-modal representation alignment, who will find the Mid-Layer Leap and the DINOv2 results directly relevant. Readers need working familiarity with LLM internals (embedding/unembedding matrices, layer-wise hidden states) and with prior lens-based interpretability methods to follow the comparisons.

Note: the provided paper content is truncated at the start of the Discussion and Conclusion section, so the paper's stated conclusions and any limitations section are not reported here.

Authors’ abstract

Transforming a large language model (LLM) into a vision-language model (VLM) can be achieved by mapping the visual tokens from a vision encoder into the embedding space of an LLM. Intriguingly, this mapping can be as simple as a shallow MLP transformation. To understand why LLMs can so readily process visual tokens, we need interpretability methods that reveal what is encoded in the visual token representations at every layer of LLM processing. In this work, we introduce LatentLens, a novel approach for mapping latent representations to descriptions in natural language. LatentLens encodes a large text corpus and stores contextualized token representations for each token in that corpus. Visual token representations are then compared to these contextualized representations and the top-nearest neighbor representations serve as descriptions of the visual token. We evaluate this method on 15 different VLMs, showing that commonly used methods, such as LogitLens, substantially underestimate the interpretability of visual tokens. With LatentLens instead, the majority of visual tokens are interpretable across all studied models and all layers. Qualitatively, we show that the descriptions produced by LatentLens are semantically meaningful and provide more fine-grained interpretations for humans compared to individual tokens. More broadly, our findings contribute new evidence on the alignment between vision and language representations and open up new directions for analyzing the latent representations of LLMs.