Vision-aligned Latent Reasoning for Multi-modal Large Language Model

AI summary

Overview

• Research area: Multi-modal large language models (MLLMs), specifically latent-space reasoning, Chain-of-Thought (CoT) reasoning, and test-time scaling for vision-language models.
• Technical level: Intermediate. The paper assumes familiarity with transformer decoders, vision encoders, and supervised fine-tuning, but its central idea (inserting "visual checkpoints" during reasoning) is conceptually simple.
• Scope: The paper proposes Vision-aligned Latent Reasoning (VaLR), a two-stage training framework that generates vision-aligned latent tokens before each Chain-of-Thought step, and evaluates it on 3D spatial reasoning and perception benchmarks.

What This Paper Is About

Multi-modal large language models lose visual information as they generate longer chains of reasoning, so they cannot take advantage of test-time scaling the way text-only LLMs can. The authors' goal is to keep the model grounded in the image throughout a long reasoning trace by generating latent (non-text) tokens before each reasoning step and training those tokens to match features from pretrained vision encoders. The result is a model that improves as it reasons longer, rather than degrading.

Key Contributions

1. Vision-aligned latent reasoning framework (VaLR): The model generates latent tokens before every Chain-of-Thought reasoning step, creating "visual checkpoints" that re-inject perceptual detail instead of relying on a single fixed visual context set at the start.
2. Representation alignment loss (REPA) for latent tokens: Intermediate hidden states of the MLLM are projected through a learnable MLP and aligned via patch-wise cosine similarity with dense features from pretrained vision encoders (DINOv2/v3, CLIP, SigLIPv2, and the 3D encoder π³). The external encoders are used only during training, not at inference.
3. Two-stage curriculum training pipeline: Stage 1 performs standard supervised fine-tuning on 450K CoT visual question-answering samples; Stage 2 inserts latent tokens before each reasoning step and adds the REPA objective.
4. Multi-encoder alignment extension: Latent tokens can be aligned to several frozen vision encoders simultaneously, with a separate projection head per encoder, allowing complementary 2D semantic/appearance and 3D spatial knowledge to be distilled into the latent reasoning space.

Main Findings

• Large gain on multi-view 3D reasoning: On VSI-Bench, VaLR raises Qwen2.5-VL-7B from 33.0% to 52.9% average accuracy, a 19.9 percentage-point gain. The single-encoder variant (VaLR-S, DINOv3-aligned) reaches 41.5%, and the multi-encoder variant (VaLR-M, DINOv3 + SigLIPv2 + π³) reaches 52.9%.
• Comparison to other latent reasoning methods on VSI-Bench: Monet reaches 14.0%, LVR 18.4%, and CoVT 18.6% average accuracy. The authors interpret this gap as evidence that latent reasoning without visual recall fails to maintain grounding on long reasoning traces.
• Comparison to API and reasoning models on VSI-Bench: GPT-4o scores 34.0%, LLaVA-NeXT-Video-7B 35.6%, Ocean-R1-7B 30.5%, and R1-OneVision-7B 16.1%. A vanilla SFT baseline on Qwen2.5-VL-7B reaches 33.7%.
• Sub-task strengths: VaLR-M reaches 66.4% on Object Counting, 64.2% on Object Size, 56.6% on Room Size, 50.0% on Relative Distance, and 40.6% on Absolute Distance, versus 40.9%, 43.4%, 20.7%, 38.6%, and 14.8% for the base model.
• Perception benchmark gains: VaLR-M scores 64.7 on BLINK, 60.3 on MMVP, 72.3 on MMStar, 86.9 on V*, and 87.6 on CVBench, versus 55.7, 56.0, 67.1, 76.4, and 74.5 for Qwen2.5-VL-7B. The paper reports that VaLR-M outperforms CoVT by 8.7 percentage points on BLINK and 8.9 percentage points on V*.
• Test-time scaling behavior: Across MathVista, MathVision, MMhalu, and MMVP, baselines peak at intermediate reasoning lengths and then degrade, while VaLR improves monotonically with reasoning length. On MMVP, Ocean-R1 drops from 62.7% to 56.5% at 300 tokens, whereas VaLR remains strong across all reasoning lengths. On MMhalu the reported hallucination rate is lower-is-better.
• Alignment matters more than the latent tokens alone: Removing visual alignment (VaLR w/o VA) gives 34.0% on VSI-Bench, barely above the base model's 33.0%. Aligning to Qwen's own native vision encoder (VaLR w/ QE) reaches 39.6% on VSI-Bench, 58.9 on BLINK, 60.0 on MMVP, 81.7 on V*, and 81.6 on CVBench, showing external encoders are not strictly required but help.
• Encoder-agnostic: Single-encoder alignment improves results with CLIP (BLINK 62.3, MMVP 59.3, MMStar 71.0, V* 83.2, CVBench 79.1), SigLIPv2 (62.8, 59.7, 71.3, 83.2, 81.9), DINOv2 (62.7, 60.0, 70.7, 83.8, 81.8), and DINOv3 (63.1, 60.3, 70.8, 86.4, 83.1).
• Encoder choice maps to task type: Adding the 3D-specialized encoder π³ drives most of the VSI-Bench improvement, while adding 2D encoders such as DINOv3 or SigLIPv2 improves perception benchmarks; using all three gives the best results across tasks (VSI-Bench 52.9, BLINK 64.7, MMVP 60.3, MMStar 72.3, V* 86.9, CVBench 87.6).
• Middle-layer alignment works best: Aligning at layer 12 (Middle) yields BLINK 63.1, MMVP 60.3, MMStar 70.8, V* 86.4, and CVBench 83.1, compared with layer 4 (Front) at 59.2, 55.7, 68.5, 83.8, 78.6 and layer 27 (Last) at 62.8, 60.0, 70.8, 85.3, 82.5.
• Data scalability: Testing 10K, 50K, 100K, 200K, and 450K samples, both VaLR variants improve with more data while vanilla SFT saturates beyond 200K samples. VaLR-M reaches comparable performance on V* with more than 20 times faster training than the vanilla SFT model.

Methodology in Plain English

The base model is Qwen2.5-VL-7B. In ordinary use, the model reads image tokens once at the start and then generates text. As the text gets longer, the influence of those original image tokens fades.

VaLR changes the generation loop in two ways. First, before each reasoning step, the model emits a fixed block of 16 special latent tokens, wrapped in <latent> and </latent> control tokens. These latent tokens are not words; instead of feeding the token embedding of the previous word, the model feeds the previous hidden state back in as the next input. This gives the model extra internal computation steps that are not constrained to language.

Second, during training, those latent hidden states are forced to stay visually meaningful. At every reasoning stage, the researchers take patch-level features from a frozen pretrained vision encoder for the relevant image, take intermediate hidden states from the MLLM's own latent tokens, upsample and project them with a small MLP, and penalize the cosine distance between the two sets of patch features. Intuitively, the latent tokens must learn to encode what the vision encoder sees.

Training proceeds in two stages. Stage 1 is standard supervised fine-tuning on 450K Chain-of-Thought VQA samples, which teaches the model to break problems into language reasoning steps. Stage 2 inserts the latent tokens before each step and adds the alignment loss on top of the standard next-token cross-entropy loss. The vision encoder and the MLLM's native vision tower are frozen; only the language decoder and the alignment MLP are trained. At test time the external vision encoder is discarded entirely — the model relies on what the latent tokens learned to represent.

For the multi-encoder version, the same alignment loss is computed separately against each of DINOv3, SigLIPv2, and π³, each with its own projection head, and averaged. Training used 4× NVIDIA Tesla A100s, with latent block size K = 16 and alignment weight λ = 0.5.

Why This Matters

Impact on research. The paper frames visual information decay during long-context generation as a fundamental blocker to test-time scaling in MLLMs. It offers a concrete mechanism — per-step latent visual checkpoints trained by representation alignment — and shows that this produces monotonic gains with reasoning length, a behavior the authors state is not observed in prior MLLMs. It also connects MLLM reasoning to the latent reasoning lineage in text-only LLMs and to external-encoder work such as PrismaticVLM, PaliGemma, and concurrent methods like CoVT and Monet.

Real-world applications (drawn from the paper's stated motivation):

• Vision-Language-Action (VLA) models for robotics, where long-horizon planning must stay grounded in what the camera sees.
• Computer-use agents (CUAs) that operate interfaces over many steps while tracking visual state.
• Spatial and 3D scene understanding assistants that must integrate information across multiple viewpoints, such as the VSI-Bench tasks (room size, relative distance, route planning, appearance order).
• General visual question answering and perception-heavy tasks such as BLINK, MMVP, MMStar, V*, and CVBench.

Industry relevance. The base model is a widely used open 7B MLLM, and the method adds only a small trainable MLP plus a modified generation loop. Because the external vision encoders are used only at training time, there is no added inference cost from them. The reported data efficiency — reaching comparable V* performance with more than 20 times faster training than vanilla SFT — is directly relevant to teams with limited compute budgets.

Future Directions

• Extension beyond static images to video and temporal reasoning. The paper evaluates on multi-view VSI-Bench but does not report results on video benchmarks; how latent checkpoints should be allocated across time is an open question.
• Combining latent visual alignment with reinforcement-learning post-training. The paper contrasts its supervised approach with RL-based MLLM reasoning methods in related work but reports no experiments combining the two.
• Understanding what the latent tokens actually encode. The alignment loss enforces correspondence with vision-encoder patch features, but the paper does not report an interpretability analysis of the learned latent representations.
• Scaling to larger base models and more encoders. All experiments use Qwen2.5-VL-7B and up to three ViT-L vision encoders; whether the gains hold at larger model scales or with additional modalities is not reported. The paper also does not include a dedicated limitations section.

Target Audience

Researchers and engineers working on multi-modal LLMs, latent-space reasoning, and vision-language training pipelines will get the most from this paper. It is also relevant to practitioners building long-horizon agents (robotics, computer use) who need models that stay visually grounded over extended reasoning, and to readers interested in the multi-modal analogue of test-time scaling. Readers should be comfortable with transformer internals and standard supervised fine-tuning concepts.

Authors’ abstract

Despite recent advancements in Multi-modal Large Language Models (MLLMs) on diverse understanding tasks, these models struggle to solve problems which require extensive multi-step reasoning. This is primarily due to the progressive dilution of visual information during long-context generation, which hinders their ability to fully exploit test-time scaling. To address this issue, we introduce Vision-aligned Latent Reasoning (VaLR), a simple, yet effective reasoning framework that dynamically generates vision-aligned latent tokens before each Chain of Thought reasoning step, guiding the model to reason based on perceptual cues in the latent space. Specifically, VaLR is trained to preserve visual knowledge during reasoning by aligning intermediate embeddings of MLLM with those from vision encoders. Empirical results demonstrate that VaLR consistently outperforms existing approaches across a wide range of benchmarks requiring long-context understanding or precise visual perception, while exhibiting test-time scaling behavior not observed in prior MLLMs. In particular, VaLR improves the performance significantly from 33.0% to 52.9% on VSI-Bench, achieving a 19.9%p gain over Qwen2.5-VL.