Mull-Tokens: Modality-Agnostic Latent Thinking

AI summary

Overview

Research area: Multimodal reasoning for vision-language models (computer vision / multimodal language models).

Technical level: Intermediate. The paper builds on latent reasoning and chain-of-thought concepts, but its central idea (a small set of special "thinking" tokens) can be understood without deep background.

Scope: The paper proposes and evaluates "Mull-Tokens," a set of modality-agnostic latent tokens that let a multimodal language model carry out intermediate reasoning internally, without explicitly decoding intermediate text or images, and tests them on spatial reasoning benchmarks.

What This Paper Is About

Multimodal models struggle with visual and spatial reasoning: text-only chain-of-thought drifts away from the visual input, while approaches that "think with images" require specialist tools, costly image generation, or hand-crafted datasets, and can be brittle. The paper's goal is a simpler alternative: a handful of special latent tokens that the model can fill with intermediate information in either image or text form, and that it uses as an internal scratchpad on the way to the answer. The claim is that these tokens improve accuracy on spatial reasoning tasks while requiring far less computation than verbose text reasoning or explicit image generation.

Key Contributions

1. Modality-agnostic latent thinking tokens. The paper introduces Mull-Tokens, a fixed-length sequence of special latent tokens z_{1:K} = (<Mull>_1, ..., <Mull>_K) that are not tied to either the text vocabulary or image features. They serve as internal compute and are never required to be decoded into readable text or images.

2. A three-stage training recipe. Stage 1 warms up the tokens with supervision from interleaved text-image reasoning traces, Stage 2 relaxes supervision and trains only on final answers so the tokens are optimized freely toward end-task accuracy, and Stage 3 refines the causal utility of the latent chain using GRPO.

3. Empirical comparison against text-only and interleaved image-text reasoning. Across BLINK, SAT-Real, VSI-Bench and ERQA, the authors report that Mull-Tokens outperform direct-answer fine-tuning, text chain-of-thought, GRPO-refined text reasoning, and an interleaved image-text latent baseline (which they describe as replicating a recent approach).

4. Ablations isolating why the method works. The paper shows that multimodal warm-up (image plus text) is essential, that discrete latent tokens beat continuous recurrent embeddings, and that the number of latent tokens at inference has an optimum beyond which performance degrades.

Main Findings

• Overall gain of +3% and up to +16%. On average, Mull-Tokens reach a +3% absolute accuracy gain over the strongest baseline, which the authors state is obtained by fine-tuning the base model directly on answers without any chain-of-thought. The largest gain reported is +16% on a puzzle-solving reasoning split (the authors report +16.01% on the BLINK Jigsaw split relative to direct-answer fine-tuning, in the row with GRPO).

• Text chain-of-thought hurts relative to direct-answer tuning. Even though text reasoning improves over the base model, it underperforms direct-answer fine-tuning (average 48.90 for TextCoT FT and 48.50 after GRPO, versus 50.87 for DirAns FT). The paper notes a nuance: text reasoning does help on some reasoning-heavy splits, such as BLINK Jigsaw and the reasoning average (by about 2.1%) and VSI-Bench reasoning (by about 0.4%).

• Interleaving image and text thoughts does not reliably help. The interleaved image-text baseline reaches 50.49 average, which improves on text-only reasoning (48.90) and shows gains on video reasoning splits and on SAT-Real and ERQA, but fails to beat direct-answer fine-tuning overall and fails on harder out-of-domain splits such as BLINK IQ Tests. The authors report that the model rarely switches to image thoughts, and that forcing it to use image thoughts reduced BLINK performance by 2%.

• Mull-Tokens are best overall. Mull-Tokens after Stage 2 reach 53.92 average, and with GRPO refinement 54.04 average, compared with 50.87 for direct-answer fine-tuning. The strongest reported improvements are on BLINK multi-view, relative depth, spatial relation, jigsaw and IQ-test splits, and on SAT-Real.

• Multimodal warm-up is the key ingredient. Warming up with both image and text achieves the highest gains (+3.05% over direct-answer fine-tuning). Mull-Tokens used purely as extra compute with no warm-up improve on the base model by 4.2% but underperform the direct-answer baseline. Text-only warm-up gives only a marginal 1.07% improvement over direct-answer fine-tuning, which the authors say is insufficient.

• Discrete tokens beat continuous embeddings. The ablation reports that discrete tokens perform better than continuous recurrent embeddings for every tested number of latents at test time, and that performance degrades with more continuous latents because errors accumulate over longer chains. Discrete tokens also allow standard token parallelism, which the paper says makes training and inference significantly faster.

• Too many latents can hurt. Performance tends to improve with more Mull-Tokens on reasoning splits, but degrades with too many, which the authors liken to degradation from overthinking. Performance scales more positively after the GRPO phase, which the authors attribute to GRPO rewarding optimal <Mull> chains on-policy unlike SFT.

• Efficiency claim. The gains come from only 20 Mull-Tokens, versus 200-500 tokens in a typical text chain-of-thought rationale; the paper states 10-40 Mull-Tokens are sufficient, framing this as Pareto-dominance (better performance plus lower inference cost).

• Generalization to other spatial benchmarks. On MMSI-Bench, the paper reports improvements in multi-step reasoning (+1.2%) and on judging attributes from different perspectives (Appr. +8.0%), plus +2.1% on SiteBench (3K randomly sampled due to compute constraints).

• Mull-Tokens can be combined with text rationales. Adding Mull-Tokens before predicting the text rationale and the answer gives an average of 51.1, versus 48.9 for predicting text reasoning directly and 50.5 for interleaved image plus text reasoning. The authors also observe the model chooses text reasoning for some tasks and only the latent tokens for others.

• No headroom on perception-heavy ERQA. All fine-tuned variants perform close to the base model on ERQA, which the authors attribute to the benchmark being perception-heavy with questions about directly visible object states or trajectories.

Methodology in Plain English

The researchers take Qwen2.5-VL (7B) as the base multimodal language model and augment its vocabulary with a fixed number of special latent tokens, <Mull>. These are appended to the input question in place of an explicit chain of thought.

Training has three stages. In Stage 1, the model is shown interleaved text-and-image reasoning traces; each <Mull> token is placed before a reasoning step and supervised to match it. If the following step is a word, the token's hidden state is pushed through the language model head and trained with cross-entropy. If the following step is a subgoal image, the image is encoded by the frozen Qwen2.5-VL image encoder with average pooling, and the token's hidden state is trained via cosine similarity to that embedding. This lets a token "anchor" to either a textual or a visual concept.

In Stage 2, the intermediate reasoning steps are removed entirely. Only the question, the <Mull> tokens, and the final answer remain, and the loss is computed only on the answer tokens. The number of latents is fixed to a small constant (for example, 20), which compresses a whole reasoning trajectory into a compact latent representation. This stage lets the model decide internally what the latents should encode, based purely on what helps answer correctly.

In Stage 3, the authors apply GRPO, a reinforcement learning method, using only the final answer as reward (a binary reward for discrete answers and a graded similarity score for numeric or continuous answers). Because the first answer token is produced from the final <Mull> state and attends to the whole latent chain through self-attention, gradients from the reward reshape the latent representations, encouraging latent trajectories that causally lead to correct answers rather than merely co-occurring with them.

Baselines are trained on the same data: direct-answer fine-tuning, text chain-of-thought (with and without GRPO), and an interleaved image-text latent approach. All training uses 8 H100 (80GB) GPUs with Deepspeed. Training data comes from Video-R1 (160K examples of general video reasoning traces), Zebra-CoT (an image-text interleaved chain-of-thought dataset), and SAT (175K simulated spatial reasoning questions and answers).

Why This Matters

Impact on research. The paper challenges two common assumptions: that explicit visual thoughts necessarily help visual reasoning, and that latent reasoning tokens need to be decoded or interpreted to be useful. It offers a training recipe that turns intermediate reasoning into a small, modality-agnostic internal computation, and it reports that a modest number of latents (20, with 10-40 found sufficient) can outperform verbose text rationales and interleaved image-text latents. The ablations specifically separate "more compute" from "multimodal information," which is a useful experimental control for the latent-reasoning literature.

Real-world applications:

• Spatial and physical reasoning assistants, such as tools that answer questions about object layouts, distances, and perspectives from photos or video.
• Robotics and embodied agents that need to reason about space, time, affordances, and route planning before acting.
• Video understanding systems, including relative direction, relative distance, and route-planning questions over video input.
• Educational or IQ-style puzzle and test solving, where the authors report their largest relative gain on a jigsaw puzzle split.
• Multi-image document or scene analysis, where the method showed gains on multi-image spatial benchmarks.

Industry relevance. Because Mull-Tokens replace hundreds of generated tokens with a few latent tokens, the approach targets inference cost as well as accuracy. The authors describe the method as Pareto-dominant (better performance, lower cost) and note that discrete tokens are compatible with standard parallel token processing in transformer frameworks, meaning training and serving infrastructure does not need to change in the way continuous recurrent latents would require.

Future Directions

1. Extending to other modalities. The framework is described as naturally extensible to 3D point clouds, trajectories, audio, and other structured data by changing the embedding function and similarity loss, but the authors leave this to future work because suitable multimodal chain-of-thought data is scarce.

2. Verifying generalization across backbones and scales. All experiments use Qwen2.5-VL (7B). The authors note that generalization to other backbones and model scales remains unverified, though they anticipate larger models would benefit more given trends in latent reasoning for language models.

3. Improving interpretability of the latents. The tokens are modality-agnostic and cannot be directly decoded into human-readable output. The paper shows they can be paired with explicit text rationales to retain accuracy while producing interpretable traces, but a more intrinsic form of interpretability remains open.

4. Integrating world models for causal reasoning chains. The authors see potential in learning from world models to discover robust causal reasoning chains, which they describe as essential for closing the remaining performance gap between current models and human capabilities.

Target Audience

Researchers and practitioners working on multimodal language models, visual and spatial reasoning, latent reasoning, or chain-of-thought methods. It is also relevant to engineers focused on inference efficiency for reasoning models, since the paper frames its contribution as improving accuracy while reducing the number of reasoning tokens generated. Readers without a background in latent reasoning or reinforcement learning will find the core idea accessible, but will need some familiarity with vision-language model training and benchmarks to follow the experimental tables.

Authors’ abstract

Reasoning goes beyond language; the real world requires reasoning about space, time, affordances, and much more that words alone cannot convey. Existing multimodal models exploring the potential of reasoning with images are brittle and do not scale. They rely on calling specialist tools, costly generation of images, or handcrafted reasoning data to switch between text and image thoughts. Instead, we offer a simpler alternative -- Mull-Tokens -- modality-agnostic latent tokens pre-trained to hold intermediate information in either image or text modalities to let the model think free-form towards the correct answer. We investigate best practices to train Mull-Tokens inspired by latent reasoning frameworks. We first train Mull-Tokens using supervision from interleaved text-image traces, and then fine-tune without any supervision by only using the final answers. Across four challenging spatial reasoning benchmarks involving tasks such as solving puzzles and taking different perspectives, we demonstrate that Mull-Tokens improve upon several baselines utilizing text-only reasoning or interleaved image-text reasoning, achieving a +3% average improvement and up to +16% on a puzzle solving reasoning-heavy split compared to our strongest baseline. Adding to conversations around challenges in grounding textual and visual reasoning, Mull-Tokens offers a simple solution to abstractly think in multiple modalities.