Boosting Reasoning in Large Multimodal Models via Activation Replay

AI summary

Overview

Research area: Large Multimodal Models (LMMs), reinforcement learning post-training (RLVR), mechanistic interpretability of activations, and training-free inference-time intervention.

Technical level: Advanced. The diagnosis relies on interpretability tooling (logit lens, activation entropy analysis), though the proposed fix itself is conceptually simple.

Scope: The paper first probes how RLVR changes internal activations in post-trained LMMs, then uses that insight to build a training-free, test-time intervention that improves multimodal reasoning without any policy optimization.

What This Paper Is About

Reinforcement Learning with Verifiable Rewards has become a standard way to make Large Multimodal Models reason better, but nobody has a clear picture of what it actually does inside the network. This paper opens that black box by inspecting how input activations change after RLVR, and finds that the changes concentrate in a specific, unexpected place: low-entropy activations shift substantially while high-entropy ones barely move. The authors connect that shift to reasoning behavior and turn it into a practical, training-free method that improves reasoning in already post-trained models.

Key Contributions

1. A mechanistic characterization of RLVR. Using the logit lens across several post-trained LMMs, the authors systematically show that RLVR unexpectedly shifts low-entropy activations while leaving high-entropy activations largely unaffected.

2. Evidence linking that shift to reasoning. Through controlled experiments, they demonstrate that these activation phenomena are associated with LMM reasoning, which motivates deliberately modulating low-entropy activations.

3. Activation Replay, a training-free method. They propose a simple approach that manipulates visual tokens at test time, replaying low-entropy activations drawn from a base LMM's input context into the RLVR-tuned counterpart, avoiding expensive policy optimization entirely.

4. Comparisons against design alternatives. The method is tested against replaying high-entropy activations instead of low-entropy ones, and against direct cross-model intervention instead of input-token manipulation, with the chosen design reported as superior.

Main Findings

• Low-entropy activations are the ones RLVR moves. Across multiple post-trained LMMs, the logit-lens investigations show RLVR shifts low-entropy activations unexpectedly, while high-entropy activations are less affected.

• The shift relates to reasoning. Controlled experiments indicate these phenomena are associated with LMM reasoning, suggesting that modulating low-entropy activations plays a potentially beneficial role.

• Activation Replay improves reasoning across diverse scenarios. The abstract names mathematics, o3-like visual agents, and video reasoning as areas where better reasoning is triggered. No quantitative results, dataset sizes, or baselines appear in the abstract.

• It broadens reasoning coverage. The method is reported to boost Pass@K and to mitigate the narrower reasoning coverage associated with RLVR.

• Design choices matter. Replaying low-entropy activations outperforms replaying high-entropy activations, and manipulating input tokens outperforms direct cross-model intervention. The abstract does not report the magnitude of these differences.

Methodology in Plain English

The authors work in two stages.

First, they diagnose the model. They use the logit lens, a technique that reads a model's intermediate activations as if they were its final predictions, to see how internal representations differ between a base LMM and the same model after RLVR post-training. They sort activations by entropy — roughly, how confident or spread-out the implied prediction is — and compare the low-entropy and high-entropy groups. They then run controlled experiments to check whether the patterns they observe track with reasoning ability.

Second, they intervene. Instead of retraining or running reinforcement learning again, they leave the post-trained model's weights untouched and modify its inputs at test time. Specifically, they take low-entropy activations associated with the visual tokens in the base model's input context and replay them into the RLVR-tuned model. This nudges the post-trained model back toward the activation pattern the base model produces, without any gradient updates. The abstract does not spell out the exact mechanics of how the replay is implemented.

Why This Matters

Impact on research. The work reframes RLVR from an opaque "it works" recipe into something with an inspectable internal signature, and shows that this signature can be exploited directly. That gives interpretability research a concrete, testable link between activation statistics and reasoning behavior, and gives post-training research an inference-time alternative to further policy optimization.

Real-world applications (domains grounded in the scenarios the abstract names):

• Mathematics and STEM problem solving that requires reading diagrams, figures, or handwritten equations from images.
• Visual agents that perform multi-step, o3-like tasks by looking at screens or interfaces.
• Video reasoning tasks where a model must interpret events unfolding across frames.
• Any multimodal assistant built on an already post-trained LMM that needs stronger reasoning without a new training cycle.

Industry relevance. Because the method is training-free and applied at test time, it can be layered onto existing post-trained models rather than requiring a fresh, expensive policy-optimization run. That lowers the cost of improving reasoning for teams that have already paid for RLVR once, and it targets a known weakness — narrower reasoning coverage — that limits how broadly deployed reasoning models generalize.

Future Directions

• Whether the low-entropy shift is a cause of improved reasoning or only a correlate, and whether controlled experiments can be extended into a fully causal account.
• Whether the same activation pattern appears under other post-training regimes, such as supervised fine-tuning or preference-based optimization, rather than RLVR specifically.
• Whether the replay mechanism can be folded into training so the benefit is baked into the weights rather than applied at inference.
• How the approach scales beyond the scenarios named in the abstract — other modalities, model families, and task types — and whether the mitigation of narrower reasoning coverage holds generally.

Target Audience

Researchers and engineers working on multimodal model post-training, reinforcement learning for language and vision models, and mechanistic interpretability of large networks. It is also relevant to practitioners who maintain already post-trained LMMs and want reasoning gains without another round of expensive optimization, and to readers interested in how training-time interventions leave detectable fingerprints in activations.

Authors’ abstract

Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach to incentivizing reasoning capability in Large Multimodal Models (LMMs), while the underlying mechanisms behind this post-training paradigm are poorly understood. We begin by exploring how input activations are affected by RLVR through the perspective of logit lens. Our systematic investigations across multiple post-trained LMMs suggest that RLVR shifts low-entropy activations unexpectedly, while high-entropy ones are less affected. We further demonstrate that such phenomena are associated with LMM reasoning by controlled experiments, suggesting a potentially beneficial role of modulating low-entropy activations. To this end, we propose Activation Replay, a novel simple yet effective training-free approach that boosts multimodal reasoning of post-trained LMMs without requiring expensive policy optimization. Our design involves manipulation of visual tokens at test time, replaying low-entropy activations from the input context of base LMMs to regulating the RLVR counterparts. Activation Replay triggers better reasoning across diverse scenarios, including mathematics, o3-like visual agents, and video reasoning. We further show that Activation Replay boosts Pass@K and mitigates narrower reasoning coverage of RLVR. Our design is compared against alternative choices, such as replaying high-entropy activations instead of low-entropy ones, or direct cross-model intervention instead of manipulating input tokens, demonstrating the superiority of our implementation. Code is publicly available at https://github.com/latentcraft/replay.