iReasoner: Trajectory-Aware Intrinsic Reasoning Supervision for Self-Evolving Large Multimodal Models

AI summary

Overview

Research area: Self-evolving large multimodal models (LMMs), reinforcement learning from intrinsic feedback, and chain-of-thought (CoT) reasoning optimization. The paper sits at the intersection of two lines of work that the authors note have remained "only loosely connected": unsupervised Proposer–Solver self-evolution for LMMs, and step-level CoT optimization.

Technical level: Advanced. The paper assumes familiarity with policy-gradient methods (REINFORCE, KL-regularized objectives), Proposer–Solver self-play, and LoRA fine-tuning, though the core idea can be stated simply.

Scope in one sentence: The paper introduces an intrinsic, trajectory-aware reward that scores agreement among intermediate reasoning steps across a model's own sampled rollouts, and shows that adding it to outcome-only self-consistency improves a base LMM across eight multimodal benchmarks in a fully unlabeled setting.

What This Paper Is About

Existing self-evolving LMM frameworks train on unlabeled images by having a model propose its own questions, sample answers, and reward itself based only on whether answers agree. The problem is that this outcome-only signal treats two rollouts that reach the same final answer as nearly equivalent, even when one is grounded in the image and the other relies on hallucinated intermediate claims that happen to cancel out. iReasoner's goal is to bring the intermediate reasoning steps into the optimization loop, rewarding step-level agreement among rollouts that already agree on an answer — without labeled data, external judges, or verifiers.

Key Contributions

1. iReasoner framework: A fully unsupervised self-evolving framework that brings intermediate reasoning into the optimization loop for Proposer–Solver self-evolution on unlabeled images, rather than supervising only final answers.

2. Intrinsic CoT Agreement Reward: A trajectory-aware reward that scores step-level alignment among Solver rollouts. It groups rollouts by the dominant (majority) answer, embeds each step, forms per-step prototypes within that group, and aggregates cosine similarities with decaying weights that emphasize early, grounding-heavy steps.

3. Empirical demonstration on a self-evolving LMM: Starting from Qwen2.5-VL-7B and training only on unlabeled images, the method yields gains of up to +2.1 points across diverse multimodal reasoning benchmarks under fully unsupervised post-training.

4. Diagnostic analysis of reasoning traces: Leave-one-out step-similarity analysis and training-dynamics tracking (see Main Findings) that characterize where reasoning diverges even when answers agree.

Main Findings

• Consistent gains over the seed model across both task groups. On general visual understanding, iReasoner improves Qwen2.5-VL-7B by +1.12 on InfoGraphic-VQA (80.44 → 81.56), +1.28 on AI2D (82.61 → 83.89), +1.62 on ScienceQA (88.30 → 89.92), and +1.26 on MMMU (51.11 → 52.37), averaging +1.32. On visual mathematics, it improves all four datasets by +1.64 on average, with the largest gain on MathVerse (+2.13, 43.78 → 45.91) and a consistent gain on MathVision (+1.38, 23.91 → 25.29).

• Step-wise reward adds transfer beyond answer-level agreement. Relative to EvoLMM, iReasoner improves all four general benchmarks: InfoGraphic-VQA (+0.50), AI2D (+0.48), ScienceQA (+0.42), and MMMU (+0.36). It improves MathVision (24.81 → 25.29) and MathVerse (44.88 → 45.91), while EvoLMM remains stronger on ChartQA and MathVista.

• A trade-off exists between answer stability and step-level structure. Answer-stability reward alone produces large gains on highly verifiable benchmarks: ChartQA +2.64 (84.00 → 86.64) and MathVista +1.94 (68.47 → 70.41). Adding the step reward improves InfoGraphic-VQA by +0.44, AI2D by +0.53, ScienceQA by +0.51, MMMU by +0.45, MathVision by +0.67, and MathVerse by +1.20 relative to answer stability alone, but is slightly lower on ChartQA (-0.86) and MathVista (-0.67).

• Step-level reward alone is weak. Using only step-wise reward yields small gains over the seed model (e.g., +0.17 on InfoGraphic-VQA; +0.09 on MathVerse) and does not improve MMMU. Combining both signals substantially improves over step-wise reward alone (e.g., +1.48 on ScienceQA; +2.04 on MathVerse).

• Step-level agreement needs continuous rewards. Adding step-level agreement to a discrete reward improves over the discrete-only baseline (InfoGraphic-VQA 80.52 → 80.78; MathVerse 42.10 → 44.18) but remains well short of iReasoner's results across all eight benchmarks.

• Warmup is the most important mechanism choice. Removing the warmup schedule produces the largest and most uniform regression (-0.68 average across benchmarks), including -0.76 on ChartQA and -0.80 on MathVerse. Removing position decay or density weighting reduces performance with smaller but consistent drops. Removing the length penalty reduces MathVerse by -0.30; removing shaping entirely (γ = 1.0) causes larger degradation (e.g., -0.90 on ScienceQA and -1.03 on MathVerse).

• Eight reasoning steps is the best default; more is not better. Increasing the step budget from 4 to 8 yields consistent gains (MathVerse 45.02 → 45.91; AI2D 83.12 → 83.89). Going to 10 steps gives diminishing returns and slight regression (e.g., MathVerse 45.71, AI2D 83.74).

• Gains hold across three model scales. Qwen2.5-VL-3B improves from 78.00 to 78.81 on InfoGraphic-VQA and 39.20 to 39.64 on MathVerse; Qwen2.5-VL-32B improves from 82.80 to 83.58 on InfoGraphic-VQA and 48.90 to 50.42 on MathVerse. The 3B and 7B models converge within roughly 2,500 steps; the 32B model benefits from around 3,000 steps.

• The dominant-answer group is usually correct, and improves. Tracked diagnostically (never during training), the dominant-answer group matches ground truth roughly 76% of the time in steps 0–1,000, roughly 86% in steps 1,000–2,000, and roughly 93% in steps 2,000–2,500. The final model answers 87% of questions it failed during training correctly when re-evaluated.

• Aligned chains do real inferential work. With CoT suppressed at inference, the iReasoner-trained 7B model still beats the baseline (e.g., InfoGraphic-VQA 80.84 vs. 80.44; MathVerse 44.52 vs. 43.78), but remains below the full-CoT setting across all eight benchmarks (e.g., 81.56 and 45.91 respectively).

• Reasoning diverges even under answer agreement, mostly mid-trace. Leave-one-out step similarity shows rollouts in the dominant group can deviate sharply at specific step indices while remaining aligned elsewhere. Aggregated disagreement concentrates in the middle of the trace (steps 2–3) and is lower in later steps.

• Training stays in a non-degenerate regime. Proposer reward remains stable around 0.3–0.5 while answer entropy stays in a moderate band (roughly 0.6–1.1 nats). The mixed Solver reward stays high (about 0.6–0.7), and the dominant-answer group size and number of valid step positions increase into the 2–3 range over training.

Methodology in Plain English

The setup is a two-role loop trained on images only. A Proposer looks at an unlabeled image and writes a question about it. A Solver then writes N = 5 reasoning attempts for that image–question pair, each formatted as an explicit chain of thought followed by a final answer.

From those five attempts, the system computes an empirical answer distribution. This gives two things. First, it feeds the Proposer an entropy-based reward that pushes toward questions of intermediate difficulty — hard enough that the Solver disagrees, but not so hard that it always fails. Second, it identifies the dominant answer group: the rollouts that landed on the most common answer.

The new ingredient is what happens next. The system parses each trace into numbered steps, embeds each step's text using the model's own internal token embeddings (an ℓ2-normalized mean with a fixed token budget), and builds a per-step prototype — the average embedding for that step index across the dominant group. Each rollout is then scored by the cosine similarity of its steps to those prototypes, weighted more heavily for early steps (since early steps tend to carry the visual grounding) and scaled by a density factor that reduces the step signal when few rollouts agree, which is exactly when error risk is highest.

The Solver's final reward mixes two terms: an answer-level self-consistency reward (with a penalty for exceeding a target pre-answer length) and the step-level agreement reward. A schedule parameter ramps the step component from 0 up to a maximum weight of 0.7 over training, so early updates lean on answer stability while later updates lean on step structure. Both roles are trained with KL-regularized REINFORCE against a frozen reference policy, with adaptive KL coefficients and LoRA adapters while the backbone stays frozen.

Training used 2.5k images drawn from ChartQA, AI2D, InfoGraphic-VQA, PlotQA, ChartX (400 each), and Geometry3K (500) — with no QA pairs, captions, metadata, or external reward models at any stage. Runs used 2.5k steps, AdamW at learning rate 10⁻⁶, weight decay 0.01, gradient clipping 1.0, in bfloat16 on 8× AMD MI250X GPUs, completing in roughly 35 hours. Evaluation used lmms-eval with identical inference settings across all models.

Why This Matters

Impact on research. The paper targets a specific gap: prior self-evolving LMM pipelines (EvoLMM, VisPlay) reward answers or whole responses, while prior CoT-optimization methods assume labeled data, external judges, or offline supervision. iReasoner shows that step-level agreement can be extracted from a model's own samples and used as a training signal, which is a genuinely new supervision channel for unlabeled multimodal self-improvement. It also provides diagnostic evidence — the mid-trace disagreement profile and the 76%/86%/93% dominant-group correctness tracking — that outcome-only rewards genuinely cannot distinguish stable from unstable reasoning within an answer mode.

Real-world applications (these follow from the benchmark domains the method was evaluated on, not from deployment claims in the paper):

• Chart, plot, and infographic understanding, where a model must read values off a figure and then reason about them — the ChartQA, PlotQA, ChartX, and InfoGraphic-VQA settings.
• Diagram and textbook-figure question answering in educational tools, corresponding to the AI2D, ScienceQA, and Geometry3K settings.
• Multimodal math tutoring, where intermediate steps matter pedagogically and a wrong step that cancels out is a real failure mode — the MathVista, MathVision, and MathVerse settings.
• Domain-specific visual assistants in scientific and engineering workflows, where the model must handle unfamiliar figures without curated labeled data, since training here requires only images.

Industry relevance. The method needs no annotation pipeline, no reward model, and no external judge — only unlabeled images and access to model internals. For organizations sitting on large volumes of proprietary images but little labeled data, that is an attractive cost profile. The paper's 8× AMD MI250X GPU, roughly 35-hour run also indicates the compute is not exotic.

Future Directions

1. Mitigating confidently-wrong dominant groups. The authors state this directly in their limitations: because iReasoner uses only intrinsic signals, it cannot directly optimize external correctness, and when the dominant-answer group is confidently wrong (from early-training noise or perception failures), step-level agreement may reinforce internally consistent but incorrect reasoning. They suggest ensemble disagreement as one possible robustness mechanism.

2. Scaling the data and the run. The limitations section calls for longer self-evolution runs, larger and more diverse unlabeled image streams, and additional model families to better characterize stability and scaling. (Note: the paper's Table 4 already reports scaling across Qwen2.5-VL-3B, 7B, and 32B, while the limitations text describes the work as reporting "results from a single backbone" — a tension the paper does not resolve.)

3. Relaxing the dependence on model internals. The training procedure assumes access to log-probabilities and a reference policy for KL regularization, which makes it most applicable to open-weight models and less directly transferable to black-box systems that do not expose token-level likelihoods.

4. More open-ended self-generated curricula. The conclusion frames iReasoner as "a starting point" and expresses hope for "reasoning-aware self-improvement and more open-ended self-generated curricula under minimal supervision." How the step-agreement signal behaves with non-fixed question-generation curricula is left open.

Target Audience

Researchers and graduate students working on multimodal foundation models, self-improving or self-play training loops, and reinforcement learning from intrinsic feedback. It is also relevant to practitioners who want to post-train open-weight vision-language models on unlabeled image collections, and to those specifically interested in making chain-of-thought supervision work without human annotations or reward models. Readers without background in policy-gradient methods and multimodal RL will find the main results accessible but the method section demanding.

Authors’ abstract

Recent work shows that large multimodal models (LMMs) can self-improve from unlabeled data via self-play and intrinsic feedback. Yet existing self-evolving frameworks mainly reward final outcomes, leaving intermediate reasoning weakly constrained despite its importance for visually grounded decision making. We propose iReasoner, a self-evolving framework that improves an LMM's implicit reasoning by explicitly eliciting chain-of-thought (CoT) and rewarding its internal agreement. In a Proposer--Solver loop over unlabeled images, iReasoner augments outcome-level intrinsic rewards with a trajectory-aware signal defined over intermediate reasoning steps, providing learning signals that distinguish reasoning paths leading to the same answer without ground-truth labels or external judges. Starting from Qwen2.5-VL-7B, iReasoner yields up to $+2.1$ points across diverse multimodal reasoning benchmarks under fully unsupervised post-training. We hope this work serves as a starting point for reasoning-aware self-improvement in LMMs in purely unsupervised settings. Our code is available at https://meghanaasunil.github.io/iReasoner.