MIRA: Reinforcing Multimodal Reasoning via Deceptive Contextual Augmentation

AI summary

Overview

Research area: Multimodal (vision-language) reasoning, reinforcement learning with verifiable rewards (RLVR), and robustness to visual hallucination in large vision-language models (VLMs).

Technical level: Advanced. The paper assumes familiarity with policy-gradient RL for language models (GRPO), reward shaping, and VLM training pipelines.

Scope (one sentence): The paper introduces MIRA, a GRPO-based training framework that injects deliberately misleading visual descriptions into a model's own reasoning context so the model learns to detect and correct perceptual errors, evaluated on math-reasoning and hallucination benchmarks using Qwen-VL backbones.

What This Paper Is About

Existing RLVR methods for multimodal reasoning make models better at logical reasoning but leave them vulnerable to errors at the perception stage: the authors cite prior work indicating roughly 70% of errors in multimodal reasoning originate from visual perception, and once a model misreads an image it reasons from a wrong premise and compounds the mistake. MIRA's goal is not to make perception itself more accurate, but to teach the model to recover from an initial misreading during its own reasoning. It does this by inserting plausible-but-wrong visual descriptions into the model's context during training and rewarding it when it notices and corrects the injected errors, all without human-annotated chain-of-thought data.

Key Contributions

1. A deceptive visual-context construction pipeline. The policy model writes its own description of the question-relevant image content, an expert model (GPT-5.4 in the main setting) perturbs that description into a plausible but incorrect one while recording before/after edit pairs, and a text-only verifier keeps only the perturbed descriptions that flip the answer from correct to incorrect (with no access to the image). No human annotations are used.
2. An intra-group reflection trigger. For each group of N rollouts, K = N/2 samples get a trigger such as "Wait, I may have misinterpreted the image." appended to the deceptive context, while the other N − K do not. Because GRPO learns from relative rewards inside a group, mixing prompted-reflection and unprompted rollouts under the same corrupted context yields a sharper learning signal; triggers are applied only to augmented samples.
3. A composite reward with a correction term. The reward combines a format reward, an answer-accuracy reward, and a correction reward computed from the recorded edit pairs via a judge model, gated by format and accuracy so it cannot be gamed on wrong answers (coefficients α = 0.4, β = 1.0, γ = 0.2).
4. An injection-position design plus empirical validation. The deceptive context is inserted as an assistant prefix (inside the reasoning block) rather than in the user prompt, and the paper tests this against user-prompt injection, several expert models, several augmentation ratios, three additional backbones, and quantitative reflection metrics.

Main Findings

• Headline benchmark gains: On Qwen2.5-VL-7B, MIRA reaches MathVista 75.2, MathVerse 53.8, MathVision 29.0, WeMath 70.7, MMMU-Pro 41.0, HallusionBench 69.8, and FREAK 30.6, for an average of 52.9. Deltas versus the base Qwen2.5-VL-7B are +7.0, +7.1, +4.7, +9.3, +2.5, +4.1, +0.4, and +5.0 respectively. The abstract describes an average 7% improvement on in-domain benchmarks; the introduction states approximately 7 percentage points on average across mathematical benchmarks.
• Competitive against specialized baselines at the same scale: MIRA's 52.9 average tops the listed reasoning methods, ahead of PAPO_G-7B (51.4), VL-Rethinker-7B (51.0), Perception-R1-7B (50.6), MM-Eureka-7B (50.2), Vision-R1-7B (49.6), SophiaVL-R1-7B (49.0), and R1-Onevision-7B (43.3). On FREAK, PAPO scores higher (31.1) than MIRA (30.6).
• Transfers across backbones: MIRA beats GRPO in 11 of 12 comparisons and ties on Qwen3-VL-8B/MathVerse. Examples: Qwen2-VL-7B MathVista 63.5 vs 62.0; Qwen3-VL-4B WeMath 78.7 vs 75.6; Qwen3-VL-8B MathVision 52.9 vs 51.1.
• Stronger expert models help: With GPT-5.4, GPT-5.4-mini, and GPT-5.1-mini as the rewriting expert, averages are 57.18, 56.53, and 56.33 respectively.
• Augmentation ratio matters, and too much hurts: On a 14K dataset, augmentation ratios of 0 (GRPO), 0.3, 0.5, and 0.7 give averages of 54.7, 55.3, 55.8, and 54.9 — best at 0.5. The main experiments use |D_aug| : |D_0| ≈ 1:2.
• Both the correction reward and the trigger are necessary: Full MIRA averages 57.2; removing the correction reward drops it to 56.7; using the full trigger without the correction reward gives 55.8; removing the trigger gives 55.1; removing the deceptive context entirely (plain GRPO) gives 55.6. Without triggers, the model learns to ignore the injected context and take shortcut answers.
• Assistant-prefix injection beats user-prompt injection: Average 53.5 vs 52.3 across MathVista, MathVerse, WeMath, MathVision, and MMMU.
• The deceptive context revives "too easy" samples: Training accuracy on easy samples drops early under MIRA, giving non-trivial gradients; the number of sample groups with zero advantage variance is substantially reduced relative to vanilla GRPO, and MIRA gradually overtakes GRPO on the normal (unaugmented) data.
• Reflection becomes internalized but is not error-free: Under controlled inputs, recall rises from 27.42 to 42.96 and SRR from 36.33 to 59.54, with augmented-set accuracy 19.48 → 50.66 and normal accuracy 43.51 → 68.69. On MathVerse, reflection rate rises 8.9% → 15.8%, SRR 57.8% → 69.8%, but false correction rate also rises from 0.30% (12/3940) to 1.07% (42/3940). On MathVista, accuracy improves 72.9 → 75.2 while SRR falls from 85.4 to 66.7.
• Qualitative case: In one training example with no trigger inserted, the model first follows an incorrect description of the relationship between two resistors, then explicitly revises its reading and answers correctly from the visual evidence — evidence of late-stage rather than superficial correction.

Methodology in Plain English

The method starts from an ordinary multimodal training set (ViRL-39K and MMK12; about 18K valid examples sampled to form the original dataset). For each question, the model being trained is asked to write a short description of the image content relevant to the question. An expert model then edits that description so it stays structurally similar and plausible but contains specific factual changes that would lead to a wrong answer, and it records exactly what it changed. A text-only checker then answers the question twice from the description alone — it must get the question right from the original description and wrong from the edited one, otherwise the pair is discarded. This filtering guarantees the corruption actually matters.

During reinforcement learning, the corrupted description is placed at the start of the model's own reasoning block, formatted as if the model had written it itself, followed in half the rollouts by a short trigger phrase. The model then continues reasoning and produces an answer. The reward has three parts: a format check, a correctness check against the ground truth, and — only when the answer is correct — a correction score from a judge model that checks whether the model explicitly rejected the altered detail and restored the original visual evidence. The deceptive prefix is used only to steer rollouts; it is not treated as an action in the policy objective, which the authors justify as avoiding off-policy bias. Original (unaugmented) samples are kept in the mix so the model does not lose general reasoning ability. Training uses GRPO for 2 epochs, learning rate 1×10⁻⁶, rollout batch size 128, and maximum generation length 2048 tokens, with Qwen3-VL-32B as the accuracy judge and temperature 0.0 at evaluation.

Why This Matters

Impact on research. The paper reframes multimodal RLVR from "improve perception" to "improve recovery from bad perception," and shows a route to visual self-correction without supervised cold-start data or human-annotated chains of thought. Its zero-variance-group analysis offers a concrete mechanism for why augmentation helps RL: it converts saturated, uninformative easy samples back into useful gradient sources. It also reports an honest cost — more reflection can mean more false corrections.

Real-world applications:

• Document, chart, and figure question answering, where a single misread axis label or table entry currently derails the whole answer.
• Medical or scientific image triage assistants, where a model should flag and revisit an inconsistent reading rather than assert it confidently.
• Industrial inspection and robotics, where the model must notice when its own visual interpretation conflicts with a physical constraint.
• Accessibility tools that describe scenes for users, where silently hallucinated attributes are the main failure mode.

Industry relevance. The pipeline uses only open-source training data and an off-the-shelf expert model, keeps the base RLVR algorithm (GRPO) intact, and adds no architecture changes — properties that make it cheap to bolt onto existing post-training stacks. The backbone-transfer results across Qwen2-VL and Qwen3-VL at 4B and 8B sizes support deployment at moderate scale.

Future Directions

• Scaling the mechanism. The authors state that validation is limited to the 7B scale due to computational constraints and that exploring scaling properties on larger foundation models is future work.
• Reducing harmful reflection. False correction rates rise on MathVerse (0.30% to 1.07%) and answer-accuracy-among-reflections drops on MathVista (85.4 to 66.7), so calibrating when to reflect — rather than reflecting more — is an open problem.
• Lowering dependence on a strong expert. Gains shrink with weaker experts (57.18 vs 56.33 average from GPT-5.4 to GPT-5.1-mini), raising the question of whether smaller or open judges/experts can sustain the effect.
• Balancing augmentation quantity against signal diversity. Performance degrades at a 0.7 augmentation ratio, and the authors attribute this to reduced training-signal diversity — how to add more corrupted data without collapsing diversity is unresolved.

Target Audience

Researchers and engineers working on post-training of vision-language models — particularly those already using GRPO or similar verifiable-reward RL — who care about hallucination robustness and self-correction. It is also useful to practitioners building reliable multimodal assistants who want a training-time technique that requires no new annotations or architecture changes, and to students of RL for LLMs interested in reward design and group-level advantage analysis. Readers without RL background will need to consult the GRPO preliminaries section first.

Authors’ abstract

While Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced multimodal reasoning, existing frameworks suffer from a critical "perception-reasoning gap." Due to an over-reliance on seemingly relevant visual tokens, even minor perceptual perturbations can propagate through the reasoning process, leading to compounding hallucinations. To address this issue, we propose MIRA, a training framework that improves robustness to erroneous visual contexts while enhancing logical reasoning ability. By injecting filtered, deceptive visual contexts during training, we construct challenging "hard traps" that stress-test the model's robustness to misleading information. Furthermore, we introduce a group-wise reflection-triggering mechanism that enables the model to autonomously detect and rectify inconsistencies between visual cues and logical constraints, which is ultimately internalized as an intrinsic policy behavior. Importantly, MIRA does not rely on human annotations. Experimental results across multiple reasoning and perception benchmarks show that MIRA yields an average 7% improvement on in-domain benchmarks, with consistent improvements on vision-centric benchmarks. These findings highlight that active error recovery is essential for robust multimodal reasoning. Comprehensive ablation studies and analyses further provide insights into why MIRA is effective.