NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation

AI summary

Overview

• Research area: Multimodal large language models (MLLMs), reinforcement learning for chain-of-thought (CoT) reasoning, computer vision (arXiv:2510.21122v3 [cs.CV]).
• Technical level: Advanced. The paper assumes familiarity with Group Relative Policy Optimization (GRPO), PPO-style clipped objectives, Gaussian/Bayesian inference, and diffusion-style noise schedules.
• Scope: A single paper proposing a multimodal RL framework that injects controllable visual noise during rollout collection and replaces standard group-normalized advantage with a Bayesian posterior advantage that fuses noise level (prior) with semantic reward (likelihood).

What This Paper Is About

Reinforcement learning methods such as GRPO can improve the CoT reasoning of multimodal large language models, but when used for general (not domain-specific) CoT the resulting models often fail to generalize beyond the training distribution—Figure 1 shows that higher training rewards do not consistently produce better evaluation performance. The authors identify two causes: rollouts collapse to near-identical outputs so exploration is insufficient, and rule-based rewards supervise only the final answer, so the policy learns shortcuts and visual hallucinations. NoisyGRPO addresses both by perturbing visual inputs during exploration and by estimating trajectory advantages through Bayesian inference rather than plain group normalization.

Key Contributions

1. A multimodal RL framework, NoisyGRPO, designed to improve general multimodal CoT reasoning rather than only narrow task-specific problems, and reported to add no additional computational overhead.
2. A noise-injected exploration policy that applies diffusion-style Gaussian noise of varying magnitude to input images during rollout collection, increasing trajectory diversity and implicitly favoring visually grounded reasoning.
3. Bayesian advantage estimation, which formulates advantage as a Bayesian inference problem where the injected noise level is the prior and the observed trajectory reward is the likelihood, producing a posterior estimate that calibrates policy updates to both noise and reward.
4. Empirical evaluation across three dimensions—CoT quality (MME-CoT), general capability (MMStar, MME-RealWorld-Lite), and hallucination (AMBER)—showing consistent gains over GRPO, with the largest gains in small-scale MLLM RL settings.

Main Findings

• Generalization gap in vanilla GRPO: Under GRPO, higher training rewards do not consistently yield better evaluation performance on MMStar over training iterations, motivating the framework.
• CoT quality gains: With Qwen2.5-VL 3B, NoisyGRPO improves CoT quality on MME-CoT by +4.4 over GRPO (F1 36.2 → 40.6; precision 53.0 → 57.3; recall 27.4 → 31.5). With Qwen2.5-VL 7B the improvement is smaller (F1 46.8 → 47.7).
• Robustness: NoisyGRPO-7B is reported to achieve higher CoT robustness scores than proprietary models GPT-4o and Kimi K1.5. Vanilla GRPO surpasses NoisyGRPO on CoT efficiency for Qwen2.5-VL 7B, which the authors attribute to NoisyGRPO prioritizing grounded reasoning over reasoning efficiency.
• General capability: On MMStar, NoisyGRPO-3B reaches 58.2 average versus 54.5 for GRPO (a +3.7 gain over GRPO) and 55.4 for the base model (+2.8 over base). NoisyGRPO-7B reaches 65.8 versus 62.6 for GRPO and 63.2 for base (+2.6 over base).
• Category-level pattern: Improvements concentrate in fine-grained visual understanding—for example, Qwen2.5-VL 3B rises from 59.6 to 66.6 on Instance Reasoning and from 49.1 to 53.2 on Fine-grained Perception, and the 7B variant from 44.6 to 48.3 on Science & Technology and 60.1 to 63.7 on Fine-grained Perception. Coarse Perception drops for both variants (3B: 70.3 → 69.5; 7B: 73.3 → 72.8).
• Hallucination: On AMBER generative metrics, NoisyGRPO-3B scores 6.6 (Cs), 67.7 (Cov.), 44.3 (Hal.), 3.4 (Cog.), 90.3 (F1), and 44.0 (Acc), against GRPO's 6.7, 68.5, 44.6, 4.2, 89.2, and 40.8. NoisyGRPO performs worse on the Cov. (coverage) metric, which the authors attribute to more concise, grounded answers.
• Real-world VQA: On MME-RealWorld-Lite accuracy, NoisyGRPO-3B reaches 44.0 (versus 42.1 base, 44.0 SFT, 40.8 GRPO); NoisyGRPO-7B reaches 44.6 (versus 43.6 base, 43.5 SFT, 42.3 GRPO).
• SFT is not the source of gains: Supervised fine-tuning on the same 13k dataset fails to bring consistent improvements, indicating the gains come from reinforcement learning rather than high-quality data alone.
• Training dynamics: The importance weight σ_s²/(σ_n² + σ_s²) decreases steadily through training, showing a shift from noise-driven prior estimation toward correctness-based reward signals. NoisyGRPO consistently produces shorter completions than GRPO while performing better.
• Ablation: Naively injecting noise ("GRPO with Noise Injection") degrades policy learning; "Naive NoisyGRPO" (setting σ_s² = σ_n², removing adaptive fusion) underperforms full NoisyGRPO, showing the dynamic prior–observation fusion is necessary.
• Hyperparameter sensitivity: NoisyGRPO is relatively insensitive to α compared to γ, suggesting embedding-reward bias is smaller than prior-estimation error from noise injection. Across τ ∈ {0.5, 0.6, 0.7}, MMStar averages are 56.5, 58.2, and 56.4, all above vanilla GRPO's 54.5.
• Noise distribution matters: Uniform noise (the default) yields stable, improved performance, whereas Gaussian noise (mean 0, variance 0.1) and fixed noise (constant 0.5) produce unstable training dynamics.
• Efficiency: Training cost is comparable to GRPO. For Qwen2.5-VL-3B, GRPO takes 6h12min / 49.6 GPU-hours / 57 GB peak memory versus NoisyGRPO's 6h40min / 53.3 GPU-hours / 53 GB. For Qwen2.5-VL-7B, GRPO is 7h58min / 63.7 GPU-hours / 75 GB versus NoisyGRPO's 7h52min / 62.9 GPU-hours / 77 GB (all on a single node with 8 A100 GPUs).

Methodology in Plain English

The framework is built on top of GRPO, which samples a group of G responses per question and normalizes their rewards within the group to form advantages. NoisyGRPO changes two things.

First, how exploration happens. For each rollout, a normalized noise level n_i is drawn from a uniform distribution U(0,1) and applied to the image through a diffusion-style forward process at timestep floor(n_i · T), where T is the total number of diffusion steps. n_i = 0 leaves the image clean; n_i = 1 corresponds to the final diffusion step, resembling pure Gaussian noise. This produces diverse responses under different noise intensities. Crucially, noise is applied only during rollout collection—policy updates use clean inputs to keep learning stable.

Second, how advantages are computed. The noise-injected rollouts create a distribution shift: the policy that gathered trajectories (noisy images) differs from the policy being optimized (clean images), so ordinary advantage estimates are biased. The authors model trajectory quality as a latent Gaussian variable r_i. The inverted noise level (r_i^n = 1 − n_i) serves as the prior mean, and the semantic reward (r_i^s, combining accuracy and format rewards) serves as a noisy observation. Both are normalized within the group. The observation variance is fixed to a constant α because the embedding-based reward's bias is relatively stable, while the prior variance is set to σ_n² = γ / (γ + std({r_i^s})²) — so when semantic rewards vary widely across the group, the prior is treated as more reliable (smaller variance). Combining these gives the posterior mean r̂_i = r̂_i^s + (σ_s²/(σ_n² + σ_s²))(r̂_i^n − r̂_i^s), and the trajectory advantage is the group-normalized posterior.

Rewards. Two rule-based rewards are used: an accuracy reward (SBERT embedding similarity between predicted and ground-truth answers, set to 0 when similarity falls below threshold τ; exact match for yes/no and multiple-choice questions) and a format reward requiring reasoning inside <think> and </think> tags.

Setup. Training uses the VQA portion of MM-RLHF: 13k samples total (1.2k yes/no, 1.3k multiple-choice, 10k open-ended). The implementation builds on the VLM-R1 training framework with Qwen2.5-VL 3B and 7B policies; G = 4 rollouts, noise upper bound σ = 1 for U(0, σ), α = 0.1, γ = 0.01, τ = 0.6. There is no validation set, so hyperparameters were chosen using MMStar results. Training takes 6 hours (3B) and 7 hours (7B) on a single node with 8 A100 GPUs.

Why This Matters

This work targets a concrete failure mode of RL post-training for multimodal models—reward improvements that do not transfer out of distribution—and offers a general-purpose fix rather than a task-specific reward design. It is notable for showing that a simple, cheap intervention (perturbing images during rollouts) can be made safe by a principled estimation scheme, and for demonstrating that the largest benefits appear in small-scale models, where RL training budgets are tightest.

Real-world applications suggested by the benchmarks and framing:

• General-purpose visual assistants built on small MLLMs (3B-class), where reduced hallucination and stronger grounded reasoning lower deployment cost.
• Document, chart, and OCR-heavy tasks, since gains concentrate in fine-grained perception and instance reasoning.
• Real-world visual question answering, evaluated with MME-RealWorld-Lite accuracy improvements.
• Hallucination-sensitive deployments such as captioning and visual verification, where AMBER generative F1 improves (89.2 → 90.3) and cognitive hallucination drops (4.2 → 3.4) for the 3B model.

Industry relevance: the method is reported to add no extra computational overhead relative to GRPO (comparable wall-clock time, GPU-hours, and peak memory), so it can be layered onto existing GRPO-based multimodal training pipelines such as VLM-R1 without new hardware requirements. The finding that SFT on the same data does not help also matters for teams deciding where to spend compute.

Future Directions

• Handling coarse-grained perception: the authors' own limitation section notes that when detailed visual information is not critical (e.g., coarse perception), injected noise may not meaningfully affect outputs, and the noise magnitude used as a prior can introduce significant bias into policy learning.
• Beyond the Gaussian/noise-correlation assumption: the core premise is that the degree of injected noise correlates with answer correctness; testing and relaxing this assumption for other task families is an open question.
• Better prior modeling: sensitivity results suggest prior-estimation error from noise injection matters more than embedding-reward bias, pointing to improved prior or uncertainty formulations (including the fixed constant α for observation variance) as a direction.
• Scaling and data scope: experiments cover Qwen2.5-VL 3B and 7B with G = 4 rollouts on a 13k-sample VQA set; whether the approach holds for larger models, larger rollout groups, and other multimodal training sets is not reported.

Target Audience

Researchers and engineers working on reinforcement learning for multimodal large language models, post-training and alignment of vision-language models, and visual chain-of-thought reasoning. It will be most useful to readers already comfortable with GRPO/PPO objectives and Bayesian estimation who are looking for training-recipe improvements that generalize across benchmarks without added compute; readers seeking a first introduction to multimodal RL will find the appendix-level preliminaries helpful but the main text assumes substantial background.

Authors’ abstract

Reinforcement learning (RL) has shown promise in enhancing the general Chain-of-Thought (CoT) reasoning capabilities of multimodal large language models (MLLMs). However, when applied to improve general CoT reasoning, existing RL frameworks often struggle to generalize beyond the training distribution. To address this, we propose NoisyGRPO, a systematic multimodal RL framework that introduces controllable noise into visual inputs for enhanced exploration and explicitly models the advantage estimation process via a Bayesian framework. Specifically, NoisyGRPO improves RL training by: (1) Noise-Injected Exploration Policy: Perturbing visual inputs with Gaussian noise to encourage exploration across a wider range of visual scenarios; and (2) Bayesian Advantage Estimation: Formulating advantage estimation as a principled Bayesian inference problem, where the injected noise level serves as a prior and the observed trajectory reward as the likelihood. This Bayesian modeling fuses both sources of information to compute a robust posterior estimate of trajectory advantage, effectively guiding MLLMs to prefer visually grounded trajectories over noisy ones. Experiments on standard CoT quality, general capability, and hallucination benchmarks demonstrate that NoisyGRPO substantially improves generalization and robustness, especially in RL settings with small-scale MLLMs such as Qwen2.5-VL 3B. The project page is available at https://artanic30.github.io/project_pages/NoisyGRPO/.