On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs

AI summary

Overview

• Research area: Multimodal machine learning — specifically reinforcement-learning (RL) post-training of vision-language models (VLMs) and the reliability of their visual reasoning.
• Technical level: Advanced. The paper assumes familiarity with RL finetuning for LLMs (GRPO, verifiable rewards), chain-of-thought (CoT) prompting, multimodal benchmarks, and information-theoretic metrics such as Shannon entropy and AUROC.
• Scope in one sentence: The paper stress-tests RL-finetuned VLMs with controlled textual perturbations (misleading captions and misleading reasoning prefixes), compares open- and closed-source models, and traces how RL finetuning itself creates a trade-off between benchmark accuracy and the faithfulness of the model's reasoning.

What This Paper Is About

RL finetuning has become a standard way to improve reasoning in large language models, and the same recipe is now being applied to vision-language models. The paper asks whether these accuracy gains reflect genuine visual grounding or whether the models are simply leaning harder on whatever text appears in the prompt. By injecting misleading captions or incorrect chain-of-thought prefixes into otherwise unchanged visual reasoning questions, the authors show that many RL-tuned models collapse or produce answers that contradict their own stated reasoning, and they then investigate how RL training dynamics produce this fragility.

Key Contributions

1. Augmented stress-test benchmarks. The authors introduce controlled textual perturbations — Stop-Think, Wrong-Think (with a "but I think" corrective variant), and Wrong-Caption (with a "but I could be wrong" disclaimer variant) — applied to visual reasoning questions. The contribution statement describes these perturbations across eight visual reasoning benchmarks targeting simple skills such as counting and 2D/3D spatial relations.
2. An accuracy–faithfulness trade-off. Through controlled RL finetuning runs on Qwen2.5 VL 7B Instruct, the paper documents a systematic disconnect: benchmark accuracy rises (particularly with more finetuning steps) while the consistency between the generated CoT and the final answer erodes.
3. An open-source versus closed-source comparison. The paper shows that closed models (o3, o4-mini, Gemini-2.5-Pro, Gemini-3.1-Pro Preview) share the same failure modes — hallucinations, overthinking, accuracy drops — but maintain markedly higher reasoning consistency and robustness, implying the gap is a shortcoming of current open-source RL finetuning rather than an inherent limit of the task.
4. Interventions on the training objective. The authors test adversarial data augmentation and a faithfulness-aware reward, showing that augmentation improves robustness without preventing faithfulness drift, and that combining faithfulness rewards with augmentation can produce unstable training that collapses onto shortcut strategies.

Main Findings

• Stop-Think effects are mixed on accuracy but uniform on entropy. Suppressing intermediate reasoning with an uninformative <think> Okay let's see. This should be the final answer. </think> tag produced model- and task-dependent accuracy changes: Video-R1 and Vision-R1 showed the largest performance deltas, while VLAA-Thinker and ViGoRL-Spatial showed minimal differences. Answer-token entropy increased for all reasoning models under Stop-Think regardless of whether accuracy rose or fell. Examples from 3DSRBench: Video-R1 rose from 0.26 to 3.05 entropy, Vision-R1 from 2.4×10⁻⁵ to 0.10, VLAA-Thinker from 0.039 to 1.15, ViGoRL-Spatial from 0.27 to 0.91, and SpaceR from 10.80 to 11.56 (SpaceR remaining high-entropy throughout).

• Misleading text degrades performance, and disclaimers partially restore it. Adding an incorrect thinking prefix (Wrong-Think) or an incorrect caption (Wrong-Caption) caused substantial performance drops; appending a corrective marker ("But I think" / "But I might be wrong") improved performance. The authors note these simple manipulations also accentuate differences between models that appear comparable under clean prompting. In the inverted ablation where the caption or thinking string embeds the correct answer, adding a disclaimer causes a drop in performance instead — reinforcing the conclusion that models lean heavily on textual context.

• Correct answers can be unfaithful. Using Qwen3-32B as a judge (a generation counts as consistent when the model's final judgment inside <think></think> matches its <answer></answer>), many correct answers under Wrong-Think were still flagged as inconsistent. The models most robust to perturbations also showed the highest inconsistency, including under clean Base conditions. Under misleading-caption perturbations, models were generally more faithful than under Wrong-Think, and their correct answers often stemmed from disregarding the caption entirely.

• Judge reliability was validated across three models. GPT-OSS-120B, Qwen3-32B, and Llama-3.1-70B-Instruct produced strict three-way agreement of 89.2% (WhatsUp) to 94.2% (CVBench), with Cohen's kappa pairwise scores from 0.783 to 0.899 and Fleiss' kappa from 0.809 to 0.884 across datasets.

• Failures are not explained by uncertainty alone. When an explicit "I'm not sure" abstention option was added, performance generally decreased, with the sharpest drops under Wrong-Caption and Wrong-Think. Stop-Think and Wrong-Caption were the conditions with the most abstentions. Abstentions sometimes increased when corrective cues were provided, indicating models remain receptive to disambiguating signals but are actively misled rather than merely unsure. Reported abstention fractions include SpaceR at 6.28% ± 5.83% and Video-R1 at 5.36% ± 4.07% under Base, rising to SpaceR at 12.25% ± 9.14% under Wrong-Caption + Disclaimer.

• Closed models are more robust but still fail. All closed models showed reduced degradation under misleading interventions, with Gemini-3.1-Pro showing the strongest performance and robustness across all datasets. Closed models frequently detected image–text conflicts but sometimes reconciled them incorrectly (for example, reinterpreting a cooler as a "bucket" to match a misleading caption), and Gemini-2.5-Pro and Gemini-3.1-Pro oscillated between prioritizing visual versus textual evidence. Hallucinations persisted, including fabricated image URLs under Wrong-Think.

• Closed models are substantially more faithful. Closed models' generations aligned more reliably with the conclusions in their reasoning traces even under perturbation, though occasional lapses remained.

• Entropy-based diagnostics separate "confidently misled" from "uncertain." Measuring letter entropy and the probability mass on the correct letter (P(Correct Letter)) over the valid options shows that Stop-Think yields the highest entropy while keeping P(Correct Letter) close to the default baseline. Adversarial prompts (Wrong-Think, Wrong-Caption) suppress P(Correct Letter); for VLAA-Thinker and Vision-R1, Wrong-Think drove P(Correct Letter) close to zero despite only moderate entropy changes, indicating degraded internal scoring of the correct option rather than diffuse uncertainty. Repair prompts restored much of the probability mass, which the authors interpret as evidence that "lost" accuracy reflects instruction-driven misalignment rather than erased knowledge. SpaceR and ViGoRL — which use spatially guided RL objectives — maintained relatively high P(Correct Letter) under adversarial inputs.

• Base-prompt confidence predicts robustness, and that has a cost. AUROC analysis shows P(Correct Letter) under the default prompt is a better predictor of robustness to perturbation than negative entropy. SpaceR's P_base reached 0.958 AUROC for Stop-Think versus 0.732 for entropy, with SpaceR exceeding 0.94 across settings ("stubborn experts" that ignore deceptive traces at the cost of CoT faithfulness). Vision-R1 showed "brittle confidence," with an AUROC of 0.565 for Stop-Think, near random chance — consistent with faithfully following the adversarial reasoning to a wrong conclusion.

• Training-recipe effects. Adding the Geometry3K visual math reasoning dataset improved baseline performance relative to SAT2 + Pixmo Count alone, and incorporating synthetic correct/incorrect captions and reasoning strings maintained competitive overall performance while substantially boosting robustness under Wrong-Caption. Even so, accuracy gains did not prevent faithfulness drift, and pairing a faithfulness-aware reward with augmentation risked collapsing onto shortcut strategies without achieving robustness.

Methodology in Plain English

The authors start with established visual reasoning benchmarks — 3DSRBench, CV-Bench, Spatial-MM, and WhatsUp — plus complementary datasets (V*-Bench, MME-RealWorld-Lite, MMBench) and a RefCOCOg bounding-box localization task. They leave the images and questions untouched but surgically edit the text: suppressing the thinking step, seeding the reasoning with a wrong trajectory, or attaching a caption that asserts an incorrect answer. To generate the misleading text they randomly sample an incorrect answer from the available options and procedurally construct the caption or thinking prefix. Correct-answer versions of the same perturbations serve as controls.

To measure whether reasoning is "faithful," they define faithfulness narrowly as consistency between the answer and the CoT — not the stronger interpretability notion of an explanation reflecting internal computation. Three separate judge models score every generation, and inter-judge agreement is reported to establish that the metric is not an artifact of one judge. An abstention variant adds an "I'm not sure" option to test whether failures come from genuine uncertainty. An entropy analysis on the first generated answer token quantifies how much probability mass remains on the correct option under each prompt type.

For the training study, the authors RL-finetune Qwen2.5 VL 7B Instruct with GRPO using the verl implementation, using a verifiable reward of 0.1 for correct formatting and 1.0 for a correct final answer (0 otherwise). Three data mixtures are compared: SAT2 + Pixmo-Count; SAT2 + Pixmo-Count + Geometry3K; and the same mixture with caption and thinking augmentation. In the augmented condition, each question has four possible augmentations (wrong thinking, correct thinking, wrong caption, correct caption), each applied with 10% probability for a 40% total augmentation rate, leaving 60% of questions unmodified. Mixing correct and incorrect context is deliberate, so the model cannot simply learn to distrust all provided text.

Why This Matters

• Impact on research. The paper argues that accuracy-only evaluation hides real vulnerabilities and motivates protocols that jointly measure correctness, robustness, and reasoning faithfulness. Its entropy and AUROC diagnostics offer an alternative lens for comparing RL finetuning recipes, and its negative results on augmentation-plus-faithfulness-reward combinations flag an open methodological problem for the field.
• Real-world applications:
  • Visually grounded planning, navigation, and acting systems, which the paper explicitly identifies as requiring stability under benign variation in textual context.
  • Multimodal assistants that receive user-supplied descriptions or captions alongside images, where an incorrect caption should not override what is visible.
  • Spatial reasoning applications such as robotics or logistics that depend on counting, object identity, and 2D/3D relation judgments remaining reliable.
  • Confidence-thresholded deployment pipelines, since the AUROC results show base-prompt probability mass can filter likely failures for some models — though the authors caution this capability appears tied to decoupling the answer from the reasoning context.
• Industry relevance. The work originates from Apple, with correspondence addresses at apple.com, and includes a comparison against commercial closed models (o3, o4-mini, Gemini-2.5-Pro, Gemini-3.1-Pro Preview). For teams shipping multimodal systems, the practical message is that headline benchmark gains from RL finetuning can coexist with reasoning that contradicts the final answer and with greater susceptibility to misleading contextual text.

Future Directions

• Designing rewards that jointly enforce robustness and faithfulness. The paper shows that faithfulness-aware rewards and adversarial augmentation interfere with each other, producing unstable dynamics and shortcut strategies, so a training objective that achieves both remains an open problem.
• Closing the open-versus-closed consistency gap. Since closed models show the same failure modes but stronger CoT consistency, the authors frame the discrepancy as a deficiency in current open-source RL finetuning — identifying what specifically produces that consistency is an unresolved question.
• Extending the analysis beyond the studied modalities and tasks. The authors report that trends extend to a different base model (InternVL3-8B) and to SenseNova-SI, as well as to other VQA benchmarks and a RefCOCOg localization task, but broader validation across architectures and training recipes is left open.
• Stronger notions of faithfulness. The paper explicitly limits itself to reasoning–answer consistency rather than mechanistic faithfulness, noting that evaluating the causal role of a generated rationale is difficult for black-box and open-source VLMs at scale.

Target Audience

Researchers and engineers working on multimodal reasoning, RL post-training for LLMs and VLMs, and benchmark design for vision-language evaluation. It is also relevant to practitioners who deploy multimodal assistants and need to know when a high-scoring model can be steered to a wrong answer by a misleading sentence in the prompt, and to those designing reward functions for reasoning models where the goal is not just a correct final answer but reasoning that actually supports it.

Authors’ abstract

Reinforcement learning (RL) finetuning has become a key technique for enhancing large language models (LLMs) on reasoning-intensive tasks, motivating its extension to vision-language models (VLMs). While RL-tuned VLMs improve on visual reasoning benchmarks, they remain vulnerable to weak visual grounding, hallucinations, and over-reliance on textual cues. We show that simple, controlled textual perturbations, including misleading captions or incorrect chain-of-thought (CoT) traces, cause substantial drops in robustness and confidence, and that these effects are more pronounced when CoT consistency is taken into account across open-source multimodal reasoning models. In contrast, closed models exhibit similar failure modes but maintain markedly greater robustness and reasoning consistency, suggesting that the gap reflects a shortcoming in current open-source RL finetuning rather than an inherent limitation of the task. To better understand these vulnerabilities, we further analyze RL finetuning dynamics and uncover an accuracy-faithfulness trade-off: finetuning raises benchmark accuracy, but can simultaneously erode the reliability of the accompanying CoT and its robustness to contextual shifts. Although adversarial augmentation improves robustness, it does not by itself prevent faithfulness drift. Incorporating a faithfulness-aware reward can restore alignment between answers and reasoning, but when paired with augmentation, training risks collapsing onto shortcut strategies and robustness remains elusive. Together, these findings highlight the limitations of accuracy-only evaluations and motivate training and assessment protocols that jointly emphasize correctness, robustness, and the faithfulness of visually grounded reasoning.