Understanding the Effects of Distractors on Reasoning Vision-Language Models

AI summary

Overview

Research area: Multimodal machine learning, specifically test-time scaling behavior and robustness of reasoning vision-language models (VLMs) under visual and linguistic distractors.

Technical level: Intermediate. The paper's concepts (chain-of-thought reasoning, test-time scaling, VQA benchmarks, attention blocking) assume some familiarity with modern language and vision-language models, but the core argument is presented in accessible terms.

Scope: The paper introduces a benchmark suite (Idis) and analysis showing that distractors affect reasoning VLMs in modality-dependent ways, and that the fraction of distractor attributes in a model's reasoning trace predicts its accuracy.

What This Paper Is About

Prior work on text-only reasoning LMs found that irrelevant information in a prompt (textual distractors) triggers "inverse scaling": models reason longer but become less accurate as more distractors are added. This paper asks whether the same failure mode occurs in reasoning VLMs when the irrelevant information arrives through the visual channel instead — as extra objects, rendered text, or background clutter. To answer this, the authors build a controlled VQA benchmark where the same underlying question is paired with systematically varied distractors, then measure how accuracy and reasoning length change together.

Key Contributions

1. Idis benchmark suite: A VQA benchmark that systematically varies distractor modality (visual object, typographic text rendered in the image, textual text added to the prompt), number (1 to 4), and semantic relationship (aligned, conflicting, irrelevant), spanning both perception-centric (Idis-perception, from ImageNet-9) and reasoning-centric (Idis-math, from MathVerse) tasks. The paper describes Idis as containing over 277k natural and synthetic images.

2. Identification of a modality-dependent distractor effect: The authors show that visual distractors degrade accuracy without substantially lengthening reasoning, shifting the entire length-accuracy curve downward — unlike the textual-distractor pattern reported for reasoning LMs, where longer traces drive the accuracy loss.

3. Attribute-centric analysis: A trace-level analysis of verbalized visual attributes showing that the distractor attribute ratio (the fraction of attributes in the trace that belong to distractor objects rather than the target) correlates strongly with final accuracy, supported by an attention-blocking experiment.

4. A prompt-based sanity check: A simple prompting strategy instructing the model to identify the main object and reason only from its own visual attributes, which reduces distractor attributes in traces and improves accuracy on Idis-perception and on Waterbirds.

Main Findings

• Visual distractors shift the curve down, not right: Adding more visual distractors consistently lowers the length-accuracy curve without substantially changing the range of reasoning lengths. This held for all four evaluated models on Idis-perception, and for Idis-math with a notably smaller marginal effect beyond one distractor. R1-OneVision maintained its overall curve shape despite the shifts.

• Conflicting distractors are the most damaging in perception tasks: On Idis-perception, semantically conflicting distractors caused the largest downward shift, while aligned distractors caused little to no degradation relative to the no-distractor baseline. The results shown are for Qwen3 with four distractors added at a time.

• In math tasks, "irrelevant" hurts most: On Idis-math, irrelevant distractors produced the largest accuracy drop, which the authors attribute to those distractors being more out-of-distribution for the geometry-focused problem space, leading to greater confusion.

• Typographic distractors behave like visual ones: Text rendered into the image reduces accuracy without substantially expanding the reasoning-length range. This trend was less pronounced on Idis-perception but clearer on Idis-math.

• Textual distractors reproduce the LM inverse-scaling pattern: Textual distractors inserted into the question prompt clearly induce test-time inverse scaling, with much of the accuracy drop driven by longer reasoning traces — closely matching the pattern reported by Gema et al. (2025) for text-only LMs.

• Modality, not content, dominates: The authors conclude that whether distracting information is presented visually or linguistically has a stronger effect on scaling behavior than whether the distractor is an object or a piece of text.

• Distractor attributes predict failure: As the number of distractors grows, the total number of attributes in a trace stays similar, but the fraction of distractor attributes steadily increases. On Idis-perception, accuracy drops to near zero when the distractor attribute ratio exceeds 50%, yet stays above 97% when the ratio is below 20%. On Idis-math, accuracy drops to zero when the distractor attribute ratio exceeds 0.4.

• Attributes plausibly guide the answer, not vice versa: Attention-blocking experiments show that blocking the final answer tokens' attention to distractor-related attributes improves accuracy, while blocking attention to target-related attributes degrades it — suggesting traces guide predictions rather than merely reflecting them.

• Distractor area matters too: The distractor attribute ratio correlates not just with the number of distractors but with the distractor area ratio, and saturates at a higher ratio when more distractors are present.

• Prompting gives modest but consistent gains: The attribute-guiding prompt improved accuracy by 0.5%p to 1.2%p on Idis-perception and by 1.8%p to 4.0%p on Waterbirds, with the gain correlating with a consistent drop in the fraction of distractor-related attributes in the trace. The authors describe the strategy as limited and characterize it as a sanity check.

Methodology in Plain English

The researchers started from two existing datasets — ImageNet-9 for object classification and MathVerse for visual math problems — and kept the original target information and answers fixed while injecting distractors. For Idis-perception, they used Gemini 2.5 Flash Image to generate realistic edits inserting distractor objects (rather than naive pixel compositing, which produced artifacts models could spot, yielding traces like "the image might be a composite or edited photo"). For Idis-math, they used deterministic PIL compositing: the original diagram is preserved on a canvas and distractors are placed in an added region so no problem information is hidden or removed. Textual distractors for math were generated by Sonnet 4.5, and irrelevant visual distractors were drawn from the table subset of LogicVista. All construction went through human-in-the-loop verification, with regeneration or exclusion of failing samples.

To measure scaling, the authors followed a sequential-scaling protocol: for each question they sampled five responses, ranked them by reasoning length, and computed accuracy at each rank, plotting each rank at its average reasoning length. This controls for question difficulty and image complexity, so the resulting curves reflect within-question variation rather than differences in problem hardness.

Four open-weight reasoning VLMs in the 7–9B parameter range were evaluated: Qwen3-VL-8B-Thinking, GLM-4.1V-9B-Thinking, Intern-S1-mini, and R1-OneVision-7B-RL. The paper notes that these models generally lack an explicit reasoning-budget control mechanism, so the study focuses on "natural overthinking"; attempts to control reasoning length via prompting were ineffective.

For the trace analysis, visual attributes mentioned in each reasoning trace were parsed and linked to target or distractor objects using DeepSeek-V3.2-Exp with structured instructions. Object pixel areas were obtained with LangSAM masks. Robustness of the extraction was checked against an alternative prompt (Pearson r = 0.803), an alternative LLM, Qwen3.5-27B (r = 0.785), and a strict lexical matcher (r = 0.476). A separate dataset, Idis-manual, was used to manually control distractor size.

Why This Matters

Impact on research: The paper challenges a generalization drawn from text-only LMs — that distractors cause harm by lengthening reasoning — by showing that in multimodal settings the modality of the distractor determines the failure mode. It contributes a reusable controlled benchmark (release code linked at github.com/effl-lab/Idis) and an attribute-level diagnostic that other researchers can apply to reasoning-trace analysis. It also extends distractor-robustness findings beyond inserted objects to background-context distractors via Waterbirds.

Real-world applications:

• Assistive and accessibility systems that describe images: misclassifying the main subject because of clutter in the scene directly harms users relying on accurate captions.
• Medical or industrial image triage, where irrelevant objects, text overlays, or artifacts in a frame could pull a reasoning model toward the wrong conclusion.
• Document and screenshot understanding, where irrelevant text rendered in an image (typographic distractors) mimics the typographic condition tested here.
• Any deployment using long chain-of-thought VLMs, where inflated reasoning-trace length is often treated as a signal of harder reasoning, but here can be uncorrelated — or even inversely related — to correctness.

Industry relevance: The finding that longer reasoning is not reliably better, and that the fraction of distractor-related content in a trace tracks failure, gives practitioners a concrete diagnostic signal. It also implies that naive "let it think longer" scaling strategies will not fix visually cluttered inputs, and that prompt-level guidance can be a cheap partial mitigation without retraining.

Future Directions

• Extending the framework beyond VQA to more complex reasoning-heavy settings such as agentic decision making and multi-step planning, which the authors explicitly flag as challenging and essential next steps.
• Developing more systematic and generalizable mitigation methods than the prompt-based sanity check, which the paper itself characterizes as limited in its ability to improve accuracy.
• Investigating mechanisms that actively suppress distractor attributes during trace generation, since the attribute-blocking experiment suggests these tokens causally influence the final prediction.
• Untangling why distractor severity (number, semantics, and size) all produce similar downward shifts in the length-accuracy curve, and whether these factors share a common underlying cause.

Target Audience

Researchers and engineers working on multimodal foundation models, chain-of-thought reasoning, or robustness evaluation will benefit most. The paper is also useful for benchmark designers interested in controlled intervention methodology and human-in-the-loop dataset construction, and for practitioners evaluating whether long-reasoning VLMs are safe to deploy in visually cluttered environments.

Authors’ abstract

How does irrelevant information (i.e., distractors) affect test-time scaling in vision-language models (VLMs)? Prior work on text-only language models has shown that textual distractors can intensify inverse scaling, causing models to reason longer but less effective reasoning traces. In this work, we investigate whether similar phenomena arise in multimodal settings. We introduce Idis (Images with distractors), a visual question-answering dataset that systematically varies distractors along semantic and numerical dimensions. Our analyses reveal that visual distractors affect reasoning VLMs in a fundamentally different way from textual distractors: although inverse scaling still emerges, visual distractors reduce accuracy without increasing reasoning length. We further show that attribute counts extracted from reasoning traces provide key insights into how distractors interact with reasoning length and accuracy. As a sanity check, we propose a simple prompting strategy that mitigates distractor-driven predictions in reasoning vision-language models.