Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models

AI summary

Overview

• Research area: Robustness and evaluation of large vision-language models (LVLMs) — specifically typographic attacks (misleading text inserted into images) and scene-text reading in visual question answering (VQA).
• Technical level: Intermediate. The core ideas are intuitive, but the benchmark design, metrics (R-CLIP-M), and training baselines assume familiarity with VQA evaluation and LVLM fine-tuning.
• One-sentence scope: The paper argues that typographic-attack robustness must be judged together with text-reading ability, and introduces a same-scene counterfactual benchmark plus a training baseline to measure and improve both at once.

What This Paper Is About

Existing evaluations of typographic attacks focus almost entirely on object recognition, so a model that simply learns to distrust all text in images can look "robust" while actually losing the ability to read anything. The authors point out that real tasks often require both skills at once — recognizing a pedestrian and reading a traffic sign — so ignoring all text is not an acceptable defense. They introduce Read-or-Ignore VQA (RIO-VQA), a task requiring a model to decide from context when to read relevant scene text and when to ignore inserted distractor text, and build RIO-Bench to measure it.

Key Contributions

1. RIO-VQA task formulation. A unified evaluation requirement in which a model must remain robust to inserted distractor text while still reading relevant scene text required by the question, deciding which to do based on context.
2. RIO-Bench benchmark. A same-scene counterfactual benchmark that holds the scene fixed and varies only two axes — question intent (object vs. text) and text condition (clean vs. attack) — producing four matched settings: Obj-Clean, Obj-Attack, Text-Clean, and Text-Attack.
3. Findings on a fundamental misalignment. The benchmark shows that object-centric defenses can gain apparent robustness by broadly suppressing text sensitivity, at the cost of text-reading performance — a failure mode that object-centric evaluation alone cannot detect.
4. A defense baseline (RIO-RT). An architecture-agnostic, data-driven defense trained on the RIO-Bench training split that improves robustness while largely preserving scene-text understanding across six recent LVLMs, outperforming prior text-ignoring baselines.

Main Findings

• Original LVLMs are vulnerable on both question types. Unmodified models show susceptibility to typographic attacks on object-centric and text-centric questions alike, which the authors attribute to over-reliance on textual cues.
• Object-centric protocols hide the trade-off. Because they only test object questions, they cannot distinguish robustness achieved by selectively ignoring misleading text from robustness achieved by degrading text sensitivity generally.
• CoT defense gives limited and uneven gains. A chain-of-thought defense (described as the only existing LVLM-focused defense) provides limited robustness improvements, often reduces Text-VQA accuracy, and increases inference cost. Its object-centric prompt design shifts focus toward object evidence.
• IT-RT improves object robustness but damages text reading. Ignore-Text Robust Training, adapted from Defense-Prefix and fine-tuned only on Obj-Attack (Overlay-Attack hard) data, substantially improves Object-VQA robustness but typically reduces text-centric performance. On LLaVA-1.5-7B, Text-Clean accuracy falls from 49.5 to 30.3; on LLaVA-1.5-13B from 54.5 to 15.3; on SmolVLM-2B from 72.3 to 35.4.
• RIO-RT improves robustness while largely preserving text reading. Trained on a balanced mixture of Obj-Attack and Text-Attack instances, RIO-RT improves object-centric robustness and also improves robustness under Text-Attack settings. For example, on Qwen3-VL-8B, Text-Clean is 84.1 versus 84.0 for the original model, while Text-Attack average rises to 84.6 (up 8%), and Obj-Attack MC rises to 98.8 (up 35%).
• Robustness transfers to unseen attack styles. Although IT-RT and RIO-RT are trained only on randomized overlay attacks, both show improved robustness under SceneTAP-Attack, which uses diffusion-based scene-coherent text insertion. Qualitative examples on Qwen-3-VL show the original model being misled while RIO-RT answers correctly.
• Read-or-ignore behavior appears to be learned mainly in the language component. In an ablation on LoRA parameter placement, tuning both vision and language (V+L) gave the most balanced performance; tuning the language model alone (L) performed comparably in many cases, whereas tuning only the vision encoder (V) failed to balance the two requirements. This contrasts with prior vision-centric defenses that modify only the vision encoder.
• Attention visualizations support selective rather than blanket suppression. Under Obj-Attack, the original LLaVA-1.5-7B strongly attends to the attack word "Remote Control" at layer 31, while IT-RT and RIO-RT shift attention toward the correct object. Under Text-Attack, RIO-RT attends to both "iMac" and the distractor "dell" but shows stronger activation on the correct text in intermediate layers (e.g., layers 10 and 21) and outputs "iMac"; IT-RT attends to both words yet produces an illogical answer.
• Distractor strength matters in controlled ways. For object questions, accuracy drops as the inserted distractor becomes more semantically similar to the ground-truth object (easy/medium/hard levels from the Open Images hierarchy). For text questions, performance generally degrades as distractors are placed closer to the key-text region needed to answer.
• Scaling effects are hidden by clean accuracy. Qwen3-VL-2B, 4B, and 8B achieve similar Obj-Clean accuracy (96.4, 96.6, 96.9) and similar Text-Clean accuracy (81.2, 82.3, 84.6), yet Obj-Attack robustness improves substantially with scale (38.3 → 65.1 → 70.7), with Text-Attack avg also rising (72.7 → 75.1 → 77.1). InternVL3.5 shows a similar trend across 8B, 14B, and 38B.
• Frontier models are more robust but do not solve the problem. GPT-5.4 and Gemini-2.5-Pro reach Obj-Attack scores of 74.3 and 77.7 respectively, but still lag their Obj-Clean performance by roughly 20 points.
• Benchmark quality was human-audited. On 400 samples (100 per stratum across {Obj/Text-VQA} × {Overlay/SceneTAP-Attack}), 95.5–97.5% of samples received majority-desirable judgments and only 0.0–4.0% received majority clear-error judgments, with 90.0–94.3% raw inter-annotator agreement and Gwet's AC1 of 0.89–0.94.

Methodology in Plain English

Building a fair comparison. The central design idea is to keep the underlying image completely fixed and change only two things: whether the question asks about an object or about text in the scene, and whether the image is clean or contains inserted distractor text. This controls for confounds like text density or layout that would arise if object questions and text questions came from different datasets.

Data source. The authors build on TextVQA, whose images come from Open Images. TextVQA supplies the text-reading questions and answers (used directly for Text-Clean), while Open Images supplies object class annotations, which are used to write matching object-centric questions for the same images (Obj-Clean). Attacks are then applied to both question intents to produce Obj-Attack and Text-Attack. The base scenes comprise 21,953 training images and 3,166 validation images; the full benchmark contains 241,483 training and 34,826 validation instances.

Question formats. Object-centric questions come in two forms: multiple-choice ("Which object is present in the image?" with three distractor options sampled at sibling, grandparent, and higher levels of the Open Images hierarchy for controlled difficulty) and open-ended ("What objects can be seen in the image?"). Text-centric questions are the original open-ended TextVQA questions.

Attack design. Distractor text is always placed outside existing text regions, using TextVQA OCR bounding boxes, so it never simply occludes the answer. Two attack families are used: (1) Overlay-Attack, controlled digital overlays with randomized font size, color, and placement, with difficulty set by semantic proximity for object questions (using MC negative labels) and by spatial proximity to the key text for text questions (with distractor phrases generated by Llama-3 to contradict the correct answer); and (2) SceneTAP-Attack, which reuses the same distractor words but inserts them scene-coherently via an image-editing generative model, with OCR text regions masked and excluded from insertion locations.

Metrics. Multiple-choice object questions use standard accuracy. Open-ended object questions use a new metric, R-CLIP-M@K, which extends CLIP-M (checking whether the ground-truth label appears among the top-K nearest class names by CLIP embedding distance) by subtracting the attack word's retrieval score, so that predictions consistent with the distractor text are penalized. Text questions use standard VQA accuracy, min(#humans agreeing / 3, 1) against 10 human answers.

Defense baseline. From the training split, the authors construct two supervised fine-tuning baselines using the same backbone and budget (16K samples, LoRA r=16, α=16, 1 epoch, AdamW, batch size 16, learning rate 1×10⁻⁴ with cosine decay, on a single H100). IT-RT is fine-tuned on Obj-Attack Overlay hard data (8K multiple-choice + 8K open-ended). RIO-RT is fine-tuned on a controlled mixture of Obj-Attack Overlay hard (4K multiple-choice + 4K open-ended) and Text-Attack Overlay hard (8K). Both are trained only on Overlay-Attack subsets.

Evaluation. Six LVLMs are benchmarked — LLaVA-1.5-7B/13B, Qwen-2.5-VL-7B, Qwen3-VL-8B, Llama-3.2-11B-Vision, and SmolVLM-2B — alongside a broader scaling study with Qwen3-VL-2B/4B/8B, InternVL3.5-8B/14B/38B, GPT-5.4, and Gemini-2.5-Pro. Attention maps of LLaVA-1.5-7B are visualized as relative image-token attention normalized by a generic captioning prompt.

Why This Matters

• Impact on research. The paper reframes typographic-attack robustness as a selective decision problem rather than a pure suppression problem. It shows that a widely used evaluation convention — measuring robustness only on object-centric tasks — can reward defenses that quietly destroy reading ability, and provides a benchmark that makes the trade-off measurable.
• Real-world applications:
  • Autonomous driving, where a system must recognize pedestrians while correctly reading traffic signs and must not be fooled by text painted on the road or on vehicles.
  • Embodied AI agents, which need to interpret written instructions or notices in the physical world to act appropriately.
  • Assistive and accessibility tools that read scene text for users, where an attacker-inserted word could otherwise override what the user is told.
  • Document, signage, and retail understanding pipelines, where both object recognition and accurate text transcription are required from the same image.
• Industry relevance. The benchmark provides a standardized testbed for teams deploying LVLMs in safety-relevant or text-heavy settings, and the RIO-RT recipe offers a concrete, architecture-agnostic training procedure that does not require redesigning a model's vision encoder. The paper's finding that strong clean accuracy does not imply selective robustness also gives a practical warning against relying on headline benchmark numbers.

Future Directions

• Broadening beyond VQA. The paper explicitly scopes RIO-Bench to LVLM×VQA and does not cover the full space of real-world typography or adversarial threat models; extending to other tasks and modalities is left open.
• Stronger and more varied training recipes. RIO-RT is presented as a minimal reference baseline, trained with a fixed supervised fine-tuning recipe and budget. Broader training data, alternative objectives, and generalization across attack types and downstream tasks remain future work.
• Generalizing across domains. The authors note that generalization across domains, attack strategies, and downstream tasks remains future work, even though preliminary out-of-benchmark checks on TypoD and additional text-VQA benchmarks showed consistent trends.
• Understanding the mechanism. The LoRA ablation suggests read-or-ignore behavior is primarily governed by the language model's reasoning capability, raising the question of how to design architectures or objectives that make this contextual prioritization reliable by default.

Target Audience

Researchers and practitioners working on vision-language model robustness, multimodal safety, and adversarial evaluation; benchmark designers who need to evaluate text-reading and object recognition jointly; and engineers deploying LVLMs in autonomous driving, embodied AI, or any setting where both recognizing objects and reading scene text matter and where inserted text could mislead the model. Readers familiar with VQA metrics and LVLM fine-tuning will get the most out of the experimental sections.

Authors’ abstract

Large vision-language models (LVLMs) are vulnerable to typographic attacks, where misleading text inserted into an image can override visual understanding. However, existing evaluation protocols and defenses are largely focused on object recognition and do not consider text-reading capability. This is a critical oversight: real-world scenarios often require both recognizing objects and reading scene text (e.g., recognizing pedestrians while reading traffic signs), where simply ignoring all text for robustness is unacceptable in practice. To address this gap, we introduce a novel task, Read-or-Ignore VQA (RIO-VQA), which jointly evaluates both requirements: models must decide, from context, when to read scene text and when to ignore inserted distractor text. To evaluate this capability, we present RIO-Bench, a same-scene counterfactual benchmark that holds the scene fixed while varying only question intent (object vs. text) and text condition (clean vs. attack), enabling direct comparisons of model behaviors with reduced confounding factors. Using RIO-Bench, we highlight a trade-off: representative defenses developed in object-centric settings can achieve robustness by suppressing text sensitivity, at the cost of text-reading performance (i.e., "ignoring" text). Motivated by this trade-off, we provide a data-driven defense baseline that improves both requirements on RIO-Bench, complementing prior text-ignoring baselines. Overall, this work highlights a fundamental misalignment between the current object-centric robustness scope and real-world multimodal requirements, providing a principled path toward reliable LVLMs.