Seeing Red, Thinking Bad: Color Bias in Vision Language Models

AI summary

Overview

Research area: Multimodal machine learning / Vision Language Model (VLM) robustness and safety, specifically cross-modal representation analysis and adversarial visual prompting.

Technical level: Intermediate. The paper assumes familiarity with VLM architectures, vision encoders, CLIP embeddings, and standard evaluation metrics (F1, sentiment classification), but its core ideas are described in accessible terms.

Scope: This paper systematically investigates how purely cosmetic visual styling of rendered text — word color and text–background contrast — shifts VLM sentiment predictions and question-answering behavior, even when the underlying text is unchanged.

What This Paper Is About

Vision language models are increasingly used in decision-making contexts such as recruitment and recommendation, yet they are largely trained on non-public data, leaving their internal biases poorly understood. Prior work has shown that VLMs can produce different answers when the "same" text is presented as plain tokens versus as rendered text inside an image. This paper asks a sharper question: if you keep the words identical and change only their visual appearance — the color of a few words, or how faded the text is — does the model's interpretation change, and can that shift be traced to the model's internal visual representation? The authors build a controlled testbed to measure these effects across four open-source VLMs and two very different tasks (subjective sentiment analysis and objective question answering).

Key Contributions

1. Stealth Visual Prompts. A methodology for perturbing the visual rendering of text (color hue and intensity, text–background contrast, saliency competition) while keeping the underlying string content, word layout, and semantics perfectly fixed — so effects can be attributed to styling alone rather than to added text or explicit instructions.

2. The Stealth Prompt Testset. A three-part benchmark probing distinct model regimes: 100 short mixed-sentiment sentences (word-level color bias, 37 visual conditions each), 100 longer structured sentences with sentiment separated across halves (discourse and positional effects), and 100 SQuAD-derived question-context pairs (contrast and saliency effects in VQA).

3. A dual diagnostic probe framework. A CLIP representation probe that reveals how hue shifts vision-encoder embeddings along human-interpretable semantic axes (valence, emotion, safety, temperature), paired with a VLM-based single-word OCR proxy that calibrates a model-dependent "readability transition" under declining contrast.

4. Quantified evidence of styling-induced failure. Demonstration that word color can systematically flip sentiment polarity, that positional heuristics can override lexical content under structured discourse, and that reduced contrast pushes several models toward copying visually salient but semantically incorrect decoy words.

Main Findings

• Color reliably biases sentiment for some models, not others. Qwen2-VL-7B showed the largest shifts, reaching +0.42 bias toward POSITIVE when positive words were colored green or blue, and −0.48 toward NEGATIVE when negative words were colored red. IDEFICS2-8B followed with a 0.52 worst-case range. The LLaVA variants were comparatively robust (0.04–0.12 range).

• The bias follows a dose-response pattern. Stronger color intensity generally amplified the magnitude of the sentiment shift, and certain hues produced consistent directional effects rather than random noise.

• Color can override contradictory lexical evidence. In mixed-sentiment sentences, coloring positive words green caused the model to underweight the negative words actually present in the text.

• Structured long sentences trigger positional heuristics that dominate color. Under discourse structure, models defaulted to either primacy or recency with high adherence: LLaVA-Mistral-7B and Qwen2-VL-7B were 100% recency-consistent, IDEFICS2-8B 93% primacy, LLaVA-Vicuna-7B 97% primacy. For three of four models the residual color bias range collapsed to near zero in this regime.

• Hue systematically shifts CLIP's vision-encoder embeddings. Projecting rendered probe words onto text-defined semantic axes showed shared, hue-dependent modulation — most visibly on valence and emotion axes — with projections rising near green (~120°) and dipping around blue (~250°), even though the rendered word never changed.

• Reduced contrast increases saliency-driven errors in VQA. As the non-salient context text faded toward white, the Induced Error Rate (rate at which the model copied a visually salient decoy word) rose for IDEFICS2-8B from 24% to 36% and for LLaVA-Vicuna-7B from 19% to 25%. LLaVA-Mistral-7B stayed flat at 24–27%; Qwen2-VL-7B remained low at 4–6%.

• Making the correct answer visually salient helps, but only where reading works at all. Qwen2-VL-7B's VQA F1 improved from 0.745 to 0.828 when the answer span was rendered in high contrast, whereas the other three models stayed near 0.05 F1 regardless — indicating limited end-to-end reading capability in this setup.

• OCR readability transitions are sharp and model-dependent. A single-word OCR proxy showed OCR accuracy rising steeply over a narrow contrast range, with the transition point varying by both model and font size — a calibration signal that small contrast changes can flip a model between low- and high-access regimes.

Methodology in Plain English

The authors render their test text onto standardized 800×600 pixel white-background images using a fixed font, fixed layout, and fixed line breaks — so that across every experimental condition, the only thing that changes is the styling of targeted words.

For color prompts, they recolor sentiment-bearing words using six canonical hues (red, green, blue, yellow, cyan, magenta) at three fixed RGB intensity levels, leaving all remaining words black. This produces 36 colored variants plus one all-black baseline per sentence. Levels are defined in raw RGB rather than perceptually matched, so the perceived distance from white can vary by hue — the authors treat intensity as a rendering-level control rather than a perceptually normalized quantity.

For contrast prompts, they render text in low-contrast grayscale at six levels (from near-black to near-white) and compute a perceptual contrast measure (CIE ΔE) from the actual rendered image. Two variants are used: Global Contrast, where the whole text is faded, and Saliency Competition, where one span is kept high-contrast black while the rest is faded — forcing the model to choose between a visually loud span and the semantically correct one. Decoy words for the saliency condition are selected as the context word with the highest CLIP semantic similarity to the true answer.

They evaluate four open-source VLMs under deterministic decoding: LLaVA-v1.6-Mistral-7B, LLaVA-v1.6-Vicuna-7B, Qwen2-VL-7B-Instruct, and IDEFICS2-8B. Sentiment bias is measured as the change in average sentiment score relative to the all-black baseline. VQA is measured with token-level F1 plus a custom Induced Error Rate that directly counts decoy copying.

To explain why these behavioral shifts occur, they add two interpretive probes: rendering single words at varying hues and projecting CLIP image embeddings onto ten bipolar semantic axes defined from CLIP text embeddings, and using each VLM as a minimal single-word reader to map out where contrast renders text effectively unreadable.

Why This Matters

Impact on research. The paper reframes visual formatting as an implicit control channel rather than cosmetic decoration. It connects behavioral oddities in VLMs to measurable shifts in vision-encoder latent space, providing a diagnostic vocabulary (stealth prompts, semantic projection probing, induced error rate, readability transitions) that other researchers can reuse. It also highlights a modality-gap-adjacent phenomenon: semantically identical content diverges in representation depending on how it is rendered.

Real-world applications:

• Document and screenshot pipelines. Any VLM that ingests scanned documents, PDFs, slide decks, or UI screenshots is exposed to styling-based manipulation — a supplier could highlight a favorable clause in dark text and fade the unfavorable one.
• Recruitment and recommendation systems. The paper explicitly cites these as motivation; if sentiment toward a candidate's description shifts based on word color, a benign-looking resume template could skew automated screening.
• Content moderation and accessibility. Faded text may be a deliberate evasion strategy, causing moderators to miss harmful passages that a human would still read.
• Adversarial robustness auditing. Stealth prompts offer a low-cost red-teaming primitive: no new words, no injected instructions, nothing a human reviewer would flag as suspicious.

Industry relevance. Because the changes survive human inspection unnoticed, standard content review and prompt-injection defenses are unlikely to catch them. The authors' suggested safeguards map directly onto production engineering work: normalize rendered text before inference, cross-validate image-based answers against OCR-extracted text, and add style-invariance checks to evaluation suites. This is immediately actionable for any team deploying VLMs on visual document input.

Future Directions

• Extend beyond English and beyond RGB. The authors acknowledge their stimuli are English-only and that intensity was defined in raw RGB rather than perceptually normalized space, leaving open whether the effects generalize across scripts, languages, and perceptually matched color spaces.
• Broaden the rendering factor space. Fonts, font sizes, weight, layout, spacing, backgrounds, and underline/highlight styling are all untouched — a natural next axis for stealth prompt research.
• Move from correlational to mechanistic explanation. The CLIP probe shows that hue shifts embeddings along semantic axes, but the paper explicitly does not claim a literal mechanism for all evaluated architectures. Causal tracing and interventions within individual VLM backbones remain open.
• Scale the OCR proxy into a real text-accessibility model. The single-word proxy does not directly model long-context VQA reading, so a more faithful model of when rendered text becomes effectively invisible to a VLM would sharpen the contrast findings.
• Develop and evaluate defenses. The proposed safeguards (normalization, OCR cross-checking, style-invariance testing) are suggestions rather than tested interventions — benchmarking their effectiveness is an obvious next step.

Target Audience

This paper is most valuable to VLM and multimodal robustness researchers, particularly those working on representation analysis, modality gaps, or adversarial visual prompting. It is also directly relevant to ML engineers and safety practitioners deploying VLMs on document, screenshot, or form-processing pipelines, who need concrete failure modes and mitigation strategies. Additionally, it will interest AI ethics and fairness researchers studying hidden sources of bias in automated decision systems, and benchmark designers looking for evaluation axes that go beyond accuracy on clean inputs. Readers with basic familiarity with vision language models and embedding spaces will get the most out of the diagnostic probe sections.

Authors’ abstract

Vision language models (VLMs) are increasingly used in industrial decision-making systems, such as recruitment support and recommendation. This motivates careful analysis of how VLMs process visual and textual information. In this work, we study how VLMs interpret text rendered as an image, and investigate the influence of visual styling biases. To this end, we introduce Stealth Visual Prompts, which subtly change visual styling of text, such as color and contrast, while preserving semantic content. Using these prompts, we systematically control the visual styling of words in text and measure their impact on the analysis performed by VLMs. We further analyze how such visual perturbations affect the latent representations of the vision encoder. From our experiments, we observed that coloring positive words in green consistently shifts sentiment predictions toward a positive direction. As a result, VLMs often fail to properly account for negative words present in the text. Our analysis suggests that this behavior is correlated with changes in the latent representations of the vision encoder induced by color variations. In addition, we show that reducing text--background contrast increases reliance on visually salient cues and leads to more incorrect Visual Question Answering (VQA) outputs. These results suggest that the visual styling of rendered text can guide VLMs' interpretation in ways that diverge from human semantic understanding. Project page: https://github.com/KohsukeIde/color-bias-vlm