ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Images

AI summary

Overview

• Research area: Computer vision and multimodal AI, specifically vision-language model (VLM) evaluation on camouflaged, Ishihara-style chromatic images.
• Technical level: Intermediate.
• Scope: This paper introduces ChromouVQA, a large-scale multi-task benchmark of 70,200 camouflage renderings built from 17,100 silhouettes, evaluates humans and a wide range of open-source and proprietary VLMs on it, and proposes a contrastive fine-tuning recipe that substantially narrows the human–model gap.

What This Paper Is About

VLMs perform well on clean, high-contrast images, but they break down when foreground and background share similar colors and textures, so that local cues fail and the model must integrate subtle chromatic differences across an entire image to recover a shape. The authors build a controlled, Ishihara-style benchmark to measure exactly this failure, then test whether a contrastive training objective that pairs clean silhouettes with their camouflaged renderings can teach models to recover global shape. The goal is a reproducible, tunable testbed for figure–ground segregation in VLMs, plus a strong baseline method for the problem.

Key Contributions

1. ChromouVQA benchmark and generation pipeline. A large-scale, multi-task dataset of Ishihara-style camouflaged images (70,200 camouflaged renderings from 17,100 silhouettes, 61 palette–shape configurations, nine VQA task types) built with a highly flexible and controllable image generation pipeline.
2. Comprehensive empirical study. Evaluations across a wide range of open-source and proprietary VLMs plus human participants, quantifying significant gaps between current models and human perception on camouflaged image tasks.
3. Contrastive fine-tuning framework. A model-agnostic recipe that uses silhouette–camouflage pairs as anchors and positives with unrelated silhouettes as negatives, encouraging global shape recovery and providing a strong baseline.
4. Extended filling algorithm. A ray casting-based filling method (odd–even test on a 0.5-pixel grid over the bounding box) that replaces sparse-vertex sampling from prior Ishihara-style generators, which left gaps in shapes with tips or concavities.

Main Findings

• Humans far outperform all evaluated VLMs. Human participants reached 88.4% overall accuracy on camouflage tasks (and 100% on silhouettes), while the best model, GPT-4o, reached only 20.4%.
• Baseline model ranking. GPT-4o leads at 20.4% overall, followed by InternVL2.5-8B at 18.6%. Qwen2.5-VL-32B performs worst at 2.1%. Other models range from Phi4-Multimodal-5.6B at 4.2% to Claude Sonnet 4 at 18.1%.
• Models are "shape-smart" but "pattern-blind." VLMs score well on clean, high-contrast silhouettes (e.g., Claude Sonnet 4 at 86.7%) but collapse on camouflaged variants, blurring hue boundaries, mis-segmenting foregrounds, and failing to integrate dispersed dots.
• Difficulty depends on palette and fill geometry. Accuracy is relatively stable across sRGB-bounded palette configurations but drops with Tri-color Ishihara settings; cross-shaped fills perform worst due to fragmented geometry.
• Task type matters. Size Comparison and Spot Difference are easier because they rely on visual contrast and local comparison, whereas Rotation, Enumeration, and Math require abstract reasoning and geometric comprehension under noise.
• Larger models are not always better. Qwen2.5-VL-32B (2.1%) trails smaller models; the authors attribute this to low-resolution, large-patch encoders blurring faint hue edges and to extra parameters reinforcing spurious correlations when camouflaged images are rare in pretraining data.
• Contrastive adaptation produces large gains. Contra-Qwen reaches 61.0% overall (an increase of 54.1% over its baseline, with gains of 58.5% on math and 86.7% on size sorting). Contra-InternVL rises to 60.6% (42.0% overall increase, counting improved by 62.4% and size comparison by 23.8%). Contra-LLaVA improves to 56.6% (48.5% overall increase, occlusion improved by 28.4% and spot-the-difference by 55.3%).
• Adapted models rival larger proprietary systems on silhouettes. All adapted models exceed 65% on silhouettes, with Contra-Qwen at 71.9%.
• The contrastive loss weight has an optimum. Ablation on Contra-Qwen shows α = 0 reduces to the baseline (31.7% overall), performance improves up to α = 0.2 (60.9%), and peaks at α = 0.5 (61.0%). Larger α values boost recognition and occlusion but reduce reasoning performance (α = 0.8 gives 60.6%, α = 1 gives 59.5%).

Methodology in Plain English

The authors generate images in the style of Ishihara color-vision test plates, but for benchmarking rather than medical use. Silhouettes (text, 3-digit numbers, and 134 animal shapes) are filled with colored dots, polygons, crosses, or stars, and the colors of the fill and the background are chosen to be similar, with tunable separation based on nine palettes extracted from real Ishihara plates plus 16 sampled color configurations. A ray casting method ensures clean fillings for shapes with tips and concavities. Each rendering is paired with a question and answer, covering nine tasks: Count, Enumeration, Spot Difference, Size Comparison, Size Sort, Pattern Recognition, Rotation-Invariant Perception, Occlusion Reasoning, and Mathematical Calculation.

They then benchmark ten human evaluators and a broad set of VLMs on the resulting test images. For the contrastive recipe, clean silhouettes act as anchors, their camouflaged counterparts as positives, and unrelated silhouettes as negatives; an InfoNCE loss aligns the vision encoder's pooled global embeddings of anchor and positive, and this is combined with a standard multimodal autoregressive answer-generation loss using a weighting parameter α. Three backbones (Qwen2.5-VL-7B, InternVL-2.5-8B, LLaVA-1.6-7B) are fine-tuned with the vision encoder, projection layers, and language model all trainable, using α = 0.5, τ = 0.7, and the 10,800 reserved silhouette–camouflage pairs, on 8 A100 GPUs with batch size 16, the Adam optimizer, learning rates of 1×10⁻⁷ (language model), 1×10⁻⁵ (projection layers), and 1×10⁻⁶ (vision encoder), for 2 epochs. Only camouflaged images are needed at inference time. Open-ended outputs are graded with GPT-4o-mini followed by human validation.

Why This Matters

The paper exposes a concrete weakness in current VLMs: they can recognize objects in clean images but fail to separate figure from ground when chromatic camouflage disrupts local cues. Because the benchmark is synthetic and fully controllable, it enables interpretable performance curves tied to color distance, dot density, and pattern geometry, and its full metadata supports reproducibility.

Real-world applications:

• Safety-critical displays: aviation cockpit displays and warning icons that appear against patterned or color-similar backdrops.
• Medical and diagnostic imaging: detecting structures embedded in low-contrast or textured tissue backgrounds.
• Search and rescue or surveillance: spotting concealed or camouflaged targets in cluttered natural scenes.
• Accessibility and color-vision research: studying how color separation and shape filling affect perception, using a controlled variant of Ishihara-style stimuli.

Industry relevance: the benchmark gives model developers a targeted test for global grouping and figure–ground segregation, areas where scaling parameters alone does not help, and the contrastive recipe offers a model-agnostic training technique, plus a reference baseline, that vendors can apply to their own vision encoders. The finding that larger models can underperform smaller ones has direct implications for architecture and resolution choices in production multimodal systems.

Future Directions

• Expanding the benchmark with more diverse patterns and task types to further challenge VLMs, as stated in the conclusion.
• Investigating why additional model parameters and larger patches sometimes hurt, and whether higher-resolution or finer-grained vision encoders preserve the faint chromatic edges that camouflaged images depend on.
• Tuning the balance between recognition/occlusion gains and reasoning losses at higher contrastive weights, since larger α improves some tasks while lowering overall scores.
• Testing whether the contrastive framework transfers beyond these three backbones, and whether it affects performance on standard, non-camouflaged VQA benchmarks (not reported in this paper).

Target Audience

Researchers and engineers working on multimodal models, VQA benchmarks, and visual perception will benefit most, particularly those interested in figure–ground segregation, robustness to low-contrast or camouflaged inputs, and parameter-efficient or contrastive fine-tuning strategies. The benchmark's controlled design also makes it useful to cognitive scientists and perception researchers who want a reproducible, human-referenced comparison against model behavior.

Authors’ abstract

Vision-Language Models (VLMs) have advanced multimodal understanding, yet still struggle when targets are embedded in cluttered backgrounds requiring figure-ground segregation. To address this, we introduce ChromouVQA, a large-scale, multi-task benchmark based on Ishihara-style chromatic camouflaged images. We extend classic dot plates with multiple fill geometries and vary chromatic separation, density, size, occlusion, and rotation, recording full metadata for reproducibility. The benchmark covers nine vision-question-answering tasks, including recognition, counting, comparison, and spatial reasoning. Evaluations of humans and VLMs reveal large gaps, especially under subtle chromatic contrast or disruptive geometric fills. We also propose a model-agnostic contrastive recipe aligning silhouettes with their camouflaged renderings, improving recovery of global shapes. ChromouVQA provides a compact, controlled benchmark for reproducible evaluation and extension. Code and dataset are available at https://github.com/Chromou-VQA-Benchmark/Chromou-VQA.