CompoSET: A Controlled Single-Edit Benchmark for Vision-Language Compositionality

AI summary

Overview

Research area: Computer vision / vision-language models — specifically compositional image-text retrieval and benchmark design.

Technical level: Intermediate. The core ideas are accessible without deep math, but familiarity with CLIP-style dual encoders, cosine similarity, and retrieval metrics helps.

Scope: The paper introduces CompoSET, a 1,776-instance benchmark of same-scene, single-edit image-caption pairs built to test whether 22 dual-encoder vision-language models can bind attributes, counts, relations, and states to the correct objects, and finds that all of them perform at or below chance.

What This Paper Is About

Vision-language compositionality is the ability to tell which property, relation, count, or state belongs to which object in an image and in a caption, and to match those bindings across the two. Prior benchmarks test this with image-text matching, but their paired images typically differ in scene, layout, and content — the paper reports 97% of pixels changing on average — so a model can score well by noticing global image differences rather than by understanding composition. The goal of this paper is to build a benchmark where each image pair keeps the scene fixed and differs by exactly one labeled edit, then measure how well current models actually do under that controlled condition.

Key Contributions

1. A reframing of compositionality evaluation as controlled single-edit experimentation. The authors argue that correlational image-caption comparison is confounded and specify seven design criteria a valid benchmark should meet: naturalistic scenes, multi-object scenes, same-scene single-edit pairs, multiple foils per base scene, a meaningful edit taxonomy, caption verbosity levels, and bidirectional retrieval scoring.

2. The CompoSET dataset. 1,776 instances across 80 naturalistic base scenes, with ~22 foils per base (22.2 on average, minimum 15 per scene), 16 edit types grouped into four semantic categories, and paired captions at three verbosity levels (short, medium, long). Base and foil images differ by 8% of pixels on average (mean 0.083, median 0.048), versus ~0.97 for different-scene benchmarks.

3. Evidence that prior progress is likely a benchmark artifact. Across 22 dual-encoder VLMs, every model scores at or below the 16.67% Group chance level on CompoSET, while many of the same models score well above chance on prior benchmarks.

4. A diagnostic structure that prior benchmarks lack. Because each scene has many foils, the authors can separate "scene-hard" failures (an entire scene is difficult) from "edit-hard" failures (specific edits fail in an otherwise tractable scene), and can compare matched modify versus swap edits on the same attribute in the same scenes.

Main Findings

• All 22 dual encoders are at or below chance. The best CompoSET Group score is 13.9% (SigLIP-2 SO400M/14, 95% CI [12.3, 15.5], n = 1,776), below the 16.67% Group chance level. DreamLIP B/16 follows at 13.7%. The same SigLIP-2 SO400M model scores 49.5% on BiVLC, 39.7% on ColorSwap, 31.1% on MMVP, and 75.8% on SugarCrepe++. CLIP B/32 reaches only 4.4% on CompoSET.

• Scale helps only modestly. CompoSET Group rises from CLIP B/32 (4.4%) to CLIP B/16 LAION-2B (7.8%), SigLIP B/16 (10.2%), and SigLIP-2 SO400M/14 (13.9%) — a real gain, but still at or below chance.

• Text-side hard negatives do not transfer. NegCLIP B/32 scores 7.0% on CompoSET, slightly below plain CLIP B/16 LAION-2B at 7.8%, because it was trained on perturbed captions only, while CompoSET also changes the image.

• Caption verbosity degrades most models. SigLIP-2 SO400M/14 drops from 14.6% (short) to 12.4% (long); SigLIP-2 L/16-384 from 14.7% to 12.0%; CLIP B/16 LAION-2B from 9.1% to 6.5%. Long captions stay within the 77-token CLIP-style limit by construction, so the drop reflects dilution of the edit phrase, not truncation. DreamLIP B/16, trained with dense recaptioning and subcaption-level alignment, is flat across levels (13.1%, 14.2%, 13.7%) and tops the long-caption ranking.

• Attribute binding collapses even where detection survives. modify_color spans 17.2–52.4% Group across the 22 encoders, while swap_color sits at or below 5.8% for 21 of them — far below chance, meaning models systematically prefer the wrong assignment rather than guessing. The modify-over-swap gap is positive for every encoder on color (+15.9 to +36.3) and material (+1.7 to +14.4, with modify_material at 1.7–16.4% against swap_material at 0.0–4.3%). DreamLIP B/16 is the exception, reaching 18.6% on swap_color against its own 52.4% on modify_color.

• Scene difficulty and edit difficulty are nearly independent. Only 4.7% of model-scene pairs are both scene-hard and edit-hard, against 6.25% expected under independence. Per-scene margin rankings correlate at mean Spearman ρ = 0.56 across the 22 encoders; scene identity explains 53% of per-scene margin variance (model identity: 10%); 17 of 80 scenes are scene-hard for a majority of models. Edit-hardness is not just edit-type difficulty — removing per-edit-type means leaves the ranking essentially unchanged (ρ = 0.95). 94.7% of per-foil success-rate variance lies within scenes, so a one-foil-per-base benchmark samples only the remaining 5.3%.

• Frontier API models close part of the gap, at per-query cost. On a stratified 200-instance subset, Claude Sonnet 4.6 reaches 42.5% Group — about 3× the best dual-encoder ceiling — while Gemini 2.5 Flash and Claude Haiku 4.5 reach 12.0% and 13.0%. These models show strong position bias (Gemini Flash: 76.5% T2I when the correct image is first versus 30.0% when second), so the authors counterbalance every call by issuing it twice with option order swapped.

• Construction is tightly quality-controlled. 22.4% of attempted foils were discarded (511 of 2,287), with the highest discard rates for modify_presence (41%), swap_material (35%), and modify_transparency (31%). A separate drift audit flags 0.67% residual drift on N = 300 pairs.

• Scenes are genuinely multi-object. OWL-ViT v2 detections (threshold 0.3, 116 categories, 50-scene subset) find a mean of 20.7 objects per CompoSET scene (median 16, IQR 11.2–26.8, range 3–66), versus 5.1 on Winoground, 6.1 on BiVLC, and 6.3 on SugarCrepe.

• Base/foil pairs are tight in CLIP image space. Within-pair CLIP image cosine is 0.986 (median 0.991, IQR 0.983–0.995), while two unrelated CompoSET base scenes score 0.609 — a within-versus-across gap of 0.38, the widest measured. Prior benchmarks range from 0.15 to 0.33 (Winoground 0.17, BiVLC 0.15, ColorSwap 0.26, MMVP 0.31, NaturalBench 0.24, What's Up 0.17, ColorBench-robust 0.33).

Methodology in Plain English

The authors build each test case as a four-way comparison, following the Winoground 2×2 structure. An instance has a base image with its caption and a foil image with its caption. A model scores all four image-caption combinations. Image-to-text (I2T) passes when each image prefers its own caption; text-to-image (T2I) passes when each caption prefers its own image; Group passes only when both hold on the same instance. This yields chance levels of 25% for I2T and T2I and 16.67% for Group, since Group passes exactly when the two matching scores are the top two of four.

Data construction uses a four-stage engine with a frontier captioning model and a frontier image-generation model: write a structured scene specification, propose taxonomy-grounded edits to objects in the scene, render the foil with a custom single-edit prompt, then write paired short/medium/long captions. Every candidate is manually reviewed by an author against four criteria — object counts preserved, no unrelated object added/removed/moved/occluded, the visual edit matches the caption edit, and ambiguous captions disambiguate the target by location or appearance.

Before testing models, the authors verify the benchmark's own properties on three axes: fraction of pixels changed between base and foil, fraction of caption tokens changed, and CLIP image cosine within pairs. They also measure object counts per scene and check that caption verbosity changes context while holding the edit phrase fixed.

Why This Matters

Impact on research. The paper argues that recent reported progress in dual-encoder compositionality may be an artifact of benchmark confounds — models that look competent on BiVLC, ColorSwap, MMVP, or SugarCrepe++ fall to or below chance when the image contrast is controlled. If correct, this resets the baseline for the subfield and gives it a clean target to improve against, rather than a leaderboard that can be climbed without solving binding.

Real-world applications:

• Image and video search over pre-indexed embeddings. If embeddings do not preserve "red tin next to blue bottle" versus "blue tin next to red bottle," retrieval will mis-rank minimally different scenes.
• E-commerce and catalog matching, where product attributes like color, material, pattern, and count must bind to the correct item.
• Safety and content moderation pipelines that rely on cheap pre-indexed similarity rather than expensive per-query model calls.
• Accessibility and captioning QA, where the correctness of a caption hinges on which object carries which property or which side of a scene it is on.

Industry relevance. Dual encoders remain central to scalable retrieval because images and captions are embedded once, indexed, and searched by similarity. The paper notes that frontier models reach 42.5% Group only under prompted pairwise selection at four API calls per instance — a cost incompatible with indexed retrieval. That makes the dual-encoder gap a practical engineering problem, not just an academic one.

Future Directions

• Improving dual-encoder training so that binding survives. The one encouraging signal is DreamLIP B/16, whose dense recaptioning with multiple sub-caption positives is flat across verbosity and shows partial swap_color signal; the paper implies this training axis deserves more investigation.
• Scaling the benchmark without losing control. CompoSET has 80 scenes and 1,776 instances, deliberately smaller than SugarCrepe and larger than Winoground or What's Up; the authors note that scaling to tens of thousands of instances without comparable human review risks reintroducing scene-level drift.
• Extending beyond static single edits. The paper explicitly leaves multi-edit composition and video compositionality open, and notes that generative VLMs required separately reported scoring primitives that were not compared head-to-head with dual encoders.
• Broadening the taxonomy and the source of images. The authors exclude age, gender, emotion, ethnicity, lighting, weather, absolute size, aesthetics, and ordinal position as hard to control cleanly or ethically, and rely on a single frontier image generator with English captions; a second rater, inter-rater reliability statistics, and multi-generator or multilingual validation are open gaps.

Target Audience

Researchers and engineers working on vision-language retrieval, CLIP-style embedding models, or multimodal evaluation design will get the most from this paper. It is directly useful to benchmark authors who need a worked example of how to control confounds, to practitioners choosing or training retrieval embeddings, and to anyone who has relied on Winoground, BiVLC, SugarCrepe, MMVP, or similar benchmarks and wants to understand what those numbers do and do not measure. Readers need only moderate familiarity with cosine-similarity retrieval; the construction and scoring are explained concretely, and the per-edit-type breakdowns make the failure analysis readable without any background in mechanistic interpretability.

Authors’ abstract

Vision-language compositionality is the ability to identify which properties, relations, quantities, and states bind to which entities in both an image and a caption, and to match those bindings across modalities. Existing benchmarks evaluate this ability through image-text matching: a model ranks candidate image-caption match similarities and succeeds when the correct match comes on top. In such datasets, paired images differ by more than a single binding: scene, layout, and content all differ, with 97% of pixels changing on average. Such pairs do not isolate compositionality: results are confounded by global image differences. We introduce Compositional Single-EdiT (CompoSET), a benchmark dataset in which each base image-caption pair has an edited counterpart, or foil, that preserves the scene but differs by a single controlled edit, such as color, count, spatial relation, or object state. Base and foil images preserve naturalistic scene complexity yet are visually near-identical: only 8% of pixels change on average. Our dataset has many foils per base scene (~22) in order to separate scene difficulty from edit difficulty, a distinction one-foil-per-base benchmarks cannot make. CompoSET contains 1,776 instances across 80 naturalistic scenes, spanning 16 edit types in 4 semantic categories, with captions at 3 verbosity levels. We test 22 dual-encoder VLMs and find that all models perform at or below chance level on CompoSET, despite scoring well on prior benchmarks. This suggests that apparent progress under prior image-text matching protocols may be an artifact arising from confounds in benchmark design, and that compositional understanding in dual-encoder VLMs is far from solved. CompoSET addresses the confounds and offers a better controlled benchmark to measure future progress.