Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models

AI summary

Overview

• Research area: Multimodal AI — specifically compositional reasoning evaluation and test-time adaptation for contrastive vision-language models (CLIP, SigLIP) and multimodal large language models (GPT-4.1, SmolVLM).
• Technical level: Intermediate. The paper assumes familiarity with image-text embeddings, similarity matrices, vision-language benchmarks, pseudo-labeling, and test-time training, but its core ideas are explained with simple matching arguments.
• Scope: The paper argues that the standard GroupScore metric systematically understates multimodal model capability on group-structured compositional reasoning benchmarks, introduces a corrected GroupMatch metric plus a translation step (SimpleMatch), and proposes an iterative self-improving algorithm, Test-Time Matching (TTM), evaluated across 16 dataset variants.

What This Paper Is About

Multimodal models have been widely reported to perform at or below random guessing on compositional reasoning benchmarks such as Winoground, where a model must decide which of several similar captions matches which of several similar images. The authors show that this conclusion is partly an artifact of the evaluation metric itself, which imposes unnecessarily strict pairwise constraints and therefore counts correct matchings as wrong. Their goal is to replace that metric with a faithful one, show that gains under the new metric can be transferred back to the old one, and then go further with a test-time algorithm that improves models without any external supervision.

Key Contributions

1. Correcting the evaluation metric. The authors show that the widely used GroupScore systematically underestimates model capability, and introduce GroupMatch, which scores the best overall image-to-caption matching instead of isolated pairwise comparisons. They prove that under random guessing, GroupScore = 1 with probability (k−1)!/(2k−1)!, while GroupMatch = 1 with probability 1/k!.
2. Translating GroupMatch back to GroupScore. They show that correctness under GroupMatch can be converted into correctness under GroupScore by overfitting to the induced matching at test time, an approach they call SimpleMatch. This enables GPT-4.1 to produce the first Winoground result surpassing the estimated human performance of 85.5.
3. Test-Time Matching (TTM). They propose an iterative, self-improving algorithm that induces matchings as pseudo-labels, retains only high-confidence ones via a margin threshold, and progressively relaxes the threshold to expand coverage — with no external supervision. This delivers additional gains on top of SimpleMatch, enabling SigLIP-B16 to surpass GPT-4.1 on MMVP-VLM, a new state of the art.
4. Broad applicability. They extend TTM to a global (non-grouped) matching formulation, use it on a generative multimodal model (SmolVLM-256M-Instruct), and demonstrate effectiveness even on benchmarks where metric-induced effects are absent, across 16 dataset variants spanning 2×2, 1×k, and non-grouped settings.

Main Findings

• GroupScore is too strict. For a k×k group, a score of 1 requires 2k²−2k constraints — each diagonal entry must be the largest in both its row and its column. For k=2, the expected random-guessing score rises from 1/6 under GroupScore to 1/2 under GroupMatch.
• SimpleMatch reveals large hidden capability. SigLIP-B16 improves from 10.25 → 67 on Winoground, 22.96 → 81.48 on MMVP-VLM, and 30.33 → 88 on ColorSwap, surpassing all previous results without additional data. GPT-4.1 improves from 69.75 → 91.38 on Winoground, 68.15 → 88.52 on MMVP-VLM, and 91.08 → 97.42 on ColorSwap.
• First result above estimated human performance on Winoground. The previous state of the art was 58.75 (scaffolding and prompt tuning GPT-4V); GPT-4.1 with SimpleMatch reaches 91.38, above the estimated human performance of 85.5 reported by Thrush et al. (2022).
• TTM adds further, non-trivial gains. SigLIP-B16 reaches 72.50 ± 0.64 on Winoground, 89.44 ± 0.96 on MMVP-VLM, and 94.25 ± 0.43 on ColorSwap. SigLIP-L16 reaches 72.75 ± 0.64 on Winoground and 96.08 ± 0.43 on ColorSwap, matching GPT-4.1 on ColorSwap.
• New state of the art on MMVP-VLM. TTM enables SigLIP-B16 (89.44) to surpass GPT-4.1 (88.52) on MMVP-VLM. The prior best reported result on MMVP was 70.7 via a GPT-4o multi-agent system with tool use; the prior best on ColorSwap without training-set access was 87.33 (95.33 with training-set finetuning).
• Gains over SimpleMatch are consistent. Relative gains from TTM reach up to 10.5% and relative error reduction up to 54.8% across models and datasets. Notably, SigLIP-L16 on ColorSwap goes from 91.33 to 96.08 with a 54.8% relative error reduction.
• TTM works on a generative multimodal model. With SmolVLM-256M-Instruct, TTM improves over SimpleMatch from 61.75 → 63.38 ± 0.67 on Winoground, 76.30 → 81.67 ± 1.52 on MMVP-VLM, and 80.00 → 85.17 ± 1.09 on ColorSwap.
• TTM works where metric changes cannot help. On 1×k benchmarks, GroupMatch and GroupScore coincide, so no metric-induced boost is possible. On SugarCrepe (1×2 groups, 4 subsets) and WhatsUp (1×4 groups, 2 subsets), TTM still delivers substantial gains, including up to 85.7% relative gains on WhatsUp.
• The 2×2 WhatsUp variants show even larger effects. Following Li et al. (2025), WhatsUp was converted into 4 directional variants with 2×2 groups; TTM yields up to 135.1% relative gains and 95.5% relative error reduction over SimpleMatch (Table 10, Section B.2).
• The global variant generalizes beyond groups. On non-grouped variants with SigLIP-B16, a single global assignment already beats raw GroupScore: Winoground 10.25 → 44.38, MMVP-VLM 22.96 → 39.63, ColorSwap 30.33 → 88.00. The global TTM variant then improves these further to 46.78 ± 1.05, 44.54 ± 2.02, and 92.00 ± 1.24 respectively.
• Threshold scheduling matters. A decaying schedule (τ₁ = 2.0 → τ_T = 0) performs best; a constant threshold (τ = 2.0) plateaus due to false negatives in later iterations; an ascending schedule (τ₁ = 0 → τ_T = 2.0) yields no gains because the model overfits to all pseudo-labels in the first iteration.
• Results are averaged over four random runs with standard deviations reported throughout.

Methodology in Plain English

The authors start by inspecting the scoring rule used by benchmarks like Winoground. In a group of k images and k captions, the standard metric awards a point only if every image is closer to its own caption than to any other, and vice versa. The authors show this is a very demanding condition — for random scores it succeeds with probability (k−1)!/(2k−1)! — and it can mark a model wrong even when the model has effectively identified the right overall pairing.

Their replacement, GroupMatch, instead asks: does the ground-truth image-to-caption pairing have a higher total similarity than every other possible pairing? This is checked by comparing the sum of similarities along the correct matching against the sums of all alternatives, which for k=2 reduces to checking whether s₁₁ + s₂₂ > s₁₂ + s₂₁. Random guessing succeeds here with probability 1/k!.

Because benchmark leaderboards use the old metric, the authors add a bridging step: once a model identifies the most likely matching under GroupMatch, that matching can be overfit at test time, which guarantees a GroupScore of 1. This is SimpleMatch.

The second half of the paper builds on this. TTM runs for T iterations. At each iteration, the current model proposes a matching for every test group; each proposal comes with a margin — the gap between the total similarity of the chosen matching and the best alternative. Only groups whose margin clears a threshold τ_t are used as pseudo-labels, and the model is finetuned on them. The threshold starts high to keep pseudo-labels precise and decays over rounds to bring in more of the test set as the model improves. In practice, the initial threshold is set so roughly 15%–30% of groups are matched, and the final threshold so more than 90% of the test set is covered; both cosine and linear decay schedules work. Runtime scales as O(T·C_ft), where C_ft is the per-iteration finetuning cost, and the authors report strong improvements even with T = 3 or 10.

For datasets without groups, they treat the whole dataset as one matching problem between all images and all captions, solved as a classical assignment problem via strongly-polynomial algorithms such as the Hungarian algorithm (Kuhn, 1955), with thresholds applied at the level of individual pairs rather than groups.

Why This Matters

Impact on research. The paper challenges a widely cited conclusion — that frontier multimodal models perform at or below chance on compositional reasoning — by attributing part of that result to the metric rather than the model. If accepted, it changes how benchmarks with group structure (particularly k×k groups) should be scored and reinterprets a body of prior negative results. It also shows that test-time self-improvement works even on 1×k benchmarks where no metric artifact exists, which is an argument that the gains are real rather than an accounting change.

Real-world applications:

• Image-text retrieval and search, where systems must decide which of several similar captions or queries best describes an image, and where near-miss distinctions (object order, attribute binding, relations) determine correctness.
• Accessibility and automatic captioning, where the difference between "a dog chasing a man" and "a man chasing a dog" is a substantive error rather than a stylistic one.
• Content moderation and safety review, where models must correctly bind attributes and relations to the right subject in an image.
• Model evaluation and procurement, where organizations comparing multimodal systems need metrics that do not systematically understate the capability of the systems being compared.

Industry relevance. The TTM procedure requires no external labels, no additional data, and no human supervision — only the test set and the model itself. It applies to contrastive models and, as demonstrated with SmolVLM-256M-Instruct, to generative multimodal models as well. That makes it attractive as a deployment-time adaptation step, with the caveat that it involves finetuning at inference time and therefore carries compute cost proportional to the number of iterations.

Future Directions

• General rectangular groups. The paper focuses on k×k and 1×k groups and defers general m×k groups to Section A.2, noting the expected random-guessing score improves from 1/k^m to (k−m)!/k!. Fully validating GroupMatch and TTM on such shapes is left open in the main text.
• Larger generative models and other similarity measures. The generative-model experiment uses one relatively small model (SmolVLM-256M-Instruct) with VQAScore. The authors note that GPT-5 did not support log-probability outputs at the time of writing (September 2025), a limitation for computing accurate similarity scores more broadly.
• Robustness of pseudo-label selection. TTM's behavior depends on balancing false positives and false negatives through the threshold schedule. How sensitive the method is to group size, margin distributions, and benchmark-specific score scales — beyond the threshold ablations reported in Section 4.6 — is a natural extension.
• Runtime and iteration trade-offs. The authors state that the runtime is comparable to standard test-time training methods and that small T works well, but the detailed analysis is deferred to Section B.3, leaving the practical scaling of T and finetuning cost a question for further work.

Target Audience

Researchers working on multimodal evaluation, compositional reasoning, and vision-language benchmarks; practitioners deploying CLIP/SigLIP-style retrieval or MLLM systems who want test-time adaptation without labels; and benchmark designers deciding how to score group-structured tasks. Readers evaluating leaderboard claims about model failure on compositional reasoning will find the metric critique and the k=2 probability arguments (1/6 versus 1/2 under random guessing) the most directly useful parts.

Note: the supplied paper content is truncated mid-Section 4.5, so results for CLIP-B32, the full non-grouped results, the Section 4.6 ablations, and the appendices are not available in the provided text.

Authors’ abstract

Frontier AI models have achieved remarkable progress, yet recent studies suggest they struggle with compositional reasoning, often performing at or below random chance on established benchmarks. We revisit this problem and show that widely used evaluation metrics systematically underestimate model capability. To correct this artifact, we introduce a group matching score that more faithfully evaluates model capability. Moreover, correctness under the new metric can be translated into correctness under existing metrics via a simple overfitting step. This adjustment enables SigLIP-B16 to surpass all previous results and GPT-4.1 to yield the first result surpassing estimated human performance on Winoground. Building on this insight, we propose Test-Time Matching (TTM), an iterative, self-improving algorithm that further bootstraps model performance without any external supervision. TTM delivers additional, non-trivial improvements: for example, TTM enables SigLIP-B16 to surpass GPT-4.1 on MMVP-VLM, establishing a new state of the art. TTM also extends beyond contrastive vision-language models, yielding clear gains on a generative multimodal model across benchmarks. Importantly, TTM remains broadly effective even on benchmarks without metric-induced effects or group structures, achieving relative gains up to 85.7% on challenging datasets such as WhatsUp. Across 16 dataset variants spanning diverse setups, our experiments demonstrate that TTM consistently improves model performance and advances the frontier of compositional reasoning.