Multimodal Reranking Is a System One Decision

AI summary

Overview

Research area: Information retrieval (cs.IR), specifically multimodal reranking — reordering the top candidates returned by a first-stage image/page retriever.

Technical level: Intermediate. The paper assumes familiarity with two-stage retrieval, vision-language models (VLMs), and logit-based scoring, but the core idea is explained without heavy math.

Scope: The paper argues and empirically tests the position that a general-purpose VLM can rerank images and document pages through a single "System One" probabilistic decision over the candidate list, without any reranking-specific training, and that decision tuning on text alone can further improve it.

What This Paper Is About

Multimodal retrieval systems normally use a dedicated, trained reranker as a separate model to reorder the top candidates from a first-stage retriever. This paper asks whether that separate model is necessary: can a general VLM, already served by a system for generation, instead rank the top 16 candidates by reading one probability per candidate from a single forward pass? The authors frame reranking as one instance of a "System One decision" — a state, a typed question, and a set of options — and test whether such decisions, plus light decision tuning on unrelated text tasks, are enough to compete with supervised rerankers.

Key Contributions

1. A reframing of multimodal reranking as a System One decision. The paper casts reranking as a Jev-style request with the query and candidates as the state, "which candidate best matches the query" as the question, and the candidate labels as options, giving one probability per candidate from a single forward pass with no decoding.

2. A controlled readout comparison. The same backbone (Qwen3.8-27B) is evaluated under a listwise "choice" readout (one prompt, labels A–P, 1 model call) and a pointwise "noul" (yes/no) readout (one question per candidate, 16 model calls), establishing that the choice format is the one that actually improves the first stage on MMEB-V2 image retrieval.

3. A controlled tuning comparison. Qwen3.8-27B is compared with JEV-27B-VL, which shares every weight except a LoRA adapter and decision head trained only on text decisions, isolating the effect of text decision tuning on image and page reranking.

4. Benchmark evidence, a scaling study, and a latency study. Results on MMEB-V2 image retrieval (12 tasks), MMEB-V2 visual-document retrieval (24 tasks) and ViDoRe V3 (8 datasets), a five-model Gemma 4 scaling sweep, per-query latency measurements, plus an explicit list of alternative views (AV1–AV5) and experiments that would test the position.

Main Findings

• The choice format beats yes/no, and yes/no can hurt. On MMEB-V2 image retrieval (hit@1, top 16 of Qwen3-VL-Embedding-2B), the first stage scores 74.85. One choice question over the 16 candidates gives 76.76 with Qwen3.8-27B and 76.80 with JEV-27B-VL; one yes/no question per candidate gives 69.50 and 71.42, both below the first stage. The choice is higher on all 12 tasks for both models, and yes/no falls below the first stage on 11 tasks for Qwen3.8-27B and 9 for JEV-27B-VL.

• A zero-shot general VLM beats the smaller dedicated reranker on all three benchmarks. Qwen3.8-27B with the choice readout scores 76.76 hit@1 on MMEB-V2 image, 84.79 nDCG@5 on MMEB-V2 visual documents and 60.94 nDCG@10 on ViDoRe V3, against 74.49, 82.72 and 59.18 for supervised Qwen3-VL-Reranker-2B — gains of 2.3 hit@1, 2.1 nDCG@5 and 1.8 nDCG@10.

• It comes close to, and on one benchmark matches, the 8B dedicated reranker. Qwen3.8-27B matches Qwen3-VL-Reranker-8B on MMEB-V2 visual documents (84.79 vs. 84.78, higher on 9 of 24 tasks). Reranker-8B leads by 1.6 hit@1 on natural images (78.36 vs. 76.76) and by 1.0 nDCG@10 on ViDoRe V3 (61.98 vs. 60.94). The paper describes these as within 1.6 points on each benchmark.

• Text decision tuning helps pages slightly and leaves natural images unchanged. JEV-27B-VL is higher than Qwen3.8-27B by 0.63 nDCG@5 on MMEB-V2 visual documents (17 of 24 tasks, two ties; two-sided sign test p = 0.017) and by 1.09 nDCG@10 on ViDoRe V3 (all 8 datasets; p = 0.008). On natural images the two tie, with JEV-27B-VL higher on 4 of 12 tasks. With the yes/no readout, tuning adds 1.92 hit@1 on images.

• Small general VLMs fail at this; the effect emerges with scale. Among the five Gemma 4 models (E2B at 5.1B, E4B at 8.0B, 12B at 12.0B, 26B-A4B at 25.8B, 31B at 31.3B total parameters), E2B and E4B lower the first stage on all three benchmarks. From 12B upward the readout improves visual-document retrieval by 1.7 to 2.9 nDCG@5 on MMEB-V2 and 2.4 to 3.4 nDCG@10 on ViDoRe V3, while on natural images only 31B improves, by 2.6 hit@1. Gains are not monotonic: 31B is the strongest Gemma on images but below 26B-A4B on both visual-document benchmarks.

• Gains concentrate on queries that contain text. Over the 12 image tasks, the choice readout on Qwen3.8-27B gains on all nine tasks whose query contains text — most on CIRR (+7.1 hit@1), then Wiki-SS-NQ (+3.8), VisDial (+3.3) and FashionIQ (+3.1), and 1.6 to 2.8 points on the other five — and loses on the three image-only query tasks: MSCOCO image-to-text (−4.1), VisualNews image-to-text (−1.7) and NIGHTS (−0.7).

• One pass is competitive but not cheaper on memory or parameters. Per query on one H200 at concurrency 1, Qwen3.8-27B takes 3.97 s with 1 model call, versus 3.67 s for Reranker-8B and 2.75 s for Reranker-2B, each making 16 calls over 29.6k input tokens. That is 1.1 times Reranker-8B with the released code and 1.5 times when the 16 pairs are batched (which brings Reranker-2B and -8B to 1.68 s and 2.61 s). JEV-27B-VL takes 4.48 s; the Gemma 4 models take 0.47 to 1.27 s at 4.4k input tokens per query.

Methodology in Plain English

The authors take the top 16 results from a fixed first-stage retriever (Qwen3-VL-Embedding-2B) and ask a vision-language model which candidate best matches the query.

• One prompt, one pass. The query and all 16 candidates are packed into a single prompt, with candidates introduced as "Candidate A:" through "Candidate P:". Each image is a set of visual tokens. The model is asked, in one sentence, which candidate best matches the query. Instead of generating an answer, the researchers look at the next-token probabilities for the letters A–P, renormalize them so the 16 probabilities sum to one, and rank the candidates by that probability.

• The yes/no alternative. The comparison readout asks the model, separately for each candidate, "does this candidate match the query?" and scores it by the probability of "yes" against "no" (summing both capitalizations). This is the readout that supervised pointwise rerankers like Qwen3-VL-Reranker use, applied here zero-shot.

• Two ways to get the probabilities. Either read the label probabilities directly from an untrained Qwen3.8-27B (thinking disabled), or use JEV-27B-VL, which is the same model plus a LoRA adapter and a decision head trained on text decisions only, with the vision encoder untouched. Because the two models share every other weight, the difference isolates the effect of text decision tuning.

• Benchmarks and metric choices. MMEB-V2 image retrieval (12 tasks, hit@1), MMEB-V2 visual-document retrieval (24 tasks, nDCG@5), and ViDoRe V3 (8 datasets, nDCG@10, queries from all six languages pooled). Video retrieval tasks in MMEB-V2 are excluded because JEV-27B-VL accepts no video. Images are capped at 1800 visual tokens, matching the dedicated rerankers, and every run uses one NVIDIA H200.

• Scaling and cost checks. The same readout is applied, with no training, to five Gemma 4 models to see how performance depends on size; then latency is measured on 100 English ViDoRe V3 queries from two domains with identical candidate sets.

Why This Matters

Impact on research. The paper draws an explicit link between multimodal reranking and the growing literature on extracting calibrated decisions from general models, and it isolates two variables (question type and decision tuning) in a way that reranking papers usually do not. It also commits to stating its confounds openly: the zero-shot 27B model is both larger and from a newer generation than the supervised baselines, so a dedicated reranker built on the same backbone might still win.

Real-world applications:

• Retrieval-augmented assistants that already serve a VLM. A system that already runs a large VLM for generation could rerank candidates with the same model, avoiding a separate model to train, serve and keep up to date.
• Visual document search (financial reports, scientific PDFs, slide decks, ESG and pharmaceutical documents), where the choice readout gives the largest measured gains over the first stage (5.6 nDCG@5 on MMEB-V2 visual documents, 7.9 nDCG@10 on ViDoRe V3 for Qwen3.8-27B).
• Image search with text descriptions, such as fashion or composed image retrieval, where gains are largest (CIRR +7.1 hit@1, Wiki-SS-NQ +3.8, VisDial +3.3).
• Cost-sensitive deployment, where smaller Gemma 4 models answer a choice prompt in 0.47 to 1.27 s at 4.4k input tokens, at the cost of quality on document benchmarks.

Industry relevance. The latency measurements give a concrete trade-off: one 27B forward pass over 16 candidates costs 3.97 s per query, comparable to the 3.67 s of a supervised 8B reranker that scores the 16 pairs one after another, and 1.5 times that reranker when it batches. The paper also notes the readout's practical ceiling of 16 candidates per prompt under labels A–P, against top-100 reranking in published Qwen3-VL-Reranker numbers.

Future Directions

• Multimodal decision tuning. Tune a VLM on decisions that include visual options — choice questions over ranked images and pages, and preference questions over two candidates of different relevance. The position is refuted if this does not improve reranking, particularly on natural images (where text tuning does not help) and on image-only queries (where the choice readout currently loses).

• Closing the gap on one backbone. Compare, at several sizes of one VLM, the zero-shot choice readout, the same model after multimodal decision tuning, and a dedicated reranker trained from that model. A gap that shrinks with size strengthens the general route; a persistent gap supports the alternative view that stronger VLMs also make stronger dedicated rerankers.

• Disentangling backbone from image budget. Run Qwen3.8-27B at Gemma 4's image budget, or Gemma 4 at a larger one, to determine whether the family gap comes from the backbone or from the budget (280 visual tokens by default for Gemma 4, up to 1120 supported, versus up to 1800 for Qwen3.8-27B).

• Characterizing the choice readout itself. Measure how the choice probabilities change with candidate order, how well they are calibrated, and whether corrections such as averaging over cyclic option rotations help — order effects that dominate the ranking would weaken the choice format as the interface for a general reranker.

Target Audience

Researchers and practitioners in information retrieval and multimodal search who are building or evaluating two-stage retrieval pipelines, especially those working on retrieval-augmented systems that already serve a vision-language model and are deciding whether a separate reranker is worth training and maintaining. It is also useful for readers interested in decision-style readouts from general models, since the paper connects reranking to that literature and is candid about what its comparisons do and do not establish. Readers wanting a drop-in recipe will need the appendices, which give the exact prompts, token ids, serving configuration and per-task numbers.

Authors’ abstract

Multimodal rerankers are usually dedicated models: a vision-language model (VLM) is trained on retrieval data to score the candidates of a query. Scoring options is not specific to reranking. System One decision models such as Jev map a state and a typed question to one probability per option without generating text, and general models can make such decisions with little or no training. Reranking is one such decision, with the query and its candidates as the state and the candidates as the options. We therefore argue that a general VLM can serve as a multimodal reranker through System One decisions, and that decision tuning can improve its reranking. We ask a VLM one choice question over the top 16 candidates of a first-stage retriever and rank the candidates by the probabilities of their labels, read from one forward pass. On MMEB-V2 image retrieval, this choice scores 5.4 to 7.3 hit@1 above a yes/no question per candidate, and the yes/no question falls below the first stage. With the choice, zero-shot Qwen3.8-27B beats the supervised Qwen3-VL-Reranker-2B on all three benchmarks and comes within 1.6 points of Reranker-8B on each. It matches Reranker-8B on MMEB-V2 visual documents (84.79 vs. 84.78 nDCG@5), at 1.1 to 1.5 times its latency. JEV-27B-VL, the same backbone tuned on text decisions only, improves page reranking by 0.6 to 1.1 points, with gains on most tasks (sign test, p <= 0.017), and leaves natural images unchanged. The zero-shot model is larger than the supervised rerankers and of a newer generation, so these comparisons are confounded; we discuss this and other alternative views, and the experiments that would test the position.