Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering

AI summary

Overview

Research area: Computer vision and multimodal retrieval, specifically first-stage retrieval for knowledge-based visual question answering (KB-VQA) and retrieval-augmented generation.

Technical level: Advanced. The paper assumes familiarity with CLIP-style dual encoders, multimodal large language models (MLLMs), contrastive learning, hard-negative mining, and distribution distillation objectives.

Scope: The paper proposes KBMR, an MLLM-based embedding retriever for KB-VQA that replaces CLIP-style visual similarity with entity-aligned semantic retrieval, and evaluates it on E-VQA, InfoSeek, and OK-VQA.

What This Paper Is About

KB-VQA systems retrieve Wikipedia-scale evidence for a query image so that a vision-language model can answer long-tail questions that its parametric knowledge cannot handle. Almost all existing pipelines use CLIP-style dual encoders for this first-stage retrieval, and those retrievers favor surface-level visual similarity, which fails when the same entity looks very different across viewpoints or styles, or when different entities look alike. The paper's goal is to break this bottleneck at the retriever level by training an MLLM to embed images into a space where similarity reflects entity identity rather than visual appearance.

Key Contributions

1. KBMR, the first MLLM-based retriever for KB-VQA. Instead of a CLIP-style dual encoder, KBMR prompts an MLLM to summarize an image into a shared semantic space using the prompt "<Image> Summary above image in one word: \n" and takes the final-token hidden state as the retrieval embedding, aligning retrieval similarity with entity-level semantic relevance under large appearance variation.

2. An MLLM-based Semantic Discriminator (SD) producing continuous entity-consistency weights. The SD judges whether a query–candidate pair refers to the same entity and converts the "Yes"/"No" logits into a calibrated weight via w_i = sigmoid((z_y^i - z_n^i)/γ), where γ is a semantic sharpness coefficient. These weights drive hard-negative mining and act as soft supervision beyond rigid binary labels.

3. A continuous semantic distillation (CSD) objective. Training minimizes the symmetric KL divergence between the retriever's query-conditioned softmax posterior over candidates and the discriminator-induced normalized semantic prior, so the model respects graded semantic relationships inside a highly confusable candidate neighborhood.

4. Extensive benchmark validation. KBMR improves retrieval recall over CLIP-family retrievers by up to +14.7 R@1, raises end-to-end VQA accuracy by up to 9.4, and sets reported SOTA of 54.7 on E-VQA, 50.8 on InfoSeek, and 79.3 on OK-VQA.

Main Findings

• Retrieval gains over CLIP and MLLM baselines. On E-VQA, KBMR (LLaVA-OV-7B) reaches 24.7 R@1 / 48.7 R@5 / 52.4 R@10 / 55.0 R@20, versus 13.3 / 31.3 / 41.0 / 48.8 for EVA-CLIP-8B, the strongest CLIP retriever reported. On InfoSeek it reaches 60.3 / 74.7 / 79.6 / 84.4, versus 45.6 / 67.1 / 73.0 / 77.9 for EVA-CLIP-8B. The paper states this is +12.8 R@1 over the best MLLM retriever baseline and +11.4 over EVA-CLIP-8B on E-VQA, and +14.7 R@1 over the strongest CLIP retriever and +19.0 over the best MLLM retriever on InfoSeek.

• Untrained MLLM embeddings underperform strong CLIP retrievers. Qwen2-VL-7B as a zero-shot retriever scores 6.1 R@1 on E-VQA and 17.5 on InfoSeek; LLaVA-OneVision scores 7.4 and 21.8; BLIP-2 scores 4.9 and 10.8. The paper attributes this to general multimodal understanding not directly yielding an effective KB-VQA retrieval space.

• End-to-end VQA improvements transfer across pipelines. Swapping only the retriever raises EchoSight (no reranking) from 19.4 to 27.6 All on E-VQA and from 27.7 to 36.0 All on InfoSeek; EchoSight with reranking from 41.8 to 51.0 (E-VQA All) and from 31.3 to 39.2 (InfoSeek All); ReflectiVA from 29.2 to 35.2 and from 40.1 to 48.6; MMKB-RAG from 35.9 to 43.1 and from 36.4 to 45.7; OMGM from 50.2 to 54.7 and from 43.5 to 50.8. The largest single-method deltas reported in Table 1 are +7.6/+8.2 (EchoSight), +9.3/+9.2 (EchoSight with reranking), +6.7/+6.0 (ReflectiVA), +7.6/+7.2 (MMKB-RAG), and +5.3/+4.5 (OMGM) for E-VQA Single-Hop / All.

• OK-VQA results. KBMR (Qwen2-VL-7B) achieves 78.1 Pseudo Recall@5 and 76.3 VQA score; KBMR (LLaVA-OV-7B) achieves 80.4 and 79.3. This is +12.7 in answer accuracy over the original OMGM, and both KBMR VQA scores were obtained by plugging its first-stage retrieval into the OMGM pipeline. PreFLMR scores 70.9 / 61.9 and Wiki-PRF-7B is not reported for Pseudo Recall@5 with 77.8 VQA score.

• SD-guided soft supervision beats hard labels. With 8 hard negatives fixed, E-VQA R@1 / R@5 is 9.1 / 18.7 for random sampling with hard one-hot weights, 14.7 / 34.9 for CLIP-similarity sampling with hard one-hot weights, 16.4 / 40.1 for SD sampling with hard one-hot weights, and 24.7 / 48.7 for SD sampling with SD weights. InfoSeek follows the same ordering: 20.3 / 36.8, 47.1 / 70.3, 53.1 / 71.6, and 60.3 / 74.7.

• Corrupting SD weights degrades performance. Inverting SD weights for 30% of sampled negatives drops E-VQA R@1 to 12.1 and InfoSeek R@1 to 43.3; inverting for 10% gives 19.9 and 54.4. The 30% setting falls below even the SD-with-hard-one-hot variant (16.4 and 53.1).

• SD weights better reflect entity identity than CLIP similarity. On a pairwise evaluation set of 10,000 positive pairs (two images of the same entity) and 10,000 negative pairs (two images of different entities), SD achieves an AUC of 0.91 versus 0.79 for EVA-CLIP cosine similarity, rescaled from [-1, 1] to [0, 1] via ŝ = (s+1)/2.

• Discriminator choice matters. With 8 negatives, Qwen2.5-VL-7B as the SD gives 24.7 R@1 on E-VQA and 60.3 on InfoSeek; InternVL3-8B gives 20.7 and 54.4; InternVL3-14B gives 24.4 and 59.1. Scaling InternVL3 from 8B to 14B helps but does not surpass Qwen2.5-VL-7B.

• Eight hard negatives is the sweet spot. For KBMR (LLaVA-OV-7B), E-VQA R@1 rises 22.5 (4 negatives), 23.3 (6), 24.7 (8), and 24.2 (10); InfoSeek R@1 rises 57.7, 59.1, 60.3, and 60.2 respectively. Going from 8 to 10 yields no significant gain.

• Sharpness coefficient γ = 1.1 is best. E-VQA R@1 / R@5 is 23.3 / 46.0 at γ=0.9, 24.1 / 47.8 at γ=1.0, 24.7 / 48.7 at γ=1.1, and 24.4 / 48.0 at γ=1.2; InfoSeek shows 57.6 / 71.2, 59.1 / 72.4, 60.3 / 74.7, and 59.4 / 73.2.

• Weight distribution is graded, not binary. Sampling 10,000 training queries and their Top-50 highest-similarity candidates shows many visually competitive candidates receive low semantic weights, while a considerable mass falls in the intermediate range, supporting the use of continuous soft supervision.

• Qualitative example. For a query image of Corvus albicollis, EVA-CLIP returns several visually similar but semantically mismatched bird species and ranks the correct entity only at position 5, whereas KBMR places the ground-truth document at rank 1.

Methodology in Plain English

The team starts from the observation that in KB-VQA, the retriever should find images of the same entity, not images that merely look alike. They therefore rebuild the retriever itself rather than adding more refinement stages on top of CLIP.

First, they turn an MLLM into a retriever. The model is shown an image together with a short instruction asking it to summarize the image in one word, and the hidden state of the final token becomes the image embedding. Because the MLLM reasons autoregressively over visual content, this embedding is intended to encode concept identity rather than pixel patterns.

Second, they build the training data. Using EVA-CLIP-8B, they retrieve the top-50 most similar non-positive candidates for each query as a pool of potential hard negatives. A separate MLLM acting as a Semantic Discriminator then judges each query–candidate pair with a yes/no question about whether both refer to the same entity. Instead of a hard yes/no, the logits of the "Yes" and "No" tokens are converted into a continuous consistency weight through a sigmoid with a sharpness coefficient γ, so that ambiguous cases land in the middle rather than at 0 or 1. Candidates whose weight is above a threshold derived from the query–positive weight (w_q,c_t − β) are removed because they are too semantically close to be useful negatives. What remains is split into four difficulty strata based on weight, and two samples are drawn from each stratum, with duplication to guarantee at least eight and discarding of samples that yield no qualifying candidate.

Third, they train with continuous distillation. The retriever turns query and candidates into embeddings, and a temperature-scaled softmax over cosine similarities gives the model's current belief distribution over the candidate set. The discriminator weights are turned into a normalized soft prior over the same set. Training minimizes the symmetric KL divergence between these two distributions, so the retriever is pulled toward the discriminator's finer-grained view of which candidates are truly the same entity. The forward KL term pushes mass toward semantically correct candidates; the reverse term keeps the posterior from collapsing into something overly sharp or biased.

Implementation: EVA-CLIP-8B supplies the potential hard negatives, Qwen2.5-VL-7B produces the entity consistency weights, and KBMR itself is trained on Qwen2-VL-7B and LLaVA-OneVision-7B separately. Training uses LoRA with rank 16 and DeepSpeed ZeRO stage-2 on 8 NVIDIA A100 (80GB) GPUs, images resized to 336×336, accumulated batch size 1024, learning rates 1e-4 for Qwen2-VL and 2e-5 for LLaVA-OneVision, 5,000 steps on 600k samples drawn from the E-VQA, InfoSeek, and OK-VQA training sets, with k = 8 hard negatives.

Evaluation uses E-VQA (natural species and landmarks, tested on the test set with BEM score), InfoSeek (information-seeking questions over Wikipedia entities from OVEN, entire validation split, VQA accuracy, knowledge base of 100K Wikipedia entries), and OK-VQA (knowledge beyond MS COCO images). Retrieval is measured by recall of the correct article in the top-k.

Why This Matters

Impact on research. The paper argues that KB-VQA retrieval has been treated as a fixed CLIP-shaped component while most innovation went into reranking, filtering, and query refinement. KBMR shows that changing the retriever itself, using an MLLM's autoregressive semantic representation instead of a dual encoder, produces gains that persist even in pipelines that already have reranking or reflection modules (EchoSight with reranking gains +9.3 on E-VQA Single-Hop). It also introduces a reusable recipe for turning an MLLM into a soft supervision source for retrieval training: continuous entity-consistency weights, stratified hard-negative mining, and symmetric KL distillation against a discriminator-induced prior. The reported AUC gap (0.91 SD versus 0.79 CLIP) is direct evidence that visual similarity is a weaker signal for entity identity than an MLLM's judgment.

Real-world applications.

• Encyclopedic and educational assistants that answer user questions about plants, animals, or landmarks photographed in the wild, where lighting, angle, and season make visual matching unreliable.
• Cultural heritage and landmark recognition systems that must distinguish buildings or monuments that look similar but carry entirely different historical knowledge.
• Product and inventory search over large catalogs where the same item appears in many poses and backgrounds while near-duplicate items differ in identity.
• Any retrieval-augmented assistant that grounds answers in an external corpus and needs the first-stage candidate pool to be semantically rather than visually faithful, reducing hallucinated or misattributed evidence.

Industry relevance. The pipeline is built on open models (Qwen2-VL-7B, LLaVA-OneVision-7B, Qwen2.5-VL-7B, EVA-CLIP-8B), trains with LoRA rank 16 and DeepSpeed ZeRO stage-2, and is reported to run on 8 A100 (80GB) GPUs, so the recipe is reproducible at moderate scale. Training is only 5,000 steps on 600k samples. The authors released code at https://github.com/realHarryX/KBMR. The work is published at ACM Multimedia (MM '26), and was supported by NSFC fund 62576190 and Shenzhen Science and Technology Project Grant KJZD20240903103210014.

Future Directions

• Scaling and architecture choices for the discriminator. The paper only compares Qwen2.5-VL-7B, InternVL3-8B, and InternVL3-14B as the Semantic Discriminator. Whether larger or differently trained discriminators, or ensembles, keep improving supervision quality is not established.

• Robustness of the sharpness and margin hyperparameters. γ = 1.1 was best on the studied benchmarks and β controls the negative-filtering margin, but the paper does not report how sensitive results are across other datasets, domains, or languages.

• Reducing dependence on a frozen CLIP retriever for candidate pool construction. The potential hard negatives are still drawn from EVA-CLIP-8B top-50 results, which means the training pipeline inherits CLIP's recall ceiling for which negatives are even considered. An end-to-end or iterative scheme that removes this dependence is an open question.

• Extending beyond Wikipedia-scale image–entity retrieval. The evaluation covers E-VQA, InfoSeek with a 100K-entry knowledge base, and OK-VQA. Applicability to larger or non-encyclopedic corpora, multi-entity queries, and non-visual evidence such as text passages is not tested.

Target Audience

Researchers and engineers working on retrieval-augmented generation, multimodal retrieval, and knowledge-based visual question answering, particularly those who build or replace first-stage retrievers in production or benchmark pipelines. It is also relevant to practitioners applying MLLM embeddings to entity-centric search, and to readers interested in soft supervision and distillation techniques as alternatives to contrastive learning with one-hot labels. Beginners will find the motivation and qualitative example accessible, but the training objective and distillation formulation require prior familiarity with retrieval and multimodal representation learning.

Authors’ abstract

Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.