Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering

AI summary

Overview

Research area: Computer Vision, specifically Knowledge-Based Visual Question Answering (KB-VQA) and multimodal Retrieval-Augmented Generation (RAG).

Technical level: Advanced. The paper assumes familiarity with vision-language models (VLMs), retrieval-augmented generation, embedding-based retrieval (CLIP/EVA-CLIP, Faiss), and reinforcement learning (GRPO, LoRA fine-tuning).

Scope: The paper proposes and evaluates Wiki-PRF, a three-stage multimodal RAG framework (Processing, Retrieval, Filtering) that uses reinforcement learning to teach a vision-language model to invoke visual tools and filter retrieved knowledge for knowledge-based VQA.

What This Paper Is About

Knowledge-based VQA asks a model to answer questions about an image using facts that are not visible in the image itself (e.g., "What is that statue made of?" requires outside knowledge). Standard RAG pipelines retrieve Wikipedia-style knowledge, but they struggle with two problems: they retrieve using only coarse whole-image features (so small but question-critical objects get drowned out by prominent background objects), and they cannot precisely filter the large volume of noisy retrieved text. This paper builds a full pipeline that fixes both problems by letting the model call visual tools before retrieval and filter the retrieved text afterward, with both behaviors learned via reinforcement learning.

Key Contributions

1. A three-stage KB-VQA framework (Wiki-PRF) consisting of a Processing stage (tool-based query enrichment), a Retrieval stage (multimodal retrieval using both visual and text features), and a Filtering stage (question-specific condensing of retrieved content into task-oriented knowledge).

2. VLM-PRF, a vision-language model trained via reinforcement learning to plan tool calls and filter retrieved knowledge. The authors state this is the first application of RL to multimodal retrieval-augmented generation.

3. A reward design combining answer accuracy and format consistency using GRPO, which requires only a small amount of training data (roughly 4K samples) rather than the near-million-sample training sets used by competing methods.

4. State-of-the-art results on three benchmarks: 36.0 on E-VQA, 42.8 on InfoSeek, and 77.8 on OK-VQA.

Main Findings

• Strong VQA gains over prior RAG methods. Wiki-PRF-7B reaches 42.8 on InfoSeek and 36.0 on E-VQA, beating EchoSight, Wiki-LLaVA, ReflectiVA, and MMKB-RAG. With InternVL3-8B as the base model, it reaches 39.2 on E-VQA.

• Tool-based processing improves retrieval recall. Direct image retrieval alone achieves 45.56% Top-1 recall on InfoSeek. Adding tool-based multi-article retrieval raises this to 53.44%, and RL training pushes it to 54.89%.

• Reinforcement learning beats supervised fine-tuning. On a 2K-sample InfoSeek subset, RL reaches 46.3 overall accuracy versus 41.8 for SFT and 40.2 for the untuned base model. The authors attribute this to RL learning general filtering principles rather than imitating surface patterns.

• Filtering works well even under perfect retrieval. In an oracle setting where ground-truth articles are provided, Wiki-PRF-7B reaches 65.8% accuracy, far above ReflectiVA (57.6%) and Wiki-LLaVA (51.5%).

• Robustness to knowledge base scale. As the KB grows from 10K to 100K entries, baseline performance collapses (e.g., 56.3 to 23.7 for the 7B base), while Wiki-PRF degrades much more slowly (60.3 to 42.8), indicating better noise tolerance.

• Tool diversity increases after RL. The number of distinct tool-call combinations grows from 34 to 53 (3B) and from 34 to 40 (7B). Captioning is the most frequently used tool, followed by grounding and flipping.

• Data efficiency. Only 4K training samples are used by default, versus the 934K (InfoSeek) and 1M (E-VQA) samples needed by competing approaches.

• Inference cost. Roughly 6.23 seconds per sample for the 3B variant and 8.77 seconds for the 7B variant, with the Filtering stage taking the largest share of time.

Methodology in Plain English

The pipeline runs in three stages on top of a base vision-language model (Qwen2.5-VL in 3B and 7B sizes).

Stage 1: Processing. Given an image and a question, a fine-tuned model called VLM-PRF reasons about which tools would help, then selects them inside structured tags. Three tools are available: a captioning tool (produces a question-focused caption of the image), a grounding tool (crops out the specific object the question is about), and a flipping tool (mirrors the image left-right to reduce orientation bias). A separate frozen base model actually executes these tools. The outputs become refined retrieval queries.

Stage 2: Retrieval. The original image and the tool-generated queries are embedded with EVA-CLIP. Using cosine similarity and Faiss for fast nearest-neighbor search, the system pulls the top matching Wikipedia articles. Articles are split into sections, and the most similar sections are selected. For grounding queries, the original question text is used directly to compute section similarity. The final retrieval context combines one article from the raw image plus top sections from five tool-retrieved articles.

Stage 3: Filtering. VLM-PRF reads the retrieved articles together with the question and produces a compact, task-oriented summary of only the relevant facts, discarding the rest. A separate base VLM then uses this summary to generate the final answer.

Training. VLM-PRF is trained with GRPO (a policy-gradient RL algorithm) with the KL constraint removed, using LoRA (rank 64) so only a small number of parameters are updated. The reward function has three weighted parts: exact-match accuracy of the final answer (weight 1.0), format compliance for tool-call tags (weight 0.3), and format compliance for filtering tags (weight 0.7). Training takes about 15 hours on 8 A800 GPUs.

Why This Matters

The paper shows that a compact, reinforcement-learned pipeline can match or exceed much larger retrieval-augmented systems while using a fraction of the training data. It also provides a concrete demonstration that RL, already popular for text-only reasoning models, transfers effectively to multimodal RAG, where the model must learn when to call tools and what to keep from noisy retrieved text.

Real-world applications:

• Search engines and digital assistants answering fact-based questions about user-uploaded images (landmarks, artwork, products).
• Accessibility tools that describe objects in a photo with accurate factual context for visually impaired users.
• E-commerce and product catalogs, where a shopper asks about a specific item in a cluttered scene and needs factual details pulled from a knowledge base.
• Cultural heritage and museum apps that identify monuments or artifacts in a photo and answer questions about their materials, history, or origin.

Industry relevance: Alibaba Cloud co-authored this work, and the techniques map directly onto production multimodal search and Q&A services. The data efficiency and slow degradation with large knowledge bases are particularly relevant for companies that maintain very large internal knowledge stores.

Future Directions

• Expand the tool set. The current system uses only three tools (captioning, grounding, flipping). Adding OCR, depth, segmentation, or domain-specific tools could improve retrieval on more varied image types.

• Reduce inference latency. The Processing/Retrieval and Filtering stages dominate runtime (roughly 2.2–4.6 seconds each). Distillation, caching, or parallel tool execution could make the pipeline practical for real-time use.

• Improve scaling under large knowledge bases. Even Wiki-PRF degrades as the KB grows from 10K to 100K entries. Better reranking or hierarchical retrieval could further flatten this curve.

• Generalize reward signals beyond exact match. The current answer reward relies on exact-match / regex matching, which limits applicability to open-ended or multi-sentence answers. Learned reward models could broaden the method's reach.

Target Audience

Researchers and graduate students in computer vision, multimodal learning, and information retrieval who work on RAG or VQA; machine learning engineers building production multimodal search or question-answering systems; and practitioners interested in applying reinforcement learning to tool-augmented vision-language models.

Authors’ abstract

Knowledge-based visual question answering (KB-VQA) requires visual language models (VLMs) to integrate visual understanding with external knowledge retrieval. Although retrieval-augmented generation (RAG) achieves significant advances in this task by combining knowledge-base querying, it still struggles with the quality of multimodal queries and the relevance of retrieved results. To overcome these challenges, we propose a novel three-stage method, termed Wiki-PRF, including Processing, Retrieval and Filtering stages. The processing stage dynamically invokes visual tools to extract precise multimodal information for retrieval. The retrieval stage integrates visual and text features to achieve multimodal knowledge retrieval. The filtering stage performs relevance filtering and concentration on retrieval results. To this end, we introduce a visual language model trained with answer accuracy and format consistency as reward signals via a reinforcement learning manner. This enhances the model's reasoning, tool invocation for accurate queries, and filtering of irrelevant content. Experiments on benchmark datasets (E-VQA and InfoSeek) show significant improvements~(36.0 and 42.8) in answer quality, achieving state-of-the-art performance. Code is available at https://github.com/cqu-student/Wiki-PRF