OAD-Promoter: Enhancing Zero-shot VQA using Large Language Models with Object Attribute Description

AI summary

Overview

Research area: Computer Vision / multi-modal reasoning — specifically zero-shot and few-shot knowledge-based Visual Question Answering (KBVQA) built on top of frozen Large Language Models.

Technical level: Intermediate. The paper assumes familiarity with VQA pipelines, LLM prompting, captioning models (BLIP2, VinVL), and standard benchmarks, but the core idea is explained in accessible terms.

Scope: The paper proposes OAD-Promoter, a three-module zero-shot VQA system that combines multi-granularity image captions with retrieved memory examples to reduce language bias and improve out-of-distribution (OOD) performance, evaluated on VQAv2, OKVQA, A-OKVQA, VQA-CP, and GQA-OOD.

What This Paper Is About

VQA models — including large language models used for knowledge-intensive questions — tend to exploit language bias, meaning they learn superficial correlations from training data (for example, the dominant answer to "What color ... bananas?" being "yellow") instead of actually looking at the image. This makes predictions unreliable and hurts generalization when the model meets questions from unfamiliar domains. The goal of OAD-Promoter is to supply the LLM with richer, multi-granularity visual descriptions and a memory of relevant stored examples so that both the bias problem and the domain-shift problem are reduced, all without using any external knowledge source or retrieved data.

Key Contributions

1. Multi-granularity captions for LLM-based VQA. The Object-concentrated Example Generation (OEG) module generates both a global caption and object-focused regional captions, using enhanced visual information to mitigate language bias inherited by LLMs.
2. A memory-augmented assistance module. The Memory Knowledge Assistance (MKA) module pre-assesses language bias in a new input and retrieves relevant stored object-attribute examples to support prediction, improving reliability in distribution-changing scenarios.
3. The OAD Prompt. A prompt design that integrates the global caption, object-concentrated examples, selected memory examples, and the input question, giving the LLM both comprehensive visual detail and auxiliary examples during inference.
4. Broad empirical validation. Experiments on OKVQA, A-OKVQA, VQAv2, VQA-CP, and GQA-OOD demonstrate effectiveness and generalizability, with the paper reporting new state-of-the-art results among zero-shot LLM-based approaches.

Main Findings

• Zero-shot VQAv2 state-of-the-art: OAD-Promoter with GPT-3 reaches 60.64 (val) and 61.98 (test) on VQAv2 in the zero-shot setting; with OPT it reaches 60.62 (val) and 61.93 (test). The paper states this establishes a new state-of-the-art under a zero-shot scenario on VQAv2.
• Knowledge-based benchmarks: In the zero-shot setting, OAD-Promoter with GPT-3 scores 43.09 (A-OKVQA val), 41.71 (A-OKVQA test), and 45.61 (OKVQA test). With OPT it scores 43.03, 40.68, and 45.58 respectively.
• Competitive in few-shot: With GPT-3 in the few-shot setting, OAD-Promoter scores 57.96 / 58.42 (VQAv2 val / test), 58.50 / 56.99 (A-OKVQA val / test), and 60.04 (OKVQA test). The paper describes these as competitive rather than best-in-table (Prophet with GPT-3, for example, reports 61.08 on OKVQA test).
• Better domain-shift capacity with stronger frozen LLMs: On VQA-CP and GQA-OOD, OAD-Promoter's results are lower than GRACE with LLaVA-1.5 and LLaMA2, but with GPT-4 it achieves the best GQA-OOD accuracy (50.21 versus GRACE's 50.19).
• Works across many LLMs: Zero-shot OKVQA results using different frozen models are 33.77 (BLOOM, 7.1B), 36.18 (OPT, 6.7B), 40.46 (OPT, 30B), 45.58 (OPT, 175B), 33.41 (GPT-Neo, 2.7B), 38.89 (GPT-J, 6B), and 45.61 (GPT-3, 175B), supporting the claim of broad LLM compatibility.
• Debiasing modules can hurt LLM-based pipelines: Integrating the LMH and CSS debiasing strategies into PICa, Prophet, and GRACE mostly lowers OKVQA accuracy (for example, Prophet 61.11 to 59.96 with LMH, then to 57.28 with LMH + CSS). The paper reads this as evidence that LLM-based methods exploit language bias on knowledge-intensive questions.
• Both modules matter: Ablation shows OKVQA scores rise from 47.33 (neither module) to 54.68 (OEG only), 48.95 (MKA only), and 60.04 (both) in the few-shot setting; in the zero-shot setting the same pattern is 42.50, 44.26, 43.64, and 45.61.
• More memory examples help: Adding K examples to MKA gives 43.64 (K=0), 43.65 (K=60), 43.92 (K=200), and 44.15 (K=400) on OKVQA zero-shot.
• Prompt layout matters: The complete-triple design CQA-CQA-CQA scores 45.61 versus 44.82 for the CCC-QAQAQA design on OKVQA zero-shot.
• Qualitative robustness: Across four cases spanning three different domains, OAD-Promoter gets 100% correctness while the compared zero-shot method Img2LLM attains 75% (failing on one case). When the case order is reversed, OAD-Promoter still maintains 100% correctness and Img2LLM also answers all four correctly, which the authors note indicates that input order influences LLM inference across domains.

Methodology in Plain English

The system has three parts that operate in sequence.

First, the OEG module looks at the input image and produces two kinds of description: a single global caption generated by a pre-trained BLIP2 model, and several object-focused regional captions produced by a VinVL detector. It then extracts potential answers from those object-focused captions (noun phrases, verb phrases, adjective phrases, numbers, and yes/no words) using a caption evaluation tool, and feeds them into a prompt that drives a pre-trained T5-large model — fine-tuned on SQuAD2.0, MultiRC, BookQA, CommonsenseQA, and Social IQa — to generate matching questions. Each resulting triple of caption, question, and answer becomes a reusable "example."

Second, the MKA module decides how to help the LLM on a new question. It runs two models: an ordinary VQA model (UpDn) and a QA model from LMH that sees no image. If the two agree, the authors treat that agreement as a sign that language bias is at work and switch to a "Negative" selection mode; if they disagree, they use a "Positive" mode. The fused visual-language features of the new input and each stored example are compared with cosine similarity, and the system selects either the top-N most similar examples (Positive) or the bottom-N least similar ones (Negative).

Third, the OAD Prompt assembles everything into one input for the frozen LLM: an instruction, the global caption, the object-concentrated examples from the current image, the selected memory examples, and the question. At the very start the memory is empty, so the prompt begins as "I / C_G / E_O / Q_O" and grows to include E_S once examples accumulate. Because the stored example pool grows as inference proceeds, the paper argues the LLM's domain-shift capacity improves over time. The entire procedure uses no external knowledge source and requires no retrieved data, making it a pure zero-shot method.

Why This Matters

Impact on research: The paper reframes language bias as a problem that persists in LLM-based KBVQA, not just in conventional VQA models, and shows that bolting standard debiasing modules onto LLM pipelines can actually reduce accuracy. It also argues that existing methods overlook combining global and regional visual information, and that a subsidiary memory module for domain-shift prediction had not been investigated before.

Real-world applications (as suggested by the general task, since the paper does not list specific deployed use cases):

• Assistive technology that answers spoken or typed questions about a user's surroundings for people with visual impairments.
• Visual customer support, where a user photographs a product and asks a question that requires common-sense or brand knowledge.
• Educational and museum guide tools that answer knowledge-intensive questions about what a camera sees.
• Inspection or field-service workflows where a technician asks questions about equipment shown in a photo.

Industry relevance: The approach works with frozen off-the-shelf LLMs including GPT-3, OPT, BLOOM, GPT-Neo, GPT-J, LLaMA2, LLaVA-1.5, and GPT-4, meaning it can be layered on existing model deployments without retraining them — an attractive property for teams that want to improve VQA quality but cannot afford large-scale fine-tuning.

Future Directions

• Scaling and managing the memory pool. The paper shows accuracy improves as K grows from 0 to 400 examples, but it does not report behaviour on much larger memory collections, nor retrieval cost or latency at scale.
• Closing the gap with large multi-modal models. The paper reports that its few-shot results on VQA-CP and GQA-OOD are inferior to GRACE with LLaVA-1.5 and LLaMA2, leaving open how to combine the memory and bias-handling ideas with newer multi-modal backbones.
• Understanding order sensitivity. The reversed-order qualitative experiment hints that the sequence in which samples arrive affects LLM inference across domains; the paper does not quantify this effect.
• Extending the bias-detection heuristic. The Positive/Negative mode is decided by whether an ordinary VQA model and an image-blind QA model agree; whether more nuanced bias estimators would help is not reported.

Target Audience

Researchers and graduate students working on VQA, knowledge-based visual reasoning, and LLM prompting; practitioners who want to improve multi-modal question answering on top of frozen commercial or open LLMs; and readers interested in debiasing and out-of-distribution robustness in vision-language systems. Readers should be comfortable with standard VQA benchmarks and prompting-based pipelines, though the paper explains its modules in accessible terms.

Note: the paper does not report inference latency, monetary cost, or memory storage requirements, and the truncated content does not include per-dataset training hyperparameters beyond those described in the implementation details.

Authors’ abstract

Large Language Models (LLMs) have become a crucial tool in Visual Question Answering (VQA) for handling knowledge-intensive questions in few-shot or zero-shot scenarios. However, their reliance on massive training datasets often causes them to inherit language biases during the acquisition of knowledge. This limitation imposes two key constraints on existing methods: (1) LLM predictions become less reliable due to bias exploitation, and (2) despite strong knowledge reasoning capabilities, LLMs still struggle with out-of-distribution (OOD) generalization. To address these issues, we propose Object Attribute Description Promoter (OAD-Promoter), a novel approach for enhancing LLM-based VQA by mitigating language bias and improving domain-shift robustness. OAD-Promoter comprises three components: the Object-concentrated Example Generation (OEG) module, the Memory Knowledge Assistance (MKA) module, and the OAD Prompt. The OEG module generates global captions and object-concentrated samples, jointly enhancing visual information input to the LLM and mitigating bias through complementary global and regional visual cues. The MKA module assists the LLM in handling OOD samples by retrieving relevant knowledge from stored examples to support questions from unseen domains. Finally, the OAD Prompt integrates the outputs of the preceding modules to optimize LLM inference. Experiments demonstrate that OAD-Promoter significantly improves the performance of LLM-based VQA methods in few-shot or zero-shot settings, achieving new state-of-the-art results.