QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models

AI summary

Overview

• Research area: Multimodal large language models (MLLMs); prompt engineering; multi-image visual reasoning and visual question answering.
• Technical level: Intermediate. The method itself is conceptually simple (it is a prompting pipeline, not a new architecture), but the paper assumes familiarity with MLLM benchmarks, chain-of-thought prompting, and multi-image evaluation setups.
• One-sentence scope: The paper diagnoses why existing prompting methods break down on multi-image tasks and proposes QG-CoC, a zero-shot prompting framework that decomposes a question into sub-questions, generates a targeted caption for each sub-question, and chains the resulting sub-answers into a final response.

What This Paper Is About

MLLMs handle single-image questions reasonably well, but they degrade on tasks that require comparing, ordering, or synthesizing information across several images at once. Existing prompting methods (scene graphs, bounding boxes, question decomposition) were largely designed for single images, and when adapted to multi-image settings they produce captions that miss spatial context, describe irrelevant objects, or stay vague. The paper's goal is to build a general, training-free prompting strategy that works with any number of images and forces the model to ground each reasoning step in the visual evidence actually needed to answer the question.

Key Contributions

1. A systematic study of captioning strategies for multi-image inputs. The authors isolate and compare caption length (concise vs. detailed), caption scope (one summary for all images vs. one caption per image), and whether the question is included during captioning. They find that detailed, per-image, question-guided captions win consistently across both open-source and closed-source models.
2. Evidence that existing prompting methods fail on multi-image tasks. Adapting DDCoT, CCoT, and CoCoT to multi-image settings reveals three recurring failure modes: missing spatial context, descriptions of objects unrelated to the question, and vague similarity/difference statements.
3. QG-CoC, a zero-shot prompting method that scales to an arbitrary number of images. The pipeline decomposes the question, produces sub-question-conditioned captions, answers each sub-question, then integrates the sub-question/sub-answer pairs as prior context for the final answer. No fine-tuning, no external tools, no separate vision modules.
4. Broad empirical validation and error analysis. Experiments span five MLLMs and five benchmarks (two multi-image, three single-image), plus an ablation isolating each pipeline component and a manual annotation of 120 failure cases.

Main Findings

• Detailed, per-image, question-guided captions are the strongest basic strategy. Across Gemini-Flash, LLaVA-OV, and Mantis on MMIU and MUIR, detailed captions beat concise ones, individual captions beat a single summarized caption, and question-guided captioning beats unguided captioning.
• QG-CoC tops the comparison on multi-image benchmarks. On MuirBench, LLaVA-OneVision-7B improves from 41.2 to 53.3 (+12.1), Mantis-idefics2 from 43.4 to 48.9 (+5.5), Qwen-2.5-VL from 62.1 to 65.3 (+3.2), GPT-4o from 70.8 to 74.9 (+4.1), and Gemini-1.5-Flash from 66.0 to 68.2 (+2.2). Gains on MMIU are smaller but still consistent, with Qwen-2.5-VL gaining +6.6.
• Gains transfer to single-image benchmarks. QG-CoC stays neutral-to-positive on ScienceQA, MMMU, and MMBench, where several baselines (generally more elaborate ones) actually lose accuracy. Gemini-Flash on MMMU, for instance, drops with DDCoT and CCoT but holds steady with QG-CoC.
• Each component contributes incrementally. An ablation on Gemini-1.5-Flash shows MuirBench accuracy climbing from 66.0 (zero-shot) to 66.5 with question decomposition, to 67.2 with question-guided captioning, to 68.2 with the full QG-CoC pipeline.
• Failure is dominated by reasoning, not perception. Of 120 sampled errors, 35.0% are wrong reasoning despite correct captions, 33.3% are wrong question understanding leading to bad decomposition, and 31.7% are inaccurate perception. Tasks requiring holistic multi-image synthesis (Difference, Scene, Action) skew toward reasoning errors; fine-grained tasks (Matching, Attribute, Counting) skew toward perception errors.
• Task-type sensitivity. On low-level semantic relationships, direct prompting can outperform heavier context injection; on high-level tasks such as causality reasoning and emotion recognition, QG-CoC wins by a wide margin. All methods, including QG-CoC, remain weak on 2D/3D spatial relations and temporal ordering.
• Modest overhead. QG-CoC adds roughly 127 extra tokens and 6.1 seconds per sample on MMIU, comparable to DDCoT and cheaper than detailed captioning or CCoT scene graphs, while outperforming all of them.

Methodology in Plain English

The authors start with a controlled diagnostic study rather than jumping to a solution. They systematically vary how images are described — short vs. long, one combined description vs. one per image, with vs. without the question included — and measure the downstream accuracy on multi-image benchmarks. This reveals three guiding principles that shape the final method.

They then take three well-known single-image prompting methods (DDCoT, CCoT, CoCoT) and hand-adapt them to multi-image inputs, documenting where each breaks down qualitatively and quantitatively.

QG-CoC itself is a three-step, two-stage pipeline:

1. Decompose. The model rewrites the original complex question into a short list of simpler sub-questions, each targeting a specific aspect (an action, an outcome, a comparison, a count).
2. Caption. For each sub-question, the model generates a caption describing only the visual evidence relevant to that sub-question. Because the caption is conditioned on a narrow, specific sub-question rather than the whole complex question, the extracted information is focused rather than scattershot.
3. Answer and integrate. The model answers each sub-question from its caption, then feeds the full list of sub-question/sub-answer pairs back as context for generating the final answer.

Evaluation covers closed-source models (GPT-4o, Gemini-1.5-Flash) and open-source ones (LLaVA-OneVision-7B, Mantis-idefics2-8B, Qwen-2.5-VL-7B), on multi-image benchmarks (MuirBench, MMIU) and single-image benchmarks (MMMU, MMBench, ScienceQA). For the weakest open-source models, the authors substitute Gemini-1.5-Flash as an "oracle" captioner in the first stage, since those models cannot reliably follow long multi-step prompts.

Why This Matters

Most deployed multimodal systems today are prompting-based and cannot be fine-tuned per task. This paper shows that a well-designed, training-free prompt structure can recover a large share of the reasoning ability that multi-image tasks demand, and it provides concrete diagnostic evidence about which captioning choices matter. It also establishes a strong baseline that fine-tuning approaches must beat, and its error taxonomy — question understanding vs. perception vs. reasoning — gives the field a clearer vocabulary for characterizing MLLM failure.

Real-world applications:

• E-commerce product comparison: Comparing a query image against catalog photos for matching attributes, style, or defects.
• Medical imaging review: Synthesizing findings across multiple scans or time points where the relevant regions differ per question.
• Document and diagram analysis: Answering questions that span several charts, tables, or schematic panels where each figure contributes a different piece of evidence.
• Video and surveillance understanding: Reasoning over sampled frames for ordering, action recognition, or change detection.
• Accessibility tooling: Generating structured, question-focused descriptions of image sets for screen-reader users.

Industry relevance: The method requires no retraining, no auxiliary vision modules (detectors, scene-graph parsers), and no external tool calls, so it can be dropped into existing API-based or self-hosted MLLM pipelines. Its token overhead is comparable to other two-stage prompting schemes, making the cost/accuracy tradeoff easy to justify. It is especially attractive for open-source deployments where fine-tuning at scale is impractical.

Future Directions

• Closing the reasoning gap. Wrong reasoning remains the single largest error category even when decomposition and captions are correct, so the field needs techniques that improve inference over already-accurate context.
• Spatial and temporal grounding. All evaluated prompting methods, QG-CoC included, perform poorly on 2D/3D spatial relations and temporal ordering. The authors explicitly call for extending the framework to geometric shapes and explicit spatial representations.
• Weaker captioner robustness. QG-CoC depends on strong captioning ability, and the paper substitutes an oracle (Gemini-1.5-Flash) for weaker open-source models. Making the pipeline self-sufficient on smaller models is an open engineering question.
• Adaptive decomposition. Question decomposition is currently generated in one pass; a learned or iterative decomposition that revises itself when captions prove uninformative could reduce the wrong-question-understanding error class.
• Broader model and benchmark coverage. The authors acknowledge that their model set, while representative, is not exhaustive, and that low-quality tasks had to be filtered from MMIU.

Target Audience

Researchers and practitioners working on multimodal LLMs, prompt engineering, or visual question answering who need a training-free method for multi-image reasoning. It is also useful for benchmark designers interested in how prompting choices confound evaluation, and for engineers building production systems that must reason over several images without fine-tuning. Readers new to MLLMs will find the diagnostic study in Section 3 an accessible entry point, though the benchmark tables and ablation analysis assume some prior familiarity with the subfield.

Authors’ abstract

Recently, Multimodal Large Language Models (MLLMs) encounter two key issues in multi-image contexts: (1) a lack of fine-grained perception across disparate images, and (2) a diminished capability to effectively reason over and synthesize information from multiple visual inputs. However, while various prompting methods aim to describe visual content, many existing studies focus primarily on single-image settings or specific, constrained scenarios. This leaves a critical gap in understanding and addressing how MLLMs tackle more general and complex multi-image reasoning tasks. Thus, we first extensively investigate how current prompting methods perceive fine-grained visual details and process visual information when dealing with multiple images. Our findings reveal that existing prompting methods fall short in attending to needed clues and seamlessly integrating perception and reasoning. Inspired by the findings, we propose a new zero-shot prompting method, Question-Guided Chain-of-Captions (QG-CoC), a generalized prompting approach that effectively handles problems with an arbitrary number of images. We evaluate our method on various open-source and closed-source MLLMs for multi-image and single-image benchmarks. Experimental results indicate that QG-CoC demonstrates competitive performance across tasks and exhibits robust improvements in the challenging scenarios where existing prompting methods fail.