Visual Grounding First, Multimodal In-context Learning Follows

AI summary

Overview

• Research area: Multimodal large language models (MLLMs) and multimodal in-context learning (ICL), with a focus on visual grounding, hallucination, and catastrophic forgetting.
• Technical level: Intermediate. Readers should know basic transformer/VLM concepts and be comfortable with reinforcement learning terms such as rewards and policy optimization, but the paper's core argument is intuitive.
• Scope: The paper argues that weak visual grounding, not missing ICL-specific training data, is the main bottleneck in multimodal ICL, and it proposes ProCoRe (a proposition-level contrastive reward trained without any ICL-formatted data) plus PICL, a new personalized ICL captioning benchmark.

What This Paper Is About

Multimodal LLMs are surprisingly bad at in-context learning: when given a few image–text demonstrations, they tend to copy the text and ignore the visual evidence, producing hallucinated outputs. Existing fixes train or steer the model on ICL-formatted datasets, which boosts scores on the exact benchmark used for adaptation but generalizes poorly and can erase skills the model already had. The authors instead ask why multimodal ICL fails, identify poor visual grounding as the root cause, and train the model to ground language in images using contrastive image pairs rather than ICL examples.

Key Contributions

1. Diagnosis of existing methods: The authors show that ICL data-dependent approaches (ICL training-based and task-vector-based methods) generalize poorly across ICL benchmarks and cause forgetting on previously well-solved tasks such as simple VQA, and that retrieval/ordering methods often underperform random demonstration selection.
2. ProCoRe: A reinforcement learning framework that improves visual grounding without any ICL-formatted training data. It generates captions for visually contrastive image pairs, decomposes them into propositions (unique to image 1, unique to image 2, or shared), and optimizes proposition-level contrastive rewards.
3. Proposition-level reward design: Rewards operate at the proposition level rather than the sentence level, combining an image–text contrastive term, an NLI-entailment-based text term, and a separate shared-content reward that prevents the model from collapsing to shortcut captions that describe only the distinguishing object.
4. PICL benchmark: A personalized multimodal ICL captioning benchmark built from PororoSV, FlintstonesSV and StoryStream, in which captions use personalized identifiers (e.g., a character name) that cannot be recovered from the query image alone, so correct answers require actually grounding the demonstrations.

Main Findings

• Headline gains over the off-the-shelf model: Despite never seeing ICL-formatted training examples, ProCoRe improves average multimodal ICL classification accuracy by 5.6% and captioning ROUGE-L by 13.8% over off-the-shelf Qwen-VL-3-8B, outperforming ICL-data-dependent state of the arts across benchmarks.
• Captioning results (Table 1, Qwen-VL-3-8B): ProCoRe reaches 46.03 ROUGE-L on MS COCO at 8-shot versus 25.65 for the off-the-shelf model, and 36.76 on PICL at 4-shot versus 29.29 for off-the-shelf.
• Classification results (Table 2): ProCoRe scores 68.08 on TrueMICL at 8-shot (off-the-shelf: 62.70) and 98.41 on ISEKAI at 12-shot (off-the-shelf: 96.42).
• Poor generalization of ICL data-dependent methods: Methods trained on the TrueMICL ICL dataset or on ISEKAI perform well when trained and tested on the same ICL dataset but degrade significantly when the training and test ICL datasets differ, sometimes dropping below the off-the-shelf model.
• Retrieval and ordering methods are fragile: RICES, VICL and MMICES frequently underperform plain random demonstration selection, consistent with prior reports.
• Generic vs. personalized grounding (Figure 1): With a generic identifier seen in pre-training (e.g., "monkey"), baselines and ProCoRe both attend to relevant regions; when the identifier is replaced with a personalized name (e.g., "Jaden"), baselines fail to attend to the relevant region and fall back to generic terms such as "character" or "monkey", while ProCoRe predicts the personalized concept with strong visual grounding.
• Better grounding and fewer hallucinations: ProCoRe achieves the highest target attention score (TAS) on ICL captioning benchmarks and consistently lower OpenCHAIR scores (fewer object hallucinations) than baselines across shot counts.
• Less forgetting: ICL training-based methods, which perform additional instruction tuning on ICL datasets, degrade on simple VQA tasks (IconQA, TextVQA), whereas ProCoRe improves these tasks while also delivering strong multimodal ICL performance.
• Personalized identifier accuracy (Table 5): On PICL-PororoSV, ProCoRe reaches 62 / 57 / 57 / 59 for 1–4 shots, the highest across all shots; the strongest baseline shown (STV) reaches 59 / 52 / 55 / 56.
• Ablations (Table 3): Supervised fine-tuning on the same contrastive pairs gives only marginal gains (MS COCO 0-shot 19.48, 8-shot 37.62), and a sentence-level contrastive reward reaches 36.02 (0-shot) and 42.58 (8-shot), both below ProCoRe's 39.12 and 46.03, isolating the benefit of proposition-level rewards.
• Hallucination-mitigation baselines fall short: PAI, VOLCANO, LURE, LRV, IBD and SPARC show only marginal gains and often underperform the off-the-shelf model; the authors attribute this to direct amplification of visual token attention, which does not target task-relevant regions.
• Human validation of the identifier metric: With 30 sampled query examples rated by 50 independent annotators, the GPT-4o-mini-based identifier-accuracy metric reached 95.91% ± 2.98 agreement, Cohen's κ of 0.92 ± 0.08, precision 96.97% ± 4.72, recall 95.43% ± 5.00, F1 95.97% ± 3.88, and a raw score of 28.77 ± 1.19 out of 30.

Methodology in Plain English

The authors start from an observation rather than a new architecture: when an MLLM fails at multimodal ICL, it is usually because its generated text is not anchored to the right part of the image. So instead of teaching the model on ICL-formatted data, they teach it to look more carefully.

Training uses pairs of visually similar images with captions, drawn from multimodal contrastive datasets (InpaintCOCO and EQBEN). The model generates a caption for each image. Those captions are then broken into propositions — statements unique to the first image, unique to the second, and statements true of both — using GPT-4o-mini. Rewards are computed on the unique propositions: a proposition should be more similar to its own image than to the other image (measured with CLIP-ViT-B/32 image and text embeddings), and it should be entailed by its own ground-truth caption more than by the other one (measured with an NLI entailment score). A separate shared-content reward keeps the propositions common to both images aligned with both images and both captions, so the model is not pushed into writing short-cut captions that mention only the distinguishing object. The combined reward is optimized with Group Relative Policy Optimization (GRPO). Because the training signal never involves demonstrations or ICL formatting, any ICL improvement must come from the underlying grounding ability.

For evaluation, the authors compare against retrieval/ordering methods, task-vector methods, ICL training-based methods and hallucination-mitigation methods, using accuracy on TrueMICL and ISEKAI for classification and ROUGE-L on MS COCO and their new PICL benchmark for captioning. Grounding is measured with target attention score (TAS) over SAM3 segmentation maps, and hallucination with OpenCHAIR. PICL is constructed by manually selecting frames from three visual storytelling datasets, replacing original character names with personalized identifiers sampled via GPT-4o-mini; each dataset contributes 1,000 training samples (for the ICL data-dependent baselines) and 100 test samples, which the paper states yields 3,300 samples per shot setting for 1–4 shots. Higher-shot settings are designed to require reasoning over increasingly complex relationships among demonstrations rather than merely supplying more context, so performance does not necessarily improve with shot count.

Why This Matters

The paper reframes a crowded research direction: instead of chasing benchmark-specific ICL adaptation, it shows that a general perceptual skill can be trained without ICL data and still transfer to ICL tasks, while preserving (and even improving) ordinary VQA performance. This matters for how the field builds and evaluates multimodal models, and it gives a concrete diagnostic — attention to task-relevant regions — that links ICL failures to hallucination.

Real-world applications:

• Accessibility captioning: describing photos for blind and low-vision users using personalized names ("Jaden is holding the ball") that cannot be inferred from generic priors.
• Personal photo and media organization: recognizing user-assigned names for pets, people, toys or recurring objects across images.
• Robotics and embodied assistants: following instructions that reference user-named objects or locations rather than generic categories.
• Customer-facing visual assistants: retailers and support tools where products carry brand- or user-specific nicknames that no pre-training corpus contains.

Industry relevance: the method requires no ICL-specific dataset curation and no extra detectors or revisers, which keeps deployment simpler than pipelines that bolt on auxiliary modules. The finding that ICL fine-tuning causes forgetting is directly relevant to teams that adapt MLLMs to customer-specific few-shot tasks, and PICL provides a harder evaluation than benchmarks whose answers can be guessed from pre-trained priors.

Future Directions

• Extending beyond contrastive image pairs: the paper reports additional results with other contrastive datasets (Sec. A.10), but whether the proposition-level reward transfers to video, multi-image, or document inputs is not established.
• Cost and dependence on external models: the pipeline relies on GPT-4o-mini for proposition decomposition and on CLIP and an NLI model for scoring; reducing this dependence and characterizing its failure modes is an open engineering question.
• Broader model and scale coverage: the paper reports results on Qwen-VL-3-8B, LLaVA-OneVision-1.5-8B and InternVL-3.5-8B, and on additional model sizes (described as 4B and 14B in one section and 4B and 13B in another); a full picture across architectures and scales remains to be consolidated.
• Benchmark scope: PICL uses ROUGE-L, which the authors note can be high even when the personalized identifier is wrong; a dedicated identifier-focused metric (they use a GPT-4o-mini judge, human-validated at 95.91% agreement) could become the primary evaluation, and the benchmark could be broadened beyond the three storytelling datasets used.

Target Audience

Researchers and practitioners working on multimodal LLMs, in-context learning, reinforcement-learning-based post-training for vision-language models, and hallucination or grounding evaluation. It is also useful for engineers who adapt MLLMs to personalized, few-shot deployment settings and need to know why ICL fine-tuning may hurt general performance, and for benchmark designers interested in evaluations that cannot be solved from pre-trained priors.

Authors’ abstract

LLMs have shown strong in-context learning (ICL) capability, but extending it to Multimodal LLMs (MLLMs) remains challenging. Prior multimodal ICL methods often rely on ICL-specific datasets for additional training or task vector extraction, which can improve performance on the same ICL benchmark used for adaptation. However, they do not generalize well to other benchmarks and often induce forgetting in previously well-solved tasks. For this reason, we identify 'visual grounding' as the key bottleneck in multimodal ICL; MLLMs often fail to attend to task-relevant visual evidence in demonstrations, instead over-relying on textual cues and producing hallucinated outputs. To address this, we propose ProCoRe to enhance visual grounding through contrastive reinforcement learning on multimodal contrastive data for improving multimodal ICL. ProCoRe generates captions for contrastive image pairs, decomposes them into propositions, and optimizes contrastive rewards that encourage alignment with corresponding images while discouraging mismatched ones. Despite never seeing ICL-formatted training examples, ProCoRe improves average multimodal ICL classification accuracy by 5.6% and captioning ROUGE-L by 13.8% over the off-the-shelf Qwen-VL-3-8B model, outperforming ICL-data-dependent state of the arts across benchmarks. Finally, we introduce PICL, a personalized multimodal ICL captioning benchmark that tests whether MLLMs genuinely understand visual demonstrations rather than relying on pre-trained semantic priors.