Re$^3$Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning

AI summary

Overview

Research area: Computer Vision / multimodal learning — specifically image captioning with Large Vision-Language Models (LVLMs), reinforcement learning (RL), and retrieval-augmented reasoning.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for language models (GRPO, PPO clipping, KL regularization), contrastive image-text retrieval (OpenCLIP), sentence embeddings (SBERT), and graph-theoretic k-core decomposition.

Scope: The paper proposes and evaluates Re³Cap, a retrieval-guided reasoning strategy that supplies LVLMs with hallucination and omission signals during RL post-training, so that the model can generate caption candidates it would not otherwise explore.

The work is a collaboration between the Taobao & Tmall Group of Alibaba, The Hong Kong University of Science and Technology (Guangzhou), and Shanghai Artificial Intelligence Laboratory. It was published as arXiv:2608.21305v1 [cs.CV] on 21 Aug 2026 under a CC BY 4.0 license. The code is stated to be released when the paper is accepted.

What This Paper Is About

Reinforcement learning can improve image captioning, but the paper argues it mostly selects the best caption from candidates the base model already produces, rather than teaching the model genuinely new reasoning strategies. This leaves RL-trained models lagging behind Supervised Fine-Tuning (SFT), which has access to task-specific annotation. The goal of Re³Cap is to inject a new reasoning signal — multi-modal retrieval — into RL training so that LVLMs generate previously unexplored caption candidates and close, or reverse, that gap without any additional annotations.

Key Contributions

1. A retrieval-based reasoning strategy for diagnosing captions. The authors show that comparing image-conditioned retrieval results with caption-conditioned retrieval results can reveal hallucinations and omissions in a generated caption, with no additional annotation required.

2. Re³Cap, a framework built on two modules. The Caption Refinement Suggester (CRS) identifies semantic elements that consistently recur across visually similar retrieved images and should therefore be kept in the caption. The Caption Quality Assessor (CQA) compares retrieval results to flag content to remove (hallucinations) and content to add (omissions).

3. An off-policy RL formulation that needs no retrieval at inference. Initial captions and guidance are removed from the policy input during policy optimization (a decoupled PPO formulation), so the optimized policy is conditioned only on the image at deployment.

4. Broad empirical validation. Experiments across LLaVA-1.5-7B, Qwen2-VL-7B, Qwen2.5-VL-7B, InternVL3-8B and Qwen3-VL-8B, three reward functions (CLIP, SC, CIM), and two benchmarks (COCO-LN500, DOCCI500) show consistent gains over GRPO, including against SFT-based pipelines.

Main Findings

• Relation reasoning improves most. Re³Cap outperforms GRPO with an average improvement of 8.64% in relation reasoning on the COCO-LN500 benchmark (abstract). Across multiple base LVLMs and reward functions, the Relations QA gain averages 8.64% on COCO-LN500 and 7.57% on DOCCI500.

• Gains are largest for weaker models and weaker rewards. With CLIP as the reward function, Re³Cap gains an average of 4.39% in Objects F1, 2.44% in Attributes F1, and 6.74% in Relations QA across base LVLMs and both benchmarks. With LLaVA1.5-7B as the base LVLM, gains average 5.91% in Objects F1, 2.06% in Attributes F1, and 8.95% in Relations QA across reward functions and benchmarks.

• Re³Cap can beat SFT where GRPO cannot. Using LLaVA1.5-7B as the base LVLM, Re³Cap outperforms SFT across multiple reward functions and both benchmarks, whereas GRPO underperforms SFT. On stronger models such as Qwen2-VL-7B and Qwen2.5-VL-7B, GRPO surpasses SFT only under certain rewards (e.g. CIM), and Re³Cap improves further on top.

• Single-stage RL beats two-stage pipelines. Against state-of-the-art methods on Qwen2-VL-7B, Re³Cap improves Relations QA by 4.08% on COCO-LN500 and 4.86% on DOCCI500, plus 1.47% in Objects F1 and 1.21% in Attributes F1 on COCO-LN500, averaged across multiple reward functions — while using only single-stage RL, compared with the SFT-then-RL two-stage pipeline used by SC-Captioner and SFT+CIM.

• Both components contribute and combine best. On COCO-LN500 with Qwen2-VL-7B and CIM reward: GRPO alone scores 75.80 Objects F1 / 58.22 Attributes F1 / 38.71 Relations QA; adding CRS reaches 76.83 / 58.98 / 40.90 (gains of 1.03%, 0.76%, 2.19%); adding CQA reaches 77.14 / 59.04 / 42.69 (gains of 1.34%, 0.82%, 3.98%); combining both reaches 78.18 / 59.28 / 44.19.

• The retrieval strategy expands the model's capability boundary. In a max@k analysis on COCO-LN500 using BLEU-4, RL with CIM as the reward is strong at k=1 but grows more slowly as k increases and eventually falls below the base model, indicating a narrowed output distribution. The retrieval-guided strategy, without any training, keeps improving with larger sampling budgets and surpasses the base model as k increases.

• Results hold on additional architectures. For InternVL3-8B, Re³Cap improves Relations QA over GRPO by an average of 6.59% on COCO-LN500 and 5.12% on DOCCI500 across reward functions. For Qwen3-VL-8B, the average gains are 3.76% on COCO-LN500 and 2.42% on DOCCI500. GRPO fails to surpass SFT under the CIM and SC rewards, while Re³Cap consistently outperforms SFT under these signals.

• The method is robust to hyperparameters. Varying the retrieval number K from 3 to 11 and the similarity threshold τ from 0.5 to 0.9 on COCO-LN500 with Qwen2-VL-7B produces only minor variation: within 0.53% (Objects F1), 0.74% (Attributes F1) and 1.44% (Relation QA) across K values, and within 1.32%, 1.48% and 1.38% respectively across τ values.

• No human evaluation is reported. Caption quality is measured with F1 over objects, attributes and relations, with relational correctness assessed by VQA-based accuracy using Qwen3.

Methodology in Plain English

The method rests on two intuitions: visually similar images tend to share semantic content, and semantically similar queries tend to produce consistent retrieval results.

Step 1 — Sample initial captions. For each training image, the LVLM generates a group of initial captions (M = 5 in the experiments).

Step 2 — Two retrievals. The image is used as a query to retrieve the top-K most similar image-text pairs from a retrieval corpus (K = 3). Each initial caption is separately used as a query to retrieve its own top-K text results.

Step 3 — Build graphs and find the k-core. For each set of retrieved results, every sentence becomes a node in a graph, and edges are added between sentences whose SBERT similarity exceeds a threshold τ (0.7 in the experiments). The k-core is the densely connected subgraph that survives pruning of long-tail descriptions (k = ⌈K/2⌉ = 2). The image-retrieval k-core captures the semantic content the image genuinely contains; the caption-retrieval k-core captures what the caption asserts; a third bipartite k-core over the union of both captures what they agree on.

Step 4 — Diagnose and revise. Content in the caption's k-core but not in the shared k-core is flagged as hallucination; content in the image's k-core but not in the shared k-core is flagged as omission. These findings become textual guidance ("keep this", "add this", "remove this") inserted into the prompt, and the model writes a refined caption.

Step 5 — Reinforce. Refined captions are scored by a reward function, and the policy is updated. Crucially, the initial caption and the guidance are removed from the policy input during optimization, so at inference time the model captions directly from the image with no retrieval and no k-core computation. Because the behavior policy (which saw the guidance) differs from the optimized policy, the authors use a decoupled PPO formulation with a separate proximal policy as a trust-region center.

Training setup. Training uses 6.5K images from RefinedCaps sampled from the COCO training split. Retrieval draws on a corpus built by augmenting RefinedCaps with DenseFusion-1M, kept disjoint from all evaluation benchmarks to avoid leakage. SBERT with an MPNet-base backbone serves as the text encoder and OpenCLIP ViT-H/14 as the image encoder. Training uses the VERL framework, Adam, two epochs, a constant learning rate of 1×10⁻⁶, rollout prompt batch size 256, mini-batch size 64, clipping ratio ϵ = 0.2, and KL penalty coefficient β = 0.001.

Evaluation. COCO-LN500 comprises 500 image-caption pairs from the Localized-narratives test set in COCO2017. DOCCI500 is a random sample of 500 image-caption pairs from the DOCCI test split, with images that largely lack human-centric content. Baselines are GRPO with three reward functions: CLIP (image-text similarity), SC (keyword-level self-correction), and CIM (similarity between the image retrieved by a caption and the original image).

Why This Matters

Impact on research. The paper challenges the view that RL for captioning is mostly a selection mechanism over pre-existing candidates, and offers a concrete way to expand a model's exploration space without extra annotation. It also reframes cross-modal caption quality assessment as a purely textual comparison problem among retrieval results, which is a reusable idea for other multimodal tasks, and it demonstrates that an RL-only pipeline can overtake SFT on some settings — a result the authors highlight as notable given that RL-based methods have generally lagged behind SFT.

Real-world applications:

• Image search and text-image retrieval: captions that more faithfully describe relations and attributes directly improve the retrieval signal described in the paper's own motivation.
• Text-to-image generation: more accurate and detailed captions function as better generation prompts.
• Visual question answering: the paper positions captioning as a foundational component feeding VQA pipelines.
• Accessibility and dataset annotation: the method targets hallucination and omission reduction in captions, which matters for alt-text and for building image-caption datasets, especially given the Alibaba e-commerce context of the authorship.

Industry relevance. The work is led by researchers at the Taobao & Tmall Group of Alibaba, where large volumes of product imagery require scalable, accurate descriptions. Its key practical property is that all retrieval and graph analysis happens only during training, so deployment requires no extra retrieval infrastructure at inference — an important consideration for production systems.

Future Directions

• Scaling and diversifying the retrieval corpus. The stated limitation is that effectiveness depends on the quality and scale of the retrieval set; when the dataset is too small, many captions fail to retrieve relevant results and the method can degenerate into plain RL.
• Extending beyond captioning. The retrieval-discrepancy signal could plausibly be applied to other multimodal reasoning or generation tasks, though the paper does not test this.
• Broader model and benchmark coverage. The paper evaluates five LVLMs and two benchmarks; generalization to other architectures, domains, and captioning styles remains open.
• Trade-offs in guidance design. How much of the improvement comes from retrieval quality versus the k-core graph construction, and how the number of sampled responses M interacts with the guidance, are questions the presented experiments do not fully isolate. Computational overhead is mentioned as being reported in Appendix B, but those numbers are not available in the content examined here.

Target Audience

Researchers and engineers working on multimodal large language models, reinforcement learning from reward signals, and image captioning — particularly those interested in retrieval-augmented reasoning, hallucination mitigation, and RL post-training that does not depend on supervised annotation. The paper is also relevant to practitioners building captioning or description pipelines at scale, who will care that the retrieval machinery is discarded at inference time. Readers without a background in RL optimization or graph-based retrieval will find the methodological sections demanding.

Authors’ abstract

Reinforcement Learning (RL) has demonstrated significant gains in image captioning, yet it is still limited in encouraging Large Vision-Language Models (LVLMs) to explore novel reasoning strategies. This limitation leads to a performance gap between RL and Supervised Fine-Tuning (SFT). In this paper, we argue that multi-modal retrieval can serve as an effective reasoning signal for caption refinement. Based on this insight, we present the Retrieval-Guided Refinement for Image Captioning (Re$^3$Cap), a retrieval-guided reasoning strategy that enhances image captioning without requiring additional annotations. Instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), this strategy identifies hallucinations and omissions in image captions, leading to more accurate and detailed descriptions. Extensive experiments demonstrate the superiority of our method in image captioning, even compared with Supervised Fine-Tuning. Especially, Re$^3$Cap outperforms GRPO with an average improvement of 8.64% in relation reasoning on the COCO-LN500 benchmark.