RefCaptioner: Multi-Reference Image-Grounded Video Captioning

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically video captioning combined with multi-reference image grounding.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, GRPO-style reinforcement learning, LoRA adaptation, and multimodal LLM architectures.

Scope: The paper introduces a new task (multi-reference image-grounded video captioning), a two-stage post-training method (RefCaptioner), and a companion benchmark (MRVBench) for evaluating captions that bind local video phrases to specific reference images.

What This Paper Is About

Existing video captioning models describe what happens in a video but cannot explicitly tie specific visual elements in that description to a set of supplied reference images. This paper defines a new task where a model receives a video plus several candidate reference images and must generate a factual caption in which each image tag (<Image_i>) is placed immediately after the phrase that the image actually depicts, while ignoring images that do not appear in the video. The goal is to make reference–video alignment explicit, which the authors argue is fundamental to multi-reference video generation, editing, and understanding.

Key Contributions

1. A new task formulation: Multi-reference image-grounded video captioning, requiring phrase-level reference grounding with three stated requirements — accuracy (tags placed after matching phrases), identification (unsupported references left unused), and consistency (multiple images of the same subject grouped after the same phrase).
2. RefCaptioner: A two-stage post-training framework built on Qwen3-VL-8B-Instruct that combines mixed-data supervised fine-tuning with Hierarchical Coverage-Discounted GRPO (HCD-GRPO), whose reward design targets reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general captioning ability.
3. A training corpus and MRVBench: A corpus of 20,000 videos and 171,354 reference images for training, plus MRVBench — 462 real-world and AI-generated videos (185 AIGC and 277 real-world), 3,831 reference images, and 2,172 question–answer pairs for factuality evaluation, with 5,846 key points across the 462 samples.
4. Empirical and human validation: Results showing RefCaptioner as the best overall open-source model on MRVBench while remaining competitive on VDC and VCapsBench, plus human evaluation via Good/Same/Bad (GSB) pairwise comparison and caption-conditioned video reconstruction.

Main Findings

• Best open-source result on MRVBench: RefCaptioner reaches an MRVScore of 0.888, compared with 0.897 for Gemini-3.1-Pro and 0.870 for GPT-5.4, and above all listed open-source baselines (for example, Qwen3-VL-32B-Instruct at 0.829 and Qwen3-VL-8B-Instruct at 0.763).
• Strong grounding metrics: RefCaptioner records Ref-Tag-P 0.994, Ref-Tag-R 0.943, Ref-Bind 0.967, Eff-Bind 0.912, Dist-Rej 0.985, FalseRef-Any 0.100, Subj-R 0.817, and Subj-F1 0.869. The paper reports it surpasses both Gemini-3.1-Pro and GPT-5.4 on Ref-Tag-R, Subj-R, and Subj-F1, and exceeds at least one of them on Ref-Tag-P, Eff-Bind, and Dist-Rej.
• Content metrics: RefCaptioner obtains KP-Cov 0.882, VQA 0.686, and VQA-Cov 0.837. The paper notes that the larger Qwen3-VL-32B-Instruct performs slightly better on VQA (0.701).
• Beats post-hoc refinement: Against a two-stage CaptionRefine pipeline, RefCaptioner leads on Ref-Tag-R (0.943), Ref-Bind (0.967), Subj-R (0.817), and Subj-F1 (0.869). Qwen3-VL-32B with CaptionRefine achieves the highest KP-Cov in that comparison at 0.911, suggesting a second pass preserves coverage but does not reliably recover phrase-level correspondences.
• General captioning is preserved or improved: On VDC, RefCaptioner scores 59.54 (camera), 50.87 (short), 73.23 (background), 73.50 (object), and 70.11 (detail) — the highest across all five dimensions, with gains of 5.66–6.08 points over its Qwen3-VL-8B base model in background, main-object, and detailed description. On VCapsBench it reaches AR 66.13, IR 13.42, and CR 76.39, improving AR by 1.30 and CR by 1.70 percentage points over the base model.
• Robustness scales with reference count: Grouping samples by reference count (2–4, 5–8, 9–12, 13+), RefCaptioner has the highest robustness score in every group, reaching 0.769 on the 13+ group versus 0.703 for GPT-5.4 and 0.398 for Qwen3.6-35B-A3B. Videos can contain up to 22 reference images.
• Ablation shows complementary rewards: The full model achieves the best VQA (0.686), Dist-Rej (0.985), and Subj-R (0.817). Removing the factual-caption reward drops VQA to 0.642; removing DAES drops Dist-Rej to 0.914; removing CRSC drops Subj-R to 0.778. SFT alone raises Subj-R from the base 0.471 to 0.702 but lowers VQA from 0.670 to 0.642.
• Human evaluation favors the model: In pairwise GSB comparison for caption-conditioned video reconstruction, RefCaptioner receives more Good than Bad judgments against every baseline, with the largest advantages over open-source models. The paper states its captions are preferred by annotators and enable more source-faithful reconstruction with both open-source and proprietary video generators.

Methodology in Plain English

The team started from an existing multimodal model, Qwen3-VL-8B-Instruct, and trained it in two stages.

Stage 1 — Mixed-data supervised fine-tuning. They fine-tuned on a 50/50 mix of two data types: multi-reference examples containing gold captions with image tags (manually checked for video factuality, tag placement, and image–phrase correspondence), and ordinary detailed video captions. The first type teaches reference selection and where to place tags; the second keeps the model good at describing video content. LoRA adapters were applied to attention and feed-forward projections with the vision encoder and projector frozen, using a LoRA rank of 64, alpha 128, and dropout 0.05, for three epochs on 8 NVIDIA H800 (80 GB) GPUs with a learning rate of 1×10⁻⁴, AdamW, weight decay 0.1, cosine decay, warm-up ratio 0.03, gradient clipping 1.0, and a maximum multimodal sequence length of 18,000 tokens.

Stage 2 — Hierarchical Coverage-Discounted GRPO. From the SFT checkpoint, the policy samples multiple candidate captions per input, scores them, normalizes rewards within each group, and updates with the standard clipped GRPO objective. The visual encoder and vision–language merger stay frozen; only the language model is updated, and without LoRA. Training used 32 NVIDIA H800 (80 GB) GPUs across four nodes, a reward-ready training set of 12,838 examples with 126 validation examples, six sampled responses for 16 prompts (96 captions) capped at 512 tokens, an actor learning rate of 1×10⁻⁶, and a KL coefficient of 0.005.

The reward design is the distinctive part. Two branches are combined with fixed weights (λ_b = 0.3, λ_d = 0.4, λ_c = 0.3), and both take a "coverage-discounted" form where useful coverage supplies the positive signal and observable errors discount it:

• Factual caption reward: image tags are stripped from the caption, and a keypoint bank (subject, appearance, action, background, camera motion, visual style) is scored by an LLM judge with values 0, 0.5, or 1 for missing/incorrect, partial, or correct coverage. A separate video-QA bank assigns error scores of 0, 0.5, or 1, and the coverage score is discounted by the average error.
• Multi-reference grounding reward: combines Correctly Bound Reference Coverage, Reference Binding Accuracy (an LLM judge checks whether each phrase preceding a tag describes the same visual content as that reference's annotation), Distractor-Aware Evidence Suppression (penalizes using any reference whose content does not appear in the video), and Cross-Reference Semantic Coherence (an entity-level IoU-style comparison between extracted and ground-truth tag sets for entities with multiple references). DAES activates only for samples containing distractors and CRSC only for samples containing an entity with multiple references.

The reward is capped when the output contains malformed or nonexistent image tags. Evaluation uses Gemini-3.1-Pro as the MLLM judge, with final scores computed programmatically from structured outputs.

Data construction for MRVBench followed a pipeline of multi-domain video collection, keyframe extraction, six-dimension-guided cropping, reference pool curation with distractor injection, local image-tag binding, and human verification. The training corpus was collected from authorized sources and will not be publicly released; the MRVBench test set and evaluation annotations will be released subject to applicable data licenses.

Why This Matters

The paper argues that explicitly expressing fine-grained reference–video correspondences has been comparatively neglected in captioning, even as reference-conditioned video generation and editing have advanced. By showing that grounding can be learned jointly with captioning rather than patched in afterward, it positions grounded captions as a practical interface between video description and controllable generation.

Real-world applications:

• Multi-reference video generation: Grounded captions with correct image tags can serve as conditioning input for generators that must reproduce specific subjects, outfits, or scenes from supplied images.
• Video editing: Knowing which phrase corresponds to which reference lets editing systems target changes at the right visual element rather than the whole clip.
• Caption-conditioned video reconstruction: The paper demonstrates a controlled reconstruction setting where captions plus cited images regenerate the source video, with human evaluators favoring RefCaptioner-based reconstructions.
• Multi-subject content creation (e.g., advertising, film pre-production, e-commerce): The subject-consistency metrics (Subj-R, Subj-F1) target exactly the scenario where several images show one character or product from multiple angles and must be grouped under a single description.

Industry relevance: The method starts from an 8B open-weight model, uses LoRA for SFT, and reports being competitive with proprietary systems such as Gemini-3.1-Pro and GPT-5.4 on grounding metrics — suggesting a cost-effective route to reference-aware captioning. The released MRVBench test set gives the community a shared evaluation target for a capability no prior benchmark covered.

Future Directions

• Human-verified benchmarks at larger scale: MRVBench contains 462 test samples; expanding to more videos, languages, and domains would test whether the reported robustness holds more broadly.
• Reducing dependence on LLM judges and proprietary APIs: Evaluation relies on Gemini-3.1-Pro as the MLLM judge plus multiple LLM-judge reward components, which raises questions about judge bias, cost, and reproducibility.
• Extending grounding beyond captions: The paper positions the work as a foundation for multi-reference video generation and editing, but grounding quality for these downstream tasks is only probed through reconstruction studies.
• Understanding the VQA trade-off: SFT alone lowered VQA (0.670 to 0.642) and the full model (0.686) still trails Qwen3-VL-32B-Instruct (0.701), leaving open how to raise factual QA accuracy without sacrificing grounding.

Target Audience

Researchers and engineers working on multimodal LLMs, video understanding, and reinforcement-learning-based post-training — particularly those building reference-conditioned video generation or editing systems. It is also relevant to benchmark designers interested in factuality and grounding evaluation, since MRVBench's metric suite (KP-Cov, VQA, VQA-Cov, Ref-Tag-P/R, Ref-Bind, Eff-Bind, Dist-Rej, FalseRef-Any, Subj-R/F1) is reusable. Readers without background in RL fine-tuning or multimodal architectures will find the method sections dense.

Authors’ abstract

Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captioning, a new task requiring factual video descriptions with phrase-level reference grounding, and propose RefCaptioner, a two-stage post-training framework for this task. RefCaptioner combines mixed-data SFT with Hierarchical Coverage-Discounted GRPO to jointly improve reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general video-captioning ability. To support training, we construct a corpus containing $20,000$ videos and 171,354 reference images. We further introduce MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding on both real-world and AI-generated videos. Experiments show that RefCaptioner achieves the best overall performance among the open-source models while remaining competitive on standard video captioning benchmarks. Human evaluation further confirms that its captions are preferred by annotators and enable more source-faithful video reconstruction with both open-source and proprietary video generators.