VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection

AI summary

Overview

• Research area: Computer Vision / Multimodal Large Language Models (MLLMs), specifically Video Detailed Captioning (VDC) and preference-based alignment.
• Technical level: Advanced. The paper assumes familiarity with MLLMs, prompt-based agents, Direct Preference Optimization (DPO), LoRA, and video-caption evaluation benchmarks.
• Scope: The paper proposes a single-model, self-improving video captioning framework that replaces human annotation and larger teacher models with agentic self-reflection, then internalizes that reflective behavior into the base model through a curriculum DPO training stage.

What This Paper Is About

Producing detailed, faithful descriptions of video content normally requires either expensive human-written captions or distillation from much larger proprietary models such as GPT-4V or Qwen-72B. The authors ask whether one model can act as its own critic, generating a caption, scoring it against written principles, revising its own prompt when the score drops, and learning from those self-generated successes and failures. The goal is a captioner that improves itself without external supervision while keeping the inference cost of the original model.

Key Contributions

1. VDC-Agent: An autonomous framework in which a single MLLM iteratively generates captions, evaluates them against textual principles, refines its own prompt, and triggers a self-reflection step when a refinement makes a caption worse, requiring neither human annotation nor a stronger external teacher.
2. VDC-Agent-19K: A preference dataset of 18,886 caption pairs with quantified quality gaps, extracted automatically from the agent's own scored iteration trajectories on the Cockatiel-4K corpus.
3. Curriculum DPO: A training strategy that uses the score gap between preferred and dispreferred captions as a difficulty signal and orders mini-batches from large gaps (easy) to small gaps (hard), replacing vanilla DPO's uniform treatment of pairs.
4. State-of-the-art results plus efficiency evidence: Reported best average performance on the VDC benchmark and top F1/Recall on DREAM-1K, with ablations showing internalization matches or beats test-time reflection at base-model inference speed.

Main Findings

• VDC benchmark state of the art: Built on Qwen2.5-VL-7B, VDC-Agent reaches an average accuracy of 49.08% and average score of 2.50, improving over its own baseline of 43.95% / 2.23 by +5.13 accuracy and +0.27 score. Reported gains are largest on detail-oriented dimensions: camera +7.91%, background +7.83%, main object +4.23%, detailed +3.96%, and smallest on short +1.73%.
• Cross-model transferability: The same framework improves three distinct backbones. LLaVA-OneVision-7B rises from 37.45% / 1.95 to 43.18% / 2.21 (+5.73 / +0.26); InternVL2.5-8B rises from 39.44% / 2.08 to 44.88% / 2.30 (+5.44 / +0.22); Qwen2.5-VL-7B rises to 49.08% / 2.50.
• DREAM-1K gains: The Qwen2.5-VL-7B variant achieves the highest reported F1 of 37.4% and Recall of 36.3%, with Precision 38.5; the InternVL2.5-8B variant attains the best Precision at 40.5%. F1 improvements span +5.2 to +7.3 across the three backbones.
• Human agreement with automatic scores: In a blind evaluation on 500 randomly sampled pairs from VDC-Agent-19K, three independent human evaluators preferred the higher-scoring caption at rates of 75.8%, 69.0%, and 67.6%. GPT-5.1 preferred it at an average of 89.8% across both input orders.
• Preference is not just length: The length distributions of preferred (y+) and dispreferred (y−) captions overlap considerably, and human raters favored y+ for semantic richness and visual grounding, indicating content quality rather than length drives the preference.
• Internalization beats test-time reflection on efficiency: On the VDC benchmark, Baseline scores 43.95% / 2.23 at 15.5s; Baseline+P reaches 46.47% / 2.35 at 22.3s; Baseline+P+R reaches 47.10% / 2.38 but takes 164.9s; VDC-Agent reaches 49.08% / 2.50 at 15.5s — the same inference time as the baseline.
• Curriculum order matters: Curriculum DPO (49.08% / 2.50) outperforms SFT (47.54% / 2.41), Random DPO (48.03% / 2.42), and Anti-Curriculum DPO, which sorts hard-to-easy and drops to 47.65% / 2.40.
• Directed reflection beats extra sampling: Under comparable compute, Best-of-N with N=12 reaches 45.12% / 2.30 in 186.0s, while Baseline+P+R at T=4 reaches 47.10% / 2.38 in 164.9s, and VDC-Agent reaches 49.08% / 2.50.
• Iteration depth trades off against cost: Accuracy rises from 46.98% / 2.38 at T=2 to 49.08% / 2.50 at T=4, then largely saturates: 49.29% / 2.52 at T=5, with dataset construction time growing from 41.6s per sample / 29.0h total at T=2 to 103.8s / 72.2h at T=6.
• Robustness to principle wording: Four independently written principle sets, including one derived from Shutterstock's official content review guidelines, all converge to roughly 49.0 accuracy and 2.5 score on VDC, suggesting the gains come from the iterative reflection mechanism rather than specific phrasing.

Methodology in Plain English

The authors give a single MLLM three jobs inside a loop. First, it writes a caption for a video using a current instruction prompt. Second, it scores that caption from 0 to 100 against a written checklist of what makes a good caption, such as covering camera motion, background, main objects, and fine-grained events, and it writes a suggestion for how to change the prompt. Third, a prompt refiner rewrites the prompt based on the suggestion, and the loop repeats. The key twist is failure handling: if the new caption scores worse than the previous one, the model enters a self-reflection stage where it re-examines the earlier reasoning that produced the bad prompt update and proposes a more reliable one instead. The loop stops once the score passes a threshold of 90 or the maximum of 4 iterations is hit.

To keep this from being slow, the authors turn the loop's output into training data. They run the agent over 4,008 high-resolution videos from Cockatiel-4K (drawn from OpenVid-1M) across five caption dimensions, producing 20,040 raw caption sets. They then automatically discard 1,078 sets whose first caption already passed the threshold, since those carry no learning signal, and 76 more due to JSON parsing errors, leaving 18,886 usable sets. From each set they keep only the highest-scoring caption as the preferred example and the lowest-scoring one as the dispreferred example, recording the score difference between them.

That score difference becomes a difficulty label. The model, Qwen2.5-VL-7B-Instruct, is fine-tuned with DPO on LoRA adapters only, and the training pairs are fed in descending order of score gap so that obvious contrasts are learned first and subtle ones later. Because the model has now absorbed the outcome of reflection during training, it can produce a refined-quality caption in a single forward pass, matching base-model latency. The authors verify the approach on three backbones and against publicly available video captioning models on the VDC and DREAM-1K benchmarks.

Why This Matters

Impact on research. The paper attacks a structural bottleneck in video captioning: the field's reliance on human annotation or on distillation from much larger proprietary models. By showing a 7B/8B-scale model can improve itself through principle-guided self-critique, it offers a supervision-free alternative that could be reproduced at lower cost. The curriculum DPO formulation, which repurposes an automatically generated score gap as a difficulty signal, is also transferable to other preference-optimization settings.

Real-world applications (drawn from the paper's stated downstream uses):

• Video retrieval, where richer and more accurate descriptions improve search over large video libraries.
• Text-to-video generation, where detailed captions serve as training or conditioning signal.
• Visual question answering over video and open-vocabulary semantic segmentation, both of which depend on fine-grained video understanding.
• Object detection and temporal localization in video, where temporally grounded descriptions provide supervision and grounding cues.

Industry relevance. The industrial connection is explicit: one of the four principle sets (P4) was derived from Shutterstock's official content review guidelines, and the pipeline still reached about 49.0 accuracy / 2.5 score, suggesting the method works with commercial-style standards rather than bespoke academic prompts. The efficiency result matters commercially too — the internalized model runs at 15.5s per sample versus 164.9s for test-time reflection, which is the difference between a research demo and a deployable captioning service.

Future Directions

• Scaling and saturation beyond T=4: Performance plateaus after roughly four iterations (49.08% at T=4 versus 49.55% at T=6) while construction time nearly doubles from 49.2h to 72.2h. What mechanism would break that plateau rather than add latency?
• Why weak at summarization: Gains on the "short" dimension are the smallest (+1.73%), which the authors attribute to existing summarization strength. Whether the reflective loop can be redesigned to target global abstraction is left open.
• Principle design as a research object: The four principle sets produce similar results, but they share the same five task dimensions. How far the method generalizes to structurally different principles, or to domains beyond video, is not established.
• Scorer reliability at scale: Human agreement on 500 sampled pairs ranged from 67.6% to 75.8%, while the GPT-5.1 judge averaged 89.8%. Closing the gap between automated scoring and human judgment, and testing the self-scoring signal on pairs nearer the threshold, remain open questions.

Target Audience

Researchers and engineers working on video-language models who want to reduce dependence on proprietary teacher models or human annotation; practitioners building video captioning, retrieval, or metadata systems at scale; and readers interested in agentic self-improvement and preference optimization, including the specific technique of using an automatically generated score gap as a curriculum signal.

Authors’ abstract

Existing Video Detailed Captioning (VDC) methods predominantly rely on costly human annotations or distillation from powerful proprietary models, creating a dependency on external supervision. In this paper, we propose VDC-Agent, an autonomous self-evolving framework that empowers a single Multimodal Large Language Model (MLLM) to generate and refine high-quality captions through principle-guided self-reflection. To overcome the inference latency inherent in iterative refinement, we further propose to internalize this reflective capability into the model. Specifically, we construct VDC-Agent-19K, a preference dataset derived from the agent's self-scored trajectories, and introduce a Curriculum Direct Preference Optimization (DPO) strategy. This strategy leverages the quality gap between generated candidates to progressively align the model from easy to hard samples. Extensive experiments demonstrate that VDC-Agent achieves state-of-the-art performance on VDC and DREAM-1K benchmarks, generating captions with superior detail and faithfulness. Crucially, our internalization strategy retains the inference efficiency of the base model while significantly enhancing its generalization capabilities, as validated by both quantitative metrics and human evaluation.