VC4VG: Optimizing Video Captions for Text-to-Video Generation

AI summary

Overview

Research area: Computer vision, specifically text-to-video (T2V) generation and video captioning data curation.

Technical level: Intermediate. The paper assumes familiarity with multimodal large language models (MLLMs), diffusion/DiT video generators, fine-tuning (SFT, LoRA, DPO), and standard generation benchmarks, but its core ideas about caption design are explained in accessible terms.

Scope: The paper proposes VC4VG, a framework that defines what a good training caption for text-to-video generation should contain, builds a benchmark to measure it, and validates the approach by fine-tuning a video generator.

What This Paper Is About

Text-to-video models need large numbers of video-text pairs to train, but most videos online have no captions or only low-quality ones, so systems rely on AI-generated "pseudo-captions." The problem is that no existing work defines systematically what those captions should contain, how to measure their quality for generation (rather than for video understanding), or how caption quality feeds back into generation quality. This paper introduces VC4VG, which decomposes captions into five dimensions essential for reconstructing a video, builds a 1,000-pair human-annotated benchmark (VC4VG-Bench) to score captions, and shows through fine-tuning experiments that better captions produce better videos.

Key Contributions

1. Dimension-aware caption optimization. The authors decompose video captioning into five dimensions critical for video reconstruction: subject attributes, environmental context, motion dynamics, camera parameters, and atmospheric/stylistic elements. They also build a 7B-parameter captioner, LLaVA-Video-Gen-7B (built on LLaVA-Video, augmented with Gemini 1.5 Pro and temporal-sensitive data from RTime), to demonstrate the framework at scale on datasets with over 10M videos.

2. VC4VG-Bench. A hierarchical, LLM-assisted benchmark with 1,000 human-annotated Video-QA pairs spanning multi-level visual content, from high-level themes to fine-grained details. It introduces a necessity-based hierarchy separating "core" (Necessity-L1) from "supplementary" (Necessity-L2) information for video reconstruction, enabling automated LLM-as-judge evaluation.

3. Closed-loop validation through T2V fine-tuning. The authors fine-tune CogVideoX-5B on a 72K-sample video-caption dataset curated from OpenVid-1M across three captioning methods (CogVLM2-Caption, LLaVA-Video-7B, and their LLaVA-Video-Gen-7B), showing that caption quality correlates with generation performance on VBench and MovieGenBench.

Main Findings

• Benchmark scores separate captioners. On VC4VG-Bench, LLaVA-Video-Gen-7B reaches an overall total of 804/57.0, ahead of LLaVA-Video-7B (651/46.2), Gemini 1.5 Pro under the free-generated setting (713/50.6), and CogVLM2-Caption (510/36.2). The highest overall score reported is Gemini 1.5 Pro-VC4VG at 972/68.9.

• Prompt strategy matters more than raw model strength. Using Gemini 1.5 Pro with VC4VG prompting (972/68.9) substantially outperforms the same model with MiraData-style prompting (878/62.3) and with no prompt guidance (713/50.6), suggesting the caption design itself drives much of the gain.

• Generation quality tracks caption quality. In VBench evaluations, CogVideoX-5B fine-tuned on LLaVA-Video-Gen captions leads on most metrics, including total score (82.50 vs 79.97 for the raw model, 81.79 for LLaVA-Video-7B captions, and 81.54 for CogVLM2-Caption captions), subject consistency (94.25%), background consistency (95.58%), aesthetic quality (65.16%), imaging quality (65.95%), object class (90.98%), and multiple objects (77.90%).

• Human evaluation favors the optimized captions. In blind GSB (Good/Same/Bad) comparisons on 200 MovieGenBench samples, LLaVA-Video-Gen outputs were rated better than CogVLM2-Caption on 37.5% of overall cases versus 11.5% worse, and better than LLaVA-Video-7B on 61% versus 10.5% worse.

• Camera motion remains hard. The authors report that while models control shot size and angles well, movement patterns are challenging due to MLLMs' limited understanding of fine-grained temporal dynamics, compounded by MovieGenBench's sparse coverage of complex camera motions.

• Ablations confirm both training stages matter. Prompt engineering alone on the base LLaVA-Video-7B actually hurt performance (599/42.5 vs the 651/46.2 baseline). SFT on the curated 200K WebVid subset raised it to 780/55.3, and adding DPO on RTime reached 804/57.0, with gains concentrated in motion and camera dimensions.

• Automated judging is reliable enough. The GPT-4o-based LLM-as-judge pipeline achieved a consistency rate over 80% with human judgments.

Methodology in Plain English

The researchers started from the observation that professional video creators think about shots in specific terms, and translated that into five caption dimensions: what the subject looks like and how subjects relate spatially; the environment (lighting, weather, time of day, layout); motion (gradual changes, limb movements, movement paths); camera behavior (shot size, angle, movement patterns, plus modifiers like slow motion); and style (emotional ambiance and stylistic descriptors like anime or cyberpunk).

To test whether captions covering these dimensions help, they built a captioner rather than relying only on off-the-shelf models. They curated 200K high-quality videos from WebVid-10M (filtering for durations of 5-15 seconds, using Qwen2VL to sample diverse content tags, then cleaning by aesthetic quality and motion intensity), discarded the original noisy WebVid captions, and had Gemini 1.5 Pro write fresh descriptions. Separately, they took the RTime dataset of 21K videos with distinct forward and reversed semantics (e.g., "opening a door" versus "closing a door") and used the verified short captions to guide Gemini 1.5 Pro into producing temporally aware long descriptions as (video, forward_caption, reversed_caption) triples. They then fine-tuned LLaVA-Video-7B with LoRA using 32 uniformly sampled frames per video, followed by DPO on the RTime triples.

For evaluation, they collected aesthetically strong videos from Pixabay (typically 5-20 seconds) and had humans write QA pairs while looking at both the video and a Gemini-1.5-Pro caption, so that human visual interpretation and model understanding cross-check each other. Temporal questions were clustered to avoid unmanageable dependency chains, and scoring points were split so partial information still earns partial credit. GPT-4o-0806 then acts as judge, extracting relevant content from a candidate caption and comparing it against reference answers.

Finally, they closed the loop: they fine-tuned CogVideoX-5B with full parameters on roughly 72K videos from OpenVid-1M (49-frame sampling, 720×480 resolution, learning rate 2e-5, 64×NVIDIA H20 GPUs, 5 epochs, about 1,600 steps), using captions from three different captioners, and evaluated the resulting models. They found peak VBench performance at 1,200 training steps and used that checkpoint for all comparisons. Inference uses 8 fps and the CogVideoXDPMScheduler with 50 steps at guidance scale 6, producing roughly 6-second videos.

Why This Matters

Impact on research: This work reframes video captioning from a video-understanding task into a data-engineering task for generation. By linking caption dimensions, a purpose-built benchmark, and actual fine-tuning outcomes in one loop, it gives the field a testable recipe rather than a collection of ad hoc prompting heuristics. It also argues explicitly that traditional metrics like BLEU and CIDEr, designed for short generic captions, are the wrong tools for this problem, and that benchmarks like AuroraCap and Dream-1K target understanding rather than generation.

Real-world applications:

• Training more instruction-faithful text-to-video models by upgrading caption pipelines instead of only scaling model size or video volume.
• Automated quality control for video data vendors and dataset builders, who can screen candidate captions with the benchmark before committing expensive GPU training runs.
• Content production tools that require controllable outputs, since the camera and subject dimensions map directly to prompt features users care about.
• E-commerce and advertising video generation, relevant given the Alibaba/Taobao affiliations, where product appearance and motion fidelity are commercially critical.

Industry relevance: The paper targets a bottleneck that matters economically, namely the cost and quality of training data. Its captioner is 7B parameters and locally deployable, framing large-scale captioning as an efficiency problem rather than one requiring only expensive proprietary APIs. The work was sponsored by the CCF-ALIMAMA TECH Kangaroo Fund (NO. CCF-ALIMAMA OF 2024007), and code and benchmark tools are released at https://github.com/qyr0403/VC4VG under CC BY-NC 4.0.

Future Directions

• Improving camera movement understanding in captioners, which the paper identifies as the weakest dimension and attributes to MLLMs' limited grasp of fine-grained temporal dynamics.
• Expanding benchmark coverage of complex camera motions, since MovieGenBench's sparse coverage is cited as a factor limiting evaluation of that dimension.
• Testing whether the dimension-aware strategy transfers beyond CogVideoX-5B to other T2V architectures and larger training scales; the paper only reports fine-tuning one model on a 72K-sample subset of OpenVid-1M.
• Reducing residual bias in the automated LLM-as-judge setup, which the limitations section acknowledges, and better controlling for variation from video processing techniques and prompt engineering strategies.

Target Audience

Researchers and engineers working on text-to-video generation, video-text dataset construction, or multimodal captioning models will get the most from this paper. It is also useful for data curation teams at companies building generative video products, and for benchmark designers interested in necessity-graded, LLM-judged evaluation schemes. Readers looking for a purely architectural contribution will not find one here; the value is in the caption design methodology, the benchmark, and the empirical link between caption quality and generation quality.

Authors’ abstract

Recent advances in text-to-video (T2V) generation highlight the critical role of high-quality video-text pairs in training models capable of producing coherent and instruction-aligned videos. However, strategies for optimizing video captions specifically for T2V training remain underexplored. In this paper, we introduce VC4VG (Video Captioning for Video Generation), a comprehensive caption optimization framework tailored to the needs of T2V models. We begin by analyzing caption content from a T2V perspective, decomposing the essential elements required for video reconstruction into multiple dimensions, and proposing a principled caption design methodology. To support evaluation, we construct VC4VG-Bench, a new benchmark featuring fine-grained, multi-dimensional, and necessity-graded metrics aligned with T2V-specific requirements. Extensive T2V fine-tuning experiments demonstrate a strong correlation between improved caption quality and video generation performance, validating the effectiveness of our approach. We release all benchmark tools and code at https://github.com/alimama-creative/VC4VG to support further research.