Factorized Learning for Temporally Grounded Video-Language Models

AI summary

Overview

Research area: Computer Vision / Video-Language Models (video LLMs) with a focus on temporal grounding and preference optimization.

Technical level: Intermediate. The paper assumes familiarity with vision-language models, autoregressive generation, and preference optimization methods such as DPO, but the core ideas are explainable in plain terms.

One-sentence scope: This paper proposes a framework (D²VLM) and a training algorithm (FPO) that separate the tasks of locating events in video and generating text answers, so that video-language models ground events more accurately and answer questions more reliably.

What This Paper Is About

Video-language models often produce answers about video content while struggling to correctly identify when the relevant events occur. The authors observe that these two capabilities — temporal grounding (locating evidence in time) and textual response (describing or answering) — have a natural logical order: you must find the right evidence before you can describe it correctly. Existing models mix these two tasks together during training with no clear structure, which leads to weaker results on both.

The paper's goal is to restructure how a video LLM produces its output — first grounding events, then answering while explicitly referring back to that evidence — and to design a training algorithm that improves both capabilities at once.

Key Contributions

1. D²VLM framework: A decomposition of the model's output into two sequential stages — a pure temporal grounding stage followed by interleaved text-evidence answer generation — that preserves the logical dependency between locating evidence and describing it.

2. Evidence tokens (<evi>): A new special token type that goes beyond timestamp representation by explicitly capturing event-level visual semantics. Visual features from frames similar to the <evi> token are pooled and added into it, so the token carries the actual visual content of the grounded event forward as context for the answer.

3. Factorized Preference Optimization (FPO): A preference-learning algorithm that extends standard DPO-style optimization by adding an explicit probabilistic term for temporal grounding, allowing the model to learn preferences over both where events are located and what text is generated.

4. Factorized preference data synthesis: An automatic pipeline that creates preferred/dispreferred response pairs by injecting controlled, event-level perturbations (temporal shifts, added/deleted events, merged events, distorted or repeated text) — the first preference data of its kind to explicitly and factorially model temporal grounding, requiring no manual annotation.

Main Findings

• State-of-the-art across multiple benchmarks: The 3.8B-parameter D²VLM outperforms much larger models (up to 13B) on the E.T. Bench suite. On grounding tasks it achieves 42.3 average F1 versus 35.3 for Qwen2.5-VL-7B and 33.5 for E.T.Chat-3.8B, and on dense captioning it reaches 37.5 F1 and 21.8 similarity.

• Strong results on individual datasets: On Charades-STA it reaches 50.3 R@1 at IoU 0.5 (versus 48.7 for VideoChat-T-7B), and on YouCook2 it achieves 26.4 F1, 10.6 CIDEr, and 3.2 SODA_c — beating TRACE-7B's 22.4 / 8.1 / 2.2.

• Decomposition alone gives large gains: Simply restructuring the objective into grounding-then-answering lifts average grounding F1 from 21.2 to 28.9. Adding interleaved text-evidence generation pushes it to 35.6, and the consistency constraint between the two stages brings it to 39.5.

• The evidence token design matters independently: Removing event-level modeling (using per-frame instead of per-interval tokens) drops grounding F1 to 26.1. Removing explicit visual semantic capture drops it to 37.1. The full design reaches 39.5.

• FPO adds measurable gains: Applying factorized preference optimization on top of the full framework improves grounding F1 from 39.5 to 42.3, dense captioning F1 from 35.0 to 37.5, and caption similarity from 21.2 to 21.8 — with a larger benefit on grounding than on text.

• The consistency constraint reinforces logical coherence: Enforcing that <evi> tokens generated during the answering stage match those from the grounding stage keeps the final response aligned with the evidence the model originally identified.

• Recognized weaknesses remain: Episodic memory F1 is only 14.4%, and YouCook2 dense video captioning is 26.4 F1, indicating substantial room for improvement on the hardest sub-tasks.

Methodology in Plain English

The researchers start from a simple observation about how video models are trained: normally, the model emits grounding tokens and text tokens interleaved throughout its output, and the training signal treats everything uniformly. This paper splits the model's response into two clearly separated phases.

In the first phase, the model produces only <evi> tokens — one per event it believes is relevant. Each <evi> token is compared against every frame's visual feature, producing a similarity score that says "this frame belongs to this event." Frames with high similarity are treated as salient; their features are averaged and added into the <evi> token itself, so the token literally carries the visual content of the event. A BCE loss trains these similarity scores against the ground-truth frame labels.

In the second phase, a </evi> marker signals the switch, and the model generates the actual answer. Crucially, the timestamps in the answer are produced by generating <evi> tokens again — not by writing timestamps as plain text — which acts as a form of "evidence referencing." A consistency loss forces the <evi> tokens in the second phase to match those from the first phase, keeping both stages logically aligned. The total objective combines standard supervised fine-tuning loss, the grounding loss, and this consistency loss.

For preference optimization, the authors needed a way to assign a probability to a temporal interval being grounded, which standard token-level preference learning can't express. They reuse the frame-level similarity scores: the probability of an interval [s, e] is the product of similarities for frames inside the interval times (1 − similarity) for frames outside it. This grounding probability is added as a second term alongside ordinary token log-probabilities inside the DPO objective, yielding FPO.

To create training pairs for FPO, they synthesize "bad" responses automatically. For each sample, they randomly pick a perturbation type — temporal (shift an interval, delete an event, add a false event, merge two events) or textual (distort key information, repeat a previous response) — and apply it to a randomly selected sub-event rather than the whole video. Because they know exactly what kind of noise was injected and where, the data is clean and controlled without any human labeling.

The model itself uses EVA-CLIP ViT-G/14 as the visual encoder, a Q-Former-style compressor, and Phi-3-Mini (3.8B) as the language model, fine-tuned with LoRA on the E.T. Instruct 164K dataset. Training takes about one day on four H100 GPUs.

Why This Matters

Impact on research: The paper reframes temporal grounding and text generation as a hierarchy rather than a flat multi-task problem, and shows that the structure of the output sequence itself is as important as the model or data. It also demonstrates that preference optimization can be extended to structured, non-textual predictions like time intervals — a general idea that could apply to other grounded generation tasks such as spatial grounding or audio localization. The finding that special tokens should carry visual semantics, not just positional encodings, is a design insight likely to influence future token-based grounding architectures.

Real-world applications:

• Video search and retrieval: Users searching for "the moment the dog jumps into the pool" need accurate timestamps, not just a description; better grounding enables precise clip retrieval in large media libraries.
• Video editing and highlight generation: Automatic sports highlight reels or social media clip extraction depend on locating the exact intervals where key events occur.
• Accessibility and media description: Audio description for blind and low-vision users requires both accurate event localization and coherent narration tied to what is on screen at each moment.
• Surveillance and safety monitoring: Identifying when specific activities occur in long recordings, with textual explanations of the evidence, supports incident review and compliance workflows.

Industry relevance: Video platforms, media asset management systems, video editing tools, and any product built on video question answering benefit directly. The 3.8B model size matters here — the approach achieves state-of-the-art results without scaling up, which lowers inference cost and makes deployment practical for companies that cannot afford 13B+ models. The synthetic preference data pipeline removes a major annotation bottleneck, since no human labeling is required to produce the training pairs.

Future Directions

• Improving the hardest sub-tasks: Episodic memory (14.4% F1) and YouCook2 dense captioning (26.4% F1) remain far from solved; the authors explicitly flag these as areas where the approach still falls short, suggesting that long-context or multi-event reasoning needs further work.

• Positive sample synthesis: The current data pipeline only generates dispreferred (negative) responses. Extending it to produce diverse, high-quality positive responses could enrich the preference signal and improve learning, particularly for rare or complex events.

• Scaling to longer videos and more events: The grounding probability is computed over frame-level similarities across the whole video; how this models long-form content with dozens of events, or overlapping and ambiguous events, is an open question.

• Generalizing factorized preference learning: The idea of defining preference objectives over structured, non-textual outputs could transfer to spatial grounding, audio-visual event localization, or any task where the model's output contains both a discrete location and a natural-language description with an inherent dependency between them.

Target Audience

This paper is most valuable to researchers and engineers working on video-language models, video question answering, and temporal grounding, particularly those interested in model architecture design or preference-based fine-tuning. It will also appeal to practitioners building video search, editing, or captioning products who need accurate timestamp output without running very large models. Readers should be comfortable with transformer-based vision-language architectures and have some familiarity with DPO or preference optimization to fully appreciate the FPO contribution; the framework design sections are accessible to a broader machine learning audience.

Authors’ abstract

Recent video-language models have shown great potential for video understanding, but still struggle with accurate temporal grounding for event-level perception. We observe that two main factors in video understanding (i.e., temporal grounding and textual response) form a logical hierarchy: accurate temporal evidence grounding lays the foundation for reliable textual response. However, existing works typically handle these two tasks in a coupled manner without a clear logical structure, leading to sub-optimal objectives. We address this from a factorized learning perspective. We first propose D$^2$VLM, a framework that decouples the learning of these two tasks while also emphasizing their inherent dependency. We adopt a "grounding then answering with evidence referencing" paradigm and introduce evidence tokens for evidence grounding, which emphasize event-level visual semantic capture beyond the focus on timestamp representation in existing works. To further facilitate the learning of these two tasks, we introduce a novel factorized preference optimization (FPO) algorithm. Unlike standard preference optimization, FPO explicitly incorporates probabilistic temporal grounding modeling into the optimization objective, enabling preference learning for both temporal grounding and textual response. We also construct a synthetic dataset to address the lack of suitable datasets for factorized preference learning with explicit temporal grounding. Experiments on various tasks demonstrate the clear advantage of our approach. Our source code is available at https://github.com/nusnlp/d2vlm.