Video Finetuning Improves Reasoning Between Frames

AI summary

Overview

Research area: Multimodal large language models (vision-language models), video understanding, and chain-of-thought reasoning.

Technical level: Intermediate. The paper assumes familiarity with LVLMs, finetuning, and benchmark evaluation, but its core ideas—frame-by-frame tokenization, temporal reasoning, and controlled comparisons—are explained concretely enough for readers with a general ML background.

Scope (one sentence): The paper asks what video finetuning actually adds to a multimodal LLM, using a new "Visual Chain-of-Thought" (vCoT) prompting method to compare image-only models against their video-finetuned twins under matched architecture.

What This Paper Is About

Multimodal LLMs that understand images are often extended to video simply by concatenating frame tokens, which does not guarantee any real understanding of what happens between frames. The authors investigate whether finetuning on video data gives a model genuine frame-to-frame temporal reasoning, or whether it just looks like it does. To test this, they build an explicit reasoning scaffold (vCoT) that inserts generated transitional event descriptions between consecutive frames, and then compare image-only and video-finetuned models with and without those cues.

Key Contributions

1. Visual Chain-of-Thought (vCoT): A two-step prompting pipeline that first identifies visual attributes shared by two adjacent frames, then infers a one-sentence "bridging event" describing the transition between them, producing text infills interleaved with the video frames.
2. A controlled image-vs-video comparison: Pairing image LLMs (LLaVA-NeXT, InternVL) with their video-finetuned counterparts (LLaVA-NeXT-Video, InternVL2) that share the same vision encoder, language backbone, and cross-modal projector, so differences can be attributed to video finetuning alone.
3. Modality-shuffling diagnostics: Visual shuffle (replacing frames with frames from unrelated clips) and text shuffle (swapping text infills between videos) to measure how much each model relies on visual versus textual evidence.
4. A transfer test to static relational reasoning: Evaluating whether video finetuning helps on i-RAVEN, a Progressive-Matrices-style suite with no temporal signals.

Main Findings

• vCoT helps image-only models substantially on long-form video QA. On the EgoSchema subset (500 three-minute egocentric videos, one multiple-choice question each), LLaVA-NeXT gained +7.4 points at 5 frames (44.0 to 51.4) and +6.2 points at 10 frames (49.2 to 55.4).
• Video-finetuned models gain far less from the same cues. LLaVA-NeXT-Video improved by only +1.6 (47.0 to 48.6) at 5 frames and +2.4 (49.0 to 51.4) at 10 frames. The authors read this as evidence that video finetuning already captures frame-to-frame transitions implicitly.
• The InternVL pair shows the same pattern, with one exception. InternVL-Image improved +2.0 (38.4 to 40.4) at 5 frames and +5.2 (37.4 to 42.6) at 10 frames; InternVL-Video improved +3.2 at 10 frames (45.8 to 49.0) but decreased by 2.2 at 5 frames (44.6 to 42.4).
• Video models are more robust to text noise. Under text shuffle on EgoSchema, LLaVA-NeXT-Video lost only 1.6 points at 5 frames, while the image-only LLaVA-NeXT lost 9.4. Both model types were sensitive to visual shuffle, with drops ranging from 3.6 to 11.6 points.
• Temporal reasoning appears to transfer to static relational reasoning. On i-RAVEN, video-finetuned InternVL and LLaVA gained +1.1 and +3.0 average points over their image-only baselines (InternVL 14.1 to 15.2; LLaVA 10.3 to 13.3). The largest single gains were InternVL's +3.8 on the indist4/out rule and LLaVA's +7.0 on U/D.
• vCoT beats plain frame captions as inter-frame text. With LLaVA-NeXT at 5 frames, captions reached 48.2 versus 51.4 for vCoT; at 10 frames, 54.6 versus 55.4.
• A LoRA control muddies the clean story. When image-only LLaVA-NeXT was LoRA-tuned (rank-128) on 100k image instructions from ShareGPT-4V and ShareGPT-4o to match the video variant's data scale, the image model lost the vCoT advantage under dense sampling: at 5 frames, stride 1, it dropped 0.8 points (45.8 to 45.0), against the video variant's +1.6. The authors suggest the original advantage may have been driven by training data scale rather than a lack of temporal understanding.
• An unexpected result about model strength: the paper notes that LLaVA-NeXT often outperformed LLaVA-NeXT-Video, which the authors call unexpected and interpret as a sign that the video variant may not be a strong representative of video LLMs.

Methodology in Plain English

The authors pick pairs of models that are identical except for one thing: whether the model was finetuned on video data. Because the vision encoder, language model, and projector are the same within each pair, any performance gap can be credited to video finetuning rather than architecture.

To probe what the models already know, they invent vCoT. For every pair of neighboring frames, they ask the model two questions in sequence: what the two frames have in common, and what plausible single event could have happened in between. A lightweight Qwen-2.5 model rewrites each inferred event into a concise sentence. These sentences are inserted between the frames like text placeholders, and the interleaved sequence plus the task question goes to the model for a final answer.

If adding these explicit descriptions boosts accuracy a lot, the model probably lacked the reasoning; if it barely helps, the model likely already has it. The authors run this on EgoSchema, where models output an option index A–D judged by Qwen2.5-7B-Chat using the official VLMEvalKit template. They then break the setup on purpose—swapping in frames from unrelated videos, or swapping in text infills from different videos—to see which modality each model actually leans on. Finally they test whether video finetuning helps on i-RAVEN, a purely static logical reasoning suite built from Progressive Matrices, where no temporal cues exist at all.

Why This Matters

Impact on research. The paper offers a diagnostic recipe, not just a benchmark number: by comparing architecture-matched image and video models under an injected reasoning scaffold, and by adding a data-matched LoRA control, it separates "the model learned temporal reasoning" from "the model has seen more data." It also raises a caution that the observed advantage may partly be a data-scale artifact, which is a useful corrective for work arguing that video finetuning confers unique inductive biases.

Real-world applications (implied by the tasks studied):

• Long-form video question answering, such as searching or querying hours of egocentric or recorded footage.
• Assistive and retrieval systems that need to describe what happened between sampled frames rather than just what is in each frame.
• Robotics or embodied agents that must infer intermediate events (an object being moved, a person starting an action) between sparse observations.
• Video summarization and captioning pipelines where inserting explicit transition descriptions can compensate for models that were never trained on video.

Industry relevance. The vCoT scaffold is a prompting-time intervention, so it can be layered onto existing image-based LVLMs without retraining. That makes the +7.4-point EgoSchema gain on LLaVA-NeXT practically relevant for teams already deploying image models on video tasks. The LoRA control result is equally relevant, showing that naively collecting more instruction data can change the conclusions a team would draw about their model's temporal abilities. The finding that video finetuning also helps on static relational reasoning (i-RAVEN) suggests video data may be a useful ingredient even for products that only serve images.

Future Directions

• Train control models from scratch. The authors state this as the main limitation: they would ideally use a small LLM (for example, 0.5B parameters) and train one variant on image data and one on video data to fully control the pipeline and ablate inductive biases such as temporal encoders and temporal positional embeddings.
• Decide which inductive biases to control. The paper argues that all inductive biases aside from those inherent to the base LLM architecture should be included if they help the model learn, so that upper-bound image and video variants can be compared fairly from the same initialization.
• Resolve the data-scale confound. The LoRA experiment only partially equalizes training data, and the authors also hypothesize that LoRA adaptation on 100k image-instruction pairs may cause catastrophic forgetting that weakens use of shortcut textual cues in EgoSchema. A cleaner data-matched comparison remains open.
• Explain why LLaVA-NeXT-Video underperforms LLaVA-NeXT. The authors flag this as unexpected and as evidence the video variant may be a weak representative of video LLMs, leaving open what a stronger video baseline would show.

Target Audience

Researchers and graduate students working on multimodal LLMs, video understanding, and evaluation methodology will get the most from this paper. It is also useful for practitioners who deploy image-based vision-language models on video inputs and want a low-cost reasoning scaffold, and for anyone designing controlled comparisons between model variants who needs an example of probing a benchmark with deliberate modality perturbations and a data-scale control.

Authors’ abstract

Multimodal large language models (LLMs) have made rapid progress in visual understanding, yet their extension from images to videos often reduces to a naive concatenation of frame tokens. In this work, we investigate what video finetuning brings to multimodal LLMs. We propose Visual Chain-of-Thought (vCoT), an explicit reasoning process that generates transitional event descriptions between consecutive frames. Using vCoT, we systematically compare image-only LVLMs with their video-finetuned counterparts, both with and without access to these transitional cues. Our experiments show that vCoT significantly improves the performance of image-only models on long-form video question answering, while yielding only marginal gains for video-finetuned models. This suggests that the latter already capture frame-to-frame transitions implicitly. Moreover, we find that video models transfer this temporal reasoning ability to purely static settings, outperforming image models' baselines on relational visual reasoning tasks.