D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decomposition

AI summary

Overview

• Research area: Computer vision / multimodal video-language models — specifically adapting image-pretrained vision-language models (VLMs) to video understanding.
• Technical level: Advanced (assumes familiarity with vision-language models, visual token compression, and VideoQA benchmarks).
• Scope: The paper introduces D-CoDe, a training-free framework that combines dynamic visual compression with question decomposition to extend the 7B image-pretrained LLaVA-NeXT to video, evaluated on seven VideoQA benchmarks.

What This Paper Is About

Video large language models are expensive to build because high-quality video-text training data is scarce, so a cheaper route is to adapt models already trained on images. However, video inputs produce far more visual tokens than images, and the paper argues this creates two problems: a perception bottleneck (uniform frame sampling and spatial average pooling discard salient content that is unevenly distributed across time and space) and token overload (even compressed video still exceeds the token capacity of image-pretrained models, so accuracy plateaus). The goal of D-CoDe is to fix both problems without any fine-tuning.

Key Contributions

1. Problem diagnosis. The authors formally characterize two failure modes of training-free image-to-video adaptation — the perception bottleneck (from static compression) and token overload (from excess visual tokens) — and illustrate each with controlled experiments on EgoSchema comparing static compression against dynamic compression, and a vanilla 7B LLaVA-NeXT against a question-decomposition variant under varying top-k activation retention ratios.
2. Dynamic compression. A module that augments uniform frame sampling with supplementary frames chosen for maximum semantic dissimilarity from the already-selected set (measured by cosine similarity between CLIP global features), then prunes spatial tokens by ℓ2 activation magnitude and greedily merges tokens whose cosine similarity exceeds a threshold by mean pooling.
3. Question decomposition. A module that reformulates the original query into focused sub-questions using a structured system prompt and a pretrained question-decomposition LLM, answers each sub-question against the same compressed visual input, and concatenates those sub-answers with the original query to produce the final answer.
4. Empirical validation across seven benchmarks. D-CoDe is reported as the first training-free method to surpass training-required models on EgoSchema, and it achieves the highest accuracy among all compared methods on MSVD-QA (80.0%) and TGIF-QA (79.1%).

Main Findings

• Multiple-choice VideoQA gains. D-CoDe reaches 68.3% on NExT-QA, 58.0% on EgoSchema, and 64.2% on IntentQA, the top score in every column of Table 2 across both training-required and training-free methods.
• EgoSchema is the headline result. D-CoDe exceeds the second-best training-free method, TS-LLaVA (50.2%), by 7.8% and surpasses the best training-required model, MovieChat+ (56.4%), by 1.6%, making it the first training-free method to outperform all training-required ones on this long-form egocentric benchmark.
• Open-ended VideoQA gains. D-CoDe achieves 80.0/4.1 on MSVD-QA, 64.2/3.5 on MSRVTT-QA, 79.1/4.1 on TGIF-QA, and 56.4/3.4 on ActivityNet-QA (accuracy/GPT-Score). Only the dynamic compression module is applied here, since the paper notes these questions are simpler and less suited to decomposition.
• One benchmark where it trails. On MSRVTT-QA, D-CoDe (64.2/3.5) performs slightly below SF-LLaVA (65.8/3.6) and TS-LLaVA (65.1/3.6), which the authors attribute to the dataset's frequent scene transitions favoring slow-fast architectures.
• Module ablation on EgoSchema (15 frames). Baseline 44.8% → + dynamic spatial token compression 50.6% → + dynamic temporal frame selection 51.8% → + question decomposition 58.0%, i.e., every module adds accuracy and question decomposition adds the largest single jump.
• Sampling strategy matters. On EgoSchema without question decomposition, uniform sampling scores 50.6%, question-aware sampling 51.4%, and the paper's supplemental-frame selection 51.8%. The authors argue question-aware sampling overemphasizes query-relevant segments and weakens broader temporal modeling.
• Merge range matters. Constraining spatial merges to ≤4 or ≤5 neighboring tokens gives 51.4%, ≤6 neighboring tokens gives 52.0%, and no constraint gives 51.8%.
• Sub-answers, not sub-questions, drive the gain. On EgoSchema, no question decomposition scores 51.8%, supplying sub-questions scores 50.4%, while supplying sub-answers scores 58.0% — the paper concludes the benefit comes from intermediate answers rather than the structured breakdown itself.
• Prompt content is what counts. The default prompt scores 58.0%; removing task/background explanation drops to 53.2%; removing "temporal and dynamic aspects" drops to 54.8%; rephrasing with different wording actually improves to 58.4%, suggesting robustness to wording.
• Decomposition can hurt on simple questions. Table 13 shows that adding question decomposition to the open-ended benchmarks lowers scores relative to the compression-only stage: MSVD-QA from 80.0/4.1 to 72.4/3.8, MSRVTT-QA from 64.2/3.5 to 62.2/3.5, TGIF-QA from 79.1/4.1 to 75.7/4.0, and ANet-QA from 56.4/3.4 to 53.8/3.3.
• Hyperparameters are sensitive. On EgoSchema, accuracy peaks at uniform sampling ratio α = 0.85 (58.0%, versus 57.2 at 0.80, 57.0 at 0.90, 56.2 at 0.95); retention ratio β = 0.625 (58.0%, versus 56.6 at 0.575, 56.4 at 0.600, 56.2 at 0.650); merge threshold τ = 0.90 (58.0%, versus 55.0 at 0.80, 55.6 at 0.85, 57.0 at 0.95); and decomposition temperature t = 0.5 (58.0%, versus 56.0 at 0.3, 56.2 at 0.7, 56.4 at 0.9).
• Attention shifts with decomposition. Visualizations on IntentQA with 5 input frames show that prompting the baseline with decomposed sub-questions changes the peaks of its attention distribution over the same visual input.
• Error analysis on scene transitions. On MSRVTT-QA, SF-LLaVA scores 65.8/3.6 on the full set and 64.0/3.5 on the top-100 scene-change samples, while D-CoDe scores 64.2/3.5 on the full set and 56.0/3.3 on those scene-change samples.

Methodology in Plain English

The authors keep the image-pretrained model frozen and change only what is fed into it.

Temporal frame selection. Rather than sampling frames at fixed intervals, they first take a fraction (α = 0.85 of the budget of N frames) uniformly for coarse coverage. For the remaining frames, they repeatedly pick the candidate frame that is least similar on average to the frames already chosen, using cosine similarity between CLIP global features. This biases the input toward segments that look different from what is already represented.

Spatial token compression. For each selected frame, they compute the ℓ2 norm of every visual token as a salience proxy and keep only the top 62.5% (β = 0.625). Among the survivors, they walk through tokens in descending salience order, treat each as an anchor, and group any still-unmerged token whose cosine similarity to the anchor is at least 0.9 (τ = 0.9). Each group is replaced by the mean of its members. The resulting compact tokens from all N frames are concatenated into one visual sequence.

Question decomposition. A fixed system prompt instructs a pretrained LLM (gpt-3.5-turbo-0125, temperature 0.5) to break the user's question into sub-questions that emphasize temporal and dynamic aspects rather than single-frame static content, returning them as a Python list of strings. Each sub-question is answered against the same compressed visual input, and those sub-answers are concatenated and fed back to the model together with the original question to produce the final answer. The number of sub-questions is unconstrained.

Setup. The base model is LLaVA-NeXT with 7B parameters; rotary position embeddings with a scaling factor of 2 extend context to 8192 tokens; frames are resized to 336 × 336. N is chosen empirically per dataset based on average video length. Everything runs on a single NVIDIA RTX A6000 GPU.

Why This Matters

Impact on research. The paper argues that training-free adaptation has been held back by two specific, fixable bottlenecks rather than by a fundamental capacity limit, and it demonstrates that a training-free method can beat fine-tuned models on a hard long-video benchmark (EgoSchema) — a result that reframes how much value additional video-text training data necessarily provides. It also isolates a counterintuitive finding: the gain from decomposition comes from the intermediate sub-answers as extra context, not from the reasoning structure of the sub-questions.

Real-world applications:

• Video question answering and search over long archives such as egocentric recordings or instructional footage.
• Assistive and accessibility tools that answer natural-language questions about video content for users who cannot watch it.
• Content moderation or compliance review that needs to reason over long, complex video rather than isolated frames.
• Video analytics on existing image-trained model deployments, where retraining on video data is prohibitively expensive.

Industry relevance. Because the method is training-free, it can be applied as an inference-time wrapper on top of already-deployed image VLMs without new labeled video data or fine-tuning compute — attractive for teams that have image models in production but no large-scale video-text pipeline. The cost is inference overhead: each query requires an extra LLM call for decomposition plus one answer per sub-question.

Future Directions

• Integrating slow-fast architectures. The authors state that D-CoDe underperforms on videos with frequent scene transitions compared to SF-LLaVA and TS-LLaVA, and suggest combining a slow-fast design with their approach to better balance temporal and spatial retention.
• Adding a memory bank. The limitations section proposes that a memory bank, commonly used in Vid-LLMs to maintain long-range context, could further improve handling of complex video inputs.
• Precise temporal grounding. The paper notes that understanding durations and timestamps remains difficult for training-free frameworks like theirs, and suggests this may require task-specific training or architectural modifications, citing LLaVA-ST as an example.
• Knowing when not to decompose. Since question decomposition lowers accuracy on simpler open-ended questions such as MSVD-QA ("What is a man sitting on?"), a natural next step is deciding adaptively when decomposition helps and when it overcomplicates a query.

Target Audience

Researchers and engineers working on multimodal video-language models will get the most from this paper, particularly those interested in training-free inference-time adaptation, visual token compression, or extending image VLMs to long video. It is also relevant to practitioners who need to add video understanding to an existing image-model deployment without collecting video-text training data. Readers should be comfortable with VLM architectures, visual tokenization, CLIP-style embeddings, and standard VideoQA evaluation protocols; the question-decomposition component itself is simple enough for a broader technical audience to follow.

Authors’ abstract

Video large language models (Vid-LLMs), which excel in diverse video-language tasks, can be effectively constructed by adapting image-pretrained vision-language models (VLMs). However, this adaptation remains challenging, as it requires processing dense and temporally extended visual inputs that exceed the capacity of image-based models. This paper identifies the perception bottleneck and token overload as key challenges in extending image-based VLMs to the video domain. To address these issues, we propose D-CoDe, a training-free adaptation framework that incorporates dynamic compression and question decomposition. Specifically, dynamic compression alleviates the perception bottleneck through adaptive selection of representative frames and content-aware aggregation of spatial tokens, thereby reducing redundancy while preserving informative content. In parallel, question decomposition mitigates token overload by reformulating the original query into sub-questions, guiding the model to focus on distinct aspects of the video and enabling more comprehensive understanding. Experiments demonstrate that D-CoDe effectively improves video understanding across various benchmarks. Furthermore, strong performance on the challenging long-video benchmark highlights the potential of D-CoDe in handling complex video-language tasks. Code is available at https://github.com/hukcc/D-CoDe.