TempCloze: Can Video-LLMs Identify the Missing Middle?

AI summary

Overview

Research area: Computer Vision — video temporal reasoning benchmarks for Video Large Language Models (Video-LLMs).

Technical level: Advanced. The paper assumes familiarity with Video-LLM architectures, vision-language evaluation protocols, and temporal reasoning benchmarks such as TempCompass, TVBench, and VideoMME. The core idea is easy to grasp, but the experimental and analytical sections are aimed at researchers in multimodal evaluation.

Scope: The paper introduces TempCloze, a video cloze benchmark of 1,521 videos in which models must identify the true missing middle clip from four candidates given only the beginning and ending clips, and uses it to evaluate 10 proprietary and 21 open-source Video-LLMs.

What This Paper Is About

Existing temporal reasoning benchmarks for Video-LLMs are mediated by language: models pick among textual answer options or write captions, which leaves room for linguistic shortcuts such as exploiting option wording, answer correlations, or language priors. For example, the paper notes that pretrained language models can exceed the random baseline by more than 25% without any multimodal context. TempCloze replaces textual answers with video candidates: given the beginning and ending clips of a video, a model must select which of four candidate middle clips actually belongs in the gap, so that success depends on visual temporal fit rather than on wording.

Key Contributions

1. A video cloze benchmark. The authors introduce TempCloze, comprising 1,521 carefully filtered videos drawn from seven public sources, where each video is instantiated with three dimensions — Semantic (S), Alignment (A), and Progression (P) — using same-source distractors that share scenes and objects with the original video.

2. A three-dimension distractor design. Semantic distractors are same-duration clips that do not overlap the missing interval (testing what should happen); Alignment distractors shift or expand the ground-truth span via Advanced, Deferred, and Expanded variants (testing when it should occur); Progression distractors use Reversed, Reordered, and Repeated edits (testing how it should unfold).

3. A large-scale evaluation. The authors evaluate 10 proprietary and 21 open-source Video-LLMs and identify temporal Alignment as the primary bottleneck across models.

4. Error pattern and behavioral sensitivity analyses. Using four representative models (Gemini2.5-Pro, Seed1.6, Qwen3.5-397B-A17B, and Qwen3VL-8B-I) on the TempCloze-Mixed and TempCloze-Hard subsets, the paper examines where errors arise and how candidate order, context direction, visible span, frame density, and test-time scaling influence model choices.

Main Findings

• Alignment is the bottleneck. Proprietary models average 48.13% on Alignment versus 70.73% on Semantic and 67.72% on Progression. Open-source models average 26.54% on Alignment versus 34.00% on Semantic and 36.97% on Progression. Even the strongest Alignment scores — 76.92% for Seed1.8 and 51.55% for Qwen3.5-35B — fall well below the 98.00% human Alignment baseline.

• Proprietary and open-source models are separated by a large group-level gap. Proprietary models average 62.19% mean dimension accuracy with 65.12% on the ≥2 cumulative metric and 33.24% on 3/3. Open-source models average 32.51% mean accuracy, 26.49% on ≥2, and 7.15% on 3/3. The random baseline is 25.00% per dimension, 57.81% on ≥1, 15.62% on ≥2, and 1.56% on 3/3.

• The best open-source system approaches proprietary Semantic performance but not the rest. Qwen3.5-397B reaches 75.94% on Semantic and 78.24% on Progression, but only 50.62% on Alignment, for a 68.27% mean.

• Complete three-dimension understanding is rare. Even the best proprietary model (Seed1.8 in thinking mode) reaches only 70.81% on 3/3, against a 92.00% human baseline. The best open-source model reaches 35.77% on 3/3.

• Proprietary and open-source models differ in their S-versus-P profile. Open-source models, despite low absolute scores, perform better on Progression than Semantic, suggesting local motion continuity can support basic Progression scores. Proprietary models sit at higher scores and favor Semantic over Progression.

• Alignment errors are dominated by "Expanded." The selected clip contains the correct event content but extends before and after the target interval. Seed1.8-I selects Expanded in 75% of its Alignment errors.

• Progression errors are dominated by "Reversed." GPT-5.4 selects Reversed in 67% of its Progression errors.

• Alignment is the most distracting alternative in mixed settings. On TempCloze-Mixed, Alignment receives the highest error share for every representative model: 19.7% for Gemini2.5-Pro, 20.3% for Qwen3.5-397B-A17B, 14.7% for Seed1.6, and 37.3% for Qwen3VL-8B-I. Semantic can be more distracting than Progression even though Semantic accuracy is generally slightly higher, suggesting incorrect progression is easier to reject than an incorrect event.

• Stable accuracy hides unstable choices. Under four candidate permutations at temperature 0 on TempCloze-Hard, mean accuracy varies by less than 4 points for every model, but the Clip Flip Rate ranges from 32.4% to 60.7% and the Flip Rate from 25.9% to 40.2%. Alignment and Progression are generally less stable than Semantic.

• Models rely more on the beginning clip than the ending clip. Ending-only context is often weakest, while beginning-only frequently approaches or exceeds using both. Seed-1.6 reaches 80% on Progression with beginning-only, close to 82% with both, but drops to 46% with ending-only.

• Longer visible spans and denser sampling do not uniformly help. As visible span expands from endpoint frames to full context, Alignment drops sharply for all models because its evidence lies near the missing gap boundaries. Increasing sampling density from 8 to 20 frames per clip produces a similar pattern: stronger models preserve or slightly improve Semantic and Progression, while Alignment declines across models.

• Test-time scaling helps, but does not reorder the dimensions. At temperature 0.7, Gemini2.5-Pro and Qwen3.5-397B-A17B show the largest gains, with Overall pass@k rising by roughly 30 points from k=1 to k=5, consistent with their higher accuracy dispersion. Alignment remains the lowest-scoring dimension regardless.

• Boundary and shortcut checks are clean. Across 1,521 videos per dimension and 18,252 candidate clips, there are zero exact boundary-frame matches. An edge-based DINOv2 baseline yields mean ground-truth probabilities of 26.35% (Edge-1) and 25.49% (Edge-4), only slightly above the uniform 25%. A single-midpoint-frame baseline collapses performance: Seed1.8 drops from 83.43% to 17.31% mean, and Gemini2.5-Pro from 74.92% to 19.07%.

Methodology in Plain English

The task. A video is split into three parts: a beginning clip, a middle clip, and an ending clip. The model sees only the beginning and the ending, plus four candidate clips, one of which is the true missing middle. Each candidate must be judged not just on whether it looks related, but on whether it fits in the exact temporal slot between the two visible clips.

Building the candidates. All four candidates come from the same source video, so they share scenes, objects, and lighting. This is deliberate: it removes appearance-based shortcuts and forces the model to compare timing rather than content. Within this setup, the authors generate three families of distractors. Semantic distractors are same-duration clips from elsewhere in the video that do not overlap the true middle. Alignment distractors are nearby clips shifted earlier (Advanced), shifted later (Deferred), or widened to include surrounding context (Expanded). Progression distractors play the true segment backward (Reversed), scramble the order of its subevents (Reordered), or duplicate a short subsegment (Repeated).

Filtering the videos. The pipeline keeps videos between 12 and 90 seconds, screens temporally dense captions with the reasoning model GPT-o3 to check that events are temporally coherent and that context can constrain a missing middle, then removes clips with bitrate below 200 kbps or sharpness below 30. The gap is sampled from the central 50% of the timeline with a length of 20% to 40% of the full duration, leaving context on both sides, and is validated with Farnebäck optical flow requiring average flow magnitude above 1.0, resampling up to three times. This leaves 1,521 videos.

Sources. Seven public sources are used: LVD-2M (515 videos), EgoLife (437), MiraData (198), FAVOR (145), CaReBench (94), Video-TT (89), and Daily-Omni (43). Long-take sources such as MiraData and LVD-2M and egocentric sources such as EgoLife and FAVOR-Bench provide temporally continuous video that reduces scene-change cues.

Evaluation. By default the authors uniformly sample 16 frames per clip (96 frames per dimension across six clips) using bin-centered sampling, so samples sit inside each clip rather than exactly at its boundaries. Open-source models run on A6000 GPUs using vLLM. Humans are used as a reference: five annotators answered 120 questions each on 100 randomly selected videos, with two independent judgments per question and strict scoring where any disagreement counts as incorrect.

Why This Matters

Impact on research. The paper argues that language-mediated temporal benchmarks let models score well through option wording, answer correlations, or language priors. By making the answer choices themselves video clips drawn from the same source, TempCloze measures visual temporal fit directly. The finding that Alignment — not Semantic content recognition — is the consistent weak point reframes where progress is needed: current Video-LLMs largely know what happened but struggle with when. The sensitivity results (unstable choices under reordering, asymmetric reliance on the beginning clip, degradation under longer spans and denser sampling) also suggest that reported accuracy numbers may overstate the stability of the underlying reasoning.

Real-world applications (implications drawn from the benchmark's design and findings):

• Video search and retrieval, where a user wants the exact moment an event occurred rather than a broadly related clip.
• Egocentric assistant systems, since EgoLife and FAVOR-Bench are among the sources and these settings require precise temporal alignment of a user's actions.
• Video editing and summarization tools that must decide where a cut belongs inside a continuous take.
• Robotics and embodied agents that need to infer what happened between two observed states rather than merely recognize plausible actions.

Industry relevance. Anyone deploying Video-LLMs for long-video understanding, video indexing, or agentic video pipelines should note that even the strongest proprietary model reaches only 70.81% on the all-three-dimensions metric, and that adding frames or context does not reliably improve Alignment. The benchmark and code are released at https://github.com/CedricPei/Temporal-Cloze under a CC BY 4.0 license.

Future Directions

1. Video-native architectures. The authors suggest that exploring architectures which do not treat video as a sequence of images could better exploit dense temporal evidence, since simply increasing sampling density did not improve Alignment.

2. Mechanisms for integrating both temporal directions. Because models rely more on forward continuation cues than backward inference, the paper calls for mechanisms that jointly use beginning and ending context.

3. Better evidence selection over long context. Longer visible spans add temporal evidence but can distract models from the most diagnostic frames, so methods for identifying critical frames are identified as a need.

4. Broadening domains and modalities. The Limitations section notes that TempCloze draws from public sources emphasizing long-take, egocentric, and fine-grained motion videos, and that future extensions could broaden video domains and modalities while preserving the visual candidate-comparison principle. The paper also states that TempCloze does not directly cover open-ended generation, dialogue-based narrative understanding, audio-grounded inference, or unconstrained future prediction.

Target Audience

Researchers and engineers working on Video-LLMs, multimodal evaluation, and video temporal reasoning; benchmark designers interested in reducing linguistic shortcuts in vision-language evaluation; and practitioners building long-video understanding, video retrieval, or egocentric-assistant systems who need to know where current models fail. Readers seeking a beginner-level introduction to video-language models would need supplementary background, since the paper assumes familiarity with the benchmark landscape it critiques.

Authors’ abstract

Temporal reasoning benchmarks for Video-LLMs are often mediated by language, leaving room for linguistic shortcuts from option wording, answer correlations, or language priors. To reduce such shortcuts, we introduce TempCloze, a video cloze benchmark for evaluating visual temporal reasoning in Video-LLMs. Given the beginning and ending clips of a video, models must identify the true missing middle from four candidates. TempCloze contains 1,521 carefully filtered videos from seven sources, mainly long-take and egocentric videos. We construct same-source distractors along three dimensions: Semantic asks what event should happen, Alignment probes when it should occur, and Progression tests how it should unfold, while shared scenes and objects reduce appearance cues. Our evaluation of 10 proprietary and 21 open-source Video-LLMs reveals Alignment as the primary bottleneck: models often recognize plausible semantic content and local event progression but struggle with temporal alignment. We further conduct error pattern and behavioral sensitivity analyses on TempCloze-Mixed and TempCloze-Hard with four representative models to examine where errors arise and how candidate order, context direction, visible span, frame density, and test-time scaling influence model choices.