What's the Catch? Evaluating Temporal Consistency in Vision-Language Models

AI summary

Overview

Research area: Multimodal vision-language models (VLMs) and temporal reasoning; natural language processing / computer vision evaluation.

Technical level: Intermediate. The benchmark design and tasks are easy to grasp, but the paper assumes familiarity with VLMs, benchmark evaluation, prompting protocols, and metrics such as accuracy and LPIPS.

Scope: A single-sentence summary — the paper introduces TimeCatch, a controlled benchmark that tests whether vision-language models can detect and localize temporal inconsistencies (swapped adjacent frames) versus frame-level anomalies (a frame replaced with Gaussian noise), and compares model performance against humans.

What This Paper Is About

Vision-language models score well on video and image-sequence benchmarks, but it is unclear whether this reflects genuine sensitivity to how visual content evolves over time, or whether models rely on shortcuts such as textual cues or a small subset of frames. The authors formulate temporal grounding as an anomaly detection problem: a sequence is either temporally consistent or contains a deliberately introduced inconsistency, and the model must say so (and, in the localization variant, say where). Their goal is a controlled test of temporal consistency that keeps the underlying visual content unchanged and minimizes reliance on language biases.

Key Contributions

1. A systematic evaluation of state-of-the-art vision-language models and human performance on temporal anomaly detection and localization.
2. A controlled comparison of VLM and human performance on temporal versus frame-level anomaly tasks, revealing a substantial gap in sensitivity to temporal consistency.
3. TimeCatch, a controlled benchmark for evaluating temporal grounding through temporal and frame-level anomaly detection, spanning four datasets in synthetic and real-world domains.
4. A preliminary fine-tuning experiment (Appendix D) showing that targeted supervision can improve temporal anomaly detection and transfer to unseen datasets.

Main Findings

• Near-chance temporal detection: In the main evaluation setting, the highest temporal anomaly detection accuracy across all evaluated models is 57.4%.
• Limited temporal localization: The highest temporal anomaly localization accuracy is 43.2%.
• Strong frame-level control performance: Frame-level anomaly detection and localization reach up to 99.6% and 99.4% respectively, showing models can attend to sequences and find salient anomalous frames.
• Humans far outperform models: On a shared subset of 72 samples from each dataset, humans achieve 75.0–91.7% on temporal anomaly detection and 83.3–88.9% on temporal anomaly localization. On that same subset, no evaluated model exceeds 70% detection accuracy or 50% localization accuracy on any dataset.
• Visual similarity is not the main bottleneck: Grouping swapped frame pairs into LPIPS bins of 0.1, performance rises only slightly for more perceptually distinct pairs, and accuracy remains far below human levels across all bins.
• A systematic localization bias, not near misses: The Qwen3-VL-8B confusion matrix shows a strong bias toward predicting position 2 regardless of the true anomaly location, rather than clustering near the diagonal.
• Model scale helps detection but not enough: Evaluating Qwen3-VL at 2B, 4B, 8B, and 32B shows consistent gains on frame-level tasks and gains in temporal detection particularly from 4B to 32B, while temporal localization remains comparatively low.
• Prompt engineering does not close the gap: Alternative prompt formulations (Prompts B–E versus the base Prompt A) have little effect on temporal detection; removing scene descriptions often yields modest improvements, and enabling reasoning produces mixed results with no consistent overall benefit.
• Longer sequences do not help: On longer CRAFT sequences up to 16 frames, temporal anomaly performance stays near chance, while frame-level detection remains high and frame-level localization degrades as the number of candidate frames grows. The final bucket (n = 16) contains only five samples and the authors caution against over-interpreting it.
• Fine-tuning can teach the task: Qwen3-VL-2B fine-tuned on the CLEVRER training split (80%) and evaluated on the held-out 20% split improved from 49.6% to 96.5% on CLEVRER, and transferred to unseen datasets: CRAFT 50.5% → 63.6%, DriveLM 50.0% → 52.7%, MTL-AQA 44.1% → 68.1% (deltas of +46.9, +13.1, +2.7, +24.0). The fine-tuned 2B model matches or exceeds zero-shot Qwen3-VL-8B (51.9 / 53.3 / 51.9 / 57.4 on those datasets).

Methodology in Plain English

The authors build an artificial but controlled failure: take an image sequence that unfolds naturally, then either swap two consecutive frames (a temporal anomaly) or replace one frame with Gaussian noise (a frame-level anomaly). For the temporal case, the swap location is sampled uniformly from the range of possible consecutive pairs; for the frame-level case, the corrupted frame is sampled uniformly from the sequence. Models face two questions per condition — does an anomaly exist (detection), and where is it (localization)? Detection accuracy is meaningful because the classes are balanced by construction.

Four datasets are curated: CLEVRER and CRAFT (synthetic 3D and 2D object interactions), DriveLM (real driving scenes), and MTL-AQA (competitive diving). Videos are converted to image sequences by temporal subsampling — every 1.5 seconds for CRAFT and CLEVRER, every 1 second for MTL-AQA. Consecutive frames with LPIPS distance below 0.05 are removed to keep anomalies perceptually meaningful, and sequences shorter than four frames are discarded. Statistics: CLEVRER 4,997 sequences at average length 7.5; CRAFT 858 at 6.9; DriveLM 696 at 5.9; MTL-AQA 338 at 4.5; plus CRAFT-Long with 1,101 sequences at average length 10.6. Scene descriptions are appended after the image tokens, drawn from dataset annotations.

Five open-weight models are evaluated zero-shot: Qwen2.5-VL-7B, Qwen3-VL-8B, Gemma-4-E4B, InternVL3-8B, and InternVL3.5-8B. Models are served with vLLM on NVIDIA A100 GPUs with constrained decoding (outputs restricted to yes/no for detection and to valid frame indices for localization). A human study recruited 24 participants through Prolific (three per dataset and task configuration), all with at least an undergraduate degree, mean age 35.2 years, spanning 16 countries, compensated above £8.00/hour against Prolific's minimum recommended £6.00/hour; mean completion time was 9.8 minutes for 15 image sequences, with three attention check sequences per participant that were excluded from results.

Why This Matters

Impact on research: The results support concerns that existing video benchmarks may overestimate temporal reasoning. A model can recognize objects, actions, or anomalous single frames while remaining insensitive to the relationships between frames, meaning strong question-answering or captioning scores do not prove temporal grounding. TimeCatch isolates that capability by perturbing only frame order and leaving visual content unchanged.

Real-world applications where decisions depend on how events unfold:

• Autonomous driving, where a reversed or impossible frame transition signals a corrupted or misordered camera stream.
• Robotics, where an agent acting on an out-of-order sequence could misjudge the progression of a task.
• Medical imaging sequences, where ordering errors could change interpretation of a study.
• Video understanding pipelines, where detecting temporally implausible input is a prerequisite to trustworthy downstream output.

Industry relevance: Any deployment that ingests video or frame sequences — content moderation, surveillance, autonomous systems, media forensics, and multimodal assistants — depends on models noticing when the sequence itself does not make sense. The systematic localization bias (predicting position 2 regardless of the true location) is directly relevant to anyone relying on a model's output for verification or logging, and the fine-tuning result suggests the limitation is at least partly a training-supervision issue rather than an inherent ceiling.

Future Directions

• Extending the benchmark with other types of frame-level controls, such as plausible distractor frames, so that the frame-level comparison is not driven merely by the salience of Gaussian noise.
• Adding evaluation of motion continuity and causal event structure, which the current adjacent-frame swap design does not capture.
• A controlled study across model scales and model families for the fine-tuning approach, which is currently preliminary and limited to a single model family (Qwen3-VL-2B, trained with LoRA via ms-swift for 3 epochs at learning rate 1×10⁻⁴, effective batch size 16, max sequence length 4096 tokens, final checkpoint step 714).
• Determining which pretraining or instruction data would provide the temporal supervision that appears to be missing, given that fine-tuning on CLEVRER transferred to unseen synthetic and real-world datasets.

Target Audience

Researchers and engineers working on multimodal models and video understanding who need to know what current VLM benchmarks do and do not measure; benchmark designers looking for a controlled diagnostic protocol; practitioners deploying VLMs on sequential visual input; and human-computer interaction or evaluation researchers interested in the human-versus-model comparison methodology described in Appendix C.

Authors’ abstract

Vision-language models (VLMs) achieve strong performance on video and image-sequence benchmarks, yet it remains unclear whether they capture temporal structure. To study this question, we formulate temporal grounding as an anomaly detection problem, providing a simple and controlled evaluation that directly tests sensitivity to temporal consistency. We introduce TimeCatch, where temporal anomalies are created by swapping consecutive frames and frame-level anomalies by replacing a frame with Gaussian noise. Models are evaluated on anomaly detection and localization tasks across four synthetic and real-world datasets, alongside a human study. Our evaluation reveals a substantial gap between frame-level and temporal anomaly detection. While VLMs consistently detect frame-level anomalies and often localize them accurately, they perform near chance on temporal anomaly detection and only modestly above chance on localization. Humans, in contrast, achieve near-ceiling performance on both tasks. Additional analyses across model scales, prompting strategies, sequence lengths, and visual similarity suggest that these failures cannot be explained solely by limitations in perception or model capacity. Together, these findings indicate that current VLMs can identify anomalies within individual frames but struggle to integrate information across frames to reason about temporal consistency. TimeCatch provides a controlled benchmark for evaluating temporal grounding in vision-language models.