FineVAU: A Novel Human-Aligned Benchmark for Fine-Grained Video Anomaly Understanding

AI summary

Overview

Research area: Computer vision, specifically evaluation benchmarks for Video Anomaly Understanding (VAU) with Large Vision-Language Models (LVLMs).

Technical level: Advanced — assumes familiarity with LVLM evaluation, captioning metrics (BLEU, ROUGE-L, CIDEr, METEOR), and LLM-as-judge protocols.

Scope: The paper proposes FineVAU, a benchmark that decomposes video anomaly understanding into three dimensions (events/What, entities/Who, location/Where), together with FV-Score (a human-aligned LLM-based metric), a FineVAU-Judge, and the Fine W³ dataset of 1544 videos.

What This Paper Is About

Existing evaluation of Video Anomaly Understanding relies either on n-gram metrics (e.g., BLEU, ROUGE-L) that measure lexical overlap rather than factual correctness, or on LLM-based judges that score language fluency and general reasoning rather than anomaly-specific content. Both fail to signal when a model produces a coherent but factually wrong description of an anomaly. The authors build a benchmark that instead checks whether a model's report correctly captures the key elements humans use to perceive anomalies: what happened, who was involved, and where it occurred.

Key Contributions

1. FineVAU, a benchmark for VAU whose evaluation is structured around three hierarchical dimensions grounded in human anomaly perception: events (What), entities (Who), and location (Where).
2. FV-Score, an LLM-based metric that performs key-element detection on LVLM answers, producing interpretable and fine-grained feedback rather than a single vague quality score. It is implemented through a FineVAU-Judge (materialized with Gemini-2.5-Flash) that assigns membership scores to each ground-truth element.
3. Fine W³, a dataset that augments existing high-quality human anomaly video annotations with What, Who, Where information using a systematic, fully automatic, LVLM-assisted pipeline (based on Gemini-2.5-Pro and UCA annotations).
4. Extensive experiments with five open-source LVLMs (Qwen2.5-VL-7B, InternVL3-9B, VideoLLaMA3-7B, LLaVA-Video-7B, LLaVA-OneVision-7B), plus a human correlation study with 8 experts and a weight ablation study.

Main Findings

• FV-Score aligns best with humans: In a study using 60 videos randomly sampled from UCF-Crime, 8 experts, and 180 ranking judgments (Pairwise Percentage Agreement of 68%), FV-Score achieved a Pearson correlation of 0.61, 1−R² of 0.63, Kendall's Tau of 0.56, and Spearman's Rho of 0.56. The strongest n-gram baseline, ROUGE-L, reached 0.47 PCC and 0.43 Kendall's Tau, and the strongest LLM judge baseline, VAU-EVAL, reached 0.53 PCC and 0.49 Kendall's Tau.
• Poor metrics can systematically disagree with humans: BLEU scored 0.19 PCC and CIDEr scored −0.63 PCC, which the authors describe as systematic disagreement with human judgment. Notably, LLM-based baselines (AnomEVAL at 0.42 PCC, VAU-EVAL at 0.53 PCC) performed similarly to n-gram counterparts.
• Humans weight entity perception highly: Weight ablations showed the configuration λ_what=1.0, λ_who=2.0, λ_where=1.0 gave the best correlation (0.61 PCC, 0.63 1−R², 0.56 Kendall's Tau, 0.56 Spearman's Rho), compared with the equal-weight setting (0.51 PCC) and λ_what=2.0, λ_who=1.0, λ_where=1.0 (0.56 PCC).
• Models handle static, coarse-grained information better: Across the five LVLMs, mean location accuracy was 61.3%, versus entity 39.8%, attribute 37.0%, and events only 12.2%.
• Fine-grained spatial and temporal understanding is the weak point: Models perform far worse on individual object attributes than on coarse object identification, and event identification is the weakest dimension, which the authors attribute to a training bias toward higher-resolution, less cluttered general-purpose video.
• Strong visual cues help: Performance on events was better for categories such as Explosions, Arson, Arrests, and Fights, which come with conspicuous visual signals, while anomalies that unfold in small spatial and temporal windows, such as Shoplifting, were much harder.
• Models are biased toward normalcy: LVLMs more often conflate abnormal events into normal ones (e.g., describing a fight as a conversation) than the reverse, and they still understand Normal events better than abnormal ones even in videos containing abnormalities.
• InternVL3 leads overall: InternVL3 achieved the highest overall score of 51.5, ahead of LLaVA-VID (47.4), Qwen2.5-VL (44.5), LLaVA-OV (44.0), and VideoLLaMA3 (27.5), with a mean across models of 43.0.
• Context size does not determine event understanding: LLaVA-OneVision and LLaVA-VID outperformed Qwen-2.5-VL and VideoLLaMA3 on events despite lower context sizes and smaller pretraining corpora, which the authors present as evidence of a disconnect between understanding static scene context and understanding anomalous events.
• Dataset scale: Fine W³ contains 1544 videos with 17813 events (13393 normal, 4420 abnormal), 59392 entities referencing 74593 individual attributes, and 7669 annotated location attributes. Videos are CCTV footage and can run up to 1h in duration.

Methodology in Plain English

The authors first define what a complete anomaly description should contain. For each video, they build a structured ground truth made of three sets: descriptions of events, descriptions of entities with attributes (clothing, age group, gender, distinguishing features for people; size, color, brand for vehicles; at least color and size for objects), and descriptions of location (environment, time of day, lighting, crowd density, and a salient identifying feature). A model's generated report is scored by checking how many of those ground-truth elements it correctly mentions.

Scoring is done by a large language model acting as a judge. It is given the report, the ground truth set, and a detailed prompt, and it assigns each ground-truth element one of a small number of scores: a binary present/absent scale for Who and Where, and a three-level scale for What (missing/incorrect, partial with minor errors, accurate and complete). The per-dimension scores are then combined with weights that control the relative importance of What, Who, and Where.

To build the dataset, the authors start from existing human-written anomaly annotations (UCA, built on UCF-Crime) and run a two-stage automatic pipeline using an LVLM, sampling frames at 1 fps. Stage one decomposes complex event sentences into chains of causally linked atomic events, adds clearly observable events or objects that were missing, and links each event to concise entity identifiers. Stage two adds fine-grained physical attributes to those entities and describes the physical properties of the location.

For validation, the authors recruit 8 experts, sample 60 UCF-Crime videos, and have each video ranked by 3 experts (180 judgments total) across three report quality levels generated by Gemini 2.5-flash, presented in randomized order. They then compute standard agreement measures between metric scores and human rankings. Finally, they evaluate five LVLMs zero-shot using lmms-eval at temperature 0, with each model's default frame sampling.

Why This Matters

Impact on research: The paper argues that progress in VAU has been held back by evaluation rather than by modeling: n-gram metrics penalize correct answers phrased differently, and LLM judges reward fluent language over factual grounding, so a model can be scored well while hallucinating normalcy. By decomposing evaluation into anomaly-specific elements, FineVAU provides a diagnostic signal that separates coarse static understanding from fine-grained temporal and spatial perception, which the authors present as a new and challenging frontier for VAU.

Real-world applications:

• Public safety and security monitoring, where systems must flag and describe unusual events in CCTV footage without human review, as highlighted in the introduction.
• Infrastructure monitoring, another application area the paper names explicitly.
• Retail loss prevention, illustrated by the difficulty models had with Shoplifting, which requires recognizing small object placement and non-payment behavior.
• Forensic or post-incident review of long recordings, since the dataset draws on CCTV footage from public streets, highways, shopping centers, and private households, often several minutes to hours long.

Industry relevance: The benchmark targets deployable vision-language systems that summarize surveillance footage, so its findings matter for vendors building anomaly alerting, video search, and automated incident reporting. The code and dataset are released at the project site, and the paper notes that the structured data could be used to train models to mitigate hallucination, giving practitioners both a diagnostic tool and a potential training resource.

Future Directions

• Targeted training against hallucination: The authors state that their structured data can support targeted training to reduce normalcy bias and induce detailed, factual understanding, and that pairing such data with rigorous benchmarks is needed for the next generation of models.
• Improving fine-grained temporal and spatial perception: The low event accuracy (mean 12.2%) and weak attribute performance point to a need for models that reason over small spatial and temporal windows, not just static scene context.
• Handling densely annotated, long videos: With 17813 events, 59392 entities, 74593 attributes, and videos up to 1h, the benchmark raises the question of how models should recall large numbers of events without simply omitting them.
• Refining the evaluation itself: The paper frames VAU as maximizing the weighted score S(R), leaving open how the dimension weights should be set in general and how the judge prompt and scales should evolve as models change.

Target Audience

Researchers and engineers working on video anomaly understanding, video-language modeling, and multimodal evaluation. It is most useful for those who build or benchmark LVLMs on surveillance and anomaly tasks, for practitioners who need evaluation metrics that reflect factual correctness rather than fluency, and for anyone studying the design of LLM-as-judge protocols for domain-specific video tasks. The human correlation study and weight ablations also make it relevant to those specifically interested in metric validation.

Authors’ abstract

Video Anomaly Understanding (VAU) is a novel task focused on describing unusual occurrences in videos. Despite growing interest, the evaluation of VAU remains an open challenge. Existing benchmarks rely on n-gram-based metrics (e.g., BLEU, ROUGE-L) or LLM-based evaluation. The first fails to capture the rich, free-form, and visually grounded nature of LVLM responses, while the latter focuses on assessing language quality over factual relevance, often resulting in subjective judgments that are misaligned with human perception. In this work, we address this issue by proposing FineVAU, a new benchmark for VAU that shifts the focus towards rich, fine-grained and domain-specific understanding of anomalous videos. We formulate VAU as a three-fold problem, with the goal of comprehensively understanding key descriptive elements of anomalies in video: events (What), participating entities (Who) and location (Where). Our benchmark introduces a) FVScore, a novel, human-aligned evaluation metric that assesses the presence of critical visual elements in LVLM answers, providing interpretable, fine-grained feedback; and b) FineW3, a novel, comprehensive dataset curated through a structured and fully automatic procedure that augments existing human annotations with high quality, fine-grained visual information. Human evaluation reveals that our proposed metric has a superior alignment with human perception of anomalies in comparison to current approaches. Detailed experiments on FineVAU unveil critical limitations in LVLM's ability to perceive anomalous events that require spatial and fine-grained temporal understanding, despite strong performance on coarse grain, static information, and events with strong visual cues.