Evaluation of Vision-LLMs in Surveillance Video

AI summary

Overview

• Research area: Computer Vision / video understanding — zero-shot anomalous action recognition in surveillance footage using small open vision-language models.
• Technical level: Intermediate. Readers should be comfortable with vision-language models, zero-shot classification, few-shot prompting, and natural language inference, though the framework itself is conceptually simple.
• Scope: A systematic evaluation of four open vision-LLMs (Gemma-3 (4B), Qwen-2.5-VL-7B-Instruct, VideoLLaMA-3-7B, NVILA-8B) as training-free, language-grounded anomaly detectors on the UCF-Crime and RWF-2000 benchmarks, under varied prompting and privacy-preserving conditions (arXiv:2510.23190v1, 27 Oct 2025).

What This Paper Is About

Surveillance cameras generate far more video than human operators can monitor, yet timely detection of anomalous or criminal events is critical for public safety. Existing anomaly detectors require large, annotated datasets and generalize poorly to rare or unseen events. This paper asks whether small, pre-trained vision-LLMs can serve as spatially grounded, zero-shot anomaly detectors by describing video in natural language and then scoring candidate labels through textual entailment — with no task-specific training.

Key Contributions

1. A training-free, language-grounded anomaly classification framework. Video is converted into a text description by a frozen vision-LLM, and a separate frozen NLI classifier scores how strongly that description entails each candidate anomaly label; the highest-scoring label wins.
2. A controlled evaluation across prompting regimes. Four open vision-LLMs are evaluated on UCF-Crime under an unguided prompt, a guided prompt (with a fixed list of 14 classes), and a guided prompt augmented with few-shot exemplars, using class-averaged Top-1 accuracy (Top-1^macro) as the primary metric.
3. A privacy-robustness study. RWF-2000 is re-generated under three privacy filters — local head/face Gaussian blur, DeepPrivacy2 GAN face-level anonymization, and the same GAN extended to full-body masks — to measure how identity obfuscation affects anomaly recognition.
4. Fault analysis and a public implementation. The paper reports false-positive rates, wrong-label rates, and batch-level AUC, and releases the evaluation code at https://github.com/pascalbenschopTU/VLLM_AnomalyRecognition.

Main Findings

• Guided prompts raise UCF-Crime accuracy but also wrong-label rates. Top-1 rises from unguided to guided for every model: Gemma3-4B 26.29% → 33.85%, NVILA-8B 13.39% → 27.00%, Qwen2.5 25.31% → 34.69%, VideoLLama3 19.94% → 34.16%. Wrong-label rates jump in the same direction (e.g., Qwen2.5 from 2.21% to 76.56%).
• Few-shot effects are model-dependent and not uniformly positive. With guided prompt plus few-shot examples, NVILA-8B improves substantially to 45.05% Top-1 (from 27.00%), while Gemma3-4B drops to 29.80% (from 33.85%) and VideoLLama3 drops to 31.44% (from 34.16%). Qwen2.5 rises to 38.87%. False positives increase sharply for several models (Gemma3-4B 21.67% → 68.67%; NVILA-8B 5.00% → 18.00%), while VideoLLama3's fall from 19.67% to 5.00%.
• The paper's own summary of few-shot prompting: few-shot examples improve accuracy on average but tend to increase the false-positive rate, with Gemma-3 and NVILA benefiting the most. Figure 1 reports Top-1 accuracies for only the classes included in the few-shot prompts (Fighting, RoadAccidents, Shooting and Stealing).
• Privacy filters degrade RWF-2000 accuracy by roughly 2–11 percentage points. Baselines with no filter are Gemma-3 (4B) 86.25% / 20.50% FP, NVILA-8B 82.50% / 14.00% FP, Qwen-2.5-VL-7B-Instruct 82.25% / 24.50% FP, and VideoLLaMA-3-7B 83.25% / 8.50% FP.
• Full-body GAN anonymization is the most disruptive filter. Accuracy deltas under GAN full body are –4.0 pp (Gemma-3), –11.3 pp (NVILA-8B), –6.5 pp (Qwen-2.5), and –8.8 pp (VideoLLaMA-3). Under blur face, deltas are –5.0, –1.8, –4.8, and –2.5 pp respectively; under GAN face, –2.8, –1.8, –1.0, and –4.5 pp.
• False positives often rise under privacy filters. Notable increases include Gemma-3 with blur (+10.5 pp FP), Qwen-2.5 with GAN full body (+11.0 pp FP), and Qwen-2.5 with blur (+9.0 pp FP). VideoLLaMA-3-7B is the exception, showing FP reductions under GAN filters (–5.5 pp for GAN face, –6.5 pp for GAN full body).
• VideoLLaMA-3 shows an outlier false-positive pattern on UCF-Crime's unguided setting: 80.67% FP with 19.94% Top-1, versus 4.17% wrong-label rate.
• Privacy filters are inconsistent frame-to-frame. With GAN-based anonymization, the same person is generated slightly differently across frames, producing inconsistent motion (Figure 3), which the authors identify as a cause of the accuracy drop.
• Models are stronger on simpler, spatially salient tasks. RWF-2000 accuracy (73.25%–86.25% across all conditions) is far higher than UCF-Crime accuracy (13.39%–45.05% across all conditions); the conclusion states these models show promise for tasks like fight detection but are not yet reliable for complex, autonomous surveillance.

Methodology in Plain English

The approach reframes anomaly classification as a reasoning task rather than a pixel-to-label mapping. A video clip is fed to a frozen vision-LLM together with a text prompt, and the model generates a short natural-language description (capped at ≤40 words in the prompts). That description is then compared against each candidate human-readable anomaly label by a frozen NLI classifier — the paper uses facebook/bart-large-mnli — which scores how strongly the description entails each label. The label with the highest entailment score becomes the prediction. No parameters are updated at any stage.

Three prompting regimes are compared: an unguided prompt that simply asks the model to identify the primary action; a guided prompt that supplies the full label list with short definitions (14 classes for UCF-Crime, 2 for RWF-2000) and requests a [Predicted Class]: description format; and a guided prompt preceded by few-shot image-and-text exemplars drawn from the official training split to avoid test leakage.

Evaluation uses class-averaged Top-1 accuracy, where a video counts as correct if the ground-truth label is the Top-1 class in at least one temporal window. Reported secondary metrics are batch-level AUC (256-frame batches, with every non-"Normal" class treated as anomaly), false-positive rate (percentage of batches predicted as something other than "Normal" in videos labeled "Normal"), and wrong-label rate (a generated label that does not correspond to the video label).

Experiments run on NVIDIA A40 or L40 GPUs with up to 46 GB VRAM, using conservative decoding (temperature 0.05–0.1, 64–128 new tokens, repetition penalty 1.5). Each experiment is run once in a single pass due to computational cost. For the privacy study, a separate dataset is pre-generated for each filter so multiple models can be tested without reapplying the transform, and all privacy evaluations use the guided prompt with sampling, aggregation, and scoring mirroring the prompting study.

Why This Matters

Impact on research. The paper tests whether the transferability that vision-LLMs show on standard action recognition extends to anomalous action recognition — a domain of rare, atypical, or criminal events that are underrepresented or absent from standard training data. It documents where zero-shot, language-grounded pipelines succeed (simple, spatially salient events) and where they falter (noisy spatial cues, identity obfuscation), and it treats privacy preprocessing as a variable that affects recognition, rather than as a neutral step.

Real-world applications:

• Surveillance triage — flagging candidate events across camera networks where the number of cameras far exceeds the number of operators.
• Forensic analysis — language-grounded descriptions of footage that can be reviewed and audited as text.
• Privacy-compliant monitoring — anomaly detection on blurred or GAN-anonymized footage, with quantified accuracy and false-positive costs.
• Public-safety deployment planning — evidence that adding labels to the candidate set requires no retraining, which matters when anomaly categories evolve.

Industry relevance. The modular design means the vision-LLM and NLI classifier can be replaced independently, so a stronger backbone can be swapped in without touching the classification module. The release of the evaluation implementation lowers the barrier for vendors and researchers to reproduce or extend these comparisons.

Future Directions

• Structure-aware prompts and spatial grounding — designing prompts that better exploit spatial relations, without any task-specific training.
• Lightweight spatial memory across clips — carrying information between temporal windows to improve temporal consistency of predictions.
• Scene-graph or 3D-pose priors during description — injecting geometric structure at the description stage to strengthen spatial grounding.
• Privacy methods that preserve action-relevant geometry — improving temporal consistency of anonymization, since full-body GAN transforms introduce frame-to-frame inconsistencies that degrade accuracy.

Open questions the paper leaves unresolved include how these models would perform on datasets beyond UCF-Crime and RWF-2000, how much the single-pass evaluation protocol affects the reported numbers, and how zero-shot pipelines compare against the supervised and weakly supervised baselines discussed in the related work.

Target Audience

Researchers and practitioners working on video anomaly detection, surveillance analytics, and vision-language model evaluation. It is also useful for privacy engineers and public-sector technologists who need to understand the accuracy cost of anonymization, and for applied machine learning teams deciding whether a zero-shot, training-free pipeline is sufficient for a given deployment. The paper assumes familiarity with prompting and entailment-based zero-shot classification; the framework itself is accessible to readers without a deep learning background.

Authors’ abstract

The widespread use of cameras in our society has created an overwhelming amount of video data, far exceeding the capacity for human monitoring. This presents a critical challenge for public safety and security, as the timely detection of anomalous or criminal events is crucial for effective response and prevention. The ability for an embodied agent to recognize unexpected events is fundamentally tied to its capacity for spatial reasoning. This paper investigates the spatial reasoning of vision-language models (VLMs) by framing anomalous action recognition as a zero-shot, language-grounded task, addressing the embodied perception challenge of interpreting dynamic 3D scenes from sparse 2D video. Specifically, we investigate whether small, pre-trained vision--LLMs can act as spatially-grounded, zero-shot anomaly detectors by converting video into text descriptions and scoring labels via textual entailment. We evaluate four open models on UCF-Crime and RWF-2000 under prompting and privacy-preserving conditions. Few-shot exemplars can improve accuracy for some models, but may increase false positives, and privacy filters -- especially full-body GAN transforms -- introduce inconsistencies that degrade accuracy. These results chart where current vision--LLMs succeed (simple, spatially salient events) and where they falter (noisy spatial cues, identity obfuscation). Looking forward, we outline concrete paths to strengthen spatial grounding without task-specific training: structure-aware prompts, lightweight spatial memory across clips, scene-graph or 3D-pose priors during description, and privacy methods that preserve action-relevant geometry. This positions zero-shot, language-grounded pipelines as adaptable building blocks for embodied, real-world video understanding. Our implementation for evaluating VLMs is publicly available at: https://github.com/pascalbenschopTU/VLLM_AnomalyRecognition