MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection

AI summary

Overview

• Research area: Computer vision, specifically online (streaming) video anomaly detection (VAD) using large language models (LLMs) and vision-language models (VLMs), with no training required.
• Technical level: Intermediate. The paper is readable without deep math, but it assumes familiarity with anomaly-detection benchmarks, AUC/AP metrics, and the basic roles of VLMs and LLMs.
• Scope: The paper introduces MoniTor, a training-free framework that produces frame-level anomaly scores from streaming video by feeding frame captions, memory summaries, an anomaly prior, and a scoring queue into an LLM, and evaluates it on UCF-Crime and XD-Violence.

What This Paper Is About

Most video anomaly detection research works offline: the model sees the whole video before scoring it. Real monitoring, however, requires judgments as footage arrives, and online VAD has received little attention because of real-time constraints and computational intensity, plus the fact that anomalies are infrequent, discontinuous, and hard to collect into training sets. MoniTor addresses this by combining a vision-language pipeline with an instructed LLM so that anomalous events in a stream can be flagged without any training, while still using the recent past to interpret the current frame.

Key Contributions

1. MoniTor, an online training-free VAD framework. It takes streaming video input and applies LLMs to real-time monitoring, generating anomaly scores at 0.6-second intervals while maintaining a 5-second end-to-end processing latency.
2. LSTM-inspired memory and prediction. Long Short-Term Memory (LSTM) mechanisms are integrated with the LLM to encode historical sequence information, improving online VAD and making the boundaries of anomalous events more precise.
3. A scoring queue plus an anomaly prior. A scoring queue mechanism is designed to reduce the LLM's dependence on explicit instructions, and an anomaly prior (definitions drawn from Wikipedia, with examples) is injected to help the LLM recognize anomalous events.
4. Extensive evaluation. Experiments show MoniTor outperforming unsupervised approaches and surpassing training-free offline methods on UCF-Crime and XD-Violence.

Main Findings

• UCF-Crime AUC: MoniTor reaches 82.57% AUC, higher than offline training-free LAVAD (80.28%) and EventVAD (82.03%), and higher than the online-LAVAD baseline (76.06%).
• Gains on UCF-Crime: The paper reports a 6.51% improvement in AUC over online LAVAD, improvements of 12.11% over offline one-class VAD and 2.81% over offline unsupervised VAD, and absolute gains of 2.29% and 0.54% in AUC when using the same ViT video features.
• XD-Violence results: MoniTor obtains 55.01% AP and 79.11% AUC, a gain of 2.38% in AP and 3.10% in AUC over online-LAVAD (52.63% AP, 76.01% AUC). Here it remains below offline training-free methods such as LAVAD (60.02% AP, 82.89% AUC) and EventVAD (64.04% AP, 87.51% AUC).
• Comparability to weakly supervised methods: MoniTor is described as competitive with online weakly supervised VAD on UCF-Crime without any training, while REWARD, a weakly supervised method, reaches a higher 86.94% AUC there.
• Efficiency: Compared with REWARD, MoniTor has a shorter decision period (0.6 s vs. 6.4 s) at a higher processing time (5.9 s vs. 0.5 s), for a lower total delay (6.5 s vs. 6.9 s). The paper states an anomaly score is available within 5 to 6 seconds per frame, faster than the general online VAD standard of 30 seconds.
• Ablation on components (UCF-Crime): Starting from a 76.06% AUC baseline, Anomaly Priors Integration adds 1.79%, Dynamic Memory Gating adds 2.82%, Standard Scoring Queue adds 2.59%, Behavior Prediction and Dynamic Analysis adds 2.24%, and Weight Assignment adds 0.96%, with all components together reaching 82.57%.
• Ablation on memory: Long-term memory alone yields 78.27% AUC, short-term memory alone 77.92%, long-term memory plus the forgetting gate 78.66%, and all three together 78.88%. The forgetting gate adds a further 0.39% AUC by filtering unimportant captions; short-term memory covers the two previous captions, about 1 second.
• Qualitative behavior: In anomalous videos the scores stay low during normal portions and rise sharply in abnormal segments; in normal videos the scores stay consistently low throughout, indicating no false alarms in the shown examples.
• Real-world tests: On randomly selected YouTube videos involving, for example, gun robberies and physical altercations, MoniTor reportedly identified anomalies across diverse settings.
• Failure mode: Roughly 60% of detection errors occur around abrupt camera transitions, with the paper attributing XD-Violence's smaller gains to its many camera transitions, which weaken the Dynamic Memory Gating Module and the Behavior Prediction and Dynamic Analysis module.

Methodology in Plain English

The pipeline processes a video frame by frame. Each frame is passed to an Online Vision-Language Model: five BLIP-2 models each produce a raw caption, and the captions from the current frame plus the previous five frames are ranked by cosine similarity (computed with ImageBind's image and text encoders) against the frame's image feature. The top 10 captions are kept and sent to GLM-4-Flash to produce a short summary of what is happening.

That summary is combined with several forms of context. An "anomaly prior" injects Wikipedia definitions of the anomaly categories that appear in the datasets (13 for UCF-Crime, 6 for XD-Violence) so the LLM knows what to look for. A Dynamic Memory Gating Module, inspired by LSTM architecture, keeps a long-term memory over a 10-frame window and a short-term memory over the two most recent frames; a forgetting gate compares the current frame's summary with earlier ones by cosine similarity and discards frames below a threshold (set to 0.5) so stale or irrelevant captions do not mislead the model. A Behavior Prediction and Dynamic Analysis module asks the LLM to predict what will happen next, creating a feedback loop between expectation and what actually occurs.

A Standard Scoring Queue holds the most recent caption for each score value in a range from 0 to 1, updated as each frame is scored, which gives the LLM concrete examples of how previous frames were rated. The LLM then produces a raw anomaly score for the current frame from the prompt, memory, queue, anomaly prior, and summary. Finally, a weight assignment step blends the current frame's score with the previous frame's score using a parameter alpha (set to 0.7), so scores change smoothly rather than jumping. The temperature of the LLM is set to 0.6, the number of parallel video jobs is 190, and the runs use two NVIDIA GeForce RTX 4090 GPUs.

Why This Matters

Research impact. The paper argues it is the first online training-free VAD framework to effectively leverage LLMs with robust temporal reasoning, showing that a memory-and-queue design can partially compensate for the weaknesses that made LLMs unreliable for streaming detection — sensitivity to prompt wording and weak temporal context. It also provides a baseline, online-LAVAD, for measuring progress in online training-free settings.

Real-world applications:

• Intelligent surveillance, where security incidents such as abuse, robbery, explosion, and fighting must be detected as they occur.
• Autonomous driving, listed among the domains where timely anomaly detection matters.
• Scenarios with data-collection or privacy constraints, where training on large labeled anomaly sets is impractical and a training-free solution is preferable.
• Real-time safety monitoring of public video sources, as demonstrated by the tests on YouTube videos of gun robberies and altercations.

Industry relevance. Because MoniTor needs no retraining and no data annotation, it can be deployed across domains and camera setups without the overhead of collecting domain-specific data. Its 0.6-second decision period is presented as suitable for real-time applications, though the LLM/VLM reliance requires substantial compute, which the authors flag as a barrier to edge deployment.

Future Directions

• Handling sudden camera transitions, which account for roughly 60% of detection errors, potentially through continual learning techniques such as Experience Replay.
• Compressing the LLM/VLM stack via quantization and pruning so that real-time processing fits on resource-constrained edge devices.
• Removing the instruction-dependency and cold-start weaknesses of LLM-based scoring; the appendix is said to contain prompt sensitivity analysis, initialization strategy studies for the cold-start problem, and video length performance analysis, though the truncated content does not report their results.
• Improving on XD-Violence specifically, where camera transitions reduce the benefit of the memory and prediction modules and where MoniTor trails offline training-free methods.

Target Audience

Researchers and graduate students working on video anomaly detection, video understanding, or surveillance analytics, particularly those interested in training-free and LLM-based pipelines. It also suits engineers evaluating whether LLM-driven, no-training detection is practical for real-time monitoring, and readers who want a concrete example of how memory modules and prompt scaffolding can make LLM outputs more temporally coherent.

Authors’ abstract

Video Anomaly Detection (VAD) aims to locate unusual activities or behaviors within videos. Recently, offline VAD has garnered substantial research attention, which has been invigorated by the progress in large language models (LLMs) and vision-language models (VLMs), offering the potential for a more nuanced understanding of anomalies. However, online VAD has seldom received attention due to real-time constraints and computational intensity. In this paper, we introduce a novel Memory-based online scoring queue scheme for Training-free VAD (MoniTor), to address the inherent complexities in online VAD. Specifically, MoniTor applies a streaming input to VLMs, leveraging the capabilities of pre-trained large-scale models. To capture temporal dependencies more effectively, we incorporate a novel prediction mechanism inspired by Long Short-Term Memory (LSTM) networks. This ensures the model can effectively model past states and leverage previous predictions to identify anomalous behaviors. Thereby, it better understands the current frame. Moreover, we design a scoring queue and an anomaly prior to dynamically store recent scores and cover all anomalies in the monitoring scenario, providing guidance for LLMs to distinguish between normal and abnormal behaviors over time. We evaluate MoniTor on two large datasets (i.e., UCF-Crime and XD-Violence) containing various surveillance and real-world scenarios. The results demonstrate that MoniTor outperforms state-of-the-art methods and is competitive with weakly supervised methods without training. Code is available at https://github.com/YsTvT/MoniTor.