Text-guided Fine-Grained Video Anomaly Understanding

AI summary

Overview

• Research area: Computer vision — video anomaly detection (VAD) and video anomaly understanding (VAU), combining large vision-language models (LVLMs) with pixel-level anomaly localization.
• Technical level: Advanced. The paper assumes familiarity with transformer-based vision encoders, vision-language pretraining, multiple-instance learning for anomaly detection, and evaluation protocols such as RBDC/TBDC and BLEU-4.
• Scope: The paper introduces T-VAU, a framework that couples a text-aligned Anomaly Heatmap Decoder with a Region-aware Anomaly Encoder so that a frozen LVLM can judge, localize, and verbally explain subtle video anomalies, together with a new fine-grained video–text anomaly dataset built from ShanghaiTech and UBnormal.

What This Paper Is About

Conventional video anomaly detection systems output coarse frame- or clip-level anomaly scores, and they leave the user without evidence about where the anomaly is, which target caused it, or how it unfolds. General-purpose large vision-language models can produce textual judgments, but they tend to be weak at grounding subtle, low-signal anomalies in pixel-level evidence, which leads to unreliable localization and potentially unfaithful descriptions. The goal of this work is to close that loop: extract spatio-temporal anomaly evidence through visual–text alignment, then inject that evidence into an LVLM as structured prompts so that detection, localization, target identification, and multi-turn explanation all happen in one pipeline.

Key Contributions

1. The T-VAU framework. A fine-grained video anomaly understanding framework built on a frozen LVLM backbone, integrating an Anomaly Heatmap Decoder (AHD) for pixel-level anomaly localization and a Region-aware Anomaly Encoder (RAE) for injecting region-aware and motion-aware evidence into language reasoning. This yields a unified solution for anomaly detection, localization, and multi-turn explanation.

2. A new fine-grained anomaly understanding dataset. Constructed from two large-scale benchmarks, ShanghaiTech and UBnormal, it supports both target localization and language-based anomaly understanding. It provides frame-wise annotations for ShanghaiTech (4,108 training / 1,028 validation) and target-aligned descriptions for ShanghaiTech (5,136 samples) and UBnormal (7,912 samples), covering abnormal targets, appearance attributes, and motion trajectories.

3. A structured dataset construction pipeline. Three stages — frame-level structured prompting with temporal aggregation into target timelines, anomaly-focused refinement using anomaly masks and background suppression (Gaussian blurring), and bidirectional cross-modal consistency verification between appearance and motion cues — produce annotations aligned with appearance, spatial localization, and motion trajectory.

4. State-of-the-art results across judgment, localization, and dialog-based explanation benchmarks. The paper reports improvements on anomaly judgment, localization, and language-based explanation relative to prior methods and ablated variants.

Main Findings

• One-shot anomaly localization on UBnormal. AHD reaches 94.5% micro-AUC and 85.2% macro-AUC, with 64.3% RBDC and 74.4% TBDC from a single exemplar.
• Fine-tuned anomaly localization on UBnormal. After fine-tuning, AHD improves to 94.8% micro-AUC and 87.8% macro-AUC, with 67.8% RBDC and 76.7% TBDC, which the authors describe as establishing SOTA performance on localization-oriented metrics.
• Gains over the strongest baseline. Against Georgescu et al. (FT), the fine-tuned model gains +26.6 points in micro-AUC (94.8 vs. 68.2), +39.1 in RBDC (67.8 vs. 28.7), and +18.6 in TBDC (76.7 vs. 58.1), while remaining competitive in macro-AUC (87.8 vs. 95.3). The paper notes the baseline's macro-AUC advantage relies on a multi-stream architecture, whereas the proposed single model performs pixel-level localization while also supporting multi-turn dialogue and explainable reasoning.
• Gains over video transformers. Compared with Bertasius et al. (FT, SR=1/32: 86.1 micro-AUC, 89.2 macro-AUC, 0.008 RBDC, 0.021 TBDC), the method yields substantially higher RBDC/TBDC together with stronger micro-AUC.
• Textual faithfulness and discriminative power on ShanghaiTech. RAE reaches BLEU-4 of 62.67 (Target) and 88.84 (Trajectory), with 97.67% Yes/No accuracy — improving by about 7 and 6 points over the strongest baseline (InternVL-8B at 55.73 / 82.65).
• Results on UBnormal. RAE achieves 50.32 / 78.10 BLEU-4 for Target / Trajectory and 89.73% Yes/No accuracy, outperforming 7B- and 8B-scale baselines under one-shot prompting.
• Zero-shot versus one-shot gap for general LVLMs. Qwen2.5-VL (7B) zero-shot scores 18.74 / 27.33 BLEU-4 and 61.03% Yes/No on ShanghaiTech, versus 50.42 / 78.91 and 92.36% one-shot; on UBnormal it scores 16.20 / 24.18 and 65.62% zero-shot versus 44.35 / 70.82 and 87.24% one-shot.
• Ablation confirms module complementarity. Full T-VAU (8,324.67M parameters) achieves RBDC 67.8, TBDC 76.7, BLEU-4 62.67 / 88.84, and 97.67% Yes/No accuracy. Removing AHD while keeping RAE (8,299.71M parameters) makes RBDC/TBDC inapplicable and yields BLEU-4 61.82 / 85.47 with 95.38% accuracy. Removing RAE while keeping AHD (8,317.13M parameters) keeps RBDC/TBDC at 67.8 / 76.7 but degrades BLEU-4 and anomaly judgment accuracy. Removing both (8,274.74M parameters) reproduces the same 61.82 / 85.47 BLEU-4 and 95.38% accuracy as the no-AHD variant.
• Qualitative behavior. AHD concentrates activation on human silhouettes with sharp boundaries in pedestrian scenes and produces elongated responses along the motion direction in traffic scenes, while decoder-driven heatmaps blur across large regions or drift toward irrelevant textures. When the anomalous target is partially occluded, AHD responses attenuate but remain anchored and re-amplify upon reappearance.
• Reported failure cases. Micro-actions with minimal displacement, highly nonrigid motion that scatters activation, and scene-dependent appearance shifts such as specularities and fog.

Methodology in Plain English

The system starts from a frozen large vision-language model — a vision encoder, a text encoder, and a language model decoder — and adds only two small trainable modules.

The first, the Anomaly Heatmap Decoder (AHD), takes intermediate visual features from several transformer blocks of the vision encoder (layers 1, 8, 16, and 32) and compares them, via cosine similarity, against sentence-level text embeddings built from simple template prompts for the two categories normal and abnormal. Similarity maps from different layers are combined with learnable weights, passed through a softmax over the category dimension, and the "abnormal" channel becomes the anomaly heatmap. Because it works by comparison rather than by a hand-tuned cut-off, the authors describe the heatmaps as threshold-free.

The second, the Region-aware Anomaly Encoder (RAE), turns those heatmaps into language-ready prompts. It first computes temporal differences between adjacent heatmaps to capture motion, runs the result through a small convolutional backbone, then pools it two ways: a 3×3 grid yields nine per-region prompts, and a spatial mean pooling yields one global prompt. These are concatenated with a shared learnable base prompt, the visual prompt, and the embedded dialogue history, and fed to the language decoder. The result is a single prompt sequence in which anomaly evidence, visual content, and conversation context all sit side by side, so that the model's textual answer is anchored to the heatmap that produced it.

To train this, the authors built supervision data rather than relying on existing labels. Starting from datasets that only had pixel-level anomaly labels (ShanghaiTech and UBnormal), they ran each frame through a structured prompt to extract target attributes and bounding boxes, linked those outputs across time into target-level timelines, refined the frames using anomaly masks and background suppression to sharpen the anomalous evidence, and finally cross-checked appearance-derived trajectories against motion-derived descriptions in both directions.

Evaluation uses frame-level micro- and macro-AUC plus Region-Based (RBDC) and Track-Based (TBDC) detection criteria for localization; BLEU-4 (reported separately for Target and Trajectory, using the best-matching reference per video), binary accuracy, balanced accuracy (bAcc = ½(TPR + TNR)), and threshold-free frame-level micro/macro ROC–AUC for the language side. All reported experiments use a one-shot setting with minimal text normalization.

Why This Matters

Impact on research. The paper argues that score-based VAD answers "is there an anomaly?" but not "where, which target, and how," and that general LVLMs answer the second question without grounding it in pixels. By making heatmaps the shared substrate for both localization and language, T-VAU provides a spatial witness for every claim the model makes, which the authors frame as enabling error analysis through spatial and temporal evidence rather than through text alone. It also contributes a labelled resource that gives the field target-level supervision on top of two widely used benchmarks, which is a prerequisite for training evidence-grounded explanation rather than post-hoc narration.

Real-world applications:

• Public security monitoring, where operators need to know which individual or vehicle triggered an alert rather than just receiving a score.
• Industrial inspection, where a subtle deviation in appearance or motion must be pointed out at the pixel level for an operator to act.
• Traffic safety analysis, for example abrupt vehicle entry, near-collisions, and jaywalking, where the trajectory of the responsible target matters as much as the alert itself.
• Interactive video review and audit, where an analyst asks follow-up questions ("which target?", "where?", "when?") and the system reuses the same evidence chain across turns.

Industry relevance. The framework keeps the LVLM backbone frozen and trains only two lightweight modules, which the paper highlights through parameter counts in the ablation (8,274.74M for the backbone-only variant versus 8,324.67M for the full model). That design keeps deployment centered on an existing general-purpose model while adding domain-specific anomaly grounding on top. The commitment to release code at https://github.com/momiji-bit/T-VAU also lowers the barrier to reproducing and adapting the pipeline.

Future Directions

• Handling hard failure cases. The paper explicitly identifies micro-actions with minimal displacement, highly nonrigid motion that scatters activation, and scene-dependent appearance shifts such as specularities and fog as cases where the method struggles. Improving robustness there is a clear next step.
• Generalization to unseen anomaly categories and cross-scene changes. The authors report that AHD preserves localization fidelity when target scale is reasonable and that T-VAU often highlights the correct target for unseen categories sharing motion primitives, but this is characterized qualitatively rather than as a quantified benchmark, leaving open how far the approach generalizes.
• Closing the macro-AUC gap. The fine-tuned model trails the strongest baseline on macro-AUC (87.8 vs. 95.3), a gap the paper attributes to that baseline's multi-stream architecture; recovering macro-level performance without giving up single-model localization and dialogue is an open problem.
• Extending the supervision pipeline beyond two benchmarks. The constructed dataset is derived from ShanghaiTech and UBnormal only; whether the three-stage construction process transfers to other surveillance and open-set benchmarks is not reported.

Target Audience

Researchers and practitioners working on video anomaly detection and understanding, vision-language model grounding, and interpretable multimodal reasoning will get the most from this paper. It is also relevant to engineers building surveillance, industrial inspection, or safety-critical monitoring systems who need anomaly localization with human-readable justification rather than a bare score, and to dataset builders interested in the structured video–text annotation pipeline. Readers without a background in transformer architectures, vision-language pretraining, or detection evaluation metrics such as RBDC/TBDC and BLEU-4 will find the methodology sections demanding.

Authors’ abstract

Subtle abnormal events in videos often manifest as weak spatio-temporal cues that are easily overlooked by conventional anomaly detection systems. Existing video anomaly detection approaches typically provide coarse binary anomaly decisions without interpretable evidence, while large vision-language models (LVLMs) can produce textual judgments but lack precise localization of subtle visual signals. To address this gap, we propose Text-guided Fine-Grained Video Anomaly Understanding T-VAU, a framework that grounds subtle anomaly evidence into multimodal reasoning. Specifically, we introduce an Anomaly Heatmap Decoder (AHD) that performs visual-textual feature alignment to extract pixel-level spatio-temporal anomaly heatmaps from intermediate visual representations. We further design a Region-aware Anomaly Encoder (RAE) that converts these heatmaps into structured prompt embeddings, enabling the LVLM to perform anomaly detection, localization, and semantic explanation in a unified reasoning pipeline. To support fine-grained supervision, we construct a target-level fine-grained video-text anomaly dataset derived from ShanghaiTech and UBnormal with detailed annotations of object appearance, localization, and motion trajectories. Extensive experiments demonstrate that T-VAU significantly improves anomaly localization and textual reasoning performance on both benchmarks, achieving strong results in BLEU-4 metrics and Yes/No decision accuracy while providing interpretable pixel-level spatio-temporal evidence for anomaly understanding. The code will be available at https://github.com/momiji-bit/T-VAU.