Learning to Tell Apart: Weakly Supervised Video Anomaly Detection via Disentangled Semantic Alignment

AI summary

Overview

Research area: Computer Vision — Weakly Supervised Video Anomaly Detection (WS-VAD), with cross-modal vision-language learning (CLIP).

Technical level: Advanced. The paper assumes familiarity with multiple instance learning, transformer architectures, contrastive learning, and CLIP-style vision-language pretraining.

Scope: The paper presents DSANet, a three-branch framework that improves both coarse-grained (anomaly localization) and fine-grained (anomaly classification) video anomaly detection by explicitly modeling normal patterns and preventing category confusion under weak supervision.

What This Paper Is About

Weakly supervised video anomaly detection uses only video-level labels (anomaly present or not) to locate and classify anomalous moments in long, untrimmed videos. Existing multiple-instance-learning approaches tend to fixate on the most visually salient segments, which blurs the boundary between normal and anomalous events and causes different anomaly categories to be confused in the feature space. This paper tackles both problems by explicitly modeling normality and by disentangling event content from background context so that anomaly classes become separable.

Key Contributions

1. Self-Guided Normality Modeling (SG-NM): A generative reconstruction branch that mines a small set of video-specific "Dynamic Normal Patterns" (DNPs) directly from each input video — no external memory bank or dataset-level priors required — and uses them to reconstruct features. Reconstruction error then serves as an anomaly cue that complements the discriminative MIL branch.

2. Decoupled Contrastive Semantic Alignment (DCSA): A mechanism that splits each video into an event-centric prototype and a background-centric prototype using the detection branch's frame-level scores, then contrastively aligns the event prototype with its true anomaly class and the background prototype with a universal "normal" class.

3. Lightweight Text Adapters: Small parallel adapter modules inserted into the early CLIP text-transformer layers, allowing domain-specific adaptation of text embeddings while preserving the pretrained CLIP knowledge.

4. Hierarchical Belief Modulation for inference: A strategy that distributes the temporal prior from the detection branch across fine-grained class predictions, anchoring classification to robust temporal boundaries.

Main Findings

• State-of-the-art coarse-grained detection: DSANet reaches 86.95% AP on XD-Violence (beating ReFLIP's 85.81% and ITC's 85.45%) and 89.44% AUC on UCF-Crime (beating ITC's 89.04% and ReFLIP's 88.57%), using the same frozen CLIP (ViT-B/16) backbone as competitors.

• Large margins in fine-grained detection: On UCF-Crime, DSANet achieves 13.01% average mAP versus 9.62% (ReFLIP) and 7.90% (ITC) — a substantial gain over prior art. On XD-Violence it reaches 28.87% average mAP versus 27.36% (ReFLIP).

• DNPs cleanly separate normal from abnormal: Normal test frames sit at low average cosine distance (0.35) to the learned normal patterns, while abnormal frames sit much farther (0.69), confirming that the DNP-defined space is a compact, discriminative representation of normality.

• Disentanglement dramatically improves alignment accuracy: DSANet's background-centric prototype aligns with the "normal" class 99.63% of the time, versus 87.63% for an adapted VadCLIP baseline. Its event-centric confusion matrix shows far stronger diagonal dominance, indicating reduced class confusion.

• Component synergy: Ablation on XD-Violence shows each module adds value (Adapter: 85.00% AP, SG-NM: 85.94%, DCSA: 85.67%), with the full model reaching 86.95% AP and 28.87% average mAP — a 2.44% AP and 4.17% average mAP gain over the VadCLIP baseline.

• Text adapters beat prompting: Adapter-based text-encoder tuning (86.95% AP) outperforms no tuning (81.57%), manual prompts (81.05%), and learnable prompts (82.88%).

• Better temporal boundaries qualitatively: Visual comparisons show VadCLIP produces fragmented, peak-focused anomaly predictions, whereas DSANet's outputs align more accurately with ground-truth anomaly intervals.

Methodology in Plain English

The framework has three cooperating branches built on a frozen CLIP encoder.

1. Anomaly Detection Branch. Frame features from CLIP are enriched with temporal context — first by a local transformer that models short-range motion within fixed time windows, then by a graph convolutional network that captures long-range relationships. A binary classifier converts these into per-frame anomaly scores. Video-level predictions are made by averaging the top-k scores, following the standard multiple-instance-learning recipe.

2. Self-Guided Normality Modeling. The detection scores are reused to pick the frames that look least anomalous. A small cross-attention module with 16 learnable queries distills these into "Dynamic Normal Patterns." A compactness loss keeps these patterns tight representatives of normalcy. A multi-layer decoder then tries to reconstruct every frame using only the normal patterns as reference. Because anomalies cannot be well reconstructed from normal prototypes, high reconstruction error signals an anomaly. A consistency loss makes the discriminative score and the reconstruction-based score agree with each other, so the two views reinforce one another.

3. Anomaly Classification Branch with DCSA. The frame-level scores are converted into soft weights, and two prototypes are computed: an event-centric one (weighted toward anomalous frames) and a background-centric one (weighted toward the rest). Meanwhile, CLIP's text embeddings for each class — including "normal" — are refined by lightweight adapters. A separation loss pushes the "normal" text embedding away from all anomaly embeddings, and a dual contrastive loss pulls the event prototype toward its true class while pulling the background prototype toward the "normal" class. This decoupling makes anomalies harder to confuse with each other or with benign context.

At inference, the detection score supplies a temporal prior, which is distributed over class probabilities to produce fine-grained predictions.

Why This Matters

Impact on research. The paper reframes WS-VAD as a two-sided problem: you need both a good model of what is anomalous and a good model of what is normal. Most prior MIL-based pipelines implicitly assume normality will emerge as "not anomalous," which the authors show is insufficient. The SG-NM design — mining normal prototypes from within each video without a memory bank — is a data-efficient, self-contained alternative that could transfer to adjacent tasks like defect detection, medical screening, and industrial monitoring, where annotated anomalies are rare.

Real-world applications:

• Public safety surveillance: Automatically flagging fights, vandalism, or accidents in hours of CCTV footage using only video-level tags.
• Industrial quality control: Detecting defective products on a manufacturing line, where normal exemplars are abundant but defective ones are rare.
• Medical video monitoring: Flagging abnormal events (e.g., seizures or falls) in patient-monitoring footage.
• Autonomous driving / traffic analytics: Identifying unusual road events like accidents or erratic driving in dashcam or traffic-camera streams.

Industry relevance. The method trains in only 10 epochs on a single NVIDIA 4090 GPU and uses a frozen CLIP backbone, keeping compute modest for a modern deep-learning pipeline. The domain-adaptive text adapters also make it practical to retarget the model to new anomaly taxonomies with minimal fine-tuning cost.

Future Directions

• Extending beyond a single "normal" prototype: The DCSA design collapses all normal context into one background prototype. Fine-grained normality (e.g., different normal scenes or activities) might benefit from multiple normal clusters.

• Handling multi-anomaly videos: The current label scheme assumes a video contains one anomaly category. Real surveillance footage may contain several distinct anomalies, requiring multi-label disentanglement.

• Robustness of the DNP mining step: Normal patterns are extracted from frames with the lowest current scores, which is a chicken-and-egg dependency on the detection branch. A curriculum or adversarial refinement of this step could improve reliability when the initial detector is weak.

• Scaling to larger and more diverse datasets: Evaluation is restricted to two benchmarks (UCF-Crime and XD-Violence). Testing on newer, larger, or more multilingual/multicultural datasets — and combining with open-vocabulary backbones — would test generality.

• Temporal efficiency: The current pipeline runs full temporal modeling plus an 8-layer decoder. Distilling these into a lighter online detector would help for real-time deployment.

Target Audience

This paper is best suited for graduate students and researchers working on video anomaly detection, weakly supervised learning, or vision-language models. It will also interest computer-vision engineers building surveillance, safety, or fault-detection systems, and anyone studying how to leverage CLIP-style pretrained models for temporal or event-level reasoning with limited labels. Readers should be comfortable with multiple instance learning, contrastive objectives, and transformer-based architectures.

Authors’ abstract

Recent advancements in weakly-supervised video anomaly detection have achieved remarkable performance by applying the multiple instance learning paradigm based on multimodal foundation models such as CLIP to highlight anomalous instances and classify categories. However, their objectives may tend to detect the most salient response segments, while neglecting to mine diverse normal patterns separated from anomalies, and are prone to category confusion due to similar appearance, leading to unsatisfactory fine-grained classification results. Therefore, we propose a novel Disentangled Semantic Alignment Network (DSANet) to explicitly separate abnormal and normal features from coarse-grained and fine-grained aspects, enhancing the distinguishability. Specifically, at the coarse-grained level, we introduce a self-guided normality modeling branch that reconstructs input video features under the guidance of learned normal prototypes, encouraging the model to exploit normality cues inherent in the video, thereby improving the temporal separation of normal patterns and anomalous events. At the fine-grained level, we present a decoupled contrastive semantic alignment mechanism, which first temporally decomposes each video into event-centric and background-centric components using frame-level anomaly scores and then applies visual-language contrastive learning to enhance class-discriminative representations. Comprehensive experiments on two standard benchmarks, namely XD-Violence and UCF-Crime, demonstrate that DSANet outperforms existing state-of-the-art methods.