FrameShield: Adversarially Robust Video Anomaly Detection

AI summary

Overview

• Research area: Machine learning security and video analytics — specifically adversarial robustness for Weakly Supervised Video Anomaly Detection (WSVAD).
• Technical level: Advanced. The paper combines a theoretical gradient analysis of Multiple Instance Learning (MIL) aggregators, a pseudo-anomaly generation module, and min–max adversarial training.
• Scope: This paper proposes FrameShield, an end-to-end adversarial training pipeline that makes weakly supervised video anomaly detectors resistant to adversarial attacks by combining chunk-level pseudo-labels from real anomalous videos with precisely labeled synthetic anomalies.

What This Paper Is About

Video anomaly detectors are trained with only video-level labels ("this video contains an anomaly") but must make frame-level decisions at inference time, and the paper shows they collapse under small adversarial perturbations — for example, RTFM drops from 98.6 to 2.4 percent AUROC on UCSD Ped2, and TEVAD from 84.9 to 0.0 percent on UCF Crime. Standard adversarial training cannot fix this, because in a MIL setting the gradient only flows through the single highest-scoring frame, so training-time attacks are far weaker than the attacks an adversary can actually mount at inference. FrameShield's goal is to enable strong, frame-level adversarial training in a weakly supervised setting without frame-level ground truth.

Key Contributions

1. FrameShield pipeline: The authors describe FrameShield as the first adversarial training pipeline designed specifically to improve the robustness of WSVAD models, operating in two stages — standard "PromptMIL" training followed by chunk-wise adversarial training.
2. Spatiotemporal Region Distortion (SRD): A pseudo-anomaly generation method that applies severe augmentations to Grad-CAM-identified foreground regions of normal videos and moves the corrupted region along a randomly curved trajectory across consecutive frames, producing synthetic anomalies with exact frame-level annotations and no extra supervision.
3. Theoretical justification: An analysis showing that using max as the MIL aggregator means gradients from adversarial attacks flow through only a single frame, leaving other high-scoring frames unprotected; the authors argue for attacking every frame instead, via a loss they link to the "boundary error" of TRADES (Zhang et al., 2019a).
4. Robustness evaluation against strong attacks: Evaluation against PGD-1000, AutoAttack, and A³ across MSAD, UCF-Crime, ShanghaiTech, TAD, and UCSD Ped2, reporting an average 71.0 percent improvement in overall AUROC against state-of-the-art methods (the contributions section separately states a near 53 percent improvement in overall AUROC for robust performance and 68.5 percent on anomaly segments).

Main Findings

• Existing detectors collapse under attack: Under PGD-1000 with ε = 0.5/255 on the full test set, RTFM falls from 98.6 to 2.4 percent AUROC on UCSD Ped2, TEVAD falls from 84.9 to 0.0 percent on UCF Crime, and MGFN falls from 96.3 to 5.0 percent on UCSD Ped2.
• FrameShield retains most of its performance under attack: On the full test set, FrameShield reports 97.1/81.3 percent (clean/PGD) on UCSD Ped2, 89.5/87.1 on Shanghai, 85.1/77.2 on TAD, 80.2/78.7 on UCF Crime, and 78.9/76.2 on MSAD.
• On anomaly sections, robustness improves sharply: For AUC_A, FrameShield reports 94.3/91.3 percent (clean/PGD) on UCSD Ped2, 62.3/61.9 on Shanghai, 50.9/30.0 on TAD, 60.1/53.4 on UCF Crime, and 64.4/60.2 on MSAD, versus baselines that typically fall to single digits under attack.
• Clean performance is sometimes lower than baselines: FrameShield's clean AUC_O of 89.5 percent on Shanghai is below RTFM's 97.21, and its clean AUC_A values on TAD (50.9) and Shanghai (62.3) are below several baselines, which the authors frame as an acceptable trade-off for large robustness gains.
• Both supervision sources are needed: The ablation over pseudo-anomaly, pseudo-label, and combined supervision shows the combination achieves the best adversarial results: on UCF Crime, AUC_O goes from 68.2 percent (pseudo-anomaly only) and 71.3 percent (pseudo-label only) to 78.7 percent (combined), and AUC_A from 39.4 and 15.4 to 53.4.
• Pseudo-labels alone are fragile on anomalies: With pseudo-label supervision only, AUC_A under PGD drops to 7.1 percent on TAD, 15.4 on UCF Crime, and 21.7 on MSAD — the failure mode SRD is designed to fix.
• Frame-level attacks are far stronger than video-level attacks: When attacking the video-level scores of a cleanly trained model, the strongest aggregators still cannot reach zero AUC (Max: 45.4/51.2/48.6; Log-Sum-Exp: 43.8/39.3/43.0; SmoothMax: 49.7/48.7/47.1; ABMIL: 43.6/45.2/44.5 on TAD/UCF Crime/MSAD), whereas a frame-level attack drives performance to 0.4, 0.0, and 0.6 percent.
• Robustness transfers to stronger attacks: Against AutoAttack and A³ at ε = 0.5/255, FrameShield reports AUC_O/AUC_A of 73.1/27.3 and 73.6/27.2 on TAD, 72.1/50.3 and 71.5/48.7 on UCF Crime, and 74.7/58.2 and 73.6/56.9 on MSAD.
• Large ε destabilizes training: On Shanghai, training with ε = 2.0/255 or 1.0/255 dropped clean detection performance to near-random levels, while ε = 0.5/255 maintained stable training and robust performance.

Methodology in Plain English

FrameShield trains in two stages.

Stage one — PromptMIL. Each video is split into chunks (chunk size 16 frames). An X-Clip feature extractor encodes each chunk, and two frozen text prompts, "Normal" and "Abnormal", are encoded as text features. The dot product between a chunk's visual feature and each text feature, passed through a softmax, gives the chunk's normality and abnormality scores. These scores are aggregated with a max operation and a single binary cross-entropy loss is applied to the maximum, which is how the model learns from video-level labels alone. Training uses a learning rate of 8×10⁻⁶ for 40 epochs with the AdamW optimizer and a cosine learning-rate schedule.

Pseudo-labeling. The trained PromptMIL model labels each chunk of the real anomalous training videos. Normal videos do not need this step, since all their frames are already known to be normal. Chunks scoring above τ = 0.5 are treated as abnormal at inference.

Stage two — SRD and adversarial training. Because the pseudo-labels are noisy, the authors generate additional anomalies with SRD: a normal video segment is selected, Grad-CAM with a pre-trained ResNet18 highlights the salient foreground, the largest connected component is turned into a masked region, and several harsh augmentations are applied only to that region. To add a temporal dimension, the corrupted region is duplicated and displaced along a randomly curved vector across consecutive frames, with each step moving a distance equal to the vector length divided by the number of frames. The result is a video with exact frame-level anomaly labels.

With chunk-level labels available for normal videos, real anomalous chunks (pseudo-labeled), and SRD-generated anomalies, training switches from MIL to a fully supervised chunk-wise binary cross-entropy summed over all chunks. Adversarial examples are crafted with PGD-10 under an l∞ constraint of ε = 0.5/255 to maximize this chunk-wise loss, and the model is optimized with a min–max objective over batches.

Why This Matters

• Impact on research: The paper identifies a structural flaw in using MIL aggregation for adversarial training in video anomaly detection — that robustness learned on the maximum-scoring frame does not transfer to other high-scoring frames — and shows a way to get frame-level supervision without frame-level annotations. It also introduces a task-specific attack that pushes anomaly scores up on normal frames and down on abnormal frames rather than optimizing a single global loss.
• Real-world applications (as named in the paper):
  • Public safety and surveillance monitoring of rare, hazardous events.
  • Healthcare monitoring settings.
  • Industrial equipment monitoring and malfunction detection.
  • Any deployment where an attacker who can perturb camera or video input could suppress alerts or trigger false ones.
• Industry relevance: Robustness is a prerequisite for deploying anomaly detectors in high-reliability settings, and this paper targets exactly that gap for the weakly supervised models that are practical to train, since frame-level labels are expensive to collect. The code is released at https://github.com/rohban-lab/FrameShield.

Future Directions

• Closing the remaining clean-performance gap: FrameShield's clean AUC_A on TAD (50.9) and Shanghai (62.3) is close to or below several baselines, so improving standard accuracy alongside robustness remains open.
• Extending beyond the tested threat model: The paper evaluates white-box PGD-1000, AutoAttack, and A³ at ε = 0.5/255 and states that black-box attacks (Appendix L) and l₂-norm evaluations (Appendix G) are covered elsewhere; broader attack families and deployment-relevant perturbations are natural next steps.
• Alternative pseudo-label generators: The paper explicitly studies using other WSVAD methods as pseudo-label generators (Appendix C) and compares against adversarially trained baseline detectors (Appendix N), suggesting further work on which generator gives the best robustness-to-noise trade-off.
• Scaling and hyperparameter sensitivity: Only 50 percent of UCF Crime was used because of computational constraints, and ε values above 0.5/255 destabilized training on Shanghai, so full-dataset training and more principled schedules for the perturbation budget (the gradual-increase strategy is discussed in Appendix E) remain open questions.

Target Audience

Researchers and practitioners working on video anomaly detection, weakly supervised learning, and adversarial machine learning, particularly those building surveillance, safety, or industrial monitoring systems; the two-stage pipeline and released code also make it useful for engineers who need to harden an existing MIL-based detector without collecting frame-level labels.

Authors’ abstract

Weakly Supervised Video Anomaly Detection (WSVAD) has achieved notable advancements, yet existing models remain vulnerable to adversarial attacks, limiting their reliability. Due to the inherent constraints of weak supervision, where only video-level labels are provided despite the need for frame-level predictions, traditional adversarial defense mechanisms, such as adversarial training, are not effective since video-level adversarial perturbations are typically weak and inadequate. To address this limitation, pseudo-labels generated directly from the model can enable frame-level adversarial training; however, these pseudo-labels are inherently noisy, significantly degrading performance. We therefore introduce a novel Pseudo-Anomaly Generation method called Spatiotemporal Region Distortion (SRD), which creates synthetic anomalies by applying severe augmentations to localized regions in normal videos while preserving temporal consistency. Integrating these precisely annotated synthetic anomalies with the noisy pseudo-labels substantially reduces label noise, enabling effective adversarial training. Extensive experiments demonstrate that our method significantly enhances the robustness of WSVAD models against adversarial attacks, outperforming state-of-the-art methods by an average of 71.0\% in overall AUROC performance across multiple benchmarks. The implementation and code are publicly available at https://github.com/rohban-lab/FrameShield.