Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks

AI summary

Overview

• Research area: Computer vision, specifically Video Anomaly Detection (VAD) and Video Anomaly Understanding (VAU), plus synthetic video benchmark construction using video generation models and Vision-Language Models (VLMs).
• Technical level: Advanced. The paper assumes familiarity with video anomaly benchmarks, weakly/semi-supervised VAD methods, video diffusion models, and multimodal large language models.
• Scope in one sentence: The paper introduces Pistachio, a benchmark built entirely through a generation pipeline — 4,962 VAD videos (1,676,822 frames, 3,896 scenes, 31 anomaly types) and 1,385 VAU videos (517,514 frames) with event-level and video-level text annotations produced with zero manual annotation.

What This Paper Is About

Existing VAD benchmarks are collected from the Internet or surveillance footage, which biases them toward urban road scenes, over-represents anomalies that circulate online (creating severe long-tail distributions), and fills "normal" segments with trivial activities like walking. They also lack structured temporal narratives, multi-step event progressions, and causal dependencies, so they cannot evaluate VAU, and building a VAU benchmark by hand is prohibitively expensive. Pistachio's goal is to replace collection-and-filter with a controlled generation pipeline that gives precise control over scenes, anomaly types, temporal narratives, and visual diversity, yielding balanced anomaly coverage and automatic semantic annotations.

Key Contributions

1. Pistachio-VAD, a generation-based VAD benchmark of 4,962 videos totaling 1,676,822 frames (29.11 hours at 16 FPS), spanning 6 major scene categories, 3,896 scenes, and 31 anomaly types, with both static surveillance-style shots and moving-camera shots. The authors describe it as the largest known semi-supervised VAD dataset and the most category-rich anomaly dataset to date.
2. A fully automated long-form video creation pipeline: Scene-Aware Classification, Anomaly Type Specification, multi-step storyline generation, and temporally consistent long-form video synthesis, producing coherent 41-second videos (16-second short videos) with minimal human intervention.
3. Pistachio-VAU, described as the first large-scale VAU benchmark built without manual annotation (1,385 videos, 517,514 frames), using generation storylines as structured semantic annotations, including 35 videos with multiple co-occurring anomalies.
4. Public release of project assets at https://pistachio-video.github.io, together with an openly characterized scope and limitations section giving concrete guidance on when synthetic data can and cannot stand in for real-world data.

Main Findings

• Dataset scale and diversity: Pistachio-VAD contains 4,962 videos, 1,676,822 frames, 29.11 hours at 16 FPS, 6 major scenes, 3,896 scenes, and 31 anomaly types. Ten anomaly types are explicitly named as absent from existing VAD benchmarks: Animal Predation, Animal Abuse, Construction Accidents, Infrastructure Failure, Natural Disasters, Avalanche, Equipment Breakdown, Extreme Weather Events, Ground Collapse, and Landslide.
• Balanced, non-long-tail design: The authors state they maintained a balanced categorical distribution, directly addressing the long-tail problem in prior datasets (they cite Street Scene's imbalance of 61 jaywalking vs. 1 motorcycle-on-sidewalk as an example).
• Normal footage is deliberately complex: Thanks to storyline design, normal samples include rich everyday activities such as handshaking and vehicle boarding, rather than only walking or light movement.
• Naive long-video pipelines fail: Direct long-form generation suffers ghosting/artifacts and prompt forgetting; fixed first-last-frame looping induces pathological motion artifacts (unnatural acceleration, stuttering, physically implausible trajectories); naive storyline chaining lacks precise control, with the model treating a modified prompt as continuation of the prior scene rather than a new event.
• VAD baselines struggle on Pistachio: Of nine baselines evaluated (RTFM, DR-DMU, MGFN, MULDE, CLIP-TSA, VadCLIP, PEL4VAD, Fed-WSVAD, VADTree), most perform suboptimally on Public Safety and Animal Incidents. PEL4VAD reaches the highest Overall AUC (83.7%) and Fed-WSVAD the highest Overall AP (71.9%). MULDE drops to 63.4% AUC / 34.9% AP, and the training-free VADTree drops to 72.52% AUC / 25.15% AP, which the authors attribute to enriched normal diversity causing false positives.
• Cross-dataset transfer is mixed: Pistachio-trained models are competitive on the synthetic Cosmos and Hailuo sets and on real-world XD-Violence and MSAD, but Pistachio-trained CLIP-TSA reaches only 61.58% AUC / 10.48% AP on UCF-Crime versus 82.56% / 21.62% when trained on UCF-Crime itself.
• Small amounts of in-domain data go far: Adding just 10% of the UCF-Crime training set to Pistachio yields 83.96% AUC / 24.25% AP on UCF-Crime, exceeding full UCF-Crime training (82.56% / 21.62%). Scaling to 30% reaches 86.19% / 28.44% and 50% reaches 86.79% / 33.04%.
• Gains are not generator-specific: Evaluating Pistachio-trained models on benchmarks built with Cosmos-2.5 and Hailuo-2.3 raises AUC from 71.54% to 85.22% on Cosmos and from 68.71% to 82.34% on Hailuo.
• VAU remains hard for current MLLMs: Using F1-Score at event and video levels across InternVL3 (1B–14B), Qwen2.5-VL (3B, 7B), Qwen3-VL-8B, and LLaVA-Next-Video-7B, the best average is InternVL3-14B at 29.39; InternVL3-8B leads at event level (32.56) and Qwen3-VL-8B leads at video level (26.65). Event-level scores consistently exceed video-level scores, and larger parameter counts do not always help (InternVL3 8B to 14B gains are marginal).
• Zero-manual-annotation VAU labels: Event-level and video-level annotations are derived from generation storylines, aggregated by Qwen3-8B and refined by Qwen3-VL-32B cross-referencing against the actual visual content.
• Human/AI filtering quality control: VideoScore is used to discard the bottom 5% of videos (average score below 3.98), followed by manual review with acceptance rates of 90% for normal videos and 80% for anomaly videos, taking approximately 2 person-days.
• Acknowledged domain gap: The authors state Pistachio cannot directly substitute for real-world data when the task is restricted to traditional fixed-camera surveillance or depends heavily on long-term character identity consistency, but can serve well for evaluating detection of sudden, unexpected anomalies across generalized scenarios.

Methodology in Plain English

The researchers avoid scraping videos from the Internet and instead build every video from a still image using generation models. Their pipeline has several stages.

First, still images from COCO 2017, augmented with MIT Indoor Scenes and web-sourced images of large infrastructure like bridges and tunnels, are sorted by a VLM into K = 6 predefined scene categories plus an "Other" category. Each scene category has a manually defined list of plausible anomaly types, and the VLM assigns an appropriate anomaly type to each image so that scene and anomaly are logically consistent.

Second, an LLM turns each image plus its scene/anomaly assignment into a storyline of descriptive segments: 7 to 8 segments for long videos and 2 to 3 for short videos. Each segment is expanded into a detailed prompt specifying camera angle, action, and temporal progression.

Third, the Wan image-to-video model generates each segment, seeded by the previous segment's last frame so the clips chain together into one continuous video. Long videos come out at 41 seconds and short videos at 16 seconds.

Fourth, generated videos pass a hybrid filter: VideoScore automatically discards the bottom 5% (average score below 3.98), and human reviewers check that anomalies follow real-life logic, that there are no obvious AI artifacts or distortions, and that camera angles, viewpoints, and lighting look realistic.

Finally, annotations are produced. VAD ground-truth labels are defined by manual human annotation, while VAU text labels come from the storylines: an LLM (Qwen3-8B) summarizes individual event descriptions into a coherent video-level summary, and a VLM (Qwen3-VL-32B) cross-checks those summaries against the visual content to correct misalignments. The whole generation run took roughly 20 days on a cluster of 32 NVIDIA A100 (80GB) GPUs.

The team then evaluated nine existing VAD methods on the benchmark (training them on the Pistachio training set and testing on Pistachio's test set

Authors’ abstract

Automatically detecting abnormal events in videos is crucial for modern autonomous systems, yet existing Video Anomaly Detection (VAD) benchmarks lack the scene diversity, balanced anomaly coverage, and temporal complexity needed to reliably assess real-world performance. Meanwhile, the community is increasingly moving toward Video Anomaly Understanding (VAU), which requires deeper semantic and causal reasoning but remains difficult to benchmark due to the heavy manual annotation effort it demands. In this paper, we introduce Pistachio, a new VAD/VAU benchmark constructed entirely through a controlled, generation-based pipeline. By leveraging recent advances in video generation models, Pistachio provides precise control over scenes, anomaly types, and temporal narratives, effectively eliminating the biases and limitations of Internet-collected datasets. Our pipeline integrates scene-conditioned anomaly assignment, multi-step storyline generation, and a temporally consistent long-form synthesis strategy that produces coherent 41-second videos with minimal human intervention. Extensive experiments demonstrate the scale, diversity, and complexity of Pistachio, revealing new challenges for existing methods and motivating future research on dynamic and multi-event anomaly understanding.