O-VAD: Industrial Video Anomaly Detection through Object-Centric Tracking and Reasoning

AI summary

Overview

Research area: Computer vision, specifically industrial video anomaly detection (IVAD) using vision-language models (VLMs) and agentic reasoning pipelines.

Technical level: Intermediate. The paper assumes familiarity with object detection, segmentation and tracking (SAM, SAM2, CropFormer), vision-language models, and chain-of-thought prompting, but its central idea—track how objects change state over time and reason about those trajectories—is easy to grasp.

Scope in one sentence: The paper introduces O-VAD, a training-free, three-stage agentic pipeline that detects industrial anomalies by grounding objects, tracking their physical state changes across frames, and reasoning over those trajectories with a VLM's built-in commonsense.

What This Paper Is About

Industrial inspection videos are hard to analyze automatically because objects are constantly transformed—cut, pressed, assembled, heated—and anomalies often show up as subtle violations of physics or procedure rather than obvious visual outliers. Existing anomaly detectors either need to be retrained on normal footage for every object class, or rely on VLMs that reason over whole frames or captions and miss fine-grained object-level details. O-VAD's goal is to detect anomalies without any training data, domain-specific knowledge, or predefined anomaly taxonomy, by having the model inspect objects the way a human quality inspector would: watch each object change over time and judge whether those changes make sense.

Key Contributions

1. A generalizable data curation pipeline applied to three object-centric industrial anomaly detection datasets (IPAD, Phys-AD, LiquidAD), producing fine-grained annotations of object trajectories across frames.

2. A training-free, three-stage agentic reasoning framework that requires no domain knowledge or fine-tuning. It performs automated object discovery, spatiotemporal tubelet construction with open-ended state-change detection, and chain-of-thought anomaly reasoning with visual verification.

3. State-of-the-art results on three IVAD datasets at both video and frame levels, outperforming frontier VLMs (Qwen3-VL-32B, GPT-5), two agentic VAD frameworks (URF-ZS-HVAA, VERA), and traditional anomaly detectors fine-tuned on the same datasets.

4. Interpretable, open-ended anomaly reports that specify anomaly type, severity, affected object, temporal localization, and a natural-language causal explanation—rather than a binary score.

Main Findings

• Object-level evidence is the real bottleneck, not language reasoning. On Phys-AD, O-VAD achieves the best training-free video-level AUROC (0.584), beating direct prompting with Qwen3-VL-32B (0.513) and GPT-5 (0.503), agentic workflows URF-ZS-HVAA (0.426) and VERA (0.456), and even the trained baselines MNAD.p (0.495) and S3R (0.555). The authors argue GPT-5's failures come from impoverished caption-level inputs, not weak reasoning.

• Largest gains occur on multi-step interaction tasks. Across the 22 Phys-AD categories, O-VAD ranks best or second-best on 16, with big jumps on Gear (0.879), Lock (0.780), Screw (0.704), and Sticky Roller (0.811)—cases where temporal state tracking or surface-level detail matters most.

• Object-centric tracking generalizes to cluttered multi-instance scenes. On LiquidAD, where up to eight pipettes dispense in parallel, zero-shot VLMs collapse to near-zero recall and a strong bias toward "normal," while O-VAD reaches the best training-free video-level AUROC (0.692) and the strongest frame-level recall and F1.

• Reference-based industrial processes also benefit. On IPAD, whose anomalies are defined as deviations from reference normals rather than physics violations, O-VAD still posts the highest average video-level (0.565) and frame-level (0.518) AUROC and ranks first on 12 of 16 scenarios.

• Removing state tracking is catastrophic. The ablation shows precision, recall, and F1 drop to zero on three of four test categories, with the system collapsing to predicting everything normal—even though BERTScore stays deceptively high (0.867–0.947) because the model emits fluent but content-free descriptions. This shows text-quality metrics alone are inadequate for evaluating anomaly detection.

• Video-level captioning helps only when anomalies are holistic. Removing the caption drops AUC by 0.247 (sticky roller) and 0.131 (rubber band), where scene-level semantics guide judgment, but slightly improves screw F1 (0.700 to 0.733), because localized instantaneous defects gain nothing from a coarse global summary.

• Structured reasoning and verification improve calibration. Swapping the six-step chain-of-thought for a generic "think step by step" prompt lowers AUC by 0.038–0.091 across all four ablation categories; removing post-hoc visual verification lowers AUC by up to 0.178, mainly helping borderline cases.

• Qualitative traces show grounded multi-defect diagnosis. In a plastic-bottle case, GPT-5 reports "no leakage" while O-VAD tracks progressive deformation and flags a high-severity loss-of-containment anomaly. In a hinge-screw case with three co-occurring defects, O-VAD captures all three streams and attributes them to a torque-control or bit-mismatch failure.

Methodology in Plain English

O-VAD works as a three-stage assembly line with no training at any point.

Stage 1 — Find the objects. Instead of analyzing one frame, the system samples frames spread across the video so that objects which appear late or start occluded still get discovered. A VLM (GPT-5) lists the objects in each sampled frame with a name, description, and location cue. These lists are merged and deduplicated. Each object is then segmented in a reference frame using SAM3, producing an initial mask.

Stage 2 — Follow each object over time. A per-frame entity segmentation model (CropFormer) partitions every frame, and SAM2 propagates those regions forward into "tubelets"—tubes of the same region across time. When an object breaks apart, releases material, or otherwise transforms so its track is lost, the system looks for newly appearing tracks that are both spatially close (overlap with SAM2's mask candidates) and semantically similar (masked CLIP feature similarity), then stitches them back onto the original object. At moments where an object's state visibly transitions, the VLM is queried with masked visualizations of two frames and asked to describe what changed—in free-form language, not from a fixed list. Each event is stored as a tuple: start frame, end frame, change type, cause, description, severity, and affected object index.

Stage 3 — Reason like an inspector. All accumulated evidence—object metadata, state-change events, an optional video caption, and sampled frames—goes into a single cascaded chain-of-thought prompt with six steps: understand the process, cite observations, state pass/fail expectations, compare candidate anomalies against those expectations, reason about causes, and finally assign a free-form anomaly type and severity. Because the VLM is never constrained to a fixed taxonomy, it can name novel anomaly types. To suppress false positives, each candidate is gated by confidence: above 0.8 it is accepted outright, below 0.2 it is discarded, and anything in between is re-verified against its evidence frames. Final confidence multiplies the original and verification scores, and only candidates above 0.3 survive into the report.

Why This Matters

Impact on research. The paper reframes industrial anomaly detection as an evidence-construction problem rather than a classification problem. Its strongest empirical signal is that a frontier VLM with rich object-level state trajectories beats the same VLM given only frame-level or caption-level input—suggesting that future gains in VLM-based anomaly detection will come from better grounding and tracking, not bigger language models or heavier prompt engineering. It also provides a benchmark-level contribution by curating three IVAD datasets with object trajectory annotations.

Real-world applications:

• Robotic assembly lines, where a manipulated part must be tracked through pressing, rotating, or grasping and any deviation flagged before it propagates.
• Automated laboratory liquid handling, where multiple pipettes dispense in parallel and per-instance precision is required to catch overflow or misfill.
• Factory equipment inspection, where deviations from a reference normal clip signal tool wear, misalignment, or incomplete operations.
• Quality control auditing, where operators need not just an alert but a causal explanation with the specific frames, object, and severity for root-cause analysis and corrective action.

Industry relevance. The framework requires no labeled data, no per-class retraining, and no expert-written domain rules—addressing the practical barrier that traditional one-class-one-model detectors must be retrained for every new product or production line. Because outputs are natural-language reports with grounded frame ranges and causes, they are directly actionable for maintenance and quality teams rather than being opaque anomaly scores. The main caveats are latency (the multi-stage pipeline is far slower than a single forward pass) and a reliance on VLM commonsense rather than specification-level knowledge.

Future Directions

• Multi-granularity captioning. The ablation shows global video captions help holistic anomalies but hurt localized instantaneous ones. Adapting caption granularity to the spatio-temporal scale of the anomaly is an explicit next step the authors flag.

• Injecting lightweight domain priors and specification references. O-VAD fails on anomalies defined by invisible physical properties or precise numeric specs that cannot be inferred from appearance alone. Few-shot in-context learning with specification examples is proposed as one route.

• Reducing latency. The pipeline's cost scales with the number of tracked objects per video, making real-time deployment on busy production lines an open engineering problem.

• Handling static or perception-ambiguous defects. Stuck buttons and degaussed magnets produce visually plausible behavior, which the authors describe as a fundamental boundary of perception-driven reasoning rather than an engineering gap.

• Scaling the object-tracking and reasoning stage. The curated trajectory annotations across three datasets open the door to learning the tracker or reasoner rather than keeping the whole pipeline training-free.

Target Audience

Researchers and practitioners working on video anomaly detection, industrial visual inspection, and VLM-based agentic systems will get the most from this paper. It is also valuable for engineers evaluating whether training-free pipelines can replace per-class trained detectors in manufacturing and lab automation, and for anyone interested in how chain-of-thought reasoning can be grounded in structured spatiotemporal evidence rather than raw frames. Readers should have working familiarity with segmentation and tracking models and with VLM prompting; the core argument, however, is accessible to anyone who understands the difference between analyzing a whole video and tracking a single object through it.

Authors’ abstract

Industrial Video Anomaly Detection (IVAD) aims to identify anomalous objects and events in an industrial process, which is crucial for modern manufacturing and quality control systems. Existing VLM-based anomaly reasoning methods are capable of detecting open-ended anomalies in general domains. However, their performance declines in industrial settings characterized by intricate object transformations, strict physics, and procedural constraints. To tackle the complexity of such interaction-intensive detection, we introduce a training-free agentic framework for anomaly detection free of domain-specific knowledge, emphasizing object state evolution like humans inspectors. It is designed to track spatial-temporal dynamics and underlying transformations of detected objects over time, and then reason over the object-wise temporal state trajectories to identify abnormal objects in grounded frames. Our method overcomes limitations of prior approaches that rely on retraining on normal clips or injecting domain knowledge as context for test-time inference. Extensive experiments on three IVAD datasets demonstrate that our method outperforms frontier VLMs, agentic frameworks, and traditional VAD methods fine-tuned on the respective datasets, while providing interpretable reports over anomaly processes and types.