STORM: Segment, Track, and Object Re-Localization from a Single Image

AI summary

Overview

Research area: Computer vision — 6D object pose estimation and tracking, reference-conditioned segmentation, and failure-aware tracking with foundation models.

Technical level: Advanced (assumes familiarity with attention mechanisms, 6D pose metrics such as ADD/ADD-S, and BOP benchmark protocols).

Scope: STORM is a unified, annotation-free framework that combines a reference-conditioned segmentation module with a learned tracking-failure verifier to track object 6D pose from as little as a single reference image.

What This Paper Is About

Estimating and tracking the 6D pose of objects in real scenes usually requires curated CAD models, hand-drawn masks, or per-object fine-tuning, and existing trackers give no signal when they have silently lost the object. STORM addresses both problems by learning to align a reference view with cluttered query views and by learning an explicit "am I still tracking the right thing?" verifier whose score triggers automatic re-localization after persistent drift.

Key Contributions

1. STORM framework. A unified, annotation-free, reference-conditioned 6D pose tracking system that does not require curated CAD models or instance-specific fine-tuning at inference time, explicitly targeting the distributional shift between clean reference data and degraded query observations.
2. Hierarchical Spatial Fusion Attention (HSFA). A task-driven architecture for variable-view reference-conditioned segmentation that combines reference-view aggregation, query-reference cross-attention, and optional VLM-derived (CLIP/LLM) semantic conditioning to resolve instance ambiguity under clutter and occlusion.
3. BCE-trained tracking verification. Tracking validity is formulated as a binary compatibility verification problem implemented in the Tracking Object Module (TOM), whose compatibility logit is converted into an energy-like score to detect persistent drift and trigger automatic re-initialization.
4. Tracking Failure Benchmark. A benchmark that evaluates trackers not only on pose accuracy but also on their ability to distinguish valid tracks from failure modes, reported to show TOM provides more reliable tracking-loss detection than fixed metric-learning baselines.

Main Findings

• Segmentation quality. SOM reaches a mean AP_C of 67.7 ± 1.44 across five BOP datasets (LM-O, T-LESS, TUD-L, HB, YCB-V), slightly above the strongest supervised baseline, ZebraPoseSAT–EffNetB4 at 67.5. Per-dataset SOM scores are 57.8 ± 2.18 (LM-O), 53.0 ± 1.67 (T-LESS), 73.3 ± 2.26 (TUD-L), 74.1 ± 0.50 (HB), and 80.3 ± 1.41 (YCB-V). The paper states SOM sets a new state of the art on four of the five benchmarks (LM-O, TUD-L, HomebrewedDB, YCB-Video) and is the strongest annotation-free method on T-LESS.
• Segmentation efficiency. SOM reports a runtime of 0.046 s, and the single-reference (k = 1), language-conditioned setting used in the main comparison runs at about 22 FPS on an H100. Competing annotation-free methods list times such as 0.221 s (CNOS with FastSAM), 0.485 s (NIDS), 0.559 s (MUSE), 0.990 s (NOCTIS), 1.276 s (Prisma-MPG-Complex), 1.700 s (ViewInvDet), 1.890 s (LDSeg), 1.847 s (CNOS with SAM), and 2.795 s (SAM6D).
• Pose estimation on LM-O. With FoundationPose as the fixed downstream estimator, STORM reaches 74.0 ± 1.28 ADD_AUC, 89.0 ± 1.25 ADD-S_AUC, and 53.0 ± 2.02 AR, versus 57.0 / 68.0 / 41.0 for FP + CNOS and 78.0 / 93.0 / 56.0 for the ground-truth-mask upper bound. The paper describes this as within 4% of the ground-truth-mask variant.
• Pose estimation on YCB-Video. STORM reaches 77.0 ± 1.25 ADD_AUC, 98.0 ± 1.20 ADD-S_AUC, and 73.0 ± 1.23 AR, versus 73.0 / 92.0 / 69.0 for FP + CNOS and 78.0 / 99.0 / 74.0 for FP + Ground Truth. The paper states STORM matches the ground-truth setup exactly on this dataset and exceeds the CNOS baseline by 5%.
• Reconstructed geometry works if aligned. On YCB-V, SAM3D (unaligned) + FP yields VSD 22.19 ± 3.3, IoU 48.27 ± 6.24, ADD-S_AUC 44.92 ± 5.23; SAM3D (aligned) + FP yields 60.72 ± 0.32, 84.63 ± 1.43, 99.12 ± 0.87; GT + FP yields 60.97, 88.50, 98.95. The authors conclude SAM3D reconstructions are usable geometry for pose registration but not exact replacements for ground-truth CAD models.
• Recovery from tracking failure. On a modified YCB-Video stress test (frame dropping for rapid motion, random masks for occlusion), STORM achieves ADD_AUC 74.64 ± 1.25, ADD-S_AUC 88.56 ± 1.89, and AR 67.85 ± 2.13 in 98 ± 3 ms, versus FoundationPose tracking at 52.76, 66.76, 50.09 in 84 ms. The paper reports a measured end-to-end tracking-loop increase of about 14 ms and a TOM verifier cost of 9.10 ± 0.04 ms per full forward pass.
• Attention improves failure detection. On the synthetic failure dataset, cosine similarity reaches 87.55% accuracy and 94.2 AUROC at 12.4 ms; TOM without attention reaches 95.23 ± 0.26% / 95.6 ± 0.16 at 12.2 ± 0.30 ms; TOM with one attention layer reaches 98.36 ± 0.21% / 96.4 ± 0.13 at 13.5 ± 0.37 ms; two layers reach 97.84 ± 0.14% / 96.2 ± 0.10 at 16.8 ± 0.43 ms. The single attention layer lifts accuracy from 87.55% to 98.36% for only 1.3 ms extra latency.
• Fixed design choices. Tracking loss is declared when the EMA-smoothed energy exceeds the threshold for L = 3 consecutive steps; the threshold is calibrated once as the 95th percentile of compatible-pair scores; the memory pool is a FIFO queue of K = 16 crops that is reset after re-localization.
• Evaluation protocol. Results are reported as mean ± std over 5 independent runs with different random seeds, in an annotation-free, adaptation-free setting where BOP test scenes and images are never used for training. The evaluation spans 5 BOP test datasets covering 95 distinct object types, and SOM can condition on between 1 and 16 rendered reference views per object.

Methodology in Plain English

STORM splits the problem into two learned modules sitting on top of frozen foundation components.

The Segmenting Object Module (SOM) learns to align a query image (occluded, transformed) with one or more reference views. Instead of concatenating features or using cosine similarity, it uses Hierarchical Spatial Fusion Attention: reference patches are first aggregated into a canonical object representation, query features then attend to that representation via cross-attention, and this fusion block is repeated (the paper describes early layers attending to raw reference features for global semantics and deeper layers attending to refined features for local detail). Optionally, a VLM generates a short text descriptor for the object, CLIP encodes it, and a zero-initialized AdaLN/FiLM-style layer injects it as a residual scale and shift on the visual tokens, with a sigmoid gate modulating reference key/value channels. Alignment is not supervised explicitly — an attention-weight matrix W propagates a reference objectness prior to produce the query mask, and standard segmentation losses are applied to the mask.

The Tracking Object Module (TOM) is a binary compatibility classifier. It takes the current observation crop and a memory pool of crops from previously successful frames, extracts DINOv3 features, and outputs a compatibility logit through a lightweight attention network. It is trained with binary cross-entropy on a synthetic verification set built from BOP annotations: positives are observation-memory pairs of the same object, negatives come from distractor swapping (different object crops) and drift simulation (randomly shifted or randomly cropped regions). At inference, the negative logit is treated as an energy-like score, smoothed with an EMA, and compared to a calibrated threshold; only persistent exceedance over three consecutive steps triggers re-localization.

Geometry comes from SAM3D, which builds a canonical mesh from the reference image and serves as a rigid anchor for lifting 2D masks into SE(3); FoundationPose then performs downstream registration and tracking. The trainable parts are SOM (HSFA plus segmentation heads) and TOM; DINOv3, the VLM/CLIP stack, SAM3D, and FoundationPose are frozen. Evaluation follows the BOP protocol with the ADD, ADD-S, Average Recall, VSD, IoU, mAP, accuracy, and AUROC metrics as described in the paper.

Why This Matters

Impact on research. The paper argues that reference-conditioned trackers currently fail "blindly" — they assume the target stays in a local basin and have no internal way to recognize drift. STORM's contribution is to treat segmentation, geometry, and failure awareness as one joint problem, and to release a Tracking Failure Benchmark that measures failure discrimination separately from pose accuracy. It also demonstrates that reconstructed geometry, rather than authored CAD, can be adequate for pose registration once coordinate frames are handled consistently.

Real-world applications (implied by the settings the paper targets):

• Robotic manipulation and pick-and-place, where STORM is motivated by 6D pose as a prerequisite for embodied agents.
• Industrial bin picking and assembly, drawing on the LM-O, T-LESS, and TUD-L / TUD-Light evaluation settings.
• Warehouse and household object handling, drawing on the YCB-Video and HomebrewedDB settings.
• Deployment in dynamic, unstructured scenes where occlusion, fast motion, and viewpoint change are common and manual masking is impractical.

Industry relevance. The pipeline reduces reliance on curated CAD models and manual annotation, both of which the paper identifies as labor-intensive deployment bottlenecks. The reported timing profile — a 0.046 s SOM runtime, roughly 22 FPS single-reference operation on an H100, a 9.10 ms TOM forward pass, and about 14 ms added to the tracking loop in the stress test — is presented as keeping the steady-state tracking loop in the real-time regime, with SOM invoked only at initialization or drift-triggered re-localization.

Future Directions

• Better reference coverage for hard objects. The authors note a single reference image is only the minimum input, that viewpoint-diverse references improve robustness, and that symmetric or textureless objects such as T-LESS can stay ambiguous for both visual matching and language descriptions.
• Reducing dependence on reconstructed geometry. The pipeline's quality is bounded by the fidelity and coordinate consistency of SAM3D reconstructions; the paper notes that CAD-based alignment was used only to report under the BOP coordinate convention, and deployment would instead need a self-consistent reconstructed object frame.
• Handling materials not well covered by the evaluation. Transparent, reflective, or thin objects are explicitly named as cases where the frozen FoundationPose registration/tracking backend can bottleneck pose quality.
• Understanding the role and cost of language conditioning. Language conditioning is optional, and the VLM/CLIP descriptor is generated once per object and cached; the paper also reports diminishing returns from stacking a second attention layer in TOM. The provided content is truncated at this point, so the remaining future-work items are not reported.

Target Audience

Researchers and engineers working on 6D object pose estimation and tracking, zero-shot or annotation-free perception, and robotic manipulation will benefit most. It is also relevant to practitioners building on BOP benchmarks and foundation-model pipelines (DINOv3, SAM3D, FoundationPose, CLIP/VLMs), and to readers interested in failure detection and automatic recovery in tracking systems. The dense notation, benchmark tables, and energy-score formulation make it an advanced rather than introductory read.

Authors’ abstract

Accurate 6D pose estimation and tracking are core capabilities for physical AI systems, yet real-world deployment remains brittle and labor-intensive. Many pipelines rely on CAD models, manual masking, or per-object adaptation, and still fail under occlusion or fast motion without a principled way to recognize failure. We propose STORM, a unified framework for reference-conditioned 6D tracking that can operate from a single reference image, with minimal manual input and improved robustness. STORM combines: (i) Hierarchical Spatial Fusion Attention (HSFA), a task-driven reference-query fusion architecture that supports both single-reference and multi-reference conditioning and can optionally use vision-language semantic conditioning to resolve instance ambiguities; and (ii) a BCE-trained tracking verifier whose continuous compatibility logit is used as an energy-like score to detect drift and trigger automatic re-initialization. Experiments on LM-O and YCB-Video show that STORM improves annotation-free pose tracking accuracy over strong baselines and recovers reliably from severe occlusions and rapid viewpoint changes with minimal overhead.