Segment Anything Across Shots: A Method and Benchmark

AI summary

Overview

Research area: Computer vision, specifically semi-supervised video object segmentation (VOS) extended to multi-shot video object segmentation (MVOS). Technical level: Intermediate — readers should be comfortable with segmentation, attention mechanisms, and benchmark metrics, but the core ideas are explained conceptually. Scope: This paper proposes a training augmentation strategy (TMA), a transition-aware segmentation model (SAAS), and a new multi-shot benchmark (Cut-VOS) to address segmentation across shot boundaries in edited or multi-camera video.

What This Paper Is About

Existing video object segmentation methods assume a video is one continuous shot, so when a video cuts to a new camera angle, location, or time, these models lose track of the target object and fail to segment it. The paper targets this gap: given a mask of the target in the first frame, the goal is to keep segmenting that same object reliably across frequent, abrupt shot changes. Because almost no annotated multi-shot training data exists, the authors also have to solve the data sparsity problem alongside the modeling problem.

Key Contributions

1. Transition Mimicking Data Augmentation (TMA): A training strategy that synthesizes realistic shot transitions on top of ordinary single-shot VOS datasets, allowing multi-shot training without any natively multi-shot annotations.
2. Segment Anything Across Shots (SAAS): The first semi-supervised VOS method built specifically for multi-shot video, adding a Transition Detection Module (TDM), a Transition Comprehension Module (TCH), and a local memory bank to the SAM2 architecture.
3. Cut-VOS benchmark: A new MVOS benchmark with 100 videos, 174 objects, 10.2K dense instance masks, 648 shots, 11 object categories, and a transition frequency 1.6x higher than the prior dataset (0.346/s).
4. YouMVOS-dagger test split: A manually annotated 30-video test split of the existing YouMVOS dataset, created because the original dataset never released its mask annotations.

Main Findings

• Shot transitions break current models badly: SAM2-B+ drops 21.4% in J&F on Cut-VOS compared to the single-shot MOSE dataset, and 16.4% in the cross-shot metric J_t compared to the annotated YouMVOS test split.
• SAAS sets a new state of the art on both benchmarks: SAAS-B+ reaches 73.5% J&F and 68.9% J_t on YouMVOS, and 60.7% J&F and 53.1% J_t on Cut-VOS, beating SAM2-B+ (67.6/63.7 and 55.2/47.2). SAAS-L similarly improves on SAM2-L on Cut-VOS (62.0 vs. 59.4 J&F, 54.0 vs. 50.7 J_t).
• TMA generalizes beyond SAAS: Applying TMA to the unrelated Cutie model raises its Cut-VOS J&F from 52.3% to 53.5%, showing the augmentation strategy is architecture-agnostic.
• Direct single-shot training helps little on truly multi-shot data: Training on YTVOS without TMA gives a small gain on YouMVOS (0.7–1.3%) but actually hurts performance on Cut-VOS, suggesting YouMVOS is not a strong test of multi-shot difficulty.
• Ablation confirms each module matters: On Cut-VOS, the local memory bank adds 2.4% J&F, TMA adds 2.8%, and TMA plus TCH together add 4.9% over the baseline (55.2 → 60.7 J&F).
• Not all transitions are equally hard: SAM2 handles "cut away" and "insignificance" well, is moderate on pitch and horizon transformations, but falls below 27% accuracy on "delayed cut in," "close-up view," and "scene change." Mean accuracy drops from 44.7% on YouMVOS to 38.8% on Cut-VOS.
• Efficiency is preserved: SAAS-B+ runs at 21 FPS versus SAM2-B+'s 22 FPS despite the added modules.

Methodology in Plain English

The work has three interlocking pieces.

Training data (TMA). Since real multi-shot videos with mask annotations are scarce, the authors fake shot transitions using existing single-shot datasets like YTVOS. During training, a normal 8-frame clip is occasionally modified: frames after a chosen cut point are given strong random transformations (scale, affine, flips) to simulate a zoom or angle change; or the clip is replaced with a different segment of the same video to simulate a time jump; or a completely different video is spliced in, optionally with the target object copy-pasted back in and gradually translated to simulate a scene change or a delayed reappearance. These patterns cover common transition types while avoiding ambiguous training samples.

Detecting and understanding transitions (TDM and TCH). A lightweight dilated-convolution module looks at adjacent frames and outputs a probability that a shot boundary has occurred. If no transition is detected, the standard SAM2 pipeline runs unchanged. If a transition is detected, a Transition Comprehension Module takes over: it gathers scene context from a bank of memories of the most recent shots, then uses a trainable query vector that attends to both the previous frame and the current frame to build a "transition state" representation. That state is used to refine the stored memory before segmentation. Two auxiliary losses — predicting whether the object is present next frame, and regressing its bounding box — force the transition state to encode meaningful object information rather than just background appearance.

Local detail memory. Many objects can be recognized after a cut by local cues like clothing or paint markings. The model builds a minimum spanning tree over the masked deep feature map, prunes low-weight edges to split the target into semantically coherent sub-regions, then uses those regions as point prompts to extract fine-grained high-resolution features. These are stored in a local memory bank and consulted specifically when a transition is detected.

Why This Matters

Multi-shot video is now the dominant format on the internet and in edited media, yet nearly all VOS research and benchmarks assume a single continuous shot. This paper closes that gap on three fronts at once — data, method, and evaluation — and the strong results on both a sampled and a purpose-built benchmark suggest the problem is tractable rather than fundamental.

Real-world applications:

• Video editing and post-production: Automatic rotoscoping or object removal across cuts, where an editor currently has to re-prompt the model at every shot boundary.
• Autonomous driving and robotics: Multi-camera systems and high-mobility platforms that need to maintain a target identity across viewpoint discontinuities.
• Video annotation assistance: Semi-automatic labeling pipelines that can propagate one initial mask across a multi-cut clip, dramatically reducing annotation cost.
• Content retrieval and analysis: Tracking a specific person, product, or vehicle through edited compilations, trailers, or multi-camera broadcasts.

Industry relevance: The benchmark and open-source code (CC BY 4.0 data, Apache 2.0 checkpoints) give media, automotive, and annotation-tool companies a directly usable evaluation target. Because SAAS builds on SAM2 and adds only modest inference overhead, the technique is deployable in existing SAM2-based pipelines without a major redesign.

Future Directions

• Extreme appearance change: The authors acknowledge their model fails when a person changes clothing or hairstyle between shots, and TMA cannot simulate this transition type. Solving it likely requires reasoning beyond visual feature matching rather than better memory banks.
• Richer transition simulation: TMA currently covers four transition patterns through affine transforms and video splicing. More realistic synthesis — lighting changes, seasonal changes, lens distortion — could close the gap to real footage.
• Scaling and backbone independence: SAAS is demonstrated on SAM2 variants. Testing whether TDM, TCH, and the local memory bank transfer to other VOS backbones or to open-vocabulary detection-based trackers is an open question.
• Joint matching and distinguishing: The paper frames the core MVOS challenge as simultaneously matching objects that look different across shots and separating objects that look alike within a shot. Methods that explicitly model this dual objective, rather than relying on appearance similarity, remain unexplored.

Target Audience

Researchers and graduate students in computer vision working on video object segmentation, tracking, or video foundation models; engineers building video editing, annotation, or multi-camera perception systems; and benchmark designers interested in how to construct transition-aware evaluation sets with dense mask annotation. Readers with a general machine learning background but no prior VOS exposure will find the paper accessible, though familiarity with memory-bank VOS architectures and metrics like J&F will make the experimental sections easier to interpret.

Authors’ abstract

This work focuses on multi-shot semi-supervised video object segmentation (MVOS), which aims at segmenting the target object indicated by an initial mask throughout a video with multiple shots. The existing VOS methods mainly focus on single-shot videos and struggle with shot discontinuities, thereby limiting their real-world applicability. We propose a transition mimicking data augmentation strategy (TMA) which enables cross-shot generalization with single-shot data to alleviate the severe annotated multi-shot data sparsity, and the Segment Anything Across Shots (SAAS) model, which can detect and comprehend shot transitions effectively. To support evaluation and future study in MVOS, we introduce Cut-VOS, a new MVOS benchmark with dense mask annotations, diverse object categories, and high-frequency transitions. Extensive experiments on YouMVOS and Cut-VOS demonstrate that the proposed SAAS achieves state-of-the-art performance by effectively mimicking, understanding, and segmenting across complex transitions. The code and datasets are released at https://henghuiding.com/SAAS/.