Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrieval

AI summary

Overview

• Research area: Efficient computer vision — specifically post-training acceleration of video object segmentation (VOS) foundation models, with a focus on the Segment Anything Model 2 (SAM2) family.
• Technical level: Advanced. The paper assumes familiarity with vision transformers, window attention, cross-attention maps, FIFO memory banks, and token sparsity.
• Scope: The paper proposes a training-light, post-training acceleration framework that sparsifies SAM2's image encoder and memory attention to improve inference speed while preserving segmentation accuracy.

What This Paper Is About

SAM2 delivers strong video object segmentation quality, but its large image encoder and frame-by-frame memory interaction make real-time video processing impractical. Most prior efficiency work retrains lightweight backbones, which is expensive and degrades accuracy; this paper instead targets post-training acceleration, exploiting the observation that SAM2's computation is dense even though its actual perceptual attention is sparse and concentrated on a small number of object-relevant regions.

Key Contributions

1. Efficient-SAM2, a post-training sparse acceleration scheme for SAM2. The paper frames the core inefficiency as a contradiction between SAM2's sparse perception pattern and its dense computation, identifying redundancy specifically in the image encoder and the memory attention block.
2. Object-aware Sparse Window Routing (SWR) for the image encoder. A window-level computation allocation mechanism that routes background windows into a lightweight shortcut branch, with routing decisions driven by saliency and consistency cues from the previous frame's mask decoder.
3. Object-aware Sparse Memory Retrieval (SMR) for memory attention. A scheme that lets only salient memory tokens participate in attention, with each memory frame's saliency pattern computed once at its first recollection and reused for subsequent frames under a FIFO caching scheme.
4. Demonstration of a favourable performance–speed trade-off. Across SA-V, DAVIS 2017, and MOSE benchmarks, the method adds negligible parameters and minimal training cost while reporting up to 1.68× end-to-end speedup on SAM2.1-L with a 1.0% accuracy drop on the SA-V test set.

Main Findings

• Sparse perception is real in SAM2. The mask decoder's prompt-to-image attention concentrates on foreground objects and potential distractors while suppressing background, whereas the earlier image encoder shows a broad attention span because it is unaware of the prompt's interest.
• Memory bank redundancy with temporal consistency. Only a small subset of memory tokens receives dominant attention, and the attention distribution over a given memory frame stays highly similar across consecutive queries (measured by cosine similarity), so repeated full-token recollection wastes computation.
• Headline speedup and accuracy. Efficient-SAM2 delivers 1.68× speedup on the SAM2.1-L model with only 1.0% accuracy drop on the SA-V test set; SWR and SMR individually provide 1.83× and 1.78× module-level speedups.
• SWR results on SAM2.1-L (Table 2). 1.83× module speedup, SA-V test 79.0 versus the original 79.8, DAVIS 89.9 matching the original, MOSE 73.4 versus 74.5. At Δt=5, SWR's DAVIS score (90.7) exceeds the original model's (89.9).
• SMR results on SAM2.1-L (Table 2). 1.78× module speedup, SA-V test 79.6 versus 79.8, SA-V val 77.7 versus 78.3, DAVIS 89.9 matching original, MOSE 74.2 versus 74.5.
• Combined Efficient-SAM2 on SAM2.1-L (Table 2). 1.80× module-level speedup with SA-V test 78.8, SA-V val 75.5, DAVIS 89.7, MOSE 72.6 at Δt=1.
• Baseline comparisons. ToMe collapses badly on SAM2.1-L (1.43× speedup, SA-V test 47.8 versus 79.8). MemPool reaches higher speedup (2.04× on L, 2.14× on B+) but with larger accuracy loss. EdgeTAM, the distillation baseline, gets 1.63× on B+ (Δt=1) with SA-V test 72.1 versus 77.7.
• SMR variants confirm the sparsity hypothesis. Random and uniform token masks (SMR-random at 1.73×/1.75×, SMR-uniform at 1.78×/1.76×) also preserve most accuracy, which the authors present as evidence for token-level sparsity in SAM2's memory bank.
• Shortcut branch design matters little; efficiency does. In Table 3, Bottleneck-FFN adds only 0.20M parameters and 0.97 training hours for a 75.0 J&F on SAM2.1-B+, versus 0.81M/1.10 hours for a full FFN (also 75.0) and 0/0 hours for a bare Identity (73.0). On SAM2.1-L, Bottleneck-FFN gives 79.0 with 0.33M parameters.
• SWR routing components help. In Table 4, using all three mask predictions instead of only the highest-IoU one adds +0.5 on SA-V test and +0.7 on MOSE; mask dilation adds +1.4 on SA-V test and +1.6 on MOSE.
• Cross-dataset sparsity varies. Table 5 reports SWR window sparsity and speedup per dataset: SA-V test 0.80/3.12× (B+) and 0.75/3.24× (L); SA-V val 0.77/2.96× and 0.74/3.14×; DAVIS 2017 val 0.61/2.17× and 0.50/2.03×; MOSE val 0.72/2.48× and 0.66/2.61×.
• Prose and table speedup figures are inconsistent in places. The Section 4.2 text states SWR at "1.65×" and Efficient-SAM2 at "1.71×" for SAM2.1-B+ at Δt=1, while Table 1 lists 1.69× and 1.74×. For SAM2.1-L the text says SWR "1.80×" and Efficient-SAM2 "1.75×", while Table 1 lists 1.83× and 1.80×. The abstract's 1.83×/1.78× figures match the table values.
• Speed measurement caveat. Speed was evaluated on an RTX A6000 at Float32, not BF16. The authors report that on their hardware/software stack, dropping roughly 80% of windows and 68% of memory tokens left end-to-end time essentially unchanged in BF16, attributing this to irregular memory access and control-flow overhead rather than dense compute.

Methodology in Plain English

The authors start by looking at where SAM2 actually spends its attention rather than where it spends its FLOPs, and find a mismatch: the expensive early encoder computes over the whole frame uniformly, but the cheap late decoder ends up caring only about the object.

Their first fix, SWR, works at the granularity of attention windows. The image encoder uses window attention, which processes non-overlapping patches independently, so windows can be treated individually without breaking the architecture. A router decides which windows contain the object, using two signals: which windows overlap the previous frame's predicted masks (all three candidate masks, after dilation, for spatial-temporal continuity), and — when tracking confidence is low — which windows had high cross-attention saliency in the previous frame's mask decoder. Object-relevant windows go through the normal transformer block; background windows go through a small two-layer bottleneck that is trained by matching the teacher model's memory-conditioned features with a simple squared-error reconstruction loss, using only 30 unlabeled SA-V training samples and about one hour on an RTX A6000.

Their second fix, SMR, works at the token level inside memory attention. When a memory frame enters the queue, its cross-attention weights are used once to pick the top (1 − s) fraction of tokens by importance (s = 0.95 per frame across all memory attention layers, giving an overall sparsity ratio of 5s/7 ≈ 0.68 once the prompt frame and latest frame are kept dense). That selected token set is cached and reused for later frames rather than recomputed, following the same first-in-first-out order as the memory queue itself. The prompt frame always stays dense and the most recent frame always stays complete, so tracking and occlusion handling are not damaged. The paper states this reduces per-layer complexity from O((m+1)NKd) to O(2NKd + (m−1)Nkd), where k = (1−s)K is much smaller than K.

Evaluation uses SAM2.1-B+ and SAM2.1-L with memory intervals Δt=1 and Δt=5, on SA-V test and validation, DAVIS 2017 validation, and MOSE validation, with J&F as the metric, following the standard semi-supervised VOS protocol where first-frame ground-truth masks serve as prompts.

Why This Matters

Impact on research. The paper argues that post-training acceleration tailored for SAM2 is largely unexplored, and that generic token-merging methods (ToMe, ALGM, ToMe4DM) transfer poorly because SAM2's encoder relies on windowed attention that makes their matching strategies incompatible. If its observations hold, the work reframes SAM2 efficiency as a sparsity-scheduling problem rather than a retraining problem, offering a cheaper alternative to distillation pipelines like EdgeTAM.

Real-world applications:

• Real-time video object segmentation and tracking on streaming footage, where frame-rate latency is the binding constraint.
• Edge and embedded deployment, where a large vision backbone plus per-frame memory interaction is otherwise prohibitive.
• Video editing and post-production workflows that need interactive, promptable object masks across long clips.
• Robotics and autonomous perception, where segmentation must run within a control loop rather than offline.

Industry relevance. The method requires no architectural redesign and approximately one hour of shortcut-branch training on a single GPU, which lowers the barrier for teams that want to deploy an existing SAM2 checkpoint faster without a large retraining budget. The reported speedups are end-to-end and measured on real hardware rather than theoretical FLOP reductions, which is the number deployment teams care about.

Future Directions

• Resolve the measurement caveat. The authors report that BF16 gives little to no end-to-end gain even after aggressive sparsification, and that the pipeline is dominated by irregular memory access and control-flow overhead. Making sparse SAM2 actually faster in low precision, or designing kernels that exploit the sparsity, is a clear open problem.
• Tune sparsity per layer or per frame. SMR applies a single sparsity ratio s = 0.95 to every memory attention layer; the paper's own finding that accuracy rises above the dense baseline until s ≈ 0.95 suggests the optimal ratio is not uniform.
• Extend beyond semi-supervised VOS. The evaluation covers SA-V, DAVIS 2017, and MOSE under a first-frame-prompt protocol; whether the same sparsity patterns hold for referring VOS, interactive multi-prompt use, or long-horizon tracking is not reported.
• Investigate the accuracy-gain effect. SWR at Δt=5 surpasses the original model on DAVIS (90.7 versus 89.9), which the authors attribute to noise suppression; they do not fully characterize when sparsification improves rather than degrades quality.

Target Audience

Practitioners and researchers working on efficient foundation models, video object segmentation, and on-device vision deployment — particularly engineers who already run SAM2 and want inference speedups without retraining a backbone. It is also relevant to readers interested in exploiting attention sparsity in transformer-based vision systems, and to those comparing post-training acceleration against distillation approaches. The paper's density of architectural notation, complexity expressions, and sparsity ratios makes it most accessible to readers with a working knowledge of transformer attention mechanisms.

Authors’ abstract

Segment Anything Model 2 (SAM2) shows excellent performance in video object segmentation tasks; however, the heavy computational burden hinders its application in real-time video processing. Although there have been efforts to improve the efficiency of SAM2, most of them focus on retraining a lightweight backbone, with little exploration into post-training acceleration. In this paper, we observe that SAM2 exhibits sparse perception pattern as biological vision, which provides opportunities for eliminating redundant computation and acceleration: i) In mask decoder, the attention primarily focuses on the foreground objects, whereas the image encoder in the earlier stage exhibits a broad attention span, which results in unnecessary computation to background regions. ii) In memory bank, only a small subset of tokens in each frame contribute significantly to memory attention, and the salient regions exhibit temporal consistency, making full-token computation redundant. With these insights, we propose Efficient-SAM2, which promotes SAM2 to adaptively focus on object regions while eliminating task-irrelevant computations, thereby significantly improving inference efficiency. Specifically, for image encoder, we propose object-aware Sparse Window Routing (SWR), a window-level computation allocation mechanism that leverages the consistency and saliency cues from the previous-frame decoder to route background regions into a lightweight shortcut branch. Moreover, for memory attention, we propose object-aware Sparse Memory Retrieval (SMR), which allows only the salient memory tokens in each frame to participate in computation, with the saliency pattern reused from their first recollection. With negligible additional parameters and minimal training overhead, Efficient-SAM2 delivers 1.68x speedup on SAM2.1-L model with only 1.0% accuracy drop on SA-V test set.