Segment and Matte Anything in a Unified Model

AI summary

Overview

Research area: Computer vision — interactive image segmentation and image matting, built as a lightweight extension of the Segment Anything Model (SAM).

Technical level: Intermediate. The paper assumes familiarity with SAM's encoder/prompt-encoder/decoder design, transformer attention, and standard segmentation and matting evaluation metrics.

Scope: The paper proposes and evaluates SAMA (Segment And Matte Anything), a single frozen-SAM-based framework that performs high-quality interactive segmentation and interactive image matting simultaneously with only 1.8% additional parameters.

What This Paper Is About

Segment Anything (SAM), trained on over one billion masks across 11 million images in SA-1B, generalizes well and accepts points, boxes, and masks as prompts, but its raw masks often lack tight boundaries and fine structural detail. Existing fixes (HQ-SAM, Pi-SAM, DIS-SAM, SAMRefiner) typically add extra post-processing models or require additional human interaction, and interactive image matting — producing a continuous alpha matte that captures hair, fur, or glass — had not been explored inside SAM.

The goal of this work is to unify both problems in one model: recover fine-grained boundary detail without sacrificing SAM's zero-shot prompting flexibility, and add an interactive matting capability to the same encoder–decoder architecture.

Key Contributions

1. A unified SAM-based framework. SAMA is presented as the first SAM-based model that jointly performs interactive segmentation and matting, adding only 1.8% to SAM's parameter count and, per the authors, only marginal latency.
2. New architectural components. A Multi-View Localization Encoder (MVLE), a Localization Adapter (Local-Adapter), and task-specific prediction heads are introduced to bridge object-level context and boundary-level detail.
3. Dual task-specific tokens and heads. SAM's original output token is replaced with two learnable SAMA tokens (one for segmentation, one for matting) that feed two lightweight prediction heads, enabling both outputs from one model without architectural modification between tasks.
4. State-of-the-art results across benchmarks. SAMA is reported to set new records on multiple segmentation and matting benchmarks while retaining SAM's prompting versatility and inference speed.

Main Findings

• Dichotomous image segmentation (DIS). On DIS-VD, SAMA reaches F_beta^max 0.942, F_beta^w 0.885, MAE 0.021, S_alpha 0.930, and E_phi^m 0.962, versus DIS-SAM's 0.920 / 0.877 / 0.031 / 0.909 / 0.948 and SAM's 0.835 / 0.782 / 0.069 / 0.808 / 0.889. On the full DIS-TE set (2000 images) SAMA scores F_beta^max 0.926, F_beta^w 0.897, MAE 0.026, S_alpha 0.925, E_phi^m 0.956.
• Harder DIS subsets. On DIS-TE4 SAMA reaches F_beta^max 0.917 and MAE 0.041, compared with DIS-SAM's 0.899 / 0.043; the paper notes fully automatic models such as BiRefNet (F_beta^max 0.904 on DIS-TE4) are often trained with significantly more epochs but cannot support interactive segmentation.
• Matting quality. On Composition-1K, SAMA records SAD 22.8 and MSE 2.9; on Distinction-646, SAD 22.4 and MSE 2.2. These are the best values among the trimap-free methods (MFC-Net: 35.6 / 8.7 and 34.5 / 7.8), and the paper describes them as comparable to leading trimap-based approaches such as VITMatte (21.5 / 3.3 and 21.22 / 2.1) without requiring any trimap.
• Point-prompt interactivity. SAMA achieves higher mean IoU than HQ-SAM and SAM on COIFT across all tested prompt configurations, with the paper reporting especially large gains when only 1, 3, or 5 points are given.
• Zero-shot interactive segmentation (COIFT, 280 samples). SAMA scores F_beta^max 0.990, F_beta^w 0.982, MAE 0.004, S_alpha 0.984, E_phi^m 0.993, ahead of DIS-SAM (0.982 / 0.969 / 0.005 / 0.978 / 0.988) and HQ-SAM (0.974 / 0.976 / 0.005 / 0.971 / 0.991).
• Zero-shot semantic matting (AM2K). SAMA reports SAD 8.04 and MSE 0.0030, improving on GFM (11.11 / 0.0031), MatAny (11.9 / 0.0033), SMat (16.84 / 0.0047), and MAM (17.30 / 0.0035) despite not being trained on animal categories.
• Zero-shot portrait matting (P3M-500). SAMA reports SAD 9.08 and MSE 0.0028. PPM-ViTAE is better on this set (SAD 7.80, MSE 0.0017), while PPM scores 13.38 / 0.0042 and MatAny 17.82 / 0.0057.
• Prompt robustness (RefMatte-RW100). With box prompts SAMA is best (SAD 25.69, MSE 0.0100); with noisy boxes it is also best (SAD 27.57, MSE 0.0111); with 10 point prompts it is second-best (SAD 39.2, MSE 0.0121) behind SMat (25.60 / 0.0120). The paper notes SMat performs only third-best on box and noisy-box prompts.
• Component ablations on DIS-VD. Removing both modules gives F_beta^max 0.872 and MAE 0.038; adding only Local-Adapter gives 0.893 / 0.029; adding only MVLE gives 0.882 / 0.027; using both gives 0.942 / 0.021.
• Matting ablations. With neither module, AM2K results are SAD 19.79 / MSE 0.028 and P3M-500 are SAD 16.06 / MSE 0.0346; with both modules these improve to 8.04 / 0.003 and 9.08 / 0.0028, and dropping either module causes a substantial decline.
• Multi-task synergy. Training on segmentation alone yields DIS-VD F_beta^max 0.917 and RefMatte-RW100 SAD 62.70 / MSE 0.054; training on matting alone yields DIS-VD 0.682 and RefMatte-RW100 SAD 34.25 / MSE 0.021; joint training yields DIS-VD 0.942 and RefMatte-RW100 SAD 25.69 / MSE 0.0100, indicating the two tasks help each other.
• Instance matting (HIM2K). On the synthetic subset, SAMA records IMQ mad 57.41, IMQ mse 77.25, IMQ grad 47.26, IMQ conn 59.29, behind InstMatt (63.59 / 78.14 / 64.50 / 67.71) but above MAM. On the natural subset, SAMA leads every listed model with IMQ mad 71.06 and IMQ mse 86.77. The remaining natural-subset rows were truncated in the available content.
• Salient object segmentation (HRSOD). The paper states SAMA consistently outperforms all competing methods, though the corresponding table as provided lists values only for LDF, HRSOD, DHQ, BiRefNet, SAM, HQ-SAM, DIS-SAM, Pi-SAM, and SAM-UQ (SAM-UQ scoring S_m .977, F_beta^x .986, E_phi^m .988, M .005).

Methodology in Plain English

SAMA keeps the entire pre-trained SAM backbone frozen and trains only small new modules, which keeps the approach data-efficient and cheap.

The core idea is to look at the image more than once. The full image is treated as a "global view," and it is also evenly cut into four non-overlapping local patches. Each patch is up-sampled back to the original resolution and pushed through the same frozen image encoder, producing four high-resolution local feature maps alongside the global one. This is the Multi-View Localization Encoder. To align the two, the global features are average-pooled at multiple receptive fields (e.g., 4, 8, 16), split into four spatial regions matching the patches, and used as keys and values in cross-attention where the local features act as queries.

The refined local features are then injected back into SAM's mask decoder through the Local-Adapter. It runs after a decoder layer and has three steps: a first cross-attention that fuses the refined local features with early-layer encoder features via a residual connection; a second cross-attention that swaps keys and values (following the global-local / local-global design of GLIP and GroundingDINO) so information flows both ways; and an output stage that computes a sigmoid-gated confidence map from the decoder output and multiplies it with the attention result, then adds it back to the decoder output. The confidence map is there to preserve SAM's zero-shot generalization and reduce overfitting or catastrophic forgetting.

For the two tasks, SAM's original output token is replaced by two trainable SAMA tokens, one for segmentation and one for matting, which are used separately at training and inference. Two lightweight prediction heads upsample and convolve the adapter output into a high-resolution segmentation mask and alpha matte.

Training freezes SAM entirely, and only one head is optimized at a time: the matting head is frozen while training segmentation, and vice versa. Segmentation uses BCE + IoU + SSIM losses; matting uses L1 + SSIM + gradient + Laplacian losses. Training data combines DIS-5K and ThinObject-5K for segmentation with Adobe Image Matting (AIM) and AIM-500 for matting, chosen over the larger but noisier SA-1B.

Why This Matters

The work argues that segmentation and matting are complementary — segmentation supplies global object cues while matting supplies local boundary precision — and shows a single frozen-SAM backbone can carry both with roughly 1.8% extra parameters. That is a practical alternative to the common pattern of bolting SAM onto a separate heavy matting network or cascading a trimap generator in front of a dedicated matting model.

Real-world applications the setup supports:

• Photo and video editing workflows that need clean cutouts of hair, fur, or semi-transparent objects such as glass.
• E-commerce and catalog imagery, where product masks and alpha mattes are needed at scale from simple box or click prompts.
• Augmented reality and virtual try-on, which require precise, prompt-driven foreground extraction.
• Content moderation, medical image analysis, and autonomous-driving perception pipelines that benefit from flexible prompting combined with fine boundaries.

Industry relevance: The authors are from the Personalization Team at Walmart Global Tech, Sunnyvale, CA, USA, indicating a commercial interest in deployable, data-efficient segmentation and matting tooling. SAMA's ability to accept user prompts while staying trimap-free and cheap to adapt is directly relevant to production systems where annotation budgets and inference cost both matter.

Future Directions

• Runtime optimization. The paper explicitly names runtime optimization as future work, which is relevant because MVLE runs the encoder over the original image plus four up-sampled local patches.
• Video segmentation. Extending SAMA from still images to video is listed as the second explicit future direction.
• Broader zero-shot evaluation. The appendix mentions extended zero-shot evaluations for both segmentation and matting, and the HRSOD comparison table as presented does not show a SAMA column even though the text claims it outperforms all competitors, leaving room for a more complete presentation.
• Matting accuracy under point prompts. SAMA is second to SMat on RefMatte-RW100 point prompts (SAD 39.2 versus 25.60) while leading on box and noisy-box prompts, raising the question of how to close the point-prompt gap.

Target Audience

This paper is best suited to computer vision researchers and engineers already familiar with SAM and its variants (HQ-SAM, DIS-SAM, Pi-SAM, MAM, MatAny), as well as practitioners in image editing, e-commerce imagery, and AR who need interactive, trimap-free matting and fine-grained segmentation in one deployable model. Readers looking for a first introduction to segmentation or matting will need supporting background on SAM's architecture and on standard metrics such as F_beta^max, S_alpha, E_phi^m, MAE, SAD,

Authors’ abstract

Segment Anything (SAM) has recently pushed the boundaries of segmentation by demonstrating zero-shot generalization and flexible prompting after training on over one billion masks. Despite this, its mask prediction accuracy often falls short of the precision required in real-world applications. While several refinement modules have been proposed to boost SAM's segmentation quality, achieving highly accurate object delineation within a single, unified framework remains an open challenge. Furthermore, interactive image matting, which aims to generate fine-grained alpha mattes guided by diverse user hints, has not yet been explored in the context of SAM. Insights from recent studies highlight strong correlations between segmentation and matting, suggesting the feasibility of a unified model capable of both tasks. In this paper, we introduce Segment And Matte Anything (SAMA), a lightweight extension of SAM that delivers high-quality interactive image segmentation and matting with minimal extra parameters. Our Multi-View Localization Encoder (MVLE) captures detailed features from local views, while the Localization Adapter (Local-Adapter) refines mask outputs by recovering subtle boundary details. We also incorporate two prediction heads for each task into the architecture to generate segmentation and matting masks, simultaneously. Trained on a diverse dataset aggregated from publicly available sources, SAMA achieves state-of-the-art performance across multiple segmentation and matting benchmarks, showcasing its adaptability and effectiveness in a wide range of downstream tasks.