MUFASA: A Multi-Layer Framework for Slot Attention

AI summary

Overview

Research area: Computer vision, specifically unsupervised object-centric learning (OCL) and unsupervised object segmentation (UOS) with slot-attention models built on pre-trained vision transformers.

Technical level: Advanced. The paper assumes familiarity with vision transformers, slot attention, DINO-style self-supervised encoders, Hungarian matching, and auto-regressive transformer decoders.

Scope: The paper introduces MUFASA, a plug-and-play multi-layer slot-attention framework that draws features from several layers of a pre-trained DINO ViT instead of only the final layer, and demonstrates improved segmentation accuracy and faster training when integrated into the DINOSAUR and SPOT models.

What This Paper Is About

Slot-attention methods such as DINOSAUR and SPOT decompose an image into a small set of latent vectors ("slots"), each intended to bind to one object, but they read features only from the last layer of a pre-trained DINO ViT. The authors argue that semantically useful information is spread across multiple encoder layers, so discarding all but the last one wastes signal. MUFASA computes slot attention on several DINO layers, aligns the resulting slots across layers, and fuses them into a single object-centric representation that is passed to the decoder.

Key Contributions

1. A multi-layer slot-attention framework. Slot attention is run independently on a subset of DINO ViT feature layers, with separate trainable parameters per layer rather than shared weights, so each module can adapt to layer-specific features.
2. M-Fusion, a slot and mask fusion technique. Slots from adjacent, Hungarian-matched layers are summed pairwise in a sliding-window fashion, concatenated along the feature dimension, and projected by an MLP into one fused set of slots; attention masks are fused by a weighted linear combination.
3. Plug-and-play integration. MUFASA replaces the single-layer slot-attention bottleneck in existing DINO-based slot-attention models without adding any new loss terms, producing DINOSAUR-M and SPOT-M, and is trained purely with the base model's own training signal.
4. New state-of-the-art results with efficiency gains. Integrating MUFASA improves segmentation across PASCAL VOC, COCO, and MOVi-C while reducing training time and adding only modest parameters and inference overhead.

Main Findings

• Broad segmentation gains on PASCAL VOC. SPOT-M reaches 59.8 mBO^c, 51.3 mBO^i, 49.4 mIoU, and 20.6 FG-ARI, compared with 55.3, 48.1, 46.5, and 19.7 for SPOT. DINOSAUR-M reaches 57.6, 49.2, 47.2, and 25.2, versus 51.2, 44.0, no reported mIoU, and 24.8 for DINOSAUR.
• Class-level mBO shows the largest improvements. The paper states that the most substantial gains across datasets appear in class-level mBO (mBO^c).
• COCO results are close but mixed. SPOT-M scores 45.5 mBO^c, 34.8 mBO^i, 32.5 mIoU, and 35.6 FG-ARI against SPOT's 44.3, 34.7, 32.7, and 37.8. The paper describes a single marginal exception in mIoU on COCO. DINOSAUR-M improves on DINOSAUR in every COCO metric: 43.0 versus 39.7 mBO^c, 32.7 versus 31.6 mBO^i, 30.5 mIoU where DINOSAUR's mIoU is not reported, and 33.9 versus 34.1 FG-ARI.
• MOVi-C gains are large. SPOT-M reaches 49.2 mBO^i, 48.2 mIoU, and 67.8 FG-ARI versus SPOT's 47.0, 46.4, and 57.9. DINOSAUR-M reaches 49.2, 48.3, and 66.4 versus DINOSAUR's 42.4, no reported mIoU, and 55.7. Class-level mBO is not applicable to MOVi-C because the required annotations are not provided.
• DINOSAUR-M beats the previous state of the art on two of three datasets. Even without SPOT's teacher-student training, DINOSAUR-M surpasses the previous state of the art on PASCAL VOC and MOVi-C.
• Modest parameter growth. DINOSAUR-M has 49.6 M trainable parameters, 20.7% more than DINOSAUR's 41.1 M. SPOT-M has 77.8 M, a 12.1% increase over SPOT's 69.5 M.
• Small memory overhead. Average GPU memory during training rises 8.1% for DINOSAUR-M and 0.4% for SPOT-M relative to their baselines.
• Faster convergence and training. On VOC, training time is reduced by 94.4% for SPOT-M and 90.2% for DINOSAUR-M. DINOSAUR-M reaches baseline-level results in 42 epochs versus DINOSAUR's 644, and peak results in 271 epochs versus 644. SPOT-M reaches baseline-level in 51 epochs versus SPOT's 944, and peak in 615 versus 944.
• Inference throughput changes are small. Throughput drops from 271.1 to 217.3 img s^-1 for DINOSAUR and from 86.1 to 84.7 img s^-1 for SPOT.
• Four layers is the sweet spot. Ablations with the last three, four, and five consecutive layers peak at four; adding more layers slightly deteriorates results. Combining earlier with later layers beats the baseline but not the last-consecutive-layer selection.
• M-Fusion outperforms alternative fusion strategies on VOC. M-Fusion scores 59.8 mBO^c, 51.3 mBO^i, 49.4 mIoU, 20.6 FG-ARI, ahead of Concat-Fusion (59.0, 50.9, 48.9, 20.0), T-Fusion (59.0, 50.7, 48.9, 19.7), and Avg-Fusion (55.6, 48.1, 46.5, 19.4), with SPOT at 55.3, 48.1, 46.5, 19.7.
• The approach generalizes across encoders. With MAE ViT-B/16, DINOSAUR-M scores 55.0 mBO^c versus DINOSAUR's 48.8, and SPOT-M scores 55.8 versus SPOT's 54.5. With DINOv2 ViT-B/14, DINOSAUR-M reaches 52.1 versus 49.5, and SPOT-M 51.0 versus 49.4. With DINO ViT-S/8, DINOSAUR-M reaches 55.2 versus 51.2, and SPOT-M 60.0 versus 55.3.
• Weaker decoders hurt the baselines more. On COCO with an MLP decoder instead of a transformer decoder, DINOSAUR drops from 39.7 to 30.9 mBO^c and from 31.6 to 27.7 mBO^i, values the paper attributes to a sharper decline than MUFASA experiences.
• Fused masks compensate for per-layer noise. Layer-wise attention visualizations show that segmentation noise appears at various layers rather than being confined to early or late ones, and the fused mask is more accurate than any individual layer mask.
• Layers play complementary roles. The paper reports that layer 10 merges the dog and the person into one slot, while later layers separate them correctly but introduce background noise that the fused mask mitigates.

Methodology in Plain English

Slot-attention models take an image, run it through a frozen pre-trained DINO vision transformer, and then let a set of latent "slots" compete through attention to claim image patches. Only the transformer's final layer normally feeds this process. MUFASA instead grabs features from several layers at once — the authors settle on the last four consecutive layers — and runs a separate slot-attention module on each, each with its own learnable weights.

Because different layers may assign the same object to slots in a different order, the authors use Hungarian matching, driven by mean Intersection-over-Union between binarized attention masks, to line up the slots so that the same index refers to the same object across layers. They then fuse the aligned slots by adding adjacent-layer pairs together in sliding-window fashion, concatenating the results, and passing them through a one-hidden-layer MLP with 768 units and GELU activations. Attention masks are fused by a weighted sum that is uniform for DINOSAUR-M and learned, with softmax normalization over the layer dimension, for SPOT-M where teacher-student distillation guides it. The fused slots go into an auto-regressive transformer decoder that reconstructs the final encoder layer's features, and training uses only the base model's existing reconstruction loss. Slot counts are dataset-specific: 6 for VOC, 7 for COCO, and 11 for MOVi-C, with a ViT-B/16 backbone initialized from DINO weights and training for 1120, 100, and 95 epochs respectively.

Why This Matters

The work shows that a well-known design habit in object-centric learning — reading only the final transformer layer — leaves useful information on the table, and that a small architectural change can improve both accuracy and training cost. It also indicates that slot-attention models are less dependent on a powerful decoder than previously assumed, since MUFASA improves the slot-attention mechanism itself.

Real-world applications:

• Robotic perception and manipulation, where a robot must split a cluttered scene into distinct objects without labeled data.
• World models and reinforcement learning, which use object-centric representations to predict how scenes evolve.
• Video and image editing, where per-object masks enable selective manipulation of individual entities.
• Explainability and compositional reasoning, where structured scene decomposition supports interpreting what a model sees.

Industry relevance: the reported reductions in training time — 94.4% for SPOT-M and 90.2% for DINOSAUR-M on VOC, with convergence in a fraction of the baseline epochs — matter for large-scale training budgets, and the framework's plug-and-play nature means existing DINO-based pipelines can adopt it without redesigning their losses. The modest parameter increases (20.7% and 12.1%) and small memory overheads (8.1% and 0.4%) keep the cost of adoption low.

Future Directions

• Beyond the last four layers. The ablations show that earlier layers alone give weak segmentation and that combining earlier with later layers helps but not as much as consecutive deep layers. Which layer combinations would work best for other backbones or domains remains open.
• Layer count and selection as a learned decision. Results deteriorate slightly beyond four layers, so automatically choosing how many and which layers to use, rather than fixing them, is a natural extension.
• Connections to other multi-layer ViT work. The paper notes that multi-layer ViT features have been used for multi-modal tasks, feature forecasting, visual correspondence, and object discovery, but their integration into slot attention was previously unexplored. Extending MUFASA into those settings is implied.
• Decoder dependence. The authors observe that SPOT's patch-order permutation strategy increases its reliance on the decoder, while MUFASA is more robust to weaker decoders. Whether MUFASA could enable simpler or cheaper decoder designs is not resolved.

Target Audience

Researchers and graduate students working on unsupervised object-centric learning, slot-attention architectures, and self-supervised vision transformer features. It is also relevant to practitioners building object-segmentation pipelines on DINO-family encoders who want accuracy and training-efficiency gains with a lightweight change, and to readers interested in how information is distributed across transformer layers.

Note on source completeness: the provided paper content is truncated partway through Table 5 (the decoder ablation on COCO), so the remaining rows of that table and any later sections are not reflected above. The project page and code are listed at https://visinf.github.io/mufasa/.

Authors’ abstract

Unsupervised object-centric learning (OCL) decomposes visual scenes into distinct entities. Slot attention is a popular approach that represents individual objects as latent vectors, called slots. Current methods obtain these slot representations solely from the last layer of a pre-trained vision transformer (ViT), ignoring valuable, semantically rich information encoded across the other layers. To better utilize this latent semantic information, we introduce MUFASA, a lightweight plug-and-play framework for slot-attention-based approaches to unsupervised object segmentation. Our model computes slot attention across multiple feature layers of the ViT encoder, fully leveraging their semantic richness. We propose a fusion strategy to aggregate slots obtained on multiple layers into a unified object-centric representation. Integrating MUFASA into existing OCL methods improves their segmentation results across multiple datasets, setting a new state of the art while simultaneously improving training convergence with only minor inference overhead.