Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning Segmentation

AI summary

Overview

• Research area: Computer Vision — text-conditioned video object segmentation, built on Multimodal Large Language Models (MLLMs) and promptable segmentation models (SAM2).
• Technical level: Advanced. Requires familiarity with transformer attention, attention rollout, and video object segmentation benchmarks.
• Scope: Presents a training-free pipeline (DecAF) that turns noisy MLLM attention maps into precise, dense video segmentation masks by decomposing and fusing attention signals across prompts, modalities, and frames.

What This Paper Is About

Multimodal LLMs can answer video questions by attending to relevant visual tokens, but extracting clean object masks from that attention is hard: raw attention is noisy, dominated by "attention sink" regions, and too low-resolution to produce sharp boundaries. This paper shows that by decomposing attention into object-versus-background and video-versus-frame components and fusing them, you can convert an MLLM's internal attention into coarse masks good enough to prompt SAM2 for fine-grained tracking — all without any retraining. The goal is video reasoning segmentation: localizing objects described by queries that require reasoning (e.g., temporal context, world knowledge) rather than simple appearance.

Key Contributions

1. Decomposed Attention Fusion (DecAF): Two complementary fusion mechanisms — contrastive object–background fusion (subtracting a background-focused prompt's attention from an object-focused prompt's attention) and complementary video–frame fusion (averaging globally normalized video attention with per-frame image attention at higher resolution).
2. Vision-aware attention rollout normalization (V-Max): A head-weighting scheme that scores each attention head by the peak attention it assigns to visual tokens, suppressing noisy heads and improving over standard rollout and prior Rollout-Max variants.
3. Training-free attention-guided SAM2 prompting: Point queries derived by thresholding refined attention maps, with NMS-based redundancy removal and a novel attention consistency score that filters spurious masks by checking whether a tracklet consistently overlaps high-attention regions.
4. Broad empirical validation: Evaluated across three MLLM families (LLaVA-OV, InternVL3, Qwen2VL/2.5VL) and five datasets, showing training-free performance that matches or exceeds several training-based methods on reasoning VOS.

Main Findings

• Contrastive fusion is the largest single gain: On Qwen2.5VL, adding object–background contrast raised attention-mask accuracy on Ref-DAVIS from 14.5 to 25.3 and on ReasonVOS from 13.8 to 20.6.
• Video and frame attention are genuinely complementary: Video-only and frame-only inputs scored 65.9 and 67.4 J&F on Ref-DAVIS, while fusing them reached 75.2; similar gains held on ReasonVOS and InternVL3.
• V-Max rollout beats prior rollout variants: 75.2 (Ref-DAVIS) / 63.9 (ReasonVOS) versus 72.9 / 60.9 for Rollout-Max and 68.4 / 56.8 for standard rollout.
• Middle LLM layers are optimal for rollout: Starting at layer 14 of 28 (Qwen2.5VL-7B) outperformed starting at layer 7 or 21; deeper layers improved InternVL3's ReasonVOS but hurt Ref-DAVIS.
• Training-free results rival trained models: With Qwen2.5VL-7B, DecAF reaches 75.2 J&F on Ref-DAVIS (outperforming VISA at 69.4 and VideoLISA at 68.8), 48.1 on MeViS (vs. 44.5 / 44.4), and 63.9 on ReasonVOS — surpassing all training-based methods including Veason-R1 (59.9), even though Veason-R1 additionally trains the same MLLM with RL.
• Reasoning-heavy datasets show the widest margin over Loc-Head: The gap over the prior training-free head-selection method grows on ReVOS Reasoning (49.7 vs. 40.8 with Qwen2.5VL), while Loc-Head retains a slight edge on Ref-DAVIS with InternVL3.
• Attention masks remain coarse even when refined: Contour accuracy (F) lags region similarity (J) on attention-only masks — the opposite of specialized segmentation models — confirming they are best used as prompts, not final outputs.
• The attention consistency score is essential: Removing it and relying only on object scores dropped Ref-DAVIS from 75.2 to 60.0.
• Robustness to thresholds and scale: Performance varies only mildly across τ_pq / τ_trk from 0.7–0.9; larger MLLMs generally perform better (InternVL3 rises from 53.5 at 2B to 63.3 at 14B on Ref-DAVIS).

Methodology in Plain English

The method treats segmentation as a question-answering problem. Given a video and a text query, the MLLM is asked "What is the main object referred to in the expression?" The researchers then trace how visual tokens influence that answer through attention rollout — propagating attention weights across layers to see which image regions the model relied on. To clean this up:

1. Contrastive subtraction. The same model is also asked to describe the background, with the identified object name inserted to explicitly exclude it. Subtracting the background attention from the object attention cancels out regions that get attention regardless of the question (attention sinks), leaving the target object highlighted.
2. Combining video and frame views. The whole rollout is run twice — once on the video and once on each frame at higher resolution — then averaged. Video attention supplies temporal context (useful when the object is temporarily absent); frame attention supplies fine spatial detail.
3. Coarse masks by thresholding. The fused map is thresholded to get rough object masks.
4. SAM2 refinement. Points above a confidence threshold become SAM2 prompts; SAM2 propagates masks through time. Duplicate masks are merged via NMS, and each remaining mask track is scored by how consistently it overlaps the high-attention areas across frames. Only high-scoring tracks are kept and propagated to produce final dense masks across all frames.

Notably, the pipeline uses no learned parameters, produces single- and multi-object outputs naturally, and requires no keyframe selection — just uniform sampling.

Why This Matters

The work demonstrates that MLLM reasoning ability transfers directly to fine-grained localization without any fine-tuning, which lowers the barrier to building video segmentation systems and sidesteps the expense and generalization loss of joint MLLM–SAM training.

Real-world applications:

• Video editing and post-production: Isolating a described subject ("the player who hits the ball") for rotoscoping or background replacement without manual masking.
• Robotics and embodied agents: Language-directed object tracking in egocentric or manipulation video, where the target must be inferred from context rather than named explicitly.
• Video search and content moderation: Retrieving or flagging segments matching complex natural-language criteria rather than keyword tags.
• Medical and sports video analysis: Tracking anatomies or players described by relational or temporal descriptions that simple appearance matching would fail on.

Industry relevance: Because the method works on top of publicly released MLLM checkpoints and SAM2 without retraining, it is attractive for teams that want to add language-conditioned segmentation to products without maintaining a separate trained segmentation model or collecting annotated video data. The finding that scaling the MLLM improves accuracy suggests a straightforward quality roadmap.

Future Directions

• Extending beyond vision: The authors note DecAF is modality-agnostic and could apply to localization in audio or other modalities with emerging Omni-MLLMs.
• Improving attention map resolution: Since contour accuracy is the main weakness, closing the granularity gap between attention maps and true segmentation is the clearest next step.
• Smarter frame selection: The current uniform sampling is a deliberate simplification; replacing it with keyframe selection (as used by top training-based competitors) could improve temporal coverage.
• Removing remaining prompt engineering: The method still relies on hand-designed object and background prompt templates; automated prompt construction or learned prompt selection could improve robustness across MLLMs and query types.

Target Audience

Researchers and engineers working on video understanding, multimodal LLMs, promptable segmentation, or training-free grounding methods. It is most valuable to readers already comfortable with transformer attention mechanics and VOS benchmarks, though the core idea — decompose and fuse attention rather than train — is accessible to anyone building language-conditioned vision systems.

Authors’ abstract

Multimodal large language models (MLLMs) demonstrate strong video understanding by attending to visual tokens relevant to textual queries. To directly adapt this for localization in a training-free manner, we cast video reasoning segmentation as a video QA task and extract attention maps via rollout mechanism. However, raw attention maps are noisy and poorly aligned with object regions. We propose Decomposed Attention Fusion (DecAF), which refines these maps through two mechanisms: (1) contrastive object-background fusion and (2) complementary video-frame fusion. This method suppresses irrelevant activations and enhances object-focused cues, enabling direct conversion of attention maps into coarse segmentation masks. In addition, we introduce attention-guided SAM2 prompting for obtaining fine-grained masks. Unlike existing methods that jointly train MLLMs with SAM, our method operates entirely without retraining. DecAF outperforms training-free methods and achieves performance comparable to training-based methods on both referring and reasoning VOS benchmarks.