MATRIX: Mask Track Alignment for Interaction-aware Video Generation

AI summary

Overview

Research area: Computer vision and generative AI — specifically text-to-video generation with diffusion transformers (DiTs), and human-object interaction (HOI) modeling.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, attention mechanisms, LoRA finetuning, and video segmentation/tracking pipelines.

Scope: The paper analyzes how video diffusion transformers internally represent subject-object interactions, curates a dataset to support that analysis, and proposes an attention-alignment regularization plus an evaluation protocol for interaction-aware video generation.

What This Paper Is About

Video diffusion transformers can generate single objects or humans well, but they fail at multi-instance or subject-object interactions — getting "who does what to whom" wrong, or losing track of which instance is which over time. The authors ask how these models internally represent interactions, and then use the answer to build a method that makes interaction generation more faithful and more temporally consistent.

Key Contributions

1. MATRIX-11K dataset. A dataset of 11K videos paired with interaction-aware captions and multi-instance mask tracks, where each instance keeps a persistent ID across the video and the union of subject and object masks defines the "interaction region." The curation pipeline combines LLM-based caption processing, GroundingDINO box proposals, VLM verification, SAM2 mask propagation, and human filtering.

2. The first systematic analysis of interaction representations in video DiTs. The authors formalize two perspectives — semantic grounding (via video-to-text attention, whether noun tokens localize to subject/object regions and verb tokens attend to their union) and semantic propagation (via video-to-video attention, whether those bindings persist across frames) — and show both concentrate in a small subset of "interaction-dominant layers."

3. The MATRIX framework. A regularization that aligns attention in interaction-dominant layers with ground-truth mask tracks, via a Semantic Grounding Alignment (SGA) loss and a Semantic Propagation Alignment (SPA) loss, applied to a LoRA-finetuned image-to-video model conditioned on a first-frame multi-instance ID map.

4. InterGenEval. An interaction-aware evaluation protocol with key interaction semantic alignment (KISA), semantic grounding integrity (SGI), semantic propagation integrity (SPI), and an overall interaction fidelity (IF) score.

Main Findings

• Both effects live in a few layers. Using CogVideoX-5B-I2V, the authors compute attention alignment scores across 42 layers and 50 denoising timesteps. Noun grounding is described as aligning in layers such as layer 11, verb grounding in layers such as layer 7, and propagation in layers such as layer 12 — examples shown in the layer visualizations. Layers are ranked by frequency of appearing in the top-10 per video and by mean AAS; the top-10 by rank-sum are called "influential." A layer is called "interaction-dominant" when its success gap is large and positive and its failure gap is large and negative.

• Attention alignment tracks success and failure. In successful generations, alignment is consistently stronger; in failures, it is weaker, yielding a clear success-failure contrast. A sanity check applying perturbation guidance to these layers sharpened attention toward instance regions and improved interaction fidelity slightly.

• MATRIX beats prior baselines on interaction metrics. On InterGenEval, MATRIX scores KISA 0.546, SGI 0.641, IF 0.593, versus CogVideoX-5B-I2V at 0.406 / 0.491 / 0.449, CogVideoX-2B-I2V at 0.420 / 0.470 / 0.445, Open-Sora-11B-I2V at 0.453 / 0.508 / 0.480, and TaVid at 0.465 / 0.522 / 0.494.

• MATRIX also leads on human fidelity and video quality in the main comparison. HA (Human Anatomy, from VBench2.0) is 0.954 for MATRIX, versus 0.937 (2B), 0.936 (5B), 0.891 (Open-Sora-11B-I2V) and 0.917 (TaVid). MS (Motion Smoothness) is 0.994 and IQ (Image Quality) is 69.73, the highest in that comparison table.

• Finetuning on the data alone helps only moderately. Ablation (III), LoRA finetuning with MATRIX-11K and no layer selection or auxiliary losses, reaches KISA 0.445 / SGI 0.526 / IF 0.486, above the untuned baseline (I) but below the full method.

• SPA alone trades off grounding. Adding SPA (IV) raises IF to 0.496 and gives the highest IQ (70.26) and MS (0.995) among the ablations, but without explicit grounding the paper reports a trade-off that weakens noun/verb alignment.

• SGA is more stable on video-to-text attention than on text-to-video. SGA applied on A^t2v (V) reaches KISA 0.486 / SGI 0.578 / IF 0.531; SGA applied on A^v2t (VI) reaches 0.509 / 0.592 / 0.550. The paper attributes the difference to a text token mapping to multiple locations, making text-to-video supervision unstable.

• Combining both losses is best. Configuration (VII), SGA plus SPA, gives the strongest interaction fidelity (0.546 / 0.641 / 0.593), the best HA (0.954), and improved quality over the baselines.

• The framework transfers to other backbones. Applying MATRIX to Wan2.1-14B-I2V improved interaction fidelity while preserving visual quality; Appendix C extends the layer analysis to HunyuanVideo-I2V and Wan2.1-14B-I2V.

Methodology in Plain English

The authors start by describing interactions formally as two things: a token-level binding (the word "man" should point at the man's pixels, and the verb should point at the area covering both the man and the object) and a temporal binding (that same pointing should stay stable frame after frame). They then read the attention matrices inside a video DiT as heatmaps and score how much attention lands on the ground-truth mask locations, calling this the attention alignment score.

Because no dataset had the right reference data, they built one. An LLM reads captions and extracts interaction triplets of the form (subject ID, verb, object ID), scores each interaction for Contactness and Dynamism on a 1-to-5 scale, keeps only interactions above thresholds, and writes a short appearance description for each instance. Then GroundingDINO proposes boxes for each ID, a VLM checks each candidate against the class label and description until one verifies, and SAM2 propagates the verified box into a full mask track. Human annotators clean up remaining errors.

To train, they take CogVideoX-5B-I2V, add a LoRA adapter, and feed it the first RGB frame plus a first-frame ID map whose colors stay fixed per instance across the clip. A small causal decoder maps attention up to pixel resolution so it can be compared directly against the mask tracks. The SGA loss pushes video-to-text attention onto subject, object, and union regions; the SPA loss pushes video-to-video attention to keep each instance's own track. Both use a weighted combination of BCE, soft Dice, and L2 terms. Only the interaction-dominant layers found in the analysis get these losses; everything else in the backbone stays frozen.

Why This Matters

Impact on research. The paper shifts analysis of video generation from output quality to internal mechanism, offering a reusable vocabulary (semantic grounding, semantic propagation, attention alignment score, interaction-dominant layers) and a layer-selection method. Its finding that alignment is sparse and layer-specific suggests future work should target specific layers rather than the whole network. The dataset and protocol also give the field a way to measure interaction fidelity instead of relying on global text-video alignment scores.

Real-world applications:

• Simulation and synthetic data generation, where a simulator must produce correct who-does-what-to-whom scene dynamics rather than plausible-looking motion.
• AR/VR content pipelines, where multiple distinct instances must remain consistent across a generated clip.
• Robotics and embodied reasoning, where generated video can serve as training or planning signal and object identity drift would corrupt it.
• Creative and advertising video production, where a user supplies a first-frame mask of specific subjects and objects and expects those exact instances to act out a requested interaction.

Industry relevance. The method is a plug-and-play adapter with small architectural changes, demonstrated on both CogVideoX-5B-I2V and Wan2.1-14B-I2V — attractive for teams that already have a video model and want to improve interaction control without retraining from scratch. The reliance on off-the-shelf segmentors at inference keeps user control practical.

Future Directions

• Scaling the analysis. The main analysis covers CogVideoX-5B-I2V; broader conclusions across many architectures would require extending the layer-dominance methodology to more backbones than the two analyzed in the appendix.
• Generalizing beyond masks. The core signal is instance mask tracks, which requires segmentation quality. Whether optical flow or depth could substitute or complement the tracks is left open.
• Beyond the closed verb set. The paper critiques relation-customization work for depending on a closed verb set; whether MATRIX handles fully open-vocabulary verbs robustly is not established in the reported results.
• Removing the causal decoder. The lightweight decoder bridges attention space and pixel space; alternatives that avoid this component could simplify the pipeline.

The paper states codes and weights will be released, and the dataset and curation pipeline will be made public. Limitations are discussed in Appendix I and future directions in Appendix J, but those appendix contents are not included in the provided text.

Target Audience

Researchers and graduate students working on video diffusion models, controllable generation, or human-object interaction synthesis — particularly those interested in mechanistic interpretability of generative models. Practitioners building video generation products who need reliable multi-instance control will also benefit from the framework and evaluation protocol, though the dense attention analysis and dataset curation details demand prior familiarity with transformer internals and segmentation pipelines.

Authors’ abstract

Video DiTs have advanced video generation, yet they still struggle to model multi-instance or subject-object interactions. This raises a key question: How do these models internally represent interactions? To answer this, we curate MATRIX-11K, a video dataset with interaction-aware captions and multi-instance mask tracks. Using this dataset, we conduct a systematic analysis that formalizes two perspectives of video DiTs: semantic grounding, via video-to-text attention, which evaluates whether noun and verb tokens capture instances and their relations; and semantic propagation, via video-to-video attention, which assesses whether instance bindings persist across frames. We find both effects concentrate in a small subset of interaction-dominant layers. Motivated by this, we introduce MATRIX, a simple and effective regularization that aligns attention in specific layers of video DiTs with multi-instance mask tracks from the MATRIX-11K dataset, enhancing both grounding and propagation. We further propose InterGenEval, an evaluation protocol for interaction-aware video generation. In experiments, MATRIX improves both interaction fidelity and semantic alignment while reducing drift and hallucination. Extensive ablations validate our design choices. Codes and weights will be released.