LVMT: Video Mask Transformer for Long-term Video Segmentation

AI summary

Overview

Research area: Computer vision, specifically online video segmentation — video instance segmentation (VIS), video panoptic segmentation (VPS), and video semantic segmentation (VSS).

Technical level: Intermediate. The paper builds directly on Vision Transformer-based segmentation models (EoMT, VidEoMT, PMT) and assumes familiarity with query-based segmentation, recurrent units, and backpropagation-through-time concepts.

Scope: The paper proposes a lightweight GRU-based temporal propagation module and a Truncated Query Propagation (TQP) training strategy, combined into the Long-term Video Mask Transformer (LVMT), and evaluates them on six video segmentation benchmarks against prior state-of-the-art models.

What This Paper Is About

Existing efficient online video segmentation models track objects by repeatedly adding the previous frame's queries to a fixed set of learnable queries. In long videos with long-term occlusions, this causes object information to be diluted, producing identity switches when objects reappear. The paper's goal is to make temporal propagation adaptive — letting the model decide what to keep in memory — and to enable training on longer videos without memory blow-ups or vanishing gradients, while preserving the speed of the efficient models it is based on.

Key Contributions

1. A lightweight GRU-based temporal propagation mechanism for video segmentation. It replaces the additive query fusion of VidEoMT and PMT with a Gated Recurrent Unit cell, where the learnable queries initialize the hidden state and the hidden state is adaptively updated using the previous frame's output queries. It captures long-range context without growing memory usage over time and with negligible parameter, GFLOPs, and speed overhead.

2. Truncated Query Propagation (TQP), a training strategy that adapts Truncated Backpropagation Through Time (TBPTT) with across-chunk gradient accumulation to query propagation. Videos are split into chunks; hidden states are detached and propagated forward across chunk boundaries during the forward pass, while backpropagation is bounded within each chunk, enabling long-video training with bounded peak memory and improved gradient stability.

3. LVMT, the resulting model, which achieves a stronger performance-versus-latency trade-off than prior methods. It is reported as 10× faster than the prior state of the art (DVIS-DAQ) while surpassing it.

4. Extensive evaluation across six benchmarks (OVIS, YouTube-VIS 2019, 2021, and 2022, VIPSeg, and VSPW) covering VIS, VPS, and VSS, including specialized tracking metrics (IDF1, AssA, MT, ML, IDS) and ablations on memory choice and training clip length.

Main Findings

• GRU propagation improves accuracy at almost no cost: Replacing PMT's query fusion with GRU-based propagation raised mean AP on OVIS from 51.8 ± 0.3 to 54.2 ± 0.4 (+2.4 AP), changing parameters from 358M to 363M, GFLOPs from 1014 to 1015, and FPS from 97 to 95.

• Naively training on longer clips hurts: Increasing the training clip length from T_train = 5 to T_train = 10 reduced AP by 1.9 points compared to the GRU-only setting (52.3 ± 0.3 vs. 54.2 ± 0.4), attributed to vanishing gradients; T_train = 10 was the longest feasible setting without out-of-memory errors.

• TQP recovers and exceeds the loss: Applying TQP at T_train = 10 yielded 56.1 ± 0.3 AP, a +3.8 AP boost over naive longer-clip training at the same clip length.

• Final LVMT configuration: With GRU plus TQP at T_train = 15, LVMT reached 56.4 ± 0.4 mean AP on OVIS — +4.6 over PMT — at 363M parameters, 1015 GFLOPs, and 95 FPS. Across ViT-L, ViT-B, and ViT-S backbones the improvements over PMT were +4.6, +5.7, and +7.8 AP respectively (PMT 51.8 ± 0.3 / 42.7 ± 0.5 / 31.5 ± 0.4 vs. LVMT 56.4 ± 0.4 / 48.4 ± 0.2 / 39.3 ± 0.3), all measured over five runs.

• State of the art on OVIS and YouTube-VIS 2022: With a frozen ViT-L encoder and DINOv3, LVMT reached 56.7 AP on OVIS (61.8 AP75, 61.6 AR10, 1015 GFLOPs, 95 FPS) and 51.5 AP^L on YouTube-VIS 2022 (58.3 AP75^L, 55.4 AR10^L, 618 GFLOPs, 121 FPS). With DINOv2: 55.6 AP on OVIS and 48.4 AP^L on YouTube-VIS 2022.

• Better than both frozen-encoder and fine-tuned competitors: LVMT exceeds CAVIS with DINOv3 (frozen encoder) by +3.3 AP on OVIS and +9.3 AP on YouTube-VIS 2022 while being much faster. It beats DVIS-DAQ (which fine-tunes the encoder) by +2.4 AP on OVIS and LOMM by +3.3 AP on YouTube-VIS 2022, while being over 10× faster than both. Against VidEoMT, LVMT gains +4.2 AP on OVIS and +8.7 AP on YouTube-VIS 2022 while being only slightly slower. On YouTube-VIS 2019 and 2021 similar trends hold with smaller absolute differences.

• Fewer identity switches: On OVIS, LVMT reduced total identity switches from 2,755 (PMT) to 2,001 — about 27% — and improved IDF1 (81.8 vs. 78.9), AssA (78.4 vs. 73.3), MT (78.8 vs. 75.7), and ML (5.5 vs. 6.3). CAVIS scored IDF1 79.2, AssA 74.7, MT 76.3, ML 6.1, 2,683 IDS; VidEoMT scored 78.7, 73.3, 75.5, 6.2, 2,801.

• State of the art on VIPSeg (VPS): With DINOv3, LVMT reached 60.3 VPQ and 53.4 STQ (2038 GFLOPs, 57 FPS), improving over DVIS-DAQ by +2.9 VPQ. With DINOv2, LVMT reached 56.6 VPQ and 51.3 STQ — a 0.8 VPQ gap with DVIS-DAQ while being over 10× faster. PMT scored 55.5 VPQ / 49.2 STQ (DINOv3) and 55.3 / 48.2 (DINOv2).

• State of the art on VSPW (VSS): With DINOv3, LVMT reached 95.3 mVC16 and 66.4 mIoU (2050 GFLOPs, 57 FPS), surpassing VidEoMT by +0.9 mVC and +2.4 mIoU and improving over PMT by +0.4 mVC and +0.7 mIoU at nearly identical FPS.

• GRU is the best memory mechanism: Among LSTM (56.0 AP, 93 FPS), S5 (55.8 AP, 96 FPS), RVM's DeepGRU (56.5 AP, 91 FPS), an explicit memory bank following GenVIS (54.5 AP, 86 FPS), and GRU (56.7 AP, 95 FPS), GRU gave the best trade-off — +0.7 AP over LSTM and +2.2 AP over the memory bank.

• GRU and TQP are complementary: At T_train = 10, TQP-only scored 54.5 AP versus 56.5 AP with both GRU and TQP. Naive long-clip training without GRU also degraded (51.1 vs. 52.6 AP at 10 frames), indicating vanishing gradients are not unique to the GRU. Without TQP, clips of 15 frames or longer produced out-of-memory errors; extending T_train from 15 to 20 frames added only +0.1 AP.

• Identity-switch analysis: Videos with the most identity switches contained about 3× more objects per video, 5× more disappearance/reappearance cases, and mean occlusion lengths 2× longer. The mean occlusion length was 13 frames, about 2.6× longer than the training clips.

Methodology in Plain English

The authors start from PMT, an efficient video segmentation model that keeps a frozen Vision Transformer encoder and uses a lightweight decoder plus a simple query-propagation rule: at each frame, the previous frame's output queries are linearly projected and added to a fixed set of learnable queries.

Their first change is to replace that addition with a GRU cell. Each tracked object corresponds to one query slot; the learnable queries serve as the initial hidden state, and at every frame the GRU takes the previous frame's segmentation queries and the current hidden state to produce the queries for the next frame. Because a GRU has gates, the model can learn what to retain and what to discard, so it can hold onto information about an object that has disappeared and use it to re-identify the object later. The hidden state has a fixed size, so memory and computation do not grow with video length — unlike an external memory bank that stores every query ever produced.

Their second change addresses training. The model is trained on short clips by default, but the occlusions it must handle at test time are longer than those clips. Simply training on longer clips caused both out-of-memory failures and worse accuracy, which they attribute to gradients vanishing across many recurrent steps. TQP instead divides a training clip of T frames into M = ⌈T/F⌉ chunks of at most F frames, processes the chunks in order within one iteration, and updates the optimizer once after all chunks using the average of the per-chunk gradients. The final hidden state of each chunk is detached from the computational graph and passed to the next chunk as its initial state, so temporal information still flows forward while gradients stay bounded within a chunk.

The default configuration uses a frozen ViT-L encoder initialized with DINOv3, the AdamW optimizer with mixed precision, chunks of F = 5 frames, an overall temporal window of T = 15 frames, and M = 3 chunks. Matching follows standard practice: each object is matched to a query in the frame it first appears and the assignment is kept for subsequent frames. Batch size, learning rate, scheduler, video resolutions, and iteration counts follow PMT and prior work. Speed is measured at batch size 1 on a single NVIDIA H100 GPU with FlashAttention v2 and torch.compile, and GFLOPs are computed with fvcore. The authors also analyze failure cases, gradient flow, memory retention, encoder freezing, and alternative memory mechanisms in the appendix.

Why This Matters

The paper shows that long-horizon temporal consistency in video segmentation can be improved through a small architectural change and a training-schedule change, without paying for it in inference speed — the central tension in this line of work. It also supplies diagnostic evidence about why identity switches happen (many objects, frequent disappearances, occlusion spans exceeding

Authors’ abstract

Existing online video segmentation methods struggle to track objects in long, complex videos with long-term occlusions. We hypothesize that this limitation is caused by (i) the inability of their temporal propagation mechanism to adaptively select the object information that is propagated across time, and (ii) their inability to be trained on long videos due to memory requirements and vanishing gradients. To address the first limitation, we propose to use a lightweight GRU-based temporal propagation module that can learn to select which information it keeps in memory and propagates across time. Second, to allow training on long videos, we introduce Truncated Query Propagation (TQP), a training strategy in which the model processes a video in chunks of frames, where information about tracked objects is propagated between chunks but backpropagation is only conducted in individual chunks, enabling longer temporal supervision without out-of-memory issues, inference overhead, or vanishing gradients. The resulting model is called the Long-term Video Mask Transformer (LVMT). Extensive experiments on six benchmarks show that LVMT sets a new state of the art across a range of video segmentation tasks, while retaining the speed of the highly efficient model it is based on, making it 10X faster than the prior state of the art. Code: https://www.tue-mps.org/lvmt