MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation

AI summary

Overview

Research area: Computer Vision and multi-modal video understanding, specifically referring video object segmentation, audio-guided video object segmentation, referring multi-object tracking, and referring motion expression generation.

Technical level: Advanced. The paper assumes familiarity with Transformer-based segmentation architectures, vision-language pre-training, large language models, and video object segmentation benchmarks.

One-sentence scope: The paper introduces MeViS (Motion expressions Video Segmentation), a large-scale multi-modal dataset of motion-focused referring expressions in text and audio, benchmarks 15 existing methods on four tasks, and proposes a baseline method LMPM++ that achieves new state-of-the-art results on the perception tasks.

What This Paper Is About

Existing referring video segmentation datasets tend to feature isolated, salient objects whose identities can be pinned down from static attributes such as color or shape, so the target can often be found in a single frame and motion plays little role. The authors argue this underuses what video actually offers: temporal motion. They build MeViS, a dataset of 2,006 complex videos annotated with motion expressions in both text and audio, where the target object or objects can only be identified by watching how things move over time.

Key Contributions

1. The MeViSv2 dataset. A large-scale multi-modal referring motion expression video segmentation dataset containing 2,006 videos, 8,171 objects, and 33,072 human-annotated motion expressions, provided in text and audio formats. It is described as the largest in the field of referring video.
2. Support for four tasks. The dataset supports referring video object segmentation (RVOS), audio-guided video object segmentation (AVOS), referring multi-object tracking (RMOT), and the newly introduced referring motion expression generation (RMEG).
3. A benchmark of 15 methods. The authors evaluate 15 existing methods across the four tasks: 6 RVOS methods, 3 AVOS methods, 2 RMOT methods, and 4 video captioning methods.
4. The LMPM++ baseline. A Language-guided Motion Perception and Matching approach for RVOS/AVOS/RMOT that uses language-conditional queries and object embeddings fed to a large language model, plus a temporal-level contrastive loss, and is reported to achieve new state-of-the-art results.

Main Findings

• Motion expressions cannot be resolved from a single frame. Expressions in MeViS deliberately prioritize motion and action clues over static attributes, so target objects generally require temporal reasoning across the video rather than recognition from one frame. Samples whose targets could be identified from a single frame were rejected during validation.
• Videos and objects are more complex than prior benchmarks. MeViS averages 4.28 objects per video, which the authors state is more than twice the number in the largest previous dataset, Refer-YouTube-VOS. Videos average 13.16 seconds versus 5.45 seconds for Refer-YouTube-VOS, and target objects average 10.88 seconds, more than two times longer than in Refer-YouTube-VOS.
• Scale is larger than prior referring video datasets. MeViS has 8,171 objects versus 7,451, 33,072 expressions versus 15,009, and 443k annotation masks versus 131k, all compared with Refer-YouTube-VOS.
• The MeViSv2 update adds difficulty beyond the conference version. MeViSv1 contained 28,570 expressions; MeViSv2 adds 4,502 new expressions for a total of 33,072. These include 999 motion reasoning expressions and 3,503 no-target expressions, the latter being deceptive descriptions that describe motion but refer to no object in the video.
• Expressions can refer to arbitrary numbers of objects. Rather than the prior "one expression, one object" rule, MeViS supports single-target, multi-target, and no-target expressions. The table reports an average of 1.58 target objects per expression for MeViSv2 (1.59 for MeViSv1), with 21,541 single-target, 8,028 multi-target, and 3,503 no-target expressions.
• Audio expressions are added. MeViSv2 provides over 150,000 seconds of audio for the expressions, mixing automatic synthesis (six TTS models and three public TTS services) and human recordings from 10 speakers reading at 100-150 words per minute, recorded at a sampling rate above 44.1KHz.
• MeViS is also positioned as the largest RMOT dataset. It offers 2,006 videos, 136,102 frames, and 33,072 expressions, compared with Refer-KITTI's 18 videos, 6,650 frames, and 818 expressions.
• Existing methods fall short. The paper states the results demonstrate weaknesses and limitations of existing methods at motion expression-guided video understanding. Specific per-method benchmark scores are not included in the available text of the paper.
• LMPM++ sets new state of the art on RVOS/AVOS/RMOT. LMPM++ is reported to achieve new state-of-the-art results on these perception tasks. Because it feeds object tokens instead of frame features into the language model, it can process much longer sequences, for example 200 frames compared with 3 or 13 frames in previous methods.

Methodology in Plain English

The authors started from several existing video segmentation datasets that already had high-quality mask annotations, and reviewed over 4,000 candidate videos. They kept only 2,006 that met two rules: each video had to contain multiple interacting objects (ideally with similar appearances, so that appearance alone would not identify a target), and those objects had to exhibit substantial motion. Videos with isolated or barely moving objects were discarded.

Annotation used GRES-style and ReferIt-style interactive two-player game approaches. An annotator picked one or more objects in a video and wrote an expression about them, or wrote a deceptive "no-target" expression without picking any object. Five guidelines governed the writing: targets must show significant motion; static attributes such as shape and color should be omitted when motion alone unambiguously identifies the target; several objects may be described together if their shared motion disambiguates them; limited static attributes may be used only when motion alone cannot differentiate; and no-target expressions must still be about motion and still relate to the video.

Validation was independent. A separate validator watched the video, read the expression, and submitted which objects they believed were referred to. A sample survived only if the validator's selection matched the annotator's. Mismatches went to a second validator, and persistent failures were discarded. Validators could also reject samples whose targets were identifiable from a single frame, or no-target expressions unrelated to the video or lacking motion.

Audio was then recorded or synthesized for every expression, with all clips checked twice by human verifiers and by speech recognition models for consistency with the text.

Finally, to serve as a baseline, the authors built LMPM++, which detects candidate objects per frame using language-conditioned queries, represents each with an object embedding, and passes those embeddings to a large language model to reason over the global temporal context. The model outputs the number of target objects and corresponding <SEG> tokens used by a mask decoder to produce object mask trajectories. A temporal-level contrastive loss was introduced so the model can distinguish motions with different temporal orders, such as "first jumping high and then jumping far" versus "first jumping far and then jumping high."

Why This Matters

Impact on research. The paper shifts the emphasis of referring video segmentation from static appearance matching toward temporal motion understanding, and supplies a benchmark with supporting metrics for four distinct tasks. It also introduces the RMEG task, reframing capationing as generating an expression that unambiguously picks out a specified target rather than merely describing a scene.

Real-world applications:

• Embodied agents and robotics. AVOS is described as suited to embodied scenarios where speaking a command to a robot is more convenient than typing text.
• Generative video editing. The paper cites VIDiff, which uses MeViS to train diffusion-based models for modifying and translating video content based on user instructions.
• Multi-modal large language models. The paper cites Merlin, which uses MeViS to train MLLMs that predict future states from present observations.
• Surveillance, autonomous systems, and tracking. RMOT over MeViS's 2,006 videos and 136,102 frames targets detecting and tracking objects from natural language descriptions across scenes far more varied than the driving-focused Refer-KITTI.

Industry relevance. Any product that needs to find, follow, or edit specific objects in video from a spoken or written instruction — video editing tools, robotics interfaces, video search, content moderation, or autonomous driving perception — could use this dataset for training and evaluation. The dataset and method source code are publicly available at https://henghuiding.com/MeViS/.

Future Directions

• Improving motion reasoning models. The benchmark shows existing methods struggle; closing the gap on motion reasoning expressions (999 of them) and no-target expressions (3,503) is an open problem.
• Exploiting audio more directly. The authors suggest integrating audio and visual signals without converting speech to text first, since that conversion discards accent, emotion, speed, and noise cues.
• Extending to richer referring tasks. The paper explicitly raises referring motion expression generation (RMEG) as a new task and discusses additional applications such as AIGC and beyond.
• Scaling and generalizing annotation. The paper states it will discuss potential directions for future video-language motion understanding research, though the specific list is not included in the available text.

Target Audience

Researchers and engineers working on video-language understanding, referring segmentation, video object tracking, audio-visual learning, and multi-modal large language models — particularly those building or evaluating systems that must locate or track objects in complex, cluttered video based on motion descriptions. It also suits practitioners in generative video editing, robotics, and autonomous systems who need a benchmark with diverse real-world scene complexity rather than isolated salient objects.

Authors’ abstract

This paper proposes a large-scale multi-modal dataset for referring motion expression video segmentation, focusing on segmenting and tracking target objects in videos based on language description of objects' motions. Existing referring video segmentation datasets often focus on salient objects and use language expressions rich in static attributes, potentially allowing the target object to be identified in a single frame. Such datasets underemphasize the role of motion in both videos and languages. To explore the feasibility of using motion expressions and motion reasoning clues for pixel-level video understanding, we introduce MeViS, a dataset containing 33,072 human-annotated motion expressions in both text and audio, covering 8,171 objects in 2,006 videos of complex scenarios. We benchmark 15 existing methods across 4 tasks supported by MeViS, including 6 referring video object segmentation (RVOS) methods, 3 audio-guided video object segmentation (AVOS) methods, 2 referring multi-object tracking (RMOT) methods, and 4 video captioning methods for the newly introduced referring motion expression generation (RMEG) task. The results demonstrate weaknesses and limitations of existing methods in addressing motion expression-guided video understanding. We further analyze the challenges and propose an approach LMPM++ for RVOS/AVOS/RMOT that achieves new state-of-the-art results. Our dataset provides a platform that facilitates the development of motion expression-guided video understanding algorithms in complex video scenes. The proposed MeViS dataset and the method's source code are publicly available at https://henghuiding.com/MeViS/