Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

AI summary

Overview

Research area: Computer vision — specifically Video Reasoning Segmentation (VRS), combining multimodal large language models (MLLMs), grounded chain-of-thought reasoning, and video mask propagation.

Technical level: Advanced. The paper assumes familiarity with referring/reasoning video segmentation, chain-of-thought prompting in MLLMs, reinforcement learning with group-relative policy optimization, and mask propagators such as SAM3.

Scope: The paper proposes Temporal Chain of Video Primitives (TCoVP), a representation that interleaves natural-language reasoning with timestamped, box-grounded entity observations so that an MLLM's reasoning trace can be converted directly into prompts for video mask prediction.

What This Paper Is About

Video Reasoning Segmentation asks a model to take an indirect natural-language query (e.g., "who clasps hands with the athlete with dyed hair before hugging that athlete") and output the pixel-level mask trajectory of the intended target. The authors identify a "video reference gap": text-only chain-of-thought can explain why a target was chosen without ever making explicit which visual evidence, at which timestamps, supports that choice. Their goal is to close that gap by forcing the reasoning trace itself to carry explicit entity references, temporal spans, and bounding boxes, and then reusing the selected evidence as segmentation prompts.

Key Contributions

1. Formulation of the video reference gap and the TCoVP representation. Each "video primitive" bundles an entity identifier, a semantic description, a keyframe timestamp, a bounding box, a temporal support interval, and an optional action/relation description; the primitives are serialized in chronological order and interleaved with language in a single autoregressive reasoning trace.
2. Executable reasoning output. The final <answer> block selects a subset of primitives, which become timestamped box prompts for mask propagation, with propagation restricted to the predicted action interval on motion-grounded queries.
3. Two new training corpora and a chain-aligned training recipe. TCoVP-SFT-15k (supervised fine-tuning) and TCoVP-RL-13k (reinforcement learning), with rewards for structural validity, temporal alignment, spatiotemporal grounding, and target primitive accuracy, built on the training splits of ReVOS, MeViS, Long-RVOS, and GroundMoRe.
4. Broad evaluation. Five VRS benchmarks (Long-RVOS, ReasonVOS, ReVOS, MeViS, GroundMoRe) across three backbones (Qwen2.5-VL-3B, Qwen3-VL-4B, Molmo2-4B), including controlled comparisons against text-only CoT with the same answer format and data volume.

Main Findings

• Large gains on motion-grounded queries (GroundMoRe). TCoVP-M-4B reaches 55.37 overall J&F, TCoVP-4B 55.21, and TCoVP-3B 53.67, versus 31.96 for the strongest listed prior VRS baseline (VideoSEG-O3-4B). Per-subset, TCoVP-M-4B scores 60.17 on Sequential, 54.51 on Counterfactual, 54.33 on Descriptive, and 53.00 on Causal.
• Strong general VRS results. The Molmo2-4B variant obtains the highest listed J&F on Long-RVOS (67.4), ReVOS (68.3), and MeViS (65.2). On ReasonVOS, Qwen3-VL-4B leads with 72.1, ahead of Qwen2.5-VL-3B at 71.6 and Molmo2-4B at 71.5. Gains over the strongest listed prior baseline are 7.7 to 10.0 points on Long-RVOS and 6.7 to 7.3 points on ReasonVOS.
• TCoVP beats text-only CoT under matched conditions. With Qwen2.5-VL-3B and identical data volume, the five-benchmark macro average rises from 57.8 (text-only CoT with the same answer-primitive format) to 62.7, with a 7.4-point gain on GroundMoRe and gains of 2.5 to 6.4 points elsewhere. Against Veason-R1-3B (macro average 48.3), the macro average rises to 62.7.
• Reinforcement learning adds substantial value over SFT alone. Adding chain-aligned RL to SFT yields gains of 8.8, 8.5, 5.4, and 9.9 points on Long-RVOS, ReasonVOS, ReVOS, and GroundMoRe respectively, with the largest improvement on GroundMoRe.
• Both temporal and spatial rewards matter. Removing both the temporal alignment and spatiotemporal grounding rewards drops the four reported scores to 62.5 / 67.4 / 63.1 / 47.9 (Long-RVOS / ReasonVOS / ReVOS / GroundMoRe); removing only the grounding reward gives 63.2 / 65.6 / 63.5 / 49.2; removing only the temporal reward gives 64.0 / 68.5 / 63.9 / 48.4; using both gives the best result at 65.6 / 71.6 / 65.3 / 53.7.
• Consistency across backbones. All three variants outperform the strongest listed prior baseline on Long-RVOS and ReasonVOS, which the authors present as evidence that the representation is not tied to one MLLM.

Methodology in Plain English

The model watches 16 uniformly sampled frames from a video along with an indirect question. Instead of writing a plain paragraph of reasoning, it "rolls out" a chronological chain: at each sampled timestamp it either emits nothing (if the frame holds no relevant evidence) or appends one or more records describing an entity. Each record reuses a stable identifier if the entity was seen before, so the chain keeps track of who is who across time, and each record carries a timestamp, a box, and a time interval during which the entity or event holds. The chain is written with tags (<obj>, <time>, <box>, <t>) interleaved with ordinary explanatory text, so a standard autoregressive MLLM can generate it with no extra network. At the end, an <answer> block names which primitives are actually the target — deliberately separating contextual entities (the athlete with dyed hair, used only as identifying context) from the object to be segmented (the coach). Those selected boxes are handed to SAM3, which propagates masks forward and backward in time; on motion-grounded data the output is clipped to the predicted action interval.

Training happens in two stages. First, supervised fine-tuning on TCoVP-SFT-15k teaches the serialization format and entity-continuity conventions. Annotation for these corpora was produced by Kimi-K2.5 from sampled frames plus task annotations, then filtered: primitives must have valid identifiers, temporal supports, keyframe timestamps, and boxes; timestamps must be ordered; identifiers must be reused consistently; unmatched references are removed; and Qwen2.5-VL-7B performs a second-stage semantic check on contextual boxes. Second, chain-aligned reinforcement learning on TCoVP-RL-13k samples eight candidate chains per query and scores them with four rewards — structural validity (format and parseability), temporal alignment (soft precision/recall over timestamps with a Gaussian tolerance, combined as an F-score), spatiotemporal grounding (IoU against nearby annotated primitives plus temporal IoU), and target primitive accuracy (Hungarian-matched pairwise temporal and spatial scores, normalized by the larger of predicted and ground-truth target counts, with special handling of zero-target cases). Advantages are group-normalized and optimized with a KL penalty to the SFT reference policy. Training used one SFT epoch, two RL epochs, LoRA for Qwen backbones at 4×10⁻⁵ (SFT) and 5×10⁻⁷ (RL), and full-parameter tuning for Molmo2 at 2×10⁻⁶ and 2×10⁻⁷.

Why This Matters

The paper reframes reasoning for segmentation as an interface problem: the bottleneck is not whether the model can reason, but whether its reasoning leaves behind machine-usable, checkable visual evidence. Making each intermediate step a parseable record with a box and a time span means the reasoning can be supervised, rewarded, and executed — and errors can be localized to a specific primitive rather than to an opaque paragraph.

Real-world applications:

• Assistive and surveillance video search: finding "the person who handed the package to the courier before the gate closed" in long footage, where the target is defined by an event, not an appearance.
• Video editing and post-production: turning an indirect instruction into a tracked mask for rotoscoping or object removal, especially when the relevant span is short.
• Robotics and instruction following: grounding referring expressions that depend on action order ("the tool used right after the lid was removed").
• Sports and coaching analytics: isolating a specific participant during a specific interaction rather than tracking them across the whole clip.

Industry relevance: the method is backbone-agnostic, uses standard autoregressive decoding, and plugs into an off-the-shelf mask propagator, so it is comparatively easy to adopt in existing MLLM-plus-segmenter pipelines. The authors state that all models and code will be publicly released, with a project page at https://victor-wjw.github.io/TCoVP/.

Future Directions

• Temporal resolution. Reasoning over 16 uniformly sampled frames means brief events can fall entirely between observations; the authors suggest adaptive temporal sampling as a remedy.
• Structured relations instead of prose. Cross-entity actions and relations are currently expressed in natural language without a dedicated structured representation, which limits how precisely interactions can be grounded or verified.
• Decoupling from the mask propagator. Final masks inherit errors from SAM3 propagation, so better or verifiable propagation backends are an open lever (the appendix reports ablations and oracle analyses of propagation backends and intermediate grounding).
• Scaling to harder video scenarios. The authors note that more complex settings may require denser spatial evidence and richer interaction representations, and the truncated appendix indicates further oracle and conventional referring-VOS analyses that would clarify where the remaining error sits.

Target Audience

Researchers and graduate students working on video reasoning segmentation, referring video object segmentation, and grounded/visual chain-of-thought in multimodal LLMs; engineers building MLLM-plus-segmenter pipelines who need event-level, temporally bounded mask extraction; and practitioners in reinforcement learning for multimodal reasoning who are interested in reward designs that supervise intermediate spatial and temporal evidence rather than only final answers. Readers wanting a gentle introduction to the area should start with the introduction and Section 3.1, since Sections 3.3 and the reward equations assume familiarity with group-relative policy optimization and mask propagation.

Authors’ abstract

Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer. Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a \emph{video reference gap} between linguistic inference and the evidence distributed across frames. We introduce \textbf{Temporal Chain of Video Primitives (TCoVP)}, a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain. Each primitive associates an entity reference with temporal support and timestamped spatial evidence, and the chain interleaves these references with language to guide reasoning about which entity is relevant, when the queried event holds, and where its evidence appears. The selected target primitives then provide box prompts and temporal support for video mask prediction. We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning. Extensive experiments across five VRS benchmarks and three model backbones demonstrate the strong performance of TCoVP, highlighting the effectiveness of organizing spatial and temporal evidence within the reasoning process. All models and code will be publicly released. Project page: \url{https://victor-wjw.github.io/TCoVP/}