UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Models

AI summary

Overview

Research area: Computer vision / multimodal large language models — specifically video-language models and fine-grained video understanding.

Technical level: Advanced. The paper assumes familiarity with multimodal LLM architecture, vision encoders, SAM-style mask decoders, referring segmentation, and temporal video grounding.

Scope: This paper introduces UFVideo, a single Video LLM that unifies global video understanding, pixel-level video referring and segmentation, and temporal video grounding, together with a new benchmark (UFVideo-Bench) for tasks that require all three granularities simultaneously.

What This Paper Is About

Existing Video LLMs are specialists: one model handles video question answering, another handles referring segmentation, and another handles locating events in time. Because these capabilities live in separate models, they cannot reinforce each other, and no single system can answer a question that demands, say, identifying an object, describing it, and saying when it appears. UFVideo's goal is to fold all three granularities — global, pixel-level, and temporal — into one architecture, one set of training tokens, and one training objective, so that they cooperate rather than compete.

Key Contributions

1. A unified Video LLM architecture. UFVideo is presented as the first Video LLM that performs global video understanding, pixel-level video object referring and reasoning segmentation, and temporal video grounding within a single model, using a common architecture and training strategy rather than task-specific heads or pipelines.

2. A token-based alignment scheme for multi-grained tasks. The authors introduce three special token types — relative temporal tokens (<Temp-τ>) for time, <Ref> placeholders for injecting object mask prompts into the LLM input, and <Seg> tokens whose hidden states are routed to a SAM2 mask decoder. This lets one next-token-prediction objective cover text answers, timestamps, and pixel masks.

3. UFVideo-Bench, a cooperative multi-grained benchmark. The paper defines three novel tasks that deliberately chain granularities: PixRQA (referring question answering with pixel output), PixHQA (adds a specified moment), and PixTRQA (adds moment retrieval). It also releases the accompanying UFVideo-Dataset for training.

4. Strong empirical results across 9 public benchmarks. UFVideo achieves state-of-the-art or near-top results on general video QA, video object referring, referring and reasoning video segmentation, and temporal grounding, using a single 7B model.

Main Findings

• Multi-grained joint training improves every individual task. The central empirical claim is that training on segmentation, temporal grounding, and QA together does not degrade performance but enhances it. In the ablation on task unification, the all-task configuration reaches 62.80 J&F on MeViS val^u, 44.72 tIoU on Charades-STA, and 32.25 J&F on PixTRQA — well above configurations where tasks are trained in isolation (e.g., 52.26 J&F segmentation-only).

• General video understanding is maintained, not sacrificed. On MVBench, UFVideo scores 67.3 average, ahead of LLaVA-ST (64.2), UniPixel (62.5), VideoChat2 (60.4), and far ahead of GPT-4V (43.5). This counters the usual assumption that adding fine-grained tasks erodes holistic comprehension.

• Video object referring improves over the base model. On VideoRefer-Bench-D, UFVideo reaches 3.59 single-frame and 3.61 multi-frame averages, surpassing its own base model VideoRefer (3.42/3.46) and UniPixel (3.47/3.48), showing that broader task exposure sharpens region-level understanding.

• Reasoning video segmentation sees the largest gains. On ReVOS val, UFVideo achieves 64.8 J&F overall versus 58.0 for RGA3 and 57.0 for ViLLa, and on MeViS val^u it reaches 62.8 J&F versus UniPixel's 59.9 — evidence that temporal and global knowledge helps interpret implicit, reasoning-heavy referring expressions.

• Temporal grounding is competitive with temporal specialists. On Charades-STA, UFVideo scores 71.2 R@0.3, 49.6 R@0.5, and 44.7 tIoU, beating LLaVA-ST by roughly 5.4% relative tIoU and all listed 7B temporal-grounding models.

• Relative temporal tokens beat absolute time numbers. The ablation shows relative tokens at length 100 substantially outperform absolute timestamps (39.19 vs 30.24 tIoU), and that scaling from 100 to 150 tokens yields diminishing returns, motivating the choice of 100 tokens for efficiency.

• Existing strong MLLMs struggle on cooperative tasks. On UFVideo-Bench, GPT-4o, Qwen3-VL-8B, and Qwen3-VL-32B produce very weak segmentation and temporal results, while UFVideo attains grounded mask quality and timestamps simultaneously. The authors frame this as evidence that cooperative understanding is genuinely hard and not solved by scale alone.

Methodology in Plain English

UFVideo builds on an existing 7B Video LLM (VideoRefer) and adds two encoders and a decoding path:

• Video and prompts are encoded into tokens. A SigLIP vision encoder converts video frames into visual tokens. Object mask prompts supplied by the user (for referring tasks) are encoded separately and compressed into a small set of "object tokens." These are injected into the language model's input wherever a <Ref> placeholder appears.

• Time is handled with a fixed vocabulary of relative tokens. Rather than writing timestamps as numbers, the authors divide each video into 100 relative bins and use <Temp-τ> tokens to both read and write time. This makes timestamps predictable in the same way as ordinary words.

• Segmentation is delegated to a mask decoder. SAM2's Hiera-L encoder processes a few sampled frames, and the hidden states at <Seg> token positions in the LLM's output are projected and fed into SAM2's mask decoder to produce pixel-level masks. Because different samples ask for different numbers of objects, the decoder training handles variable object counts per batch.

• One loss for text, another for masks. The language model is trained with standard next-token prediction; the mask decoder is trained with binary cross-entropy plus DICE loss against ground-truth masks. The combined objective is a weighted sum.

• Training proceeds in two stages. Stage 1 (about 128K samples) teaches temporal grounding and segmentation alignment; Stage 2 mixes roughly 3 million samples covering general QA, referring, segmentation, temporal grounding, and the new cooperative tasks. Training used 32 A800 GPUs with DeepSpeed.

• The benchmark chains capabilities deliberately. PixRQA gives one frame's mask as a hint and asks for answers plus masks elsewhere; PixHQA pins the question to a specific moment; PixTRQA additionally requires the model to find the right moment before answering and segmenting.

Why This Matters

Research impact. The paper provides concrete evidence for a claim that had been mostly asserted before: fine-grained video capabilities are complementary, not competing. It also offers a reusable design pattern — special tokens plus a shared next-token objective — for merging structured outputs (timestamps, masks) into a language model without task-specific heads. UFVideo-Bench sets a harder evaluation target that current frontier MLLMs largely fail, giving the field a clearer measure of progress toward genuinely general video intelligence.

Real-world applications:

• Video editing and post-production: locating a described person or object across footage and returning precise masks and time ranges for rotoscoping or effects work.
• Video search and surveillance review: answering questions like "when did the person in the red jacket enter, and where are they?" in a single query rather than chaining separate detectors and trackers.
• Sports and coaching analytics: identifying a specific athlete, describing their action, and returning the exact interval and silhouette for technique review.
• Robotics and embodied agents: grounding language instructions in both space and time when acting on video streams, which requires exactly the three granularities UFVideo unifies.
• Accessibility and content moderation: generating temporally anchored, region-specific descriptions of video content.

Industry relevance. The result is a single 7B model replacing what would otherwise be a pipeline of separate referring, segmentation, and grounding systems, reducing latency and integration complexity. The all-public-benchmark wins and the demonstrated weaknesses of GPT-4o on cooperative tasks are directly relevant to companies building video analytics, editing tools, or multimodal assistants.

Future Directions

• Scaling and data efficiency. UFVideo relies on roughly 3 million stage-2 samples. How much of the cooperative ability comes from data volume versus the unified token design, and whether smaller curated datasets suffice, remains untested.

• Longer videos and denser events. The current design uses 100 fixed temporal bins and samples a limited number of frames. Extending to hour-long videos with many events, and refining timestamp precision beyond bin resolution, is an open problem.

• Richer and more compositional cooperative tasks. UFVideo-Bench covers three task chains. Multi-object, multi-event, and multi-hop reasoning scenarios — plus spoken audio — are natural extensions that the current formulation does not touch.

• Generalization and failure analysis. The benchmark is still small, and the paper does not deeply analyze where UFVideo fails or how it behaves under distribution shift. Understanding its systematic error modes is a prerequisite for deployment in safety-critical settings.

Target Audience

Researchers and graduate students working on multimodal LLMs, video-language understanding, and referring segmentation; engineers building video analytics or video editing products who need region- and time-aware language interfaces; and benchmark designers interested in evaluating compositional, multi-granularity video reasoning. Readers without background in vision encoders, mask decoders, or LLM tokenization will find the architectural details hard going, but the framing of the problem and the benchmark results are accessible to a broader technical audience.

Authors’ abstract

With the advancement of multi-modal Large Language Models (LLMs), Video LLMs have been further developed to perform on holistic and specialized video understanding. However, existing works are limited to specialized video understanding tasks, failing to achieve a comprehensive and multi-grained video perception. To bridge this gap, we introduce UFVideo, the first Video LLM with unified multi-grained cooperative understanding capabilities. Specifically, we design unified visual-language guided alignment to flexibly handle video understanding across global, pixel and temporal scales within a single model. UFVideo dynamically encodes the visual and text inputs of different tasks and generates the textual response, temporal localization, or grounded mask. Additionally, to evaluate challenging multi-grained video understanding tasks, we construct the UFVideo-Bench consisting of three distinct collaborative tasks within the scales, which demonstrates UFVideo's flexibility and advantages over GPT-4o. Furthermore, we validate the effectiveness of our model across 9 public benchmarks covering various common video understanding tasks, providing valuable insights for future Video LLMs.