FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos

AI summary

Overview

Research area: Computer vision and vision-language modeling — specifically automated dataset curation and fine-tuning for fine-grained object motion and spatial reasoning in video.

Technical level: Advanced. The paper assumes familiarity with vision-language models (VLMs), object detection and tracking pipelines, and benchmark evaluation.

Scope in one sentence: The paper presents FoundationMotion, a fully automated pipeline that turns raw videos into motion trajectories, captions, and question-answer pairs, and shows that fine-tuning open VLMs on the resulting data improves motion understanding across six benchmarks.

What This Paper Is About

Current vision-language models can often identify what an object is or what is happening, but they struggle with how motion unfolds — for example, recognizing that "the car is turning right" or how water pours from a bottle into a glass. The paper argues this gap comes primarily from a lack of "how" motion data, because manually annotating fine-grained motion is prohibitively expensive (the authors cite an estimate that a team of 10 people would need roughly 100 days to annotate 100,000 videos).

FoundationMotion addresses the gap with a fully automated curation pipeline that detects and tracks objects, converts their trajectories into structured annotations, and uses large language models to produce captions and multiple-choice QA pairs at scale. The authors then fine-tune open-source models on this data and evaluate them on public and newly hand-annotated motion benchmarks.

Key Contributions

1. FoundationMotion, a fully automated data curation pipeline. A unified pipeline for detecting, tracking, and understanding object behavior, which the authors describe as producing approximately 500K question-answer pairs (QAs) and captions, collectively the FoundationMotion Dataset.

2. A manually annotated "how" motion benchmark suite. The authors collect videos of varying lengths and annotate QAs across multiple domains: hand motion in daily activities, robot motion during manipulation, and car and hand motion in autonomous driving. This yields four zero-shot benchmarks named AV-Car, AV-Hand, Daily, and Robotics.

3. Fine-tuning results across open-source VLMs. NVILA-Video-15B, NVILA-Video-8B, and Qwen-2.5-VL-7B were fine-tuned on the FoundationMotion Dataset and evaluated on public benchmarks (MotionBench, VLM4D) plus the four self-labeled benchmarks, with comparisons against Gemini-2.5-Flash, Qwen-2.5-VL-72B, and an equal-budget fine-tune on PLM data.

4. Release commitment. The authors state they will release all code, data, and benchmarks, with the stated goal of establishing a standard for motion understanding and fostering community development.

Main Findings

• Consistent gains for NVILA-Video-15B. Fine-tuning with FoundationMotion lifted MotionBench by +1.0 (45.7 to 46.7), VLM4D by +0.1 (51.8 to 51.9), AV-Car by +7.1 (84.4 to 91.5), AV-Hand by +0.6 (58.1 to 58.7), Daily by +2.4 (76.2 to 78.6), and Robotics by +14.9 (21.4 to 36.3).

• Consistent gains for NVILA-Video-8B. Improvements were +0.6 on MotionBench (42.3 to 42.9), +3.4 on VLM4D (49.0 to 52.4), +1.7 on AV-Car (88.9 to 90.6), +6.8 on AV-Hand (54.6 to 61.4), +2.0 on Daily (79.1 to 81.1), and +17.8 on Robotics (20.4 to 38.2).

• Broad gains for Qwen-2.5-VL-7B. Gains were +2.1 on MotionBench (39.1 to 41.3), +3.2 on VLM4D (41.7 to 44.9), +1.8 on AV-Car (80.3 to 82.1), +5.6 on AV-Hand (47.2 to 52.8), +11.7 on Daily (61.4 to 73.1), and +4.2 on Robotics (28.3 to 32.5).

• FoundationMotion data beats an equal budget of PLM data. On NVILA-Video-15B, FoundationMotion is reported as better on AV-Car (+7.1 vs. -5.0), AV-Hand (+0.6 vs. -2.5), Daily (+2.4 vs. +0.9), and Robotics (+14.9 vs. +6.0), while PLM was slightly better on MotionBench (+1.0 vs. +1.8) and VLM4D (+0.1 vs. +1.1). On NVILA-Video-8B, FoundationMotion led on VLM4D (+3.4 vs. +0.1), AV-Car (+1.7 vs. -1.0), AV-Hand (+6.8 vs. +1.4), Daily (+2.0 vs. -4.1), and Robotics (+17.8 vs. +6.1), while PLM was slightly better on MotionBench (+0.6 vs. +1.3).

• Mid-sized tuned models surpass much larger baselines on several tasks. NVILA-Video-15B + FoundationMotion reached 91.5 on AV-Car, above Gemini-2.5-Flash (84.1) and Qwen-2.5-VL-72B (83.3). The same model also exceeded Qwen-2.5-VL-72B on VLM4D (51.9 vs. 50.5) and AV-Hand (58.7 vs. 56.5). Baseline scores reported for Gemini-2.5-Flash were 55.6 / 54.7 / 84.1 / 72.7 / 75.4 / 36.1 and for Qwen-2.5-VL-72B were 61.4 / 50.5 / 83.3 / 56.5 / 80.2 / 36.7 across MotionBench, VLM4D, AV-Car, AV-Hand, Daily, and Robotics respectively.

• Structured bounding-box JSONs improve generated QA quality. Evaluated by GPT-4 on a 0–10 scale averaged over three runs, adding bounding-box JSONs to video input raised fine-grained action accuracy from 5.8 to 8.4 (+2.6), motion detail and specificity from 6.1 to 8.7 (+2.6), temporal coherence from 6.5 to 8.9 (+2.4), question relevance from 6.9 to 8.5 (+1.6), and overall QA quality from 6.3 to 8.6 (+2.3).

• Different QA types contribute complementary gains. With Qwen2.5-7B fine-tuned on 2,000 samples per experiment, the baseline was 48%. Motion Recognition and Action Order each reached 52% (+8.3% over base), Motion-related Objects and Location-related Motion each reached 53% (+10.4%), and Repetition Count delivered the largest gain at approximately 55% (+14.6%). The aggregated setting (ALL) also reached 55%.

• Dataset statistics. The FoundationMotion Dataset comprises 46.7k videos and 467k QAs, each QA containing a question, four options, an answer, and a category. Correct answers are evenly distributed across the four options. Most questions fall between 30 and 80 characters. Video durations are mostly concentrated within 3–7 seconds. Training data was drawn from InternVid by randomly extracting 5-second clips, giving 467K caption/QA-video pairs.

• Appendix statistics. Average video duration is 17.51 seconds, average questions per video is 10.04, average annotation density is 1.671 questions per second, and average question length is 55.9 characters.

Methodology in Plain English

The pipeline has four stages.

Video preprocessing. For each input video, the system extracts a temporal segment of 5–10 seconds. Videos of 5 seconds or less are kept whole; longer videos get a segment of duration drawn uniformly from 5 to min(10, duration), centered near the video midpoint with a small random temporal offset. Because camera motion makes object motion hard to describe even for humans, VGGT predicts camera poses across sampled frames and computes a motion score from translation and rotation changes (both average and maximum). Videos exceeding a threshold of 0.3 are discarded.

Object detection and tracking. Open-vocabulary detection uses Qwen2.5-VL-7B to name salient objects in the first frame by natural language generation, then Grounded-DINO to localize them. The authors query Grounded-DINO with one object class at a time rather than a concatenated prompt, to enforce a one-to-one alignment between boxes and labels. A separate human-centric branch runs Cascade Mask R-CNN with a ViTDet-H backbone for person detection at confidence 0.8, ViTPose+ for whole-body keypoints including 42 hand keypoints (initial hand regions expanded by 1.5x), and the Hands23 model to detect hands with hand side and contact state. Hand-person associations require IoU above 0.3.

Tracking uses SAM2 with a two-stage strategy. Persons get IDs in the range 0–99 with sub-IDs for body parts (ID x 10 for the person, ID x 10 + 1 for the left hand, ID x 10 + 4 for the right hand), while objects get IDs starting from 1000. A refined stage incorporates hand and hand-object detections at keyframes (every fifth frame) to prevent tracking drift.

Caption generation. GPT-4o-mini receives video frames sampled at 2 fps, a JSON of normalized bounding-box trajectories, and visual overlays with color-coded boxes. Its prompt covers seven dimensions of motion: action and gesture recognition, temporal sequencing, object-action associations, spatial context, repetition patterns, motion dynamics (direction, distance, velocity, trajectory), and evolving spatial relationships.

QA generation. GPT-4o-mini is prompted with both captions and video frames to produce multiple-choice questions in five categories: motion recognition, temporal ordering, action-object association, location-based motion, and repetition counting. Each question has four options with distractors drawn from video content, and correct answers are randomly distributed in the released data to avoid position bias (the generation prompt instructs the model to always place the correct answer first).

For evaluation, the authors fine-tune on 8 A100 GPUs. Qwen-related training uses llamafactory with a learning rate of 10^-5; NVILA-related training follows official settings with a learning rate of 1.5 x 10^-5. Both use cosine annealing with Adam and no weight decay.

Why This Matters

Impact on research. Motion benchmarks such as MotionBench (5,385 videos, 8,052 human-annotated QA pairs across six motion tasks) and FAVOR-Bench (1,776 curated videos) show that leading models score poorly on fine-grained motion — MotionBench reports state-of-the-art video VLMs below 60%, and FAVOR-Bench evaluations across 21 multimodal LLMs showed performance far below human levels. FoundationMotion reframes the bottleneck as a data problem rather than purely a modeling problem, and shows that automated curation at scale can close some of that gap. It also extends beyond "what" recognition to spatial relations and relative trajectories, which the authors argue prior benchmarks under-emphasize.

Real-world applications:

• Autonomous driving: understanding car and hand motion, evaluated here via AV-Car (1,968 QA pairs derived from Nuscenes) and AV-Hand (108 QA pairs).
• Robotics and manipulation: understanding how robot hands move during tasks, evaluated via the Robotics benchmark (102 QA pairs collected from YouTube).
• Human daily activity understanding: hand motion and hand-object interaction, evaluated via the Daily benchmark (832 QA pairs from 100 Days of Hands).
• Embodied agents and physical reasoning: the paper frames motion understanding as fundamental to inferring dynamics and predicting future states.

Industry relevance. The pipeline is designed for scalability: the authors cite manual annotation costs (minutes per 3-second video; roughly 100 days for a team of 10 to label 100,000 videos) as the motivation for automation. Demonstrating that mid-sized open models fine-tuned on FoundationMotion can beat Gemini-2.5-Flash and Qwen-2.5-VL-72B on several motion tasks is directly relevant to teams that cannot rely on large closed-source models for deployment in driving or robotics.

Future Directions

• Extending from 2D to 3D. The authors state explicitly that current spatial understanding is limited to 2D and that understanding how objects move in 3D is a challenging but essential next step.
• Dexterous hand motion in 3D. The limitation section singles out understanding how each joint moves to form dexterous hand motions in 3D space as a direction that would greatly benefit robotics.
• Refining the codebase and dataset. The conclusion commits to continuous efforts and improvements to the FoundationMotion codebase and dataset beyond the initial release.
• Broadening evaluation coverage. The self-labeled benchmarks vary substantially in size (from 102 to 1,968 QA pairs) and are drawn from limited source datasets, leaving room to expand the range of domains and motion types tested.

Target Audience

Researchers and engineers working on video-language models, motion understanding, and spatial reasoning will get the most from this paper. It is also relevant to practitioners in autonomous driving and robotics who need models that reason about how objects move rather than only what they are; to dataset builders looking for an automated alternative to manual annotation; and to benchmark designers interested in the distinction between "what" and "how" motion evaluation. Readers should be comfortable with detection, tracking, and VLM fine-tuning concepts to follow the pipeline details.

Authors’ abstract

Motion understanding is fundamental to physical reasoning, enabling models to infer dynamics and predict future states. However, state-of-the-art models still struggle on recent motion benchmarks, primarily due to the scarcity of large-scale, fine-grained motion datasets. Existing motion datasets are often constructed from costly manual annotation, severely limiting scalability. To address this challenge, we introduce FoundationMotion, a fully automated data curation pipeline that constructs large-scale motion datasets. Our approach first detects and tracks objects in videos to extract their trajectories, then leverages these trajectories and video frames with Large Language Models (LLMs) to generate fine-grained captions and diverse question-answer pairs about motion and spatial reasoning. Using datasets produced by this pipeline, we fine-tune open-source models including NVILA-Video-15B and Qwen2.5-7B, achieving substantial improvements in motion understanding without compromising performance on other tasks. Notably, our models outperform strong closed-source baselines like Gemini-2.5 Flash and large open-source models such as Qwen2.5-VL-72B across diverse motion understanding datasets and benchmarks. FoundationMotion thus provides a scalable solution for curating fine-grained motion datasets that enable effective fine-tuning of diverse models to enhance motion understanding and spatial reasoning capabilities.