LAMP: Language-Assisted Motion Planning for Controllable Video Generation

AI summary

Overview

Research area: Computer Vision — controllable text-to-video generation, motion planning, and language-model-based program synthesis.

Technical level: Advanced.

Scope: The paper presents LAMP, a framework that uses a large language model (LLM) as a motion planner to convert natural-language scene descriptions into symbolic motion programs, which are deterministically mapped to 3D object and camera trajectories that condition a pretrained video generator.

What This Paper Is About

Video generators can now be steered by text, layouts, depth, and camera parameters, but users still have no practical way to specify how objects and cameras should move together over time. Describing a coordinated shot — for example a chase scene where a runner's path and the pursuing camera are defined relative to each other — requires spatial planning that ordinary text prompts, video-derived annotations, or 2D sketches cannot express. LAMP addresses this by treating motion control as a language-to-program synthesis problem: an LLM writes structured motion programs in a cinematography-inspired domain-specific language (DSL), which are then converted into explicit 3D trajectories for both the object and the camera.

Key Contributions

1. A cinematography-inspired motion DSL. Motion is expressed as ordered tags built from four base primitives (free-form, orbit track, tail track, rotation track) plus key–value modifiers covering translation (lat, vert, depth), rotation (yaw, pitch, roll), and stylistic/temporal cues (speed_fast, ease_in, jitter_low). Programs are deterministically mapped to 3D trajectories.

2. An open procedural data generation framework and large-scale text–motion dataset. The corpus contains 400K text–motion pairs — 100K samples each for free-form and object-relative motions, in both raw and LLM-paraphrased form — built by sampling and composing motion primitives and paraphrasing descriptions with an auxiliary LLM.

3. A decoupled, editable motion planning interface. Because planning happens before synthesis, users can issue relative refinements such as "move the camera lower" or "zoom out slightly" and inspect trajectories before committing to costly video generation.

4. The first framework to generate both object and camera trajectories from natural language. Prior work focuses on layout generation or camera trajectory synthesis alone; LAMP unifies both in a shared 3D space, which the authors argue is required for cinematographically coherent video synthesis.

Main Findings

• Camera trajectory accuracy on DataDoP: Training only on the procedural dataset (no DataDoP training data), LAMP reaches a Revised F1-Score of 0.763, versus 0.360 for GenDoP (exp), 0.400 for GenDoP, and 0.391 for Director3D, both of which were trained on DataDoP. LAMP's CLaTr score is 36.29, above GenDoP (36.18) and GenDoP (exp) (35.91). After finetuning on a DSL-converted DataDoP subset, the Revised F1-Score rises to 0.776 and CLaTr to 36.52. The paper notes that the official GenDoP train/test split was not released, so LAMP and GenDoP (exp) were evaluated on a representative public subset.

• Camera trajectory accuracy on the ET benchmark: LAMP obtains F1 0.976 and CLaTr 35.10 on the pure split and F1 0.769 and CLaTr 36.59 on the harder mixed split, outperforming CCD (0.27 / 3.21 pure; 0.17 / 6.26 mixed), MDM (0.76 / 21.26; 0.34 / 18.32), ET-DirB (0.86 / 23.10; 0.39 / 20.78), and ET-DirC (0.80 / 21.49; 0.48 / 21.95). Finetuning on a DSL-converted ET subset changes results only marginally (0.978 / 35.02 pure; 0.779 / 36.95 mixed).

• Object trajectory accuracy on the procedural test split: LAMP reports Coarse Translation F1 of 0.9983, Fine Translation F1 of 0.9293, and Fine Rotation F1 of 0.975.

• DSL beats direct trajectory regression: In an ablation on the procedural dataset, DSL-based prediction scores 0.996 coarse translation F1 (27 classes), 0.966 fine translation F1 (343 classes), 0.950 coarse rotation F1, and 0.753 fine rotation F1 (1728 classes) with a per-axis rotation MAE of 3.507 degrees. Direct per-frame trajectory regression — with rotation and translation discretized into bins similar to the GenDoP formulation — scores 0.847, 0.781, 0.730, and 0.626 with a rotation MAE of 8.333 degrees.

• Metrics used: text–trajectory alignment is measured with CLaTr-CLIP from ET, realism/naturalness with CLaTr-FID and coverage, and semantic action capture with the motion tagging F1 score proposed in ET.

• Video-level user study: 16 participants rated videos on three criteria — camera-motion alignment with text, object-motion alignment with text, and overall plausibility. LAMP was consistently preferred over baselines. Against the strongest baseline (GPT-DSL, where GPT 5 (thinking) is given the full DSL specification and prompted to generate camera programs from the same object motions), LAMP was preferred 52% of the time and both options were preferred 25% of the time.

• Video baselines compared: a text-only Wan 2.1 model conditioned on combined object and camera descriptions; GPT-DSL; GPT-Traj (the same MLLM given an example trajectory and asked to synthesize trajectories directly); and the full LAMP model. The MLLM baselines used LAMP's object motions and generated only camera motion.

• Qualitative test set: 14 textual descriptions selected from CameraBench, focused on object motion, camera motion, or both, with object-motion descriptions added automatically by a vision–language model.

• Model-agnostic trajectory interface: The same DSL-derived motion signal was integrated with CameraCtrl (text-to-video), EPiC (image-to-video), and ReCamMaster (video-to-video) without modifying the video generators.

• Procedural data augmentation from real datasets: After DSL conversion of noisy DataDoP trajectories, filtering by original-vs-reconstructed similarity and removing static or near-static cases yielded 10K usable samples; a comparable process on ET yielded 21K samples. In both cases, the paper reports only marginal gains, which it attributes to limited data volume and, for ET, the absence of rotational camera motion.

• F1 metric revision: The authors revised the F1 computation used with DataDoP. Instead of evaluating on trajectories reconstructed by CLaTr, they evaluate direction and rotation of predicted trajectories directly, removing reconstruction bias; the paper states this revision was verified with the GenDoP authors and yields a significant improvement in observed F1 scores.

• Runtime characteristics (supplementary): An MLLM call (Qwen2.5-VL-7B) typically takes under 5 seconds and trajectory rendering takes 2–3 seconds, so motion can be validated in near real-time. Final video synthesis with VACE-1.3B takes around 6 minutes for 81 frames on an NVIDIA A40 48GB.

• Distribution statistics: The procedural corpus spans 27 coarse classes (3 motion types × 3 directions) and 343 fine-grained classes (7 motion types × 3 directions) for translation; camera rotations are sampled densely across the full [-180, 180] degree range. The distribution is intentionally imbalanced, with common single-axis motions such as forward and backward appearing more often than multi-axis composites, mirroring real datasets such as E.T. and GenDoP.

Methodology in Plain English

Rather than training a model to spit out raw coordinate numbers for camera and object paths, the authors teach an LLM to write short motion programs in a small, formal language modeled on how filmmakers talk about movement. The DSL has four base primitives — free-form (unconstrained 6-DoF motion), orbit track (camera circles a target), tail track (camera follows an object with a configurable time offset), and rotation track (camera rotates in place) — and each primitive is refined with key–value modifiers for translation, rotation, speed, easing, and jitter.

Training data is generated procedurally. Motion sequences span T = 21 frames split into four temporal segments, with a primitive possibly changing per segment. Object motion sequences hold up to N = 4 motion tags and are limited to translation plus pitch and yaw; camera sequences allow full 6-DoF control. Given a symbolic program, a deterministic converter produces physically coherent 3D motion: free-form primitives integrate translation and rotation per frame relative to the origin, while tracking primitives synthesize the object trajectory first and then compute the camera relative to the object before transforming to world coordinates. Text templates describing canonical cinematographic behaviors (for example, "the camera pans slowly right") are paraphrased by an auxiliary LLM to broaden linguistic variety.

At inference, the LLM planner factorizes the problem: it first predicts the object program from the object-centric part of the description, then predicts the camera program conditioned on the object program and the camera-centric text, with the coordinate system anchored to the first camera frame. This mirrors the hierarchy of cinematography, where the subject defines the scene dynamics and the camera adjusts to preserve framing and continuity. Generating tags instead of coordinates keeps the output syntactically valid, interpretable, and editable.

For video synthesis, the predicted trajectories are rendered into a frame-aligned control video compatible with the VACE diffusion generator: 3D object bounding boxes are projected into the current view as 2D overlays, and the edges of a fixed global cube are projected to represent the horizon and camera orientation. This control video, the text prompt, and optionally a first frame condition the pretrained generator. For long-horizon results, LAMP is prompted sequentially across temporal segments, concatenating trajectories by initializing each segment with the previous endpoint and running VACE iteratively with the last frame as conditioning. Multi-object results generate independent trajectories per object, followed by camera motion relative to one of them, with manual alignment of initial object positions; automating relational multi-object trajectory generation is left for future work.

Why This Matters

Impact on research. The paper reframes motion control in video generation from coordinate regression to language-conditioned program synthesis, and argues the real bottleneck is not the generative models but the interfaces used to control them. Its procedural dataset generation framework and DSL are offered as reusable infrastructure: the dataset and models are stated to be publicly released, and the resulting trajectories are shown to plug into existing backbones such as CameraCtrl, EPiC, ReCamMaster, and VACE. The DSL extends the CameraBench taxonomy, connecting video generation to established geometric camera planning practice in computer graphics.

Real-world applications:

• Film and animation previsualization: simplified geometric proxies and explicit camera/object trajectories mirror standard blocking and previz practice, letting creators iterate on staging and shot composition before rendering.
• AI-assisted content creation: short-form video, advertising, and social media production where a written scene description needs to produce a specific, repeatable camera move.
• Virtual production and game cinematics: camera paths generated as explicit 3D trajectories can drive virtual cameras or feed downstream engines rather than being baked into pixels.
• Interactive creative tools: because planning is decoupled from synthesis and takes seconds rather than minutes, an editing interface can let users nudge motion with follow-up text instructions before paying the cost of video generation.

Industry relevance. Video generation is becoming commoditized on visual fidelity, so controllability and iteration speed are natural differentiators. LAMP's model-agnostic trajectory interface means its output can be layered onto whichever generator is cheapest or best at a given moment, and the roughly 2–3 second trajectory rendering plus sub-5-second planner call, compared with roughly 6 minutes for 81 frames of VACE-1.3B synthesis, is a favorable economics profile for interactive products.

Future Directions

• Richer, more realistic motion programs. The authors note that current DSL-based object and camera programs are limited in diversity and realism, lacking fine-grained orientation data, semantic object categories such as car versus human, and rich multi-object interactions. Moving toward dynamic, in-the-wild datasets is proposed to capture real-world motion variety.

• Iterative refinement and shot recomposition. The paper suggests letting users iteratively refine trajectories and recompose shots, with options for supplying appearance specifications or locking identities over space and time.

• Automated relational multi-object trajectory generation. Currently, multi-object results require manually aligning initial object positions. Automating this would let LLMs define complex temporal and spatial scene layouts and interactions directly.

• Longer-horizon synthesis. LAMP already generates long-horizon motion by chaining segments and re-conditioning VACE on the last frame, but the paper expects visual quality to improve only as video models become capable of generating longer sequences, and argues explicit trajectory control will remain essential for multi-stage movement planning.

Target Audience

Researchers and practitioners in generative computer vision working on controllable video generation, text-to-video diffusion, and camera/motion control. It is also relevant to graphics and virtual production engineers interested in camera planning and previsualization workflows, and to researchers studying LLMs as planners — the paper positions its planner against LLM-based approaches in robotics, navigation, and embodied AI, as well as against captioning-oriented uses of LLMs for video generation such as ET and GenDoP. Readers will get the most from it with prior familiarity with diffusion-based video models, camera parameterization, and motion representation benchmarks.

Authors’ abstract

Video generation has achieved remarkable progress in visual fidelity and controllability, enabling conditioning on text, layout, or motion. Among these, motion control - specifying object dynamics and camera trajectories - is essential for composing complex, cinematic scenes, yet existing interfaces remain limited. We introduce LAMP that leverages large language models (LLMs) as motion planners to translate natural language descriptions into explicit 3D trajectories for dynamic objects and (relatively defined) cameras. LAMP defines a motion domain-specific language (DSL), inspired by cinematography conventions. By harnessing program synthesis capabilities of LLMs, LAMP generates structured motion programs from natural language, which are deterministically mapped to 3D trajectories. We construct a large-scale procedural dataset pairing natural text descriptions with corresponding motion programs and 3D trajectories. Experiments demonstrate LAMP's improved performance in motion controllability and alignment with user intent compared to state-of-the-art alternatives establishing the first framework for generating both object and camera motions directly from natural language specifications. Code, models and data are available on our project page.