Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising

AI summary

Overview

Research area: Computer vision and generative AI, specifically controllable video synthesis with diffusion models.

Technical level: Intermediate. The paper assumes familiarity with diffusion sampling, noise schedules, and image-to-video (I2V) conditioning, but its central idea is architectural rather than mathematical.

One-sentence scope: The paper introduces a training-free sampling procedure that gives any off-the-shelf image-to-video diffusion model precise, region-specific motion and appearance control by treating a crude user-made animation as the target and denoising different regions at different noise levels.

What This Paper Is About

Diffusion-based video generators now produce convincing footage, but users can only steer motion through text prompts or indirect cues. Methods that offer real motion control — following a trajectory or a camera path — usually require fine-tuning a large model on motion-conditioned data, which is expensive, degrades the base model, and locks the technique to one backbone. This paper asks whether precise motion control can instead be obtained purely at inference time, by exploiting the fact that a crude animation (a dragged object, a depth-reprojected frame) already carries the intended motion, just not photorealistic appearance.

Key Contributions

1. Training-free motion control from crude animations. The authors show that simple user manipulations — cut-and-drag selections or depth-based reprojection of a single image — are effective motion proxies when fed to a video diffusion model through an SDEdit-style noise injection, with the original image used to anchor appearance.
2. Region-dependent dual-clock denoising. A sampling scheme that assigns two different noise levels to two regions of the same video: strong alignment (less noise) inside the user-specified mask, weak alignment (more noise) outside it. This enforces user intent where it exists while letting the background create plausible supporting dynamics.
3. Joint motion-and-appearance control. Because the conditioning signal is a full reference frame rather than a sparse trajectory or flow field, the method can dictate not only where things move but also how they look (color, shape, inserted objects) — a capability prior trajectory-based methods lack.
4. Backbone-agnostic validation. The method is demonstrated unchanged on three I2V backbones (SVD ≈1.5B, CogVideoX 5B, WAN 2.2 14B), matching or beating training-based baselines on object-motion and camera-motion benchmarks with no runtime overhead.

Main Findings

• Training-free matches or beats training-based methods. On the MC-Bench object-motion benchmark, TTM achieves the lowest CoTracker Distance (best trajectory adherence) apart from SG-I2V, while beating DragAnything, MotionPro, and Go-with-the-Flow (GWTF) on most quality metrics — despite none of the competing methods being trained with motion conditioning in its case.
• Dual-clock denoising is the decisive design choice. Ablations show a single noise level either under-constrains motion (CoTracker Distance 27.3 when set too weak), freezes the background (5.5 but no dynamics when set too strong), or, in a RePaint-style variant that keeps injecting the warped mask, produces unnatural-looking output (Imaging Quality 0.578). The dual-clock configuration gives the best trade-off (CTD 7.97, Dynamic Degree 0.427, Imaging Quality 0.617).
• Large gains on camera-motion control. On a 150-scene subset of DL3DV-10K, TTM reduces pixel MSE by 33% and FID by 15.5% relative to the best GWTF setting, and also lowers optical-flow error, meaning generated motion better matches the prescribed camera path.
• SG-I2V's apparent success is partly an artifact. Its low object-tracking error comes with very low background–object CoTracker Distance, indicating the whole scene moves together (camera drift) rather than the specified object.
• Appearance control works as a byproduct. Conditioning on full frames lets users recolor a chameleon while it walks a drawn path, place a hat on a cowboy (correctly appearing in a mirror reflection), and deform clouds — none of which text prompts or flow fields handle reliably.
• Efficiency is the headline. TTM adds no computation over standard sampling; by contrast, GWTF requires roughly 7,680 A100-80GB GPU-hours of fine-tuning to reach motion control on CogVideoX-5B.

Methodology in Plain English

The pipeline has three parts.

Making the motion signal. The user draws a mask over an object in the first frame and drags it along a desired path. The system forward-warps the original image to produce a "warped reference video": the object moves as requested, but pixels are smeared, stretched, and full of holes where occluded background is revealed. The same reference can be built automatically by estimating depth from the single image and reprojecting it into new camera poses for camera-motion tasks.

Injecting motion via noise. Drawing on SDEdit, the authors argue that if you add noise to a rough reference up to the point in the denoising trajectory where the model decides motion, the model will adopt that motion. So they start sampling from a noised version of the warped video instead of pure noise. The catch: at that noise level, appearance detail is destroyed. Using an image-to-video model solves this, because the model keeps referring back to the clean first frame while denoising, preserving identity and texture.

Splitting the clock. A single noise level cannot serve both the dragged region and the untouched background. The warped video is first noised to a high level, t_weak, and denoising begins. While the current step is between t_strong and t_weak, the masked region is overwritten at every step with the warped reference noised to that step — pinning it to the intended motion. Outside the mask, the model denoises freely. Once the step count falls to t_strong, the override stops and normal sampling continues, letting the model blend the two regions into a coherent video. The whole thing is expressed as a single masked blending equation, so no retraining or architectural change is needed.

Why This Matters

Impact on research. The paper shows that a substantial capability — precise spatial motion control — can be obtained by reorganizing the inference procedure rather than training a new model. That reframes a problem the field has largely attacked with fine-tuning (ControlNets, LoRAs, noise-warping training) as a scheduling problem, and provides an architecture-agnostic baseline that future trained methods must beat.

Real-world applications.

• Interactive content authoring: dragging objects in a still image to produce short animated clips, with fast feedback and no GPU cluster.
• Post-production and VFX: re-timing or re-routing camera moves from a single plate, or animating props along director-specified paths.
• Advertising and e-commerce: product videos where both the motion of the object and its color or finish change on demand.
• Animation and storyboarding: turning rough layout sketches or grain-of-truth drags into watchable previews while keeping the artist's intended staging.

Industry relevance. Foundation-model providers can ship motion control as a sampling-time wrapper on top of an existing I2V model, avoiding seven-figure retraining budgets and keeping pace with rapid backbone releases (the paper demonstrates this by porting to WAN 2.2 14B shortly after its release). Studios and tool vendors get controllable generation without maintaining a bespoke fine-tuned checkpoint per model version.

Future Directions

• Removing hyperparameter tuning. The dual-clock scheme still requires manually chosen (t_weak, t_strong) per backbone; an automatic or adaptive schedule would make it truly plug-and-play.
• Richer spatial control. Generalizing the clock to many regions, soft masks, or smoothly varying noise schedules rather than a binary mask with two discrete levels.
• Beyond first-frame anchoring. Identity is preserved only for content visible in frame one; objects entering later cannot be anchored. Extending conditioning to multi-frame or reference-image anchors is an open problem.
• Broader edit and motion types. The authors leave stylistic transformations, articulated and deformable motion, and long-horizon generation for future work, and note that full masks are currently required — unlike methods trained to work from partial or point-level markings.

Target Audience

Graduate students and researchers working on diffusion-based video generation and controllable synthesis; engineers at companies building creative tools, VFX pipelines, or generative video products; and practitioners who want motion control from an existing image-to-video model without collecting data or paying for fine-tuning. Readers unfamiliar with diffusion sampling will need background in denoising schedules and image conditioning to follow the methodology section.

Authors’ abstract

Diffusion-based video generation can create realistic videos, yet existing image- and text-based conditioning fails to offer precise motion control. Prior methods for motion-conditioned synthesis typically require model-specific fine-tuning, which is computationally expensive and restrictive. We introduce Time-to-Move (TTM), a training-free, plug-and-play framework for motion- and appearance-controlled video generation with image-to-video (I2V) diffusion models. Our key insight is to use crude reference animations obtained through user-friendly manipulations such as cut-and-drag or depth-based reprojection. Motivated by SDEdit's use of coarse layout cues for image editing, we treat the crude animations as coarse motion cues and adapt the mechanism to the video domain. We preserve appearance with image conditioning and introduce dual-clock denoising, a region-dependent strategy that enforces strong alignment in motion-specified regions while allowing flexibility elsewhere, balancing fidelity to user intent with natural dynamics. This lightweight modification of the sampling process incurs no additional training or runtime cost and is compatible with any backbone. Extensive experiments on object and camera motion benchmarks show that TTM matches or exceeds existing training-based baselines in realism and motion control. Beyond this, TTM introduces a unique capability: precise appearance control through pixel-level conditioning, exceeding the limits of text-only prompting. Visit our project page for video examples and code: https://time-to-move.github.io/.