FlashRender: Few-Step Generative Rendering via Camera-Controlled Video MeanFlow

AI summary

Overview

Research area: Computer vision / generative video modeling — specifically camera-controlled generative rendering (video-to-video re-rendering along a new camera trajectory) accelerated with few-step diffusion/flow distillation.

Technical level: Advanced. The paper assumes familiarity with flow matching, MeanFlow, distribution-matching distillation (DMD/DMD2), diffusion transformers (DiT), and visual geometry models (VGGT).

Scope: A three-stage training framework (RETA alignment, MeanFlow fine-tuning, on-policy flow map distillation) that lets a camera-controlled video generative rendering model produce high-quality retakes in 4-NFE instead of 50-NFE.

What This Paper Is About

Generative rendering models can re-shoot an existing video along a user-specified camera trajectory, but they need many sampling steps, which makes inference expensive. Naively reducing the step count introduces "sampling-step-dependent camera control": the same target trajectory produces different realized camera motion (altered scene scale, inconsistent spatial grounding of dynamic objects) depending on how many steps are used. FlashRender's goal is to make few-step camera-controlled generative rendering both accurate and controllable, and it attributes the failure to discretization error along a highly curved denoising trajectory.

Key Contributions

1. Identification and diagnosis of sampling-step-dependent camera control. The authors name step-dependent camera control as a prominent manifestation of discretization error in existing multi-step generative rendering models, and show that resolving it substantially lowers denoising trajectory curvature, making later step distillation easier.

2. Representation Transformation and Alignment (RETA). A training objective that aligns intermediate source-video representations of the generative model with target-video features from a frozen VGGT encoder (further processed by a single global self-attention layer with camera tokens). RETA encodes the source-to-target geometric transformation directly in the source-video stream, giving timestep-independent target-view geometry and consistent camera control across sampling steps.

3. MeanFlow fine-tuning on the lower-curvature trajectory, using interpolated timestep conditioning and a finite-difference approximation of the Jacobian-vector product term instead of JVP, so the model learns average velocity fields that shortcut the denoising trajectory.

4. On-policy flow map distillation for fixed four-step sampling, where the MeanFlow model acts as a backward simulator and is optimized on its own rollout samples with DMD and an adversarial (DMD2-style) objective, correcting training-inference mismatch. Together the three components play complementary roles.

Main Findings

• Few-step performance matches multi-step at 25× lower sampling cost. FlashRender matches multi-step baselines in video quality and geometric consistency with 25× lower sampling cost while achieving superior camera controllability. On DAVIS (Table 1), FlashRender at 4-NFE records Aesthetic 0.5182, Imaging 0.6654, Dyn-MEt3R 0.8571, MEt3R 0.3071, TransErr 0.0122, RotErr 1.236, versus 6.5-minute, 2×50-NFE baselines such as GeoAlign (0.5250 / 0.6683 / 0.8532 / 0.3111 / 0.0149 / 1.495) and ReDirector (0.5149 / 0.6668 / 0.8477 / 0.3073 / 0.0165 / 1.666). Reported runtime is 0.2 minutes for FlashRender versus 6.5 minutes for the multi-step models.

• The multi-step model FlashRender-MS is already competitive. It achieves superior MEt3R, TransErr, and RotErr compared with GeoAlign, which applies reward-based post-training to ReDirector (FlashRender-MS: 0.5214 / 0.6625 / 0.8491 / 0.3059 / 0.0143 / 1.486 at 6.5 minutes).

• Few-step baselines built from the same distillation framework underperform. Applying the framework to ReDirector (ReDirector-FS) and GeoAlign (GeoAlign-FS) at 4-NFE yields worse results (e.g., ReDirector-FS MEt3R 0.3492, RotErr 1.746; GeoAlign-FS MEt3R 0.3161, RotErr 1.528) than FlashRender. The authors attribute this to MeanFlow training without RETA failing to sufficiently mitigate discretization error, which in turn limits on-policy distillation.

• NeoVerse, the few-step explicit baseline, has high imaging quality but poor geometry. NeoVerse achieves the best Imaging Quality (0.6715) among few-step methods but poor geometric consistency and camera accuracy (Dyn-MEt3R 0.7523, MEt3R 0.3619, TransErr 0.0343, RotErr 3.495), which the authors attribute to its post-hoc few-step LoRA adaptation. Qualitatively, it often duplicates or hallucinates objects.

• Every component is necessary (ablations). Removing RETA (w/o RETA: MEt3R 0.3367, RotErr 1.685) or replacing it with VideoREPA (MEt3R 0.3390, RotErr 1.514, worse even than no alignment) degrades results. Skipping Stage 2 (w/o MeanFlow: Imaging 0.6312, MEt3R 0.3215, RotErr 1.294) or Stage 3 (w/o distillation: MEt3R 0.3119, RotErr 1.311) also degrades performance. MeanFlow-based distillation beats FlashRender-MS + DMD2 (MEt3R 0.3190, RotErr 1.544) and FlashRender-MS + DMD2 + DFD (MEt3R 0.3138, RotErr 1.324).

• RETA lowers denoising trajectory curvature. The paper reports (Fig. 3) that RETA consistently reduces trajectory curvature throughout the 50-step sampling process relative to FlashRender-MS without RETA.

• Generalization to out-of-distribution conditions. On DyCheck, using the first 81 frames as input and the provided camera poses as the target trajectory, FlashRender at 4-NFE achieves the best Aesthetic (0.4454), Imaging (0.7487), Dyn-MEt3R (0.8424), and MEt3R (0.3323) among compared methods, while achieving the best fidelity among few-step baselines (PSNR 10.47, LPIPS 0.669). Its PSNR is lower than multi-step models (e.g., ReDirector 10.82), which the authors attribute mainly to scene-scale mismatch with the ground-truth video.

Methodology in Plain English

FlashRender is built by fine-tuning Wan2.1-1.3B-CamCtrl and trained in three stages.

Stage 1 — a better multi-step model. The model takes a source video and a target camera trajectory. Each source frame is expressed in the coordinate system of its corresponding target view by adding the frame-wise source-to-target relative pose to the source-video latent, and both video streams share the same camera-conditioned positional encoding (RoCE on the target trajectory, with shared 3D RoPE). Building on this, RETA aligns the model's intermediate source-video representations (taken from the eighth DiT block) with target-video features from a frozen VGGT encoder using cosine similarity, which forces the model to internalize the source-to-target geometric transformation. The stage-1 loss is flow matching plus a weighted RETA term (λ = 0.1).

Stage 2 — learning shortcuts. Because RETA yields a less curved denoising trajectory, the authors fine-tune with the MeanFlow objective to predict the average velocity over an interval, which directly fights discretization error. Timestep r is injected via interpolated conditioning (g = 0.75), and the expensive Jacobian-vector product is replaced by a finite-difference approximation (δt = 0.005). Classifier-free guidance is baked into the target velocity with guidance scale ω = 5, and an adaptive loss weight from TMD is used.

Stage 3 — fixing self-rollout errors. The MeanFlow model generates its own samples using exactly the four-step inference schedule (intermediate timesteps 0.249, 0.499, 0.749) and is then optimized on those samples with DMD plus a DMD2-style adversarial loss (λ_GAN = 0.03), against a fake-score model and discriminator. This closes the gap between how the model is trained and how it is actually used.

Setup. Training uses the MultiCamVideo and SynCamVideo datasets on eight RTX PRO 6000 GPUs with global batch size 8; all videos contain 81 frames at 480×832, and all three stages take about two weeks. Stage 1 and Stage 2 run 20k iterations (learning rates 10⁻⁴ and 3×10⁻⁵), Stage 3 runs 5k iterations (10⁻⁵). Evaluation follows ReDirector's protocol: 50 DAVIS source videos × 10 target camera trajectories = 500 cases, with VBench Aesthetic/Imaging Quality, Dyn-MEt3R, frame-wise MEt3R, TransErr, and RotErr. Inference uses a sigma shift of 5.0 and CFG scale 5 for multi-step rendering.

Why This Matters

Impact on research. The paper reframes a practical failure mode of few-step generative rendering — camera control that changes with the number of sampling steps — as a geometric-representation problem rather than purely a distillation problem. It connects representation alignment (REPA-style) to step distillation, arguing that reducing trajectory curvature is a prerequisite for effective few-step approximation. It also supplies a recipe for making MeanFlow-scale training tractable in video by replacing JVP with finite differences.

Real-world applications.

• Virtual production and filmmaking: re-shooting existing footage from physically impractical camera angles.
• Stabilization of shaky footage by re-rendering along a smoothed camera path.
• Novel-view synthesis for dynamic scenes under occlusions and unseen regions.
• Rapid pre-visualization and shot iteration, where generating retakes in seconds (0.2 minutes reported for FlashRender) rather than minutes changes the editing workflow.

Industry relevance. The 25× reduction in sampling cost at 4-NFE, plus the reported ability to match multi-step baselines and to handle out-of-distribution target trajectories, targets production cost and latency constraints. The runtime is measured on a single NVIDIA B200 GPU at 480×832, and the method inherits an existing open camera-controlled model (Wan2.1-1.3B-CamCtrl), making it comparatively straightforward to adopt.

Future Directions

• Extending beyond fixed four-step sampling. The third stage is deliberately specialized for fixed four-step inference (unlike AnyFlow, which targets any-step generation); generalizing the distillation to arbitrary step counts while retaining the camera-control benefits is left open.
• Closing the fidelity gap on out-of-distribution data. On DyCheck, FlashRender's PSNR is lower than multi-step models, which the authors trace mainly to scene-scale mismatch with ground-truth video — a target for future scale-alignment work.
• Reducing reliance on auxiliary geometry models. RETA depends on a frozen VGGT encoder and on source camera trajectories that are either given or estimated with ViPE; the sensitivity to estimated camera poses and depth is not resolved.
• Broadening evaluation. Results are reported on DAVIS (50 videos × 10 trajectories) and DyCheck (first 81 frames); whether the curvature-reduction argument transfers to other domains, longer videos, or higher resolutions is not established.

Target Audience

Researchers and engineers working on video diffusion/flow models, camera-controlled video generation, and few-step distillation will get the most from this paper. It is also relevant to graphics and virtual-production practitioners interested in fast camera re-rendering of real footage, and to readers tracking MeanFlow and DMD2-style distillation applied at video scale. Readers without background in flow matching or diffusion distillation will find the methodology section demanding.

Authors’ abstract

We present FlashRender, a few-step generative rendering framework that retakes a source video along a target camera trajectory in seconds. We identify sampling-step-dependent camera control as a prominent manifestation of discretization error in existing multi-step generative rendering models and show that resolving this inconsistency substantially lowers denoising trajectory curvature, facilitating subsequent step distillation. To this end, we introduce Representation Transformation and Alignment (RETA), which aligns hidden source-video representations with target-video features from a frozen visual geometry model. This directly encodes the geometric transformation within the source-video stream, enabling sampling-step-consistent camera control. We then fine-tune the model with the MeanFlow objective on the lower-curvature denoising trajectory induced by RETA, allowing the model to more effectively address discretization error. Finally, we apply on-policy flow map distillation to correct self-rollout errors under fixed few-step sampling. Extensive experiments show that RETA, MeanFlow, and on-policy flow map distillation play complementary roles in few-step generative rendering. Together, they enable our approach to match multi-step baselines in video quality and geometric consistency at 25x lower sampling cost while achieving superior camera controllability, even under out-of-distribution target camera trajectories.