WorldCast: World-Consistent Character Video Generation with Large-Angle Camera Control

AI summary

Overview

Research area: Computer vision — controllable video diffusion models, specifically character-centric video generation with explicit camera-trajectory control and multi-view consistency.

Technical level: Advanced. The paper assumes familiarity with latent video diffusion, flow matching, diffusion transformers, camera intrinsics/extrinsics (Plücker ray encoding), point-cloud back-projection and splatting, and multi-view geometry estimation.

Scope (one sentence): The paper introduces CLACT, a character-centric dataset and benchmark for large-angle camera trajectories, and WorldCast, a video generation framework that pairs scale-calibrated 3D geometric guidance with a geometry-aligned multi-view character memory to keep a character's geometry, appearance, and identity stable while the camera follows demanding paths.

What This Paper Is About

Existing camera-controlled video generators can follow simple or moderate camera moves, but they break down when a trajectory combines large rotation, translation, changing orbit radius, and an arbitrary starting azimuth — and those same viewpoint changes expose parts of a character that were never visible in the first frame, causing distortion, appearance loss, and identity drift. The authors build a dataset and benchmark (CLACT) to study these two coupled problems together, then propose WorldCast, which renders the first frame as a 3D point cloud along the requested trajectory for geometric guidance and retrieves view-relevant character features from additional character views to fill in what the first frame cannot show.

Key Contributions

1. CLACT dataset and benchmark. A character-centric collection of large-angle camera trajectories with accurate camera geometry and multi-view character observations: an 80,000-sample training corpus (50,000 Unreal Engine 5 camera-motion sequences plus 30,000 Blender-rendered samples with multi-view character anchors, spanning 99 characters and 33 environments), and an evaluation set of 42 character scenes (17 synthetic, 25 real-world) each tested under six camera trajectories with different initial azimuths.

2. WorldCast framework. A two-branch generation framework that couples scale-aligned geometric guidance (depth back-projection, scale calibration against the reference video, point-cloud rendering with coverage masks) with multi-view character memory, built on the Wan2.1-I2V-14B backbone.

3. Geometry-Aligned Memory Attention (GAMA). A mechanism that estimates camera poses jointly from the initial frame and the character anchors with a pretrained multi-view geometry model, aligns anchor geometry to the target trajectory with an Umeyama similarity transform, and uses shared Plücker ray encoding so that video tokens retrieve view-relevant character features from the aligned memory.

4. Systematic ablations. Component-level studies showing the contribution of GAMA, anchor geometry alignment, anchor foreground masking, UE5 large-angle training data, Plücker ray conditioning, and the number of character anchors.

Main Findings

• Best camera control among compared methods. On the CLACT benchmark, WorldCast records ATE 0.065, RTE 0.020, and RRE 0.896. The strongest camera-controlled baseline, Uni3C, records ATE 0.098, RTE 0.033, RRE 1.069; GEN3C records ATE 0.130, RTE 0.038, RRE 1.255; Lyra 2.0 records ATE 0.142, RTE 0.034, RRE 1.741. CamI2V is far behind at ATE 0.351 and RRE 11.490.

• Best cross-view geometric consistency and identity similarity. WorldCast reaches MEt3R 0.228 and ArcFace 0.555, against 0.306/0.433 for GEN3C, 0.316/0.394 for Lyra 2.0, and 0.324/0.374 for Uni3C. Subject consistency is 0.917 and DINO 0.719, both the highest in Table 1; CLIP is 0.830, the highest reported.

• Visual quality is competitive rather than dominant. WorldCast's image quality is 0.719, while Lyra 2.0 reports 0.725 — slightly higher. The paper describes its own visual quality as "competitive," not superior.

• Extra reference views do not reliably help prior methods. Adding anchors to ViewCrafter changed ATE from 0.353 to 0.361 and ArcFace from 0.223 to 0.195; for MVGenMaster, ATE moved from 0.125 to 0.218. The authors attribute this to the difficulty of aligning sparse cross-view geometry under large viewpoint changes.

• GAMA is the single most important component. Removing the entire GAMA branch degrades ATE from 0.065 to 0.100, MEt3R from 0.228 to 0.333, and ArcFace from 0.555 to 0.370, showing that multi-view character memory drives most of the consistency gain.

• Anchor geometry alignment matters most for rotation. Without it, RRE rises from 0.896 to 1.374 (the worst RRE in the ablation table) while ArcFace stays relatively high at 0.547, indicating that identity can survive while the geometric relationship between anchors and target views breaks down.

• Foreground masking helps across the board. Removing anchor masking gives ATE 0.078, MEt3R 0.247, IQ 0.709 — worse than the full model on camera control, consistency, and quality, supporting the benefit of isolating the character from scene background.

• Large-angle training data and ray conditioning both matter. In the first-stage ablation, dropping the UE5 collection raises ATE from 0.100 to 0.359, RTE from 0.031 to 0.156, and RRE from 1.149 to 4.295; dropping Plücker conditioning gives ATE 0.275, RTE 0.056, RRE 2.113.

• More anchors help, with diminishing returns. Increasing character anchors from one to four improves cross-view geometry, identity, and appearance similarity; the gain from three to four anchors is smaller, suggesting additional views contribute less new information.

• Qualitative behavior. In a large-angle orbit combined with dolly motion, Uni3C shows severe character distortion, particularly in facial regions, and MVGenMaster — despite using multi-view references containing both character and scene — loses fine character detail; WorldCast follows the prescribed motion more reliably and better preserves identity, 3D structure, and appearance.

Methodology in Plain English

The setup. The system takes one character image from any starting angle, a text prompt, a target camera path (intrinsics plus camera-to-world poses per frame), and a small set of extra views of the same character (front, back, side). It must produce the remaining video frames.

Step 1 — Make the depth map and the camera path agree on scale. A pretrained depth estimator gives the depth of the first frame, but its scale may not match the scale of the camera translations. The authors try several candidate scales, re-render the first frame as a point cloud along each rescaled path, measure how much image motion each candidate produces (corner tracking with pyramidal Lucas–Kanade optical flow, RANSAC-filtered affine fitting), and pick the scale whose motion magnitudes best match the reference video's motion.

Step 2 — Turn the first frame into 3D guidance. With the calibrated scale, every pixel with valid depth is back-projected into a colored 3D point cloud, then rendered along the target trajectory using point splatting with depth buffering. This yields an RGB guide image per frame plus a coverage mask marking pixels the point cloud could not support — exactly the regions the generator must invent.

Step 3 — Feed guidance into the diffusion model. The rendered guide is VAE-encoded and concatenated channel-wise with the noisy latent, the initial-frame latent, and the resized coverage mask, then projected by a 1×1×1 convolution into the patch embedding. Camera geometry is added separately as Plücker ray grids in a normalized frame, projected and added element-wise to the token features.

Step 4 — Build a memory of the character from other views. Character foregrounds are segmented out of the first frame and all anchors to suppress backgrounds. A pretrained multi-view geometry model estimates their camera poses jointly. A similarity transform (Umeyama alignment, using camera-center correspondences where the first frame is the shared reference) maps the anchor geometry into the target trajectory's coordinate frame, and the same camera normalization is applied to both, so anchors and target views live in one shared frame.

Step 5 — Retrieve from that memory during generation. Each masked anchor is VAE-encoded, given a Plücker ray grid from its aligned pose, and turned into tokens. Twenty memory-attention blocks are inserted after backbone blocks 0, 2, …, 38; memory and video tokens are concatenated, projected from 5120 to 1024 dimensions, timestep-modulated via AdaLN, and jointly attended with eight heads of dimension 128. The residual output is projected back and added only to the video tokens, with each output projection zero-initialized so the pretrained backbone is preserved at the start of adaptation.

Step 6 — Train in two stages. Stage 1 trains camera control on 76,698 rendered SpatialVID sequences plus the 50,000 UE5 sequences for 5 epochs at batch size 64, updating only the geometric projection, patch embedding, backbone self-attention, and Plücker projection. Stage 2 initializes from that checkpoint and trains GAMA on the 30,000 Blender samples with anchors for 8 epochs at batch size 32, optimizing only the memory-attention bank and anchor adapter. Both use AdamW at a constant learning rate of 10⁻⁴ with zero weight decay, on 81-frame clips at 832×480.

Why This Matters

Impact on research. The paper reframes large-angle camera control and character consistency as one coupled problem rather than two separate ones: the trajectory determines both the geometry to follow and which unseen character regions must be synthesized. CLACT supplies paired camera-motion sequences and multi-view character observations, which the authors argue existing camera-motion datasets do not consistently provide, enabling joint measurement of trajectory accuracy and cross-view consistency. The result is a benchmark where ATE, MEt3R, identity similarity, and image quality are reported together.

Real-world applications:

• Cinematic and previsualization workflows that need a director-specified camera move around a character without reshooting.
• Virtual production and game cinematics, where a character asset must survive an orbiting or dollying camera.
• Interactive or personalized content creation from a single portrait plus a few reference views.
• 3D-aware character asset preview, where turntable-style inspection must not distort the subject.

Industry relevance. The work comes from Wuhan University together with MiLM Plus at Xiaomi Inc., and the paper states an intent to release benchmark generation and evaluation code, pose-alignment and identity-evaluation procedures, trajectory configurations, training and inference configurations, and a checkpoint. The paper also includes an ethics statement noting that the training benchmark is built primarily from controlled simulations using licensed or publicly available 3D assets, that released real-world evaluation data will exclude personally identifiable private individuals, and that deceptive impersonation is discouraged.

Future Directions

• More flexible anchor acquisition. The conclusion names this explicitly: the current pipeline depends on a supplied set of character views (front, back, side) plus precomputed poses, so obtaining anchors more freely — fewer views, or generated ones — is an open problem.
• More complex dynamic scenes. The method is demonstrated on character-centric scenes with a prescribed trajectory; extending to scenes with independent motion is listed as future work.
• Longer and interactive camera trajectories. The current experiments use 81-frame clips and fixed trajectory families; longer paths and user-in-the-loop camera control are proposed next steps.
• Closing the visual-quality gap. WorldCast leads on camera and consistency metrics but trails Lyra 2.0 slightly on image quality (0.719 vs 0.725), leaving room to improve fidelity without sacrificing control. The paper also defers additional discussions and supplementary experiments to an appendix not included in the provided content.

Target Audience

Researchers and advanced practitioners in generative video and 3D vision — particularly those working on camera-controllable diffusion, novel-view and re-rendering systems, and identity-consistent subject generation — who want a benchmark for large-angle camera motion and a concrete recipe for combining explicit geometric guidance with multi-view memory. It is also useful for technical leads in film, virtual production, or consumer content tools evaluating whether current camera-control methods are reliable enough for large viewpoint changes. Readers without a background in diffusion transformers and multi-view geometry will find the method sections dense, though the problem framing and benchmark design are accessible.

Authors’ abstract

Precise camera control is essential for cinematic character video generation, yet existing methods struggle to follow large-angle trajectories with combined rotation, translation, and radius variation from arbitrary initial azimuths. Such viewpoint changes also reveal character regions unseen in the initial frame, causing geometric distortions, appearance degradation, and identity drift. To systematically study these coupled challenges, we introduce CLACT, a large-scale Character-centric dataset and benchmark covering diverse Large-Angle Camera Trajectories, with accurate camera geometry and multi-view character observations for training and evaluation. We further propose WorldCast, a framework for world-consistent character video generation under diverse large-angle camera motions. Specifically, we back-project the initial frame into a 3D point cloud using estimated depth, align the point-cloud scale with the target camera trajectory, and render it along the trajectory to obtain explicit geometric guidance and coverage masks. To complement the incomplete character information, we introduce Geometry-Aligned Memory Attention (GAMA). Using relative camera poses estimated jointly from the initial frame and multi-view character anchors, we express anchor and target-view geometry in a common coordinate frame. GAMA uses this geometric alignment to guide the retrieval and injection of view-relevant character features during generation. Evaluations across diverse large-angle camera trajectories demonstrate accurate trajectory following while preserving character geometry, appearance, and identity.