OmniShowcase: Omnidirectional Subject-Consistent Video Generation from Multi-View Images

AI summary

Overview

Research area: Computer vision — subject-driven video generation (subject-to-video, or S2V), combining video diffusion transformers with feed-forward 3D reconstruction and novel-view synthesis.

Technical level: Advanced. The paper assumes familiarity with diffusion/rectified-flow models, latent video VAEs, LoRA tuning, and multi-view 3D reconstruction (pointmaps, 3D Gaussians).

Scope (one sentence): The paper introduces OmniShowcase, a system that takes four reference images of an object and generates a video that follows a commanded camera path — including full 360-degree orbits — while keeping the object's identity and 3D shape intact, backed by a new 4,000-video training corpus (Showcase-4K) and a held-out benchmark (ShowcaseBench).

What This Paper Is About

When a video model is asked to show a specific object from new angles, any surface the input images never captured has to be invented — and models usually invent it badly, so the object changes shape, grows extra parts, or morphs as the camera moves. Existing multi-view methods feed in several reference images but never state the 3D structure that ties those views together, and their training videos do not guarantee that the same underlying 3D object is preserved across a viewpoint change. OmniShowcase solves both halves of this: it builds training videos by rendering one real 3D mesh along many camera paths, and at generation time it hands the model an explicit geometric cue — a rotating "turntable" of surface-normal maps reconstructed from the reference views.

Key Contributions

1. An automated 3D-consistent data synthesis pipeline. From curated 3D assets, Blender renders a target camera path plus four reference views at 90-degree yaw intervals; the subject is then composited into a prompt-driven, realistic background by a video inpainting model, while the rendered subject pixels stay unchanged. Every stage runs without human intervention, with vision-language model gates filtering degenerate assets, floating subjects, and broken reconstructions.

2. Showcase-4K and ShowcaseBench. Showcase-4K is a corpus of 4,000 videos covering twelve camera motions across 88 trajectory configurations, each paired with source meshes, reference images, and reconstruction-derived surface-normal sequences. ShowcaseBench is a held-out evaluation set of 50 subjects spanning four 360-degree camera trajectories that shares no 3D asset with the training corpus, and it ships with the source meshes and camera parameters used to render each video.

3. Geometry-aware in-context conditioning without frame alignment. The model reconstructs geometry from the four reference views at both training and inference time, renders it as a 25-frame surface-normal turntable, and appends it as its own span in the transformer's context — so no frame-by-frame correspondence between the 25 normal frames and the 80 target frames is required.

4. State-of-the-art results on identity and 3D consistency. Against single- and multi-view baselines on both ShowcaseBench and NAVI, OmniShowcase achieves the best DINO, CLIP, MEt3R, and GT-CD scores while keeping text alignment and VBench video-quality metrics competitive.

Main Findings

• Best identity preservation on both benchmarks. On ShowcaseBench, DINO 0.799 and CLIP 0.925 versus the next-best 3DreamBooth at 0.707 / 0.907. On NAVI, DINO 0.828 and CLIP 0.918 versus MV-S2V at 0.770 / 0.912.

• Best 3D consistency by a wide margin. On ShowcaseBench, MEt3R 0.190 and GT-CD 0.0824, compared with 3DreamBooth at 0.325 / 0.1717 and MV-S2V at 0.351 / 0.2306. On NAVI, MEt3R 0.249 and GT-CD 0.0975, versus MV-S2V at 0.350 / 0.1816.

• Competitive but not dominant on everything. Aesthetic quality on ShowcaseBench is 0.574, below MAGREF-MV (0.614) and 3DreamBooth (0.598); on NAVI it is 0.551, below MAGREF-MV (0.574) and Phantom-MV (0.572). Imaging quality is the highest of the compared methods on both sets (74.29 and 74.58).

• No baseline is strong on every axis. Phantom-MV and MAGREF-MV trail on identity and camera travel, and the lead between MV-S2V and 3DreamBooth flips between the two benchmarks. Qualitatively, Phantom-MV and MAGREF-MV barely orbit the subject; MV-S2V stops short of the full rotation, grows a second spout on a teapot, and snaps back to an earlier frame; 3DreamBooth misses both identity and geometry.

• Removing geometry conditioning hurts. Dropping the normal turntable entirely drops ShowcaseBench DINO from 0.799 to 0.752, worsens MEt3R from 0.190 to 0.265, and roughly doubles GT-CD from 0.0824 to 0.2092.

• Coloured renders are worse than normals. Replacing normals with a textured render of the same reconstruction gives DINO 0.755 and MEt3R 0.270 — the reconstructed colour competes with the reference images for appearance, and text alignment falls furthest of the three ablations on ShowcaseBench (ViCLIP 0.236).

• Thinning the turntable costs coverage. Reducing the turntable from 25 to 9 views gives DINO 0.752, MEt3R 0.263, GT-CD 0.1777, and RotCoverage 0.698 (versus 0.665 for the full turntable).

• Human raters agree with the metrics. In a study with 44 participants comparing 24 held-out videos across five methods, OmniShowcase takes between 70.9% and 79.4% of the votes on all four criteria, while no baseline passes 12.7%.

• Data efficiency. Showcase-4K is only 4K samples, versus 14K for MV-S2V, 200K for 3DreamBooth, 1M for Phantom, and "large" for MAGREF — yet it outperforms them on the reported metrics, attributed to the geometric consistency of rendered data.

• Rendered training clips score better than generator-produced ones. Scored under the same metrics with the same reference views and ground-truth geometry, the paper's 3D-asset renders beat paired Uni3C clips (DINO 0.8030 vs 0.7167; MEt3R 0.1871 vs 0.3052).

• Fewer references still work, but four is best. Performance degrades gracefully from n=4 down to n=1 (ShowcaseBench DINO 0.799 at n=4 versus 0.702 at n=1), with GT-CD at 0.0824 for n=4 versus 0.1704 for n=1. NAVI identity stays nearly flat across reference counts because its subjects are small and largely convex, and ViCLIP is saturated at every count on both benchmarks.

Methodology in Plain English

The pipeline has two halves that mirror each other: how training data is made, and how the model is conditioned.

Making training data from 3D assets. The team starts with meshes from Objaverse, the Digital Twin Catalog, and Google Scanned Objects. A heuristic filter drops degenerate meshes, then a vision-language model inspects four rendered views of each candidate and keeps only assets with valid geometry, plausible textures, and consistent category. For each kept mesh, Blender renders an 81-frame target video along a camera path plus four reference views at 90-degree yaw intervals, so references and target are guaranteed to depict the same shape. Because raw renders have clean backgrounds, a video inpainting model paints a realistic background around the fixed subject using its alpha-channel mask, with the scene prompt built from an LLM-written scene bank combined with the sample's camera motion and lighting. The target is trimmed to 80 frames to fit the video VAE's temporal constraint.

Making the geometry cue the same way at train and test time. Rather than using the source mesh as the condition — which would not be available at inference — the pipeline reconstructs geometry from only the four reference views: it masks the subject, predicts aligned pointmaps with a feed-forward multi-view estimator, fuses them into 3D Gaussians, and renders a yaw orbit at 15-degree elevation at 25 uniformly spaced poses. Each rendered depth map is converted to a surface normal, giving a 25-frame normal "turntable."

Feeding it to the model. The base model is a pretrained image-to-video diffusion transformer (Wan2.1 I2V-14B-480P) with frozen weights and LoRA adapters (rank 128). The four reference images are encoded independently; the 25 normal frames and 80 target frames are each encoded by the video VAE separately, producing 7 and 20 latent frames. Concatenating along the temporal axis gives a 31-position sequence (4 + 7 + 20). A binary mask marks the first 11 positions as given, so only the target latents are noised during training and only they contribute to the loss. Three input streams are stacked channel-wise — the partially noised latents, the mask expanded to 4 channels, and the clean conditioning latents — giving 16 + 4 + 16 = 36 input channels. The crucial design choice is that the normal turntable gets its own span rather than extra channels: a channel-wise condition would have to be read frame by frame, forcing an alignment between 25 normals and 80 target frames that does not exist, since the turntable is a fixed yaw sweep and the target follows whatever trajectory the prompt names.

Training and inference. Training runs 33,000 steps at batch size 2 per device on four NVIDIA B200 GPUs (about 50 hours) at 480×832 resolution. At inference, the same reconstruction and rendering steps are applied to a new subject's reference views, 50 sampling steps are run with three-way classifier-free guidance (reference stream 5.0, text stream 5.0), and the 27-frame latent sequence is jointly decoded into 1 + 4(27−1) = 105 frames — 25 normal-condition frames followed by 80 target frames.

Why This Matters

Impact on research. The paper reframes subject-to-video generation as a geometry problem rather than a pure appearance-matching problem, and it argues that the source of training data matters more than its size — 4K geometrically consistent rendered clips outperform corpora an order of magnitude larger. It also shows a practical pattern: reconstruct geometry from the same sparse inputs available at test time, so the training and inference conditioning signals match exactly. The release of Showcase-4K, ShowcaseBench, and the synthesis code gives the field a benchmark with ground-truth meshes and camera parameters, which is exactly what consistency metrics need.

Real-world applications:

• E-commerce and product showcases: generating a 360-degree video of a single product from a handful of photos, without shipping it to a studio.
• Game and VFX asset previews: turning a scanned or modelled prop into orbit footage for pitch material before final rendering.
• Cultural heritage and museum catalogues: producing rotating documentation of scanned artefacts for online exhibitions.
• Advertising and social content: repurposing existing product photography into camera-motion clips on demand.

Industry relevance. The approach avoids per-subject optimization (unlike 3DreamBooth), which makes it a feed-forward, deployable pipeline, and the LoRA-on-a-frozen-backbone recipe is the standard cheap fine-tuning pattern that production teams already use. The automated asset filtering, VLM gating, and captioning stages also mean the data engine can be scaled without manual annotation, which is the main cost driver in this kind of work.

Future Directions

• Better backgrounds. The authors state that generated backgrounds sometimes look synthetic, which points to stronger scene compositing or joint background-and-subject generation.
• Non-rigid and deformable subjects. The method assumes the subject is rigid across the four references; handling objects that bend, articulate, or deform is explicitly left to future work.
• Loosening the fixed turntable assumption. The normal condition is a fixed yaw sweep at 15-degree elevation, so the geometry cue is trajectory-independent by design — a natural next question is whether trajectory-conditioned or elevation-varied geometry cues help further.
• Extending beyond four references and toward scenes. The reference-count ablation shows graceful degradation and NAVI's small convex objects do not benefit from extra views; multi-object scenes and complex occlusion are open territory.

Target Audience

Researchers and graduate students working on video diffusion, subject-driven generation, and personalized generative models, who will care most about the conditioning architecture (a separate context span instead of channel concatenation) and the dataset construction argument. Also useful for 3D vision researchers interested in closing the loop between feed-forward reconstruction and generative video, and for applied engineers in e-commerce, advertising, or content tooling who need a feed-forward, no-per-subject-optimization way to produce orbit videos from a few photographs. Readers should already be comfortable with diffusion transformers and latent video VAEs; the 3D reconstruction and evaluation sections assume some background in multi-view geometry and point-cloud metrics.

Authors’ abstract

Generating videos of a specific subject from a few reference images remains challenging when the camera reveals surfaces that are not visible in the input. Early subject-to-video (S2V) methods condition on a single reference image and must therefore infer the subject's unseen appearance and geometry as the viewpoint changes. To address this limitation, recent methods extend S2V to multiple reference views, providing more information about the subject from different viewpoints. However, they do not explicitly specify the 3D structure that connects these observations. In addition, existing training data do not guarantee that viewpoint changes preserve the same underlying 3D subject. We present OmniShowcase, built on two components: (i) a fully automated data synthesis pipeline that produces training videos along diverse camera trajectories, with viewpoint variation driven by underlying subject geometry, even spanning full 360-degree orbits; and (ii) a geometry-conditioning method that provides the video model with a turntable of surface-normal maps reconstructed from the reference views without requiring temporal alignment with the generated frames. We construct Showcase-4K, a corpus of 3D-asset-based videos covering twelve camera motions across 88 trajectory configurations with reconstruction-derived surface-normal conditioning, and ShowcaseBench, a held-out evaluation set with the source meshes and camera parameters used to render each video. Experiments show state-of-the-art subject identity and 3D consistency while maintaining competitive video quality and motion. Project page: justin4ai.github.io/OmniShowcase