Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator

AI summary

Overview

• Research area: Computer vision / generative 3D modeling, specifically text-to-3D generation built on latent diffusion models (LDMs), video generation, and feedforward 3D reconstruction.
• Technical level: Advanced. The paper assumes familiarity with latent diffusion models, VAEs, multi-view generation, 3D Gaussian splats (3DGS), pointmaps, and reward-based finetuning.
• Scope (one sentence): The paper introduces VIST3A, a framework that stitches a pretrained feedforward 3D reconstruction network onto the latent space of a text-to-video VAE to form a 3D decoder, then aligns the video generator to that decoder with direct reward finetuning, and evaluates the resulting text-to-3DGS and text-to-pointmap models across several benchmarks, video generators, and 3D foundation models.

What This Paper Is About

Existing end-to-end text-to-3D latent diffusion models take a pretrained 2D image or video model and train a custom VAE-style decoder to turn multi-view 2D latents into 3D outputs. The authors argue this is the weak link: the decoder must learn 3D reconstruction more or less from scratch (requiring extensive training and large, hard-to-obtain datasets), and it is only weakly aligned with the generative model, since the generator is finetuned with generative losses that only indirectly promote 3D-consistent latents. The goal of this work is to build a strong 3D decoder by reusing a pretrained feedforward 3D model instead of training one, and then explicitly aligning the video generator to that decoder.

Key Contributions

1. A model stitching procedure for constructing 3D VAEs. The authors identify the layer in a pretrained feedforward 3D model whose activations are most linearly related to the video VAE's latent space, fit a single linear stitching layer in closed form, discard the front portion of the 3D model, and keep the downstream portion as the 3D decoder.

2. Self-supervised stitched-decoder finetuning. The stitched decoder (the stitching layer plus the retained downstream part) is finetuned to reproduce the outputs of the original 3D model as pseudo-targets, using a weighted sum of ℓ1 losses over all outputs, with no labels required.

3. Direct reward finetuning for generator-decoder alignment. The video generator is aligned to the stitched decoder using a three-part reward (multi-view image quality, 3D representation quality, and 3D consistency) backpropagated through the denoising trajectory, with a stabilization scheme generalizing DRTune, randomized timestep sampling, and randomized gradient backpropagation steps.

4. Demonstrated generality and a new output modality. The stitching scheme is shown to work across several video generators (Wan 2.1 T2V large, CogVideoX, SVD, HunyuanVideo) and several 3D foundation models (MVDUSt3R, VGGT, AnySplat), enabling both text-to-3DGS and, by choosing a suitable 3D base model, high-quality text-to-pointmap generation.

Main Findings

• Text-to-3DGS results beat prior methods. On T3bench (object-centric), VIST3A with Wan + MVDUSt3R reaches 58.83 Imaging Quality, 56.55 Aesthetic Quality, 32.75 CLIP, and Alignment/Coherence/Style scores of 3.56/3.89/3.56, while VIST3A with Wan + AnySplat reaches 57.03/54.11/31.38 and 3.36/3.68/3.17. For comparison, the best baselines on those columns are Director3D (54.32 Imaging, 53.33 Aesthetic, 3.25 Align, 3.43 Coher, 3.05 Style) and VideoRFSplat (30.13 CLIP).

• Scene-level synthesis improves markedly. On SceneBench, VIST3A Wan + MVDUSt3R scores 62.08 Imaging, 55.67 Aesthetic, 30.26 CLIP and 3.72/3.97/3.47 for Alignment/Coherence/Style; Wan + AnySplat scores 64.87/56.96/30.18 and 3.67/3.86/3.40. The authors report Imaging Quality above 60 and Coherence above 3.8 for their models.

• Large gains on long, detailed prompts. On DPG-Bench, VIST3A Wan + MVDUSt3R scores 81.82 Global, 84.31 Entity, 86.13 Attribute, 68.93 Relation, and 54.55 Other; Wan + AnySplat scores 78.79/85.58/84.12/76.70/45.45. The paper states their models mostly score above 75, often around 85, values it describes as previously out of reach.

• Stitching improves novel view synthesis. On RealEstate10K, stitching AnySplat onto any tested video model improves over AnySplat alone (AnySplat: 20.85 PSNR / 0.695 SSIM / 0.238 LPIPS). Results include Hunyuan + AnySplat 21.17/0.710/0.242, SVD + AnySplat 21.48/0.720/0.218, CogVid + AnySplat 21.32/0.716/0.222, and Wan + AnySplat 21.29/0.718/0.232. The stitched VAEs also outperform SplatFlow (19.10/0.671/0.278), VideoRFSplat (19.05/0.674/0.281), and Prometheus3D (19.56/0.683/0.277), despite those methods using camera poses and large-scale training data.

• Stitching preserves 3D reconstruction ability. The paper reports that both pointmap quality and camera pose accuracy barely change when video encoder latents are used as input, confirming that pretrained 3D capabilities are repurposed rather than rebuilt.

• MSE is a better stitching-layer criterion than CKA. Ablating stitching index for Wan + VGGT, the authors find lower stitching residual correlates with better pointmaps, with best performance at layer 2. CKA captures the overall degradation with increasing layer index but is less precise, failing to identify layer 2 as best. This is consistent with Theorem 1 in Insulla et al. (2025), which bounds stitching risk by the MSE at the stitching layer scaled by the Lipschitz constant of the downstream part.

• Human evaluation favors VIST3A. In a study with 28 participants evaluating 14 randomly selected samples from T3Bench, SceneBench, and DPG-Bench, VIST3A achieved the lowest average rank on both criteria: 1.54 for Text Alignment and 1.45 for Visual Quality, versus 3.03/2.99 for Director3D, 3.38/3.88 for SplatFlow, 3.25/3.71 for Prometheus3D, and 2.74/2.92 for VideoRFSplat. Participants ranked VIST3A top in more than 68% of cases for text alignment and more than 87% for visual quality.

• Emergent properties. VIST3A inherits prompt-based camera control from the video backbone (for example, responding to instructions like "aerial droneshot"), can generate coherent large-scale scenes by extending the number of frames generated by the LDM even without specific training on very long image sequences, and its finetuning does not degrade — and even slightly improves — video generation quality as measured by VBench.

• Text-to-pointmap is evaluated qualitatively. The paper states that no established benchmarks or baselines exist for text-to-pointmap models, so those results are qualitative only.

Methodology in Plain English

The system has two halves. The first half is a standard text-to-video latent diffusion model that turns a text prompt into a set of multi-view latent codes. The second half must turn those latents into 3D geometry.

Rather than train that second half, the authors borrow it. They take an off-the-shelf feedforward 3D reconstruction model — one that already maps images to pointmaps, Gaussian splats, camera poses, or depth — and search for the layer inside it whose internal activations most closely resemble the video VAE's latent space. "Closely resemble" is made concrete by fitting a simple linear map from latents to that layer's activations by least squares, and choosing the layer with the smallest residual error. They then cut the 3D model at that point: the early layers are thrown away, and the rest becomes the decoder. Only a small dataset and no labels are needed for this step, and because latents can be obtained by pushing images through the encoder, the denoising loop is skipped during stitching.

Because the match is approximate, they finetune the stitching layer and the retained part of the 3D model to reproduce the original 3D model's outputs as pseudo-targets. This finetuning is self-supervised (no labels), the stitching layer is restricted to a 3D convolution, and LoRA is used on the retained part to avoid large deviations from the pretrained weights.

The remaining problem is that at inference the latents come from denoising rather than from the encoder, so they may be off-distribution for the decoder. The authors therefore align the generator using direct reward finetuning, in which samples are generated by unfolding the full denoising path and rewards are backpropagated through it. The reward has three parts: a CLIP-based and HPSv2 score on multi-view images decoded by the original video decoder; the same metrics applied to 2D views rendered from the generated 3D representation; and an ℓ1 plus LPIPS comparison, for each shared viewpoint, between decoded multi-view images and renders of the 3D representation using the camera poses predicted by the feedforward model. The total loss combines the conventional generative loss (which needs paired multi-view images and prompts) with the negative reward (which needs only the prompt). To keep optimization stable, gradients are detached from the inputs to the generative model but retained during the update to the next denoising state, generalizing DRTune; the optimizer is made more efficient by sampling fewer timesteps than at inference and randomizing which denoising steps receive backpropagated gradients.

Training data: stitched VAEs are finetuned on DL3DV-10K and ScanNet without 3D labels; the generative loss for alignment uses DL3DV-10K with prompts from the HPSv2 training set.

Why This Matters

Impact on research. The paper reframes the decoder problem in text-to-3D latent diffusion models: instead of training a custom decoder from multi-view 2D latents to 3D outputs, it argues one should reuse the 3D reasoning already encoded in pretrained feedforward 3D foundation models. As 3D foundation models keep improving, this means a text-to-3D system can inherit those improvements rather than falling behind, and it removes the need for large labeled 3D datasets and massive training runs for the decoder. The work also demonstrates model stitching — previously studied as an analysis tool and for resource-constrained reassembly or multi-scale networks — as a practical construction method for generative 3D systems.

Real-world applications (as named by the paper): AR/VR, gaming, robotics, and simulation. Text-to-3D generation is described as a new research frontier with applications in these four areas.

Industry relevance. The approach reuses pretrained weights rather than rebuilding capability, which matters for compute and data budgets. The authors note that replicating recent feedforward 3D capabilities as part of VAE training would be difficult and costly, and that the LDM-based design enables efficient 3D scene generation with a compact, well-amortized decoder. The framework's demonstrated generality across multiple video backbones (Wan 2.1 T2V large, CogVideoX, SVD, HunyuanVideo) and 3D models (MVDUSt3R, VGGT, AnySplat) suggests a route to upgrading products as new base models are released.

Future Directions

• Extending stitching to more base models and combinations. The authors show results across several video generators and three 3D foundation models, but the framework is presented as general; whether arbitrary pairings stitch well, and how to predict that in advance, remains open.

• Better criteria for choosing the stitching layer. MSE outperformed CKA in their ablation, but CKA still tracked overall degradation; the paper leaves open what other transferability measures might identify the best layer more reliably or more cheaply.

• Stronger alignment that accounts for the full denoising trajectory. The paper motivates direct reward finetuning by arguing that prior rendering-loss approaches rely on single-step sampling and thus align weakly at inference; further improvements to reward design or optimization stability are a natural continuation.

• Broader use of stitching to combine foundational networks. The authors state they see great potential for model stitching as a general tool to combine two or more foundational neural networks, including latent generative models, into powerful end-to-end solutions.

• Benchmarking text-to-pointmap generation. Text-to-pointmap results are qualitative only because no established benchmarks or baselines exist, leaving an obvious gap for future evaluation work.

Target Audience

Researchers and practitioners in generative 3D modeling, diffusion/video generative models, and 3D reconstruction who want to understand how pretrained 3D foundation models can be repurposed as decoders for latent text-to-3D systems. It is also relevant to engineers building text-to-3D assets for AR/VR, gaming, robotics, and simulation who care about compute-efficient reuse of pretrained models. The paper is not beginner-friendly: it assumes working knowledge of latent diffusion, VAEs, model stitching, and 3D representations such as Gaussian splats and pointmaps.

Authors’ abstract

The rapid progress of large, pretrained models for both visual content generation and 3D reconstruction opens up new possibilities for text-to-3D generation. Intuitively, one could obtain a formidable 3D scene generator if one were able to combine the power of a modern latent text-to-video model as "generator" with the geometric abilities of a recent (feedforward) 3D reconstruction system as "decoder". We introduce VIST3A, a general framework that does just that, addressing two main challenges. First, the two components must be joined in a way that preserves the rich knowledge encoded in their weights. We revisit model stitching, i.e., we identify the layer in the 3D decoder that best matches the latent representation produced by the text-to-video generator and stitch the two parts together. That operation requires only a small dataset and no labels. Second, the text-to-video generator must be aligned with the stitched 3D decoder, to ensure that the generated latents are decodable into consistent, perceptually convincing 3D scene geometry. To that end, we adapt direct reward finetuning, a popular technique for human preference alignment. We evaluate the proposed VIST3A approach with different video generators and 3D reconstruction models. All tested pairings markedly improve over prior text-to-3D models that output Gaussian splats. Moreover, by choosing a suitable 3D base model, VIST3A also enables high-quality text-to-pointmap generation.