Latent Diffusion Model without Variational Autoencoder

AI summary

Overview

Research area: Computer vision — generative modeling (latent diffusion models) and self-supervised visual representation learning.

Technical level: Advanced.

Scope: The paper proposes SVG, a latent diffusion framework that replaces the conventional VAE latent space with a semantically structured feature space built from frozen DINOv3 self-supervised features plus a lightweight residual branch, and shows this speeds up diffusion training and few-step inference while preserving the representation's usefulness for perception and understanding tasks.

What This Paper Is About

Almost all modern image and video generators follow one recipe: train a Variational Autoencoder (VAE) to compress pixels into a low-dimensional latent space, then train a diffusion model inside that space. The authors argue that the VAE latent space is the hidden bottleneck — its features are semantically entangled (different object classes overlap heavily), which makes the diffusion model's learning problem harder, requires enormous training budgets, and forces slow many-step sampling. Their goal is to build a latent space that is simultaneously semantically discriminative and good enough for pixel-accurate reconstruction, so that a single representation can serve generation, perception, and understanding.

Key Contributions

1. A systematic diagnosis of VAE latent spaces. Through t-SNE visualizations of SD-VAE, VA-VAE, and MAR-VAE against DINOv3, plus a controlled toy example of velocity fields, the authors show that weak semantic separation in the latent space directly causes ambiguous denoising targets, slower convergence, and the need for more sampling steps.

2. SVG: a latent diffusion model without a VAE. The method freezes a DINOv3 encoder and augments it with a lightweight Vision Transformer "Residual Encoder" whose outputs are concatenated channel-wise with the DINO features to recover fine-grained detail. A distribution-alignment step normalizes residual statistics to match the DINO features, preventing the added channels from destroying the semantic structure.

3. A unified feature space spanning generation and perception. Experiments show the same SVG encoder supports high-quality image generation, ImageNet classification, ADE20K semantic segmentation, and NYUv2 depth estimation, matching or slightly exceeding the original DINOv3 on the downstream tasks.

4. Demonstrated efficiency gains. SVG reaches competitive FID at 25 sampling steps after only 80 training epochs, whereas comparable SiT baselines need 250 steps and 1,400 epochs, and SVG degrades far more gracefully under 5- and 10-step sampling.

Main Findings

• VAE latents are semantically entangled; DINOv3 latents are not. t-SNE plots of 100 ImageNet classes show VAE representations from different classes heavily mixed, while DINOv3 features form well-separated clusters. The authors annotate linear-probe Top-1 accuracy alongside each visualization to make the discrimination gap concrete.

• Semantic dispersion makes the denoising field easier to learn. In a two-class toy example, a separated latent space produces mean velocity directions that are consistent within a class and clearly divergent between classes. The authors argue this reduces discretization error during ODE sampling, which is why few-step generation works.

• SVG-XL achieves strong generation with 25 steps. After 80 training epochs, SVG-XL reaches gFID 6.57 without classifier-free guidance and 3.54 with CFG, versus 22.58 (w/o CFG) for a reproduced SiT-XL in SD-VAE space at the same step count. Extending to 500 epochs yields 3.94 / 2.10, and 1,400 epochs yields 3.36 / 1.92.

• Few-step sampling is dramatically better. At 5 steps, SVG-XL scores 12.26 (w/o CFG) and 9.03 (w/ CFG) against 69.38 / 29.48 for SiT-XL in SD-VAE space. At 10 steps, SVG-XL reaches 9.39 / 6.49 while SiT-XL (VA-VAE) sits at 17.41 / 6.79.

• Scaling behavior is preserved. Across 130M, 458M, and 675M parameter backbones, SVG is better than SiT at every size, and the relative improvement does not shrink — SVG-B at 25 steps (21.90 / 11.49) already beats SiT-B at 250 steps (33.00 / 13.40).

• The Residual Encoder helps reconstruction but needs distribution alignment. Ablating DINOv3 alone gives rFID 1.17 and gFID 6.12. Adding the residual branch cuts rFID to 0.78 but worsens gFID to 9.03 because the concatenated feature distribution becomes imbalanced. Adding the alignment step recovers rFID 0.65 and gFID 6.11.

• DINOv3 is the right base representation. In an encoder comparison, SigLIP2 reconstructs poorly (rFID 4.05) despite strong global semantics, MAE reconstructs best among baselines (rFID 1.69, PSNR 25.04) but trails DINO on dense prediction, and DINOv2/DINOv3 reconstruct weakly. SVG (29M + 11M parameters) gets rFID 1.60 with PSNR 21.77.

• Downstream capabilities survive. With the encoder frozen and only a linear or MLP decoder trained, SVG features give ImageNet-1K Top-1 of 81.80, ADE20K mIoU of 46.51, and NYUv2 A.Rel of 0.101 — matching or marginally beating frozen DINOv3 (81.71, 46.37, 0.101).

• The feature space is continuous and editable. Linear and spherical latent interpolations both produce smooth, coherent images, and an SDEdit-style trajectory inversion supports zero-shot class-conditioned editing without task-specific finetuning.

Methodology in Plain English

The approach has two stages.

Stage one builds the latent space. A DINOv3 ViT-S/16+ encoder is frozen and used as-is. Because DINO features were trained for discrimination, not reconstruction, they throw away color and high-frequency texture. A small trainable Vision Transformer — the Residual Encoder — is added in parallel to capture exactly that missing information. Its output is concatenated to the DINO features along the channel dimension. To stop the decoder from ignoring the DINO channels and leaning entirely on the residual channels, the residual output is rescaled each batch to match the mean and standard deviation of the DINO features. The decoder is a standard VAE-style decoder mapping back to pixels.

Stage two trains the diffusion model directly on this space. No VAE is involved at generation time. The authors use the SiT flow-matching setup (v-prediction, Euler ODE solver) with QK-Norm and per-channel normalization for stability, and replace SiT's patch-embedding layer with a linear projection that maps the 384-channel SVG feature into the model dimension.

A notable detail: the SVG feature for a 256×256 image is 16×16×384, versus 16×16×4 for DiT's VAE latent — roughly a 96× increase in latent dimensionality. The authors point out this does not cost extra inference because transformer hidden widths in these models are already larger than 384, so the patch embedding collapses the channels anyway. The benefit of the higher-dimensional, semantically ordered space is that the model converges in far fewer training iterations.

Why This Matters

Impact on research. The paper challenges a near-universal assumption in generative modeling: that perceptual compression (VAE) and semantic representation (self-supervised encoders) must be separate, specialized systems. If one space can serve both, it reframes latent design as a first-class research problem rather than a preprocessing detail, and it suggests that the "reconstruction versus generation" trade-off that plagues VAE scaling may be avoidable. The diagnostic methodology — using t-SNE separation and velocity-field geometry to predict sampling efficiency — is also reusable.

Real-world applications.

• Fast image and video synthesis: 5- to 10-step generation at competitive quality materially reduces inference cost for interactive tools such as image editors, concept design assistants, and real-time creative software.
• Unified perception-plus-generation pipelines: robotics and autonomous systems need to both understand scenes and imagine outcomes; a single shared representation simplifies such architectures.
• Semantic image editing: the demonstrated zero-shot class-conditioned editing, plus smooth latent interpolation, maps directly onto region-based editing, style transfer, and attribute manipulation products.
• Data augmentation and simulation: compact training budgets (80 epochs rather than 1,400) lower the barrier for domain-specific generative models in medicine, remote sensing, or industrial inspection.

Industry relevance. The work is a collaboration between Tsinghua University and Kuaishou's Kling team, which signals direct applicability to commercial video generation. The efficiency claims — both in training GPU-hours and in sampling steps — target the two dominant cost drivers in deployed generative media systems. If SVG-style latents extend to video, the savings compound across frames and temporal dimension.

Future Directions

• Reduce SVG feature dimensionality. The 16×16×384 latent is much larger than a VAE latent; the authors flag compression as an open efficiency question, particularly once resolutions and sequence lengths grow.

• Improve or replace classifier-free guidance. The authors observe that CFG is less effective in the SVG framework than in VAE-based diffusion, and call for better alternatives — an important gap given how central CFG is to state-of-the-art image quality.

• Refine the residual branch. The residual encoder currently trades reconstruction quality against semantic preservation, mediated by a batch-statistics alignment. A more principled fusion could improve both simultaneously.

• Scale beyond ImageNet 256×256. Text-to-image, text-to-video, larger datasets, and higher resolutions are explicitly listed as underexplored. Whether DINO-based latents retain their advantages when the text conditioning signal is the primary driver of the distribution is an open empirical question.

• Establish the limits of task generality. The paper validates classification, segmentation, and depth estimation; whether the same space supports retrieval, detection, or multimodal alignment remains untested.

Target Audience

This paper is for researchers and engineers working on diffusion-based generative models, particularly those interested in latent space design, training efficiency, and sampling acceleration. It will also interest self-supervised representation learning researchers curious about whether discriminative features can be repurposed for synthesis, and practitioners building production image or video generation systems where training cost and inference latency are the binding constraints. Readers should be comfortable with flow matching, transformer architectures, and standard diffusion evaluation metrics (FID, IS) to get the most out of the experimental sections.

Authors’ abstract

Recent progress in diffusion-based visual generation has largely relied on latent diffusion models with variational autoencoders (VAEs). While effective for high-fidelity synthesis, this VAE+diffusion paradigm suffers from limited training efficiency, slow inference, and poor transferability to broader vision tasks. These issues stem from a key limitation of VAE latent spaces: the lack of clear semantic separation and strong discriminative structure. Our analysis confirms that these properties are crucial not only for perception and understanding tasks, but also for the stable and efficient training of latent diffusion models. Motivated by this insight, we introduce SVG, a novel latent diffusion model without variational autoencoders, which leverages self-supervised representations for visual generation. SVG constructs a feature space with clear semantic discriminability by leveraging frozen DINO features, while a lightweight residual branch captures fine-grained details for high-fidelity reconstruction. Diffusion models are trained directly on this semantically structured latent space to facilitate more efficient learning. As a result, SVG enables accelerated diffusion training, supports few-step sampling, and improves generative quality. Experimental results further show that SVG preserves the semantic and discriminative capabilities of the underlying self-supervised representations, providing a principled pathway toward task-general, high-quality visual representations. Code and interpretations are available at https://howlin-wang.github.io/svg/.