SVG-T2I: Scaling Up Text-to-Image Latent Diffusion Model Without Variational Autoencoder

AI summary

Overview

Research area: Computer vision / generative modeling — specifically large-scale text-to-image (T2I) diffusion trained directly in the feature space of a visual foundation model (VFM) rather than in a variational autoencoder (VAE) latent space.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, flow matching, diffusion transformers (DiT), VAEs, and self-supervised visual encoders such as DINOv2 and DINOv3.

Scope (one sentence): The paper provides the first large-scale study showing that a text-to-image diffusion transformer can be trained from scratch directly on frozen DINOv3 features, reaching competitive scores on GenEval and DPG-Bench while removing the VAE from the pipeline, and it open-sources the full pipeline and weights.

What This Paper Is About

Nearly all state-of-the-art image generators compress images into a VAE latent space before running diffusion. VAE latents are good at pixel reconstruction but carry weak semantic structure, so separate encoders (e.g., SigLIP for understanding) are still needed for other tasks. Prior work — SVG (Shi et al., 2025) and RAE (Zheng et al., 2025) — showed that diffusion can instead run directly inside a high-dimensional VFM feature space, but only demonstrated this on relatively small-scale, low-resolution ImageNet class-conditioned generation. This paper asks whether that same VFM feature space is genuinely usable for large-scale, high-resolution, text-conditioned image synthesis.

Key Contributions

1. First large-scale validation of text-to-image latent diffusion inside a VFM feature space. The authors train SVG-T2I end-to-end on text-to-image data with no VAE, using DINOv3 features as the latent manifold.

2. Two released autoencoder configurations. autoencoder-P (Pure) uses frozen DINOv3 features directly; autoencoder-R (Residual) keeps an optional ViT-S-RoPE residual branch from SVG to compensate for high-frequency detail and color cast. Both share the same decoder.

3. Full open-source release. The project includes the autoencoder and generation model, their training, inference, and evaluation pipelines, and pre-trained weights of multiple sizes (GitHub: KlingTeam/SVG-T2I; Hugging Face: KlingTeam/SVG-T2I).

4. A diagnostic analysis of VFM features across resolutions. The paper identifies resolution-dependent feature instability in DINOv2/DINOv3 as the central obstacle to scaling this paradigm, and characterizes it against VAE features.

Main Findings

• GenEval overall score of 0.75. Table 5 reports SVG-T2I at Single Obj. 0.94, Two Obj. 0.89, Counting 0.49, Colors 0.89, Position 0.69, Color Attri. 0.62, Overall 0.75. The paper's Section 4.2 prose states the model "attains an overall score of 0.74," which differs from both the abstract and Table 5; the discrepancy is not explained.

• DPG-Bench overall score of 85.78. Table 6 reports SVG-T2I at Global 88.50, Entity 91.00, Attribute 91.86, Relation 92.21, Other 91.86, and Overall 85.78.

• Comparable to VAE-based diffusion models. On GenEval the paper places SVG-T2I alongside SD3-Medium (0.74 overall in Table 5) and well above SDXL (0.55) and DALL-E 2 (0.52). On DPG-Bench it sits near HiDream-I1-Full (85.89) and below Lumina-Image 2.0 (87.20) and FLUX.1-dev's 0.82 GenEval overall.

• High-resolution reconstruction from DINOv3 is strong; low-resolution is not. Visualizations show reconstructions from low-resolution inputs degrade in fine structures, while high-resolution inputs produce substantially more detailed and faithful results. The authors conclude the DINOv3 encoder alone is relatively sufficient at high resolution, making the residual encoder optional.

• VAE features are far more resolution-stable than VFM features. Cross-resolution cosine similarity is close to 1.0 for VAE features, whereas DINOv3 and DINOv2 features vary more substantially. This is attributed to fixed patch size (e.g., 16×16) causing semantic granularity and effective compression ratio to shift with absolute image scale, since VFM encoders are optimized for discriminative tokens rather than uniform local detail.

• Quality improves steadily across the four training stages, as shown by visual comparisons of stage outputs.

• Documented failure modes. The model struggles with highly detailed human faces (eyes, eyebrows), anatomically accurate fingers, and reliable text rendering. The authors attribute this to insufficient coverage of these fine-grained cases in the training corpus and to the computational cost of modeling high-frequency patterns.

Methodology in Plain English

The encoder. Instead of a VAE, SVG-T2I uses a frozen DINOv3-ViT-S/16+ encoder that maps an H×W×3 image to an (H/16)×(W/16)×384 feature grid. A decoder maps those features back to pixel space; its channels are [512, 256, 256, 128, 128] with 3 output channels and 43M parameters. Pure mode has a 384-dim latent and a 29M-parameter frozen encoder; residual mode adds a ViT-S-RoPE branch for a 392-dim latent and a 51M-parameter encoder. Both are optimized with Adam at lr 1e-4 and betas (0.5, 0.9).

The generator. The diffusion transformer is the 2.6B-parameter Unified Next-DiT architecture from Lumina-Image-2.0, a single-stream design that treats text and image tokens as one joint sequence. It has patch size 1, hidden dimension 2304, 24 heads, 8 KV heads, 26 layers, and M-RoPE positional embedding. Text conditioning comes from Gemma2-2B. Training uses flow matching with α_t = 1−t, σ_t = t, v-prediction, an Euler sampler, AdamW at lr 2e-4 and betas (0.9, 0.95).

Two-stage training. Stage one trains autoencoder-P and autoencoder-R separately: 40 epochs on ImageNet (Data A, 1.2M samples) at 256×256, then multi-resolution fine-tuning on a 3M-sample dataset (Data B) at an anchor of 512×512 for 10M seen images and 1024×1024 for 6M more. Stage two trains the DiT on top of autoencoder-P.

Progressive DiT schedule. Four stages: 256×256 on 60M samples (Data C, 91K steps, batch 1536, 140M seen samples), then 512×512 on the same 60M (90K steps, batch 768, 70M seen), then 1024×1024 on 15M samples (Data D, 44K steps, batch 768, 34M seen), and finally high-quality tuning at 1024×1024 on 1M high-aesthetic samples (Data E, 40K steps, batch 768, 30M seen). Captions are bilingual with a fixed 0.2 Chinese / 0.8 English sampling ratio and short/middle/long sample ratios of (0.10, 0.35, 0.55) for Data C and D and (0.00, 0.00, 1.00) for Data E. Maximum text token length is 256 for the first three stages and 512 during high-quality tuning.

Evaluation. All evaluation images are generated at 1024×1024 and scored on GenEval and DPG-Bench following official protocols.

Why This Matters

Impact on research. The result is a proof of concept that the latent space used for image generation need not be a VAE. If generation, reconstruction, and semantic understanding can share one encoder, the field can stop maintaining separate task-specific encoders (the paper names SigLIP for understanding, VAE for generation, and VGGT for geometry). The paper also isolates a concrete obstacle — resolution-dependent instability of VFM features — that future work must solve before that unification is practical.

Real-world applications (derived from what the paper generates and demonstrates):

• Commercial product and food/beverage imagery. Figure 1 and Appendix B show prompts for still-life advertising shots, wine-glass compositions, and dessert photography.
• Fashion and portrait editorial. One shown example is a full specification of a gown, headpiece, hair, and lighting setup for portrait editorial use.
• Landscape and nature photography replacement. The prompts include alpine lake scenes and macro botanical photography.
• Illustration and decorative art. Appendix B includes watercolor botanical illustration prompts intended for uses such as textiles, wallpapers, or stationery.
• Text-to-image research tooling. The open training, inference, evaluation pipelines and weights let other groups reproduce and extend representation-space generation without building it from scratch.

Industry relevance. The work comes from Tsinghua University's Department of Automation and the Kling Team at Kuaishou Technology, and it targets exactly the efficiency argument that matters commercially: prior work on SVG and RAE reported better generation quality with higher efficiency during both training and inference than VAE-based LDMs. Removing the VAE and potentially consolidating encoders across tasks is directly relevant to the cost structure of large-scale image and video generation systems.

Future Directions

1. Scale invariance for VFM encoders. The conclusion calls this out explicitly: existing VFM encoders such as DINOv2 and DINOv3 produce poorly internally consistent representations when the same image is encoded at different input resolutions, and the paper argues future research must focus on scale-invariance.

2. Training mechanisms that stabilize cross-resolution feature geometry. Section 4.3 suggests the training pipeline may need built-in mechanisms that encourage consistent feature geometry and preserve fine-grained detail fidelity across scales.

3. Fine-grained failure modes. Faces, fingers, and text rendering remain unreliable; the authors state that addressing these will require more specialized datasets and additional training compute.

4. A truly unified representation across visual tasks. The paper frames its result as a step toward a native unified general vision model, in which one architecture jointly supports low-level perceptual fidelity, reconstruction, semantic understanding, and generation without sacrificing performance on any of them — the first of the two open challenges it poses but does not resolve.

Target Audience

Researchers and engineers working on diffusion models, latent-space design, and visual foundation models who want to understand whether VAE-free generation is viable at scale. It is most useful to people who already understand flow matching and transformer-based diffusion, since the paper moves quickly over those preliminaries. It is also relevant to practitioners who intend to build on the released code and weights, and to anyone investigating the unification of visual understanding and generation encoders. Readers looking for a beginner-level tutorial, detailed ablation studies, or a discussion of the GenEval 0.74 vs. 0.75 discrepancy will not find those here.

Authors’ abstract

Visual generation grounded in Visual Foundation Model (VFM) representations offers a highly promising unified pathway for integrating visual understanding, perception, and generation. Despite this potential, training large-scale text-to-image diffusion models entirely within the VFM representation space remains largely unexplored. To bridge this gap, we scale the SVG (Self-supervised representations for Visual Generation) framework, proposing SVG-T2I to support high-quality text-to-image synthesis directly in the VFM feature domain. By leveraging a standard text-to-image diffusion pipeline, SVG-T2I achieves competitive performance, reaching 0.75 on GenEval and 85.78 on DPG-Bench. This performance validates the intrinsic representational power of VFMs for generative tasks. We fully open-source the project, including the autoencoder and generation model, together with their training, inference, evaluation pipelines, and pre-trained weights, to facilitate further research in representation-driven visual generation.