On the Diffusibility of High-Dimensional Latents

AI summary

Authors: Chao Feng, Zhiyang Xu, Bowei Chen, Yuanjun Xiong, Xiyao Wang, Jui-Hsien Wang, Richard Zhang, Zhe Lin, Andrew Owens, Yijun Li Affiliations: Cornell University, Adobe, Virginia Tech, University of Washington, University of Maryland arXiv: 2609.28473v1 [cs.CV], 23 Sep 2026 Project page: https://cfeng16.github.io/on_the_diffusibility/

Overview

• Research area: Latent diffusion and flow matching for text-to-image generation, specifically the design of latent spaces built from pretrained visual encoders ("representation autoencoders").
• Technical level: Advanced. The paper combines flow-matching parameterization theory, singular value decomposition of patch embeddings, and large-scale text-to-image training.
• Scope in one sentence: The paper identifies an "effective dimensionality collapse" in reconstruction-tuned high-dimensional visual latents and shows that switching from velocity prediction to clean-data ($\boldsymbol{x}_0$) prediction restores efficient training and improves text-to-image results.

What This Paper Is About

Diffusion models typically generate images in a compressed latent space, and a recent line of work replaces the usual VAE latent with features from pretrained semantic vision encoders such as DINOv2 or SigLIP2. Those encoders were not trained to reconstruct images, so they lose fine details like text, fine textures, and small objects. The obvious fix is to finetune the encoder with a reconstruction objective, but the authors find that this makes standard flow-matching training surprisingly hard to optimize. The paper diagnoses why, and shows that simply changing what the network predicts (clean representation instead of velocity) fixes the problem while keeping the richer high-dimensional representation.

Key Contributions

1. Identification and quantification of effective dimensionality collapse. The authors show that reconstruction-tuned high-dimensional representations concentrate near a much lower-dimensional subspace, measured as the minimum number of principal components needed to explain 90%, 95%, or 99% of feature variance, and show this collapse correlates with slow convergence of standard velocity prediction.

2. A mathematical analysis of why $\boldsymbol{x}_0$-prediction suits this regime. The velocity target in high dimensions decomposes into a manifold-aligned component plus an orthogonal component $\frac{1}{t}(I - QQ^{\top})\boldsymbol{z}_t^{h}$ that carries little information about the clean representation but can dominate when the ambient dimension is much larger than the intrinsic dimension. Predicting the clean representation mathematically bypasses this orthogonal term.

3. Demonstration that $\boldsymbol{x}_0$-prediction improves text-to-image generation. Across two strong-reconstruction tokenizers (finetuned DINOv2-L and an MAE-based RAE), $\boldsymbol{x}_0$-prediction improves GenEval, DPG-Bench, and COCO-30k FID, letting a reconstruction-finetuned model surpass its frozen semantic baseline counterpart.

4. Evidence that reconstruction quality matters for generation. Comparing a frozen DINOv2-L tokenizer against the reconstruction-finetuned version shows the finetuned one produces better structure and color alignment, and that a low-dimensional bottleneck variant trades image quality for text alignment.

Main Findings

• Reconstruction finetuning collapses effective dimensionality. For DINOv2-L at 1024 dimensions, $R_{90}$ drops from 672 (frozen) to 129 (finetuned), and $R_{95}$ drops from 811 to 302, while $R_{99}$ moves from 965 to 726. MAE-based encoders show the same pattern even at large scale: MAE-B-RAE trained on IN-1k has $R_{90}$ of 103 and $R_{95}$ of 252 at 768 dimensions, and MAE-B pretrained on Instagram-3B has $R_{90}$ of 197 and $R_{95}$ of 348. Encoders not tuned for reconstruction stay high-dimensional: DINOv2-B-RAE is 507/611/726 and SigLIP2-B-RAE is 460/578/715, both at 768 dimensions; DINOv1-B is 502/595/701.

• Better reconstruction, slower optimization under velocity prediction. Finetuned DINOv2-L reaches 29.12 PSNR versus 17.34 for the frozen DINOv2-L, yet in a single-image overfitting test, $\boldsymbol{v}$-prediction converges slowly in the finetuned and MAE-RAE spaces while $\boldsymbol{x}_0$-prediction optimizes more efficiently.

• $\boldsymbol{x}_0$-prediction rescues finetuned DINOv2-L generation. With 1024-dimensional latents, finetuned DINOv2-L with $\boldsymbol{v}$-prediction scores 30.96 GenEval, 67.47 DPG-Bench, and 29.97 COCO-30k FID. Switching to $\boldsymbol{x}_0$-prediction raises this to 39.48 GenEval, 71.83 DPG-Bench, and 16.80 FID, exceeding the original DINOv2-L with $\boldsymbol{v}$-prediction (37.63 GenEval, 70.21 DPG-Bench, 18.34 FID).

• The same effect appears for MAE-RAE. With 768-dimensional latents, MAE-B-RAE with $\boldsymbol{v}$-prediction scores 36.17 GenEval, 67.54 DPG-Bench, and 22.20 COCO-30k FID, while $\boldsymbol{x}_0$-prediction scores 40.89 GenEval, 73.90 DPG-Bench, and 17.24 FID.

• Compressing to low dimensions helps one metric but hurts another. A variant that down-projects the 1024-dimensional finetuned DINOv2-L features to 32 dimensions and uses $\boldsymbol{v}$-prediction achieves 40.99 GenEval and 73.04 DPG-Bench (the strongest text-alignment numbers in that comparison) but a worse FID of 17.64, versus 16.80 for the full 1024-dimensional $\boldsymbol{x}_0$-prediction approach. Its reconstruction PSNR is 25.83.

• Reconstruction improves image quality over the frozen encoder. Comparing the frozen DINOv2-L tokenizer (with $\boldsymbol{v}$-prediction) to the finetuned tokenizer (with $\boldsymbol{x}_0$-prediction) on the same three benchmarks, the finetuned version wins on all three, and qualitative examples show the frozen version failing on color (a purple backpack) and object integrity (a bicycle).

• The method scales. Trained on a 90M-image internal dataset with a 1B DiT and 1024-dimensional latents at 256 resolution, the model reaches 43.84 GenEval and 76.15 DPG-Bench. Adding supervised finetuning on 60k examples raises this to 78.62 and 80.13; a 512-resolution variant reaches 79.69 and 81.15. The paper lists Scale-RAE (2.4B DiT, 1152-dimensional latents, 224 resolution) as a reference at 77.43 and 78.47.

• Qualitative difference between predictors. For a prompt such as "a photo of a pink car," the authors report that $\boldsymbol{v}$-prediction produces amorphous, blob-like structures while $\boldsymbol{x}_0$-prediction maintains structural integrity and geometrically coherent objects.

Methodology in Plain English

The setup is a three-part pipeline. First, a pretrained visual encoder (DINOv2-L or an MAE) is paired with a decoder and trained so that features can be decoded back into pixels. In one configuration the encoder stays frozen; in the other, following AlignTok, it is unfrozen and finetuned with a combination of a semantic preservation loss (keeping its features close to the original encoder's), an L1 pixel loss, a perceptual loss, and an adversarial loss. Second, the resulting high-dimensional features (1024 dimensions for DINOv2-L, 768 for MAE) become the latents for a diffusion transformer. Third, text conditioning is produced by a frozen vision-language model (Qwen3-VL-2B-Instruct) that reads learnable query tokens, feeding a randomly initialized DiT (Lumina-Next, 1B parameters, 24 layers, 24 attention heads per layer, hidden dimension 1536, 64 query tokens).

To diagnose the training difficulty, the authors sample 200k patches from ImageNet, L2-normalize the per-patch features, subtract the mean, and run an SVD on the centered patch matrix. The fraction of total variance explained by each component gives a cumulative curve, and they define effective dimensionality $R_{\tau}$ as the smallest number of components covering at least $\tau$% of the variance, for $\tau \in {90, 95, 99}$.

The theoretical argument treats the high-dimensional feature $\boldsymbol{x}_0$ as living exactly on a low-dimensional subspace: $\boldsymbol{x}_0 = Q\boldsymbol{c}_0$ for an orthonormal $Q$. Under the linear interpolation used in flow matching, the noisy latent splits into an in-subspace part plus a perpendicular part. The optimal velocity predictor then has to fit a term $\frac{1}{t}(I - QQ^{\top})\boldsymbol{z}t^{h}$ that grows as $t$ shrinks; since this term is uninformative about the signal, capacity is wasted on it. The optimal clean-data predictor, by contrast, simplifies to $Q\boldsymbol{x}{\theta}^{l}(\boldsymbol{z}_t^{l}, t)$, which depends only on the manifold component.

Training therefore uses a loss of the form $\mathbb{E}|(\boldsymbol{h}{i,0} - \boldsymbol{x}{\theta}(\boldsymbol{h}_{i,t}, t, \boldsymbol{y}_i))/t|^2$, with $t$ clamped to a minimum of 0.05 to avoid numerical instability, following JiT. Other settings: AdamW with learning rate $10^{-4}$, global batch size 1024, uniform time schedule with shift using $n = 4096$, 90k training steps, 256×256 resolution unless noted, run on either 64 NVIDIA A100 or 32 H200 GPUs. Training data is a 34M subset of the public BLIP-3o pretraining dataset, whose full size is 39.3M and which combines web data such as CC12M, SA-1B, and JourneyDB. Evaluation uses a 100-step Euler sampler with COCO-30k FID, GenEval, and DPG-Bench.

Why This Matters

Impact on research. The paper reframes a practical training headache as a property of latent geometry, giving a measurable diagnostic (effective dimensionality) and a one-line fix ($\boldsymbol{x}_0$-prediction) that requires no architecture change. It also challenges the prevailing habit of compressing reconstruction-tuned latents down to low dimensions, showing that the full high-dimensional space can be diffused if the target is chosen correctly, and it connects the pixel-space argument of JiT to representation spaces.

Real-world applications. The paper does not test or claim any of these directly, but they are the downstream contexts where this capability matters:

• Text-to-image content creation and graphic design tools that need legible text rendering and fine textures.
• E-commerce and product imagery, where small objects, logos, and correct color rendering are critical.
• Creative and media workflows that depend on faithful reproduction of fine-grained detail rather than only semantic plausibility.
• Any system that wants one visual encoder serving both understanding and generation, which the authors name as a future direction.

Industry relevance. Training large text-to-image models is expensive, and the paper's central claim is that the right prediction target removes an optimization bottleneck without extra modules, adapters, or dimensionality reduction. The scaling table (1B DiT, 90M images) and the comparison against a 2.4B DiT baseline speak directly to cost-quality tradeoffs that production teams care about.

Future Directions

• Unified understanding-and-generation encoders. The authors state they will explore building visual encoders that serve both generative and unified (understanding plus generation) models, which this paper does not attempt.
• Scaling further. The 90M-image run is described as showing "potential scalability"; how the approach behaves with larger datasets, larger DiTs, and higher resolutions beyond the 512-resolution variant is left open.
• Retaining reconstruction detail without losing semantics. The low-dimensional adapter variant gains on GenEval and DPG-Bench but loses on FID, and the finetuned encoder relies on a semantic preservation loss to avoid collapse; how to get both properties cleanly is unresolved.
• Whether effective dimensionality can be controlled during training. The paper measures $R_{90}$, $R_{95}$, and $R_{99}$ and links them to convergence, but does not report a method for steering the encoder's geometry to a target value.

The paper does not report a dedicated limitations section or an explicit failure-case study beyond the qualitative comparisons.

Target Audience

Researchers and engineers working on latent diffusion, flow matching, or representation autoencoders who need to understand why high-dimensional reconstruction-tuned latents are hard to train and what to change. It is also useful for practitioners choosing a prediction parameterization for a new latent space, and for readers following the line of work on unifying visual representation learning with generative modeling. Readers without background in flow matching, singular value decomposition, or diffusion parameterizations will find the theoretical section demanding.

Authors’ abstract

Representation Autoencoders (RAEs) enable diffusion models to operate in the feature spaces of pretrained visual encoders. However, many off-the-shelf encoders are not optimized for faithful reconstruction, discarding fine-grained visual details. As expected, finetuning these encoders for image reconstruction recovers such details. However, perhaps counterintuitively, this procedure reduces the effective dimensionality of the resulting representation, and the altered geometry has downstream effects on generation. Specifically, we show that using the standard velocity prediction in flow matching in this high-dimensional space requires the model to fit orthogonal noise directions outside the low-dimensional signal manifold, making optimization inefficient. This motivates using the clean data parameterization ($\boldsymbol{x}_{0}$-prediction) instead, which focuses learning on the underlying signal manifold. Across experiments with multiple strong-reconstruction encoders, we show that $\boldsymbol{x}_{0}$-prediction consistently improves text-to-image generation performance.