REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion

AI summary

Overview

Research area: Computer vision and generative modeling — specifically latent diffusion models (LDMs) for class-conditional and unconditional image synthesis, and how pretrained Vision Foundation Model (VFM) representations can be injected into the diffusion process.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching objectives, Transformer backbones, VAEs, and self-supervised vision features (DINOv2, DINOv3, CLIP-style encoders).

Scope: The paper proposes REGLUE, a single-backbone latent diffusion framework that jointly models VAE image latents, compactly compressed patch-level VFM features, and a global image-level [CLS] token, evaluated on ImageNet 256×256 with SiT-B/2 and SiT-XL/2 backbones.

What This Paper Is About

Latent diffusion models are trained with a reconstruction-style denoising objective, which supervises semantics only indirectly, so high-level structure (objects, layout, relations) emerges slowly and image quality suffers. Recent methods try to fix this either by aligning the denoiser's internal features to a frozen Vision Foundation Model (REPA), by modeling a single image-level [CLS] token inside the diffusion process (REG), or by jointly modeling linearly PCA-projected patch features (ReDi). REGLUE argues that all of these under-use the rich, nonlinear, multi-layer, spatial semantics available in a VFM, and proposes to jointly model a nonlinearly compressed patch-level semantic representation alongside the VAE latent and a global token.

Key Contributions

1. REGLUE (Representation Entanglement with Global–Local Unified Encoding): a unified diffusion framework that jointly models image-level (global) and patch-level (local) VFM semantics together with VAE image latents in a single SiT backbone, boosting generative performance.

2. A lightweight semantic compressor: a shallow convolutional autoencoder that aggregates multi-layer VFM patch features and maps them to a compact, semantics-preserving, spatially structured representation — enabling patch-level joint modeling without letting the semantic modality dominate the diffusion model's capacity.

3. A systematic decomposition of VFM signal design: the paper shows that (a) patch-level local semantics, (b) image-level global [CLS] semantics, and (c) REPA-style representation alignment act synergistically, delivering gains in image quality and convergence while leaving the diffusion model's parameter count and inference-time compute essentially unchanged.

4. Efficiency result on ImageNet 256×256: SiT-XL/2 + REGLUE reaches the 1M-step performance of ReDi and REG using less than 30% and 80% of their iterations, respectively.

Main Findings

• Local semantics beat global semantics when modeled directly. At 400K training steps with SiT-B/2 and no external alignment, modeling only the global [CLS] token (setting c) gives 25.7 FID, whereas modeling patch-level features (setting d, ReDi, linear PCA) gives 21.4 FID. Both beat the vanilla SiT-B/2 baseline at 33.0 FID (setting a), but the gap shows spatial semantics matter most.

• Non-linear compression is the key to unlocking local guidance. Replacing ReDi's linear PCA with the paper's non-linear semantic compressor reaches 14.3 FID (setting k) with no alignment loss — a 7.1 FID absolute reduction over ReDi (setting d, 21.4) — and this already surpasses the REG baseline (setting h, 15.2) even though REG also uses external alignment.

• Multi-layer aggregation adds further gains. Aggregating multi-layer VFM patch features before compression (setting n) reduces FID to 13.3 without any global token or external alignment.

• Alignment on local features dominates; alignment on global features alone is unstable. Original REPA (setting b) improves the SiT-B/2 baseline from 33.0 to 24.4 FID. Adding local-only alignment to a model that jointly models the global [CLS] token (setting g) reduces FID from 25.7 to 15.5, and adding the global component to the alignment (setting h, REG) reaches 15.2. Aligning only global information without local alignment (setting f) degrades performance from 25.7 to 33.7 FID.

• External alignment becomes a mild complement once strong spatial semantics are modeled. Adding REPA on top of non-linear patch modeling improves FID from 14.3 (setting k) to 14.1 (setting l).

• Naïvely stacking components backfires. Adding components without the non-linear compressor (setting i) gives 17.4 FID, and using multi-layer PCA (setting j) gives 23.1 FID, both far worse than REGLUE's 12.9 FID (setting o).

• REGLUE's full unified setting reaches 12.9 FID, and a stronger VFM helps further. The final configuration, combining the global [CLS] token, local and global alignment, and multi-layer aggregation before compression, achieves 12.9 FID with DINOv2-B. Swapping in DINOv3-B (setting p) gives the best result at 12.3 FID; DINOv2-B is kept as the default for fair comparison with prior work.

• Faster convergence at SiT-B/2 scale (no CFG). REGLUE reaches 14.5 FID at 300K steps, surpassing REG's 15.2 at 400K with 25% fewer iterations, and reaches 12.9 at 400K. At 400K this is a 60.9% FID reduction vs. vanilla SiT-B/2 (33.0), 47.1% vs. REPA (24.4), and 39.7% vs. ReDi (21.4). Throughput (3.8 img/s, measured on 1×A100 with batch size 64 and 250 sampling steps) and parameter count (132M vs. 130M for SiT-B/2, 4.1 img/s) remain essentially unchanged.

• SiT-XL/2 results without CFG (Table 3). At 200K steps REGLUE achieves 4.6 FID, beating REG (5.0), REPA (11.1) and ReDi (12.5). At 400K it reaches 3.2 vs. REG's 3.4. At 700K it reaches 2.7 FID, matching REG's 1M-step result (2.7) with 30% fewer iterations. At 1M steps REGLUE's 2.5 is the best reported in that comparison vs. REG (2.7), ReDi (5.1) and REPA (6.4).

• Unconditional generation also improves. With SiT-B/2 at 400K steps and no CFG, REGLUE achieves 28.7 FID vs. SiT-B/2 at 59.8, ReDi at 43.6, and REG at 29.7 — reported as 52%, 34.2%, and 3.4% improvements respectively. REGLUE's unconditional 28.7 FID also outperforms the conditional SiT-B/2 baseline (33.0 FID).

• State-of-the-art comparison with CFG (Table 4). At 80 epochs REGLUE lowers FID to 1.59 vs. REG's 1.86, matching 160-epoch REG. At 160 epochs it improves to 1.46 vs. 1.59. Despite training 5× fewer epochs than the 800-epoch REPA, ReDi and REG variants, the 160-epoch REGLUE remains competitive with REPA (FID 1.42) and REG (FID 1.36). The table also lists RAE at 1.13 FID and REPA-E at 1.12 FID at 800 epochs, both marked as not using SD-VAE latents (REPA-E also uses a class-balanced evaluation protocol).

Methodology in Plain English

The authors start from a SiT (Scalable Interpolant Transformer) diffusion backbone, which generates images in the compressed latent space of a frozen VAE. Instead of asking the denoiser to learn semantics implicitly, they feed it three entangled signals at once:

1. The VAE image latent (the thing the model actually needs to generate, denoted z).
2. A compact patch-level semantic map (denoted s) derived from a frozen VFM. Rather than using raw VFM features — which are far wider than the VAE latent and would swamp the model — they first train a small, shallow convolutional autoencoder offline to reconstruct the multi-layer concatenated VFM features, then keep only the encoder and freeze it. This encoder compresses the concatenation of several VFM layers (blocks 9–12 of DINOv2-B, giving a 3072-channel 16×16 map) into a 16-channel 16×16 latent, which is bilinearly resampled to the VAE latent grid (32×32).
3. A single global [CLS] token (denoted cls) taken from the VFM, representing image-level semantics.

All three are corrupted with the same noise schedule, and the SiT backbone is trained with a multimodal velocity objective: three separate velocity predictions (one per modality) with weighting coefficients, so the model learns to denoise the image latent, the semantic map, and the [CLS] token jointly.

For tokenization, the latent and semantic map are patchified with 2×2 patches and projected to a shared model width; instead of concatenating them along the sequence dimension (which would double sequence length and quadruple self-attention cost), the authors merge them channel-wise by summing, keeping a single grid of N tokens, with the [CLS] as one extra token. Separate lightweight linear heads decode the velocity for each modality.

On top of this, they add a REPA-style external alignment loss at a chosen Transformer block (block 4 for SiT-B/2, block 8 for SiT-XL/2): a small MLP projects the backbone's hidden tokens and a cosine similarity loss pulls them toward the clean, frozen VFM targets (global [CLS] plus last-layer patch features). The total objective is the multimodal velocity loss plus a weighted alignment term, with λ_s = 1, λ_cls = 0.03, and λ_rep = 0.5. At sampling time, only the generated VAE latents are decoded — the semantic and global branches are training-time scaffolding, so inference cost is unchanged.

Evaluation follows the standard ADM protocol on ImageNet 256×256 (center-cropped and resized, encoded with SD-VAE-FT-EMA into 4×32×32 latents), reporting FID, sFID, IS, Precision and Recall over 50,000 generated samples using Euler–Maruyama SDE sampling with 250 steps, and CFG scale w = 2.3 over interval [0, 0.9] when guidance is used. Batch size is 256 throughout.

Why This Matters

Impact on research. The paper reframes a design question that has been answered in three different ways by REPA, REG and ReDi: what semantic signal should a diffusion model see, and how should that signal be compressed? Its ablation table isolates each choice — spatial vs. global, linear vs. nonlinear, single-layer vs. multi-layer, internal joint modeling vs. external alignment — and shows that the nonlinear spatial path is where most of the gain lives. It also demonstrates a rare property for a quality-improving method: no added inference-time parameters or throughput cost. This gives follow-up work a clear, cheap target to build on.

Real-world applications (plausible directions, not claims made by the paper):

• Lower-cost training of text-to-image and class-conditional generators, since reaching a given FID in fewer iterations directly reduces GPU-hours for foundation model training runs.
• Controllable and structured image synthesis, where spatially grounded semantic latents could make layout, object placement, and region-level editing more reliable.
• Data augmentation and synthetic dataset generation for downstream perception models, where better sample fidelity and diversity (Precision/Recall) translate into more useful synthetic data.
• Efficiency-constrained deployment, since the method adds no inference-time compute,

Authors’ abstract

Latent diffusion models (LDMs) achieve state-of-the-art image synthesis, yet their reconstruction-style denoising objective provides only indirect semantic supervision: high-level semantics emerge slowly, requiring longer training and limiting sample quality. Recent works inject semantics from Vision Foundation Models (VFMs) either externally via representation alignment or internally by jointly modeling only a narrow slice of VFM features inside the diffusion process, under-utilizing the rich, nonlinear, multi-layer spatial semantics available. We introduce REGLUE (Representation Entanglement with Global-Local Unified Encoding), a unified latent diffusion framework that jointly models (i) VAE image latents, (ii) compact local (patch-level) VFM semantics, and (iii) a global (image-level) [CLS] token within a single SiT backbone. A lightweight convolutional semantic compressor nonlinearly aggregates multi-layer VFM features into a low-dimensional, spatially structured representation, which is entangled with the VAE latents in the diffusion process. An external alignment loss further regularizes internal representations toward frozen VFM targets. On ImageNet 256x256, REGLUE consistently improves FID and accelerates convergence over SiT-B/2 and SiT-XL/2 baselines, as well as over REPA, ReDi, and REG. Extensive experiments show that (a) spatial VFM semantics are crucial, (b) non-linear compression is key to unlocking their full benefit, and (c) global tokens and external alignment act as complementary, lightweight enhancements within our global-local-latent joint modeling framework. The code is available at https://github.com/giorgospets/reglue .