InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators

AI summary

Overview

Research area: Computer vision / generative modeling — specifically cross-model latent representation alignment and denoising-trajectory transfer among text-to-image diffusion and flow-matching generators.

Technical level: Advanced. The paper assumes familiarity with flow matching, latent diffusion, variational autoencoders, denoising schedules, on-policy distillation, and reward-based preference optimization.

Scope: The paper proposes a lightweight, model-specific latent interface that lets frozen, architecturally heterogeneous image generators exchange intermediate denoising states and dynamics without pair-specific mappings, and demonstrates the interface on acceleration, multi-preference composition, cross-architecture distillation, and cross-model composition.

What This Paper Is About

Modern image generators (diffusion and flow models) are built independently, with different latent parameterizations, architectures, and training recipes, and they are almost always studied and deployed in isolation. The authors ask whether independently trained generators actually share enough latent structure and denoising behavior to be connected. Their goal is an interface that translates one model's intermediate denoising state into another model's latent space so that a denoising trajectory can hand off from one frozen generator to another.

Key Contributions

1. Empirical evidence of partial cross-model compatibility. The authors show that independently trained generators (Qwen-Image, SD3, FLUX.1) exhibit closely related latent spatial organization, and that directly handing off an intermediate state between unaligned models — with no learned alignment — preserves the main subject and coarse spatial layout even though it introduces local artifacts.
2. InstantFusion, a shared latent interface. Model-specific Latent AutoEncoders (LAEs) project intermediate states from different generators into a shared latent space and decode them into a target generator's native latent space, constrained by objectives that align states from the same image at matched noise levels and synchronize their local denoising velocities.
3. Pair-free, extensible design. Once trained, denoising transitions across frozen generators without a dedicated mapping for every model pair; new generators can be incorporated through their own LAE.
4. Demonstration across four settings. The interface is applied to generation acceleration, multi-preference composition, cross-architecture on-policy distillation, and cross-model composition.

Main Findings

• Direct handoff preserves coarse content but not fidelity: Halting SD3 mid-denoising and handing the state to SD3.5 or FLUX.1 yields a seamless image for same-family handoff, while cross-family handoff introduces local artifacts yet clearly preserves the central cup and its coarse layout. Native latent spaces cannot be directly interchanged.
• Reconstruction is high fidelity but degrades over repeated round trips: For the SD3–Qwen shared space, one trip gives PSNR 31.375, SSIM 0.9417, LPIPS 0.04028; 5 trips give 22.977 / 0.8431 / 0.12630; 10 trips give 20.043 / 0.7670 / 0.18892. FLUX.1–Qwen: 32.579 / 0.9496 / 0.03663 at 1 trip, degrading to 19.106 / 0.7621 / 0.21014 at 10. SD3–FLUX.1–Qwen: 32.397 / 0.9511 / 0.03257 at 1 trip, 20.119 / 0.7808 / 0.17591 at 10.
• Cross-model composition improves weaker generators: Qwen-Image guidance raises SD3's GenEval-v2 from 22.14 to 35.06 (+12.92) and FLUX.1's from 22.86 to 36.54 (+13.68). CVTG-2K improves from 0.3682 to 0.6488 for SD3 (+0.2806) and from 0.5180 to 0.6983 for FLUX.1 (+0.1803). OneIG-Bench averages improve by 0.0619 and 0.0591 respectively.
• Acceleration without quality collapse: Qwen-Image alone takes 18.84 s with GenEval-v2 41.73. Pairing it with Z-Image-Turbo or FLUX.2-Klein reduces latency to 4.373 s and 2.545 s with GenEval-v2 of 33.34 and 27.19. T-Stitch takes 20.76 s with GenEval-v2 7.762. The abstract reports a 1.8–3.3× speedup.
• Orthogonal to training-free caching: Applying DiCache to an InstantFusion configuration reduces latency from 10.64 s to 5.700 s, with GenEval-v2 changing only from 32.70 to 32.46.
• Cross-architecture on-policy distillation works: Transferring priors from Qwen-Image to SD3 raises SD3's HPSv2 average from 26.27 to 26.46 (Qwen-Image teacher: 27.92).
• Multi-preference composition rivals joint training: Combining GenEval-v2- and HPSv2-optimized experts yields GenEval-v2 30.97 and HPSv2 average 30.68, outperforming direct velocity fusion and closely matching jointly trained multi-reward GRPO.
• All three training objectives are necessary: Without the dynamics loss, training diverges during cross-architecture on-policy distillation even with aligned latent spaces; without the reconstruction loss, cross-model translation still works but reconstruction quality collapses (SD3 average PSNR 10.4743, SSIM 0.3627, LPIPS 0.9491; Qwen-Image average 8.6137, 0.4794, 0.9328).
• Time-conditioned residual blocks matter: Replacing them with standard convolutional layers reduces GenEval-v2 by 7.20 for Qwen-Image→SD3 and by 19.17 for Qwen-Image→FLUX.1.
• Rollout order is largely stable: Across all six permutation schedules among SD3, FLUX.1, and Qwen-Image, GenEval-v2 ranges from 19.99 to 33.13 and HPSv2 average from 22.95 to 25.86, with the same relative ordering of model quality.
• No extra denoiser evaluations: Each handoff only translates the latent state. Latency at 1024×1024 ranges from 3.61 s (SD3) and 10.70 s (FLUX.1) to 20.82 s (Qwen); hybrids range from 7.16 s to 15.7 s (two-model) and 11.53 s (three-model), with theoretical NFEs of 50 for single models, 25+25 for two-model hybrids, and 17+17+16 for the three-model hybrid.

Methodology in Plain English

The authors start from two observations: different generators organize their intermediate latents in similar spatial ways, and a raw state handoff between unaligned models keeps the broad layout of the image even though it looks broken. They turn that partial compatibility into a trained bridge.

Each frozen generator gets its own small Latent AutoEncoder (LAE) with an encoder and a decoder, made of time-conditioned residual blocks. The encoder maps a noisy intermediate state into a shared representation, and a decoder maps that shared representation back into a specific generator's latent space. Because all generators share the same frozen VAE and time parameterization, the authors can build paired training states from the same image, the same noise sample, and the same time step.

The LAEs are trained with four terms: a reconstruction loss so the round trip preserves information (otherwise the alignment objective could be satisfied trivially by discarding everything); a pairwise alignment loss that pulls encoded states from different models together; a dynamics loss that aligns each model's one-step displacement in the shared space so that translated states follow the receiver's denoising trajectory; and a small norm-consistency term to keep representation magnitudes comparable. The generator backbones stay frozen; only the LAEs are optimized.

At inference, a sender state is encoded by the sender's encoder and decoded by the receiver's decoder. Rather than fixing the handoff step in advance (which varies by model pair and fails if the sender has not yet formed stable structure), the method selects the switch step online by tracking the temporal consistency of the sender's predicted clean latents across consecutive steps, and switches at the first step meeting a similarity threshold. Training uses 50,000 image-text pairs from VisPrompt, with LAEs of 32 shared channels, 64 hidden channels, a 128-dimensional sigma embedding, and 1.532M trainable parameters.

Why This Matters

Impact on research: The paper reframes heterogeneous generators as parts of one composable ecosystem rather than isolated systems. It offers empirical evidence that independently trained latent spaces share structure, and it shows that aligning denoising dynamics — not just static states — is a distinct and necessary problem. Prior cross-model alignment work focused on pairwise mappings of isolated states or assumed models lived in the same latent space; this work targets multi-model trajectories without pair-specific mappings.

Real-world applications:

• Faster high-quality image generation by running most denoising steps in a cheap model and finishing in a stronger one.
• Combining independently trained preference experts (for example, prompt-adherence vs. aesthetic-quality models) without jointly retraining a single model.
• Distilling a strong generator's behavior into an architecturally different student to improve the student, rather than retraining it from scratch.
• Building generation pipelines that chain several specialist models, since the interface is stable across rollout orders.

Industry relevance: The interface is orthogonal to existing training-free acceleration such as DiCache and only translates latents rather than adding denoiser evaluations, so it can be layered onto existing inference stacks. The authors note that hybrid inference requires hosting multiple frozen backbones simultaneously, which the paper flags as memory and storage overhead — a practical consideration for deployment. Code is available at https://github.com/InternLM/InstantFusion.

Future Directions

• Extending the interface to heterogeneous autoencoders; the current formulation relies on a shared autoencoder and compatible time parameterization across generators.
• Optimizing dynamic model loading and offloading to reduce the memory and storage cost of hosting multiple frozen backbones during hybrid inference.
• Developing the learned shared space into a reusable pretrained interface so new generators can be incorporated with minimal adaptation and share a common starting point for composition, knowledge transfer, and downstream optimization.
• Understanding and mitigating the fidelity degradation observed over repeated latent round trips, which the per-model appendix analysis is reported to track.

Target Audience

Researchers and engineers working on diffusion and flow-matching image generation, latent-space representation alignment, model composition, and inference acceleration. It is also relevant to practitioners interested in distillation across incompatible architectures and in composing independently trained reward-specialized models without joint multi-reward training. Readers without background in flow matching, latent diffusion, or denoising schedules will find the methods section demanding.

Authors’ abstract

Diffusion/flow models have emerged as the de facto paradigms for image generation, giving rise to a diverse family of modern generators with increasingly heterogeneous architectures and representations. Despite their remarkable generation capabilities, the cross-model alignment of latent representations and the consistency of their underlying denoising dynamics remain underexplored. In this work, we uncover an intriguing cross-model compatibility: independently trained generators exhibit similar latent spatial organization, while directly handing off intermediate states between unaligned models can still preserve coarse visual content. Motivated by these observations, we propose InstantFusion, a lightweight shared latent interface that connects the denoising trajectories of heterogeneous image generators. Specifically, to enable flexible cross-model state translation, model-specific Latent AutoEncoders (LAEs) are introduced to project intermediate states from different generators into a shared latent space and decode them into the latent space of a target generator. This shared space is constrained by two complementary objectives: aligning intermediate states derived from the same image at matched noise levels and synchronizing their corresponding local denoising velocities. Once trained, InstantFusion allows denoising to seamlessly transition across frozen generators without requiring a dedicated mapping for every model pair. The versatility of InstantFusion is demonstrated across acceleration, multi-preference composition, cross-architecture on-policy distillation, and cross-model composition, achieving competitive performance across these diverse settings.