More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models

AI summary

Overview

• Research area: Computer vision — unified diffusion models for image generation and monocular depth estimation (generative perception).
• Technical level: Intermediate. The core idea is conceptually simple, but the paper assumes familiarity with latent diffusion models, transformer blocks, cross-attention, and standard depth evaluation metrics.
• Scope: A single paper proposing MERGE, a plug-and-play adapter framework that adds depth estimation to a frozen text-to-image diffusion model using roughly 12–18% additional trainable parameters, without retraining or degrading the original generation ability.

What This Paper Is About

Generative depth estimation methods reuse the visual priors inside pre-trained text-to-image diffusion models, but they fine-tune all of the model's parameters to do so, which destroys the model's original ability to generate images. Meanwhile, existing methods that do support both tasks either run two parallel diffusion models (expensive at inference) or train a unified model from scratch on enormous multi-task datasets (100M samples in the case of OneDiffusion). This paper asks whether a text-to-image model can be extended to depth estimation effortlessly, by inserting a small number of learnable converters while leaving the original model completely frozen.

Key Contributions

1. A play-and-plug conversion framework. MERGE inserts a small, learnable transformer block (a "converter") before each text-to-image transformer block in a frozen DiT-based T2I model. Running the converters switches the model into depth estimation mode; skipping them restores the original text-to-image generator exactly, with zero drift in generation quality.

2. The Group Reuse Mechanism (GRE). Based on the observation that adjacent transformer blocks produce highly similar output features (measured by cosine similarity), MERGE groups consecutive blocks and shares a single converter within each group. This roughly halves the additional parameter count at a small performance cost.

3. Empirically simplified converters. The authors show that the cross-attention sub-layer is redundant because depth estimation uses an empty text prompt, removing it saves ~25% of converter parameters, and reducing the feed-forward expansion rate from 4 to 1 saves another ~36% with negligible accuracy loss. Self-attention and the feed-forward network remain essential.

4. State-of-the-art results among unified generation-perception models. MERGE outperforms OneDiffusion (trained from scratch on 100M samples) and the parallel dual-model methods JointNet and UniCon on NYUv2, ScanNet, and DIODE, while using roughly 1/1000th of OneDiffusion's training data.

Main Findings

• Generation ability is fully preserved. Because the pre-trained T2I model's weights never change, MERGE can switch back to generation mode by skipping converters. Marigold-style full fine-tuning and JointNet/UniCon's dual-model feature interaction both alter the original feature distribution irreversibly.

• Depth accuracy beats other unified models. MERGE-L (FLUX.1-dev backbone, 1.4B trainable params, 12% of the base model) reaches 5.9 A.Rel / 95.4% δ1 on NYUv2 and 31.4 / 76.3 on DIODE, beating OneDiffusion's 6.8 / 95.2 and 29.4 / 75.2. MERGE-B (PixArt backbone, 110M params, 18%) beats JointNet and UniCon comfortably.

• Comparable to full fine-tuning at a fraction of the cost. Marigold fine-tuned on the same PixArt backbone (596M parameters, 100%) scores 7.4 A.Rel on NYUv2; MERGE-B matches this at 110M parameters (7.5 A.Rel), and MERGE-B-28 (224M parameters) exceeds it at 7.0 A.Rel — while retaining generation capability that Marigold-P loses.

• Better than low-rank adaptation at equal parameter budget. At a matched 110M parameter count, LoRA (rank 128) scores 8.7 A.Rel on NYUv2 and DoRA scores 8.6, while MERGE-B scores 7.5. The authors attribute this to MERGE's structured, block-level converter design versus LoRA/DoRA's finer-grained per-layer modulation.

• Still behind dedicated discriminative models, but the gap narrows indoors. DepthAnything-v2 reaches 4.4 A.Rel / 97.9% δ1 on NYUv2 versus MERGE's 5.9 / 95.4, and the gap widens on outdoor benchmarks like DIODE. The advantage of MERGE is that it also generates images.

• Generalizes to surface normal estimation. With the same framework and 59K training samples, MERGE-L achieves mean angular errors of 20.1° on NYUv2, 18.3° on ScanNet, 18.2° on iBims-1, and 36.9° on Sintel — competitive with or better than task-specific full-fine-tuning methods like Lotus and StableNormal, while still preserving generation.

• Group Reuse is the critical design choice. Without GRE (a converter only before the first block of each group, same parameter count), A.Rel collapses from 7.5 to 15.6 and δ1 from 94.2 to 78.8. More groups give better accuracy (28 groups: 7.0; 14: 7.5; 7: 7.8; 4: 9.3) at proportionally higher parameter cost.

• Stacking converters backfires. Using 1, 2, or 3 stacked converters per insertion point yields A.Rel of 7.5, 7.6, and 8.2 respectively — increasing model depth makes optimization harder rather than adding capacity.

• Pre-trained initialization matters. Initializing converters from the corresponding pre-trained T2I block outperforms random initialization (7.5 vs 7.9 A.Rel).

• Richer text prompts give small gains. Empty prompt: 7.5 A.Rel; fixed "depth map": 7.4; LLaVA-generated dense captions: 7.3. The authors keep the empty prompt by default because the extra captioning cost outweighs the marginal benefit.

Methodology in Plain English

The researchers start with a frozen, off-the-shelf text-to-image diffusion transformer such as PixArt or FLUX. They do not touch a single pre-trained weight. Instead, before each transformer block in the denoising network, they insert a converter — an extra copy of that same kind of transformer block, this time with trainable weights. The converter's job is to nudge the latent features from "this looks like an image-generation feature" toward "this looks like a depth-estimation feature." If you want image generation, you simply skip all the converters and the network is byte-for-byte the original model.

To keep the extra parameter count small, they exploit a property they measured empirically: neighbouring transformer blocks in a pretrained model produce nearly identical output features. So instead of one converter per block, they split the blocks into groups and let each group share one converter. They also strip parts of the converter that turn out to be unnecessary — cross-attention goes away because depth estimation always uses an empty text prompt, and the feed-forward expansion factor drops from 4 to 1.

Training is done only on depth data (74K synthetic samples from Hypersim and Virtual KITTI), for 30K iterations on 8 NVIDIA H20 GPUs. The patchify layer's input channels are doubled to accept image conditioning, but that layer is swapped back for the original one at generation time, so nothing leaks into the generation path.

Why This Matters

Impact on research. The paper challenges two dominant assumptions in unified generation-perception modelling: that you must either fine-tune the whole backbone (destroying generation) or bolt on a second parallel model (doubling inference cost), and that unification requires massive multi-task training data. It reframes "unification" as a lightweight, reversible feature-space steering problem rather than a retraining problem, and provides a template that could plausibly extend to other dense prediction tasks.

Real-world applications:

• Augmented and virtual reality: a single on-device diffusion model that both synthesizes scene imagery and recovers depth for spatial mapping, without shipping two networks.
• Robotics and autonomous navigation: depth prediction for obstacle avoidance and manipulation, sharing a backbone with a generative model used for sim-to-real scene synthesis.
• 3D content creation and photo editing: tools that generate an image and immediately produce a consistent depth map for relighting, novel-view synthesis, or 2.5D parallax effects.
• Mixed-reality compositing and image restoration pipelines where depth and generation are used in alternating steps and switching cost matters.

Industry relevance. The 12–18% parameter overhead matters commercially: it means existing deployed checkpoints, quantization schemes, and inference servers for PixArt or FLUX can be extended rather than replaced. The training cost (74K samples, 30K iterations, 8 consumer-grade datacentre GPUs) is within reach of small teams, unlike the 100M-sample regime of OneDiffusion. The code is released publicly.

Future Directions

1. Extending to other dense prediction tasks. The authors explicitly state that semantic segmentation remains difficult — mapping semantic IDs through a stochastic denoiser is unstable, and bit-encoding schemes like LDMSeg are incompatible with the existing VAE. Solving the multi-task version of MERGE is the obvious next step.

2. Adaptive or learned grouping. The current grouping strategy is a fixed, evenly divided heuristic chosen "to avoid model-specific designs." A data-driven or similarity-driven grouping rule could tighten the accuracy-versus-parameter trade-off, especially since the paper shows accuracy improves monotonically with more groups.

3. Closing the gap to discriminative specialists. MERGE still trails DepthAnything-v2 notably on outdoor benchmarks like DIODE. Whether this reflects a fundamental limitation of generative depth or simply the small synthetic training set (74K samples) is an open question.

4. Better text conditioning. Dense captions gave a small but consistent improvement, suggesting the empty-prompt convention inherited from Marigold leaves accuracy on the table. Studying prompt design or lightweight text-conditioned converters could be a cheap source of gains.

5. Broader backbone coverage. The framework is demonstrated on DiT-based models (PixArt, FLUX). Whether the converter-plus-frozen-backbone recipe transfers to UNet-based diffusion architectures or to video diffusion models is untested.

Target Audience

Researchers and engineers working on diffusion models, 3D vision, or unified multi-task architectures will get the most from this paper — particularly those who need a practical way to extend an existing generative model with perception capability without retraining or maintaining two networks. It is also useful for practitioners evaluating how far lightweight adapters (LoRA, DoRA, control adapters) can be pushed before full fine-tuning becomes necessary. Readers should already understand how latent diffusion denoising works, what a DiT block contains, and how depth metrics like absolute relative error and δ1 are defined.

Authors’ abstract

Generative depth estimation methods leverage the rich visual priors stored in pre-trained text-to-image diffusion models, demonstrating astonishing zero-shot capability. However, parameter updates during training lead to catastrophic degradation in the image generation capability of the pre-trained model. We introduce MERGE, a unified model for image generation and depth estimation, starting from a fixed pre-trained text-to-image model. MERGE demonstrates that the pre-trained text-to-image model can do more than image generation, but also expand to depth estimation effortlessly. Specifically, MERGE introduces a play-and-plug framework that enables seamless switching between image generation and depth estimation modes through simple and pluggable converters. Meanwhile, we propose a Group Reuse Mechanism to encourage parameter reuse and improve the utilization of the additional learnable parameters. MERGE unleashes the powerful depth estimation capability of the pre-trained text-to-image model while preserving its original image generation ability. Compared to other unified models for image generation and depth estimation, MERGE achieves state-of-the-art performance across multiple depth estimation benchmarks. The code will be made available at https://github.com/H-EmbodVis/MERGE