Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge

AI summary

Overview

• Research area: Generative modeling for modality translation (MT) — cross-modal generation across images, 3D shapes, audio, and scene representations — with a focus on diffusion bridge models and contrastive representation learning.
• Technical level: Advanced. The paper builds on stochastic differential equations, Doob's h-transform, score matching, and contrastive estimation, although its core design choices (shared latent space, encoder-decoder transformer, alignment loss) can be understood at an intermediate level.
• Scope: The paper proposes LDDBM (Latent Denoising Diffusion Bridge Model), a task-agnostic framework that bridges any two paired modalities through a shared latent space, and evaluates it on four benchmark tasks plus architecture and loss ablations.

What This Paper Is About

Denoising Diffusion Bridge Models (DDBMs) can translate between two data distributions, but they assume both endpoints live in the same vector space (for some d), which rules out translating between modalities that differ in dimensionality, structure, and semantics. This paper introduces a latent extension of DDBMs that encodes each modality into a shared latent space, learns a diffusion bridge between the two encodings, and decodes back to the target modality — so arbitrary modality pairs can be translated without shared dimensionality, Gaussian source priors, or modality-specific denoiser architectures. The paper's goal is a general-purpose, theoretically grounded baseline for modality translation rather than a new task-specific state-of-the-art.

Key Contributions

1. Latent Denoising Diffusion Bridge Model (LDDBM): a latent-variable extension of DDBMs that operates on embeddings and can bridge arbitrary modality pairs where the source and target dimensions differ, and where paired data may not share an underlying manifold.
2. Two new objectives for bridge training: a contrastive alignment loss (infoNCE, temperature τ = 0.5) that pulls paired source/target latents together and pushes unpaired examples apart, and a predictive loss that compares the decoded output of the full encode–bridge–decode pipeline to the ground-truth target.
3. A domain-agnostic denoiser architecture: an encoder-decoder Transformer in which a Transformer encoder produces a "memory" embedding from the target-modality tokens, which the Transformer decoder consumes via cross-attention; the decoder input combines the noisy latent tokens with learnable [MASK] (context-summarization) tokens, and timestep embeddings modulate self-attention, cross-attention, and feedforward outputs.
4. A study of training strategies: two-step training (autoencoders then bridge), fully end-to-end training, and an iterative scheme that alternates between reconstruction and bridge alignment, which the authors report as the best trade-off between stability and performance.

Main Findings

• Multi-view to 3D shape generation (ShapeNet, four rendered views, following the protocol of prior work): LDDBM reaches the lowest 1-NNA of .508 ± .005 (lower is better) and the highest IoU of .664 ± .002 (higher is better), versus EDM (.532 ± .013 / .631 ± .006), 3D-EDM (.575 ± .009 / .602 ± .003), DiT (.548 ± .004 / .613 ± .011), and SiT (.563 ± .007 / .604 ± .003). The task-specialized Pix2Vox-A is reported with IoU 0.697 and no 1-NNA value.
• Zero-shot low-to-high resolution (trained on FFHQ, evaluated on CelebA-HQ, 16×16 to 128×128): LDDBM reports PSNR 25.6 ± 0.4, SSIM 0.68 ± .03, and LPIPS 0.32 ± .01, against EDM (23.1 ± 0.7 / 0.58 ± .05 / 0.41 ± .02), DiWa (23.3 / 0.65 / 0.39, with standard deviations not extractable), DiT (22.2 ± 1.1 / 0.52 ± .07 / 0.49 ± .01), and SiT (21.5 ± 0.4 / 0.57 ± .02 / 0.51 ± .03).
• Cross-modal biometric translation (face–voice): LDDBM achieves 71.2 for Face→Voice and 75.1 for Voice→Face, outperforming the general-purpose SiT baseline (65.7 and 68.3) but trailing the task-specific method of the benchmark being followed (79.5 and 81.0).
• Image-to-image translation (Edges→Bags, batch size 256): LDDBM reports FID 4.17 and inference time 7.8 seconds, versus DDBM's FID 2.93 and 16.9 seconds — a quality/efficiency trade-off where LDDBM is more than 2× faster at competitive quality.
• Architecture ablation: U-Net gives ShapeNet IoU .635 / 1-NNA .518; switching to decoder-only DiT drops to .613 / .548; adding an encoder-decoder reaches .651 / .518; adding spatial embedding gives .658 / .522; adding the [MASK] token gives the best scores .664 / .508. On CelebA-HQ the full model reaches PSNR 25.6, SSIM 0.68, LPIPS 0.32, compared with DiT's 22.2 / 0.52 / 0.49 and U-Net's 23.2 / 0.57 / 0.42. On nuScenes the full model gives IoU .233, though its 1-NNA of .807 is not the best in that column (U-Net is .818, DiT .825, encoder-decoder .821, spatial embedding .812).
• Qualitative and latent-space analysis: the paper shows super-resolution examples where the basic objective produces artifacts and loses high-frequency detail (an example given is a yellow hair ornament), and t-SNE plots indicating that the basic formulation clusters within modalities but fails to align semantically similar multi-view and 3D shape samples across modalities.

Methodology in Plain English

The method reframes cross-modal translation as three steps. First, each modality gets its own encoder to map data into a shared latent space (the only modality-specific component); in experiments these are simple neural networks or pre-trained models, and a decoder maps the latent back to the target modality. Second, instead of translating directly between raw data spaces, the model learns a diffusion bridge in the latent space, where the process starts from the encoded source and is trained to reach the encoded target using the score-matching objective of DDBMs under a variance-exploding scheme. Third, training combines three terms: the bridge score-matching loss, a predictive loss that runs a sample through encode–bridge–decode and compares the result to the ground truth in pixel space, and a contrastive infoNCE loss over the batch that treats the paired latents as a positive pair and all other batch samples as negatives. The denoiser is a Transformer encoder-decoder: the encoder summarizes the target-side latent tokens into a memory embedding, and the decoder cross-attends to that memory while denoising the noisy latent tokens, with learnable [MASK] summary tokens and shared positional encodings to keep the input and output tokens spatially aligned. At inference, the memory is computed once for the whole Heun sampling loop, the first denoising step replaces the noisy tokens with the source tokens, and sampling proceeds autoregressively.

Why This Matters

• Research impact: The work removes the same-dimensionality assumption of DDBMs, turning bridge models from a same-modality translation tool into a general cross-modal framework, and it introduces contrastive and predictive supervision for latent bridges that other bridge formulations could adopt. It positions itself against concurrent flow-matching approaches (CrossFlow, FlowTok) and latent bridge work (DPBridge), which the authors argue are tied to specific modality pairs or backbones.
• Real-world applications:
  • Generating 3D assets from a handful of photographs.
  • Upscaling or restoring low-resolution imagery without task-specific fine-tuning (demonstrated zero-shot from FFHQ training to CelebA-HQ evaluation).
  • Face–voice cross-modal matching, relevant to biometrics and forensic workflows.
  • Multi-view scene and occupancy synthesis for embodied and driving settings, given the authors' Bosch AI Center affiliation and the nuScenes experiments.
• Industry relevance: Because the only modality-specific parts are the encoders and decoders, and the denoiser is domain-agnostic, the framework is a plug-in template for product teams that already have per-modality encoders and want one translation engine rather than one model per task. The reported inference speed advantage (7.8 s versus 16.9 s on Edges→Bags) is directly relevant to deployment costs.

Future Directions

• Extending beyond paired supervision: the paper notes that its framework operates in the paired translation paradigm and that unpaired and weakly paired setups have received substantial attention, leaving those regimes open.
• Swapping the bridging module: the authors state the framework is agnostic to the choice of bridge and could be extended to alternatives such as optimal transport, standard diffusion, or other samplers referenced in the paper.
• Resolving the encoder–bridge conflict: the paper describes the tension between trainable encoders reshaping the latent distribution and the bridge's fixed-marginal assumption, motivates the iterative training scheme from adversarial training practice, and points to stability as an ongoing concern.
• Improving the fidelity of latent bridging: the authors identify loss of fine-grained high-frequency detail as a limitation of latent-domain bridging, and their architecture ablation shows the [MASK] token and spatial embeddings matter, suggesting further architectural refinement — note that the paper's loss ablation results are referenced but the numerical details are not contained in the provided content.

Target Audience

Researchers and practitioners working on diffusion models, bridge models, and multimodal generation who want a task-agnostic recipe for translating between heterogeneous modalities; engineers in 3D content creation, image restoration, biometrics, and autonomous driving who need a single translation backbone rather than one model per task; and readers interested in latent-space alignment objectives such as contrastive and predictive losses applied to generative bridges.

Authors’ abstract

Recent advances in generative modeling have positioned diffusion models as state-of-the-art tools for sampling from complex data distributions. While these models have shown remarkable success across single-modality domains such as images and audio, extending their capabilities to Modality Translation (MT), translating information across different sensory modalities, remains an open challenge. Existing approaches often rely on restrictive assumptions, including shared dimensionality, Gaussian source priors, and modality-specific architectures, which limit their generality and theoretical grounding. In this work, we propose the Latent Denoising Diffusion Bridge Model (LDDBM), a general-purpose framework for modality translation based on a latent-variable extension of Denoising Diffusion Bridge Models. By operating in a shared latent space, our method learns a bridge between arbitrary modalities without requiring aligned dimensions. We introduce a contrastive alignment loss to enforce semantic consistency between paired samples and design a domain-agnostic encoder-decoder architecture tailored for noise prediction in latent space. Additionally, we propose a predictive loss to guide training toward accurate cross-domain translation and explore several training strategies to improve stability. Our approach supports arbitrary modality pairs and performs strongly on diverse MT tasks, including multi-view to 3D shape generation, image super-resolution, and multi-view scene synthesis. Comprehensive experiments and ablations validate the effectiveness of our framework, establishing a new strong baseline in general modality translation. For more information, see our project page: https://sites.google.com/view/lddbm/home.