StereoSpace: Depth-Free Synthesis of Stereo Geometry via End-to-End Diffusion in a Canonical Space

AI summary

Overview

Research area: Computer vision — generative models for monocular-to-stereo image synthesis (converting a single 2D image into a left-right stereo pair).

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, Plücker coordinates, rectified stereo geometry, and modern evaluation metrics such as MEt3R and iSQoE.

Scope: The paper proposes StereoSpace, a depth-free diffusion framework that learns stereo geometry through viewpoint conditioning in a canonical rectified space, and a leakage-free evaluation protocol for stereo generation.

Authors: Tjark Behrens, Anton Obukhov, Bingxin Ke, Fabio Tosi, Matteo Poggi, Konrad Schindler (affiliations listed include ETH Zurich, University of Bologna, and HUAWEI Bayer Lab). Project page: https://hf.co/spaces/prs-eth/stereospace

What This Paper Is About

Generating a stereo pair from one monocular image is usually done by estimating depth, warping pixels into a second viewpoint, and inpainting the holes. The authors argue this shortcut inherits every failure of the depth estimator, especially on layered or non-Lambertian scenes such as glass, transparency, and reflections, where a single disparity map cannot describe multiple surfaces at once. StereoSpace instead reformulates the task as image-conditioned 3D generation: the model is told where the target camera sits, expressed in physical units inside a canonical rectified frame, and the diffusion process learns the correspondence and disocclusion behavior end to end without any explicit depth volume.

Key Contributions

1. StereoSpace, a single-image conditional generator of the counterpart stereo view that is free of explicit geometric shortcuts — no depth estimation, no warping, no inpainting stage.
2. An end-to-end training procedure for efficiently transferring a task-agnostic foundation prior to stereo synthesis, built on a dual U-Net initialized from Stable Diffusion 2.0 and supervised with velocity, photometric, and warp-consistency losses.
3. A novel perceptual and geometry-aware evaluation protocol that combines iSQoE (perceptual comfort) and MEt3R (geometric consistency), and that excludes any ground truth or proxy geometry estimates at test time, avoiding the test-time leakage the authors attribute to earlier protocols.
4. A canonical "StereoSpace" formulation in which the stereo rig is fixed at the origin with both cameras on the x-axis separated by the baseline B, letting a user set the baseline directly in physical units at inference.

Main Findings

• Depth-free wins on layered scenes: On Booster and LayeredFlow — datasets with transparency and multi-layer depth — StereoSpace attains the best scores on both metrics: Booster iSQoE 0.6764 and MEt3R 0.1013, LayeredFlow iSQoE 0.7489 and MEt3R 0.1619. StereoDiffusion (0.7248 / 0.2011 and 0.8046 / 0.3074), ZeroStereo (0.7503 / 0.3171 and 0.8108 / 0.3630), GenStereo (0.6901 / 0.1457 and 0.7678 / 0.2275), and Lyra (0.6989 / 0.1293 and 0.7802 / 0.1877) all trail.
• Best on indoor and driving benchmarks: On Middlebury 2014, StereoSpace scores iSQoE 0.6829 and MEt3R 0.0893, versus GenStereo 0.6933 / 0.1339 and Lyra 0.7184 / 0.1163 — which the paper describes as improvements of more than 30% and 20% respectively on MEt3R. On DrivingStereo it reaches iSQoE 0.7829 and MEt3R 0.0717, ahead of GenStereo (0.7850 / 0.0728) and ZeroStereo (0.7964 / 0.0798).
• Plücker ray conditioning is the best conditioning choice: On Middlebury, text-prompt baseline conditioning gives 0.6841 / 0.0907, PRoPE-style projective attention gives 0.6865 / 0.0937, combining Plücker with PRoPE gives 0.6828 / 0.0945, and Plücker alone gives the best 0.6823 / 0.0901. Even the weakest conditioning variant still surpasses GenStereo.
• Multi-baseline training data helps: Removing the multi-baseline sources (NeRF-Stereo, SceneSplat) degrades results to 0.6907 / 0.1095, though that reduced-data variant still beats GenStereo.
• Auxiliary disparity supervision trades comfort for geometry: Adding a disparity loss yields a modest MEt3R improvement (0.0893) with a slight iSQoE degradation (0.6829).
• Conventional metrics mislead: In Figure 4 on Middlebury, GenStereo records PSNR 18.3323 / SSIM 0.6061 against StereoSpace's PSNR 14.3611 / SSIM 0.4211, despite visibly worse artifacts. The supplementary table shows the same pattern across all four datasets (for example Booster PSNR 21.74 for GenStereo and 23.21 for Lyra versus 21.91 for StereoSpace).
• Human judgment agrees with the proposed metrics: A user study with 70 participants and a total of 1.4K pairwise comparisons (20 pairs per subject) ranks StereoSpace first under a Bradley-Terry preference model, with a roughly 60-40 winrate over Lyra.
• Baseline control works in metric units: Because StereoSpace is canonicalized, the target baseline in world units acts as conditioning at inference, giving predictable control across baselines rather than requiring a fixed calibrated rig.

Methodology in Plain English

The authors start from the observation that in a rectified stereo setup the only thing that matters is the relative calibration — focal length f and metric baseline B — not absolute camera poses. So they define a canonical space where the stereo rig sits at the origin and both cameras lie on the x-axis separated by B.

They then treat target-view synthesis as conditional diffusion. Given a source image, the model predicts the target image while being told exactly where the target camera is. That information is supplied as dense per-pixel Plücker ray embeddings (a 6D moment-and-direction vector per ray), injected both through Adaptive Layer Normalization inside the ResNet blocks of a dual U-Net and by concatenation to the input latents. The architecture is a reference U-Net plus a denoising U-Net, both initialized from Stable Diffusion 2.0, with the first convolution widened to accept 10 channels (a 4-channel VAE latent plus the 6D Plücker embedding). CLIP ViT-H/14 features provide semantic conditioning.

Training mixes roughly 750K single-baseline stereo pairs from 12 synthetic and photorealistic datasets — the largest sources being TartanAir (306K pairs), Dynamic Replica (145K), IRS (103K), Falling Things (61K), and LayeredFlow (31K) — with multi-baseline data: 27K multi-view tuples from NeRF-Stereo and 5K from SceneSplat-7K. Each tuple holds 5-7 rectified views along a shared baseline direction, producing 10-21 stereo pairs. Smaller datasets are resampled to 10% of the largest, and multi-baseline tuples are weighted by 10 × the number of tuples.

Optimization combines the velocity-matching objective with a photometric term (SSIM plus ℓ1) and a masked warp-consistency term that back-warps the predicted target into the source frame using ground truth disparity, with a left-right consistency mask. The model trains for 3 epochs (~48.6K optimizer steps) at 768×768 resolution, and unlike competing methods it is trained in both stereo directions.

Evaluation is deliberately end-to-end. No method may see ground truth geometry at test time; instead, per-scene calibration searches over a baseline or depth-to-disparity scale (search range [0.025, 1]) to minimize RMSE between SGBM disparity maps. All four test sets — Middlebury 2014, DrivingStereo, Booster, LayeredFlow — are downsampled to 512×512 before metrics are computed.

Why This Matters

Impact on research. The paper challenges the dominant warp-and-inpaint paradigm for monocular-to-stereo conversion, showing that a diffusion model's generative capacity can absorb the geometry problem entirely rather than depending on a depth estimator's generalization. It also argues that PSNR, SSIM, and LPIPS reward over-smoothed, pixel-aligned predictions — GenStereo scores better on them than StereoSpace in every reported setting — and proposes iSQoE plus MEt3R as more decision-relevant alternatives. The end-to-end, leakage-free protocol is a methodological contribution that applies beyond this specific model.

Real-world applications.

• 2D-to-3D conversion for cinema, where stereo acquisition is expensive and requires precise alignment, synchronization, and calibration.
• Virtual reality and augmented reality content pipelines, where viewing comfort (captured by iSQoE) matters as much as geometric accuracy.
• Consumer devices and streaming services converting legacy 2D libraries into stereo without per-scene depth estimation.
• Difficult real-world captures — glass, transparent surfaces, specular or non-Lambertian materials — where warping pipelines generate geometric drift and blur.

Industry relevance. Because the method needs no depth network at inference, the deployment stack is smaller and avoids cascading failures from a separate depth model. Metric baseline conditioning in world units gives content creators direct, physical control over the strength of the 3D effect, which matters for stereo authoring standards where excessive disparity causes viewer discomfort.

Future Directions

• Stereo video generation. The conclusion explicitly states that future work will aim at extending StereoSpace to stereo video.
• Alternative conditioning parameterizations. The paper notes the method is not tied to Plücker embeddings and that other camera-aware parameterizations, or mixtures of them, can be substituted without substantial differences; exploring which work best remains open.
• Metric quality versus perceptual comfort. The disparity-loss ablation shows a modest MEt3R gain at a slight iSQoE cost, leaving the question of how best to balance geometric alignment against viewing comfort unresolved.
• Conventional-metric gaps. StereoSpace scores lower than GenStereo on PSNR, SSIM, and LPIPS across Middlebury, DrivingStereo, Booster, and LayeredFlow, so how much of that gap is genuine fidelity loss versus a bias of pixel-alignment metrics is still an open question the paper raises rather than settles.

Target Audience

Researchers and engineers working on novel view synthesis, diffusion-based generative vision, and stereo imaging for XR, VR, and 3D cinema. It is most useful to readers who already understand latent diffusion and stereo geometry and want to see how viewpoint conditioning can replace depth-based warping pipelines, as well as to practitioners interested in evaluation protocols for stereo quality.

Authors’ abstract

We introduce StereoSpace, a diffusion-based framework for monocular-to-stereo synthesis that models geometry purely through viewpoint conditioning, without explicit depth or warping. A canonical rectified space and the conditioning guide the generator to infer correspondences and fill disocclusions end-to-end. To ensure fair and leakage-free evaluation, we introduce an end-to-end protocol that excludes any ground truth or proxy geometry estimates at test time. The protocol emphasizes metrics reflecting downstream relevance: iSQoE for perceptual comfort and MEt3R for geometric consistency. StereoSpace surpasses other methods from the warp & inpaint, latent-warping, and warped-conditioning categories, achieving sharp parallax and strong robustness on layered and non-Lambertian scenes. This establishes viewpoint-conditioned diffusion as a scalable, depth-free solution for stereo generation.