Boosting Monocular Metric Depth Estimation via Bokeh Rendering

AI summary

Overview

Research area: Computer vision, specifically monocular metric depth estimation (MMDE) and physically grounded bokeh (defocus) rendering, drawing on thin-lens optics, diffusion/rectified-flow image generation, and Vision Transformer depth backbones.

Technical level: Advanced — the paper assumes familiarity with diffusion transformers, DPT-style depth decoders, depth-from-defocus theory, and metric depth evaluation protocols.

Scope: The paper introduces BokehDepth, a two-stage framework that first generates depth-free but physically calibrated bokeh stacks from a single sharp image and then injects the resulting defocus cues into existing monocular metric depth encoders through a plug-in attention module.

What This Paper Is About

Monocular metric depth estimation and bokeh rendering are linked by lens imaging geometry, but current systems exploit this link poorly: bokeh pipelines depend on noisy depth maps that produce artifacts, while depth models either lack absolute scale (generative diffusion approaches) or fail in textureless and distant regions where defocus blur could supply geometric information. The paper's goal is to break this circular dependency by treating synthetic defocus as a supervision-free geometric cue — generating calibrated blur without any depth input, then using those blur variations to sharpen and stabilize metric depth predictions.

Key Contributions

1. A depth-free, calibrated bokeh renderer. Stage-1 augments a pretrained image-editing backbone (FLUX.1-Kontext) with a bokeh cross-attention adapter driven by a single scalar bokeh strength K derived from the thin-lens circle-of-confusion model. Trained on a unified real-and-synthetic data pipeline, it produces multi-strength bokeh stacks from a single sharp image without any depth map.

2. A plug-and-play defocus-aware module for depth encoders. Divided Space Focus Attention (DSFA) is inserted into the encoder of a monocular metric depth model. It performs spatial attention within each bokeh frame and focus attention across frames at aligned spatial locations, with FiLM conditioning on the bokeh strength, while leaving the decoder and metric head unchanged.

3. A formal depth-from-bokeh-sweep statement. Proposition 4.1 states that when only the calibrated bokeh strength K is swept with fixed pose and focus distance, the measured bokeh radius at every pixel is proportional to that pixel's inverse-depth offset from the focal plane, and the ordinary-least-squares slope of radius on K is an unbiased and consistent estimator of that offset (up to the front/behind-focus sign). The full proof is stated to be in the supplementary material.

4. Demonstrated gains in both bokeh fidelity and depth accuracy. The combined BokehDepth framework improves visual fidelity over depth-map-based bokeh baselines and consistently boosts the metric performance of strong monocular depth models in in-domain and zero-shot settings.

Main Findings

• Stage-1 bokeh quality on real photographs: On the exposure-aligned EBB! Val200 split, BokehDepth reports the best PSNR (25.91), the lowest LPIPS (0.185) and the lowest DISTS (0.076), with SSIM 0.856. For comparison, GenFocus reports PSNR 25.48, SSIM 0.886, LPIPS 0.278, DISTS 0.081; DiffCamera reports 25.07 / 0.867 / 0.311 / 0.102; BokehDiff 24.35 / 0.802 / 0.279 / 0.112; BokehMe 24.13 / 0.751 / 0.390 / 0.143; DrBokeh 22.61 / 0.735 / 0.435 / 0.178; unmodified FLUX.1-Kontext 19.92 / 0.645 / 0.165 / 0.427.

• Stage-1 bokeh quality on synthetic scenes: On the SYNTHEBOKEH300 benchmark, DrBokeh achieves the strongest PSNR (29.70) with direct depth access and layered rendering, while BokehDepth reaches PSNR 29.12 and reports the best SSIM (0.901), LPIPS (0.139) and DISTS (0.073). Other entries: DiffCamera 25.26 / 0.880 / 0.238 / 0.120, BokehDiff 26.21 / 0.807 / 0.288 / 0.142, BokehMe 22.83 / 0.809 / 0.279 / 0.188, GenFocus 21.54 / 0.895 / 0.317 / 0.153, FLUX.1-Kontext 20.34 / 0.685 / 0.351 / 0.155.

• Zero-shot metric depth across six benchmarks: Table 2 evaluates on HAMMER, IBims-1, Middlebury, Make3D, Sintel and ETH3D. Adding DSFA improves both base backbones. Depth Anything V2 goes from δ1 0.828 / AbsRel 0.134 (HAMMER) to 0.894 / 0.105; from 0.938 / 0.080 (IBims-1) to 0.960 / 0.064; from 0.618 / 0.240 (Middlebury) to 0.675 / 0.215; from 0.726 / 0.217 (Make3D) to 0.740 / 0.206; ETH3D from 0.910 / 0.091 to 0.954 / 0.077. UniDepthV2 goes from δ1 0.645 / AbsRel 0.293 (HAMMER) to 0.895 / 0.094; IBims-1 from 0.945 / 0.082 to 0.978 / 0.039; Middlebury from 0.535 / 0.288 to 0.716 / 0.205; Make3D from 0.739 / 0.263 to 0.786 / 0.228; Sintel from 0.344 / 0.496 to 0.671 / 0.391; ETH3D from 0.852 / 0.160 to 0.963 / 0.074.

• In-domain NYUv2 depth accuracy: With a UniDepthV2 backbone, BokehDepth reports Abs_Rel 0.039, RMS 0.043 and Log10 0.016 on the NYUv2 validation set, versus UniDepthV2 at 0.047 / 0.180 / 0.020, Metric3Dv2 at 0.047 / 0.183 / 0.020, DepthAnythingv2 at 0.056 / 0.206 / 0.024 and ZoeDepth at 0.077 / 0.278 / 0.033.

• Comparison to dedicated depth-from-defocus methods: On NYUv2 validation, BokehDepth reports the highest δ1 (0.978) and the lowest RMS (0.043). Reference points: SDNet 0.964 / 0.026 / 0.201, DFF-DFV 0.967 / 0.445 / 0.232, DAIF-Net 0.950 / 0.170 / 0.325, Deep-Optics 0.930 / 0.087 / 0.433, 2HDED:Net 0.914 / 0.029 / 0.244. The paper notes that every competing DfD method builds its evaluation focal stack by rendering RGB-D pairs through a thin-lens or PSF forward model, implicitly encoding ground-truth depth, whereas BokehDepth generates its stack from a single sharp image with no depth access.

• Ablation — the bokeh source matters: On VKITTI2 training and the KITTI Eigen split, the full method reaches δ1 0.943 / AbsRel 0.084. Removing the Focus branch gives 0.620 / 0.215, removing the Space branch 0.567 / 0.236, removing FiLM conditioning 0.602 / 0.223, and repeating the all-in-focus image across the stack yields 0.855 / 0.119. Replacing Stage-1 with BokehMe using ground-truth sparse LiDAR depth gives 0.914 / 0.094, with predicted dense depth (Pred-BokehMe) 0.918 / 0.094, and uncalibrated FLUX.1-Kontext renderings collapse to 0.608 / 0.227.

• Computational profile: UniDepthV2 baseline is 353.83M parameters, 0.045 s/img latency, 2.48 GiB memory; with DSFA this becomes 464.07M, 0.114 s/img, 3.25 GiB. Stage-1 bokeh rendering alone is 12.32B parameters, 48.663 s/img and 33.52 GiB (30 diffusion steps), and the full pipeline is 12.78B, 48.777 s/img, 33.52 GiB. Stage-1 is described as fully offline, with outputs precomputable and cacheable, leaving online inference nearly identical to the baseline.

Methodology in Plain English

The authors separate the two halves of the problem instead of letting them depend on each other.

Stage 1 — making blur without knowing depth. Instead of predicting depth and then blurring accordingly, they fine-tune a strong pretrained image-editing model (FLUX.1-Kontext) to produce blurred versions of a sharp input. To keep the blur physically meaningful, they collapse the camera settings that determine defocus — focal length f, aperture number N and focus distance S1 — into a single number K, using the thin-lens circle-of-confusion formula scaled into pixel units. Training data comes from three sources mapped onto the same K scale: real defocused photos with EXIF metadata, physically rendered blur from BokehMe, and paired datasets (DPDD, BLB) whose aperture settings are re-parameterized. Conditioning is implemented as an extra "bokeh branch" inside the model's unified attention, using small LoRA-style projections of a K-derived conditioning vector, and added to the normal attention output. The result is a stack of the same scene at increasing blur strengths, generated without any depth map.

Stage 2 — reading depth out of blur. The sharp frame plus the bokeh stack are passed through a shared ViT encoder. A small module called Divided Space Focus Attention is inserted inside the encoder and works in two steps: first it attends spatially within each frame (prepending a token that encodes that frame's K value), then it attends across frames at each matching pixel location, using FiLM scale-and-shift modulation conditioned on the frame's K. Only the reference frame's refined tokens are kept and sent to the unchanged DPT-style decoder, which predicts metric depth. The intuition is that at a given pixel, blur changes with K in a way that depends on how far that pixel is from the focal plane, so comparing across the stack exposes a depth cue that does not depend on texture.

Evaluation. Stage-1 is tested on real photos (EBB! Val200) and on a constructed synthetic benchmark (SYNTHEBOKEH300, with foreground/background layers rendered using accurate depth maps) using PSNR, SSIM, LPIPS and DISTS. Stage-2 is plugged into Depth Anything V2-L and UniDepthV2-L; cross-domain zero-shot runs fine-tune on Hypersim only, while in-domain runs fine-tune and test on NYUv2. Ablations are trained on VKITTI2 and evaluated on the KITTI Eigen split. The bokeh stacks used contain three frames with K ∈ [10, 30].

Why This Matters

Impact on research. The work reframes defocus as a supervision-free geometric signal rather than an artifact to be cleaned up. If defocus stacks can improve metric depth without ground-truth depth at inference, that suggests a route to depth models that learn image formation and geometry jointly, and it offers a way to exploit weak-texture and distant regions where appearance-based single-frame cues are ambiguous. It also challenges the standard dependency of bokeh rendering on depth estimators.

Real-world applications (as suggested by the setup and evaluations):

• Portrait and smartphone computational photography, where shallow depth-of-field effects must be produced without a reliable depth map.
• Indoor 3D scene understanding and reconstruction, given the indoor benchmarks used (NYUv2, Hypersim, IBims-1, Middlebury).
• Autonomous driving and outdoor robotics, given the KITTI and VKITTI2 evaluations and the real-time constraints of metric depth for navigation.
• Augmented and mixed reality, where consistent metric scale and clean layer separation around occluding edges matter.

Industry relevance. The plug-in design means a depth model can be upgraded without retraining its decoder or changing its interface, which lowers adoption cost. The reported overhead once bokeh stacks are cached is small (DSFA adds 110.24M parameters and roughly 0.069 s/img latency on the UniDepthV2 backbone in the reported profile), which makes the approach plausible as an offline augmentation stage in existing production pipelines.

Future Directions

• Making Stage-1 fast. The paper states the current Stage-1 latency comes from the unoptimized multi-step diffusion schedule (30 steps) rather than an architectural bottleneck, and points to one-step or few-step distillation as a path to real-time bokeh-stack synthesis. This is not demonstrated in the reported experiments.
• Resolving the depth sign ambiguity. Proposition 4.1 recovers depth "up to the usual front/behind-focus sign," so determining whether a pixel is in front of or behind the focal plane remains an open question.
• Broadening evaluation scope. Results are reported for two backbones (Depth Anything V2-L and UniDepthV2-L), with zero-shot fine-tuning on Hypersim only and in-domain work on NYUv2. Whether the gains generalize to other backbones, larger bokeh stacks, or wider K ranges is not reported.
• Moving beyond single images. The framework operates on static scenes with fixed camera pose and focus distance; extension to video, where the bokeh stack would need temporal consistency, is not addressed. Several benchmark dataset sizes and the full supplementary proof are also not included in the provided content.

Target Audience

Researchers and practitioners in computer vision and computational photography working on monocular depth estimation, depth-from-defocus, or neural bokeh rendering; engineers building depth-aware camera features for smartphones, AR/VR, or autonomy; and graduate-level readers already comfortable with Vision Transformers, diffusion/rectified-flow models, and metric depth evaluation metrics, who want to understand how optical defocus can be turned into a usable geometric signal.

Authors’ abstract

Bokeh rendering and depth estimation share a fundamental optical connection, yet existing methods fail to fully exploit this reciprocity. Conventional bokeh pipelines rely heavily on noisy depth maps that inevitably introduce visual artifacts. Conversely, existing monocular depth models typically follow two flawed paradigms. Generative diffusion-based frameworks often lack consistent metric scale. Meanwhile, feed-forward metric depth models frequently fail in textureless or distant regions where defocus blur can provide geometric information. We propose BokehDepth, a two-stage framework that treats synthetic defocus as a supervision-free geometric signal. In the first stage, a physically grounded generative model produces calibrated bokeh stacks from a single sharp input without requiring prior depth input. Subsequently, a lightweight defocus-aware aggregation module integrates these stacks into the encoder of a depth estimation framework. This mechanism allows the model to extract consistent geometric features from the defocus dimension while keeping the decoder architecture unchanged. Experiments demonstrate that BokehDepth achieves superior visual bokeh fidelity compared to depth-dependent rendering baselines and consistently enhances the metric accuracy of state-of-the-art monocular depth models.