Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation

AI summary

Overview

• Research area: Computer vision, specifically monocular video depth and surface-normal estimation for transparent and highly reflective objects, with validation in robotic grasping.
• Technical level: Advanced. It assumes familiarity with diffusion models, flow matching, latent VAEs, DiT backbones, and LoRA adapters.
• One-sentence scope: The paper introduces a synthetic transparent-object video dataset (TransPhy3D) and repurposes a pretrained video diffusion model (WAN) via LoRA fine-tuning to produce temporally consistent depth and normal videos for transparent and reflective scenes, which it then tests on real benchmarks and a real robot.

What This Paper Is About

Transparent and reflective objects break the assumptions behind stereo, time-of-flight, and discriminative monocular depth estimation, producing holes and temporally unstable depth maps. The authors' premise is that modern video diffusion models already synthesize convincing transparent phenomena, implying they have internalized optical rules such as refraction and reflection, and that these priors can be repurposed for perception. Their goal is a model that estimates depth (and, in a variant, surface normals) from arbitrary-length videos containing transparent objects, trained purely on synthetic data yet generalizing zero-shot to real benchmarks.

Key Contributions

1. TransPhy3D, described as the first synthetic video dataset for transparent and highly reflective objects: 11,000 videos and 1,320,000 frames (120 frames per scene), rendered with Blender/Cycles and OptiX denoising, with paired RGB, depth, and normal ground truth.
2. A rendering pipeline and asset bank combining 5,574 assets collected from BlenderKit (filtered down to 574 high-quality assets using an aesthetic score from Qwen2.5-VL-7B) with shape-rich procedural parametric assets, paired with glass, plastic, metal, and glazed-ceramic materials.
3. DKT, the first foundation model for transparent-object video depth estimation, built by repurposing a pretrained video diffusion model (WAN) with lightweight LoRA adapters and an RGB-conditioning latent concatenation, plus a co-training strategy that mixes frame-wise synthetic image datasets (HISS, DREDS, ClearGrasp) with TransPhy3D.
4. Comprehensive benchmarking and real-robot validation, showing zero-shot state-of-the-art results on ClearPose, DREDS (CatKnown/CatNovel), and TransPhy3D-Test, a normal variant (DKT-Normal) that leads video normal estimation on ClearPose, and grasping experiments where DKT's depth improves success rates across translucent, reflective, and diffuse surfaces.

Main Findings

• Depth accuracy on ClearPose: DKT reports REL 9.72, RMSE 14.58, δ1.05 38.17, δ1.10 65.50, δ1.25 93.04, with an average rank of 1.8. The paper states it outperforms the second-best method by 5.69, 9.13, and 3.1 for δ1.05, δ1.10, and δ1.25 respectively.
• Depth accuracy on TransPhy3D-Test: DKT reports REL 2.96, RMSE 19.50, δ1.05 87.17, δ1.10 97.09, δ1.25 98.56, rank 1.2. The stated margins over the second-best method are 55.25, 40.53, and 9.97 for δ1.05, δ1.10, and δ1.25.
• Depth accuracy on DREDS: On DREDS-STD-CatKnown, DKT reports REL 5.30, RMSE 4.96, δ1.05 53.86, δ1.10 84.93, δ1.25 99.89, rank 1.0. On DREDS-STD-CatNovel, DKT reports REL 5.71, RMSE 4.66, δ1.05 52.12, δ1.10 79.51, δ1.25 99.84, rank 1.2.
• Baselines compared: Depth4ToM (ICCV23), Depth-Anything-v2 / DAv2 (NeurIPS24), Marigold-E2E-FT (WACV25), MoGe (CVPR25), VGGT (CVPR25), and DepthCrafter (CVPR25). For instance, on ClearPose DepthCrafter reports δ1.05 31.92 and DAv2 reports 32.21, versus DKT's 38.17.
• Normal estimation: DKT-Normal-14B reports mean error 26.03, median 18.59, and thresholds 11.25° 30.06, 22.5° 59.63, 30° 70.98 on ClearPose, compared with NormalCrafter (27.08, 20.29, 26.10, 55.37, 68.81) and Marigold-E2E-FT (27.08, 19.40, 29.78, 57.22, 69.30).
• Ablation on training strategy: Naive fine-tuning of the 1.3B model gives REL 11.86, RMSE 26.54, δ1.05 30.48, δ1.10 54.03, δ1.25 88.30; LoRA fine-tuning of the same 1.3B model improves to REL 11.17, RMSE 17.45, δ1.05 33.16, δ1.10 58.02, δ1.25 90.65; scaling to 14B with LoRA gives REL 9.72, RMSE 14.58, δ1.05 38.17, δ1.10 65.50, δ1.25 93.04.
• Inference speed: At 832×480, DKT-1.3B runs at 167.48 ms per frame total (encoding 46.53, denoising 52.88, decoding 68.07), versus DAv2 at 277.75 ms and DepthCrafter at 565.55 ms. DKT-14B takes 411.71 ms total (encoding 46.53, denoising 297.11, decoding 68.07). Peak GPU memory for DKT-1.3B is 11.19 GB. The abstract phrases the 1.3B speed as 0.17 s/frame.
• Inference steps: Increasing the number of inference steps does not yield significant performance gains; fewer steps cause inaccurate predictions and more steps lose important details, so 5 steps is set as the default.
• Robotic grasping success rates: RAW depth 0.47 (translucent), 0.18 (reflective), 0.56 (diffusive), mean 0.384; DAv2 0.60, 0.27, 0.56, mean 0.46; DepthCrafter 0.67, 0.23, 0.625, mean 0.48; DKT-1.3B 0.80, 0.59, 0.81, mean 0.73.
• Temporal consistency: The authors present temporal profiles (depth values sliced along the time axis) to show DKT's predictions are more stable across frames than the baselines.

Methodology in Plain English

The authors start from the idea that a video diffusion model trained on internet-scale video already "knows" how light behaves around glass and mirrors. Rather than training a network to regress depth directly, they treat depth estimation as a video-to-video translation problem.

Data. They assemble a 3D asset bank from two sources: category-rich static assets pulled from BlenderKit (5,574 collected, filtered to 574 high-quality transparent/reflective assets using Qwen2.5-VL-7B aesthetic scoring) and shape-rich procedural parametric assets generated through a custom pipeline. Objects are given glass, plastic, metal, or glazed-ceramic materials and dropped into scenes using Blender's physics engine so they settle into physically plausible arrangements. Cameras follow circular trajectories around the objects with sinusoidal perturbations, and rendering uses Blender's Cycles ray tracer with NVIDIA OptiX denoising to produce RGB, depth, and normal videos. The result is TransPhy3D: 11,000 scenes, 120 frames each, 1,320,000 frames total.

Model. They build on the WAN video diffusion model, which has a VAE, a diffusion transformer with DiT blocks, and a text encoder, and which uses a flow-matching objective (interpolating between noise and clean latents, and predicting the velocity between them). During training they encode both the RGB video and the depth video into latents, normalize both to [-1, 1] to match the VAE's training space, convert depth to disparity, add noise to the depth latent, and concatenate the noisy depth latent with the clean RGB latent along the channel dimension before feeding it into the DiT. The loss is the MSE between the predicted velocity and the ground-truth velocity for the depth stream. Only a small set of LoRA parameters in the DiT are trained; everything else stays frozen, which prevents catastrophic forgetting of the pretrained priors.

Co-training. To reduce rendering costs, they mix frame-wise synthetic image datasets (HISS, DREDS, ClearGrasp) with the video dataset. Frame counts are sampled as F = 4N + 1 with N drawn uniformly from 0 to 5; if F equals 1, a batch of paired RGB/depth data is drawn from both image and video datasets (the video containing one frame), otherwise sampling comes only from video datasets.

Training and inference details. Training uses a learning rate of 1e-5 with AdamW, batch size 8, resolution 832×480, 70K iterations, and 8 Nvidia H100 GPUs for two days. Inference uses 5 denoising steps and, following DepthCrafter's inference strategy, splits arbitrary-length input into overlapping segments stitched together with complementary weights on the overlap. Evaluation aligns predictions to ground truth with a global scale and shift and reports δ1.05, δ1.10, δ1.25, REL (as percentages) and RMSE (in centimeters).

Grasping deployment. An RGB image from a RealSense D435 is processed by a depth model, rescaled to metric depth with AprilTag, fed with the RGB into AnyGrasp to produce a 7-DoF grasp pose, and executed by a PiPER Arm after CuRobo trajectory planning on a Cobot Magic system.

Why This Matters

• Impact on research: The work argues that generative video priors can be repurposed into perception heads cheaply and without real-world labels. It shows a purely synthetic training pipeline reaching zero-shot state of the art on real transparent-object benchmarks, which challenges the assumption that transparent-object depth requires large real RGB-D collections. It also provides a video dataset where only frame-wise image datasets for transparency previously existed.
• Robotics and manipulation: Reliable depth on transparent and reflective items supports bin picking, grasping, and placement where glassware, bottles, or wrapped goods are common.
• Warehousing and logistics: Many packed goods use shrink wrap, plastic film, or glossy packaging that confuses depth sensors.
• Household and service robots: Robots handling dishes, cups, and containers on reflective or translucent tabletops need stable depth over time, not just per-frame accuracy.
• Augmented and virtual reality, plus 3D reconstruction: Transparent-object depth and normal estimation feed scene reconstruction and rendering of glass surfaces.
• Industry relevance: The 1.3B variant runs at 167.48 ms per frame with 11.19 GB peak GPU memory, which the authors describe as acceptable for most robot computational platforms, and it is integrated into an off-the-shelf grasping stack (AnyGrasp, CuRobo, PiPER Arm, RealSense D435).

Future Directions

• Closing the temporal-consistency gap in circular camera paths: The paper attributes the particularly large TransPhy3D-Test margin to the circular rendering trajectories, which raise the bar for inter-frame consistency under global alignment; improving consistency under such motion is a natural next step.
• Extending beyond depth and normals: The same fashion of repurposing a video diffusion model could be applied to other physical properties of transparent scenes, such as segmentation, material classification, or reflectance.
• Reducing rendering cost: The paper notes that the co-training design exists partly to alleviate the rendering burden, implying further work on cheaper or more diverse synthetic generation, and on the domain gap between the parametric and static asset sources.
• Scaling and efficiency trade-offs: The authors show 5 inference steps is enough and that the 1.3B model is fastest, but with a 14B model still required for peak accuracy on ClearPose; narrowing that accuracy-efficiency gap is an open question.
• Broadening real-world validation: Grasping results come from a single dual-arm setup with a fixed-view RealSense D435 across three tabletop surface types; the paper does not report the number of trials, objects, or statistical significance, so wider and more rigorous real-robot evaluation would strengthen the claims.

Target Audience

This paper is most useful for computer vision researchers working on depth estimation, generative models, and physically based rendering, especially those interested in transparent materials; for robotics engineers building perception and grasping pipelines for challenging surfaces; and for graduate students and practitioners already comfortable with diffusion models, latent VAEs, and flow matching. Readers looking for a beginner-level introduction to depth estimation will find the model internals and training recipe demanding, though the dataset construction and grasping results are accessible on their own.

Authors’ abstract

Transparent objects remain notoriously hard for perception systems: refraction, reflection and transmission break the assumptions behind stereo, ToF and purely discriminative monocular depth, causing holes and temporally unstable estimates. Our key observation is that modern video diffusion models already synthesize convincing transparent phenomena, suggesting they have internalized the optical rules. We build TransPhy3D, a synthetic video corpus of transparent/reflective scenes: 11k sequences rendered with Blender/Cycles. Scenes are assembled from a curated bank of category-rich static assets and shape-rich procedural assets paired with glass/plastic/metal materials. We render RGB + depth + normals with physically based ray tracing and OptiX denoising. Starting from a large video diffusion model, we learn a video-to-video translator for depth (and normals) via lightweight LoRA adapters. During training we concatenate RGB and (noisy) depth latents in the DiT backbone and co-train on TransPhy3D and existing frame-wise synthetic datasets, yielding temporally consistent predictions for arbitrary-length input videos. The resulting model, DKT, achieves zero-shot SOTA on real and synthetic video benchmarks involving transparency: ClearPose, DREDS (CatKnown/CatNovel), and TransPhy3D-Test. It improves accuracy and temporal consistency over strong image/video baselines, and a normal variant sets the best video normal estimation results on ClearPose. A compact 1.3B version runs at ~0.17 s/frame. Integrated into a grasping stack, DKT's depth boosts success rates across translucent, reflective and diffuse surfaces, outperforming prior estimators. Together, these results support a broader claim: "Diffusion knows transparency." Generative video priors can be repurposed, efficiently and label-free, into robust, temporally coherent perception for challenging real-world manipulation.