Soft Shadow Diffusion (SSD): Physics-inspired Learning for 3D Computational Periscopy

AI summary

Overview

Research area: Non-line-of-sight (NLOS) imaging, passive computational periscopy, computational photography, inverse problems, diffusion models, and 3D point cloud reconstruction.
Technical level: Advanced.
Scope: This paper proposes a separable nonlinear least squares formulation, a gradient-based solver, and a physics-inspired diffusion network called Soft Shadow Diffusion (SSD) for reconstructing 3D hidden occluders and 2D non-occluding scenes from a single passive soft-shadow photograph.

What This Paper Is About

Conventional imaging needs a line of sight, but NLOS imaging reconstructs hidden scenes from indirect measurements. Passive NLOS methods can use an ordinary photograph of the subtle soft shadow cast onto a visible wall by a hidden scene, but they have been limited to 1D or low-resolution 2D color imaging or to localizing a hidden object whose shape is approximately known. This paper generalizes that class of methods to 3D reconstruction of a hidden scene from an ordinary NLOS photograph, including joint reconstruction of 3D light-occluding objects and 2D non-occluding objects.

Key Contributions

1. A reformulation of computational periscopy as a separable nonlinear least squares (SNLLS) inverse problem that decomposes the hidden scene into light-occluding and non-light-occluding components, enabling 3D reconstructions from a single penumbra photograph.
2. A gradient-based optimization method that solves the inverse problem using a uniform voxel representation of the hidden occluding volume.
3. Soft Shadow Diffusion (SSD), a generative model based on denoising probabilistic diffusion for reconstructing 3D shapes as high-resolution 3D point clouds from a single photograph of their soft shadows, plus a physics-inspired neural network inversion pipeline.
4. The first experimental demonstration of joint reconstructions of 3D occluding and 2D non-occluding objects in a real experimental scene.

Main Findings

• Joint 3D and 2D reconstruction: The methods reconstruct both a 3D image of hidden light-occluding objects and a 2D image of hidden non-occluding objects from passive measurements.
• Real experimental validation: In real experiments, the non-occluding portion of the hidden scene was shown on a monitor at a distance of 1.08 meters to the visible wall, with 3D occluders placed between them. Measurements were obtained by photographing the visible wall with a 4-megapixel camera.
• SSD generalization: SSD was trained in simulation but generalizes well to unseen classes in simulation and real-world NLOS scenes. The training dataset was roughly 260,000 simulated instances based on thousands of 3D models from ShapeNet.
• Robustness to background illumination: The approach degrades slowly for reasonable signal-to-background ratios (SBRs) in simulated experiments. 3D reconstructions remain accurate even in SBRs of around 10 dB, while 2D reconstructions degrade gradually for SBRs below 20 dB. Similar trends are observed for real data reconstructions.
• Robustness to measurement noise: SSD shows similar robustness to measurement noise, with 3D reconstruction quality relatively unaffected even at signal-to-noise ratios as low as 10 dB.
• Metrics used: The paper reports mean squared error (MSE) for 2D reconstructions and Chamfer Distance (CD) for 3D occluding object reconstructions.
• Gradient-based method limitations: The gradient-based method does not scale well for more complex occluder shapes requiring finer discretizations. It uses a 10×5×10 discretization of the occluding region. A sparse matrix approach mitigated limitations in simulations, but it was less effective in real-world scenarios.
• No direct comparisons: The paper states that direct and fair comparisons are not possible because prior methods focus on different acquisition scenarios, such as access to video frames, pre-calibrated occluder visibility functions, known occluder shapes, or fully known occluders. The authors note their 2D reconstructions will, in expectation, be no better than existing methods because they are solving a harder problem.

Methodology in Plain English

The researchers model how light from a hidden scene bounces toward a visible wall. Hidden objects that block light create a soft shadow, or penumbra, on that wall, while other hidden objects reflect or emit light. They divide the hidden scene into two parts: 3D light-occluding structures and a 2D non-occluding plane of light reflectors/emitters. They define a computational field of view that includes the 3D volume where occluders can cast a shadow within the camera’s view, plus the 2D plane for non-occluding objects.

They use a pinspeck, or anti-pinhole, model to represent how small occluding elements block light. In the gradient-based approach, they divide the occluding volume into uniform voxels, relax the binary presence/absence of each voxel with a sigmoid function, and alternate between estimating the 2D non-occluding image, the background, the occluder voxels, and the regularization parameter. This is implemented in PyTorch, with a sparse matrix representation for the visibility contributions.

In the SSD approach, a conditional diffusion model is trained to generate 3D point clouds from an encoded soft-shadow photograph. The pipeline has three stages: first, generate a 3D shape from the diffusion model; second, localize the generated shape by finding a 3D translation; third, convert the translated point cloud into a mesh and solve a TV-regularized problem for the 2D non-occluding image. The diffusion model is conditioned on a latent representation of the soft-shadow photograph using a CrossFormer image encoder. The model itself uses a UNet-based architecture with 1D convolutions. To create meshes, they train a signed distance field (SDF) neural network with an autoencoder architecture and apply marching cubes.

Training data comes from ShapeNet 3D models rendered in Blender from 3 random camera angles. Each 3D model is normalized to a bounding unit cube, rendered as RGB images, and paired with point clouds. Dense point clouds of 50,000 points are sampled along each object’s surface and then resampled to 2,048 evenly spaced points. The rendering equation is applied to all possible pairs of rendered images and 3D models, and pairs are randomly selected to simulate diverse training examples.

Why This Matters

This work advances passive NLOS imaging from 1D or low-resolution 2D reconstructions, or localization with approximately known occluder shapes, to 3D reconstructions of unknown hidden occluders from a single photograph. It provides a new way to recover hidden occluder structure, which can supply prior knowledge for other passive NLOS methods. It also demonstrates that a model trained entirely in simulation can generalize to real-world NLOS scenes and remain robust to noise and ambient illumination, while using simpler passive hardware than active NLOS methods that rely on ultrafast pulsed-laser illumination and time-resolved single-photon detection.

Real-world applications listed in the paper include:

• Collision avoidance systems in autonomous navigation.
• Detection of hidden adversaries in military operations.
• Danger assessment in search-and-rescue operations.
• Arterial blockage assessment in biomedical imaging.

Industry relevance: the method points toward cheaper, simpler passive NLOS sensing than active systems, which can be limited by equipment complexity and cost. It could benefit robotics, autonomous vehicles, defense, search-and-rescue, and biomedical imaging, especially where direct line of sight is impractical, dangerous, or impossible.

Future Directions

• Scale the gradient-based method to complex shapes and higher discretizations, since it is currently limited by computational demand. The sparse matrix approach helped in simulations but was less effective in real-world scenarios.
• Improve localization and handle unknown occluder size. SSD predicts a view-consistent 3D point cloud without information about the true location of points in the NLOS configuration. The current pipeline partitions occluder space into multiple 3D bounding boxes and uses a grid search, but the occluding object’s size is typically unknown in real-world uncontrolled scenarios.
• Further test and improve robustness to noise, ambient illumination, and unseen shape classes in more real experimental settings.
• Integrate the recovered 3D occluder structure with existing passive NLOS methods that assume known or calibrated occluders, to provide the prior knowledge those methods need.

Target Audience

This paper is most useful for advanced researchers and practitioners in computational imaging, NLOS imaging, computer vision, computational photography, inverse problems, machine learning for physics-based reconstruction, and diffusion models for 3D point cloud generation. It is also relevant to engineers and scientists working on autonomous navigation, defense, search-and-rescue, and biomedical imaging who are interested in passive sensing beyond the line of sight.

Authors’ abstract

Conventional imaging requires a line of sight to create accurate visual representations of a scene. In certain circumstances, however, obtaining a suitable line of sight may be impractical, dangerous, or even impossible. Non-line-of-sight (NLOS) imaging addresses this challenge by reconstructing the scene from indirect measurements. Recently, passive NLOS methods that use an ordinary photograph of the subtle shadow cast onto a visible wall by the hidden scene have gained interest. These methods are currently limited to 1D or low-resolution 2D color imaging or to localizing a hidden object whose shape is approximately known. Here, we generalize this class of methods and demonstrate a 3D reconstruction of a hidden scene from an ordinary NLOS photograph. To achieve this, we propose a novel reformulation of the light transport model that conveniently decomposes the hidden scene into \textit{light-occluding} and \textit{non-light-occluding} components to yield a separable non-linear least squares (SNLLS) inverse problem. We develop two solutions: A gradient-based optimization method and a physics-inspired neural network approach, which we call Soft Shadow diffusion (SSD). Despite the challenging ill-conditioned inverse problem encountered here, our approaches are effective on numerous 3D scenes in real experimental scenarios. Moreover, SSD is trained in simulation but generalizes well to unseen classes in simulation and real-world NLOS scenes. SSD also shows surprising robustness to noise and ambient illumination.