Thermal Polarimetric Multi-view Stereo

AI summary

Overview

Research area: Computer vision / physics-based vision — 3D shape reconstruction using long-wave infrared (LWIR, "thermal") polarimetric imaging combined with neural implicit surface representation and multi-view geometry.

Technical level: Advanced. The paper combines Mueller-matrix polarization calculus, Fresnel/Kirchhoff emission physics, differentiable implicit surface rendering, and a multi-view tangent-space constraint. Readers need comfort with polarization optics and neural implicit reconstruction to follow Section 3.

Scope: The paper formulates a general theory of polarization observation across visible and LWIR spectra, argues that LWIR emission polarization avoids the ambiguities that plague visible-light shape-from-polarization, and proposes a thermal polarimetric multi-view stereo method that reconstructs detailed 3D shapes from multi-view thermal AoLP images alone.

What This Paper Is About

Most 3D reconstruction methods — multi-view stereo, structured light, photometric stereo, visible-light shape from polarization — depend on external illumination and on assumptions about a surface's optical properties, which limits them to textured or opaque or specially lit objects. Thermal imaging sidesteps illumination entirely because any warm object emits its own LWIR light, and most materials are opaque in the LWIR spectrum, but existing thermal reconstruction methods tend to be either low-accuracy, model-based, or require slow active heating and cooling. The goal of this paper is to recover fine 3D shape in a steady state (no heating or cooling cycle) using the polarization state of thermally emitted LWIR light, which the authors show depends only on the surface normal for the angle-of-linear-polarization (AoLP) cue.

Key Contributions

1. A unified theoretical framework for LWIR polarization observation. The authors write the observed Stokes vector as a sum of specular, diffuse, transmission, and emission components, then show that in the LWIR spectrum transmission can be ignored (most materials are opaque) and, when no significant hot sources surround the target, the observation reduces to the emission component alone — analytically representable without the specular–diffuse ambiguity of visible polarization.

2. The proposal of thermal polarimetric multi-view stereo. Multi-view thermal AoLP images are used to estimate a signed distance function (SDF) represented implicitly by an MLP and optimized via differentiable rendering, adapting the tangent space constraint (TSC) from multi-view azimuth stereo (MVAS) to thermal polarization.

3. A demonstration that AoLP in LWIR is material-independent. In their observation figure, visible DoLP/AoLP images of spheres are contaminated by environment reflections, background texture, and subsurface scattering depending on material, whereas LWIR AoLP images remain consistent across materials.

4. An experimental comparison against state-of-the-art baselines on real heterogeneous objects, showing their method outperforms visible-light MVAS and thermal IDR on Chamfer distance and mean angular error for all captured objects.

Main Findings

• Visible polarization is unstable; thermal AoLP is consistent. The paper's Figure 2 shows spheres of black stone, glass, and translucent plastic: visible DoLP is heavily affected by environment reflections for black stone, background textures for glass, and surface texture and subsurface scattering for translucent plastic, whereas LWIR AoLP stays consistent across materials.

• LWIR polarization has no specular–diffuse ambiguity. Because transmission is negligible and emission dominates in typical environments, the observation reduces to a single emission term, so AoLP equals the azimuth angle of the surface normal directly.

• AoLP is the robust cue, DoLP is material-dependent. AoLP is given by half the arctangent of the ratio of the Stokes parameters and equals the projected azimuth angle; DoLP depends on the refractive index as well as the zenith angle, so the paper uses only AoLP images for reconstruction.

• Reconstructed detail that baselines miss. In the owl (ceramic with clear coating), eyes and beak are missing from both visible MVAS and thermal IDR but recovered by the proposed method; the relief pattern on a glass container is clearly visible; and for a black tumbler with a white lid, the lid's surface — unreconstructible and wavy under visible polarization — is recovered because of a unique AoLP in the LWIR.

• Baseline failure modes. Visible MVAS tends to produce wavy artifacts because AoLP is noisy, and thermal IDR tends to produce swollen shapes because it does not use surface normal information.

• Quantitative outcomes. The proposed method outperforms the other methods in both Chamfer distance and mean angular error for all 7 objects. The truncated content does not report the specific numeric values of these metrics.

• Notable artifact. The authors report artifacts in thin parts, particularly the neck of the bottle, which they attribute to the low spatial resolution of the thermal camera, and suggest a higher-resolution camera or more views could reduce it.

Methodology in Plain English

Step 1 — Model what the camera sees. The authors write the polarization state of light as a four-element Stokes vector and describe how reflection, diffuse scattering, transmission, and thermal emission each transform it via a Mueller matrix. In the LWIR spectrum, they argue, transmission drops out because most materials are opaque, and emission dominates over reflection when nothing very hot surrounds the object. Emission polarization follows Kirchhoff's law: emissivity for each polarization equals one minus the corresponding Fresnel reflectance, so it can be written analytically from the refractive index and the angle to the surface.

Step 2 — Extract surface normals from polarization. The angle-of-linear-polarization (AoLP) of the emitted light equals the azimuth of the surface normal. This relationship is independent of material properties, and unlike the visible-light case it is not confounded by mixing specular and diffuse reflection.

Step 3 — Represent shape implicitly and optimize it. The object surface is the zero level set of a signed distance function represented by an 8-layer MLP with softplus activations, positional encoding of dimension 10, and a skip connection from the input 3D position and encoded features to the 4th layer. The MLP is trained within an implicit differentiable renderer.

Step 4 — Enforce consistency across views. For each pixel of each view, a "projected tangent vector" is computed from the camera pose and the observed AoLP; the true surface normal must be perpendicular to it. Summing the squared violation of that perpendicularity over all visible views gives the tangent space consistency loss. Two additional losses are used: a silhouette loss that matches the rendered outline against the input mask via cross entropy, and an Eikonal regularization that pushes the SDF gradient magnitude toward 1. The total loss is L_TSC + λ1·L_silhouette + λ2·L_Eikonal, with λ1 = 50 and λ2 = 0.1.

Step 5 — Train and capture data. Optimization uses Adam at a learning rate of 0.001 for 50 epochs with a batch size of 4096 pixels, halving the silhouette loss weight every 10 epochs. Training takes about 2 hours on a single RTX A6000 GPU.

Capture system. A thermal camera (FLIR Boson 320) with a 15 mm lens has a wire-grid polarizer (Thorlabs WP25M-IRC) mounted on a motorized rotation mount (Thorlabs K10CR2) between the sensor and the imaging lens to mitigate narcissus reflections. Four polarization images are captured at 0, 45, 90, and 135 degrees. A monochrome visible-light polarization camera (FLIR Blackfly S BFS-U3-51S5P) sits adjacent and is used for the visible baseline and for camera pose estimation.

Data and evaluation. Seven objects of varied shapes and materials, including transparent and low-reflective ones, were placed on a rotating table and captured from roughly 20 to 30 views, with the objects kept warm during measurement. Aruco markers on the rotation stage give the visible-camera poses, and thermal-camera poses are obtained by transforming those using a stereo calibration estimated with Zhang's method against a calibration board of a white-painted aluminum pegboard in front of a black surface heater. Thermal masks come from thresholding pixel values; visible masks come from Segment Anything Model 2. Ground-truth shapes come from a structured light scanner (EinScan-SP V2), with objects painted white to be opaque for measurement. Evaluation uses Chamfer distance for geometry and mean angular error for normals, after ICP alignment of the estimated and ground-truth meshes.

Baselines. MVAS (visible-light polarization, with the TSC loss modified to account for the ±π/2 normal ambiguity present in visible polarization) and IDR (a neural implicit surface reconstruction method) with thermal image input.

Why This Matters

Impact on research. The paper extends shape-from-polarization into the LWIR spectrum and provides a theoretical argument for why the thermal case is cleaner than the visible case: no transmission component, emission-dominated observation, and an AoLP cue that is independent of material and illumination. It also connects the polarization literature with the neural implicit surface reconstruction and thermal-NeRF literature, positioning the proposed method in a comparison table as illumination-independent, material-independent, accurate, and requiring only easy steady-state measurement.

Real-world applications (as motivated by the paper's framing):

• Reconstructing transparent and translucent objects — the paper highlights a glass container with and without interiors, a transparent vase, and a bottle, materials that are typically difficult for visible-light methods but opaque in the LWIR spectrum.
• Low-reflective and dark objects — a black tumbler with a white lid is reconstructed where visible AoLP on the lid is too noisy to use.
• Stealth or unilluminated observation — the related-work discussion notes thermal imaging is effective for stealth observation because thermal radiation does not rely on external illumination.
• Inspection or digitization of heterogeneous objects — scenes mixing multiple materials, which the paper reports its method handles.

Industry relevance. Potential users include 3D scanning and digitization for objects that defeat visible-light scanners, industrial inspection, and robotics or autonomous systems that need geometry in darkness or without controlled lighting. The reliance on an off-the-shelf thermal camera and a rotating polarizer means the approach is demonstrated with commodity components rather than a bespoke imaging rig, though the authors note the capturing system takes time to rotate the polarization filter.

Future Directions

1. Remove the emission-dominance assumption. The method assumes the emission component is sufficiently strong relative to reflection. This breaks down when hot objects sit near the target or when the target is cooler than its environment. The authors suggest decomposing emission and reflection components before shape recovery, or integrating additional cues.

2. Handle metallic and rough surfaces. The paper states the method is less effective where the polarization signal becomes unstable or noisy, specifically metallic and rough surfaces.

3. Speed up capture with single-shot LWIR polarimetry. The current rig rotates a polarizer and captures four images, which takes time. The authors suggest a single-shot polarized LWIR camera could accelerate thermal polarization analysis, as has happened in the visible spectrum.

4. Improve spatial resolution. The reported artifacts in thin parts, such as the neck of the bottle, are attributed to the low spatial resolution of the thermal camera; higher-resolution cameras or more camera views are proposed as mitigations.

Target Audience

Researchers and graduate students in computer vision, computational imaging, and physics-based vision who work on 3D reconstruction, shape from polarization, or neural implicit surfaces. It is also relevant to practitioners in 3D scanning and industrial inspection who deal with transparent, translucent, dark, or mixed-material objects that defeat conventional visible-light pipelines, and to readers interested in thermal or LWIR imaging more broadly. The paper is not beginner-friendly: the theory section assumes familiarity with Stokes parameters, Mueller matrices, Fresnel equations, and Kirchhoff's law, and the method section assumes familiarity with signed distance functions, differentiable rendering, and Eikonal regularization.

Authors’ abstract

This paper introduces a novel method for detailed 3D shape reconstruction utilizing thermal polarization cues. Unlike state-of-the-art methods, the proposed approach is independent of illumination and material properties. In this paper, we formulate a general theory of polarization observation and show that long-wave infrared (LWIR) polarimetric imaging is free from the ambiguities that affect visible polarization analyses. Subsequently, we propose a method for recovering detailed 3D shapes using multi-view thermal polarimetric images. Experimental results demonstrate that our approach effectively reconstructs fine details in transparent, translucent, and heterogeneous objects, outperforming existing techniques.