MRD: Using Physically Based Differentiable Rendering to Probe Vision Models for 3D Scene Understanding

AI summary

Overview

Research area: Computer vision model interpretability (explainable AI), combining physically based differentiable rendering (PBDR) from computer graphics with the classical vision-science concept of metamerism.

Technical level: Advanced. The paper assumes familiarity with the rendering equation, Monte Carlo integration, BSDFs (specifically the Principled/Disney BSDF), path tracing, differentiable rendering, and deep network latent representations.

Scope: The paper introduces MRD (metamers rendered differentiably), a proof-of-concept method that optimizes 3D scene parameters through a differentiable physically based renderer so that rendered images match a frozen vision network's latent representation of a target scene, then uses the resulting reconstructions to characterize what those networks are sensitive to and invariant to. The provided content is truncated before the results section, so specific quantitative outcomes (beyond the abstract's qualitative statements) are not reported here.

What This Paper Is About

Deep networks are trained on 2D images but are often assumed to build an implicit understanding of the 3D scene behind them, yet it is hard to explain or inspect that understanding. MRD attacks this by searching for model metamers: physically different 3D scenes that produce the same internal activation in a frozen network. Because the search happens in the space of physical scene parameters rather than pixels, the resulting reconstructions can be interpreted in physical units (shape, material, lighting), letting researchers isolate which physical cause a network does or does not care about.

Key Contributions

The authors state two headline contributions and frame the work explicitly as a proof of concept:

1. A new interpretability method (MRD) that links a neural network's activations to physical environmental properties by optimizing scene parameters through a physically based differentiable renderer, enabling the search for representational invariants (model metamers). The optimization objective is the distance between network latents for a target scene and a reconstructed scene, not the distance between pixels.
2. A proof-of-concept evaluation of contemporary vision models using this method, probing six frozen networks spanning CNNs (ResNet-50, ResNet-50-SIN), perceptual metrics (LPIPS with a VGG backbone, plus the standalone VGG), and vision transformers (CLIP, DINO/DINOv2), in two settings: surface material (BSDF) recovery and shape (geometry) recovery.
3. An operational definition of a model metamer under physically constrained stochastic optimization: because exact latent equality is never reached, a reconstruction is declared a metamer when its latent similarity to the target reaches or exceeds a baseline level obtained from a reconstruction optimized directly in image space against the ground-truth scene (a baseline that does not use the network in its loss), assessed with several complementary metrics rather than a single scalar.
4. A separation of two outcomes in the evaluation: whether MRD finds a model metamer at all (the primary criterion), and, separately, how far the optimization recovers the ground-truth scene parameters (a secondary outcome).

Main Findings

• Material metamers are found reliably: Across multiple networks, MRD reliably finds metamers for material (Principled BSDF parameters plus albedo and, where present, anisotropy textures).
• Material metamers are frequently not the ground-truth material: The reconstructions often do not recover the target's actual physical parameters. The authors interpret this as evidence that the networks' equivalence classes for material are rather broad. They also note that material reconstruction is metameric by definition in their setup, because the target materials are not themselves Principled BSDFs (some, such as Brushed Metal and Aurora, are goniophotometer recordings with no ground-truth parametric Principled BSDF), so there is generally no single parameter setting that reproduces the target exactly.
• Shape reconstruction often fails the metamer criteria: For shape, reconstructions often fail to meet any of the metamer criteria, although clear success cases exist, for example with simple geometries.
• The criterion requires calibration: The authors state that exact latent equality is never reached under stochastic, physically constrained optimization, which is why the metamer threshold is defined relative to an image-space baseline rather than to exact latent match.
• The evaluated hypothesis about shape bias is not resolved in the available content: The authors motivate comparing ResNet-50 and ResNet-50-SIN with the hypothesis that the shape-biased network should reconstruct 3D shape better, but the truncated content does not report the outcome of that comparison.
• Quantitative results are not reported in the provided content: The retrieved text ends within the Methods (Section 2.2), so numbers such as metamer success rates, per-network comparisons, and scene-parameter recovery errors are not available here.

Methodology in Plain English

The pipeline works as follows. Start with a scene whose parameters are known (for example, a dragon model), sample camera positions on a unit sphere, and render a set of views with a differentiable physically based renderer. Then initialize a second scene from a different state — the paper's example is a sphere shape instead of a dragon — and render the same views. Pass both image sets through a frozen neural network to obtain latent representations, and minimize the distance between those latents (mean squared error for ResNet-50, ResNet-50-SIN, CLIP and DINO; the native multi-layer perceptual distance for VGG and LPIPS). Because the renderer is differentiable, the gradient of this latent loss with respect to the scene parameters can be backpropagated through the light transport simulation to update the target scene's parameters, while other parameters (such as lighting) are held constant. This separates physical causes: you can vary shape while holding material and lighting fixed, or vary material while holding geometry fixed.

Concrete implementation details: the work uses Python 3.12, PyTorch, and Mitsuba 3, with a path tracing integrator for material reconstruction — necessary because translucent effects cannot be captured by compute-efficient integrators such as Path Replay Backpropagation — and the direct projective path replay backpropagation integrator of Zhang et al. for shape reconstruction, which handles the visibility discontinuities that arise when object boundaries move. Shape optimization follows the Large Steps method of Nicolet et al., starting from an icosphere proxy mesh with 16000 vertices and moving vertices, with the authors adjusting the regularization strength lambda and step size because they use Mitsuba 3 instead of nvdiffrast and PyTorch. Principled BSDF parameters that are only defined on [0, 1] (roughness, specular transmission, metalness, anisotropy) are clamped to keep the optimization physically valid. Rendered HDR radiance is converted to linear sRGB, tonemapped with Reinhard tonemapping, and gamma-encoded with the sRGB transfer function so the images are in-distribution for the networks; ground-truth renders are denoised with the Optix Denoiser. For shape, camera origins are sampled with a Fibonacci lattice on a unit sphere; for material, the object is viewed from front, back and both sides. An early stopping criterion halts an experiment if the loss does not improve for 50 epochs. All network weights are frozen. Shape experiments ran on a single RTX 4090 with 24 GB of VRAM under Ubuntu 24.04.5 LTS; material experiments used an RTX 6000 with 48 GB of VRAM.

Why This Matters

Impact on research. MRD offers an alternative to pixel-based synthesis methods for explaining network representations. Instead of interpreting noisy pixel images, researchers obtain reconstructions expressed in the physical units of the generating scene, which supports targeted probing of invariance and equivalence classes and makes it possible to ask how physical scene parameters cause changes in model responses. The authors also note that the fully separated physical causes available through physics-based rendering distinguish this from neural inverse-rendering approaches, which are not grounded in light transport.

Real-world applications (as the framework implies, not as deployed in the paper):

• Evaluating and stress-testing perceptual similarity metrics and foundation-model backbones on physically controlled scene variations rather than natural-image datasets.
• Graphics and content-creation pipelines where material and geometry parameter recovery is needed and where understanding which appearance differences a network treats as equivalent informs asset optimization and compression.
• Auditing models used in safety-critical perception (for example, AR/VR, robotics, autonomous systems) to reveal which physical scene attributes a model ignores.
• Bridging to human vision science, where metamerism has been used to identify perceptual invariants.

Industry relevance. The work speaks directly to two industries: computer graphics and rendering (where differentiable renderers such as Mitsuba 3 are already used, and where the appearance model is the Principled BSDF used in production), and machine-learning tooling for model evaluation, where the paper's framing of "model metamer" provides a principled criterion for representational equivalence.

Future Directions

• Making PBDR converge reliably against deep-network objectives. The authors state this is itself a substantial technical challenge and position their work as a proof of concept marking where it succeeds and where it does not; closing this gap is the main prerequisite for broader use.
• Improving shape reconstruction. Shape reconstructions often fail the metamer criteria except for simple geometries, so extending success to complex geometry is an open problem.
• Testing the shape-bias hypothesis. Comparing ImageNet-trained and Stylized ImageNet-trained ResNet-50 was motivated by the hypothesis that the shape-biased network should reconstruct 3D shape better due to more disentangled shape and material representations; the available content does not report whether this held.
• Extending the approach. The authors suggest the method opens possibilities to fine-tune existing models on specific scene properties, to investigate which physical scene attributes models are sensitive to or invariant to, and to advance understanding of both computer and human vision.

Target Audience

Researchers and practitioners in computer vision interpretability and explainable AI; computer graphics researchers working with differentiable and inverse rendering; cognitive and vision scientists interested in metamerism and in comparing machine and human visual representations; and machine-learning engineers who evaluate perceptual metrics or foundation-model backbones and need a physically grounded characterization of what those models encode. Readers need a working knowledge of rendering and neural network latents to follow the derivations, though the high-level argument is accessible to a general technical audience. Code and plots for all experiments are stated to be available at https://doi.org/10.5281/zenodo.21374307, and the paper is licensed CC BY-NC-SA 4.0.

Authors’ abstract

While deep learning methods have achieved impressive success in many vision benchmarks, it remains difficult to understand and explain the representations and decisions of these models. Though vision models are typically trained on 2D inputs, they are often assumed to develop an implicit representation of the underlying 3D scene (for example, showing tolerance to partial occlusion, or the ability to reason about relative depth). Here, we introduce MRD (metamers rendered differentiably), an approach that uses physically based differentiable rendering to probe vision models' implicit understanding of generative 3D scene properties, by finding 3D scene parameters that are physically different but produce the same model activation (i.e. are model metamers). Unlike previous pixel-based methods for evaluating model representations, these reconstruction results are always grounded in physical scene descriptions. This means we can, for example, probe a model's sensitivity to object shape while holding material and lighting constant. As a proof-of-principle, we assess multiple models in their ability to recover scene parameters of geometry (shape) and bidirectional reflectance distribution function (material). The results show high similarity in model activation between target and optimized scenes, with varying visual results. Qualitatively, these reconstructions help investigate the physical scene attributes to which models are sensitive or invariant. MRD holds promise for advancing our understanding of both computer and human vision by enabling analysis of how physical scene parameters drive changes in model responses.