Integrating Multi-view Multi-light Surface Reconstruction into Cultural Heritage Workflows

AI summary

Overview

Research area: Computer Vision / 3D Surface Reconstruction, applied to Cultural Heritage digitisation.

Technical level: Intermediate — the paper is an integration and tooling contribution, so the core idea is accessible, but it assumes familiarity with photogrammetry, photometric stereo, and neural surface reconstruction concepts.

Scope: The paper describes how existing multi-view photometric stereo (MVPS) methods were assembled into a usable, open-source pipeline inside the photogrammetry software Meshroom, and reports qualitative results on four real archaeological objects.

What This Paper Is About

Cultural heritage documentation has two complementary photographic techniques: photogrammetry, which recovers overall 3D shape from many viewpoints but smooths out fine relief, and reflectance transformation imaging (RTI), which captures illumination-varying images from a single fixed viewpoint and reveals engravings and tool marks but produces no 3D geometry. Combining them has long been recognised as valuable but difficult, because it requires careful registration between two separate modalities.

The authors' goal is not to invent a new reconstruction algorithm, but to break down the integration barrier: they take state-of-the-art computer vision methods for multi-view, multi-light surface reconstruction — which currently exist only as research prototypes and unpackaged code — and expose them as ready-to-use nodes inside Meshroom, an open-source photogrammetry framework that heritage practitioners already use.

Key Contributions

1. A complete photometric stereo ecosystem exposed as Meshroom nodes, spanning three lighting regimes: classical calibrated photometric stereo with automatic calibration-sphere detection and light-direction estimation; self-calibrated near-light photometric stereo that recovers light position, intensity and anisotropy on the fly from a coarse geometric proxy; and three universal neural methods (UniMS-PS, SDM-UniPS, LINO-UniPS) that need no light calibration at all.

2. Automatic object masking and a multi-view normal-and-reflectance integration node, which together close the chain from multi-light images to a single textured, high-fidelity mesh. Masking uses SAM 3-based semantic segmentation; integration is provided both as the reference RNb-NeuS2 and as OpenRNb, a permissively licensed (MPL-2.0) from-scratch reimplementation.

3. An open-source release of all nodes and ready-to-use MVPS pipelines, with sample code that encourages other researchers to plug in further photometric stereo and integration methods. The nodes are implemented in Python as Meshroom plugins.

4. Qualitative validation on four real archaeological objects — a Cypriot warrior head, an engraved bone from the Gourdan cave, a lithic biface, and a ceramic head ("woman with a calathos") — each exercising a different photometric stereo method.

Main Findings

• Fine relief is recovered where photogrammetry fails. On the Cypriot warrior head, the proposed pipeline separates the individual strands of the braided beard that dense multi-view stereo fuses and smooths, at a comparable image budget (156 multi-light images versus 121 photogrammetric views). The reference photograph confirms the relief is genuine rather than neural hallucination.

• Universal photometric stereo works without any light calibration. The engraved bone from the Gourdan cave was reconstructed using UniMS-PS with no photogrammetric prior and no light calibration. The faint incised engravings, barely visible in RealityScan, Metashape, or COLMAP reconstructions from the same 23 views, become legible in the normal map and preserved in 3D.

• Results approach structured-light scanning. On the lithic biface, the pipeline recovers the sharp ridges of shallow intersecting flake scars at a level of detail very close to an Artec Micro 2 structured-light scan, while using only 12 viewpoints under 12 illuminations each compared to 162 photogrammetric views.

• Micro-relief can exceed scan quality at fine scales. On the ceramic head, rendered under identical shading, the pipeline brings out a fine surface granularity that neither photogrammetry nor the Artec Spider structured-light scan (50 µm point accuracy) renders at that scale. Relighting under a grazing light matched to the reference photograph confirms the recovered relief casts shadows consistent with the real object.

• Texture is shading-free. Because photometric stereo separates reflectance from shading, the colour map applied to the mesh is intrinsic surface reflectance rather than a bake of the acquisition lighting — unlike standard photogrammetric texturing, which darkens surfaces that were lit from behind.

• Practical runtime. The full chain — raw development, structure-from-motion, photometric stereo, and integration — runs in a few hours per object on a single GPU at 45.7 MP image resolution, comparable to a conventional photogrammetric reconstruction.

• OpenRNb nearly matches RNb-NeuS2. On the controlled DiLiGenT-MV benchmark, the two integration backends recover essentially the same sub-millimetre surfaces, but OpenRNb's mean Chamfer distance is higher: 0.24 mm versus 0.18 mm on UniMS-PS normals, and 0.30 mm versus 0.22 mm on SDM-UniPS normals.

Methodology in Plain English

The authors treat the problem as a software engineering and orchestration challenge rather than an algorithmic one.

They build on Meshroom's nodal architecture, where a reconstruction pipeline is a directed acyclic graph in which each node wraps a tool, intermediates are cached, downstream nodes are automatically invalidated when inputs change, and standard file formats pass between stages. They graft a multi-light reconstruction chain onto the existing structure-from-motion stage.

The data flow begins with multi-view, multi-light acquisition: several viewpoints, each photographed under several light directions, using an RTI dome, an open-hardware camera-and-turntable rig, or a hand-held flash in the field. SAM 3-based segmentation produces per-view object masks from a lightweight text prompt, which propagate through the whole chain and also fix a common structure-from-motion failure mode on turntables, where the background and object move differently. Fiducial markers (CCTag or AprilTag) improve pose robustness and set metric scale.

Structure-from-motion then recovers camera parameters and a sparse point cloud, optionally with a coarse dense surface mesh. That coarse proxy is fed to the self-calibrated near-light photometric stereo method for its light estimation. Next, per-view normal maps (and reflectance maps, when the chosen method supports it) are computed by whichever photometric stereo node the practitioner selects.

Finally, a multi-view normal-and-reflectance integration node represents the scene as a signed distance function whose zero level set is the surface, and optimises it so the SDF gradient agrees with the per-view normal maps. Reflectance is folded into the same objective by re-parametrising normals and reflectance as a simulated radiance field. The mesh is extracted by marching cubes. The key architectural choice is that all photometric stereo variants — calibrated, self-calibrated, and universal — feed into this same integration node, so they are directly interchangeable and comparable.

Evaluation is qualitative: because unique heritage objects have no metric ground truth, the authors compare against photogrammetric reconstructions under matched viewpoints and image budgets, and against structured-light scans as an alternative digitisation modality rather than as a reference target. They render reconstructed geometry under controlled shading rather than colour, since fine relief is the object of interest.

Why This Matters

Impact on research. The paper addresses a persistent gap between computer vision research output and practical adoption in an adjacent field. By exposing interchangeable methods behind a common interface, it creates a natural comparison platform for photometric stereo and multi-view normal integration methods on real, difficult, non-Lambertian, weakly textured objects — data that controlled benchmarks like DiLiGenT-MV do not fully represent. It also sidesteps the registration problem that dogged earlier attempts to fuse RTI and photogrammetry, by feeding multi-light images directly into photometric stereo instead of registering two separately produced representations.

Real-world applications:

• Archaeological documentation and analysis — reading faint engravings, inscriptions, and tool marks on lithics, bone, and ceramics, where surface relief carries as much information as overall form.
• Museum conservation and condition monitoring — recording micro-relief and surface granularity at a level of detail that can track deterioration or inform treatment decisions, with illumination-neutral texture that does not lock the object into acquisition lighting.
• Digital replicas and reproduction — producing high-fidelity meshes for study, exhibition, or fabrication, where relief fidelity matters more than texture colour accuracy.
• Heritage at scale — allowing institutions already using RTI domes to reuse their existing multi-light captures for 3D reconstruction, rather than acquiring a new modality.

Industry relevance. The pipeline reuses interfaces, file formats, and execution models that heritage practitioners already know from photogrammetry, which lowers adoption friction considerably. Support for camera RAW, high-dynamic-range OpenEXR, and dispatch across a render farm via a pluggable submitter mechanism makes it viable for institutional workflows. The dual integration backend matters commercially: RNb-NeuS2 is faster and more accurate but inherits a non-commercial NVIDIA research licence, so the MPL-2.0 OpenRNb reimplementation gives commercial users a legally usable path.

Future Directions

• Broaden the method ecosystem. Wrap additional photometric stereo and multi-view normal integration methods behind the same interface, including emerging approaches that reconstruct surfaces directly from multi-light images without an explicit photometric-stereo-then-fusion split. The goal is to let practitioners compare methods on their own data.
• Propagate per-pixel reliability. Push an estimated confidence for each photometric measurement through the pipeline and expose it as a confidence map in the mesh texture, so downstream analysis can flag regions to interpret with caution.
• Build an open evaluation platform. Because every intermediate output is exposed in open formats, the pipeline could support shared, frequency-resolved quality metrics — a natural answer to the current lack of metric ground truth on heritage objects.
• Close the OpenRNb accuracy gap. The permissively licensed reimplementation is close to but not on par with RNb-NeuS2; narrowing that Chamfer-distance difference needs further investigation.
• Run a user study. A structured study with conservators and archaeologists would help refine presets, documentation, and acquisition guidelines for different heritage settings, and would surface deployment constraints such as the current requirement for NVIDIA/CUDA GPUs.

Target Audience

This paper is most valuable to heritage practitioners and technical staff — archaeologists, conservators, museum digitisation specialists, and heritage technicians — who already use photogrammetry software and want to access multi-light reconstruction without a computer vision background. It is equally relevant to applied computer vision researchers and engineers working on photometric stereo or neural surface reconstruction who want to see their methods deployed in a real workflow, and to developers building open-source reconstruction tools who can reuse the node implementations and orchestration logic. Readers evaluating commercial versus academic licensing trade-offs for integration backends will find the RNb-NeuS2 versus OpenRNb comparison directly useful.

Authors’ abstract

Cultural heritage documentation increasingly relies on image-based 3D surface reconstruction, with photogrammetry software making such workflows accessible to archaeologists, conservators, and heritage technicians. These tools have been successful for conventional multi-view acquisition, but they do not routinely exploit richer multi-view, multi-light data, despite its potential for improving fine-scale surface reconstruction. This limitation is particularly relevant in heritage contexts, where controlled-light acquisition devices such as RTI domes are already used to capture illumination-varying image sets. The challenge is therefore to connect these existing acquisition practices with recent computer vision methods in a form that can be used within operational heritage workflows. In this work, we address this need by integrating state-of-the-art components from computer vision for multi-view, multi-light surface reconstruction into Meshroom, an open-source photogrammetry framework. Rather than proposing a new reconstruction algorithm, our contribution is to assemble and expose existing advanced methods, namely a complete photometric stereo ecosystem (calibrated, self-calibrated and universal), automatic object masking, and multi-view normal-and-reflectance integration, within a usable heritage-oriented workflow. The proposed system thus provides an intermediate software layer between computer vision research code and practical cultural heritage applications, making recent techniques easier to use and evaluate.