Image Valuation in NeRF-based 3D reconstruction

AI summary

Overview

Research area: Computer Vision — data valuation applied to Neural Radiance Fields (NeRFs) for 3D scene reconstruction from in-the-wild image collections.

Technical level: Intermediate. Readers should be comfortable with NeRF training pipelines (per-image ray sampling, PSNR/L1 validation metrics) and with general concepts from data valuation (Shapley values, influence functions, DVRL, sample importance).

Scope: The paper proposes and empirically evaluates an image contribution scoring scheme, DV_psnr, that measures each training image's individual impact on NeRF reconstruction quality for in-the-wild photo collections, validated on 4 scenes from the PhotoTourism dataset.

What This Paper Is About

Not every image in a casually captured photo collection helps a NeRF reconstruct a scene; some contain transient objects, poor quality, or inconsistent appearance. Existing pipelines such as NeRF-W discard images using hand-crafted heuristics (a NIMA aesthetic threshold and a DeepLab v3 transient-object area threshold) applied before training, which never measures an image's actual effect on the reconstruction. This paper assigns a continuous, explicit contribution score to each image by tracking how validation PSNR changes each time that image is used during training, then tests whether training on the highly scored images actually improves results.

Key Contributions

1. An adaptation of data valuation to NeRF training. The authors repurpose the reward signal idea from Data Valuation using Reinforcement Learning (DVRL) into a direct, explicit continuous contribution score per image — DV_psnr (aggregated validation PSNR change) and DV_loss (aggregated validation L1 loss change) — rather than training a separate learned valuation policy.

2. A systematic examination of what makes a contribution score "fair." Three strategies for fair attribution are analyzed: weighting contributions by training progress, aggregating impact across epochs, and reverting the model to its pre-update state. Only cross-epoch aggregation was adopted, with the first epoch excluded.

3. A reproducibility study of contribution scores across random seeds. Correlation between independent runs is measured for the Brandenburg Gate scene and for three additional scenes, with the finding that reproducibility improves with more epochs and larger per-image sampling.

4. An empirical test of whether the scores are useful. Images scored as positive contributors are used to build new training sets, which are compared against NeRF-W's training sets on held-out test images across four scenes.

Main Findings

• Only the aggregated PSNR metric was reproducible. Across runs with different random seeds, DV_psnr achieved a correlation coefficient of 0.8 on the Brandenburg scene. Aggregated L1 loss difference, last-epoch PSNR difference, and per-image PSNR difference from an identical trained state did not hold up consistently.

• Reproducibility depends on sampling density. Correlation across seeds improved with more epochs and larger per-image sampling, but this trend held consistently for only one metric. DV_psnr showed satisfactory correlation when 500 pixel-rays per image were used in each iteration.

• Cross-seed correlations for the other three scenes were high. Using a subset of 100 images valuated twice with different seeds, correlation coefficients were 0.90 (Sacre Coeur), 0.87 (Trevi Fountain), and 0.86 (Taj Mahal).

• 716 of the valuated Brandenburg images were scored as positive contributors. For that scene, all images except the test and validation images were valuated, totalling 1310 images.

• Validation-set gains did not always transfer to the test set. On Brandenburg, DV_psnr selection reached higher validation PSNR (19.96 vs 19.27) but lower test PSNR (16.79 vs 17.72) on 43 held-out test images, compared with the NeRF-W training set. The authors attribute the gap to a larger test set that includes finer details.

• On the other three scenes, DV_psnr selection beat NeRF-W on test PSNR. Sacre Coeur: 16.23 vs 15.62. Taj Mahal: 16.55 vs 16.22. Trevi Fountain: 17.50 vs 17.33. Validation PSNR was also higher in all three (17.45 vs 16.94; 18.33 vs 17.93; 17.65 vs 17.51). These scenes have near-balanced validation and test sets (Sacre Coeur 11:11, Taj Mahal 14:13, Trevi Fountain 10:9).

• Distant viewpoints benefited from DV_psnr selection. In the Brandenburg example, the DV_psnr-trained model reached PSNR 19.51 versus 17.59 for the NeRF-W-trained model. Example PSNR values in Figure 3 were 21.3 vs 20.7 (Sacre Coeur), 17.31 vs 17.03 (Trevi Fountain), and 24.38 vs 22.9 (Taj Mahal), where the DV_psnr-rendered Taj Mahal image showed no obvious visual differences but a higher PSNR.

• The method disagreed with NeRF-W's heuristic filtering in both directions. One image included in NeRF-W's training set — despite containing numerous transient objects — was scored as harmful by DV_psnr, while an image excluded from NeRF-W's training, validation and test sets scored highly. The authors note the brighter "harmful" image may reveal more fine detail in unoccluded regions, while the highly scored image is darker with some less distinct areas but significantly fewer occlusions, and that image downscaling during analysis may also influence the difference.

• Scoring adds measurable overhead. On an NVIDIA GeForce RTX 3060 (12GB), a standard training step takes 0.185 seconds, with DV_psnr scoring adding approximately 0.5 seconds per step. For a typical 50,000-iteration run this amounts to about 9.5 hours of total training time — considerably faster than methods such as Data Shapley that demand multiple re-runs.

Methodology in Plain English

The scheme requires a NeRF variant that samples rays from a single image per training iteration. Standard NeRF training already works this way: images are shuffled into a new order each epoch and used once per epoch. That structure is what makes attribution possible — because one image drives each update, a change in validation quality can be tied to that specific image.

After every training step, the authors evaluate a fixed validation set of 10–14 images and record the PSNR. When image I appears again later in training, they subtract its previously recorded PSNR from the current one, giving a "delta" for that appearance. Summing these deltas for an image across all epochs — skipping the first epoch to avoid the outsized influence of early, barely trained updates — yields a single number, DV_psnr, per image. A parallel score, DV_loss, is built the same way from validation L1 loss. PSNR itself is derived from mean squared error between rendered and ground-truth images; L1 loss is used as the training objective and is described as more robust to outliers because it penalizes errors linearly.

To assess fairness, the authors considered three options. Weighting contributions by training progress was rejected because any choice of weighting function (linear, exponential, heuristic decay) would be arbitrary and would still ignore the non-linear interaction between image and model state. Reverting the model to a consistent pre-update state before each measurement was rejected as computationally intensive and unstable, with high variance across random seeds in the final epoch. Cross-epoch aggregation was selected because the model rapidly settles to a stable performance baseline (validation PSNR of approximately 14) after the first epoch, so later images are processed under a comparably trained model.

The evaluation uses 4 scenes from the PhotoTourism dataset. Brandenburg Gate (1363 images) drives the consistency study; NeRF-W reserved 10 images for testing, 763 for training, and 96 for validation, with the remainder excluded. For the other three scenes, all images except test and validation images were valuated once to measure the impact of training composition. The resulting scores were used to build new training sets, which were compared against NeRF-W's sets on held-out test images.

Why This Matters

The paper sits at the intersection of two growing concerns: making 3D reconstruction pipelines robust to messy real-world photo collections, and assigning economic value to individual data contributions.

Impact on research: It shows that a data valuation signal can be extracted from NeRF training without retraining the model or computing Shapley values, and it provides an honest accounting of where such scores are and are not reproducible. The finding that one metric survives seed variation while three others do not is a useful methodological caution for anyone building valuation schemes on top of stochastic training. The disagreement between DV_psnr and NeRF-W's NIMA-plus-DeepLab heuristics also suggests that pre-training image filtering may discard images that would have helped and keep images that hurt.

Real-world applications (drawn from the paper's own framing):

• Virtual tourism — reconstructing explorable 3D scenes from internet-sourced landmark photos of varying quality.
• Cultural heritage preservation — building digital twins of monuments from crowdsourced or archival image collections.
• Healthcare — 3D reconstruction applications in medical settings.
• Data marketplaces for XR and digital media — enabling fair compensation to different data providers by quantifying how much each contributed.

Industry relevance: The acknowledgement of funding under the Horizon Europe "XReco" project (grant 101070250) and the framing around data monetization place this work in the context of commercial XR media ecosystems. The reported cost of roughly 0.5 extra seconds per step is a practical figure for teams deciding whether continuous per-image valuation fits their training budget, and the method's model-agnostic design means it can be layered onto existing NeRF pipelines provided the one-image-per-step constraint is met.

Future Directions

1. Inferring contribution scores from a learnable model. The authors state that future work aims to optimize cost by directly inferring contribution scores rather than measuring them step by step.

2. Extending beyond PhotoTourism. The study was confined to this dataset; the authors call for applying DV_psnr to other datasets to validate generalizability.

3. Testing across NeRF variants and 3D Gaussian Splatting. The valuation was evaluated with two experimental NeRF pipelines inspired by HA-NeRF and NeRF-W, which reduce training time but do not reach state-of-the-art accuracy. Broader architectural coverage, including newer 3D Gaussian Splatting techniques, is left open.

4. Resolution transfer. Because valuation was performed on downscaled images, the authors hypothesize that data valuation scores may be transferable to other resolutions — a claim they observed held at double the valuation resolution but did not fully establish.

Target Audience

Researchers and practitioners working on neural rendering and 3D reconstruction from in-the-wild imagery will get the most from this paper, particularly those concerned with dataset curation for NeRF pipelines. It is also relevant to the data valuation and data marketplace communities, since it demonstrates how a valuation principle developed for conventional machine learning (DVRL) can be adapted to a rendering pipeline with very different computational constraints. Engineers building XR or phototourism products from user-contributed photos will find the reproducibility results and timing measurements directly actionable, while readers seeking state-of-the-art reconstruction numbers should note the authors' own caveat that the underlying NeRF pipelines were chosen for efficiency rather than peak accuracy.

Authors’ abstract

Data valuation and monetization are becoming increasingly important across domains such as eXtended Reality (XR) and digital media. In the context of 3D scene reconstruction from a set of images -- whether casually or professionally captured -- not all inputs contribute equally to the final output. Neural Radiance Fields (NeRFs) enable photorealistic 3D reconstruction of scenes by optimizing a volumetric radiance field given a set of images. However, in-the-wild scenes often include image captures of varying quality, occlusions, and transient objects, resulting in uneven utility across inputs. In this paper we propose a method to quantify the individual contribution of each image to NeRF-based reconstructions of in-the-wild image sets. Contribution is assessed through reconstruction quality metrics based on PSNR and MSE. We validate our approach by removing low-contributing images during training and measuring the resulting impact on reconstruction fidelity.