Nonlinear Noise2Noise for Efficient Monte Carlo Denoiser Training

AI summary

Overview

Research area: Computer vision and computer graphics, specifically machine learning-based Monte Carlo rendering denoising and weakly supervised learning theory.

Technical level: Advanced. The paper derives analytical minima for loss functions, proves pseudoconvexity, and applies Jensen gap bounds (Liao and Berg), the Bhatia-Davis inequality, and Popoviciu's inequality. Readers need comfort with probability theory, convexity arguments, and rendering terminology such as samples per pixel (spp) and high dynamic range (HDR).

Scope in one sentence: The paper develops a theoretical framework showing that certain nonlinear functions (such as tone mapping) can be applied to noisy targets in Noise2Noise training with bounded, minimal bias, and demonstrates this on an existing Monte Carlo denoiser (SBMC) trained only on noisy image pairs.

What This Paper Is About

Noise2Noise lets denoisers train on pairs of noisy images instead of requiring expensive clean reference images, but it was believed to break whenever a nonlinear function is applied to the noisy targets, because the expected value of a nonlinear function differs from the function of the expected value. Since nonlinear operations like tone mapping are exactly what makes HDR Monte Carlo rendering training stable, this limitation blocked a useful technique. The paper builds a theory for when these nonlinearities introduce only small bias, and shows that HDR Monte Carlo denoising can be trained from noisy data alone.

Key Contributions

1. A theoretical framework for nonlinear Noise2Noise. Using the Jensen gap, defined as the difference between the expected value of a nonlinear function and the function of the expected value, the authors analyze how loss functions behave when a nonlinear function T(v) is applied to both model output and noisy target.

2. A characterization of a class of low-bias nonlinear functions. The paper specifies that T(v) must be twice differentiable, strictly monotonic, and nonnegative for v ≥ 0, and shows that functions with low curvature (especially where clean targets lie) produce small Jensen gap bounds. Bounded distributions permit additional curvature near the minimum and maximum values.

3. Analytical minimum values and bias terms for three losses. The authors derive the minimizer of L2, relative mean squared error (rMSE), and LHDR losses, with and without nonlinearity, and prove pseudoconvexity so that the local minima are global minima.

4. Demonstration on an existing Monte Carlo denoiser. The method is applied to Sample-Based Monte Carlo Denoising (SBMC), whose original implementation was trained with high-sample count reference images. The authors report results that approach the original implementation using only noisy training data. They state that this work and previous experimental work by the first author (Tinits, 2022) are, to their knowledge, the first to successfully apply nonlinear functions to noisy targets in Noise2Noise training.

Main Findings

• Nonlinearity shifts the L2 minimizer to a transformed mean. For the L2 loss with a nonlinear function T applied, the single global minimum occurs at T⁻¹( E[T(ŷ)] ), meaning the bias is expressed in the nonlinear space and mapped back through the inverse function.

• The rMSE loss remains pseudoconvex. With and without tone mapping, the rMSE loss (formulated as (T(ỹ) − T(ŷ))²/(T(ŷ) + ε)²) has a single global minimum given by a ratio of expectations of T(ŷ)/(T(ŷ)+ε)² and 1/(T(ŷ)+ε)², passed through T⁻¹. Pseudoconvexity for ỹ ≥ 0 follows from Mangasarian (1965).

• The LHDR loss carries a bias equal to the index of dispersion. Without tone mapping, the LHDR minimizer simplifies to y + Var(ŷ)/y, i.e., the variance divided by the mean. Because the denominator can be separated from the expectation, the authors note LHDR can be redefined by setting the gradient of the denominator to zero, removing the bias; they call this variant L*HDR and evaluate it separately in Section 5.2.

• The Noise2Noise authors' LHDR loss was insufficient in their setting. The paper states that in their testing this proposed solution to HDR training issues still performed poorly, and attributes the difference to their more complex model and their significantly larger and more varied training dataset.

• Tone mapping can be combined with Noise2Noise. The previously assumed incompatibility (Lehtinen et al., 2018) is addressed by showing that certain combinations of loss function and tone mapping reduce the effect of outliers while introducing minimal bias, with experimental results confirming the approach.

• Bias bounds scale with variance and curvature. Using the method of Liao and Berg (2019), the Jensen gap is bounded between J₋(μ)·Var(X) and J₊(μ)·Var(X), where J depends on the curvature of the nonlinear function and μ is the mean. The bound is small when the function is near-linear around the mean or when the variance is small.

• Curvature placement matters for tone mapping design. The paper argues that effective dynamic range reduction requires more curvature near zero and less toward infinity, which also yields low J(y) values over most of the domain except near zero. Because HDR clean targets y are often large, these tone mapping functions tend to have low J(y) for HDR images.

• Variance is bounded by a parabola for bounded distributions. Via the Bhatia-Davis inequality, Var(ŷ) ≤ (M − y)(y − m) = −y² + My for minimum m = 0, forming a parabola with roots at y = 0 and y = M and a maximum of M²/4 at y = M/2, which is also known as Popoviciu's inequality. This bound is small near y = 0 and y = M.

• "Clean" targets are never truly noiseless. The authors observe that in long-exposure photography and Monte Carlo rendering, zero noise is reached only after infinite time, so some noise remains even in so-called clean targets — meaning Noise2Noise training and its Jensen gap concerns apply even when training with clean references.

• Denoising quality approaches the supervised baseline. Figure 1 presents an 8 spp Monte Carlo rendering denoised with their models, denoised with SBMC, and a reference rendered at 8192 spp; the models are trained with only noisy data while SBMC uses clean reference images. The quantitative denoising metrics for the experiments are not reported in the available text (the content is truncated during the training-progress description in Section 5.1).

Methodology in Plain English

The researchers start from a mathematical quirk: if you average noisy images, you recover the clean image, but if you first push those noisy images through a nonlinear function, the average of the outputs is not the same as the function applied to the clean image. That mismatch is the Jensen gap. They work out, for three different loss functions, exactly what value a network converges to when a nonlinear function sits between the network output and the loss, and then use an existing bound from the literature to say how far that converged value can stray from the ideal answer.

The key practical lever is curvature. If the nonlinear function is nearly straight in the region where the image values actually live, the gap is tiny; if the variance of the noisy targets is small, it is also tiny. That tells the researchers what kind of tone mapping function to pick: one that compresses hard near zero, where the values are cramped anyway, and stays flat for large values where HDR outliers sit. This also reduces the influence of HDR outliers on training.

For the experiments, they take an existing neural Monte Carlo denoiser, SBMC, and reproduce its training setup as closely as possible while swapping clean references for noisy pairs. Scenes are randomly generated with the SBMC scene generator, restricted to outdoor scenes because the indoor dataset SunCG is no longer available; they deliberately did not substitute another indoor dataset, to avoid factors that could artificially improve their results. Rendering uses PBRT v2 with the SBMC patch, saving samples separately and collecting extra feature data. Each scene is rendered twice at 8 spp with different random seeds to form the noisy input/target pair. The training set has 283,831 example pairs at 128 × 128 resolution, roughly 300,000 as intended, with the shortfall due to rejection sampling in the SBMC scene generator; for SBMC the noisy inputs were rendered at 8 spp and references at 4096 spp. A second training set was generated with a different starting seed, adding 284,016 example pairs. They then train SBMC models across combinations of loss functions and tone mapping functions and compare denoising performance.

Why This Matters

Impact on research. The paper removes a long-standing restriction on Noise2Noise training, namely that nonlinear preprocessing of targets was considered off-limits. It gives a principled way to decide which nonlinear functions are safe, which could make weakly supervised denoising viable across domains where clean references are effectively unobtainable. The authors also note their idea can be combined with single-image methods such as Noise2Void, Noise2Self, and the work of Laine et al. (2019), which rely on the same statistical property.

Real-world applications:

• Monte Carlo rendering production pipelines: film, animation, and architectural visualization can denoise by rendering two cheap low-spp images instead of spending time on high-sample reference renders.
• Real-time and interactive rendering: the paper notes that denoising has contributed to adapting Monte Carlo rendering for real-time applications, where both training references and per-frame sample budgets are tight.
• Long-exposure photography: the paper cites photography as a setting where clean images are hard to obtain because the noise only vanishes with lengthy capture.
• Biomedical imaging: the paper mentions single-image denoising methods being useful in biomedical imaging where noisy image pairs are difficult to obtain, and its nonlinearity analysis is stated to apply to those methods as well.

Industry relevance. The practical claim is economic: expensive high-sample-count reference images can be replaced by pairs of cheap renderings, which reduces the compute cost of building denoiser training sets. The authors' integration into an existing published denoiser (SBMC) with a publicly available test set, and their deliberate choice not to change other variables, makes the result easier for practitioners to assess against a known baseline.

Future Directions

• Completing the loss-function comparison. The paper states that the bias-removed L*HDR definition is evaluated separately in Section 5.2, but the reported outcomes of that evaluation are not present in the available text.

• Closing the gap to supervised training. The authors report that their noisy-only models approach, but do not match, the original SBMC implementation trained on high-sample references. The quantitative size of the remaining gap is not reported in the available content.

• Scene diversity and generalization. The training data contains only outdoor scenes because SunCG is no longer available and the authors declined to substitute an indoor dataset. They explicitly flag that this could affect generalization to indoor scenes and should be considered when comparing against SBMC.

• Tighter bias bounds. The paper notes that tighter Jensen gap bounds can be derived using higher central moments than the variance (Lee et al., 2021), but that these moments are less interpretable for arbitrary distributions — leaving an open trade-off between bound tightness and practical usability.

Target Audience

Researchers and graduate students in computer graphics, rendering, and image processing who work on denoising, Monte Carlo sampling, or weakly supervised learning; machine learning engineers building denoisers where clean ground truth is expensive; and rendering engineers in film, animation, or game development who need to train denoisers without high-sample reference render sets. The theoretical sections (Sections 3 and 4) suit readers comfortable with probability and convexity arguments, while the experimental setup in Section 5 is readable by practitioners who mainly want to know what was trained and on what data.

Authors’ abstract

The Noise2Noise method allows for training machine learning-based denoisers with pairs of input and target images where both the input and target can be noisy. This removes the need for training with clean target images, which can be difficult to obtain. However, Noise2Noise training has a major limitation: nonlinear functions applied to the noisy targets will skew the results. This bias occurs because the nonlinearity makes the expected value of the noisy targets different from the clean target image. Since nonlinear functions are common in image processing, avoiding them limits the types of preprocessing that can be performed on the noisy targets. Our main insight is that certain nonlinear functions can be applied to the noisy targets without adding significant bias to the results. We develop a theoretical framework for analyzing the effects of these nonlinearities, and describe a class of nonlinear functions with minimal bias. We demonstrate our method on the denoising of high dynamic range (HDR) images produced by Monte Carlo rendering. Noise2Noise training can have trouble with HDR images, where the training process is overwhelmed by outliers and performs poorly. We consider a commonly used method of addressing these training issues: applying a nonlinear tone mapping function to the model output and target images to reduce their dynamic range. This method was previously thought to be incompatible with Noise2Noise training because of the nonlinearities involved. We show that certain combinations of loss functions and tone mapping functions can reduce the effect of outliers while introducing minimal bias. We apply our method to an existing machine learning-based Monte Carlo denoiser, where the original implementation was trained with high-sample count reference images. Our results approach those of the original implementation, but are produced using only noisy training data.