Image Restoration and Inverse Problems

Restoration begins by modeling how the latent scene became the measured image.

The chain runs one way. Step 2 destroys information and step 5 cannot put it back. It returns one member of the set of scenes still compatible with the observation. PULSE, a super-resolution method published in 2020 by Menon and four colleagues at Duke, makes that explicit by construction. It does not estimate the lost pixels at all. It searches the latent space of a pretrained StyleGAN for a high-resolution image that downscales back to the low-resolution input. Any candidate that downscales correctly is an acceptable output. Step 3 does not determine step 5. The prior in step 4 does.

Both approaches resolve ambiguity by preferring some solutions over others.

The third column is where the preference becomes visible. In June 2020 a downsampled photograph of Barack Obama went through the PULSE-based Face Depixelizer and came out a white face. The authors answered by adding a Bias section and a model card to their paper: “While we initially chose to demonstrate PULSE using StyleGAN (trained on FFHQ) as the generative model for its impressive image quality, we noticed some bias when evaluated on natural images of faces outside of our test set.” Then they measured it. Success — finding a generator output that downscales back to the input — ran at 79.2% for Black faces on the FairFace dataset, sampling 100 examples per subgroup. It ran at 83.4% for White, 87.0% for East Asian and Middle Eastern, 87.4% for Indian and Southeast Asian, and 90.2% for Latino/Hispanic. They also quoted Salminen and colleagues' finding that 72.6% of StyleGAN's generated pictures represent White people, against 13.8% Asian, 10.1% Black and 3.4% Indian, and concluded that this bias extends to any downstream application of StyleGAN, PULSE included.

So the line below reading “May hallucinate structures” is not a caveat about texture. It is a statement about whose face the prior supplies when the measurement cannot decide.

Treat each degradation as a separate measurement model.

The compression family is not hypothetical. On 2 August 2013 David Kriesel showed that Xerox WorkCentre and ColorQube scanners were silently substituting digits in scanned documents. JBIG2 “Pattern Matching & Substitution” in lossy mode groups visually similar glyph segments and stores only one representative. That representative is then pasted in place of the others. On his WorkCentre 7535 and 7556 test machines a 65 was reproduced as 85, and a 60 as 80, in cleanly rendered type. No blur, no ringing, nothing for a reader to notice. Xerox confirmed the defect across the ColorQube 87XX/89XX/92XX/93XX and WorkCentre 5030, 5050, 51XX, 56XX, 57XX, 58XX, 6400, 7220, 7225, 75XX, 76XX, 77XX and 78XX families, the WorkCentrePro 2XX and BookMark 40/55. Its on-device disclaimer for lower quality settings read: “The normal quality option produces small file sizes by using advanced compression techniques. Image quality is generally acceptable, however, text quality degradation and character substitution errors may occur.” After further testing the company stated that even the factory-default and highest-quality modes did not completely eliminate the substitutions on stress documents. A prior that swaps a real measurement for a plausible substitute leaves no visible sign of having done so.

Averaging plausible outputs may improve mean squared error while producing blur; perceptual or adversarial losses can sharpen texture while moving farther from the exact target.

This tension has been proved, not merely observed. In 2017 Blau and Michaeli set out to “prove mathematically that distortion and perceptual quality are at odds with each other”. Take the probability of “correctly discriminating the outputs of an image restoration algorithm from real images”. Their result: “as the mean distortion decreases, this probability must increase (indicating worse perceptual quality)”. And the trade is not an artifact of the metric you happened to choose — “this result holds true for any distortion measure, and is not only a problem of the PSNR or SSIM criteria”. The version published a year later adds that for some measures the effect “is less severe”. Their response was not a better single number but a two-axis evaluation. They “propose a new methodology for evaluating image restoration methods” and use it “to perform an extensive comparison between recent super-resolution algorithms”.

Choose metrics according to the use case. A visually pleasing photograph, a quantitative microscopy measurement, and an OCR preprocessing stage require different evidence. None of them is served by one scalar that has to buy its score on one axis by paying on the other.

A restoration benchmark should test both reconstruction and decision consequences.

Step 1 is settled by real acquisition data. In 2018 Abdelhamed and two colleagues built the Smartphone Image Denoising Dataset. It holds “~30,000 noisy images from 10 scenes under different lighting conditions using five representative smartphone cameras”. Its ground truth came from “a systematic procedure for estimating ground truth for noisy images”, not from an assumption. Benchmarking on it showed “that CNN-based methods perform better when trained on our high-quality dataset than when trained using alternative strategies, such as low-ISO images used as a proxy for ground truth data”. The proxy was the problem.

Steps 4 and 5 have a benchmark of their own. The 2020 fastMRI challenge distributed 7,299 clinical brain scans from NYU Langone Health and held back the fully sampled data from 894 of them. It received 19 submissions from eight groups. Radiologists reading the reconstructions found anatomy that was not in the patient: a false vessel, a bright line mimicking a cerebrospinal-fluid cleft, a false sulcus. Muckley and colleagues wrote it into the discussion: “Such hallucinatory features are not acceptable and especially problematic if they mimic normal structures that are either not present or actually abnormal. These images had high SSIM scores, indicating that even though these images are considered well-optimized according to this metric, they are not optimized regarding hallucination features.” The metric passed the image. The radiologist did not. Bhadra and colleagues later gave the failure a formal treatment, in 2021, decomposing a reconstruction into generalized measurement and null components and introducing a “hallucination map”.

What a regulator accepted for a learned reconstruction runs in the same order as these five steps. On 10 April 2020 the FDA cleared GE Healthcare's AIR Recon DL on the SIGNA Premier MRI system, under 510(k) K193282, as substantially equivalent. The nonclinical package used a digital reference object and phantom imaging to test SNR, sharpness, low contrast detectability and noise spectral content. It also carried a specific check that the network had not quietly moved the numbers clinicians read off the image: “Analysis was performed to confirm that the feature does not introduce significant bias that might impact quantitative measurements based on signal intensity.” Only then came the downstream test — a blinded radiologist reader study comparing images reconstructed with the algorithm against images reconstructed without it, including shorter-scan versus longer-scan pairs. An independent evaluation two years later, by Zerunian and colleagues on 50 volunteers at Sapienza University of Rome, reported SSFSE T2 SNR of 181.40 ± 135.09 with the algorithm versus 109.79 ± 108.98 without (P = 0.0012). Overall image quality was 4.94 ± 0.23 versus 3.26 ± 0.75 on a five-point Likert scale (P < 0.0001).

Night cameras produce shot noise, motion blur, clipped highlights, and aggressive compression. Plan the evaluation the way the AIR Recon DL clearance did: reference conditions first, a bias check on whatever quantity anyone will read off the image, and a blinded downstream decision test last.