FlareX: A Physics-Informed Dataset for Lens Flare Removal via 2D Synthesis and 3D Rendering

AI summary

Overview

Research area: Computer vision and computational photography, specifically dataset construction and deep-learning-based image restoration for lens flare removal.

Technical level: Intermediate. The paper is readable without a physics background, but it assumes familiarity with image restoration benchmarks (PSNR, SSIM, LPIPS, NIQE, BRISQUE), physically based rendering, and monocular depth estimation.

Scope: The paper introduces FlareX, a mixed 2D-synthesized and 3D-rendered dataset for training and evaluating lens flare removal models, together with a masking strategy for collecting real-world flare-free ground truth.

What This Paper Is About

Lens flare appears when a camera shoots toward strong light, degrading image quality and hurting downstream vision tasks. Because true paired flare-corrupted and flare-free photographs are very hard to capture, prior datasets build training pairs by pasting artificial 2D flare templates onto background photos, which limits flare diversity and ignores physics such as how flare brightness relates to the light source's distance and angle. This paper's goal is to build a more physically realistic dataset by generating flare data through parameterized template creation, illumination-law-aware 2D synthesis, and physics-engine-based 3D rendering, plus a new way to measure real-world performance.

Key Contributions

1. A wider catalogue of flare templates. The authors create 9,500 flare templates derived from 95 flare types with different parameter settings, produced in the Blender 3D graphics engine using flare patterns from the Flared plugin, with mutual constraints among flare components (light source, streak, iris, glare) so that moving the light source changes the pattern in a physically plausible way.

2. Illumination-aware 2D synthesis. They improve the standard 2D synthesis pipeline by incorporating the laws of illumination, using an estimated depth map to relate flare brightness to the light source's spatial position and angle of incidence, rather than adding flares with random affine transformations only.

3. A 3D-rendered complement dataset. They construct 60 3D scenes with flares placed in plausible locations (near light sources, in the sky, outside windows) and keyframe a camera path to render 3,000 flare image pairs from multiple viewpoints.

4. A masking approach for real-world evaluation. They use an eye-exam occluder to block the direct light source and capture flare-free images under the same conditions as the corrupted ones, collecting 100 image pairs at 3024 × 3024 resolution, with the occluded region excluded from metrics. They also collect 63 off-screen flare images from the Internet.

Main Findings

• Dataset composition: FlareX contains 9,500 2D templates from 95 flare patterns plus 3,000 flare image pairs rendered from 60 3D scenes, abbreviated as Flare-2D and Flare-3D. The authors state it covers both daytime and nighttime scenarios, multiple reflections, and light source annotations, whereas they describe Wu et al. (2021) as daytime-only and Flare7K and FlareReal600 as nighttime-only.

• Comparison with existing datasets (Table 1): Wu et al. (2021): 3 types, 5,001 count; Flare7K: 35 types, 7,000; FlareReal600: 600; Flare-R: 962; SDFRD: 3 types; FlareX: 95 types, 9,500 + 3,000.

• Quantitative comparison on the proposed test set (Table 3): Training on FlareX gives the best scores for HINet, MPRNet, Uformer, and Restormer. Uformer trained on FlareX reaches PSNR 25.459, SSIM 0.692, LPIPS 0.133; HINet 25.388 / 0.682 / 0.131; Restormer 25.096 / 0.688 / 0.131; MPRNet 23.882 / 0.660 / 0.138. The paper reports that Uformer surpasses the second-best model by nearly 1.38 dB in PSNR, achieves a 3.75% increase in SSIM, and reduces LPIPS by 5%.

• Test-set alignment with human judgment (Table 2): On the proposed test set, models trained on Flare7K score 7% and 5% on PSNR and user study respectively, while models trained on Flare7K++ score 93% and 95%. The paper states that nearly one-third of samples in existing test sets fail to accurately reflect the improvement from Flare7K to Flare7K++.

• Off-screen flare removal (Table 4): With no-reference metrics, Uformer trained on FlareX achieves the lowest NIQE (3.967 vs. input 4.144, Wu et al. 4.023, Flare7K 4.050, Flare7K++ 4.026) and the lowest BRISQUE (28.354 vs. input 31.894, Wu et al. 29.070, Flare7K 30.621, Flare7K++ 29.766).

• Ablation on the laws of illumination (Table 5): Adding illumination laws improves Uformer on Flare7K from 23.386 / 0.667 / 0.141 to 23.711 / 0.675 / 0.138, and on FlareX from 25.122 / 0.681 / 0.135 to 25.459 / 0.692 / 0.133.

• Ablation on dataset composition (Table 6): For Uformer, Flare-2D alone gives 24.853 / 0.690 / 0.140, Flare-3D alone gives 23.647 / 0.687 / 0.137, and both together give 25.459 / 0.692 / 0.133. Restormer shows the same ordering: 24.457 / 0.616 / 0.139, 23.728 / 0.665 / 0.141, and 25.096 / 0.688 / 0.131. The paper attributes the weak Flare-3D-only result to its limited amount.

• Downstream effect on object detection: In visual comparisons, flare-removed images reveal occluded objects such as bicycles and motorcycles that were not detected in flare-corrupted images, and a case where flares caused bicycles to be misclassified as motorcycles is resolved after removal.

• Stated limitation: Extremely heavy flare remains a significant challenge, often leaving visible artifacts after restoration (Figure 11).

Methodology in Plain English

The authors build the dataset in three stages. First, they use the Blender 3D graphics engine and the Flared plugin to assemble flares from components such as light source, streak, iris, and glare. They bind the light source to a spatial point and rely on Blender's preset mutual constraints among components, then render on a black background in flat mode to produce flare templates. Re-rendering after removing all components except the light source yields the corresponding light source templates and hence annotations. Manually adjusting camera focal length and component parameters produces 95 flare types, each yielding 100 templates.

Second, for 2D synthesis they apply random affine transformations to flares, estimate a depth map of the background image with a pre-trained monocular depth estimation model, and pass the results through a Brightness Adjustment Module. The module uses the laws of illumination (illumination equals luminous intensity times the cosine of the incidence angle, divided by the squared distance) to compute each flare's brightness. Spatial Position Estimation uses the average depth of the light source's pixels as the distance to the lens, and the horizontal field of view to estimate the incidence angle. Multiple adjusted flares are then added to the background and clipped to the [0, 1] range. Because the field of view is adjustable, the pipeline can synthesize data for a specific camera. Background images are sampled from the 24K Flickr image dataset.

Third, for the 3D part they construct scenes in Blender, place flares where they are likely to occur rather than randomly, keyframe a camera path, and render flare-corrupted images along the path, then repeat with the flares removed to get the paired ground truth.

For evaluation, they train the same models on five datasets using an identical data aggregation approach, with an L1 loss, a perceptual loss using a pre-trained VGG-19, and a reconstruction loss. Training runs on two NVIDIA GeForce RTX 3090 GPUs with 24GB of memory, on 512 × 512 crops, with batch size 2 for 30,000 iterations. Models compared include HINet, MPRNet, Uformer, and Restormer, with the addition of AST mentioned in the experimental setup.

Why This Matters

Impact on research: The paper argues that flaws in synthetic data and, more importantly, in test-set ground truth distort reported progress in flare removal. Its masking-based evaluation protocol, which excludes the occluded region from metrics, offers a way to measure model quality on real images containing reflective and severe flares that lens-wiping cannot remove. The dataset and code are released at https://github.com/qulishen/FlareX under a CC BY 4.0 license.

Real-world applications:

• Cleaning photos taken against strong light, including sunsets, night city scenes, and indoor shots with non-spherical light sources.
• Improving smartphone camera pipelines, where the paper notes off-screen flare from large-aperture flagship phones has been widely discussed online and is not effectively removed by existing methods.
• Preprocessing for high-level vision systems such as object detection, where the paper shows flares can hide or misclassify objects like bicycles and motorcycles.
• Supporting camera and lens design evaluation by synthesizing data for a specific field of view, which the authors present as an alternative to expensive anti-reflective coatings.

Industry relevance: The work targets consumer-grade photography, where anti-reflective lens coatings are described as costly and of limited effectiveness, making software-based flare removal attractive to camera and phone manufacturers.

Future Directions

• Handling extremely heavy flare, which the authors identify as a remaining challenge that leaves visible artifacts, possibly by incorporating physical priors or transferring structural cues from cleaner regions.
• Improving the accuracy of the estimated depth map, which the authors cite as motivation for adding the 3D rendering branch in the first place.
• Expanding Flare-3D, since training on Flare-3D alone performs poorly because of its limited amount (3,000 pairs from 60 scenes) compared with Flare-2D.
• Extending the masking-based evaluation protocol and the collected off-screen flare images toward broader real-world coverage, given that off-screen flare images have no ground truth and require no-reference metrics such as NIQE and BRISQUE.

Target Audience

Researchers and engineers working on image restoration, computational photography, and camera image signal processing, particularly those who build or benchmark datasets for flare removal and other lens-artifact tasks. The paper is also useful for practitioners evaluating how synthetic training data and flawed test-set ground truth affect measured model performance, and for anyone interested in physically based rendering as a source of training data.

Authors’ abstract

Lens flare occurs when shooting towards strong light sources, significantly degrading the visual quality of images. Due to the difficulty in capturing flare-corrupted and flare-free image pairs in the real world, existing datasets are typically synthesized in 2D by overlaying artificial flare templates onto background images. However, the lack of flare diversity in templates and the neglect of physical principles in the synthesis process hinder models trained on these datasets from generalizing well to real-world scenarios. To address these challenges, we propose a new physics-informed method for flare data generation, which consists of three stages: parameterized template creation, the laws of illumination-aware 2D synthesis, and physical engine-based 3D rendering, which finally gives us a mixed flare dataset that incorporates both 2D and 3D perspectives, namely FlareX. This dataset offers 9,500 2D templates derived from 95 flare patterns and 3,000 flare image pairs rendered from 60 3D scenes. Furthermore, we design a masking approach to obtain real-world flare-free images from their corrupted counterparts to measure the performance of the model on real-world images. Extensive experiments demonstrate the effectiveness of our method and dataset.