MESS: Multi-Exposure Sequence Synthesis for Generalizable Image Enhancement

AI summary

Overview

Research area: Computational photography and low-level vision — specifically data synthesis for image illumination enhancement, covering low-light image enhancement (LLIE), exposure correction (EC), and backlit image enhancement (BIE).

Technical level: Intermediate to Advanced. The high-level idea (generate better training pairs) is accessible, but the implementation combines an emulated camera ISP pipeline, Haar discrete wavelet transforms, a LoRA-adapted Stable Diffusion Turbo backbone, and FiLM-based conditioning.

Scope: The paper proposes MESS, a framework that transfers exposure variation learned from high-bit RAW images onto ordinary 8-bit RGB photos to mass-produce paired training data, then shows that a lightweight enhancement network trained on 500K such pairs generalizes across five enhancement benchmarks.

What This Paper Is About

Enhancement models for badly lit photos (too dark, too bright, backlit) are usually trained on hand-captured, carefully aligned image pairs, which are expensive to collect and narrow in scene coverage, so a model trained for one task often fails on another. Synthetic alternatives exist, but RAW-domain synthesis is limited by how little RAW data is available, and RGB-domain synthesis cannot faithfully undo the camera pipeline once dynamic range has been clipped or compressed. MESS closes this gap by using RAW images only as a source of realistic exposure supervision, then learning a generator that reproduces those exposure changes on any in-the-wild RGB image at scale.

Key Contributions

1. A RAW-to-RGB supervision transfer framework. MESS derives multi-exposure sequences (MES) from high-bit RAW images via an emulated ISP, then trains a diffusion-based RGB-to-MES generator so the same exposure behaviour can be synthesized from a single 8-bit RGB image, combining RAW realism with RGB scalability.
2. A three-stage MES synthesis pipeline. A frequency-based enhancement module built on Haar DWT with channel attention produces style-robust scene representations; a diffusion module (SD-Turbo with LoRA and a learnable pseudo-word embedding) restores details lost to clipping; an exposure-conditioned renderer applies FiLM-modulated affine parameters to wavelet sub-bands to control the target exposure.
3. A 500K-pair synthetic dataset. For each source image, one sampled exposure variant serves as the degraded input and the original high-quality RGB as the target.
4. Evidence that data, not model size, drives generalization. A lightweight FourLLIE backbone trained only on MESS data outperforms models trained on in-domain real datasets, RAW-based synthetic datasets, and RGB-based synthetic datasets when averaged over five test sets.

Main Findings

• Best average full-reference scores with the FourLLIE backbone. MESS-generated training data reaches an average PSNR of 21.923, SSIM of 0.846, and LPIPS of 0.121 across UHDLL, LSRW, SICE, MSEC, and BAID. The strongest in-domain baseline average is MSEC at 20.079 PSNR; the strongest RGB-synthetic baseline is UEC-based at 19.645 PSNR; the RAW-synthetic RCD reaches 19.101 PSNR.
• In-domain training still wins on its own benchmark, but only there. On the UHDLL test set, the UHDLL-trained model scores 23.694 PSNR versus 23.416 for MESS; on LSRW, 20.849 versus 20.514; on SICE, 22.385 versus 21.769; on MSEC, 22.116 versus 21.982. MESS is ahead on BAID (21.935 versus 21.674) and clearly ahead on the average.
• Best no-reference scores on every metric. MESS averages ARNIQA 0.709, MUSIQ 55.510, and MANIQA 0.467, versus 0.692 / 53.920 / 0.450 for the next-best source (MSEC).
• Performance scales with dataset size. Macro-averaged PSNR rises from 20.220 (MESS-5K) to 20.590 (MESS-10K) to 21.199 (MESS-50K) to 21.923 (MESS-500K); SSIM rises 0.822 to 0.846 and LPIPS falls 0.143 to 0.121. Even MESS-5K, roughly the size of LSRW or SICE, beats both real-data baselines (LSRW 19.663 PSNR / 0.816 SSIM / 0.149 LPIPS; SICE 19.965 / 0.816 / 0.149).
• RGB expansion beats RAW-only data at matched scale. With 39,642 scenes and pairs in both, RAW-only data (exposure-perturbed inputs with MEF-generated targets, following an approach similar to UNICE) gives 20.764 PSNR / 0.829 SSIM / 0.139 LPIPS, while matched-size MESS-RGB gives 21.083 / 0.837 / 0.131, and the full 500,000-pair set gives 21.923 / 0.846 / 0.121.
• Qualitative gains. In cross-dataset low-light tests on LSRW, the UHDLL-trained model amplifies JPEG artifacts and the LSRW-trained model blurs details, while the MESS-trained model avoids both. On a SICE exposure-correction example, MSEC-trained data fail to enhance effectively and UEC-trained data produce unnatural colour, while MESS gives more natural exposure and better colour fidelity.
• Existing datasets are small. The paper's Table 1 lists LOLv2Real at 789 images / 263 scenes, UHD-LL at 2,150 / 724, LSRW at 5,650 / 4,157, SICE at 4,800 / 589, BAID at 3,000 / 1,782, Backlit300 at 305 / 305, LCDP at 1,700 / 1,700, and MSEC at 24,330 / 5,000.
• Reported limitations. MESS depends on a fixed set of RAW datasets for its exposure priors, and it selects the best-rendered image using several NR-IQA metrics rather than a vision-language model.

Methodology in Plain English

The trick is to separate what exposure looks like from what a photo looks like.

First, the authors gather high-bit RAW images from seven public datasets (AODRAW, FiveK, HDRP, PASCAL-RAW, PPR10K, RAISE, RAW-NOD) and filter out bad ones. Each RAW frame is rendered at seven exposures from −3 to +3 EV. From these they build two kinds of supervision: a set of style-varied renderings (different tone mapping and multi-exposure fusion settings applied to the same seven-exposure stack), and a camera-consistent rendering at a randomly chosen target exposure, which serves as the reference answer.

Second, they train a three-stage generator on these triplets. The first stage splits an image's features into wavelet sub-bands and reweights them with channel attention, learning to produce a representation that stays the same even when the rendering style changes — enforced by an L1 consistency loss between two style-varied renderings of the same scene. The second stage pushes that representation through a frozen Stable Diffusion Turbo VAE and a LoRA-adapted U-Net guided by a single learnable pseudo-word, so the generative prior can hallucinate plausible texture where the 8-bit input has lost information. The third stage takes the target exposure value, encodes it with a small MLP, and uses a FiLM predictor to produce per-channel scale and shift parameters applied to the wavelet sub-bands, so brightness and tone can be dialled continuously. Training uses the squared-error reconstruction loss against the camera-consistent RAW reference, with the consistency term weighted equally.

Third, once trained, the generator is applied to high-quality in-the-wild RGB photos: sample an exposure value, synthesize the variant, use it as the degraded input and the untouched original as the target. Repeat 500,000 times and train a small enhancement network on the result.

The MESS generator itself was trained on 512×512 crops for 200K iterations with AdamW on 8 NVIDIA A100 GPUs, batch size 64, learning rate from 1×10⁻⁴ to 1×10⁻⁶ by cosine schedule, target EV sampled uniformly from [−3, 3]. Downstream models use 512×512 crops, AdamW at 2×10⁻⁴, batch size 64, cosine decay, an ℓ1 loss, and 300K iterations. Evaluation uses six backbones (FourLLIE, RetinexFormer, Multinex for LLIE; ExSlot and CSEC for EC; RAVE for BIE); the main paper reports only FourLLIE results, with the other backbones in Appendix C, which is not included in the supplied content.

Why This Matters

Impact on research. The paper reframes the generalization problem in illumination enhancement as a data problem rather than an architecture problem, and offers a concrete recipe for decoupling physical realism (from RAW) from scale (from RGB). It also provides a 500K-pair resource and a reproducible evaluation protocol spanning three tasks and five test sets, which gives the field a common yardstick for judging synthetic data quality.

Real-world applications.

• Smartphone camera pipelines, where tone mapping and exposure correction must handle mixed under- and over-exposed regions in a single frame.
• Backlit photography such as portraits against a bright window or sky, where faces are dark and backgrounds are blown out.
• Night-time and low-light capture in consumer devices, dashcams, and surveillance cameras.
• Batch photo editing and gallery apps that need one model to fix arbitrary badly exposed images rather than one model per scenario.

Industry relevance. Two of the four authors are affiliated with Qualcomm Technologies and OPPO Research Institute, and the work is partly supported by the PolyU-OPPO Joint Innovative Research Center — a clear signal that scalable training-data synthesis is a direct concern for mobile imaging teams. If a lightweight network trained on synthetic pairs can generalize across tasks, OEMs can avoid per-device paired-data collection campaigns, which are the dominant cost in tuning camera pipelines.

Future Directions

• Broaden the RAW prior pool. The authors state MESS relies on a fixed set of RAW datasets; more diverse RAW sources would improve the realism and coverage of synthesized exposure variation.
• Replace NR-IQA-based data filtering with vision-language models. The current pipeline picks the best-rendered image using several no-reference IQA metrics; the authors suggest that VLMs could yield more robust selection.
• Test whether the transfer idea extends beyond exposure. MESS transfers exposure behaviour specifically; whether the same RAW-to-RGB generator scheme can carry noise, motion blur, or sensor-specific priors is left open.
• Report and analyse the other downstream backbones. The main paper shows only FourLLIE results, deferring RetinexFormer, Multinex, ExSlot, CSEC, and RAVE to an appendix and omitting TOPIQ scores from the main tables, so the breadth of the claim rests on material not visible in the supplied content.

Target Audience

Graduate students and applied researchers in computational photography and low-level vision who work on low-light enhancement, exposure correction, or backlit enhancement; practitioners building camera pipelines or mobile photo-editing features who need scalable paired training data; and anyone interested in using generative priors, wavelet decompositions, or ISP emulation as a data-synthesis tool rather than as an inference-time model. Readers should be comfortable with diffusion models, camera ISP concepts, and standard restoration metrics.

Authors’ abstract

Despite the significant progress in image enhancement under adverse illuminations, such as low-light image enhancement, exposure correction, and backlit image enhancement, existing models show limited generalizability to different scenarios. A major bottleneck lies in the lack of large-scale and diverse training data, as these tasks typically rely on manually captured, precisely aligned image pairs that are expensive to scale up. Although synthetic data have been explored as an alternative, existing synthesis methods are limited in modeling complex adverse illumination conditions and camera imaging pipelines. To address this issue, we propose a novel framework to synthesize realistic training pairs for generalizable image enhancement in the wild. Specifically, we first derive multi-exposure sequences (MES) from high-bit RAW images by rendering them under different exposure settings through an emulated ISP pipeline. Using these RAW-derived MES as supervision, we train a diffusion-based RGB-to-MES generator to synthesize plausible exposure sequences from a single 8-bit RGB image, extending the synthesis pipeline from limited RAW collections to large-scale in-the-wild RGB data. For each source image, we randomly sample one synthesized exposure variant as the degraded input and use the original high-quality RGB image as the target. With the proposed MES Synthesis (MESS) approach, we construct a dataset of 500K training pairs and train a lightweight network, which demonstrates significantly better generalization performance than existing models across various image enhancement tasks. Code and data are available at https://github.com/PolyU-VCLab/MESS.