Rethinking Nighttime Image Deraining via Learnable Color Space Transformation

AI summary

Overview

• Research area: Computer vision, specifically nighttime image deraining (removing rain streaks and raindrops from images captured in low-light scenes).
• Technical level: Intermediate. The paper assumes familiarity with color spaces, convolutional/Transformer encoder-decoder architectures, and standard restoration metrics, but its central ideas can be grasped without deep mathematical background.
• Scope: The paper introduces a new nighttime rainy-image benchmark (HQ-NightRain) built with an illumination-aware synthesis pipeline, and a two-stage network (CST-Net) that performs deraining in the Y channel of a learnable YCbCr color space rather than in RGB.

What This Paper Is About

Nighttime rain is hard to remove because rain is only visible near artificial light sources such as streetlights and headlights, and because existing synthetic datasets add rain uniformly across the whole image, making them unrealistic. The authors argue that prior nighttime deraining methods still operate in RGB even though nighttime rain stands out far more in the luminance (Y) channel of YCbCr. Their goal is therefore twofold: build a more realistic nighttime rainy benchmark, and design a network that performs deraining in a color space it learns for itself.

Key Contributions

1. A new benchmark, HQ-NightRain. The authors construct a high-quality nighttime deraining dataset whose rain is distributed non-uniformly according to illumination, rather than added linearly and uniformly as in prior work. It contains 11,200 image pairs (10,000 training, 900 validation, 300 testing), split into rain streak (RS), raindrop (RD), and mixed rain streak plus raindrop (SD) subsets, with backgrounds selected from BDD100K using images labeled "night." A real-captured subset of 512 images and a 20-pair natural-scene synthetic night rain subset are also provided.

2. A learnable color space transformation framework (CST-Net). The method removes degradation in the Y channel of YCbCr instead of RGB, using a learnable Color Space Converter (CSC) whose conversion weights are not fixed constants but learnable variables passed through an MLP.

3. Implicit Illumination Guidance (IIG). An implicit neural representation branch encodes illumination information from pixel coordinates to guide rain removal in complex nighttime scenes, without relying on an explicit illumination model.

4. Broad empirical validation. Experiments are reported on HQ-NightRain, GTAV-NightRain, RealRain-1k, RainDS-real, Multi-Weather6K, and BDD350-Night, covering synthetic, real-world, multi-weather, and downstream detection settings.

Main Findings

• Y channel separates nighttime rain best. Histogram analysis shows the Y channel in YCbCr has the most significant difference between rainy and rain-free nighttime images relative to other channels, which motivates doing deraining in that channel. In daytime scenes the equivalent histogram difference is described as relatively small.

• Strong results on the proposed benchmark. On HQ-NightRain, CST-Net reaches an average of 39.0652 PSNR, 0.9778 SSIM, and 0.0477 LPIPS. On the mixed SD subset it reaches 40.4984 PSNR, 0.9881 SSIM, 0.0248 LPIPS, which the paper states is 0.81 dB PSNR above the state-of-the-art comparison method NeRD-Rain.

• Best reported performance on GTAV-NightRain. CST-Net scores 38.9378 PSNR, 0.9786 SSIM, 0.0320 LPIPS on the public GTAV-NightRain set1 dataset, which the paper describes as 1.1 dB above NeRD-Rain.

• Generalization to real data. On RealRain-1k, CST-Net reports 27.3064 PSNR / 0.8891 SSIM / 0.3805 LPIPS; on RealRain-1k-L 25.0456 / 0.7065 / 0.2715; on RealRain-1k-H 22.7280 / 0.6499 / 0.3479; and on RainDS-real 23.8114 / 0.8062 / 0.4877.

• The dataset itself improves real-world transfer. With IDT as the test model, training on HQ-NightRain gives 26.94 PSNR / 0.8873 SSIM when tested on RealRain-1k-L, versus 26.47 / 0.8640 for GTAV-NightRain and 25.63 / 0.8408 for RoadScene-rain.

• The learnable CSC and IIG both help. The full model reaches 40.4984 PSNR / 0.9881 SSIM on HQ-NightRain; removing the second stage drops it to 35.0858 / 0.9650, removing the first stage to 36.4385 / 0.9740, and keeping both stages but dropping IIG to 39.8767 / 0.9866.

• YCbCr beats other color spaces. In the ablation comparing color spaces with a fixed converter, RGB gives 38.7507 PSNR / 0.9838 SSIM, HSV 39.0317 / 0.9843, HSL 39.1613 / 0.9844, YUV 39.1932 / 0.9846, and YCbCr with a fixed converter 39.5959 / 0.9857, all below the learnable CSC variant.

• Perceptual metrics on natural scenes. On the natural-scene subset using non-reference metrics, CST-Net reports PaQ-2-PiQ 63.8756 and MANIQA 0.5079, compared with IDT (63.4735 / 0.5056), Restormer (63.7980 / 0.5010), and MSGNN (56.8076 / 0.4483).

• Extension to multi-weather restoration. On the Multi-Weather6K dataset, CST-Net reports 33.82 PSNR / 0.9642 SSIM, versus Restormer 31.80 / 0.9228, TransWeather 30.75 / 0.9468, and PromptIR 31.69 / 0.9169.

• Synthesis pipeline components matter. Ablating the data pipeline with CST-Net, linear addition (D1) yields 25.38 PSNR / 0.8705 SSIM, convolutional merge alone (D2) 28.58 / 0.9456, and convolutional plus illumination merge (D3) 36.55 / 0.9754. The full pipeline including the defocus blur term is reported at 31.91 / 0.9493; the paper frames this as a trade-off where simulating more challenging nighttime scenarios yields a balanced improvement.

• Downstream detection. Using a pre-trained YOLOv8 model on BDD350-Night, the authors report that their method achieves the highest precision, recall, and IoU among the compared approaches; specific numeric values are not reported in the provided content.

Methodology in Plain English

The work proceeds in two parts.

Building a more realistic dataset. The authors note that nighttime light radiates outward from artificial sources and, following the first law of illumination (illuminance falls off with the square of distance), rain is only visible near those sources. Their synthesis pipeline therefore extracts an illumination coefficient matrix from the background image: it converts the background from RGB to HSV, takes the V channel, and normalizes it. A masking operation then suppresses both very dark areas and the brightest cores of light sources, using illumination thresholds τ1 = 0.2 and τ2 = 0.8. The rain streak and raindrop masks are multiplied by this illumination matrix so that rain appears only where it would realistically be lit. A defocus blur is also applied to the background to mimic refraction and scattering through raindrops, and the background and rain mask are merged with a 3×3 convolution rather than simple addition.

Removing the rain. CST-Net has two stages: a degradation removal stage and a color refinement stage, both built on a 4-level Transformer-based encoder-decoder. A learnable Color Space Converter first transforms the input RGB image into YCbCr and splits the channels. The Y channel goes into the degradation removal stage, while the Cb and Cr channels are held back for the color refinement stage. Where a standard RGB-to-YCbCr conversion uses fixed constants (0.299, 0.587, 0.114, and so on), the CSC instead uses a 3×3 matrix of learnable one-dimensional variables that are passed through an MLP, allowing the transformation to adapt to each scene. Between the two stages, an Implicit Illumination Guidance module encodes features from a shared encoder and combines them with pixel coordinates weighted by distance from the central pixel, then decodes through an MLP that predicts Y-channel values rather than RGB values. The result is converted back to RGB and refined in the second stage.

Training setup. Training used an NVIDIA GeForce RTX 3090 GPU, the Adam optimizer with default parameters, an initial learning rate of 2×10⁻⁴ cosine-annealed down to 1×10⁻⁶, 500 epochs, 128×128 patches, and a batch size of 4. Evaluation uses PSNR, SSIM, LPIPS, PaQ-2-PiQ, and MANIQA. Comparisons are made against 11 deraining methods, including PReNet, RCDNet, SPDNet, MPRNet, IDT, Restormer, SFNet, DRSformer, RLP, MSGNN, and NeRD-Rain.

Why This Matters

Impact on research. The paper challenges two assumptions in nighttime deraining: that rain should be synthesized uniformly, and that deraining should be performed in RGB. It provides a benchmark intended to reduce the domain gap between synthetic and real nighttime rainy images, plus a concrete mechanism (learnable color space conversion) that other restoration tasks could adopt.

Real-world applications mentioned in the paper:

• Autonomous driving, where nighttime rain degrades the images used for perception.
• Video surveillance, which the paper cites alongside driving as a downstream task affected by nighttime rain.
• Object detection, evaluated using a pre-trained YOLOv8 model on the BDD350-Night set.
• Film and game production, where the authors apply their rain synthesis pipeline to generate realistic rainy scenes.

Industry relevance. Because the dataset is derived from BDD100K, a driving-focused dataset, and because the authors explicitly test on a detection model, the work targets the automotive perception pipeline. The multi-weather extension to Multi-Weather6K suggests the approach may be attractive for systems that must handle several degradation types with one model, and the low-light luminance-channel emphasis is relevant to camera systems operating at night.

Future Directions

• Reconciling PSNR and realism in the synthesis pipeline. The ablation shows the full pipeline at 31.91 PSNR / 0.9493 SSIM while the variant without defocus blur scores 36.55 / 0.9754, so how to keep high perceptual realism without losing quantitative performance remains open.
• Evaluating the promised downstream gains numerically. The detection experiment on BDD350-Night is described qualitatively; reporting precision, recall, and IoU values would let others compare directly.
• Extending beyond YCbCr. The paper tests RGB, HSV, HSL, YUV, and YCbCr; whether other learnable spaces, such as the LAB space mentioned in the motivation analysis, could perform similarly or better is untested.
• Scaling and generalizing the benchmark. HQ-NightRain's real-captured subset contains 512 images and its natural-scene synthetic subset only 20 pairs, so broadening real-world coverage and validating across more sensors and geographies is a natural next step.

Target Audience

Researchers and graduate students working on image restoration, low-light vision, and adverse-weather perception; engineers building nighttime camera pipelines for autonomous driving or surveillance; and anyone who needs a nighttime deraining dataset or a strong baseline on GTAV-NightRain, RealRain-1k, or RainDS-real. Readers should be comfortable with standard restoration metrics and with encoder-decoder network terminology, but the paper's core insight about the Y channel and illumination-aware rain synthesis is accessible without deep mathematical background.

Authors’ abstract

Compared to daytime image deraining, nighttime image deraining poses significant challenges due to inherent complexities of nighttime scenarios and the lack of high-quality datasets that accurately represent the coupling effect between rain and illumination. In this paper, we rethink the task of nighttime image deraining and contribute a new high-quality benchmark, HQ-NightRain, which offers higher harmony and realism compared to existing datasets. In addition, we develop an effective Color Space Transformation Network (CST-Net) for better removing complex rain from nighttime scenes. Specifically, we propose a learnable color space converter (CSC) to better facilitate rain removal in the Y channel, as nighttime rain is more pronounced in the Y channel compared to the RGB color space. To capture illumination information for guiding nighttime deraining, implicit illumination guidance is introduced enabling the learned features to improve the model's robustness in complex scenarios. Extensive experiments show the value of our dataset and the effectiveness of our method. The source code and datasets are available at https://github.com/guanqiyuan/CST-Net.