Leveraging Multispectral Sensors for Color Correction in Mobile Cameras

AI summary

Overview

Research area: Computational color imaging and computer vision, specifically camera color correction using multispectral sensing for mobile and consumer devices.

Technical level: Advanced. The paper assumes familiarity with camera image formation models, CIE XYZ and ProPhoto RGB color spaces, white balancing, and deep learning for image-to-image translation.

Scope (one sentence): The paper proposes a unified, learning-based framework that fuses a high-resolution RGB sensor with an auxiliary low-resolution multispectral sensor to perform the entire camera color correction pipeline in one model, and releases a purpose-built simulated dataset of 116,688 RGB–MS image triplets for training and evaluation.

What This Paper Is About

Camera color correction converts raw sensor values into accurate, device-independent color. Conventional pipelines split this into automatic white balancing (illuminant estimation plus illuminant discounting) followed by a color space transformation, and most multispectral approaches use the spectral data only for the first step and then discard it. The paper's goal is a single end-to-end model that keeps multispectral information in play throughout the entire pipeline, producing more color-accurate and more stable outputs on mobile-style camera modules.

Key Contributions

1. A dual-input, end-to-end color correction framework that fuses high-resolution RGB images with low-resolution multispectral images and jointly models illuminant estimation, illuminant discounting, and color space transformation in one network.
2. A physically grounded dataset of 116,688 RGB–MS image pairs with ground-truth color references, covering a wide range of illuminants and camera sensitivities, built by aggregating and repurposing public hyperspectral reflectance datasets and rendering them under multiple RGB camera sensitivities.
3. A misaligned variant of that dataset, produced by applying homographies sampled from the Zurich dataset (168 real misaligned image pairs) to the MS images, to emulate real cross-sensor spatial misregistration.
4. Refactoring of two lightweight image-to-image architectures — LPIENet and cmKAN — into the dual-input setting, demonstrating that the framework is not tied to a specific backbone.

Main Findings

• Large accuracy gains over RGB-only and MS-driven baselines: On the aligned dataset, the proposed models reach mean ΔE00 of 1.60 (cmKAN-light, mirrorless), 1.47 (cmKAN-light, mobile), 1.74 (LPIENet, mirrorless) and 1.66 (LPIENet, mobile), versus 3.17 and 3.16 for SpectralConvMean, the strongest multispectral baseline, and 3.25/3.19 for SpectralFC4. The paper reports that the proposed models achieve an average ΔE00 roughly 50% lower than competing methods, whose average performance is comparable to the worst-quartile (W-25) results of the proposed approaches.
• Statistical baselines are far behind: Gray-World reaches mean ΔE00 of 7.80 (mirrorless) and 7.71 (mobile); White-Patch reaches 5.08 and 5.04 respectively.
• Robustness to spatial misalignment: On the misaligned dataset, the proposed models lose only a small amount of accuracy (for example, LPIENet goes from 1.74 to 1.84 mean ΔE00 on mirrorless, cmKAN-light from 1.60 to 1.84), while remaining the top-performing methods by a significant margin. The MS illuminant estimation baselines are largely unaffected because their architectures strongly downsample the input.
• Multispectral data is the main driver of the improvement: Removing the spectral encoder raises mean ΔE00 for every proposed model — for example, LPIENet goes from 1.74 to 3.23 (mirrorless) and 1.66 to 3.29 (mobile), and cmKAN-light from 1.60 to 3.06 (mirrorless) and 1.49 to 3.03 (mobile). The paper states improvements of up to 50% over the RGB-only setup.
• Modest degradation under reduced exposure: Reproduction Error increases slightly at lower exposure (α = 0.75 and α = 0.5) but stays below the strongest baseline. For example, cmKAN-light moves from 2.91 to 2.98 to 3.18 on mirrorless, and 2.84 to 2.90 to 3.08 on mobile, while SpectralConvMean moves from 4.24 to 4.25 to 4.30, and 4.35 to 4.35 to 4.36.
• Model compactness: LPIENet uses approximately 220K trainable parameters in its original configuration and 60K in the "small" configuration; cmKAN-light uses only 18K trainable parameters.
• Qualitative confirmation: Visual comparisons show the framework produces perceptually more consistent results across scenes, illuminants, and cameras than FC4 and SpectralConvMean.

Methodology in Plain English

The problem is framed with the standard image formation model: what a sensor measures in each channel is the integral over wavelength of surface reflectance times the illuminant's spectral power distribution times that channel's spectral sensitivity. Because RGB cameras take only three measurements of this integral, the problem is underdetermined and reflectance and illumination cannot be cleanly separated. Multispectral sensors sample the spectrum more densely, adding constraints that make estimation of both reflectance and illumination more reliable.

The authors take two existing lightweight image-to-image networks and modify them to accept a second input:

• LPIENet follows a U-Net-like topology with three encoder blocks, two decoder blocks, skip connections, and a final refinement block, built around Inverted Residual Attention blocks. A separate spectral encoder mirroring the original encoder (three IRA blocks, no downsampling) processes the MS image, and its features are added element-wise to the RGB encoder features at the skip connections before each decoding stage.
• cmKAN uses a hypernetwork generator that predicts spatially varying spline parameters for a Kolmogorov-Arnold Network layer. It contains an Illumination Estimator, a Color Transformer with Multi-Scale Color Attention, and a Color Feature Modulator. A compact spectral encoder of three 3×3 convolutional layers with ReLU activations injects spectral features at two feature levels via element-wise addition.

Because accurate color ground truth cannot be obtained from RGB or MS captures alone, the authors simulate it. They start from two public hyperspectral reflectance datasets with densely sampled spectra (which already contain realistic noise, blur, and acquisition artifacts), mask calibration patches, and manually remove corrupted or incomplete data, leaving 1,144 valid scenes. Each scene is re-illuminated under 102 illuminants from the Barnard et al. dataset. RGB images are rendered at 512×512 pixels using sensitivities of Google Pixel 3, iPhone Xs Max, Huawei Mate 20 Pro, Samsung Galaxy Note 9, Canon R5, Sony α9 III, and Nikon Zf. MS images are rendered at 64×64 pixels using the Spectricity S1 sensor, which has 15 narrowband channels across the visible range. Ground truth is rendered in CIE XYZ under the canonical D65 illuminant. Splits are done scene-wise to prevent leakage: 80% of scenes for training and 20% for testing, with 20% of the training subset reserved for validation.

Training uses the Adam optimizer with a learning rate of 1×10⁻⁴, early stopping with a maximum of 300 epochs and patience of 5, and cosine annealing scheduling. Illuminant estimation networks are optimized on Angular Error, while image-to-image models are trained on ΔE76. Evaluation uses ΔE00 and Reproduction Error.

For a fair comparison, the baselines — Gray-World, White-Patch, Shades-of-Gray, General Gray-World, first- and second-order Gray-Edge, FC4, ConvMean, and QU — are treated as illuminant estimation methods only, and the remaining pipeline steps follow the traditional correction procedure: von Kries adaptation for discounting, then a 3×3 color space transformation matrix interpolated between presets calibrated at 2500°K and 6500°K using Macbeth ColorChecker targets.

Why This Matters

The paper argues that prior work separates the stages of color correction and often throws away multispectral data after illuminant estimation, and that recent joint approaches operate directly on multispectral images, which limits them on mobile platforms where the MS sensor has low spatial resolution and is only an auxiliary modality. By keeping spectral cues active through every stage and by decoupling the framework from any specific backbone, the work suggests future, more powerful architectures can be dropped into the same formulation.

Real-world applications:

• Smartphone and consumer camera photography, where a compact snapshot multispectral sensor can complement a primary high-resolution RGB camera across many lighting conditions.
• Retail and e-commerce product imaging, where accurate device-independent color reproduction under uncontrolled lighting matters commercially.
• Augmented and mixed reality capture, where mismatched colors between virtual and real content are conspicuous.
• Industrial and automotive vision, where color fidelity under varied illumination and imperfect multi-sensor alignment must be maintained.

Industry relevance: The dataset was rendered under the spectral sensitivities of actual shipping mobile devices (Google Pixel 3, iPhone Xs Max, Huawei Mate 20 Pro, Samsung Galaxy Note 9) and mirrorless cameras (Canon R5, Sony α9 III, Nikon Zf), and the MS data uses the Spectricity S1 sensor. The models are deliberately lightweight (18K to 220K parameters), and results are reported separately for mobile and mirrorless camera categories, indicating a deployment-oriented framing. Code, models, and dataset are released publicly.

Future Directions

• Interpretability and intermediate outputs: The authors state as a limitation that the end-to-end design does not rely on explicit intermediate estimates such as illuminant or reflectance, offering less diagnostic or reusable information than modular pipelines. Recovering interpretable intermediates without sacrificing accuracy is an open question.
• Stronger backbones: The authors state that any future backbone, if more powerful, can be seamlessly integrated into the same formulation to further enhance performance, implying architecture search and larger models as a natural extension.
• Practical robustness beyond exposure and misalignment: Only two robustness dimensions are tested here (exposure scaling at α = 0.75 and α = 0.5, and simulated homography misalignment). Other real-world perturbations are not evaluated.
• Bridging simulation and reality: The dataset is entirely simulated from hyperspectral reflectance renderings because no real dataset currently satisfies the requirements of the task. Validating on real dual-sensor captures, and closing the simulation-to-reality gap, remains open.

Target Audience

Researchers and engineers working on camera image signal processing pipelines, computational photography, and color science, particularly those interested in multispectral or spectral sensing on constrained hardware. It is also relevant to practitioners in mobile imaging and smartphone camera development, and to readers interested in end-to-end learned alternatives to modular color correction. Readers without background in color spaces, white balancing, or image-to-image networks will need to consult the cited references to follow the quantitative results.

Authors’ abstract

Recent advances in snapshot multispectral (MS) imaging have enabled compact, low-cost spectral sensors for consumer and mobile devices. By capturing richer spectral information than conventional RGB sensors, these systems can enhance key imaging tasks, including color correction. However, most existing methods treat the color correction pipeline in separate stages, often discarding MS data early in the process. We propose a unified, learning-based framework that performs end-to-end color correction and jointly leverages data from a high-resolution RGB sensor and an auxiliary low-resolution MS sensor. Our approach integrates the full pipeline within a single model, producing coherent and color-accurate outputs. We demonstrate the flexibility and generality of our framework by refactoring two different state-of-the-art image-to-image architectures. To support training and evaluation, we construct a dedicated dataset by aggregating and repurposing publicly available spectral datasets, rendering under multiple RGB camera sensitivities. Extensive experiments show that our approach improves color accuracy and stability, reducing error by up to 50% compared to RGB-only and MS-driven baselines. Code, models and dataset available at: https://lucacogo.github.io/Mobile-Spectral-CC/.