Bilevel Layer-Positioning LoRA for Real Image Dehazing

AI summary

Overview

Research area: Real image dehazing (low-level vision), with a focus on parameter-efficient domain adaptation and vision-language guidance.

Technical level: Advanced. The method combines CLIP-based cross-modal losses, LoRA parameter-efficient fine-tuning, differentiable architecture search, and bilevel optimization with an implicit-function-theorem hypergradient estimator.

Scope: The paper proposes a haze-to-clear text-directed (H2C) unsupervised loss and a Bilevel Layer-Positioning LoRA (BiLaLoRA) adaptation strategy that automatically selects which network layers to adapt, targeting synthetic-to-real generalization in real image dehazing.

What This Paper Is About

Dehazing models trained on synthetic hazy images degrade noticeably when applied to real photographs, because paired clean ground-truth images are essentially unavailable in real scenes and fully fine-tuning a whole network is expensive. The paper's goal is an adaptation method that needs no paired real data and updates only a small number of parameters, while automatically deciding which layers in the network actually need adapting rather than relying on manual heuristics.

Key Contributions

1. The H2C loss. A haze-to-clear text-directed unsupervised loss built on CLIP's cross-modal capability, which reframes dehazing as a semantic alignment task in latent space and permits optimization without paired real haze/clear data.

2. BiLaLoRA. An adaptation strategy that treats adapter injection-layer selection and adapter weight optimization as a single bilevel optimization problem, automatically pinpointing and fine-tuning performance bottleneck layers without manual configuration.

3. Efficient synthetic-to-real transfer. BiLaLoRA achieves its transfer with minimal computational and storage overhead; its plug-and-play nature supports rapid switching across multiple target domains, balancing performance, efficiency, and flexibility.

4. Breadth of validation. The method is demonstrated across four dehazing architectures, four synthetic source datasets, and three real-world benchmarks, plus generalization tests on HazyDet, Dense-Haze, and O-Haze.

Main Findings

• H2C loss works across scenes and prompts. On the real-scene examples in Figure 2 (daytime rows from URHI and Fattal, nighttime rows from NHRW), the H2C loss improves pre-trained model outputs; for nighttime scenes the prompt is changed to "a photo with nighttime haze."

• The performance bottleneck shifts by architecture. After fine-tuning MSBDN and DEA with the H2C loss and grafting adapted modules back to measure MUSIQ improvements on RTTS, the encoder dominated, with its final block accounting for most of the gain — but the specific layers responsible differed substantially across architectures. The bottleneck is therefore dynamic, not static.

• Cross-model flexibility. BiLaLoRA was applied to MSBDN, DeHamer, ConvIR, and DEA, all uniformly pre-trained on THaze, and improved existing pre-trained models while automatically positioning bottleneck layers (Figure 4).

• Cross-domain stability. Using DEA as baseline across four synthetic datasets (RIDCP, ITS, OTS, Haze4K), BiLaLoRA consistently improved performance regardless of source domain (Figure 5), with visual confirmation in Figure 6 on Fattal data.

• Comparable performance at far lower training cost. In Table 1, full fine-tuning versus BiLaLoRA: FADE 0.610 vs. 0.638 (a 4.59% change), BIQME 0.617 vs. 0.611 (0.97% change), Entropy 7.569 vs. 7.572 (0.04% change), MUSIQ 64.43 vs. 64.40 (0.05% change). Training time dropped from 4.215 to 0.940 hours (77.70% reduction). Parameters went from 3.653M to 3.764M (3.03% increase), FLOPs from 34.04G to 34.08G (1.18% increase), and runtime from 3.702 ms to 3.735 ms (0.89% increase), with FLOPs and runtime measured on 256×256 input.

• Strong benchmark standing. Averaged over RTTS (4,322 images), URHI (150 images), and Fattal (31 images), BiLaLoRA reaches FADE 0.638, BIQME 0.611, Entropy 7.572, and MUSIQ 64.40, ranked first or second across key metrics against specialized dehazing methods and all-in-one restoration models. Per-dataset: RTTS (0.752, 0.611, 7.576, 61.77), URHI (0.881, 0.615, 7.599, 63.52), Fattal (0.281, 0.607, 7.541, 67.92).

• Qualitative advantages. Compared with DAD, D4, and KANet (which struggle with colored haze), PSD (overexposure and color shift), and PHATNet (visual artifacts), BiLaLoRA removes haze while better preserving fine details and natural appearance.

• Generalization under harder conditions. On HazyDet, BiLaLoRA recovers haze-obscured details without color distortion; on Dense-Haze and O-Haze, methods such as CoA and IPC deteriorate substantially, while BiLaLoRA maintains consistent performance.

• Ablation results. The full configuration (positive and negative prompts plus bilevel positioning) scores FADE 0.638, BIQME 0.611, Entropy 7.572, MUSIQ 64.40. Removing all components gives 1.018, 0.582, 7.438, 62.05. Using only the negative prompt (Q_e) yields 0.774, 0.561, 7.533, 60.47; bilevel positioning outperforms naïve joint learning in every matched pairing.

• Prompt asymmetry matters. Dropping the negative prompt drives outputs toward a single positive semantic target, producing color distortion artifacts; dropping the positive prompt leads to over-dehazing from excessive suppression of haze features.

• Three adapter layers is optimal. Performance improves steadily with more adaptation layers, peaks at three, then flattens with diminishing marginal returns, adding parameter redundancy and computational overhead.

Methodology in Plain English

The researchers start from a pre-trained dehazing network. For training on real data without ground truth, they encode both the hazy input and the network's dehazed output with CLIP's image encoder, and encode two text prompts — "a photo with haze" and "a clear photo" — with CLIP's text encoder. The direction from haze to clear in the text embedding space serves as the target; the loss (Equation 1) is simply 1 minus the cosine similarity between the image-space displacement and that target text direction. If the network's transformation points the right way semantically, the loss is small.

For efficiency, they add LoRA modules to the network. Each candidate LoRA module carries a learnable gating parameter α, squeezed into (0,1) by a sigmoid, that scales its contribution alongside a fixed scaling factor γ (Equation 2). Because the choice of which layers to adapt depends on how well the weights in those layers can be trained, the two are entangled — so the authors set it up as a bilevel problem: the lower level trains the LoRA weights, and the upper level chooses the architecture parameters based on validation performance. Computing the required hypergradient directly would need an expensive Hessian inverse, so they approximate it with a rank-one outer product (Equations 5–7) that uses only first-order derivatives.

Implementation runs in two stages. First, the bilevel layer-positioning stage alternates updates of α and the LoRA weights up to a switch epoch, after which the top-k layers are selected (k = 3 in the experiments). Then, in the LoRA fine-tuning stage, only the selected modules are trained. The experiments used DEA as baseline, Adam with β1 = 0.9, β2 = 0.999, ε = 1×10⁻⁸, pre-training on THaze with ℓ1 loss and a learning rate annealed from 1×10⁻⁴ to 1×10⁻⁶, and adaptation with learning rate 1×10⁻⁶, γ = 2, r = 8, on 500 real daytime haze images and 100 nighttime images from NHRW split equally into training and validation. Training data was augmented with random 256×256 crops, 90°/180°/270° rotations, and horizontal flipping, all on a single NVIDIA 4090 GPU.

Why This Matters

Impact on research. The paper argues that the layer most responsible for domain-gap performance loss is not fixed — it depends on architecture and scene — which challenges the common practice of hand-picking LoRA injection points. It also shows that a CLIP-based semantic direction can substitute for paired supervision in a low-level restoration task, and that bilevel optimization outperforms naïve joint architecture-and-weight learning in this setting.

Real-world applications:

• Autonomous driving and advanced driver-assistance systems operating in fog, haze, or smog.
• UAV and aerial imaging, directly evaluated here on the HazyDet UAV-perspective dataset.
• Surveillance and outdoor camera networks that need reliable visibility in poor weather.
• Nighttime photography and mobile imaging pipelines, where the paper trains a dedicated nighttime adapter rather than one monolithic model.

Industry relevance. The 77.70% reduction in training time, negligible inference overhead (1.18% FLOPs and 0.89% runtime increases on 256×256 input), and plug-and-play domain switching make this attractive for deployments where models must be re-targeted to new camera sites or weather regimes quickly and cheaply. Storing a small adapter per domain is far cheaper than storing a fully fine-tuned model per domain.

Future Directions

• Applying BiLaLoRA to other low-level vision tasks beyond dehazing, in diverse real-world scenarios, as stated in the concluding remarks.
• Developing more refined semantic guidance mechanisms to replace or extend the current two-prompt H2C formulation.
• Advancing cross-domain adaptation techniques to handle severe degradation conditions.
• An open question raised by the results: whether the rank-one hypergradient approximation remains accurate for much larger backbones, and whether the observed optimum of three adapter layers transfers to architectures not tested here.

Target Audience

Researchers and practitioners in low-level vision and image restoration, particularly those working on domain adaptation, unpaired/unsupervised dehazing, and parameter-efficient fine-tuning. It is also relevant to engineers deploying restoration models on constrained hardware, and to readers interested in applying vision-language models such as CLIP as supervisory signals for pixel-level tasks. The bilevel optimization formulation, with its implicit-function-theorem derivative and rank-one approximation, will interest readers working on differentiable architecture search.

Authors’ abstract

Learning-based real image dehazing methods have achieved notable progress, yet they still face adaptation challenges in diverse real haze scenes. These challenges mainly stem from the lack of effective unsupervised mechanisms for unlabeled data and the heavy cost of full model fine-tuning. To address these challenges, we propose the haze-to-clear text-directed loss that leverages CLIP's cross-modal capabilities to reformulate real image dehazing as a semantic alignment problem in latent space, thereby providing explicit unsupervised cross-modal guidance in the absence of reference images. Furthermore, we introduce the Bilevel Layer-positioning LoRA (BiLaLoRA) strategy, which learns both the LoRA parameters and automatically search the injection layers, enabling targeted adaptation of critical network layers. Extensive experiments demonstrate our superiority against state-of-the-art methods on multiple real-world dehazing benchmarks. The code is publicly available at https://github.com/YanZhang-zy/BiLaLoRA.