RobuMTL: Enhancing Multi-Task Learning Robustness Against Weather Conditions

AI summary

Overview

Research area: Computer vision / multi-task learning robustness under adverse weather (semantic segmentation, surface normals, saliency, human part segmentation, edge detection, depth estimation).

Technical level: Advanced. The paper assumes familiarity with multi-task learning, LoRA low-rank adaptation, mixture-of-experts routing, and Swin Transformer backbones.

Scope: The paper introduces RobuMTL and RobuMTL+, parameter-efficient architectures that dynamically select hierarchical LoRA experts based on detected input perturbations, and evaluates them on PASCAL and NYUD-v2 against single-task, MTL, and state-of-the-art baselines.

What This Paper Is About

Multi-task learning (MTL) models that share one encoder across several vision tasks degrade badly when images are corrupted by rain, snow, fog, noise, or blur, and most existing fixes (adversarial training, restoration pre-processing, static inference paths) either hurt clean accuracy or add heavy latency. The authors build a system where a small classifier inspects the input image, scores which weather/degradation types are present, and then blends the matching lightweight LoRA expert modules into the frozen MTL backbone before inference. The stated goal is to gain robustness under adverse and mixed weather conditions without sacrificing clean-data performance.

Key Contributions

1. RobuMTL pipeline: A method that dynamically selects and applies lightweight hierarchical LoRA experts to the encoder based on input characteristics, using an MoE-inspired expert aggregation strategy to mitigate weather-induced degradation.
2. RobuMTL+ extension: Incorporates dynamically selected modular adaptations in which perturbation-specific hierarchical LoRA experts are complemented by task-adaptive expert modules per perturbation (an "expert squad").
3. Dynamic Modular LoRA Selector (DMLS): A lightweight policy network (48,742 parameters) that assigns importance scores to LoRA experts and parameter experts based on the detected perturbation type, invoked once before the MTL backbone rather than at every layer.
4. Empirical analysis and comparison: An evaluation of traditional MTL and single-task models under several adverse weather conditions on two datasets, plus comparison against state-of-the-art methods, including a mix of rank configurations and ablation of each architectural component.

Main Findings

• PASCAL gains: RobuMTL delivers a +2.8% average relative improvement under single perturbations and up to +44.4% under mixed weather conditions compared to the MTL baseline. In Table 1, RobuMTL+ reaches Δm_adv of +2.80% and Δm_clean of +13.50%, versus −2.60% and +7.70% for MTL-Full FT.
• NYUD-v2 gains: RobuMTL+ achieves a +9.70% Δm on NYUD-v2 across tasks (Table 3), compared with −3.20% for MTLoRA and −22.10% for MTLMoE.
• Absolute task scores on PASCAL (Table 1): RobuMTL+ records 63.86 mIoU semantic segmentation, 18.17 RMSE normals, 62.64 mIoU saliency, and 55.01 mIoU human parts, against 61.61 / 18.97 / 61.97 / 53.78 for the Single Task baseline.
• Parameter and compute efficiency: RobuMTL achieves a 3.64× reduction in parameters and 3.43× lower computation cost over single-task training. RobuMTL+ uses 30.92M parameters and 21.50 G FLOPs at 57 FPS on an NVIDIA V100 (batch size = 1), versus 112.60M parameters, 73.70 G FLOPs, and 65 FPS for the single-task model.
• Clean-data behavior: On clean data, clean RobuMTL(+) improves accuracy by +3.18% over the clean single-task baseline.
• Optimal expert count: Top-K selection shows k=1 works best for single perturbations, while k=6 is optimal for mixed conditions. In Table 5, Δ_adv under single perturbations falls from +3.10 at k=1 to −3.10 at k=6, while mixed-condition Δ_adv rises from +44.00 to +44.40 across k=1 to k=6.
• Mixed weather (rain + fog, PASCAL, Table 4): RobuMTL+ reports 64.78 mIoU semantic segmentation, 64.18 saliency, 19.05 normals RMSE, 54.99 mIoU human parts, and Δm of +44.40, compared with +16.24 for MTLoRA and 0 for the single-task model.
• Router quality: Conventional MoE routers fail to differentiate perturbation patterns; replacing them with DMLS yields up to a 60% performance boost (Table 6: MTLMOE+ours Δm of −1.4, STMOE+ours Δm of +0.6).
• DMLS classification performance (Table 8): Accuracy 94.44%, precision 93.36%, recall 92.58%, F1-score 92.65%, with 48,742 parameters. The paper notes a 6% difference in accuracy affecting snow-versus-clean discrimination, mitigated by batch-level voting that reaches 99.9% accuracy.
• Component ablation (Table 7): Adding DMLS with SE but without MEPF yields Δm of −5.71; adding MEPF raises it to +1.5; adding Squad reaches +2.8. The base MTL + clean LoRA expert starts at −12.78.
• Rank configuration: Forward hierarchical rank configurations outperform uniform and reverse configurations. The r[8,16,32,64] setup delivers the highest relative gains under perturbations across all tasks, while r[16,32,64,128] is the best balance of clean-scene performance and robustness and is the configuration selected for Table 1. PASCAL uses (16,32,64,128) and NYUD-v2 uses a hierarchy starting at 64 (64,128,256,512).
• Backbone scaling (Table 2, Swin-B/ViT-B): RobuMTL+ reaches Δm of +5.00 with 63.50 G FLOPs and 101.62M parameters, versus +1.40 for both TaskPrompter (1134.08 G FLOPs, 418.33M parameters) and MLoRE (890.90 G FLOPs, 259.13M parameters).
• Comparison against restoration-based inference: The paper notes UIR-LoRA has 95.2M parameters and that incorporating such an approach raises total complexity to 125.256M parameters including 30M MTL parameters, motivating the parameter-efficient design instead.
• Cross-dataset generalization: DMLS is trained on one dataset and generalizes to another without retraining; Figure 4 shows diagonal dominance in expert activations on NYUD-v2.

Methodology in Plain English

The starting point is a pre-trained MTL model with a shared Swin-ViT encoder and small task-specific decoders. Instead of fine-tuning the whole thing, the authors insert LoRA low-rank adapter matrices into the attention, projection, and feed-forward layers, following the placement used by MTLoRA. Their twist is a hierarchical rank schedule: small ranks in early layers and larger ranks in deeper layers, an "inverted-triangle" layout that keeps early layers from overfitting low-level degradation while giving deeper layers more capacity to learn semantics. Adapters are applied only in the final stage of the Swin Transformer.

A separate, small CNN classifier called the Dynamic Modular LoRA Selector (DMLS) looks at the input image and predicts which perturbation classes are present. It has three parts: a depthwise-separable convolutional backbone that expands features to 128 channels, a squeeze-and-excitation block that reweights channels (with softmax replacing the usual sigmoid), and two fully connected layers producing a probability distribution over perturbation classes via softmax. It is trained with a cross-entropy loss against known synthetic perturbation labels. Because it runs once before the backbone, it avoids the instability and overhead of routing at every layer.

The Multiple Expert Parameter Fusion (MEPF) module then uses those scores to blend the top-K selected LoRA experts by weighted averaging, with normalization by the sum of the selected scores. Expert squads in RobuMTL+ bundle each perturbation-specific LoRA with its own normalization and decoder head parameters. Two training strategies are described: RobuMTL trains one hierarchical LoRA expert per perturbation type (rain, snow, noise) with a shared decoder, warming up on clean data and mixing in a small portion of other data; RobuMTL+ additionally trains paired task adaptation modules per perturbation. During training, non-relevant experts are deactivated so only the needed expert updates. Evaluation uses PASCAL and NYUD-v2, with corrupted replicas generated through the imgaug library simulating Gaussian noise, average/motion/Gaussian blur, rain, snow, and fog. Tasks are scored with mIoU for segmentation-type tasks and RMSE for normals/depth, with edge detection reported as error, all aggregated into a Δm metric relative to single-task baselines.

Why This Matters

Research impact. The paper claims to be the first work investigating robustness for MTL under adverse weather conditions, and reframes robustness not as a full fine-tuning problem but as a routing and parameter-selection problem. It also argues that most of the performance drop in standard multi-task MoE models comes from router failure rather than expert capacity, which points the field toward better routers instead of bigger expert pools.

Real-world applications:

• Autonomous vehicles and driving systems, which the paper explicitly names as a target setting where distribution shifts from adverse weather or sensor noise degrade performance.
• Virtual reality systems, cited alongside autonomous vehicles as needing simultaneous execution of multiple tasks under constrained resources.
• Robotics and embedded perception platforms, where the 3.64× parameter reduction and 3.43× lower computation cost over single-task training matter for deployment.
• Any deployment needing segmentation plus depth/normals plus saliency from one model under varying image quality, including surveillance and outdoor monitoring where weather is uncontrolled.

Industry relevance. The efficiency comparison is the practical hook: RobuMTL+ runs at 57 FPS with 30.92M parameters and 21.50 G FLOPs, while the paper positions restoration-based pre-processing pipelines as adding latency and memory unsuitable for real-time MTL. Comparisons against TaskPrompter and MLoRE on Swin-B show roughly 63.50 G FLOPs and 101.62M parameters versus 1134.08 G and 418.33M, and 890.90 G and 259.13M respectively. The authors also note that LoRA squad members are aggregated on the CPU and only the aggregated squad is transferred to the GPU at inference, which is a concrete deployment detail.

Future Directions

• Extending RobuMTL to real-world weather datasets, which the conclusion names as the planned next step beyond synthetic perturbations generated with imgaug.
• Improving discrimination between snow and clean images, where the DMLS shows a 6% accuracy difference attributed to shared properties and small snowflake patterns.
• Better handling of the single-perturbation versus mixed-perturbation trade-off, since merging all experts (k=6) reduces performance on single perturbations while improving mixed-condition robustness.
• Revisiting rank allocation policy, given that r[8,16,32,64] gives the highest robustness but lower clean accuracy, while r[16,32,64,128] is chosen as the middle ground; the paper states a full breakdown of this trade-off lives in supplementary materials rather than the main text.
• The paper compares several task-conflict methods (NashMTL, PCGrad, CAGrad) and notes gains, but does not report combining those conflict-resolution objectives with the DMLS routing scheme, leaving that combination open.

Target Audience

Researchers and practitioners working on multi-task learning, parameter-efficient fine-tuning, mixture-of-experts routing, and perception robustness for autonomous systems. It is most useful to readers already comfortable with LoRA mechanics and MoE routing who want a concrete, efficiency-focused recipe for making shared multi-task backbones resilient to weather degradation. Engineers deploying perception models on resource-constrained hardware will also find the FPS, parameter, and FLOPs comparisons directly relevant.

Authors’ abstract

Robust Multi-Task Learning (MTL) is crucial for autonomous systems operating in real-world environments, where adverse weather conditions can severely degrade model performance and reliability. In this paper, we introduce RobuMTL, a novel architecture designed to adaptively address visual degradation by dynamically selecting task-specific hierarchical Low-Rank Adaptation (LoRA) modules and a LoRA expert squad based on input perturbations in a mixture-of-experts fashion. Our framework enables adaptive specialization based on input characteristics, improving robustness across diverse real-world conditions. To validate our approach, we evaluated it on the PASCAL and NYUD-v2 datasets and compared it against single-task models, standard MTL baselines, and state-of-the-art methods. On the PASCAL benchmark, RobuMTL delivers a +2.8% average relative improvement under single perturbations and up to +44.4% under mixed weather conditions compared to the MTL baseline. On NYUD-v2, RobuMTL achieves a +9.7% average relative improvement across tasks. The code is available at GitHub.