Causal-Tune: Mining Causal Factors from Vision Foundation Models for Domain Generalized Semantic Segmentation

AI summary

Overview

Research area: Computer vision — Domain Generalized Semantic Segmentation (DGSS), parameter-efficient fine-tuning (PEFT) of Vision Foundation Models (VFMs), and frequency-domain causal representation learning.

Technical level: Advanced. The paper assumes familiarity with transformer-based VFMs (DINOv2, CLIP, EVA02, SAM), adapter-style fine-tuning, attention mechanisms, and spectral transforms (DCT, FFT, HWT).

Scope (one sentence): The paper identifies artifacts in VFM features as non-causal factors living in the low- and high-frequency bands of the feature spectrum, and proposes Causal-Tune, a fine-tuning strategy that filters those bands out with a Gaussian band-pass filter and refines the remaining causal components with learnable tokens in the frequency domain.

What This Paper Is About

Fine-tuning a large frozen Vision Foundation Model with a small number of trainable parameters works well for semantic segmentation on unseen domains, but the paper observes that long-term pretrained VFMs produce artifacts in their features — and existing adapter-based methods leave those artifacts in place because they fine-tune all feature components indiscriminately. The authors reframe these artifacts as non-causal factors that sit in the extreme low- and high-frequency parts of the feature spectrum, and argue that only the remaining "causal" band carries the domain-invariant structure needed for generalization. The goal is a fine-tuning method that explicitly separates the two and keeps only the causal part.

Key Contributions

1. A causal-perspective investigation of Parameter-Efficient Fine-Tuning of VFMs for Domain Generalized Semantic Segmentation, motivated by the observation that frozen DINOv2 features contain artifacts that persist after existing adapter-based fine-tuning.
2. Causal-Tune, a fine-tuning strategy that mines causal factors and removes non-causal factors from VFM features using a Gaussian band-pass filter applied to the Discrete Cosine Transform (DCT) spectrum, with non-causal components discarded.
3. A set of causal-aware learnable tokens that operate in the frequency domain to refine the retained causal components before an inverse DCT returns the refined features to the spatial domain for the next layer.
4. Extensive cross-domain experiments showing effectiveness, including +4.8% mIoU over the baseline in the 'Snow' condition; code is released at https://github.com/zhangyin1996/Causal-Tune.

Main Findings

• Adverse-weather gains dominate: Under Cityscapes (C.) → ACDC, Causal-Tune improves over the baseline Rein by +0.3 (Night), +4.8 (Snow), +1.8 (Fog), and +2.7 (Rain), for a +2.4 average (72.0% vs. 69.6%). The 'Snow' improvement runs from 70.6% to 75.4%, and 'Rain' from 72.5% to 75.2%.
• Best results on three of four ACDC conditions: Causal-Tune achieves the best results on 'Snow' (75.4), 'Fog' (81.3), and 'Rain' (75.2), outperforming SET and FADA by (1.8% & 1.9%), (1.2% & 1.1%), and (0.4% & 0.2%) respectively.
• Real-to-real transfer sets new state of the art: C. → BDD100K (B.) reaches 66.28% and C. → Mapillary (M.) reaches 76.05%, versus baseline Rein at 63.54% and 74.03% (+2.74 and +2.02).
• Synthetic-to-real is weaker: For GTA5 (G.) → Cityscapes the method scores 66.22%, a −0.18 change relative to baseline Rein (66.40) and below FADA (68.23); the authors attribute this to difficulty mining causal factors under synthetic-to-real generalization. G. → B. gives 61.80% and G. → M. gives 68.21% (best result on G. → M., versus FADA).
• Overall ranking against VFM-based methods: The paper states Causal-Tune achieves the best results in 7 out of 10 cases and second-best in 1 case among VFM-based methods, and outperforms all ResNet-based and Transformer-based methods compared.
• DCT beats FFT and HWT: In the frequency-transform ablation on C. → ACDC, DCT reaches 72.0 average versus FFT 69.5 and HWT 69.2. FFT is described as simply better at removing the non-causal factor of 'Night' (57.1 for FFT vs. 56.2 for DCT), while HWT is described as unable to reveal causal and non-causal factors well.
• Both frequency extremes matter: Removing only low frequencies improves 'Night' (57.3) and 'Snow' (72.1) but degrades other conditions (68.4 average); removing only high frequencies helps 'Snow', 'Fog', and 'Rain' (70.4 average); the band-pass filter (removing both low and high) gives the best average, 72.0.
• Non-causal factors are condition-specific in the spectrum: 'Fog' and 'Rain' non-causal factors concentrate in high frequencies, 'Night' in low frequencies, and 'Snow' in both low and high frequencies.
• Cutoff frequency sensitivity: The accuracy matrix under C. → ACDC is reported as showing better generalization when the low cutoff is 0.2 and the high cutoff is 0.8, with degradation as both increase; the paper sets the default to R_L = 0.2 and R_H = 0.7. The conclusion explicitly notes the model is somewhat sensitive to these cutoff frequencies.
• Artifacts are visually reduced across VFMs: Feature-map visualization shows that fine-tuning with Causal-Tune removes artifacts in DINOv2 and CLIP and significantly reduces the artifacts in EVA02.

Methodology in Plain English

The pipeline sits between the frozen layers of a VFM, in the same place an adapter would go. For each layer, the output feature is converted into the frequency domain with the Discrete Cosine Transform. A Gaussian band-pass filter then splits the spectrum into two parts using a low cutoff and a high cutoff (R_L = 0.2, R_H = 0.7 by default): the extremes are treated as non-causal and thrown away, while the middle band is treated as causal and kept.

The kept causal band is then refined. A learnable token per layer, built as the product of two small matrices, plays the role of Key and Value in an attention operation where the causal frequency features are the Query. The token is projected by one MLP, added back through a residual connection, and passed through a second MLP with another residual connection to produce the refined causal features. An inverse DCT converts those refined features back to the spatial domain, and they are passed to the next VFM layer.

The design rests on an empirical observation: when various non-causal factors are injected into images, then transformed by DCT, filtered at high and low frequencies, and inverted, the injected factors concentrate in the extreme bands, while the residual middle band retains structural and textural patterns that stay stable across domains. The default backbone is DINOv2 with Mask2Former as segmentation head, trained with AdamW at a base learning rate of 1e-4 for 40000 iterations, batch size 4, and 512×512 inputs, implemented in MMSegmentation on a single RTX3090 GPU using 14 GB of memory. Experiments are averaged over three independent repetitions.

Why This Matters

Impact on research: The paper connects two lines of work that had been separate — PEFT of VFMs for DGSS and causal-mechanism domain generalization. It challenges the assumption that adapter-based fine-tuning improves features uniformly, showing instead that long-pretrained VFMs carry redundancy that adapters leave behind, and it argues frequency-based methods using FFT or HWT miss implicitly non-causal factors that DCT can separate. It also supplies an ablation over filtering choices (low only, high only, band-pass) that maps which weather conditions live in which part of the spectrum.

Real-world applications:

• Autonomous driving and driver assistance in adverse weather — night, snow, fog, and rain — which is exactly the ACDC setting where the method's largest reported gain (+4.8 mIoU on Snow) appears.
• Sim-to-real deployment, where a model is trained on synthetic data from a game engine (GTA5, 24966 images) and must run on real street scenes (Cityscapes, 2975 training images) — the paper notes this setting is where the approach is weakest.
• Large-scale street-level mapping and urban scene understanding, relevant to the Mapillary (18000 training / 2000 validation images) and BDD100K (70k training / 10k validation images) benchmarks used.
• Any deployment of a frozen foundation model that must be adapted to a new camera, city, or weather regime without retraining the backbone.

Industry relevance: Because only lightweight tokens and MLPs are trained while the VFM stays frozen, adaptation is cheap — the paper reports fitting the whole setup in 14 GB on a single RTX3090. For teams shipping perception systems across many regions and conditions, a per-layer frequency filter plus a small token set is a far lower-cost adaptation route than full fine-tuning or collecting labeled data for every target domain.

Future Directions

• Dynamic cutoff frequencies. The conclusion names this directly: generalization is somewhat sensitive to the high and low band-pass cutoff frequencies, and the authors plan a dynamic cutoff so the model adapts across conditions without hand-tuned values.
• Improving synthetic-to-real transfer. The method underperforms on G. → Cityscapes (66.22%) relative to baseline Rein (66.40%) and FADA (68.23%); understanding why causal factors are harder to mine from synthetic sources is an open problem.
• Extending beyond the default backbone. The paper visualizes artifact reduction for DINOv2, EVA02, and CLIP, but the quantitative study uses DINOv2 with Mask2Former; testing Causal-Tune systematically across other VFMs and segmentation heads remains open.
• Comparing DCT against other separations more deeply. FFT outperformed DCT on the 'Night' condition while HWT lagged overall, which raises the question of whether a combination or condition-aware transform would be more robust.

Target Audience

Researchers and graduate students working on domain generalization, semantic segmentation, and parameter-efficient fine-tuning of vision foundation models; practitioners adapting frozen VFM backbones to new domains under compute constraints; and readers interested in how causal-inference framings translate into concrete architectural operations such as spectral filtering and learnable tokens.

Authors’ abstract

Fine-tuning Vision Foundation Models (VFMs) with a small number of parameters has shown remarkable performance in Domain Generalized Semantic Segmentation (DGSS). Most existing works either train lightweight adapters or refine intermediate features to achieve better generalization on unseen domains. However, they both overlook the fact that long-term pre-trained VFMs often exhibit artifacts, which hinder the utilization of valuable representations and ultimately degrade DGSS performance. Inspired by causal mechanisms, we observe that these artifacts are associated with non-causal factors, which usually reside in the low- and high-frequency components of the VFM spectrum. In this paper, we explicitly examine the causal and non-causal factors of features within VFMs for DGSS, and propose a simple yet effective method to identify and disentangle them, enabling more robust domain generalization. Specifically, we propose Causal-Tune, a novel fine-tuning strategy designed to extract causal factors and suppress non-causal ones from the features of VFMs. First, we extract the frequency spectrum of features from each layer using the Discrete Cosine Transform (DCT). A Gaussian band-pass filter is then applied to separate the spectrum into causal and non-causal components. To further refine the causal components, we introduce a set of causal-aware learnable tokens that operate in the frequency domain, while the non-causal components are discarded. Finally, refined features are transformed back into the spatial domain via inverse DCT and passed to the next layer. Extensive experiments conducted on various cross-domain tasks demonstrate the effectiveness of Causal-Tune. In particular, our method achieves superior performance under adverse weather conditions, improving +4.8% mIoU over the baseline in snow conditions.