Exploiting Domain Properties in Language-Driven Domain Generalization for Semantic Segmentation

AI summary

Authors: Seogkyu Jeon, Kibeom Hong, Hyeran Byun (Yonsei University; Sookmyung Women's University) arXiv: 2512.03508v2 [cs.CV], 18 Dec 2025 Code: https://github.com/jone1222/DPMFormer

Overview

Research area: Computer Vision — Domain Generalized Semantic Segmentation (DGSS) combined with Vision-Language Models (VLMs).

Technical level: Advanced. The paper assumes familiarity with transformer-based mask classification (Mask2Former), contrastive learning, and CLIP-style text/image encoders.

Scope: The paper proposes DPMFormer, a framework that generates domain-aware text prompts from the input image and enforces prediction consistency under synthetic texture perturbations, to improve single-source domain generalization for semantic segmentation.

What This Paper Is About

Domain generalized semantic segmentation trains a model on one source dataset and expects it to work on unseen target environments, but the text prompts used to describe object categories are fixed — either a handcrafted template or a single learned embedding — and therefore encode the source domain's characteristics. This rigidity creates a semantic mismatch between what the model reads in the text and what it sees in a target image (for example, a "car" at night in the rain). The goal of this work is to make the textual description adapt to the domain properties of each input image, and to make the visual predictions stable when the image's texture and appearance change.

Key Contributions

1. Domain-aware prompt learning. An auxiliary network h_θ (a shallow BatchNorm-Linear-ReLU-Linear module) takes the class token from a frozen CLIP visual backbone and produces a domain-specific prompt embedding π_x, which is added to the learned context prompt p to form p_x = p + π_x. This prompt then conditions the frozen text encoder, so the textual object descriptions change with the input image.

2. Domain-aware contrastive learning with texture perturbation. To obtain diverse domain properties from a single source dataset, training images are stylized using photometric transformations (strong color jittering, Gaussian blur, and noise injection) following RobustNet. A contrastive loss is then applied on the context embeddings: original-domain images treat other original images as positives and all augmented images as negatives, encouraging the prompt generator to capture domain-specific rather than instance-specific information.

3. Domain-robust consistency learning. A consistency loss penalizes discrepancies between the predictions for an original image and its texture-perturbed counterpart, decomposed into a mask consistency term (binary cross entropy) and a class consistency term (Jensen-Shannon divergence). This loss is applied at every one of the S transformer decoder blocks so that errors in early layers do not propagate.

4. State-of-the-art DGSS results. DPMFormer achieves the best or comparable results across synthetic-to-real and real-to-real benchmarks with both CLIP (ViT-B) and EVA02-CLIP (EVA02-L) backbones.

Main Findings

• Synthetic-to-real with GTAV source, CLIP (ViT-B): DPMFormer reaches 59.00 mIoU on Cityscapes, 51.80 on BDD100K, and 63.62 on Mapillary, for an average of 58.14, compared with the previous state-of-the-art TQDM at 57.50, 47.66, 59.76, and 54.97. The paper reports this as a 3.17% average mIoU improvement over the previous state of the art, with the largest gains attributed to BDD100K and Mapillary, where environmental variation is higher.

• Synthetic-to-real with GTAV source, EVA02-L backbone: DPMFormer scores 70.08 on Cityscapes, 60.48 on BDD100K, and 70.66 on Mapillary (average 67.07), versus TQDM at 68.88, 59.18, and 70.10 (average 66.05). The paper notes a 1.2% gain on Cityscapes and 0.56% on Mapillary relative to TQDM.

• Synthetic-to-real with SYNTHIA source: DPMFormer with EVA02-L achieves 58.92 on Cityscapes, 54.39 on BDD100K, and 60.08 on Mapillary (average 57.80), surpassing TQDM (57.99, 52.43, 54.87; average 55.10) by 2.7% average mIoU, with a 5.21% improvement on Mapillary.

• Real-to-real with Cityscapes source, ViT-B: DPMFormer reaches 54.81 on BDD100K and 67.72 on Mapillary (average 61.27), improving over TQDM (50.54, 65.74; average 58.14) by 4.27% and 1.98% on the two targets, and 3.13% on average.

• Real-to-real with Cityscapes source, EVA02-L: DPMFormer attains the highest Mapillary mIoU at 76.67% but slightly lower performance on BDD100K, giving an average of 70.44%, which the paper describes as comparable to TQDM (70.44%).

• Ablation on GTAV with ViT-B: Starting from a baseline of 54.97 average mIoU, texture perturbation adds 0.65% (55.38), adding the consistency loss raises it to 56.15 (+0.77%), and adding the domain-aware contrastive loss brings the final result to 58.14 (+1.99%).

• Comparison with prompt learning methods: Using sim(π, π) as the similarity target gives 58.14 average mIoU, better than sim(t, v) (57.34), sim(t, t) (56.57), PromptSRC (56.78), CoCoOp (56.36), MaPLe (56.34), CoCoOp+ (56.20), and no contrastive loss (56.13). The analysis argues CoCoOp's design makes the generated prompt instance-specific rather than domain-specific.

• Qualitative robustness: On BDD100K and Mapillary, FAMix and TQDM mislabel roads as sidewalks or miss low-brightness sidewalks, while DPMFormer segments correctly. On artistic styles (Minimalist, Pop Art, Bauhaus, Cubism) generated with ChatGPT, DPMFormer reportedly predicts correctly even under severe style shift where TQDM confuses road, sidewalk, and terrain.

Methodology in Plain English

The model is built on Mask2Former, a mask-classification architecture with an image encoder, a pixel decoder, and a transformer decoder with N = 9 layers. The image encoder and a text encoder are initialized from pretrained VLMs (CLIP ViT-B with patch size 16, or EVA02-CLIP EVA02-L with patch size 14), and the text encoder stays frozen. Object queries for the decoder are initialized from text embeddings of class names, where the class name is wrapped in a context prompt.

The key change is that this context prompt is no longer fixed. A small auxiliary network looks at the global visual feature of the input image and outputs a domain-specific embedding, which is added to the prompt before the text encoder sees it. This means the same category gets a text description that reflects the appearance of the current image.

Because there is only one training domain, the authors manufacture extra "domains" by applying photometric transformations — color jittering, blur, and noise — that change texture but preserve image structure. Each batch mixes original and perturbed images. A contrastive loss then teaches the prompt generator to produce embeddings that cluster by domain: original images are positives for each other, perturbed images are negatives. Separately, a consistency loss forces the model to produce the same class predictions and mask predictions for an image and its perturbed version, applied at every transformer decoder layer.

Training uses AdamW with learning rates of 1×10⁻⁵ for synthetic datasets and 1×10⁻⁴ for real datasets, linear warm-up for 1,500 iterations, rare class sampling, 20,000 total iterations, batch size 8, and 512×512 crops. The loss weighting factors are λ_reg = 1, λ_contra = 1, and λ_cons = 10. Evaluation uses mean Intersection over Union (mIoU). Texture perturbation is used only during training.

Why This Matters

Impact on research: The paper challenges the assumption that a single learned context prompt, optimized on one source domain, is sufficient for language-driven segmentation. It shows that prompt information should be conditioned on the input image's domain at the cost of a very small auxiliary network, and that combining domain-aware prompts with consistency regularization targets two distinct failure modes (semantic mismatch and prediction instability) rather than one.

Real-world applications:

• Autonomous driving perception that must work across day, dawn, and night conditions, and across different sensor and weather conditions not seen in training.
• Robotics and mobile platforms that operate across indoor, outdoor, and seasonal environments where labeled data for every setting is unavailable.
• Remote sensing and aerial imagery, where a model trained on one region must transfer to new geographies and imaging conditions.
• Medical imaging and industrial inspection, where models trained on one scanner, site, or lighting setup must be deployed elsewhere without target-domain labels.

Industry relevance: The method uses pretrained VLM backbones that many teams already deploy, adds an auxiliary network that the paper describes as a shallow multi-layer structure, and requires no target-domain data. That makes it a practical upgrade path for existing Mask2Former-based segmentation pipelines.

Future Directions

• Extending the domain-aware prompt idea beyond a single-source setting to multiple source datasets, an assumption the paper keeps fixed.
• Testing whether the texture perturbation family (color jittering, Gaussian blur, noise) can be replaced by learned or more diverse augmentation strategies, since the ablation shows perturbation alone contributes 0.65% average mIoU.
• Addressing the inconsistent behavior of the largest backbone on real-to-real transfer, where the paper reports a slightly lower BDD100K result than a competing method despite a higher Mapillary score.
• Investigating whether the generated domain prompts can be interpreted or reused across tasks, given the PCA visualization of textual prompts presented in Figure 2 but not fully exploited elsewhere.

Target Audience

Researchers and practitioners working on domain generalization, semantic segmentation, and vision-language model adaptation. The paper is most valuable to readers already familiar with Mask2Former-style architectures and CLIP-style contrastive pretraining; readers new to segmentation will need background in mask classification, object queries, and mIoU evaluation to follow the technical sections.

Authors’ abstract

Recent domain generalized semantic segmentation (DGSS) studies have achieved notable improvements by distilling semantic knowledge from Vision-Language Models (VLMs). However, they overlook the semantic misalignment between visual and textual contexts, which arises due to the rigidity of a fixed context prompt learned on a single source domain. To this end, we present a novel domain generalization framework for semantic segmentation, namely Domain-aware Prompt-driven Masked Transformer (DPMFormer). Firstly, we introduce domain-aware prompt learning to facilitate semantic alignment between visual and textual cues. To capture various domain-specific properties with a single source dataset, we propose domain-aware contrastive learning along with the texture perturbation that diversifies the observable domains. Lastly, to establish a framework resilient against diverse environmental changes, we have proposed the domain-robust consistency learning which guides the model to minimize discrepancies of prediction from original and the augmented images. Through experiments and analyses, we demonstrate the superiority of the proposed framework, which establishes a new state-of-the-art on various DGSS benchmarks. The code is available at https://github.com/jone1222/DPMFormer.