Do We Need Perfect Data? Leveraging Noise for Domain Generalized Segmentation

AI summary

Authors: Taeyeong Kim, SeungJoon Lee, Jung Uk Kim, MyeongAh Cho (Kyung Hee University, Republic of Korea) arXiv: 2511.22948v1 [cs.CV], 28 Nov 2025

Overview

• Research area: Computer vision, specifically Domain Generalized Semantic Segmentation (DGSS) — training segmentation models that transfer to unseen driving environments using only source-domain data.
• Technical level: Advanced. The paper assumes familiarity with semantic segmentation, diffusion-based image generation, contrastive learning, morphological operations, and standard domain generalization benchmarks.
• Scope: The paper proposes FLEX-Seg, a framework that deliberately exploits the spatial misalignment between diffusion-generated synthetic images and their semantic masks instead of trying to eliminate it, and evaluates it across five real-world driving datasets with two backbone families.

What This Paper Is About

Semantic segmentation models trained on one set of driving conditions (weather, lighting, camera) often break down on unseen conditions. A popular fix is to generate extra synthetic training images with diffusion models, but these generators produce images from masks rather than masks from images, so fine structures in the generated image do not line up pixel-for-pixel with their labels — especially at object boundaries. Rather than treating this misalignment as something to be corrected or ignored, the paper asks whether it can be used as a training signal, and builds a framework that learns robust boundary representations despite (and partly because of) the imperfect alignment.

Key Contributions

1. FLEX-Seg framework (FLexible Edge eXploitation for Segmentation), which reframes synthetic-data misalignment as a source of robust, domain-invariant learning rather than an error to be suppressed.
2. Granular Adaptive Prototypes (GAP), a class-shape token coordinate system that builds a prototype bank of size C × 3 × 256 (C classes × 3 boundary granularities), updated with a momentum factor m = 0.99 and trained with an imbalance-aware InfoNCE contrastive loss.
3. Uncertainty Boundary Emphasis (UBE), which uses prediction entropy to modulate per-pixel loss weights on boundary regions, avoiding manual tuning of boundary weights across weather conditions.
4. Hardness-Aware Sampling (HAS), a training sampler that transitions from random to loss-based sampling via sigmoid decay, maintaining a hardness score per image updated every 50 iterations with an EMA decay factor β = 0.9.
5. Empirical validation across five real-world datasets (Cityscapes, BDD100K, Mapillary Vistas, ACDC, Dark Zurich) with ResNet-101 and MiT-B5 backbones, plus ablations and supplementary studies on hyperparameters, dataset size, sampling temperature, and applicability to ALDM-generated data.

Main Findings

• Boundary regions dominate the error: Error rates are consistently higher at boundaries than in interiors under both normal and adverse conditions, and the gap widens in adverse conditions (Figure 1).
• Simple boundary upweighting already helps: Applying a fixed boundary loss weight of α = 5 improved delineation of object structure despite substantial misalignment, motivating the more adaptive GAP and UBE modules (Figure 2).
• Gains on adverse-condition benchmarks: FLEX-Seg achieves 2.44% and 2.63% mIoU gains on ACDC and Dark Zurich (abstract-level figures, corresponding to HRDA-based results).
• HRDA + FLEX-Seg results: ACDC 48.51% mIoU (vs. 46.07% for HRDA + DGInStyle), Dark Zurich 28.16% (vs. 25.53%), Avg2 38.34%, Cityscapes 59.49%, BDD100K 52.48%, Mapillary Vistas 61.71%, Avg5 50.07%.
• DAFormer + FLEX-Seg results: ACDC 46.56% (vs. 44.04%), Dark Zurich 29.51% (vs. 25.58%), Avg2 38.04%, Cityscapes 56.84%, BDD100K 52.06%, Mapillary Vistas 57.93%, Avg5 48.58%. The ACDC gain over DGInStyle is +2.52% and the Dark Zurich gain is +3.93%.
• Outperforms CLIP-pretrained baselines: FLEX-Seg surpasses FAMix (32.74% on ACDC) and SCSD (35.66% on ACDC), even though those methods use a ResNet-50 backbone initialized with CLIP pretrained weights.
• Ablation — GAP is the strongest single module: Starting from the DGInStyle baseline of 34.81% Avg2 / 54.25% Avg3 / 46.47% Avg5, GAP alone gives 36.33% / 55.26% / 47.69% (+1.52% Avg2), while UBE alone gives 35.05% / 55.33% / 47.21% (+0.24% Avg2).
• Ablation — full combination wins overall: GAP+UBE reaches 36.15% Avg2 / 56.07% Avg3 / 48.10% Avg5; adding HAS gives the best overall result of 38.04% Avg2 / 55.61% Avg3 / 48.58% Avg5, a +3.23% improvement over baseline on challenging domains.
• HAS trades standard-domain score for adverse robustness: Adding HAS slightly reduces Avg3 (−0.46%) but substantially boosts challenging domains (+1.89%).
• Dataset size saturates: Avg2 rises from 35.82 at N = 2,000 generated images to 38.04 at N = 10,000, then dips to 37.88 at N = 12,000; Avg5 keeps rising to 48.73 at N = 12,000.
• Optimal hyperparameters found: UBE amplification α = 3.0 peaks at 48.58% average mIoU; GAP loss weight λ_gap = 0.5 peaks at 48.58% (up from 47.92% at λ_gap = 0.1); HAS sampling temperature τ = 1.0 peaks at 48.58%, while τ = 0.01 collapses to 31.24%.
• Sigmoid decay beats alternatives: Sigmoid decay gives 38.04% Avg2 / 48.58% Avg5, versus 36.82% / 47.51% for no decay (random sampling) and 36.28% / 47.15% for linear decay — linear decay was worse than no decay at all.
• Method generalizes beyond DGInStyle: With ALDM-generated synthetic data, FLEX-Seg improves the baseline from 35.30% to 36.74% on Avg2 (+1.44%) and from 45.48% to 46.31% on Avg4 (+0.83%), excluding Cityscapes from evaluation because ALDM was pretrained on it.
• Qualitative improvements: Sharper object contours (vehicles, pedestrians), better handling of thin structures such as poles and traffic signs, and more coherent predictions under snow and nighttime where baselines fragment.
• Sampling stays balanced by condition: Although night and foggy conditions are sampled more frequently in later stages, all conditions are sampled at roughly 20% each over training without bias.

Methodology in Plain English

The pipeline starts by taking the GTA source dataset (24,966 synthetic images with 19-class annotations) and using Rare Class Sampling with probability threshold T = 0.01 to pick masks, crop them into 512×512 patches centered on rare classes, and generate 10,000 new images covering standard and adverse weather. These generated images are combined with 6,000 images selected from the original GTA set by rare-class criteria to form the training corpus.

Two mechanisms then handle the imprecise boundaries. GAP treats each boundary pixel as a point in a two-dimensional coordinate space — one axis is the semantic class, the other is the boundary's geometric thickness. Morphological dilation/erosion with different kernel sizes produces thin, medium, and thick boundary masks, and features from those pixels are pulled toward prototypes at the matching (class, granularity) cell of a C × 3 × 256 memory bank, updated by momentum. A contrastive loss pulls same-cell boundary features together regardless of which domain they came from, with an automatically computed weight that boosts under-represented class-granularity cells.

UBE measures per-pixel prediction entropy, standardizes it using batch-wise boundary statistics (mean and standard deviation), and passes it through a sigmoid so that uncertain boundary pixels receive up to 1 + α times the normal loss weight, while interior pixels keep weight 1. Because the emphasis comes from the model's own uncertainty, one α value works across fog, rain, snow, and night.

HAS stores a running difficulty score for each training image. It uses a sigmoid-decay threshold to decide, at each iteration, whether to sample randomly or to sample images proportional to their hardness (via a softmax with temperature τ). This keeps early training stable while progressively concentrating on hard cases later.

The total objective is the sum of the UBE loss and a weighted GAP loss (λ_gap). Training used a single RTX 3090 for DAFormer and a single A6000 Ada for HRDA, with other settings and 3 random runs following the DGInStyle/HRDA configurations.

Why This Matters

• Impact on research: It challenges the implicit assumption behind boundary-aware segmentation methods — that image structures and masks correspond perfectly — and shows that imperfect generative data can be exploited rather than filtered, corrected, or discarded. It also provides a reusable recipe (entropy-driven weighting plus multi-granularity prototypes) that is not tied to a single diffusion generator, as the ALDM experiment demonstrates.
• Autonomous driving in bad weather: More reliable segmentation of pedestrians, vehicles, and lane-relevant structures under fog, rain, snow, and night, where ACDC and Dark Zurich results show the largest gains.
• Robotics operating outdoors: Boundary-aware perception supports safe navigation in environments where lighting and weather change unpredictably and target-domain data cannot be collected in advance.
• Reduced data collection and annotation cost: If synthetic data does not need pixel-perfect alignment, teams can generate training data with cheaper or faster pipelines and still get usable models.
• General synthetic data pipelines: Any system that generates images from masks or from layout conditions (layout-to-image models, simulation engines) can adopt these modules, since the misalignment problem is a property of the generation direction, not of one specific method.

Industry relevance: Companies that rely on large labeled driving datasets for perception stacks can lower their dependency on target-domain annotation and on expensive perfect synthetic labels, using entropy-based weighting and multi-granularity prototypes as add-on training modules to existing segmentation architectures such as DAFormer or HRDA. The reported gains on adverse conditions are directly relevant to safety cases where performance under fog, rain, and night is the limiting factor.

Future Directions

• Adaptive prototype mechanisms: The authors explicitly name exploring adaptive prototype mechanisms as a next step, suggesting the fixed class × granularity structure of GAP is not necessarily the final form.
• Extension to other dense prediction tasks: The conclusion proposes applying the framework where boundary precision is critical in other dense prediction problems, such as depth estimation or other per-pixel tasks.
• Broader generator coverage: Only DGInStyle and ALDM are tested as generation backends. Testing more diffusion or layout-to-image generators would clarify how general the misalignment-as-signal premise is.
• Reconciling the HAS trade-off: HAS improves challenging domains by +1.89% but lowers the standard-domain average by 0.46%, leaving open how to schedule sampling so both improve together.

Target Audience

• Domain generalization researchers working on segmentation and other dense prediction tasks who want a boundary-aware alternative to strict alignment or filtering of synthetic data.
• Autonomous driving and robotics perception engineers concerned with robustness to fog, rain, snow, and nighttime conditions, who can plug GAP, UBE, and HAS onto existing DAFormer/HRDA-style training pipelines.
• Practitioners building synthetic data pipelines with diffusion models who need to know whether the residual image-mask misalignment in their generated datasets is a fatal flaw or a manageable, even useful, property.
• Graduate students and advanced readers comfortable with segmentation benchmarks and contrastive learning; the paper is not beginner-friendly because it assumes fluency with domain shift terminology, morphological boundary extraction, and InfoNCE-style losses.

Authors’ abstract

Domain generalization in semantic segmentation faces challenges from domain shifts, particularly under adverse conditions. While diffusion-based data generation methods show promise, they introduce inherent misalignment between generated images and semantic masks. This paper presents FLEX-Seg (FLexible Edge eXploitation for Segmentation), a framework that transforms this limitation into an opportunity for robust learning. FLEX-Seg comprises three key components: (1) Granular Adaptive Prototypes that captures boundary characteristics across multiple scales, (2) Uncertainty Boundary Emphasis that dynamically adjusts learning emphasis based on prediction entropy, and (3) Hardness-Aware Sampling that progressively focuses on challenging examples. By leveraging inherent misalignment rather than enforcing strict alignment, FLEX-Seg learns robust representations while capturing rich stylistic variations. Experiments across five real-world datasets demonstrate consistent improvements over state-of-the-art methods, achieving 2.44% and 2.63% mIoU gains on ACDC and Dark Zurich. Our findings validate that adaptive strategies for handling imperfect synthetic data lead to superior domain generalization. Code is available at https://github.com/VisualScienceLab-KHU/FLEX-Seg.