Self-Training with Dynamic Weighting for Robust Gradual Domain Adaptation

AI summary

Overview

• Research area: Machine learning, specifically gradual domain adaptation (GDA) — transfer learning where a model moves from a labeled source distribution to an unlabeled target distribution through a sequence of intermediate domains. arXiv:2510.13864v1 [cs.LG], published 13 October 2025.
• Technical level: Advanced. The paper defines a formal optimization objective with a time-varying weighting parameter, a cyclic batch-matching scheme, and a stepwise "dynamic osmosis" update rule, and it compares against self-training, optimal-transport, and test-time adaptation baselines.
• Scope (one sentence): The paper introduces Self-Training with Dynamic Weighting (STDW), a gradual domain adaptation method that schedules a hyperparameter ϱ from 0 to 1 to shift the training loss from the source domain toward the target domain while refining pseudo-labels batch by batch, and it evaluates this on Rotated MNIST, Color-Shift MNIST, Portraits, Cover Type, CIFAR-10-C, and CIFAR-100-C.

What This Paper Is About

Gradual domain adaptation tries to move a classifier from a labeled source domain to an unlabeled target domain by passing through intermediate domains, because jumping directly across a large distribution shift fails. Existing GDA methods either propagate knowledge inefficiently between neighboring domains or label the whole unlabeled set in one pass, which lets mislabeled examples compound into errors across successive domains. The paper's goal is a training scheme that migrates knowledge smoothly and stably from source to target by dynamically reweighting the two domains' losses as training progresses, combined with incremental pseudo-label generation.

Key Contributions

1. A time-varying weighting framework. The paper introduces a hyperparameter ϱ ∈ [0,1] that linearly increases from 0 to 1 during adaptation, balancing the loss on the current domain against the loss on the next domain, so the model progressively hands emphasis from source to target.
2. Dynamic pseudo-labeling. Instead of labeling the entire unlabeled dataset in a single pass (the static paradigm, which discards a fixed proportion such as 10% of low-confidence samples), STDW splits the data into T disjoint mini-batches and alternates between generating pseudo-labels for a batch and updating the classifier, so later batches benefit from a better model.
3. Cyclic batch matching across neighboring domains. Batches from a left domain and a right domain are paired in a deterministic periodic order using index functions based on modulo arithmetic, giving a controlled schedule of cross-domain updates.
4. Empirical validation on six benchmarks. STDW is compared against UDA baselines (DANN, DeepCoral, DeepJDOT), GDA baselines (GST, IDOL, GOAT, GGF, CNF), and test-time adaptation methods (TENT-continual, AdaContrast, CoTTA, GTTA-MIX), plus ablation of the ϱ schedule (Equal vs. Fixed, Rand, Sorted). Code is released at https://github.com/Dramwig/STDW.

Main Findings

• Top accuracy on all four core benchmarks: STDW reaches 97.6% on Rotated MNIST, 98.3% on Color-Shift MNIST, 87.1% on Portraits, and 74.2% on Cover Type. The paper states these are absolute improvements of 11.2%, 6.5%, 0.94%, and 4.3% over the second-best methods.
• Baseline gap: On Rotated MNIST the strongest listed competitor is IDOL at 87.5%, with GGF at 67.72% and CNF at 62.55%; on Color-Shift MNIST the next best is GOAT at 91.8%; on Portraits GGF is next at 86.1%; on Cover Type GST is next at 73.5%.
• Corruption robustness: At severity level 5, STDW achieves the lowest mean error rate of 15.5% on CIFAR-10-C and 25.8% on CIFAR-100-C, versus a source-only baseline of 43.5% and 46.4% respectively, and versus the closest competitor GTTA-MIX at 15.6% and 28.9%.
• Consistency across corruption types: STDW is best on 11 of 15 corruption types for CIFAR-10-C and 14 of 15 for CIFAR-100-C. For high-frequency corruptions such as Gaussian noise, shot noise, and impulse noise, the paper reports error-rate reductions of 3.3 to 11.6 percentage points over the next best method.
• Intermediate domains help, and more domains help more: On Rotated MNIST, going from 2 given domains to 3 jumps from roughly 81.5–83.6% to 95.9–96.9%. Color-Shift MNIST rises from the 86.2–96.5% range at 2 domains to 98.1–98.3% at 3 or more. Cover Type rises from 69.7–71.8% at 2 domains to as high as 74.4% at 3 domains. Portraits stays in a narrower band, from 83.7% at 2 domains to 86.1% at 6 domains.
• Stability: Across five independent runs, reported standard deviations are all ≤ 1.2% in the multi-dataset domain-count experiments.
• Monotonic ϱ scheduling beats alternatives: In the ablation on Rotated MNIST and Portraits with 0 to 4 inter-domain steps, the "Equal" schedule reaches 88.1% ± 1.5 on Rotated MNIST at Step 4, versus 84.1% for Fixed (ϱ = 0.5 held constant) and 86.1% ± 0.4 for Rand (ϱ sampled from U(0,1)), and it also has far lower variance (e.g. ± 1.5 versus ± 7.0 for Rand at Step 1 on Rotated MNIST).
• Adding intermediate domains stabilizes training: The ablation across 2 to 6 domains shows that adding unlabeled intermediate domains improves accuracy in most settings and reduces run-to-run standard deviation.
• Correctness on CIFAR-10-C at severity 5: STDW with 6 given domains reaches 84.5% mean correctness, compared with 71.9% for GST, 56.6% for GOAT, and 56.5% for the source-only model.

Methodology in Plain English

The method is built on standard self-training, where a model labels unlabeled data with its own confident predictions and then trains on those labels. STDW changes three things about how that loop runs.

First, instead of labeling everything at once, the unlabeled pool is split into small batches. The model labels one batch, updates its parameters on that batch, then moves to the next batch using the improved model. This means each batch is labeled by a slightly better classifier than the last, which reduces how much label noise accumulates.

Second, when adapting between two neighboring domains, batches from the "left" domain and the "right" domain are paired in a fixed cyclic order rather than randomly. This gives a repeatable, evenly spread schedule of cross-domain updates instead of an arbitrary one.

Third, the loss is a weighted sum: (1 − ϱ) times the loss on the left-domain batch plus ϱ times the loss on the right-domain batch. At the start ϱ = 0, so the model only cares about the current domain; as training proceeds, ϱ increases in equal increments (a step size of 1/s, where s is the number of inter-domain migration steps) until ϱ = 1, at which point the model only cares about the next domain. This produces a gradual handover of knowledge rather than an abrupt jump.

The implementation uses a convolutional network with three 32-channel convolutional layers and two 256-unit fully connected layers for image tasks (Rotated MNIST, Color-Shift MNIST, Portraits), a 128-256-512 fully connected architecture for the tabular Cover Type data, WideResNet-28 for CIFAR-10-C, and ResNeXt-29 for CIFAR-100-C. Models use ReLU activations, batch normalization, dropout, and the Adam optimizer, and all experiments run on NVIDIA RTX 4090 GPUs with fixed random seeds. The number of inter-domain adaptation steps is varied from 0 to 4.

Why This Matters

The work targets a practical failure mode in deployed machine learning: models trained on one distribution quietly degrade when the world drifts, and labels for the drifted data are unavailable. STDW's contribution is a scheduling rule — one hyperparameter that slides from source to target — plus incremental pseudo-labeling, which together let a model track a slowly changing environment without a hard, error-prone jump.

Real-world applications suggested by the benchmarks used:

• Progressive visual transformations: the Rotated MNIST and Color-Shift MNIST setups mirror gradual changes in camera angle or lighting, relevant to vision systems whose sensors or viewpoints drift.
• Historical and archival imagery: the Portraits Dataset spans 1905–2013 across nine chronological domains, relevant to digitizing and classifying photographs across changing photographic styles.
• Tabular environmental/land-cover prediction: the Cover Type Dataset constructs domains by ecological proximity to water, a setting where measured distributions shift smoothly across geography.
• Robustness to sensor and image degradation: the 15 corruption types in CIFAR-10-C and CIFAR-100-C (noise, blur, weather, compression) model real degradation in image pipelines.

Industry relevance: any deployed system facing slow, continuous distribution shift — rather than a single abrupt change — is a candidate, especially where unlabeled data is abundant and labeling is expensive. The released code and the reliance on established datasets and architectures make the approach relatively easy to reproduce. The paper's work on Shuai Zhang is supported by National Science Foundation (NSF) #2349879.

Future Directions

• Reducing dependence on intermediate domains. The authors note that STDW relies heavily on the availability and quality of intermediate domains, which may not be accessible or well-defined in practice. Generating those intermediate distributions automatically is an open step.
• Automating the ϱ schedule. The dynamic weighting adds hyperparameters, notably how ϱ is scheduled, that need careful tuning and can increase training cost. The authors list automatic hyperparameter selection as future work.
• Handling abrupt shifts. The method assumes smooth transitions, so highly discontinuous or abrupt domain shifts are not covered.
• Extending beyond images and tables. All experiments use image or tabular data; text, time-series, and other modalities are unexplored. The truncated appendix also references an ImageNet-C setting using ResNet

Authors’ abstract

In this paper, we propose a new method called Self-Training with Dynamic Weighting (STDW), which aims to enhance robustness in Gradual Domain Adaptation (GDA) by addressing the challenge of smooth knowledge migration from the source to the target domain. Traditional GDA methods mitigate domain shift through intermediate domains and self-training but often suffer from inefficient knowledge migration or incomplete intermediate data. Our approach introduces a dynamic weighting mechanism that adaptively balances the loss contributions of the source and target domains during training. Specifically, we design an optimization framework governed by a time-varying hyperparameter $\varrho$ (progressing from 0 to 1), which controls the strength of domain-specific learning and ensures stable adaptation. The method leverages self-training to generate pseudo-labels and optimizes a weighted objective function for iterative model updates, maintaining robustness across intermediate domains. Experiments on rotated MNIST, color-shifted MNIST, portrait datasets, and the Cover Type dataset demonstrate that STDW outperforms existing baselines. Ablation studies further validate the critical role of $\varrho$'s dynamic scheduling in achieving progressive adaptation, confirming its effectiveness in reducing domain bias and improving generalization. This work provides both theoretical insights and a practical framework for robust gradual domain adaptation, with potential applications in dynamic real-world scenarios. The code is available at https://github.com/Dramwig/STDW.