Factorizable joint shift revisited

AI summary

Overview

Research area: Machine learning / domain adaptation — specifically distribution shift (dataset shift) between training and test distributions, with a focus on factorizable joint shift (FJS) and generalized label shift (GLS).

Technical level: Advanced. The paper is formulated in measure-theoretic probability (σ-algebras, Radon–Nikodym derivatives, conditional expectations), and is aimed at readers comfortable with that formalism.

Scope: A single-author theoretical paper (Dirk Tasche, Centre for Business Mathematics and Informatics, North-West University, South Africa; arXiv:2601.15036v4 [cs.LG]) that extends the analysis of factorizable joint shift from categorical labels to general label spaces, thereby covering regression as well as classification.

What This Paper Is About

Machine learning models trained on a source (training) distribution P frequently fail when deployed on a target (test) distribution Q, because P and Q differ. Making progress without label observations in the target domain requires an assumption about how the two distributions differ. This paper studies factorizable joint shift, an assumption that generalises both covariate shift and label shift, and generalises earlier FJS results — which had been confined largely to categorical labels — to general label spaces. It also presents a general-label-space version of the expectation maximisation (EM) algorithm for estimating target label distributions, and re-examines generalized label shift in the same general setting.

Key Contributions

1. A framework for non-specific distribution shift with general label spaces. The paper introduces a measure-theoretic setting covering Euclidean, categorical, and mixed Euclidean–categorical label spaces, in order to demarcate and analyse specific types of distribution shift.

2. A generalisation of the FJS characterisation (Theorem 3.5). The earlier characterisation by Tasche is extended so that both classification and regression problems are covered.

3. A general-label-space EM algorithm for label distribution estimation (Theorem 3.8). This generalises the expectation maximisation algorithm for estimating target prior class probabilities by Saerens et al., and the paper shows it is fit for purpose.

4. A generalisation of a relation between GLS and FJS (Proposition 4.1). This extends a result of He et al. on how generalized label shift relates to factorizable joint shift.

Main Findings

• FJS is defined by a factorised density. Under the setting of the paper, P and Q are related through factorizable joint shift if there is an ℋ-measurable function h̄: Ω_X → [0, ∞) and a 𝒢-measurable function ḡ: Ω_Y → [0, ∞) such that h̄(X) ḡ(Y) is a density of Q with respect to P. FJS implies the paper's Assumption 2.1 (absolute continuity of Q with respect to P), and the two factors are unique only up to a constant factor c > 0.

• Covariate and label shift are special cases of FJS, and conversely. Both covariate shift and label shift imply FJS. Conversely, FJS with ḡ = 1 implies covariate shift, and FJS with h̄ = 1 implies label shift (Proposition 3.3).

• FJS decomposes into consecutive label and covariate shifts. Proposition 3.4, described as a streamlined version of Theorem 1 of Dong et al. in the more general setting, constructs probability measures Q_L (with dQ_L/dP = ḡ(Y)/E_P[ḡ(Y)]) and Q_C (with dQ_C/dQ_L = h̄(X)/E_{Q_L}[h̄(X)]), shows Q = Q_C, shows that P and Q_L are related through label shift, and shows that Q_C and Q are related through covariate shift.

• The FJS factors relate to the marginal densities by conditional expectation. Theorem 3.5 (i) gives h(X) = h̄(X) E_P[ḡ(Y) | σ(X)] and g(Y) = ḡ(Y) E_P[h̄(X) | σ(Y)], where h is a density of Q_X with respect to P_X and g is a density of Q_Y with respect to P_Y. Theorem 3.5 (ii) gives a converse: if densities h and g of prescribed measures Q_X* and Q_Y* satisfy these relations with some h̄ and ḡ, then h̄(X) ḡ(Y) is a probability density and the resulting Q has Q_X = Q_X* and Q_Y = Q_Y*.

• FJS is more homogeneous than non-specific distribution shift. Remark 3.7 interprets ḡ not as a marginal label density (as under label shift) but, after normalisation, as a conditional density of Q_{Y|X=x} with respect to P_{Y|X=x} for all x; symmetrically, h̄ is a conditional density of Q_{X|Y=y} with respect to P_{X|Y=y} for all y. This yields an explicit representation of the target conditional density q_{Y|X=x}(y) = ḡ(y) / ∫ ḡ(z) P_{Y|X=x}(dz), and, for real-valued labels, the conditional expectation E_Q[Y | X = x] = ∫ y ḡ(y) P_{Y|X=x}(dy) / ∫ ḡ(z) P_{Y|X=x}(dz). The paper notes that Tasche previously suggested calling FJS "scaled density ratios" shift on the basis of a similar observation.

• Two problem settings are distinguished. Problem 1: the target feature marginal Q_X is known, but neither Q_Y nor the posteriors Q_{X|Y} and Q_{Y|X}; this is described as an unsupervised domain adaptation problem. Problem 2: both Q_X and Q_Y are known but neither posterior; this is described as less frequently studied in machine learning and akin to copula estimation.

• Problem 1 can in principle be solved by assuming covariate shift, by setting dQ/dP = h(X) for the pre-specified density h of Q_X. The paper notes Storkey's remark that the benefit of doing something different under covariate shift may be computational rather than a fundamental modelling benefit.

• A constrained-FJS alternating iteration replaces uncontrolled FJS under Problem 1. The paper constrains FJS by requiring ḡ(Y) = g(Y) in the relation above — which includes label shift (h̄(X) = 1) as well as more general FJS subject to 1 = E_P[h̄(X) | σ(Y)]. This leads to an alternating iteration: initialise g_0(Y) > 0 (with g_0(Y) = 1 given as the default, described as the only possible choice absent specific information about P_Y); compute h_n(X) = E_P[g_n(Y) | σ(X)]; then compute g_{n+1}(Y) = g_n(Y) E_P[h(X)/h_n(X) | σ(Y)]; and repeat until a stop criterion is fulfilled.

• The EM generalisation is shown to be well behaved. Theorem 3.8, described as generalising the appendix "Derivation of the EM Algorithm" of Saerens et al. from categorical to general label spaces, establishes (as far as the provided content shows) that 0 < h_n(X) < ∞ and 0 < g_{n+1}(Y) < ∞ P-almost surely for all n ≥ 0; that g_n(Y) and h_n(X) are probability densities under P; that f_n = h(X) g_n(Y) / h_n(X) is a probability density under P; and that the measures Q^(n) (dQ^(n)/dP = g_n(Y)) and R^(n) (dR^(n)/dP = f_n(X,Y)) satisfy dQ_Y^(n)/dP_Y = g_n, dQ_X^(n)/dP_X = h_n, dR_X^(n)/dP_X = h, and dR_Y^(n)/dP_Y = g_{n+1}. Corollary 3.10 is said to suggest a stop criterion for the iteration.

• GLS is re-examined for general label spaces. Section 4 analyses the relation between generalized label shift and FJS, with Proposition 4.1 generalising a result of He et al.

• An illustrative example is given, not a benchmark. Section 3.5 presents an illustrative example of how the EM algorithm deals with real-valued labels and categorical features. The provided content reports no datasets, dataset sizes, benchmark names, accuracy figures, or other quantitative experimental results; the paper presents theory plus an illustrative example.

Methodology in Plain English

The paper works from first principles rather than from experiments. It sets up a probabilistic setting with feature variables X and label variables Y, a fully known source distribution P, and a partially known target distribution Q, and it assumes that Q is absolutely continuous with respect to P — so that a density f = dQ/dP exists. This density is what carries information from training to test data (the paper notes this is often called "importance reweighting" and f the "importance weights"). Under this assumption the paper derives how the marginal densities of X and Y transform, how conditional expectations transform, and what additional assumptions (in the appendices) make these quantities numerically usable. It then defines factorizable joint shift as the requirement that this density splits into a factor depending only on X and a factor depending only on Y, proves how this relates to covariate shift, label shift, and their composition, and derives the formulas relating the factors to the marginal densities. Finally, it specialises the general relations to a constrained case and derives an alternating (EM-style) iteration for estimating the target label density, proving convergence-relevant properties of the iterates. Proofs are placed in Appendix D; Appendix C describes a classification-based approach to density estimation, and Appendices A and B cover technical background and extra assumptions on P.

Why This Matters

The paper strengthens the theoretical footing of a distribution-shift assumption that is more flexible than pure covariate shift or pure label shift, and it removes the earlier restriction to categorical labels — meaning the same machinery now applies to regression, not only classification. This matters because FJS had been criticised as hard to justify; the description of FJS as a composition of label and covariate shifts, and the explicit decomposition in Proposition 3.4, makes the assumption more plausible. The EM generalisation also extends a widely used class-prior estimation technique to non-categorical label spaces, where the paper positions it as a potential alternative to existing inverse-operator and segmentation/discretisation approaches.

Real-world applications that the paper's setting speaks to include:

• Deployed classification models facing test populations whose class proportions differ from training (quantification, counting, class probability re-estimation, or class prior estimation).
• Regression models under shift where labels are real-valued rather than categorical, and the target label distribution must be estimated without target labels.
• Domain adaptation with unlabelled target data, including the "missing at random" sample selection setting where covariate shift is the only reasonable assumption.
• Copula-type problems, which the paper links to Problem 2, where both marginals are known but the dependence structure is not.

Industry relevance: the paper's concern — models that perform well on training data but fail on test data — is directly relevant to any organisation deploying predictive models on shifting populations, and to anyone who needs target-domain label distributions without target labels. The provided content does not report deployments, products, or benchmark results.

Future Directions

• Establishing existence and uniqueness conditions for the FJS density factors h̄ and ḡ that satisfy the relations of Theorem 3.5, which the paper explicitly notes Theorem 3.5 (ii) does not address.
• Further analysis of Problem 2, where both the target feature marginal and the target label marginal are known, and the connection to copula estimation.
• Development of stop criteria and convergence analysis for the alternating iteration, which the paper points to via Corollary 3.10.
• The paper's Section 5 closes with a summary and suggestions for further research, including the empirical behaviour of the general EM algorithm in non-categorical settings beyond the illustrative example in Section 3.5.

Target Audience

Researchers and graduate students in machine learning working on distribution shift, dataset shift, and domain adaptation, particularly those comfortable with measure-theoretic probability; statisticians interested in label distribution estimation, quantification, and class prior estimation; and readers who want the technical derivation behind FJS and its relationship to covariate shift, label shift, and generalized label shift.

Authors’ abstract

Factorizable joint shift (FJS) represents a type of distribution shift (or dataset shift) that comprises both covariate and label shift. Recently, it has been observed that FJS actually arises from consecutive label and covariate (or vice versa) shifts. Research into FJS so far has been confined mostly to the case of categorical labels. We propose a framework for analysing distribution shift in the case of a general label space, thus covering both classification and regression models. Based on the framework, we generalise existing results on FJS to general label spaces and present and analyse a related extension to label distribution estimation of the expectation maximisation (EM) algorithm for class prior probabilities. We also take a fresh look at generalized label shift (GLS) in the case of a general label space.