Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression

AI summary

Overview

Research area: Statistical machine learning — transfer learning theory, high-dimensional linear regression, and the theory of benign overfitting in overparameterized models.

Technical level: Advanced. The paper is written for readers comfortable with linear algebra, random matrix theory arguments, sub-Gaussian concentration, effective rank conditions, and non-asymptotic excess risk analysis.

Scope (one sentence): The paper proposes and theoretically analyzes a two-step "Transfer MNI" estimator that pre-trains on a source task's minimum-ℓ2-norm interpolator and fine-tunes on target data, characterizing when this transfer beats training on target data alone.

Authors and venue information: Yeichan Kim (Yonsei University), Ilmun Kim (KAIST), and Seyoung Park (Yonsei University, co-corresponding author). Posted to arXiv as 2510.15337v2 [stat.ML] on 16 Jan 2026, under a CC BY 4.0 license.

What This Paper Is About

In high-dimensional linear regression where the number of features exceeds the sample size (n < p), the minimum-ℓ2-norm interpolator (MNI) fits the training data exactly yet still generalizes well — a phenomenon called benign overfitting. Separately, transfer learning improves a target task by borrowing from related source tasks. This paper asks whether transfer learning can further improve the generalization of such interpolators, and answers by designing a new estimator and deriving exactly when the borrowed source knowledge helps, costs nothing extra, or actively hurts.

Key Contributions

1. A new two-step Transfer MNI (TM) estimator. The method pre-trains a source-only MNI and then fine-tunes it by interpolating target data while staying as close as possible to the pre-trained model. The authors derive an interpretable "retain-plus-transfer" decomposition showing the estimator keeps target-learned signal in the span of the target samples and injects source information only into the target's null space.

2. Non-asymptotic excess risk theory under model and covariate shift. The paper provides bias and variance decompositions and excess risk bounds for TM when each single-task MNI is benignly overfitted, including a characterization of the trade-off between bias reduction and variance inflation.

3. Identification of when transfer wins, and how much. Under isotropic covariates with model shift, the authors give necessary and sufficient conditions for TM to beat the target-only MNI, derive the optimal transfer sample size, and quantify the maximal achievable improvement in excess risk.

4. A "free-lunch" covariate shift regime and a data-driven multi-source method. The paper shows a covariate shift (uniform upscaling of the source covariance) that reduces variance inflation while leaving bias unaffected, and it proposes a cross-validation procedure to detect informative sources plus an ensemble that aggregates multiple informative Transfer MNIs with data-adaptive weights.

Main Findings

• Transfer always costs variance, but can buy more bias reduction. The TM variance equals the target-only MNI variance plus a strictly positive "variance inflation" term. Positive transfer occurs only if the bias reduction from the source outweighs this inflation.

• Exact conditions for positive transfer (isotropic case). Under isotropic covariates (Σ⁽⁰⁾ = Σ⁽q⁾ = I_p) with Gaussian designs and p > (n₀+1) ∨ (n_q+1), TM outperforms target-only MNI in expected excess risk if and only if SSR_q < 1 and SNR_q(1 − SSR_q) > p/(p − (n_q+1)). Here SSR_q = ‖δ⁽q⁾‖²/‖β⁽⁰⁾‖² and SNR_q = ‖β⁽⁰⁾‖²/σ_q².

• Negative transfer when model shift dominates. If SSR_q ≥ 1, so that the source–target model contrast is at least as large as the target signal, transfer hurts.

• More source data is not always better. The improvement Δ(n_q) is strictly concave over n_q ∈ [1, p−1). It grows up to an optimal transfer size n_q* = p − 1 − √( p(p−1) / (SNR_q(1 − SSR_q)) ) and then declines, so transferring beyond n_q* degrades the benefit.

• Maximal improvement is quantified. When SNR_q(1 − SSR_q) ≥ p(p−1)/(p−2)², the maximal improvement equals ((p − n₀)/p) · ((n_q*)²(1 − SSR_q)/(p(p−1))) · ‖β⁽⁰⁾‖², which is positive.

• Bounds under general sub-Gaussian covariates. With benign covariate assumptions, the TM bias vanishes under conditions including r₀(Σ⁽⁰⁾) ≪ n₀, r₀(Σ⁽⁰⁾) ≍ r₀(Σ⁽q⁾), and ‖δ⁽q⁾‖² ≪ ‖β⁽⁰⁾‖² with n₀ < n_q, in which case TM can converge faster than target-only.

• A "free-lunch" covariate shift exists. If the source covariance is a uniform α-fold upscaling of the target covariance (Σ⁽q⁾ = αΣ⁽⁰⁾ with α > 1), the variance inflation is multiplied by α⁻¹ — reduced by a factor of α — while the bias upper bound stays the same up to a constant independent of α (case A), or the exact bias is unchanged while exact variance inflation is multiplied by α⁻¹ (case B, when all eigenvectors align).

• The free-lunch alignment condition can be relaxed. The leading-τ* eigenvector alignment requirement can be loosened to a spectral-norm condition relating the leading eigenvector blocks to the ratio λ_{τ*+1}⁽⁰⁾/λ₁⁽⁰⁾.

• Multi-source transfer needs source selection. Because the informative-source set is unknown in practice, the authors use K-fold cross-validation (with K = 5 noted as a common choice) to estimate which sources induce positive transfer, then ensemble informative Transfer MNIs with data-adaptive weights.

• Empirical comparison. Finite-sample experiments, whose specific datasets and numeric results are not reported in the available text, indicate robustness to model and data heterogeneity and report advantage over transfer baselines including the pooled-MNI and the SGD-based transfer method.

Methodology in Plain English

The authors work with a standard linear model where each task's response is a linear function of covariates plus noise. They consider one target task and Q source tasks, all with p features but usually fewer samples than features, so the ordinary least-squares solution is not unique. In this regime the natural estimator is the minimum-ℓ2-norm interpolator — the fitted model with the smallest Euclidean length among all models that fit the data exactly.

Their proposed estimator, Transfer MNI, works in two steps. First, fit an MNI using only the source data. Second, fit a model using the target data, but among all models that interpolate the target data exactly, pick the one closest in Euclidean distance to the pre-trained source model. This "staying close" step is what lets them analytically separate what the target data provides from what the source data adds, and it yields the additive decomposition into a target-only term and a projected source term.

To evaluate the estimators, they define excess risk as the conditional mean squared prediction error on a fresh target sample, and split it into squared bias and variance. The transfer estimator's variance always includes an extra positive term, which they call variance inflation. They compute expected bias and variance exactly in the isotropic Gaussian design case (Theorem 1), giving closed-form formulas that reveal a trade-off governed by the feature dimension p, the source sample size n_q, the shift-to-signal ratio (SSR), and the signal-to-noise ratio (SNR). Comparing these formulas gives the exact conditions and optimal transfer size in Corollary 1.

For general covariances, they avoid assuming the source and target share an eigenbasis, and instead use effective rank quantities — r_k and R_k, which summarize how quickly the covariance spectrum decays. Assuming each single-task MNI is benign (rapid spectral decay up to an intrinsic dimension k*, with heavy effective rank in the tail), they derive non-asymptotic bounds on the bias, the variance inflation, and a lower bound on the excess risk, valid with stated high probability over the random design and noise. The free-lunch covariate shift result then follows from tracking how uniform rescaling of the source covariance propagates into those bounds. Finally, they turn the theory into practice with a cross-validation scheme that compares held-out target losses of the transfer and target-only estimators, using the resulting comparison as a proxy for the unknown excess risks, and weight an ensemble of the surviving transfer estimates accordingly.

Why This Matters

This work opens a largely unexplored intersection: prior work studied benign overfitting and transfer learning separately, with the closest related efforts either ignoring target data during training (out-of-distribution MNI analysis), ignoring model shift, focusing on underparameterized regimes, or pooling sources without guarding against harmful shifts. The paper's central message is that transfer in this regime is not automatically beneficial — it carries a provable variance cost, has a precise positive-transfer region, and has an optimal amount of source data that is generally not "as much as possible."

Real-world applications suggested by the setting:

• High-dimensional prediction with related cohorts: borrowing from auxiliary datasets (for example, different experimental batches or subpopulations) when fitting a model with more features than observations.
• Scientific regression with small target studies: domains such as genomics, imaging, or sensor studies where the target dataset is small but related datasets are abundant.
• Multi-source data aggregation pipelines: deciding which external datasets to include, and how much of each, rather than pooling everything indiscriminately.
• Overparameterized model deployment: settings where interpolation-based, regularization-free models are used and practitioners want a principled transfer recipe rather than a heuristic fine-tuning.

Industry relevance: The paper supplies both a diagnostic (cross-validated detection of informative sources) and a constructive estimator, which are directly relevant to teams that fine-tune large models on small target datasets and need guidance on whether more source data helps or harms. The finding that exceeding an optimal transfer size degrades performance gives a concrete counter-narrative to "more data is always better."

Future Directions

• Tightening the variance-inflation bound. The authors note that their upper bound on variance inflation contains a reciprocal of the smallest source eigenvalue, reflecting the lack of simultaneous diagonalizability; they leave the search for a tractable covariance structure that sharpens this bound to future work.

• Extending beyond the Euclidean norm. The paper defers discussion of minimum-norm interpolators under norms other than ℓ2 to its appendix, suggesting further generalization of the transfer mechanism.

• Theory for the ensemble and source-selection procedure. The paper proposes the cross-validated informative-source detector and weighted ensemble as operational tools; a natural next step is a theoretical characterization of their risk relative to the oracle informative set.

• Beyond the linear interpolator setting. The related-work discussion points to open questions around stochastic-gradient-descent-based pre-training and fine-tuning, and around covariate shifts beyond the uniform-upscaling regime studied here, including shifts between non-commuting covariances.

Target Audience

This paper is most valuable to theoretical statisticians and machine learning researchers working on high-dimensional regression, transfer learning theory, and the mathematics of overparameterization and benign overfitting. It will also interest methodologically oriented practitioners — for example, researchers in scientific or biomedical data analysis — who routinely transfer models from related datasets to small target datasets and need principled criteria for which sources to use and how much data to draw from them. Readers without a background in concentration inequalities, effective rank conditions, and excess risk decomposition will find the theoretical sections demanding, though the two-step estimator and the qualitative conclusions (variance cost of transfer, optimal transfer size, free-lunch covariate shift, and informative-source detection) are accessible on their own.

Authors’ abstract

Transfer learning is a key component of modern machine learning, enhancing the performance of target tasks by leveraging diverse data sources. Simultaneously, overparameterized models such as the minimum-$\ell_2$-norm interpolator (MNI) in high-dimensional linear regression have garnered significant attention for their remarkable generalization capabilities, a property known as benign overfitting. Despite their individual importance, the intersection of transfer learning and MNI remains largely unexplored. Our research bridges this gap by proposing a novel two-step Transfer MNI approach and analyzing its trade-offs. We characterize its non-asymptotic excess risk and identify conditions under which it outperforms the target-only MNI. Our analysis reveals free-lunch covariate shift regimes, where leveraging heterogeneous data yields the benefit of knowledge transfer at limited cost. To operationalize our findings, we develop a data-driven procedure to detect informative sources and introduce an ensemble method incorporating multiple informative Transfer MNIs. Finite-sample experiments demonstrate the robustness of our methods to model and data heterogeneity, confirming their advantage.