Transfer Learning in Nonparametric Regression with Deep ReLU Networks

AI summary

Overview

Research area: Statistical machine learning theory — specifically nonparametric regression, transfer learning, and approximation/estimation theory for deep ReLU neural networks.

Technical level: Advanced. The paper is a theoretical statistics contribution built on empirical process arguments, covering-number complexity conditions, and hierarchical composition function classes. Readers need comfort with $\mathcal{L}_2$ risk bounds, nonparametric minimax rates, and neural network approximation theory.

Scope in one sentence: The paper proposes a two-stage "offset" transfer learning framework for nonparametric regression across multiple groups, proves general $\mathcal{L}_2$ error bounds and explicit convergence rates when deep ReLU networks are used, and shows when transfer yields strictly faster rates than estimating each group separately.

Publication details: arXiv:2608.20255v1 [stat.ML], 20 Aug 2026, licensed CC BY 4.0. Authors: Junpeng Ren and Oscar Hernan Madrid Padilla (UCLA, Department of Statistics and Data Science), Carlos Misael Madrid Padilla (Washington University in St. Louis), Yanzhen Chen (Hong Kong University of Science and Technology).

What This Paper Is About

Modern datasets often pool observations from several related groups that behave similarly but not identically. The paper asks how to estimate each group's own regression function in that setting, without throwing away the shared information and without assuming a rigid parametric form. The proposed answer: learn one overall mean function from all the data pooled together, then learn a small per-group correction on top of it — and prove how fast that two-step procedure converges when the estimators are deep ReLU networks.

Key Contributions

1. A general transfer learning theory for nonparametric regression. Theorem 3.2 gives an $\mathcal{L}_2$ error upper bound for a broad class of nonparametric estimators under mild complexity and noise conditions that accommodate sub-exponential noise. The framework allows the number of groups $L$ to grow with the total sample size $n$, unlike existing transfer learning results that focus on a fixed number of groups.

2. Explicit rates for deep ReLU networks that overcome the curse of dimensionality. Theorem 3.4 and Corollary 3.8 give explicit upper bounds when both stages are dense ReLU networks under hierarchical composition models, showing that the ability to beat the curse of dimensionality is preserved inside the transfer learning framework.

3. Identification of regimes where transfer is provably beneficial. The paper characterizes conditions under which two-stage pretraining achieves strictly faster convergence than estimating each group's function separately, including cases where the offset function is simpler than the target and where pooling increases effective sample size.

4. New and recovered guarantees as byproducts of the general framework. Appendix D.1 yields what the authors describe as the first convergence guarantees for trend filtering (Tibshirani, 2014) in a transfer learning setting, and Appendix D.2 recovers existing convergence rates of Wang et al. (2016) for orthogonal series regression using Sobolev sieves, up to logarithmic factors.

Main Findings

• The two-stage rate decomposes additively. Under the general framework, Theorem 3.2 bounds $|g_{0,\ell}-\hat{g}\ell|{\mathcal{L}2}^2$ by terms for the first-stage error $r_n$ plus second-stage approximation error $\phi{\ell,n}$ and estimation error $\mathcal{B}n^2\delta^2 + \mathcal{U}n^2 \eta{\ell,n}(\delta)/(n\underline{\pi}\ell)$, valid provided $\delta^2 n \underline{\pi}_\ell \to \infty$.

• Neural network transfer learning preserves dimension-free structure. With hierarchical composition models, the first-stage approximation error is $\phi_n = \max_{(p,K)\in\mathcal{P}0} n^{-2p/(2p+K)}$ and the second-stage error is $\phi{\ell,n} = \max_{(p,K)\in\mathcal{P}\ell} (n\underline{\pi}\ell)^{-2p/(2p+K)}$, up to logarithmic factors — rates governed by the intrinsic smoothness and intrinsic dimension rather than the ambient dimension.

• Ignoring logarithms, the ReLU pretraining rate is $\phi_n + \phi_{\ell,n}$. This holds for the sample-splitting variant in Corollary 3.8, and for Theorem 3.4 up to a factor involving $\underline{\pi}\ell$. Transfer helps when the overall mean $\bar{f}$ and offset $G\ell$ are less complex than the group function $g_{0,\ell}$ itself.

• Sample splitting relaxes a complexity constraint and can give a tighter rate. Theorem 3.7 replaces one localized complexity condition with an additional $\mathcal{A}n^2 \log n/(n\underline{\pi}\ell)$ term; Corollary 3.8 removes the $\underline{\pi}_\ell$ factor from the denominator that appears in Theorem 3.4.

• Concrete example shows neural transfer learning is robust to vanishing group proportions. In a 10-dimensional example with three groups, where $g_{0,\ell}$ and $\bar{f}$ have smoothness $p=1$ and ambient dimension $K=10$ while the offset $G_\ell$ has smoothness $p=3/2$ and intrinsic dimension $K=4$, Table 1 reports: classical no-transfer and neural no-transfer rates of $(n\pi_1)^{-1/6}$; classical transfer degrading to $(n\pi_1)^{-3/13}$ for $n^{-11/18}\lesssim \pi_1 \lesssim n^{-5/18}$; and neural transfer remaining at $n^{-1/6}$ until $\pi_1 \ll n^{-11/18}$, then transitioning to $(n\pi_1)^{-3/7}$ — strictly faster than classical transfer learning.

• The authors do not claim optimality. Remark 3.6 states that lower bounds for the hierarchical function class are only known in very specific cases, citing Remark 2 of Kohler and Langer (2021).

• A conjecture about an artifact in the proof. Remark 3.9 conjectures that the appearance of $\underline{\pi}\ell$ in Theorem 3.4 is an artifact of the proof, noting that in practice the estimator performs better without sample splitting, though both results coincide when $\underline{\pi}\ell \asymp 1$.

• Simulations and two real-data experiments are reported. The paper states that simulations across diverse scenarios and two real-data applications show the proposed estimators consistently outperform a range of competitors. The specific datasets, sample sizes, and error values are not reported in the provided content.

Methodology in Plain English

The paper sets up a regression model in which each observation belongs to one of $L$ groups, indexed by $z_i \in {1,\dots,L}$. The response is written as a sum of two parts: a shared function $f_0(x)$ common to every group, plus a group-specific deviation $f_{0,\ell}(x)$ that only applies to group $\ell$. The quantity of interest is the group-specific conditional mean $g_{0,\ell}(x) = f_0(x) + f_{0,\ell}(x)$.

The estimation procedure has two stages, echoing the pretrain-then-fine-tune pattern familiar from modern machine learning:

1. Stage one (pooling). Fit a single function $\hat{f}$ by least squares on data from all groups combined. This targets the overall mean $\bar{f}(x) = \mathbb{E}(Y \mid X=x)$, which equals $f_0(x)$ plus a covariate-dependent weighted average of the group deviations. The fitted function is then clipped at a level $\mathcal{A}_n$ to control extremes.

2. Stage two (offsets). For each group $\ell$, fit a separate function $\hat{f}\ell$ to the residuals within that group only — that is, regress $y_i - \hat{f}{\mathcal{A}n}(x_i)$ on $x_i$ using only observations with $z_i = \ell$. This targets the offset $G\ell(x)$, defined in equation (6) as the group deviation minus the weighted average of all group deviations.

The final estimator is the additive combination $\hat{g}\ell(x) = \hat{f}{\mathcal{A}n}(x) + \hat{f}{\ell,\mathcal{B}_n}(x)$.

For the theory, the authors work on $\mathcal{X} = [0,1]^d$ and impose an overlap assumption (Assumption 3.1) that each group's membership probability given covariates stays bounded away from 0 and 1 — analogous to the overlap condition in causal inference. They then bound the error of the first stage using a result from Padilla et al. (2024a), and propagate that bound into the second stage using localized covering-number complexity conditions on each group's function class. When specialized to neural networks, both stages use dense fully connected ReLU networks $\mathcal{F}(M,\nu)$ with shared hidden-layer width, and the target functions are assumed to satisfy a hierarchical compositional structure (Assumption 3.3) with component functions of potentially different smoothness $p_m = q_m + s_m$ and different input dimensions $K_m$.

Why This Matters

Impact on research. The paper sits at the intersection of three active lines of work — transfer learning, nonparametric regression, and deep network approximation theory — and gives a unified framework rather than an analysis of one specific estimator. Two features distinguish it from prior nonparametric transfer results (Wang and Schneider, 2015; Wang et al., 2016; Du et al., 2017; Lin and Reimherr, 2024; Cai and Pu, 2024), which largely use kernel-based or local polynomial estimators: the general bound applies to a broad estimator class rather than one method, and the neural network instantiation achieves rates that overcome the curse of dimensionality under hierarchical composition. The paper also differs from representation-learning-based nonparametric transfer with deep ReLU networks (Jiao et al., 2024) by adopting an additive formulation and relaxing conditions on noise and network constraints. Beyond that, it unifies transfer learning (data-scarce target) and multi-task learning (all groups estimated on equal footing) within one theory, and produces the first convergence guarantees for trend filtering in a transfer setting.

Real-world applications cited by the paper:

• Natural language processing — transfer learning has seen notable empirical success here (Daumé III, 2007; Howard and Ruder, 2018).
• Computer vision — image recognition and domain adaptation (Gong et al., 2012; Tzeng et al., 2017).
• Bioinformatics — leveraging related biological datasets (Schweikert et al., 2008).
• Transportation — transfer across traffic or mobility domains (Lu et al., 2019).
• Epidemiology — transfer in disease modeling (Apostolopoulos and Mpesiana, 2020).

Industry relevance. The theoretical justification for pretraining matters commercially because modern architectures — ImageNet (Deng et al., 2009), ResNet (He et al., 2016), and GPT (Brown et al., 2020) — all rely on the pretrain-then-adapt pattern. The paper's results formalize when that pattern pays off: when the pooled mean and the per-group offset are simpler objects than the group function itself, and when pooling enlarges the effective sample size for the shared component even as the per-group sample shrinks. Table 1's example is directly relevant to practitioners facing many small, similar datasets, showing that neural transfer learning keeps a favorable rate while classical estimators degrade badly when a group becomes rare.

Future Directions

• Lower bounds and optimality. The paper explicitly does not claim optimality (Remark 3.6), noting that lower bounds for the hierarchical function class are only known in very specific cases. Establishing matching lower bounds for the two-stage transfer framework is the natural next question.

• Resolving the sample-splitting gap. Remark 3.9 conjectures that the $\underline{\pi}_\ell$ factor in Theorem 3.4 is an artifact of the proof, since practice favors the non-sample-split estimator. A proof technique that removes it without splitting would close the gap between theory and observed behavior.

• Relaxing structural assumptions. The framework relies on the additive model in equation (1) and on the overlap condition in Assumption 3.1. Extending the theory to nonlinear links between shared and group-specific functions — as Du et al. (2017) do for kernels — and to settings where group proportions approach the boundary is an open direction.

• Broadening the estimator class further. The authors note the general result already covers trend filtering and Sobolev-sieve orthogonal series regression in Appendices D.1 and D.2, suggesting the framework could be instantiated with additional estimator families and architectures beyond dense ReLU networks.

• The paper's conclusion (Section 5) discusses potential extensions, but the specific extensions proposed are not included in the provided content.

Target Audience

This paper is aimed at theoretical statisticians and machine learning theorists working on nonparametric estimation, transfer learning, or deep network approximation theory. It is most useful to readers already familiar with empirical process techniques, covering numbers, and minimax convergence rates, who want to understand when pretraining provably beats training a model per group. Applied researchers with a strong mathematical background — for example, those designing multi-group prediction systems in healthcare, transportation, or NLP — can also extract the practical takeaway that transfer is most valuable when the shared structure and the group-specific corrections are each simpler than the unpooled target, but should treat the technical derivations as reference material rather than a how-to guide.

Authors’ abstract

This paper develops a general transfer learning framework for nonparametric regression with data consisting of multiple groups. Under the assumption that groups share a common structure along with group-specific deviations in additive form, the proposed method employs a two-stage offset learning procedure: the first stage pools data from all groups to estimate an overall mean function, and the second stage estimates offsets for each group, yielding final group-level estimators through additive combination. Upper bounds on the $\mathcal L_2$ error are established for the proposed framework, covering a broad class of nonparametric estimators under mild complexity and noise conditions. When instantiated with deep ReLU networks, explicit convergence rates are derived under hierarchical composition models, demonstrating the ability to overcome the curse of dimensionality. Conditions that enable positive transfer with faster rates are considered, including learning with simpler functions and data augmentation through pooling samples across groups. Various simulations and real-data experiments further validate the effectiveness of the proposed method.