Co-Regularization Enhances Knowledge Transfer in High Dimensions

Thank you for your comments! Please see our responses in the following.

1. The paper only discussed method and theory for the GLMs, while even the small datasets may have more complicated hypothesis space. Thus, further discussion for more general model is necessary. Discussion for more general model is suggested. Will the CoRT still work for more complicated non-linear model? What about the convergence?

We appreciate the insightful comment.

The key idea of CoRT can actually generalize beyond GLMs naturally. This is because the core of CoRT is to simultaneously regularize the complexity of (1) the bias between source and target models and (2) the target model itself; see Eq. (1). Therefore, for more generic cases, we can modify the objective to

$$\sum\_{k \in \mathcal{S}} \left( \text{Loss}( f^{(0)} + \delta^{(k)}, \mathcal{D}^{(k)} ) + \lambda\_{k} \\|\delta^{(k)} \\|\_{\mathcal{H}\_{1}} \right) + \text{Loss}( f^{(0)} , \mathcal{D}^{(0)} ) + \lambda\_{0} \\| f^{(0)} \\|\_{\mathcal{H}\_{2}}$$

where $\mathcal{H_{1}}$, $\mathcal{H}_{2}$ represent generic hypothesis spaces and $f$ denote the parameters/models of interests. Therefore, the non-linear models can be incorporated into the CoRT framework naturally by setting the hypothesis spaces as specified function spaces, like RKHS, class of DNN, etc. Besides, the adaptive techniques (Algorithm 2) for excluding outlier sources are also model-agnostic, which thus can be generalized well to other models and used broadly by the community.

We focused on high-dimensional GLMs for two primary reasons.

• First, since our motivation is to provide a better knowledge transfer training scheme than the two-step seminal frameworks [1] and [2] paradigm, we follow their setting which allows us to make head-to-head comparison both theoretically and empirically. Besides, the generalization of CoRT is analogous to the generalization of the two-step frameworks, where the techniques of regularizing the bias between source and target parameters in GLMs have already been naturally extended to different parametric/non-parametric models, from graphic models, to autoregressive models, and even neural networks.

• Second, the high-dimensional GLMs has already been a relatively broad model family and has shown its excellent versatility in various scientify fields such as biological and social applications. Therefore, we believe the study of transfer learning in GLMs provides researchers from different fields with powerful tools, while the extension of CoRT to other models can be natural.

For the convergence results, we believe the form of the error bounds for different models should stay in a similar fashion as long as it applies to the CoRT framework, but their exact form should be case by case since the model assumptions and analysis tools are different. This is consistent with prior literature: when the two-step method was extended from GLMs to nonparametric regression [3], autoregressive models [4], or Fréchet regression [5], the form of the convergence rate was preserved, with only the model-specific rates being updated, e.g., replacing the sparse learning parametric rate by the nonparametric rate in [3]. Therefore, we expect the form of convergence rates for using CoRT on different models will stay similar asymptotically, but just having the components being more model-specified. We believe this can be an interesting future direction.

2. The objective of the CoRT, i.e. Eq. (1), seeks for optimal solution of both source and target model, which is harder than only training the optimal target model. The gradients may cause negative impact to each other. Further, I am curious about whether the training of target model will be disturbed by the gradient of the source models, making it harder to reach the optimal solution.

We thank the reviewer for this insightful question.

As the objective indicates, the training is not to seek optimal solutions for target and source models, i.e., $\beta^{(0)}$ and $\beta^{(k)}$. Instead, we reparameterize $\beta^{(k)}$ as $\beta^{(0)} +\delta^{(k)}$, seeking for optimal solution for the target coefficient and the bias terms, i.e., $\delta^{(k)}$. While simultaneously optimizing them can be challenging, our optimization procedure, based on coordinate descent algorithms (for nonconvex penalized regression [6] via package ncvreg), iteratively updates the target model $\beta^{(0)}$ and each bias $\delta^{(k)}$ term. It effectively decouples the optimization steps and ensures a more stable convergence path by isolating the high-dimensional gradient updates. This procedure guarantees convergence to a stationary point [7].

We also mentioned this in the appendix, see line 659.

3. The idea of Co-Regularization seems to be similar to the multi-task learning. The similarities and differences between them should be discussed.

It is worth noting that multi-task learning is closely related to transfer learning, though their objectives differ. Multi-task learning aims to solve multiple learning tasks simultaneously by leveraging shared structure across tasks, like the Data Shared Lasso framework [8, 9] in high dimensions. In contrast, transfer learning focuses on improving performance on a specific target task by transferring knowledge from source data.

We believe that ``CoRT is similar to multi-task learning'' is due to the fact that the objective of CoRT, i.e., Eq. (1), looks like the objective in multi-task learning. However, we would like to note that the CoRT objective is actually asymmetric w.r.t. the source and target, as the last two terms put specific emphasis on the target model (thus focus on optimizing the learning over the target domain). This is the core of CoRT, as it adopts the spirit of multi-task learning via a shared objective function across domains while having its own special focus on the target. We will add related discussion in the revised version once more space is allowed.

4. Since all models are known in the simulation experiment, numerical validation of the error bounds and the convergence behavior will more strongly support the main theorems of the paper.

Yes, all the simulation studies, as well as the real data study, were designed to support the theoretical results. The trends in the experiments all illustrate the convergence behavior of the main theorems, and are discussed in the result description in Sections 4 and 5.

Reference

[1] Li, S., Cai, T. T., and Li, H. (2022). Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology.

[2] Tian, Y. and Feng, Y. (2023). Transfer learning under high-dimensional generalized linear models. Journal of the American Statistical Association.

[3] Lin, H., & Reimherr, M. (2024). Smoothness adaptive hypothesis transfer learning. ICML.

[4] Ma, M., & Safikhani, A. (2025). Transfer Learning for High-dimensional Reduced Rank Time Series Models. AISTATS

[5] Zhang, K., Zhang, S., Zhou, D., & Zhou, Y. (2025). Wasserstein Transfer Learning. arXiv.

[6] Breheny, P., & Huang, J. (2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. The annals of applied statistics.

[7] Loh, P. L., & Wainwright, M. J. (2015). Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima. The Journal of Machine Learning Research.

[8] Samuel M Gross and Robert Tibshirani. (2016). Data shared lasso: A novel tool to discover uplift. Computational Statistics & Data Analysis.

[9] Edouard Ollier and Vivian Viallon. (2017). Regression modelling on stratified data with the lasso. Biometrika.

Thank you for your comments! Please see our responses in the following.

1. The practicality of the paper is of concern, as even though simultaneously training on source and target domain would intuitively leads to better estimation error, having access to both source data at the time of learning in target domain is a rather restrictive assumption. As the establishment of two-step training paradigm is partially conditioned on the difficulty of training on both source and target data. For example, the proposed approach would make switching to new target domain extremely expensive, as the source data need to be involved all the time.

We thank the reviewer for the insightful comment. We agree that the proposal CoRT may appear restrictive in some scenarios, but we believe the approach remains both meaningful and practical for the following reasons:

(1) First, we’d like to point out that those seminal works for the two-step paradigm for high-dimensional data, such as Trans-GLM [1], also necessitated access to source datasets during their training phase. In these works, having access to source data is not only common but often necessary to ensure more robust knowledge transfer across domains as they need to control the training on source domains. Therefore, our work actually considers the same setting as those seminal transfer learning works in high-dimensional GLMs. The proposed joint training scheme (co-regularization in our model context) advances the two-step paradigm under the same setting by being more tolerant to less similar sources (better utilities knowledge from these sources.)

(2) Second, having access to both source and target data is common in the field of transfer learning. Methodologically, researchers from core-ML, statistics and deep learning community have make such assumptions to achieve design algorithm that can transfer knowledge better; see [2,3,4,5,6] and more reference therein. Besides, this assumptions is also practically satisfied in some specific studies, such as biological applications. For example, [7] investigates genetic effects on obesity in a multicenter study where data from both Black/African American (target) and White (source) populations are available. In [8], methods for multi-omics data treat one omic as the target while the others serve as sources.

(3) From a theoretical aspect, access the source data ensures the error bounds, such as convergence rates, to incorporate both source and target size. This allows us to provide insight into when transfer learning outperforms the non-transfer case without directly making assumptions on pre-trained models (like those domain adaptation works ranging from 2006 to 2010s).

(4) Finally, although CoRT requires retraining when switching to new target tasks, distributed techniques, like divide-and-conquer, can be used to practically improved the computational efficiency. The algorithm and theory for divide-and-conquer high-dimensional statistical models have been well-developed with theory to guarantee their performance. These distributed algorithms can be directly plug-in into the CoRT framework in practice when computation expense is a concern. However, this is out of the scope of current manuscript.

At a high level, our vision of the proposed CoRT is that it is a complement to the modern transfer learning community, rather than a replacement of the successful two-step paradigm. We believe our work does provide a new perspective to better leverage the source knowledge. To provide a balanced perspective, we will also dedicate a discussion to the potential limitations and challenges of this method, together with potential distributed style solutions in practice, in the revised version.

2. Even though the specific combinations of components are unique, the idea of one step learning for both source and target domain are not exactly brand new in the literature of domain adaptation.

We agree with the reviewer that the one step learning for both source and target is not exactly brand new in the community, especially in computer vision field. These one step approaches are also typically require access to both source and target data and retraining when switching to new target task (this actually also reasonalizes our setting better in question 1).

Our contribution, however, is specifically situated within the high-dimensional $(p\gg n)$ setting, where the research landscape is markedly different.

In this high-dimensional regime, the community has almost exclusively followed the two-step paradigm since the seminal works of [1,4,9]. This two-step approach has been the default for adapting transfer learning to various high-dimensional models. Our work, CoRT, challenges this status quo by introducing a fundamentally different, one-step paradigm tailored for this setting. The novelty of our work is therefore twofold:

• Methodological contribution: The design of CoRT departs from the dominant paradigm, leverages the source knowledge better, and accommodates less similar source, which are all illustrated by the theories and experiments.

• Technical Contributions: Analyzing a joint-training scheme requires a completely different theoretical approach than analyzing a two-step procedure. We develope new mathematical tools to derive the results for joint training (CoRT), which, we believe, can benefit the community in the future.

3. There are some minor presentation issues, including really small font size in Figure 1, making them hard to read.

Thank you for pointing this out. We admit that it is hard to trade off between informativeness and readability with multiple figures are presented. We will revise the figures to improve readability once more spaces are allowed.

Reference

[1] Tian, Y. and Feng, Y. (2023). Transfer learning under high-dimensional generalized linear models. Journal of the American Statistical Association, 118(544):2684–2697.

[2] Dai, Wenyuan, et al. (2006) Eigentransfer: a unified framework for transfer learning. ICML.

[3] Du, Simon S., et al. (2017) Hypothesis transfer learning via transformation functions. NeurIPS.

[4] Li, S., Cai, T. T., & Li, H. (2022). Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology.

[5] Sun, B., & Saenko, K. (2016, October). Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision.

[6] Ganin, Yaroslav, et al. (2016) Domain-adversarial training of neural networks. Journal of machine learning research.

[7] Li, S., Cai, T., & Duan, R. (2023). Targeting underrepresented populations in precision medicine: A federated transfer learning approach. The annals of applied statistics.

[8] Lac, L., Leung, C. K., & Hu, P. (2024). Computational frameworks integrating deep learning and statistical models in mining multimodal omics data. Journal of Biomedical Informatics.

[9] Bastani, H. (2021). Predicting with proxies: Transfer learning in high dimension. Management Science.

Thank you for your comments! Please see our responses in the following.

1. The explanation of the proposed method is somewhat difficult to understand. For example, in line 49 of the introduction, it should be clarified how adaptive identification of outlier sources is achieved and how this information is utilized during training.

Thanks for the comment. Due to the space limit, we can only briefly mention that it is achieved via the majority vote mechanism in line 51. This information is then utilized by excluded identified outlier sources in the Co-regularization process. We will add additional details once there is more space.

2. An explanation is needed for what "E" and the "acute" symbol in line 94 mean.

E stands for expectation, and $\psi’$ is the first-order derivative of the cumulant function used in GLMs, see [1].

3. The effectiveness of the proposed method could be further emphasized by presenting results for each source without the adaptive technique.

We believe the reviewer is referring to Figure 2. As we mentioned on line 298: The Model starts with ``pooled'', which means that we directly pool all sources without an adaptive procedure. Therefore, Pooled-CoRT and Pooled-Trans-GLM are the results without the adaptive techniques, i.e., all sources are directly used in the proposed co-regularization scheme. Note that the goal of transfer learning is to estimate the estimation error of the target data. Current results on Figure 2 show the averaged error of all targets, which should have similar patterns to the results of each target.

4. In what scenarios is the proposed method more effective (excluding the size of the target data)?

As Table 1 indicates, to outperform single-task learning (naive-Lasso), either a similar source (small $h$) or a larger transferable source sample size (larger $N$) can lead to smaller errors. Compared to the existing two-step TL, we allow less similar sources to achieve the same improved error bounds.

5. The authors mention that their method would be effective when applied to neural networks. What considerations should be made when applying it to DNNs?

In the discussion, we only mentioned the limitation of DNN when applied to a very small dataset. Small-sized datasets with extremely large-sized covariates are common in certain fields like social and biological sciences, where high-dimensional statistical models (and the TL techniques in high-dimensions) usually outperform DNN-based methods.

However, our framework can generalize to other models naturally as the key idea is to simultaneously regularize (1) the distance between source and target parameters and (2) the target parameter itself; see Eq. (1).

Therefore, for more generic cases, we can modify the objective to

$$\sum\_{k \in \mathcal{S}} \left( \text{Loss}( f^{(0)} + \delta^{(k)}, \mathcal{D}^{(k)} ) + \lambda\_{k} \\|\delta^{(k)} \\|\_{\mathcal{H}\_{1}} \right) + \text{Loss}( f^{(0)} , \mathcal{D}^{(0)} ) + \lambda\_{0} \\| f^{(0)} \\|\_{\mathcal{H}\_{2}}$$

where $\mathcal{H_{1}}$, $\mathcal{H}_{2}$ represent generic model spaces and $f$ denote the parameters/models of interests. When extended to DNN, we just need to set these $f$ as the neural network. One consideration would be that these DNN needs to be parameterized similarly.

6. Instead of dropping specific source domains, is there a way to maximize the knowledge that can be obtained from each source domain (e.g., sample-wise)?

One way to maximize the knowledge from each source domain is to aggregate models from each source domain by different aggregation techniques, like [2] and [3]. Sample-wise approach could be possible, but it is more related to the topic in active learning.

That being said, either aggregation or sample-wise knowledge transfer typically comes with extensive computational cost. For example, aggregation-based needs to calculate source models multiple times, and sample-wise typically requires one to score each sample within the source domain. On the other hand, source-wise selection achieves a better trade-off between performance and efficiency.

Reference

[1] Loh, P. L., & Wainwright, M. J. (2015). Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima. The Journal of Machine Learning Research.

[2] Li, S., Cai, T. T., and Li, H. (2022). Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology.

[3] Lin, Haotian, and Matthew Reimherr. (2024) On Hypothesis Transfer Learning of Functional Linear Models. ICML.

Thank you for your comments! Please see our responses in the following.

1. Although the authors claim that the methodology can be extended to other types of models, the paper is focused on GLMs, so it becomes unclear how broadly CoRT can generalize beyond this family.

We believe the key idea of co-regularization can generalize beyond GLMs naturally, as the key idea is to simultaneously regularize (1) the distance between source and target parameters and (2) the target parameter itself; see Eq. (1).

Therefore, for more generic cases, we can modify the objective to

$$\sum\_{k \in \mathcal{S}} \left( \text{Loss}( f^{(0)} + \delta^{(k)}, \mathcal{D}^{(k)} ) + \lambda\_{k} \\|\delta^{(k)} \\|\_{\mathcal{H}\_{1}} \right) + \text{Loss}( f^{(0)} , \mathcal{D}^{(0)} ) + \lambda\_{0} \\| f^{(0)} \\|\_{\mathcal{H}\_{2}}$$

where $\mathcal{H_{1}}$, $\mathcal{H}_{2}$ represent generic hypothesis spaces and $f$ denote the parameters/models of interests. This allows us to incorporate most linear/non-linear models that learned via empirical risk minimization. Besides, the adaptive techniques (Algorithm 2) for excluding outlier sources are also model-agnostics, which thus can be generalized well to other models and used broadly by the community.

We focused on high-dimensional GLMs for two primary reasons.

• First, since our motivation is to provide a better knowledge transfer training scheme than the two-step seminal frameworks [1] and [2] paradigm, we follow their setting which allows us to make head-to-head comparison both theoretically and empirically. Besides, the generalization of CoRT is analogous to the generalization of the two-step frameworks, where the techniques of regularizing the bias between source and target parameters in GLMs has already been naturally extended to different parametric/non-parametric models, from graphic models, to autoregressive models, and even neural networks.

• Second, the high-dimensional GLMs has already been a relatively broad model family and has shown its excellent versatility in various scientify fields such as biological and social applications. Therefore, we believe the study of transfer learning in GLMs provides researchers from different fields with powerful tools, while the extension of CoRT to other models can be natural.

2. There’s no discussion on computational complexity of the derived algorithms. Please elaborate on runtime complexity and scalability to large K (number of sources) or p (dimensionality).

To optimize objective Eq. (1), we leverage the package ncvreg [3], which contains a very efficient and robust optimization coordinate-descent algorithm.

We conduct the following two additional experiments to investigate the computational cost w.r.t. $K$ and $p$, with the same settings in Section 4. For our proposal, we consider two regularizers, Lasso and SCAD. In scenario 1 (Table 1), we fix $p=300$ and let the number of sources range from 10 to 30. In scenario 2 (Table 2), we fix $n_{0} = 50$ and let $p$ range from $300$ to $700$. Results are in seconds and averaged from 100 replicates.

The computation time of our proposed CoRT (Lasso) and CoRT (SCAD) increases as K or p increases. This is because we use the coordinate descent algorithm [3] to implement our methods. In either case, the increasing |S| or p leads to more parameters to be estimated since our methods need to find the contrast $\delta^{(k)} = \beta^{k} - \beta^{0}$. However, we note that for those high-dimensional data where $p \gg n$, the CoRT still achieve great scalability given its running time.

We will include the results in the revised version.

3. The real-world dataset (COVID-19 mortality) is insightful but relatively small.

Thank you for the acknowledgement. Datasets in biological or social sciences typically come with small sizes (at each site/domain). While DNN-based TL methods typically work well in large datasets, they tend to perform poorly on small datasets (small size but large dimensionality). Our experiments actually illustrate its effectiveness to unique challenges in high dimensions.

4. Regarding the derived theoretical guarantees, how do these relate to or differ from discrepancy-based domain adaptation methods in terms of assumptions and guarantees?

This is a great question. It can be long to have a detailed discussion, so we try to make it concise but clear. At a high level, the theoretical study of domain adaptation (transfer learning) consists of two lines.

The first is, as you mentioned, ``discrepancy-based’’ results; see [4,5] and etc. They obtain model-agnostic and divergence-centric bounds which characterize worst-case performance, making bounds tend to be general and overly pessimistic, particularly when data from the target domain is available, making them less instructive.

The second one (where our work falls in) is based on more refined assumptions than the first line. For example, works in this line typically assume distribution shifts (our work is a combination of model and covariate shifts) and access to unlabeled/labeled data on the target domain. These settings bring more refined control over the discrepancy between domains, allowing researchers to design specific transfer learning algorithms. The statistical properties and guarantees are also more optimistic, as we can conduct inference on the parameters (Section 2.3) and also investigate the minimax optimality.

5. Are you willing to release the source code implementing your methods and experiments?

Yes, we will release the corresponding codes after acceptance. Thank you very much.

Reference

[1] Li, S., Cai, T. T., and Li, H. (2022). Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology.

[2] Tian, Y. and Feng, Y. (2023). Transfer learning under high-dimensional generalized linear models. Journal of the American Statistical Association.

[3] Breheny, P., & Huang, J. (2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. The annals of applied statistics.

[4] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. (2006) Analysis of representations for domain adaptation. NeurIPS.

[5] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms.