Transfer Learning Under High-Dimensional Graph Convolutional Regression Model for Node Classification

We thank the reviewer for the dedicated and insightful review. Please see below for our response.

Comment: (1) Comparison with more transfer learning methods. (2) Hyperparameter tuning for each method.

Response: Thank you for your great suggestions. In response, we have made the following improvements to our manuscript: (1) We have added an additional transfer learning approach to our experiments: the Pretrained GNN method as described in [1]. This method involves pretraining a Graph Neural Network on the source domain and then fine-tuning it on the target domain, serving as a strong baseline in graph transfer learning tasks. (2) We have employed a cross-validation strategy for each method to select the optimal hyperparameters. Specifically, we performed grid searches over key hyperparameters such as learning rates, regularization parameters, and the number of layers, using a validation set to assess performance.

The updated results, incorporating the additional method and refined hyperparameter tuning, are presented in Table R4 below. As shown in Table R4, our proposed method, Trans-GCR, consistently outperforms the added Pretrained GNN method and other baseline models. This demonstrates the superior transfer learning capability of our approach in leveraging information from the source domain to improve performance on the target domain. In the updated manuscript, we have updated Table 1 to include these results.

Table R4: Averaged Micro F1 score (%) of various methods, over 10 replicates, with source training rate fixed at 0.75 and target training rate fixed at 0.03.

[1] W Hu, B Liu, J Gomes, M Zitnik, P Liang, V Pande, and J Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations (ICLR), 2020.

We thank the reviewer for the dedicated and insightful review. Please see below for our response.

Comment: GCR is based on an indeed very strong assumption, that is, there is a linear relationship between aggregated features and labels.

Response: Thank you for your comment. We agree with you that linear assumption is a restriction. In line "513-515" in the original manuscript, we have acknowledged that one limitation of our paper is restricted to linear relationship between aggregated features and labels. However, as the statistician George Box famously stated, "All models are wrong, but some are useful." While nonlinear models are indeed more general and capable of capturing complex relationships, linear models offer simplicity, interpretability, and computational efficiency, which are valuable in many practical applications.

The linear model for aggregated features and labels has been proven to be useful by existing literature. For example, the work by Wu et al. in "Simplifying Graph Convolutional Networks" (2019) demonstrated that removing nonlinearities between GCN layers and simplifying the architecture can still achieve competitive performance. Their findings suggest that much of the representational power of GCNs arises from the network structure and feature aggregation process rather than from complex nonlinear transformations.

Furthermore, the primary goal of our paper is not to propose a model that outperforms existing GCNs but to propose a statistical transfer learning method in a simplified linear network convolutional regression model. By focusing on a linear relationship, we aim to create a foundational model that is both analytically tractable and interpretable. This simplification allows us to derive theoretical guarantees of the transfer learning process on graphs, which will be more challenging with complex nonlinear models.

In summary, while the linear assumption is indeed a limitation, it allows us to provide a clear and interpretable framework for statistical transfer learning on graphs. Our linear approach serves as a starting point that can be extended in future work to incorporate nonlinearities.

We thank the reviewer for the dedicated and insightful review. Please see below for our response.

Comment: Triviality of Convergence Proof: The theoretical proof for convergence appears trivial. Although the paper claims that $Z$ is non-i.i.d., Assumption 4.1 indicates that $X$ is i.i.d., leading to a clear covariance matrix for $Z$ as $A^TA/np$. Following Modification 1 in Appendix C, previous theorems can be trivially adapted with straightforward modifications. Thus, the convergence proof lacks substantial novelty and rigor.

Response: Thank you for your comment. We will respond to each of your points individually.

1. Covariance matrix of $Z$. We would like to clarify that $A^TA/np$ is not related to the covariance matrix of $Z$. To see this, note that each observation $Z_i$ is $d$-dimensional, so the covariance matrix of $Z$ is of size $d \times d$, whereas $A^TA/np$ is $n \times n$. Additionally, $Z$ is obtained by pre-multiplying $X$ with $A$, not post-multiplying. Therefore, the covariance matrix of $Z$ is not equal to $A^TA/np$. The pre-multiplication by $A$ introduces dependence among the rows $X_1, \dots, X_n$, not the columns of $X_i$, which significantly complicates the analysis.

2. Following analysis after modification 1. While Modification 1 addresses certain aspects of the analysis, it is not the only adjustment required. In fact, Modification 3 involves the most substantial effort. A central challenge is that the observations ${Z_i}, {1 \leq i \leq n}$ are dependent. One of the critical aspects of analyzing high-dimensional models is establishing the restricted strong convexity/restricted eigenvalue condition (e.g., see [1]). Even for i.i.d. or independent observations, proving this condition has only recently been achieved through considerable work (e.g., see [2], [3]). For our case, the dependence among ${Z_i}, {1 \leq i \leq n}$ makes this significantly more challenging. Existing techniques for time series data with mixing conditions are not directly applicable due to the special network-type dependency structure in our problem. Specifically, the dependencies are governed by the network's adjacency matrix $A$, which introduces complex correlations that are neither time-based nor easily characterized by standard mixing conditions. To address this, we developed novel techniques to handle the network dependencies and to establish the RSC condition for our model. This forms a critical part of our proof, particularly in Lemma C.2.

[1] Bickel, Peter J., Ya’acov Ritov, and Alexandre B. Tsybakov. "Simultaneous analysis of Lasso and Dantzig selector." (2009): 1705-1732.

[2] Rudelson, Mark, and Shuheng Zhou. "Reconstruction from anisotropic random measurements." Conference on Learning Theory. JMLR Workshop and Conference Proceedings, 2012.

[3] Kasiviswanathan, Shiva Prasad, and Mark Rudelson. "Restricted isometry property under high correlations." arXiv preprint arXiv:1904.05510 (2019).

Comment: Limited Parameterization: According to Equation 2.2, it seems that only the final logits layer contains parameters $\beta$, while the preceding $S^MX$ lacks parameters. The absence of parameters in these earlier layers raises concerns about why only the last layer is parameterized, which could lead to over-smoothing due to unparameterized iterations of $SX$ and consequently limit the model’s expressiveness.

Response: Thank you for the insightful commets. Equation 2.2 is motivated by [1], which simplifies GCN by removing nonlinerity between GCN layers. Thus the $M$-layer GCN is simplified as $\hat{Y} = Softmax(SS \cdots S X \beta^{(1)}\beta^{(2)}\cdots\beta^{(M)})$. These weight parameters can be reparametrized as a single weight matrix $\beta= \beta^{(1)}\beta^{(2)}\cdots\beta^{(M)}$.

By reparametrizing the multiple parameters into one, we effectively capture the cumulative effect of parameterization across all iterations without explicitly parameterizing each step. This approach helps in reducing the complexity of the model and prevents issues like over-smoothing or overfitting that could arise from excessive parameterization, especially in scenarios with limited training data.

[1] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pp. 6861–6871. PMLR, 2019.

Comment about $\mathbf{S}^M$

Response: Thank you for pointing out the potential issue with $\mathbf{S}^M$ converging to a matrix of all ones as $M$ increases and for suggesting the APPNP-inspired formulation using $\alpha < 1$ to mitigate this indistinguishability problem.

We agree that incorporating such a mechanism could provide better control over the propagation process. We acknowledge its importance and plan to explore this direction in future research. We will include the following statement in the discussion section (section 7) of the manuscript.

" As $M$ increases, $\mathbf{S}^M$ converges to a matrix of ones, resulting in oversmoothing, where node representations become indistinguishable. Future work could explore adaptive propagation mechanisms, such as APPNP, to mitigate oversmoothing and preserve meaningful differentiation between node representations in deeper architectures. "

Thank you for your thoughtful feedback.

Comment about $\lambda$

Thank you for your comment. The differences arise because we reran the experiments during the revision process. Each result is averaged across 10 independent replications, where the training, validation, and testing splits differ for each replication. This introduces some natural variability, which accounts for the numerical differences observed.

Despite numerical differences, the optimal $\lambda$ values remain largely consistent between the original and revised versions, as shown in the following table. We have three observations: (1) For 6 out of 9 tasks, the optimal $\lambda$ values remain identical ($0.0005$ or $0.0015$) between the original and new results. (2) In three tasks ($D \rightarrow C$, $D \rightarrow A$, and $A \rightarrow C$), minor differences in the optimal $\lambda$ (between $0.0005$ and $0.0015$) ) are observed. These variations are likely due to the fine-scale sensitivity of these specific configurations to random data splits during replication. (3) Tasks involving multiple source domains demonstrate perfect stability, consistently maintaining $\lambda = 0.0005$ across both versions.

Table: Comparison of optimal lambda between the old version and the new version.

We thank the reviewer for the dedicated and insightful review. Please see below for our response.

Comment: In line 260, the authors state that $p$ is the expected degree, but according to the definition in Assumption 4.2, the network connectivity parameter $p$ should be a probability.

Response: Thank you for pointing out this typo. We have corrected it in the revised manuscript. In line 260, $p$ is the network connectivity probability, and $np$ is the expected degree.

Comment: I find the condition $p\log d \to 0$ as $n \to \infty$ in Assumption 4.2 somewhat perplexing. It would be helpful for the authors to explain why the feature dimension $d$ is relevant when defining network connectivity.

Response: Thank you for your insightful comment! The main reason for imposing the condition on the dimension is that we do not assume the dimension d to be fixed; instead, d can be significantly larger than the sample size n. Our theoretical results are established in an ultra-high dimensional regime, where d can grow as large as $e^{n^c}$ for some $c < 1$. Therefore, to take care of this growing dimension, we require this additional condition. When the dimension is fixed or grows slowly, the condition $p\log{d} \to 0$ simplifies to $p \to 0$, aligning with the standard framework in network regression analysis.

In an ultra-high dimensional feature space, the impact of each additional feature can amplify the complexity of the network structure. Thus, to ensure that the network remains sufficiently sparse for our theoretical results to hold, $p$ must decrease at a rate that compensates for the growth in $\log d$. The condition $p\log d \to 0$ as $n \to \infty$ ensures that, despite the increasing number of features, the overall connectivity remains controlled. Thank you again for pointing this out, and we hope this clarification helps address your concern.

Comment: Since one of the contributions of the paper is providing theoretical guarantees, and the authors have made several modifications to the proof approach compared to previous work, a brief sketch of these changes would be helpful for readers to understand the arguments better.

Response: Thank you very much for your comment! We have now included a roadmap in the Appendix (Section B) that highlights the key differences and modifications made in the proof to account for the dependency among the observations.

2. Summary of revisions in the paper:

Main text:

(1) Additional Method for Comparison: We have included an additional method, Pretraining GNNs [1], to provide a more comprehensive comparison in our experiments.

(2) Updated Table 1: We have updated Table 1 by implementing each method using cross-validation to select optimal hyperparameter settings, ensuring a fair and rigorous evaluation.

(3) Typo Correction: A typo regarding the parameter $p$ on page 5 has been corrected.

Appendix:

(1) Clarification of Notation: An explanation of the notation $\mathbf{X}_j$ has been added to Table S1 to enhance clarity.

(2) Proof Roadmap: Section B has been added to include a roadmap of the proof, providing readers with a structured overview of our theoretical analysis.

(3) Additional Evaluation Metrics: We have added the Table S2 which includes two additional evaluation metrics—True Positive Rate (TPR) and False Discovery Proportion (FDP)—to offer a more detailed assessment of model performance.

(4) Extended Hyperparameter Range: We have updated Table S6 to explore a broader, exponential range of the regularization parameter $\lambda$, specifically the values: 0.00005, 0.0001, 0.0005, 0.001, 0.0015, 0.01, and 0.1.

(5) Scalability Investigation: We have added the Table S7, which includes two additional large datasets to investigate the scalability of our method, demonstrating its effectiveness on a larger scale.

[1] W Hu, B Liu, J Gomes, M Zitnik, P Liang, V Pande, and J Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations (ICLR), 2020.