TFGDA: Exploring Topology and Feature Alignment in Semi-supervised Graph Domain Adaptation through Robust Clustering

We appreciate Reviewer R5oT for the thorough review of the manuscript and the valuable comments that will aid in enhancing our paper.

Q1: Include some discussions on practical applications of graph transfer learning in the Sec.1.
A1: Graph Transfer Learning can be applied in various practical scenarios, such as recommender systems [K1], molecular property prediction [K2], social and academic networks analysis [K3], cross-modal retrieval [K4], and human parsing [K5].

Ref.[K1] Learning adaptive node embeddings across graphs. TKDE 2022.
Ref.[K2] Transfer learning with graph neural networks for improved molecular property prediction. NC, 2024.
Ref.[K3] A comprehensive survey on graph neural networks. TNNLS, 2020.
Ref.[K4] Cross-Domain Transfer Hashing for Efficient Cross-modal Retrieval. TCSVT, 2024.
Ref.[K5] Graphonomy: Universal Human Parsing via Graph Transfer Learning. CVPR 2019.

Q2: What dimensions of topological features did the authors use in STSA strategy? How was this determined?
A2: As described in Section A.2 of the Appendix (lines 508-511), we use 0-dimensional topological features in our STSA strategy to align the topological structures of the input and latent spaces. Because some preliminary experiments have shown that using higher-dimension topological information does not lead to clear accuracy improvements but noticeably increases the model’s training time.
Additionally, the topological theory [K6-K7] has demonstrated that low-dimensional topological features (e.g., 0-dimensional) can roughly reflect the topological structure of the data, while high-dimensional topological features (e.g., 3-dimensional and 4-dimensional) are responsible for capturing some complex details of topological structure. Thus, to balance computational efficiency and model performance, we choose to retain 0-dimensional topological features in our STSA strategy.

Ref.[K6] Computing persistent homology. 2004.
Ref.[K7] Persistent homology-a survey. 2008.

Q3: What distance metric was used when constructing the Vietoris-Rips simplicial complex in the STSA strategy?
A3: We use Euclidean distance as the distance metric to construct the Vietoris-Rips simplicial complex in our STSA strategy.

Q4: What advantages does the RNC strategy have compared to the K-means algorithm?
A4: Compared to the K-means clustering algorithm, our RNC strategy has two main advantages:

1) Using the K-means algorithm in neural networks results in a two-stage clustering process [K8]. Specifically, the deep features extracted by the network are first clustered using the K-means algorithm to generate pseudo-labels. These pseudo-labels are then used to guide the model training. This clustering process inevitably introduces pseudo-labels noise into the model during training, limiting the model's generalization ability. On the contrary, our proposed RNC strategy is an end-to-end clustering method that does not involve any pseudo-labels, thereby enhancing the model's robustness in face of different transfer scenarios.

2) When combined with representation learning, the K-means algorithm is prone to degenerate clustering solutions [K8-K10], where all unlabeled nodes are assigned to the same cluster, severely affecting the construction of decision boundaries. The proposed RNC strategy is designed based on the mutual information theory, which effectively guides unlabeled nodes toward robust clustering and naturally avoids degenerate clustering solutions. As shown in Section A.3.3 of the Appendix (lines 566-579), we have provided a detailed theoretical analysis that explains how our RNC strategy can avoid degenerate clustering solutions. Furthermore, as mentioned in Section 4.5 (lines 234-236), the experimental results in Figure 2 and Figure 3 have also validated this point.

Ref. [K8] Deep Clustering for Unsupervised Learning of Visual Features. ECCV 2018.
Ref. [K9] On strategies to fix degenerate k-means solutions. Journal of Classification 2017.
Ref. [K10] How to Use K-means for Big Data Clustering. PR 2023.

Q5: Discuss the scalability of TFGDA on larger graph datasets, such as the AliCVR dataset.
A5: Based on your valuable suggestion, we follow Ref.[K1] to utilize a real-world large-scale recommender systems dataset Ali-CVR to conduct additional experiments. Specially, Ali-CVR can be divided into 3 large-scale graphs: AN, A11 and AB11. We report the node classification accuracy in the target graph, and the results in Table R1 demonstrate the scalability and stability of our method.

Table R1: Node classification accuracy (%) in the target graph. Source graph label rate: 10%. [Source graph $\rightarrow$ Target graph]

Q6: It is suggested to compare and discuss the TFGDA with the GIFI method.
A6: Following your suggestion, we compare the transfer performance of our method with the GIFI on the challenging transfer scenario, where only 5% of the source graph nodes are labeled. The results in the Table R2 demonstrate the effectivenss of our method.

Table R2: Transfer performance (%) measured by Macro-F1 on some transfer tasks. Source graph label rate: 5%.

Dear Reviewer R5oT:

We thank your response and appreciation of our work and rebuttal. We will make sure to incorporate the new results and discussions into our revision to enable it to be a high-quality paper.

Best Regards, Authors of 504.

We thank the Reviewer byDA for the careful reading of the manuscript and the related comments, which are helpful to improve our paper.

W1: Further exploration can be conducted on other types of graph datasets.
A1: Based on your suggestion, we conduct additional experiments on a real-world recommender systems graph dataset called AliCVR to validate the generalizability of our method. Please refer to the answer A5 to Reviewer R5oT.

W2: Include a discussion and comparison of the model’s inference efficiency.
A2: We compare model's inference time on the transfer task A$\rightarrow$C in the following Table R3. Due to the addition of the spherical space \mathbb{S}\_\{r}^{d-1}, our TFGDA exhibits a slightly longer inference time (1.02x) than the SOTA competitor SGDA [S1], which does not significantly increase the inference time but brings a clear performance gain. Additionally, it is worth mentioning that if the spherical space \mathbb{S}\_\{r}^{d-1} is removed, during inference the network architecture of our TFGDA is almost identical to that of SGDA, as we will disable the shift parameters perturbation branch \xi\_\{s} and \xi\_\{t}.

Table R3: Model's Inference time on Citationv1 (C).

Ref. [S1] Semi-supervised Domain Adaptation in Graph Transfer Learning. IJCAI 2023.

W3: Include additional quantitative metrics to better demonstrate the transfer ability of the method, such as the $\mathcal{A}$-distance.
A3: Following your advice, we utilize the $\mathcal{A}$-distance as a metric to evaluate the transfer ability of different methods in Figure 2. Table R4 shows the results of the $\mathcal{A}$-distance on A$\rightarrow$C task. Since our method can accurately align feature distributions of two domains at the class-level, it achieves the smallest $\mathcal{A}$-distance.

Table R4: $\mathcal{A}$-distance on A$\rightarrow$C task.

W4: Add a high-level algorithm table to summarize the SDA strategy.
A4: Thanks for the valuable advice. We will include a a high-level algorithm table for the SDA strategy in the revised version to make the paper more comprehensive.

Q1: During inference, is the shift parameter perturbation branch $\xi$ of RNC activated? Does the graph data flow through the regular branch or the perturbation branch before being fed into the classifier $\mathcal{C}$ for inference?
A5: As mentioned in Section A.2 of the Appendix (line 513), during the inference process, we will disable the shift parameters perturbation branches $\xi_{s}$ and $\xi_{t}$. Thus, during the inference process, the graph data will be fed through the regular branch (i.e., without any feature perturbation) into the classifier $\mathcal{C}$ for the final prediction.

Q2: What is the intrinsic reason for the need to introduce structure information in graph transfer learning frameworks? Is it because the structure information of the data tends to be lost as the network layers go deeper?
A6: Notably, unlike images and time series data, graph data usually contains rich structure information that encodes complex relationships among nodes and edges. Most existing graph transfer learning works directly use graph convolutional networks (GCNs)-based feature extractors to learn transferable node features. However, recent studies [S2-S4] have indicated that GCNs are insufficient in capturing the sophisticated structure information in graph, which means that the graph structure information may be lost or destroyed after passing through the GCNs-based feature extractor. Therefore, we propose the STSA strategy to encode these intrinsic structure information in graph (i.e., from the input space) into the latent space by aligning the topological structures of the input space and the latent space, effectively improving the model's transfer performance.

Ref. [S2] Coco: A coupled contrastive framework for unsupervised domain adaptive graph classification. ICML 2023.
Ref. [S3] Graph Kernel Neural Networks. TNNLS 2024.
Ref. [S4] Theoretically improving graph neural networks via anonymous walk graph kernels. WWW 2021.

Q3: In the SDA strategy, why is the SSW distance better than the wasserstein and the sliced wasserstein distance metrics in measuring domain differences?
A3: 1) As described in lines 182-184, due to numerous nodes in the graph (e.g., ACMv9 dataset has over 9000 nodes), directly using the classical Wasserstein distance to compute feature distributions discrepancy is computationally expensive, rendering it impractical for graph transfer learning task.

2) To eliminate the negative influence of feature norm discrepancy on the learning of transferable node features, our SDA strategy thus guides feature distributions alignment in spherical space. As described in lines 528-530, the sliced Wasserstein distance focuses on calculating discrepancy between distributions in Euclidean space [S5], making it difficult to precisely measure feature distributions discrepancy in spherical space. While our SSW distance fully considers the manifold structure of data in spherical space when assessing distribution discrepancy [S6], thereby facilitating a more precise quantification of difference in feature distributions. Furthermore, the detailed ablative experiments results in Figure 5 of the Appendix (lines 518-532) have already validated that our SSW distance is superior to the sliced Wasserstein distance in measuring domain discrepancy.

Ref.[S5] Sliced Wasserstein Kernels for Probability Distributions. CVPR 2016.
Ref.[S6] Spherical Sliced-Wasserstein. ICLR 2023.

Dear Reviewer byDA:

Thanks so much again for the time and effort in our work. According to the comments and concerns, we conduct the corresponding experiments and further discuss the related points. As the discussion period is nearing its end, please feel free to let us know if there are any other concerns. We would be happy to provide more details and clarifications.

Best Regards, Authors of 504.

We extend our heartfelt gratitude to Reviewer aMJG for the careful reading of the manuscript and the valuable feedback provided, which are helpful to improve our paper. Our detailed point-by-point responses are provided below.

W1: While the authors claim to be the first to consider graph structure, similar approaches have been explored in previous work that also utilize graph structural information.
A1: Thanks for the valuable comment. We primarily showcase the innovation of our method through the following three aspects:

1): Although many graph convolutional networks (GCNs)-based methods have already been proposed to exploit graph structure information to promote the learning of features, these methods heavily rely on a large amount of labeled data. Moreover, these methods overlook the substantial domain discrepancy between training and testing datasets in real-world scenarios, making them unsuitable for semi-supervised graph domain adaptation (SGDA) scenario. Specifically, these GCNs-based models trained only with the limited source labeled nodes (i.e., SGDA task scenario) will suffer severe performance degradation when directly applied to a new target domain.

2): In addition, these GCNs-based methods [X1-X6] typically mine the graph structure information in the deep feature space by designing well-crafted GCN architectures or introducing some complex modules. However, recent studies [X6-X8] have pointed out that GCNs are insufficient in capturing the sophisticated structure information in graph, which means that the graph structure information may be lost or destroyed after passing through the GCNs-based feature extractor. Thus, directly mining graph structure information from the deep feature space is a suboptimal way, which affects the learning of transferable node features in our SGDA setting.

The proposed STSA strategy aims to extract the graph structure information directly from the input space and encode these powerful information into the latent spherical space by aligning the topological structures of the two spaces. This method does not lose or destroy the graph structure information during training. Furthermore, our STSA strategy does not introduce any changes to the network architecture, effectively avoiding an increase in model's complexity and ensuring its adaptability to integration with other methods.

3): What's more, these GCN-based methods typically model graph structure by considering the similarity between features of adjacent nodes, making it difficult to capture the sophisticated high-order structure information in graph [X5,X6]. In contrast, our work attempts, for the first time, to model the graph structure information from a persistent homology (i.e., topological data analysis) perspective, enabling a more precise capture of the complex structures inherent in graph.

In summary, our proposed STSA strategy is an innovative solution for mining graph structure information and facilitating graph transfer learning.

Ref.[X1] Distilling Knowledge From Graph Convolutional Networks. CVPR 2020.
Ref.[X2] Graph-in-Graph Convolutional Network for Hyperspectral Image Classification. TNNLS 2022.
Ref.[X3] Multigraph Fusion for Dynamic Graph Convolutional Network. TNNLS 2022.
Ref.[X4] Knowledge Embedding Based Graph Convolutional Network. WWW 2021.
Ref.[X5] Scattering GCN: Overcoming Oversmoothness in Graph Convolutional Networks. NIPS 2020.
Ref.[X6] Coco: A Coupled Contrastive Framework for Unsupervised Domain Adaptive Graph Classification. ICML 2023.
Ref.[X7] Graph Kernel Neural Networks. TNNLS 2024.
Ref.[X8] Theoretically Improving Graph Neural Networks via Anonymous Walk Graph Kernels. WWW 2021.

W2&W3: The experimental results seem excessively positive. The paper does not provide open-source code.
A2&A3:
1): It is worth noting that semi-supervised graph domain adaptation (SGDA) is a relatively novel task scenario in graph transfer learning, first introduced in Ref.[X9]. Currently, there has been little exploration in this field. As stated in lines 248-251, the main compared methods adopted in Table 1 are some semi-supervised methods and graph domain adaptation methods. Due to the label scarcity of source graph and the huge domain discrepancy in the SGDA scenario, these semi-supervised methods are unable to tackle these challenges, resulting in severe performance degradation on the target domain.

Although graph domain adaptation can alleviate domain discrepancy, they inevitably encounter the overfitting issue. Because these methods typically assume that the source domain is fully labeled, making it difficult to effectively utilize a large number of unlabeled nodes, as stated in lines 265-267. Our method effectively addresses these challenges, thus achieving SOTA performance.

2): Regarding the pioneering work SGDA [X9], it struggles to achieve superior performance because it simply utilizes the classical adversarial learning to reduce domain discrepancy, and its clustering strategy inevitably introduces pseudo-label noise into the model (as verified in lines 307-308 and 518-532).

Ref.[X9] Semi-supervised Domain Adaptation in Graph Transfer Learning. IJCAI 2023.

3) Source code: Based on your valuable suggestion, we will provide AC with the source code of our method in order to validate the authority of our work. We commit to publicly release the code after the paper is accepted.

If you have additional concerns, please let us know and we will do our best to address them. We appreciate your time and efforts in reviewing our work.

Dear Reviewer aMJG:

Thanks so much again for the time and effort in reviewing our work. According to your valuable comments and concerns, we provide the source code and further discuss the related points. As the discussion period is nearing its end, we would like to kindly ask the reviewer if our rebuttal has addressed some of the reviewer's questions or concers and if any of them remain unaddressed. We would be happy to provide more details and clarifications.

Best Regards, Authors of 504.

We would like to express our gratitude to Reviewer 9hdx for their comprehensive evaluation of the manuscript and the insightful feedback provided, which will greatly contribute to the improvement of our paper. Kindly refer to our specific responses outlined below.

W1: This paper proposes to extract spherical features of subgraphs in STSA strategy. Please further explain the reasons for using spherical features and analyze the necessity. The authors should further clarify the motivation behind using spherical features of their proposed method.
A1: Thanks for your valuable advice.

1) explain the reasons: Notably, several studies [Y1] in transfer learning have demonstrated that the domain shifts are mainly caused by the feature norm discrepancy between the source and target domains, and the model degradation on the target domain is primarily caused by its excessively smaller feature norms than the source domain. To eliminate the negative effects of feature norm differences, we map features into the spherical space to guide the alignment of feature distribution (as stated in lines 170-173), which can significantly facilitate the learning of domain-invariant features and improve the model's transfer performance.

Furthermore, Refs.[Y1-Y2] have pointed out that highly informative features typically display more significant feature norms, and task-specific features with larger norms are more transferable. However, due to the large parameter search space, it is difficult to find the optimal spherical radius for different transfer tasks. To address this issue, we follow the lower bound theorem from Ref.[Y3] to set an appropriate radius $r$ for the spherical space $\mathbb{S}_{r}^{d-1}$, as mentioned in lines 173-176 of the manuscript.

Ref.[Y1] Larger Norm More Transferable: An Adaptive Feature Norm Approach for Unsupervised Domain Adaptation. ICCV 2019.
Ref.[Y2] Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers. ICLR 2018.
Ref.[Y3] Unsupervised and Semi-supervised Robust Spherical Space Domain Adaptation. TPAMI 2022.

2) analyze the necessity: According to your advice, we conduct additional ablation experiments to validate this point. As described in Table 2 of the paper, variant TFGDA-T (\mathcal{L}\_\{cls}+\mathcal{L}\_\{stsa}) implements the STSA strategy in spherical space \mathbb{S}\_\{r}^{d-1}. Based on variant TFGDA-T, we remove the spherical space \mathbb{S}\_\{r}^{d-1} and implement the STSA strategy in regular space, resulting in a new variant called TFGDA-T (w/o sphere). The experimental results are shown in the Table A of the global response PDF file. We can observe that the introduction of spherical space \mathbb{S}\_\{r}^{d-1} can effectively enhance the model's transfer ability, which is consistent with the aforementioned analysis.

W2: The interplay between loss functions could affect the extracted features and model performance. It would be beneficial if the authors could provide a thorough analysis for the combination of loss functions and theoretically explanation of the success.
A2: Thanks for your valuable suggestion. It is worth noting that our ablation study results in Section 5.3 (lines 271-285) have validated the contribution of each loss term and the complementary relationship among different loss terms (i.e., different combinations of loss terms). Moreover, the t-SNE visualization results in Figure 2 have also confirmed the importance the combination of loss terms \mathcal{L}\_\{rnc}, \mathcal{L}\_\{stsa} and \mathcal{L}\_\{sda}. We have also conducted a sensitivity analysis on the trade-off parameters of these loss terms in Section A.3.2 of the Appendix (lines 557-565).

Theoretically Explanation: Based on your insightful advice, we provide a theoretical analysis of our method. Due to the characters limit, we will present this theoretical analysis in the global rebuttal area.

Dear Reviewer 9hdx:

Thanks so much again for the time and effort in reviewing our work. As we are getting closer to the end of the author-reviewer discussion phase, we would like to kindly ask the reviewer if our rebuttal has addressed some of the reviewer's questions or concers and if any of them remain unaddressed. We would be happy to provide more details and clarifications.

Best Regards, Authors of 504.

Dear AC and all Reviewers:

Based on the valuable suggestion of Reviewer aMJG, we provide the source code and pre-trained models of our method to demonstrate the authority of our work. Code and pre-trained models are available at: https://anonymous.4open.science/r/TFGDA-003D

We appreciate your time and efforts in reviewing our work.

Best Regards, Authors of 504.

Thank you to the authors for their detailed response to my questions. After reviewing all the comments and the rebuttal materials, my concerns have been well addressed. This process has further deepened my appreciation for the design and reproducibility of the proposed approach. I have no further questions. I am happy to adjust my rating to reflect my positive feedback.