MPHIL: Multi-Prototype Hyperspherical Invariant Learning for Graph Out-of-Distribution Generalization

Q4: I'm a bit confused here, can you explain the rationale for designing prototypes? Is it relying exclusively on the several losses that have been subsequently proposed to be corrected to avoid the situation I have described?

A4: Thanks for your valuable question! As we explained in A3, the introduction of multiple prototypes is aimed at achieving more robust classification. Beyond that, in our formulation, the misalignment of a sample with an incorrect prototype can be interpreted as a signal of environmental interference, while successful alignment with the correct prototype indicates the capture of stable and invariant features. By optimizing the two proposed loss functions, we ensure:

• Each prototype within a class captures a sufficient number of samples, guaranteeing accurate classification into the correct class.
• Prototypes of different classes are as far apart as possible, maximizing inter-class separability and enabling highly confident classifications.

In summary, by optimizing the two proposed loss functions, we can effectively avoid the classification confidence issue you mentioned, as it ensures that all prototypes of the correct class consistently exhibit higher classification confidence compared to prototypes of other classes.

W1: There are some terms and formulas that are not presented accordingly, such as I(z;y) in Eq. 2, which I understand to mean mutual information. However, to increase readability I would suggest that these symbols should be explained the first time they appear.

A1: Thanks for your valuable comments, and we sincerely apologize for overlooking the explanation of some essential terms and formulas. In the revised manuscript, we have included the necessary clarifications (page 3 line 160). As you correctly pointed out, $I(z;y)$ represents the mutual information between two random variables, quantifying the amount of shared information between them.

W2: Apart from this, there is no explanation of the significance of some of the designed variables, such as separation score S. Why add it additionally to H? What is the benefit of doing so?

A2: Thanks for your valuable question! We designed the separation score to be more efficient in obtaining the invariant representation. Most existing methods aim to obtain invariant representations from the input graph by learning a mask matrix to obtain invariant subgraph and encode it, which is time-consuming and inefficient. To address this limitation, we encode the entire graph directly into a vector representation and then utilize the separability score $S$ to recover the invariant representation. This design not only improves computational efficiency but also ensures the accuracy of the extracted invariant representations, as evidenced by the ablation study in Table 2 (w/o Inv. Enc.).

W3: Figure 3 does not significantly show the advantage of your proposed method over baseline, can you consider a different visualization example that more intuitively reflects the difference between the two?

A3: Thanks for your valuable question! In the GOODHIV dataset, which is a binary classification task, the blue and green points represent class 0 samples from the training and test sets after dimensionality reduction with t-SNE, while the orange and red points represent class 1 samples. We observe that, with MPHIL, the representations of the same class (blue and green, or orange and red) are more tightly clustered, and the separation between different classes (blue vs. orange, and green vs. red) is larger, whether for In-distribution(training set) or Out of-distribution (test set) data.

We acknowledge that Fig. 3 may be challenging to interpret due to the imbalance in the number of point, as training sets are generally much larger than test sets, especially in out-of-distribution generalization tasks. To address this and further quantify MPHIL's advantages, we calculated the 1-order Wasserstein distance [1] between samples of the same category and those of different categories. The results clearly demonstrate the effectiveness of our method in achieving the principles of 'intra-class compactness and inter-class separation.

[1] Villani C. Optimal transport: old and new[M]. Berlin: springer, 2009.

W4:You included a link to the code in your manuscript, but I found that many of the modules are called, but the relevant code doesn't exist and doesn't work.

A4: Thanks for your valuable question! We apologize for not clearly explaining how to use the code provided in the anonymous link. Some modules that are referenced but not included in the files may need to be installed via pip. To address this, we have added a requirements.txt file to the anonymous GitHub repository, specifying all the necessary dependencies. In addition, installation descriptions of the two benchmarks covered in this article have been added to the latest Readme file.

Q1: This paper seems to closely follow the recent research "HYPO: Hyperspherical Out-Of-Distribution Generalization", e.g., "intra-class variation and inter-class separation principles". The novelty of the paper seems to be limited.

A1: Thanks for your valuable comments! We would like to further clarify the novelty of our work and highlight the differences with the HYPO method.

1.Novelty of Motivation. Our work is not a simple application of hyperspherical space but rather an attempt to address two specific challenges in graph OOD generalization: the difficulty in capturing environmental information and the semantic cliff across different classes. HYPO, on the other hand, is applied to image data, where accurate environmental information is accessible. Therefore, the challenges and motivations behind our work differ significantly from those of HYPO.

2.Novelty of Methodology. Since reliance on environmental information is not feasible, specific adjustments are made in the graph OOD generalization approach to implement the 'intra-class variation and inter-class separation principles.' Specifically, the MPHIL framework proposed in this work consists of two novel components:

• Hyperspherical Invariant Representation Learning (Section 3.2): This component improves the separability and informativeness of the learned invariant representations, making it more effective at capturing invariant features for graph OOD tasks compared to original representation learning methods.
• Multi-Prototype-based Classification (Section 3.3): This component eliminates the need for explicit environmental modeling and enables more flexible decision boundaries, allowing for better prediction.

These components work together and optimize two novel objective functions: the invariant prototype matching loss and the prototype separation loss to ensure correct prototype matching and maximize class separability in the hyperspherical space. Through these innovative components, our method provides a new perspective and solution for graph OOD generalization, setting it apart from existing methods in computer vision like HYPO in terms of goals and implementation strategies.

Q2:The visualization results do not seem clear (fig 3). Why the proposed method is better than the compared method.

A2: Thanks for your valuable comments! To visualize the learned representations, we use t-SNE for dimensionality reduction. In the GOODHIV dataset, which is a binary classification task, the blue and green points represent class 0 samples from the training and test sets, while the orange and red points represent class 1 samples. We observe that, with MPHIL, the representations of the same class (blue and green, or orange and red) are more tightly clustered, and the separation between different classes (blue vs. orange, and green vs. red) is larger, both in the training and test sets.

We acknowledge that Fig. 3 seems to be challenging to understand due to the imbalance in the number of points, as training sets are typically much larger than test sets, especially in out-of-distribution generalization tasks. To further quantify the advantage of MPHIL, we also calculate the 1-order Wasserstein distance [1] of the same category and different categories. The result demonstrates the effectiveness of our approach in achieving 'intra-class variation and inter-class separation principles'.

[1] Villani C. Optimal transport: old and new[M]. Berlin: springer, 2009.

Dear Reviewer pafp:

We sincerely appreciate your valuable contributions during the review process and your recognition of and suggestions for our work. To save you time, we provide a summarized version of our response to facilitate a quicker understanding:

• Emphasis on Novelty: To emphasize the distinctions between our approach and vision-domain methods like HYPO, we address both the motivation and design. Unlike visual OOD generalization, graph OOD generalization faces two unique challenges: the difficulty in capturing environmental information and the semantic cliff across different classes. To tackle these issues, we propose two innovative modules: hyperspherical invariant learning and multi-prototype-based classification, combined with two novel loss functions, enabling maximally class-separable, environment-free graph OOD generalization

• Supplemental Baseline Models: We have added three state-of-the-art graph OOD generalization methods for comparison, as suggested. The experimental results continue to highlight the superior performance of MPHIL.

• The Interpretation of Figure 3: We have refined our interpretation of Figure 3 and included additional quantitative analyses to better demonstrate the advantages of MPHIL.

We have carefully response all your questions and truly appreciate the opportunity to clarify and improve our work based on your feedback. With some time remaining before the rebuttal period ends, please do not hesitate to reach out if you have any additional concerns that you would like us to discuss further. If you feel that our responses have adequately resolved your concerns, we would be deeply grateful if you might consider improving your score. Your support and recognition are incredibly important to us, and we sincerely thank you for your thoughtful review and constructive input.

Dear Reviewer pafp,

Thank you for your efforts in reviewing and for raising valuable questions. We would like to remind you of the extended discussion period. In the first-round response, we provided detailed replies and a summarized version to address your concerns regarding novelty, baseline settings and provide a detailed explanation about Fig. 3.

We sincerely hope you will reconsider your score based on these responses, as this is crucial for us, especially since some reviewers have already increased their scores after our clarifications. If you have any further concerns, please do not hesitate to reach out to us, and we will provide additional explanations.

Dear Reviewer pafp,

We greatly appreciate your contribution to the review of our manuscript, as well as your valuable suggestions. As the discussion phase is nearing its end, we kindly remind you to comfirm whether our responses have addressed your concerns.

We will deeply appreciate your response and look forward to further discussions to address any remaining concerns you may have.

Q1: The novelty of this paper is limited. This paper introduce invariant learning into a hyperspherical space and introduces multiple prototypes to improve class separability, these ideas extend previously established techniques in a relatively straightforward manner. However, the use of hyperspherical embeddings for enhancing model performance of OOD has been explored [1], [2] and [3].

A1: Thank you for pointing out the application of hyperspherical space in OOD detection [1,3] and OOD generalization [2]. We'd like to emphasize that our work is not a direct extension of previously established techniques (hyperspherical space). Instead of that, we incorporate several novel designs driven by distinct motivations, underscoring the originality of our approach.

1. Distinct Motivation. While the referenced work [2] you mentioned is indeed effective in image-based OOD generalization, it rely heavily on the availability of accurate and sufficient environmental information. This critical requirement, however, is not feasible in the graph domain, where the environmental information is often unavailable or ambiguous. Our motivation is not merely to extend existing techniques from computer vision to graphs but to address two unique challenges in graph OOD generalization: the inherent difficulty in capturing environmental information and the semantic cliffs across different classes. These challenges require tailored solutions that go beyond direct application of image-based methods.

Besides, the reference works [1, 3] are focused on OOD detection, which aims to determine whether the test data deviates from the training distribution. In contrast, our work addresses OOD generalization, which is concerned with maintaining robust predictive performance even when the test distribution differs from the training distribution. These are different tasks, and therefore, the challenges to address and the motivations for model design are also different.

2. Novel Design. To address these challenges, we propose a novel method MPHIL which contains two key innovations: (1) hyperspherical invariant learning for robust feature extraction and prototypical learning in a highly discriminative space. (2)multi-prototype-based classification to mitigate the semantic cliff issue and eliminate the need for explicit environment modeling. We propose two novel objective functions: the invariant prototype matching loss and the prototype separation loss to ensure correct prototype matching and maximize class separability in the hyperspherical space.

We hope our explanation of the differences in motivation and the corresponding solutions above can highlight the unique novelty of our approach and address your concerns. All the references you mentioned we will add to the latest version.

Dear Reviewer X7hK:

We sincerely appreciate your valuable contributions during the review process and your recognition of and suggestions for our work. To save you time, we provide a summarized version of our response to facilitate a quicker understanding:

• Emphasis on Novelty: Regarding the novelty of our approach, we emphasize its unique motivation and design, distinct from existing hypersphere learning methods. Unlike OOD generalization in computer vision (e.g., the reference [2] you mentioned), graph data presents unique challenges, including limited access to environmental information and separability issues caused by the semantic cliff. To address these, we introduce two key modules—hyperspherical invariant learning and multi-prototype-based classification—alongside two novel loss functions. These innovations enable intra-class consistency and inter-class separability, supporting maximally class-separable, environment-free graph OOD generalization.

• Supplemental Baseline Models: We added three state-of-the-art graph OOD generalization methods for comparison, as per your suggestion. The experimental results confirm that MPHIL continues to achieve superior performance.

Finally, we have included the references you pointed out, recognizing their value in enhancing our work.

We have carefully response all your questions and truly appreciate the opportunity to clarify and improve our work based on your feedback. With some time remaining before the rebuttal period ends, please do not hesitate to reach out if you have any additional concerns that you would like us to discuss further. If you feel that our responses have adequately resolved your concerns, we would be deeply grateful if you might consider improving your score. Your support and recognition are incredibly important to us, and we sincerely thank you for your thoughtful review and constructive input.

Dear Reviewer X7hK,

Thank you for your efforts in reviewing and for raising valuable questions. We would like to remind you of the extended discussion period. In the first-round response, we provided detailed replies and a summarized version to address your concerns regarding novelty and baseline settings.

We sincerely hope you will reconsider your score based on these responses, as this is crucial for us, especially since some reviewers have already increased their scores after our clarifications. If you have any further concerns, please do not hesitate to reach out to us, and we will provide additional explanations.

Q3: I wonder the performance when only using one prototype.

A3: Thanks for your valuable comments! In the ablation study of the original manuscript (Table 2), we included the results for 'w/o Multi-P,' which corresponds to using only a single prototype for classification. The results indicate that this approach significantly impacts model performance, as it blurs decision boundaries and prevents the loss function $\mathcal{L}_{\mathrm{PS}}$ from being effectively optimized, ultimately compromising inter-class separability.

Q4: The prototype updating mechanism is complicated, what if directly use the representation of the sample froraining set after training.

A4: Thanks for your valuable comments! In the original manuscript, we conducted an ablation study (Table 2, w/o Update) to investigate the impact of removing the prototype update mechanism described in Section 3.3. Specifically, this is equivalent to directly using the mean of same-class samples as the prototype (the number of prototypes for each class is reduced to one). To further address your concerns, we additionally designed two alternative prototype construction mechanisms: (1) Randomly selecting $K$ samples from each class as prototypes. (2) Clustering the samples of each class and using the $K$ cluster centers as prototypes. The experimental results are shown in the table below.

The results show that the random sampling method leads to the most significant performance drop, as it fails to effectively ensure inter-class separability. While the mean and k-means methods perform slightly better, they still struggle to generate optimal prototypes due to their inability to fully capture the intra-class diversity. This further demonstrates the superiority of our proposed prototype update method.

Q5: The explaination of some symbols is missing, e.g. the $w_{k}^{c}$ in Eq (12).

A5: Thanks for your valuable comments, and we apologize for the oversight in explaining some of the notation. We have now clarified this in the latest manuscript (page 7 line 324). Specifically, $w_{k}^{c}$ denotes the weight assigned to the $k$-th prototype of class $c$ for the current sample.

Q1: Intuitively, the accuracy the class prediction relies on the fastest prototype point in the same class prototype set. What if use the mean/sum/min for predicting the class? More ablation study can make your work more convincing.

A1: Thanks for your valuable comments! As you mentioned, intuitively, the prototype most similar to the sample provides the most representative and discriminative information, facilitating faster convergence of the sample towards its correct class space. To further justify this intuition, we used to conducted experiments replacing the max operation with min, median, sum, and mean in our approach （Equation 12）.

The results show that the max operation consistently achieves the best performance, followed by median with min performing the worst. We believe this is because the median is relatively robust to outliers and can still capture central tendencies to some extent, while sum and mean dilute the impact of the most relevant prototype by averaging contributions from all prototypes. The min operation, on the other hand, focuses on the least similar prototype, which misrepresents the true relationship between the sample and its class and slows down the convergence of the optimization process.

Q2: The novelty of this proposed method is limited. I don't think the projection to the hypersphere here is novel. Actually, it is a simple cosine distance. How about using other distances like L2, L1, more analysis should be included.

A2: Thanks for your valuable comments! While hyperspherical learning may initially appear similar to cosine similarity, its core principle is the normalization of high-dimensional representations into unit vectors on a hypersphere. This approach suppresses amplitude interference and optimizes angular relationships, thereby enhancing intra-class consistency and maximizing inter-class separability. These properties make hyperspherical learning, though seemingly simple, particularly effective for OOD generalization and has been demonstrated in computer vision, while our ablation experiments (w/o project) also illustrate its effectiveness for graph OOD methods.

While a direct application of hyperspherical space may have limited novelty, we want to emphasize that the primary motivation of our paper is not a simple application of hyperspherical spaces but rather to tackles two unique challenges in graph OOD generalization: the difficulty in capturing environmental information and the semantic cliff across different classes. To overcome these challenges, we further propose two novel and effective designs, i.e., hyperspherical invariant feature learning and multi-prototype classification mechanism. These components work together to ensure the learning of environment-independent, maximally class-separable invariant features. Ablation studies (w/o Inv.Enc., w/o Multi-P) demonstrate their critical contributions, showcasing how our method advances the state-of-the-art in graph OOD generalization.

Furthermore, MPHIL does involve multiple similarity computations (Eq. 12-14), all based on cosine similarity. Following your suggestion, we further replaced cosine similarity with $L1$, $L2$, and $L_\infty$ distances to analyze the impact of different similarity metrics. The results are summarized in the table below.

The experimental results demonstrate that cosine similarity is the most effective measure, as it leverages the directional properties of hyperspherical space to distinguish features with high precision. In contrast, measures like $L1$, $L2$, and $L_\infty$ distances primarily focus on magnitude differences, which are less relevant in hyperspherical representations where all features are normalized to have unit norms.

Dear Reviewer gRW6:

We sincerely appreciate your valuable contributions during the review process and your recognition of and suggestions for our work. To save you time, we provide a summarized version of our response to facilitate a quicker understanding:

• Expanded Ablation Experiments: As per your suggestion, we replaced the max operation with other statistics, substituted cosine similarity with alternative similarity measures, and explored additional prototype update strategies. The results of these experiments consistently validate the effectiveness of our proposed design.
• Clarification on Novelty: While you expressed concerns about the simplicity and novelty of hyperspherical space, we further explain its unique properties and why it is particularly well-suited for OOD generalization tasks. More importantly, our proposed MPHIL is far from a simple application of hyperspherical spaces. It directly tackles two critical challenges in graph OOD generalization: the difficulty in capturing environmental information and the semantic cliff across different classes. Our contributions include hyperspherical invariant feature learning and a multi-prototype classification mechanism, supported by two novel loss functions. These innovations ensure intra-class consistency and inter-class separability, enabling us to achieve better performance in graph OOD generalization.

We have carefully response all your questions and truly appreciate the opportunity to clarify and improve our work based on your feedback. With some time remaining before the rebuttal period ends, please do not hesitate to reach out if you have any additional concerns that you would like us to discuss further. If you feel that our responses have adequately resolved your concerns, we would be deeply grateful if you might consider improving your score. Your support and recognition are incredibly important to us, and we sincerely thank you for your thoughtful review and constructive input.

Dear Reviewer gRW6

Thank you for your efforts in reviewing and for raising valuable questions. We would like to remind you of the extended discussion period. In the first-round response, we provided detailed replies and a summarized version to address your concerns regarding novelty and requests for more ablation experiments.

Thank you for your previous response, but we would still like to know your specific concerns about novelty, or why our previous answer did not address your concerns. We hope to do our best to solve your confusion to prompt you to reconsider the score, as this is crucial for us, especially since some reviewers have already increased their scores after our clarifications. If you have any further concerns, please do not hesitate to reach out to us, and we will provide additional explanations.