GDL-DS: A Benchmark for Geometric Deep Learning under Distribution Shifts

We express our sincere gratitude to reviewer 1MFG for reviewing our paper, appreciating our work and raising valuable comments. We address your questions as follows.

1). How do the findings of this study compare with earlier research on CV and NLP tasks concerning distribution shifts?

Firstly, we’d like to appreciate earlier OOD works on CV and NLP tasks, many of which have shed valuable insights into our research. We have meticulously connected our multiple conclusions with the existing OOD work in CV and NLP and put details in the appendix. The details are as follows.

• In Sec. 4.2, we observe that fine-tuning can sometimes result in negative effects when the labeled OOD data is quite limited. This is consistent with catastrophic forgetting in [1]. To mitigate this issue, the strategy of surgical fine-tuning raised in [2] is a potential solution for this problem.

• In Sec. 4.2, we observe that multiple OOD generalization methods in our No-Info level find it hard to provide significant improvement across various applications. We can find similar observations in existing CV/NLP works, including [3], [4]. Actually, this could be even more severe in GDL compared to CV tasks, considering the intricate nature of irregularity and geometric prior (information on the structure space and symmetry properties like invariance or equivariance) inherent in geometric data.

• In Sec. 4.3, we conclude that “For the OOD generalization methods to learn robust representations, the more informatively the groups obtained by splitting the source domain $S$ indicate the distribution shift, the better performance these methods may achieve.” This is consistent with previous CV works like [5], which revealed the importance of appropriate subgroup partitioning for invariant learning. And we note that [6] proposed to leverage additional auxiliary variables for enhancing OOD generalization. This may also shed insights into future research on scientific GDL.

Moreover, certain shifts in scientific GDL are different from common shifts in CV/NLP, which leads to challenges when addressing these shifts with methods initially proposed for CV/NLP tasks. Here are some examples in our work:

• Shifts related to graph/geometric structure: For instance, the I-conditional shift can be interpreted as a size shift (the number of nodes in data), where the model trained in data with smaller sizes is to generalize to data with larger sizes. We observed that the methods designed for CV struggle to perform well. In addition, the features in GDL additionally incorporate the geometric structure. The methods in CV work on feature representations without explicitly handling the change in geometric structure, which may result in suboptimal solutions.

• Fidelity shift, which indicates the change of causal correlation between $X_c$ and Y, corresponds to a significant challenge in material property prediction. However, we rarely encounter the shift in the causal relationship in CV, so most methods in our benchmark find it hard to handle such a shift, except the pertaining-finetuning strategy.

The above two points of difference indicate the necessity to develop OOD methods specifically designed for GDL. Overall, We thank the reviewer’s suggestion and the comparisons made above would enhance the connection between our findings with earlier research in CV/NLP domains and shed more insights for future research.

[1] Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.

[2] Lee, Yoonho, et al. "Surgical Fine-Tuning Improves Adaptation to Distribution Shifts." ICLR, 2022.

[3] Gulrajani, Ishaan, and David Lopez-Paz. "In Search of Lost Domain Generalization." ICLR, 2020.

[4] Koh, Pang Wei, et al. "Wilds: A benchmark of in-the-wild distribution shifts." ICML, 2021.

[5] Creager, Elliot, Jörn-Henrik Jacobsen, and Richard Zemel. "Environment inference for invariant learning." ICML, 2021.

[6] Lin, Yong, et al. "ZIN: When and How to Learn Invariance Without Environment Partition?." NeurIPS, 2022.

We express our gratitude to reviewer xYh1 for acknowledging the contribution of our benchmark and recognizing the clarity and technical soundness of our paper. We also appreciate the valuable advice provided by the reviewer regarding the inclusion of more detailed information about the characteristics of the datasets in our benchmark. Basic dataset information, including data size and strategies for domain/subgroup splits, has been included in Appendix C. In our refined draft, we will provide more granular information that better reflects the characteristics of the constructed datasets and distribution shift. This will involve details covering:

1. the average number of tracks and particles for each pileup level in Pileup Shift (Track Dataset)
2. the average signal radius of each type of signal in Signal Shift (Track Dataset)
3. the average number of atoms for ID and OOD Dataset in Size Shift (DrugOOD-3D Dataset)
4. the average band gap value for each fidelity level in Fidelity Shift (QMOF Dataset)

We will collect and add the above 4 types of information to the refined draft before the deadline for rebuttal. Feel free to let us know if more details should be provided.

We thank reviewer GGxS for recognizing the compelling aspect of our work in developing OOD benchmarks for GDL in science, well-organized contents, and sufficient experimental analyses. Based on the review, we acknowledge that the reviewer has read our paper thoroughly and pointed out questions, in particular identifying our improper underemphasis of some previous works’ coverage. However, we respectfully disagree with the reviewer’s comments on the soundness of our assumption of categorizing various distribution shifts and concerns regarding our experiment settings. We summarize the reviewer’s concerns and provide our detailed explanations in the following response.

1). Use of terms: “The use of critical terms should be better considered.”

We thank the reviewer for raising this confusion based on our paper presentation. We are not aiming to modify the existing well-established terms. Instead, we would like to further explain and extend the common definition of these well-established terms under the context of our selected application datasets. We have clarified in our refined draft by first following the well-established definitions of these distribution shifts and then highlighting that we further extend the definition to what we present in the paper given our assumption.

2). Soundness of causal statements in section 3.1

Overall, the reviewer mentioned some alternatively correct definitions of causal relationships that are not covered by this paper. However, the existence of those alternative definitions does not imply the lack of soundness of our ways of definitions. Note that our definitions in Sec. 3.1 are to model the scientific GDL datasets benchmarked in this work, which indicate how these datasets were curated and how their distribution shifts are like. Our definitions are not to cover all possible causality relationships. The alternative definitions mentioned by the reviewer, though existing in theory, either are not reflected by those datasets or may introduce more confusion. We are sorry for not clarifying this point in Sec. 3.1, which caused such confusion. In the following, we respond to each bullet point raised by the reviewer and have adjusted the exposition in Sec. 3.1 accordingly to avoid the similar confusion made by the reviewer.

• 2.1 soundness of causal statements: “The statement that $X$ consists of two disjoint parts and $X_i \bot Y|X_c$ does not hold”

This review confuses “the settings this work is targeting” and “other possible settings that this work is not to cover”. Note that such conditional independence just describes the settings approximately indicated by our datasets. We agree that there exist other types of dependence, while they are out of the scope of this work because we did not find interesting scientific GDL datasets that reflect those types of dependence. Moreover, limiting the scope properly allows for more in-depth study without losing much generality.

First, such conditional independence is derived from real-world scientific application and holds in our established scenarios. Take our track Dataset in particle physics as an example. The entire point cloud can be split into 2 parts, namely the signal part used to determine whether there is a specific signal we are looking for (determine the label), and the background part, which itself is independent of the label given the signal part. This exactly follows the conditional independence assumption as $X_i \bot Y|X_c$. The other two datasets also comply with the assumption that the label is determined by only a part of the data entry, with another part being statistically correlated but conditionally independent of the label given the causal part. Therefore, we respectfully disagree with the argument that “X cannot be divided into two disjoint parts that are causal/independent”, which is at least properly reflected by the datasets considered in this work.

Second, this setting of conditional independence is of a broad range of research interest to the machine learning community. For instance, we can find similar formulations in DIR [1], Invariant Rationalization [2], etc. Take DIR for example, Equation 1 in this paper indicates the conditional-independence property of the oracle rationale, where $S$ is “statistically associated with $Y$ but non-causal to $Y$”, and is independent of $Y$ given the causal part $C$.

In a word, the example $X_c\to Y\to X_i$ raised by the reviewer exists, but it is beyond the scope of this work, especially when we did not find a scientific GDL dataset that obeyed such a setting. We are happy to enrich our benchmark in the future if the reviewer may suggest some scientific GDL datasets that reflect the settings the reviewer would like to cover.

[1] Wu, Yingxin, et al. "Discovering Invariant Rationales for Graph Neural Networks." ICML, 2021.

[2] Chang, Shiyu, et al. "Invariant rationalization." ICML, 2020.

Many thanks to the response of the reviewer. We will also give more explanations in the first concern:

The reviewer comments "The shifts in OoD-Bench, GDS, and GOOD are conceptually closer to the conditional shift". However, the original papers formalized the shift by using a concept shift notion: In OoD-bench, the authors formalized $p(y | z) \ne q(y | z)$ and in GOOD, the authors formalized $P ^{train}(Y |X) \ne P ^{test}(Y |X)$. These are concept shifts according to their explicit formulations, and we tried to follow their notions.

Thank you for your response!

Dear reviewer GGxS,

After carefully considering your comments, we provide point-to-point answers to address all your concerns. As the discussion period is closing, we would appreciate it if you could give some feedback. Your participation is important to the review process.

the authors