Enhancing Graph Invariant Learning from a Negative Inference Perspective

Dear Reviewer vzxf,

Thank you so much for taking your valuable time to review our manuscript. We greatly appreciate the positive feedback on our work. Here we will carefully address your concerns.

W1&Q1. Intuitive analysis

Prompt learning and negative learning collaboratively contribute to performance improvement, which is studied in Apendix E.4. Given your valuable comment, we will move the discussion from the Appendix to the main text in the revised version of our manuscript. Specifically, we designed two variants of NeGo: PoGo uses a positive learning mechanism, and PoGo (w/o. prompt) masks the prompt learning module. The performance comparison results we obtained are as follows: NeGo > PoGo > PoGo (w/o. prompt). Therefore, using only negative learning mechanism or only the prompt module does not achieve the maximum performance.

W2&Q2. Learnable parameters

The reasons for choosing learnable parameters as prompt guidance are as follows:

• Limitations of manual design. Traditional manual prompts require expert knowledge and may not generalize well to new data or tasks. Especially in scenarios with data distribution shifts, manually designed prompts often struggle to generalize across all distributions.

• A data-augmented perspective. The learnable prompt enables implicit enhancement by dynamically (learnably) updating the input prompts without directly altering the original data distribution. This design is particularly well-suited for tasks involving data distribution shifts.

W3&Q3. The efficiency of model

We support that model efficiency is a crucial evaluation criterion. In the next version of the manuscript, we will incorporate Appendix E.5 and following table into the main text.

Thank you again for taking the time to review our work! We will carefully consider your suggestions and further revise our manuscript to satisfy the high standards of ICML community.

Dear Reviewer sxw9,

Thank you for taking valuable time to review our work. We have carefully considered your comments, and the following are our detailed responses. We hope these details could address your concerns.

W1. The environment in the graph data.

The environment factors in graph data are highly informative and extend well beyond just graph scale. In graph-structured data, environment variables are typically defined as all information (spurious subgraph) beyond the causal subgraph. In synthetic datasets, researchers commonly simulate environment variations by increasing the size of spurious subgraph. For example, in GOOD-Motif dataset, base graphs (label-irrelevant subgraphs) can be seen as environment factors and motifs (label-determining subgraphs) are be consider as the casual factors. In real molecular datasets, the environment variables are often characterized by identifying molecules with complex structure and a diverse range of atomic types.

Actually, the complex environments discussed in our manuscript encompass not only changes in scale but also variations in subgraph types. The environment variables for each dataset are detailed in the table below.

W2. A discussion of experiment phenomenon.

Thank you for your insightful comment. In real-world tasks, causal variables extend beyond the subgraph itself and also incorporate additional contextual information. We aruge that negative learning may cause the model to actively suppress certain features that are beneficial for ID scenarios but ineffective in OOD scenarios. The phenomenon where our NeGo achieves suboptimal performance on ID val but delivers the best results on OOD val validates our view.

W3. FIIF and PIIF.

FIIF and PIIF represent two possible interactions between invariant and spurious features during graph generation, as modeled by SCMs [1]. FIIF and PIIF have been accepted by many studies for verifying whether the proposed model demonstrates comprehensive invariant learning ability [2,3].

Under the FIIF assumption, the invariant feature $C$ fully contains all the information about the label $Y$, i.e., $(S, E) \perp Y \mid C$. This means that given $C$, $S$ and $E$ are independent of $Y$. In the structural causal model under FIIF, we have $Y := f_{\text{inv}}(C)$, $S := f_{\text{spu}}(C, E)$, $G := f_{\text{gen}}(C, S)$. Here, $f_{\text{inv}}$ is the label generation function, $f_{\text{spu}}$ describes how $S$ is influenced by $C$ and $E$, and $f_{\text{gen}}$ is the graph generation function. Under the PIIF assumption, the invariant feature $C$ only partially contains the information about the label $Y$, i.e., $S$ is indirectly dependent on $C$ through $Y$. In the structural causal model under PIIF, we have $Y := f_{\text{inv}}(C)$, $S := f_{\text{spu}}(Y, E)$, and $G := f_{\text{gen}}(C, S)$. Here, $f_{\text{inv}}$ is the label generation function, $f_{\text{spu}}$ describes how $S$ is influenced by $Y$ and $E$, and $f_{\text{gen}}$ is the graph generation function.

In both FIIF and PIIF, the interaction between $C$ and $S$ is different, leading to distinct distribution shift patterns. In FIIF, $C$ directly controls $S$ , while in PIIF, $C$ indirectly controls $S$ through $Y$ . Understanding these differences is crucial for addressing distribution shifts. In our work, we also validate that our NeGo possesses comprehensive causal learning ability through theoretical proof. In the next version of our manuscript, we will introduce more discussions on the importance of studying causal learning under the FIIF and PIIF assumptions.

Typos.

Many thanks for your careful review. We will certainly correct the typos you pointed out and proceed to check our manuscript in greater detail. We will carefully consider your comments and revise our manuscript to improve it further. Looking forward to your feedback!

References:

[1] Chen, Yongqiang, et al. "Learning causally invariant representations for out-of-distribution generalization on graphs." Advances in Neural Information Processing Systems 35 (2022): 22131-22148.

[2] Chen, Yongqiang, et al. "Does invariant graph learning via environment augmentation learn invariance?." Advances in Neural Information Processing Systems 36 (2023): 71486-71519.

[3] Gui, Shurui, et al. "Joint learning of label and environment causal independence for graph out-of-distribution generalization." Advances in Neural Information Processing Systems 36 (2023): 3945-3978.

Dear Reviewer wCN4,

Thank you for taking valuable time to review this work. We have carefully considered your comments, and the following are our detailed responses. We hope these details could address your concerns.

W1. Parameterized networks

${f_\phi }$ represents the negative prompter to learn negative embedding, ${g_\xi }$ and ${g_\theta }$ are subgraph extractor and final predictor. We will update Fig. 2 by adding the parameter networks to the structure figure in the next version.

W2. Negative learning and negative sampling

The differences between our negative learning (NL) and negative sampling (NS) is as follows:

• NS primarily aims to optimize training efficiency, while NL focuses on enhancing the model's generalization and distinguishment ability.

• NS involves randomly selecting a small number of negative samples from all negative samples for training. In contrast, our NL design negative loss to explicitly guide the model in learning how to distinguish between positive and negative samples, and use prompt mechanism to increase the contribution of negative samples to the model's generalization.

W3&4. Quantitative evaluation and Training efficiency

We will add the following in the description: Our NeGo achieves optimal performance across four datasets, with the highest performance improvement of 1.17%. In the next version of the manuscript, we will incorporate Appendix E.5 (Efficiency Analysis) into the main text.

W5. typo Many thanks for your careful review. We will certainly correct the typos you pointed out and proceed to check our manuscript in greater detail.

Q1. Unique design of graph-structured data

The unique prompt design of our NeGo lies that the prompt outputs serves as a latent embedding for extracting subgraphs, rather than being directly used for prediction. Given that label-related information in graph-structured data corresponds to subgraphs, we employ prompt learning with the goal of disentangling causal subgraph from spurious subgraph.

Q2. Intuitive validations

The effectiveness of negative prompting in NeGo can be intuitively understood from following several perspectives:

• Broadening the Environment Inference Space. Negative prompting allows the model to consider all extra-class samples as potential environments. This significantly expands the scope of environment factors that the model can infer, beyond just the spurious subgraphs within the same class.

• Our Case Studies and Visualization. In Appendix E.4, we show that the model consistently identifies the ground-truth subgraphs even in complex environments. This demonstrates that negative prompting helps the model accurately distinguish between causal and spurious factors.

Q3. Transferability

• Transferability Across Tasks. The negative inference mechanism of NeGo not only focuses on extracting causal subgraphs but also learns environment factors through negative prompting, a mechanism that may have a certain degree of universality across different tasks. Experimental results show that NeGo performs well in different tasks, such as molecular property prediction (e.g., GOOD-HIV, DrugOOD) and social sentiment analysis (e.g., GOOD-SST2, GOOD-Twitter). This shows that the design of NeGo can accommodate the demands of different tasks and has a certain level of transferability across tasks.

• The design of NeGo is based on causal theory and negative inference, theories that have broad applicability across different fields (such as chemistry, biology, social sciences). Experimental results indicate that NeGo performs well in molecular property prediction tasks in the chemical domain and social sentiment analysis tasks in the social domain. This suggests that the design of NeGo can adapt to data distribution shifts in different fields and has a certain level of transferability across domains.

Thanks again for your constructive comments, which are valuable in helping us further improve our work. Look forward to your further feedback!

Dear Reviewer cMu6,

We appreciate your time on reviewing our work. We will provide further clarification to address your concerns.

W1 and Q1. Complex environments.

Our proposed NeGo remains effective in other complex environment scenarios, not limited to the size-dependent environment cases. Actually, we have discussed the effectiveness of NeGo in scenarios where spurious subgraph styles become more complex in Sec 6.3. As shown in Tab. 3, we observe the superior performance of NeGo in style-dependent complex environment cases, achieving a 1.29% performance improvement.

W2 and Q2. Prompt learning and negative learning.

Prompt learning and negative learning collaboratively contribute to performance improvement, which has been studied in Apendix E.4. Given your valuable suggestion, we will move the discussion from the Appendix to the main text in the revised version of our manuscript. Specifically, we designed two variants of NeGo: PoGo uses a positive learning mechanism, and PoGo (w/o. prompt) masks the prompt learning module. The performance comparison results we obtained are as follows: NeGo > PoGo > PoGo (w/o. prompt). Therefore, using only negative learning mechanism or only the prompt module does not achieve the best performance.

W3. Detailed discussion of related works.

We will emphasize the researches on graph OOD from an environment perspective with more details in Sec 3.

GALA investigates the problem of invariant graph representation learning from the perspective of environment augmentation. LECI introduces the environment discrimination module and utilizes an adversarial learning mechanism to achieve invariant learning. Our proposed NeGo share the same research goal with those works, achieving better invariant learning mechanisms through environment modeling. However, our research persepctive and techniques for achieving this goal are completely different with them. Specifically, we not only focus on environment shift issues in linear mixed scenarios, but also place greater emphasis on model performance under complex and nonlinear environment shifts. Technically, we use prompt learning and negative learning mechanisms to capture the environment.

Thank you again for your positive comments on our work, which are invaluable for improving our manuscript. We look forward to your further feedback.