Capturing substructure interactions by invariant Information Bottle Theory for Generalizable Property Prediction

Thank you for the valuable feedback. I will address each of your concerns in detail.

W1&W2&Q1: Equation (19) needs revision, and mutual information symbols should be consistent

Thank you for your suggestion. In fact, Equation 19 corresponds to the first term in Equation 12, which represents the maximization of mutual information between the extracted core subgraph and the label $Y$, thereby ensuring that $\mathcal{G} _ {\mathrm{sub}}$ retains the most relevant information for $Y$. The second term in Equation 12 is the compression term, which aims to minimize the structure of $\mathcal{G} _ {\mathrm{sub}}$, retaining only the most essential parts, $\mathcal{G} _ {\mathrm{IB}}$.

We have revised the corresponding section to improve the readability and coherence of the manuscript. Additionally, the mutual information symbols have been standardized in the revised version for consistency.

W1&W2&Q2: $\tilde{G}_{IB}$ is defined as a graph, yet in Equations (20)– (22), it is treated as a multivariate vector apparently. with no explanation of this difference within the paper

Thank you for your professional suggestion. We have clarified the distinction between graph representation and vector representation in revised version: “Additionally, the set2set network is utilized to pool $\widetilde{\mathcal{G}} _ \text{IB}$, $\widetilde{\mathcal{G}} _ \text{env}$, and $\widetilde{\mathcal{G}}$, resulting in the substructure representation vectors $s _ {\text{IB}}$, $s_{\text{env}}$, and $s _ {\mathcal{G}}$.” Furthermore, the related formula symbols in the revised manuscript have been updated for improved rigor and consistency throughout the paper.

W1&Q3: Furthermore, the description of $q(\tilde{G} _ {IB})$ in Equation (20) as “the distribution of data under environment $\tilde{G} _ {IB}$” is too vague to understand. With respect to what variable is the expectation E taken?

In our method, we assume that the merged molecular graph $\widetilde{\mathcal{G}}$ can be decomposed into two parts: the rationale and the environment. The non-core part is considered the latent environment $\widetilde{\mathcal{G}} _ \text{env}$. The $\widetilde{\mathcal{G}} _ \text{env}$ can be combined with various rationales to form new molecules representation, collectively constituting the distribution $q(\widetilde{\mathcal{G}} _ \text{env})$.

Directly computing the expectation $\mathbb{E}$ by traversing all molecules in the distribution $q(\widetilde{\mathcal{G}} _ \text{env})$ is challenging due to the infinite nature of the space. To address this issue, we transform the problem into traversal sampling within a finite space. By doing so, we map the infinite potential environments into a fixed number of $M$ environments, enabling the expectation to be solvable according to Equation 23.

W1&W2: defining $\tilde{G} _ {IB}$ as a subgraph of $\tilde{G}$ while expressing $(\tilde{G}{IB}, Y) \sim q(\tilde{G}_{IB})$ is unclear.

We sincerely apologize for the unclear expression that may have caused confusion. The notation $(\widetilde{\mathcal{G}}, \mathbf{Y}) \sim q(\widetilde{\mathcal{G}} _ \text{env})$ indicates that $\widetilde{\mathcal{G}}$ is sampled from the space defined by $q(\widetilde{\mathcal{G}} _ \text{env})$.

W1&Q3: Furthermore, the description of $q(\tilde{G}{IB})$ in Equation (20) as “the distribution of data under environment $\tilde{G} _ {IB}$” is too vague to understand. With respect to what variable is the expectation E taken?

In our method, we assume that the merged molecular graph $\widetilde{\mathcal{G}}$ can be decomposed into two parts: the rationale and the environment. The non-core part is considered the latent environment $\widetilde{\mathcal{G}} _\text{env}$. The $\widetilde{\mathcal{G}} _ \text{env}$ can be combined with various rationales to form new molecules representation, collectively constituting the distribution $q(\widetilde{\mathcal{G}} _\text{env})$.

Directly computing the expectation $\mathbb{E}$ by traversing all molecules in the distribution $q(\widetilde{\mathcal{G}} _ \text{env})$ is challenging due to the infinite nature of the space. To address this issue, we transform the problem into traversal sampling within a finite space. By doing so, we map the infinite potential environments into a fixed number of $M$ environments, enabling the expectation to be solvable according to Equation 23.

W1&W2: defining $\tilde{G}{IB}$ as a subgraph of $\tilde{G}$ while expressing $(\tilde{G}{IB}, Y) \sim q(\tilde{G}_{IB})$ is unclear.

We sincerely apologize for the unclear expression that may have caused confusion. The notation $(\widetilde{\mathcal{G}}, \mathbf{Y}) \sim q(\widetilde{\mathcal{G}} _ \text{env})$ indicates that $\widetilde{\mathcal{G}}$ is sampled from the space defined by $q(\widetilde{\mathcal{G}} _ \text{env})$.

Thank you very much for your response. We are glad to hear that our reply has addressed most of your concerns. Regarding the remaining issue, we are more than happy to provide further clarification to facilitate your understanding.

1. As indicated by its inclusion in the Keywords section, I believe the concept of Out-of-Distribution (OOD) is one of the key focuses of this paper.

We have accepted your suggestion. In the revised version, we have removed references related to OOD words, including Keywords section. The revised version is still being finalized, and we will upload it promptly once completed.

2. In the main text, the authors state that “the OOD problem can be formally defined as follows,” which resulted in the main problem defined by Eq (20). I guess that the solvable approximation of (20) would be Eq (25).

In the original version, Equation (20) is approximately solved by Equation (24), i.e., $\mathcal{L} _ {inv}$. This equation represents placing the extracted $\mathcal{G} _ {\mathrm{IB}}$ into different environments $env _ {i}$ learned through VQ modules to obtain robust representations.

Equation (25) defines the overall loss $\mathcal{L} _ {total}$, which includes $\mathcal{L} _ {inv}$, along with $\mathcal{L} _ {pre}$, $\mathcal{L} _ {MI}$, and $\mathcal{L} _ {vq}$:

• $\mathcal{L} _ {pre}$ can be modeled as the cross-entropy loss for classification tasks and the mean squared error loss for regression tasks.
• $\mathcal{L} _ {MI}$ represents the KL divergence between the extracted substructures and the remaining subgraph, encouraging substructure compression.
• $\mathcal{L} _ {vq}$ aims to optimize the environment vectors in the codebook by mapping multiple environments from the training set into a finite set of discrete environment codes.

3. What exactly does $\tilde{G}_{env}$ in the main problem of Eq. (20) mean? A graph or a subgraph? A vector? A set of graphs or vectors? Or something else? is Eq (20) just an abstract metaphor or something?

In Equation (20), we assume that the merged graph can be decomposed into substructures $\mathcal{G} _ {\mathrm{IB}}$ and $\mathcal{G} _ {\mathrm{env}}$, where $\mathcal{G} _ {\mathrm{IB}}$ represents the core substructure and $\mathcal{G} _ {\mathrm{env}}$ represents the non-core part (i.e., $\mathbf{h}^{r} _ {i}$ in Equation (16)). In the subsequent implementation, we pool $\mathcal{G} _ {\mathrm{env}}$ into vectors $\mathcal{s} _ {\mathrm{env}}$ (In revised version). Equation (20) could be considered as an abstract concept that conveys our intention to traverse all environments, aiming to minimize prediction error (risk function) even in the worst-case scenario. Here, "environment" is an abstract concept, and in our approach, we consider the non-core substructure $\mathcal{G} _ {\mathrm{env}}$ as the environment source.

Specifically, the environment vector $\mathcal{s} _ {\mathrm{env}}$ is obtained by applying a pooling operation to $\mathcal{G} _ {\mathrm{env}}$. Since there are numerous $\mathcal{G} _ {\mathrm{env}}$ structures in the training data, this also generates many $\mathcal{s} _ {\mathrm{env}}$ vectors. To manage this complexity, we use a VQ (Vector Quantization) module to map them to a finite set of $M$ embedding vectors in the codebook $W$.

In the subsequent sections, specifically Equations (21-23), these operations should be treated as vector operations rather than $\mathcal{G} _ {\mathrm{env}}$ and $\mathcal{G} _ {\mathrm{IB}}$. Therefore, we have corrected the equations in the revised version to reflect this more rigorously.

To enhance the clarity of Section 3.3, we have reorganized and revised the content to improve readability and prevent misunderstandings.

4. Is the support E an infinite set?

$\mathbf{E}$ depends on the number of samples in the training set. Theoretically, it represents an infinite potential chemical space; however, in practice, it is equal to the number of training samples.

Thank you for your response! I feel I now have a better understanding of the paper’s content. However, there’s one big issue related to the main claim of the paper that remains unclear, so I’d like to ask for further clarification.

As indicated by its inclusion in the Keywords section, I believe the concept of Out-of-Distribution (OOD) is one of the key focuses of this paper. In the main text, the authors state that “the OOD problem can be formally defined as follows,” which resulted in the main problem defined by Eq (20). I guess that the solvable approximation of (20) would be Eq (25).

Regarding this main formulation, the meaning and assumptions behind the term "environment" in Eq (20) still remain unclear and not well defined to me, as pointed out by most other reviewers.

• What exactly does $\tilde{G}_{env}$ in the main problem of Eq. (20) mean? A graph or a subgraph? A vector? A set of graphs or vectors? Or something else?
• Is the support E an infinite set?
• Or, is Eq (20) just an abstract metaphor or something?
• When the minimization of Eq (20) is approximated by Eq (23), the paper refers to "environmental samples." Does "out-of-distribution" (or environments) in this paper simply refer to the internal variations within the training distribution, and is it correct to interpret the assumption of this paper as these variations being well-represented by M codes in the code book? (I still felt that this was not "out of" the training distribution at all.)

Thank you for the clarification. I understand the explanation about modeling variations in the environment outside the core part using VQ in the training data to stabilize predictions. However, if that’s the case, Observation 7 seems like an expected outcome. This paper appears to focus more on technically implementing robustness, and the discussion on how much of the chemical mechanisms of interaction—the original motivation mentioned in the Introduction and Figure 1—has been captured feels insufficient. Furthermore, removing “OOD prediction” arguments, which was one of the main motivations at the time of submission, and revising the manuscript accordingly seems like a major revision that would typically require re-review.

Thank you for the valuable feedback. I will address each of your concerns in detail.

W1: The method has not been evaluated on common DDI benchmarks.

Thank you for your feedback. Following reference [1], we have incorporated two additional datasets, DrugBank and Twosides, to evaluate the performance of I2Mole (In Acc index).

Clearly, I2Mole outperforms current baselines significantly on the DrugBank and Twosides datasets, with improvements of 0.319% and 0.15%, respectively, in the transductive setting. More importantly, under inductive settings, I2Mole demonstrates even more substantial improvements compared to the second-best model. Specifically, it achieves gains of 0.622% and 2.89% in the Type 1 scenario, and 1.22% and 5.9% in the Type 2 scenario. These results highlight I2Mole's superior predictive accuracy and stronger generalization capability.

In terms of this work, we utilize graph embedding-based techniques to acquire potentially effective molecular features. Then we evaluated on ZhangDDI, ChchMiner, and DeepDDI, which are also widely used benchmark datasets for molecular relationship learning [2,3,4]. Since the primary focus of this work is on the out-of-distribution (OOD) generalization of DDI tasks, we designed a series of experiments to assess model performance. First, we conducted evaluations under both transductive settings (where both molecules appear in the training set) and inductive settings:

• Type 1: Predicts potential interaction properties between known and unseen drugs.

• Type 2: Predicts potential interaction properties between unseen drugs.  

These settings not only evaluate the model's performance under conventional testing methodologies but also provide insights into its generalization ability to unseen molecules, a key aspect of OOD testing [2]. Further, we conducted generalization tests based on scaffold splitting and size splitting, similar to conventional single-molecule systems [5,6]. We also performed domain shift tests, following the design of DSIL-DDI [6], to validate the model's domain generalization performance. It is worth noting that the experimental setup for DDI tasks varies depending on the specific research focus. For example, ZeroDDI [8] focuses on zero-shot DDI event prediction, and KnowDDI [9] aims to uncover DDI relationships under long-tail distributions, where dataset splits are based on classes.  

Given our emphasis on generalization, our experimental setup differs from these approaches, as we aim to explore a different aspect of the DDI problem [10]. We hope this provides clarity regarding the reason behind our experimental design.

[1] Comprehensive Evaluation of Deep and Graph Learning on Drug–Drug Interaction Prediction, Briefings in Bioinformatics, 2023

[2] SSI–DDI: substructure–substructure interactions for drug–drug interaction prediction, Briefings in Bioinformatics, 2021

[3] Conditional Graph Information Bottleneck for Molecular Relational Learning. ICML 2023

Thank you for the valuable feedback. I will address each of your concerns in detail.

W1&Q1: why intra-molecular and inter-molecular message passing need to be separated after constructing the merged graph

Thank you very much for your insightful suggestion. The separation of intra-molecular and inter-molecular message passing is motivated by the distinct physical and chemical characteristics of these interactions. Intra-molecular interactions, such as covalent bonds and specific geometric constraints, often adhere to well-defined physical rules, whereas inter-molecular interactions involve weaker forces like hydrogen bonds, van der Waals forces, and other non-covalent interactions [1,2].

Handling these two types of message passing separately allows us to better capture their unique properties. Moreover, this separation enables the design of tailored optimization strategies or network architectures (e.g., differing retention rates for relational edges) to efficiently model each type of interaction. For example, in our approach, we employed MPNN for intra-molecular message passing to effectively model the structured, rigid interactions within molecules, while using TGAT for inter-molecular message passing to account for the transient and weaker nature of inter-molecular interactions.

This division is particularly important because integrating these two types of interactions without distinction could introduce unwanted noise. Specifically, insufficient incorporation of inter-molecular information could weaken the learning of inter-molecular interactions, while excessive inclusion might disrupt the intrinsic properties of intra-molecular relationships, as demonstrated in Table 10. Additionally, handling an excessive number of relational edges simultaneously could unnecessarily increase model complexity.

Considering these factors, we concluded that separating intra-molecular and inter-molecular message passing is essential for effectively capturing their respective characteristics and improving model efficiency.

In revised version, we have added relevant descriptions in the Introduction section to facilitate readers' understanding.

[1] Chi Z. Toxic interaction mechanism between oxytetracycline and bovine hemoglobin. J. Hazard, 2010

[2] Chen D. Algebraic graph-assisted bidirectional transformers for molecular property prediction. Nat.Commun, 2021.

W1&Q1: Further elaboration on the role and benefits of vector quantization in this context.

Thank you for your thoughtful suggestion. As mentioned in the Introduction, randomly simulated noise and indiscriminate noise injection can introduce potential issues, such as instability and lack of chemical interpretability. To address these concerns, we leveraged the non-core substructures of the merged molecular graph $\widetilde{\mathcal{G}}_\text{env}$ as latent environmental information. This adjustment not only enhances model robustness and generalization but also aligns with the underlying chemical properties of the data.

Given the vast and largely unknown nature of the potential chemical space, collecting all possible environments is impractical. Therefore, the VQ module clusters an infinite number of potential environments into a finite set. This clustering enables us to approximate a diverse range of environment, derived from pooling results of less important nodes (Eq. 22), thus providing a more computationally feasible and chemically meaningful representation.

To further illustrate the role of the VQ module, we introduced extra two variants for comparison in this revised version:

1. RD Noise Variant: In this version, noise in the environment codebook is entirely random, mimicking the effects of random noise injection.

2. Instance-Dependent (ID) Noise Variant: Here, we sampled new environments from the environment codebook and added them as small perturbations to the instance-dependent environment. The Gaussian distribution of this noise is determined by the mean and variance of subgraph node vectors, emphasizing instance-dependent noisy perturbations.

Our experimental results demonstrate the superiority of the VQ module and the proposed optimization strategy, particularly in improving the model's robustness and ability to generalize across diverse chemical environments.

Thank you for the valuable feedback. I will address each of your concerns in detail.

W1: Some of the math notations are confusing in equation (1), (2) and (12). And "env" and "E" are not clearly defined.

Thank you for your feedback. We have adopted a more standardized format to redefine Equation (1) and Equation (12). These changes have been incorporated into the revised version for clarity and consistency.

Equation (1):

Intuitively, where $\mathcal{S}$ represents the set of $\mathcal{G} _ {\mathrm{sub}}$, $\mathcal{G} _ {\mathrm{sub}}$ is the general subgraph.

Equation (12):

where $\mathcal{\widetilde{S}}$  represents the set of $\mathcal{\widetilde{G}} _ {\mathrm{sub}}$. Each term indicates the prediction and compression terms respectively, which should be minimized during training, as outlined below.

Equation (2):

Here, $\mathbf{E}$ represents the set of environments, i.e., the collection of all possible environments $\mathcal{G} _ {\text{env}}$. $\mathcal{G} _ {\text{env}}$ is a specific instance of an environment, $env$ is an environment variable in environment codebook $W$, and $\mathbb{E}$ represents the mathematical expectation.

W2: More clarification is needed to explain how to combine these components to construct $\mathcal{\widetilde{G}} _ {\mathrm{IB}}$ in details.

We sincerely apologize for the writing error. The correct statement should be: $\mathbf{h}^{r} _ {i}$ is the irrelevant substructure node which would be used to construct $\mathcal{\widetilde{G}} _ {\mathrm{env}}$ (in section 3.3). Here, $h _ {i}$ denotes the original node feature, and $\lambda _ {i}$ is the weight assigned to the node.

During the construction of node embeddings $z _ {i}$ in $\mathcal{\widetilde{G}} _ {\mathrm{IB}}$, it is formed by the combination of $\lambda _ {i}h _ {i}$ and $(1-\lambda _ {i})\epsilon$, where $\lambda_ {i}h _ {i}$ represents the rationale information, i.e., the core features. In this work, $(1-\lambda_ {i})\epsilon$ is considered as the environmental feature. This environmental feature originates from the molecule itself but lacks robustness. To address this, we subsequently leverage $\mathbf{h}^{r} _ {i}$ to identify new environments and construct the final representations, as described in Section 3.3.

W3: How to decide M.

$M$ represents the environmental vector number in the environment codebook, which is treated as a hyperparameter, as recorded in Table 8. We have conducted hyperparameter experiments in original version, as presented in Figure 6, to determine the optimal value for $M$.

We greatly appreciate your insightful and helpful comments, as they will undoubtedly help us improve the quality of our article. If our response has successfully addressed your concerns and clarified any ambiguities, we respectfully hope that you consider raising the score. Should you have any further questions or require additional clarification, we would be delighted to engage in further discussion. Once again, we sincerely appreciate your time and effort in reviewing our manuscript. Your feedback has been invaluable in improving our research.

Dear Reviewer b1KJ,

Thank you for your positive feedback and continued support for our paper! We appreciate your thoughtful review and are glad that we have adequately addressed your concerns.

Best regards,

Authors.