Molecule Generation by Heterophilious Triple Flows

Thank you for providing constructive feedback that has allowed us to clarify essential aspects of this paper. We believe our response has adequately responded to your concerns.

Q1: There are many literatures showing that heterophily is not always harmful. How does my paper align with these?
A1: We thank the reviewer for pointing out these works. We are amending the related work section to include these works. We summarize the relation to these works as follows:

• Paper [1] points when the cross-class neighbour similarity is stronger than intra-class similarity, which keeps the post-aggregated embeddings of different classes distinguishable, then the data heterophily is not that harmful for node classification.
• Paper [2] claims their newly designed aggregated homophily is a good indicator of the model expressiveness. However, it’s not very surprising that this homophily indicator is highly correlated with the performance of the model on certain data, since it combines the information from the data (node features, aggregation operator and labels) and model structure (assumption on model type: simplifying graph convolutional networks). In the model design, we share a similar idea: trying to process lower/higher frequency information separately in different channels. At the same time, we are different: For example, while handling high-frequency info, we avoid the information exchange between highly different nodes but keep smoothing the similar clusters. ACM capture high-frequency data by the graph Laplacian, which cares more about the graph topology (this idea comes from graph signal processing)
• Paper [3] verifies that compared with homophily, GNN performance on node classification is more related to intra- and inter-class Node Distinguishability (ND). In molecular generation, graph distinguishability is much more important than ND.
• Paper [4] defines the majority pattern (homophilic nodes in homophilic graph/ heterophilic nodes in the heterophilic graph) and shows that GNN is superior to graph-agnostic models on the majority patterns but worse on minority patterns. It tries to reveal how does the heterophily influence the GNN performance on node classification.

In general, all these papers [1,2,3,4]:

• Focus on the deeper insights between homophily and the GNN model performance on the node classification task. The heterophily matters in different styles: the difference with node classification and graph generation is discussed at the general reply.
• They try to discover the more essential factor (compared with heterophily) which affects the model performance. At the same time, we develop our methods, motivated by noticing the possible lossing information during training from the inspiration of heterophily.

Q2: What is the problem of data heterophily in molecular generation tasks? any evidence or reference? A2: Key to the generative model is learning the data distribution, which possibly contains special patterns underlying the data domain. Here the data is graph-structured, so an expressive model can distinguish different graphs. The smoothing embeddings from the message passing scheme will decrease the expressiveness of models. Thus it is natural to build structure to mitigate this problem. This is the motivation for us to design different flows to capture more information and avoid missing certain types of information. We have included an example analysis in the general reply to all reviewers.

Q3: In equation (2), what is $S$?
A3: In the ACLs (affine coupling layers), the input $X$ is split to two parts $X_1, X_2$ by the masking matrix. One part $X_1$ is updated by another part $X_2$ in the following style. For reversibility, the transformation is chosen to be an affine map, then $X_1$ is updated by the factorization matrix $S$ and translation matrix $T$ to be $SX_1+T$. And $S, T$ are generated from $f(X_1)$, where the $f$ is the coupling function. To avoid the negative factor, we choose $\log S, T = f(X_1)$, as mentioned in Equation (2). The coupling function $f$ is defined to be the GNN with heterophilous message passing layer in Equation (9): $ACL^{(GNN_i^\gamma, \mathbf{M}_i)}$

Q4: Ablation study is missing. How does each component in heterophilious message passing contribute to the performance gain?
A4: Our model adds structure caring about heterophily/homophily based on MoFlow. And MoFlow utilizes the traditional GNN so loses the high-frequency information on heterophilic data. Thus MoFlow could be viewed as a single flow (HTFlows without heterophily and homophilic flows) as ablation. And there is also an example analysis in the general reply.

References the same as the reviewer given.

We appreciate your valuable feedback, which has assisted us in clarifying crucial elements of this paper. We trust that our reply could adequately deal with your apprehensions and results in an improved rating

Q1: Weak experiments: more SOTA experiments should be included.
A1: Many of the SOTA methods do not have code available for reproducing the runs to generate the chemoinformatics metrics. Also copying over partial results from the papers directly does not seem feasible or fair as the evaluation setups differ. For example, in STGG (in the appendix) FCD-qm9 is 0.58 and FCD-zinc is 0.2778, which are even better than the ground truth values we get. They also utilised chemical rules in the learning process, while we hope to learn the data distribution without hand-tailoring this prior knowledge.

Like most of the methods, GDSS, DiGress and GraphARM don’t provide the comprehensive cheminformatics metrics evaluation on QM9 or ZINC, which brings difficulties to merge the results. The main baseline for our work is MoFlow, on which we demonstrate the benefits of the heterophilous flow approach and show to outperform it.

Q2: Why the proposed flow network better at generating heterophilious graphs? A2: Our model is not aiming to be better at generating heterophilious graphs, but to avoid lose information from heterophilious data points during training. Molecular generation cares more about distribution learning. Please see our general reply to all reviewers, where there is an example analysis about problems on heterophilous data and how our model benefits.

Q3 Why not just utilize heterophilic GNNs but design this structure?

A3 The difference between node classification and graph generation task are discussed at general reply. For molecular generation task, graph distribution learning requires avoiding any possible information loss. The existing heterophilic GNN could mitigate oversmoothing problem for node classification. The heterophilic GNNs are good at making heterophilic nodes distinguishable. But it’s not enough for simply replacing traditional GNN by it in current problem scenorias, because it may ignores the subtle low-frequency in the homophilic data. Keeping both traditional GNN (in central flow) and extra two flows helps the model get distinguishable embeddings (from both low and high frequency) for all types of graph-structured data points.

Thank you for your comments.

1. Please note that the baseline MoFlow we compared with is known to strongly outperform baselines such as RNNs/LSTMs and advanced flow models such as GraphAF, and methods such as Junction tree VAEs etc. Since we improve upon MoFlow, we inherit the benefits with respect to these baselines.

2, 3) Please see our global response, where we give a concrete example of how GNNs fail to take into account the effect of heterophily in generative settings, and how our modelling helps overcome this issue.

Thank you again for your feedback, and we hope this addresses your concerns.

Thank you for your constructive feedback that helped us elucidate some important aspects of this work. We hope our response has sufficiently addressed your concerns, and translates into an increased score.

Q1: The potential insonsistency between node embeddings and labels. The reviewer finds defining cosine similarity as homophily is problematic.
A1: This inconsistency appears in the node classification task as discussed in the general reply. Here the node features are the one-hot encoding of the node label (atom type). Since the node label is the learning target during the molecule generation, we need to design a substitute way to approximate the homophily/heterophily. The flow maps the sampled Gaussian embeddings back to the one-hot encoding of atom type (label), to be consistent with the definition of homophily of the final layer, we extend this concept to the intermediate layers as the cosine similarity between the node embeddings.

1. The node features $x_v$ of node $v$ is initialized by the one-hot encoding of atom type. we can write this to be this node $x_v = [1_{v=C}, 1_{v=N}, 1_{v=O}, 1_{v=F} ]$ e.g. for atom C, the node feature is $[1,0,0,0]$.

2. In the flow, the node features are mapped into latent space with a Gaussian distribution. During the molecular generation process, we sample Gaussian distributed random vectors and map them back to node features at the final layer, then this feature is transferred to a node label. It means we don't know the node label until the final layer. To define the homophily for a node in the intermediate layer without a label, we choose the similarity between the half-transferred embeddings to be the homophily. This definition style is also consistent with the final layer.

Q2.1: Need to show the designed heterophilic flows expressive on heterophilic graphs.
A2.1: This is not a node classification task but a generation task. It could be easy to show the ratio of the heterophilic nodes is correctly classified. But here for the generation task, the model is supposed to learn the data distribution, with as least missing information as possible. Traditional GNNs will smooth the high-frequency information on the graph easily. We provide example analysis and discussion in the general answer.

Q2.2: Why design the heterophilic flows but not directly adapt existing heterophilic GNN?
A2.2: The difference between node classification and graph generation task are discussed at general reply. For molecular generation task, graph distribution learning requires avoiding any possible information loss. The existing heterophilic GNN could mitigate oversmoothing problem for node classification. The heterophilic GNNs are good at making heterophilic nodes distinguishable. But it's not enough for simply replacing traditional GNN by it in current problem scenorias, because it may ignores the subtle low-frequency in the homophilic data. Keeping both traditional GNN (in central flow) and extra two flows helps the model get distinguishable embeddings (from both low and high frequency) for all types of graph-structured data points.

Q3: The ablation study on different flows suggested.
A3: Our model adds structure caring about heterophily/homophily based on MoFlow. And MoFlow utilizes the traditional GNN so loses the high-frequency information on heterophilic data. Thus MoFlow could be viewed as a single flow (HTFLows without heterophily and homophilic flows) for ablation study.

Q4: How does the model overcomes the issue of oversmoothing?
A4: There is disscusion and example analysis in the general reply.

Q5: Suggestion: more content about heterophily in the related work section.
A5: Following reviewer VenB's comments, we are amending the related work section with more details and background on heterophily.

We thank the reviewer for their comments. We reply to each concern/question below in the order received.

Q1: Limited improvement on all metrics.
A1: We focus on showing the benefits of our HTFlows framework over MoFlow, which can be seen as a base. Rather than cherry-picking just a few metrics where we excel, we on purpose show a wide range of chemoinformatics metrics. Some of the metrics are a bit of a trade-off; improvement on one metric can have a negative influence on other metrics. For example, if a model would always generate molecules with only carbons, it will get 100% validity, but the diversity should be low. Especially on QM9 (that other works typically test on), we achieve very good performance overall.

Q2: What are the connections between the heterophily and the improved metrics?.
A2: Generative model aims to learn the distributions related to several important patterns from the dataset. Most GNN-based models fail to capture enough information from the heterophilic graph data since the message-passing scheme smooths the graph embeddings after several layers. These improved metrics are good indicators showing the better distribution learning of our model.

Q3: The HTFlows' structure: why design mixing ACL iteratively but not in parallel?
A3: The different flows learn different types of information. We design the interactions for different channel so that they could also utilize the information from other patterns. However, for the reversibility, ACL updates one part of data by another part, which means there is always an order for which part is updated (impossible to update all at the same time, otherwise the structure is not reversible anymore). More layers of the current design can also make sure enough interactions compared to parallel design.

Q4: Inconsistency between weight histogram (Fig 6,7) and tables (Table A4, A5)
A4: They are consistent. As described in App. C, the metrics in Table A4–A5 are the Wasserstein distance/discrepancy metric between the generated molecules and the original data set, summarizing the histogram result into one number.

We have now updated our submission to include the promised changes in the literature review (see Section 2 Related Work).