MAGNet: Motif-Agnostic Generation of Molecules from Shapes

 

Dear Reviewer,

We greatly appreciate your time and effort in reviewing our manuscript. Your feedback has been valuable in improving our work, and we address your concerns in our answer below. Your questions regarding clarifications of the proposed generation approach aided in revising the methods section. In our answers, we indicate where we changed the manuscript to address the raised concerns.

 

Connection between our work and other "shape" works that were not discussed

• Note that the mentioned works focus on a different task where shapes refer to surfaces in 3D space while we consider untyped 2D graph as shapes. We are familiar with these works but originally refrained from including them, to avoid confusion with respect to the task we address. Adams et al., in particular, use MoLeR to create sufficient priors for their method. The same is possible with our approach and, due to the introduction of shapes, even with more flexibility. We realise that not including these works also has the potential for confusion. Hence, we differentiate our methods from the mentioned "shape" works in the updated version of the related work section, cf. end of Section 3.

 

In the following, we discuss your questions and concerns regarding the method description.

How do you generate the shape multiset?

• We train a 2-layer transformer decoder to generate the shape multiset. We use the decoder to learn the shape set in an autoregressive manner. We updated the "shape-level" paragraph in Section 2.2 and provide a more detailed explanation of the multiset generation in the Appendix B.2.

Why is an atom type present on the shape level?

• The requirement of the atom type on the shape's adjacency helps to ensure that the generated shape allocations can be joined on the atom level. Strictly speaking, this is not required and we updated the "Factorising the data distribution" paragraph in Section 2 accordingly. We analysed the effect of this design choice and provide the results for this experiment here. Including the atom type of the join improves the quality of generated molecules by 16% according to the FCD score. We explain the reason for this design choice more thoroughly in the "shape-level" paragraph of Section 2.2.

How do we extract the shapes? Can you provide more examples?

• We clarified the discussion about the fragmentation and extraction of shapes in Section 2.1 of the manuscript. We further provide additional samples of the shape set in Appendix A, where we illustrate examples that we only discussed verbally in the initial manuscript.

How the Normalising flow is applied is confusing

• We observed that solely optimising the KL-term leads to over-pruning [1]. Thus, we apply a normalising flow to the latent space. To do so, we rely on Conditional Flow Matching [2, 3]. We have clarified this point at the end of Section 2.3 of the revised manuscript.

Terminology: shapes -> topology

• We apologise for the confusion caused by the terminology. After internal discussion, we decided to stick with the shape term in the current version of the manuscript. However, we provided a clarifying statement at the beginning of Section 2, "Factorising the data distribution", when defining shapes and refer to the concept of topology in various other parts other the discussion.

 

We hope to have addressed your concerns and answered your questions adequately and we are looking forward to discussing our work further with you.

 

[1] Serena Yeung, Anitha Kannan, Yann Dauphin, Li Fei-Fei, Tackling Over-pruning in Variational Autoencoders, arxiv.org/abs/1706.03643
[2] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le, Flow matching for generative modeling, International Conference on Learning Representations, 2023, arxiv.org/abs/2210.02747
[3] Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio, Improving and generalizing flow-based generative models with minibatch optimal transport, arxiv.org/abs/2302.00482

Dear Reviewer,

Thank you for your response. We appreciate your feedback. However, it remains unclear to us what requires significant revision. Below, we provide detailed answers to the raised points to clarify our understanding of the feedback.

 

I still find the paper challenging to grasp due to the scattered presentation of details.

Based on the feedback we received, we believe that our (revised) manuscript follows a clear structure. Hence, we are unsure about the scattered details that were mentioned in the feedback. If the criticism is related to referring to the appendix multiple times from the main manuscript, we would like to clarify that the appendix A only provides additional examples and supporting information that are not crucial for the main part of the manuscript and were asked for in the initial review. The same applies, for example, to the appendix sections B which provides implementation details that will become less relevant once the code is publicly available. We hope this clears up any confusion.

An improved overview section could be beneficial in this regard.

We would like to understand why the provided overview paragraphs are insufficient. We include multiple overview paragraphs throughout the main manuscript to help guide the reader:

• In the introduction, we clearly state our contributions, followed by a detailed explanation of the claims made in the paper and how they are reflected in the experiment section.
• Section 2 introduces our contributions on the methodological level by first discussing the data factorisation and nomenclature, followed by our fragmentation procedure and MAGNet as a generative model fitting the fragmentation and proposed factorisation. This structure is again highlighted at the beginning of Section 2.
• In Section 3, after discussing key aspects of our manuscript in two paragraphs of related work, we put MAGNet in a broader context and distinguish our work from others which also was refined based on the feedback we got in the the initial review.
• Section 4, the experiment section, starts again with an overview of the different experiments, introducing their aspects and outlining how they are meant to analyse key aspects of the generative approach.

Additionally, the paper should include ablation studies to demonstrate the contribution of each MAGNet component, e.g., normalising flow, atom type on the shape level.

We are happy to include ablation studies as we already have provided the results for the atom type inclusion in our intial response. Regarding the normalising flow, we find the analysis on active units and insufficiency of the KL-regularisation more convincing. However, we have also analysed the generative performance of a MAGNet model without the normalising flow:

• We observed that the performance for the experiments from section 4.1 and 4.3 and 4.4 remained largely unchanged.
• The unconditional generation, however, as analysed in 4.2 decreased to a score of 0.65 according to the FCD metric.

As these analyses do not directly reinforce the primary arguments of the paper, we have included them in appendix section C.4.

 

In our understanding, the raised criticism pertains solely to the details of the method and stylistic aspects of the manuscript, not its content or main contributions. We believe that the current score does not reflect this adequately, and we would be grateful if you would reconsider your evaluation.

In essence, the paper's comprehensiveness is hindered by two key factors:

• Unfamiliar Terms: Several technical terms, such as 'leaves,' 'joins,' and 'OS,' are introduced without adequate explanation. These terms appear in Section 2.2 and Table 1, respectively, leaving readers without a clear understanding of their usage and significance.
• Model Design Rationale: The paper lacks a thorough explanation of the rationale behind the model's design. The current presentation resembles a summary of experiments rather than a comprehensive discussion of the model's architecture and underlying principles. For instance, the justification for adopting a two-stage approach for shape generation instead of a more end-to-end approach remains unclear.

While I acknowledge the authors' efforts and the potential contributions of their work, the current presentation of the paper falls short of the bar of ICLR. A revised overview section that clearly defines key terms and elaborates on the design principles of the model would significantly enhance its comprehensibility. In the absence of such revisions, I maintain my current score.

 

Dear Reviewer,

Thank you for taking the time to review our work and providing us with valuable feedback. We appreciate your constructive criticism and insightful comments that will help us improve our manuscript.

We are pleased to observe that you appreciate the strengths of our work. Regarding the weaknesses you have pointed out, we have noted them and addressed them point-by-point. Additionally, we indicate where we changed the manuscript to address the respective concern.

 

Comparison with diffusion models

• We initially decided against adding a paragraph on diffusion models as these primarily focus on molecule generation in 3D space [1,3,4], which is different from the focus of our work. However, we agree that it is beneficial for the reader to present the entire spectrum of molecule representations (SMILES, graphs, geometries) and have added in a paragraph in Section 3 (Related Work, "Molecule generation") of the updated manuscript.
• We highlight DiGress [2] specifically as it is a discrete diffusion model for general graphs that is applicable to 2D molecular graphs. However, similar to SMILES-LSTM, a comparison between the models is difficult due to its implicit learning of representations. DiGress, in its current form, is not applicable to the majority of our experimental evaluations and downstream applications, such reconstruction experiments in Section 4.1 and 4.3, scaffold/shape conditioning in Section 4.4, or the interpolation between molecules in Appendix C.
• However, DiGress works for unconditional generation and we will include it in the presented benchmark, see Section 4.2. After training for >2,5 days on one A100 GPUs (already 2x the compute resources of any other model), the model achieves an FCD score of 0.6 (MAGNet has 0.76). For the benchmark computation, we discarded all invalid molecules, which corresponds to about 26% of the generated samples. Note that sampling is 33x slower than MAGNet.

Missing surveys citations

• We are aware of these works [5,6,7] and apologise for the oversight. To put MAGNet in a broader context, we updated the manuscript accordingly, see Section 3 "Related work" and Section 1 "Introduction".

Unusual terminology

• We acknowledge your comment regarding terminology and updated our manuscript to follow the survey from Zhu et al. [6], that is, we replace "AAO" with "OS". Importantly, "AAO" refered to "all-at-once" (Section 3) rather than "all-atom-object", which was motivated by the work of Yang et al. [5].

Discuss the variety of representations for molecules

• In the initial manuscript, we focused on the differentiation between SMILES and graphs for molecule generation. We added a clarifying sentence on using descriptors for molecule representation in Section 3 to address your comment. For this, we followed the survey from Du et al. [7].

 

We appreciate your feedback and hope we addressed all raised concerns to your satisfaction in the revised manuscript. Once again, thank you for your time and effort in reviewing our work.

 

[1] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, Max Welling, Equivariant Diffusion for Molecule Generation in 3D, arxiv.org/abs/2203.17003
[2] Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, Pascal Frossard, DiGress: Discrete Denoising diffusion for graph generation, arxiv.org/abs/2209.14734
[3] Clement Vignac, Nagham Osman, Laura Toni, Pascal Frossard, MiDi: Mixed Graph and 3D Denoising Diffusion for Molecule Generation, arxiv.org/abs/2302.09048
[4] Minkai Xu, Alexander Powers, Ron Dror, Stefano Ermon, Jure Leskovec, Geometric Latent Diffusion Models for 3D Molecule Generation, arxiv.org/abs/2305.01140
[5] Nianzu Yang, Huaijin Wu, Junchi Yan, Xiaoyong Pan, Ye Yuan, Le Song, Molecule Generation for Drug Design: a Graph Learning Perspective, arxiv.org/abs/2202.09212
[6] Yanqiao Zhu, Yuanqi Du, Yinkai Wang, Yichen Xu, Jieyu Zhang, Qiang Liu, Shu Wu, A Survey on Deep Graph Generation: Methods and Applications,arxiv.org/abs/2203.06714
[7] Yuanqi Du, Tianfan Fu, Jimeng Sun, Shengchao Liu, MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design, arxiv.org/abs/2203.14500

Dear Reviewer,

Thank you for the kind response and the update of your score. For completeness, we would like to follow up on the intermediate results that we have reported for the DiGress model. At the time of reporting, the model had been training for 2.5 days and achieved an FCD score of 0.6. After an additional 2 days and completing 1000 epochs of training, this has not further improved.

Again, we would like to thank you for your valuable feedback.

 

Dear Reviewer,

We appreciate your comprehensive review of our manuscript and valuable feedback. It is encouraging to see that you appreciate our work in multiple ways. We answer your questions below and further indicate how we addressed your feedback in the revised manuscript.

 

Unclear why KL-term does not work and how normalising flow is fitted

• It is important to note that we conducted an extensive investigation into the ineffectiveness of the KL-term. During our analysis of the latent space, we found that optimising the KL-term leads to over-pruning, meaning many latent factors fail to learn anything and become inactive [1]. Despite trying different methods, such as introducing dropouts or epitomes, we could not solve this optimisation issue satisfactorily. As a result of this analysis, we chose to fit a normalising flow post-hoc to the VAE which was trained with low KL regularsation. We agree that this clarification would be beneficial to include in the manuscript and have updated Section 2.3 accordingly, providing the analysis of active units in Appendix B.3 as well as an ablation study on this in Appendix C.4.
• Regarding details on the normalising flow, we clarify its optimisation in the revised version of Section 2.3 and put it in context with the above explanations.

Idea: decouple shapes and atom-level also in latent space

• We appreciate your suggestion regarding decoupling shapes and atom-level in latent space. As the computed features with respect to shape and atom graph are extracted from the same molecule, they will be correlated. Therefore, it is not easily possible to disentangle the representations for this hierarchy. However, we followed up on your suggestions and split the latent dimensions to decode for the distinct properties of the molecular graph according to our factoriation. Here, we investigated the setting of splitting into shape and atom representations. In our experiment, the performance of our MAGNet model was unchanged, and we hypothesise that modelling shape and atom level truly independently, one has to change the datastructure (train the hierarchy differently) and/or introduce explicit disentangling losses.

Fig 3b

• Thank you for your feedback on Fig 3b's axis label. The figure had a typo, and we intended to refer to the ratio. As you correctly pointed out, MAGNet is designed to capture the structural variety of shapes. However, note that this design choice introduces additional complexity for generating shape allocations, which MAGNet captures to such an extent that it even outperforms other methods, highlighting its greater flexibility shown in Figure 4. We demonstrate MAGNet's ability to learn meaningful atom allocations both qualitatively (4a) and quantitatively (4b). To clarify this point, we updated Section 4.1 ("MAGNet reliably decodes shapes") of our manuscript.

Condition MAGNet on a particular motif

• One can readily use MAGNet for this kind of conditional task. We have shown results for this task in Figure 5 (top) of the initial manuscript. As this might have been unclear before, we improved the experiment discussion in the corresponding paragraph of Section 4.4.
   

Once again, we thank you for your insightful feedback and hope we have addressed your concerns satisfactorily in the revised manuscript.

 

[1] Serena Yeung, Anitha Kannan, Yann Dauphin, Li Fei-Fei, Tackling Over-pruning in Variational Autoencoders, arxiv.org/abs/1706.03643

 

Dear Reviewer,

We have acknowledged your feedback and are surprised by the tone of the review. Nevertheless, we believe many of the points raised in this review are based on misunderstandings, which we would like to clarify.

 

Questions on the General Relevance of the GuacaMol/MOSES benchmarks for Drug Discovery, Evaluation on the Open Graph Benchmark

• Our experimental section focuses on critically assessing how well the underlying distribution is learned. We employ two standard benchmarks for this purpose. The GuacaMol distribution learning benchmark "encompasses measuring the fidelity of the models to reproduce the (property) distribution of the training sets" [1]. Further, the MOSES benchmark evaluates "the quality and diversity of generated structures" [2]. Both benchmarks are specifically designed for de-novo molecular generation models. These benchmarks have become standard for 2D de-novo molecule generation over the past few years, with most works published at ICLR or similar venues relying on them for critical evaluation of their methodology [3,4,5,6].
• MAGNet challenges a common perception in the domain of graph-based molecule generation: the requirements of fragments. We discuss the limitations of the standard benchmarks in Section 4.2 and complement the benchmark results comprehensively in the rest our experimental section.

Why did you not train on large-scale benchmarks, do the authors have a reasonable argument for why at scale it would help to have their model

• The "Open Graph Benchmark" is a benchmark for property prediction on the edge, node, and graph level, it is not a benchmark for graph generation. The encoder we use as part of MAGNet has already been evaluated on the OGB benchmark [7].
• The core motivation for the abstraction to shapes is that representing a distribution of molecules by motifs suffers from increasing combinatorial complexity (by also considering edge and node features). As we show in our experimental evaluation, the ZINC dataset is sufficient to expose drawbacks to fragment-based methods and demonstrate the advantages of our factorisation as demonstrated by MAGNet.
• We are certain that these factors are applicable in the case of a large amount of data as well. For instance, the MAGNet model only requires a set of less than 400 shapes to decode for more than 98% of the 1.5 million molecules in the GuacaMol dataset, whereas the MICAM model requires more than 20,000 motifs for the ZINC dataset alone.
• Our work provides a new perspective on fragment-based generation methods. While we are interested in further improving our method, e.g. the shape extraction process towards an even more concise vocabulary, our experimental evaluation has shown the superior performance of MAGNet. We consider the task of "scaling up 2D molecule generation methods" a natural step for future work which is, however, beyond the scope of the current manuscript.

Look at the internal representation

• We would be keen to know in what way such an analysis would add additional insights beyond the already conducted experiments.

What the authors call "shape" is the so-called Murcko scaffold of individual fragments

• MAGNet does not operate with Murcko scaffolds. Rather than utilising the simplest feature allocation (Murcko scaffolds), MAGNet works with featureless abstract graph objects (shapes). We deemed this more suitable for the audience of an ML conference like ICLR. Nevertheless, this misunderstanding about MAGNet's use of shapes has prompted us to improve Section 2 in a general fashion.

 

We appreciate your feedback and hope to have addressed all raised concerns in the revised manuscript. We look forward to your response.

 

[1] Brown et al., GuacaMol: Benchmarking Models for de Novo Molecular Design, Journal of Chemical Information and Modeling, 2019
[2] Polykovskiy et al., Molecular sets (MOSES): a benchmarking platform for molecular generation models, Frontiers in pharmacology, 2020
[3] Kong et al., Molecule Generation by Principal Subgraph Mining and Assembling, Advances in Neural Information Processing Systems, 2022
[4] Geng et al., De Novo Molecular Generation via Connection-aware Motif Mining, International Conference on Learning Representations, 2023
[5] Maziarz et al., Learning to Extend Molecular Scaffolds with Structural Motifs, International Conference on Learning Representations, 2022
[6] Jin et al., Hierarchical Generation of Molecular Graphs using Structural Motifs, International Conference on Machine Learning, 2020
[7] Shi et al., Masked label prediction: Unified message passing model for semi-supervised classification, 2020