MAGNet: Motif-Agnostic Generation of Molecules from Scaffolds

We want to thank the reviewer for their thoughtful review. We particularly appreciate your recognition of the significance of the problem we address and our contribution in highlighting the limitations of current motif-based approaches. We provide detailed answers to the raised questions and concerns in the following.

Is the problem of structural diversity fully resolved? & Significance of experimental results

• We believe that making the community aware of critical limitations of fragment-based methods together with our novel method and evaluation are important contributions and we hope that our work will spark followup research that will eventually resolve the challenges regarding structural diversity.
• Problem and Evaluation: In our work, we propose a new perspective on the evaluation of molecular generative methods. While standard benchmarks rely heavily on metrics like FCD scores, our work demonstrates that this focus is insufficient. A similar observation has also been made in the context of computer vision [1]. Accordingly, we introduce novel evaluation criteria to account for the importance of structural expressivity, which is an overlooked aspect of molecular generation.
• Approach: Our scaffold-motif approach represents a step towards addressing these limitations, even if it does not fully resolve them. The drop in standard metrics like FCD and SA actually highlights an important trade-off: methods optimised for these conventional metrics sacrifice structural expressivity. With MAGNet, we aim to strike a balance that is overall beneficial for molecule generation and, thus, drug discovery.

Impact on separation of structure and features on validity of generated samples, Q: Validity and Connectivity

• We agree that the impact of the changed inductive bias has to be analysed thoroughly.
  • For the effect on sample quality, we specifically dedicated our analysis in Sec.5.3 to the accurate prediction of features both in terms of generation (Fig.4 a,b) and reconstruction (Fig.4c). We clearly demonstrate that the free featurisation as implemented through MAGNet is not a limitation but a feature and beneficial for sample diversity.
  • We want to further support these findings by referring the reviewer to App. D.3, Tab.4 in particular, which complements the benchmark from Sec. 5.2. Unfortunately, we missed to make this reference in the initial submission and have updated the manuscript accordingly, cf. L419. Due to validity constraints, MAGNet achieves 100% validity. Like other methods, we always consider the largest connected component for analyses. For the complete generated output, as shown in Fig.12, the retention percentage for MAGNet is >93%, with 80% of the generated molecules being fully connected.

Writing

• We structured the manuscript to separate the general mathematical framework of our factorisation (Sec. 3) from the model (MAGNet) that matches this factorisation (Sec. 4). Following the reviewer’s feedback, we have clarified the manuscript structure with an introductory note in Section 3 and detailed the network architectures throughout Section 4, making the distinction between theoretical framework and implementation more explicit.
• Many thanks for pointing out the incorrect panel order in Fig.3. We have corrected this in the updated pdf.
• We have updated all references to cite the published versions of the papers and revised the manuscript accordingly.

Q: Generation speed:

• We conducted a quantitative analysis on the generation speed. While the reported results certainly reflect aspects of the methodological design, the specific generation times also heavily depend on the chosen framework and implementation. However, our choice to design MAGNet in a hierarchical OS fashion relies more on the advantages connected to its ability to consider the entire molecular context at all generation steps. Please refer to our response to reviewer hxc4 (W1) for additional details. |Method|s/10 samples| |-|-| |PS-VAE|0.293| |MiCaM|0.583| |MoLeR|2.760| |MAGNet|1.200|

Q: Vocabulary Size:

• Using our fragmentation approach, the vocabulary reduces to 347 distinct scaffolds, which corresponds to 7371 motifs. Note that our fragmentation is more effcient than those used in MoLeR and MiCaM, which amount to roughly 14000 and 15600 motifs, respectively.
• For a fair comparison, we chose the same vocabulary size (350) for all baselines. Inspired by the question of reviewer w2Yn, we also compare MAGNet against a MoLeR model with a vocabulary size of 2k, which we discuss in our general response.

We hope that we have adequately addressed the raised concerns and believe the reviewer's feedback has enhanced the quality of our manuscript. We look forward to further discussion.

[1] Jiralerspong et al., Feature Likelihood Score: Evaluating Generalization of Generative Models Using Samples, 2023

Thank you for the prompt response and engaging in a discussion with us. We believe there has been a fundamental misconception regarding Fig. 1 and apologise that we misunderstood the reviewer's main concern. In the following, we would like to clarify this essential aspect of our work.

Based on my understanding, your method still struggles to generate complex fused ring systems as shown in Figure 1a, correct?

• That is indeed not correct. Quite the opposite, with our approach complex scaffolds such as fused rings become accessible.
• In Fig.1a we only highlight the shortcomings of existing motif-based methods (in particular MoLeR) on the reconstruction of approved drugs with complex scaffolds. This presentation was meant to motivate the necessity for rethinking motif-based molecular generators.
• In App.D.2 (Figure 10), we demonstrate how MAGNet can decode the complex scaffolds shown in Fig.1a.
• We also explicitly evaluate the decoding capabilities of such scaffolds quantitatively in Sec.5.1, Fig.3 in particular. What is referred to as uncommon cyclic scaffolds are exactly those complex scaffolds the reviewer mentions and we improve upon baselines significantly.
• It is important to highlight that our vocabulary is much more expressive than existing ones, containing scaffolds of structures like Pentacene, Triphenylene and Benzo[a]pirene, and generally scaffolds of fused rings with up to 6 rings, as well as bicyclic scaffolds.
• To avoid future misunderstandings and if the reviewers finds this beneficial, we could imagine to swap the presentations of Fig.1a and Fig.10. This change would aid in showing what MAGNet can do in the main part of the manuscript and highlight limitations of existing works only in the appendix.

We sincerely hope that this clears up the misunderstanding regarding MAGNet’s capabilities and our contributions. In light of this clarification, we would appreciate if the reviewer reconsiders their evaluation and look forward to further discussion.

We are grateful for the reviewer's insightful feedback and are pleased with the positive comments on our work. In the following, we would like to clarify the remaining questions and concerns.

Absence of synthesizability considerations

• Thank you for this constructive feedback. We agree that synthesizability warrants discussion and have added a paragraph to the limitations section, cf. L285-288. While our manuscript focuses on standard small molecule datasets, we acknowledge that the absence of synthesizability considerations affects the practical application of fragment-based methods, particularly when moving from computational predictions to experimental validation. We believe that synthesizability is inherently linked to the training data, and that utilising a dataset of multi-component reactions [1] could help address this limitation.

Missing metrics (Novelty, Uniqueness, Validity)

• We completely agree with the reviewer’s feedback and have missed to refer the reader to Appendix D.3, Tab. 4 in particular, which includes the mentioned metrics for Novelty and Uniqueness. Moreover, the number of samples to obtain 10k valid molecules can be deduced from the reported validity and amounts to roughly 12k for DiGress.

Clarification of the benchmarking procedure

• As we outline in L321-322, all models are trained on the ZINC data and we conduct the GuacaMol and MOSES benchmarks on the respective test set of the ZINC data. We updated our manuscript to better reflect this setup, cf. L322. Note that we use the other mentioned datasets as part of the zero-shot experiment (Appendix D.6 and Figure 8) that investigates the ability of MAGNet and the baseline models to represent molecules from different distributions. Across datasets, we show that the inductive bias as set through our fragmentation and scaffold approach leads to more faithful reconstruction of molecules from their latent codes.

Inclusion of methods that specifically optimise for goal-directed generation of molecules

• We appreciate this suggestion. In our manuscript, we focus on the comparison with fragment-based methods to establish a clear methodological context. For this, we delineate between the optimisation scheme (Gradient ascent and MSO) and the underlying generative procedure (MAGNet, MiCaM, MoLeR). Following the reviewer’s suggestion, we checked the setting used in [2]. Comparing our results for the goal-directed benchmark with those reported in [2], MAGNet should perform competitively. However, the goal-directed generation setting in our experiments and the results reported in [2] are not directly comparable due to the varying number of oracle calls. For the final version of the manuscript, we would be happy to include a thorough evaluation of MAGNet following the benchmark setting in [2].

Clarifications

• We updated the caption of Fig.1 and aligned it better with the structure (a,b,c) of the figure as suggested by the reviewer.
• In structuring the manuscript, we intended to distinguish between the general mathematical formulation of our factorisation approach (Section 3) and MAGNet's specific implementation details (Section 4), with Fig. 2 illustrating the key concepts of our factorisation, cf. L130-133. We would appreciate more specific guidance on which aspects are not sufficiently visualised in its current version.
• Following the reviewer’s suggestions we tried to provide more specific information about the main baselines through the manuscript. In particular, we would like to refer the reviewer to L300-307.
• Thank you for highlighting a lack of clarity for the conditioning setting presented in Fig. 5. In our response to Reviewer hxc4, we provide a more comprehensive discussion of the practical relevance and potential applications of this conditioning approach. We furthermore updated Sec.5.4 with these details to better illustrate how MAGNet enables flexible conditioning.

Minor Comments

• We have addressed all minor comments and typos in the updated PDF. Thank you for thoroughly reviewing the manuscript and providing such detailed feedback.

Impact of vocabulary size and experiments analysing the effect

• We thank the reviewer for their suggestion. We believe an analysis of this kind greatly supports the claims and contributions of our work and have included a detailed discussion in our general response to highlight these results for all reviewers.

We appreciate the positive feedback and are excited to move forward with these improvements. We hope that we addressed all outstanding concerns satisfactorily and welcome further discussion.

[1] Graziano et al., Multicomponent reaction-assisted drug discovery: A time-and cost-effective green approach speeding up identification and optimization of anticancer drugs, 2023
[2] Gao et al., Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization, 2022

We thank the reviewer for their detailed feedback and thoughtful insights. We are grateful for the reviewer’s acknowledgement of scaffolds as an original and valuable proposal, as well as their recognition of the technical execution and soundness of our work. In the following, we would like to address the reviewer’s main concern: the categorisation of MAGNet as a one-shot (OS) model.

One-Shot (OS) terminology

• In our manuscript, we aimed to adopt the established terminology to classify different methods. However, we acknowledge that the strict definitions do not perfectly apply to our approach, nor PS-VAE or DiGress. We sincerely appreciate the reviewer highlighting this.
• Fundamentally, we argue that there is a difference between (i) the underlying neural network architecture used to generate aspects of the molecule and (ii) the process of molecule generation (sequential or one–shot). These two are orthogonal concepts in our view.
  • Regarding (i), MAGNet and PS-VAE for example utilise, among others, recurrent and autoregressive architectures such as RNNs to generate specific, independent components of the molecule, e.g. the set of motifs.
  • Regarding (ii), methods like MAGNet and PS-VAE divide the molecular graph into distinct parts, e.g. atom and bond types, which are predicted once and define different characteristics of the molecular graph. In contrast, sequential generation builds molecules step-by-step from parts that share the same characteristics, e.g. motifs or single atoms. This distinction is well reflected in the factorisations of these methods:
    • MAGNET divides $P(G)$ into distinct components scaffolds $S$, their connectivity $A$ and representation $M$, joins $J$, and leaves $L$.
    • Similarly, PS-VAE, factorises $P(G)$ into nodes (atoms and motifs) and bonds. We classify this method as OS for the same reason, even though it decodes e.g. the nodes with an autoregressive module.
    • We also consider DiGress, while iterating over the graph generation in many steps, as an OS model, as it does not condition on a partial molecule over the generation, cf. L114-115.
    • MoLeR, a sequential approach, generates the molecule step-by-step $P(M_i | M_{i+1}, …, M_1, z)$, attaching motif after motif. Each generation step is designed identically and consists of multiple modules, which are repeatedly called over the process of the generation. Importantly, such methods can condition on the intermediate state of a molecule, which OS methods cannot.

• Although we believe that a better terminology would distinguish between i) attaching nodes or fragments to a partial graph and ii) fully generating specific aspects of a molecule, e.g. the set of motifs, we chose to adopt established terminology from [Zhu et al., 2022] to ensure clarity and maintain consistency. However, we agree with the reviewer that this might lead to misunderstandings and misguide the interpretation of the experimental evaluation.
• To avoid misunderstandings, we removed or toned down all claims that rely on MAGNet being an OS model:
  • We updated L185-186 to remove the classification of MAGNet as OS model
  • We updated L412-413, such that MAGNet is described to perform on par with motif-based approaches

• The classification of the methods in Table 1 was intended to help the reader contextualise the results, as the classification is consistent in itself (MAGNet, PS-VAE and DiGress as OS models), and aims to make up for inconsistencies in the literature. We acknowledge that this discussion needs more room in the paper and we describe the categorisation of methods in more detail in App. C of the updated PDF. However, if the reviewer believes that the classification for this experiment can lead to confusion more than to help the reader, we are happy to remove the classification from this table entirely.
• Moreover, we do not consider MAGNet being an OS model to be our main contribution, cf. L74-76 as well as L84-92. To us, leveraging scaffolds instead of motifs as well as our experimental evaluation, which was also highlighted by the reviewer, are the most essential aspects of our work.

We want to thank the reviewer again for raising their concern and encouraging us to discuss the categorisation of methods in more detail. We hope that our updated manuscript as well as our clarifications have addressed the reviewer’s remaining concern satisfactorily.

We truly appreciate your thoughtful review and support for our paper's acceptance at ICLR. Your insights, regarding method classification in particular, were valuable, and we are grateful for your constructive feedback.

Thank you for the constructive feedback on our submission. In our response, we aim to clarify the remaining questions and concerns of the reviewer.

MAGNet is not purely one-shot (W1)

• We agree with the reviewer’s description and classification of our MAGNet being “not purely one-shot” in the classical sense. As prompted also by reviewer Ry7i, we extend the discussion of this aspect in the updated manuscript and would like to refer the reviewer to both the response to reviewer Ry7i and App. C of the updated PDF.

Is there any inherent value that comes from being an OS model? (W1)

• As the reviewer correctly pointed out, sequential models incur additional computational overhead due to the repeated encoding of partial molecules, generally making them slower than models that do not condition on partial molecules. This difference is evident when comparing generation speeds, which we discuss in more detail in our response to reviewer 8uqV.
• However, we believe that models that avoid conditioning on partial molecules have another key advantage: their latent representation must fully capture the essential features of the molecule. In contrast, when conditioning on both the partial molecule and the latent representation, the influence of the latent code is not guaranteed. In OS models like MAGNet, the latent code is central to the generation process and dictates all subsequent steps, ensuring faithfulness to the representation, which we, for example, demonstrate through our displacement analysis presented in Fig.7.
• The hierarchical architecture is essential for MAGNet. Scaffold representations can vary significantly based on their position within the molecule. For example, the Murcko scaffold CC(C)(C)C often appears as CS(=O)(=O)C in a central position, but as CC(F)(F)F in less central contexts. MAGNet requires a complete understanding of the coarse-grained molecular arrangement to assign the correct representations to each scaffold.

Could [this conditioning scenario] be an issue causing the model to add scaffolds that do not fit well with the partial molecule? (W2)

• The hierarchical design of MAGNet, which avoids conditioning on partial molecules, offers greater flexibility for tasks like the one presented in Figure 5, where the model needs to condition on disconnected scaffolds. This type of multi-scaffold conditioning is challenging for sequential models, whereas it aligns naturally with MAGNet's factorised approach. Following the reviewer’s feedback, we have extended the description of this experiment in the updated PDF. In general, we share the concern that within our factorisation it might occur that scaffolds are connected such that they do not match with the conditioning fragment. This is likely linked to the complexity of the conditioning scenario.
• While we have not experienced a mismatch between the partial molecule and generated molecule so far, we recognise that it may arise in other scenarios or datasets. To support the model in such scenarios, we can envision:
  • Refined Latent Space Navigation: Sampling latent codes in a subspace of the latent space that is in accordance with the conditioning fragments.
  • Filtering: Incorporating a post-hoc filtering mechanism to eliminate incoherent samples.
• It would also be possible to incorporate additional architectural changes as listed below, while preserving the reliance on hierarchical decoding with scaffolds as building blocks:
  • Additive Conditioning: Introducing an auxiliary term, like a scaffold embedding, or constraint in the generation process that explicitly accounts for scaffold compatibility with the partial molecule.
  • Hierarchical Conditioning: Augmenting the model’s latent space to encode more detailed inter-scaffold relationships, ensuring compatibility even for disconnected scaffolds.
  • Correction Step: Allowing the model to iterate over its prediction to ensure consistency between the condition and its final prediction.

Nitpicks

• We sincerely thank the reviewer for carefully pointing out the typos in our manuscript. We have corrected these issues and believe that it improves the clarity and presentation of our work.

In conclusion, we appreciate the reviewer’s positive evaluation of our work and hope to have addressed the concerns raised to the reviewer’s satisfaction.