SDDBench: A Benchmark for Synthesizable Drug Design

W1: The analysis and discussion in Section 5, which presents the benchmark, lacks depth.

A1: We already provided molecule property analysis in Table 8 in Appendix.

W2: Lastly, the paper does not provide guidelines on how the synthesizability objective could be balanced with other (potentially opposing) objectives.

A2: Yes, our findings show that molecules generated by pocket2mol have the highest likelihood of being solved in a single synthesis step. Smaller molecules tend to be more easily solved within a shorter number of steps. The average synthetic route length for molecules proposed by Pockert2Mol is 2.93. 14.7% of them proposed Pocket2Mol by are building blocks.

W3: Other metric candidates compared with SA score.

A3: Firstly, the SA score is the most widely used indicator of synthesizability, and it is utilized by the SBDD model and all drug design models for this purpose. Our goal is to replace the SA score, and the experiment clearly demonstrates the adequacy of our approach.

W4: Inference Speed

A4: We have provided time analysis in Line 498-501.

W5: Presentation

A5: With the current 10-page limit, we can only present a limited amount of information. Many concepts are drawn from previous papers and don't fit well into the main text. While we considered placing them in the appendix, this still requires readers to jump back and forth several times.

Q1: Why is the search success rate not a reliable metric (line 252)? This is an interesting point but the paper could discuss this claim more and ideally provide empirical evidence.

A6: We have provided short analysis in Section 2.6. Reviewers are encouraged to refer to [1] for further details.

Q2: The definition of the round-trip score using Tanimoto similarity...

A7: Eq. 1 represents the joint probability: the probability that the drug design model first generates a molecule and the probability that the retrosynthetic planner then generates a synthesis pathway for this molecule. It is unrelated to the round-trip score.

Q3: Have you also experimented with established tools (e.g. AIZynthFinder) or pretrained models?

A8: Since a retrosynthesis model must be trained to generate a feasible synthesis route, other tools rely on the search success rate, which, as we have already explained, is flawed.

Q4: What is the typical run time to compute this score? How does it compare to other synthesizability scores?

A9: We have reported the time in Lines 498-502, utilizing the KV cache, which allows for very fast processing of a single molecule. Its runtime matches the time required for the retrosynthetic planner to predict synthesis routes for a single molecule.

[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning. ICML 2023.

We believe the primary challenge in synthesis is identifying a feasible synthetic route for the molecule. It appears this reviewer may lack sufficient background knowledge on the topic.

W1: Difference from previous works

A1: Previous work [3] has primarily followed a top-down approach to drug design and synthesis planning. In contrast, our method begins with the desired product molecule and uses a retrosynthetic planner to build a synthesis route in a top-down manner. We then employ a forward reaction model to evaluate the feasibility of the route. In this regard, we are indeed the first to adopt this approach. Additionally, when it comes to evaluating the feasibility of the synthetic route, [1,2] were the pioneers in this area. We have directly incorporated the findings from these two ICML-accepted papers into our methodology, which is novel we believe.

SA, etc. score do not ensure that a viable synthetic route for a molecule can be identified, as we have discussed in this paper

Q1: When classifying the test set molecules as “successfully” solved...

A2: This is a challenge that remains unsolved within the community. Previous retrosynthetic planners have relied on search success rate to evaluate retrosynthetic methods, but many of the routes identified are not feasible, meaning the target molecule cannot actually be synthesized. This issue is discussed in [1,2]. In [1], a more robust metric was introduced that matches the predicted starting materials with those in a reaction database. However, this approach has its own limitations, as the reaction database may not cover all potential routes. Nevertheless, compared to search success rate, the matching-based method represents a significant improvement.

Q2: It would be informative for context to include molecule size/number of heavy atoms as another metric.

A3: We agree with the reviewer’s observation. Our findings show that molecules generated by pocket2mol have the highest likelihood of being solved in a single synthesis step. Smaller molecules tend to be more easily solved within a shorter number of steps.

Q3: A specific question about Pocket2Mol is in Table 1, the authors show that the Search Success Rate is highest. What is the route length distribution of these "successful" molecules? Are any directly in the building blocks?

A4: The average length is 2.93. 14.7% of the molecules proposed Pocket2Mol by are building blocks.

Q4: How many molecules out of the 10,000 generated molecules do not have a solved retrosynthesis route according to NeuralSym?

A5: We have already reported the search success rate.

Q5: What are the building blocks used?

A6: They ares to determine the molecules in the leaf nodes of retrosynthetic routes are building blocks or not.

[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning. ICML 2023.
[2] Preference Optimization for Molecule Synthesis with Residual Conditional Energy-based Models. ICML 2024.
[3] Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design. ICLR 2022.
[4] Planning chemical syntheses with deep neural networks and symbolic AI, 2018 Nature.

Thank you to the authors for the revised draft. I have read through the new sections and overall, I think the comparison and justification for the round-trip score has improved a lot. Thank you also for adding information on the average number of atoms and the ratio of starting material in Table 2 (if may help to clarify what this metric means in the caption). It is also great that the authors additionally assess feasibility by a manual SciFinder search as this is what is commonly done by synthetic chemists.

I updated my score to 5 as I still have some concerns over the superiority of round-trip vs. search success rate. I will start with some comments for clarity before going into specific questions.

Clarity

• In lines 364-365: it might be better to express that search success rate does not have an additional check for feasibility (as in the forward model in round-trip). The templates themselves are assessing feasibility. Reactions are not feasible if the templates are imperfect (which I agree they are not perfect). I think this can help convey the authors’ position in the paper that coupling a forward model to the retro predictions is beneficial to getting more true positives.

• Lines 355-369: Are the numbers supposed to sum to 95 since the retrosynthetic planner returned no routes for 5/100 molecules? For the search success numbers, they sum to 95

Questions (all related to section 4.1)

1. Given the limited time, I can appreciate assessing more than 100 molecules is difficult. But N=100 is a relatively small sample size to draw large conclusions

2. Why was the retrosynthetic planner restricted to not exceed the maximum depth of the reference route? Since the authors also state that retrosynthesis is a “one-to-many” task, a longer route does not necessarily mean an infeasible route. Perhaps it is not ideal due to a longer synthesis, but it can still lead to the correct product. How would the metrics look without this restriction?

3. For True/False Negatives, round-trip != 1, what is the distribution of the scores? Do the False Negatives have a distribution of round-trip scores closer to 1? If so, this can give support to the > 0.9 similarity threshold as this.

Or as a question: “For feasible routes that do not have a round-trip score of 1, is the round-trip score generally closer to 1? If so, this supports the use of round-trip as a proxy signal for reaction feasibility” If there is no statistically significant difference in the distribution of round-trip scores for True/False negatives, it suggests that the > 0.9 threshold may not be giving a proxy for the intended feasibility.

1. Clarity

We appreciate the suggestions and will include the sentence: "Templates are not perfect for reaction feasibility checks, so we use a forward model to validate feasibility."

Additionally, we acknowledge an error in our initial explanation. We will revise the number of False Negatives (FN) to 17.

2. Given the limited time, I can appreciate assessing more than 100 molecules is difficult. But N=100 is a relatively small sample size to draw large conclusions

Querying 100 molecules is highly time-consuming, taking us two days to complete. For evaluating the search success rate, the search algorithm in [1] was applied to 189 molecules. Therefore, we use 100 molecules for our evaluation.

3. Why was the retrosynthetic planner restricted to not exceed the maximum depth of the reference route? Since the authors also state that retrosynthesis is a “one-to-many” task, a longer route does not necessarily mean an infeasible route. Perhaps it is not ideal due to a longer synthesis, but it can still lead to the correct product. How would the metrics look without this restriction?

Because we need the reference routes in the test set to check feasibility. Therefore, the predicted route cannot exceed the maximum depth of the reference routes. With this restriction, we can search for routes for 95 molecules. Without this restriction, we can search for routes for at most 5 other molecules, but we think it will not change the results of the other 95 molecules. Because the deeper the route, the lower its cumulative probability will be in the beam search, and the lower it will be in the top-k ranking.

4. For True/False Negatives, round-trip != 1, what is the distribution of the scores? Do the False Negatives have a distribution of round-trip scores closer to 1? If so, this can give support to the > 0.9 similarity threshold as this.

For True Negatives, the mean is 0.6148 (std: 0.1406, max: 0.8387). The distribution across bins is as follows:

0.3–0.4: 1
0.4–0.5: 3
0.5–0.6: 3
0.6–0.7: 6
0.7–0.8: 1
0.8–0.9: 3

For False Negatives, the mean is 0.7058 (std: 0.1027). The distribution across bins is:

0.4–0.5: 1
0.5–0.6: 3
0.6–0.7: 8
0.7–0.8: 6
0.8–0.9: 6.

Notably, the round-trip score of infeasible routes can be 1. We set a round-trip score threshold of 0.9 to ensure a higher likelihood of finding feasible routes among the filtered candidates.

[1] Retro*: Learning Retrosynthetic Planning with Neural Guided A* Search

W1: It seems to me that the "wrong" target is set for that metric

A1: Thank you for your suggestion. Currently, the reliability of our metric relies on the amount of reaction data used by our model. In theory, the more reaction data available, the more dependable the metric becomes. As discussed in the introduction, while certain natural molecules are theoretically synthesizable, feasible synthetic routes may remain undiscovered by humans. Based on this, we define a synthesizable molecule as one with a feasible synthetic route identifiable by a retrosynthetic planner. Feasibility is assessed by comparing the synthetic route predicted by the retrosynthetic planner with a reference route in the reaction database.

Our goal is to predict the synthetic routes of molecules generated by various drug design models directly through the retrosynthetic planner. We then evaluate the reliability of the round-trip score. In the first category of molecules, you observed that 99.5% achieved a round-trip score of 1, indicating that our round-trip score correctly identifies 99.5% of synthesizable molecules. We agree on the importance of evaluating the accuracy of the round-trip score for negative examples (molecules known to be unsynthesizable). However, defining unsynthesizable molecules poses significant challenges. Unsynthesizable could mean that the molecular structure cannot exist at room temperature and pressure, or that a synthesis route cannot be found with current human knowledge. Each definition has its limitations.

However, our primary goal is to encourage drug design models to use our metric to identify molecules for which the retrosynthetic planner can find synthetic routes. Once identified, these molecules will indeed have findable synthesis routes. From this perspective, our metric is accurate and highly reliable. Other metrics, such as the SA score, cannot guarantee the discovery of synthetic pathways, even at high values.

W2: Some claims made in the first paragraph on the introduction should be supported by appropriate citations.

A2: We will cite more papers.

W3: the claim on lines 74-76

A3: We agree with your point and will make the necessary corrections to this statement.

W4: Related Work

A4: We will introduce more.

W5: Guo's work

A5: We have thoroughly reviewed the details of this work. However, in our view, the synthetic metric used in this paper is flawed. The SA score does not ensure that a viable synthetic route for a molecule can be identified, as we have discussed in this paper. AiZynthFinder, on the other hand, assesses the solvability of a molecule's synthesis using the search success rate, which has limitations discussed in [1].

Q6: Why would we expect retrosynthesis planners to be less limited than template-based synthesis planners?

A6: Their approach begins with a building block, predicting the reaction template (a classification problem with fewer than 100 categories) to generate the next intermediate molecule. Then, by iteratively predicting reaction templates through each intermediate molecule, they construct a synthetic pathway to the final molecule. The final molecule is then quantified to generate a reward, and they use reinforcement learning (RL) from scratch to train, with a limit on the number of reaction steps.

We attempted a similar method, expanding the reaction template to thousands of types, but encountered challenges with convergence in the loss function, as RL training can be quite difficult. In contrast, our approach first generates molecules using the drug design model and then employs our proposed synthesis reward withour limit on the reaction template for filtering or post-training optimization—a clearly more efficient method. For reference, consider this ICML talk, https://icml.cc/virtual/2024/invited-talk/35253, which highlights that RL with a strong starting point (post-training) outperforms RL from scratch.

[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning. ICML 2023.

I wish to thank the authors for their response. I have read the revised manuscript in its entirety as well as the ongoing discussions.

I have found the revised manuscript improved and easier to follow. In particular, the aspect of current retrosynthesis-based synthesizability evaluations that the authors propose to improve upon is now more clearly explained. The evaluation of the proposed synthesizability metric presented in Section 4.1 is also much more sound than what was presented in the first version of the manuscript.

However, some of my concerns remain incompletely addressed.

Evaluation of the presented metric

While improved in the revised manuscript, the evaluation of the proposed metric remains in my opinion insufficient. This paper proposes a new metric to assess synthesisability of molecules generated in silico. A thorough evaluation of this metric compared to existing work thus represents the central and most important validation necessary to support the claims made in this work.

First, on line 314, a split comprising less than 0.1% of the data for the validation and test sets (only 100 samples each) seems insufficient and the authors do not motivate this choice. The authors also report using their own domain knowledge to validate some predicted routes (lines 342-343) but do not report these routes (e.g. in Appendix) for the reader and the community to validate the correctness of this assessment.

In general, the evaluation of the proposed metric in Section 4.1 falls short of appropriate results reporting. For exemple, the authors mention that the SA-score of the final molecules for both the feasible and infeasible routes are similar, but do not report that data. This can easily be presented by plotting the distribution of SA-score for both of these groups in a graph. The SC score is also easy to report and would give additional context. Moreover, given that only 100 routes are used for testing, I would expect a substantial number of these routes to be presented in appendix (give multiple exemples of feasible and infeasible routes, identify which ones are correctly classified by the round-trip score, which ones are missed, discuss hypothesis for these failing modes, etc.).

Finally, a remaining question: the scores derived from the proposed Round-Trip score for various SBDD models presented in Table 2 seem highly correlated to the "Ration of Starting Material". Have the authors identified exemples that would favour the round-trip score compared to this much simpler metric?

Incomplete review of the literature

Despite being raised by several reviewers, the authors do not appropriately cover synthesis-space constrained models in their literature review (see my comment and link to exemples of such works [1,2,3,4,5,6] in my initial review). Instead of adequately covering this subfield in the literature review and discussing the advantages and limitations of these methods compared to the proposed approach, the authors have added citations to some of these papers here and there in support of different claims.

Finally, in some sections, the Related Work reads more like an enumeration meant to defend against "missing citations" than an honest effort at guiding the reader through the relevant literature for this work. For exemple, the paragraph on SBDD (lines 449-454) mentions that such models can be split into diffusion models and other types of models and proceeds to enumerate a number of papers without guiding the reader through the timeline over which these works were developed and contrasting the advantages and limitations of both methods. In the paragraph on Retrosynthesis Models, the authors describe the field as comprising three main categories but do not bother categorising for the reader the 30+ citations that are listed right above.

Minor points

• \citet instead of \citep on line 59
• the sentence from lines 68 to 71 is too long and hard to follow
• Figure 2 is helpful but could be improved. First, it should appear later in the paper (it appears more than one page before being mentioned in the text, leading to confusion). Second, it could be made clearer by further annotations highlighting the fact that both leaf nodes are valid starting materials but the issue is that the intermediate reaction of one of the routes in wrong.

---

For the reasons listed above, I maintain my score of 5. I believe that the proposed idea has merits and can be of interest to the community, but the current evaluation of the proposed metric remains insufficient to withhold scrutiny, and important elements of the relevant literature is not appropriately covered. In my opinion, this work in its current state does not meet the standards of the conference.

W1: Citation

A1: We will cite more papers if this reviewer can give some references.

W2: Claim

A2: Previous work [3] has primarily followed a top-down approach to drug design and synthesis planning. In contrast, our method begins with the desired product molecule and uses a retrosynthetic planner to build a synthesis route in a top-down manner. We then employ a forward reaction model to evaluate the feasibility of the route. In this regard, we are indeed the first to adopt this approach. Additionally, when it comes to evaluating the feasibility of the synthetic route, [1,2] were the pioneers in this area. We have directly incorporated the findings from these two ICML-accepted papers into our methodology, which is novel we believe.

W3: Guo's work

A3: We have thoroughly reviewed the details of this work. However, in our view, the synthetic metric used in this paper is flawed. The SA score does not ensure that a viable synthetic route for a molecule can be identified, as we have discussed in this paper. AiZynthFinder, on the other hand, assesses the solvability of a molecule's synthesis using the search success rate, which has limitations discussed in [1].

W4: the use of retrosynthesis tools in previous works.

A4: We want to emphasize that we introduce a new data-driven synthetic metric, whereas previous studies continue to rely on SA Score and SCScore. The limitations of these existing metrics are discussed in detail in this paper.

W5: the added value of round trip score in the molecule-wise setting over just reporting retrosynthesis success rates

A5: The limitations of Search Success Rate are discussed extensively in Section 2.6, and [1] devotes significant attention to criticizing this metric as flawed. While Round-trip Score evaluates individual points, Search Success Rate is batch-oriented. Furthermore, without reference routes, Round-trip Score serves as a more rigorous and effective metric than Search Success Rate.

W6: Limitation

A6: The primary limitation is the insufficient reaction data to cover all possible molecules.

W7: The authors mention provided code in Section A.1

A7: We are referring to the code for the drug design model, which is available in their paper.

W8: Data Leakage

A8: Please give the detailed dataset you mention so that we can answer your question

W9: In Table 1, the metrics of the training data and test set could be included.

A9: We evaluate the molecules generated by the SBDD model on the test set targets. Any issues that arise are attributable to the SBDD model itself. Could you provide the ChEMBL molecule resources for evaluation?

Q10: Figure 5

A10: The retrosynthesis model is Neuralsym. The forward reaction model is Transformer Decocder.

Q11: Figure 6

A11: Figure 6 highlights that the SA score cannot effectively distinguish between successful and failed cases, resulting in an overlap between them.

Q12: There is no background information on any of the other benchmarking efforts in drug discovery

A12: Please refer to Line 475-482.

Q13: There are no citations for models in the “two-stage learning paradigm”.

A13: Semi-template-based method employ two-stage learning paradigm........, which we have discussed in Line 454-457.

[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning. ICML 2023.
[2] Preference Optimization for Molecule Synthesis with Residual Conditional Energy-based Models. ICML 2024.
[3] Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design. ICLR 2022.

We have rewritten this paper.

We have provided a comparison between the round-trip score and the search success rate in Section 4.1. Our analysis concludes that the round-trip score performs significantly better.

Please check it. Thanks!