Thank you for your valuable review. We have addressed your concerns as outlined below:

Regarding synthesizing issue

Thank you for bringing up our discussion on the challenges of synthesizing molecules generated by current SBDD models.

As a benchmark paper, we are not able to provide a solution to improve the synthesizability of current SBDD models. However, from how our evaluation is done, we have indeed proposed a systematic approach to address this issue. Our evaluation metrics focus not on directly using the generated molecules in wet-lab experiments but on their potential utility within the drug discovery pipeline. Specifically, we evaluate whether the generated molecules can be easily modified into synthesizable and effective compounds or serve as references for virtual screening to identify existing molecules that are effective against the target.

In our approach, synthesizability of the generated molecules is not a primary concern because they are not restricted to be directly used in wet-lab experiments. Instead, the focus is on their ability to inspire the discovery of practical compounds. This perspective aligns with several successful virtual screening-based studies that have leveraged molecules generated by SBDD models. For instance, studies such as TamGen[1] and Pocket Crafter[2] have effectively used generated molecules as references for virtual screening, leading to the identification of viable and effective compounds.

Thus, our evaluation framework prioritizes the broader utility of the generated molecules in the drug discovery pipeline rather than their immediate synthesizability.

[1] Wu, Kehan, et al. "TamGen: drug design with target-aware molecule generation through a chemical language model." Nature Communications 15.1 (2024): 9360.

[2] Shen, Lingling, et al. "Pocket Crafter: a 3D generative modeling based workflow for the rapid generation of hit molecules in drug discovery." Journal of Cheminformatics 16.1 (2024): 33.

Regarding Whether Virtual Screening-Based Metrics Can Be Misleading

We acknowledge that no evaluation metric can fully replace wet-lab experiments, which remain the definitive test of a molecule’s usefulness. All metrics have inherent limitations and the potential to be misleading. However, our goal is to develop evaluation metrics that are less prone to bias and more closely aligned with the actual success rates of wet-lab experiments. Importantly, the virtual screening metrics we propose are evaluated using compounds validated in real wet-lab settings, ensuring they simulate real-world scenarios rather than relying solely on theoretical estimations. For this reason, we believe our proposed metrics are inherently less biased or misleading.

Furthermore, we fully agree that relying on a single metric can lead to misleading conclusions. For example, prior studies and peer-review processes have often overemphasized the importance of achieving lower (better) docking scores. This is precisely why we propose our evaluation metrics, which avoid exclusive reliance on theoretical estimation metrics based on specific factors. Instead, as detailed in our paper, we evaluate the effectiveness of SBDD models from three distinct levels. We introduce the delta score and DrugCLIP score to complement the conventional docking score for theoretical estimation-based metrics, and we also use similarity to known actives as another evaluation level. By considering these metrics together, we believe the risk of misleading conclusions is minimized. And for the virtual screening-based metics itself, we employ various methods, including 2D fingerprints, 3D fingerprints, and different deep learning-based molecular encoders. We believe that combining these approaches helps mitigate the risk of drawing misleading conclusions.

Overall, while we acknowledge that no metric can be perfect, we are confident that our proposed evaluation framework complements existing evaluation processes and contributes to the better development of SBDD models.

Dear reviewer,

Thank you once again for your thoughtful and constructive feedback. During the rebuttal process, we have incorporated additional real-world case studies that demonstrate how using machine-generated molecules as reference ligands for virtual screening can successfully lead to the identification of hit molecules. This approach also provides a systematic solution to the previously identified challenge that molecules generated by SBDD models are often difficult to synthesize.

Furthermore, we have clarified the nature of our work as a model-level benchmark, emphasizing the extensive efforts we have made to ensure the benchmark minimizes the potential for misleading conclusions. We also provided detailed explanations addressing concerns about the risk of overfitting.

We sincerely hope that our responses have satisfactorily addressed your questions and concerns. If you have any remaining reservations or require further clarification, please let us know. We are more than happy to engage in additional discussions to further refine and strengthen the contribution of our work. Thank you again for your invaluable input.

Dear reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper and rebuttals. We apologize for any lack of clarity in our explanations and would like to provide additional details to address any remaining concerns or questions you may have.

First, We sincerely appreciate your suggestion that "relying solely on previously published studies raises concerns about the transferability of your work." To clarify, the previously published studies we reference are intended to demonstrate real-world cases where using generated molecules as references for virtual screening has successfully led to the discovery of hit molecules. Regarding whether the evaluated models in our paper could lead to the discovery of active molecules in future applications, we believe our evaluation framework and the cases we cited demonstrate this potential. However, as a machine learning benchmark paper, it is beyond our scope to do actual wet-lab experiments to show that using molecules generated by these models for virtual screening results in finding real active molecules.

We would like to emphasize that our paper serves as a model-level benchmark for deep learning-based generative models. The scope of such work is to test the performance of the models on a fixed test set. Regarding the reviewer’s point about transferability, we understand this refers to the generalization ability of machine learning models. We provide explanations from both the computational perspective and the real-world wet-lab validation perspective.

Computational perspective

From a computational perspective, in our work, this generalization ability is both tested and ensured with the following methods:

1. The training set does not include any proteins that are structurally similar to those in the virtual screening test set. We constructed our dataset starting from PDBbind and performed a strict split. We remove pockets in the training set that are structurally similar to those in the test set. Structural similarity was measured using the pocket alignment method FLAPP [1]. We adopted two different FLAPP score thresholds, 0.6 and 0.9, for the splitting process to create two datasets for comparison, aiming to explore the impact of homology on testing performance. Details about this process are provided in Section 3.3 of the paper.

Thus, the targets in the test set are unseen new proteins for the model. Our metric is to measure the model’s performance in virtual screening for these new targets. Therefore, if a model performs well on our virtual screening metric, it already demonstrates the model good at conducting virtual screening on previously unseen targets.

2. The test set we used includes over 100 targets distributed across 8 categories, including Kinase, Protease, GPCR, Ion Channel, Nuclear Receptor, Cytochrome P450, Other Enzymes, and Miscellaneous [2]. The diversity of actives calculated by (1- RDK fingerprint similarity) is shown below. We compare the active sets for targets, and the molecules generated by MolCRAFT and TargetDiff for targets. The mean values are shown here:

Results show that the diversity between the active molecules actually has a better diversity than the molecules generated by models. We also provide several distribution plots to demonstrate such diversity:

• logp: https://anonymous.4open.science/r/tmp-23B1/logp.jpeg
• molecule weight: https://anonymous.4open.science/r/tmp-23B1/mw.jpeg
• hydrogen bond acceptors: https://anonymous.4open.science/r/tmp-23B1/hba.jpeg
• hydrogen bond donors: https://anonymous.4open.science/r/tmp-23B1/hbd.jpeg

In conclusion, the diversity of target types and the active molecules in the test set provides a reasonable level of confidence in the generalizability of our evaluation.

Moreover, this is the standard procedure for evaluating machine learning models across all domains. We understand the reviewer’s concern that a model performing well on our test set might still fail when applied to a new target. However, this limitation is inherent to all evaluation metrics and cannot be entirely avoided. As we have stated, there are no metrics or evaluation methods that can guarantee that a model that performs well on a fixed test set will always perform well on new data. Even the actual wet lab experiment cannot guarantee it. What we can do is to design a good test set and good split to fairly evaluate the generalization ability of models, and it has been done in our work.

Thank you for your thorough responses to my review comments and for addressing the points I raised. I appreciate the detailed clarifications and the effort to incorporate additional points into your framework. After reviewing your rebuttal and revisions, I would like to provide some feedback:

1. For real-world studies, relying solely on previously published studies raises concerns about the transferability of your work.
2. I suggest including a supplementary approach or metric to evaluate framework performance.

Thank you again for your thoughtful responses and for addressing the original feedback. The score remains unchanged.

Best,

real-world wet-lab validation perspective

The active and inactive molecules in our evaluation framework have been real-world wet-lab validated [2,3], ensuring that our evaluation metrics—such as those assessing similarity to known actives and virtual screening capabilities—directly simulate real-world scenarios. This allows us to provide a robust and meaningful benchmark for assessing machine learning models in a way that is both practical and relevant to real-world applications.

We can also provide more evidence on the reliability of using virtual screening-based metrics. Several studies have demonstrated an alignment between virtual screening metrics and real-world wet-lab performance, even for completely new targets. For instance, [4] and [5] proposed virtual screening methods that achieved high scores on DUD-E and LIT-PCBA benchmarks while in wet-lab experiments, [4] successfully identified non-covalent inhibitors for the challenging protein target GPX4, which has a flat surface, with an IC50 of 4.17 μM. Meanwhile, [5] identified 12 diverse molecules with significant affinities for NET, a newly solved protein structure. It achieved a hit rate of 15% in the wet-lab experiments which matched the computational metric estimation.

We hope this clarifies our intentions and addresses concerns regarding the transferability and scope of our work. If there are additional steps we should take to further demonstrate the validity of our benchmark, we would greatly appreciate your guidance. Would the reviewer kindly specify the case studies you would like us to conduct? If there are specific requirements or details for case studies you would like us to address, we would greatly appreciate and happy to provide additional analyses wherever feasible.

[1] Sankar, Santhosh, Naren Chandran Sakthivel, and Nagasuma Chandra. "Fast local alignment of protein pockets (FLAPP): a system-compiled program for large-scale binding site alignment." Journal of Chemical Information and Modeling 62.19 (2022): 4810-4819.

[2] Mysinger, Michael M., et al. "Directory of useful decoys, enhanced (DUD-E): better ligands and decoys for better benchmarking." Journal of medicinal chemistry 55.14 (2012): 6582-6594.

[3] Tran-Nguyen, Viet-Khoa, Célien Jacquemard, and Didier Rognan. "LIT-PCBA: an unbiased data set for machine learning and virtual screening." Journal of chemical information and modeling 60.9 (2020): 4263-4273.

[4] Wang, Zhen, et al. "Enhancing Challenging Target Screening via Multimodal Protein-Ligand Contrastive Learning." bioRxiv (2024): 2024-08.

[5] Jia, Yinjun, et al. "Deep contrastive learning enables genome-wide virtual screening." bioRxiv (2024): 2024-09.

Regarding the evaluation of the whole framework

Regarding the suggestion for a metric to evaluate framework performance, we are unsure of its intended meaning.

If the suggestion implies that we need a metric to assess how effective our evaluation framework is, we believe this is not feasible. As a model-level benchmark, our work focuses on providing insight into which model performs better according to our evaluation metrics. However, the “ground truth” of which model is definitively better is inherently unknown, making such an evaluation impossible.

Alternatively, if the suggestion is to provide a metric that summarizes the overall performance of models across different metrics within our framework, we can address this. For example, we could present a sum of ranks for each model across all metrics or calculate a success rate based on thresholds for various metrics.

Would the reviewer kindly clarify which of our interpretations is correct? We would be happy to provide further explanations.

Thank you again for the valuable feedback. We would be delighted to further discuss any remaining concerns or questions the reviewer may have.

Dear Reviewer,

Thank you once again for your thoughtful review and feedback. We would like to take this opportunity to follow up on our previous response to further clarify and reinforce our position.

Firstly, we would like to reaffirm that our framework is not subject to issues of transferability or generalization, as it is specifically designed as a benchmark paper. Our evaluation operates at the model level, rather than assessing the potential of individual molecules for specific protein targets. The targets we selected serve as standardized "exam questions" to which all models must respond, ensuring fairness and validity in our comparisons. Without established knowledge, it is impossible to evaluate newly developed models effectively.

Furthermore, in the context of a benchmark paper, we have made considerable efforts to implement a rigorously designed data split and simulations that closely mimic real-world scenarios. The active molecules used in our study have been validated in wet-lab experiments, as previously detailed.

As demonstrated in our work and highlighted by another reviewer, numerous studies and case reports have confirmed that utilizing generated molecules in virtual screening is a reliable and feasible strategy for drug discovery [1, 2, 3, 4, 5]. Based on this evidence, we firmly believe that there are no transferability concerns with our evaluation framework.

To further illustrate the relevance and applicability of our evaluation metrics, we conducted a simulation study on the relatively unexplored target, Ras-related protein Rab-2A (UniProt ID: P61019), generating new molecules using the complex structure with PDB ID 1Z0A. In this case study, one of the models evaluated in our framework, MolCRAFT, successfully generated a molecule that retrieved the only ChEMBL-listed active compound from the ChemDiv molecule library, which contains 1.64 million compounds. When the generated molecule was used as a reference for virtual screening, the actual active compound ranked among the top 0.15% of all screened molecules, as determined by similarity using Morgan fingerprints. This result highlights the model’s ability to generate biologically relevant molecules, even for targets that have not been extensively explored in prior wet-lab studies. The visualization for the ChemDiv library, the active molecule, and the generated molecule can be found here: https://anonymous.4open.science/r/tmp-23B1/visualization.jpg.

Interestingly, when evaluated using docking scores, the generated molecule achieved a moderate-to-low docking score of -6.0. This underscores the comprehensive nature of our evaluation framework, which assesses a model’s performance across multiple facets of the drug discovery pipeline, rather than relying solely on docking performance. This broader approach ensures that generative models are thoroughly evaluated in scenarios that are more directly related to their practical utility in real-world drug discovery applications.

In summary, the methods and metrics used in our evaluation framework are well-supported by numerous previous studies and can be further validated through research on targets that lack prior experimental studies. As a benchmark, the validity and transferability of this work are ensured by our rigorous dataset splitting process, which enables reliable model evaluation, as well as the use of wet-lab-validated active compounds as labels in our assessment. We hope these explanations address any concerns you may have regarding the robustness of our benchmark. Thank you again for your time and effort in reviewing our paper!

[1] Wu, Kehan, et al. "TamGen: drug design with target-aware molecule generation through a chemical language model." Nature Communications 15.1 (2024): 9360.

[2] Shen, Lingling, et al. "Pocket Crafter: a 3D generative modeling-based workflow for the rapid generation of hit molecules in drug discovery." Journal of Cheminformatics 16.1 (2024): 33.

[3] Ding, Jian, et al. "Discovery and structure-based design of inhibitors of the WD repeat-containing protein 5 (WDR5)–MYC interaction." Journal of Medicinal Chemistry 66.12 (2023): 8310-8323.

[4] Putin, Evgeny, et al. "Adversarial threshold neural computer for molecular de novo design." Molecular Pharmaceutics 15.10 (2018): 4386-4397.

[5] Bo, Weichen, et al. "Local scaffold diversity-contributed generator for discovering potential NLRP3 inhibitors." Journal of Chemical Information and Modeling 64.3 (2024): 737-748.

Thank you for taking the time to read our manuscript and for providing detailed review comments. Thank you for acknowledging our perspective on the need for improved metrics in SBDD. We have carefully considered your perspectives on the SBDD metrics. To provide you with a more comprehensive understanding of our work and minimize potential misunderstandings, we will first introduce the motivation behind our proposed set of metrics. Following that, we will respond in detail to your discussion and viewpoints on each type of metric. Finally, we will offer some reflections on the questions you raised. We hope this information helps to further clarify our work, we look forward to engaging in further discussions with you.

1.Further explanation of our motivation:

First and foremost, we do not intend for the proposed metrics to entirely replace existing ones, nor do we expect that introducing and optimizing a single perfect metric would solve the challenges in the SBDD field. Instead, our goal is to provide an additional evaluation perspective, which stems from our consideration of how SBDD models can be practically applied. We want to

Currently, the primary method for deploying SBDD models is synthesizing the generated molecules for wet-lab experiments. However, the synthesis of these molecules and the wet-lab experiments themselves are very costly. And those generated molecules that cannot be synthesized are treated as useless. We believe that beyond using SBDD-generated molecules directly as drug candidates, they can be used in more practical ways. One such application is using the generated small molecules as templates for ligand-based virtual screening, where similar compounds from chemical libraries are identified and tested as drug candidates in wet-lab experiments. Additionally, human experts can leverage their domain knowledge to modify and improve upon the generated molecules. Although some generated molecules cannot be synthesized, they can still be effective in drug discovery pipeline. We believe these deployment strategies for SBDD models have its value in pharmaceutical contexts, providing information on potentially useful compounds or functional groups. Furthermore, given the economic costs and challenges associated with synthesis, these approaches may be more practical and feasible for real-world implementation.

We hope to provide a more comprehensive evaluation of whether current SBDD models are useful and effective in the whole drug discovery pipeline. However, current evaluation metrics primarily focus on assessing the quality of generated molecules as a drug candidate directly, without considering other deployment strategies for SBDD models. We aim to fill this blank by proposing metrics that can evaluate model performance in these alternative, yet impactful, applications.

There always exists generated molecules that cannot be synthesized for experimentation, making them useless in that context. However, these molecules can still be useful as they may contain good fragments or substructures, and our metrics can assess this usefulness. Therefore, our evaluation provides a broader perspective for assessment.

Fortunately, we have observed that some recent wet-lab studies have already begun deploying SBDD models in these ways, reinforcing the importance of developing evaluation metrics for these applications. For example, the TamGen team [1] sought commercially available compounds similar to those generated by their model from a 446k compound library. They successfully identified 159 analogs, and five of these analogs demonstrated significant inhibitory effects in a wet-lab ClpP1P2 peptidase activity assay. Similarly, the Pocket Crafter project [2] employed SAR enrichment analysis to extract valuable scaffolds from the generated molecules, using these scaffolds to conduct ligand-similarity-based virtual screening. They done wet-lab experiment on the 2029 compounds searched from Novartis internal archived diverse library and led to the finding of WM-662 and Compound 1 [3], the known WBM pocket binding molecules.

Similarity Metrics (including Virtual Screening)

2.5 Regarding the similarity metrics

Firstly, we would like to clarify that our proposed metrics are designed to evaluate the effectiveness of current generative models. In other words, our framework is intended for evaluation at the model level, rather than the individual molecule level, and is not meant to serve as optimization objectives during model training or sampling. Optimizing directly for these metrics would not be meaningful.

In machine learning, if there is an explicitly defined objective or reward function, it is almost certain that models, with proper optimization methods, can generate molecules that satisfy those specific properties or objectives. However, our focus is on evaluating models, not directly optimizing for these metrics. Indeed, not all evaluation metrics in machine learning are suitable to be used as optimization targets.

Additionally, our goal is not to re-discover known molecules. As a benchmark, our goal is to test whether the models have the ability to generate active molecules. The ability to generate molecules similar to known actives is intended as a simulated test for the generative models. During sampling, the model does not have access to the information of known actives. If, under these conditions, the model can generate molecules similar to active compounds or produce references that can be used for virtual screening to identify actives, it demonstrates the model’s potential to generalize to new targets that have not been thoroughly studied or have no known actives. For new targets without any known actives, such a model could still generate molecules that similary to their potential actives and help identify potentially active compounds.

For the dataset we used for testing, we are not limited to using the reference ligand as the active molecule. Instead, for each target, there is an average of 224 active molecules, with an average pairwise similarity of approximately 0.2. We believe this constitutes a diverse and abundant enough library to encompass different binding molecules. On the other hand, if a model lacks the ability to generate molecules similar to or capable of retrieving at least one of these actives, then its generative power should be called into question.

In conclusion, our framework is not about forcing the model to re-discover known molecules. Instead, it is about evaluating whether the model has the capability to identify active molecules and provide valuable starting points for drug discovery.

2.6 Regarding similarity to FDA approved ligand

We believe that the 2582 molecules currently approved by the FDA as small-molecule drugs possess certain favorable drug-like chemical properties, such as stability, activity in the human body, toxicity profile, absorption, and proper metabolism. These properties characterize a distribution within a chemical space, and whether the molecules fall into that chemical space can characterize the possibility a new molecule can also be a drug. We agree that drugs are specific and drugs targeting 2 different proteins can be vastly different in structure, but here the similarity to FDA drugs is not a protein-specific metric.

However, we also acknowledge that similarity to known FDA-approved drugs can be a too strict requirement. It is more from a point that to predict whether the generated molecule can ultimately be a real drug, which may not be the scope of SBDD models. This is not the core of our evaluation metrics, and we are also open to moving it to the appendix section if needed.

2.7 Regarding reviewer's "similarity to known molecules is not necessarily the best metric and can be hacked"

We agree that similarity to known molecules is not necessarily the best metric. However, we are uncertain why it would be considered easy to hack. It is important to emphasize that we are not solely relying on similarity to known actives. We also provide virtual screening metrics, which evaluate whether the generated molecules are more similar to active compounds than inactive ones.

Regarding concerns about hacking these evaluation metrics during training or sampling, we believe this is unlikely. Following the evaluation process, there should be no information leakage about the known actives, meaning the similarities or virtual screening abilities cannot be precomputed or exploited.

First of all, thank you for the detailed replies and the effort put into the rebuttal. Some points have been clarified. However, I have follow-up questions and concerns that ultimately are centred around the novelty of the proposed metrics and use cases. There have been many papers in cheminformatics journals that have addressed and commented on certain points raised by the authors. As a result, I believe many limitations are a result of the optimization objectives used at machine learning conferences. I will follow the same points to reply.

1. Further explanation of our motivation

Thank you for clarifying the intended use case of the proposed metrics and I apologize if I misinterpreted. The response from the authors in this section is centred on the proposition that generated molecules can be useful even if not directly used. The authors cite TamGen and Pocket Crafter which either use information from the generated molecules or look for similar analogues in commercial libraries for experimental validation. I agree that this is a valid use case but this has been done for decades. Before TamGen and Pocket Crafter and before machine learning was popular in drug discovery, cheminformatics workflows already use similarity based on reference molecules to find other hits. In the context of generative models, this has been done in these papers [1, 2] for example. In the CACHE challenge, this is another example of using a generative model's output to find analogues [3]. This is also very similar to transfer learning to use known actives to generate similar analogues (in some sense similar to analogue searching). This review paper detailed examples of such papers that have showed experimental validation [4]. I appreciate the authors' motivation in proposing useful metrics to assess generative models but the idea of analogue searching is something that has existed for a long time.

2. Docking/synthesizability: Regarding the metrics optimization problem

In the new optimization-based method the authors performed, they state that the Vina docking scores improved but Glide scores worsened. This is not entirely surprising if Vina and Glide model different things and Vina is often much more crude of an affinity estimator. If the optimization were performed with Glide instead, then the Glide scores would not worsen. This is a limitation of the docking algorithm.

The authors next state that going from Vina to Glide, MMGBSA, FEP still does not guarantee better molecules. Models that get better scores than the reference ligand are not necessarily successful molecules. I agree with these statements but this is exactly a limitation of the predictive ability. I appreciate this is where the authors are trying to propose extra metrics for, but in practice, these binding affinity estimates are rarely used as is, as is done in most machine learning papers. Here is an example of a paper showing that many docking algorithms are not even correlated with binding affinity [5]. Here is an example of a paper where a better docking algorithm can give better FEP estimations [6]. My intended message here is that the problem with the metrics used is that they are not validated beforehand. The virtual screening metric proposed by the authors can be used to retrospectively validate models but what about for any arbitrary protein? In the end, I believe the molecules generated from SBDD models will need to be tested with FEP or something similar. Regarding models not generating molecules similar to known actives, this is handled by validating that the optimisation objective is modelling the correct dynamics. I refer again to the review paper for all the experimentally validated SBDD methods [4]. Here is also an example where this is explicitly showed [5]. Therefore, my concern is that the proposed metrics do not remove the need for validating optimisation objectives for prospective discovery. As a result, I think there should be more encouragement on validating the optimisation objectives instead. Can the authors share their thoughts about this? I am happy to continue the discussion.

[1] https://pubs.acs.org/doi/10.1021/acs.molpharmaceut.7b01137

[2] https://pubs.acs.org/doi/full/10.1021/acs.jcim.3c01818

[3] https://cache-challenge.org/challenges/app/61f7e09526539

[4] https://www.nature.com/articles/s42256-024-00843-5

[5] https://jcheminf.biomedcentral.com/articles/10.1186/s13321-021-00563-7

[6] https://www.nature.com/articles/s42004-023-00859-9

We thank the reviewers for their thoughtful feedback. The reviewer recognized that we are addressing a crucial issue in SBDD, which is very encouraging for us. We address reviewer comments below and will incorporate all feedback.

Q: More non-deep-learning methods should be tested.

We appreciate the reviewer’s valuable feedback.** We have added an evaluation of RGA[1], which is an SBDD method based on reinforcement learning and genetic algorithms.**

Table 1: results for RGA for the first round and the final round.

As shown in the table, it is obvious that after several rounds of optimization， RGA model can generate molecules with better Vina docking scores. However, the virtual screening metrics are not improved. And the Delta Score is still quite limited.

[1] Fu, Tianfan, et al. "Reinforced genetic algorithm for structure-based drug design." Advances in Neural Information Processing Systems 35 (2022): 12325-12338.

Q: A detailed explaination of Table 1.

Thank you for your comments. We have reorganized and provided more detailed explanations for Table 1 in hopes of accurately conveying our ideas. The revised version is as follows:

Caption: Results from virtual screening using docking software, and using real ligands as templates for similarity searches. Similarity is determined through various molecular fingerprints and deep learning encoders. BEDROC and EF@1 are metrics of vitrual screening. The results demonstrate that using real ligands as templates for virtual screening yields better outcomes compared to virtual screening with the docking software Vina.

If you feel this requires further clarification, please let us know. We would be more than willing to provide additional details.

Q: A explaination of FLAPP

Thank you for your suggestions. We have further elaborated on flapp and integrated it into the paper. If there are any other terms you believe require further explanation, please feel free to let us know. Here is the description of FLAPP:

FLAPP (Fast Local Alignment of Protein Pockets) [1] is a tool used to estimate the structural similarity (alignment rate) between two pockets. We calculated all the FLAPP scores between the training set and test set pockets and removed all pockets from the training set with a FLAPP score greater than 0.6 or 0.9 relative to any test set pocket. This step was taken to assess the SBDD model’s ability to generalize across structurally diverse pockets.

[1] Sankar, Santhosh, Naren Chandran Sakthivel, and Nagasuma Chandra. "Fast local alignment of protein pockets (FLAPP): a system-compiled program for large-scale binding site alignment." Journal of Chemical Information and Modeling 62.19 (2022): 4810-4819.

3. Regarding the scenario where docking scores and similarity metrics do not align.

We believe this also addresses your question regarding how our method would evaluate a generated molecule with a structure vastly different from known actives but demonstrating strong binding affinity in docking simulations. Our similarity-based and virtual screening-based metrics are not intended for molecule-level evaluation but rather serve as benchmarks for model-level assessment. In other words, if an individual molecule receives a low similarity score to all known active compounds, we are not suggesting that the molecule should be dismissed outright. However, if a model predominantly generates such molecules, its overall capability would be called into question.

Come back to the question. If a molecule exhibits a significantly different structure from known actives yet demonstrates strong binding affinity in docking simulations, our framework allows for additional evaluation using the Delta Score and DrugCLIP score. We believe these metrics provide a more robust and reliable assessment of binding affinity at the molecule level compared to conventional docking scores.

4. Regarding the question of why we are so confident that this is a practical alternative to wet-lab experiments

Similar to the previous response, we want to clarify that we did not want to claiming our metrics can replace wet-lab experiments. We apologize for any lack of clarity and have updated the paper to revise the sentence that may have caused this confusion.

Our claim is that our metrics provide a better simulation of real-world experiments. For example, in the virtual screening metrics, all active and inactive molecules used in benchmarks like DUD-E and LIT-PCBA are wet-lab validated, meaning that better virtual screening performance directly correlates with higher wet-lab success rates. In contrast, traditional metrics like the Vina docking score are purely theoretical estimations based on specific formulas and are not directly indicative of wet-lab outcomes.

Fortunately, some approaches using SBDD models for virtual screening have already been validated by wet-lab experiments. For example, the TamGen team [1] sought commercially available compounds similar to those generated by their model from a 446k compound library. They successfully identified 159 analogs, and five of these analogs demonstrated significant inhibitory effects in a wet-lab ClpP1P2 peptidase activity assay. Similarly, the Pocket Crafter project [2] employed SAR enrichment analysis to extract valuable scaffolds from the generated molecules, using these scaffolds to conduct ligand-similarity-based virtual screening. They done wet-lab experiment on the 2029 compounds searched from Novartis internal archived diverse library and led to the finding of WM-662 and Compound 1 [3], the known WBM pocket binding molecules.We hope that our metrics can provide a broader perspective for evaluating the usefulness of SBDD models.

[1] Wu, Kehan, et al. "TamGen: drug design with target-aware molecule generation through a chemical language model." Nature Communications 15.1 (2024): 9360.

[2] Shen, Lingling, et al. "Pocket Crafter: a 3D generative modeling based workflow for the rapid generation of hit molecules in drug discovery." Journal of Cheminformatics 16.1 (2024): 33.

[3] Ding, Jian, et al. "Discovery and structure-based design of inhibitors of the WD repeat-containing protein 5 (WDR5)–MYC interaction." Journal of Medicinal Chemistry 66.12 (2023): 8310-8323.

Thank you for taking the time to read our paper and for your thoughtful review. We greatly appreciate your recognition of the importance of the problem we are addressing and your acknowledgment of the value of our new perspective on evaluation metrics. We have corrected all the typos you mentioned in the manuscript. We have corrected some errors in lines 118, 307, 485, as well as in Figures 5 and 6, and we have provided a clearer description of the section you mentioned. Thank you for your careful reading. Next, we will provide detailed responses to your questions and concerns regarding our work.

1.Regarding the statement on alternative to wet-lab experiments.

We sincerely apologize for any confusion caused by this claim. The original text stated, “This provides a practical alternative to testing deep learning-generated molecules directly in wet-lab experiments.” We acknowledge that this sentence may have been ambiguous.

To clarify, we did not intend to claim that our evaluation framework can replace wet-lab experiments. What we intended to say is that since models like MolCRAFT demonstrate reasonable performance in virtual screening, then it is not mandatory to directly use the generated molecules in wet-lab experiments. Alternatively, the generated molecules can serve as references for virtual screening, and the screened molecules can then be tested in wet-lab experiments.

We have updated this sentence in the paper to eliminate any ambiguity. We apologize again for any lack of clarity in the original text.

2.Regarding the limitation of similarity to known active compounds

We appreciate the reviewer for raising this question, allowing us the opportunity to provide further clarification.

The motivation behind proposing the similarity metric is that human experts modifying generated molecules is also a method of deploying SBDD models. Therefore, if a model’s output is closer to a truly active one, it becomes easier for human experts to modify these molecules into drug candidates.

From a machine learning perspective, the model does not have access to information about actives during inference and sampling. Our rationale is that if the model can generate molecules similar to experimentally validated actives for a well-studied target, then it should be able to generalize to new targets that have not been thoroughly investigated or lack known actives, producing molecules similar to the true ligands of these new targets. The functional groups and scaffolds present in these generated molecules can assist human experts in identifying truly binding compounds. After all, there are still many targets that have not been sufficiently studied.

Come back to your concern. Of course, there might be entirely new drug-like small molecules that do not resemble any known drugs. However, our similarity metric is not aimed at evaluating a single molecule but rather assesses a generative model by taking the maximum similarity across many generated molecules. Our reasoning is that if a model consistently generates small molecules across various targets that are all far from the known actives, the model’s ability to generate actives may be questionable.

Finally, we acknowledge that during evaluation, if a generated molecule is highly desirable yet significantly different from known active compounds, our similarity-based metrics may not assign a high score to that molecule. However, whether such a molecule is genuinely effective remains uncertain without actual wet-lab experiments. From an evaluation standpoint, this does not invalidate our evaluation framework, as its purpose is to provide a structured and fair assessment based on available data.

That said, we recognize that no metric is perfect. Our similarity metric is not intended to replace theoretical estimation-based metrics like Vina docking scores or QED. Instead, it is designed to provide an additional perspective that supplements these metrics.

In our evaluation framework, we assess the effectiveness of SBDD models across three distinct levels. We introduce the Delta Score and DrugCLIP Score to complement traditional docking scores as theoretical estimation-based metrics. Additionally, we incorporate similarity to known actives as another evaluation dimension. By combining these metrics, we aim to minimize the risk of drawing incorrect conclusions and provide a more comprehensive evaluation.