PoseCheck: Generative Models for 3D Structure-based Drug Design Produce Unrealistic Poses

Major:

• Recent strong baselines are missing [1][2]. A more comprehensive evaluation would be needed for a higher score.
• I was wondering if the authors could provide a more indicative metric, similar to the PoseBusters passing rate [3]. Since metrics like steric clashes and strain energies are distributed within a certain range, it is not directly evident how those baseline models perform when they are not very significant outliers.

Minor:

• Citation format needs more careful handling. Misuses of \citet (\cite) are common in this paper which should be \citep instead.
• Line 653: by ”pocket similarity’ via Pocketome -> by ``pocket similarity'' via Pocketome

[1] Protein-Ligand Interaction Prior for Binding-aware 3D Molecule Diffusion Models. https://openreview.net/forum?id=qH9nrMNTIW

[2] MolCRAFT: Structure-Based Drug Design in Continuous Parameter Space. https://proceedings.mlr.press/v235/qu24a.html

[3] PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences. https://pubs.rsc.org/en/content/articlehtml/2024/sc/d3sc04185a

1. As a benchmark, the evaluation in this article is far from comprehensive. Many recent works on antibody design and optimization are missing from the comparison, including GraphBP[1], VoxBind [2], MolCraft [3], and D3FG [4]. To the best of my knowledge, [2] and [3] are state-of-the-art methods, and all of the codes for these methods are open-sourced which I have tested. The lack of discussion and comparison with such a substantial body of related work is a major weakness.

2. Additionally, I believe the evaluation method in this article is far from comprehensive. First, aside from interactions, evaluating the molecular topology and structure in both 2D and 3D dimensions is essential. If the generated molecule is merely in a low-energy state (as described by DiffBP, where larger molecules can produce more interactions, thus lowering affinity) but significantly deviates from real drug data in terms of structure and chemical functional groups, can we truly consider that the generated molecule has an advantage? Here, I have listed several methods and their examples in evaluating 3D geometric properties and 2D structural properties, refered to recently proposed benchmark paper [5], none of which have been considered in this article. | Method | Substructure | Geometry | |--------------|---------------------|---------------------------| | LIGAN | Figure. S6 | Figure. S7, S8, S9 | | POCKET2MOL | Table. 2 | Table. 3; Figure. 4 | | GRAPHBP | - | Table. 2; Figure. 5 | | TARGETDIFF | Table. 2 | Table. 1; Figure. 2 | | DIFFBP | Table. 3 | - | | DIFFSBDD | - | Figure. 8, 9 | | FLAG | Table. 3 | Table. 2; Figure. 4 | | D3FG | Table. 1, 3; Figure. 3 | Table. 2 | | DECOMPDiff | - | Table. 1, 2; Figure. 3 | | MOLCRAFT | Table. 1 | Table. 2 | | VOXBIND | - | Figure. 7 |

3. The conclusions in this paper also have significant issues. For example, in line 376, it states, “Interestingly, DiffSBDD and TargetDiff, which are considered state-of-the-art based on mean docking score evaluations.” However, DiffSBDD performs poorly in terms of the Vina score compared to other methods, indicating that its generated initial conformations are quite unstable. Yet, its final redocked energy is very low. Could this be due to the fact that the molecules generated by DiffSBDD have a higher molecular weight than those generated by other methods? If so, is this comparison truly fair? The Vina score is an essential metric for evaluating the quality of the generated initial conformations, and it should not be overlooked. Overall, the conclusions and analyses are overly simplistic and lack comprehensiveness.

4. The writing and presentation of this article are quite mediocre. For instance, the generation strategies, detailed introductions of methods include the model architecture and generative models, and classifications of these methods are not reflected in the tables, making the table arrangement appear rather arbitrary and unstructured.

In summary, I believe the evaluation in this article is incomplete, as it lacks many essential and reproducible methods. The conclusions are not sufficiently in-depth, and for existing methods, the article merely conducts a single test, followed by evaluations from various angles. In terms of both workload and quality, this paper falls short of the standards required for a high-level conference like ICLR. Therefore, I recommend rejection.

[1] Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding. ArXiv, abs/2204.09410, 2022.

[2] Pedro O. Pinheiro, Arian Jamasb, Omar Mahmood, Vishnu Sresht, and Saeed Saremi. Structure-based drug design by denoising voxel grids, 2024a.

[3] Yanru Qu, Keyue Qiu, Yuxuan Song, Jingjing Gong, Jiawei Han, Mingyue Zheng, Hao Zhou, and Wei-Ying Ma. Molcraft: Structure-based drug design in continuous parameter space. ICML 2024, 2024.

[4] Haitao Lin, Yufei Huang, Haotian Zhang, Lirong Wu, Siyuan Li, Zhiyuan Chen, and Stan Z. Li. Functional-group-based diffusion for pocket-specific molecule generation and elaboration. ArXiv, abs/2306.13769, 2023

[5] Haitao Lin, Guojiang Zhao, Odin Zhang, Yufei Huang, Lirong Wu, Zicheng Liu, Siyuan Li, Cheng Tan, Zhifeng Gao, Stan Z. Li, CBGBench: Fill in the Blank of Protein-Molecule Complex Binding Graph

1. On the rationale of the strain energy metric: The energy change should ideally be observed as a whole, considering both the protein and the small molecule before and after binding, rather than focusing solely on the strain energy of the small molecule. "..., generally speaking, lower strain energy results in more favourable binding interactions and potentially more effective therapeutics." This assumption is inaccurate. When evaluating whether a protein and a small molecule are likely to bind and form a complex, the energy change of the protein cannot be neglected. Furthermore, as the authors pointed out, the generated poses are often problematic. The strain energy introduced by the authors may be influenced more by the generated pose itself than by the intended evaluation of the binding affinity or stability of the protein-ligand complex.
2. Regarding interaction fingerprinting, the authors present a distribution analysis of interaction fingerprints for several current models, noting significant deviations from the CrossDocked benchmark. However, the sheer number of hydrogen bond donors and acceptors does not determine the quality of the generated molecules. For instance, an excess of hydrogen bond donors and acceptors may reduce the selectivity of the small molecule, while selectivity is crucial in real-world pharmaceutical applications.
3. These metrics cannot directly guide drug discovery in real-world scenarios. Regarding redocked RMSD, the importance of this metric depends on how we define the function of SBDD models and what we expect for them. From a practical application perspective, the role of an SBDD model is to provide potential candidate molecules, without necessarily ensuring the accuracy of their binding conformations.

1. Although the proposed evaluation metrics have been comprehensively assessed on several existing methods, to my knowledge, some of the more advanced SBDD methods developed in the past two years have not been included.
2. Regarding interaction fingerprints, I agree that it is meaningful and interpretable to observe the interactions present in the generated complex conformations. However, the analysis and conclusions in this section are not sufficiently clear or thorough. For instance, I still do not understand how to evaluate a method in relation to the distribution of interaction counts. Is a higher number better, or is it more favorable for the distribution to be closer to that of the test set?