Diff-Shape: A Novel Constrained Diffusion Model for Shape-based De Novo Drug Design

Major weaknesses

1. **Limited control strategy.**Diff-Shape’s major proposed technical contribution is using a ControlNet scheme to constrain the 3D shapes of generated molecules when using an unconditional generative 3D model like MIDI. To constrain the shape of the generated molecules, the authors provide to their ControlNet module a masked/noisy version of the reference ligand. A major limitation of this approach is that by conditioning on a noisy/masked 3D graph structure of the reference ligand, the generated molecules are greatly biased to only display minor chemical differences compared to the reference ligand (even if these chemical differences induce low measured chemical similarity). This is because rather than strictly conditioning on the shape of the reference ligand, Diff-Shape conditions on the entire 3D graph of the reference ligand; generated molecules are thus biased to have similar 3D graphs, not just similar shapes. This is qualitatively evident in the visualized examples of generated molecules in Figure 4. Whereas the baseline models MIDI, SQUID, and Shape-Mol all generate molecules with meaningfully different chemical graphs (e.g., different scaffolds with entirely different bonding arrangements) compared to the reference molecule, Diff-Shape generates molecules that are generally quite similar in terms of their 3D graphs — sometimes containing only a few different atoms. While these chemical differences may be enough to sufficiently reduce the evaluated 2D chemical similarity beneath the (arbitrary) 2D similarity thresholds, the qualitative scaffold-hopping ability of Diff-Shape is clearly limited, since Diff-Shape doesn’t actually sample from an otherwise unbiased shape-conditioned chemical space. It is also important to note that even when masking all the atoms and bond types, conditioning on the exact 3D coordinates of the reference ligand severely reduces the solution space of molecular graphs, as using the exact 3D coordinates of the reference molecule as the shape representation is extremely constraining. Diff-Shape’s limited ability to generate substantially novel molecules is also reflected in the generally low Uniqueness scores for the Diff-Shape models in Table 2 — the low uniqueness rates are likely caused by Diff-Shape generating molecules that are either exact copies of the reference molecule, or which are only changed in small ways. To demonstrate otherwise, the authors should provide many more examples of generated molecules for a given reference molecule (e.g., at least 10 random examples of generated compounds for each reference molecule, without any filtering or manual selection).

2. Potentially unnecessary control strategy. Because Diff-Shape uses its Constrain/Control module to condition MIDI on a noisy representation of the reference molecule’s 3D graph, Diff-Shape’s control module is potentially unnecessary. Indeed, it is possible to directly use MIDI to condition on noisy versions of the reference molecule’s 3D molecular graph. Since MIDI is a diffusion model, it can use inpainting to condition the generation of new molecules on the point cloud coordinates of a reference 3D molecule: just inpaint new atom and bond types around the reference 3D coordinates until a specified timestep t>=0. Stopping the inpainting at time steps close to 0 would result in lower-diversity samples that have higher 3D shape similarity compared to the reference molecule, whereas stopping inpainting at time steps t>>0 would lead to more diverse molecules that are less shape-similar. This simple strategy would achieve the same function of Diff-Shape’s control strategy without requiring any additional training or changes to MIDI’s architecture. I would recommend the authors to compare against this simple inpainting strategy to justify their use of ControlNet for this task.

3. **Limited evaluations of the geometric quality of generated molecules.**The authors do not seem to evaluate the geometric stability of the generated 3D molecules. In general, it is not useful to generate a 3D molecular structure that has a similar shape compared to a reference shape if that generated structure is unstable. The authors should thus evaluate the shape similarity of the generated molecules to the reference molecule both before and after relaxing the generated structures. Ideally, this local geometric relaxation should be performed at the same level of theory as used to generate the training data (e.g., xTB for models trained on GEOM-DRUGS). At the very least, this should be done with a cheap force field method like MMFF94.

4. Unfair comparisons to baseline approaches. Diff-Shape is trained on GEOM-DRUGS, but both SQUID and Shape-Mol were trained on conformers from MOSES. Since Diff-Shape’s evaluations use test-set molecules from GEOM-DRUGS, it is unfair to compare Diff-Shape against SQUID and Shape-Mol without retraining these methods on the same dataset. For instance, reference molecule #2 in Figure 4 has ~38 non-hydrogen atoms, but MOSES only contains molecules with <27 non-hydrogen atoms. Hence, SQUID and Shape-Mol are being evaluated in an out-of-distribution task, whereas Diff-Shape is being evaluated in-distribution. I would recommend the authors to retrain MIDI/Diff-Shape on MOSES in order to perform a fair comparison.

5. **Evidence of plagiarism.**There is clear evidence of plagiarized text in the abstract and introduction sections. For instance, the following sentences from Diff-Shape are only slight modifications of corresponding text from the SQUID paper (Adams and Coley, Equivariant Shape-Conditioned Generation of 3D Molecules for Ligand-Based Drug Design, ICLR, 2023):

   Case 1

   (Diff-Shape) “Shape-based virtual screening is a widely utilized method in ligand-based de novo drug design, aiming to identify molecules in chemical libraries that share similar 3D shapes but simultaneously possess novel 2D chemical structures compared to the reference compound.”

   (SQUID) “Shape-based virtual screening is widely used in ligand-based drug design to search chemical libraries for molecules with similar 3D shapes yet novel 2D graph structures compared to known ligands.”

   Case 2

   (Diff-Shape) “In recent years, generative model has emerged as new paradigm for de novo drug design and has revolutionized computer-aided drug design (CADD) by enabling efficient exploration of chemical space and goal-directed molecular optimization (MO) in a data driven manner.”

   (SQUID) “Generative models for de novo molecular generation have revolutionized computer-aided drug design (CADD) by enabling efficient exploration of chemical space, goal-directed molecular optimization (MO), and automated creation of virtual chemical libraries”

   Case 3

   (Diff-Shape) “Skalic et al and Imrie et al. trained networks to generate 1D SMILES strings and 2D molecular graphs, respectively, conditioned on CNN encodings of 3D pharmacophores ignoring Euclidean symmetries. Zheng et al. used supervised molecule-to-molecule translation on SMILES strings for scaffold hopping tasks and evaluated the generated scaffolds’ 3D shape similarity to the reference. Papadopoulos et al. sampled molecules with high shape similarity to a target by SMILES based reinforcement learning in REINVENT, requiring re-optimization of the agent model for each target shape. Roney et al. fine-tuned a 3D generative model on the hits of a shape based virtual screen of more than 10^10 drug-like molecules to shift the learned distribution towards a particular shape. Yet, this expensive screening approach must be repeated for each new reference shape”

   (SQUID) “Skalic et al. (2019) and Imrie et al. (2021) train networks to generate 1D SMILES strings or 2D molecular graphs conditioned on CNN encodings of 3D pharmacophores. However, they do not generate 3D structures, and the CNNs do not respect Euclidean symmetries. Zheng et al. (2021) use supervised molecule-to-molecule translation on SMILES strings for scaffold hopping tasks, but do not generate 3D structures. Papadopoulos et al. (2021) use REINVENT (Olivecrona et al., 2017) on SMILES strings to propose molecules whose conformers are shape-similar to a target, but they must re-optimize the agent for each target shape. Roney et al. (2021) fine-tune a 3D generative model on the hits of a ROCS virtual screen of > 10^10 drug-like molecules to shift the learned distribution towards a target shape. Yet, this expensive screening approach must be repeated for each new target.”

Minor weaknesses

1. **Poor paper presentation.**Numerous citations, quotations, and punctuation marks are formatted incorrectly, reducing the overall quality of the paper’s presentation. There are also numerous spelling/grammar mistakes that should be corrected to improve the paper’s readability.

2. **Unclear methods section.**The methods section is difficult to parse, and requires constant referencing back-and-forth between the ControlNet and MIDI papers to understand how Diff-Shape works. For the reader’s benefit, I recommend that the authors adjust the text of the methods sections to make the text more stand-alone. Moreover, many important details are not explained. For instance, MIDI uses discrete diffusion for atom/bond types. How is discrete diffusion compatible with the linear mixing between the locked “3D generative model” and the trainable “Constrain Module“? The paper’s clarity would be vastly improved if the authors could provide a detailed Algorithm that outlines the important steps of the forward pass of Diff-Shape.

3. **Missing content.**In section 2.4, the paper references experiments that are not included in the main text or appendices. For instance, the authors state: “for the tasks of structure-based drug design, we also evaluate the docking score of generations in target pockets with the GLIDE module”, but no such experiments are included in the submission.

Note on reviewer rating

I would have assigned a rating of 3 (reject, not good enough) because of the first four major weaknesses, but I reduced my rating to 1 (strong reject) due to the clear evidence of plagiarism. I will consider changing this rating upon ethics review.

Lack of Clarity in Evaluation Metrics and Results

• Explanation of Metrics: Although the study measures various important metrics (e.g., validity, uniqueness, shape similarity), these metrics are not adequately defined. Each metric's purpose, importance, and interpretative value should be clarified for the reader, especially for domain-specific measures like 2D graph similarity. Explaining how these metrics translate to chemical or biological relevance would add clarity and context.

• Presentation of Results (Tables and Figures): Tables and figures, particularly Table 3 and Figures 3 and 4, need clearer labels and captions to improve interpretability. For example, in Figure 3, in-figure labels and descriptive legends would make key findings more accessible. Table 3 could be reorganised to highlight the comparison across models more explicitly (possibly making certain results bold for which is the best performing architecture). Table 3, in particular, is very difficult to assess if the proposed model results are competitive.

Insufficient Background on Key Concepts

• Explanation of 2D Graph Similarity: While low 2D graph similarity is cited as a goal, the authors do not provide a rationale for its significance. It is crucial, as the authors do, to explain that minimizing 2D similarity while maximizing 3D shape alignment promotes structural novelty, which is beneficial for discovering new ligands. This discussion should be expanded in the background section to establish a clearer objective for readers unfamiliar with scaffold-hopping.

• Background on Zero-weighted MLP and Graph ControlNet: The paper introduces zero-weighted MLPs within Graph ControlNet, which may be unfamiliar to many readers. Including a brief background on this concept would provide context on its role in stabilizing training and gradually integrating shape constraints.

Weak Conclusions and Reflection on Results

The conclusions section could be enhanced by reflecting more critically on the limitations of Diff-Shape, such as its dependency on a pre-trained model and challenges related to low validity when fuzzy operations are applied. Additionally, providing insights into the real-world applicability of Diff-Shape and potential improvements (e.g., exploring alternative conditioning mechanisms) would strengthen the overall impact.

Minor

• GitHub Repository: The repository breaches anonymity as the authors’ full names and contact details are in the setup.

• Grammatical, Citation & Spelling Errors:

  • In-text citation should be before a period.
  • Authors are not consistent in using a space after a comma.
  • There are many occasions of missing articles in sentence structures (e.g., "chosen for doing following experiments in our current study").
  • Incorrect grammar on multiple occasions (e.g., "called as Diff-Shape," "in later case").
  • GEOM-Drugs dataset was not cited.

Equations

• After equation 5, where the graph embedding, $G^{t+1}$, is defined, $A^{t+1}$ and $C^{t+1}$ are not defined. As a result, $a_{i}^{t-1}$ and $c_{i}^{t-1}$ are not defined either.

• In Equation 8, $\lambda$ is not defined, and it is unclear what kind of weighting is allocated to the "weighted sum of the components" in the loss function. Is it randomized or parameterized?

● Definitions: Please define the terms "validity" and "uniqueness" in the context of your work.

● Drug-Likeness of Molecules: The paper does not provide sufficient information on whether the generated molecules adhere to drug-likeness criteria, which is essential for their potential application in drug discovery.

● Responsiveness to Shape Variation: There is a lack of analysis regarding how the model's generation is affected by variations in the size of the reference shape. Demonstrating responsiveness in this context would strengthen the model's validity.

● Synthesis Feasibility: The paper does not address the synthetic feasibility of the generated molecules. Assessing how easily these compounds can be synthesized is critical for practical applications in drug discovery.

● Performance with Rotatable Bonds: There is no insight into how the model performs with varying numbers of rotatable bonds in both the seed and generated compounds. This information is important for understanding the flexibility and potential conformational diversity of the generated molecules.

● Limited Insight into Model Behavior: While the work is impressive, the lack of detailed discussion around the aforementioned points raises questions about the model's comprehensive applicability and effectiveness in real-world scenarios.

Weak Novelty:

• Studies utilizing shape as the input condition to generate 3D molecule already exist, as author mentioned (e.g., SQUID, ShapeMol - see below (1) (2) (3) papers, etc). Therefore, novelty must be highlighted as performance superiority rather than conceptional novelty.

Details : The authors, inspired by ControlNet, applied its principles to a 3D molecular generation model, proposing a state-of-the-art approach that uses shape as an input to preserve the shape in the generated molecules. While there is a novel aspect in utilizing graph-based ControlNet in the molecular generation domain, the paper lacks clear experimental evidence to justify the necessity of this approach. Furthermore, it is not the first to use shape constraints as input for 3D molecular generation, as several studies have already explored similar approaches. This raises questions about the conceptual novelty, which is not well articulated in the paper. For instance, the paper does not clearly explain how it conceptually differs from existing models that use encoders to interpret shape constraints for 3D molecular generation (see related studies below (1) (2) (3)). The authors acknowledge this and instead emphasize the model’s superior performance compared to other shape-constrained generation models (see lines since 097). However, without clear performance improvements (see weakness 2), the advantages or novelty of this model over other shape-constrained generation models remain unclear.

(1) Equivariant Shape-Conditioned Generation of 3D Molecules for Ligand-Based Drug Design [https://openreview.net/forum?id=4MbGnp4iPQ / https://arxiv.org/abs/2210.04893]

(2) ShapeMol : Shape-conditioned 3D Molecule Generation via Equivariant Diffusion Models [https://arxiv.org/pdf/2308.11890]

(3) Shape-Based Generative Modeling for de Novo Drug Design [https://pubs.acs.org/doi/10.1021/acs.jcim.8b00706]

Unclear Performance Improvement:

• In Figures 5 or 6, while the proposed method significantly increases shape similarity (advantage), it also increases graph similarity (disadvantage). The performance improvement would be validated if shape similarity increased while maintaining graph similarity, but this is not the case, making it ambiguous to claim a performance enhancement.

• Table 2 shows a significant decrease in Validity * Uniqueness compared to MIDI. It is unclear whether the proposed method clearly outperforms MIDI.

• In Table 3, while the result showing low graph similarity for shape similarity above 0.8 is interesting, there is concern about its parametric nature. For instance, if the comparison is based on shape similarity above 0.4 instead of 0.8, the proposed method might exhibit higher graph similarity than other methods.

[Details] In Table 3, the value in the fourth column (P^{Simg<1.0}_{Sim3D>0.8}) seems to best highlight the strengths of this approach. However, this advantage does not appear consistent across different thresholds (e.g., Sim3D > 0.4). Figure 3 shows that graph similarity unfortunately does not consistently increase alongside shape similarity. If the graph similarity had been lower or comparable to existing methods, while shape similarity had been higher, the superiority of this approach would have been clearer. Unfortunately, this is not the case, and thus the paper falls short of providing clear evidence of performance superiority, which may hinder its acceptance at ICLR. Additionally, beyond the limitations in Validity and Uniqueness compared to other shape-constrained generation models in Table 3, the paper fails to demonstrate that the "graph similarity/shape similarity" metric consistently outperforms other methods across thresholds. This makes it difficult to identify the distinct technological advantages of this approach.

• There is a lack of detailed explanation regarding whether the experiments for existing studies (SQUID, ShapeMol) were conducted under fair conditions with the same input conditions and without pre-trained checkpoints. Additionally, the absence of base code makes it difficult to verify this. Performance comparisons under the same conditions as those presented in existing studies should also be provided.

======

Depending on the responses from other reviewers and the authors, if the concerns can be sufficiently addressed, I will consider adjusting the review score accordingly.