Decomposed Direct Preference Optimization for Structure-Based Drug Design

Thank you for your detailed feedback. Please see below for our responses to the comments.

Q1: Include relevant prior work PILOT.

A1: Thank you for your suggestion! We have added discussion of PILOT [1] in Related Work section.

Q2 & Q4.2: Concern related to Vina Dock.

A2: We appreciate the opportunity to clarify this aspect. As we illustrated in the paper, DecompDPO is a perference alignment framework that designed for aligning practical pharmaceutical needs using multi-granularity preference. While we demonstrated the remarkable performance of DecomDPO in terms of several prevalently recognized molecular properties, its application scope is not only limited to optimize these properties. Besides, compared to DecompOpt [1] and EvoSBDD [2], which only focus on optimizing model towards a specific target protein, DecompDPO is also applicable for molecule generation, highlighting its generalisability of providing better distribution without additional model adjustment toward each target. To provide a more comprehensive discussion of current search of molecular optimization, we have added EvoSBDD in Related Works section. To better illustrate the effectiveness of DecompDPO, we extended the training of our model by an additional epoch, totaling 30,000 fine-tuning steps, which aligns with AliDiff. The results are shown as follows:

As the result shows, DecompDPO achieves significantly improved results, with higher binding affinity related metrics compared to AliDiff, after longer training. For general molecular properties, the Complete Rate after two epochs of optimization is 77%, a slight decrease from 83.6% after one epoch. These results demonstrates the potency of DecompDPO in optimizing binding affinity related metrics effectively, including Vina Score, while not sacrifycing much molecular properties.

Q3: Evaluation using PoseCheck benchmark.

A3: Thank you for your suggestion. We reported the quartiles of Strain Energy (SE), the average number of Steric Clashes, and the number of key interactions, i.e., Hydrogen Bond Donors (HB Donors), Hydrogen Bond Acceptors (HB Acceptors), van-der Waals contacts (vdWs) and Hydrophobic interactions (Hydrophobic) calculated using PoseCheck.

As the table shows, DecompDPO achieves better Strain Energy compared to TargetDiff and DecompDiff. Admittedly, although DecompDPO performs slightly better than DecompDiff, the mean number of Steric Clashes is still high due to few anomalies beta priors used in sampling. We will address this issue in our future work.

Q6: Concerns about the JS Divergence of bond distance and angle.

A6: Thank you for your question. Indeed, as we stated in Section 3.3, the purpose of introducing physics-informed energy terms in preference alignment is to maintain reasonable molecular conformations instead of optimizing the geometry of generated molecules. So the bond and angle energy in Formula (11) and (12) is used as constriaint terms to penalizing molecules with high rewards but poor conformations rather than an optimization objective in preference alignment. In Table 5 and 6, we demonstrate that with physically-constrained optimization, DecompDPO generally achieves comparable performance with DecompDiff in terms of molecular conformation, which is align with our motivation. To demonstrate the usefulness of physics-informed energy terms, we have further conducted an experiment without using these terms.

As the results show, the energy differences for both rigid fragments and generated molecules significantly increase when not using energy terms for penalizing unreasonable conformations. These results suggest that physics-informed energy terms effectively prevents the model from optimizing toward molecules with high rewards but unreasonable conformations without sacrificing much optimization performance.

Q7: Could a similar DPO scheme be applied to TargetDiff?

D7: Please refer to the result we provided in Answer to Questions Q4

[1] Cremer, J., Le, T., Noé, F., Clevert, D. A., & Schütt, K. T. (2024). PILOT: Equivariant diffusion for pocket conditioned de novo ligand generation with multi-objective guidance via importance sampling. arXiv preprint arXiv:2405.14925.

[2] Zhou, X., Cheng, X., Yang, Y., Bao, Y., Wang, L., & Gu, Q. (2024). DecompOpt: Controllable and Decomposed Diffusion Models for Structure-based Molecular Optimization. arXiv preprint arXiv:2403.13829.

[3] Reidenbach, D. (2024). EvoSBDD: Latent Evolution for Accurate and Efficient Structure-Based Drug Design. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations.

Q4: Why does the multi-objective DPO fair worse than the energy-focused version of AliDiff? Could the regressions stem from the IPDiff vs DecompDIff underlying architecture and training? A deeper study of the underlying DPO method agnostic to the architecture would be very interesting as unless one is to use DecompDiff can someone use DecompDPO?

A4: Thank you for your question. We respectfully disagree with Reviewer koEE that the multi-objective DecompDPO is fair worse than the energy-focused version of AliDiff, with QED and SA of AliDiff decreased compared to IPDiff in binding affinity optimization, while multi-objective DecompDPO achieves improvement across all properties.

In multi-ojbective optimization, the gradient of different objectives can conflict with each other, leading to less effective performance on each objective compared to single objective optimization. So we think it is more fair to compare DecompDPO with AliDiff in terms of binding affinity optimization. Admittedly, the performance of preference alignment could be affected by the base model. However, as we demonstrated in previous Q2, with the same length of training steps, DecompDPO achieves better performance in terms of all binding-affinity related metrics compared to AliDiff.

We further conducted an experiment using TargetDiff as base model to answer the question regarding different model architecture. Please refer to Q4 in Answer to Questions.

Q5: Concerns about the Complete Rate

A5: Thank you for bringing up this concern. First, we would like to clarify that DecompDPO is evaluated in two settings: In molecule generation, as shown in Table 7 of our paper, DecompDPO achieves a Complete Rate of 73.26%, which is slightly higher than DecompDiff. Besides, as reported in Single-Objective Optimization section, DecompDPO achieves a Complete Rate of 83.6%. These results demonstrate that DecompDPO generally maintains robust performance in generating valid molecules after fine-tuning for different objectives in molecule generation.

Admittedly, in molecular optimization, which optimize the fine-tuned model more aggressively towards each target protein, we observe that the Complete Rate decreases to 65%. As preference alignment push the generated molecules into regions of the chemical space with high rewards that may deviate from the training distribution, in more aggressive optimization, the likelihood of producing invalid or disconnected molecules could be increased. These results suggest that careful consideration is needed to balance trade-offs between optimization objective and general molecular properties during optimization. However, we believe that in molecular optimization, the significant improvements in targeted property optimization could justify the slight 7% reduction in validity.

Q1: DecomDPO performs iterative DPO, optimizing the model for each target protein in the test set. Does this mean a new model is trained for each test set pocket? If so, what happens if you only do standard DPO fine-tuning as done in AliDiff?

A1: There appears to be a misunderstanding regarding how DecompDPO is trained and utilized. As we emphasized in our paper, DecompDPO is evaluated in two distinct settings: molecule generation and molecular optimization. In molecule generation setting, we fine-tuned DecompDPO on 63k generated preference pairs from the training data for one epoch, resulting in a single fine-tuned model for evaluation. In molecular optimization setting, we further validated the potential of DecompDPO, optimizing the fine-tuned model specifically for each target protein in the test set. In our paper, except for the results presented in Table 2, all evaluations in our paper are based on a single fine-tuned model.

Q2: On line 350 it appears to describe that DecompDPO samples 500 molecules per 100 test pocket for table 1. Do the benchmarks reflect the average over all 500 or is it some filtered subset?

A2: We would like to clarify that from line 346 to 352 in our original paper, we described how we generated preference pairs for DecompDPO instead of how we sampled molecules for evaluation. Specifically, DecompDPO is first fine-tuned using 63,092 valid pairs generated on the training set. The fine-tuned model is used for molecule generation. Then we further optimized the fine-tuned DecompDPO for molecular optimization using preference pairs constructed from 500 molecules sampled for each target protein. For model evaluation, we sampled 100 molecules for each target protein in the test set for both molecule generation and molecular optimization.

Q3: What is the inference time, and how does it compare to prior methods factoring in the complete rate?

A3: Thank you for your question. The inference time of DecompDPO depends on the base model DecompDiff used in our experiment. As we demonstrated in Answer to Weakness Q5, the Complete Rate of DecompDPO and DecompDiff are generally on the same level. We test the inference efficiency for generating 20 molecules on average. The inference time for DecompDPO and DecompDiff are 506s.

Q4: What happens if you fine-tune TargetDiff on the DPO fine-tuning dataset as done in AliDiff? It would be interesting to see of the decomposed objective yields better training data.

A4: Thank you for your question. We agree that more studies that evaluating DecompDPO on different model architectures would further demonstrate its effectiveness. The GlobalDPO with energy constraints can be easily adapted to general model structures. The LocalDPO requires an additional fragmentation procedure to generate decomposed preferences.

Regarding applying DecompDPO to TargetDiff, DPO expects that perference data is sampled from the optimal distribution and usually starts from the reference model distribution in practice. Sampling preference pairs from TargetDiff’s distribution incurs high computational costs, which we could not accommodate alongside other ablation studies during the rebuttal.

For now, to provide preliminary insights, we conducted a toy experiment using DecompDPO's 63k fine-tuning dataset. Specifically, we first randomly sampled half of the data to fine-tune TargetDiff, and term the fine-tuned model as TargetDiff - ft. This step is to reduce the gap between the reference model and preference data distributions. Then we use the remaining half of the dataset to optimize TargetDiff towards Vina Minimize, resulting in the model termed TargetDiff - dpo.

As the result shows, the performance of TargetDiff - ft decreased compared to the original TargetDiff, suggesting that fine-tuning directly on a subset of preference data introduces challenges in aligning TargetDiff with the DecompDiff distribution that generates the preference data. Using a more general data distribution instead of top-tail data could yield better results. Regarding the performance of TargetDiff - dpo, while it is better than TargetDiff - ft, it only partially recovers the performance of the original TargetDiff, indicating that preference alignment introduces potential benefits but requires further refinement for full integration.

Q5: Where are results showing fine-tuning pretrained diffusion models for molecule generation across various protein families (line 21-22)? There is no notion of protein families discussed in existing CrossDocked2020 benchmarks.

A5: Thank you for your question. Following Luo et al. [4], only proteins with less than 30% sequence identity are selected in the dataset to ensure protein diversity, which is likely to include a wide range of protein families with different structural and functional characteristics. In the CrossDocked2020 test set, the test proteins such as 3L3N, 4XLI, and 5MGL belong to different protein families. The reference to protein families in our paper is intended to highlight the broad applicability of DecompDPO across different targets and to distinguish the two application scenarios.

[4] Luo, S., Guan, J., Ma, J., & Peng, J. (2021). A 3D generative model for structure-based drug design. Advances in Neural Information Processing Systems, 34, 6229-6239.

Thank you for your prompt feedback. We're glad that our rebuttal has resolved many of your previous questions and concerns. Regarding your remaining concerns, we will address it as follows:

Re: "Given all prior 3D generative models are unable to generate 100% valid and complete molecules, the impact of validity should be reflected in the benchmarks. Otherwise, it's impossible to know is the high average due to an overall good model or a few good samples or due to filtering."

Thank you for your suggestion. We have moved the validity to the table in the main text.

Re: "My concerns with Vina Dock are also still not addressed....As a result, the only meaningful metrics here are Vina score and the structure-based distributional metrics. To this end, in Table 3 AliDiff shows better Vina Score (-7.07 to -6.31), QED, and diversity with DecompDPO having better SA." Re: "Given Table 3, why does DecompDPO generate worse structures that AliDiff that Vina is able to optimize to predict much higher Vina Dock scores? This is something that would be incredibly valuable to the space as several methods exhibit preference for redocking with generating poor raw structures."

We would like to kindly remind the reviewer that we have already demonstrated DecompDPO's remarkable performance in optimizing binding affinity with a higher Vina Score, Vina Min, Vina Dock, and High Affinity compared to AliDiff, as demonstrated in the answer to Q2 & Q4.2 in Weakness Part 1. For your convenience, we have reiterated the relevant results below:

As shown in the table above, after fine-tuning DecompDPO for an additional epoch (a total of 30k steps, matching the fine-tuning steps used in AliDiff), DecompDPO outperforms AliDiff across all binding-related metrics. This demonstrates the effectiveness of DecompDPO in generating molecules with superior structures.

Re: Why does the DPO training and finetuning hurt TargetDiff here whereas AliDiff's DPO help it?

We would like to clarify that the results we provided in the answer to Q4 are preliminary proof-of-concept experiments aimed at demonstrating the potential of applying our idea to TargetDiff. The key takeaway from this preliminary experiment, is that TargetDiff-DPO outperforms TargetDiff-FT. However, fully implementing TargetDiff-DPO would require a more thorough execution of the idea, which was not feasible within the given time constraints.

Additionally, we chose to apply our idea to DecompDiff because it outperforms TargetDiff across major metrics and serves as a stronger baseline. We believe that when a stronger baseline is available, it is both reasonable and more impactful to demonstrate the effectiveness of our method using that baseline, as it better highlights the full potential of our approach. Therefore, we do not consider it necessary or fair to be required to apply our method to TargetDiff, which is weaker than DecompDiff..

Dear Reviewer koEE,

Thank you for your valuable feedback and the time you've invested in reviewing our work.

As we near the end of the discussion phase, we wanted to follow up to see if you've could review our responses to your comments. If there are any further concerns we can address, please let us know.

We look forward to your feedback and are happy to address any further questions.

Sincerely,

The Authors

Q3: Details of molecular fragmentation

A3: Thank you for your question. We follow the fragmentation algorithm proposed in DecompDiff [1] to decompose molecules. Specifically, we first use Alphaspace2 [2] to extract target protein subpockets and use BRICS to decompose ligands into fragments. Then, terminal fragments (with only one connection site) are assigned to subpockets by a linear sum assignment. Arms centers are defined as the centroids of terminal fragments and any remaining subpockets, and scaffold center is defined as the farthest fragment from all arm centers. Finally, the nearest neighbor clustering is performed to tag fragments as arms or the scaffold. We have added the details of molecular fragmentation in Appendix.

Q4: Ablation study on the physics-informed reward term.

A4: Thank you for your question. Please refer to our response to Q3 in Answer to Weakness.

Q5: More extensive evaluation could also include methods like REINVENT or other reinforcement learning/GFlowNets-based approaches.

A5: Thank you for your suggestion! We appreciate the importance of including more baseline methods in molecular optimization. Currently, we are still running the experiment. We are committed to include comparisons with REINVENT in the future version of the paper.

Q6: Response to Minor Comments

A6: Thank you for your careful reading and valuable suggestions to improve our manuscript. We have addressed all the minor comments in the revised version.

[1] Guan, J., Zhou, X., Yang, Y., Bao, Y., Peng, J., Ma, J., ... & Gu, Q. (2024). DecompDiff: diffusion models with decomposed priors for structure-based drug design. arXiv preprint arXiv:2403.07902.

[2] Katigbak, J., Li, H., Rooklin, D., & Zhang, Y. (2020). AlphaSpace 2.0: representing concave biomolecular surfaces using β-clusters. Journal of chemical information and modeling, 60(3), 1494-1508.

[3] Buttenschoen, M., Morris, G. M., & Deane, C. M. (2024). PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences. Chemical Science, 15(9), 3130-3139.

Q1: Why is the model only fine-tuned for one epoch? How does longer training affect the target metrics and other molecular properties?

A1: Thank you for your question regarding the number of fine-tuning epochs. We initially fine-tuned the model for one epoch (15,000 steps) to balance computational efficiency with performance. To assess the impact of longer training, we extended the fine-tuning to two epochs (totaling 30,000 steps), aligning with the fine-tuning steps of AliDiff for a fair comparison. The results are as follows:

As the results demonstrate, extending the fine-tuning significantly enhances the binding affinity-related metrics. The median Vina Dock score improved from -9.77 to -10.66, and the High Affinity increased to 100%. For general molecular properties, the Complete Rate after two epochs of optimization is 77%, a slight decrease from 83.6% after one epoch. There is also a minor decrease in QED when training for longer steps. These findings suggest that careful consideration is needed to balance trade-offs between optimization objectives and general molecular properties. However, the substantial improvements confirm that DecompDPO is highly effective in aligning the model with preferences without sacrificing much on overall molecular quality.

Q2: Is there a trade-off between the target metrics and other molecular properties? Does the general molecular quality degrade when the reward is optimized more aggressively? Posebuster could be used to answer this question.

A2: Thank you for your insightful question. There is indeed a trade-off between optimizing target metrics and maintaining general molecular properties. When the reward is optimized more aggressively, such as in our molecular optimization setting where DecompDPO is further fine-tuned for each target protein, we observe significant improvements in target metrics, achieving a 52.1% Success Rate as shown in Table 2. To assess whether general molecular quality degrades under this aggressive optimization, we evaluated the Complete Rate and the percentage of valid molecules verified by PoseBuster [3] (PB-Valid):

As the result shows, both the Complete Rate and PB-Valid decrease in molecular optimization compared to the general fine-tuned DecompDPO and baseline model. These results suggest that while more aggressive optimization improves target metrics, it may compromise general molecular properties. Therefore, the trade-off between optimization objectives and general molecular properties should be handled carefully during optimisation.

Third, regarding the observed increase in QED and SA, we attribute this to the flexibility introduced by LocalDPO in the decomposed chemical space. To expalin the observed improvements in QED and SA when optimizing for binding affinity, we calculated the Pearson correlation coefficients between the differences in Vina Minimize scores and the differences in QED and SA scores for each pair in our dataset. We found that the correlations between differences in the decomposed arms' Vina Minimize and the corresponding molecules' differences in QED and SA are -0.12 and -0.11, respectively. These statistics suggest that the improvements in QED and SA are potentially due to the benefits of LocalDPO, where the flexibility introduced by the decomposed chemical space alleviates conflicts between SA and binding affinity, allowing the model to navigate the chemical space effectively even when optimizing for binding affinity. Besides, in multi-objective optimization, the gradients of different objectives may conflict with each other, leading to less effective results and potentially hindering the optimization towards QED and SA. Admittedly, in DecompDPO, we do not investigate the optimal approach for multi-objective optimization, which is reserved for our future work.

[1] Qian, H., Huang, W., Tu, S., & Xu, L. (2024). KGDiff: towards explainable target-aware molecule generation with knowledge guidance. Briefings in Bioinformatics, 25(1), bbad435.

Molecular optimization focuses on generating better molecules for each specific target protein by optimizing more aggressively with specific information about the target binding site. We employ molecular optimization to further validate the effectiveness of DecompDPO in addition to molecule generation, demonstrating its versatility in different settings.

Regarding the comparison with Pocket2Mol, we acknowledge that the performance of preference alignment is influenced by the base model, making it challenging for diffusion models to achieve the exceptionally high SA scores of Pocket2Mol. However, we believe that DecompDPO's superior ability is demonstrated by its outstanding performance, achieving the highest Success Rate among all generative models and the highest SA score among diffusion-based methods, ranking second only to Pocket2Mol among all generative methods. This highlights DecompDPO's effectiveness in balancing high binding affinity with favorable SA score within the diffusion model framework.

Q3: Ablation studies: 1) How longer DPO training affects the results; 2)Provide empirical evidence for the benefits of physics-informed energy terms in maintaining reasonable molecular conformations during optimization; 3) QED and SA values in the single-objective case are as good or even better than in the multi-objective setup.

A3: Thank you for your detailed review. We have further provided experiment results with longer training steps. Please refer to Q1 in Answer to Questions.

Second, we have further conducted an experiment without using physics-informed energy terms to penalize dpo reward. We summarized the conformation related metrics in the following table:

The results showed significant increases in energy differences for both rigid fragments and generated molecules when the physics-informed terms were omitted. Specifically, the energy difference for generated molecules increased from 976.33 to 12,981.34. These findings indicate that the inclusion of physics-informed energy terms effectively prevents the model from optimizing toward molecules with high rewards but unreasonable conformations, thereby maintaining more realistic molecular structures during optimization. We also evaluated the binding affinity and molecular properties, which are shown in the following table:

The results indicate that physics-informed energy terms are beneficial in maintaining reasonable molecular conformations without sacrificing much in terms of optimization performance.

Thank you for your constructive and detailed feedback. Please see below for our responses to the comments.

Q1: Empirical results: The performance improvement compared to the base model is moderate in terms of QED and SA. Will more DPO fine-tuning iterations/epochs further increase the optimized scores? The evaluation should include additional sanity checks such as Complete Rate and PoseBusters filters.

A1: Thank you for your comment. First, we want to emphasize that in our evaluation, performance of generative models is evaluated across 100 proteins with 100 molecules generated for each protein, ensuring the observed improvement of DecompDPO is robust in statistics and reflects the improvements in distribution. As shown in Table 1, DecompDPO's effectiveness has been validated by its highest SA across all the diffusion based models. To further illustrate its potential, we have further trained DecompDPO for another epoch. Please refer to Q1 in the Answer to Questions.

Second, we reported the Complete Rate, Novelty, Similarity, and Uniqueness in Table 7 to evaluate the model's capability in designing novel and valid molecules. DecompDPO generally maintained similar Complete Rate compared with DecompDiff. To further check the sanity of generated molecules, we reported the PB-Valid checked by the PoseBusters. Please refer to Q2 in the Answer to Questions.

Q2: Baselines: 1) In Table 1, DecompDPO is compared with baselines that trained to reproduce the training data distribution. More relevant baselines are only considered in the Molecule Optimization section. 2) DecompDPO does not show larger improvements compared to the presented baseline methods, for example Pocket2Mol seems to achieve better SA than DecompDPO.

A2: Thank you for bringing up this concern. First, as discussed in our original paper, molecule generation emphasizes the generalizability of models, aiming to generate desired molecules across various protein families. This is analogous to evaluating the performance of large language models independently of whether they are fine-tuned using reinforcement learning methods. Out of this concern, we compared DecompDPO with generative models designed for molecule generation to assess its broad applicability.

To provide a more comprehensive comparison, we have included additional target-aware generative methods in the binding affinity optimization setting. We have further incorporated a new baseline, KGDiff [1], with binding knowledge guided generation. The results are summarized in the following table:

As the table shows, DecompDPO, initialized from DecompDiff, achieves significant improvements on all binding-affinity-related metrics. Compared to AliDiff and KGDiff, DecompDPO attains the highest Vina Dock scores and High Affinity percentages without sacrificing much on QED and SA values, demonstrating its effectiveness in generating molecules with both high binding affinity and desirable drug-like properties.

Q3:

Re: "Thank you for providing the additional ablations. ... why is the model not trained for longer and the stronger empirical results reported in the paper? Are there trade-offs that are currently not discussed in the paper and responses?"

Thank you for recognizing our efforts in providing additional ablations. Regarding training time, as explained in Q1 in Question Part 1, our current results already demonstrate significant improvements over the base model. Out of the concern of computational efficiency, we believe the current training time is sufficient to validate the strong optimization capabilities of DecompDPO.

Additionally, we have discussed potential trade-offs between stronger performance of optimization objective and other general molecular properties in the answer to Q1 in Question Part 1. We reiterate here for clarity: While extending training to two epochs enhances the optimization objective, there are slight decreases in general molecular properties, with the Complete Rate dropped from 83% to 77%, and the mean QED decreased from 0.48 to 0.47.

Re: "Furthermore, I do not find the explanation for my question about QED and SA values in single-objective vs. multi-objective setups convincing. The authors say that "the gradients of different objectives may conflict with each other, leading to less effective results and potentially hindering the optimization towards QED and SA". If a model attempts to optimize a certain property explicitly but does so less successfully than a model that completely disregards the same property (apart from the very small correlation the authors mentioned in their rebuttal), this calls into question the entire approach in my opinion. Besides, the statement "the flexibility introduced by the decomposed chemical space [LocalDPO] alleviates conflicts between SA and binding affinity" is unsubstantiated because the results in Table 4, which explicitly compares local and global DPO, show there is effectively no difference of the QED and SA values in the two setups."

Thank you for your detailed feedback. We would like to further address your concerns by emphasizing the following key aspects:

As demonstrated in Q3 of Weakness Part 3, our dataset shows correlations between the optimization objectives: The correlations between the differences in the decomposed arms' Vina Minimize and the corresponding molecules' differences in QED and SA are -0.12 and -0.11, respectively. In contrast, the correlation between differences in molecules' Vina Minimize and SA is 0.27, and between Vina Minimize and QED is -0.14. This shift indicates that decomposition fundamentally changes the way Vina Minimize optimization interacts with QED and SA. In the single-objective optimization of Vina Minimize, the negative correlation introduced by decomposition may indirectly lead to improvements in QED and SA, as observed in our results.

We would like to further clarify that multi-objective optimization remains effective and provides balanced improvements across metrics. It does not prioritize a single metric, but seeks a compromise that improves the overall molecular quality. The results of the two-epoch optimization provide clear evidence: When DecompDPO is optimized more aggressively towards Vina Minimize, QED decreased and became lower compared to multi-objective optimization results. This result demonstrates that more aggressive optimization of Vina Minimize may result in reduced secondary benefits for QED and SA due to the inherent trade-offs between molecular properties.

Questions

A1 & A2: Please refer to Weakness Q3 above.

A3 & A4: Thank you for recognizing our efforts.

A5: Thank you for your question. We will try to provide the result before the end of the discussion period and plan to include REINVENT 4 in the Related Work section to provide a more comprehensive overview.

We would also like to clarify an important distinction: our work, along with the baselines in the paper, focuses on 3D molecular generation, whereas REINVENT 4 operates in a 1D space using SMILES. The challenges and goals of 3D and 1D molecular generation are very different. Compared to 1D or 2D generation methods, 3D modelling involves capturing spatial and geometric information, which is inherently more complex.

As noted in EvoSBDD [1], a 1D optimization method based on SMILES, 3D generation methods generally perform worse than 1D or 2D generation methods across all metrics. However, this does not diminish the importance of 3D methods. On the contrary, 3D modelling is indispensable in SBDD, where understanding spatial relationships is crucial for evaluating and optimizing binding affinity to target proteins. Given these differences, we believe a direct comparison between DecompDPO and REINVENT 4 is not fair or meaningful.

To better contextualize our work, we will include discussions in the revised version to clarify the unique contributions of 3D methods. While we will make efforts to explore comparative analyses, we hope to emphasize the inherent differences between 3D and 1D methods should be considered in the comparison. This distinction is critical for understanding the complementary roles of these approaches in addressing the unique challenges of SBDD.

A6: Thank you for your efforts in reviewing our paper and your detailed suggestions! In our newly uploaded version, we have adjusted all these suggestions. Please let us know if you have any further questions.

Remaining Concerns

Thank you for your suggestion. We will update all these results in the revised version of paper.

Supplementary Questions

Q1: $\beta$ schedule: shouldn't regularization be stronger at the end of the trajectory when molecular features are only refined but the global structure is more or less decided already? The optimization objective probably doesn't depend on small geometric details.

A1: In Direct Preference Optimization (DPO), we focus on optimizing the model parameters, not the molecules directly. This makes our model a generative one, different from traditional molecule optimization methods. Our goal is to adjust the distribution generated by DecompDiff to favor molecules with better properties.

Diffusion-DPO uses the forward process to approximate sample molecules efficiently. During fine-tuning, there might be slight deviations, which could affect this approximation. The steps at the start of the trajectory affect those at the end, but not the other way around.

Thus, we propose a beta schedule that emphasizes optimization at the end of the trajectory. This keeps the Diffusion-DPO approximation reliable while allowing enough flexibility to optimize the model effectively.

Q2: The DecompDPO model trained for 2 epochs seems to perform better. Why is it not trained even longer?

A2: Please refer to our response to Q3 in Weakness.

Thank you once again for your detailed feedback. Please let us know if you have any further questions and suggestions.

[1] Reidenbach, D. (2024). EvoSBDD: Latent Evolution for Accurate and Efficient Structure-Based Drug Design. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations.

Thank you for your prompt feedback. We are pleased to hear that our rebuttal has addressed many of your previous questions and concerns. Regarding your remaining concerns, we would like to them as follows:

Weaknesses

Q1 & Q2:

Re: "I would also recommend to include either complete rate or PB-valid directly in the main tables to present this data more transparently."

Thank you for your suggestion. We have moved the Complete Rate to the table in the main text and will update in the revised version of paper.

Re: "However, my main concern about DecompDPO's performance is not about the statistical significance of the results but rather about absolute differences. As I see it, all the presented performance gains are negligible because some baselines perform as well or better than DecompDPO (e.g. Pocket2Mol on QED and SA) even though they are not directly aiming to optimize these properties whereas DecompDPO is. Even the differences to its own base model, DecompDiff, appear to be too small to make a difference in practice."

We would like to emphasise a few key points to address concerns about the performance of DecompDPO:

While Pocket2Mol achieves higher QED and SA scores, its performance on binding affinity-related metrics is notably lower compared to diffusion-based models. This discrepancy highlights an inherent difference between autoregressive models and diffusion models. Autoregressive models tend to excel in chemical property optimization, such as QED and SA, whereas diffusion models are more adept at generating molecules with higher binding affinities.

Therefore, evaluating model performance solely based on a single metric like QED or SA does not provide a complete picture. We believe that the Success Rate, which reflects the overall balance of desirable properties, is a more meaningful indicator of a model's ability. DecompDPO consistently demonstrates a higher Success Rate compared to baseline models, validating its ability to generate high quality molecules.

Furthermore, we would like to reiterate that DecompDPO achieves improvements across multiple metrics, including QED, SA, and all binding-related metrics. Its ability to balance these properties while demonstrating remarkable performance in binding optimization reinforces its utility in SBDD and highlights DecompDPO's potential in optimizing models to generate better molecules.

Re: "Unfortunately, the newly added baselines (e.g. KGDiff) do not strenghten the claims of the paper either. Importantly, DecompDPO does not perform better than this baseline in the Vina Minimize metric which it is explicitly optimising for. I consider this the core metric as it aligns best with the training objective."

Thank you for your comment. We would like to kindly remind the reviewer that, as addressed in Q1 of Questions Part 1, the results from DecompDPO fine-tuned for two epochs (a total of 30k steps, matching the fine-tuning steps used in AliDiff and KGDiff) demonstrate that DecompDPO achieves higher performance on Vina Minimize. For your convenience, we reorganize the results below:

As the table shows, DecompDPO fine-tuned for two epochs achieves higher Vina Minimize, Vina Dock, and High Affinity compared to AliDiff and KGDiff, confirming the effectiveness of DecompDPO in aligning with the core optimization objective.

Dear Reviewer HgKr,

We are grateful for your insightful comments and the time you've taken to review our submission.

As the deadline for the discussion phase approaches, we kindly ask if you could review our responses to your comments. Please let us know if there are any additional questions or concerns we can help clarify.

We greatly appreciate your time and are committed to addressing any remaining concerns you may have.

Sincerely,

The Authors

Q1: Provide complete rate for DecompDPO and DecompDiff

A1: Thank you for your question. We report the Complete Rate, along with Novelty, Similarity, and Uniqueness in Table 7 in the original paper. We have further checked the sanity of generated molecules using PoseBuster. We computed the percentage of valid molecules (PB-Valid) checked by the PoseBuster [1] and reported the result in the following table:

As the table shows, DecompDPO achieves slightly improvement compared to DecompDiff in terms of Complete Rate and PB-Valid, confirming its effectiveness in generating valid molecules.

[1] Buttenschoen, M., Morris, G. M., & Deane, C. M. (2024). PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences. Chemical Science, 15(9), 3130-3139.

Q3: Effectiveness of Local DPO

A3: Thank you for pointing out this concern. Both Reviewer fiLe and Reviewer bSNV raised questions about the effectiveness of LocalDPO, and we appreciate the opportunity to clarify its advantages. In multi-objective optimization, gradients from different objectives may conflict, potentially hindering the optimization process and limiting the model's ability to improve all desired properties simultaneously. This conflict can make the benefits of LocalDPO less apparent in certain evaluations.

To better illustrate the effectiveness of LocalDPO, we considered a more focused setting in binding affinity optimization. In this context, AliDiff can be regarded as employing a variant of GlobalDPO enhanced with Exact Energy Preference Optimization. To ensure a fair comparison, we extended the training of our model by an additional epoch, totaling 30,000 fine-tuning steps. The results are as follows:

As the results show, with LocalDPO optimizing Vina Minimize, our model achieves very significant improvements across binding affinity-related metrics, including Vina Score. The Complete Rate after 2 epochs of optimization is 77%. These results demonstrate LocalDPO's superior performance in binding optimization without sacrificing general molecular properties, highlighting its unique advantage in effectively navigating the optimization with decomposed perference.

Thank you for your valuable feedback. Please see below for our responses to the comments.

Q1: Comparsion with KGDiff

A1: Thank you for your suggestion! We noticed that KGDiff reported the performance of binding knowledge guided generation benchmarked on CrossDocked2020. So we add the comparision with KGDiff in Single-Objective Optimization section.

As shown in the table, DecompDPO achieves the highest Vina Dock and High Affinity compared to AliDiff and KGDiff without sacrificing QED and SA, indicating its effectiveness. We will be more cautious about the pharsing when stating the performance of DecompDPO in both the Main Result section and the Single-Objective Optimization section.

Q2: Performance on Vina Score

A2: Thank you for bringing up this concern. We appreciate the opportunity to clarify our experimental choices and results regarding the Vina Score. In our experiment, Vina Minimize is selected as optimization objective rather than Vina Score, so the optimization does not prioritise Vina Score. There are several reasons for this choice: First, in SBDD, which is still in the early stages of pharmaceutical development, docking related metrics optimized by force field are crucial indicators for assessing the success of designed molecules, as it evaluates pose validity for downstream applications. Vina Minimize was chosen as optimization objective because it effectively reflects the binding affinity between the generated molecule and the protein without incurring high computational costs. Besides, in multi-objective optimization tasks, gradients from different objectives can conflict, potentially hindering overall performance. By focusing on Vina Minimize, which enables local optimization of atom positions, DecompDPO allows for greater flexibility and mitigates gradient conflicts.

We want to emphasize that even though DecompDPO optimized towards Vina Minimize, which not prioritise the optimization towards Vina Score, the median Vina Score still showed a significant improvement compared to DecompDiff.

To further illustrate the effectiveness of DecompDPO in optimizing binding affinity, including Vina Score, we extended the training for an additional epoch, totaling 30,000 fine-tuning steps, aligning with AliDiff. The experiment result is provided in Q3. As the result shows, DecompDPO with longer training achieves higher Vina Score compared to AliDiff, but with a slight trade-off in QED. These results suggest that DecompDPO can effectively optimize for higher binding affinity, but careful consideration is needed to balance trade-offs between optimization objective and general molecular properties during optimization.

Dear Reviewer bSNV,

Thank you for the time and effort you've invested in evaluating our submission.

In response to your valuable suggestions, we have added a comparative analysis with KGDiff and conducted additional experiments to demonstrate the effectiveness of DecompDPO in optimizing binding affinity.

With the discussion phase coming to an end, we wanted to check if there are any further concerns we can address. If our responses have addressed your concerns, we kindly hope you might reconsider your evaluation of our submission.

Sincerely,

The Authors

Q5: Use physics-informed energy terms as additional penalty loss

A5: Thank you for your question. We had conducted an experiment where we used physics-informed energy terms as additional penalty losses during the fine-tuning. In this setup, the physics-informed terms were directly applied as penalty losses to the output of the fine-tuning model, rather than being used as constraints for penalizing the DPO rewards.

As the result shows, using physics-informed energy terms as penalty losses leads to improvements over the baseline model, but does not perform as well as our proposed method where these terms are used as reward penalties in optimization. We also compared the models in terms of molecular conformation:

Using physics-informed energy terms as penalty losses resulted in lower energy differences and slightly better RMSD values, indicating better preservation of molecular conformations. We hypothesize that this occurs because applying the energy terms as penalty losses more directly penalizes unreasonable outputs, but at the expense of reduced optimization effectiveness toward the desired objectives.

Q6: How do authors select the size of the sampled molecules? Is the distribution of sizes in samples the same with and without DPO?

A6: Thank you for your question. For both molecule generation and molecular optimization, we employ the same Opt Prior used in DecompDiff, ensuring the same samples size with and without DPO. Opt Prior is defined as a mixture of Ref Prior, which is determined by the reference ligand, and Pocket Prior, which is defined by a prior generation algorithm using AlphaSpace2 [3] toolkit, depending on whether Ref Prior passes the Success threshold (QED > 0.25, SA > 0.59, Vina Dock < −8.18). For Ref Prior, the generated molecule size is the same as the reference molecule, and for Pocket Prior, the generated molecule size is determined by the size of the detected pocket. We have added the details of molecule size selection in the Experimental Details section in Appendix.

[3] Katigbak, J., Li, H., Rooklin, D., & Zhang, Y. (2020). AlphaSpace 2.0: representing concave biomolecular surfaces using β-clusters. Journal of chemical information and modeling, 60(3), 1494-1508.

Q1 & Q2: How are the fragments without rotatable bonds computed? In Figure 4 (right), what does fragment size on x-axis mean?

A1: Thank you for your question. In evaluating the median RMSD and energy difference of rigid fragments, we first fragmentation each molecule by rotatable bonds. Then we group rigid fragments by their nubmer of heavy atoms and calculate the median RMSD and energy difference. In Figure 4 (right), the x-axis is the number of heavy atoms contained in rigid fragments.

Q3: Distributions of the scores in winning and losing examples in the created dataset for preference alignment.

A3: Thank you for your question. We have further provided winning and losing molecules' distribution of QED, SA, and Vina Minimize in Figure 13 in the Appendix.

Q4: Clarify the beta schedule.

A4: Thank you for your question. Yes, in our linear $\beta$-schedule, we set a large $\beta$ when $t$ is large to ensure strong regularization from the reference model during the initial stages, maintaining stability and adherence to the learned data distribution. As $t$ decreases, moving toward the later stages of denoising, we gradually reduce $\beta$. This smaller $\beta$ allows for more aggressive optimization guided by the specific objectives, enabling the model to align more closely with the desired properties. This strategy effectively balances the influence of the reference model with the optimization goals throughout the denoising process.

Q9: Report evaluation of the number of steric clashes. Report binding efficiency scores such as docking scores divided by the number of heavy atoms.

A9: Thank you for your suggestion. To provide a more comprehensive evaluation, we benchmarked DecompDPO using PoseCheck [1]. We reported the quartiles of Strain Energy (SE), the average number of Steric Clashes, and the number of key interactions, i.e., Hydrogen Bond Donors (HB Donors), Hydrogen Bond Acceptors (HB Acceptors), van-der Waals contacts (vdWs) and Hydrophobic interactions (Hydrophobic).

As shown in the table, DecompDPO achieves better Strain Energy compared to TargetDiff and DecompDiff. Regarding Steric Clashes, although DecompDPO performs slightly better than DecompDiff, we acknowledge that there are a few targets with extreme clashes due to anomalies beta priors detected by auto process. Addressing this issue is part of our future work.

In terms of binding efficiency, while the number of heavy atoms in molecules relates to binding affinity, their correlation is not strictly linear. To better illustrate the potency of the generated molecules, we further drew scatter plots of heavy atom numbers versus Vina Dock scores for DecompDPO and baseline models. As Figure 10 in Appendix shows, DecompDPO generally achieves better Vina Dock scores compared to DecompDiff for molecules with the same number of heavy atoms.

Q10: Report validity, connectivity, and percentage of the samples that pass PoseBusters filters.

A10: Thank you for your question. In the original paper, we report the Complete Rate, which is defined as the percentage of valid and connected molecules, in Table 7. To further evaluate the performance of DecompDPO, we computed the percentage of valid molecules (PB-Valid) checked by the PoseBusters [2].

As the table shows, DecompDPO achieves slightly improvement compared to DecompDiff in terms of Complete Rate and PB-Valid.

[1] Harris, C., Didi, K., Jamasb, A., Joshi, C., Mathis, S., Lio, P., & Blundell, T. (2023). Posecheck: Generative models for 3d structure-based drug design produce unrealistic poses. In NeurIPS 2023 Generative AI and Biology (GenBio) Workshop.

[2] Buttenschoen, M., Morris, G. M., & Deane, C. M. (2024). PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences. Chemical Science, 15(9), 3130-3139.

Q7: Have another table in the main text with the key geometric scores: JS divergence between distance, JS divergence between angles, Avg. conformer energies.

A7: Thank you for your suggestion. We have further summarized JSD of pairwise distance distributions provided in Figure 4 (left), and RMSD and energy difference of rigid fragments and the whole molecule provided in Figure 4 (right), 6, 7, 8 in the following table:

We have added Table 7 in the Appendix to provide numerical evidence in addition to the distributional evidence for molecular conformations and will adjust the layout in the future version. For JSD between bond distance and angles, we provide the evaluation results in Table 5 and 6 in Appendix.

Q8: Try physical constraints with $\sigma$ threshold

A8: Thank you for your insightful question. We have further conducted an experiment using physical constraints with $\sigma$ threshold for multi-objective optimization.

As we emphasized before, the physical constraints are proposed for penalizing unreasonable conformations during preference alignment. The results show that the stricter physical constraints with $\sigma$ threshold basically does not affect optimization performance, with a very slight drop in binding related metrics, and leads to lower energy difference for both rigid fragments and generated molecules compared to $3\sigma$ threshold.

Q5.1: How to explain the improvements of QED and SA when optimizing for binding affinity?

A5.1: Thank you for your insightful questions. Typically, optimizing a model toward a specific objective like binding affinity can lead to trade-offs with other molecular properties such as QED and SA. For instance, when AliDiff is optimized for binding affinity, both QED and SA scores tend to decrease. However, we observed that when DecompDPO is optimized solely for binding affinity, the QED and SA scores actually improve, which is unusual in single-objective optimization. To explain this, we calculated the Pearson correlation coefficients between the differences in Vina Minimize scores and the differences in QED and SA scores for each pair in our dataset. We found that the correlations between differences in the decomposed arms' Vina Minimize and the corresponding molecules' differences in QED and SA are -0.12 and -0.11, respectively. These statistics suggest that the improvements in QED and SA are potentially due to the benefits of LocalDPO, where the flexibility introduced by the decomposed chemical space alleviates conflicts between SA and binding affinity, allowing the model to navigate the chemical space effectively even when optimizing for binding affinity. Besides, in multi-objective optimization, the gradients of different objectives may conflict with each other, leading to less effective results, which could also affect the optimization performance of QED and SA.

Q5.2: The single objective optimization result for QED and SA.

A5.2: Thanks for your questions. We have further conducted experiments optimizing QED and SA separately, the results are shown below. As QED and SA are non-decomposable, we used GlobalDPO for the optimizaiton.

As shown in the experimental result, DecompDPO achieves 6.7% and 10% in the corresponding objective for QED and SA optimization, respectively. As we discussed in Q5.1, there are positive side effects between QED, SA, and Vina arise from the correlation in our dataset, so the binding affinity is slightly improved. Overall, the single objective optimization experiments demonstrate the effectiveness of DecompDPO, especially in the setting that using LocalDPO to optimize Vina Minimize.

Q5.3: Discussion of connectivity and validity in Single-Objective Optimization.

A5.3: Thank you for your question. In Section 4.3 Single-Objective Optimization, we reported the Complate Rate to demonstrate that the improvement of molecular properties is not at the expense of the Complete Rate. In Table 7, we have reported the Complate Rate, Novelty, Similarity, and Uniqueness of DecompDPO to demonstrate its ability in designing novel and valid molecules.

Q6: Put Figure 4 (left) to appendix. Typo in Line 472: Table 4 -> Table 3.

A6: Thank you for pointing this out! We will carefully proofread our manuscript and revise these.

Thank you for your constructive feedback. Please see below for our responses to the comments.

Q1: In Table 4, the contribution of LocalDPO is not evident. The current evaluation does not highlight the advantages of LocalDPO and its unique feature – locality. Why should LocalDPO be used instead of GlobalDPO?

A1: Thank you for bringing up this concern. We appreciate the opportunity to further clarify the advantages of LocalDPO. In multi-objective optimization, the gradients for different objectives may conflict with each other, potentially hinder the optimization process and limit the model's ability to improve all desired properties effectively, resulting in the advantage of LocalDPO not highlighted very significantly. The effectiveness of LocalDPO could be better illustrated by comparison with AliDiff, which can be regarded as employing a variant of GlobalDPO enhanced with Exact Energy Preference Optimization to improve results. To ensure a fair comparison, we extended the training of our model by an additional epoch, totaling 30,000 fine-tuning steps. The results are shown as follows:

As the result shows, with LocalDPO optimizing Vina Minimize, our model achieves significant improvements across binding affinity related metrics while not sacrifycing molecular properties. This demonstrates LocalDPO's superior performance in binding optimization and highlights its unique advantage in effectively navigating the optimization with decomposed perference.

Q2: Tables 5 and 6 suggest that DPO doesn't improve the geometry of the molecules.

A2: We appreciate the opportunity to further clarify the motivation of our physically-constrained optimization. Indeed, as we stated in Section 3.3, the purpose of introducing physics-informed energy terms in preference alignment is to maintain reasonable molecular conformations instead of optimizing the geometry of generated molecules. Out of this consideration, we only penalize unreasonable conformations that digress away from $3\sigma$ threshold. In Table 5 and 6, we demonstrate that the physically-constrained optimization generally achieves comparable performance with DecompDiff in terms of molecular conformation, which is align with the motivation.

Q3: Calrify the objectives used for QED, SA, and Vina Minimize Score regarding LocalDPO and GlobalDPO

A3: As stated in Section 3.2 DECOMPDPO, we use LocalDPO or GlobalDPO according to each optimization objective's decomposability in multi-objective optimization. The total loss is a weighted sum over all loss terms for different objectives. In experiment, we used GlobalDPO for QED and SA since they are non-decomposable, and used LocalDPO for Vina Minimize optimization.

Q4: The argument for improved molecular conformation is unsubstantiated.

A4: Thank you for your detailed review and highlighting this point. As we stated in Q2, as preference alignment inevitably optimizes model distribution towards high rewards and away from the original distribution, our initial purpose of introducing physical constraints is to maintain reasonable conformations during optimization. For the sentence in line 431, our intent is to illustrate the experiment result we observed from Figure 4 (right), with DecompDPO achieves lower energy difference for rigid fragments with more heavy atoms. We will be more cautious about our phrasing and have revised this sentence in the paper.

Dear Reviewer fiLe,

We sincerely appreciate your detailed and constructive comments on our submission.

In our response, we have further clarified the motivation and advantages of DecompDPO, provided a more comprehensive evaluation, and conducted additional experiments as you suggested.

As the discussion phase is nearing its conclusion, we would greatly appreciate any additional feedback you might have. If we have properly addressed your concerns, we kindly hope you might reconsider your evaluation of our submission.

Sincerely,

The Authors