Harmonic Prior Flow Matching for Multi-Ligand Docking and Binding Site Design

Thank you for the detailed comments and the clear distillation of several strengths of our method, as well as your two suggestions that we address below!

We are glad to see your appreciation of the novelty of the binding site design task in the ML community and the novelty of our joint flow over discrete and continuous data, considering that there has not been any discrete flow matching work yet.

Weakness:

“There are many other tricks used for training FlowSite and HarmonicFlow, such as many auxiliary losses similar to those in Alpafold2. But there is no related ablation study.”

• We agree that our ablations focused on HarmonicFlow and thus added the following additional ablations for both FlowSite and HarmonicFlow that show how all components are relevant for performance:

• FlowSite binding site design (Table 5):

1. Dropping the auxiliary side-chain torsion angle loss
2. Dropping the fake ligand data augmentation
3. Using equivariant instead of invariant layers throughout the architecture
4. Dropping the auxiliary refinement loss of the Equivariant Refinement TFN layers
5. Adding noise to the backbone coordinates

• 3D structure generation with Harmonic Flow (Table 4):

1. Using isotropic Gaussian prior instead of harmonic prior
2. Dropping the auxiliary refinement loss of the Equivariant Refinement TFN layers

Questions:

“Is there any post hoc method to generated side-chains for the designed binding site?”

A great question! Multiple methods exist for the task of sidechain packing; however, they commonly are not aware of the ligand. In the updated paper, we improved our discussion in section 4.3 on binding affinity evaluation with respect to how, e.g., multiple residue types might be able to bind a particular ligand. We attempted to address this by developing “BLOSUM score,” which takes evolutionary and physicochemical similarity into account. Ultimately, we think a final evaluation has to be biological validation.

Thank you for the productive discussion so far! We hope that the improvements might warrant raising your score and we are happy to continue the conversation. Please also find other significant improvements in our general response above and have a great week!

We authors thank you for your initial review and would be very grateful for any further discussion or acknowledgment as to whether any concerns remain after our improvements and responses!

Thanks for your response! Thanks for providing the extensive ablation results. However, the unavailability of the side-chains of the generated pocket is a weakness, which is also a concern of other reviewers. Due to this limitation, it may be hard to fairly or properly compare FlowSite and HarmoicFlow with some of the baselines. Experiments in [1] show that side chains play an important role in designing protein pockets. The technical novelty is limited based on the comments of Reviewers XYXw and k1W9. Considering the above two points, I decided to change my rating to 5.

References:

[1] Zhang, Zaixi, et al. "Full-Atom Protein Pocket Design via Iterative Refinement." NeurIPS 2023 Spotlight.

Thank you for the constructive questions that we now better address in the updated text and below! We hope the improvements warrant accepting the paper and are happy to make any further changes.

1, First bullet: “The problem of multi-ligand binding is interesting, but I am not sure how important this task is [...] more detailed explanations can be helpful”

• Multi-ligand binding is indeed an important problem since it is crucial for, e.g., fragment-based drug design, and in our context, it is of interest for enzyme design where we would desire to design a binding site for multiple reactants / starting molecules of a chemical reaction that the enzyme should catalyze.

2, Second bullet: “the authors compared HarmonicFlow with DiffDock, but they are solving different problems: one for multi-ligand docking and the other for protein-ligand (both single- and multi-ligand) docking.”

• We apologize for the following being unclear in the text and improved our description: The current implementation of DiffDock can actually only do single ligand docking. We compare HarmonicFlow (which can do both single- and multi-ligand docking) with DiffDock’s product space diffusion on a single-ligand benchmark. HarmonicFlow is not specific to only multi-ligand docking. Meanwhile, for our multi-ligand experiments, we compare with the diffusion process of EigenFold in Table 4. We hope the updated manuscript makes this sufficiently clear and welcome any further feedback!

3, Third bullet: “Does this vector field u in Eq 1 have any physical meaning?”

• An excellent question! In general we want to be careful with such physics interpretations. The direction the vector field points in is where the expected x_1 is given that we currently are at x_t. At times very close to t=1 this can become similar to the direction of the forces on the atoms if our training data is from the Boltzmann distribution of the molecules.

4.1 Fourth bullet first part: “summarize the essential difference between score matching / denoising diffusion with the objective (Eq 1&3) in this paper?”

• Another great question in our opinion! Flow matching and denoising diffusion are indeed similar, with flow matching being a generalization of diffusion models. The quantity $u_t(x | x_0, x_1)$ that we regress against in equation 3 is a conditional vector field that points to a data sample (obtained by, e.g., the vector $x_1 - x_0$). Meanwhile, in diffusion models, we regress against the conditional score, which is a vector pointing toward the steepest ascent in a noised distribution. This way, in flow matching, we learn a drift that we can integrate to sample while in diffusion models, we have a predefined arbitrary drift that we have to combine with the learned score to sample. Please let us know if we can clarify this any further and if any questions remain!

4.2 Fourth bullet second part: “is the proposed HarmonicFlow specific for multi-ligand docking?”

• HarmonicFlow is not specific to multi-ligand docking and can also be used for single-ligand docking which we hope is now better made clear in the updated text!

5, Fifth bullet: “Many baselines, such as the related works mentioned in Sec 2, are missing in the experiments.”

• We adapt our related work section and baseline explanation in the experiments to better reflect the following:

• With respect to HarmonicFlow: the works in section 2 (apart from DiffDock’s diffusion process, which we compare with) all require sidechain positions of the protein as input which is not available in our final application of interest — binding site design — where we aim to predict the side-chain identities.

• With respect to FlowSite: of the mentioned inverse folding methods, we chose PiFold since it currently has the state-of-the-art recovery rate.

Thank you for the great questions! We hope the improvements that arose from them might warrant raising your score and we are happy to continue the conversation. Please also find other significant improvements in our general response above!

We authors thank you for your initial review and would be very grateful for any further discussion or acknowledgment as to whether any concerns remain after our improvements and responses!

I appreciate the authors for the detailed replies to each of my questions.

However, some of the concerns remain:

1. The architecture proposed here (with flow matching) is not specific to the multi-ligand docking problem.
2. The algorithm has no knowledge of the physics. I read more literature papers in the past few days, especially the ones cited in the related work section. I think this is the issue of the whole research line: all the existing papers claim that they are handling the docking problem, which is a physical process. But indeed, only statistics modeling of the data distribution is considered.

I also appreciate the authors for the detailed response. Thus, combining all the pros and cons, I would like to keep my score.

We appreciate your time taken for reviewing and are glad you find the fake ligand augmentation interesting and see the ablation studies as a strength.

Regarding the Dual submission ethics concerns:

• We think there might be a confusion about the GenBio workshop having proceedings. The ICLR 2024 dual submission policy (https://iclr.cc/Conferences/2024/CallForPapers) states, “papers that [...] have been presented at workshops (i.e., venues that do not have publication proceedings) do not violate the policy.” Does this address your Ethics concerns?

Weaknesses:

1, “Overall the paper is much of a summarization of known techs”

• We argue that there are 3 critical points of novelty that might be overlooked. We provide the first flow matching process to jointly generate discrete and continuous data, which we consider a novelty given that there is no work on discrete flow matching yet. Secondly, we think next to technical novelty, the novelty of the application is also important - our work is the first to tackle binding site design with deep learning. Thirdly, ours is (among another ICLR submission) the first to investigate flow matching for biomolecular structures, which exhibit (also in diffusion models) different behavior than CV.

2, “I'd expect some impressive case studies of multiple docking [...] I would suggest the authors be more focused”.

• Thank you for the suggestion! We now included several case studies that were randomly picked from the test set and show how HarmonicFlow produces more realistic bond lengths and bond angles and respects physical constraints better. We acknowledge that our work indeed does not focus on multi ligand docking - we describe the problem and evaluate it since we see it as important for binding site design when the goal is to design binders for multiple molecules, such as reactants in an enzyme. In our updated version, we clarify this focus - we are happy to also change the title if you think that is a better description of the work.

3, “Backbone flexibility is not allowed in the entire picture. All experiments are done on PDBBind, which provides accurate holo-structures of proteins.”

• We agree that for a full docking pipeline the performance on apo structures is interesting next to the holo performance. In our work, we evaluate docking performance only for the purpose of binding site design as required for our task definition which we chose for its alignment with the state-of-the-art traditional binding site and enzyme design approach of (https://www.nature.com/articles/s41586-023-05696-3) where the starting point is also a fixed backbone structure that is not altered.

4, “Baselines are weak.”

• We added another new stronger baseline to Table 3 based on the anonymous ICLR 2024 submission DiffDock-Pocket [1]. It operates similar to our ground truth position comparison model, except that the fixed positions are given by a sample from DiffDock-Pocket, a recent state-of-the-art pocket level docking tool. We note that, as we are the first to consider this task, there are no existing baselines for it yet.

Questions:

1, Are sidechains configurations by any means included as data inputs?

• They are not used as inputs and only a quantity that we predict as auxiliary target.

2, “I would expect some analysis of the ODE dynamics”

• Thank you for the suggestion! We now included a plot (Figure 11) of the evolution of FlowSite's x_1 prediction's RMSD and the evolution of the output entropy of the residue type probabilities. The results show how a more determined structure prediction correlates with a decrease in uncertainty of the residue type prediction.

Thank you for the constructive conversation! We hope the improvements that arose from your suggestions might warrant raising your score and we are happy to continue the discussion.

Thank you for the clarification of ethical concerns and other issues.

1. the ethical concerns are resolved.

2. thanks for the ode analysis which basically solves my concerns in the black-box issue about the ODE.

3. the analysis of binding site design and multiple ligand docking is addressed, but the binding site design part is not well benchmarked against other inverse folding / inpainting baselines beyond PiFold. In fact, I would expect some larger elevation in the recovery rate because most of the sequences are given. The visualization results themselves are also not convincing to display the model's accuracy.

4. about the concern “overall the paper is much of a summarization of known techs”, the author's reply still does not resolve my concerns. Indeed applications of techs transfering from one domain to another is interested, but current results are far from being strong enough to persuade a drug designer in believing the outputs from the model.

overall I would give this a 5.

Thank you for the time taken to review our work and for the excellent questions that helped improve the paper!

We understand how these questions arose - we updated the text to contain the following answers to your questions. We hope this makes the paper easy to follow and the experiment setup sufficiently clear. Please let us know if further changes are required.

1.1 Question: “Why do you think that the quality of docking tells much about the quality of side-chain prediction (DiffDock can perform this task without this information)?”

• A great question! While DiffDock does not use side-chain atom positions, it does use the residue identities as features, which improves performance as the experiments without residue identities in Table 5 show. We are confident that a good structure prediction aids residue-type prediction since the recovery rate is 51.4% when using the ground truth ligand positions as input compared to 41.8% when using random ligand positions.

1.2 Regarding the concern of comparing with DiffDock’s product space diffusion, which was originally developed for Blind docking.

• This is an important concern! However, the work “DiffDock-Pocket: Diffusion for Pocket-Level Docking with Sidechain Flexibility” (https://openreview.net/forum?id=1IaoWBqB6K) now shows that the product space diffusion employed in a similar way as in our work is a strong pocket level docking method (although they use side-chain atom positions which are not available in our application). We now cite this paper to clarify this.

2, Regarding using amino acid recovery for known pockets as metrics and that there can be multiple right answers, not only the residues in the ground truth data.

• This is correct and an important point! The recovery metric has been shown to be a good proxy for downstream performance in the inverse folding literature (https://www.science.org/doi/10.1126/science.add2187 PMPNN, https://arxiv.org/abs/2306.16819 GRADEIF) and we try to improve over it with our BLOSUM metric which allows for multiple solutions by taking evolutionary and physicochemical similarity of residues into account. We now better discuss your concern in the 4.3 Metrics paragraph, and in our conclusion, we acknowledge how, ultimately, biological validation is required to fully evaluate, which is ongoing work.

3, Question “Have you tried to sample many different noise vectors and compare the outputs?”

• We indeed generate 10 samples per complex and evaluate the average quality. In the updated text, we now also provide the standard deviation over these samples, which is 1.16 Angstrom RMSD for Distance-Pockets on the time split (similar for all settings) and 2.5% for the recovery rate and 1.8 Blosum score on PDBBind.

4, Estimating binding free energy or using DiffDock’s confidence model as proxy.

• Our discussion of the following was unclear in the text, and we improved it in the update: DiffDock’s confidence model and common computational binding affinity correlates would require the ligand and side chain atom positions as input, which none of the methods (except ours) produce. Thus, we could only produce a number for FlowSite with no comparison. Further, DiffDock’s confidence model was never trained on negatives, meaning that it cannot distinguish between binders and non-binders, and there is no evidence for it as a correlate for binding affinity.

5, “you mentioned the oracle method that has access to the ground truth ligand structure. What exactly is this method?”

• Thank you for pointing out our insufficient explanation, which we now improved! This is the same architecture as FlowSite, but instead of using the predicted ligand positions, the ground truth crystal structure ligand positions are used as input and not altered. In practice, this information would not be known during inference.

6, Question: “How much time does it take to train your model on PDBBind and MOAD?”

• We now included this in the paper: on PDBBind, FlowSite took 58.6 hours, and on MOAD 115.6 hours. Both on an RTX A600 GPU.

Thank you for the productive discussion! We hope that the improvements might warrant raising your score and we are happy to continue the conversation and make further changes. Please also find other significant improvements in our general response above!

We authors thank you for your initial review and would be very grateful for any further discussion or acknowledgment as to whether any concerns remain after our improvements and responses!

Thank you for the positive review! We are glad you find the improvements significant and appreciate our discussion of FlowSite’s limitations.

Weaknesses:

“Most of the technology is just an application of the existing methods which prevents the paper from getting higher scores in machine learning conferences.”

We think our work provides 3 main important pieces of novelty: 1. We provide the first flow matching process to jointly generate discrete and continuous data, which we consider a novelty given that there is no work on discrete flow matching yet. 2. We think next to technical novelty, the novelty of the application is also important - our work is the first to tackle binding site design with deep learning. 3. Ours is (among another ICLR submission) the first to investigate flow matching for biomolecular structures.

Questions:

“How fast are the two methods in terms of inference speed?”

An excellent question and we provide these details in the updated manuscript! Averaged over 4350 generated samples, the average runtime of HarmonicFlow is 0.223 seconds per generated structure and that of FlowSite is 0.351 seconds per sequence.

Thank you for the discussion so far! We hope the clarifications and our other significant improvements in our general response might warrant raising your score!

We authors thank you for your initial review and would be very grateful for any further discussion or acknowledgment as to whether any concerns remain after our improvements and responses!