Generating Molecular Conformer Fields

We thank the reviewers for the constructive reviews. Please find below our detailed responses to the weaknesses and questions:

1. Limited novelty relative to DPF paper

   • In this work we generalize the problem setting in DPF to deal with functions on graphs (as opposed to functions on Euclidean spaces). Our proposed approach is conceptually simple to implement yet very powerful in its application. Note that MCF sets the current state-of-the-art for conformer generation. We believe that approaches that are both simple to implement and powerful in their empirical results tend to have the longest lasting impact in the field.

2. Several claims are not very well substantiated

   • What is the cost of enforcing domain-specific inductive biases (e.g. periodic domain of possible torsional angles in Torsion Diff)

     • Torsional diffusion requires breaking molecules into fragments connected by rotatable bonds. This process relies on rule-based methods that can fail, especially for complex molecules. For example, in our replica of torsional diffusion on GEOM-XL using their public codebase, we only successfully generate conformers of 77 molecules out of 102 in the database. Besides, torsional diffusion utilizes cheminformatic methods to predict the local 3D structures. Therefore, the accuracy of the model is dependent on the local models and may neglect effects of long-range interatomic interactions from other substructures. We have further clarified the cost in Introduction section.
     • Furthermore, we trained an MCF model with updated hyper-parameters as reflected in Tab. 4. We applied updated latent dimensions as well as number of latents and trained the model for more epochs. This model now obtains state-of-the-art performance, outperforming torsional diffusion on challenging the GEOM-DRUGS and GEOM-XL datasets (shown in updated Tab. 2 & 3).

   • Is it really more scalable than Torsional Diffusion?

     • First, more scalable means MCF operates directly on raw data in 3D Euclidean space without the domain-specific efforts to pre-processing the data. Such pre-processing can lead to failure cases for complex molecules and forces the conformer generation adhered to other substructure models. These priors can prevent the models from scaling up to larger and more complex molecular systems. In addition, MCF is built upon PerceiverIO, an attention-based Transformer model. PerceiverIO includes a latent array that cross attends with input context and output query. In our experiments, we set the number of latent as 512. For a molecule with 100 atoms, the model has degree of freedom up to 512 which is much greater the number of atoms. Both aspects provide the flexibility of our proposed MCF to scale up.

   • The paper claims that the ablation study shows what factors are important for molecular-conformer generation, but this is a pretty grandiose claim.

     • We thank the reviewer for pointing this out, this is a misscommunication that is clarified in the updated version of the paper. Our claim is to show what implementation factors and node features are important for MCF to obtain a good performance. We have rephrased the claim to clarify our contributions.

   • The paper claims that explicitly enforcing roto-translation equivariance is not a strong requirement for generalization, and that equivariance is not particularly useful for molecular-conformer generation. Although the authors have shown similar and comparable performance between an equivariant and non-equivariant architecture, it is not yet clear whether or not equivariance really affects performance. For example, one could ask whether MCFs could outperform Torsional Diffusion if it had used a roto-translationally equivariant architecture (not just PerceiverIO).

     • We have updated Tab. 2 after training an MCF model with updated hyper-parameters (see Tab. 4). MCF now surpasses all baseline models by a large margin on GEOM-DRUGS. The latest results further demonstrate that roto-translational equivairance is not a strong requirement for achieving superior performance on molecular conformer generation. Our model, which is built on a general-purposed Transformer-based architecture, can achieve better performance than other models which are roto-translational equivariant.
     • Also, we would like to underline that to the best of our knowledge, our work is the first to highlight the question whether equivariance is required for solving molecular conformer generation problem. Previous works have focused on building equiavriant neural networks and baking inductive biases (e.g. modeling torsional angles) for conformation generation. Indeed current work doesn’t completely answer the question whether equivariance is needed for conformer generation. But we hope this work can shed a light on future investigations of the role equivariance plays in such tasks.

------- see following comments below -------

We thank the reviewer for engaging with the discussion and bringing up valuable comments. Please find the our detailed replies below:

1. The cost of enforcing domain-specific inductive biases

• Torsional diffusion relies on cheminformatic methods to find the rotatable bonds and predict the 3D structure of substructures. As the reviewer mentioned, such cheminformatic methods can fail in some cases, especially for large molecules. In particular, in GEOM-XL dataset, which contains large molecules with more than 100 atoms, Torsional Diffusion only successfully generates 77 out of 102 molecules when reproduced using their public codebase. While MCF can generate conformers of all molecules without failure cases. This provides an example of the cost of domain-specific inductive biases and the advantages of MCF with simple architecture and training recipe. MCF achieves better performance than Torsional Diffusion on GEOM-XL while successfully generating conformers for all molecules. To the best of our knowledge, our work is the first to raise the question of whether strong inductive bias is required in achieving superior performance on molecular conformer generation.

2. Scalability compared to TorsionalDiff

• First, as discussed in the previous question, MCF can generate conformers for large molecules where Torsional Diffusion. This indicates the capability of scaling MCF to large molecules. Second, in terms of computational cost, we have added the experiments of using efficient sampling technique (i.e. DDIM) in Appendix of updated manuscript. It shows that using the same 20 denoising steps (which is the same number of steps as main results reported in Torsional Diffusion), MCF achieves significantly better performance than Torsional Diffusion. Under this setting, MCF takes approximately 0.5s on one A100 GPU and 100s on a CPU to sample one conformer in GEOM-DRUGS. Given the common accessibility of GPUs, we believe benchmark on GPU time is a reasonable practice, which is much faster than acquiring the ground truth conformers using DFT. Also, it should be noted that the benchmark was conducted with batch size 1. In practice, one can always stack multiple molecules as a batch and further improve the sampling efficiency. In terms of training cost, MCF is trained for 300 epochs on GEOM-DRUGS on 8 A100 GPUs. Indeed, it is more costly than Torsional Diffusion in training. But it should be noted that training is a one-time cost and MCF sets the new state-of-the-art on standard molecular conformer generation tasks. Lastly, since MCF doesn’t rely heavily on cheminformatic methods to generate conformers, we hope the framework can be scaled to large molecules like proteins in the future work.

3. Continuity of MCF

• We agree with the reviewer that current results don’t reflect physically meaningful structures and it’s still an open question to be investigated. Here, we include the experiment of interpolated eigenfunctions to conceptually investigate the flexibility of defining conformer generation problem as a field. It's an interesting observation that MCF can generate feasible conformers even when the input interpolated eigenfunctions have never been seen during training. Such that MCF is not over-fitted to certain eigenfunctions. It provides a hint that MCF may learn to generate distributional aspects of atomic positions purely from correlations from training data. Admittedly, we recognize this is highly speculative and needs further empirical investigation to substantiate in future works. Also, when provided molecular conformer data with distribution of electron density, MCF may be extended to predict electron density beyond atomic positions in future works as well. Following the suggestion, we will move the discussion to Appendix and further clarify the scope of the experiments.

We thank the reviewers for the constructive reviews. Please find below our detailed responses to their concerns:

1. The main formulation and model architecture are same to DPF

   • In this work we generalize the problem setting in DPF to deal with functions on graphs (as opposed to functions on Euclidean spaces). Our proposed approach is conceptually simple to implement while being very powerful in its application. Note that MCF sets the current state-of-the-art for conformer generation. We believe that approaches that are both simple to implement and powerful in their empirical results tend to have the longest lasting impact in the field.

2. More explanation about background of DPF

   • We have included a sub-section 3.1 in the Preliminaries introducing diffusion models and diffusion probabilistic fields.

3. Why should the context pair also be corrupted? Is there any motivation behind it?

   • During training, the context pair provides the information about the field/function at a particular timestep $t$. Drawing an analogy to the image case, the input to the model at timestep $t$ is a noisy image. Similarly, the input to MCF at timestep $t$ is a set of noisy context pairs that explicitly encode the field. At inference time, sampling from MCF starts from gaussian samples for both context and query pairs. Therefore, noisy pairs are required to effectively model the distribution of fields/functions.

4. How can a function be used as the variable of Gaussian distribution?

   • In our case, we sample a conformer field $f_0 \sim q(f_0)$. Here, $q(f_0)$ denotes the ground truth distribution of conformer fields, namely the implicit functions that map points from molecular graph space to 3D Euclidean space, which is not a Gaussian distribution. We sample noise from Gaussian distribution $\epsilon \sim N(0, I)$ to corrupt the signal (i.e., 3D atomic coordinates) in context and query pairs rather than the field itself.

5. Why can the proposed method outperform GeoDiff?

   • As the reviewer mentioned, the superior performance of MCF comparing to GeoDiff roots from both the field formulation and the relaxation of using equivariant constraints in the architecture. MCF models the distribution of fields instead of distributions over discrete graphs. In addition, our backbone architecture (ie. PerceiverIO), is an efficient Transformer version, which can model the global interactions between atoms. On the other hand, GeoDiff is built upon an equivariant graph neural network which is known to suffer from over-smoothing and hard to model long-range interactions. Besides, our transformer backbone provides more flexibility of scaling up the number of parameters. In our case, we implement the number of latents as 512 which widens the bandwidth and degree of freedom in modeling the conformer field beyond number of atoms (which is usually in the order of 10).

6. Missing related works

   • We thank the reviewer for point us to the related work, we have included that in the revised manuscript.

We thank the reviewers for the constructive reviews. Please find below our detailed responses to the weaknesses and questions:

1. Inconsistency in Chemical Properties Experiment In Section A.3.3, which pertains to the chemical properties experiment, this paper adopts experimental results from "Torsional Diffusion." However, there's a discrepancy in the subsets used. Specifically, the paper does not use the identical subset as "Torsional Diffusion," implying that they have selected different molecules. This raises concerns about the validity of their experimental comparison. For it to be persuasive, it should be more convincingly aligned.

   • We have added experimental results of MCF and reproduced Torsional Diffusion on same subset in Appendix A.3.3. It indicates that on the aligned subset, MCF generate conformers that are closer to the ground truth than Torsional Diffusion in terms of the ensemble properties. This further validates that MCF learns to model the distribution of physically/chemically valid molecular conformers.

2. Errors in Previous Studies

   • We thank the reviewer for pointing out this. We have revised the description regarding GeoMol in section 1 and 2.

3. Insufficient Performance The GEOM-DRUGS dataset is considered a more challenging metric in this field. The method proposed in this paper demonstrates enhanced performance only regarding recall. Recall measures the capability to locate ground-truth conformers within the generated ones, and this metric is susceptible to influences from the training data. Notably, Torsional Diffusion depends on the local generated by RDKit, utilizing only 30 normalized conformers for each molecule during training. Although both the proposed method and Torsional Diffusion use the same dataset and split, the mean count of conformers in the GEOM-DRUGS dataset is approximately 100. This suggests that the boost in recall could result from the proposed method being trained on over 300% more conformers and having double the training epochs rather than the inherent efficacy of the method.

   • We have updated the results on GEOM-DRUGS in Table 2, after training MCF with updated hyper-parameters (see Tab. 4). MCF now surpasses Torsional Diffusion on all metrics regarding recall and precision. On more challenging datasets like GEOM-DRUGS, MCF shows its advantage over other baselines.
   • Also, we did not use the all the conformers in GEOM-DRUGS dataset during training. We follow the setting of GeoMol [1] and randomly pick at most 10 and 20 conformers for each molecule in training set of GEOM-QM9 and GEOM-DRUGS, respectively. Compared with 30 conformers per molecule in training Torsional Diffusion, our MCF is actually trained on less number of conformers. Trained on less data while achieving superior performance, MCF demonstrates the effectiveness in molecular conformer generation.

4. Furthermore, in terms of precision, the proposed method is worse than Torsional Diffusion, especially evident when Torsional Diffusion undergoes 50 denoising steps and MCF requires 1000 denoising steps.

   • Following the reviewer’s suggestion, we have included experiments with DDIM sampling [2] with small number of timesteps in Appendix A.4 and Tab. 9. Our MCF demonstrates competitive performance using a reduced number of denoising steps. When using 20 denoising steps, which is the main setting Torsional Diffusion reported in the paper, MCF achieves better performance than Torsional Diffusion on the challenging GEOM-DRUGS dataset. Also, as shown in the Tab. 9 MCF surpasses Torsional Diffusion with 50 sampling steps. Indeed, Torsional Diffusion show advantage when the number of sampling steps is very small (3 or 5 steps). We attribute this to the fact that Torsional Diffusion uses a pre-processing step in which a cheimformatics method predicts local substructure in 3D spaces before applying their model. So even before inference the cheimformatics prediction provides a strong prior. In addition, we would like to mention there are lines of works concerning distillation of diffusion models [3] so that it can effectively sample in even one single step. Such techniques can be applied to MCF and further increase the sampling efficiency.

------- see following comments below -------

We thank the reviewers for the constructive reviews. Please find below our detailed responses to the weaknesses and questions:

1. If we have n atoms in one molecule, then the conformation space is $\mathbb{R}^{3n}$ instead of $\mathbb{R}^3$. in the paper authors indicate $f: G → \mathbb{R}^3$.

   • Here, the notation represents a field (or implicit function) that maps any node from a graph to 3D Euclidean space. Namely it means mapping an atom (node) in a molecular graph to its 3D coordinates in the Euclidean space instead of mapping a molecular graph to a 3D conformer.

2. Section 4.1. Score field network: the author cite and use perceiverIO net and explain why they use that. this is good. the author should explain further for the benefits of the readers, what line 8 of algorithm 1 entails. \epsilon_q ~ N(0,I), line 8 want \epsilon_\theta to map to N(0,I). what is the physical significance in the context of molecule conformations?

   • The loss in line 8 of algorithm 1 is adapted from DDPM loss. $\epsilon_q$ are sampled Gaussian noises that are injected to corrupt the output space of conformer field (i.e, 3D atomic coordinates). We train the score network $\epsilon_\theta$ to predict the Gaussian noise added to the coordinates. Therefore, the gradient is not intended to push $\epsilon_\theta$ to a standard Gaussian, but rather predict the sampled noise $\epsilon_q$ given the corrupted context and query pairs at current timestep $t$. So that in inference, MCF can generate realistic molecular conformers starting from random Gaussian noise.

3. The authors claimed that the strength of their method is that there is no need to know some domain knowledge such as torsion angles of molecules. in my opinion this is a serious weakness. would the predictions generate physically non-viable confirmations if it disregard some basic physics and chemistry?

   • We updated Tab. 2 with results from an updated MCF model, which demonstrates superior performance over all previous approaches on the challenging GEOM-DRUGS dataset. Also, in updated Tab. 8, MCF achieves competitive performance on ensemble property prediction. We note that although MCF does not enforce explicit inductive biases (eg. modeling torsional angles) it generates physically/chemically realistic samples. In addition other machine learning domains like computer vision [1,2] and natural language processing [3,4] that attention-based transformers and its variants can generalize well to various downstream applications when pre-trained on sufficiently large datasets. This indicates that if the domain knowledge is properly included in sufficient training data (e.g., the molecular conformers with low energies investigated in this work), the model can implicitly learn inductive biases from data without the need to explicitly enforcing them into the architecture. Besides, it should be noted that integrating domain knowledge also comes at a cost. For example, in Torsional Diffusion, one needs to use rule-based method to find the rotatable bonds which can fail in some cases especially for large complex molecules. It also relies on cheminformatic methods to predict local substructures which may neglect long-range interactions between substructures and restrict the conformer quality by the local methods. On the other hand, MCF provides a framework that directly operates on Euclidean coordinates which is easier to scale up both in terms of number of parameters as well as larger molecular systems.

4. Since diffusion model will generate a distribution of conformations, does the distribution follows the distribution of statistical physics? e.g. Boltzmann distribution. Diffusion generative models are trained to model an empirical distribution of training data. Following the setting of previous work [5], we randomly pick at most 10 and 20 conformers for each molecule during training for GEOM-QM9 and GEOM-DRUGS datasets [6], respectively. Therefore, the training data doesn’t follow the Boltzmann distribution and our current MCF does not exactly generate Boltzmann distributed conformers. However, if provided training data that follows Boltzmann distribution, diffusion models like MCF are expected learn to the model the distribution as shown in some recent works [7]. Also, we can apply weighted sampling during training if Boltzmann weights are available for training conformers, which can force generating conformers in Boltzmann distribution. Another extension could be integrating flow matching [8] framework which provides exact estimation of log-likelihood. Flow matching provides the flexibility to map between arbitrary distributions. Therefore, instead of sampling from a standard Gaussian distribution, one may start from a Boltzmann distribution in inference. We have added discussions to Appendix A.1 as future works to extend MCF for generating conformers following Boltzmann distributions.

------- see following comments below -------

We thank the reviewer for the constructive reviews. Please find below our detailed responses to the concerns:

1. The novelty is limited from a technological perspective. I see little difference between the diffusion probabilistic field and the proposed molecular conformer field.

   • In this work we generalize the problem setting in DPF to deal with functions on graphs (as opposed to functions on Euclidean spaces). Our proposed approach is conceptually simple to implement while being very powerful in its application. Note that MCF sets the current state-of-the-art for conformer generation. We believe that approaches that are both simple to implement, scalable and powerful in their empirical results tend to have the longest lasting impact in the field (ref the bitter lesson by Rich Sutton http://www.incompleteideas.net/IncIdeas/BitterLesson.html) .

2. Using eigenvectors of the position of the graph in the field seems dangerous for the following two reasons. (1) This modeling does not take key information, such as atomic numbers, into consideration. (2) The eigenvector based position cannot effectively indicate distance in the graph without the corresponding eigenvalues.

   • In MCF, we use atomic features which are concatenated with the graph Laplacians in both query and context pairs, including atomic numbers, aromatic, degree, hybridization, formal charge, etc. Therefore MCF has included essential atomic information that models the molecules. We have added an extra section A.2.2 in Appendix to clarify the atomic features used in this work. Though we didn’t include eigenvalues as the input to MCF, the eigenvectors are ranked based on eigenvalues. This can provide the extra information regarding the distance and connectivity between atoms. Finally, in practice we see that even without eigenvalues MCF sets the new state-of-the-art for conformer generation.

3. Can this proposed method distinguish two different 2D molecular graphs that have the same eigenvectors but different eigenvalues?

   • As mentioned in response to the previous question, eigenvectors are ranked based on eigenvalues. Therefore, MCF can still distinguish graphs with same eigenvectors but different eigenvalues. In addition, we use atomic features as input to MCF which solves this type of problem in practice.