MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectra

Q8 Discuss different normalization procedures for peak intensities and whether alternatives were tested.

Response: Thank you for your feedback regarding our use of $\log_{10}$ normalization for peak intensities. In our experiments, we indeed explored different normalization techniques, including the min-max normalization you mentioned. Additionally, we experimented with truncating particularly high intensities in each spectrum based on a threshold before applying either $\log_{10}$ or min-max normalization.

Our findings indicated that $\log_{10}$ normalization consistently outperformed min-max normalization. This is primarily due to the significant differences in magnitude often observed between peaks in molecular spectra, with some spectra containing a few exceptionally high peaks. In such cases, min-max normalization tends to preserve information from only a few very high peaks, leading to the loss of information from the majority of the peaks. Conversely, $\log_{10}$ normalization effectively mitigates these magnitude differences, allowing for the retention of more comprehensive peak information across the spectrum.

We appreciate your insights and hope this clarifies our approach.

Additional Feedback 1 Include a discussion on the limitations of the method and potential future research directions.

Response: Thank you for your insightful feedback. We have outlined the limitations of our method and potential future directions, which have been added to the Appendix D of our paper.

One significant limitation is the availability, scale, and diversity of molecular spectral data. Our current dataset includes geometric structures of 134,000 molecules, each with three types of spectra (UV-Vis, IR, Raman). To effectively explore the scaling laws of pre-training methods, larger and more diverse molecular spectral datasets are necessary. Encouragingly, molecular spectroscopy has been gaining increasing attention in the research community, with larger and more diverse datasets being released, such as the recent multimodal spectroscopic dataset [1]. This development aids in advancing molecular representation learning and other related tasks.

Another limitation is that our proposed SpecFormer can currently only handle one-dimensional molecular spectra. For higher-dimensional spectra, such as two-dimensional NMR and two-dimensional correlation spectra, further development of sophisticated encoder architectures is needed.

Looking ahead, we envision several future directions in this field: 1. Exploring the scaling laws of pre-training on larger and more diverse molecular spectral datasets. 2. Expanding the encoding of molecular spectra to a broader range, including NMR, mass spectra, and two-dimensional spectra. 3. While we have developed a pre-trained spectral encoder, we have so far only applied the pre-trained 3D encoder to downstream tasks. The pre-trained spectral encoder can be utilized for molecular spectrum-related downstream tasks, such as automated molecular structure elucidation from spectra.

Additional Feedback 2 Correct the typo on page 8 line 427/428 from "yiels" to "yields".

Response: Thank you for bringing this to our attention. We have corrected this typo in the revised paper

Other questions (including W3 and Q4).

Response: We sincerely appreciate the reviewer's suggestions on other aspects of experiments, including the architecture of the structural encoder (W3), and the scaling performance of the pre-training dataset (Q4)). Conducting experiments on pre-training and fine-tuning methods requires more time than training from scratch, and due to the limited time available for the rebuttal, we are currently unable to provide the relevant results now. However, we assure you that we will address these as soon as possible.

References

[1] Unraveling Molecular Structure: A Multimodal Spectroscopic Dataset for Chemistry, NeurIPS Datasets and Benchmarks Track 2024

Q1 Clarify if the TorchMD-Net architecture used for MolSpectra differs from the one in Tables 2 and 3.

Response: Thank you for your insightful feedback. We would like to clarify that the backbone architecture used for structural representation learning in our method is indeed TorchMD-Net, and we have not made any modifications to it. This approach aligns with recent methods in molecular 3D representation pre-training, such as Coord. Our method can be understood as integrating molecular spectra into Coord, which uses TorchMD-Net as its backbone and employs denoising as its pre-training objective. Therefore, comparing our method with Coord more effectively demonstrates the advantages of our approach.

Q2 Explain the choice of UV-Vis, IR, and Raman spectra, and consider discussing the potential benefits of including other spectra types like NMR and mass spectra.

Response: We chose UV-Vis, IR, and Raman spectra for two main reasons. First, the information contained in these energy absorption spectra aligns well with our motivation to learn the energy level structures of molecules. Second, our method requires data sets that include both the 3D geometric structures and their corresponding energy spectra. The QM9S dataset provides this comprehensive information for the spectra we selected, whereas datasets for other types of spectra often lack 3D geometric structures. We fully agree that our method has the potential to be extended to other types of spectra, such as NMR and mass spectra, which could further enhance molecular representation learning.

Q5 Offer a deeper analysis of the learned representations, such as analyzing attention patterns or visualizing the latent space in SpecFormer.

Response: Thank you for your insightful suggestion. We have visualized the attention patterns and learned spectra representations in SpecFormer, and have added a new Section 4.5 to our paper to present the results and related analysis. We kindly invite you to refer to Section 4.5 for detailed visualizations. Based on the visualizations provided in Figure 4, we have made the following observations:

In Figures 4(a), (b), and (c), different attention heads in SpecFormer model distinct dependencies. Since the input to SpecFormer includes three types of spectra, the attention weights within the three boxes along the main diagonal reveal intra-spectrum dependencies, while those outside the boxes reveal inter-spectrum dependencies. The concepts of intra- and inter-spectrum dependencies are introduced in Section 3.1. In our visualizations of the attention maps from three attention heads in the second layer of SpecFormer, it can be observed that Head 11 primarily models intra-spectrum dependencies, Head 13 primarily models inter-spectrum dependencies, and Head 12 models both types of dependencies simultaneously. Additionally, because the intensity peaks and dependencies in molecular spectra are relatively sparse, the attention maps in SpecFormer are generally sparse as well.

In Figure 4(d), we visualize the spectra representations output by the final layer of SpecFormer using t-SNE. It can be observed that the distribution of representations in the latent space is relatively uniform and forms several potential clusters. This well-shaped distribution of representations reveals effective spectra representation learning and supports the structure-spectrum alignment.

We hope these additions and analyses address your concerns and provide a clearer understanding of the learned representations in SpecFormer.

Q6 Specify the source of spectra used for spectral representations in section 4.1/Table 1.

Response: Thank you for your feedback. The spectra used in the experiments in Section 4.1 are indeed the UV-Vis spectra provided by QM9S. We have revised the paper to explicitly state this in Section 4.1.

W2 Direct comparison with the same model architecture trained without spectral information is needed.

Response: Thank you for your constructive suggestion. We have added an ablation study to more rigorously demonstrate the improvements brought by quantum spectral data. In this study, we retain only the denoising loss in the objective during the second-stage pre-training of MolSpectra, excluding the MPR loss and contrastive loss. The only difference between this variant and MolSpectra is whether molecular spectra are incorporated into the pre-training.

Because pre-training and fine-tuning on molecular 3D structures require a relatively long time, this ablation study is still in progress. We assure you that we will provide the experimental results and analysis before the rebuttal period ends. We will also revise Section 4.4 of the paper to include this ablation study.

W4 The claim that MolSpectra learns molecular dynamics through energy level transitions is unsupported by direct evidence and should be explicitly tested using molecular dynamics simulations, e.g., power spectrum.

Response: Thank you very much for introducing the concept of the power spectrum to us. We completely agree that testing on the power spectrum can provide valuable insights into the molecular dynamics knowledge learned by the pre-trained model. We will incorporate these tests into our experiments.

In our current experiments, we have already conducted tests on the MD17 dataset, where we predicted force labels based on molecular conformations from molecular dynamics trajectories. As shown in Table 3, the comparison between our method and Coord demonstrates that our approach captures more information about molecular dynamics.

W5 It is unclear which energy function is used for MolSpectra in section 2.2, and this should be explicitly stated for clarity.

Response: We apologize for not making this point clear. In MolSpectra, we use the energy function and denoising objective provided by Coord, specifically variant I as described in Section 2.2. We have added clarification on this in Appendix C.2.

W6&Q3 The paper lacks an analysis of the computational complexity and resource requirements of the proposed method compared to baseline methods.

Response: Thanks for your suggestion. For downstream fine-tuning, the overhead of fine-tuning our pre-trained model is identical to that of previous methods. For upstream pre-training, the comparison of pre-training time cost and memory cost is provided as follows:

* The time costs are all measured on a single 3090 GPU.

Furthermore, a single pre-training session can provide a pre-trained model for numerous downstream tasks. We think that this increase in time cost is acceptable for a pre-training method.

W7 A more comprehensive ablation study is needed to evaluate the contributions of different spectral modalities (UV-Vis, IR, Raman) to the model's performance.

Response: Thank you for your suggestions. We have added an ablation study in Section 4.4 to evaluate the contributions of different spectral modalities (UV-Vis, IR, Raman) to the model's performance.

Because pre-training and fine-tuning on molecular 3D structures require a relatively long time, this ablation study is still in progress. We assure you that we will provide the experimental results and analysis before the rebuttal period ends. We will also revise Section 4.4 of the paper to include this ablation study.

W8 The paper primarily compares against methods using only 3D structure information.

Response: Compared to 1D SMILES and 2D molecular graphs, molecular spectra are more closely related to the 3D geometry of molecules, offering better potential for representational alignment. Additionally, the denoising targets in the 3D modality have explanations in terms of molecular energy, and molecular spectra complement previous methods by providing information on quantum energy states. Therefore, we prioritized the 3D modality. We appreciate your suggestion and agree that molecular spectra also have the potential to enhance molecular representation learning in other modalities.

We sincerely appreciate the time and effort you have dedicated to reviewing our paper, as well as your detailed and professional comments. We truly believe that your insights will significantly enhance the quality of our paper. In response to your suggestions, we have made every effort to provide thorough additions and clarifications. Please find detailed responses below.

W1 The best results in Table 2 are not correctly highlighted.

Response: Thank you very much for pointing out the formatting issues in Table 2. We have updated the results in Table 2, and ensured that the best results are highlighted in bold. Additionally, since our method introduces molecular spectra on top of Coord, we have used background colors to emphasize the improvements over Coord. The updated experimental results and formatting have been incorporated into the revised paper PDF. In the revised Table 2, our method achieves state-of-the-art performance in 8 out of 12 properties and outperforms Coord in 10 out of 12 properties.

Revised Table 2:

W1 Some current SOTA models are not included.

Response: In our two-stage pre-training schedule introduced in Section 3.4, the first stage uses only the denoising objective and can directly adopt any previous denoising-based method (e.g., Coord, Frad, and SliDe). The second stage can then be viewed as a plug-in to these existing methods, performing spectra-aware post-training on them.

Considering that Coord is the most fundamental, representative, and efficient denoising-based method, we adopted the denoising objective provided by Coord as the foundation for developing our method. By comparing our method with Coord, we can effectively demonstrate the advantages of incorporating molecular spectra.

We acknowledge that including recent SOTA geometrically equivariant models could further illustrate the effectiveness of our approach. We greatly appreciate your suggestion and commit to including these results in our experiments.

W1 The meaning of the horizontal separator in the table should be clearly explained.

Response: The purpose of the horizontal separators in Tables 2 and 3 is to divide the compared methods into two groups: training from scratch and pre-training. Thank you for your suggestion, and we have added explanations of this in the table captions.

W3 The experimental conditions are not clearly defined, such as the number of negative samples and noise generation parameters.

Response: We apologize for not clearly defining some experimental details. We provide the missing details here and have also included them in Appendix C.2:

• Number of Negative Samples: Following SimCLR [1], the contrastive loss in our Equation 7 is implemented using in-batch contrastive loss, where positive and negative pairs are constructed within each data batch. Therefore, for each anchor representation in a batch, there is one positive sample and $bs-1$ negative samples, where $bs$ is the batch size. In our method, $bs=128$.

• Noise Generation: In both pre-training stages, we use the noise generation method and denoising objective provided by Coord. The noise is added to atom positions as scaled mixture of isotropic Gaussian noise, with a scaling factor of 0.04. The denoising objective is defined in Equation 2.

Q3 Further analysis or ablation study is needed to prove the contribution of quantum spectral data to the model's performance.

Response: Thank you for your constructive suggestion. We have added an ablation study to more rigorously demonstrate the improvements brought by quantum spectral data. In this study, we retain only the denoising loss in the objective during the second-stage pre-training of MolSpectra, excluding the MPR loss and contrastive loss. The only difference between this variant and MolSpectra is whether molecular spectra are incorporated into the pre-training.

Because pre-training and fine-tuning on molecular 3D structures require a relatively long time, this ablation study is still in progress. We assure you that we will provide the experimental results and analysis before the rebuttal period ends. We will also revise Section 4.4 of the paper to include this ablation study.

References

[1] A Simple Framework for Contrastive Learning of Visual Representations, ICML 2020

We sincerely appreciate the time and effort you have dedicated to reviewing our paper, as well as your constructive comments. Please find detailed responses below.

W1&Q1 The model is likely to be computationally demanding due to high-dimensional spectral data and resource-intensive transformer architecture, as well as the numerous data combinations in the contrastive loss.

Response: We completely understand your concerns regarding efficiency. For downstream fine-tuning, the overhead of fine-tuning our pre-trained model is identical to that of previous methods. For upstream pre-training, we would like to clarify the following points:

• Although the original spectral data is high-dimensional, in Section 3.1, we propose a method to divide the original spectrum into patches based on patch length and stride. This significantly reduces the number of tokens fed into the transformer-based encoder. Specifically, for a 601-dimensional UV-Vis spectrum, by using a patch length of 20 and a stride of 10, we obtain only 58 tokens that need to be fed into the SpecFormer. We can further balance effectiveness and efficiency by adjusting these hyper-parameters.

• We employ an in-batch contrastive loss, which constructs positive and negative pairs only within each randomly sampled data batch. This approach can be seen as a negative sampling method to avoid numerous negative pairs and is widely used in many representative contrastive learning methods, such as SimCLR [1].

• As mentioned in Section 3.3, the scoring function in our contrastive loss (Equation 7) uses an inner product. Therefore, the scores for all in-batch positive and negative pairs can be obtained by performing matrix multiplication between the encoded 3D representation matrix and the encoded spectra representation matrix. Thanks to our use of in-batch contrastive loss and the optimizations for matrix multiplication provided by CUDA and PyTorch, this computation does not introduce a heavy computational overhead.

Overall, the additional computational overhead introduced by our method due to spectra is comparable to that of previous pre-training methods based solely on denoising. The empirical comparison of pre-training time cost is as follows:

* The time costs are all measured on a single 3090 GPU.

Furthermore, a single pre-training session can provide a pre-trained model for numerous downstream tasks. We think that this increase in time cost is acceptable for a pre-training method.

W2&Q2 The experiments do not include comparisons with recent models, such as SliDe or Frad.

Response: In our two-stage pre-training schedule introduced in Section 3.4, the first stage uses only the denoising objective and can directly adopt any previous denoising-based method (e.g., Coord, Frad, and SliDe). The second stage can then be viewed as a plug-in to these existing methods, performing spectra-aware post-training on them.

Considering that Coord is the most fundamental, representative, and efficient denoising-based method, we adopted the denoising objective provided by Coord as the foundation for developing our method. By comparing our method with Coord, we can effectively demonstrate the advantages of incorporating molecular spectra.

We acknowledge that implementing our method using the denoising objectives provided by Frad and SliDe, and comparing with them, could further illustrate the effectiveness of our approach. We greatly appreciate your suggestion and commit to including these results in our experiments.

W3 Tables 4 and 5 show that the performance of the proposed method is sensitive to hyper-parameter values, considering the improvement over the best baseline method.

Response: We have updated the experimental results in Tables 2, 4, and 5. The improvements of our method over the best baseline are now more pronounced in Table 2, and our method demonstrates greater robustness to value changes in hyper-parameters, including patch length, stride, and mask ratio. The results of the sensitivity analysis are updated as follows, and have also been incorporated into the revised paper. (Some additional experiments are still in progress, but they do not affect our conclusions. We will make sure to include the results of these experiments as soon as possible before the rebuttal period ends.)

Table 4: Sensitivity of patch length and stride.

Table 5: Sensitivity of mask ratio.

The main baseline method being compared, Coord, has results of 17.7, 14.7, and 31.8 for homo, lumo, and gap, respectively. Our method consistently outperforms it when hyper-parameters change, as shown in the tables above.

W3 In Table 4, we suspect that row 1 column overlap ration the number should be swapped with row 3 same column.

Response: This is indeed the mistake we made in writing the paper. Thank you very much for helping us find this issue, we have corrected it in the revised paper.

W4 It is unclear if parameter tuning was done using the pre-training dataset or downstream tasks, as this is not clarified in the text.

Response: We apologize for not clarifying this point. Our goal is to incorporate spectral information of molecules into 3D representations through representation alignment during pre-training, and then use only the 3D structure in downstream tasks to obtain 3D representations that contain spectral information, without directly using molecular spectral data in downstream tasks. This allows us to benefit from spectral knowledge even when molecular spectral data is not available downstream. The hyper-parameters we tuned (including patch length, stride, and mask ratio) are related to the spectral data, so they were tuned on the pre-training dataset, not on the downstream dataset. We have added relevant clarification in Appendix C2.

We sincerely appreciate the time and effort you have dedicated to reviewing our paper, as well as your constructive comments. Please find detailed responses below.

W1&Q1 Why choose a two-step pre-training schedule, which may lead to additional complexity?

Response: The adoption of the two-stage pre-training schedule is motivated by the presence of molecular spectra in different datasets and the need to evaluate the impact of these spectra. Existing 3D molecular representation pre-training methods primarily perform denoising-based pre-training on the PCQM4Mv2 dataset, which contains 3.4 million molecules. This dataset provides only the equilibrium 3D structures of molecules without any spectral data. To compare with these existing methods and verify the improvements brought by incorporating molecular spectra, our first stage of pre-training also involves denoising-based pre-training on the PCQM4Mv2 dataset, aligning with these methods in terms of training strategy and dataset.

In the second stage of pre-training, we utilize the QM9S dataset containing over 134,000 molecules with available molecular spectra. We introduce spectral data and using the complete objective in Equation 8 to further pre-train the model. Thus, our approach adds a second stage of pre-training on top of the previous denoising pre-training methods.

In experiments, our first stage of pre-training can be conducted independently or by directly using pre-trained models from previous denoising-based methods. When we use existing denoising pre-trained models, our second-stage pre-training acts as a plug-in to these existing approaches, performing post-training on them.

If we only perform one-step pre-training on QM9S, it would be unfair to compare it with baseline methods pre-trained on the large-scale PCQM4Mv2 due to the difference in the number of molecules in the pre-training datasets.

W2 In Table 2, some experimental results are not highlighted correctly.

Response: Thank you very much for pointing out the formatting issues in Table 2. We have updated the results in Table 2, and ensured that the best results are highlighted in bold. Additionally, since our method introduces molecular spectra on top of Coord, we have used background colors to emphasize the improvements over Coord. The updated experimental results and formatting have been incorporated into the revised paper PDF. In the revised Table 2, our method achieves state-of-the-art performance in 8 out of 12 properties and outperforms Coord in 10 out of 12 properties.

Revised Table 2:

In our initial response, some additional experiments for W3 were still in progress. Now, we have completed all the experiments and are providing a supplemental response to W3.

W3 Tables 4 and 5 show that the performance of the proposed method is sensitive to hyper-parameter values, considering the improvement over the best baseline method.

Response: We have updated the experimental results in Tables 2, 4, and 5. The improvements of our method over the best baseline are now more pronounced (as shown in Table 2), and our method demonstrates greater robustness to value changes in hyper-parameters, including patch length, stride, and mask ratio. The results of the sensitivity analysis are updated as follows, and have also been incorporated into the revised paper.

Table 4: Sensitivity of patch length and stride.

Table 5: Sensitivity of mask ratio.

The main baseline method being compared, Coord, has results of 17.7, 14.7, and 31.8 for homo, lumo, and gap, respectively. Our method consistently outperforms it when hyper-parameters change, as shown in the tables above.

We hope this addresses your concerns. If you have any further questions, please feel free to ask, and we would be more than delighted to answer.

Dear Reviewer vWL4,

Thank you very much for your precious time and valuable comments. Since we are in the last two days of author-reviewer discussions, we eagerly await your feedback. We have made every effort to address your concerns and hope that our responses have clarified any questions or issues. If there are any further concerns, please let us know. We look forward to hearing from you and appreciate any feedback.

Best regards,

Authors

Dear Reviewer vWL4,

As the author-reviewer discussion phase is nearing its conclusion, we are eager to receive your feedback on our response and revised paper. Your feedback is extremely valuable to us.

In response to your concerns and questions, we have made the following efforts:

1. Clarification and Additional Experimental Details: We have clarified the motivation and rationale behind the two-step pre-training schedule and provided detailed information on hyperparameter tuning.
2. Optimized Presentation of Experimental Results: We have corrected the formatting error in Table 2 regarding the bold highlights and fixed the incorrect placement of the overlap ratio in Table 4.
3. Additional Experiments: Our updated ablation studies demonstrate that the improvements of our method are robust to changes in hyperparameters.

We are keen to know if our efforts have addressed your concerns. Thank you once again for your valuable comments and the time you have dedicated to reviewing our work. We look forward to receiving your feedback.

Best regards,

Authors