Dimension Debate: Is 3D a Step Too Far for Optimizing Molecules?

Thanks for your review! We hope that our response below can convince you further. If you have any additional doubts, please let us know!

1. Methodological Flaw in MolFormer Prompting: Thank you for identifying the tokenization issue in MolFormer. We acknowledge the flaw (and the typo), and have proceeded to fix this. We re-ran experiments with corrected implementations and observed identical trends, reaffirming our conclusions. Interestingly, MolFormer’s robustness to tokenization errors suggests resilience in its embedding process, and we believe this is something worth further researching. We now explicitly discuss these results and their implications in the revised manuscript.

2. Conformational Variability: We used single conformers per molecule across all datasets to ensure consistency and comparability. This choice eliminates potential confounding effects due to conformational variability. We clarified this methodological detail and its implications in the revised text.

3. Clarity of Experimental Setup: The absence of MolFormer results in certain figures, such as Figure 3, reflects compatibility constraints with specific surrogate models. MolFormer only made use of the LLA surrogate, as such, it has now been included also in the LLA column. These limitations are now explicitly detailed in the manuscript for greater clarity.

4. Generalization of Claims: While our study primarily focuses on benchmarking embedding performance across diverse datasets and experimental setups, we recognize the potential for developing methods that dynamically guide feature selection during the optimization process. For example, incorporating metrics to evaluate embedding performance early in the BO pipeline could help practitioners decide when to prioritize 2D or 3D representations based on task-specific needs or available data. Exploring such adaptive strategies is a promising direction for future research but out of scope for this manuscript, and as such, we have added further discussion of these possibilities to the revised conclusions.

Thanks for your review! We hope that our response below can convince you further. If you have any additional doubts, please let us know!

1. Innovation and Contribution:
   We appreciate your observations regarding the scope of innovation. While comparing representations is a classical problem, it is also a well-defined challenge that arises repeatedly in the well-established framework of molecular screening during the design timeline (Dara et al, 2021). Our results have broad implications for practitioners, as they provide actionable insights into how to select embeddings under typical constraints faced in real-world applications. By revealing counter-intuitive findings, such as the better performance of 1D and 2D embeddings over their 3D counterparts, we contribute new knowledge that challenges assumptions and guides decision-making within this widely used framework

2. Physical Relevance of 2D and 3D Features:
   We agree that 3D embeddings may show advantages in tasks such as protein docking. However, our study deliberately focuses on tasks that ensure a fair comparisons across embeddings. These are tasks that allow for the three levels of representation, while on the other hand, tasks like protein docking depend on the 3D representation. Furthermore, including trajectory-based datasets like MD17, as suggested, would introduce dependencies on parameters such as conformer generation, making the evaluation less controlled. Datasets like MD17 target molecular dynamics, which is fundamentally distinct from the optimization objectives in molecular design. We clarified this scope in the revised manuscript.

3. Focus on Bayesian Optimization (BO):
   BO is a practical and widely used tool for molecule screening within pre-defined libraries, allowing efficient exploration of complex search spaces (Tom et al, 2024). While generative models are excellent for proposing novel candidates, they address a different problem. Our use of Gaussian Processes and Bayesian Neural Networks as surrogates ensures a focus on embedding performance rather than model-specific mechanisms. We clarified this point in the revised manuscript to address your concerns.

4. Model Configurations and Comparisons:
   We ensured parity in parameter magnitude across models to maintain fairness(Hunter et al, 2012). Although 3D models like Equiformer-V2 are highly expressive, they require significantly more data to perform comparably to simpler 2D models. These findings suggest practical advantages for practitioners working under typical constraints. We expanded the discussion on this topic and included an analysis of architectural trade-offs.

Thank the author for their response. I apologize that I can not change my score.

1. In computational chemistry, it is not counterintuitive that 1D and 2D embeddings outperform 3D in some tasks. As I mentioned in my first review, 3D embeddings are suitable for describing some QM properties, such as the Boltzmann distribution, force fields, which 2D embeddings cannot represent. A 2D molecule may correspond to many 3D structures, and if you only measure one of them with a 3D model, it indeed struggles to learn the macroscopic features of 2D, such as thermodynamic statistical characteristics. Essentially, 2D and 3D embeddings are suitable for different tasks, and they should not be compared together. In practice, in some industrial scenarios that do not involve quantified features, researchers prefer to use 2D characterization because capturing 3D conformations are expensive.

2. There is not a single table with specific numerical values throughout the paper, which makes many of your conclusions hard to verify accurately. I hope the authors can point the detail evaluation metrics in Figures 3, 4, and 5 (the caption for Figure 5 is missing GEOM). I do not understand the meaning of three figures. The results in QM7 reached -500 kcal/mol; if this is the error between the predicted results and the ground truth, it is too large. Similarly, the results for QM9 are 12 eV, and GEOM's results are 20 Ha. Equiformer v2's errors in property prediction are measured in meV on QM9, and GEOM's energy prediction MAE is smaaler than 1. I hope the authors can explain the meaning of these figures to me before the rebuttal ends.

3. Equiformer v2 does not require more data. In the field of computational chemistry, MD17 and QM9 are both very simple datasets, especially MD17, which has only 950 data in the training set, yet Equiformer v2 can still achieve SOTA. For industrial practitioners, which model they use should mainly depend on which level of molecular features they are focused on.

Thanks for your review! We hope that our response below can convince you further. If you have any additional doubts, please let us know!

1. Generative Models vs. BO: While generative models are complementary to BO, our focus is rooted in pre-defined library screening, a practical and widely used scenario in molecular design workflows (Tom et al, 2024). It is not only that BO is the main tool in these workflows, but it is also the method of choice for practitioners precisely because it aligns with the constraints of synthesizability. Pre-defined virtual libraries provide a realistic and constrained search space where synthesizability can be carefully considered, making the BO framework particularly well-suited. Our study reflects these practical considerations, ensuring that our findings are directly relevant to the molecular optimization tasks practitioners face today. This distinction between BO and generative approaches is now explicitly discussed in the revised manuscript.

2. Applications of 3D Information: While 3D embeddings can be valuable for symmetry-sensitive tasks or those involving intricate spatial relationships their utility is not universal. For molecular optimization tasks, especially those focused on steady-state properties within the BO framework, our findings reveal that 2D embeddings strike a better balance between computational efficiency and performance. Additionally, the use of 3D embeddings introduces dependencies on conformer generation and sampling, which can add complexity and uncertainty to the pipeline. By highlighting these trade-offs, our study offers a nuanced perspective on when 3D embeddings are beneficial versus when simpler 2D representations suffice. These insights help practitioners to make informed decisions based on the specific demands of their optimization tasks. We have explicitly highlighted this limitation in our discussion to ensure readers understand the task-dependent nature of 3D embeddings’ utility.

3. Model Selection: The inclusion of MolFormer, MPNN, and Equiformer-V2 reflects a deliberate choice to balance relevance and diversity in our evaluation. MolFormer represents a state-of-the-art foundation model for 1D embeddings, while Equiformer-V2 is a cutting-edge model for 3D embeddings. MPNN, though not state-of-the-art for 2D embeddings, represented the first iteration of models we analysed. We decided to show our results for this 'general' architecture as our findings demonstrate that even with the inclusion of a non-SOTA 2D model, the performance gap between 2D and 3D embeddings underscores the practical implications of embedding dimensionality. This observation suggests that higher-dimensional embeddings may not always justify their computational cost, even when paired with advanced architectures. We expanded the justification for these model choices in the revised manuscript.

4. Training Strategy: Our choice to use pretrained weights for MolFormer and train MPNN and Equiformer-V2 from scratch ensures consistency with typical practices in the field. Pretraining allows MolFormer to leverage large-scale prior knowledge, reflecting a practical scenario where practitioners may use foundation models for embedding generation. Conversely, training MPNN and Equiformer-V2 from scratch ensures that these models are evaluated based solely on their ability to extract features specific to the datasets used in this study. This mixed strategy aligns with real-world applications where practitioners must decide between leveraging pretrained embeddings or developing task-specific models. This comparison provides valuable insights into the trade-offs practitioners face when selecting embedding strategies, and we have clarified this in the revised manuscript.

Thanks for your review! We hope that our response below can convince you further. If you have any additional doubts, please let us know!

1. Generality of Conclusions: We recognise the need for further investigation to generalise our findings to broader scenarios and to frame our findings carefully. We clarified in the revised manuscript that our conclusions apply specifically to the datasets and tasks analysed, such as steady-state molecular property prediction under BO.

2. Fair Comparison of 2D and 3D Features: We agree that 3D embeddings may show advantages in tasks such as protein docking. However, our study deliberately focuses on tasks that ensure a fair comparisons across embeddings. These are tasks that allow for the three levels of representation, while on the other hand, tasks like protein docking depend on the 3D representation.

3. LLM Training: We agree there is a blurry line when discussing LLMs as molecular feature extractors. To maintain coherence with our design choices, a pre-trained, but not fine-tuned, MolFormer was chosen. This is because this version of MolFormer has been trained on canonicalised SMILES only. We have further clarify this in our updated manuscript and further discussed the specifics when judging a 1D representation with LLMs.