Reply to Summary

We sincerely appreciate the reviewer i3nd for the feedback and comments to improve the quality of the paper. Before proceeding further, we think it is necessary to reformulate the summary to reiterate the motivation of our paper. Instead of extending denoising pre-training techniques, investigating the pre-training task choice, or developing model architectures, our paper aims to answer a previously unanswered research question in graph representational learning, which fits the ICLR venue:
"Are pre-trained all-atom geometric graph neural network (GNN) representations transferable to protein modeling, and how expressive are they?"

To answer this research question, our contributions are as follows:

1. Scaling Behaviors of Geometric GNNs:
   We studied the scaling behaviors of state-of-the-art geometric GNNs in unsupervised, self-supervised, and supervised setups, instead of focusing on developing new model architectures or pre-training objectives.
2. Demonstrating Transferability:
   We pre-trained these GNNs on small molecular datasets and demonstrated their transferability to proteins with all-atom resolution, highlighting their expressiveness in these settings.

Pre-training and Zero-Shot Transfer Learning

In terms of pre-training, we studied scaling behaviors with various model configurations. In the zero-shot transfer learning setup, we inferred the atomistic embeddings of all-atom peptides and proteins, coarse-grained to residue-wise embeddings, and organically combined them with other architectures for conformational kinetic modeling (VAMPNet) and fold classification tasks. In both tasks, pre-trained all-atom embeddings demonstrated excellent transferability.

Small Molecule Setups

In the small molecule setups, we studied molecular property prediction (QM9) and molecular force field prediction (xxMD) with both pre-trained and non-pre-trained models. In all the aforementioned setups, we did not find predictable power-law scaling (not just on the pre-training task). During pre-training, we observed that shallow models exhibit much higher pre-training loss compared to deeper models with similar parameter counts (hence, why we mentioned under-reaching). Additionally, we found that the benefits of increasing depth diminish after six layers (hence, why we mentioned over-smoothing).

QM9 and Kinetic Modeling Observations

In the QM9 experiments, we noted that when computing QM9 labels—such as the HOMO-LUMO gap—different quantum chemical methods produce ev-scale differences (Page 19, Figure 5). Therefore, we should not expect further improvements from machine learning models if they already reach the data uncertainty limit.

In contrast, in the kinetic modeling task (Page 5, Table 1), we found that the "benefit" of scaling in the no-mixer setup cannot compare to the improvement from adding a few layers of MLP or Transformer, which allows direct modeling of interdependence among structural units.

Reply to Experiments

Regarding the Comparison of ‘Molecular Dynamics’

First of all, we are unsure about what ‘molecular dynamics’ the reviewer 5LXW is referring to. Timewarp [1] is an enhanced sampling technique that proposes larger MD timesteps to accelerate the simulation, which is not directly comparable to the kinetic modeling task (if this is what the reviewer 5LXW refers to) included in this paper. Could the reviewer clarify what we are trying to compare here?

Regarding the Splitting and Comparison with More Models

As we have explained in the paper, randomly splitting the data without considering generalizability is not meaningful. Scaffold splitting is prevalent in benchmarks such as MoleculeNet. Furthermore, the xxMD paper [2] contains the results of DimeNet++, MACE, etc.

Regarding the Lack of Extensive Comparison

We carefully examined the performance and scaling behaviors of different pre-trained GNNs across diverse tasks:

• Conformational variety of single molecules (force field regression, kinetic modeling).
• Chemical variety of many molecules (quantum chemical property prediction).
• Conformational variety of peptides and proteins (kinetic modeling).
• Biological variety of many proteins (folding classification).

In detail, we provided more than 100 data points of pre-training experiments covering two GNN architectures with variations in model depth, width, aspect ratio, and radius cutoff configurations on both equilibrium and non-equilibrium molecular datasets:

• xxMD Experiments: We tested two GNNs (non-pre-trained and pre-trained on PCQM and Denali, respectively) across hidden dimensions on four molecular systems.
• QM9 Experiments: We tested two GNNs (non-pre-trained and pre-trained on PCQM) across hidden dimensions on five tasks.
• VAMPNet Experiments: We evaluated three systems (molecules to proteins) with different embedding dimensions and token mixers.
• Folding Classification Task: We tested ProNet with and without pre-trained all-atom embeddings across different dimensions, representing the most comprehensive and detailed study among relevant works.

Regarding the xxMD-Temporal Benchmarks

In the caption of Table 2 (Page 8), we indicated that we compared models pre-trained (with prefix PT) and not pre-trained across different dimensions and targets. In Table 3 (Page 9), we noted that only models pre-trained on the Denali dataset (PT-Denali) could positively transfer. ViSNet, without being pre-trained, performs better on xxMD-Azobenzene and dithiophene subsets.

To better support our claims, we have added an additional table in the appendix with the complete results. We summarized the ViSNet/ET results on azobenzene, stilbene, malonaldehyde, and dithiophene across 64, 128, 256, and 384 dimensions with no pre-training, pre-trained on PCQM, and pre-trained on Denali setups in [Appendix L, Page 31, Table 9].

Reply to minor points

As we have explained in the paper (Page 5 Line 237-240) and the main figure (Page 2 Figure 1), In each window, atomic structures are treated as individual graphs and processed by the pre-trained Geom-GNN to extract atomic-level features, which are aggregated into residue-level representations or “tokens.” The architecture can employ self-attention (SA), multi-layer perceptron (MLP), or message-passing mechanisms to enhance representational power.

Reply to zero-shot definition

Using masked language modeling as an example, language models trained to complete the sentence in English can transfer to complete the sentence in Chinese. (Same task, different data domain) In our setup, the backbone network is trained on small molecules with roughly 10-20 heavy atoms with denoising objective, and the backbone network is transferred to infer the atomistic embedding of peptides and proteins, and then a separate head is trained with a different objective. (Different task, different data domain) Since the backbone network has never seen protein systems, we think it’s appropriate to claim it as zero-shot transfer for embedding inference.

Reply to pre-training vs. fine-tuning

Pre-training datasets are broad and diverse, aiming to build general representations, while downstram datasets are task-specific and focused, helping the model adapt to particular applications.

[1] Klein, Leon, Andrew Foong, Tor Fjelde, Bruno Mlodozeniec, Marc Brockschmidt, Sebastian Nowozin, Frank Noé, and Ryota Tomioka. "Timewarp: Transferable acceleration of molecular dynamics by learning time-coarsened dynamics." Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2024. [2] Pengmei, Zihan, Liu, Junyu, and Shu, Yinan. "Beyond MD17: the reactive xxMD dataset." Scientific Data, vol. 11, no. 1, 2024, p. 222. Nature Publishing Group UK London.

Reply to the paper representation

We sincerely appreciate the reviewer 5LXW’s opinions on the paper structure, we restructured the introduction and the abstract of the paper to emphasize our RQ.

Reply to the Novelty Issues

We appreciate reviewer 5LXW for their thoughtful and constructive feedback and the opportunity to clarify the novelty and contributions of our work.

On the Novelty of Our Study

While previous studies, such as Zaidi et al. (2022) [1], have explored the benefits of pre-training in GNNs, our work differentiates itself in several significant ways:

Comprehensive Study of Scaling Behaviors in Geometric GNNs

1. First of Its Kind:
   To our knowledge, this paper presents the first extensive examination of scaling laws specifically for all-atom geometric GNNs in molecular learning and the first study on how pre-trained all-atom graph embeddings can be transferred to protein modeling tasks.

2. Wide Range of Configurations:
   We investigate not only model size but also aspect ratios, radial cutoffs, architectural choices, and pre-training datasets, providing practical insights into the design and configuration of geometric GNNs.

3. Diverse Applications:
   Our analysis spans multiple tasks, including kinetic modeling, protein folding classification, force field predictions, and quantum chemical property predictions, showcasing the versatility and limitations of scaling in different contexts.

Insights into Transferability and Expressiveness

We delve into how pre-trained all-atom graph embeddings transfer across various downstream tasks, studying the role of embedding dimensionality and the incorporation of token mixing modules (e.g., MLPs, Transformers, GNNs) for protein modeling. Our findings reveal:

• Simply increasing embedding dimensionality yields diminishing returns.
• Architectures that model interdependencies among structural units lead to significant performance gains.

Novel Perspectives and Benchmarks

1. Improved Evaluation Strategies:
   By employing scaffold splitting in QM9 and utilizing non-equilibrium datasets like xxMD, we challenge conventional evaluation methods, providing more rigorous assessments of model generalization capabilities.

2. Analysis of Scaling Benefits:
   We demonstrate that scaling up model parameters does not uniformly improve performance, particularly when data uncertainties impose fundamental limits—a nuance that prior work has not thoroughly explored.

On Formal Connections to GNN Properties (Oversmoothing, Underreaching)

We acknowledge that we did not explicitly investigate formal properties like oversmoothing or underreaching, as these topics are not central to our study. Instead, our focus lies on:

• Empirical scaling behaviors.
• Transferability of pre-trained embeddings.
• Practical implications for molecular and protein modeling tasks.

We believe that understanding these empirical trends is a critical step before delving into formal theoretical analyses, providing a foundation for future investigations into these topics.

[1] Zaidi, Sheheryar, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. "Pre-training via denoising for molecular property prediction." arXiv preprint arXiv:2206.00133, 2022

Sorry, it has been a busy time for me. I have read the paper again in light of the review comments and responses to them. Overall the author's research goals are more clear to me now. Especially it was not my intention "to frame their work as a continuation of pre-training methods or graph scaling laws, despite our repeated clarifications", but might have misunderstood it.

We kindly ask reviewer 5LXW to clarify the meaning of comparing kinetic modeling "with respect to the speeding up of molecular dynamics" or "how effective the pre-training is."

Here the authors must have misunderstood me completely with what I meant; However let's not dwell on it further.

with respect to the dataset for kinetic modeling tasks: I find it strange when one has to compare with a tutorial example. It raises concerns, whether this is far from real-world.

with respect to Random Split QM9: Here the authors must have misunderstood me. I even marked it as "Strengths", that they considered not only random splitting. However, even if results are overoptimistic, many researchers evaluated their methods for this benchmark (which does not mean, that I think it's necessarily a good dataset or benchmark, but), which could nevertheless give additional insights.

I must acknowledge, however, that I am not an expert in each of the specific subfields from which the authors have drawn their datasets. Apparently, other reviewers found the experiments on the selected datasets to be scientifically more valuable than I did (or at least they did not question it). That is one reason why I am considering possibly raising the score.

clarity of the contributions:

• To me it seems that Geom-GNN is the same as shown in Figure 1 and refers to a specific network architecture. Is this true?
• Is Geom-GNN a new architecture, a generic term, or, an established architecture from somewhere else?
  • If it is a new architecture, the authors should declare it as a contribution.
  • If it is a generic term, then the authors should throughout the paper be more specific which network architecture they are actually referring to.
  • If it is an established architecture from somewhere else, they should properly cite it.

There are further questions on hyperparameter selection:

• When an architecture with pretrained features is compared to an architecture with non-pretrained features, is the hyperparameter selection individual for each of the two cases? Maybe architectures with non-pretrained features need more layers. What is the authors' opinion on this?

Question on Scaffold-QM9: Why are only 5 columns shown in Table 2? Does Scaffold-QM9 have less targets than QM9? If it does not have less targets, why did you skip the other targets?

presentation of the paper:

• There is a typo in the first sentence of the paper: In silico molecular computation and simulation are indispensable tools in modern research for biology, chemistry, and material sciences capable of accelerating the discovery of new drugs and materials, as well as prediction of protein structures and functions
• This sentence seem weird to me: As we have studied a few preliminary application of pre-trained graph embedding, we wonder if the power of features given by Geom-GNNs with varying architectures and configurations.
• The paper is hard to read as many figures, etc. are in the appendix and it is not clear with the figure numbers, whether the figure is expected to be found in the main text or in the appendix

Regarding the reviewer’s suggestion of adding an additional comparison of pre-trained models (apple) with non-pre-trained models with a different number of layers (pear), we believe this would not be a fair comparison as the variables are not controlled. Additionally, we are not hiding any details about hyperparameter selection, and we kindly refer the reviewer to Appendix F for all the hyperparameters.

I am not suggesting new comparisons. But, I wonder, whether it has to be declared as a limitation of your work.

In case you are purely doing a comparison of a fixed network architecture and other hyperparameters (up to those network architecture design parameters and hyperparameters of study) once with pre-trained features and once without and study, e.g. the effect of model width, it seems ok to my understanding.

However, I wonder, e.g., for Table 3. In case you are doing a general comparison in the way like "Do pretrained features have advantages over non-pretrained features", then one would expect an individual hyperparameter optimization for both of these options. I am not absolutely sure, what the goal with Table 3 exactly was, though.

to be more clear, what I mean, I formulate it as an exemplary question to you: Consider you would need to give advice to a machine learning engineer, whose task is to train a model with a minimum validation error. The engineer can either take pretrained features or non-pretrained features. Would you recommend the engineer to a) optimize hyperparameters once for pretrained and once for non-pretrained features or would you recommend b) that the best hyperparameters found with one type of the features also have to be used for the other type of features?

If you opt for a) then you should possibly declare lack of individual hyperparameter optimization as a limitation of your work, in case your goal is/was to show that one type of features is better than the other one. Otherwise it could be misleading to my opinion.

I am not super-convinced the paper should be accepted; Due to the discussion with the authors their research goal became more clear to me. I have the impression that the relevance of this work is a little bit limited. The empirical investigations might have a clear novel aspect, although to me unclear how interesting they are to the community.

At the same time, my point of view might be subjective and I have to recognise that others see the manuscript much more positive. Since I am not the person, who wants to block a potentially useful publication, I raise my score to 6.

for others who read my one or two initial reviews: There might have been misunderstandings from my side. I do not change the text (and the initial ratings except the overall rating) at this stage any more, but please do not necessarily rely/refer to it.

Summary

We sincerely appreciate the reviewer i3nd for the careful proofreading of our paper, which have helped improve the quality of the manuscript. our paper aims to answer a previously unanswered research question in graph representational learning: "Are pre-trained all-atom geometric graph neural network (GNN) representations transferable to protein modeling, and how expressive are they?"

To answer this research question, our contributions are as follows:

1. Scaling Behaviors of Geometric GNNs:
   We studied the scaling behaviors of state-of-the-art geometric GNNs in unsupervised, self-supervised, and supervised setups, rather than focusing on creating new architectures or pre-training objectives.
2. Demonstrating Transferability:
   We pre-trained these GNNs on small molecular datasets and demonstrated their transferability to proteins with all-atom resolution, highlighting their expressiveness in these settings.

Regarding the figure formatting

We took the reviewer's advice and revised the figures accordingly in the manuscript for clearer representation. Per other reviewers’ suggestions, we also restructured the introduction and abstract sections to highlight our RQ.

Regarding the comparison of model depth

We appreciate the reviewer for suggesting comparing the effect of model aspect ratio in the fine-tuning stage. We did the following extra experiments on QM9 to illustrate the effect by fixing the parameter count and varying the depth of the model (this table is integrated in Appendix L as Table 10 on Page 32):

Interestingly, we found there is no obvious trend of aspect ratio effect on the results of blank models, but there is a trend for pre-trained models where deeper models perform better than shallower models. We would also like to refer the reviewer to Figure 2 in Li et al. 2024 [1], where deeper models generally perform better than shallower models on force field tasks.

[1] Li, Yunyang, Yusong Wang, Lin Huang, Han Yang, Xinran Wei, Jia Zhang, Tong Wang, Zun Wang, Bin Shao, and Tie-Yan Liu. "Long-short-range message-passing: A physics-informed framework to capture non-local interaction for scalable molecular dynamics simulation." arXiv preprint arXiv:2304.13542, 2023.

Summary

We appreciate the thoughtful comments and summary from reviewer KcDu, which have helped improve the quality of the manuscript. Before proceeding to further discussions, we would like to clarify several key points.

Our paper does not focus on extending denoising pre-training techniques, investigating pre-training task choices, or developing new model architectures. Instead, it addresses a previously unanswered research question in graph representational learning:
"Are pre-trained all-atom geometric graph neural network (GNN) representations transferable to protein modeling, and how expressive are they?"

To answer this research question, our contributions are as follows:

1. Scaling Behaviors of Geometric GNNs:
   We studied the scaling behaviors of state-of-the-art geometric GNNs in unsupervised, self-supervised, and supervised setups, rather than focusing on creating new architectures or pre-training objectives.
2. Demonstrating Transferability:
   We pre-trained these GNNs on small molecular datasets and demonstrated their transferability to proteins with all-atom resolution, highlighting their expressiveness in these settings.

Performance Increase on VAMP and Fold Classification

We believe the observed performance increases on the VAMP objective and fold classification task do not stem from specific pre-training techniques. Instead, the improvements result from the fact that we pre-trained geometric GNNs, then transferred them in a zero-shot fashion and organically combined with higher-level architectures (Page 5 Table 1). Similarly, the reason why the fold classification results (Page 30 Figure 20) got improved originates from the fact that the inferred graph embedding from pre-trained GNNs contain rich all-atom information instead of applying denoising pre-training to downstream tasks. (And we did not)

Zero-shot Transfer Learners

We appreciate the reviewer for noticing the difference of zero-shot learning setup here when comparing to natural language modeling and computer vision. Using masked language modeling as an example, language models trained to complete the sentence in English can transfer to complete the sentence in Chinese. (Same task, different data domain) In our setup, the backbone network is trained on small molecules with roughly 10-20 heavy atoms with denoising objective, and the backbone network is transferred to infer the atomistic embedding of peptides and proteins and then a separate head is trained with a different objective. (Different task, different data domain) Since the backbone network has never seen protein systems, we think it’s appropriate to claim it as zero-shot transfer for embedding inference.

About other possible pre-training objectives

We appreciate the reviewer for suggesting other possible pre-training objectives. Indeed, there are many other pre-training strategies, such as Noisy Node [1], GraphMVP [2], Frad [3], Uni-Mol [4], etc. The scope of our paper does not lay in comparing the effectiveness of various pre-training strategies nor absolute performance. We chose coordinate denoising as our pre-training objective for its proven effectiveness and simplicity, as the coordinate denoising just has the level of additive Gaussian noise as the only hyperparameter. Combining other objectives inevitably requires more complicated hyperparameter scanning. Considering our extensive experiments in both pre-training and downstream tasks, it would be impractical to contain those comparisons in this paper given the computational budget, where including those comparisons could easily multiplicatively increase the workload.

Adding a node mask objective

In kinetic modeling task, we do not think add a node attribute masking pre-training task (e.g. flipping the atom types) would considerably affect the results, since VAMP aims to find the slow modes of the objective movements, as structural information in a single molecular system where all atom types are fixed across samples. Generally, if the two pre-training objectives focus on different perspectives (coordinate denoising and guessing the right atom type), more parameters would be needed to reach the same loss on each pre-training objective.

[1] Godwin et al. "Simple GNN Regularisation for 3D Molecular Property Prediction and Beyond." International Conference on Learning Representations (ICLR), 2022 [2] Liu et al. "Pre-training molecular graph representation with 3D geometry." arXiv preprint arXiv:2110.07728, 2021. [3] Feng et al. "Fractional Denoising for 3D Molecular Pre-training." International Conference on Machine Learning (ICML), 2023, pp. 9938–9961. PMLR. [4] Zhou et al. "Uni-mol: A universal 3D molecular representation learning framework." 2023.

About including invariant feature-based networks

Let’s reiterate here, we did not focus on a specific GNN architecture nor a specific pre-training technique, and we did not propose a new GNN architecture nor a new pre-training technique in this paper. May the reviewer clarify what we should compare?

Though this discussion is far from the focus of the paper, we can briefly talk about different forms of geometric GNNs. Theoretically, O(d)-equivariant functions can be universally expressed by a collection of scalars [5]. And practically the performance of GemNet/Equiformer/ViSNet/MACE/etc. are not that different on various benchmarks. In line with GVP, PaiNN/ET/ViSNet directly track the vector features by parameterizing the displacement vectors, and those features can be used to predict vector quantities. As the reviewer suggested, the expressiveness of invariant-feature based GNNs increases by the order of features used (SchNet: bond length, DimeNet: bond length/bond angle, GemNet: bond length/bond angle/dihedrals), but higher-order features, for instance, dihedrals require enumerating four indices at the same time. ViSNet includes higher-order features by taking the product of vector representations of source and target nodes, resulting in higher efficiency. As invariant networks do not directly track the vector features, we would have to take the derivative of the scalar output with respect to the input coordinates to predict vector values, where we do not see a point to do so. Regarding group-equivariant networks, we do not foresee considerable differences as explained earlier except for more hyperparameters tuning and computational cost. Again, we would like to emphasize the fact that comparing different GNN architectures or pre-training objectives is not the goal of this paper, and adding this comparison could also easily multiplicatively increase the workload for evaluating every pre-training and downstream task.

[5] Villar, Soledad, David W. Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. "Scalars are universal: Equivariant machine learning, structured like classical physics." Advances in Neural Information Processing Systems (NeurIPS), edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, 2021.

Thank you for your detailed response. However, I still have some concerns based on your explanations.

Zero-shot Transfer Learners

I agree with your claim only to the extent that it represents "zero-shot transfer for embedding inference." If this is indeed what you mean, I strongly recommend explicitly clarifying this in the paper to prevent confusion among readers. Please highlight that the "zero-shot transfer" applies specifically to embedding inference, as the current phrasing may mislead readers into thinking it applies to the entire model.

However, I maintain that this setup aligns more closely with a standard downstream fine-tuning approach. The model is pre-trained using self-supervised tasks and subsequently fine-tuned with task-specific heads. For example, the kinetic modeling task, as you described, involves further training on the VAMP score. This implies that the entire network(with the backbone parameter frozen) used for the task is trained specifically for this task and has seen task-specific data during fine-tuning. Therefore, using the term "Zero-shot Transfer Learning" throughout the paper without clear qualifications may be misleading. I advise you to avoid such potentially thrilling terminology without sufficient explanation and to be precise about where and how zero-shot transfer applies in your work.

Scaling Behavior

I raised concerns regarding the experimental settings, particularly about why other pretraining objectives and models were not utilized. While I understand that pretraining on the PCQM4M dataset is computationally expensive and it may not be feasible to conduct additional experiments during the review, I believe this limitation affects the paper's broader claims. As an experiment-driven paper, such conclusions should be supported by broader experimental settings and benchmarks. It is insufficient to simply discuss how information is processed and aggregated and then claim that models are philosophically similar. Such statements do not substantiate general claims about scaling behaviors.

The experimental settings in this work are limited and do not justify general conclusions. Furthermore, as cited by the authors, existing works [1,2,3] have already explored scaling laws on graphs, testing a range of models to ensure the generalizability of their findings. A similar approach would strengthen the current paper's claims, ensuring that its results are broadly applicable

References:

[1] Frey, Nathan C., et al. “Neural scaling of deep chemical models.” Nature Machine Intelligence 5 (2023): 1297–1305. [2] Chen, Dingshuo, et al. "Uncovering neural scaling laws in molecular representation learning." Advances in Neural Information Processing Systems 36 (2024). [3] Liu, Jingzhe, et al. "Neural scaling laws on graphs." arXiv preprint arXiv:2402.02054 (2024).

Additional Minor Points

The paper claims to characterize the scaling behaviors of Geometric GNNs in unsupervised setups. However, I could not find any unsupervised tasks presented or analyzed across the paper. This claim is mentioned in the abstract and conclusion but is unsupported in the main text.

We appreciate the reviewer for engaging in the discussion and going through all the responses. We wish you a good break :)

Kind Regards, Authors

Thank you for the detailed response to my comments. I appreciate the clarifications provided regarding the experimental setups, the scope of the study, and the comparisons to related works. The authors’ comprehensive exploration of pre-trained and non-pre-trained 3D GNNs across diverse tasks, coupled with their use of rigorous dataset splits, thoroughly addresses my concerns. Additionally, I acknowledge that the distinctions between the supervised, self-supervised, and unsupervised setups have already been reiterated and clarified in the article. These points, combined with the authors’ thorough responses, address my concern, and I am willing to raise my score.

Summary

We sincerely appreciate reviewer GND7 for their thoughtful feedback, which has significantly improved the manuscript's quality. We are grateful for their recognition of our contribution and the novelty of using self-supervised geometric graph neural networks (GNNs) as general molecular/amino acid descriptors containing all-atom information.

Our paper addresses a previously unanswered research question in graph representational learning:
"Are pre-trained all-atom geometric GNN representations transferable to protein modeling, and how expressive are they?"

Rather than extending denoising pre-training techniques or focusing on pre-training task choices or model architectures, our contributions are as follows:

1. Scaling Study of Geometric GNNs:
   We examined the scaling behaviors of state-of-the-art geometric GNNs across unsupervised, self-supervised, and supervised setups, focusing on their general applicability rather than developing new architectures or pre-training objectives.

2. Transferability to Protein Modeling:
   We pre-trained these GNNs on small molecular datasets and demonstrated their transferability to protein modeling tasks with all-atom resolution, showcasing their expressiveness in these settings.

Per the reviewers’ suggestions, we restructured the abstract and introduction to emphasize this research question.

To explore the expressiveness of these representations, we thoroughly evaluated their performance and scaling across various tasks, including:

• Kinetic Modeling/Markov State Modeling
• Protein Folding Classification
  (trained on equilibrium small molecules, transferred to non-equilibrium and equilibrium protein structures)
• Molecular Force Field and Property Predictions
  (both fine-tuning and training from scratch)

Response to Concerns About Limited Molecular Diversity in Evaluation

We thank the reviewer for highlighting this important concern. We agree that QM9, while historically significant, is a limited benchmark due to its age and the prevalent use of random splitting strategies, which favor memorization over generalization. To address this, we adopted scaffold splitting (Page 8, Table 2), as MoleculeNet [1], to better evaluate generalization across molecular scaffolds. Under this more rigorous evaluation, our results show that increasing the number of parameters does not consistently improve performance, in contrast to prior findings.

We also concur that xxMD, which includes non-equilibrium molecular trajectories, is a more informative benchmark than MD17/rMD17. As noted in the xxMD paper [2] (Figure 2 and S4), random splits of poorly sampled trajectories result in minimal distinction between training and test sets.

While we appreciate the reviewer’s suggestions to explore other benchmarks such as Polaris and TDC, these datasets provide only 2D SMILES strings, which are incompatible with geometric GNNs that require 3D molecular information. Furthermore, we are unaware of prior work benchmarking geometric GNNs on these datasets. Single conformers in drug-like property prediction pose additional challenges due to stereochemistry (e.g., chirality) significantly influencing functionality. The limited availability of 3D molecular benchmarks reflects the current focus of all-atom geometric GNNs on quantum chemical property prediction.

To address these gaps, we provide:

• Insights on pre-training GNNs and leveraging their embeddings for downstream tasks.
• Guidance on integrating these embeddings with other architectures in protein modeling (Figure 2/3/14–20, Table 1).
• A new section (Appendix M, Page 31–32) discussing optimal model configurations and dataset splitting strategies for robust evaluation.

[1] Wu, Zhenqin, Ramsundar, Bharath, Feinberg, Evan N., Gomes, Joseph, Geniesse, Caleb, Pappu, Aneesh S., Leswing, Karl, and Pande, Vijay. "MoleculeNet: a benchmark for molecular machine learning." Chemical Science, vol. 9, no. 2, 2018, pp. 513–530. Royal Society of Chemistry. [2] Pengmei, Zihan, Liu, Junyu, and Shu, Yinan. "Beyond MD17: the reactive xxMD dataset." Scientific Data, vol. 11, no. 1, 2024, p. 222. Nature Publishing Group UK London.

Response to Concern About Lack of General Evaluations

We thank the reviewer for raising this point. Our study thoroughly examines the performance and scaling behavior of pre-trained GNNs across diverse tasks, including:

• Single-molecule conformational variety: Force field regression and kinetic modeling.
• Multi-molecule chemical variety: Quantum property prediction.
• Peptide/protein conformational variety: Kinetic modeling.
• Protein biological variety: Folding classification.

We present over 100 pre-training experiments, testing two GNN architectures with varied depth, width, aspect ratio, and radius cutoff configurations on equilibrium and non-equilibrium datasets. Key experiments include:

1. VAMPNet: Three systems (molecules to proteins) tested with different embedding dimensions and token mixers.
2. Folding classification: ProNet (with/without pre-trained all-atom embeddings) tested across dimensions, representing the most comprehensive study in relevant literature.
3. Force field prediction: Two GNNs (non-pre-trained and pre-trained on PCQM and Denali) evaluated across hidden dimensions on 4 molecular systems from xxMD.
4. Molecular property prediction: Two GNNs (non-pre-trained and pre-trained on PCQM) evaluated across hidden dimensions on 5 tasks.

Key Insights About Scaling

Embedding Transfer for Protein Modeling

Pre-trained embeddings without token mixing (e.g., simple summation) show fictitious scaling benefits due to higher-dimensional embeddings retaining more information. Introducing token mixers (e.g., MLPs or Transformers) enables structural unit interdependence modeling, significantly boosting performance. While larger models show slight gains with mixers, there are no sustained scaling benefits, as confirmed by folding classification results (Page 30, Figure 20).

Label Uncertainty Bottlenecks

1. xxMD: Pre-training benefits vary. ViSNet, already near the label uncertainty limit, shows minimal pre-training gains (2 of 4 sets benefit), while the ET architecture shows consistent gains across all 4 sets (Page 9, Table 3; Page 32, Table 9). Scaling data, rather than models, proves more impactful in low-data regimes.

2. QM9: For tasks like quantum chemical property prediction (e.g., homo-lumo gap), the inherent uncertainty in labels limits test-set accuracy regardless of pre-training or model expressiveness. For example, common density functionals exhibit ev-scale differences (Page 16, Figure 5), making milli-ev-scale accuracy unrealistic for QM9.

Additional Data Provided

We also include xxMD-temporal benchmark data (dimensions 64–384) across all setups (non-pre-trained, pre-trained on PCQM/Denali) in the appendix (Page 32, Table 9).

Dear Reviewer GND7,

We just would like to ask if our response has addressed your concerns? If not, we are very happy to discuss and further clarify.

Kind Regards, Authors