Large Language-Geometry Model: When LLM meets Equivariance

We sincerely thank you for the time and careful consideration you have given to providing detailed and constructive feedback. Your valuable insights have greatly improved both the technical accuracy and clarity of our manuscript. We have meticulously revised the paper to incorporate your suggestions. Below, we respond to each of your comments point by point.

Responses to Weaknesses

Q1: The explanation for this choice.

In our experiments, we follow the mainstream settings on three different domains. For MD17 and Human Motion Capture datasets, future frame prediction is a common setting that used in many studies, such as Eqmotion[1] and ESTAG. This approach offers a more direct method to assess model capabilities without relying on external solvers. For the antibody design task, we benchmarked against MEAN (ICLR 2023) and GeoAB (ICML 2024), which represent the current mainstream and state-of-the-art approaches in this domain.

Q2： Including additional baselines.

Thank you for your suggestion. Due to the tight time limitation, we have included additional experimental results for only one recent model, Equiformer, on the MD17 dataset. The results in Table F indicate that our EquiLLM still outperforms Equiformer on by a large margin. We'll include more experiments across three tasks to further strengthen credibility.

Table F. The performance of Equiformer on MD17.

Q3: Computational cost & Include the Uni-Mol, AlphaFold, etc.

We appreciate your comments. As shown in Table G on antibody desgin task, our comparative analysis of inference times reveals that EquiLLM requires slightly more computation than state-of-the-art methods (MEAN and GeoAB), but this modest overhead is justified by its substantial accuracy gains. However, it is worth to note that the primary contribution of this paper is not the optimization of computational cost, but rather the integration of Pretained Large Language Models into geometric learning. We excluded large pretrained models (Uni-Mol, AlphaFold) due to their extensive domain-specific pretraining and significantly higher computational costs, which would result in an unfair comparison.

Table G. The inference time on RAbD.

Responses to Questions For Authors:

Q1: More details

Great suggestion! We will include detailed descriptions of the MLP dimensions in the manuscript's appendix and will open-source the code upon paper acceptance.

Q2: Do Invariant features capture "spatial information"

Thank you for your question. In our Equivariant Encoder, equivariant and invariant features interact through message passing and feature updating, with 3D spatial distances explicitly encoded. As established in PAINN (Section 3.3), incorporating distance information across stacked layers implicitly models angular relationships, enabling the output invariant features to inherently capture spatial geometric information.

To validate our design, we have conducted ablation studies by replacing the Equivariant Encoder with two types of invariant encoders: (1) a standard GNN (see Table A, Response to Reviewer c49n) and (2) a canonicalization approach converting equivariant vectors to invariant forms (see Tables D&E, Response to Reviewer M747). Both variants underperformed our original model, confirming that the advantage of equivariant encoder against the invariant counterparts in effectively capturing spatial information.

[1] EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning, CVPR2023.

We are deeply grateful for the time and effort you have dedicated to offering valuable feedback. We have revised the paper to address all your comments. Below, we respond to each of your points in detail.

Questions in Claims And Evidence

Q1:

In our experiments, we evaluated multiple LLMs (GPT, Gemini, and DeepSeek) by directly inputting 3D systems, all of which performed significantly worse than our method. This finding aligns with CrystalLLM's [1] conclusion that direct fine-tuning of LLMs for 3D structure modeling leads to suboptimal performance across 5/8 evaluation metrics.

Q2

While our work focuses on equivariant 3D tasks requiring directional vector outputs, the works you mentioned primarily address invariant 3D tasks. That said, we will expand our discussion in the revised manuscript to incorporate recent advances in 3D structure tokenization.

Q3

We have expanded our discussion of canonicalization in the manuscript and moderate the original claim accordingly.

Q4

We have revised the claim in our paper in the format as you suggested.

Q5

Regarding the claim in the Related Work section, our intention was primarily to contrast our EquiLLM with the LLM+GNN approaches mentioned in the same paragraph. In the revised version, we will add a detailed discussion of ESM3 and removed the original claim to ensure a more precise and rigorous presentation.

Questions in other sections:

Q6: Sanity check.

We have computed the equivariance error for each module, and the overall model indeed satisfies equivariance.

Q7: Ablation study.

We have replaced the equivariant GNN with a normal GNN, with the results shown in Row 2 of Table A in our response to Reviewer c49n. The model exhibits performance degradation, indicating the importance of maintaining E(3) equivariance

Q8: Lack of other, stronger baselines.

As requested, we have conducted the following experiments:

1. Using pretrained models with canonicalization: On the MD17 dataset, we first subtract the mean from coordinates to ensure translational invariance, then perform SVD decomposition for rotational invariance. We directly feed this canonicalized data into GPT-4o-mini, with results shown in Row 2 of Table D. The results demonstrate that while canonicalization indeed improves model's predictive capability, there remains a remarkable performance gap compared to our EquiLLM. This suggests that direct prediction of 3D coordinates remains suboptimal for current LLMs.

Table D. Results of the pretrained models with canonicalization on MD17.

1. Fine-tuning the LLM: Following CrystalLLM, we finetune the llama-7b model on MD17. Due to token length limitations in our prediction task (predicting 10 frames), we select the smallest molecule, Ethanol (3 heavy atoms) for evaluation. We have investigated three experimental settings: (1) 500 samples (the original paper's setup) trained for 10 epochs; (2) 30,000 samples trained for 1 epoch; (3) 30,000 samples trained for 1 epoch with canonicalization. The results in Table E reveal that without canonicalization, 500-sample and 30,000-sample fine-tuned models perform poorly, lagging behind EquiLLM by two orders of magnitude. Remarkably, when we incorporate canonicalization as suggested, the model's predictive performance improved by a factor of 100, even surpassing GPT-4o-mini. This compelling result demonstrates that the combination of canonicalization with direct LLM fine-tuning is indeed promising and warrants further investigation.

Table E. Results of fine-tuning LLM on MD17.

Q9: The clarity of the paper.

In our current implementation, the LLM module remains pretrained (and fixed), while all other module parameters are learnable and trained from scratch.

Q10: The related work also needs work.

In the revised manuscript, we will modify the Related Work section by providing a clearer organization, including a comprehensive discussion of ESM3 and other language model-based methods for structural tasks. We will also provide detailed descriptions of ESTAG in the experimental section.

Q11: Typos & Contextualize their experimental results.

We have implemented all revisions in accordance with your suggestions in the revised manuscript.

Q12: Ensure wasn’t data leakage

Our study utilizes exclusively 3D structural data, whereas all comparative LLMs were pretrained on textual corpora alone. This fundamental modality difference potentially effectively eliminates the risk of data leakage.

[1] Fine-Tuned Language Models Generate Stable Inorganic Materials as Text, ICLR24

We sincerely appreciate your recognition of our work! We are deeply grateful for the time and thoughtful effort you have dedicated to offering such detailed and constructive feedback. Your valuable suggestions have significantly enhanced both the scholarly rigor and presentation of our manuscript. Having carefully revised the paper to reflect your comments, we now address each point in detail below.

Q1: Role of "External Knowledge "& how LLM knowledge influences outcomes1 Role of "External Knowledge.

Thank you for raising this point. Directly explaining how the LLM's knowledge affects the results is a challenging task. Here, we indirectly demonstrate, through ablation experiments, that the model's performance suffers significantly without properly designed prompts to activate the LLM's knowledge. Specifically, when removing antigen, light chain, and heavy chain feature descriptions from antibody design prompts (Table C, Row 1), we observe clear performance degradation, highlighting how domain-specific knowledge enhances EquiLLM's geometric modeling capabilities.

Q2: Independence of Added Model Capacity.

We sincerely appreciate your thoughtful review. We have removed the LLM module for further ablation study as you suggested, with results presented in Row 2 of Table C. The model exhibits significant performance degradation, underscoring the critical role of LLM in our framework. We have included these ablation results in the revised manuscript.

Table C. Further ablation studies on RAbD.

Q3: Formal Proofs of Equivariance & How is E(3)-equivariance assured once data flows through the LLM?

We apologize for any lack of clarity in the current manuscript. To clarify, since the LLM exclusively processes invariant features, its outputs remain strictly invariant. These invariant outputs are then concatenated with the original equivariant features from the encoder through a skip connection, and subsequently processed by the equivariant adaptor. Throughout this data flow, we rigorously maintain E(3)-equivariance. In the revised version, we will add the following contents: (1) a rigorous mathematical proof of the framework's equivariance properties, and (2) a detailed analysis of how data flow maintains E(3)-equivariance throughout the architecture.

Q4: Which aspects of prompt engineering bring the greatest advantages & How sensitive is the method to changes in prompts.

Thank you for this valuable comment! We have conducted more detailed prompt ablations on the RAbD dataset, to investigate the impact of different prompt components on model performance. For antibody design task, the object statistical information encompasses two hierarchical levels:

1. Chain-level features:
   1. Inter-chain centroid distances (prompt 1)
   2. Maximum residue-residue distances within each chain(prompt 2)
2. Residue-level features:
   1. Statistics (max/min/mean) of residue-to-centroid distances per chain(prompt 3)

As shown in Rows 3-5 of Table C, the results demonstrate that chain-level features contribute more significantly to performance improvement compared to residue-level features. We hypothesize that this discrepancy arises because chain-level features provide macroscopic structural information that better facilitates global 3D structure understanding and modeling. These comprehensive ablation results will be included in the revised manuscript.

Q5: How "geometry-aware " prompts are formulated to remain invariant.

Nice question! To guarantee the invariance of geometry-aware prompts input to the LLM, we exclusively employ distance-based statistical measures, which are inherently rotation- and translation-invariant. The three prompt types mentioned above are deliberately designed as distance metrics to preserve E(3)-invariance.

We sincerely appreciate the time and effort you have devoted to providing detailed and constructive feedback. Your insightful comments have been invaluable in improving both the technical quality and clarity of our manuscript. We have carefully revised our paper to incorporate your suggestions. Below, we address each of your points individually.

Q1: Method is built on the LLaVA approach.

Thank you for your comment. There may be some misunderstandings here—our method does not simply replace LLaVA's encoder with an equivariant GNN encoder, as that would compromise the framework's overall equivariance. Instead, EquiLLM introduces an innovative design, as shown in Fig.1. First, the equivariant GNN encoder extracts both equivariant and invariant features, but only the invariant features are fed into the LLM, unlike LLaVA where the LLM receives all encoder outputs. Then, after LLM processing, the output is concatenated with the encoder's equivariant features via a skip connection and passed to the equivariant adapter module to generate both equivariant and invariant predictions.

Q2: The impact of fine-tuning the LLM layers.

Nice suggestion! We set the LLM's parameters to be trainable and fine-tune the model on the SAbDab dataset. However, the experimental results (Table A, first row) show performance degradation, suggesting that fine-tuning may compromise the original information encoded in the LLM, particularly since the dataset used for fine-tuning is not large enough.

Table A. EquiLLM with different backbones.

Q3: Different encoding layers & normal GNN encoders.

Thank you for raising this point. We additionally replace the original equivariant GNN encoder with a normal GNN encoder. As shown in Row 2 of Table A, the model exhibits significant performance degradation, demonstrating the importance of maintaining E(3) equivariance when modeling 3D structures.

Q4: Baseline models are relatively outdated and weak models.

Thank you for this constructive comment. We would like to clarify that ESTAG (NeurIPS 2023) remains the SOTA model on MD17, while MEAN (ICLR 2023) and GeoAB (ICML 2024) are also leading methods on the RAbD dataset. That said, we agree that additional baselines could further validate our approach. To address this, we have included the results of Equiformer (ICLR 2023) on MD17 in Table B, where it still significantly underperforms our model.

Table B. The performance of Equiformer on MD17.

Q5: All hyperparameters are kept consistent.

Thank you for your comment. Our experimental settings follow the ESTAG paper, using identical parameter configurations across all baseline models to ensure fair comparisons. We also explored various hyperparameter choices for the baselines on MD17, but the performance improvements remained marginal compared to our model's results.

Q6: Discuss/acknowledge EGNN.

Great suggestion! We will provide a more comprehensive discussion of EGNN in the revised version.

Q7: Lack of more technical details.

Thank you for this insightful suggestion. In our EquiLLM framework, the LLM module parameters remain frozen during training. In the revised version, we will provide more technical details and a brief introduction to the specific LLM employed in our work.

Q8: The language model is very outdated.

Thank you for your insightful observation. To address this point, we conducted additional evaluations using the Qwen2.5-3B model (see Table A, Row 3). While it shows marginal improvement in AAR, we observe slight decreases in RMSD and TM-score performance. We hypothesize that the language model's capability remains constrained by limited text-3D structure paired data; otherwise, upgrading the LLM component could yield significant gains. We leave this exploration for future work.

Q9: The training size

Nice question! Our experimental setup primarily follows established conventions in the field. For the MD17 and Human Motion datasets, we adopt the same configurations as the EGMN and ESTAG papers, while for the SAbDab dataset, we maintain the settings used in MEAN and GeoAB.