Geometry Informed Tokenization of Molecules for Language Model Generation

Dear Reviewer dPhN,

Thank you for your constructive comments and valuable suggestions! We have made much effort to thoroughly improve our work accordingly and provide responses for each concern here. Please also refer to the revised paper PDF.

W1. Limited contributions

We first extend our gratitude for sharing your specific concerns on contributions. We understand that contribution concerns are very subjective. However, the contribution consideration should be also comprehensive. In order to demonstrate the contributions of our paper clearly, we mainly focus on the distinctions between this paper and the previous works.

1. The track is under-explored: Different from mainstream methods, we convert 3D molecules to 1D discrete sequences and generate using LMs for 3D molecule generation tasks, positioning in an under-explored field.
2. Technique distinctions: The only comparable work using LMs applies coordinates from XYZ files directly. In contrast, Geo2Seq achieves structural completeness and geometric invariance for LM training, which is distinct from all existing works.
3. Flexibility & comprehensive architecture: Geo2Seq operates solely on the input data, which allows independence from model architecture and training techniques and provides reuse flexibility. To further demonstrate the universal applicability of our method, we not only include conventional Transformer architecture of CLMs, but also employ SSMs as LM backbones.
4. Effectiveness & Efficiency: Results under both architectures prove the effectiveness of Geo2Seq. In addition, compared to diffusion models, our method outperforms them with significantly higher efficiency, i.e., 100x faster generation speed.

Although this work stands on the shoulders of techniques from multiple fields, we have demonstrated our unique contributions, which is beneficial to the 3D molecular community.

W2. Does not require extensive theoretical exposition...no new conclusions presented here

Thank you for your comment.

• We understand the importance of balancing theoretical depth with readability. To address this, we have restructured the paper to move a few theoretical details to the appendix, while highlighting necessary contents and the methodology. Please refer to the revised paper.
• Still, we would like to clarify our theoretical contributions the importance of the theoretical formulations presented:
  • While isomorphism and invariance are concepts in graph theory, our paper extends these concepts to another domain by introducing 3D graph isomorphism and its application to molecular systems. Specifically, we propose an extension that incorporates SE(3)-invariance into graph isomorphism, ensuring equivalence of molecular modeling. To the best of our knowledge, this has not been explicitly studied in prior works. The proposed 3D graph isomorphism is a more general and rigorous definition of 3D molecular equivalence and provides a framework for comparing molecules at multiple levels of abstraction.
  • The theoretical formulations also serve as the foundation for our methodology, ensuring the correctness and invariance of the sequence representation. We demonstrated how our theoretical framework directly supports the development of Geo2Seq, which provides a complete SE(3)-invariant mapping of 3D molecular structures into sequences, also a new conclusion we present. The guarantee for this mapping is established via our theoretical derivations.
  • Our work introduces a theoretical contribution that bridges graph theory, geometry, and language models for 3D molecular modeling. The conclusions we present benefit the development of these communities.

Dear Reviewer aUY2,

Thank you for your constructive comments and valuable suggestions! We have made much effort to thoroughly improve our work accordingly and provide responses for each concern here. Please also refer to the revised paper PDF.

W1: ... it would be beneficial to thoroughly discuss and compare their applications as documented in existing literature. This analysis should be included in the preliminary and related work section of the paper ...

• Thanks for your constructive suggestions. We understand your point that canonical ordering and spherical representation have been used in existing literature. We would like to first emphasize that our major contribution lies in providing a solution for LM learning of 3D molecules which can achieve both structural completeness and geometric invariance. Theoretically we demonstrate that using canonicalization and spherical representation can achieve our goal.
• Following your advice, we added a more thorough literature review as follows, which is also included in the paper revision:
  • Spherical representation has been applied in various molecule-related tasks [1], including molecular property prediction [2, 3] and molecule generation [4]. A crucial step in using spherical representation is defining the coordinate frame, i.e. x, y, and z axes. One straightforward approach is to directly use the frame as for the input coordinates, i.e. (1, 0, 0) as x-axis, but this method fails to ensure SE(3) invariance. Instead, SphereNet [2] defines local frames based on a central edge and one reference node to ensure invariance [3]. Similarly, G-SphereNet defines local frames based on focal atoms to compute distance and angle for model generation [4]. These approaches demonstrate the importance of carefully designing the coordinate frame when working with spherical representations in molecule-related tasks.
  • Canonical labeling (CL) has been adopted from the graph theory and used in molecular representation, enabling the conversion of molecules into 1D sequences. This allows for efficient processing and analysis of chemical structures. One of the most popular canonical sequence is canonical SMILES [5], which represents molecules as a string of characters based on the Morgan Algorithm [7] and additional defined rules. SELFIES [6] provides a more robust sting representation to overcome the limitation that some strings do not correspond to valid molecules. However, the application of CL to 3D molecules has yet been studied.

W2: ... it would be beneficial to have additional ablation studies compare the proposed tokenization method with some traditional word-based or subword tokenization methods.

• Thanks for your insightful comment. We agree that different tokenization approaches may significantly affect the generation quality, and we have included ablation study on tokenization approaches in our initial version (see Appendix C, Table 5), including Comp-tokenization (taking the complete real number as a token) and Sub-tokenization (splitting a real number by the decimal point and treat every part as an individual token). We have also tried Char-tokenization (treating each character as a token) and BPE during our method design, but they didn't work well. A detailed comparison is show in the following table, which is also included in the paper revision:

We sincerely thank you for your time! Hope we have addressed your concerns through practical efforts and shown the contributions and significance of our work. We look forward to your reply and further discussions, thanks!

Sincerely,

Authors

[1] Fast and accurate protein structure search with Foldseek, 2024

[2] Spherical message passing for 3D graph networks, 2022

[3] PiFold: Toward effective and efficient protein inverse folding, 2022

[4] An autoregressive flow model for 3D molecular geometry generation from scratch, 2022

[5] SMILES, a chemical language and information system, 1988

[6] Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation, 2020

[7] The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service, 1965

Dear Reviewer Sjkt,

Thank you for your constructive comments and valuable suggestions! We have made much effort to thoroughly improve our work accordingly and provide responses for each concern here. Please also refer to the revised paper PDF.

W1. Why not use SMILES canonicalization

• Thank you for the point. As we specified in Sec 3.1 and Appx. C, our theoretical analyses and derivations apply to all rigorous CL algorithms. We do not propose a new CL algorithm, and all rigorous CL algorithms are usable here, while our contribution lies in achieving structural completeness and geometric invariance for LM learning of 3D molecules. Thus we believe that the choice of CL algorithm is not a weakness of our work. In the paper, we implement Nauty Algorithm for 3D molecules because:
  1. Its implementation has the best time efficiency among all existing CL algorithms.
  2. Canonical SMILES is based on the Morgan CL Algorithm, which is proven to be incomplete for isomorphism corner cases (such as two triangles versus one hexagon). While canonical SMILES solve corner cases by manual restrictions, it's a bit less elegant than Nauty Algorithm.
• We have studied the influence of ordering algorithms in Ablation on atom order of Appendix C. Following your advice, we further extend to include canonical SMILES order, as shown below.

• Our implemented Nauty Algorithm for 3D molecules can perform partitioning of graph vertices (a step in Nauty) with/without locality considered. Canonicalization with locality considered can lead to better results, due to the importance of neighboring atom interactions in molecular evaluations. Given the similar nature, canonical SMILES produces a very similar ordering with "Nauty with locality", thus close in performances.

W2. No strict definition of bonds in quantum chemistry...graph isomorphism necessity?

• Thank you for your thoughtful question. While molecular bonds in our paper are generated using RDKit, the introduction of graph isomorphism serves a foundational role for several reasons:
  • In graph theory, graph isomorphism is not restricted to comparing the presence or absence of edges; the concept allows us to model and compare node attributes, and thus features such as molecular geometry or spatial configurations. Graph isomorphism provides a general framework for comparing molecules at multiple levels of abstraction, beyond bond-centric representation.
  • Also, the introduction of graph isomorphism lays necessary theoretical groundwork for our proposed 3D graph isomorphism, which extends to incorporate spatial geometry through SE(3)-invariant transformations. The defined 3D graph isomorphism allows us to compare 3D molecular graphs, ensuring that we account for both topological and geometric equivalence of molecules, and can generalize to different bonding schemes.
  • The introduction of 3D graph isomorphism is a more general and rigorous definition of 3D molecular equivalence. We believe this is meaningful theoretical contribution providing a unified perspective. Our method, Geo2Seq, also uses this theoretical foundation to guarantee the correctness and invariance of the sequence representation. The process uses the 3D graph isomorphism to ensure that equivalent 3D molecular graphs are mapped to the same representation, thus achieving SE(3)-invariant sequences.

W3 & Q2. Limiting the range...distance remains

Thank you for the insightful question and advice! We consider W3 and Q2 together here.

• It is true that in our current setting, the distance is not restricted numerically like the azimuth and polar angles. We appreciate your suggestion regarding adopting a local coordinate system to manage distances. To further analyze, we provide ablation studies on local distances, where our distances to the global frame are replaced with relative distances to the previous atom (except for the first atom) while the angles remain the same. This reduces the scale of the distances.
• We provide experiment results using 8-layer GPT models on QM9 dataset:

Dear Reviewer Sjkt,

Thank you for replying to our rebuttal! We are glad to know that most of your concerns have been addressed, and would like to respond to your remaining concern as follows.

Local distances ... relative distances to the previous atom while keeping the angle component in the global frame ... reconstruction task cannot be completed

• We would like to clarify that in our ablation studies on local distances, when we replace distances to the global frame with relative distances to the previous atom, the angles are not constructed in the global frame but constructed along with the same relativity. You are definitely correct in that reconstruction cannot be completed if relative distances are used while keeping the angle component in the global frame. The distances and two angles need to be constructed under the same frame, thus we adopt relative distances and use the same way as Geo2Seq to define azimuth and polar angles under relativity. Specifically:
  • For each atom $A_i$, we compute local distance $d = \| \mathbf{r}\_{i} - \mathbf{r}\_{i-1} \|$ where $\mathbf{r}\_{I}$ and $\mathbf{r}\_{i-1}$ are the positions of $A\_i$ and $A\_{i-1}$. We compute angles, where polar angle $\theta$ is the angle between $\mathbf{r}\_i - \mathbf{r}\_{i-1}$ and the local $z$-axis, and Azimuth angle $\phi$ is the angle between the projection of $\mathbf{r}\_i - \mathbf{r}\_{i-1}$ onto the local $x-y$ plane and the local $x$-axis. For the locality, we define a reference frame with local axes using two adjacent atoms, $z$-axis as $\mathbf{v} = \mathbf{r}\_{i-1} - \mathbf{r}\_{i-2}$, $x$-axis as vector orthogonal to $\mathbf{v}$, and $y$-axis as $\mathbf{y} = \mathbf{z} \times \mathbf{x}$. For $A\_1$, the global frame is still used as the reference.
  • For reconstruction we can do the following steps. First we perform global frame initialization similar to Geo2Seq; the global coordinate of the first atom is $\mathbf{r}\_1 = [0, 0, 0]$, and the coordinate of the second atom is set along direction $\mathbf{r}\_2 = [d\_2, \pi/2, 0]$. The third atom defines the initial reference frame. We place it in 3D space based on the distance $d\_3$ and angles $\theta\_3, \phi\_3$, $\mathbf{r}\_3 = \mathbf{r}\_2 + d\_3 \cdot \begin{bmatrix} \sin(\theta\_3) \cos(\phi\_3) \\\\ \sin(\theta\_3) \sin(\phi\_3) \\\\ \cos(\theta\_3) \end{bmatrix}.$ For $i \geq 4$, compute the local frame at $A\_{i-1}$, where local $z$-axis is $\mathbf{z}\_{i-1} = \frac{\mathbf{r}\_{i-1} - \mathbf{r}\_{i-2}}{\|\mathbf{r}\_{i-1} - \mathbf{r}\_{i-2}\|}$, local $x$-axis is $\mathbf{x}\_{i-1} = \frac{\mathbf{x}\_g - (\mathbf{z}\_{i-1} \cdot \mathbf{x}\_g) \cdot \mathbf{z}\_{i-1}}{\|\mathbf{x}\_g - (\mathbf{z}\_{i-1} \cdot \mathbf{x}\_g) \cdot \mathbf{z}\_{i-1}\|}$, and local $y$-axis: $\mathbf{y}\_{i-1} = \mathbf{z}\_{i-1} \times \mathbf{x}\_{i-1}$. We can reconstruct global cartesian coordinate for $A\_i$. Using $d\_i, \theta\_i, \phi\_i$, calculate the Cartesian displacement of $A\_i$ in the local reference frame $\Delta \mathbf{r}\_i = d\_i \cdot \begin{bmatrix} \sin(\theta\_i) \cos(\phi\_i) \\\\ \sin(\theta\_i) \sin(\phi\_i) \\\\ \cos(\theta\_i) \end{bmatrix}.$ Transform this displacement into the global frame using the local axes, $\Delta \mathbf{r}\_i^\text{global} = \Delta \mathbf{r}\_i \cdot \begin{bmatrix} \mathbf{x}\_{i-1} \\\\ \mathbf{y}\_{i-1} \\\\ \mathbf{z}\_{i-1} \end{bmatrix}^T.$ And compute the global coordinate $\mathbf{r}\_i = \mathbf{r}\_{i-1} + \Delta \mathbf{r}\_i^\text{global}$. Wo do this for all subsequent atoms $A\_{i}$ to reconstruct the global coordinates for the molecule.
• At your request, we would also like to share our code through this anonymous link: https://anonymous.4open.science/r/DGTHM/

We sincerely thank you for your time! Hope we have addressed your concerns through practical efforts and shown the contributions and significance of our work. We believe our work would be beneficial to the 3D molecular community and would genuinely appreciate your recognition. We hope you can let us know if we have addressed your concern and reconsider the score if we have. We also welcome any additional discussions or feedback, thanks!

Sincerely,

Authors

Dear Reviewer Y6S8,

Thank you for your appreciation of our work and insightful comments! We have made much effort to thoroughly improve our work accordingly and provide responses for each concern here. Please also refer to the revised paper PDF.

W1 & W2: more (LM-based) baselines:

• Thanks for your valuable comments. We would like to clarify that the track of using LMs for 3D molecule generation is largely under-explored, and the only comparable work we noticed is [1], which applies coordinates from XYZ files directly. The result is included in the Ablation study (please see Table 4 original coordinates, implemented by ourselves).
• In addition, we have included 3 more baselines, JODO [2], MiDi [3], and Symphony [4] in our initial paper with many additional evaluation metrics [2,3,4]. Please refer to Appendix D.3, Tables 6, 7, 8, 9, 10, and 11. Vast experiments show the effectiveness of our method. Note that JODO and MiDi are diffusion models jointly generating 2D and 3D molecular information. Since the settings of these papers are not the same as ours, we exclude them in experiments of the main paper and provide in the appendix.

Q1: ... There can be more case studies and discussion on how additional capabilities (e.g. in-context learning) of LLMs can be leveraged with this representation. Do the authors see any emergent abilities? Are prompt-based approaches possible?

• Thanks for your insightful comment. First, we would like to emphasize that one of our major motivation is to make use of the rapidly developing power of LMs; thus our Geo2Seq is orthogonal to the selected LM architectures. In the paper, we focus on the design of Geo2Seq and evaluate it using two backbone models (GPT and Mamba). Since our current method is well grounded and has shown empirical significance and great performances, we leave the exploration of further LLM techniques as a future direction.
• However, we agree that some advanced LLMs techniques and case studies can introduce valuable insights into the field. We have provided some scaling law studies (Appendix E.1) since this provides typical insights regarding LMs. As shown in Table 14 and 15, LMs' performances on molecules grow significantly with parameter size increase, which is very similar to the emergent abilities widely-recognized in NLP tasks. We also provide case studies ( Appendix E.2) in our paper, where we show error cases with repetition or hallucinations, similar to those observed in NLP tasks. We observed these cases especially when not trained to best convergence, but they will be rarer if the model well converges.
• Regarding prompt-based approaches, we believe that they can be applied here. In fact, for the controllable generation setting, we provide the model with property values as the prompt, and let the model generate novel molecules with this desired property. Moreover, we can leverage in-context learning to perform controllable generation on new properties without retraining a model for new properties. In-context learning can be also be extended to property prediction task, which could be an interesting research direction and we leave it for future work.

We sincerely thank you for your time! Hope we have addressed your concerns through practical efforts and shown the contributions and significance of our work. We look forward to your reply and further discussions, thanks!

Sincerely,

Authors

[1] Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files, 2023

[2] Learning joint 2d & 3d diffusion models for complete molecule generation, 2023

[3] Midi: Mixed graph and 3d denoising diffusion for molecule generation, 2023

[4] Symphony: Symmetry-equivariant point-centered spherical harmonics for molecule generation

Dear Reviewer EoEL,

Thank you for your constructive comments and valuable suggestions! We have made much effort to thoroughly improve our work accordingly and provide responses for each concern here. Please also refer to the revised paper PDF.

W1: As stated in the strength section, even though the combination is novel, the individual components have been used before.

• We appreciate your acknowledgement of the novelty of our proposed Geo2Seq. We understand your point that parts of our method have been used in previous application. However, we would like to emphasize that our major contribution lies in providing a solution for LM learning of 3D molecules which can achieve both structural completeness and geometric invariance. Theoretically we demonstrate that using canonicalization and spherical representation can achieve our goal.
• More specifically, we summarize the key distinctions of our work as follows:

1. The track is under-explored: Different from mainstream methods, we convert 3D molecules to 1D discrete sequences and generate using LMs for 3D molecule generation tasks, positioning in an under-explored field.
2. Technique distinctions: The only comparable work using LMs applies coordinates from XYZ files directly. In contrast, Geo2Seq achieves structural completeness and geometric invariance for LM training, which is distinct from all existing works.
3. Flexibility & comprehensive architecture: Geo2Seq operates solely on the input data, which allows independence from model architecture and training techniques and provides reuse flexibility. To further demonstrate the universal applicability of our method, we not only include conventional Transformer architecture of CLMs, but also employ SSMs as LM backbones.
4. Effectiveness & Efficiency: Results under both architectures prove the effectiveness of Geo2Seq. In addition, compared to diffusion models, our method outperforms them with significantly higher efficiency, i.e., 100x faster generation speed.

• In short, although this work stands on the shoulders of techniques from multiple fields, we have demonstrated our unique contributions, which is beneficial to the 3D molecular community.

W2: ...From table-4 (ablation study on different atom orders) the gap between the top 3-4 methods is not very large. ... should efficiency and memory ... be used to pick a winner? Did the author do such an analysis?

• Thank you for your comment. We agree that the differences may not seem very large, but we would like to point out that there are still noticeable differences of around 0.5% (atom stability), 5.8% (molecule stability), 2.5% (valid), 8.6% (valid & unique) between these top methods (Canonical-locality vs BFS). Notably, atom stability is a relatively easy metric, thus most methods can achieve 90%. Moreover, while the absolute improvement in these metrics is not large, utilizing canonical rigorousness to achieve these performance gains towards SoTA is both theoretically and practically meaningful. Thus we emphasize the information completeness and select canonical ordering, which grounds our contributions in well-defined theoretical principles.
• Regarding efficiency, yes, we take it into consideration when evaluating our method. In fact, the results presented in the main tables (Table 1 and Table 2) are based on Canonical-locality (Nauty algorithm) atom order because of its top performance, high efficiency, and rigorousness. In detail:

Dear Reviewer JuWn,

Thank you for your constructive comments and valuable suggestions! We have made much effort to thoroughly improve our work accordingly and provide responses for each concern here. Please also refer to the revised paper PDF.

W1: Controllable generation experiment was under-explained ... analysis for each of these cases of instances where it succeeded and other methods failed or ... some explanation for each property of how the property relates to the 3-d configuration and what information relevant to that configuration is preserved or removed under the new framework, especially for readers with general knowledge of LLM's but without knowledge of chemistry.

Thanks for your comments.

• As shown in Table 2, among all six properties, our method outperforms the diffusion methods EDM and GEOLDM by a large margin, moving a significant step towards the lower bound. We can observe that, our method has significant improvement over baseline methods on the controllable generation, while the improvement on random generation is relatively small. This means that although baseline methods (diffusion-based methods) can generate novel molecules, they can't use conditional information appropriately, showing the advantage of our LM-based methods.

  In detail, compared to LMs which can use prompt as conditions, baseline methods (e.g. diffusion based methods) take a property c as additional input which is concatenated to the nodes features. This makes the model hard to train and limit their performance.

• Regarding the properties, all properties are related to the 3D configurations. For example, for different conformers of the same molecules (considering the same atoms and chemical bonds, but some bonds are rotated, therefore, the 3D structure becomes different), the property value of mu (Dipole moment) can change a lot. This is because the dipole moment is a vector quantity that depends on the orientation of the molecule in 3D space. As the theoretical analysis guarantees, Geo2Seq can achieve structural completeness and geometric invariance with the numerical precision of $ϵ\leq|10-b|/2$, since canonical labeling reduces graph structures to 1D sequences with no information loss regarding graph isomorphism, and SE(3)-invariant spherical representations that ensure no 3D information loss. Thus atomic information and nearly-precise 3D information are preserved, which the properties depend on. Below are the details of these properties:

  • α Polarizability: Tendency of a molecule to acquire an electric dipole moment when subjected to an external electric field.
  • HOMO: Highest occupied molecular orbital energy.
  • LUMO: Lowest unoccupied molecular orbital energy.
  • Gap: The energy difference between HOMO and LUMO.
  • µ: Dipole moment which depends on the orientation of the molecule in 3D space.
  • Cv: Heat capacity at 298.15K which is related to the vibration of molecules

• We have added these information in the paper revision.

Dear Reviewer J6yi,

Thank you for your constructive comments and valuable suggestions! We have made much effort to thoroughly improve our work accordingly and provide responses for each concern here. Please also refer to the revised paper PDF.

W1. Stylistic points

1. Thank you for your question. "Molecular science" in this context refers to the interdisciplinary study of molecules, their structures, properties, interactions, and functions. This encompasses fields such as computational chemistry, drug discovery, material science, and biochemistry, where understanding and modeling molecular behavior are crucial. In our paper, we specifically highlight how language models (LMs) contribute to molecular science by representing molecules and enabling tasks like molecular property prediction, molecule generation, and interaction modeling. We use this term following prior works such as [1,2] and chemistry journals such as [3,4,5].
2. Thank you for the suggestion. It's true that both terms have been used for similar tasks [6,7,8], and we choose to use "controllable generation" following our baselines EDM[9] and GeoLDM[10] in order to keep consistency. To improve clarity, we have added the discussion on the equivalence of these two terms.
3. By "NLP and beyond," we refer to the application of large language models not only in traditional natural language processing (NLP) tasks such as text generation, sentiment analysis, and machine translation but also in domains outside conventional NLP. These include areas like daily Q&A, education, software development, scientific discovery, and multi-modal tasks. We have added explanations in the paper revision.

W2. Position embeddings for token understanding

• Thank you for the question. We would like to clarify that one of our major motivation is to make use of the rapidly developing power of LMs; thus our goal is to design a model-agnostic method, without any need of model-level modification. We admit that position embeddings for coordinates might help the model achieve similar token understanding; however, it requires specific embedding design, which needs to be done separately for each LM architecture and can be infeasible for modern LMs released via APIs. Geo2Seq operates solely on the input data, which allows independence from model architecture and training techniques and provides reuse flexibility. In this case, we can apply Geo2Seq on newly released LMs, which we believe is more efficient compared to the workload of positional embedding design.
• To improve clarity, we have further emphasized this motivation and advantage of Geo2Seq in the paper revision.

W3. Vocabulary size... overfitting?

• First, we would like to clarify that the vocabulary size reported ('1.8K for QM9' and '16K for Geom-Drug') is not for single digit number; we use the precision of 2 decimal places, and this vocabulary covers the numerical token range of distance and angle values, as specified in Sec 3.3. In addition, we believe a vocabulary size of 1.8k/16k is not abnormal for LMs with 80M/100M parameters, since for a 100M-parameterLM in NLP, a vocabulary size of 10k to 50k tokens is typical [11,12].
• Next, we would like to clarify that our results are not from overfitting. We use the cross entropy loss for LM training, which does not have direct mathematical relationships with any of the chemical metrics, such as validity, stability, uniqueness, novelty, and so on. We use a validation set to monitor whether overfitting happens. For QM9 dataset, we select checkpoint before overfitting happens to perform final evaluation so that we get good performances across all metrics. Further experiments show that one metric validity does get better as the LM overfits, but other metrics such as stability, uniqueness and novelty would suffer when the LM overfits. For GEOM-DRUG dataset, we have not observed overfitting throughout our training of 20 and 25 epoch for GPT and Mamba models, respectively. We include these information in the paper revision as well.

Dear Reviewer 3p53,

Thank you for your appreciation of our work and insightful comments! We provide responses for each concern here. Following other reviewers' advice, we have made much effort to thoroughly improve our work. Please also refer to the revised paper PDF and our responses to other reviewers. Also, we welcome any additional discussions or feedback. Thank you!

Sincerely,

Authors