Thanks for your valuable comments! We’ll denote the reviewer's concerns with Q and our responses with R. We appreciate the reviewer finding our architectural and theoretical contributions compelling. The main concern of the reviewer focuses on the scalability of the HELM models, which we address below.

Q: Regarding the size of the trained models and training corpora:

• The model size studied ranges from 100m to 1B in the current experiments. I would like to see results on 7B/14B models…would distortion manifest in meaningful way?
• The current pre-training dataset only contains 5B tokens from English Wikipedia….pre-training would not saturate for at least 100B tokens for 1B sized models.

R: Thank you for the feedback and for sharing your personal experience! We agree that the models can benefit from increased parameter count and going beyond 100B tokens combining datasets like C4. Given how computationally demanding this is, however, we focused on the high-quality tokens in Wikipedia to showcase the merits of our architecture in providing a first step towards making such large-scale hyperbolic models a reality.

As the first work to take a fully hyperbolic approach to LLMs and pretraining at the scale of billions of parameters and tokens, our work focuses on demonstrating the effectiveness of HELM under computational expenses that are reasonable within the academic research context. When going from 100M parameters to 1B, we found that the advantages of hyperbolic LLMs persist when compared to Euclidean baselines. Our architecture and implementation, such as the theoretical guarantees and hyperbolic word embedding mentioned in Sec. 5, enables the training stability needed to be the first to train hyperbolic models at this scale.

We also do reasonably expect that the advantage of hyperbolic LLMs would not diminish with increasing parameter counts. Previous works have shown theoretically that Euclidean space will always incur significant distortion when embedding tree-like data regardless of dimensionality and token count [b], while hyperbolic space can achieve arbitrarily low distortion [a]. Thus, there is a theoretical upper bound on the performance of the Euclidean baselines at larger scales, which is not present for HELM models. We plan on releasing well-documented code, and we hope our architecture helps pave the path for 7B/14B models on the 100B of tokens scale to be trained in the future, in settings that are not limited by the computational resources of academic research.

Q: The paper only has results on multi-choice benchmarks...better to also include some instruction following, math and coding benchmarks.

R: Thank you for your valuable suggestion. We agree with the reviewer that additional benchmarks, such as instruction following, would be interesting. Due to limited computational resources in academic research settings, we focused on pretraining on the high-quality tokens from English Wikipedia, instead of much larger-scale data or post-training, such as instruction fine-tuning. We chose the benchmarks due to the simplicity of evaluation on multi-QA as they align with the Wikipedia tokens. Thus, they better reflects the true performance of the learned model, instead of random noise. These benchmarks are also wide used by previously accepted by NeurIPS[49,2], popular open LLMs[19, 11], and LLMs of similar size[52]. To achieve meaningful results in math or coding would require much larger computational resources that would be out of our compute abilities. For this reason, we focused on benchmarks that models can, in fact, answer based on the high-quality Wikipedia tokens it has been trained on.

Q: It is unclear how the HELM model compares to the Euclidean models efficiency wise in terms of both memory and computation.

R: Our proposed methods are efficient in both runtime and memory usage. HELM models are within 1.55X in runtime and 1.11X in memory usage of the Euclidean counterparts for both the 100M and 1B models. In the table below, we show that runtime and memory comparisons between HELM and the Euclidean baselines for one training iteration (roughly 2M tokens). The results shown are for our exact experimental setup ran on 4 A100 GPUs, where we show the worst runtime and memory usage across the ranks. The results are averaged over 10 runs, and standard deviations are not shown since they are within 0.2 of the results. We will include the results in the revision.

Additionally, there has been very little effort to develop optimized tools and libraries for hyperbolic foundation models [23]. In contrast, there is an abundance of the same for Euclidean models (e.g., LLM-Foundry, Flash Attention, DeepSpeed). Development of comparable hyperbolic libraries would further reduce the gap between HELM models and Euclidean baselines.

Finally, while there is overhead for models of the same size, non-Euclidean models require fewer dimensions to embed complex structures when compared to Euclidean models [a,b]. This shows promise for non-Euclidean models to match the performance of Euclidean models with fewer parameters to build parameter-efficient foundation models despite the additional overhead. There is also potential to follow scaling law relationships between parameters and model performance. On the other hand, Euclidean models face heavy dimension-distortion tradeoffs [c].

Q: What are the expected benefits of hyperbolic models for frontier models? Are there any indications of this distortion negatively affecting the frontier models in any tangible ways?

R: The advantage of hyperbolic space is that it could better capture the hierarchical and scale-free properties in texts and language, as shown by prior works. To the best of our knowledge, there currently lacks conclusive work directly connecting numerical distortion to downstream performance in foundation models. Some previous works have explored how using manifolds that more accurately estimate the structure of the data can enhance performance in tasks related to GNNs and word embeddings (e.g., [d]).

Nevertheless, our investigation into the final embedding distribution of our pre-trained HELM models suggests that hyperbolic LLMs can better represent the semantic hierarchies than the Euclidean baselines. Below are examples of how the 1B HELM-MiCE model organizes semantics as compared to the 1B DeepseekV3 model:

As we can see, for HELM-MiCE, more generic words (e.g., subject) are clustered closer to the origin than more specific words (e.g., biology), which has a smaller norm than even more specific words (e.g., photosynthesis). However, this does not necessarily hold for the DeepseekV3 1B model, demonstrating how HELM-MiCE better handles semantic hierarchies. We will provide a low-dimensional visualization of these embeddings in our revision.

This difference manifests when embeddings questions HELM-MiCE correctly answer but DeepseekV3 struggles. For instance, below is a question in the MMLU benchmark categorized by its final embedding norm:

For HELM-MiCE, general words have a smaller norm (e.g., how, if) while the more specific graph theory words are further away from the origin (e.g., connecting, graph). However, for DeepseekV3, general and specific words are sometimes intertwined in the embedding space. The hierarchical organization of the space enables the HELM models to sometimes better navigate the embedding space and obtain better performance.

Q: Is it possible to retain some experts in flat space in the MiCE structure?

R: Thank you for your valuable suggestions! We agree that incorporating Euclidean experts into MiCE is an exciting future direction. Our current implementation of MiCE initializes the experts with a variety of curvature values (from -0.1 to -2.0), where the lower curvature values can simulate a flatter space (like Euclidean space). To truly extend MiCE to Euclidean space, the projection coefficients in equation Sec. 4.2 can be derived from log and exp mappings. As the tangent space of hyperbolic spaces are universally Euclidean, Eq.6 can be extended to incorporate Euclidean space by lifting points to and from the tangent spaces. We will add these discussions to the revision.

References:

[a] Sarkar, R. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pp. 355–366. Springer, 2011.

[b] Lee, J. R., Naor, A., and Peres, Y. Trees and markov convexity, 2007.

[c] Matousek, J. Lectures on Discrete Geometry. Springer, 2002.

[d] Gu, A., Sala, F., Gunel, B., and Re, C. Learning mixed-curvature representations in product spaces. ICLR, 2019.

Thanks for your valuable comments! Thanks for the valuable edit suggestion for our writing as well, we will incorporate the edits into our revision. We also appreciate the reviewer for pointing out the additional related works, which we will incorporate into our revision as well. We’ll denote the reviewer's concerns with Q and our responses with R.

Q: It is still not completely clear why hyperbolic geometry results in a better performance. What has improved in the hyperbolic model? Which type of errors have decreased? In addition, adding qualitative analysis would bring benefits to better understand why the hyperbolic model is performing better.

R: Prior works of hyperbolic language and vision models have found that hyperbolic models better learn the hierarchies in semantic structures [13, 47, 35], where more generic words/concepts tend to cluster in areas of smaller norm and more specific words/concepts tend to have larger norms. Our investigation of final-layer embedding distributions have found that HELM learns representations in a similar way. Below are examples of how the 1B HELM-MiCE model organizes semantics as compared to the 1B DeepseekV3 model:

As we can see, for HELM-MiCE, more generic words (e.g., subject) are clustered closer to the origin than more specific words (e.g., biology), which has a smaller norm than even more specific words (e.g., photosynthesis). However, this does not necessarily hold for the DeepseekV3 1B model, demonstrating how HELM-MiCE better handles semantic hierarchies. We will provide low-dimensional visualization of these embeddings in our revision.

This difference manifests when embeddings questions HELM-MiCE correctly answer but DeepseekV3 struggles. For instance, below is a question in the MMLU benchmark categorized by its final embedding norm:

As we can see, for HELM-MiCE, general words have a smaller norm (e.g., how, if) while the more specific graph theory words are further away from the origin (e.g., connecting, graph). However, for DeepseekV3, general and specific words are sometimes intertwined in the embedding space. The hierarchical organization of the space enables the HELM models to sometimes better navigate the embedding space and obtain better performance.

Q: The reviewer has concerns about the scalability of the proposed approach.

R: We appreciate the reviewer recognizing that this is the first work to train hyperbolic LLMs and hyperbolic models at the scale of billions of parameters and tokens. To address the reviewer’s concerns, we compare the runtime and memory consumption of HELM models and the Euclidean baselines, which we will include in the revision. We found that our proposed methods are efficient and within 1.55X in runtime and 1.11X in memory usage of the Euclidean counterparts. In the table below, we show that runtime and memory comparisons between HELM and the Euclidean baselines for one training iteration (roughly 2M tokens). The results shown are for our exact experimental setup ran on 4 A100 GPUs, where we show the worst runtime and memory usage across the ranks. The results are averaged over 10 runs, and standard deviations are not shown since they are within 0.2 of the results.

As we can see, HELM does not incur significant training overhead across the different model sizes. Additionally, while some efforts have tried to develop tools for hyperbolic foundation models [23], there currently still lacks libraries that optimize training hyperbolic foundation models, whereas there are many that are optimized for training Euclidean models (such as LLM-Foundry, Flash Attention, Deepspeed). Development of comparable hyperbolic libraries would further reduce the gap between HELM models and the Euclidean baselines.

Finally, while there is overhead for models of the same size, non-Euclidean models require fewer dimensions to embed complex structures when compared to Euclidean models [a,b]. This enables the potential for non-Euclidean models to match the performance of Euclidean models with fewer parameters to build parameter-efficient foundation models, countering the computational overhead from non-Euclidean operations and offering the potential to continue the scaling law relationship between parameters and model performance. On the other hand, Euclidean models face heavy dimension-distortion tradeoffs [c].

[a] Sarkar, R. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pp. 355–366. Springer, 2011.

[b] Lee, J. R., Naor, A., and Peres, Y. Trees and markov convexity, 2007.

[c] Matousek, J. Lectures on Discrete Geometry. Springer, 2002.

Q: What is y in equation 2?

R: The y here is f(x), we will fix all of our typos in the revision.

Q: Xj is not defined in Proposition 4.1?

R: The i and j subscripts in the proposition index through the tokens, queries, and keys. We will make this clearer in the revision.

Q: Confusion regarding equations on lines 886 and 887 and lines 912 and 913?

R: The equation on these utilizes the simplified squared Lorentz distance formula from [28]

Q: Why is there no power 2 in the distance in Proposition 4.2?

R: We apologize for our typo; there should be an exponential in the distance

Q: Where did equation 4 come from?

R: In the case of the 2D Euclidean RoPE, the formulation depends on properties of Euclidean rotation operations. Let $R_{\theta_i}$ denote a 2D rotational matrix of angle $\theta_i$ One property of this matrix is that $R_{\theta_i}^TR_{\theta_j} = R_{\theta_i-\theta_j}$ for some other angle $\theta_j$. This identity allows relative position information to be encoded directly into the dot-product-attention: $(R_{\theta_i}q_i)^T(R_{\theta_j}k_j) = (q_i^T)(R_{\theta_i-\theta_j}k_j)$. The properties of rotational operations also enable other theoretical guarantees of RoPE. The higher-dimensional RoPE operations generalize this by putting the 2D matrices on the diagonal. Our proposed HoPE operation takes inspiration from RoPE and is based on Loretzian rotational operations (see [6]). This formulation enables HoPE to encode relative positional information similar to RoPE and achieve similar theoretical guarantees.

Thanks for your valuable comments! We’ll denote the reviewer's concerns with Q and our responses with R.

We appreciate that Reviewer 2Div finds our method novel and compelling. The concerns raised are mostly due to reservations about the significance of our experimental results. We focus our rebuttal below on solidifying our experiments and including the reviewer’s feedback in our new experimental results.

Q: The difference in performance with Euclidean models is minimal and not supported by statistical significance analyses, which, I think, given the relatively small difference in performance, is necessary. How do you ensure that the reported results and improvements are statistically significant?

R: We appreciate the reviewer raising this important point. To further strengthen our results, we performed additional training runs for the DeepseekV3 and HELM-MiCE at the 100M, where the statistics are computed over 3 runs as shown below:

The results show that HELM-MiCE outperforms the Euclidean DeepseekV3 model with statistical significance. We will include these results, along with similar statistics for the 100M dense models, in the revision.

As for going up to the 1B scale parameters, performing multiple training runs is rather compute-heavy, requiring thousands of GPU hours for multiple training sessions of both our model and the baselines. As performing multiple runs at the scale of billions of parameters becomes extremely expensive, it is typically evaluated for a single run across prior literature, even those that were previously accepted by NeurIPS[49,2], popular open LLMs[19, 11], and LLMs of similar size[52]. To showcase the significance and consistency of the results, though, we provide the results at different checkpoints to showcase how the performance gap is consistent between the two and remains beneficial to our framework:

The results demonstrate that the HELM-MiCE model consistently outperforms the Euclidean DeepseekV3 model across training stages, ranging from 4 to 5 billion tokens. We will incorporate these statistics in the revision.

Q: While the authors perform a comparison with Euclidean models of the same size, there is a missing comparison with previous Hyperbolic transformers such as HAT, HNN++, and HyboNet, HypFormer. I believe such a comparison is necessary (even considering different tasks and setups, such as relation prediction) to fully validate the quality of their contribution. How does HELM empirically compare with previous hyperbolic transformers on different tasks? And why were previous hyperbolic approaches not considered in the experiments?

R: We’d like to thank the reviewer for bringing attention to these prior works. Of these prior works for hyperbolic Transformers, three of which (namely HAT, HNN++, and HyboNet) each lacked hyperbolic counterparts of essential modules in modern LLMs. In comparison, all of these modules are available in HELM. We summarize the modules below:

• HAT focuses on proposing hyperbolic self-attention mechanism without the rest of the hyperbolic Transformer architecture (e.g., LayerNorm, residual connection, positional encoding, linear layers), which is used in HELM-D where the operations are mathematically equivalent (as proven in HNN++)
• HNN++ lacks positional encoding, layer normalization, and residual connection
• HyboNet lacked layer normalization, and its positional encoding and residual connection could only be used in conjunction with its linear layers, as they are added as bias terms in the linear layers, greatly limiting the usability of these methods.

Given these limitations, we have not considered these models in LLM baselining of HELM. For completeness, we compare our HELM-D model against these prior works in the machine translation tasks from HyboNet, where the choice of using the dense version of HELM comes from the fact that the prior works were all dense models. The results are shown below. The results for the baselines are taken from [6], and the original study did not show statistical information.

As for Hypformer, which does have all the required modules for language modeling, the differences that set HELM-D apart from Hypformer are:

• Their positional encoding methods (relative for the former and our proposed HoPE for the latter).
• Our proposed hyperbolic RMSNorm and the hyperbolic SwiGLU feedforward network (Appendix B.3.)

For the former, we have provided an ablation study between the positional encoding method proposed in Hypformer and HoPE in our appendix (Table 4), where we created an ablation model HELM-D-L that replaced HoPE with Hypformer’s relative encoding method. HELM-D with HoPE consistently outperformed HELM-D-L. For the latter, we recognize that a direct comparison between HELM and Hypformer would be informative. Thus, we compare a 100M Hypformer model against HELM-D 100M below, where the choice of HELM variant is based on the fact that Hypformer is a dense model:

As shown above, the HELM-D model consistently outperforms Hypformer. Additionally, in comparison to Hypformer, the ablation model HELM-D-L still outperformed Hypformer in 4 out of the 5 tasks (Table 2), highlighting the effectiveness of the hyperbolic RMSNorm and hyperbolic SwiGLU feedforward network.

Q: Could you provide qualitative examples of how your model improves upon Euclidean ones? In particular, how does HELM capture and leverage hierarchical information to deal with the natural language tasks you evaluate on?

R: Prior works of hyperbolic language and vision models have found that hyperbolic models better learn the hierarchies in semantic structures [13, 47, 35], where more generic words/concepts tend to cluster in areas of smaller norm and more specific words/concepts tend to have larger norms. Our investigation of final-layer embedding distributions has found that HELM learns representations in a similar way. Below are examples of how the 1B HELM-MiCE model organizes semantics as compared to the 1B DeepseekV3 model:

As we can see, for HELM-MiCE, more generic words (e.g., subject) are clustered closer to the origin than more specific words (e.g., biology), which has a smaller norm than even more specific words (e.g., photosynthesis). However, this does not necessarily hold for the DeepseekV3 1B model, demonstrating how HELM-MiCE better handles semantic hierarchies. We will provide a low-dimensional visualization of these embeddings in our revision.

This difference manifests when embedding questions HELM-MiCE correctly answers, but DeepseekV3 struggles with. For instance, below is a question in the MMLU benchmark categorized by its final embedding norm:

As we can see, for HELM-MiCE, general words have a smaller norm (e.g., how, if) while the more specific graph theory words are further away from the origin (e.g., connecting, graph). However, for DeepseekV3, general and specific words are sometimes intertwined in the embedding space. The hierarchical organization of the space enables the HELM models to sometimes better navigate the embedding space and obtain better performance.

Thanks for your valuable comments! We’ll denote the reviewer's concerns with Q and our responses with R.

Q: HELM introduces non-trivial engineering overhead (Lorentz ops, curvature routing, expert load-balancing), yet the paper gives no timing or memory benchmarks against standard implementations, leaving practitioners uncertain about the real compute-efficiency trade-offs.

R: While HELM inevitably introduces computational overhead from operations that respect the curvature of the embedding space, our proposed methods are efficient in both runtime and memory usage. HELM models are within 1.55X in runtime and 1.11X in memory usage of the Euclidean counterparts for both the 100M and 1B models. In the table below, we show that runtime and memory comparisons between HELM and the Euclidean baselines for one training iteration (roughly 2M tokens). The results shown are for our exact experimental setup ran on 4 A100 GPUs, where we show the worst runtime and memory usage across the ranks. The results are averaged over 10 runs, and standard deviations are not shown since they are within 0.2 of the results. We will include the results in the revision.

Additionally, there has been very little effort to develop optimized tools and libraries for hyperbolic foundation models [23]. In contrast, there is an abundance of the same for Euclidean models (e.g., LLM-Foundry, Flash Attention, DeepSpeed). Development of comparable hyperbolic libraries would further reduce the gap between HELM models and Euclidean baselines.

Finally, while there is overhead for models of the same size, non-Euclidean models require fewer dimensions to embed complex structures when compared to Euclidean models [a,b]. This shows promise for non-Euclidean models to match the performance of Euclidean models with fewer parameters to build parameter-efficient foundation models despite the additional overhead. There is also potential to follow scaling law relationships between parameters and model performance. On the other hand, Euclidean models face heavy dimension-distortion tradeoffs [c].

Q: The paper does not provide error analysis, case studies, or interpretive examples to show how hyperbolic geometry changes model behavior or what kinds of errors it corrects. Could you provide low-dimensional projections (e.g., Poincaré-disk plots) of the learned embeddings to show how curvature and expert routing organize semantic structure?

R: Prior works of hyperbolic language and vision models have found that hyperbolic models better learn the hierarchies in semantic structures [13, 47, 35], where more generic words/concepts tend to cluster in areas of smaller norm and more specific words/concepts tend to have larger norms. Our investigation of final-layer embedding distributions has found that HELM learns representations in a similar way. Below are examples of how the 1B HELM-MiCE model organizes semantics as compared to the 1B DeepseekV3 model:

For HELM-MiCE, more generic words (e.g., subject) are clustered closer to the origin than more specific words (e.g., biology), which have a smaller norm than even more specific words (e.g., photosynthesis). However, this does not necessarily hold for the DeepseekV3 1B model, demonstrating how HELM-MiCE better handles semantic hierarchies. We will provide low-dimensional visualizations of these embeddings in our revision.

Let us now consider tokens from questions from MMLU that HELM-MiCE correctly answers but DeepseekV3 gets wrong. We examine their final embedding norms:

For HELM-MiCE, general words have a smaller norm (e.g., how, if) while the more specific “graph theory” keywords are further away from the origin (e.g., connecting, graph). However, for DeepseekV3, general and specific words are sometimes intertwined in the embedding space. The hierarchical organization of the space enables the HELM models to sometimes better navigate the embedding space and obtain better performance.

Q: Are each expert’s curvature values fixed or learnable, and how sensitive are results to that choice?

R: For our experiments with MiCE, the experts’ curvatures are trainable and initiated uniformly from -0.1 to -2.0 (e.g., with 4 experts, the curvature values were -0.1, -0.733, -1.367, -2.0). We did not observe the curvature to converge to values that were far from their initial values (within $\pm0.1$ of the initial values). This is presumably due to the wide range of initial curvatures, which we have selected to better capture the geometric variation from token to token. Our ablations in Table 2 demonstrate the effectiveness of variable curvatures for each expert.

Q: All models are trained on only 5B tokens...do you expect the hyperbolic–Euclidean gap to widen, stay flat, or shrink when trained on ≥100 B tokens? And what stability issues could emerge at that scale?

R: The results for the 1B models from checkpoints taken at 4B and 5B tokens demonstrate that HELM-MiCE consistently outperforms the Euclidean baselines, as shown below:

As we scale up data to 5B, the gap between HELM and Euclidean baselines did not show any behavior or signs of diminishing, and the outperformance remained consistent. While one can only fully answer the reviewer’s question by increasing the training corpus to 100B tokens, these results give us more confidence in HELM to outperform the Euclidean baselines at larger scales. Our work presents a stable foundation to achieve training hyperbolic LLMs at such scales, paving the way for future explorations. It is worth noting that Euclidean space will always incur significant distortion when embedding tree-like data regardless of dimensionality and token count [b], while hyperbolic space can achieve arbitrarily low distortion [a]. Thus, there is a theoretical upper bound on the performance of the Euclidean baselines at larger scales, which is not present for HELM models.

In regard to stability, we have employed several techniques to ensure training stability and stable convergence of the losses. Notably, as described in section 5, our modified hyperbolic token embedding method – where the embeddings are learned along the space dimension – improves stability over prior methods.

Q: What motivated using the Lorentz representation over alternatives like the Poincaré ball or Klein model, and did you observe concrete numerical or training-stability advantages?

R: The Lorentz hyperboloid comes with a wide availability of fully hyperbolic operations in the Lorentz space. These are hyperbolic operations that are performed directly on the manifold without requiring repeated mapping to and from the tangent space, and are not generally available in the Poincaré ball or Klein model. Many prior works have found that these Lorentz operations have several advantages, including better computational efficiency, higher expressiveness, and fewer mapping errors [24, 48, 6, 4]. These prior works have found that the fully hyperbolic operations concretely improve numerical stability, whereas tangent space mappings can result in NaN/overflow [24, 4]. Our models are built on top of these efforts and operations (e.g., fully hyperbolic linear layer and residual connection).

Our proposed methods also take advantage of properties that are specific to the Lorentz space, such as formulating of HoPE based on Lorentzian rotations and the interactions between the time and space dimensions. These properties enable us to prove theoretical guarantees for HoPE and hyperbolic RMSNorm that are comparable to the Euclidean counterparts.

References: [a] Sarkar, R. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pp. 355–366. Springer, 2011.

[b] Lee, J. R., Naor, A., and Peres, Y. Trees and markov convexity, 2007.

[c] Matousek, J. Lectures on Discrete Geometry. Springer, 2002.