Hyperbolic Fine-Tuning for Large Language Models

Q: Incremental improvements may not justify increased complexity and computational overhead in practical deployments.

A: We thank the reviewer for raising the practical consideration of the trade-off between performance gains and computational overhead. We agree that any added complexity must be justified by meaningful improvements.

While HypLoRA introduces hyperbolic projection operations, we have taken care to ensure its practical overhead is minimal and justified by its performance benefits. In terms of model complexity, the number of additional trainable parameters in HypLoRA is negligible, remaining nearly identical to that of the standard LoRA method (as shown in Tables 3 and 7).

Regarding computational overhead, our analysis in Section 5.2 shows that while inference time is slightly increased compared to LoRA, HypLoRA remains more efficient than other competitive methods like DoRA (Figure 2). Furthermore, memory consumption is comparable to LoRA (Table 7) and in the table below, ensuring our method remains a practical choice for deployment.

GPU Allocated Memory (Finetuning, in GB)

Besides, the performance improvements offered by HypLoRA are not merely incremental, but are particularly significant on more challenging reasoning tasks. For instance, on the complex GSM8K and AQUA arithmetic benchmarks, HypLoRA delivers substantial gains over LoRA across multiple models, such as a +9.8% and +12.2% improvement with Gemma-7B, respectively (Table 3). Similarly, on commonsense reasoning, we observe consistent gains of +1.8-3.0% on average (Table 4). This demonstrates that the benefits of explicitly modeling the latent hyperbolic geometry are most pronounced where complex, hierarchical relationships must be understood. Therefore, we believe the slight and manageable increase in computational cost is a worthwhile trade-off for the significant and targeted performance enhancements HypLoRA provides on difficult reasoning tasks.

Thank you for your thorough review and constructive feedback. In what follows, we use “Q” to denote each question, weakness, or issue raised and “A” for our response.

Q1: Base models’ Performance

A1: To address the reviewer’s concerns and in the limit of time, we report the arithmetic reasoning performance across four base LLMs, and on Gemma 3-4B for the commonsense reasoning evaluations.

Arithmetic reasoning task:

Commonsense reasoning task:

These baseline results clearly demonstrate the value of fine-tuning. More importantly, these benchmarks further highlight the contribution of our proposed method, HypLoRA. Compared to both the base LLM and the LoRA baseline, HypLoRA consistently outperforms across most evaluations in both arithmetic and commonsense reasoning tasks.

Q2: Motivation is disconnected from experimental analysis: The experimental analysis can be better aligned with the core motivation of the paper.

A2: Thanks for your suggestion. In this response, we present a quantitative study of the embeddings finetuned using HypLoRA. We extend our hyperbolicity analysis beyond the input embeddings. We examine the relative hyperbolicity of the final hidden layer representations of Gemma 7B and Gemma 3-4B across five datasets, two focused on math reasoning (AQuA, GSM8K) and three on commonsense reasoning (ARC-Challenge, Winogrande, OpenbookQA). We conduct this analysis on both the base models and their fine-tuned counterparts using three PEFT methods: LoRA, DoRA, and HypLoRA.

First, as a baseline, the hyperbolicity of the initial, non-contextualized token embeddings is presented in Table 1.

Hyperbolicity of Inital Token Embeddings (Table1)

Hyperbolicity of Final Hidden Layer Embeddings in Gemma 7B: (Table2-1)

Hyperbolicity of Final Hidden Layer Embeddings in Gemma3 4B: (Table2-2)

The results for the final hidden layer embeddings are shown in Table 2-1 and .Table 2-2. Our analysis reveals several key findings: First, the final hidden states of the base models exhibit notable hyperbolicity, though it is generally less pronounced (i.e., higher $\delta$ values) than in the initial embeddings. Second, HypLoRA consistently learns final representations with a higher degree of hyperbolicity (lower δ values) across almost all datasets and models. This provides strong empirical evidence that our method actively preserves and enhances the hierarchical structure of the representations throughout the model, aligning the final contextualized embeddings with the geometric biases that are beneficial for reasoning. We also observe that as dataset complexity increases, the final hidden layer exhibits correspondingly higher relative hyperbolicity value.

For instance, AQuA, a particularly challenging dataset, shows a relative hyperbolicity of 0.31 in Gemma 7B and 0.17 in Gemma 3-4B. When these models were fine-tuned with HypLoRA, the degree of hyperbolicity increased (i.e., the relative hyperbolicity value decreased). In Gemma 7B, the drop from 0.31 to 0.22 led to a significant improvement in accuracy, reaching 46.5%, compared to LoRA’s 34.3%, which had a relative hyperbolicity of 0.24. Similar trends are observed in Gemma 3-4B as well. Moreover, in commonsense reasoning tasks, the effect is even more pronounced: while LoRA and DoRA tend to reduce the degree of hyperbolicity relative to the base LLM, HypLoRA increases it. As a result, the performance improvement with HypLoRA is both evident and consistent.

Q3: As shown in Table 3, the LLaMA3-8B model fine-tuned with LoRA achieves 69.8% accuracy on GSM8K. However, the official technical report for LLaMA-3 (https://ai.meta.com/blog/meta-llama-3/) shows an 8-shot CoT accuracy of GSM8K is about 79.6%. Can you explain the evaluation settings that lead to this result? This significant difference raises potential concerns about the strength of the LoRA baseline used for comparison. If the baseline is not optimally tuned, the gains reported by HypLoRA may be partially due to a suboptimal starting point rather than simply the advantages of the hyperbolic method.

A3: We follow the experimental setup outlined in (Hu et al.)[1], which has been adapted by various works. For arithmetic and commonsense reasoning tasks, we finetune the LLMs using Math10k and Commense170k datasets, respectively proposed in (Hu et al.)[1]. For the PEFT baselines, we adopt standard hyperparameters: lora rank 32, lora alpha of 64 and apply it at QKVUD. We use a batch size of 16, micro-batch size of 4, a learning rate of 3e-4, and train for 3 epochs with a maximum sequence length of 256 tokens. These settings are consistent with Hu et al[1]., DoRA, and are also similar to those used in the S2FT[2] framework. Notably, similar LoRA performance on GSM8K can be observed in the S2FT paper. Importantly, the 8-shot accuracy reported in the official LLaMA-3 blog refers to the chain-of-thought prompting, which is not directly comparable to our setting, where the model is fine-tuned using Math10K, a dataset that includes GSM8K, AQuA, and MAWPS. Thus, drawing a direct comparison between these two settings would not be valid, as they represent different evaluation paradigms.

[1]Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.P., Bing, L., Xu, X., Poria, S. and Lee, R.K.W., 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. EMNLP 2023

[2]Yang, X., Leng, J., Guo, G., Zhao, J., Nakada, R., Zhang, L., Yao, H. and Chen, B., 2024. S $^{2}$FT: Efficient, scalable and generalizable LLM fine-tuning by structured sparsity. Advances in Neural Information Processing Systems, 37, pp.59912-59947.

Q: Regarding the stability of fine-tuning: It is well known that the performance of LoRA can be sensitive to hyperparameter choices (e.g., learning rate, rank "r"). The paper compares the final performance of LoRA and HypLoRA, but a deeper analysis of training stability would be helpful. Can you comment on the stability of HypLoRA compared to LoRA? Are the reported results based on the average of multiple runs with different random seeds to explain the variance?

A: We thank the reviewer for the suggestion. The reported results were from the best single run across different curvature values (0.5, 1.0, 2.0). To address stability concerns and in limit of time, we report mean and std over 3 runs for LLaMA-3B (LoRA vs. HypLoRA) and for Gemma3-4B (HypLoRA). While HypLoRA shows slightly higher variance in some cases, it consistently outperforms LoRA on most arithmetic tasks. Improving stability is an exciting direction for future work.

Thank you for your constructive feedback. In the follows, "Q” denotes any question raised, and “A” denotes response.

Q1: Explanation of cancellation effect

A1: The cancellation effect occurs because standard hyperbolic neural network [1,2] approaches apply transformations in the tangent space at the origin, requiring the sequence : Euclidean embedding → exponential map to hyperbolic space → logarithmic map to tangent space → linear transformation → exponential map back to hyperbolic space → projection to Euclidean space.

When these operations are chained together, the maps are mutually inverse and effectively cancel out, reducing the entire sequence to approximately the original Euclidean transformation BAx without preserving the beneficial hyperbolic geometry. Logrithmic map is the inverse of exponentail map.

This is why we developed Direct Lorentz Low-rank Transformation (LLR) that operates directly on the hyperbolic manifold without intermediate tangent space mappings, preserving the hyperbolic structure.

[1] Hyperbolic Graph Convolutional Neural Networks

[2] Hyperbolic Neural Networks

Q2: Motivation of direct LLR; the reason of transforming the space-like dimensions

A2: Regarding the decision to transform the hidden states back to Euclidean space, this is a fundamental requirement for our method's design as a parameter-efficient adapter. HypLoRA is intended to fine-tune existing, pre-trained LLMs, whose core components—such as the attention mechanism, MLP layers, and residual connections—are frozen and operate exclusively on Euclidean vectors. For our adapter to be compatible with this architecture, its output must be projected back into Euclidean space so it can be integrated into the model's residual stream. This "project-transform-project back" approach allows us to leverage the benefits of hyperbolic geometry in a localized, "plug-and-play" manner for specific tasks, avoiding the prohibitive cost and complexity of redesigning and retraining the entire language model in a new geometry.

For the second question about transforming space-like dimensions, as detailed in Section 5 and Appendix D, our spatial-like transformation can be interpreted as a constrained Lorentz rotation that preserves the hyperbolic manifold structure while enabling low-rank adaptation. The space-like components contain the semantic information analogous to the original token embeddings, while the time component is recomputed to maintain the Lorentz constraint (x₀² - ||xs||² = K). This design is inspired by established hyperbolic neural network architectures and allows us to apply linear transformations BA directly to the meaningful semantic dimensions while preserving hyperbolic geometry.

Q3: Details of GNN models

A3: We use two-layer GCN to construct a GNN models.

Q4: Meaning of “prompt level”

A4: Prompt-level means that for each input prompt (e.g., a mathematical problem from GSM8K), we extract the token embeddings for all tokens within that specific prompt, treating each token embedding as a point in the metric space.

The "prompt level" refers to this per-prompt analysis rather than computing hyperbolicity across the entire vocabulary or dataset. We repeat this process for all prompts in each dataset and then average the resulting δ values to obtain the overall hyperbolicity measure reported in Table 2. We will revise this section in the manuscript to detail this procedure more clearly.

Q5: Linear transformation in lines 266-268

A5: Thanks for your comment. We direct your attention to the comprehensive mathematical formulation provided throughout Section 5 and Appendix D for the complete technical details.

Our Direct Lorentz Low-rank Transformation (LLR) operates directly on the hyperbolic manifold using the formula LLR(BA, x^H) = (√(||BAx^H_s||² + K), BAx^H_s), where x^H_s represents the space-like component of the hyperbolic representation and BA are the low-rank adaptation matrices (Equation 9). This transformation applies the linear adaptation BA to the semantic space-like dimensions while recomputing the time component to preserve the Lorentz manifold constraint x₀² - ||xs||² = K. As detailed in Appendix D, this can be interpreted as a constrained Lorentz rotation that maintains hyperbolic geometry while enabling parameter-efficient adaptation.

The linear transformation is inspired by established hyperbolic neural networks, which allows us to capture the hierarchical inductive bias of hyperbolic geometry while maintaining the efficiency and compatibility.

Q6: GPU usage comparison is either missing or mislabeled.

A6: We apologize for this mislabeling error.

Q7: The reason of using Poincaré disk model for discussing the connection between power-law distribution and hyperbolic geometry.

A7: Thanks for your comments. While our main methodology employs the Lorentz model throughout due to its numerical stability and advantages for performing transformations within neural networks, the theoretical connection in Section 4.3 utilizes the Poincaré disk model because it provides a more intuitive geometric framework for explaining the relationship between power-law distributions and hyperbolic curvature.

The Poincaré disk model's polar coordinate system (r, θ) naturally aligns with the hierarchical organization we observe in token embeddings, where radial distance from the origin correlates with token specificity as demonstrated in our empirical analysis (Figure 1, Table 1).

Importantly, the Lorentz and Poincaré disk models are isometric; they are different representations of the same underlying hyperbolic geometry and share equivalent geometric properties. Therefore, the choice of the Poincaré disk model for the conceptual derivation in Section 4.3 does not affect the mathematical validity of the connection we draw between power-law distributions and hyperbolic space.

To prevent any potential confusion for readers, we will revise the manuscript accordingly.

Q8: Connection between the power-law distribution and hyperbolic geometry

A: Thanks for your detailed and constructive feedback. 2D Poincaré disk model offers an intuitive visualization of exponential volume growth—a key property that generalizes to n-dimensional hyperbolic spaces.

The generalization to higher dimensions follows standard hyperbolic geometry principles where the exponential volume growth property (C(r) ∼ e^r and A(r) ∼ e^r as r → ∞) holds in n-dimensional hyperbolic space, not just 2D.

Our core argument builds upon the established work of Krioukov et al. [19], which demonstrates that embedding random geometric graphs in an n-dimensional hyperbolic space naturally gives rise to scale-free networks with a power-law degree distribution. The polar coordinate system is a conceptual framework used in that work to model this phenomenon. Our assignment of "radial coordinate correlating with token frequency" is motivated by our empirical findings in Section 4.1. The angular coordinate encoding semantic similarity follows from the hierarchical tree-like structure we demonstrate through δ-hyperbolicity analysis (Table 2).

Q9: In Equations 6, 7, and the following lines, the symbol is used without explanation of what kind of approximation or relation it indicates.

A9: The symbol "∼" indicates asymptotic equivalence as r approaches infinity, meaning the ratio of the left-hand side to the right-hand side approaches 1 in the limit. We will make it clear in the improved version.

Q10: In line 263, the paper refers to Appendix E.1 for the stereographic projection formulas, but the actual definitions of and are in Appendix E.2.

A10: Sorry for the typo. Line 263 should refer to Appendix E.2.

Q11 Missing referred information of Appendix F in main text

A: The purpose of Appendix F is to provide the formal mathematical derivation that substantiates Proposition 1.

Specifically, it demonstrates how the non-linear projections inherent in HypLoRA introduce higher-order, norm-dependent terms into the adaptation, which is the core mechanism behind its improved performance. Thanks for the reviewer’s comments. We will revise the main text to more clearly introduce the purpose of the appendix when it is referenced.

Q12: Proof of Proposition 1

A: The analysis in Appendix F serves as the mathematical derivation that substantiates Proposition 1. The core of the argument is that the sequence of non-linear projections to and from hyperbolic space results in an effective transformation that is dependent on the input token's L2 norm. As shown in Appendix F.1 (Eq. 30), the final update term from HypLoRA can be approximated as $\Delta Q^{\text{Hyp}} \approx BAx + \frac{||x||^2}{6K} \cdot BAx$. The second term, $\frac{||x||^2}{6K} \cdot BAx$, is the higher-order modification introduced by our method. This term is directly proportional to the square of the L2 norm ($\|x\|^2$) of the input token embedding $x$. This explicit dependency is the key mechanism that allows HypLoRA to perform a more nuanced adaptation than standard LoRA, which only applies the linear transformation $BAx$.

The link between this norm-dependent modification and "capturing hierarchical relationships" stems directly from our empirical findings in Section 4.1. In our analysis (Figure 1, Table 1), we demonstrate that token embeddings are organized hierarchically, where token specificity correlates with L2 norm: general, high-frequency tokens (e.g., "to", "and") have smaller norms, while specific, low-frequency tokens (e.g., "banana", "purple") have larger norms. Because the higher-order term in HypLoRA is proportional to $\|x\|^2$, our method applies a larger, more significant adaptation to these specific, large-norm tokens. This allows the model to differentially weight the importance of tokens based on their position in the semantic hierarchy, enabling a more refined and context-aware fine-tuning.

Thank you for your suggestion. We will include a brief introduction on Poincaré and Lorentz models and their isometric relationship as well as emphasizing the cancellation effect using clear mathematical notation.

For the difference with exisiting hyperbolic dimension reduction, we appreciate the opportunity to better position our work within the broader hyperbolic learning landscape.

While both HypLoRA and HoroPCA[1] operate on hyperbolic representations, their fundamental objectives and technical methodologies differ significantly. Our primary goal is supervised, parameter-efficient fine-tuning of pre-existing Euclidean LLMs for specific downstream tasks. This motivation differs from that of HoroPCA, which focuses on unsupervised dimensionality reduction to identify an optimal low-dimensional hyperbolic subspace that preserves the variance of the original data.

HypLoRA learns an AB matrix decomposition, where A functions as a dimensionality reduction matrix and B serves as a dimensionality expansion matrix. In contrast, HoroPCA (as well as other related methods, like PGA[2], EPGA[3]) exclusively involves the dimensionality reduction process, and the reduced representations cannot be directly aligned with the original dimensions of the LLMs, making it unsuitable for our fine-tuning objective.

Furthermore, this technical difference also exhibit distinct computational efficiency characteristics. HoroPCA requires an iterative optimization process to identify principal directions (ideal points) while maximizing a variance objective, which becomes computationally intensive, when applied across multiple Transformer components (query, key, value, and feed-forward layers) in LLMs. Conversely, HypLoRA avoids the need to search for optimal subspaces and instead directly learns a low-rank update (BA) for the LLM's existing weight matrices. This approach enables HypLoRA to maintain computational complexity comparable to standard LoRA.

Thank you for your insightful suggestion again. We will incorporate these into our revised version. Should you have any follow-up questions or additional suggestions, we would be happy to discuss them further.

[1] Chami, Ines, Albert Gu, Dat P. Nguyen, and Christopher Ré. "Horopca: Hyperbolic dimensionality reduction via horospherical projections." In International Conference on Machine Learning, pp. 1419-1429. PMLR, 2021.

[2] Fletcher, P. Thomas, Conglin Lu, Stephen M. Pizer, and Sarang Joshi. "Principal geodesic analysis for the study of nonlinear statistics of shape." IEEE transactions on medical imaging 23, no. 8 (2004): 995-1005.

[3] Sommer, Stefan, François Lauze, Søren Hauberg, and Mads Nielsen. "Manifold valued statistics, exact principal geodesic analysis and the effect of linear approximations." In European conference on computer vision, pp. 43-56. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010.

Thank you for your thorough review and constructive feedback. In what follows, we use “Q” to denote each question, weakness, or issue raised and “A” for our response.

Q1: Representative power of LoRA and HypLoRA and the advantage of HypLoRA

A1: Thank you for your thorough observations and constructive feedback. The main differences lies in the higher-order interaction term( $||\mathbf{x}||^2BA\mathbf{x}$), which represents a qualitatively different type of transformation that cannot be replicated and covered by standard LoRA.

The higher-order term ($||\mathbf{x}||^2BA\mathbf{x}$) creates a multiplicative coupling between the input embedding norm and the low-rank transformation matrices, fundamentally changing how the adaptation interacts with different tokens. This is not simply a scaled version of BAx—it's an entirely different functional form that introduces input-dependent adaptation strength. For example, for tokens with larger norms (more specific concepts like "dog", "banana" with norm ~0.569 from Table 1), the term ($||\mathbf{x}||^2BA\mathbf{x}$) provides significantly stronger modifications, while for tokens with smaller norms (general function words like "to", "in" with norm ~0.353), the adaptation is naturally attenuated. This creates an adaptive mechanism where the fine-tuning automatically emphasizes tokens that are more semantically specific and likely crucial for downstream task performance.

Q2: why not applying riemmanian optimization && does all the transformations on the hyperbolic space before the final layers

A2: Thanks for your critical suggestions. While these are theoretically appealing alternatives, they face significant practical limitations that motivated our design choices. Fine-tuning embeddings with Riemannian optimization on hyperbolic manifolds would require substantial computational overhead and careful handling of the manifold constraints, while training fully hyperbolic networks from scratch is prohibitively expensive for large-scale LLMs and would sacrifice the valuable pretrained representations that already capture hierarchical structure (as shown in our Section 4 analysis).

Our HypLoRA approach balances theoretical rigor with practical feasibility: we preserve the pretrained Euclidean embeddings and weights that already exhibit hyperbolic characteristics, while selectively applying hyperbolic transformations only to the low-rank adaptation matrices A and B. This hybrid design leverages the best of both worlds—maintaining compatibility with existing pretrained models while capturing hyperbolic inductive biases through the learnable curvature parameter $K$ and norm-dependent scaling terms.

Q3：The frequency of the tokens seem to be largely depend on the corpus or the context considered. I am wondering how the observation that the frequency of tokens tends to be inversely proportional to the norm can help for fine-tuning on new datasets.

A3: While it is true that absolute token frequencies are corpus-dependent, our core argument relies on the stable, underlying hierarchical structure of the embedding space, which is a fundamental property learned during the LLM's large-scale pre-training, not just on the specific fine-tuning dataset. Our cross-model and cross-dataset analysis in Appendices A and B demonstrates that this principle is a remarkably consistent feature across different model families and task domains.

This pre-existing geometric organization is precisely what makes our approach effective for fine-tuning on new datasets. Fine-tuning does not occur in a vacuum; it leverages the rich geometric landscape established during pre-training. When HypLoRA is applied to a new task, its norm-dependent update mechanism automatically exploits this latent structure. The higher-order term, $||\mathbf{x}||^2BA\mathbf{x}$, naturally applies a stronger adaptation to the tokens that the pre-trained model has already encoded as more specific (i.e., those with larger norms). These are often the most informative tokens for mastering the nuances of the new task.

Q4: I am wondering how you choose or estimate the curvature constant and how it affects the final result.

A4：We treat curvature $C=-1/K$ as a learnable hyperparameter to allow the model to adapt to the specific geometric needs of each task. During our experiments, we initialize $K$ by searching over the candidate set {0.5, 1.0, 2.0} and subsequently make it trainable during the fine-tuning process. This approach allows the model to automatically discover a suitable curvature, effectively balancing the hyperbolic inductive bias with the geometric characteristics of a given dataset.

Thank you for follow-up and for this insightful question. While direct injection $(c_x + 1)BAx$ with $c_x = \|x\|^p$ appears similar, our hyperbolic approach is theoretically grounded rather than ad-hoc. Compare with direct injection, the proposed method has a clear geometric meaning and the curvature parameter provides principled control through learned geometric properties.

In terms of representational power: LoRA applies uniform scaling $BAx$ while HypLoRA introduces norm-dependent adaptation $\|x\|^2 BAx$ + $BAx$ that standard LoRA cannot represent and cover.

Beyond the methodological innovation, our paper makes two key contributions: (1) comprehensive investigation of hyperbolic characteristics in LLM token embeddings, revealing power-law distributions and tree-like structures across multiple models and datasets, and (2) HypLoRA methodology that performs low-rank adaptation directly on hyperbolic manifolds while preserving geometric properties.

Thanks again for your critical feedback, we will add this discussion in our revised version and weclome further discussion.

We sincerely appreciate Reviewer 8MJ1's valuable suggestion. Given the time constraints, we conducted series of experiments on Gemma 7b by removing the corresponding mappings and observed a significant decrease in model performance, confirming that these mappings are indeed essential components. We will incorporate this analysis into our revised manuscript. Weclome further discussion.

Thank you for your thorough review and constructive feedback. In the follows, we use “Q” to denote any question, weakness, or issue raised, and “A” to denote our response.

Q1: The analysis of hierarchical structure and hyperbolicity appears to focus predominantly on the input embeddings. It would strengthen the work to examine whether these non-Euclidean properties persist or transform across deeper layers of the model. (as well as with the token context) . It could also be interesting to see how such behavior may differ when using HypLoRA as opposed to other parameter efficient finetuning methods.

A1: We thank the reviewer for this excellent suggestion. To extend our hyperbolicity analysis beyond the input embeddings, we further examine the relative hyperbolicity of the final hidden layer representations of Gemma 7B and Gemma 3-4B across five datasets, two focused on math reasoning (AQuA, GSM8K) and three on commonsense reasoning (ARC-Challenge, Winogrande, OpenbookQA). We compared the base models against versions fine-tuned with LoRA, DoRA, and our proposed HypLoRA across a diverse set of five reasoning datasets.

First, as a baseline, the hyperbolicity of the initial, non-contextualized token embeddings is presented in Table 1.

Hyperbolicity of Inital Token Embeddings (Table1)

Hyperbolicity of Final Hidden Layer Embeddings in Gemma 7B: (Table2-1)

Hyperbolicity of Final Hidden Layer Embeddings in Gemma3 4B: (Table2-2)

The results for the final hidden layer embeddings are shown in Table 2. Our analysis reveals several key findings:

• First, the final hidden states of the base models exhibit a lower degree of hyperbolicity (i.e., larger hyperbolicity value than in the initial embeddings).
• Second, HypLoRA consistently learns final representations with a higher degree of hyperbolicity (lower δ values) across almost all datasets and models. This provides strong empirical evidence that our method actively preserves and enhances the hierarchical structure of the representations throughout the model, aligning the final contextualized embeddings with the geometric biases that are beneficial for reasoning.

Q2: Is there any relationship between the degree of hyperbolicity in an LLM’s embeddings and the amount of performance improvements observed with HypLoRA?

A2: We observe that as dataset complexity increases, the final hidden layer exhibits correspondingly higher relative hyperbolicity value.

For instance, AQuA, a particularly challenging dataset, shows a relative hyperbolicity of 0.31 in Gemma 7B and 0.17 in Gemma 3-4B. When these models were fine-tuned with HypLoRA, the degree of hyperbolicity increased (i.e., the relative hyperbolicity value decreased). In Gemma 7B, the drop from 0.31 to 0.22 led to a significant improvement in accuracy, reaching 46.5%, compared to LoRA’s 34.3%, which had a relative hyperbolicity of 0.24.

Similar trends are observed in Gemma 3-4B as well. Moreover, in commonsense reasoning tasks, the effect is even more pronounced: while LoRA and DoRA tend to reduce the degree of hyperbolicity relative to the base LLM, HypLoRA increases it. As a result, the performance improvement with HypLoRA is both evident and consistent.

Q3: Could the insights from this work be applied beyond the low-rank adapter setting, for example in other fine-tuning strategies or pretraining?

A3: The insights from HypLoRA have broad applicability beyond low-rank adaptation. The core principle—using norm-dependent scaling (||x||²) to automatically emphasize semantically specific tokens—can enhance any parameter-efficient fine-tuning method including adapters, prefix-tuning, and prompt-tuning, replacing their uniform scaling with hierarchically-aware adaptations that align with natural language structure.

For pretraining, the potential is also transformative. Since our analysis reveals pretrained embeddings naturally develop hyperbolic characteristics, incorporating hyperbolic inductive biases from the start—through curvature-aware attention, hyperbolic positional encodings, or learnable curvature parameters—could accelerate the emergence of beneficial hierarchical structure.

Rather than discovering these geometric properties implicitly through millions of training steps, models could be architected to respect linguistic hierarchy from initialization, potentially reducing training costs while improving the quality of learned representations and enabling more interpretable, structured language understanding.

Thank you for the reviewer's follow-up. We are pleased to hear that most of your concerns and questions have been resolved. We are currently investigating case of hyperbolicity on Winogrande and exploring potential reasons underlying this behavior.

We observed that all fine-tuning methods, including standard LoRA, resulted in a higher degree of hyperbolicity (lower δ values) compared to the base model on most datasets, with Winogrande on Gemma 7B being a notable exception. This phenomenon suggests there may be a discrepancy between the pre-trained objective and new fine-tuning tasks, resulting in differing hyperbolicity trends across different datasets and task types. Comparing Winogrande results between Gemma 7B and Gemma 4B suggests this may also be related to the base model.

However, we have observed a consistent pattern where HypLoRA typically exhibits higher degree of hyperbolicity than LoRA across all datasets and base models, demonstrating that hyperbolic operations provide more effective geometric regularization regardless of task-specific requirements. Since our main focus here is on fine-tuning and exploiting the underlying hyperbolicity, this consistent pattern better reflects the rationality and effectiveness of our proposed method.