Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks

Thank you for your insightful comments and suggestions.

W1: There is another work ...

Reply: Thank you for pointing out this recent work. We have included it in the related work section of our revised manuscript.

W2: I am interested in ...

Reply: Low-precision and mixed-precision training are standard techniques to accelerate LLM pre-training while reducing memory usage. In our experiments:

• For LLaMA pre-training, we used bfloat16 precision, which offers a wide dynamic range and is well-suited for stable large-scale training.
• For OPT pre-training, we used a mixed-precision setup, combining bf16 and TF32 to leverage fast matrix operations on modern hardware while preserving numerical stability.

Although low-precision training greatly improves efficiency, it can increase sensitivity to instability. We observed that LoRA and GaLore occasionally experienced training instability, which may be partially due to interactions between low-rank updates and reduced precision.

W3: Comparison to GaLore ...

Reply: Thank you for the suggestion. In the revised manuscript, we have moved the relevant results and analysis from the appendix into the main body.

W4: Other Comments Or Suggestions

Reply: Due to character limit, couldn't address this part in detail. We have carefully revised the manuscript accordingly. Thank you again for your constructive feedback.

W5: It seems to me ...

Reply: The motivation behind the Enhanced Gradient is to decouple the updates of singular vectors and singular values, allowing singular vectors with smaller singular values to still receive meaningful updates. From another perspective, this effectively corresponds to using a larger learning rate for less dominant directions, similar in spirit to adaptive optimization methods like Adam, where updates are rescaled by the inverse of the magnitude. Enhanced Gradient remains faithful to the model’s computation in the sense that when the gradient of W approaches zero, the Enhanced Gradients of U and V also approach zero. This ensures convergence behavior remains aligned with the model’s optimization objective.

W6: What happens with ...

Reply: Due to the page limit, we had originally omitted the update formulas for $V^T$ in Equations 6–8, but now added them back.

To clarify, the indices for updating U and $V^T$ are shared. This is necessary because the gradient update of $U_i$ depends on $V_i$ and vice versa, which promotes faster convergence.

W7: In 4.3 ...

Reply: Thank you for the insightful question. We conducted additional experiments where A and B in ReLoRA* are initialized with random values instead of zeros. The results (https://anonymous.4open.science/r/sparse_spectral_training-6A2C/relora_random.png) show clear improvement over the original ReLoRA*, but performance still falls short of SST.

One possible reason is that SST uses singular vectors of the weight matrix W , which may better align with the update directions. In contrast, randomly initialized vectors in high-dimensional space are typically close to orthogonal and may not capture meaningful structure. We're also testing random-initialized ReLoRA* on LLaMA and will update results as they become available.

W8: In L323--325...

Reply: Thank you for the question. The key difference from standard fine-tuning setups is that LoRA typically replaces only the query and value projection layers during fine-tuning, while freezing the rest of the model. In contrast, for pre-training, all linear layers, including Q, K, V, O projections and FFN, must be trainable. Therefore, to apply low-rank methods like LoRA, ReLoRA*, or SST to pre-training fairly, we replace all torch.nn.Linear layers of original model with their corresponding low-rank implementations to maintain parameter efficiency across the entire model.

We confirm that the "Full" benchmarks are not affected by this replacement; they use standard full-rank training with all linear layers unchanged.

W9: What's going on...

Reply: Thank you for pointing this out. Table 5 was originally intended as part of the Appendix but was referenced in the main text, so it wasn’t placed under any specific appendix section. We have revised the manuscript and moved it into the main body for clarity and consistency.

W10: Why did you feel...

Reply: Thank you for the question. Hyperbolic models often suffer from overfitting and instability in high-dimensional settings due to the exponential growth of space. Prior work has typically restricted them to low-dimensional use cases for this reason. We included hyperbolic models to show that SST, by limiting the parameter search space, mitigates overfitting and improves stability. Compared to both full-rank and other low-rank methods, SST is more robust and consistently delivers better performance in high-dimensional hyperbolic setups.

Thank you for your detailed review and valuable insights.

Weakness 1: Result analysis should provide insights into future work and underlying reasons for the experimental results in the performance tables, not only provide numerical values already shown in the table.

Reply: Thank you for this valuable feedback. We agree that the original version of our paper focused too heavily on numerical results and lacked sufficient analysis. In the revised version, we have incorporated more discussion on the underlying causes of the observed performance differences, and we highlight two key examples below:

First, in our ablation study on sampling mechanisms (Appendix E.2), we examined why multinomial sampling outperforms other strategies such as uniform, sequential, and top-$r$ selection. As shown in Table 17, top-$r$ performs the worst because it repeatedly updates only a fixed subset of singular vectors, restricting the search space to a narrow low-rank subspace. In contrast, stochastic strategies like multinomial sampling encourage exploration across a broader spectral domain, leading to improved generalization and convergence. This observation suggests future work could explore adaptive or learnable sampling strategies to further enhance training dynamics.

Second, in our extended comparison of SST and GaLore (Appendix G), we analyzed why SST consistently outperforms GaLore in low-rank regimes. GaLore relies on a low-rank projection of the gradient derived from a single batch, which can be noisy and unstable due to sampling variability. SST, on the other hand, uses the SVD of the weight matrix $\Vec{W}$, a more stable and aggregated representation over training steps, and updates all singular values at every iteration. This makes SST more robust, especially at small ranks. These findings also suggest a potential future direction in combining SST’s spectral stability with GaLore’s gradient-based projection approach.

We have incorporated these analyses into the main text in the revised version, utilizing the additional page allowed for the final version.

Weakness 2: Detailed analysis is needed for rank choice.

Reply: Thank you for raising this important point. We agree that a more in-depth discussion of rank is valuable. In our study, the rank is not treated as a tunable hyperparameter but rather as a practical constraint determined by available GPU memory. While increasing the rank generally improves performance, it also significantly increases memory consumption. To ensure fair comparisons, we used the same rank across LoRA, ReLoRA*, and SST in all experiments.

To offer further insight, we report BLEU scores on the IWSLT'14 dataset using a Transformer with model dimension 128:

These results show that SST consistently outperforms both LoRA and ReLoRA across all rank settings, particularly in the low-rank regime. This demonstrates SST’s robustness and practical utility for memory-efficient training under constrained compute budgets.

Weakness 3: Experiments on Hyperbolic graph neural networks are limited. Comparison between hyperbolic transformer and graph neural networks needs furtheranalysis.

Reply: Thank you for this suggestion. We acknowledge that our current experiments on hyperbolic graph neural networks and hyperbolic transformers are limited in scope. However, the results presented already highlight the advantages of SST in this setting.

SST consistently outperforms other low-rank methods across all tested configurations of hyperbolic models (with the exception of node classification on Airport with $r=2$), and in several cases, especially for the hyperbolic Transformer, it even surpasses full-rank training. As shown in Table 1, baseline methods like LoRA and ReLoRA suffer from training instability (e.g., NaN losses), whereas SST maintains stable convergence throughout.

Geometrically, hyperbolic space exhibits exponential volume growth with distance, unlike the polynomial growth of Euclidean space. This makes hyperbolic models highly expressive but also prone to overfitting, particularly in high dimensions. By selectively updating a small subset of singular vectors, SST constrains the parameter search space, helping to regularize the model and prevent overfitting.

We appreciate the reviewer’s suggestion and are actively running additional experiments to further evaluate SST on other hyperbolic architectures. We will share updated results as soon as they are available.

Thank you for your insightful comments and suggestions. We are grateful for the opportunity to clarify and improve our work.

Weakness 1: There are not enough baselines and they are not particularly advanced. Consider introducing Dora, HIRA, Vera, FourierFT (at least Dora and Vera).

Reply: Thank you for the suggestion. In the original manuscript, we did not include Dora and Vera as baselines because their parameter search spaces are restricted to the low-rank subspace. As discussed in Section 3.1 "Limitation of LoRA," this constraint limits their expressiveness, while it may suffice for simple tasks like fine-tuning, it becomes a major bottleneck for more complex tasks such as training from scratch, which is the focus of our work. Since ReLoRA and GaLore are explicitly designed for pre-training, we prioritized comparing with them as the most relevant baselines.

To further demonstrate the limitations of Dora and Vera in pre-training, we conducted additional experiments comparing them to SST:

BLEU scores (higher is better) on IWSLT’14 with Transformer models:

These results show that while Dora slightly improves over LoRA, it still falls short of SST. Vera performs poorly across all settings.

Dora decomposes the pre-trained weights into magnitude and direction components, but this does not overcome the fundamental limitation of the low-rank subspace. Vera further constrains the trainable parameters to a single vector, which significantly reduces memory but fails to capture the complexity required for effective pre-training.

We appreciate this valuable suggestion and have added this analysis and experiment to the revised manuscript.

Weakness 2: I would like to discuss ... because the baselines are all PEFT methods).

Reply: Thank you for raising this important point. We agree that training and fine-tuning are fundamentally different tasks and should be clearly distinguished. In this paper, our focus is specifically on parameter-efficient pre-training, rather than parameter-efficient fine-tuning (PEFT) or sparse training.

There are two main reasons we emphasize pre-training. First, pre-training is considerably more challenging than fine-tuning, as it starts from random initialization without any prior knowledge. The model must learn meaningful representations entirely from scratch, making it more sensitive to optimization stability and expressiveness constraints. Methods that work well in fine-tuning often fail to generalize to this more difficult setting.

Second, pre-training requires significantly more computational and memory resources, which has become a major bottleneck. Many academic labs today lack the resources to conduct large-scale pre-training. SST is motivated by the desire to lower the resource barrier, making pre-training more accessible to the broader research community.

We appreciate the reviewer’s suggestion and have revised our manuscript to explicitly clarify that our work focuses on parameter-efficient pre-training. We have also updated the wording from "Previous memory reduction training techniques" to "Previous parameter-efficient training techniques", since LoRA is a PEFT method and ReLoRA is designed for pre-training. We hope these changes enhance the clarity of our contributions.