Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining

We express our deepest gratitude to the reviewer for the time and effort in reviewing our work. It is very encouraging for us to see your evaluation of our work as 1). having clear motivation, 2). simple and practical, 3). theoretically rigorous, and 4). having clear presentation. Below, we want to address your concern about our work.

Weakness 1: The idea of analyzing low-rank subspaces to improve performance is an established paradigm, particularly in work on improving LoRA. This positions the contribution as an effective but arguably incremental advancement.

Response to Weakness 1: We appreciate your observation that analyzing and modifying low-rank subspaces is an established paradigm, especially in the context of LoRA. We agree that this is a widely explored area. However, we would like to clarify that the main novelty of our work is to address a newly identified challenge within this paradigm—namely, the frozen dominant subspace phenomenon.

This phenomenon was recently formalized in [1, 2], and to the best of our knowledge, our paper is the first to propose a theoretically grounded and empirically validated method to mitigate it. Specifically, while prior methods like GaLore and $Q$-GaLore project gradients onto the dominant subspace (defined by leading singular vectors), we show empirically (e.g., Figure 3, Table 1) that such a strategy can lead to stagnation in subspace evolution during pretraining, ultimately limiting the expressiveness of weight updates.

To tackle this, we introduce an importance sampling mechanism inspired by classical sketching techniques in randomized linear algebra. Given a big data matrix $A \in \mathbb{R}^{n \times d}$, sketching aims to reduce dimensionality by sampling a small matrix $SA$, where $S \in \mathbb{R}^{s \times n}$ and $s \ll n$. A principled way to construct $S$ is by leveraging leverage scores, which quantify each row’s contribution to the row space of $A$. Rather than selecting the top-$k$ leverage rows (which could lead to overfitting to a narrow subspace), sketching theory recommends sampling proportionally to leverage scores, which better preserves the full subspace structure [3].

We draw a direct analogy between this and our method: we treat the singular values (quantifying each singular vector’s contribution) of the gradient as analogous to leverage scores, and sample left singular vectors without replacement, proportional to their singular values. This avoids always selecting the same top-$k$ directions, like in GaLore and $Q$-GaLore, and instead promotes diversity across training steps. Our importance sampling method is not a heuristic adjustment made purely for empirical performance. Rather, it is a theoretically motivated approach rooted in randomized linear algebra and directly addresses the frozen subspace problem. While LoRA-related work focuses on augmenting low-rank structure, our work focuses on dynamically diversifying it during optimization, which is a crucial distinction. As far as we know, no prior work explores importance-sampled low-rank subspace selection in this way, particularly within the context of LLM pretraining.

Moreover, SVD is a well-established and widely used technique in machine learning. Many methods—such as principal component analysis (PCA) [4] and attention approximation techniques [5]—rely on projecting data onto the top-$k$ left singular vectors of a data matrix to reduce dimensionality. Our randomized importance sampling strategy offers a novel perspective: instead of deterministically selecting the top directions, probabilistic selection may enhance subspace diversity and performance. We believe this insight could inspire future work in improving other applications that currently rely on fixed top-$k$ subspace projections and in theoretically and/or empirically studying the similarities and differences between our importance sampling and leverage score sampling.

Weakness 2:

2a. The experiments do not extend to larger models.

2b. The evaluation is confined to perplexity. It critically lacks evaluation on a suite of downstream fine-tuning tasks, which is the standard for demonstrating the quality and generalizability of a pretrained model.

2c. There is a crucial absence of any ablation or sensitivity analysis on the rank $r$. (a key hyperparameter).

Response to Weakness 2: Thank you very much for your insightful comments. We currently report experimental results on models with 60M, 130M, 350M, and 1.1B parameters (Tables 1 and 2), demonstrating that SARA scales effectively across a range of model sizes. Due to limitations in time and computational resources, it is very challenging to present results on larger models at this stage. Even conducting experiments on the 1.1B model takes more than 10 days by using all of our GPUs. Nevertheless, we would like to emphasize that our theoretical convergence guarantee (Theorem 3.4) is independent of model size and thus supports general scalability.

Although we lack the time and resources to run very large-scale experiments, we have fully utilized the rebuttal period to present the following experimental results that are computationally efficient and feasible within this limited timeframe. The following results can address your concerns in 2b and 2c.

$\bullet$ Hyperparameter sensitivity: Demonstrated stable performance across critical hyperparameters (see our Response to Weakness and Question 2 to Reviewer ZP7Q).

$\bullet$ Additional metrics beyond validation perplexity: Confirmed enhanced downstream task performance (see our Response to Weakness and Question 1 to Reviewer ZP7Q).

$\bullet$ Comparison with uniform sampling: Highlighted advantages of our sampling strategy (see our Response to Weakness and Question 2 to Reviewer yFMY).

$\bullet$ Multiple random seeds: Verified the robustness and reliability of our method across different runs (see our Response to Weakness and Question 2 to Reviewer yFMY).

Weakness 3: The practical significance of the improvements is questionable in some cases. On the 130M model, SARA's perplexity gain over the Fira-Adam baseline is minimal (22.22 vs. 22.37).

Response to Weakness 3: We thank you very much for pointing this out. We want to highlight that the key measure is not the raw perplexity improvement, but the reduction in the performance gap between the low-rank optimizer and full-rank Adam. In the 130M case, SARA reduces this gap by 42.25%, which is substantial given the tight margins typically observed in large-scale LLM pretraining. Also, SARA consistently improves performance across a wide range of model sizes (60M to 1.1B) and optimizer variants (Fira, GaLore, Adafactor, Adam-mini, 8-bit), as shown in Table 1 and Table 2. This consistency indicates that the improvements are not isolated noise but the result of meaningful enhancements to subspace exploration during optimization.

Additionally, we only pretrain LLaMA models with a certain number of tokens, so we can only observe relatively small performance gains in our paper. In fact, the performance gain of our method compared to GaLore increases as the language models are trained with more tokens. Therefore, once the language models are fully trained on the pretraining task, we expect to see much larger performance gains.

Weakness 4: "The paper claims SARA "does not bring extra overhead". This claim requires empirical validation with wall-clock time and memory usage comparisons.

Response to Weakness 4: Thank you for pointing this out. Therefore, we conducted an additional experiment to address your concern. In the experiment, we measure the wall-clock time of SVD computation and sampling. We calculate the average wall-clock time of ten runs; the result for SVD is 0.34 seconds for a 2048 by 2048 matrix, and the result for sampling is 0.0005 seconds. The empirical result shows that the additional computational cost brought by sampling is negligible.

Weakness 5: The paper's own theory suggests SARA's convergence rate is slightly worse than GoLore's by a constant factor, yet SARA consistently outperforms GoLore in experiments.

Response to Weakness 5: We thank you very much for pointing out this important point. SARA picks singular vectors proportional to their singular values: this can not only ensure that we pick useful information (high singular values) but also address frozen dominant subspace phenomenon. GoLore, on the other hand, projects gradients onto a random low-rank subspace using unbiased random projections. This random projection cannot guarantee that they can pick so useful information compared to our SARA. Additionally, our theoretical guarantee considers the worst case. In practice, "the gradient lies predominantly in the smaller, top subspace" [2], and our importance sampling concentrates on informative directions, leading to a better effective convergence behavior than the worst-case suggests. We have included the above discussion to our paper.

Once again, we are deeply grateful for your insightful comments. We have addressed all identified weaknesses and questions, and we respectfully and genuinely hope you will consider raising your score.

[1] Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. "Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients." Preprint'24.

[2] Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. "Gradient descent happens in a tiny subspace." Preprint'18.

[3] Michael W. Mahoney "Randomized algorithms for matrices and data." Foundations and Trends in Machine Learning'11.

[4] Jianqing Fan, Qiang Sun, Wen-Xin Zhou, and Ziwei Zhu. "Principal component analysis for big data." Preprint'18.

[5] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. "Linformer: Self-attention with linear complexity." Preprint'20.

We express our deepest gratitude to the reviewer for the time and effort in reviewing our work. It is very encouraging for us to see your evaluation of our work as 1). having clear motivation, 2). simple and useful, and 3). containing theoretical support and strong experiments. Below, we want to address your concerns about our work.

Question 1 and Weakness 3: Why are the sampling probabilities proportional to singular values, for instance, why not proportional to exp(.) of singular values to resemble a softmax function? Have you tried uniform sampling? An ablation would clarify this design choice.

Response to Question 1 and Weakness 3: The frozen dominant subspace phenomenon was recently formalized in [1, 2], and to the best of our knowledge, our paper is the first to propose a theoretically grounded and empirically validated method to mitigate it. Specifically, while prior methods like GaLore and $Q$-GaLore project gradients onto the dominant subspace (defined by leading singular vectors), we show empirically (e.g., Figure 3, Table 1) that such a strategy can lead to stagnation in subspace evolution during pretraining, ultimately limiting the expressiveness of weight updates.

To tackle this, we introduce an importance sampling mechanism inspired by classical sketching techniques in randomized linear algebra. Given a big data matrix $A \in \mathbb{R}^{n \times d}$, sketching aims to reduce dimensionality by sampling a small matrix $SA$, where $S \in \mathbb{R}^{s \times n}$ and $s \ll n$. A principled way to construct $S$ is by leveraging leverage scores, which quantify each row’s contribution to the row space of $A$. Rather than selecting the top-$k$ leverage rows (which could lead to overfitting to a narrow subspace), sketching theory recommends sampling proportionally to leverage scores, which better preserves the full subspace structure [3].

We draw a direct analogy between this and our method: we treat the singular values (quantifying each singular vector’s contribution) of the gradient as analogous to leverage scores, and sample left singular vectors without replacement, proportional to their singular values. This avoids always selecting the same top-$k$ directions, like in GaLore and $Q$-GaLore, and instead promotes diversity across training steps. Our importance sampling method is not a heuristic adjustment made purely for empirical performance. Rather, it is a theoretically motivated approach rooted in randomized linear algebra and directly addresses the frozen subspace problem. While LoRA-related work focuses on augmenting low-rank structure, our work focuses on dynamically diversifying it during optimization, which is a crucial distinction.

Moreover, SVD is a well-established and widely used technique in machine learning. Many methods—such as principal component analysis (PCA) [4] and attention approximation techniques [5]—rely on projecting data onto the top-$k$ left singular vectors of a data matrix to reduce dimensionality. Our randomized importance sampling strategy offers a novel perspective: instead of deterministically selecting the top directions, probabilistic selection may enhance subspace diversity and performance. We believe this insight could inspire future work in improving other applications that currently rely on fixed top-$k$ subspace projections and in theoretically and/or empirically studying the similarities and differences between our importance sampling and leverage score sampling.

Weakness and Question 2: No comparison with alternative sampling methods. The paper does not benchmark SARA against simpler schemes such as uniform, top-$k$, or random-$k$ sampling, leaving it unclear how much of the gain is due to importance sampling versus just adding noise or diversity.

Response to Weakness and Question 2: We run experiments with uniform sampling, our proposed importance sampling, and top-$r$ selection (the subspace selection used in GaLore) using three different seeds on the LLaMA-130M model with a sequence length of 256. The models are trained for 2 billion tokens. The results are provided in the table below. We can see that importance sampling outperforms both uniform sampling and top-$r$ selection.

We average the subspace metrics across all layers, and the resulting averages are presented in the table below. We observe that the overlap between the anchor subspace and subsequent subspaces is lower for GaLore-SARA-Adam compared to GaLore-Adam. This indicates that SARA encourages the optimization trajectory to explore a more diverse set of subspaces relative to using the dominant subspace. The averaged anchor subspace overlap metrics for GaLore-Adam and GaLore-SARA-Adam across all layers are summarized as follows:

Question 3: How frequently do you compute the SVD and update the sampling probabilities? I could not find details about the hyperparameter $\tau$ in the paper. I assume it matches the baseline methods, but please clarify and report it in Appendix B. Also, a rough estimate of the added runtime or FLOP overhead would be helpful.

Response to Question 3: SVD is computed every $\tau$ iterations, where $\tau$ is the subspace update period of our algorithm. In our experiments, we set $\tau$ to the same value as in GaLore’s original paper, which is 200. We have included it in the updated version of our paper.

The additional overhead introduced by sampling is minimal. To address your concern, we conducted an additional experiment. In the experiment, we measure the wall-clock time of SVD computation and sampling. We calculate the average wall-clock time of ten runs, the result for SVD is 0.34 seconds for a 2048 by 2048 matrix, and the result for sampling is 0.0005 seconds. The empirical result shows that the addition computational cost brought by sampling is negligible.

Weakness 4 and Question 4: Does the added randomness affect training stability or convergence? Showing results across multiple seeds would help. Also, does random sampling affect the optimal value for the hyperparameter $\tau$?

Response to Weakness 4 and Question 4: Sampling is stable across runs. In our Response to Weakness and Question 2, we show the results of GaLore-SARA-Adam across three seeds, demonstrating its stability. We also test whether random sampling affects the optimal subspace update frequency, i.e., $\tau$. Specifically, we train the LLaMA-130M model with different $\tau$ values on the C4 dataset with a sequence length of 256. The results are shown in the table below. We find that the optimal $\tau$ remains 200. This indicates that random sampling does not affect the optimal value of $\tau$, which is an important hyperparameter.

Also, in our experiment, we did not find that sampling has any negative effect on distributed training.

Weakness 1 and Question 5: Organization and typos. Section 4.3 visualizes subspace metrics for just one parameter group, with the rest of the figures relegated to the appendix. The paper should aggregate and discuss results across all layers and groups in the main text (for example, plotting the average subspace overlap per layer and the average overlap over training steps for each parameter group) to give a clearer, more comprehensive view of the method’s behavior. The $x$-axis in Figure 4 (and similar plots in the appendix) is labeled "update steps," but it appears to show singular value indices. Please clarify or correct the label.

Response to Weakness 1 and Question 5: We sincerely appreciate your valuable comments on the organization and typos in our work. We completely agree with your valuable comments, and we have fixed these issues in the revised version of our paper.

Once again, we are deeply grateful for your insightful comments. We have addressed all identified weaknesses and questions, and we respectfully and genuinely hope you will consider raising your score.

[1] Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. "Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients." Preprint'24.

[2] Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. "Gradient descent happens in a tiny subspace." Preprint'18.

[3] Michael W. Mahoney "Randomized algorithms for matrices and data." Foundations and Trends in Machine Learning'11.

[4] Jianqing Fan, Qiang Sun, Wen-Xin Zhou, and Ziwei Zhu. "Principal component analysis for big data." Preprint'18.

[5] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. "Linformer: Self-attention with linear complexity." Preprint'20.

We express our deepest gratitude to the reviewer for the time and effort in reviewing our work. It is very encouraging for us to see your evaluation of our work as 1). clear and well-motivated, 2). simple and has principled modification, and 3). well-integrated with existing low-rank optimizers. Below, we want to address your concerns about our work.

Weakness and Question 1: Besides validation perplexity, more metrics are needed to show the model's performance.

Response to Weakness and Question 1: Thank you for your insightful comments. We have conducted additional experiments using multiple downstream tasks from the GLUE benchmark to further evaluate the effectiveness of our method. The table below compares fine-tuning performance on GLUE tasks using pre-trained RoBERTa-Base with different optimizers and LoRA configurations. Results for LoRA and GaLore-Adam are from Zhao et al. [1]. GaLore-SARA-Adam uses identical hyperparameters to GaLore-Adam for the fairness of comparison.

Weakness and Question 2: More analysis of hyperparameter sensitivity is needed.

Response to Weakness and Question 2: We thank you for pointing this out. We have performed additional sensitivity analysis on the subspace refresh frequency $\tau$ and the sampled rank $r$ by training a LLaMA-130M model on the C4 dataset with sequence length 256. The sensitivity of performance to subspace refresh frequency $\tau$ is presented in the table below, with an optimal setting around $\tau=200$. Additionally, when $\tau$ varies from 100 to 500, our algorithm's performance remains stable:

The impact of the sampled rank $r$ on perplexity is demonstrated in the table below, confirming our intuition that a higher rank consistently yields lower evaluation perplexity:

Weakness and Question 3: It’s better to scale up the model size to prove the generalization ability of the method.

Response to Weakness and Question 3: Thank you very much for your insightful comments. We currently report experimental results on models with 60M, 130M, 350M, and 1.1B parameters (Tables 1 and 2), demonstrating that SARA scales effectively across a range of model sizes. Due to limitations in time and computational resources, it is very challenging to present results on larger models at this stage. Even conducting experiments on the 1.1B model takes more than 10 days by using all of our GPUs. Nevertheless, we would like to emphasize that our theoretical convergence guarantee (Theorem 3.4) is independent of model size and thus supports general scalability.

Although we lack the time and resources to run very large-scale experiments, we have fully utilized the rebuttal period to present the following experimental results that are computationally efficient and feasible within this limited timeframe.

$\bullet$ Hyperparameter sensitivity: Demonstrated stable performance across critical hyperparameters (see our Response to Weakness and Question 2 to Reviewer ZP7Q).

$\bullet$ Additional metrics beyond validation perplexity: Confirmed enhanced downstream task performance (see our Response to Weakness and Question 1 to Reviewer ZP7Q).

$\bullet$ Comparison with uniform sampling: Highlighted advantages of our sampling strategy (see our Response to Weakness and Question 2 to Reviewer yFMY).

$\bullet$ Multiple random seeds: Verified the robustness and reliability of our method across different runs (see our Response to Weakness and Question 2 to Reviewer yFMY).

Once again, we are deeply grateful for your insightful comments. We have addressed all identified weaknesses and questions, and we respectfully and genuinely hope you will consider raising your score.

[1] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. "Galore: Memory-efficient llm training by gradient low-rank projection." ICML'24.

We express our deepest gratitude to the reviewer for the time and effort in reviewing our work. It is very encouraging for us to see your evaluation of our work as 1). novel and compelling, 2). clearly written, and 3). containing sufficient theoretical and experimental results. Below, we want to address your concerns about our paper.

Weakness and Question 1: Adding experiments on larger language models—for instance, Llama-3.1-8B—would strengthen the empirical evidence for the method’s scalability.

Response to Weakness and Question 1: Thank you very much for your insightful comments. We currently report experimental results on models with 60M, 130M, 350M, and 1.1B parameters (Tables 1 and 2), demonstrating that SARA scales effectively across a range of model sizes. Due to limitations in time and computational resources, it is very challenging to present results on larger models at this stage. Even conducting experiments on the 1.1B model takes more than 10 days by using all of our GPUs. Nevertheless, we would like to emphasize that our theoretical convergence guarantee (Theorem 3.4) is independent of model size and thus supports general scalability.

Although we lack the time and resources to run very large-scale experiments, we have fully utilized the rebuttal period to present the following experimental results that are computationally efficient and feasible within this limited timeframe.

$\bullet$ Hyperparameter sensitivity: Demonstrated stable performance across critical hyperparameters (see our Response to Weakness and Question 2 to Reviewer ZP7Q).

$\bullet$ Additional metrics beyond validation perplexity: Confirmed enhanced downstream task performance (see our Response to Weakness and Question 1 to Reviewer ZP7Q).

$\bullet$ Comparison with uniform sampling: Highlighted advantages of our sampling strategy (see our Response to Weakness and Question 2 to Reviewer yFMY).

$\bullet$ Multiple random seeds: Verified the robustness and reliability of our method across different runs (see our Response to Weakness and Question 2 to Reviewer yFMY).

Weakness 2: While the theorem is proven for SGD momentum, the experiments use Adam for optimization. I understand that it is difficult to prove convergence guarantees for Adam given its non-projection-free property, but it would be beneficial to mention this in the limitations (Appendix C).

Response to Weakness 2: We thank you for your insightful comments. We use Adam since it is a widespread adoption in large-scale LLM pretraining for its empirical performance. We completely agree with your insightful comments that proving convergence guarantees for Adam, especially under low-rank projection with importance sampling, remains challenging due to its complex dynamics and lack of a projection-free formulation. We have included this discussion in the limitations part of our paper. We thank you again for pointing this out!

Weakness 3: Some missing citations [1–4]. Algorithm 1, line 7: The formatting could be refined for readability. Tables 1–4: Sorting the results in descending order would make the comparisons clearer. Line 469: The statement should involve inequalities rather than equalities.

Response to Weakness 3: Thank you very much for pointing out these wonderful papers [1–4]. We have carefully read these recent works, and we believe they are highly related to our work. [1] analyzes the sparsity-based parameter-efficient fine-tuning (SPEFT) for LLMs, which is an alternative method of LoRA. [2] designs a novel method for memory-efficient fine-tuning for LLMs. It has been shown that it can outperform LoRA and full-parameter training in many cases. Similarly, [3] proposes another novel fine-tuning method called Half Fine-Tuning (HFT), which can mitigate "catastrophic forgetting" in LLMs during sequential training and instruction tuning. Finally, [4] also proposes a novel memory-efficient fine-tuning method by strategically selecting layers to update based on outlier statistics. Our paper also considers the memory and convergence behavior, but the difference is that we focus on the LLM pretraining instead of fine-tuning. We have included the above discussion in the revised version of our paper.

Regarding Algorithm 1 line 7, we have updated it as the following two lines:

$

\mathcal{S} &\gets \\{V_l^{(t - 1)}, M_l^{(t - 1)}, x_l^{(t)}, P_l^{(t)}, G_l^{(t)}, \beta_1, \beta_2, \xi, \eta, \alpha\\}. ~~~~~~~~~\text{Input parameters}\\\\
x_l^{(t)} &\gets \text{GaLore-Adam}(\mathcal{S}) \text{ or } \text{Fira-Adam}(\mathcal{S}).

$

Thank you for your insightful comments.

Also, we have fixed our Tables 1–4 to make "the results in descending order" to make them clearer.

Finally, regarding Line 469, we first want to sincerely thank you for carefully checking the details of our work, even though this is in the appendix. In Line 469, we have:

$

\mathbb{E} \left[ \left\| \tilde{M}_l^{(0)} - \nabla_l f(x^{(0)}) \right\|_F^2 \right] &= \mathbb{E} \left[ \left\| \beta_1 P_l^{(0)} (P_l^{(0)})^\top \left( G_l^{(0)} - \nabla_l f(x^{(0)}) \right) \right\|_F^2 \right] \\ &\quad + \mathbb{E} \left[ \left\| \left( \beta_1 P_l^{(0)} (P_l^{(0)})^\top - I \right) \nabla_l f(x^{(0)}) \right\|_F^2 \right].

$

This is because in general $\mathbb{E}[\\|A + B\\|_F^2] = \mathbb{E}[\\|A\\|_F^2] + \mathbb{E}[\\|B\\|_F^2] + 2\mathbb{E}[\langle A, B \rangle]$, but here, we have $\mathbb{E}[\langle A, B \rangle] = 0$, where $A = \beta_1 P_l^{(0)} (P_l^{(0)})^\top \left( G_l^{(0)} - \nabla_l f(x^{(0)}) \right)$ and $B = \left( \beta_1 P_l^{(0)} (P_l^{(0)})^\top - I \right) \nabla_l f(x^{(0)})$. The reason is that $A$ has ``zero-mean'' since we have $G_l^{(0)}$ is unbiased, as our assumption.

Question 2: Do the improvements in subspace diversity lead to any performance gains for downstream tasks, such as math, commonsense reasoning, or instruction-following?

Response to Question 2: Thank you for your insightful comments. We have conducted additional experiments using multiple downstream tasks from the GLUE benchmark to further evaluate the effectiveness of our method. The table below compares fine-tuning performance on GLUE tasks using pre-trained RoBERTa-Base with different optimizers and LoRA configurations. Results for LoRA and GaLore-Adam are from Zhao et al. [7]. GaLore-SARA-Adam uses identical hyperparameters to GaLore-Adam for the fairness of comparison.

Question 3: I am wondering why the similarity between adjacent projection matrices is large, given the diversity in sampled batches, and the noise is expected to be a dominant term in the gradient in the small batch training setting?

Response to Question 3: The randomness in mini-batch gradients can indeed introduce some variation in the dominant subspaces across steps. However, this typically does not have a significant impact on the similarity between adjacent projection matrices, as demonstrated empirically in [5]. In the problem setting of [6], the subspace spanned by the top-$k$ singular vectors tends to stabilize early in training and remains nearly fixed, even though individual singular vectors may vary. The gradient dynamically adapts to reside within this top subspace, resulting in a consistent projection structure across steps [6].

We sincerely thank you for raising this important point. We have incorporated this clarification into the final version of our paper, as we believe it strengthens the motivation behind our work.

Once again, we are deeply grateful for your insightful comments. We have addressed all identified weaknesses and questions, and we respectfully and genuinely hope you will consider raising your score.

[1] Xinxin Liu, Aaron Thomas, Cheng Zhang, Jianyi Cheng, Yiren Zhao, and Xitong Gao. "Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models." ACL'25.

[2] Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. "Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning." NeurIPS'24.

[3] Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Weiran Xu, Yu Sun, and Hua Wu. "Hft: Half fine-tuning for large language models." Preprint'24.

[4] Pengxiang Li, Lu Yin, Xiaowei Gao, and Shiwei Liu. "Owlore: Outlier-weighed layerwise sampled low-rank projection for memory-efficient llm fine-tuning." Preprint'24.

[5] Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. "Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients." Preprint'24.

[6] Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. "Gradient descent happens in a tiny subspace." Preprint'18.

[7] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. "Galore: Memory-efficient llm training by gradient low-rank projection." ICML'24.