GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

Thank you for your review and for highlighting both the strengths of our work and areas for improvement. We have addressed each of your comments in detail below.

(W1 & Q1) Why increasing the rank intensifies the entangled influence of outliers

As discussed in Section 2.1 and 2.2 of the paper, we analyze the gradient dynamics of LoRA to explain why it suffers at higher ranks, particularly in the presence of outlier channels.

Given the LoRA weight update $R = BA^\top$ where $B \in \mathbb{R}^{M \times r}$, $A \in \mathbb{R}^{N \times r}$, and input $X \in \mathbb{R}^{N \times T}$, the gradients respect to $A$ and $B$ are: $\frac{\partial{L}}{\partial{A^\top}} = B^\top \frac{\partial{L}}{\partial{Y}}X^\top, \ \frac{\partial{L}}{\partial{B}}=\frac{\partial{L}}{\partial{Y}}X^\top A$ The corresponding gradient of $R$ becomes: $\frac{\partial{L}}{\partial{R}}=\frac{\partial{L}}{\partial{B}}A^\top + B\frac{\partial{L}}{\partial{A^\top}}=\frac{\partial{L}}{\partial{Y}}X^\top AA^\top + BB^\top \frac{\partial{L}}{\partial{Y}}X^\top$ Note that $X$ appears inside both terms. In particular, the first term $\frac{\partial{L}}{\partial{Y}}X^\top AA^\top$, shows that $X$ is multiplied between two. If $X$ contains an outlier channel, a row with abnormally high magnitude, the channel will influence the entire update due to the matrix multiplications.

Moreover, as the rank $r$ increases, the norm of the Gramm matrx $AA^\top$ also increases. This amplifies the effect of the outlier channel, widening the scale gab between outlier and non-outlier contributions and distorting the overall gradient landscape. In other words, the entanglement becomes worse as $r$ grows, further suppressing gradients from informative but less dominant channels.

This phenomenon is empirically validated in Figure 4, where increasing rank leads to disproportionately larger gradients, driven by the outlier channel. These observations highlight the structural limitation of LoRA at high ranks, motivating the design of GraLoRA to localize such influence.

(W3 & Q2) Additional experiments with new baselines on varying tasks and models

Thank you for the suggestion. While the main comparisons in our paper focused on low-rank decomposition methods, we have since conducted additional experiments to evaluate the generality and robustness of GraLoRA across a broader range of model architectures, tasks, and PEFT baselines—including BOFT and FourierFT.

1. Commonsense Reasoning with Extensive PEFT Baselines

We conducted a new evaluation on the commonsense reasoning benchmark using the LLaMA3.2–3B model, comparing GraLoRA with a wide spectrum of 10 PEFT baselines, including full fine-tuning and all models were trained with rank 32. For consistency, we used the results reported in the LoRA-SB paper for LoRA-XS, LoRA-SB, rsLoRA, and PiSSA. For other methods, we conducted a learning rate sweep over {2e-4, 4e-4, 1e-3, 2e-3}, selecting the best-performing configuration per method. All other hyperparameters followed the settings in LoRA-SB.

GraLoRA achieves the highest average accuracy and ranks first on 5 out of 8 tasks. These results, combined with those in Table 3 of the main paper, confirm GraLoRA’s scalability and consistent improvements across model sizes, ranks, and PEFT baselines.

2. Mathematical Reasoning

We further evaluated GraLoRA on a mathematical reasoning task using the MetaMathQA for training and testing on the MATH data. Two models, LLaMA3.2–1B and Qwen2.5–1.5B, were fine-tuned, mostly following the settings from Hu et al. [1], with ranks 64 and 128. Following our paper’s heuristic, we used $k=4$ for both.

GraLoRA consistently outperforms LoRA, especially on the larger model, highlighting its robustness across architectures and tasks requiring symbolic reasoning.

3. General Language Understanding (GLUE)

We evaluated GraLoRA on the GLUE benchmark using RoBERTa-base, an encoder-only architecture with eight subtasks. To ensure fair comparison with recent PEFT methods designed for parameter efficiency, we included two additional baselines: VeRA and FourierFT.

Following the protocol of prior work, we excluded MNLI and QQP—two time-intensive tasks—which also meant we did not apply the MNLI-based tricks for MRPC, RTE, and STS-B (as used in the original LoRA paper). Accordingly, we retrained LoRA on these tasks without this optimization and report updated results.

While VeRA and FourierFT involve fewer trainable parameters, their training time is comparable to or even longer than LoRA with rank 8. Therefore, we set the LoRA and GraLoRA ranks to 8. Since this is a relatively low-rank setting, we also evaluated Hybrid GraLoRA by splitting the rank equally between LoRA and GraLoRA (i.e., 4+4), which is expected to be beneficial under such constraints.

All hyperparameters for GraLoRA followed those used in the VeRA implementation, except for learning rate, which was reduced by a factor of 5 to 10.

GraLoRA demonstrates strong performance in this low-rank regime, outperforming all baselines in average score. Hybrid GraLoRA delivers the most robust results, achieving the best performance on 4 out of 6 tasks. These results indicate that GraLoRA maintains high effectiveness even under constrained parameter budgets and in non-LLM architectures.

4. Diffusion Model Fine-Tuning

Finally, we applied GraLoRA to diffusion models by fine-tuning SDXL for personalization. We followed the official training setup from the HuggingFace diffusers repository, using the lambdalabs/naruto-blip-captions dataset. The dataset was split 90% for training and 10% for evaluation.

GraLoRA consistently outperformed LoRA in both CLIP and DINOv2 similarity scores, further demonstrating its generality and effectiveness beyond LLMs—including vision-text and generative architectures like diffusion models.

(W2) Analysis for Hyperparameter $k$

We acknowledge that introducing $k$—which determines the granularity of matrix partitioning—adds a degree of tuning not present in vanilla LoRA. However, we empirically found that setting $r/k^2 \approx 8$ provides consistently strong results across different models and tasks. Based on this, we use $k=2$ for ranks up to 32, and $k=4$ for higher ranks (64 and 128). All experiments in the main paper and during the rebuttal strictly follow this heuristic.

Thus, while $k$ is a new hyperparameter, we provide a practical and robust guideline that eliminates the need for exhaustive tuning in most settings.

(Limitations)

We appreciate the suggestion to more explicitly frame future directions around current limitations. Two key limitations were initially considered: Effectiveness at low ranks (e.g., $r=8$) and Hyperparameter sensitivity of $k$.

In small ranks, the per-block expressiveness of GraLoRA may diminish. To address this, we proposed the Hybrid GraLoRA architecture, which retains the fine-grained structure of GraLoRA while incorporating standard LoRA to maintain sufficient expressivity. Empirical results confirm that this approach remains competitive even in low-rank regimes (see GLUE experiment).

As noted, we provide a simple and effective heuristic for setting $k$. However, to eliminate manual tuning altogether, we believe this limitation can be addressed more fundamentally. As suggested in the paper’s future work section, adaptive or learned partitioning could allow the model itself to determine the optimal granularity, replacing the need for a fixed $k$. This would significantly enhance the usability and flexibility of GraLoRA in practical deployments.

To conclude, we sincerely thank the reviewer for the thoughtful and detailed feedback. Your comments have helped us clarify the theoretical foundations of GraLoRA and broaden our empirical validation across diverse model families, tasks, and PEFT baselines. In response, we have incorporated additional theoretical analysis on LoRA’s gradient behavior and GraLoRA’s mitigation strategy, and we have conducted extensive new experiments—including comparisons against BOFT, FourierFT, and applications to GLUE and diffusion models. These results, along with revised discussions on tuning complexity and limitations, will be included in the updated manuscript.

[1] Hu et al., "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models", EMNLP 2023

Respectfully, I believe that the equation (4) is mathmatically wrong. As we know

$$\frac{\partial L}{\partial R} = \frac{\partial L}{\partial (R+W)} = \frac{\partial L}{\partial Y} X^T,$$

if equation (4) holds, we have

$$\frac{\partial L}{\partial R} = \frac{\partial L}{\partial R} AA^T + BB^T\frac{\partial L}{\partial R},$$

which does not hold for all A and B (e.g., if $AA^T=BB^T=I$). Due to this flaw in the theory, which should be a cornerstone of the whole paper, I tend to reject this paper.

We sincerely thank you for your encouraging evaluation and the valuable suggestions for improvement. Below, we address each of your points and questions clearly.

(W1) GraLoRA's gradient dynamics on matrices

In the paper, we identify a fundamental misalignment between LoRA updates and the true gradient landscape, particularly in the presence of outlier channels in the input. Prior studies have shown that outlier channels and weights are common in large models [1][2], and the impact of such outliers has been actively addressed in areas like quantization [3][4].

Our empirical analysis focuses on the down-projection layer of LLaMA3.1–8B, based on the findings from [2] where most outliers mostly existed in the down projection layers. However, any linear projection matrix with LoRA adapters—such as the Q, K, V projections in self-attention—can exhibit similar behavior. These matrices are functionally equivalent in structure and are therefore equally susceptible to the effects of outlier activations.

To further support this claim, we measured the kurtosis of input for the first 10 layers in LLaMA3.1-8B. High kurtosis is indicative of a heavy-tailed distribution, which implies the presence of strong outliers.

Among these, down-projection layers exhibited the highest kurtosis values, indicating an extremely heavy-tailed distribution and thus a high likelihood of dominant outlier channels. However, most other layers are also leptokurtic, meaning they too are likely to be influenced by outlier activations.

This supports our central claim: the phenomenon of outlier-induced gradient distortion is not limited to the down-projection layer, but is broadly relevant across various parts of the model that use LoRA adapters.

(W3 & Q2) Experiments on additional tasks with different Optimizer

1. Mathematical Reasoning

We further evaluated GraLoRA on a mathematical reasoning task using the MetaMathQA for training and testing on the MATH data. Two models, LLaMA3.2–1B and Qwen2.5–1.5B, were fine-tuned, mostly following the settings from Hu et al. [5]. Following our paper’s heuristic, we used k = 4 for both rank 64 and 128. In addition, we replaced the LionW to AdamW, since LionW itself can mitigate the influence of large gradients.

As shown in the table, all GraLoRA showed superior performance for math task, for both models. This highlights the robustness of GraLoRA, for different architecture and hyper-parameters including the optimizer.

2. General Language Understanding (GLUE)

We evaluated GraLoRA on the GLUE benchmark using RoBERTa-base and included two additional baselines: VeRA and FourierFT.

Following the protocol of prior work, we excluded MNLI and QQP—two time-intensive tasks—which also meant we did not apply the MNLI-based tricks for MRPC, RTE, and STS-B (as used in the original LoRA paper). Accordingly, we retrained LoRA on these tasks without this optimization and report updated results.

While VeRA and FourierFT involve fewer trainable parameters, their training time is comparable to or even longer than LoRA with rank 8. Therefore, we set the LoRA and GraLoRA ranks to 8 for fair comparison. Since this is a relatively low-rank setting, we also evaluated Hybrid GraLoRA by splitting the rank equally between LoRA and GraLoRA (i.e., 4+4), which is expected to be beneficial under such constraints. All hyperparameters followed those used in the VeRA implementation, except for learning rate, which was reduced by a factor of 5 to 10.

GraLoRA demonstrates strong performance in this low-rank regime, outperforming all baselines in average score. Hybrid GraLoRA delivers the most robust results, achieving the best performance on 4 out of 6 tasks. These results indicate that GraLoRA maintains high effectiveness even under constrained parameter budgets and in non-LLM architectures.

(W2 & Q3) Fair Comparison with Hyper-parameter search

We conducted an additional experiment on the commonsense reasoning task using the Qwen2.5–1.5B model, extending the number of training epochs from 2 to 3. In response to the concern about fixed learning rates, we also performed a grid search over learning rates {1e-4, 3e-4, 5e-4} for each method and selected the best-performing configuration individually.

GraLoRA continues to achieve the highest average accuracy and ranks first on 4 out of 8 tasks, demonstrating robust performance even under altered training schedules and per-method parameter tuning.

Interestingly, we observed that overall accuracy slightly decreased compared to the 2-epoch setting reported in the main paper. We believe this is due to overfitting, since we observed that training loss continued to decrease in the third epoch, while evaluation loss began to increase.

3. Commonsense Reasoning with Extensive PEFT Baselines

We conducted a new evaluation on the commonsense reasoning benchmark using the LLaMA3.2–3B model, comparing GraLoRA with a wide spectrum of PEFT methods, including full fine-tuning, LoRA, LoRA-XS, LoRA-SB, rsLoRA, PiSSA, BOFT, MELoRA, MoRA, and RaSA. All models were trained with rank 32. For consistency, we used the results reported in the LoRA-SB paper for full fine-tuning, LoRA-XS, LoRA-SB, rsLoRA, and PiSSA. For other methods, we conducted a learning rate sweep over {2e-4, 4e-4, 1e-3, 2e-3}, selecting the best-performing configuration per method. All other hyperparameters followed the settings in LoRA-SB, including the usage of AdamW optimizer.

GraLoRA achieves the highest average accuracy and ranks first on 5 out of 8 tasks. These results, combined with those in Table 3 of the main paper, confirm GraLoRA’s scalability and consistent improvements across model sizes, ranks, and PEFT baselines.

4. Diffusion Model Fine-Tuning

Finally, we applied GraLoRA to diffusion models by fine-tuning SDXL for personalization. We followed the official training setup from the HuggingFace diffusers repository, using the lambdalabs/naruto-blip-captions dataset. The dataset was split 90% for training and 10% for evaluation.

GraLoRA consistently outperformed LoRA in both CLIP and DINOv2 similarity scores, further demonstrating its generality and effectiveness beyond LLMs—including vision-text and generative architectures like diffusion models.

(Q1) Heuristic for Hyperparameter $k$

We have further analyzed the hyper-parameter sweep result in the ablation study section. Based on the sweep result and experience, $r/k^2 \approx 8$ yielded stable performance under varying configurations including model and task. Thus, we set GraLoRA $k=2$ for ranks under 32, and $k=4$ for higher ranks (64, 128) in the experiments.

To conclude, we sincerely thank the reviewer for the thoughtful feedback, which has significantly contributed to improving our work. In response, we have added new theoretical analysis on the generality of gradient distortion beyond down-projection layers, extended evaluations on mathematical reasoning and GLUE, and included comprehensive comparisons with regularized LoRA variants such as MELoRA. We have also investigated the impact of training duration and hyperparameter tuning, and validated GraLoRA’s generality through experiments on diffusion models. These additions will be incorporated into the revised manuscript to reflect the valuable insights raised in your review.

[1] Bondarenko, et al., "Understanding and Overcoming the Challenges of Efficient Transformer Quantization" [2] Yu, et al., "The Super Weight in Large Language Models" [3] Xiao et al., "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models" [4] Liu et al., "SpinQuant: LLM quantization with learned rotations" [5] Hu et al., "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models"

We are grateful for your careful reading of our paper and your positive remarks. Your comments have helped us clarify several important aspects.

(W1) Influence of Outlier Channels

As mentioned, prior studies have shown that outlier channels and weights are common in large models [1][2], and the impact of such outliers has been actively addressed in areas like quantization [3][4]. We also agree that existence of outlier channels and weights are nature, which usually show high correlation with tokens like ".", or special tokens.

On the other hand, while the existence of outlier is not a problem in general situations, it can give negative influences when training LoRA adapters. As illustrated in Figure 2, while full fine-tuning localizes the gradient impact, LoRA’s entire gradient update becomes disproportionately influenced by the single outlier.

Such structural limitation of LoRA leads to misalignment between LoRA updates and the gradient landscape shaped by full fine-tuning. In addition, since outlier only exists in certain layers, it results in different gradient scale across layers making the model harder to learn.

(W2) Flexibility and Capacity analysis of GraLoRA

To comapare the flexibility and capacity, we examine the column spaces of the weight update matrices from LoRA and GraLoRA within the framework of Grassmannian manifold.

Let $R_{LoRA} = B_{LoRA}A_{LoRA}^\top \in \mathbb{R}^{M \times N}$ and $R_{GraLoRA} = B_{GraLoRA}A_{GraLoRA}^\top \in \mathbb{R}^{M \times N}$, as illustrated in Figure 5 of the paper. In vanilla LoRA, assuming column-wise linear in both $B_{LoRA}$ and $A_{LoRA}$, the rank of $R_{LoRA}$ is $r$. Thus, its column space lies in the Grassmannian manifold $\mathcal{C}(R) \in \mathcal{Gr}(r, M)$, which denotes the set of all $r$-dimensional linear subpaces in $\mathbb{R}^M$.

In GraLoRA, each sub-block $R_{i,j}$ has rank $\frac{r}{k}$, as shown in Section 3.2. The rank of each column block $R_i = [R_{i,1} R_{i,2} \cdots R_{i,k}]$ is there fore $r$ under the assumption that all sub-blocks are linearly independent, maximum rank of $R_{GraLoRA}$ can reach up to $kr$. Consequently, the column space of the GraLoRA update list in $\mathcal{C}(R_{gralora}) \in \mathcal{Gr}(kr, M)$.

In terms of dimension, the Grassmannian manifold $\mathcal{Gr}(r, M)$ has dimension $r(M-r)$, while $\mathcal{Gr}(kr, M)$ has dimension $kr(M-kr)$. Under the common setting where $r \ll M$, this implies that:

$$**dim**({\mathcal{Gr}(r, M)}) < **dim**({\mathcal{Gr}(kr, M)})$$

Therefore, the subspace learned by GraLoRA spans a higher-dimensional manifold, providing greater representational flexibility and capacity than standard LoRA. This geometric perspective further supports our theoretical claim that GraLoRA enhances expressivity by expanding the effective rank of the adaptation subspace.

(W4 & Q1 & Q3) Experiments with New Baselines and Variance Evaluation

1. Commonsense Reasoning with Extensive PEFT Baselines

We conducted a new evaluation on the commonsense reasoning benchmark using the LLaMA3.2–3B model, comparing GraLoRA with a wide spectrum of PEFT methods, including full fine-tuning, LoRA, LoRA-XS, LoRA-SB, rsLoRA, PiSSA, BOFT, MELoRA, MoRA, and RaSA. All models were trained with rank 32. For consistency, we used the results reported in the LoRA-SB paper for LoRA-XS, LoRA-SB, rsLoRA, and PiSSA. Here rsLoRA, BOFT, and MELoRA are popular regularization techniques in LoRA. For BOFT, MELoRA, MoRA, RaSA, and GraLoRA, we conducted a learning rate sweep over {2e-4, 4e-4, 1e-3, 2e-3}, selecting the best-performing configuration per method. All other hyperparameters followed the settings in LoRA-SB.

GraLoRA achieves the highest average accuracy and ranks first on 5 out of 8 tasks. These results, combined with those in Table 3 of the main paper, confirm GraLoRA’s scalability and consistent improvements across model sizes, ranks, and PEFT baselines.

2. Mathematical Reasoning

We further evaluated GraLoRA on a mathematical reasoning task using the MetaMathQA dataset for training and testing on the MATH benchmark. Two models, LLaMA3.2–1B and Qwen2.5–1.5B, were fine-tuned, mostly following the settings from Hu et al. [1], with ranks 64 and 128. Following our paper’s heuristic, we used $k=4$ for both.

GraLoRA consistently outperforms LoRA, especially on the larger model, highlighting its robustness across architectures and tasks requiring symbolic reasoning.

3. General Language Understanding (GLUE)

We evaluated GraLoRA on the GLUE benchmark using RoBERTa-base, an encoder-only architecture with eight subtasks. To ensure fair comparison with recent PEFT methods designed for parameter efficiency, we included two additional baselines: VeRA and FourierFT.

Following the protocol of prior work, we excluded MNLI and QQP—two time-intensive tasks—which also meant we did not apply the MNLI-based tricks for MRPC, RTE, and STS-B (as used in the original LoRA paper). Accordingly, we retrained LoRA on these tasks without this optimization and report updated results.

While VeRA and FourierFT involve fewer trainable parameters, their training time is comparable to or even longer than LoRA with rank 8. Therefore, we set the LoRA and GraLoRA ranks to 8. Since this is a relatively low-rank setting, we also evaluated Hybrid GraLoRA by splitting the rank equally between LoRA and GraLoRA (i.e., 4+4), which is expected to be beneficial under such constraints.

All hyperparameters for GraLoRA followed those used in the VeRA implementation, except for learning rate, which was reduced by a factor of 5–10, as VeRA uses a learning rate approximately 10× larger than LoRA.

GraLoRA demonstrates strong performance in this encoder-only setting and low-rank regime, outperforming all baselines in average score. Hybrid GraLoRA delivers the most robust results, achieving the best performance on 4 out of 6 tasks. These results indicate that GraLoRA maintains high effectiveness even under constrained parameter budgets and in non-LLM architectures.

(W3 & Q2) Image Task Experiments

4. Diffusion Model Fine-Tuning

We applied GraLoRA to diffusion models by fine-tuning SDXL for personalization. We followed the official training setup from the HuggingFace diffusers repository, using the lambdalabs/naruto-blip-captions dataset. The dataset was split 90% for training and 10% for evaluation.

GraLoRA consistently outperformed LoRA in both CLIP and DINOv2 similarity scores, further demonstrating its generality and effectiveness beyond LLMs—including vision-text and generative architectures like diffusion models.

To conclude, we sincerely thank the reviewer for their valuable suggestions and thoughtful critique. Your feedback prompted us to clarify the role of outlier channels in LoRA, deepen the theoretical analysis of GraLoRA’s expressivity via Grassmannian geometry, and conduct comprehensive new experiments—including comparisons against regularized LoRA variants such as MELoRA, as well as evaluations on mathematical reasoning, GLUE, and diffusion tasks. These additions, derived directly from your comments, will be incorporated into the revised manuscript to strengthen both its theoretical depth and empirical breadth.

Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of our contributions and the insightful points raised.

(W1) Theoretical justification of GraLoRA's gradient dynamics

While the gradient dynamics of LoRA are theoretically formulated in Equation (4) and Figure 2, and empirically validated in Figures 3 and 4, the analysis for GraLoRA in the main paper has thus far focused on empirical results (Figures 3 and 6). We now provide a more rigorous treatment of GraLoRA’s gradient dynamics.

GraLoRA divides the adapter into $k^2$ sub-blocks. Each sub-block is defined as

$$R_{i,j} = B_{i,j}A_{i,j}^\top, \ B_{i,j} \in \mathbb{R}^{\frac{M}{k} \times \frac{r}{k}}, \ A_{i,j} \in \mathbb{R}^{\frac{N}{k} \times \frac{r}{k}}$$

Given an input submatrix $X_i \in \mathbb{R}^{\frac{N}{k} \times T}$, the gradient of the parameters is:

$$\frac{\partial{L}}{\partial{A_{i,j}^\top}} = B_{i,j}^\top \frac{\partial{L}}{\partial{Y_j}}X_i^\top, \ \frac{\partial{L}}{\partial{B_{i,j}}}=\frac{\partial{L}}{\partial{Y_j}}X_i^\top A_{i,j}$$

The corresponding gradient of the composed update matrix $R_{i,j}$ is:

$$\frac{\partial{L}}{\partial{R_{i,j}}}=\frac{\partial{L}}{\partial{B_{i,j}}}A_{i,j}^\top + B_{i,j}\frac{\partial{L}}{\partial{A_{i,j}^\top}}=\frac{\partial{L}}{\partial{Y_j}}X_i^\top A_{i,j}A_{i,j}^\top + B_{i,j}B_{i,j}^\top \frac{\partial{L}}{\partial{Y_j}}X_i^\top$$

This closely resembles the gradient dynamics of LoRA, maintaining the same underlying structure of interaction between the input and parameter gradients.

Now if an outlier exists in input slice $X_{out}$, only $k$ sub-blocks $[R_{out, 1}, \dots , R_{out, k}]$ directly process the outlier input and receive the amplified gradient signals. In contrast, the remaining $k^2-k$ blocks remain largely unaffected, as they operate on non-outlier slices aligning from the observations in Figure 6. This selective exposure sharply contrasts with standard LoRA, in which a single outlier channel can influence the entire low-rank adapter due to global entanglement.

(W2 & Q1) Independence of all columns in the submatrices

The independence of columns in the submatrices $B_{(i,k)}$ and $A_{(i,k)}$ is a critical condition to achieve the maximum rank $kr$ in GraLoRA, comparable to the condition in standard LoRA, where the rank is maximized when all columns of $B$ and $A$ are linearly independent.

To justify why this condition holds in practice, we refer to results from random matrix theory. Consider a random matrix $X \in \mathbb{R}^{r \times r}$, where each entry is sampled independently from a continuous distribution (e.g., Gaussian). The event that such a matrix is rank-deficient is equivalent to it being $\det(X) = 0$. Since determinant function is a multivariate polynomial over $r^2$ dimensional space of the matrix entries, the set of rank-deficient matrices forms a lower-dimensional algebraic hypersurface of dimension $r^2-1$, which has Lebesgue measure zero in $\mathbb{R}^{r^2}$. Therefore:

$\mathbb{P}(\mathrm{rank}(X) < r) = \mathbb{P}(\det(X) = 0) = 0$ which implies $\mathbb{P}(\mathrm{rank}(X) = r) = 1$

This logic extends naturally to non-square matrices $X \in \mathbb{R}^{m \times n}$. Rank-deficiency in such cases would require all $k \times k$ minors (where $k = \min(m, n)$) to vanish simultaneously, which again corresponds to a measure zero. Therefore, the condition of independence in both LoRA and GraLoRA is well-supported by probabilistic arguments rooted in random matrix theory.

(W2) analysis with Grassmannian geometry

To compare the subspaces learned by the adapters, we examine the column spaces of the weight update matrices from LoRA and GraLoRA. Let $R_{LoRA} = B_{LoRA}A_{LoRA}^\top$ and $R_{GraLoRA} = B_{GraLoRA}A_{GraLoRA}^\top$, as illustrated in Figure 5 of the paper. In vanilla LoRA, assuming column-wise linear in both $B_{LoRA}$ and $A_{LoRA}$, the rank of $R_{LoRA}$ is $r$. Thus, its column space lies in the Grassmannian manifold $\mathcal{C}(R) \in \mathcal{Gr}(r, M)$, which denotes the set of all $r$-dimensional linear subspaces in $\mathbb{R}^M$.

In GraLoRA, each sub-block $R_{i,j}$ has rank $\frac{r}{k}$, as shown in Section 3.2. The rank of each column block $R_i = [R_{i,1} R_{i,2} \cdots R_{i,k}]$ becomes $r$ under the assumption that all sub-blocks are linearly independent, and $R_{GraLoRA}$ can reach the maximum rank $kr$. Consequently, the column space of the GraLoRA update list in $\mathcal{C}(R_{gralora}) \in \mathcal{Gr}(kr, M)$.

In terms of dimension, the Grassmannian manifold $\mathcal{Gr}(r, M)$ has dimension $r(M-r)$, while $\mathcal{Gr}(kr, M)$ has dimension $kr(M-kr)$. Under the common setting where $r \ll M$, this implies that: $**dim**({\mathcal{Gr}(r, M)}) < **dim**({\mathcal{Gr}(kr, M)})$ Therefore, the subspace learned by GraLoRA spans a higher-dimensional manifold, providing greater representational flexibility and capacity than standard LoRA. This geometric perspective further supports our theoretical claim that GraLoRA enhances expressivity by expanding the effective rank of the adaptation subspace.

(W3) GraLoRA on resource-constrained scenario

As discussed in Section 3.4 (Tradeoff Analysis), GraLoRA introduces no additional computational or memory overhead during training. This is because GraLoRA maintains the same number of parameters and FLOPs as standard LoRA.

During inference, GraLoRA adapters can be statically fused into the base model weights, just like in LoRA. Once merged, inference proceeds identically to that of the original model with updated weights, incurring no runtime penalty. Thus, for single-adapter deployments, such as in typical on-device or resource-constrained settings, GraLoRA remains as efficient as standard LoRA.

(W3) GraLoRA on multiple adapter scenario

We agree that multi-adapter settings introduce practical challenges due to the need to dynamically load and apply adapter weights without fusion. However, we emphasize that GraLoRA introduces no additional burden compared to LoRA in this scenario. Both methods involve the same number of parameters and identical FLOPs, resulting in equivalent memory and compute requirements per adapter.

To empirically validate this, we measured the inference latency of GraLoRA and LoRA under a multi-adapter setting using the LLaMA3.1–8B model with rank 32. We simulated a realistic scenario by generating 128 output tokens from 128 input tokens using a single NVIDIA H100 GPU, with adapters loaded dynamically and applied without fusion. The average end-to-end inference times were 5.1 seconds for LoRA and 5.3 seconds for GraLoRA.

This negligible difference demonstrates that GraLoRA scales effectively in multi-adapter deployments, and does not suffer from significant computational or architectural limitations compared to existing PEFT methods. We will include this measurement in the revised paper to clarify GraLoRA’s practicality in such scenarios.

(Q2 & Q3) Initialization and hyper-parameter settings of GraLoRA

Thank you for pointing this out. The initialization and hyperparameter settings for GraLoRA follow the same convention as in standard LoRA. Specifically, each sub-block adapter $A_{i,j}$ is initalized using Kaiming uniform initialization, while $B_{i,j}$ is initialized to zero. Regarding the scaling factor, we currently apply the same LoRA alpha value uniformly across all blocks. Consequently, each sub-block’s output is scaled identically, ensuring consistent training dynamics across the adapter grid.

(Q4) Applicability of VeRA and LoRA-SB to GraLoRA

Both VeRA and LoRA-SB primarily aim to reduce the number of trainable parameters, but they do not fundamentally address the gradient sensitivity to outlier channels. In VeRA, the shared matrices $A$ and $B$ still multiply the full input $X$, so outliers can dominate the gradient signal. LoRA-SB adopts the structure $R = B R A^\top$, training only $R$, but since $A$ still interacts with the entire input, it also remains vulnerable to outliers.

Nevertheless, both styles can be incorporated into GraLoRA. A VeRA-style GraLoRA would share $A$ and $B$ across sub-blocks, increasing parameter sharing. While this raises the latent dimensionality from $r$ to $kr$, the blockwise structure of GraLoRA still localizes the influence of outliers, offering improved robustness. For LoRA-SB-style GraLoRA, each sub-block can be reparameterized as $R_{i,j} = B_{i,j} R_{i,j} A_{i,j}^\top$, preserving GraLoRA’s outlier isolation while benefiting from reduced trainable parameters.

To further validate GraLoRA’s generality, we have conducted additional experiments across diverse tasks, including commonsense reasoning, mathematical reasoning, GLUE, and diffusion model fine-tuning. These evaluations incorporate VeRA and LoRA-SB as baselines and consistently demonstrate GraLoRA’s robustness and effectiveness across domains. As this response focuses on mathematical and theoretical aspects, the corresponding experimental results are provided in our responses to other reviewers.

To conclude, we sincerely thank the reviewer for the thoughtful and constructive feedback. The insightful comments guided us to develop additional theoretical analysis of GraLoRA’s gradient dynamics, independence assumptions, and subspace structure, which we have now included. Furthermore, we conducted new experiments, including inference latency and multi-task evaluation, that address practical concerns raised in the review. These theoretical clarifications and empirical results will be incorporated into the revised manuscript to improve clarity, completeness, and impact.