From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications

We would like to thank you for your time to review our work. Next, we will try to address the weakness pointed by you one-by-one as follows:

1. Discussion of Sparsity in Section 2.4 and typoes: Thank you for identifying this mistake. In Section 2.4, we mean low-rank representation and we promise to correct this in our camera-ready version of our submission. We will also correct additional typoes including threshold.

2. Rescaling Figure 1: We would like to highlight that we intentionally have not rescaled all subfigures in Figure 1 to effectively capture/illustrate the variation in eigenvalues across different component types (MLP vs attention) of the model.

3. Clarification regarding Equation 1: $S_{W_l}$ is an array of singular values of weight matrix $W$ of layer $l$. $(S_{W_l} < k)$ will be a $0/1$ array indicating which singular values are less than k which will be truncated or compressed. $sum(S_{W_l} < k)$ is the sum of 1’s in the array illustrating the reduction ratio of layer l. Summing across all the layers (i.e., $\sum_l sum(S_{W_l} < k)$) will give the total rank reduction in the entire model. $len(S_{W_l})$ is the indicator of the number of singular values in the weight matrix of layer $l$ and it may vary depending upon the model architecture choices.

We would first like to thank you for the time to review our work. We would now address your weakness point by point as follows:

1. Computational overhead of SVD on large matrices: Thank you for raising this point. We would like to highlight that WeLore-COMP is a one-shot data-agnostic compression technique. WeLore-COMP requires one-time SVD estimation of all weight matrices of the pretrained checkpoint which can be saved and various compression ratios (e.g., 10%, 20%, 30%, etc.) can be achieved by a simple linear search of threshold k (Algorithm 1) without any need for re-estimation of SVD.

1. For empirical estimates of LLaMa2-7B using a Nvidia A6000 RTX GPU, a 4096 * 4096 matrix SVD decomposition takes ~2.6895 seconds to complete. Given 7 weight matrices for each transformer layer of approximately similar dimensions in LLaMa2, SVD time cost for decomposition of a transformer layer will be ~19 seconds. With 32 layers, the total time taken will be approximately ~10 minutes. Note that SVD estimation of each layer can be executed independently and parallel. Therefore, with 8 GPUs, we can reduce the time 10/8 = ~1.25 minutes in total.

2. For empirical estimates of LLaMa2-70B using a Nvidia A6000 RTX GPU, a 8192 * 8192 matrix SVD decomposition takes ~2.799 seconds to complete. Given 7 weight matrices for each transformer layer of approximately similar dimensions in LLaMa2, SVD time cost for decomposition of a transformer layer will be ~20 seconds. With 80 layers, the total time taken will be approximately ~26 minutes. Note that SVD estimation of each layer can be executed independently and parallel. Therefore, with 8 GPUs, we can reduce the time 26/8 = ~3.25 minutes in total.

We promise to include additional computational cost discussion on SVD in our final draft.

2. Low-rank patterns and Datasets: We would like to clarify that low-rank patterns emerging in model checkpoints are subjected to their pre-training. WeLore exploits these existing patterns for data-agnostic compression and PEFT. Experimentally through a wide-range of tasks in our experiments ranging from commonsense tasks to open-ended tasks (MT-Bench with code-centric questions - Table 2), we found that WeLore consistently perform well for the existing low-rank patterns in pre-trained checkpoints.

3. Freezing Weights and Underfitting: Thank you for raising this point. Across all the experiments with WeLore-PEFT with different compression ratios, we didn’t observe the underfitting issue and WeLore-PEFT can closely mimic the training trajectory of full-finetuning. For example, as it can be seen from Table 6, with 30% compression, which approximately freeze 70% weights of the model during PEFT, WeLore outperform full-finetuning across several tasks which indicate that task adaptations can be adopted by only a few components in the model.

4. Extension for encoder-decoder architecture: Thank you for this suggestion. Decoder-only architectures like LLaMa and Mistral family are most popular and scaled up architecture, that’s why we choose them as our experimental focus. We would like to highlight that while deeper cross-attention layers or an encoder-decoder structure might exhibit differences in rank collapse behavior compared to decoder-only models, these differences do not fundamentally invalidate the mechanism underlying WeLore’s compression or PEFT strategy. The low-rank collapse is driven by the stabilization of the gradient subspace and the emergence of a clear Hessian spectral gap—a phenomenon that is intrinsic to the training dynamics rather than being solely an artifact of a particular architecture. Even if cross-attention layers show a less pronounced rank collapse (due to their role in integrating encoder context, which can diffuse gradient signals), WeLore’s design is inherently adaptive. In other words:

• Even if some layers (such as deep cross-attention ones) deviate from the typical rank collapse pattern, the thresholding in WeLore is computed on a per-layer basis, automatically preserving higher ranks where needed and applying more aggressive compression where the low-rank structure is clear.

• The core mechanism—gradient subspace alignment with dominant Hessian directions—remains relevant across different architectures. Thus, while encoder-decoder models might introduce variations, they do not “disrupt” the phenomenon in a way that would render adaptive thresholding ineffective.

Due to time limit of rebuttal, we couldn't complete our experiments but we are working towards it and we will include in our final version.

We would first like to thank you for the time to review our work. We would first like to thank you for finding our work to provide extensive experiments to establish robustness and have practical significance. We would now address your weakness point by point as follows:

1. Computational overhead of SVD on large matrices: Thank you for raising this point. We would like to highlight that WeLore-COMP is a one-shot data-agnostic compression technique. WeLore-COMP requires one-time SVD estimation of all weight matrices of the pretrained checkpoint which can be saved and various compression ratios (e.g., 10%, 20%, 30%, etc.) can be achieved by a simple linear search of threshold k (Algorithm 1) without any need for re-estimation of SVD.

1. For empirical estimates of LLaMa2-7B using a Nvidia A6000 RTX GPU, a 4096 * 4096 matrix SVD decomposition takes ~2.6895 seconds to complete. Given 7 weight matrices for each transformer layer of approximately similar dimensions in LLaMa2, SVD time cost for decomposition of a transformer layer will be ~19 seconds. With 32 layers, the total time taken will be approximately ~10 minutes. Note that SVD estimation of each layer can be executed independently and parallel. Therefore, with 8 GPUs, we can reduce the time 10/8 = ~1.25 minutes in total.

2. For empirical estimates of LLaMa2-70B using a Nvidia A6000 RTX GPU, a 8192 * 8192 matrix SVD decomposition takes ~2.799 seconds to complete. Given 7 weight matrices for each transformer layer of approximately similar dimensions in LLaMa2, SVD time cost for decomposition of a transformer layer will be ~20 seconds. With 80 layers, the total time taken will be approximately ~26 minutes. Note that SVD estimation of each layer can be executed independently and parallel. Therefore, with 8 GPUs, we can reduce the time 26/8 = ~3.25 minutes in total.

We promise to include additional computational cost discussion on SVD in our final draft.

2. Generalizability to other architectures such as GPT-style models: We would like to have additional clarification on this weakness. Our experiments across various tasks,compression ratios on LLaMa & Mistral illustrate that WeLore generalizes to GPT-style decoder models. Decoder architectures are most popular and scaled up architecture, that’s why they are our experimental focus.

3. Theoretical framework relies on a reversibility assumption: The current proof employs the reversible network structure primarily as a sufficient condition to ensure the boundedness and stability of both gradients and Hessians. In particular, the reversibility assumption is invoked to prevent unbounded growth of norms and to guarantee stable spectral properties of the Hessian sequence $\{H_t\}$. Notably, if boundedness and stability can be established through alternative means, strict reversibility may not be necessary.

From a purely mathematical standpoint, the key requirements of the proof are as follows:

1. Boundedness of $||G||$, $||W_t||$, and $||H_t||$:
   It is imperative to ensure that the norms of gradients, weights, and Hessians do not diverge. When these quantities remain bounded, the spectral gap property and the Lipschitz continuity of the Hessian ensure stable eigen-decomposition and validate the application of Davis–Kahan perturbation theory.
2. Stable Spectral Gap:
   For $H_t=\nabla^2 L(W_t)$ with eigenvalues in descending order, there is a uniform $\gamma>0$ such that $\min_{1 \le i \le r,\, j>r}|\lambda_i(H_t)-\lambda_j(H_t)|\ge \gamma$.

It is important to emphasize that these conditions do not inherently require reversibility. Many standard optimization settings—such as strongly convex problems or well-conditioned neural architectures—employ mechanisms that naturally ensure boundedness and prevent degenerate Hessians. Consequently, if uniform boundedness of parameters, gradients, and Hessians can be secured by other means (for example, weight regularization, strong convexity near minima, or alternative architectural constraints), then the strict assumption of reversibility is not essential.

4. Benefit from other compression techniques on N-LRCs layers: Thank you for raising this point and we agree that exploration of mixed compression strategy can be highly interesting with WeLore. We have indeed conducted experiments to study WeLore in conjunction with pruning. A careful observation of popular non-uniform layerwise pruning algorithms (https://arxiv.org/pdf/2310.05175) reveals that the majority of middle transformer blocks can be subjected to a higher pruning ratio which is complementary to WeLore low-rank reduction ratio that favours terminal blocks being low-rank friendly (Appendix B.1). Our experimental results in Appendix B.2 reveals that WeLore can be used in conjunction with SoTA pruning techniques like SparseGPT, Wanda etc. due to existing orthogonal properties in mixed compression settings with minimal performance degradations.

We would first like to thank you for the time to review our work. We greatly appreciate that you have found our theoretical analysis novel and experimental section comprehensive while identifying the unique proposition of WeLore as a cohesive method to handle compression and fine-tuning in a unified way. We would now like to address the weaknesses pointed by you one by one as follows:

1. Issues with Figure and Visualization Clarity: We appreciate your concerns regarding about some mislabel and missing texts in our figures in the submitted draft. We promise you that we will address all these issues in the camera-ready version of our submission.

2. Hyperparameter sensitivity (k): Thank you for raising this point. We would like to highlight that our hyperparameter k is uniquely determined/searched based on the effective rank reduction ratio (ERR) required. We use a linear search with np.linspace(0, 1, 0.005) using the algorithm described in Appendix D3 (Algorithm 1) to preserve normalized singular values greater than the threshold k so that we can achieve the ERR. In our experiments, we have found that the precision 0.005 is sufficient to select optimal threshold K for a given ERR without notable variation in results. We will provide this additional clarification in the final draft of our submission. For additional information, we have included the pre-estimated singular value thresholds (k) for LLaMa-2 7B and 13B as follows:

3. WeLore and other compression techniques: We appreciate your interest about WeLore comparison with other compression techniques like pruning and quantization. WeLore is a low-rank compression method with its own unique benefit of hardware-friendly acceleration and it is unclear what compression ratio of low-rank compression will equate to what compression ratio of pruning and quantization. However, we have indeed conducted experiments to study WeLore in conjunction with pruning. A careful observation of popular non-uniform layerwise pruning algorithms (https://arxiv.org/pdf/2310.05175) reveals that the majority of middle transformer blocks can be subjected to a higher pruning ratio which is complementary to WeLore low-rank reduction ratio that favours terminal blocks being low-rank friendly (Appendix B.1). Our experimental results in Appendix B.2 reveals that WeLore can be used in conjunction with SoTA pruning techniques like SparseGPT, Wanda etc. due to existing orthogonal properties in mixed compression settings with minimal performance degradations.