3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs

We would like to thank the reviewer for the high-quality review, highlighting the strengths of our paper, and the useful suggestions to improve the manuscript that led to an insightful finding regarding using OWL for (S+LR) methods.

Weaknesses:

W1: You are making a very good point. If one expands the expression of equation (1), it is equivalent to minimizing $1/2Tr((\hat{W} - S - L)^T (X^TX+\lambda I) (\hat{W} - S - L))$. So it behaves like adding a regularization term to the local layer-wise reconstruction error Hessian $X^TX$. The aim here is to make the Hessian matrix $X^TX + \lambda I$ invertible (which is satisfied for any $\lambda > 0$) and reduce the numerical errors when applying equation (4).

From a global optimization perspective, we aim to minimize the layer-wise reconstruction error $*without deviating too far*$ from the original weights. It has also been used in previous (S + LR) compression methods like Hassle-free.

W2.1: Our method is indeed generalizable to unstructured sparsity patterns. We experiment with a "less aggressive compression" configuration (0.5 + 128) and study the differences after further LoRA fine-tuning. We believe that it is interesting to further explore these settings as they are "near-lossless" configurations.

Table 1: Comparison of the the perplexity of (S+LR) algorithms before and after LoRA fine-tuning on the configuration (0.5 + 128) on the model Llama3.2-1B. For perplexity (lower is better, ↓). If a dataset has the suffix "-LFT", we report the perplexity on that dataset after further LoRA fine-tuning on the C4 dataset [this can decrease the performance on WT2 and PTB datasets if the model has a performance comparable to dense model--3BASiL seems to be the only one to have this property].

W2.2: Our method can also support non-uniform sparsity distributions and we have added support for OWL for unstructured sparsity. This is a great insight and we would like to thank the reviewer for raising this important remark. We have tried to activate OWL for the configuration (0.7 + 64) for Llama3-8B. It turns out that this does improve the results of 3BASiL. This is not trivially true because OWL deltas were optimized for pure pruning methods. We believe that this is an insightful finding for the community. It opens further research directions as to how to create similar deltas for the low-rank components, so thanks to the reviewer for this suggestion!

Table 2: Comparison of the performance of our proposed method 3BASiL with and without activating OWL on the configuration (0.7 + 64) on Llama3-8B. For perplexity (lower is better, ↓). For accuracy (higher is better, ↑).

We intend to add a section showcasing the value of activating OWL for highly sparse unstructured configuration [e.g. (0.7 + 64)] and adding a discussion motivating the question of non-uniform sparsity and non-constant rank allocation as a future research direction.

W3: 80 alternating minimization steps is the default value in the papers of OATS and Hassle-free. A sensitivity analysis has been conducted in OATS for instance [see figure 1 of OATS paper [1]], and shows that the performance plateaus after 80 minimization steps. For these alternating-minimization based methods, increasing the number of steps generally lowers the objective (1) and improves the performance. If we were to reduce the number of steps, the performance would degrade.If we were to increase it, the performance gain is negligible (plateau reached) in our experiments but the runtime still increases.

Our algorithm is different from alternating minimization [used by OATS/HASSLE-free] and the number of iterations is not directly comparable to OATS and HASSLE-free. That being said, 3BASiL is both faster and achieves a better performance than OATS and HASSLE-free [see figure 2 in our submission].

[1] Zhang et al., OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition (ICLR'25)

We would like to thank the reviewer for the extensive feedback, highlighting the strengths of our paper, and the valuable suggestions to improve the manuscript.

Weaknesses and Questions

W1 & Q1: Our method is indeed generalizable to unstructured sparsity patterns. We experiment with a "less aggressive compression" configuration (0.5 + 128) and study the differences after further LoRA fine-tuning. We believe that it is interesting to further explore these settings as they are "near-lossless" configurations.

Table 1: Comparison of the the perplexity of (S+LR) algorithms before and after LoRA fine-tuning on the configuration (0.5 + 128) on the model Llama3.2-1B. For perplexity (lower is better, ↓). If a dataset has the suffix "-LFT", we report the perplexity on that dataset after further LoRA fine-tuning on the C4 dataset [this can decrease the performance on WT2 and PTB datasets if the model has a performance comparable to dense model--3BASiL seems to be the only one to have this property].

W2 & Q2: Thank you for pointing out this very important point! We are currently running experiments on the model OPT-30B which we intend to include in the revised paper [see Table 2 below for a comparison between 3BASiL and Hf-SparseGPT--it seems that this type of compression has less impact on performance for larger models].

According to SLOPE [1], we should expect an inference acceleration of up to 1.25× and up to 0.63 memory reduction for this model if we use their custom designed CUDA kernels during inference.

Table 2: Comparison of the the perplexity of Hf-SparseGPT and 3BASiL on (2:4 + 64) configuration for a OPT-30B model.

Q3: In the revised paper, we are going to expand our experiments to report more unstructured sparsity (S+LR) configurations similar to Table 1 above. We intend to include more pure pruning results to showcase the universality of Transformer Matching (see Table 2 of our response to reviewer JUTM). In addition, we are going to include a subsection showcasing the applicability of our method to non-uniform transformer compression (see Table 2 of our response to reviewer J8EX). Finally, we will report some results on the model OPT-30B to see the performance of 3BASiL on larger models [similar to Table 2 above].

[1] Mozaffari et al., SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs (ICLR'25).

We would like to thank the reviewer for the thorough review, for highlighting the strengths of our submission, and for raising important fundamental questions about our compression algorithm.

Weaknesses & Questions

W1: Thanks for pointing out this remark. We also think that including plots for the convergence of the objective and iterates will further improve the quality of the paper, we will add such plots in the revised version.

Since we are not allowed to include plots for this rebuttal per NeurIPS policy, we wanted to share simple logs we used to follow the convergence of 3BASiL [for the compression of self_attn.q_proj of the first transformer of Llama3-8B under a (2:4 + 128) configuration].

For iteration 009, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 1952.1258544921875

For iteration 049, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 664.7432861328125

For iteration 149, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 176.23971557617188

For iteration 199, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 119.56678771972656

For iteration 299, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 87.93989562988281

For iteration 399, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 87.01194763183594

For iteration 479, the true loss ||X(W_old - (W_S + B @ A))||_F^2 = 86.82907104492188

[479 is the last iteration, note that iterations in our algorithm are extremely efficient and are not comparable to the default value 80 of alternating minimization steps used in HASSLE-free and OATS].

W2: Thank you for pointing out this very important point! We are currently running experiments on the model OPT-30B which we intend to include in the revised paper [see Table 2 below for a comparison between 3BASiL and Hf-SparseGPT--it seems that this type of compression has less impact on performance for larger models].

According to SLOPE [1], we should expect an inference acceleration of up to 1.25× and up to 0.63 memory reduction for this model if we use their custom designed CUDA kernels during inference.

Table 2: Comparison of the the perplexity of Hf-SparseGPT and 3BASiL on (2:4 + 64) configuration for a OPT-30B model.

Q1: Thank you for making the connection between our optimization formulation and the RPCA literature. Our problem is quite different from RPCA as we aim to match the $**outputs**$ of each weight matrix $||X(W - S - L)||_F$ as opposed to the actual matrix weights matching $||W - S - L||_F$. A block in ADMM is introduced to this end. Please also note [line 154 of our paper] that our 3-block ADMM approach can be reformulated as a standard 2-block because the Lagrangian is separable with respect to 2 blocks introduced for the optimization.

Q2: This is a typo, thanks for catching it. The correct inequality should be

$| L^{(t+1)} - L^{(t)} |_F \leq \frac{3C_F^2 C_A}{\rho_{t-1}}$,

which directly uses the bound in Eq (17). The constant $C$ should be correspondingly changed to $C=3C_F^2C_A$ (since $C_F\ge 1$). We deeply appreciate the reviewer's careful examination and have corrected this typo in our revised manuscript.

Q3: This is a fundamental question related to one-shot compression methods! It is true that $S$ and $L$ can be updated with the original loss function ($L$ is updated during LoRA fine-tuning for example) or a knowledge distillation loss from the original model. However, if one aims to update $S$, one needs to do a full back-propagation on the entire LLM which is very memory intensive. Our entire pipeline: compression of LLama3-8B with 3BASiL, refinement with TM and LoRA fine-tuning can be done on a single A100 GPU. However, full back-propagation even with batch=1 results in a cuda memory error. That's why Transformer Matching introduces an intermediate [memory-efficient] loss function which provides is a good trade-off between the local-layerwise reconstruction loss and the original training loss.

[1] Mozaffari et al., SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs (ICLR'25).

We would first like to thank the reviewer for the thorough examination of the paper, highlighting the strengths of our submission and proposing missing pieces that would improve the manuscript.

Weaknesses -- Questions

W1 & Q2: Thank you very much for pointing EoRA which should indeed be included in the related works of our paper. Upon inspection of this important method, we noted that HASSLE-free [a considered competing method] with a special configuration (alternating minimization steps=1 and using our improved implementation--see Table 4 discussion) reduces to EoRA. In that case, Hf-SparseGPT-ours would (i) compress the model weights to 2:4 sparsity using the SparseGPT algorithm and (ii) use the closed-form rank-update (equation 4) for "compensation" as discussed in the EoRA paper [1]. In EoRA, $Q^\prime$ corresponds to $H^{1/2}$ in our paper (up to a minor notation difference, as they consider $XX^T$ to be the hessian whereas we use $X^TX$).

We launched experiments with EoRA, Hf-SparseGPT-ours (default AM steps=80) and 3BASiL for Llama3-8B under a (2:4 + 128) configuration [similar to Table 2 in EoRA table] (we also compare the effect of using TM on top of EoRA to show universality).

We obtain the following results:

Table 1: Comparison of (S+LR) methods with (2:4 + 128) configuration on Llama3-8B. For perplexity (lower is better, ↓). For accuracy (higher is better, ↑).

Note that the results of our implementation of EoRA are slightly different than reported in the EoRA paper.

(i) Our EoRA implementation has a slightly worse WikiText ppl than reported in the EoRA paper (11.07) because our calibration data (X matrix) is C4 whereas they consider WikiText2 (training data). Despite this fact, 3BASiL-TM [with calibration C4] largely outperforms EoRA [with calibration WikiText2] on the WikiText2 test perplexity task (from 11.07 to 8.82) under the same constraints.

(ii) Our EoRA implementation has a slightly better ARC-C accuracy than reported in the original EoRA paper. For this metric, we both use the same calibration data C4. That being said, we use more calibration data (128 segments vs. 64 in EoRA paper). Our method 3BASiL-TM improves the accuracy post-compression from 35.75 to 44.88.

We would like to thank the reviewer again for pointing us to this important method. We intend to add it to the related works discussion (Exact Low-Rank updates for layer-wise compression). We also think it is helpful for the compression community if we add a paragraph discussing the underlying connections between our improved implementation of Hf-SparseGPT-ours and EoRA as well as more experiments in the Appendix.

W2 & Q1: This is a very good remark. We have launched preliminary results for the universality of Transformer Matching to pure pruning methods. The results in Table 3 of the paper for pure pruning methods have been extracted from HASSLE-free paper [2]. During the rebuttal, we have added support for SparseGPT and ALPS with TM. The results are below.

Table 2: Comparison of pure pruning methods with 2:4 and 4:8 configurations on Llama3-8B. For perplexity (lower is better, ↓). For accuracy (higher is better, ↑).

We agree with the reviewer that further experiments should be included to pure sparsity algorithms to showcase the universality of our proposed Transformer Matching procedure. In the revised paper, we will add support to all reported pruning methods and their enhanced TM versions.

W3: Thank you for pointing out these issues. They have been fixed in the revised manuscript.

We want to reiterate our appreciation for the thorough feedback of the reviewer. We believe that incorporating the suggestions and addressing the questions and weaknesses discussed above will greatly improve the quality of the revised paper.

[1] Liu et al., EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation.

[2] Makni et al., A unified framework for Sparse plus Low-Rank Matrix Decomposition for LLMs (CPAL'25).