SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

Dear Reviewer 8cQQ,

We sincerely appreciate your valuable comments and suggestions on our paper. We have made every effort to address your concerns and hope these updates align with your expectations.

Although we have not yet received further responses, we remain motivated and are continuously working to improve our paper. Specifically, we compared SVD-LLM*—SVD-LLM without the proposed sequential LoRA fine-tuning—with current pruning-based LLM compression methods. As demonstrated in the results, SVD-LLM continues to outperform pruning-based methods even without LoRA. We promise to include this analysis in our new revised version upon acceptance. We believe the current set of experiments is sufficiently fair, as it compares SVD-LLM against both SVD-based and pruning-based methods, with and without LoRA.

We also deeply appreciate your acknowledgement that Thanks to the authors for addressing all questions. in the last response. Your recognition means a great deal to us. Since there are fewer than 24 hours remaining in the discussion period, we kindly ask if you have any additional concerns. We would be grateful for the opportunity to address them promptly and hope this might lead you to reconsider your score.

Thank you very much!

Best wishes,

All authors of Submission 2301

Hi,

Let's be clear: I am still open to adjusting my score. It’s just that, given there is still roughly a week left, more discussion would give me greater confidence in my decision.

To convince me, I suggest that the authors address the following.

1. Argue the novelty/contribution from areas other than the optimal compression part.
2. Acknowledging more the work of DRONE, which to me is still quite conceptually similar. Whether improving the computation/storage is a significant contribution is subjective. Calling this an improvement is fine, but like stating that DRONE is unusable as in Sec 3.1 is a bit too much.

I think the problem is actually point 1, but we are discussing point 2 because of the lack of a strong argument for point 1. We can still do the discussion on 2 though.

SVD-LLM is NOT just fixing some edge cases of DRONE.

I don't know why you take my sentence out of context.

So the thing to be discussed here is whether SVD-LLM and DRONE will always output the same thing, disregarding time complexity, memory usage, and numerical error.

You claim that there is a case where outputs are different and the output of SVD-LLM is better. I questioned that and asked for the significance of the case, but you did not respond to me.

If you can prove that SVD-LLM and DRONE output differently for significance cases and the output of SVD-LLM is better, then I am convinced, and you probably want to tell the author's of DRONE to fix their paper and do some kind of clarification as they seem to claim optimality.

Otherwise, I think of this as time complexity/memory usage/numerical error optimization of more or less the same conceptual algorithm, where the degree of contribution is subjective.

DRONE can NOT even compress a 7B LLM using a NVIDIA A100 GPU with 80GB memory.

I feel like this statement should be quantitative instead of qualitative, so like on what dataset and what is your memory estimation of DRONE. For example, openbookqa only has roughly 5000 samples, which DRONE should probably work. There is larger ones, but it is weird to say DRONE always don't work for LLM.

It is also kind of weird you are arguing applicability to LLM by how you differ in handling the data dimension.

To me, there are many ways to fix the memory usage, but basically you don't have to store them all on GPU. You only need the activation X one at a time, so you can just store it else where or you might even able to do the computation on CPU. This is not too important though, as I stated this is just computation/memory optimization, where there are contributions but the significance is debatable.

If the reviewer believes DRONE can compress LLMs by using its original algorithm or some easy fix, please let us know how it can be done.

Sure. The authors claim that the main difference is whether to cache $X$. Let's see whether DRONE really need to do that. As we can see, DRONE actually only want $U_X$ and $S_X$, while it is $V_X$ for the data dimension that is causing the memory issue you mentioned.

I believe there should be lots of established way to do this. But one should be storing and eigen decompose $XX^T$ to get $U_X$ and $S_X$. It is kind of similar to what you do in SVD-LLM, but this is pretty standard, so I don't feel like I am copying your solution.

I am slightly tired so I might get this wrong. Actually, I hope I am wrong. Otherwise, this feels easier than I originally thought.

Sincerely yours, Reviewer 3PXg

Dear reviewer 3PXg,

Thank you for the comments. We believe the discussion is mutually beneficial in the sense that we learn from each other’s perspective, and understand how novelty and contributions can come from different aspects.

Let’s start with your suggested method to help fix DRONE’s applicability issue to compressing LLMs. Basically, what you suggested to improve DRONE is EXACTLY what we did in SVD-LLM. In other words, what you suggested is directly borrowing the idea proposed in SVD-LLM to help DRONE. Therefore, we do not think your suggested method is fair to SVD-LLM. What we asked in our previous comment is whether the reviewer could suggest a method that is not borrowed from SVD-LLM.

Next, let’s follow your suggestion on being quantitative, and use the example you used (openbookqa with 5000 samples) to show you quantitative evidence so that we can avoid subjective conjecture such as “DRONE should probably work”. The average sequence length of data in openbookqa is about 256. For LLaMA-7B, the maximum dim of input activation is its intermediate_size = 11,008. Therefore, if we use DRONE, the total memory for caching the activation X for a single weight matrix at a time, as suggested by the reviewer, is 5000 (data number) x 256 (seq_len) x 11008 (dim) x 32 (fp32) / 1024/1024/1024 = 419GB, which is more than 5 times larger than the memory provided by NVIDIA A100 GPU, which has 80GB memory. In contrast, SVD-LLM only requires 11008x11008x32/1024/1024/1024 = 3.6GB. Therefore, compared to DRONE, SVD-LLM achieves a memory reduction of 419GB/3.6GB, which is 116.38 times less than DRONE. In our paper, we used WikiText-2 as the dataset. If we use DRONE, the total memory to cache the activation of a single weight matrix is larger than 24,600GB. In contrast, SVD-LLM still requires 3.6GB memory, which is over 6,000 times less than DRONE. These memory reduction improvements are quantitative and objective; it is simply NOT fair to count these quantitative and objective contributions as subjective contributions claimed by the reviewer.

Lastly, the “optimal compression part” you referred to in your point 1 should consist of two components: (1) the theoretical optimal output, and (2) the time complexity/memory usage/numerical stability that allows the proposed algorithm to be able to generate the theoretical optimal output. The reason is simple: talking about one while ignoring the other is not fair simply because your real model compression results depend on BOTH. DRONE is a great work that contributed to component (1), no doubt about it. We will follow the reviewer’s suggestion to acknowledge more and revise the statement of DRONE’s contributions in Section 3.1. We really appreciate the reviewer’s suggestion and respect Drone’s contribution. Fairly speaking, SVD-LLM achieves the same theoretical minimum as DRONE, and is able to deliver much better model compression results due to its algorithm’s benefits on memory reduction and numerical stability. If you look at the model compression literature, quite a significant number of published papers make contributions to time complexity/memory usage/numerical stability with quantitative and objective results. Therefore, we believe it is NOT fair to treat contributions on time complexity/memory usage/numerical stability as subjective and weaken its contributions to model compression.

Once again, we are deeply grateful for your timely response and detailed review provided for our paper. Thank you very much!

Best regards,

All authors of Submission 2301

Dear Reviewer 3PXg,

First, we are deeply grateful to the reviewer for raising the score. Given that the deadline for submitting revision has passed, we are not able to submit another revised version to OpenReview. However, we will deliver our promise in our next version to acknowledge more of the contribution of DRONE. To do so, here is our specific revision plan:

1. We will replace the inaccurate statement in Section 3.1 "which has been achieved by Drone on small model but unable to be applied for LLM." with "Compared with DRONE, SVD-LLM makes improvements in terms of memory, speed, and numerical stability."

2. We will include the proof that DRONE could achieve the same optimal compression loss as SVD-LLM for further acknowledgement in Section 3.1.

3. We will also include the analysis of memory usage of DRONE and SVD-LLM in Section 3.1 for a clearer illustration since memory is the most significant improvement of SVD-LLM over DRONE. We kindly remind the reviewer that due to page limit, the detailed comparison between DRONE and SVD-LLM is provided in Appendix A.10.

Second, we want to note that the total number of lines of Algorithm 1 in DRONE is 7. Even including the detailed computation of M, the total number of lines is less than 11. Hence, relatively speaking, changing 5 lines of Algorithm 1 is NOT a slight change.

Lastly, our discussions allow us to learn from each other’s perspective on novelty/contributions. We understand and fully respect the reviewer’s perspective. At the same time, as we showed in our previous comment on the memory requirement comparison between DRONE and SVD-LLM, we want to note that identifying the key bottleneck of existing methods is also important to pushing the state-of-the-arts forward.

Once again, we are thankful to the reviewer for taking the time and efforts in deeply engaging in the discussion as well as providing suggestions and advice on how our paper writing should be revised. We promise to release the code of SVD-LLM after the paper’s acceptance to better benefit the community.

Best wishes,

All authors of Submission 2301

We would like to thank the PC for making this comment visible. We also sincerely appreciate the SAC and all reviewers for the engagement during the discussion phase. In short, SVD-LLM is significantly superior to the work, Drone, that was mentioned by the SAC. Compared with Drone, SVD-LLM is also optimal with the same theoretical compression loss. The detailed proof is provided in the response to the reviewer 3PXg. Moreover, SVD-LLM has four key advantages. We explain them in detail below.

• Advantage#1: Only SVD-LLM can perform LLM compression; Drone cannot achieve this. Drone was designed to compress small models like BERT. It needs to cache all the input activations for the full dataset during compression. As such, it will cause out-of-memory to compress LLMs like LLaMA-7B even with 80GB GPU memory. In contrast, SVD-LLM incrementally updates its $XX^T$ matrix by adding the $xx^T$ of each new input x. As such, SVD-LLM eliminates the need to store the full activations, requiring only the storage of the $XX^T$ matrix, which is considerably smaller than even a single input activation. Due to this advantage, SVD-LLM is far more practical to compress LLMs of size 7B or larger compared to Drone.

• Advantage#2: SVD-LLM has better numerical stability, which leads to superior empirical compression loss. While SVD-LLM shares the same theoretical compression loss as Drone, Drone’s higher complexity—stemming from additional SVD operations and inverse calculations on large-scale matrices—makes it less numerically stable compared to SVD-LLM. This often results in higher empirical compression losses in practice. To illustrate this, we compare SVD-LLM and Drone in terms of the empirical compression losses for randomly generated matrices of various shapes. We also include the theoretical minimum value, represented by the rank-k Frobenius norm loss of $WX$. The results are summarized in the following table. As shown, we observe that SVD-LLM achieves lower empirical compression losses than Drone, underscoring its superior numerical stability. | Loss | [128 x128] x [128 x 128] | [2048 x2048] x [2048 x 2048] | [4096 x 4096] x [4096 x 4096] | |---------|--------------------------|------------------------------|-------------------------------| | Theoretical Minimum | 276.1130 | 17784.2637 | 50321.9141 | | Drone | 276.1130 | 17785.6992 | 50337.2148 | | SVD-LLM | 276.1130 | 17784.2676 | 50321.9727 |

• Advantage#3: SVD-LLM incurs much shorter compression time compared to Drone. Drone involves more complex matrix operations, leading to longer compression time compared to SVD-LLM. To illustrate this, we measured the time required by Drone and SVD-LLM to compress randomly generated weight and activation matrices of varying shapes under a 50% compression ratio. The results show that SVD-LLM is approximately three times faster than Drone. | Time | [128 x128] x [128 x 128] | [2048 x2048] x [2048 x 2048] | [4096 x 4096] x [4096 x 4096] | |---------|--------------------------|------------------------------|-------------------------------| | Drone | 0.07 seconds | 5.81 seconds | 30.35 seconds | | SVD-LLM | 0.02 seconds | 1.98 seconds | 10.37 seconds |

• Advantage#4: The theoretical optimal compression loss cannot always be obtained by Drone’s formula but can always be obtained by SVD-LLM. This is because Drone formulates the rank-k minimum compression loss as the rank $k$ truncation of $Z = S_{W,r}V^T_{W,r}V_{X,t}S_{X,t}$, where $r,t$ represents the rank of $W, X$, $S_{W,r}, S_{X,t}$ represents the diag matrix with non-zero singular values, $V_{W,r},V_{X,t}$ represents the valid corresponding singular vectors. However, if $k$ is larger than $r$, the rank-$k$ truncation of $Z$ will incur errors. Therefore, in this case, the theoretical optimal compression loss cannot be obtained by Drone. In contrast, SVD-LLM computes the rank-$k$ truncation of $WS$ with the same shape of $W$, which will not raise errors and hence is able to obtain the theoretical optimal compression loss.

In the revised sbumission, we have also discussed Drone in Section 2, mentioned the same theoretical minimum SVD-LLM achieves as Drone in Section 3.1 and provided the detailed illustration in Appendix A.10.

We sincerely thank you for the time and effort you have invested in helping us enhance this paper. We are willing to provide further clarification or additional revisions. Thank you once again for your insightful contributions to our work.

Bests,

Your friends,

Team of 2301