We thank the reviewer for taking the time to review our paper and provide valuable feedback. We are glad that the reviewer thinks the paper is reasonably well-written, easy to follow, and contains a systematic treatment of the SVD approach for LLMs. We are pleased to address the reviewer’s questions and concerns. The comments will make the paper stronger.

Question 1. The submission slightly overstates its novelty.

A1. We thank the reviewer for bringing the referenced literature and we have cited them in our revision. We respectfully suggest that the reviewer may have slightly overlooked the novelty related to the activation in our work.

1. Our Activation-Aware Framework for SVD Differs Fundamentally from Previous Activation-Aware Frameworks to other compression methods.

   Activation-aware and low-rank techniques have been extensively studied in quantization (e.g., LLM.int8(), SmoothQuant, AWQ, SqueezeLLM) and pruning (e.g., NetAdapt, WandA, adaptive activation-based structured pruning). These methods treat activation distance as an optimization goal and approach the learned truncation as a form of compression-aware regularization as noted by reviewer uXRu. This perspective is also employed in ASVD and SVD-LLM.

   However, we would like to emphasize that our method extends beyond the aforementioned framework: we are the first to directly truncate activations. Building on the foundational EYM theorem of SVD, we rigorously prove that directly truncating activations is optimal, surpassing all other approaches, including weight truncation and activation-aware methods, as detailed in Lines 195 to 206. This approach has not been explored in prior SVD-based methods or even across a range of non-SVD-based techniques. Consequently, we believe both the optimal direct-truncation solution and the accompanying theoretical explanation of its optimality make our work uniquely stand apart from previous activation-aware frameworks.

2. Why haven’t previous works directly truncated activations?

   Because they can not easily reconstruct low-rank weights from the truncated activations!

   In addition to the emphasizing that we are the first to prove that directly truncating activations is the optimal solution for minimizing ∣∣A−A′∣∣, we would like to highlight that reconstructing low-rank weights from truncated activations was previously impossible. In our Appendix A4.1 “Theoretical Support for Updating Weights Using IPCA” Line 937-958, we theoretically demonstrate how weights can be reconstructed from truncated activations. Furthermore goes beyond the theoretical solution, we also address practical challenges of footprint explosion issues through a novel use of IPCA. Additionally, the Appendix A.4, "Dobi-SVD's New Perspective 1: A Novel Path from Activation to Weight" and Fig. 4 provide a more intuitive understanding.

3. Dobi handles singular value information during the "learned truncation" process differently from the "coarse" approaches of previous methods. For instance, ASVD and SVD-LLM predefine singular value selection ratios (e.g., 0.1, 0.2, ..., 0.9) for each matrix and use a search algorithm to determine the best combination under such constraints. In contrast, as reviewer sX49 commented, we are the first work considering all singular values to attempt identifying the "optimal" truncation position. This is a natural idea, and others may have considered it, but no prior work implemented it. The reason is that they were unable to achieve the efficiency and robustness in backpropagation for general matrix SVD as Dobi did. Additionally, we provided experiments in our response to reviewer sX49's first question, showing the sensitivity of the truncation position to the results, further supporting the necessity of investigating the "optimal" position.

Based on the above discussion, we believe our work demonstrates greater innovation and significance than initially evaluated by the reviewer. We hope our explanations provide the reviewer with a more comprehensive understanding, highlighting that: (1) our approach is fundamentally distinct from previous SVD-based compression methods, and (2) we make both theoretical and engineering contributions to achieve and implement the optimal solution effectively.

Question 2. Fine-tuning over WikiText2 data and then measuring PPL over the same dataset is not fair.

A2. We politely point out the reviewer may have the misconception for our experiment setting.

1. We are not a fine-tuning method.

   Different from the fine-tuning method that optimizes relatively heavy weights with many samples, we adopt a standard approach similar to previous activation-aware compression methods like SqueezeLLM and SVD-LLM. Specifically, we collect only 256 activation samples to optimize the truncation rank and use these same samples to reconstruct the weights. In contrast to fine-tuning-based methods, our optimization is parameter efficient (just 224 parameters for LLAMA1 and LLAMA2) and sample-efficient (end-to-end only using 256 calibration samples)

2. Our setting is consistent with Baseline. Our experimental setup is consistent with the baselines we compare against. We followed SVD-LLM to randomly select 256 samples from WikiText-2 as the calibration data to do post-training. Our ppl evaluation code and setting are the same as SVD-LLM. Additionally, SVD-LLM uses fine-tuning and we do not.

3. We perform better even under unfair comparison. We did not report our actual experimental results because the SVD-LLM results we obtained were much worse than those reported in their paper. For Llama-7B, under the same experimental conditions, the actual results we obtained are as follows:

As shown in the table, the results for SVD-LLM are significantly worse than those reported in their paper. Despite our best efforts to reproduce their performance, we were unable to complete the "fine-tuning" phase of SVD-LLM due to issues documented in their GitHub repository (#6 and #7), specifically the error: "cannot compute the inverse of the scaling matrix." We hypothesize that this may stem from GPU compatibility or numerical precision issues. To avoid any perception of bias or accusations of intentionally worsening their results, we have opted to use the results reported in their arXiv paper directly. Note that our method is robust to these issues of compatibility and numerical stability. Additionally, we recently saw a new pruning method MoDeGPT [1] make comparisons in the same way. And Dobi-SVD still outperforms it: At a compression ratio of 0.6, MoDeGPT achieves 9.39, while Dobi-SVD achieves 8.12; at 0.8, MoDeGPT is 6.53, and Dobi-SVD is 6.08.

Besides, It is also noteworthy that even under this unfair comparison—where SVD-LLM uses LoRA (ICLR 2025 submitted version, L254)—our results still outperform theirs (both their arXiv version and the newly submitted ICLR version). This not only demonstrates the robustness and applicability of Dobi-SVD but also highlights the effectiveness and competitiveness of our method derived through optimal theoretical analysis.

We have made our experiment settings more clear in the appendix.

[1] MoDeGPT: Modular Decomposition for Large Language Model Compression (arXiv Table 6)

We hope this message finds you well and in good spirits. We would like to express our sincere gratitude for your invaluable contribution as a reviewer. Your detailed feedback and thoughtful suggestions have been immensely helpful. Over the past two weeks, we have incorporated a significant amount of experimental data and theoretical explanations, and we are pleased to inform you that we have thoroughly addressed all the concerns raised. As the rebuttal phase nears its end, we look forward to your feedback.

Below, we have provided a summary of the current version of our paper, which we hope will assist in your evaluation.

Thanks to the invaluable suggestions and support from the reviewers, we have refined Dobi-SVD, especially the two novel perspectives: 'A Novel Path from Activation to Weight' and 'Fully Unlocking SVD’s Potential for Data Compression by Addressing a Long-Overlooked Limitation.' We believe that the current version of Dobi-SVD is backed by solid theory, presents novel insights, and offers significant contributions, deserving of a higher score. Specifically, we highlight the following key points:

• Innovative Approach: Dobi-SVD does not follow any existing quantization, pruning, or SVD-based methods but instead introduces a new way of handling activations and weights based on the fundamental theory of SVD (the EYM theorem).
• Efficient and Robust SVD Backpropagation: We have developed an efficient and robust method for general matrix SVD backpropagation, which serves as a direct reference for others exploring low-rank, high-dimensional, and multi-form SVD matrices.
• Addressing a Long-Overlooked Yet Critical Limitation: By leveraging the Gaussian distribution of singular value vectors, we’ve remapped singular value information to storage space, overcoming a longstanding limitation overlooked by SVD-based methods and criticized by non-SVD approaches, thus unlocking SVD’s full compression potential.
• Comprehensive Comparisons: We have provided a broad comparison across various model compression methods, demonstrating superior practical performance.
  • Not only do we outperform existing SVD-based methods (General Response Part 4), but we also generally outperform a wide range of pruning methods(General Response Part 2).
  • In the comparison related to quantization (General Response Part 3), we demonstrate that our approach reduces FLOPs and highlight that more effective integration of SVD-based methods and quantization is a promising direction for future exploration.
• Generalizability: Dobi-SVD’s successful application to Vision-Language Models (VLMs) highlights its broad applicability, extending beyond LLMs to areas like robotics. To the best of our knowledge, this is the first demonstration of such generalizability.

Once again, we deeply appreciate the time and effort you have dedicated to reviewing our work, and we wish you a wonderful day ahead.

4. We think that using standardized accuracy alone to evaluate model effectiveness is inappropriate.

   We politely disagree with "In order to lead to significant speedups, your method drops significant accuracy." Actually, we believe the observed accuracy drop is relative to other SVD and pruning methods. As shown in Tables 2 and 3 of our manuscript, Dobi-SVD achieves better performance compared to existing SVD compression and pruning techniques. This result is both exciting and meaningful within the SVD model compression domain, effectively advancing the development of SVD-based model compression.

   While we aim to pursue lower compression losses in future work, we consider our current experimental results to be sufficiently robust and promising. Additionally, we do not agree with the claim that "standardized accuracy being zero on harder tasks implies the model is unusable." This indicates that these tasks are inherently challenging for this model, rather than suggesting our method makes it unusable. Furthermore, test sets like MathQA are inherently difficult for existing non-compressed LLMs, and pursuing usability on such benchmarks for model compression methods may be impractical. Instead, we believe that the drop relative to the original model is the appropriate measure of a compression method's effectiveness, as used by a wide range of works including ShearedLLAMA and SliceGPT that the reviewer mentioned. We believe the necessity of this metric itself is open to question.

Once again, we thank the reviewer for taking the time and effort to review our revision and sharing concerns. We hope that our clarifications and explanations can eliminate some misunderstandings. The two novel perspectives we proposed, 'A Novel Path from Activation to Weight' and 'Fully Unlocking SVD’s Potential for Data Compression by Addressing a Long-Overlooked Limitation', have already taken a great step in a promising direction, and as the reviewer anticipated, achieving more effective and low-cost model compression is indeed our next goal.

References:

[1]Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." Advances in Neural Information Processing Systems 36 (2024).

[2] Frantar, Elias, et al. "Marlin: Mixed-precision auto-regressive parallel inference on large language models." arXiv preprint arXiv:2408.11743 (2024).

[3] Han, Song, et al. "Retrospective: EIE: Efficient Inference Engine on Sparse and Compressed Neural Network." arXiv preprint arXiv:2306.09552 (2023).

[4] Xia, Mengzhou, Zexuan Zhong, and Danqi Chen. "Structured pruning learns compact and accurate models." arXiv preprint arXiv:2204.00408 (2022).

[5]Kwon, W., Kim, S., Mahoney, M. W., Hassoun, J., Keutzer, K., & Gholami, A. (2022). A fast post-training pruning framework for transformers. Advances in Neural Information Processing Systems, 35, 24101-24116.

We fully respect the reviewer’s authority to evaluate our work and greatly appreciate the reviewer’s willingness to detail the remaining concerns. However, regarding some of the concerns and shortcomings raised, we believe that certain points are not based on accurate facts. We would like to provide the following clarifications and explanations.

1. We believe the reviewer’s accusation of our quantization comparison being deceptive is unfounded.

   First, regarding the comparison with BNB, we provided PPL results in our revised paper (Table 20 and Table 21) and also in General Response（Part 3/5), contrary to the claim that "you provide FLOPS and throughputs, but not accuracies." Additionally, our results demonstrate that compared to BNB, we achieve higher throughputs at similar perplexity levels.

   Second, BNB’s assertion of being "essentially lossless" relies on the use of QLoRA [1] mapping, which requires LoRA fine-tuning to restore performance. Without employing LoRA, BNB shows an increase in perplexity of 6.3% and 26.2% on WikiText2 for 8-bit and 4-bit configurations, respectively, when applied to Llama2-7b (Table 21 in our paper).

   Furthermore, we disagree that "FLOP comparison doesn't make sense." We believe that FLOPs is a meaningful metric. Specifically, when the model inference batch size is large, the model transitions from being memory-bound to computation-bound, making the reduction of FLOPs significant. Regarding existing quantization methods, let’s remember that when the batch size is greater than one, the performance of current quantization kernels varies greatly across different hardware platforms. Even with small batch sizes, acceleration on certain hardware, such as some NVIDIA GPUs, falls far short of expectations. For example, with a batch size of 128, classic quantization kernels like AWQ, exllamav2, and BNB achieve less than 0.5× acceleration [2]. In contrast, our method benefits from lower FLOPs and hardware versatility, achieving 1.4× and 1.6× acceleration at compression ratios of 0.6 and 0.4, respectively. Therefore, as we mentioned in our previous response, our advantage lies in the fact that our method does not require selecting kernels based on specific hardware constraints and provides higher acceleration with large batch sizes.

2. We think comments regarding performance are unreasonable.

   First, non-quantization compression methods are not unuseful. In fact, current non-quantization methods at high compression ratios inevitably experience some performance loss (for example, SliceGPT and LLM-Pruner reduce performance on commonsense reasoning by 25.4% and 17.2%, respectively, when applied to the Llama2-7b model with a 20% pruning ratio). **However, this does not mean they are without value. On the contrary, the field of non-quantization LLM compression has been continuously developing and has attracted increasing attention from researchers, including many esteemed scholars $3$

$4$

$5$ .** Our statistics show that in 2022, 2023, and 2024, there were 37, 54, and 93 papers published in ICLR, NeurIPS, ICML, and CVPR with "model pruning" in their titles, respectively. This rapidly increasing trend indicates that non-quantization compression methods are gaining recognition and receiving more attention as the scale of LLMs grows. Therefore, we believe it is unreasonable for the reviewer to dismiss the practical significance of non-quantization compression methods solely due to performance loss. Such a viewpoint actually negates the substantial efforts of researchers in the field of non-quantization compression, which is clearly incorrect.

Second, Dobi-SVD has already surpassed existing pruning and SVD methods in performance. Although the reviewer considers an 8.7% accuracy reduction for DobiSVD at a compression ratio of 0.6 on Llama 3.1-8B to be unacceptable, we would like to point out that, without employing fine-tuning, this loss is already smaller than that of other SVD/pruning methods at the same compression ratio. For instance, other pruning methods, including SliceGPT, reduce accuracy by between 33.3% and 43.5% (as shown in Table 15). Furthermore, as demonstrated in Tables 2 and 3 of the main text and in subsequent comparisons on other models, DobiSVD and all SVD-based methods achieve the best current results compared to popular pruning methods across different models. Therefore, from an accuracy perspective, we believe that DobiSVD delivers reasonable and competitive results. Combining these points, we consider the performance of DobiSVD to be both instructive and practically significant.

Additionally, we would like to remind the reviewer that current methods claiming to achieve "lossless" compression rely on post-training techniques, such as LoRA fine-tuning, to recover the performance losses caused by compression. For example, ShearedLLAMA requires extensive training on 50 billion tokens after compressing LLaMA2-7B to Sheared-LLaMA-2.7B. These training processes consume substantial time and computational resources. Therefore, while we believe that combining our method with fine-tuning techniques like LoRA could further enhance performance, our ultimate goal is to achieve model compression at a low cost.

3. We believe there are misunderstandings regarding our experiments and statements about VLM.

   Firstly, our experiments on VLM did not claim to "improve performance" as our on-the-top assertion. In fact, our exact wording in line 157 of the paper is: "we find that our Dobi-SVD not only enhances efficiency but also improves VLM performance on the Pope-Random dataset," and in General Response (Part 3/5), we describe: "for example, even at a 0.4 compression rate, it achieved nearly zero loss on the Pope dataset," which are consistent with the experimental results.

   Secondly, our outstanding experimental performance is consistently stable and not the result of the reviewer uXRu’s statement “random variability in the evaluation”. During the VLM testing process, we evaluated all data within the datasets using the standard testing procedures of LLaVA-v1.5. Furthermore, to ensure the fairness of our experimental results, we conducted 10 tests on TextQA with a random seed using a 0.4 compression ratio on LLaVA-v1.5. The maximum performance variance observed was 0.2, which validates the fairness and reliability of our testing.

   Lastly, we would like to reiterate the significance of Dobi-SVD's applicability to VLM. The addition of version modules and alignment modules in VLM means that not all compression methods can be directly transferred and remain effective. Moreover, VLMs are typically larger in size compared to LLMs and require more compression, with broader applications such as in robotics. The applicability of Dobi-SVD to VLM demonstrates its extensive potential and versatile application scenarios in real-world settings.

   We would like to emphasize that we have dedicated significant time and effort to develop Dobi-SVD in order to address the practical needs we face. Although the VLM experiment was not conducted at the request of any reviewer, but rather as part of our own research goals.

Dear authors,

Thank you again for your response. Overall, I would like to note that the response simply re-iterates points that you have already made either in your initial response or in responses to other reviews.

From my perspective, there is a very simple experiment that would clarify things, which should also be doable within a week, if your claims of simplicity and scalability for your method hold:

• Let's take the Llama-2 family of models, with the 7B and 13B variants. Please compress the 13B variant to a level that would make it comparable to Llama-2-7B in terms of FLOPs or runtime (you get extra points for the latter metric, but the first is fine).

• Now, please compare the accuracies (PPL and standard zero-shots) between the two models. Is the Dobi-SVD 13B-shrunk-to-7B model better? Bonus question: how do the accuracies compare to 4bit quantization of the 13B model using standard techniques (BNB, AutoGPTQ, AutoAWQ)?

If the above comparison would show that your method is Pareto-competitive, then I will consider the point addressed and increase my score. Notice that this is a basic test of validity for the method, as it reflects how a user would choose which model to use.

(You may use other models for this comparison, such as Qwen, if more convenient.)

I will briefly comment on some of the responses, since they are either taking my statements out of context or based on misunderstandings by the authors:

the reviewer’s accusation of our quantization comparison being deceptive is unfounded.

All I was saying was that you conveniently omitted the baseline in the table you shared with me. Viewing results in context suggested that your method leads to catastrophic accuracy loss for those benchmarks, relative to the baseline.

comments regarding performance are unreasonable. First, non-quantization compression methods are not unuseful.

This is simply not what I said. Non-quantization methods can be useful, but -- as your very own results show -- they often seem require fine-tuning to recover non-random levels of accuracy on standard tasks. Your method claims to do this without fine-tuning: I just pointed out that the evals suggest this might not be the case. Please note that, as I said before, ShearedLlama also released a pruned checkpoint before fine-tuning. My suggestion was to compare with that one.

standardized accuracy alone to evaluate model effectiveness is inappropriate.

Again, I did not suggest using standardized accuracy alone. However, looking at standardized accuracy suggests that the models you produce have a lot of accuracy drop, to the point where they are producing random responses. Since we do not have the option of interacting with the model or conducting user testing, we have to use such tests to validate the models.

Finally, a note on standards, since the authors seem to be questioning mine. While the submitted paper has value (reflected in my increased evaluation of 6), it is clearly not the first in the area of SVD compression. Therefore, the standards for it are inherently higher than for earlier work. In my view, it is not sufficient for the current work to be simply slightly improving over prior work on SVD in terms of abstract metrics--otherwise, we would just get an endless stream of incremental papers. To deserve publication into a top-tier venue such as ICLR, the work has to do so significantly, in the sense that it would make SVD a viable competitor to e.g. quantization. (The authors seem to agree on this as well, since they chose this as a comparison point.) The standard I set is therefore Pareto-competitiveness: the method should be competitive with other approaches leading to the same level of speedup. If the authors can meet this, then their work should be published. Otherwise, they may try to improve improve their results and try again.

Best regards!

We appreciate reviewer uXRu's experimental suggestions and additional explanations, which included workloads described as "which should also be doable within a week." However, these suggestions were provided on the final day of the review process, rather than at the beginning. Despite as stated in the review policy email sent by ICLR PC to reviewers on 11/26, “No requests for new experiments are allowed”, we tried our best to accommodate the reviewer's request by conducting the proposed experiments. The results are presented in the table below.

Experiments show that it is evident that whether comparing within the LLaMA series or against other model series, models compressed with Dobi-SVD show higher accuracy with competitive or even higher throughput. This suggests the practical value for real applications.

Some facts that the reviewer could easily verify:

1. Table 2: When the ratio is 0.8, Dobi-SVD outperforms the previous best method, SVD-LLM, by reducing PPL by 23.4%. At a ratio of 0.6, it reduces PPL by 38.1%, and at 0.4, it reduces PPL by 81.5%. Note that Dobi-SVD does not use fine-tuning, while SVD-LLM does.
2. General Response (Part 4/5): On the Llama3-8B 0.8 MMLU, the original model achieved 63.3, LLRC achieved 24.8, while Dobi-SVD achieved 60.1.
3. Among all quantization, pruning, and SVD-based methods, Dobi-SVD is the first to demonstrate compression for both LLM and VLM, with performance that is noteworthy. For example, even at a compression rate of 0.4, it achieved nearly zero loss on the Pope dataset.
4. For the past 34 years, since 1990 [1], Dobi-SVD is the first work to provide an effective solution to the long-overlooked limitation of SVD for compression.

[1] Demmel, J. and Kahan, W. (1990). Accurate Singular Values of Bidiagonal Matrices. SIAM J. Sci. Statist. Comput., 11 (5), 873-912.

We would like to thank the reviewer for taking the time to review our paper and provide valuable feedback. We are glad that the reviewer liked the writing style of this paper and thought it presented a good approach to model compression. We are excited to have the chance to address the reviewer’s questions and concerns. These edits will make the paper stronger.

We respectfully point out that reviewer otMy has significantly underestimated the novelty and contributions of our work. Contrary to the summary, “I think the core idea in this paper is gained from the way to let the 0-1 function (activation function) become a sigmoid function to provide a gradient for the neural network. They also build a multi-objective function to balance two targets,” our work goes far beyond these summarized points.

Specifically, our contributions can be summarized as follows:

1. Proving the optimality of directly truncating activations.
2. Developing an end-to-end differentiable approach to determine the optimal truncation positions.
3. Reconstructing weights from the truncated activations.
4. Addressing the limitations of truncation boundaries through quantization.

We would like to emphasize that this represents an entirely new paradigm. The uniqueness of our paradigm lies at four aspects:

1. We are the first to establish the theoretical optimality of directly truncating activations, which fundamentally differs from previous activation-aware methods that merely treat activation distances as an optimization target.

2. We are the first to enable an end-to-end differentiable approach for determining the optimal truncation position, significantly reducing the engineering complexity associated with SVD-based compression methods

3. While direct activation truncation is theoretically optimal, reconstructing weights from truncated activations is challenging. We are the first to demonstrate how to effectively and efficiently achieve this reconstruction.

4. We introduce a novel application of quantization to overcome the under-mapping issue of truncation rank and memory storage constraints.

From the high-level conceptual paradigm to the low-level technical implementation, we believe our work makes significant contributions that go beyond the points highlighted by reviewer otMy.

We then answer the remaining concerns from the reviewer otMy below:

1. Gradient explosion with Taylor expansion.

   We kindly disagree that the method is commonly used to solve gradient explosion. Indeed, the gradient explosion issue has been widely explored in previous literature. However, using Taylor expansion to address the gradient explosion is not common, especially in the context of SVD. As far as we know, there is only one work [1] utilizing the Taylor expansion to resolve this issue. We have discussed and cited this work in the revised version. While their focus was on symmetric matrices and computer vision tasks, we made a novel use of Taylor expansion for general matrices and LLM compression. Unlike [1], which dealt with low-dimensional image features, our work addresses the challenges of calculating large-dimensional matrices in LLMs. To meet these demands of large matrices in LLM, we developed a more efficient, parallelized algorithm. The algorithm is detailed in the appendix A.6 “Dobi-SVD's Robust and Efficient Differentiable SVD Algorithm for General Matrices”. We will release the full code, and we believe our engineering contributions in this direction can benefit a broader community interested in SVD-related optimization techniques.

2. More illustration of overcoming truncation limitations. We are pleased to see that reviewer otMy understands our motivation for this part, as reflected in the statement:“Low-rank methods require truncating half of the singular values, and the model degradation is rooted in this type of approach.” This perspective can be viewed from another angle: when aiming for a compression ratio of $<1$ while keeping $k \in [0, \min(m, n))$, it is evident that the storage space required is $k \cdot \max(m, n)$. However, the practical memory footprint is $k \cdot (m + n)$. To bridge the gap and remap the storage from $k \cdot (m + n)$ to $k \cdot \max(m, n)$, we make a novel use of the Gaussian distribution properties of the $U$ and $V$ matrices and implement a quantization method to reduce the footprint requirements.

3. Typo (about first question). We sincerely appreciate your derivation and verification. It is a typo error in our notation, also noted by the reviewer sX49 and bQbW. We have revised it in our new version.

Reference: [1] Robust Differentiable SVD (TPAMI 2021)

We greatly appreciate the reviewer pointing out this issue. We sincerely apologize for not recognizing this problem during the initial rebuttal.

In the latest revised version of the paper, we analyze and demonstrate the superiority of directly truncating activations over truncated weights from both a module-level and model-level perspective. We highlight that our conclusion of the superiority of directly truncating activations over truncated weights still holds!

• At the module level, the Eckart-Young-Mirsky Theorem guarantees that the directly truncating activation gives $A_k$, the optimal $k$-rank approximation of activation.

• At the model level, we model the training loss and account for the reviewer's suggestion to fully consider the impact of θ. To achieve this, we directly compare the complete formulas:

  $\Delta \mathcal{L}_W = \frac{\partial \mathcal{L}}{\partial A} (A - xW_k) \quad \text{and} \quad \Delta \mathcal{L}_A = \frac{\partial \mathcal{L}}{\partial A} (A - A_k).$

  • Experimental setup: We truncate weights and activations with k = {100, 500, 1500, 2000, 2500, 3000} across layers {5, 10, 15, 20, 25} on LLaMA2-7B and evaluate the model's performance loss on 256 samples from WikiText2.
  • Results: The results show that across different layers and k-values, the performance loss caused by $A_k$ is consistently smaller than that caused by $W_k$.

By analyzing and validating from both module and model perspectives, we demonstrate the superiority of directly truncating activations. Notably, this is a more comprehensive comparison, as previous works only analyzed the effects of processing activations at the module level (e.g., xW==A), without considering the model level. Due to space limitations, the detailed analysis and experimental results are presented in Appendix A.10.

Previously, this issue had not been explored in SVD-based methods. We are very grateful to the reviewer for raising it, which allowed us to improve our paper and make it more rigorous. This forms one of the foundations of Dobi-SVD and combined with various theoretical analyses, we propose two novel and interesting perspectives: A Novel Path from Activation to Weight and Fully Unlocking SVD’s Potential for Data Compression by Addressing a Long-Overlooked Limitation. If the reviewer has further questions about other theoretical analyses, we warmly welcome the feedback.

First of all, over-packaging or over-claiming is the most important factor that causes confusion in the part 3.3, i.e., part of quantizing UV matrix. I need to spend 4-6 hours in all to figure out what is a LONG-OVERLOOKED TRUNCATION LIMITATION problem, because in my opinion, rank < mn/(m+n) is a precondition that should be taken for granted in low-rank decomposition, especially in real scenarios. This will definitely be considered. When reading the article, the introduction of quantization operation in this paragraph is not connected.

So I suggest that the author just say that the UV matrix is ​​quantized in title in this part. The speed is improved by quantization. Forcibly introducing concepts such as remapping and bijection will cause confusion to readers, so this paragraph is over-packed. As I said, the title of this paragraph can directly say that the performance is further improved by combining quantization technology.

Further, the problem whether the accuracy loss caused by quantizing UV matrix can be compensated by the increase of rank, what is the upper bound can be further discussed and are innovative. Because in industrial scenarios, we need the trade off the rank, model accuracy, model speed and quantization degree. This is a topic worth discussing. Most of the current quantization work is based on direct quantization weights rather than UV matrices.

This topic is not expanded in this paper, and it is difficult for me to see these contents in the end-to-end experiments. I also think this point will be a very innovating and be worth 6-9 pages paper. This point is more innovative than learning the truncation position. But the end-to-end experiments cover up all these contents.

Therefore, this paper actually hides a content that has not been fully discussed, so I tend to reject it. Although a large number of end-to-end experiments cannot fully show the content that needs to be discussed, some traces can be seen. So here I will provide my opinions to AC, who will judge whether these traces can fully prove the effectiveness of this method.

As for EYM, ok, in truncation position indeed need this to keep optimally. So, this paper just delete the claim that "theoretically analyze the optimality of truncating weights and truncating activations" and the paper is ok. There is no theoretical proof in the paper that quantized activation is optimal. The paper still uses experimental methods to prove it, that is, eq10, eq11 and subsequent analysis in Appendix a10. Even eq10/11 here is still redundant, because the inner product of the vector is not the product of the inner product modulus, and the content in the appendix also says that this part is proved by experiment (line 1549-1556).

I think at least you should tell me the table of quantization bits of UV, rank of UV and model performance. We want to see if the rank calculated by the algorithm is exactly equal to the rank that just covers the quantization loss and the memory size of quantized UV is smaller with more rank.

But just the table of quantization bits of UV, rank of UV and model performance in different model and how to determine the rank of UV in quantization manner is another valuable topic. This paper analysis just show me analysis how to low rank in non-quantization process. Current paper I think mixing them together.

If the rank calculated by the algorithm is exactly equal to the rank that just covers the quantization loss, I think this paper would be important because they show that quantization and non-quantization analysis are the same.

Thirdly, in response to our rebuttal, the reviewer has acknowledged the importance of the EYM theorem and further elaborated on the concerns, which, however, stem from a misunderstanding.

We respectfully note that the reviewer’s comments are still not based on the accurate facts presented in the paper.

In our revision, we addressed the advantages of directly truncating activations at two distinct levels:

1. Modular Level: This is a level commonly used in the compression field, where most papers analyzing weights and activations are situated. At this level, we applied the EYM theorem to demonstrate that, compared to truncating weights or using other activation-aware methods, directly truncating activations is indeed optimal. It is important to highlight that we provide a rigorous theoretical analysis in this section.
2. Model Level: This level, which is less frequently addressed in existing literature, involves a theoretical model and comparative formula, supported by experimental validation. It is worth noting that we modified our formulas according to the reviewer’s earlier suggestions. We are uncertain why this has been perceived as "redundant," nor do we fully understand how the statement "the inner product of the vector is not the product of the inner product modulus" led to this conclusion of redundancy.

Regarding “quantized activation,” we would like to clarify that our pipeline does not involve the quantization of activations. If the reviewer is referring to the quantization of the left and right singular value vectors and their effect on the optimal truncation position, we have already provided an explanation in the above section.

Finally, in relation to the abstract's statement "theoretically analyze the optimality of truncating weights and truncating activations," we plan to revise this in the new version to: “theoretically and experimentally analyze the optimality of directly truncating activations”, as we believe this phrasing more accurately represents the work we have presented.

Reference:

[1] Demmel, J. and Kahan, W. (1990). Accurate Singular Values of Bidiagonal Matrices. SIAM J. Sci. Statist. Comput., 11 (5), 873-912.

[2] Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." Advances in Neural Information Processing Systems 36 (2024).

We would like to thank the reviewer for taking the time to review our paper and provide valuable feedback. We are glad that the reviewer thinks our paper is generally well written and easy to understand, and the idea of determining the optimal truncation position for each weight matrix is novel. We are excited to have the chance to address the reviewer’s questions and concerns. These edits will make the paper stronger.

Concern 1. Could you please add the result of pure GPTQ in Table 5?

A1. We provide the result of pure GPTQ and the results of GPTQ + Dobi-SVD. Besides, we also provide the results of one of the most popular quantization methods BnB and the results of the combination with Dobi-SVD .

We politely disagree that Dobi-SVD degrades the performance when it is combined with the quantization methods. We emphasize that Dobi-SVD offers an alternative approach to overcoming the limitations of quantization. As demonstrated in the GPTQ table, reducing quantization from 4-bit to 3-bit is challenging and often results in significant performance degradation. However, Dobi-SVD provides a viable alternative, achieving a similar memory footprint while delivering better perplexity (PPL), as shown in the first and last rows of the table. Besides, compared to the BnB method, our approach not only reduces memory requirements but also improves performance. These results highlight the potential of combining Dobi-SVD with quantization as a promising direction to break through the inherent limitations of quantizations.

By further increasing the compression ratio, Dobi-SVD can achieve near 2-bit memory levels while still maintaining acceptable performance. This demonstrates the significant potential of SVD-based methods to enhance the efficiency of LLM.

We would also like to emphasize that, compared to quantization methods, Dobi-SVD is a more generalizable compression technique that is adaptable to a variety of hardware platforms. In response to Question 9 from reviewer uXRu, we demonstrate how our method improves latency across general hardware setups.

Concern 2. Some typos in the equation (1), (2), (3)

A2. Thank you for pointing that out. These have been corrected in Line 1540~1548.

We hope this message finds you well and in good spirits. We would like to express our sincere gratitude for your invaluable contribution as a reviewer. Your detailed feedback and thoughtful suggestions have been immensely helpful. Over the past two weeks, we have incorporated a significant amount of experimental data and theoretical explanations, and we are pleased to inform you that we have thoroughly addressed all the concerns raised. As the rebuttal phase nears its end, we look forward to your feedback.

Below, we have provided a summary of the current version of our paper, which we hope will assist in your evaluation.

Thanks to the invaluable suggestions and support from the reviewers, we have refined Dobi-SVD, especially the two novel perspectives: 'A Novel Path from Activation to Weight' and 'Fully Unlocking SVD’s Potential for Data Compression by Addressing a Long-Overlooked Limitation.' We believe that the current version of Dobi-SVD is backed by solid theory, presents novel insights, and offers significant contributions, deserving of a higher score. Specifically, we highlight the following key points:

• Innovative Approach: Dobi-SVD does not follow any existing quantization, pruning, or SVD-based methods but instead introduces a new way of handling activations and weights based on the fundamental theory of SVD (the EYM theorem).
• Efficient and Robust SVD Backpropagation: We have developed an efficient and robust method for general matrix SVD backpropagation, which serves as a direct reference for others exploring low-rank, high-dimensional, and multi-form SVD matrices.
• Addressing a Long-Overlooked Yet Critical Limitation: By leveraging the Gaussian distribution of singular value vectors, we’ve remapped singular value information to storage space, overcoming a longstanding limitation overlooked by SVD-based methods and criticized by non-SVD approaches, thus unlocking SVD’s full compression potential.
• Comprehensive Comparisons: We have provided a broad comparison across various model compression methods, demonstrating superior practical performance.
  • Not only do we outperform existing SVD-based methods (General Response Part 4), but we also generally outperform a wide range of pruning methods(General Response Part 2).
  • In the comparison related to quantization (General Response Part 3), we demonstrate that our approach reduces FLOPs and highlight that more effective integration of SVD-based methods and quantization is a promising direction for future exploration.
• Generalizability: Dobi-SVD’s successful application to Vision-Language Models (VLMs) highlights its broad applicability, extending beyond LLMs to areas like robotics. To the best of our knowledge, this is the first demonstration of such generalizability.

Once again, we deeply appreciate the time and effort you have dedicated to reviewing our work, and we wish you a wonderful day ahead.

We would like to thank the reviewer for taking the time to review our paper and provide valuable feedback. We are glad that the reviewer thinks our paper thoroughly studied the impact of each of these three parts and compares favorably with existing LLM compressed SVD methods. We are excited to have the chance to address the reviewer’s questions and concerns. These edits will make the paper stronger.

Concern 1. The term "differentiable optimization" should be "gradient-based learning".

A1. Thank you for your feedback! While we understand the term "differentiable optimization" may sometimes cause confusion, it aligns well with the nature of our work.

Differentiable optimization refers to embedding optimization problems as differentiable components within a larger computation graph, enabling end-to-end training through automatic differentiation. This is well-documented in works like [1], [2], and the tutorial [3]. Our work, which dynamically optimizes the number of singular values in SVD within a learning pipeline, fits this framework. Unlike gradient-based learning, this approach treats optimization as an integral module rather than solely focusing on parameter updates. We hope this clarifies our motivation, and we’d be happy to discuss further if needed!

[1] OptNet: Differentiable Optimization as a Layer in Neural Networks， Amos & Kolter, 2017 [Citations: 1089]

[2] Differentiable Convex Optimization Layers，Agrawal et al., 2019 [Citations: 709]

[3] IJCAI 2022 Tutorial: Differentiable Optimization: Integrating Structural Information into Training Pipeline

Question 1. Is memory footprint a common concern in SVD compression for LLMs?

A2. We appreciate the reviewer for raising this issue. We believe that memory footprint is a common concern in SVD compression for LLMs, for the following reasons:

1. SVD Decomposition Requires Significant Memory. The SVD decomposition process requires a substantial amount of memory. Currently, in Python, SVD decomposition is only supported for 32-bit precision matrices, and memory footprint requirements grow exponentially with matrix dimensions. Therefore, performing SVD on larger matrices often requires a large amount of memory. For example, decomposing a 4096×11008 matrix and backpropagating it requires approximately 10GB of memory. Since SVD compression for LLMs involves numerous SVD operations, memory footprint is a common concern. This is also why we proposed using IPCA to address the issue. With IPCA, SVD decomposition for LLMs can be performed on low-performance GPUs (such as the NVIDIA Titan Xp 12GB).

2. Reducing Memory Footprint as an Important Goal of LLM Compression. Reducing memory footprint is itself an important objective of LLM compression. The large memory required for LLM inference limits its use on resource-constrained devices. One of the key goals of model compression is to reduce memory footprint during LLM inference to address potential memory-bound issues. Therefore, we emphasize in our results that Dobi-SVD significantly reduces the memory footprint of the compressed model during inference, enabling better deployment on resource-limited devices.

We hope this message finds you well and in good spirits. We would like to express our sincere gratitude for your invaluable contribution as a reviewer. Your detailed feedback and thoughtful suggestions have been immensely helpful. Over the past two weeks, we have incorporated a significant amount of experimental data and theoretical explanations, and we are pleased to inform you that we have thoroughly addressed all the concerns raised. As the rebuttal phase nears its end, we look forward to your feedback.

Below, we have provided a summary of the current version of our paper, which we hope will assist in your evaluation.

Thanks to the invaluable suggestions and support from the reviewers, we have refined Dobi-SVD, especially the two novel perspectives: 'A Novel Path from Activation to Weight' and 'Fully Unlocking SVD’s Potential for Data Compression by Addressing a Long-Overlooked Limitation.' We believe that the current version of Dobi-SVD is backed by solid theory, presents novel insights, and offers significant contributions, deserving of a higher score. Specifically, we highlight the following key points:

• Innovative Approach: Dobi-SVD does not follow any existing quantization, pruning, or SVD-based methods but instead introduces a new way of handling activations and weights based on the fundamental theory of SVD (the EYM theorem).
• Efficient and Robust SVD Backpropagation: We have developed an efficient and robust method for general matrix SVD backpropagation, which serves as a direct reference for others exploring low-rank, high-dimensional, and multi-form SVD matrices.
• Addressing a Long-Overlooked Yet Critical Limitation: By leveraging the Gaussian distribution of singular value vectors, we’ve remapped singular value information to storage space, overcoming a longstanding limitation overlooked by SVD-based methods and criticized by non-SVD approaches, thus unlocking SVD’s full compression potential.
• Comprehensive Comparisons: We have provided a broad comparison across various model compression methods, demonstrating superior practical performance.
  • Not only do we outperform existing SVD-based methods (General Response Part 4), but we also generally outperform a wide range of pruning methods(General Response Part 2).
  • In the comparison related to quantization (General Response Part 3), we demonstrate that our approach reduces FLOPs and highlight that more effective integration of SVD-based methods and quantization is a promising direction for future exploration.
• Generalizability: Dobi-SVD’s successful application to Vision-Language Models (VLMs) highlights its broad applicability, extending beyond LLMs to areas like robotics. To the best of our knowledge, this is the first demonstration of such generalizability.

Once again, we deeply appreciate the time and effort you have dedicated to reviewing our work, and we wish you a wonderful day ahead.

We sincerely thank the reviewer for taking the time to review our paper and provide valuable feedback. We are glad that the reviewer thinks our paper is computationally efficient and addresses the most common shortcomings of SVD-based methods, while proposing a stable back-propagation technique. We are pleased to address the questions and concerns and we have revised the manuscript accordingly.

Major Concern 1. Significant overlap with existing literature.

A1. We appreciate reviewer sX49 for providing the referenced literature. However, we respectfully disagree with the assessment that our work has significant overlap with it. Below, we outline the fundamental differences with each cited work and have incorporated this discussion into the revision in Appendix A.1.

• Regarding paper [1] LQER (Zhang, Cheng, et al.), it addresses quantization error using SVD along with a scaling matrix to adjust the activation-based distribution. While ASVD and SVD-LLM also incorporate a scaling matrix strategy, these methods involve a matrix inversion operation, which can result in unstable training due to the presence of outlier matrices. In contrast, Dobi-SVD avoids the matrix inversion by establishing our method on a rigorous EYM theorem coupled with a novel differentiable optimization strategy and an IPCA reconstruction. We provide a more detailed discussion of this issue and the novelty of our strategy in our response to reviewer uXRu’s first question.
• Regarding paper [2] the matrix compression method LPLR (Saha, Rajarshi, et al.), we point out the key differences and novel contributions of our work lie in both the motivation and the methodologies. LPLR is designed to optimize matrix quantization by refining the projection process and quantizing the projected matrix, effectively reducing errors in low-rank, low-precision representations. In contrast, Dobi-SVD addresses the mismatch between rank values and storage sizes in SVD-based compression, focusing on mitigating information loss caused by this mismatch. Methodologically, LPLR introduces an approach using a randomized rangefinder to approximate SVD, while Dobi-SVD makes a novel use of the normal distribution of positive-definite matrix column vectors to independently quantize projection spaces, thereby avoiding projection bias.
• Regarding paper [3] LRQ (Lee, Jung Hyun, et al.), they use quantization to compress models by leveraging low-rank weight scaling to increase training sample size and reduce training parameters. In contrast, Dobi-SVD takes a completely different approach: using SVD to compress models and addressing theoretical bottlenecks in SVD compression by exploring the quantization-friendly nature of orthogonal matrix column vectors.
• Regarding paper [4] the model compression method CALDERA (Saha, Rajarshi, et al.), our theoretical analysis and the construction of new weight matrices are fundamentally different. CALDERA represents the weight matrix as $W$\approx$Q + LR$ using LPLR+LoRA, where the new matrix dimensions exceed the original. Consequently, they must fully quantize $Q$, $L$, and $R$ to achieve the compression goal. In contrast, Dobi-SVD's optimal theoretical analysis, distinct from LPLR (as shown in our comparison with paper [2]), naturally achieves excellent results without requiring fine-tuning methods like LoRA. Additionally, quantization is not a core issue for Dobi-SVD. Even without remapping, we achieve compression (see Table 2 in Dobi-SVD) since our new matrices have reduced dimensions.

[1] Zhang, Cheng, et al. "LQER: Low-Rank Quantization Error Reconstruction for LLMs." arXiv preprint arXiv:2402.02446 (2024).

[2] Saha, Rajarshi, Varun Srivastava, and Mert Pilanci. "Matrix compression via randomized low rank and low precision factorization." Advances in Neural Information Processing Systems 36 (2023).

[3] Lee, Jung Hyun, et al. "LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices." arXiv preprint arXiv:2407.11534 (2024).

[4] Saha, Rajarshi, et al. "Compressing Large Language Models using Low Rank and Low Precision Decomposition." arXiv preprint arXiv:2405.18886 (2024).

(Continuing the response to Major 1)

Secondly, we appreciate that the reviewer raised an interesting question: Whether one requires a stricter and more rigorous approach to the selection of the truncation position? Our answer is YES, we absolutely do need a rigorous approach to select it! In combination with the reviewer’s first question, How sensitive is the task performance to the choice of $k$?, we designed an experiment to validate this necessity.

We first obtained the optimal truncation position for Llama-2-7b at 0.4 using Dobi-SVD and randomly selected 10 layers with the following code:

```python

import random

random.seed(5809) # 5809 is our submission number

print(random.sample(range(32), 10))

# Output: [6, 10, 27, 11, 17, 0, 13, 20, 7, 2]

```

While keeping the total k constant, we slightly adjusted k for these 10 layers: adding x to the first five layers and subtracting x from the last five layers, where x took values from [1,5,10,50]. The corresponding adjustment percentages (x/4096) were 0.024%, 0.122%, 0.244%, and 1.221% .

Conclusion: even with fine-grained rank adjustments (0.024% to 1.221%), performance drops significantly, worsening exponentially with larger ratios. These experiment results demonstrate that it is very necessary to obtain the optimal truncation position.

It is noteworthy to mention that previous search-based methods have a coarse minimum adjustment range of 10%, causing substantial performance loss so they require additional fine-tuning to recover the performance. Instead, Dobi-SVD, built on our optimal theoretical analysis, adjusts ranks at 0.024% granularity and operates in an end-to-end manner without additional fine-tuning.

We sincerely appreciate the reviewer’s kind recognition of our recent efforts in incorporating extensive experiments and providing deeper theoretical explanations. We are pleased that these additions have successfully addressed the concerns raised, and we are grateful for the constructive feedback and the increased scores.

We fully agree with the reviewer’s insightful comment that “although it certainly is helpful to compare methods across the board to make an informed choice when selecting a compression method.” We greatly value this perspective, as it aligns with our efforts to provide a comprehensive comparison of different methods. Within the limited rebuttal period, in addition to comparing the previous works and quantization-related comparisons mentioned and suggested by the reviewers, we proactively searched for recently published works, including MoDeGPT [1], LLM Pruner[2], SliceGPT[3], Bonsai[4], Wanda-SP[5] and FLAP[6] . Not only do we outperform existing SVD-based methods, but we also generally outperform a wide range of pruning methods:

1. When comparing compression performance on the PPL of LLaMA-7b with Wiki-text2, Dobi-SVD outperforms MoDeGPT. At a compression ratio of 0.6, MoDeGPT achieves 9.39, while Dobi-SVD achieves 8.12; at 0.8, MoDeGPT is 6.53, and Dobi-SVD is 6.08.
2. Across different tasks (various commonsense reasoning and language modeling datasets) and models (Llama3.1-8B, Llama2-7B, Llama-13B, Llama2-13B), Dobi-SVD consistently outperforms pruning methods such as LLM Pruner, SliceGPT, Bonsai, Wanda-SP, and FLAP at the same compression ratios (General Response Part 1/5 and 2/5).
3. Unfair Comparison to Showcase High Compression Advantage: To emphasize Dobi-SVD's superiority at higher compression ratios, we even performed an "unfair" comparison: when compressing to a ratio of 0.4, Dobi-SVD outperforms LLM Pruner, SliceGPT, Bonsai, and Wanda-SP at 0.5 and 0.6 ratios across various commonsense reasoning datasets (General Response Part 2/5, second table).

We are delighted that the reviewer acknowledges the potential of “effectively combining the two approaches to uniformly outperform other methods in both memory consumption and latency improvements.” This observation reinforces the importance of future work in this direction, and we are eager to explore it further. While quantization is a powerful tool, it is not a one-size-fits-all solution and is bottlenecked by lower-bit quantization now, and moreover, cannot be simply nested. The experiment results of combining Dobi-SVD and quantization encourages us to break through the limitation and we believe their combination is the next critical area for exploration.

Thanks to the invaluable suggestions and support from the reviewers, we have refined Dobi-SVD, especially the two novel perspectives: 'A Novel Path from Activation to Weight' and 'Fully Unlocking SVD’s Potential for Data Compression by Addressing a Long-Overlooked Limitation.' We believe that the current version of Dobi-SVD is backed by solid theory, presents novel insights, and offers significant contributions, deserving of a higher score. Specifically, we highlight the following key points:

• Innovative Approach: Dobi-SVD does not follow any existing quantization, pruning, or SVD-based methods but instead introduces a new way of handling activations and weights based on the fundamental theory of SVD (the EYM theorem).
• Efficient and Robust SVD Backpropagation: We have developed an efficient and robust method for general matrix SVD backpropagation, which serves as a direct reference for others exploring low-rank, high-dimensional, and multi-form SVD matrices.
• Addressing a Long-Overlooked Yet Critical Limitation: By leveraging the Gaussian distribution of singular value vectors, we’ve remapped singular value information to storage space, overcoming a longstanding limitation overlooked by SVD-based methods and criticized by non-SVD approaches, thus unlocking SVD’s full compression potential.
• Comprehensive Comparisons: We have provided a broad comparison across various model compression methods, demonstrating superior practical performance.
• Generalizability: Dobi-SVD’s successful application to Vision-Language Models (VLMs) highlights its broad applicability, extending beyond LLMs to areas like robotics. We believe this is the first demonstration of such generazability.

[1] MoDeGPT: Modular Decomposition for Large Language Model Compression. arXiv:2408

[2]Llm-pruner: On the structural pruning of large language models. NeurIPS 2023

[3]Slicegpt: Compress large language models by deleting rows and columns. arXiv:2401.

[4]Everybody prune now: Structured pruning of llms with only forward passes. arXiv:2402.

[5]Fluctuation-based adaptive structured pruning for large language models. AAAI 2024

[6]Fluctuation-based adaptive structured pruning for large language models. AAAI 2024

To conclude, we summarize our revisions as follows:

1. Typos: Fixed formula errors and expressions.

2. Added the necessary citations and relevant discussions.

3. Experimental Additions:

   1. Detailed experimental settings and experimental details.

   2. Evaluations on more tasks (MMLU, versus popular pruning methods) on more models:

      1. Llama2-7B (Table 12, Table 13, Table 14).
      2. Llama3-1.8B (Table 11, Table 13, Table 15).

   3. Evaluations on larger models:

      1. Llama-13B (Table 16, Table 18).
      2. Llama2-13B (Table 17, Table 19).

   4. Quantization:

      1. Combined with quantization (Table 20).
      2. Direct comparisons with quantization (Table 21).

   5. Vision-language model (VLM) compression: LLaVA-v1.5-7B (Table 22, Table 23).

4. Theoretical and Analytical Additions: Truncation Sensitivity Analysis (Table 10)

5. Algorithm Additions:

   • Taylor expansion to address gradient explosion in SVD backpropagation: A.6 “Dobi-SVD's Robust and Efficient Differentiable SVD Algorithm for General Matrices” and Algorithms 4 and 5.
   • Remapping-based precise storage and weight reconstruction: A.5 “Dobi-SVD's New Perspective 2: Fully Unlocking SVD's Potential for Data Compression by Addressing a Long-Overlooked Limitation” and Algorithm 3.

6. To further validate the generalizability of our approach, we added a description and experiments on Dobi-SVD in VLM compression:

   • Abstract: Lines 33–38.
   • Introduction: Lines 155–161.
   • A.8 “EXPERIMENTAL RESULTS ON VLM”, including Table 22 and Table 23.

7. On the topic of “Some New Perspective”:

   • Introduction: Lines 107–111.
   • Section 3.4 Conclusion: “Two new perspectives of Dobi-SVD” (Lines 366–377).
   • A.4 “Dobi-SVD's New Perspective 1: A Novel Path from Activation to Weight”
     • A.4.1 “Theoretical Support for Updating Weights Using IPCA.”
   • A.5 “Dobi-SVD's New Perspective 2: Fully Unlocking SVD's Potential for Data Compression by Addressing a Long-Overlooked Limitation.”

8. We added a conceptual figure (Figure 1, Lines 162), which visually illustrates the relationship between activation and weight in SVD-based methods and explains the theoretical basis of the "cosmic wormhole" concept for updating weights and IPCA’s practical implementation.

9. More comprehensive and accurate discussion on directly truncating activation and weight: A.10 "Analysis of directly truncating activations over weights"

Reference:

comparing with various pruning-based methods:

[1]Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Llm-pruner: On the structural pruning of large language models." Advances in neural information processing systems 36 (2023): 21702-21720.

[2]Ashkboos, Saleh, et al. "Slicegpt: Compress large language models by deleting rows and columns." arXiv preprint arXiv:2401.15024 (2024).

[3]Dery, Lucio, et al. "Everybody prune now: Structured pruning of llms with only forward passes." arXiv preprint arXiv:2402.05406 (2024).

[4]An, Yongqi, et al. "Fluctuation-based adaptive structured pruning for large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024.

ICLR 2025 submitted:

[5] Low-Rank Compression of Language Models Via Gradient-based Rank Selection

[6] ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models

[7] SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

Older:

[8] The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction (ICLR 2023)

[9] Zhang, Cheng, et al. "LQER: Low-Rank Quantization Error Reconstruction for LLMs." arXiv preprint arXiv:2402.02446 (2024).

[10] Saha, Rajarshi, Varun Srivastava, and Mert Pilanci. "Matrix compression via randomized low rank and low precision factorization." Advances in Neural

Information Processing Systems 36 (2023).

[11] Lee, Jung Hyun, et al. "LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices." arXiv preprint arXiv:2407.11534 (2024).

[12] Saha, Rajarshi, et al. "Compressing Large Language Models using Low Rank and Low Precision Decomposition." arXiv preprint arXiv:2405.18886 (2024).

[13] LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)

[14] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

[15] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

[16]Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers (NeurIPS 2021)

[17] Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix decomposition. (2020)

Here we highlight some of the updates:

1. [PPL on WikiText2 Across Different Models and Compression Ratios] We conducted comprehensive experiments on a variety of models, including LLAMA1-7B, LLAMA1-13B, LLAMA2-7B, LLAMA2-13B, and LLAMA 3.1-8B. The detailed experimental results are provided in Table 2, Table 11,Table 12, Table 16and Table 17 of the revised version.

   Below, we summarize the key results on Wikitext2.

   Compression Ratio	Llama-7b	Llama-2-7b	Llama-3.1-8b	Llama-13b	Llama-2-13b
   1.0	5.68	5.52	6.30	5.17	4.96
   0.8	6.08	5.92	6.90	5.43	5.25
   0.6	8.12	7.88	8.53	6.50	6.45
   0.4	9.95	9.47	15.8	11.3	29.3

2. [Performance Comparison of Dobi-SVD and Popular Pruning Methods on Various Tasks] We conducted comprehensive comparisons with various pruning-based methods, including LLM Pruner[1], SliceGPT[2], Bonsai[3] and Wanda-SP[4] on different commonsense reasoning datasets. For details and comparisons with other baseline models, please refer to Table 2, Table 14, Table 15, Table 18, and Table 19.

   Here we present the main comparison results on LLAMA3.1-8B and LLAMA2-7B across five zero-shot reasoning datasets, along with their average accuracy. The experimental results demonstrate that Dobi-SVD significantly outperforms pruning-based methods by a large margin.

Results show that, on Llama3.1-8B, Dobi-SVD with a high pruning ratio of 0.4 outperforms other pruning methods with ratios of 0.6 and 0.5. And on Llama2-7B, Dobi-SVD with ratios of 0.6 and 0.4 surpasses other pruning methods with ratios of 0.6 and 0.5, respectively.

Furthermore, we present the main comparison results of LLAMA3.1-8B and LLAMA2-7B on Wikitext2 compared with popular pruning methods.

5. Finally, we summarize the ICLR 2025 submitted SVD-based methods below. (N.A.：Not Available)

$9$

$11$ | Partial rank | Every value | NO | 24.8 | 29.3 | N.A. | NO | | $6$ ASVD | WS (activation-aware, using scaling matrix S) | YES, $10$

$12$

$13$

$14$

$15$ | Partial rank | 10% 20% ... 100% | NO | N.A. | 28.15 | N.A. | NO | | $7$ SVD-LLM | WS (activation-aware, using scaling matrix S) | YES, $10$

$12$

$13$

$14$

$15$ | Partial rank | 10% 20% ... 100% | Yes | N.A. | N.A. | Arxiv: 53.74 ICLR(new): 13.31 | NO | | Dobi-SVD (Ours) | A (directly truncating activation) | No | Full rank | Every value | NO | 60.1 | 38.6 | 9.95 | Yes |

*Relevant analysis can be found in our responses to reviewer sX49 and reviewer uXRu's first questions, as well as in Lines 195~206, where Dobi theoretically and experimentally establishes a fundamentally different third paradigm for SVD-based compression methods—directly truncating activations. Further details are provided in Appendix A.4, "Dobi-SVD's New Perspective 1: A Novel Path from Activation to Weight."

+ Other SVD-based methods have not addressed the long-overlooked limitation in SVD, which prevents them from analyzing and utilizing all singular value information. This issue is not limited to SVD-based methods but is also encountered in other low-rank approaches, such as [6], [9], and [10].