NoWag: A Unified Framework for Shape Preserving Com- pression of Large Language Models

We thank Reviewer H7VY for their constructive feedback and for acknowledging the clarity of our writing and the performance benefits of our approach. Below we address the concerns raised in detail, due to the number of reasons raised, we split our response into several comments due to the character limit

Reason to Reject 1: There's no inference speedup evaluation

We would like to emphasize that a central goal of model compression is reducing the memory needed for inference. NoWag-P at 50% sparsity results in a corresponding halving of memory needed to store the weights, and NoWag-VQ at 2 bits per value reduces the memory needed by 8x compared to fp16.

Nevertheless, we still observe inference speedups from the removal of memory bottlenecks and sparse matrices. We have included below an evaluation of the speedup of 2:4 NoWag-P as an example. We measured the wall clock of the matrix multiplication times on an A6000, which was the same setup as sparseGPT and Wanda and AQLM for inference speedup evaluation. Below we have included the benchmarks for matrix vector multiplication on a standard layer (gate projection) of the Llama-2 7B/13B/70B models.

Reason to Reject 2: The technique contribution is insufficient, as I believe the only contribution is that normalization can help gain good performance for VQ and pruning.

We respectfully disagree that our only contribution is showing that normalization helps. To put it succinctly, we propose lightweight normalization and a data aware proxy loss. This proxy loss is elementwise decomposable, allowing for a local/global optimal solution to be found quickly.

For VQ, this provides additional benefits; existing SOTA VQ methods rely on computationally expensive Hessian-based proxy losses that require iterative optimization and are sensitive to calibration data. In contrast, NoWag-VQ eliminates these dependencies entirely through normalization, achieving comparable performance in blockwise finetuning and superior performance in the challenging one-shot regime where Hessian-based methods struggle most. This represents a significant methodological advance: we achieve SOTA results while avoiding the core computational and stability issues that plague current approaches.

We sincerely thank Reviewer yN5X for the thoughtful and encouraging feedback. We're especially grateful for your positive remarks on the clarity of the paper and the usefulness of the illustrations, particularly Figure 3. We’re glad that our exposition made the proposed method accessible and enjoyable to read.

Regarding the question on sensitivity to the calibration dataset:

We ran an ablation on the NoWag-VQ dataset using 128 samples of the RedPajamas dataset and two other popular calibration datasets, the train split of the Wikitext2 and C4 datasets. Due to time constraints we only performed the ablation on the small models, Llama-2 7B and Llama-3 8B. In essence the results regardless of calibration dataset is SOTA, however RedPajamas offers the best overall performance since it is the closest to the original pretraining dataset of the Llama models.

Wikitext 2 Validation Perplexity (↓)

C4 Validation Perplexity (↓)

We appreciate Reviewer sxCa thorough review. Below we address the concerns raised in detail,

Reasons To Reject 1: The results are a bit mixed.

We respectfully disagree with the characterization that the results are too mixed to merit acceptance. While NoWag achieves performance comparable to AQLM and Wanda in some settings, it offers significant practical advantages that we believe warrant recognition.

Quantization: Although NoWag matches AQLM in performance, it does so with dramatically reduced resource requirements. Specifically, NoWag uses 16× less calibration data and achieves quantization of a 7B model in 10 hours on a single A100 GPU—compared to AQLM, which requires over 1 day under the same conditions. These gains in efficiency are critical in real-world deployments, where compute and memory constraints are major bottlenecks.

Pruning: It is important to note that structured pruning patterns like 2:4 and 4:8 inherently restrict the sparsity patterns we can induce, limiting the benefit of breaking up weight matrix structures that our normalizer provides. As expected, under more flexible patterns such as 8:16, 16:32, and 32:64, NoWag significantly outperforms Wanda (see Figure 4). Moreover, even at the constrained 2:4 setting, NoWag achieves competitive performance with Wanda while providing the additional benefit of using a metric that is compatible with quantization —something Wanda does not support. This unified applicability across compression paradigms is a key contribution of our method.

Reasons To Reject 2: Lack of baselines.

Quantization We conducted additional evaluations comparing the Wikitext2 (wiki2) and C4 perplexity between NoWag-VQ and ClusComp and GPTVQ. Perplexities were calculated with a context length of 2048, rather than the native context length of the models, since both ClusComp and GPTVQ reported perplexities calculated with a context length of 2048. GPTVQ did not report C4 perplexities. For ClusComp we compared against ClusComp$^-$ the non fine tuned results, and ClusComp, the block wise fine tuned results.

One/Zero-shot

Blockwise Finetuned

These evaluations show that NoWag-VQ significantly outperforms both GPTVQ and ClusComp in the respective non fine tuned and fine tuned settings. Furthermore, the non fine tuned NoWag-VQ performs competitively with the fine tuned ClusComp.

Differences with ClusComp: There are two main differences between NoWag and ClusComp. First, ClusComp does not incorporate any normalization, unlike NoWag-VQ. Second, ClusComp performs vanilla vector K-means, rather than weighted K-means, meaning that ClusComp has no data awareness and treats all input channels with equal importance. This is why especially in the one-shot regime, ClusComp$^-$ performs so poorly compared to NoWag-VQ

Pruning

Respectfully, we don’t believe that the two baselines proposed are relevant baselines to compare against NoWag. DARE focuses on pruning to more effectively merge Supervised Finetuned LLMs on different tasks together. SLERP focuses on Composed Image Retrieval, we checked the main text of the paper and there is nothing related to pruning or quantization. Could you please clarify how SLERP and DARE are relevant papers to use as baselines to compare against NoWag-P?

Thank you for the detailed rebuttal. I'm convinced by the discussion, and willing to raise my score.

About the pruning baselines, the authors are right. my referred baselines are more about merging baselines. You can ignore this part.

We thank the reviewers 5R73 for their thoughtful and constructive feedback. We are delighted that you found our paper clear and significant.

[Minor] Reason to Reject 1: "Unified Framework" Scope:

Respectfully, we disagree with the assertion that the unified framework wording is not accurate. As you mentioned in your review, the “unified framework” refers to the normalization and conceptual objective. This allows for performance that is superior/comparable performance for pruning and compression using common, well explored, algorithms for each compression measure (K-means for VQ and greedy pruning for Pruning). However we remain open to suggestions for changes for more accurate wording.

[Minor] Reason to Reject 2: AQLM vs NoWag for Llama-2 70B.

We are planning on quantizing Llama-2 70B with a $d=8$ codebook for an apples to apples comparison with AQLM, however because the time taken to perform the quantization scales exponentially with $d$, with our computational resources it may not be finished before the end of the rebuttal period

Reason to Reject 3 and Question 1: Pruning Score

The pruning scores used by NoWag-P is $(\bar{W}_ij)^2 ||X_j||_2^2$. However we note that pruning based on $|\bar{W}_ij| ||X_j||_2$ is equivalent to pruning based on $(\bar{W}_ij)^2 ||X_j||_2^2$, as the latter is simply the square of the former, which is non-negative. As a result, the resulting pruning mask will be the same whether we prune against $|\bar{W}_ij| ||X_j||_2$ or against $(\bar{W}_ij)^2 ||X_j||_2^2$.

Question 2: Sensitivity to $T$ and less iterations

We conducted an ablation for $T=1$, $T=10$, $T=40$. Because of the time and computational constraints, we selected the smaller models, Llama-2-7b/13b and Llama-3 8B to conduct this ablation study. We listed the Wikitext2 and C4 perplexities below, bolding the smallest $T$ for which NoWag outperforms QuIP#. For Llama-2 7b and Llama-3 8B NoWag with only one iteration of K-means was able to outperform QuIP#, for Llama-2 13B, after 10 iterations, NoWag-VQ was able to outperform QuIP#. We also note that the performance of the model continues to increase as we increase the number of iterations.

Question 3: Computational Complexity Compared with Other VQ algorithms

Yes you are correct that the overall computational complexity is $O(N_{\mathrm{TotalWeights}} T 2^{k})$. Because all VQ algorithms need to sweep over all the weights and all the vectors in the codebook, they all have a computational complexity of at least $O(N_{\mathrm{TotalWeights}} 2^{k} C)$ where $C$ is the number of iterations of this sweep that is performed. QuIP#, VPTQ, GPTVQ etc have an additional overhead of inverting the hessian, which is $O(d_{in}^3)$ for each weight matrix. Assuming that on average $d_{in} \approx d_{out}$ for the whole model, this roughly translates to an additional overhead of $O(N_{\mathrm{TotalWeights}}^{3/2})$ overhead, In practice for $T=100$ we don’t see any significant speedup compared with QuIP#, we do see a roughly 2x speedup compared with AQLM. However compared to both QuIP# and AQLM we use over an order of magnitude less calibration data.