ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning

Dear Reviewer dutm,

Thank you for your thorough review. We are really encouraged by your highlights:

1. Our paper is well-organized and well-presented, with clear problem statement, insight and method.
2. The $\approx$ 1 bit performance of ClusComp is especially impressive by only using a modicum calibration data.
3. The related works are thorough, and we compare ClusComp to all relevant and popular-used baselines.
4. We conduct thorough evaluation with convincing and clearly presented results.
5. Our idea is simple and easy to understand.
6. Our paper is worth publishing, since we set a north star for further hardware-friendly improvements.

$~$

---

$~$

Strength 2. How does SqueezeLLM/GPTQ/QuIP fare, if it was plotted in Figure 1?

We add these baselines in Figure D.3 in the new submission. One can observe that SqueezeLLM and QuiP are two comparable baselines for >=3 bit. For lower bit levels, ClusComp outperforms them.

In Figure 1, we aim to only include the most competitive baselines. Thanks for your question, we realize that QuIP outperforms OmniQuant at the 2-bit level and have added QuIP to Figure 1.

$~$

Weakness: Lack of latency or power benchmarks. If a custom kernel for ClusComp would actually be faster than its FP16 baseline on modern GPUs.

One of the future plans outlined in our paper was to design a new CUDA kernel optimized for ClusComp (Line 930). We are pleased to share that we successfully implemented this kernel before the rebuttal deadline.

Summary: The new CUDA kernel speeds up ClusComp's inference, making it > 17% faster than the 16-bit LLM.

Firstly, we would like to thank you for your thorough review. They are very inspired, and help us come up with two solutions (one is done and the other needs more time) to speed up the ClusComp's inference.

1. For GPU with small SRAM, like RTX3090-24GB, we mainly optimize the kernel by fusing all operations (index, reshape and matrix multiplication). We can obtain a speedup over the 16-bit LLM, as shown in Table R1. However, we also admit that this speedup mainly comes from the smaller cost of transfer from global GPU memory to cache, mainly working for GPUs whose memory transfer is a severe bottleneck.

2. For some advanced GPUs (like H100), we notice there is a new feature (distributed shared memory) in the CUDA document that states "Thread block clusters introduced in compute capability 9.0 provide the ability for threads in a thread block cluster to access shared memory of all the participating thread blocks in a cluster." We expect this new feature to offer a fundamental solution for ClusComp. However, due to the time limit and limited access to H100, we couldn't finish this part by now.

We want to re-state that one of the key contributions of ClusComp lies in its ability to deliver significantly better performance at lower bit levels, where uniform quantization often fails. For example, the uniform quantization (GPTQ and AWQ) obtains 0% accuracy on 2-bit LlaVa in Table 3 of the paper, while ClusComp's accuracy is 53.5%. We hope this quality-latency trade-off doesn't hurt our contribution.

$~$

Table R1. Inference speed on RTX3090-24GB GPU. Setup: Generate 128 tokens with a batch size of 1.

$~$

Question 1. What does "time" mean in Figure 1?

Sorry for the confusion. The time (GPU-hour) in Figure 1 means the time required for compression instead of the inference time. ClusComp requires two steps for compression, i.e. clustering and block-wise error minimization. The shown GPU-hour is used for these two steps. The orange bar is for clustering and the green bar is for reconstruction.

$~$

What does "≤" mean in Table 2?

Sorry for this confusion. ClusComp uses a non-uniform quantization technique, which means we can't control the bits-per-parameter to any specific number. Taking "<=4.14" in Table 2 as an example, "<=" means that the largest bits-per-parameter for all LLMs compressed by ClusComp is 4.14. If you go to Table C.1, you can notice that the bits-per-parameter is 4.14 for Llama-1-7B and Llama-2-7B, but it is 4.09 for Llama-2-13B and 4.03 for Llama-2-70B.

$~$

---

$~$

If you are interested in how ClusComp approximates the 16-bit weight distribution, we highly recommend you to Figure D.1 for the visualization of weight distribution.

Thank you for your thoughtful discussions. We have incorporated the new results in the updated version.

If these revisions address your concerns, we kindly request a reconsideration of the scores. Should you have any further questions, we are happy to assist.

Dear Reviewer dutm,

We are really encouraged by your further confirmation that our paper is worthy of publishing, and thank you for your suggestion of focusing more on low-bit post rebuttal.

In the following, we want to highlight our other contributions and new latency results that you are interested.

1. ClusComp as parameter and memory-efficient finetuning method: Since most of the discussion is about the post-training quantization performance of ClusComp, we want to metion our another core contribution: ClusComp can also serve as a memory-efficient and parameter-efficient finetuning method that outperform LoRA, QLoRA, GaLore and LISA.. One can finetune a 70B LLM on a GPU with mere 48GB memory, and achieve promising accuracy as full finetuning.

2. Thanks to the suggestion from Reviewer h5Zw who recommends us to further quantize the codebook from 16-bit (originally) to lower bit. I.e. after the block-wise error minimization, we further quantize each cluster in the codebook with symmetric quantization. (Notably, we quantize the codebook mainly for faster inference. If one wants to do finetuning, we still need to save it in FP16 for finetuning, and then quantize them for faster inference.)

From Table R2, we can observe: (1) 8-bit codebook offers the same perplexity as 16-bit’s; (2) 4-bit codebook slightly hurts the performance, but is still comparable to the best baseline at the 4-bit level and significantly outperforms the best baseline at the 2-bit level; (3) The results of the 2-bit codebook are not acceptable.

Since the quantization further reduces the codebook size, we can obtain better inference speed. As shown in Table R3, we obtain 40% speedup for a 7B LLM and 22% for a 70B LLM. Notably, our kernel is mainly optimized for the 16-bit codebook. We expect a higher speedup with further optimization, and will continue this work.

$~$

Table R2. The perplexity of ClusComp with quantized codebook on WikiText2. The best baseline is taken from Table C.1 in the paper. Please refer to Table D.3 in the paper for better reading experience and more results (70B LLM offers the same observation).

Table R3. Inference speed on RTX3090-24GB GPU.

Dear Reviewer dutm,

thanks to the extension of the rebuttal deadline and your suggestion of focusing more on low-bit optimization, we could finally achieve a ≈ 100% inference speedup after carefully optimizing the CUDA kernel. Please refer to the general response for more details.

If this work further addresses your concern, and it is not too much trouble for you, we would really thank you for any update to the score.

Dear Reviewer ZEQz,

Thank you for your thorough review. We are really encouraged by your highlights:

1. Our proposed method, ClusComp, is a simple alternative to scalar quantization, and suitable for scenarios where memory is the bottleneck instead of latency.
2. ClusComp can serve as an efficient finetuning method to train the entire LLM.

$~$

---

$~$

Weakness 1. ClusComp doesn't give latency benefits.

One of the future plans outlined in our paper was to design a new CUDA kernel optimized for ClusComp (Line 930). We are pleased to share that we successfully implemented this kernel before the rebuttal deadline.

Summary: The new CUDA kernel enhances ClusComp's inference speed, achieving a >17% improvement compared to the 16-bit LLM.

In the initial submission, we had not yet developed this kernel, and the inference speed of ClusComp was comparable to (or slightly slower than) the 16-bit LLM. After submission, we focused on designing an efficient CUDA kernel for ClusComp, integrating all necessary operations (indexing, reshaping, and matrix multiplication). As shown in Table R1, our preliminary kernel implementation now improves inference speed by over 17%.

While the speedup remains lower than that of uniform quantization methods, this is expected, as uniform quantization is inherently more GPU-friendly. However, one of the key contributions of ClusComp lies in its ability to deliver significantly better performance at lower bit levels, where uniform quantization often fails. For instance, in Table 3, we show that uniform quantization methods (e.g., GPTQ and AWQ) achieve 0% accuracy on 2-bit LLaVA, whereas ClusComp attains an impressive 53.5% accuracy.

$~$

Table R1. Inference speed on RTX3090-24GB GPU. Setup: Generate 128 tokens with a batch size of 1.

$~$

Weakness 2. Add missing related works, like SqueezeLLM [2] and Any-Precision LLM [3].

These two works are indeed very related. We have added them at Line-61 and Line-124 in the new submission.

$~$

Question. Does finetuning the codebook affect the generalization performance of the model?

This is a good point. In contrast, our finetuning results on the general-domain tasks demonstrate a better generalization of ClusComp.

In Table 6 and 7, we finetune the LLMs on Alpaca-GPT3.5 and Alpaca-GPT4, and evaluate the finetuned LLMs on MMLU and AGIEval. Alpaca contains 52K instruction-following samples. MMLU and AGIEval is a large collection of varied tasks. Specifically, AGIEval contains 20 tasks that focus on history, math, biology, law, logic, chemistry, English and so on. MMLU has a similar focus with even more tasks. We can safely assume that the data distribution in Alpaca is not fully aligned with that in MMLU and AGIEval. So overfit to Alpaca doesn't guarantee a better performance on these benchmarks.

We confirm this by finetuning Llama-2-7B compressed by ClusComp with one more epoch. (Notably, our reported results in the paper strictly follow the number of epochs as our baselines). The results become worse, from 47.0 to 46.4 on MMLU and from 26.5 to 26.1 on AGIEval.

ClusComp's better results on MMLU and AGIEval against quantization-related finetuning methods (QA-LoRA, GPTQ-LoRA) and memory-efficient finetuning methods (LISA, LoRA, GaLore) actually show that ClusComp maintains a good generalization, since these two large benchmarks couldn't easily be fit.

We further show the weight distribution of ClusComp in Figure D.1 of the new submission, one can observe that ClusComp closely approximates the 16-bit weight distribution. This is a major reason for ClusComp's better performance since it better preserves the pretrained knowledge.

$~$

---

$~$

Thank you for your thoughtful questions. We have incorporated the new results in the updated version.

If these revisions address your concerns, we kindly request a reconsideration of the scores. Should you have any further questions, we are happy to assist.

Dear Reviewer wLyR,

Thank you for your thorough review. We are really encouraged by your highlights:

1. The motivation and method are sound and clear. Our paper is easy to understand.
2. Our proposed method, ClusComp, saves memory and has limited accuracy loss compared to the 16-bit LLM.
3. ClusComp shows promising results with a wide range of models, tasks and comparisons.

$~$

---

$~$

New results on the inference speed of ClusComp.

One of the future plans outlined in our paper was to design a new CUDA kernel optimized for ClusComp (Line 930). We are pleased to share that we successfully implemented this kernel before the rebuttal deadline.

Summary: The new CUDA kernel enhances ClusComp's inference speed, achieving a >17% improvement compared to the 16-bit LLM.

In the initial submission, we had not yet developed this kernel, and the inference speed of ClusComp was comparable to (or slightly slower than) the 16-bit LLM. After submission, we focused on designing an efficient CUDA kernel for ClusComp, integrating all necessary operations (indexing, reshaping, and matrix multiplication). As shown in Table R1, our preliminary kernel implementation now improves inference speed by over 17%.

While the speedup remains lower than that of uniform quantization methods, this is expected, as uniform quantization is inherently more GPU-friendly. However, one of the key contributions of ClusComp lies in its ability to deliver significantly better performance at lower bit levels, where uniform quantization often fails. For instance, in Table 3, we show that uniform quantization methods (e.g., GPTQ and AWQ) achieve 0% accuracy on 2-bit LLaVA, whereas ClusComp attains an impressive 53.5% accuracy.

$~$

Table R1. Inference speed on RTX3090-24GB GPU. Setup: Generate 128 tokens with a batch size of 1.

$~$

Weakness 1. Include more comparisons to recent works, like SqueezeLLM [1] and AdaDim [2].

Thank you for the suggestion of including more recent baselines.

Summary: ClusComp outperforms both SqueezeLLM and AdaDim.

As shown in Table R2, R3 and R4, ClusComp (with a similar level of bit or lower bit) outperforms SqueezeLLM and GPTQ-AdaDim, with a more evident difference for a lower bit. We have added these results to Appendix D.1 in the new submission.

$~$

Table R2. The perplexity on WikiText2. The values in the brackets are the exact bits of ClusComp for different LLMs. The SqueezeLLM results are taken from [1]. Please refer to Table D.1 in the paper for better reading experience.

Table R3. The accuracy on Llama-2-7B. CSR (commonsense reasoning) here only contains 4 tasks (PIQA, HellaSwag, WinoGrande and ARC-easy) as [2]. The GPTQ-AdaDim results are taken from [2]. Please refer to Table D.2 in the paper for better reading experience.

Table R4. The accuracy on Llama-2-13B.

$~$

Weakness 2. Add analysis to explain how the codebooks are distributed to better capture salient weights.

Really thank you for this suggestion. We have added new figures for visualizing the weight distribution of ClusComp to Appendix D.2 in the new submission.

Summary: ClusComp’s weight distribution of different bit-levels can better simulate the 16-bit weight distribution than uniform quantization.

$~$

[1] Kim, et al., "SqueezeLLM: Dense-and-Sparse Quantization", ICML 2024.

[2] Heo, et al., "Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models", ICLR 2024

(1/n)

Dear wLyR,

We are really encouraged by your increasing score. Here we answer your concern about the improvement room for the kernel optimization.

Summary: Further improvement is possible, according to our finished and in-plan experiments. Currently, we can achieve maximum 40% speedup for 7B model and 22% for 70B model.

To address the latency issue, we conduct 3 parallel attempts (two are done and one is in progress):

1. (Done) The results from the first attempt were already shown in our initial repsonse, i.e. 29% speedup over FP16 for a 7B model and 17% for a 70B model. For this attempt, we mainly fuse all necessary operations (indexing, reshaping, and matrix multiplication).
2. (Done) Thanks to the suggestion from Reviewer h5Zw who recommends us to further quantize the codebook from 16-bit (originally) to lower bit. Therefore. after the block-wise error minimization, we further quantize each cluster in the codebook with symmetric quantization.

From Table R4, we can observe: (1) 8-bit codebook offers the same perplexity as 16-bit’s; (2) 4-bit codebook slightly hurts the performance, but is still comparable to the best baseline at the 4-bit level and significantly outperforms the best baseline at the 2-bit level; (3) The results of the 2-bit codebook are not acceptable.

Since the quantization further reduces the codebook size, we can obtain better inference speed. As shown in Table R5, we obtain 40% speedup for a 7B LLM and 22% for a 70B LLM. Notably, our kernel is mainly optimized for the 16-bit codebook. We expect a higher speedup with further optimization. Due to the time limit, we couldn't finish it by now.

3. (In progress, fundamental solution) Before introducing our last attempt, we offer some background knowledge to explain why it is hard to speed up the codebook-based quantization. For faster matrix multiplication, it's better for us to save the codebook in the L1 cache (shared memory) of a thread block that provides fast access to reading. However, the L1 cache in most GPUs is too small to save the codebook. Therefore, we have to save the codebook in the L2 cache (in the previous 2 attempts) that is slower for reading but has a larger storage. Recently, we notice there is a new feature from CUDA, i.e. distributed shared memory. As the terminology implies, we can gather the shared memory from different thread blocks to enlarge the storage. In this way, we can have a larger L1 cache and fast reading access at the same time. We believe this is the fundamental solution to codebook-based quantization method, and might have a great impact to the follow-up research. However, we are still working on it, and haven't obtained the results by now.

$~$

Table R4. The perplexity of ClusComp with quantized codebook on WikiText2. The best baseline is taken from Table C.1 in the paper. Please refer to Table D.3 in the paper for better reading experience and more results (70B LLM offers the same observation).

Table R5. Inference speed on RTX3090-24GB GPU.

Following response (1/n)

$~$

Question 1. Could further quantization of the codebook improve the efficiency of the model? Would combining ClusComp with conventional quantization lead to better results?

This is a very inspired suggestion. We have added these new results to Appendix D.4 in the new submission.

Summary: 8-bit or 4-bit codebook offers a similar perplexity as 16-bit codebook, and they further speed up the inference.

Experiment setup: We apply symmetric quantization to the codebook parameters after the block-wise error minimization step.

From Table R2, we can observe: (1) 8-bit codebook offers the same perplexity as 16-bit’s; (2) 4-bit codebook slightly hurts the performance, but is still comparable to the best baseline at the 4-bit level and outperforms the best baseline at the 2-bit level. (3) The results of the 2-bit codebook are not acceptable.

Since the quantization further reduces the codebook size, we can obtain better inference speed, as shown in Table R3. Notably, our kernel is mainly optimized for the 16-bit codebook. We expect a higher speedup with further optimization. Due to the time limit, we couldn't finish it by now.

$~$

Table R2. The perplexity of ClusComp with quantized codebook on WikiText2. The best baseline is taken from Table C.1 in the paper. Please refer to Table D.3 in the paper for better reading experience and more results.

Table R3. Inference speed on RTX3090-24GB GPU.

$~$

Question 2. Why does the 2.01-bit ClusComp version perform better than the 2.14-bit version on Llama3-8B?

This is an insightful observation. We believe it can be attributed to two key factors:

1. Unlike perplexity, accuracy is a more discrete and inherently noisier metric. In Table C.4, for instance, 4-bit AWQ notably outperforms the 16-bit Llama-3-8B on WinoGrande (74.0 vs. 72.8). Given that the differences between 2.01-bit and 2.14-bit are small, coupled with the larger performance variance typically observed in low-bit LLMs (as seen in lm-eval), the slightly higher accuracy for the 2.01-bit configuration could stem from these factors.

2. As shown in Figure D.1 of the updated submission, ClusComp maintains a weight distribution at the 2-bit level that closely resembles the distribution at 16-bit precision. This suggests that a minor reduction in bit level does not necessarily result in performance degradation.

$~$

Question 3. Additional ablation studies on hyperparameters, such as the number of clusters ($n$) and the cluster dimension ($g$).

Summary: The performance is more sensitive to the number of clusters than the cluster dimension.

We vary $g$ and $n$ on Llama-2-7B with ClusComp$^-$ and show the perplexity in Table R4, R5 and R6. Increasing $n$ has a more positive effect on performance compared to reducing $g$. This finding underpins our choice of $n \approx 2^{16}$ (like 65500). However, while $n$ plays a crucial role in enhancing performance, selecting $n > 2^{16}$ would necessitate using 32-bit storage for the code $q$, substantially increasing the bits-per-parameter and adversely affecting memory efficiency. So we always choose $n<2^{16}$. We have added these results to Appendix D.5 in the new submission.

$~$

Table R4. Same $n=65500$ different $g$. Perplexity changes smoothly. Please refer to Table D.4 for better reading experience.

Table R5. Same $g$ different $n$. Perplexity changes dramatically.

Table R6. Similar bit level. Perplexity is more sensitive to $n$.

$~$

---

$~$

Thank you for your thoughtful suggestions. We have incorporated the new results in the updated version.

If these revisions address your concerns, we kindly request a reconsideration of the scores. Should you have any further questions, we are happy to assist.

Dear Reviewer h5Zw,

Thank you for your thorough review. We are really encouraged by your highlights:

1. Our proposed method, ClusComp, achieves superior performance at low-bit precisions, outperforming alternative methods acrosss various LLMs and VLM with extensive experiments.
2. ClusComp demonstrates significant memory savings, making it feasible to deploy LLMs on resource-constrained devices, and crucial for practical deployment.

$~$

---

$~$

Weakness 1. ClusComp maintains the same computational load during inference as the FP16 version.

One of the future plans outlined in our paper was to design a new CUDA kernel optimized for ClusComp (Line 930). We are pleased to share that we successfully implemented this kernel before the rebuttal deadline.

Summary: The new CUDA kernel enhances ClusComp's inference speed, achieving a 29% speedup for a 7B LLM and a 17% speedup for a 70B LLM.

In the initial submission, we had not yet developed this kernel, and the inference speed of ClusComp was comparable to (or slightly slower than) the 16-bit LLM. After submission, we focused on designing an efficient CUDA kernel for ClusComp, integrating all necessary operations (indexing, reshaping, and matrix multiplication). As shown in Table R1, our preliminary kernel implementation now improves inference speed by > 17%.

While the speedup remains lower than that of uniform quantization methods, this is expected, as uniform quantization is inherently more GPU-friendly. However, one of the key contributions of ClusComp lies in its ability to deliver significantly better performance at lower bit levels, where uniform quantization often fails. For instance, in Table 3, we show that uniform quantization methods (e.g., GPTQ and AWQ) achieve 0% accuracy on 2-bit LLaVA, whereas ClusComp attains an impressive 53.5% accuracy.

$~$

Table R1. Inference speed on RTX3090-24GB GPU. Setup: Generate 128 tokens with a batch size of 1.

$~$

Weakness 2. What is the meaning of the marker colors in Figure 1.

We use different colors to distinguish the baselines from ClusComp, i.e. using blue color for all baselines and using red color for ClusComp. Sorry for this confusion, we have added a legend in the new submission for Figure 1.

$~$

Weakness 2. The histogram in Figure 3 appears unnecessary; a simple text explanation could suffice.

Thank you for this suggestion. We try to support our claim with empirical observation. We will move it to the appendix for our next version.

$~$

Weakness 3. Block-wise error minimization lacks novelty.

We acknowledge that block-wise error minimization is not novel and is a widely adopted approach for reducing quantization error, as stated in L277-278: "Block-wise error minimization... has emerged as a standard, efficient, and effective approach... (Egiazarian et al., 2024; Tseng et al., 2024; van Baalen et al., 2024; Shao et al., 2024; Liao & Monz, 2024a)".

Our intent in making this explicit was to clarify that we do not claim to have introduced this technique. Rather, it serves as an intermediate step to enhance the quality of ClusComp, consistent with practices in prior works such as AQLM, Quip#, GPTVQ, OmniQuant, ApiQ, and others. These methods also employ block-wise error minimization to mitigate quantization error.

The core contributions of our work lie elsewhere. Specifically, we propose a simple yet effective model compression method based on clustering, which surpasses existing baselines across multiple bit levels. Moreover, our method doubles as a parameter-efficient fine-tuning approach (requiring only 1% of trainable parameters) and a memory-efficient fine-tuning technique (reducing memory usage for a 70B LLM from >140GB to 42GB), achieving superior results compared to prior parameter and memory-efficient fine-tuning methods.

$~$

(1/n)

Dear Reviewer h5Zw,

We sincerely thank you for your positive feedback and the time you dedicated to reviewing our rebuttal. It brings us great joy to learn that our response has addressed your concerns and contributed to increasing the score from 3 to 5.

As the score is still borderline reject, we are wondering if there are any major concerns regarding our current revision. It would be our great pleasure to provide further clarifications and results to address any additional doubts.

Your suggestions really help a lot to improve our work and make the justification of our method more complete. We also greatly appreciate your recognition of the superior performance at low-bit precisions and significant memory savings of our algorithm.

Once again, we would like to express our appreciation for your valuable comments during the reviewing process.

Best regards,

Authors

Thanks to the extension of rebuttal deadline. Our currently optimized CUDA kernel for ClusComp can achieve ~100% speedup over the FP16 model.

Experimental setting: We further quantize the codebook (originally in 16-bit) to lower bit (4-bit) after block-wise error minimization (suggested by Reviewer h5Zw). Such a design further reduces the codebook size with minimal performance degradation, therefore leading to higher speedup with a carefully optimized kernel. We measure the time required for generating 128 tokens with a batch size of 1 (2048 tokens offer a similar result).

Perplexity performance: As shown in Table OR1, we can observe: (1) 8-bit codebook offers the same perplexity as 16-bit’s; (2) 4-bit codebook slightly hurts the performance, but still achieveing the same perplexity to the best baseline for 4-bit LLMs and significantly outperforming the best baseline for 2-bit LLMs. (3) The results of the 2-bit codebook are not acceptable.

Inference speedup: As shown in Table OR2, the inference speedup is 110% for a 7B LLM, and 81% for a 70B LLM.

$~$

Table OR1. The perplexity of ClusComp with quantized codebook on WikiText2. The best baseline is taken from Table C.1 in the paper. Please refer to Table D.3 in the paper for better reading experience and more results (70B LLM offers the same observation).

Table OR2. Inference speed on a A6000-48GB GPU.