LLM-Codebook for Extreme Compression of Large Language Models

Table 1: Zero-shot performance of the compressed LLaMA-7B and LLaMA-13B. ‘rec’ indicates whether LLM-codebooks use the recovery stage. (𝐝,𝐤) represents each codebook has 𝐤 codewords of dimension 𝐝.

---

Table 2: Combining low-bit post-training quantization (PTQ) with LLM-Codebook without recovery stage. ‘g128’ represents PTQ method with a group size of 128 is used for all the codebooks. (𝐝,𝐤) represents each codebook has 𝐤 codewords of dimension 𝐝.

Dear reviewer,

I apologize for the delayed response. Your insightful feedback has been immensely valuable in guiding the refinement of our work. Despite the constraints of limited laboratory resources, I have endeavored to conduct as many supplementary experiments as possible to address your comments comprehensively.

---

The updated part of our method:

Inspired by the approaches of AWQ[1] and SqueezeLLM[2] in preserving important weights, we upgrade Hessian-aware K-means algorithm to retain the weight and position of the salience value in the first 0.66% in a compressed sparse row. This significantly enhances the compression performance with a 2% reduction in the compression ratio. The preserved weights are excluded from the Hessian-aware k-means clustering, which also reduces clustering errors.

---

A1: In the codebook-based LoRA during the recovery stage, the original weights of each linear layer are frozen, and an adaptor is assigned to each layer. During the fine-tuning process of all adaptors, a linear layer is randomly selected, and its original weight is merged with the adaptor for compression, resulting in a codebook and index map. Subsequently, the compressed weight is reconstructed based on these components. The compressed weight is no longer updated, meaning that the codebook and index map of that specific linear layer are also not updated. To compensate for the compression error of the layers that have already been compressed, the other linear layers that have not yet been compressed continue to update their adaptors. The above process is repeated until all the linear layer are completely compressed.

---

A2: The memory cost is primarily divided into two components: the base model and the adaptor. Typically, the adaptor accounts for about 1 to 3% of the base model's memory requirement, depending on the chosen value of the rank parameter. The memory consumption of LLaMA-70B exceeds 130GB, and regrettably, we lack sufficient computational resources to support the compression of this model.

---

A3: Intel's Incremental Network Quantization[3] adopts a progressive quantization strategy, which divides weights into two groups. One group is quantized and then frozen, while the other is fine-tuned to compensate for the quantization loss of the former. This process is repeated until all weights are fully quantized. Inspired by the aforementioned strategy, the codebook-based LoRA in the recovery stage also represents a form of progressive compression, with specific procedures as outlined in answer1.

---

A4: I apologize for the inadequacy of our figures in clearly demonstrating the differences. In fact, there are distinctions between the two weight maps; for instance, the weights in the top right corner differ in color. The issue arises from our approach of using heatmaps based on the min/max values of the weights. Given that these min/max values are similar for both sets of weights, the RGB differences corresponding to minor compression errors may be challenging to discern.

---

A5: In Table 1 of Official Comment 2, we have supplemented the low-bit compression comparison with Omniquant[4], SqueezeLLM[2], and AWQ[1], demonstrating that our work achieves optimal performance in 2/3-bit compression of LLaMA-7b and LLaMA-13B. We do not include low_bit_llama[5] in our comparison due to the absence of open-source compression code and the lack of pre-compressed LLaMA1 models. Although QA-LoRA[6] represents an excellent piece of work, the absence of open-source code precludes comparison. In exploring the potential of combining low-bit compression with structural clustering for further compression, we conducted experiments with Codebook PTQ. As shown in Table 2 of Official Comment 2, the results for W8A16 and W4A16 are very close to those of W16A16.

---

Reference:

[1] Lin J, Tang J, Tang H, et al. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration[J]. arXiv preprint arXiv:2306.00978, 2023.

[2] Kim S, Hooper C, Gholami A, et al. SqueezeLLM: Dense-and-Sparse Quantization[J]. arXiv preprint arXiv:2306.07629, 2023.

[3] Zhou A, Yao A, Guo Y, et al. Incremental network quantization: Towards lossless cnns with low-precision weights[J]. arXiv preprint arXiv:1702.03044, 2017.

[4] https://github.com/GreenBitAI/low_bit_llama

[5] Xu Y, Xie L, Gu X, et al. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models[J]. arXiv preprint arXiv:2309.14717, 2023.

Dear reviewer,

I apologize for the delayed response. Your insightful feedback has been immensely valuable in guiding the refinement of our work. Despite the constraints of limited laboratory resources, I have endeavored to conduct as many supplementary experiments as possible to address your comments comprehensively.

---

The updated part of our method:

Inspired by the approaches of AWQ[1] and SqueezeLLM[2] in preserving important weights, we upgrade Hessian-aware K-means algorithm to retain the weight and position of the salience value in the first 0.66% in a compressed sparse row. This significantly enhances the compression performance with a 2% reduction in the compression ratio. The preserved weights are excluded from the Hessian-aware k-means clustering, which also reduces clustering errors.

---

A1: We sincerely apologize for our insufficient research, as we failed to recognize that the concept of Hessian-aware k-means has already been utilized in previous works [3][4]. Our implementation of Hessian-aware k-means differs in several key aspects. First, by performing a Taylor expansion on the loss function, we obtain three terms: the first related to the gradient, the second to the Hessian matrix, and the third being a redundant term. Previous works often omit the first term, assuming gradients are close to zero. However, in our case, since the dataset used for computing gradients and Hessian matrices is not extracted from the original training set, gradients are not negligible and should be considered. We regret that our naming convention may have led to the misconception that the significance of weights is solely determined by Hessian calculations. As shown in Table1 of Official Comment 2, we conducted experiments showing that considering both the first and second terms is more effective than considering them separately. Additionally, in our salience-based centroid weighting update, we noticed that many weights have salience values close to zero, leading to significant compression errors. To mitigate this, we added salience.max() to all salience values for smoothing.

Furthermore, our codebook-based LoRA approach in the recovery stage marks a significant innovation. This method deviates from earlier practices in CNN-related research, where updating the codebook was done using gradients of compressed weights through backpropagation – a method not feasible for large language models (LLMs). Instead, we have adopted a strategy inspired by Intel's Incremental Network Quantization, which employs a progressive quantization strategy[5]. We merge linear layer weights with adaptors, compress them, and then freeze them. The compression loss is compensated by adaptors in other uncompressed layers. This effectively encodes LoRA's information into the codebook without necessitating additional memory.

---

A2&A3: In Table 2 of Official Comment 2, we have supplemented the low-bit compression comparison with Omniquant[6], SqueezeLLM, and AWQ, demonstrating that our work achieves optimal performance in 2/3-bit compression of LLaMA-7b and LLaMA-13B. Although QuIP represents an excellent piece of work, the absence of open-source code of LLaMA precludes comparison. As shown in Table 3 of Official Comment 2, in exploring the potential of combining low-bit compression with structural clustering for further compression, we conducted experiments with Codebook PTQ, indicating that the results for W8A16 and W4A16 are very close to those of W16A16.

---

Reference:

[1] Lin J, Tang J, Tang H, et al. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration[J]. arXiv preprint arXiv:2306.00978, 2023.

[2] Kim S, Hooper C, Gholami A, et al. SqueezeLLM: Dense-and-Sparse Quantization[J]. arXiv preprint arXiv:2306.07629, 2023.

[3] TOWARDS THE LIMIT OF NETWORK QUANTIZATION ICLR 2017, SAMSUNG

[4] https://arxiv.org/abs/2306.07629

[5] Zhou A, Yao A, Guo Y, et al. Incremental network quantization: Towards lossless cnns with low-precision weights[J]. arXiv preprint arXiv:1702.03044, 2017.

[6] Shao W, Chen M, Zhang Z, et al. Omniquant: Omnidirectionally calibrated quantization for large language models[J]. arXiv preprint arXiv:2308.13137, 2023.

Table 1: Comparation of calculation mode of weight salience. 'First term' refers to the calculation of salience utilizing the first term (related to the gradient) in the Taylor expansion of the loss function. 'Second term' denotes the use of the second term (related to the Hessian matrix) for this purpose. 'Mix' represents the addition of the first and second terms. (𝐝,𝐤) represents each codebook has 𝐤 codewords of dimension 𝐝. ‘Recovery’ indicates whether LLM-codebooks use the recovery stage.

---

Table 2: Zero-shot performance of the compressed LLaMA-7B and LLaMA-13B. ‘rec’ indicates whether LLM-codebooks use the recovery stage. (𝐝,𝐤) represents each codebook has 𝐤 codewords of dimension 𝐝.

---

Table 3: Combining low-bit post-training quantization (PTQ) with LLM-Codebook without recovery stage. ‘g128’ represents PTQ method with a group size of 128 is used for all the codebooks. (𝐝,𝐤) represents each codebook has 𝐤 codewords of dimension 𝐝.

Table 1: Comparative analyses focusing on the sources of data and the number of samples used. ‘Recovery’ indicates whether LLM-codebooks use the recovery stage. (𝐝,𝐤) represents each codebook has 𝐤 codewords of dimension 𝐝.

---

Table 2: Ablation experiments by omitting the use of salience. ‘Recovery’ indicates whether LLM-codebooks use the recovery stage. (𝐝,𝐤) represents each codebook has 𝐤 codewords of dimension 𝐝.

---

Table 3: Zero-shot performance of the compressed LLaMA-7B and LLaMA-13B. ‘rec’ indicates whether LLM-codebooks use the recovery stage. (𝐝,𝐤) represents each codebook has 𝐤 codewords of dimension 𝐝.

Dear reviewer,

I apologize for the delayed response. Your insightful feedback has been immensely valuable in guiding the refinement of our work. Despite the constraints of limited laboratory resources, I have endeavored to conduct as many supplementary experiments as possible to address your comments comprehensively.

---

The updated part of our method:

Inspired by the approaches of AWQ[1] and SqueezeLLM[2] in preserving important weights, we upgrade Hessian-aware K-means algorithm to retain the weight and position of the salience value in the first 0.66% in a compressed sparse row. This significantly enhances the compression performance with a 2% reduction in the compression ratio. The preserved weights are excluded from the Hessian-aware k-means clustering, which also reduces clustering errors.

---

A1: In Table 1 of Official Comment 2, we conducted comparative analyses focusing on the sources of data and the number of samples used. We found that data derived from the Bookcorpus dataset yielded more accurate salience calculations compared to those obtained from Wiki and the C4 corpus. Additionally, increasing the number of samples further enhanced the precision of salience estimation. As shown in Table 2 of Official Comment 2, we performed ablation experiments by omitting the use of salience. These experiments demonstrated that preserving weights with higher salience significantly improves compression performance in the recovery stage.

---

A2: In Table 3 of Official Comment 2, we conducted comparative experiments with different LoRA ranks, attempting to achieve optimal performance using only the conventional K-means algorithm while minimally increasing the number of trainable parameters. The results indicate that merely employing the standard K-means algorithm does not suffice to attain optimal performance. Contrary to playing a secondary role, Hessian-aware K-means is significantly influential in model compression. In fact, its impact on compression performance exceeds that of the recovery stage at 4/3-bit compression. Conversely, the recovery stage has a more pronounced effect in 2-bit compression scenarios.

---

A3: The QLoRA and LLM-Pruner are based on LoRA tuning, while GPTQ, SparseGPT, AWQ, SqueezeLLM, and Omniquant[3] represent Post-Training Quantization (PTQ) methods or pruning methods not based on LoRA. In Table 3 of Official Comment 2, we have provided data that excludes the use of a recovery stage, and the results surpass those of the aforementioned methods. However, incorporating LoRA into PTQ methods involves substantial engineering efforts, and the adaptor weights fine-tuned are in fp16 format, which does not qualify as strict low-bit quantization. It indicates that our codebook-based LoRA approach in the recovery stage marks a significant innovation. This method deviates from earlier practices in CNN-related research, where updating the codebook was done using gradients of compressed weights through backpropagation – a method not feasible for large language models (LLMs). Instead, we have adopted a strategy inspired by Intel's Incremental Network Quantization, which employs a progressive quantization strategy[4]. We merge linear layer weights with adaptors, compress them, and then freeze them. The compression loss is compensated by adaptors in other uncompressed layers. This effectively encodes LoRA's information into the codebook without necessitating additional memory.

---

A4: At this juncture, with a compression ratio of 95%, our method collapses due to substantial compression errors, resulting in the compressed model yielding garbled inference outcomes, which fail to reflect the impact of salience on model compression.

---

Reference:

[1] Lin J, Tang J, Tang H, et al. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration[J]. arXiv preprint arXiv:2306.00978, 2023.

[2] Kim S, Hooper C, Gholami A, et al. SqueezeLLM: Dense-and-Sparse Quantization[J]. arXiv preprint arXiv:2306.07629, 2023.

[3] Shao W, Chen M, Zhang Z, et al. Omniquant: Omnidirectionally calibrated quantization for large language models[J]. arXiv preprint arXiv:2308.13137, 2023.

[4] Zhou A, Yao A, Guo Y, et al. Incremental network quantization: Towards lossless cnns with low-precision weights[J]. arXiv preprint arXiv:1702.03044, 2017.