Thanks for your questions.

For question 2.

Representative works on mixed precision include Yao et al., Agile-Quant, and SpQR, which focus on applying mixed precision to model weights and activations, and Bablani et al. who apply mixed precision across different layers, while we adopt different precisions across different singular values in the feature space. Recently, Apple intelligence has also applied mixed precision across different layers. As the reviewer Wicf said, "While neither low-rank nor mixed precision quantization techniques are particularly novel, the combination of the two for delta-compression is novel and clever." As far as we know, we are the first to propose mixed precision compression in the feature space and in multi-modal settings.

For question 3

We compress our backbone into 4-bit, and test their performance shown in the following table. Our method can still retain performance though the backbone is compressed in low-bit format.

For question 4

Due to limited pages, we did not include the detailed process of the decision of the number of different bit-widths in the current version. When setting $𝑟_{𝑏𝑒𝑔𝑖𝑛}$ and $𝑟_{𝑒𝑛𝑑}$, we decided based on minimizing the error between the activations of the compressed model and the original model. When searching for "Double Precision", using two 8-bit singular values results in the smallest error, while for "Triple Precision", 32 3-bit singular values results in the smallest error.

Ablation on 8-bit in "Double Precision", changing the number of 8-bit from 1-8

Ablation on 3-bit in "Triple Precision", changing the number of 3-bit from 8-64

Based on the reviewers' insights, we regard the mixed precision issue as a multi-objective optimization problem, considering single precision to be a special case of mixed precision. We developed a genetic algorithm to address this problem, using the bit count of single precision as the initial solution. We use the following objective function, $f = min PPL(x1, x2, x3, x4, x5)$ where x1, x2, x3, x4, x5 indicating the number of 16-bit, 8-bit, 4-bit, 3-bit, 2-bit and $PPL(.)$ means we calculate perplexity in 128 samples randomly chosen form C4 dataset. For each aligned model, we can automatically determine the mixing strategy through the genetic algorithm. The results demonstrate that the genetic algorithm can yield better results than greedy search, making our method easy to be applied to many different aligned models.

We also conducted experiments on the 13B model, where the genetic algorithm yielded better performance.

Thank you very much for your insightful suggestions. We have provided detailed responses to your question. If you could participate in the discussion period, we would be very grateful.

Thanks for your comment.

For weakness

Due to limited pages, we did not include the detailed process of the decision of the number of different bit-widths in the current version. When setting $𝑟_{𝑏𝑒𝑔𝑖𝑛}$ and $𝑟_{𝑒𝑛𝑑}$, we decided based on minimizing the error between the activations of the compressed model and the original model. When searching for "Double Precision", using two 8-bit singular values results in the smallest error, while for "Triple Precision", 32 3-bit singular values results in the smallest error.

Ablation on 8-bit in "Double Precision", changing the number of 8-bit from 1-8

Ablation on 3-bit in "Triple Precision", changing the number of 3-bit from 8-64

Based on the reviewers' insights, we regard the mixed precision issue as a multi-objective optimization problem, considering single precision to be a special case of mixed precision. We developed a genetic algorithm to address this problem, using the bit count of single precision as the initial solution. We use the following objective function, $f = min PPL(x1, x2, x3, x4, x5)$ where x1, x2, x3, x4, x5 indicating the number of 16-bit, 8-bit, 4-bit, 3-bit, 2-bit and $PPL(.)$ means we calculate perplexity in 128 samples randomly chosen form C4 dataset. For each aligned model, we can automatically determine the mixing strategy through the genetic algorithm. The results demonstrate that the genetic algorithm can yield better results than greedy search, making our method easy to be applied to many different aligned models.

We also conducted experiments on the 13B model, where the genetic algorithm yielded better performance.

For question

1. Compress aligned models directly using SVD can easily achieve bad performance in low-bit setting. As the following table using SVD compress the model into 1-bit or 2-bit settings resulting in a performance drop to 0 in math and code tasks. | Tasks | GSM8K | Math | HumanEval | Mbpp | |---------------|-------|------|-----------|------| | Compress 1-bit| 0 | 0 | 0 | 0 | | Compress 2-bit| 0 | 0 | 0 | 0 |
2. Recently Onebit (Xu et al.) find compress aligned model needs more sophisticated algorithms and post-training is also needed. Even in this carefully designed setting, there is still a significant performance drop. For example, in Table 2 of OneBit, the average performance on 6 benchmarks decreased from 64.06 to 51.33 and from 66.39 to 55.17 on Llama-2-7B and Llama-2-13B, respectively.
3. In our experiments, delta exhibits a long-tail distribution characteristic, and using SVD compression algorithms can retain the model in a near-lossless manner, compressing it to an equivalent of 1-bit. Therefore, compressing delta-weights can achieve better performance.

Thank you very much for your insightful suggestions. We have provided detailed responses to your question. If you could participate in the discussion period, we would be very grateful.

Thanks for your questions, weaknesses, and limitations which can bring much improvement to our paper.

For question 1.

Due to limited pages, we did not include the detailed process of the decision of the number of different bit-widths in the current version. When setting $𝑟_{𝑏𝑒𝑔𝑖𝑛}$ and $𝑟_{𝑒𝑛𝑑}$, we decided based on minimizing the error between the activations of the compressed model and the original model.

Ablation on 8-bit in "Double Precision", changing the number of 8-bit from 1-8

Ablation on 3-bit in "Triple Precision", changing the number of 3-bit from 8-64

For question 2.

Based on the reviewers' insights, we regard the mixed precision issue as a multi-objective optimization problem, considering single precision to be a special case of mixed precision. We developed a genetic algorithm to address this problem, using the bit count of single precision as the initial solution. We use the following objective function, $f = min PPL(x1, x2, x3, x4, x5)$ where x1, x2, x3, x4, x5 indicating the number of 16-bit, 8-bit, 4-bit, 3-bit, 2-bit and $PPL(.)$ means we calculate perplexity in 128 samples randomly chosen form C4 dataset. For each aligned model, we can automatically determine the mixing strategy through the genetic algorithm. The results demonstrate that the genetic algorithm can yield better results than greedy search, making our method easy to be applied to many different aligned models.

We also conducted experiments on the 13B model, where the genetic algorithm yielded better performance.

The genetic algorithm demonstrated better performance, with the average performance on the 7B model even slightly surpassing "Aligned models". However, even with the same settings without genetic algorithm, our method's performance is close to that, indicating its generalization ability.

Recently, mixed precision is being more widely used such as Apple intelligence (Gunter, Tom, et al. ) allocate different bits to different layers which achieves 3.5-bit on average and Jinhao Li et al. employ mixing 2-bit, 4-bit and 16-bit that ultimately attains an average of 2-bit. Recently, mixed precision has received more attention than before, and the use of mixed precision methods has also been widely applied to model compression. We believe that our method could also be further applied to model compression, which is the direction of our next exploration.

For question 3.

Thanks for your great question. We will give a more proper title.

For question 4.

Ablation on batch, we set sequence length to 128, our method achieves 3x speedup than Pytorch. Further, Han Guo et al. and Jianyu et al. have implemented more advanced kernels. We can draw on their methods to achieve higher acceleration ratios.

Ablation on batch, we set sequence length to 128, our method achieves 3x speedup than Pytorch.

Ablation on hidden_size, we set batch 8.

For limitation

Thanks for your great insight, we conducted a more granular exploration, compressing the model up to 32×. We use WizardMath-7B in GSM8K task and results are shown in the following table. When the compression ratio is within 20×, our method still performs well. However, at a compression ratio of 32×, there is a noticeable decline in performance, but it still outperforms low-rank and low-bit methods, which only achieve a 16× compression ratio.

For weakness

We have used sample decoding algorithm and set decoding temperature = 0.2. We run 5 times and calculate the mean value and confidence interval shown in the table.

Thanks for your questions

For question 1

In delta-compression, we consider compressing $U$ and $V^{T}$. For $U$ and $V^{T}$, their inputs are crucial for adjusting weights during the quantization process, as illustrated in GPTQ. In a forward pass $Y = W X + (UΣV^{T}) X$, the input of $V^{T}$ is $X$, while the input of $U$ is $ΣV^{T}X$. Therefore, the quantization of $V$ is $Quant(V, X)$, while the quantization of $U$ is $Quant(U, ΣV^{T}X)$.

For question 2

Due to limited pages, we did not include the detailed process of the decision of the number of different bit-widths in the current version. When setting $𝑟_{𝑏𝑒𝑔𝑖𝑛}$ and $𝑟_{𝑒𝑛𝑑}$, we decided based on minimizing the error between the activations of the compressed model and the original model. When searching for "Double Precision", using two 8-bit singular values results in the smallest error, while for "Triple Precision", 32 3-bit singular values results in the smallest error.

Ablation on 8-bit in "Double Precision", changing the number of 8-bit from 1-8

Ablation on 3-bit in "Triple Precision", changing the number of 3-bit from 8-64

Based on the reviewers' insights, we regard the mixed precision issue as a multi-objective optimization problem, considering single precision to be a special case of mixed precision. We developed a genetic algorithm to address this problem, using the bit count of single precision as the initial solution. We use the following objective function, $f = min PPL(x1, x2, x3, x4, x5)$ where x1, x2, x3, x4, x5 indicating the number of 16-bit, 8-bit, 4-bit, 3-bit, 2-bit and $PPL(.)$ means we calculate perplexity in 128 samples randomly chosen form C4 dataset. For each aligned model, we can automatically determine the mixing strategy through the genetic algorithm. The results demonstrate that the genetic algorithm can yield better results than greedy search, making our method easy to be applied to many different aligned models.

We also conducted experiments on the 13B model, where the genetic algorithm yielded better performance.

Thank you very much for your insightful suggestions. We have provided detailed responses to your question. If you have any other questions, we sincerely invite you to participate in the discussion period if possible. Thanks very much!