ARB-LLM: Alternating Refined Binarizations for Large Language Models

Q3-1: On Figure 1, authors show Pareto curve and demonstrate that ARB-LLM-RC outperforms the same-size FP16 models and previously published binarized models. So authors show an improvements over previous models binarization, but it does not show the whole picture in terms of pareto curve: they only compare binarized models vs fp16 (on Figure 1). But it will be informative to show also 4bits and 8bits models. For example paper "The case for 4-bit precision: k-bit Inference Scaling Laws" shows that in terms of pareto curve, for the same model size, 4bits has the best accuracy. In addition to 4bits and 8bits it would be great to plot 3bits too on pareto curve (given that 3bits is already presented in multiple Tables in this paper).

A3-1: Thank you for your valuable suggestion. We add the 2-bit, 3-bit, 4-bit, and 8-bit models to the Pareto curve. Due to the page limit, we put this big Pareto curve in Figure 4 of the refined supplementary file.

1. Among GPTQ models, the 4-bit version achieves the highest accuracy for a given memory budget compared to its 2-bit, 3-bit, and 8-bit counterparts. However, our ARB-LLM still outperforms 4-bit GPTQ on the Pareto curve.
2. Low-bit quantization methods like GPTQ suffer significant accuracy degradation at extremely low bit levels (e.g., 2-bit). In contrast, our ARB-LLM excels in such scenarios, delivering superior performance while using less memory.

Q3-2: Is it ARB second-order binarization is called in the paper " higher-bit representation, i.e., second-order binarizatio". Is effectively 2bits quantization?

A3-2: Thank you for your question. We provide additional explanation below.

1. "ARB second-order binarization" is equal to "higher-bit representation, i.e., second-order binarization" in our paper.
2. It is different from 2-bit quantization.
   • Second-order binarization employs two 1-bit matrices (-1, +1) with scaling and shifting to approximate the full-precision matrix, whereas 2-bit quantization uses a single 2-bit matrix (0, 1, 2, 3) along with scaling and shifting to approximate FP16.
   • As claimed in DB-LLM [ref1], second-order binarization might be more informative compared to standard 2-bit quantization. Additionally, leveraging second-order binarization to approximate salient weights enhances consistency within a binarization framework.

[ref1] DB-LLM: Accurate Dual-Binarization for Efficient LLMs, ACL, 2024.

Q3-3: Please explain why BiLLM accuracy in Table 2 does not match the results in the original BiLLM paper (shown in Table 3 LLaMA and LLaMA2)

A3-3: Thank you for your question. We provide the explanation below.

1. We strictly follow the BiLLM codebase to reproduce the results. However, the experiments were conducted on a different GPU, and some package versions may differ. These slight variations in the experimental environment are likely the primary cause of any discrepancies.
2. For these two models, more than half of the reproduced results are better than those reported in the original paper. Whether compared against the original results or the reproduced ones, our ARB-LLM consistently outperforms BiLLM. We also add this table to Table 10 of the supplementary file.

Q2-4: Could the authors introduce the meaning of each term in equation (6) so that the reader can understand without referring to prior papers?

A2-4: Thank you for your suggestion. We add the explanation of each item in the refined version (highlighted in red color).

Q2-5: Can the authors clarify the actual memory consumption and compare it to PTQ methods under the same memory budget?

A2-5: Thanks for your valuable feedback.

1. Compared to low-bit quantization, our ARB-LLM-RC outperforms 2-bit GPTQ while using less memory.

2. Compared to binarization methods, our ARB-LLM-RC largely outperforms PB-LLM and BiLLM with less memory.

We have responded to another similar question, Q2-1. Please refer to A2-1 for more details.

Q2-1: The authors claim that the weight bit is 1.11 bits, but based on the memory comparison, the model takes approximately 3GB of memory on the Llama-7B model, which is similar in size to GPTQ-3bit of the same model, with inferior performance. Can the authors clarify this?

A2-1: Thank you for your question. For simplicity and consistency, we did not include storage optimization when calculating memory. This ensures a fair comparison among the binarization methods since all of them utilize bitmaps. In practice, as shown in prior works like PB-LLM and BiLLM, these methods employ the sparse matrix format CSR for bitmap storage. Therefore, when comparing to low-bit quantization methods that do not use bitmaps, it would be fair to include the CSR compression for bitmaps. Below, we provide additional results incorporating the compressed bitmap on LLaMA-7B.

1. The optimized memory usage of ARB-LLM and other binarized PTQ methods is comparable to GPTQ-2bit, while achieving significantly improved performance.

2. Our ARB-LLM-RC significantly outperforms GPTQ-2bit and other binarized PTQ methods, while using less memory. We also clarify this in section 4.4 of the refined version.

Q2-2: It seems the approach works better on the older OPT model than the Llama family, with Llama3's performance, in particular, being much worse than floating-point. Can the authors showcase the performance on more recent models like Phi/Llama3.2 to address this concern?

A2-2: Thank you for your valuable suggestion. We conduct additional experiments by binarizing Phi-3-mini (3.8B), and Phi-3-medium (14B), and evaluate their perplexity (PPL) on WikiText2.

1. In the recent Phi models, our ARB-LLM consistently outperforms previous binarization methods, PB-LLM and BiLLM.

2. The performance gap between binarized and FP16 models is reasonable. Compared to the binarization of OPT, the results for Phi-3-mini (3.8B) surpass those of OPT (2.7B), and the results for Phi-3-medium (14B) outperform OPT (13B).

Q2-3: Can the authors provide any runtime inference metrics to showcase the practical feasibility of the proposed implementation?

A2-3: Thank you for raising the question about runtime inference metrics. We acknowledge that evaluating runtime performance is crucial for demonstrating the practical feasibility of our proposed implementation. Unfortunately, previous works such as BiLLM and PB-LLM, did not report runtime performance due to the lack of a CUDA kernel for matrix multiplication between FP activation and 1-bit weights. We use the BitBLAS [ref1] codebase to benchmark our method and comparable approaches, providing detailed runtime evaluations. We evaluate the runtime inference metrics by measuring the latency (ms) of various linear layers in LLaMA-7B and LLaMA-13B. The sequence length of input tensor X is 2048, and experiments are conducted on an NVIDIA A6000 GPU.

1. Our method demonstrates significant improvements in inference speed compared to FP16 and PB-LLM. PB-LLM is slower due to the Int8-to-FP16 matrix multiplication.

2. Both ARB-LLM-X and ARB-LLM-RC achieve a speed similar to BiLLM, while largely improving the performance.

[ref1] https://github.com/microsoft/BitBLAS

Dear Reviewer H9m3,

We sincerely appreciate your valuable time and insightful comments. Regarding your remaining concern about the end-to-end speedup of Binary LLMs, we have taken your suggestions into account and conducted additional experiments to further demonstrate the practical benefits of our method.

The results show that our approach achieves a 6x speedup in the Prefill phase and a nearly 5x speedup in the Decode phase, compared to FP16. These improvements are notably significant, especially when compared to the 2x speedup achieved by the 4-bit quantization method QUIK.

We believe our responses have covered your concerns, and we would appreciate your feedback on whether they adequately resolve the issue. Please let us know if there are any remaining aspects of our work that require further clarification.

Thank you once again for your time and constructive feedback.

Best,

Authors

Dear Reviewer H9m3,

We sincerely appreciate the time and effort you dedicated to reviewing our work, as well as your insightful comments. We have referred to your suggestions and conducted additional experiments to validate the end-to-end speedup of our method, which we believe have covered your concerns.

Due to the approaching deadline for rebuttal and reviewer mmbQ responding that our response can address the concerns, we also hope to discuss further with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.

Best,

Authors

Dear Reviewer H9m3,

We sincerely appreciate the time and effort you have invested in reviewing our work, as well as your insightful comments. In response to your suggestions, we have conducted additional experiments to validate the end-to-end speedup of our method, which we believe address your concerns.

With the rebuttal deadline approaching and Reviewer mmbQ indicating that our response adequately addresses the raised issues, we would greatly appreciate any further feedback from you to ensure that all your concerns have been fully resolved. Please let us know if there are still any aspects of our work that remain unclear.

Best，

Authors

Q1-1: The main weakness of the paper is the evaluation seems limited to perplexity and average accuracy. The paper would be more interesting if there were trade-offs on more tasks.

A1-1: Thank you for your valuable feedback. In line with previous work, we primarily evaluate using PPL and accuracy. Yet, the evaluation can be extended to other metrics. We conduct additional experiments on LLaMA-7B, measuring the F1 score on the SQuADv2 dataset and chrF on the WMT2014 (En-Fr) dataset.

Our ARB-LLM significantly outperforms previous binarization methods, PB-LLM and BiLLM, in both F1 score and chrF metrics, further demonstrating the effectiveness of our proposed method.

Q1-2: Does the proposed method perform consistently on 7 zero-shot QA datasets?

A1-2: Thank you for your question. We have presented a detailed accuracy comparison on seven zero-shot QA datasets in Table 5 of the supplementary file. Please refer to it for comprehensive details.

1. Compared to the SOTA method BiLLM, ARB-LLM demonstrates superior performance on most datasets, while achieving comparable results on one or two remaining datasets.
2. In the field of LLM quantization, it is a standard practice to evaluate performance across multiple datasets and emphasize average accuracy to mitigate the effects of randomness. For example, there is a fluctuation in accuracy between PB-LLM and BiLLM on individual datasets—PB-LLM outperforms BiLLM on some, while BiLLM leads on others. However, the average accuracy clearly shows that BiLLM significantly outperforms PB-LLM overall. Similarly, ARB-LLM surpasses BiLLM and even 2-bit GPTQ by a considerable margin in terms of average accuracy.

Q1-3: Is the binarized model able to get reasonable performance on long context, math and reasoning dataset?

A1-3: Thank you for your question. We conduct additional experiments on the long context dataset SQuADv2, math dataset MMLU College Mathematics, and reasoning dataset SWAG.

Our ARB-LLM also outperforms the previous binarization methods PB-LLM and BiLLM on these datasets, narrowing the performance gap with FP16, especially on the long-context SQuADv2 dataset.