FlexBCQ: Flexible Binary-coding Quantization for Large Language Models

[R5-4] What is the best value for periodic remapping? In the main experimental section, p=2 is adopted for all models except Llama-3 70B. Does even less frequent remapping help? It would be good to have a plot performance vs p for the sake of completeness.

The periodic remapping technique is a technique that performs weight remapping to updated quantization levels not at every step during Blockwise Output Reconstruction, but at intervals of $p$ steps. Table R5-3 presents the performance variations of FlexBCQ across different remapping periods.

Table R5-3. MMLU benchmark accuracies of 3-bit Llama-3 8B models quantized by FlexBCQ with different remapping period $p$.

The experimental results indicate that, with the exception of $p=4$, most settings achieve comparable accuracy, and $p=2$, as employed in the paper, demonstrates strong performance. Interestingly, we observe the highest average accuracy when we use less frequent settings such as $p=8$.

We recommend using $p=2$, as it was utilized to achieve the outstanding results presented in the paper for all cases except the Llama-3 70B model, where $p=1$ is used. Slight performance gains may be achieved by tuning $p$ to suit specific scenarios. We will include the plot regarding hyperparameter tuning over $p$ in the revised paper.

[R5-5] How was the duration of the optimization procedure chosen? 20 epochs could be excessive, as the amount of data is small and the model may start to overfit (the number of trainable parameters is small either). I would suggest adding an ablation on this. What is the runtime of the method for the models considered?

[Epochs]

In our experimental setup, we adopt OmniQuant’s SGD-based optimization framework, similar to FlexBCQ, and set the number of epochs to 20. The table below illustrates the impact of varying the number of epochs on FlexBCQ’s performance:

Table R5-4. Accuracy variation on the MMLU benchmark for the 3-bit quantized Llama-3 8B model using FlexBCQ across different epoch settings.

The experimental results demonstrate that the 20-epoch setting used in our experiments achieves the highest average accuracy. Overfitting appears to occur at approximately 30 epochs, further validating that training with 20 epochs is a proper choice.

[Running Time]

Before explaining about running time of FlexBCQ, we clarify some main points. Our study primarily focuses on achieving superior accuracy at a given bit-width. For LLMs, the quantization process is performed only once, and the model can be used repeatedly thereafter. Consequently, as long as the quantization process is not prohibitively expensive, we consider a model with higher accuracy to be superior.

We compare the time required to quantize the Llama-3 8B model to 2, 3, and 4 bits using FlexBCQ and its main competitor, FlexRound. The comparison results are summarized in Table R5-5.

Table R5-5. Running times in seconds for quantizing Llama-3 8B models into 2 to 4bits using FlexRound and FlexBCQ.

FlexBCQ employs non-uniform quantization levels, requiring weights to be mapped by identifying the nearest quantization level, rather than using the rounding function typically used in uniform quantization. This process of finding the nearest quantization level is more time-consuming than rounding, resulting in longer running times. Specifically, the GPU-unfriendly BST-based algorithm proposed by Xu et al. [1] (Algorithm 1 in Xu et al. [1]) leads to execution times that are practically infeasible to large language models.

To mitigate this issue, we replace the BST-based approach with a GPU-friendly algorithm that computes the distances between each weight and all quantization levels simultaneously to identify the nearest quantization level. This replacement significantly reduces the running time while preserving FlexBCQ’s superior performance. Further exploration of methods to optimize running time without compromising performance is a promising direction for future research, which we plan to address in subsequent studies. We will include a discussion about the running time of FlexBCQ in the paper during the discussion period.

[R5-3] MMLU and GSM8k are quite representative benchmarks. However, for a comparison with prior art, I would suggest reporting Wikitext2/C4 perplexity for quantized models (or a subset of 0-shot benchmarks from lm-eval-harness).

To further validate the performance of FlexBCQ across diverse scenarios, we compare the perplexity (PPL) of quantized models on the WikiText-2 and C4 benchmarks. These experiments were conducted using the Llama-3 8B model, and the results are presented in Table R5-2.

Table R5-2. Perplexities on WikiText-2 and C4 benchmarks of 2 to 4bit Llama-3 8B models quantized using FlexBCQ and competitors.

Experimental results indicate that FlexBCQ consistently achieves the lowest perplexity (demonstrating the best performance) across all experiments compared to its competitors. In particular, for 2-bit quantization, all competing methods exhibit a perplexity of 50 or higher, reflecting substantial performance degradation, whereas FlexBCQ maintains a significantly lower perplexity of approximately 15. These results will be integrated into the paper to enable a more comprehensive and rigorous comparison.

[R5-6] It would be insightful to present aggregated statistics over different weights of the value $|I(w)-I(w_f)|$ across model - i.e percentage of weights for each value.

Thank you for suggesting this intriguing analytical topoc. To quantify the occurrence of flexible mapping, we analyze the percentage of flexibly mapped weights in (1) the entire model, (2) different block locations, and (3) various module types. The corresponding results are summarized in Tables R5-6, R5-7, and R5-8, respectively. Furthermore, to compare the mapping patterns of FlexBCQ and FlexRound, we compute statistics for both methods.

Table R5-6. Proportion of weights exhibiting flexible mapping across the entire model. Each column indicates the index change in the quantization level resulting from flexible mapping.

Table R5-7. Proportion of weights exhibiting flexible mapping across different block positions. Each column corresponds to the index change in the quantization level resulting from flexible mapping. Bold values indicate the highest proportion within each column.

Table R5-8. Proportion of weights exhibiting flexible mapping across different module types. Each column corresponds to the index change in the quantization level resulting from flexible mapping. Bold values indicate the highest proportion within each column.

(Continues in the next response)

We sincerely appreciate your thoughtful suggestions regarding the experiments in our paper. In response, we have conducted all the proposed experiments and we are in the process of integrating the results into the revised manuscript. The updated version, including these findings, will be updated during the discussion period, and we will provide a summary of the changes upon its release.

[R5-1] Typos

Thank you for finding typos in our paper. we revised the typos and the revised version of the paper will be uploaded soon.

• (Table 1) We modify the benchmark accuracies of the fp16 baseline of Mistral 7B as follows: MMLU 5-shot: 70.48$\rightarrow$62.57 MMLU 0-shot 68.17 $\rightarrow$ 60.22. We double-checked the scores of Mistral 7B, and there are no further typos.
• (line 224) Blokwise $\rightarrow$ Blockwise. We fix the typo.

[R5-2] The paper claims that GPTQ and AWQ yield significantly worse performance than OmniQuant. However, in my personal experience, GPTQ and AWQ appear to be pretty robust at 4 bit quantization and typically lose only 1-2% relative to the original model on MMLU and GSM8k. I would suggest checking experimental setup for these methods.

Thank you for your valuable suggestion. In our initial experiments, we excluded GPTQ and AWQ due to their significantly lower performance compared to OmniQuant on the W2A16 and W3A16 settings (Table 1 in OmniQuant [1]). However, in light of your feedback, we have conducted additional experiments incorporating GPTQ and AWQ. These experiments were performed on the Llama-3 8B model, and the results are presented in Table R5-1.

Table R5-1. Perplexities on WikiText-2 and C4 benchmarks of 2 to 4bit Llama-3 8B models quantized using FlexBCQ and competitors. Bold values represent the highest accuracies.

The experimental results confirm that AWQ and GPTQ demonstrate respectable performance. However, in all experiments except for the 4-bit 0-shot scenario, FlexBCQ outperforms both algorithms. Notably, for 2-bit quantization, FlexBCQ achieves a performance improvement of at least 9.94% compared to both AWQ and GPTQ. These findings will be integrated into the paper to enable a more thorough and precise comparison.

We summarize the above comparisons in Table R4-1. Furthermore, we will incorporate this discussion into the paper and notify you of the updated sections once the revisions are complete.

Table R4-1. Comparison of quantized results, the number of bits for saving, and the inference process of quantized models in UQ, BCQ, FlexBCQ, and FlexBCQ with Composition Theorem. $g$ is the group size and $k$ is the bit width.

[R4-6] How do the running time and inference speed of FlexBCQ compare with those of baseline methods? I am particularly interested in whether this method is both time and memory efficient.

[Running Time]

Our study primarily focuses on achieving superior accuracy at a given bit-width. For LLMs, the quantization process is performed only once, and the model can be used repeatedly thereafter. Consequently, as long as the quantization process is not prohibitively expensive, we consider a model with higher accuracy to be superior.

We compare the time required to quantize the Llama-3 8B model to 2, 3, and 4 bits using FlexBCQ and its main competitor, FlexRound. The results of this comparison are summarized in Table R4-5.

Table R4-5. Running times in seconds for quantizing Llama-3 8B models into 2 to 4bits using FlexRound and FlexBCQ.

FlexBCQ employs non-uniform quantization levels, requiring weights to be mapped by identifying the nearest quantization level, rather than using the rounding function typically used in uniform quantization. This process of finding the nearest quantization level is more time-consuming than rounding, resulting in longer running times. Specifically, the GPU-unfriendly BST-based algorithm proposed by Xu et al. [1] (Algorithm 1 in Xu et al. [1]) leads to execution times that are practically infeasible to large language models.

To mitigate this issue, we replace the BST-based approach with a GPU-friendly algorithm that computes the distances between each weight and all quantization levels simultaneously to identify the nearest quantization level. This replacement significantly reduces the running time while preserving FlexBCQ’s superior performance. Further exploration of methods to optimize running time without compromising performance is a promising direction for future research, which we plan to address in subsequent studies. We will include a discussion about the running time of FlexBCQ in the paper during the discussion period.

[Inference Speed]

In terms of inference speed, FlexBCQ matches the speed of conventional BCQ and benefits from acceleration via the LUT-GEMM kernel. The inference speed results for the LUT-GEMM kernel are summarized in Table R4-6.

Table R4-6. Latency comparison of the first FFN layer on OPT-175B model with various precision and corresponding kernel selections with m = 12288 and g = 128 on A100-80GB-GPU (Table 1 in LUT-GEMM).

In summary, LUT-GEMM exhibits superior acceleration capabilities compared to existing UQ kernels such as AWQ and GPTQ. However, since LUT-GEMM delivers equivalent acceleration for both BCQ and UQ, all methods listed in Tables 1 and 2 achieve identical acceleration performance at the same bit-width.

[References]

[1] Xu, Chen, et al. "Alternating multi-bit quantization for recurrent neural networks." arXiv preprint arXiv:1802.00150 (2018).

[R4-4] The absence of code and insufficient implementation details in Appendix B raise concerns about reproducibility. Providing the code or more comprehensive details would significantly aid in evaluating the robustness and applicability of the proposed method.

Regrettably, due to the current circumstances of the authors, we are unable to release the code at this time. Nevertheless, we are making every effort to eventually release the code as open-source, enabling others to benefit from the superior performance of FlexBCQ. If the paper is accepted, we aim to make the code publicly available by that time.

In addition, we are updating Appendix B to include detailed information on the implementation of FlexBCQ. Given the current unavailability of the code, we will provide comprehensive details on performance variations with respect to hyperparameters to facilitate the reproduction of our results without requiring access to the code.

[R4-5] Key innovations of FlexBCQ.

We will sequentially explain the problem FlexBCQ aims to solve and the method it employs, starting from the background of its development.

1. Background: Lack of Optimization Algorithms for BCQ. Although BCQ theoretically offers superior representational power, it suffers from low performance due to the lack of optimization algorithms specifically designed for it. In contrast, UQ, despite its limited representational power, achieves high performance thanks to the development of many effective optimization techniques. A representative example is the flexible mapping shown in Figure 1(a).

2. Intuition: Transferring Optimization Techniques. We hypothesize that transferring UQ's advanced optimization techniques to BCQ would enable BCQ to achieve superior performance compared to existing algorithms.

3. Challenge: Difficulty in Transferring Optimization Techniques. However, transferring UQ's optimization techniques to BCQ poses a significant challenge due to the fundamental differences in their quantization schemes. These schemes differ in both the quantization parameters they use and the methods for utilizing these parameters. Without additional ideas, transferring optimization techniques between them is infeasible.

4. Observation: Analyzing the Characteristics of Quantization Schemes. To address the challenge in (3), we analyze the quantization processes of UQ and BCQ. As shown in Figures 3(a) and (b), we observe that UQ's superior optimization techniques stem from its transformation process, while BCQ's representational power is derived from its mapping process to non-uniform quantization levels.

5. Main Idea: Integrating Quantization Processes. Based on the observations in (4), we integrate the quantization processes of UQ and BCQ. Specifically, quantization is performed by first applying the transformation process from UQ and then proceeding with BCQ's mapping process. This algorithm is named FlexBCQ, which combines UQ's optimization techniques with BCQ's representational power.

6. Main Idea 2: Composition Theorem for Efficiency A straightforward integration of the two quantization processes would result in an increased number of quantization parameters and a more complex inference process, leading to significant memory and latency overhead. To prevent this, we utilize FlexBCQ's integrated quantization process only during the optimization (calibration) phase. For storage, we replace the integrated process with a single BCQ representation. This replacement process is formalized in our proposed Composition Theorem, which ensures that the integrated quantization process can be simplified into a single BCQ representation.

7. Contribution: Outstanding Performance. The FlexBCQ algorithm proposed in (5) demonstrates superior performance across various benchmarks (Tables 1, 2, R4-2, and R4-3). Furthermore, it effectively utilizes UQ's optimization techniques and BCQ's representational power as intended, as illustrated in Figures 4(a) and 4(b).

This concludes our explanation of FlexBCQ, from its background to its contributions. We will revise the paper to ensure that readers can understand the above content without requiring additional explanations. These revisions will be completed during the discussion period, and we will provide a summary of the changes once they are finalized.

[R4-3] Performance of FlexBCQ (2-bit quantization and perplexity performance)

In our study, we rigorously evaluate the effectiveness of FlexBCQ using the general knowledge benchmark (MMLU) and the task-specific benchmark (GSM8K). The results demonstrate that FlexBCQ consistently outperforms other methods in most scenarios. Notably, FlexBCQ excels in challenging tasks, such as the 3-bit quantization experiments reported in Table 1 and the GSM8K benchmark in Table 2, which involves answering open-ended questions.

To further substantiate the robustness of FlexBCQ, we conduct experiments on 2-bit quantization and perplexity (PPL) performance. These experiments were performed using the Llama-3 8B model.

[Extreme Quantization (2-bit)]

Tables R4-2 and R4-3 summarize the accuracy of 2-bit quantized Llama-3 8B models on MMLU and GSM8K benchmarks. As illustrated in the table, FlexBCQ exhibits significantly higher accuracy than competitors in both 5-shot and 0-shot, exhibiting 8.88%p higher average accuracy than the second-best algorithm. Furthermore, FlexBCQ is the only algorithm that shows an accuracy of over 1% among all competitors, exhibiting more than 22% higher accuracies than the second-best algorithm.

Table R4-2. The accuracies on MMLU 5-shot and 0-shot benchmarks of 2-bit Llama-3 8B models quantized with FlexBCQ and competitors.

Table R4-3. The accuracies on GSM8K benchmark of 2-bit Llama-3 8B models quantized with FlexBCQ and competitors.

[PPL Scores]

We summarize the perplexities of quantized models on WikiText-2 and C4 benchmarks in Table R4-4. We use 2 to 4-bit for quantization to compare the performance of FlexBCQ and competitors exhaustively. As a result, FlexBCQ exhibits the lowest perplexities (the best performance) in all settings, especially in 2-bit constraints.

Table R4-4. Comparison of perplexities on WikiText-2 and C4 benchmarks of Llama-3 models quantized using FlexBCQ and competitors under 2 to 4-bit constraints. Lower perplexities mean better performances.

In summary, FlexBCQ exhibits outstanding performance not only in the experimental results highlighted in the paper but also in 2-bit quantization and perplexity benchmarks.

We sincerely thank you for carefully reviewing our paper and providing thoughtful comments. Your feedback has guided us in revising the text and figures, as well as conducting additional experiments, thereby improving the overall quality of our paper. Below, we address the comments you raised. Revisions related to writing and figures are currently in progress, and we will provide detailed information on these updates once they are finalized. All revisions will be completed within the discussion period.

[R4-1] Figures 1 and 2 fail to clarify the logic and pipeline of FlexBCQ. A more abstract diagram that highlights the main innovations of FlexBCQ and outlines its functionality would be beneficial. Unfortunately, Figures 1 and 2 are quite confusing as currently presented.

As you suggested, we are revising Figures 1 and 2 to adopt a more abstract representation, aiming to convey their intended message to readers more intuitively. Furthermore, we are refining the captions to improve overall readability. We will provide detailed updates on these revisions once they are finalized.

[R4-2] The main purpose of Theorem 1 (Composition Theorem) is unclear. The theorem lacks subsequent analysis or illustration, making it difficult to understand its application to BCQ or FlexBCQ. The section appears to discuss results related to FlexBCQ, but the text refers to BCQ. Further clarification on the detransformation function and analysis of these results is needed to grasp their significance.

Thank you for your comments regarding the Composition Theorem. The Composition Theorem is introduced to ensure that models quantized with FlexBCQ achieve the same inference speed and memory usage as the original BCQ without any additional overhead. To clearly illustrate this effect, we will compare the inference speed and memory usage with and without the application of the Composition Theorem.

[Background]

Before delving into the comparison, let me explain the storage and inference processes of UQ and BCQ:

1. Uniform Quantization (UQ). As illustrated in Figure 3(a), each weight is stored as a $k$-bit integer $\tilde{w}$, along with quantization parameters such as UQ’s scale factor $\Delta\_{(k)}$ and zero-point $z\_{U,(k)}$, for reconstruction. During inference, the weights $\tilde{\boldsymbol{w}}$ stored as integers are detransformed using the quantization parameters before continuing with further inference.

2. Binary-coding Quantization (BCQ). As illustrated in Figure 3(b), each weight is stored as a $k$-bit binary code $B$, and quantization parameters such as BCQ’s scale factors $\boldsymbol{\alpha}\_{(k)}$ and shifting factors $z_{B,(k)}$ are stored for reconstruction. During inference, these parameters are used to reconstruct the weights, after which the inference proceeds. (While the exact BCQ inference process may vary depending on the kernel, this description assumes the simplest method for explanatory purposes.)

3. FlexBCQ. As illustrated in Figure 3(c), FlexBCQ integrates UQ’s transformation process with BCQ’s mapping process. Each weight is stored as $k$-bit binary codes $b$, and quantization parameters include UQ’s scale factor $\Delta\_{(k)}$, zero-point $z\_{U,(k)}$, BCQ’s scale factors $\boldsymbol{\alpha}\_{(k)}$, and shifting factors $z_{B,(k)}$. During inference, FlexBCQ reconstructs weights in two steps: (1) reconstruction using BCQ’s parameters, followed by the (2) UQ's detransformation process. This additional step increases both the number of quantization parameters stored and inference latency compared to traditional BCQ.

[The role of Composition Theorem]

To mitigate these drawbacks, the Composition Theorem enables the combined reconstruction and detransformation processes of FlexBCQ to be streamlined into a single BCQ reconstruction process. This eliminates additional memory overhead and ensures that FlexBCQ achieves the same inference speed as standard BCQ. Consequently, the Composition Theorem provides a fundamental theoretical foundation that allows FlexBCQ to deliver its performance improvements without introducing additional latency or memory costs.

(Continued on the next response)

We appreciate your reviews. Below are our responses to your concerns. All of the contents in this response will be reflected in our paper, highlighted in blue color.

[R3-1] Guidelines for selecting hyperparameters in FlexBCQ.

As you correctly noted, gradient filtering and periodic remapping introduce two additional hyperparameters (one for each), potentially leading to increased hyperparameter tuning costs. To address this concern, we demonstrate throughout the paper that FlexBCQ achieves strong performance using only two fixed settings, without requiring extensive hyperparameter tuning for these components. Furthermore, additional performance gains can be achieved with further tuning.

To provide insights into how FlexBCQ’s performance varies with changes in hyperparameters, we conducted the following supplementary experiments.

[Gradient filtering]

We adopt a gradient filtering technique to exclude gradients that could negatively impact performance during optimization and introduce the hyperparameter $\tau$ as the filtering threshold. The selection of $\tau$ in FlexBCQ takes into account the mapping process and can be explained as follows:

1. Mapping in FlexBCQ: In FlexBCQ, the mapping process is performed in the space of transformed weights by mapping each weight to its nearest quantization level. If integers in the range $[0, 2^k-1]$ are used as quantization levels, as shown in Figure 2(a), this corresponds to uniform quantization (UQ). Alternatively, if a binary tree is used to represent non-uniform quantization levels, as shown in Figure 2(b), it corresponds to FlexBCQ.

2. Uniform Quantization for Illustration: For simplicity, consider the case of uniform quantization. Weights within the clipping range $[w_{m,c}, w_{M,c}]$ are transformed to real values within $[0, 2^k-1]$, while weights outside this range take on values beyond, e.g. $w_m$ and $w_M$. These out-of-range weights cause substantial mapping errors. In the transformed space, the interval between quantization levels in UQ is 1, and each level has a range of 0.5 on either side. Thus, it is reasonable to allow a margin of 0.5 even for the extreme levels $0$ and $2^{k}-1$.

3. Generalization to FlexBCQ: For FlexBCQ, the value of 0.5 is replaced by the smallest scale factor $min(\alpha\_{(k)})$ in BCQ. Therefore, $\tau$ is assigned as the smallest scale factor in each weight group, automatically providing a threshold suited to the group’s characteristics.

To evaluate the effectiveness of the $min(\alpha\_{(k)})$ criterion described in (3), we compare it against the UQ-based threshold of 0.5 from (2), as well as alternative thresholds of 0.25 (half of 0.5) and 1.0 (double of 0.5). The results of these experiments are presented in Table R3-1.

Table R3-1. MMLU benchmark accuracies of 3-bit Llama-3 models quantized by FlexBCQ with different gradient filtering threshold $\tau$. $min(\alpha_{(k)})$ represents that $\tau$ is equal to the minimum value of $\alpha_{(k)}$ in each weight group.

As a result, $min(\alpha\_{(k)})$, which is employed in our paper, shows the highest accuracy by providing the adaptive gradient filtering threshold for each weight group. Therefore, we recommend users to use $\tau=min(\alpha\_{(k)})$.

[R3-2] Potential generality of Unified Initialization.

FlexBCQ is the only algorithm designed to enhance the performance of BCQ by transferring training techniques from UQ. To support this novel category of algorithms, we propose Unified Initialization algorithm. A key advantage of FlexBCQ lies in its potential to transfer not only UQ techniques such as FlexRound but also other algorithms like OPTQ and OmniQuant into the BCQ framework. Although unified initialization cannot be directly applied to the quantization parameter initialization of conventional UQ algorithms, it becomes applicable when these algorithms are adapted to BCQ through the FlexBCQ methodology.

Given the development of fast inference kernels for BCQ [1,2] and the strong performance demonstrated by FlexBCQ, transferring various UQ techniques to BCQ represents a promising research direction. In this context, unified initialization is expected to play a critical role. We will incorporate this discussion into the paper and provide you with the exact sections where these revisions have been made.

[R3-3] Writing and readability of Mathematical expressions.

We appreciate your observation regarding the need to improve the readability of the paper. To address this, we will enhance the explanations of the equations to ensure that our intended message is communicated more clearly. In addition to your specific suggestions, we are also incorporating feedback from other reviewers to comprehensively refine the manuscript. These updates are underway, and we will inform you once the revisions are complete. All updates will be finalized within the discussion period.

[R3-4] Related works beyond UQ and BCQ.

Uniform Quantization (UQ) has become the most widely studied quantization format, supported by the number of publicly available kernels optimized for fast inference. Recent advancements [1,2], including LUT-GEMM[1], have demonstrated the ability to achieve faster inference speeds than existing UQ kernels at the same bit-width by leveraging the unique computational characteristics of Binary-Coding Quantization (BCQ). Given this context, our work focuses on enhancing the performance of BCQ, and we therefore restrict the scope of our paper to UQ and BCQ.

Below, we provide a summary of data on the inference speed of LUT-GEMM.

Table R1-4. Latency comparison of the first FFN layer on OPT-175B model with various precision and corresponding kernel selections with m = 12288 and g = 128 on A100-80GB-GPU (Table 1 in LUT-GEMM [1]).

One potential category to be included in this paper is rotation-related UQ techniques [3,4]. These techniques involve an additional transformation that multiplies the weight matrix with an orthonormal matrix and can also be applied to FlexBCQ; in QuaRot paper, they compare the performance of QuaRot+RTN and QuaRot+GPTQ, illustrating that QuaRot serves as an auxiliary technique (Table 1 in QuaRot). As such, our study does not include comparisons with these methods.

[References]

[1] Park, Gunho, et al. "Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models." arXiv preprint arXiv:2206.09557 (2022).

[2] You, Haoran, et al. "ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization." arXiv preprint arXiv:2406.05981 (2024).

[3] Ashkboos, Saleh, et al. "Quarot: Outlier-free 4-bit inference in rotated llms." arXiv preprint arXiv:2404.00456 (2024).

[4] Liu, Zechun, et al. "SpinQuant--LLM quantization with learned rotations." arXiv preprint arXiv:2405.16406 (2024).

Dear reviewer HKka,

We updated the revised version of our paper including all discussions to address your concerns. The overall improvements are summarized in the Global Response, and we have addressed the your feedback as outlined below.

• [R3-1] Hyperparameter selection: We include sensitivity analysis regarding $p$, $\tau$, sample dataset, sample size, epochs, and clipping strategies in Appendix C.3 to C.6.
• [R3-2] Unified Initialization: We include the potential of unified initialization in lines 509-511.
• [R3-3] Readability: We made every effort to improve the writing of the paper. The overall improvements are summarized in the Global Response [G2]. We clean the mathematical notations by removing redundant equations and introducing the notation $\Theta$ for representing a set of quantization parameters.
• [R3-4] Quantization schemes: We clearly state that our focus is on UQ and BCQ algorithms which are supported by LUT-GEMM in lines 99-104.

These are our responses to your feedback. We kindly ask that you review the responses and the revised paper. If you have any unresolved concerns, we would be happy to discuss them further.

[R2-10] In lines 160-161, is $s$, $s_r$ for a whole weight or a single weight block?

$s$ and $s_r \in \mathbb{R}^{+}$ are scale factors that divide each weight before mapping to enable flexible mappings. $s$ is assigned individually for each weight, while $s_r$ is assigned for each row in the weight matrix, i.e. all weights in a row share the same $s_r$. If we have a weight matrix $\boldsymbol{W}\in\mathbb{R}^{3\times 4}$, then we have twelve $s$ and three $s_r$. Note that our main idea is to adapt FlexRound, which is designed for UQ, to BCQ. Therefore, using $s$ and $s_r$ are the ideas of FlexRound and we follow their settings. We will clarify about $s$ and $s_r$ in lines 158-160.

These are our responses to your review. We sincerely appreciate your thoughtful feedback, which has brought to light several key aspects that were not sufficiently addressed in our original submission. We are actively working to incorporate your suggestions into our paper and will notify you as soon as the revisions are finalized.

[References]

[1] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International Conference on Machine Learning. PMLR, 2023.

[2] Ashkboos, Saleh, et al. "Quarot: Outlier-free 4-bit inference in rotated llms." arXiv preprint arXiv:2404.00456 (2024).

[3] Lin, Haokun, et al. "Duquant: Distributing outliers via dual transformation makes stronger quantized llms." The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.

[R2-8] Figure 1: What do the gray bars and blue arrows represent? This figure seems to suggest that a point can be attributed to quantized values, but I'm not entirely certain. Could you explain if this interpretation is correct?

In Figure 1. (b) and (c), we aim to represent that quantization levels (black bars) of BCQ are adaptive to the weights. Grey bars represent the uniformly distributed quantization levels, which are represented as black bars in (a). Blue arrows mean that BCQ moves the uniformly distributed quantization levels (grey bars) to become closer to the weights (circles).

[R2-9] (1) Are 4.(a) and 4.(b) for the same weight? (2) Are there any correspondences between their distributions? (3) It’s unclear if Figure 4(b) alone sufficiently demonstrates that FLEXBCQ outperforms FlexRound. (4) It is best to explain the y-axis of Fig. 4(b) in the title.

Our objective in Figure 4 is to illustrate that FlexBCQ successfully utilizes (1) adaptive quantization levels aligned with the weight distribution and (2) the flexible mapping optimization technique from UQ, as we intended. The overall contribution of (1) is highlighted in Table 4, while the contribution of (2) is examined through the ablation study in Table 3. Thus, Figure 4 provides a case study to visually confirm that (1) and (2) are operating as designed.

In particular, the goal of Figure 4(b) is not to demonstrate that FlexBCQ is superior to FlexRound. Instead, it aims to show that FlexBCQ effectively utilizes the flexible mapping technique introduced in FlexRound to the same degree as FlexRound itself.

Building on the explanation above, we have addressed your questions as follows.

q1. Are 4.(a) and 4.(b) for the same weight?

Figures 4(a) and 4(b) depict visualizations derived from the same weight matrix, specifically the up-projection weight matrix in the second block of Llama-3 8B, quantized to 3 bits. These figures highlight two weight groups within the matrix. While FlexBCQ successfully leverages both flexible mapping and adaptive quantization levels, not all weight groups are equally effective for visualization. Hence, we selected an arbitrary weight matrix and identified the weight groups that best visually demonstrate the impact of these techniques.

It is important to note that this does not imply that flexible mapping and adaptive quantization are confined to the selected weight group. Their contributions to overall model performance can be observed in Tables 3 and 4, respectively.

q2. Are there any correspondences between their distributions?

Thank you for providing an insightful analytical perspective. As you suggested, comparing the trends between effective adaptive quantization levels and frequent flexible mapping yields meaningful insights. However, since both techniques are optimized with the shared objective of reconstructing the outputs of Transformer blocks, disentangling their individual contributions and analyzing their trends independently poses significant challenges.

If we hypothesize that the effectiveness of adaptive quantization levels can be measured by minimizing weight-level quantization error, we can indirectly analyze its trend. Since flexible mapping aims to reduce block-level output reconstruction error rather than weight-level quantization error, these two factors are likely to exhibit a negative correlation.

q3. It’s unclear if Figure 4(b) alone sufficiently demonstrates that FLEXBCQ outperforms FlexRound.

The purpose of Figure 4(b) is not to demonstrate that FlexBCQ outperforms FlexRound, but rather to show that FlexBCQ effectively applies flexible mapping to the same extent as FlexRound. The figure illustrates that FlexBCQ achieves a comparable amount of flexible mapping, indicating its ability to leverage this optimization technique adequately.

To ensure that flexible mapping is not perceived as being confined to a single weight group, we conducted a statistical analysis of the extent of flexible mapping across all weight groups within the model.

Table R2-3. Comparison of the proportion (%) of weights undergoing flexible mapping in models quantized using FlexBCQ and FlexRound. The title of each column indicates the differences in the mapped indices between weights with flexible mapping and those without.

The results of this analysis confirm that FlexBCQ effectively utilizes flexible mapping not only in the weight group depicted in Figure 4(b) but also consistently throughout the entire model.

q4. It is best to explain the y-axis of Fig. 4(b) in the title.

We agree with your suggestion. We will explain the meaning of y-axis in the title of Fig. 4(b).

[R2-3] Clipping Strategy in Algorithm 1: In Algorithm 1, only the minimal values are iteratively clipped, rather than clipping both minimal and maximal values. Given that outliers could also appear on the maximal side, this may not be the optimal strategy. Please provide further explanation or an ablation study to justify this design choice.

We sincerely appreciate your detailed review of our algorithm. Our clipping range search algorithm fixes the minimum value while proportionally adjusting the maximum value during the search process. Before we begin the explanation, we need to revise line 4 in Algorithm 1.

• (Revised in line 4) $\Delta_{(k)} \gets (\gamma (w_M - w_m))/(2^{k}-1)$ and $z_{U, (k)} \gets \lfloor -w_m / \Delta_{(k)} \rceil)$

When we design Unified Initialization, there are three possible clipping strategies:

• The Fixed Minimum method. which fixes $w_m$ and adjusts only $w_M$
• The Fixed Maximum method. which fixes $w_M$ and adjusts only $w_m$
• The Balanced method. which adjusts both $w_m$ and $w_M$

The performance of these strategies for compressing the Llama-3 8B model to 3 bits is presented in Table R2-2. Experimental results reveal that while all three strategies deliver comparable performance, the Fixed Minimum strategy achieves the the higher score than the Balanced strategy. Therefore, Fixed Minimum, used in our paper, is a proper strategy for initialization. Furthermore, as shown in the table, you can further improve the performance of FlexBCQ if you use Fixed Maximum.

Table R2-2. MMLU 5-shot and 0-shot benchmark scores of 3-bit quantized Llama-3 8B models with three clipping strategies.

As you correctly noted, the Fixed Minimum approach may result in performance degradation if extreme outliers occur at the higher end of the range. However, in large language models (LLMs), it is well established that those extreme outliers predominantly occur in activations [1-3], and extreme outliers are rare in weights.

Based on the results of the R2-2 experiment, we have chosen to use the Fixed Minimum method. A discussion of this rationale will be added to Section 3.3 and Appendix B.1 (Implementation Details of FlexBCQ) of the paper. Furthermore, we will introduce an argument about clipping strategy in our source code to enable users to freely select the clipping strategies that they want to use.

[R2-4] I assume that the terms weight group and blockwise refer to the same concept in the context of FLEXBCQ, correct?

We acknowledge that the definitions of key terms in our paper may not have been sufficiently clear, and we apologize for any confusion this may have caused. To address this, we have organized the terminology used in the paper by ascending order of scale, starting from the smallest unit, "weight," to the largest unit, "model":

• Weight: the smallest unit, representing an individual numerical weight value.
• Weight group: a collection of weights grouped by a specified group size, all of which share the same quantization parameters.
• Weight matrix: a two-dimensional matrix composed of weights, containing multiple weight groups.
• Layer: a component that performs affine transformations using a weight matrix and a bias vector. For example, Q, K, V, O, U, G, and D in Table 4 of the paper represent individual layers.
• Module: a collection of layers that performs a specific function. In Transformers, modules include Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP).
• Block: a fundamental unit of a Transformer, consisting of one MHA module and one MLP module.
• Model: a complete language model composed of multiple blocks.

It is important to note that weight groups and blocks refer to different levels of grouping, with a block encompassing numerous weight groups. To improve clarity, we will refine the descriptions of these terms where they appear in the paper and include a comprehensive glossary in the Appendix.

We sincerely appreciate your meticulous review. Your detailed feedback, including insights into aspects not explicitly addressed in the paper, has greatly helped to enhance the clarity and readability of our paper. Below, we have systematically addressed the points you raised. All discussions from the discussion period will be reflected in the revised manuscript, with updates regarding readability to be included in the "Global Response" section upon completion.

[R2-1] Readability Issues: The reading experience could be improved; please refer to the Questions section for specific details.

We greatly appreciate your insightful and detailed questions. To address your concerns, we will revise the paper to ensure that the points you raised can be understood clearly by readers without requiring additional explanation. Readability improvements are currently underway, and once the revisions are finalized, we will provide a comprehensive summary outlining how your feedback has been incorporated.

[R2-2] Please include the bit width calculations in Table 1. A fair comparison with the baselines should account for these extra storage requirements.

Thank you for highlighting this crucial aspect. As you noted, BCQ offers higher representational capacity than UQ but requires a greater number of quantization parameters. Specifically, the average number of bits per weight for models quantized to 3 and 4 bits has been calculated for both UQ and BCQ, as summarized in Table R2-1. BCQ is observed to utilize approximately 8% more bits than UQ. Detailed calculations supporting these results are provided at the end of this response.

Table R2-1. The average number of bits per weight after quantization by BCQ and UQ under 3-bit and 4-bit weight quantization settings.

Regarding this observation, we would like to address the following two key points:

(1) In this study, we prioritize acceleration capabilities over model storage size. The LUT-GEMM kernel, which outperforms existing UQ kernels such as AWQ and GPTQ in speed, ensures identical inference speeds for BCQ and UQ. Therefore, we believe that comparing the accuracy of UQ and BCQ algorithms under the same bit-width conditions is a fair and reasonable approach.

(2) We define a "weight group" by grouping a specific number of weights and assigning quantization parameters to each group. As the weight group size increases, the proportion of bits allocated to quantization parameters becomes smaller relative to the total storage bits. For example, with column-wise quantization, compressing the Llama-3 8B model to 4 bits results in an average of 4.02 bits per weight, with quantization parameters contributing only 0.02 bits per weight—an almost negligible amount.

In line with this rationale, we base our comparisons on the bits used for weight compression rather than the average bits per weight. As per your suggestion, we will add details regarding the storage usage differences between UQ and BCQ, as well as the calculation of bits for quantized models, to the Appendix. Additionally, we will reference these bit differences when discussing Table 1. These updates are currently being implemented, and we will notify you once they are complete.

Below, we detail the process for calculating Table R2-1.

[Calculation of Table R2-1]

When a weight group $w\in \mathbb{R}^{g}$ consisting of $g$ weights is quantized into $k$-bit using either the UQ or BCQ format, the total number of bits required for storage is calculated as shown below. In these calculations, the quantization parameters ($\boldsymbol{\alpha}_{(k)}$, $z\_{B,(k)}$, $\Delta\_{(k)}$, and $z\_{U,(k)}$) are set to 16 bits, consistent with the value used in our study.

(# bits for BCQ) = $gk$ (for $\boldsymbol{b}\_{(k)}$) + $k$ * 16 (for $\boldsymbol{\alpha}\_{(k)}$) + 1 * 16 (for $z\_{B,(k)}$) = $(gk + (k+1)16)$

(# bits for UQ) = $gk$ (for $\widetilde{w}$) + 16 (for $\Delta\_{(k)}$) + 16 (for $z\_{U,(k)}$) = $(gk + 2 * 16)$

In this context, $\boldsymbol{b}\_{(k)}\in\\{-1,1\\}^{g\times k}$, $\boldsymbol{\alpha}_{(k)}\in\mathbb{R}^{k}$, $z\_{B,(k)}\in\mathbb{R}$ denote the binary codes for weights, BCQ’s scale factor, and BCQ’s shifting factor, respectively. $\widetilde{w}\in ([0,2^{k-1}]\cap\mathbb{Z})^{g}$, $\Delta\_{(k)}\in\mathbb{R}$, and $z\_{U,(k)}\in\mathbb{R}$ represent the weights mapped to integers, UQ’s scale factor, and UQ’s zero-point, respectively. Therefore, we have the average number of bits per weight for each quantization format as follows.

(average # bits per weight for BCQ) = $(gk + (k+1)16) / g$ = $k + 16(k+1)/g$

(average # bits per weight for UQ) = $(gk + 2 * 16) / g$ = $k + 32/g$

We appreciate your thoughtful and constructive review. We answer your reviews below. All modifications will be reflected in our paper during the discussion period.

[R1-1] It is claimed that prior work on BCQ (Xu et. al., 2018) independently updates $\boldsymbol{\alpha}\_{(k)}$ and $\boldsymbol{b}\_{(k)}$ are updated independently. Are there any ablation results that show how it performs for LLM quantization?

We experiment to compare the effect of updating $\boldsymbol{\alpha}\_{(k)}$ and $\boldsymbol{b}\_{(k)}$ to clarify which one has the greater impact on reducing the quantization error. Before we explain the experimental results, we clarify the updating process of Xu et al. [1]. Xu et al. begins with initializing $\boldsymbol{\alpha}\_{(k)}$and $\boldsymbol{b}\_{(k)}$ using Greedy algorithm (Equations (2) to (4) in Xu et al. [1], which is noted in Greedy in Table 1 in our paper]. After that, they alternatingly update $\boldsymbol{\alpha}\_{(k)}$ and $\boldsymbol{b}\_{(k)}$ by repeating the following two steps (Algorithm 2 in Xu et al. [1]).

[Step 1] Update $\boldsymbol{\alpha}\_{(k)}$ while keeping $\boldsymbol{b}\_{(k)}$ fixed

[Step 2] Update $\boldsymbol{b}\_{(k)}$ while keeping $\boldsymbol{\alpha}\_{(k)}$ fixed

In our statement that “$\boldsymbol{\alpha}\_{(k)}$ and $\boldsymbol{b}\_{(k)}$ are updated independently”, we intended to convey that one is fixed while updating the other. We measure the quantization error at each updating step to evaluate the impact of updating $\boldsymbol{\alpha}\_{(k)}$ and $\boldsymbol{b}\_{(k)}$. We summarize the experimental results in Table R1-1 and a detailed experimental setup is as follows:

[Setup]

• The number of alternating updates: 15 (same as the paper)
• Bit-width: 3
• Target weight: query projection’s weight in the first decoder block of Llama-3 8B
• Quantization error: $||\boldsymbol{W}-\widehat{\boldsymbol{W}}\_{(3)}||_F^2 \times 10^6$
  • $\boldsymbol{W}$: a pretrained weight, $\widehat{\boldsymbol{W}}\_{(3)}$: 3-bit quantized weight
• Reduction: (quantization error of the previous step) - (quantization error of the current step)
• Step 0 indicates the quantization error after Greedy initialization

Table R1-1. Quantization error in each update step over 15 iterations of alternating updates.

As a result, updating $\boldsymbol{\alpha}\_{(k)}$ reduces more quantization errors than updating $\boldsymbol{b}\_{(k)}$, showing the importance of finding proper quantization levels. Figure 4 (a) in our paper shows that FlexBCQ finds the proper quantization levels during optimization, resulting in high accuracy.

[R1-2] It might be worth considering ablations on the choice of calibration dataset, especially for Llama-3 models. Can you provide some justification as to how these datasets were chosen? Usually, for compression, subsets of the pre-training datasets are presumed to be good choices as calibration dataset. So was there any particular reason why C4 was chosen and not RedPajama (which is the pre-training dataset for OpenLlama)? Also, any ablations on the size of the pre-training dataset?

For smaller language models such as ResNet or BERT, it is standard practice to perform compression using a subset (or entire) of either the pretrained dataset [2] or the fine-tuning dataset [3], as you noted, and to evaluate the performance of the compressed models on the same dataset. However, for large language models (LLMs), which are designed to generalize well to unseen datasets, data not used during training can still be effectively leveraged for quantization. Moreover, evaluations are typically conducted on datasets that were excluded from training, such as the MMLU benchmark or GSM8K.

The table below summarizes the evaluation datasets used in major LLM quantization studies. As highlighted, the C4 dataset is the most commonly utilized and representative dataset in previous research. For FlexBCQ, we employed the C4 dataset to ensure consistency and fairness in comparisons with existing studies.

Table R1-2. Sample datasets used in previous LLM quantization algorithms.

To illustrate the effect of sample dataset size on the performance of a quantized model, we quantized the Llama-3 8B model to 3 bits using sample datasets of varying sizes and evaluated its performance on the MMLU benchmark. We use sample datasets ranging in size from 32 to 256, with 128 being the specific size used in our paper. Each sample dataset comprises 2,048 tokens. We summarize the result in Table R1-3.

Table R1-3. MMLU 0-shot and 5-shot accuracies of 3-bit quantized models

Experimental results indicate that the performance of the quantized model remains largely stable except in cases where the sample dataset size is extremely small (e.g., 32). This is attributed to the fact that even the largest sample dataset (with 256 samples), consisting of $5.24 \times 10^5$ tokens, represents only approximately one ten-millionth of the $1.5 \times 10^{13}$ tokens used to train the Llama-3 8B model [7]. As such, marginal increases in dataset size do not result in significant performance changes. Based on these observations, the dataset size of 128, used in our study, is appropriate for comparing the accuracy of different quantization methods.

We will explain the reason for selecting sample datasets in Section 4.1 (Experimental Setup). Additionally, we will include a discussion about the impact of sample dataset size on performance in Appendix.

[R1-3] Is it possible to provide a comparison of the average number of bits per parameter used by FlexBCQ, along with the other strategies it is being benchmarked against?

In the rightmost column of Figure 3 in our paper, we summarize the values to store after quantization through UQ and BCQ formats, and FlexBCQ stores identical values to BCQ according to the Composition Theorem (Theorem 1 in our paper).

When a weight group $w\in \mathbb{R}^{g}$ consisting of $g$ weights is quantized into $k$-bit using either the UQ or BCQ format, the total number of bits required for storage is calculated as shown below. In these calculations, the quantization parameters ($\boldsymbol{\alpha}_{(k)}$, $z\_{B,(k)}$, $\Delta\_{(k)}$, and $z\_{U,(k)}$) are set to 16 bits, consistent with the value used in our study.

(# bits for BCQ) = $gk$ (for $\boldsymbol{b}\_{(k)}$) + $k$ * 16 (for $\boldsymbol{\alpha}\_{(k)}$) + 1 * 16 (for $z\_{B,(k)}$) = $(gk + (k+1)16)$

(# bits for UQ) = $gk$ (for $\widetilde{w}$) + 16 (for $\Delta\_{(k)}$) + 16 (for $z\_{U,(k)}$) = $(gk + 2 * 16)$

In this context, $\boldsymbol{b}\_{(k)}\in\\{-1,1\\}^{g\times k}$, $\boldsymbol{\alpha}_{(k)}\in\mathbb{R}^{k}$, $z\_{B,(k)}\in\mathbb{R}$ denote the binary codes for weights, BCQ’s scale factor, and BCQ’s shifting factor, respectively. $\widetilde{w}\in ([0,2^{k-1}]\cap\mathbb{Z})^{g}$, $\Delta\_{(k)}\in\mathbb{R}$, and $z\_{U,(k)}\in\mathbb{R}$ represent the weights mapped to integers, UQ’s scale factor, and UQ’s zero-point, respectively. Therefore, we have the average number of bits per weight for each quantization format as follows.

(average # bits per weight for BCQ) = $(gk + (k+1)16) / g$ = $k + (k+1)16/g$

(average # bits per weight for UQ) = $(gk + 2 * 16) / g$ = $k + 32/g$

Based on the results above, the average number of bits per weight for UQ and BCQ at the 3-bit and 4-bit levels is summarized in the table below.

Table R2-1. The average number of bits per weight after quantization by BCQ and UQ when the bit width of weights is 3 or 4 bit.

BCQ requires approximately 8% more bits compared to UQ. However, as derived above, the average number of bits per weight for BCQ is given by $k + (k+1)/g​$. This indicates that the additional bit usage diminishes significantly as the group size $g$ increases. For example, if a single column, with 4096 weights, of the weight matrix in the Llama-3 8B model is treated as one group, the average number of bits per weight for 4-bit quantization is approximately 4.02. In this scenario, the bit consumption for the quantization parameters becomes negligible.

Moreover, in the context of recent studies on LLM compression [8], acceleration capabilities are prioritized over memory consumption. Notably, when utilizing LUT-GEMM kernels, BCQ achieves the same computational speed as UQ despite its higher average bit count. Therefore, we directly compare the accuracy of the quantized models in BCQ and UQ formats denoted in Tables 1 and 2 without losing fairness.

Discussion about average bits per weight of UQ and BCQ will be included in the Appendix during the discussion period.

[R1-4] Is it possible to use the kernels from LUT-GEMM to see an actual improvement in inference throughput?

The official implementation of LUT-GEMM [9] kernel is open-sourced in GitHub [10]. We reorganize some useful results in LUT-GEMM as follows to show the practical effectiveness of BCQ-quantized models.

Table R1-4. Latency comparison of the first FFN layer on OPT-175B model with various precision and corresponding kernel selections with m = 12288 and g = 128 on A100-80GB-GPU (Table 1 in LUT-GEMM).

* LUT-GEMM takes the same time for computing UQ and BCQ format values if they have the same bit-width

Table R1-5. End-to-end latency per token for Llama family models (Table 7 in LUT-GEMM).

Table R1-4 exhibits that LUT-GEMM outperforms conventional kernels that support only UQ kernels and Table R1-5 demonstrates that LUT-GEMM provides end-to-end inference acceleration for diverse sizes of Llama models. Therefore, BCQ format, which is supported by LUT-GEMM, is practically valuable.

We will include the discussion about the inference speed of LUT-GEMM in Appendix for completeness.

[R1-5] I initially had about why despite its greater expressive power, BCQ significantly underperforms compared to UQ, which has limited expressive capabilities for LLM compression (as mentioned in lines 67-68)? This was perhaps answered in 199-200 -- Is this a hypothesis and are there concrete datapoints to support this? I might have missed this in the reading of the paper.

Historically, the absence of efficient kernels for BCQ has limited studies regarding accurate BCQ algorithms for LLMs. Consequently, recent studies such as GPTQ, AWQ, and OmniQuant have focused on UQ, leveraging publicly available acceleration kernels. To the best of our knowledge, there have been no BCQ-based quantization techniques specifically developed for LLMs.

The existing BCQ algorithm applicable to billion-scale LLMs is Xu et al. (2018). However, this algorithm has notable limitations

1. Neglecting input distribution. The quantization process does not account for the input distribution, potentially leading to suboptimal performance.
2. Independently updating weight groups. Each weight group is updated independently, disregarding interdependencies among weights, which can adversely affect model accuracy.

These shortcomings are evident in Equations (2) through (5) of Xu et al.'s paper, where the absence of input considerations and the isolated treatment of weight groups are apparent.

In this context, the development of LUT-GEMM, a kernel capable of accelerating BCQ to match the speed of UQ while surpassing existing UQ acceleration kernels, marks a significant advancement; LUT-GEMM, which is published in ICLR’24, enables efficient BCQ inference for LLMs. We propose FlexBCQ, the first BCQ quantization algorithm tailored for LLMs, to leverage the effectiveness of LUT-GEMM. FlexBCQ offers several key improvements:

1. Considering input distribution. By accounting for input distribution during the quantization process, FlexBCQ effectively reduces reconstruction error.
2. Reflecting dependencies between weight groups. The algorithm simultaneously updates the quantization parameters in different weight groups, considering the dependencies between them.
3. Integration of UQ Training Techniques. FlexBCQ incorporates useful training methodologies traditionally used for UQ, e.g., flexible mapping, further improving its effectiveness.

As a result, FlexBCQ demonstrates superior performance, outperforming the approach by Xu et al. by up to 67%, as evidenced in Tables 1 and 2 of our paper.

We will revise lines 199-200 to clarify the limitations of Xu et al as follows.

• Before update: The only low-cost algorithm that is applicable to LLM (Xu et al. 2018) with BCQ scheme independently updates alpha and b, while ignoring the dependencies between them.
• After update: The only low-cost algorithm that is applicable to LLM (Xu et al. 2018) with BCQ scheme does not take input distribution into account and ignores dependencies of different weight groups, resulting in low accuracy.

[R1-6] There is a potential typo in the definition of $\Delta_{(k)}$. Perhaps it should be $(w_{M,c}-w_{m,c})/2^{k-1}$?

We appreciate your detailed review! We fix the typo and the fixed version will be available soon.

[R1-7] In Line 195 - 196, please point to a more precise Theorem/Lemma in Park et. al. for reference.

"Appendix C: Conversion of Uniform Quantization Into BCQ” in LUT-GEMM [9] contains the conversion rules for transforming UQ into BCQ. This clarification will be reflected in the revised version, which will be available for review shortly.

[R1-8] How is $\boldsymbol{\alpha}\_{(k)}$ chosen? How is the blockwise output reconstruction done?

We update the quantization parameters of FlexBCQ through the blockwise output reconstruction process where the block indicates a composition of an MHA and MLP modules; Llama-3 8B model has 32 blocks. The quantization parameters of weight groups in the same block are updated simultaneously using stochastic gradient descent (SGD) with our proposed two optimization techniques (Gradient Filtering and Periodic Remapping).

The details of our block reconstruction process are as follows:

• Objective function: $||B\_i(X\_i) - \widehat{B}\_i(\widehat{X}\_i)||\_F^2$

• Optimization target: quantization parameters of FlexBCQ

  • (UQ’s scale factor $\Delta_{(k)}$, UQ’s zero-point $z_{U,(k)}$, scale factors $s$ and $s_r$ for flexible mapping, and BCQ’s scale factor $\boldsymbol{\alpha}\_{(k)}$)

• Optimization algorithm: stochastic gradient descent (SGD)

• Additional optimization techniques: Gradient Filtering and Periodic Remapping

• Optimization strategy: sequential blockwise optimization from the bottom block to the top block

$B\_i$ and $\widehat{B}\_i$ are the functions for the $i$-th unquantized and quantized blocks, respectively. $\boldsymbol{X}\_i$ and $\widehat{\boldsymbol{X}}\_i$ are the inputs for the $i$-th unquantized and quantized blocks, respectively. We will elaborate more details about Blockwise output reconstruction in Section 3.4.

[R1-9] The gradient filtering section can benefit from a little more explanation. Does it mean that when the mapping error exceeds epsilon, the gradients are zeroed out?

You are correct. When the difference between the transformed weight $\bar{w}\_{(k)}$ and the mapped weight $\widetilde{w}\_{(k)}$ exceeds the gradient filtering threshold $\tau$, i.e. $|\bar{w}\_{(k)}-\widetilde{w}\_{(k)}| > \tau$, the gradient with respect to w is set to zero. Importantly, this process does not affect forward propagation.

We have explicitly clarified in lines 333-334 that the filtered gradient becomes zero. This clarification will be reflected in the revised version, which will be available for review shortly.

[R1-10] Please specify all the learnable parameters in FlexBCQ explicitly in a sentence somewhere.

FlexBCQ contains both UQ’s and BCQ’s quantization parameters to leverage both UQ’s training technique (flexible mapping) and adaptive quantization levels. Therefore, FlexBCQ has UQ’s scale factor $\Delta_{(k)}$, UQ’s zero-point $z_{U,(k)}$, scale factors $s$ and $s_r$ for flexible mapping, BCQ’s scale factor $\boldsymbol{\alpha}\_{(k)}$, and BCQ’s shifting factor $z_{B,(k)}$. We update all quantization parameters of FlexBCQ except for $z_B$. See response [R1-8] for details about updating the quantization parameters of FlexBCQ. We will specify the learnable parameters of FlexBCQ at the beginning of Section 3.4 (Blockwise Output Reconstruction) before explaining the optimization process.

[R1-11] The LLM quantization problem formalization in the beginning of Sec. 2.1 needs more description.

We modify Section 2.1 (LLM Quantization Problem) to provide more details about our problem. We specify that we are focusing on improving the accuracy of the quantized models in UQ and BCQ format since there is an efficient kernel (LUT-GEMM) that runs UQ and BCQ at the same speed. The following are the problem definitions before and after revision.

• [Before] We have an accurate LLM $f$, a desired bit-width $k$, and a sample dataset $\mathcal{D}$ of input token sequences. Our goal is to find an accurate $k$-bit quantized model $\hat{f}$

• [After] We have an accurate LLM $f$, a desired bit-width $k$, and a sample dataset $\mathcal{D}$ of input token sequences. Our goal is to find an accurate $k$-bit quantized model $\hat{f}$. In this paper, we focus on UQ and BCQ schemes. We directly compare the accuracies of UQ and BCQ algorithms since there is an effective kernel, LUT-GEMM, which runs UQ and BCQ at the same inference speed when they have the same bit-width.

These are our responses to your review. We hope our responses resolve your concerns. If you have any further inquiries or require additional clarifications, please feel free to ask us. Thanks,

[Reference]

[1] Xu, Chen, et al. "Alternating multi-bit quantization for recurrent neural networks." arXiv preprint arXiv:1802.00150 (2018).

[2] Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding." arXiv preprint arXiv:1510.00149 (2015).

[3] Park, Seungcheol, Hojun Choi, and U. Kang. "Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models." The Twelfth International Conference on Learning Representations. 2024.

[4] Raffel, Colin, et al. "Exploring the limits of transfer learning with a unified text-to-text transformer." Journal of machine learning research 21.140 (2020): 1-67.

[5] Gao, Leo, et al. "The pile: An 800gb dataset of diverse text for language modeling." arXiv preprint arXiv:2101.00027 (2020).

[6] Merity, Stephen, et al. "Pointer sentinel mixture models." arXiv preprint arXiv:1609.07843 (2016).

[7] https://ai.meta.com/blog/meta-llama-3/

[8] Song, Jiwon, et al. "SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks." arXiv preprint arXiv:2402.09025 (2024).

[9] Park, Gunho, et al. "Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models." arXiv preprint arXiv:2206.09557 (2022).

[10] https://github.com/naver-aics/lut-gemm

[G3] Exhaustive Verification of FlexBCQ

Many reviewers suggested additional experiments. We conduct all suggested experiments and summarize some key results below. Remarkably, FlexBCQ outperforms in every case. We appreciate all the reviewers for their excellent experimental suggestions, and we trust these results further demonstrate FlexBCQ's effectiveness. All experiments were conducted using Llama-3 8B.

Table G-1. Perplexities of quantized models.

Table G-2. 2-bit quantization results.

Table G-3. Channel-wise quantization results.

• 5-shot and 0-shot represent MMLU 5-shot and MMLU 0-shot benchmarks, respectively.
• Higher accuracies on MMLU and GSM8K benchmarks represents better performances
• Lower perplexities on WikiText-2 (Wiki) and C4 benchmarks represents better performance

Dear Reviewers,

We have updated the revised version of the paper to reflect all of your comments. We greatly appreciate your constructive feedback, and we believe the quality of the paper has been significantly enhanced, particularly in terms of readability, verification, and analysis. We sincerely thank you for your efforts and kindly ask that you review the updated version to ensure all of your suggestions have been adequately addressed. Should you have any additional points or concerns, we would be happy to address them.

Sincerely,

The Authors

Dear reviewers,

We have carefully incorporated your valuable feedback and summarized the updates in individual comments. The revisions made to the paper based on your comments are summarized in the Global Response [G2]. In [G3], we have presented compelling additional experimental results that highlight the superiority of FlexBCQ. We kindly ask you to review these updates before the discussion period concludes.

Sincerely,

The Authors