We greatly appreciate the comments and questions of the reviewer. We thank the reviewer for acknowledging our locally optimal block clustering approach for quantization that achieves current state-of-the-art during 4-bit quantization of both weights and activations across several LLMs and datasets.

The reviewer has raised valid criticism regarding the terminology of block quantization vs group quantization. First, we would like to apologize for this confusion and clarify that the group quantization and block quantization methods are the same. As quantization for LLMs is a rapidly evolving field, there hasn’t been a consensus on unified terminology for this approach. Further, we would like to emphasize that LO-BCQ improves over group-quantization that typically quantizes each group to a given number format by allowing each group to map to a set of codebooks that minimize quantization error.

To demonstrate the advanced nature of our method, we perform detailed comparison against the weight-only quantization proposals listed by the reviewer. While the techniques in Omniquant, GPTQ, AWQ and QuiP have been demonstrated in weight quantization, we would like to highlight that LO-BCQ can be applied to both weights and activations. That is the codebooks we calibrate through LO-BCQ are universal across operand types and models. Further, while LO-BCQ is a post-training quantization (PTQ) method, GPTQ and QuiP involves finetuning to recover accuracy loss from quantization. We have compared the Wikitext perplexity loss of LO-BCQ against these works in Figure 1 of our paper.

In the table below, we compare weight-only LO-BCQ to AWQ and GPTQ, all with group size = 128 for a fair comparison. As shown, LO-BCQ achieves comparatively lower loss in perplexity. Further, we evaluate this loss on Wikitext-103 dataset, which is a much larger dataset compared to Wikitext2 which is used by GPTQ and AWQ.

Wikitext PPL Loss Compared to Unquantized Baseline

Additionally, in the table below, we compare against Omniquant and QuiP across various datasets such as Wikitext, PIQA and Winogrande. As shown, LO-BCQ achieves significant perplexity improvement over OmniQuant and QuiP. Further, the loss in accuracy in PIQA and Winogrande datasets are maintained to be <1% with 8 codebooks.

Wikitext PPL Loss Compared to Unquantized Baseline

PIQA Accuracy

Winogrande Accuracy

Finally, we quantitively compare LO-BCQ to GPTQ[6], which is an effective method for 4-bit weight-only PTQ. Here, GPTQ quantizes weights to 4-bits with group-size of 128 while maintaining activations in 8-bits. In contrast, LO-BCQ quantizes both weights and activations to 4-bits with group size of 64. With >=8 codebooks and effective bitwidth of 4.5 and 4.625, respectively, LO-BCQ achieves lower perplexity loss compared to GPTQ despite quantizing activations to 4-bits. Further, we evaluate LO-BCQ on Wikitext-103, which is a significantly larger datatset compared to Wikitext-2 used in evaluating GPTQ. By demonstrating lower perplexity loss in larger dataset, we show that LO-BCQ is a significantly more efficient quantization method compared to GPTQ.

PPL Loss Compared to Unquantized Baseline

References: [1]: Rouhani et al, “Microscaling data formats for deep learning, 2023” [2]: Rouhani et al. “With shared microexponents, a little shifting goes a long way”. ISCA ’23 [3]: Keller et al, “A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nm”. JSSC’23 [4]: Chee et al, “QuIP: 2-Bit Quantization of Large Language Models With Guarantees, 2024 [5]: Egiazarian et al, “Extreme compression of large language models via additive quantization, 2024.” [6]: Frantar et al, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers”

Since there is no feedback, I keep my rating to reject the work.

LO-BCQ (Group Size=64): Loss compared to unquantized baseline (lower is better)

ATOM[1] (Group Size = 128): Loss compared to unquantized baseline (lower is better)

MX4[2] (Group Size = 16): Loss compared to unquantized baseline (lower is better)

MXFP4[3] (Group Size = 32): Loss compared to unquantized baseline (lower is better)

3. Impact on language-modeling task (perplexity on Wikitext-103) with universally calibrated codebooks: We calibrate our codebooks on activations from GPT3-126M model and freeze the codebooks across our evaluations. As demonstrated by the perplexity achieved by LO-BCQ on Wikitext-103 dataset, the pre-calibrated codebooks are effective in quantizing activations of various models we considered.

Wikitext-103, Group size=64 : Loss compared to unquantized baseline (lower is better)

To validate this further, we compare the perplexity achieved by codebooks calibrated individually for each operand (weights and activations) in each layer of Llama2-7B. As shown, the universally calibrated codebooks perform as well as layer-wise calibrated codebooks especially with >= 8 codebooks.

Wikitext-103, Group size=64, Llama2-7B: Loss compared to unquantized baseline (lower is better)

4. Impact of clustering, indexing and look-up steps of LO-BCQ on inference performance:

We agree with the reviewer that the clustering and identifying codebooks is an expensive procedure and will impact inference performance. However, we would like to clarify that the clustering and calibration of codebooks is performed prior to inference (offline). We freeze the codebooks identified by LO-BCQ algorithm across our evaluations.

Further, the codebook selectors are pre-determined for the weights and are computed on-the-fly for activations. The cost of determining the codebook selector is amortized by the large size of dot-product dimension (e.g. 8192 in Llama2-70B).

Finally, each 4-bit scalar is decoded to 6-bit integer during GEMM/GEMV computations. As a result of a small number of codebooks utilized by LO-BCQ (footprint = 0.19KB), one can potentially store them in shared memory of each SM resulting in a small look-up cost. A dedicated hardware design can amortize the cost of this lookup through spatial and temporal re-use of operands.

We thank the reviewer for their thorough comments and questions. We are glad to know that the reviewer acknowledges our extensive experimental evaluations across several LLMs and datasets. We also appreciate the reviewer emphasizing the fact that we have achieved <1% accuracy loss in several inference tasks when quantizing both weights and activations to 4-bits.

The main concern of the reviewer is regarding the group-size or ‘block division’ of quantization. We agree with the reviewer that a small group-size such as 8 can lead to large computational overheads due to high precision scale factor multiplication per-group. We would like to clarify that the block-array size denoted by ‘$L_A$’ in our work is equivalent to group-size, that is, each block array shares a scale-factor. Therefore, in our work, scale factors are shared among groups of size 64, rather than 8, which has much smaller overhead. Indeed, one of the strengths of our work is using different granularities for scaling and codebook indexing (the latter which has size of 8). Therefore, LO-BCQ leverages the benefits of finer-grained quantization while avoiding a prohibitive overhead associated with scaling factors. We clarify this in more detail in part 1 of our response.

We agree with the reviewer that Atom[1], a group-quantization based proposal, demonstrates impressive results across several inference tasks. We have now cited this work in our paper. With LO-BCQ, we demonstrate comparatively better inference accuracy than previous group quantization proposals across several benchmarks, including Atom[1] in part 2 of our response. The superior accuracy achieved by LO-BCQ can be attributed to flexibly mapping each block to a codebook that achieves least quantization error. In contrast, group quantization proposals associate a single number format to each group. Additionally, the reviewer has asked an important question regarding the calibration of codebooks for activations that are typically generated on-the-fly during inference. We have provided the performance of LO-BCQ on language modelling task with universally calibrated codebooks and compared against calibrating layerwise in part 3 of our response.

Finally, the reviewer has raised valid concerns regarding the performance of our clustering procedure to identify the locally optimal codebooks. We would like to clarify that these steps are performed during the calibration phase (offline) and does not impact the latency of inference. We discuss this further in part 4 of our response.

1. Block-size ($L_b$) vs Block Array size ($L_A$): LO-BCQ shares a codebook per-$L_b$-element block and a scale-factor per-$L_A$-element block array. Here, block array is equivalent to “group size” in previous proposals such as Atom[1]. During inference, we perform $L_A$ wide inner dot-products which are scaled by the per-block array scale factors. Each 4-bit scalar within the block array is decoded into a 6-bit integer by looking up a 16-entry codebook. This codebook is selected by the codebook selector that is shared by each block of $L_b$ (typically = 8) elements within the block array. We would like to emphasize that, as a result of a small number of codebooks (<= 16) in our work, the cost of this lookup is considerably small compared to other codebook-based proposals such as QuIP#[4] and AQLM[5] where the number of codebooks are as large as $2^{16}$.

2. Comparing LO-BCQ to other group quantization proposals: We present the inference accuracy achieved by LO-BCQ with a large group size of 64 across several datasets and models. Across these benchmarks, we demonstrate significantly better inference accuracy compared to other group quantization proposals such as Atom[1], MX[2] and MXFP[3]. We find that with a group size of 64, LO-BCQ achieves significantly lower perplexity loss on Wikitext datatset compared to other works. We share an 8-bit scale-factor across 64 element block arrays, and each block of 8 shares a codebook selector. With 2, 4, 8 and 16 codebooks, the effective bit-width of LO-BCQ is 4.25, 4.375, 4.5 and 4.625, respectively. Similarly, we demonstrate significantly lower accuracy loss in datasets such as PIQA, BoolQ, HellaSwag and Winogrande. The superior performance of LO-BCQ is from effectively designing codebooks that minimize quantization error and flexibly mapping each block to a codebook that best quantizes it.

Additionally, with a group size of 128 for LO-BCQ, we would like to present a comparison to Atom. Here, LO-BCQ with $2$, $4$, $8$ and $16$ codebooks result in an effective bitwidth of $4.19$, $4.31$, $4.44$ and $4.56$, respectively, for both weights and activations.

Wikitext PPL Loss compared to unquantized baseline

We greatly appreciate the comments and questions of the reviewer. We thank the reviewer acknowledging that our codebook-based approach is universally applicable for quantizing any tensor, at any layer, across different models.

We have conducted some preliminary studies with LO_BCQ for sub-4-bit quantization of weights (W2A4 quantization). In this regime, we find that quantization-aware training is needed to bridge the accuracy gap from unquantized baseline. Further, reducing the block array size ($L_A$) and block size ($L_b$) result in improved accuracy at the cost of increased effective bit-width. This is because at smaller $L_A$ and $L_b$, the per-block array scale-factors and per-block codebook selectors are shared among fewer scalars.

The reviewer is correct is pointing out that when $L_A$ is large, the range of values within some codebooks are smaller than [-1,1] and these codebooks are used primarily for blocks with relatively small values. Although we have not experimented with $L_A$ smaller than 16, we would like to discuss the general trends. We find that the inference accuracy improves as $L_A$ decreases at the cost of increased bit-width overhead for storing the per-block array scale factors. Similarly, the inference accuracy improves when $N_c$ (number of codebooks) increases at the cost of increased bit-width overhead for storing the per-block codebook selectors. Empirically, we find that it is worthwhile to increase the number of codebooks $N_c$ rather than decrease $L_A$. For, instance the table below reports the Wikitext-103 perplexity on Llama2-70B model with various $L_A$ and $N_c$ values for $L_b$=8. With $L_A$=64, increasing the number of codebooks to 16 results in better perplexity (lower is better), compared to decreasing $L_A$ to 16, where the effective bit-width in both these configurations is 4.625.

Wikitext-103 perplexity on Llama2-70B model