LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid

We appreciate the reviewer’s thoughtful feedback and recognition of our contributions to accuracy and scalability in LLM quantization. Below, we address the concerns raised and provide additional evidence. We hope these responses clarify the issues and encourage reconsideration of the score.

[W1] Quantization isn't solely about reducing memory usage or data transfer

Thank you for raising this point. While quantization does not inherently reduce the computational workload, it significantly accelerates LLM inference due to two key factors:

LLM Inference as an IO-Bound Operation: As highlighted in [1], LLM inference is primarily IO-bound rather than compute-bound. This means that loading weights into memory, rather than performing computations, is often the primary bottleneck for latency. By significantly reducing the size of weights, quantization alleviates this bottleneck and accelerates inference.

Reduced Communication Overhead in Multi-GPU Settings: When an LLM is split across multiple GPUs, quantization reduces the number of GPUs required for inference. This reduces the communication overhead between GPUs, further improving efficiency and scalability.

Existing research, such as [2], demonstrates that quantized models can achieve substantial speedups (2-4x) over full-precision models. Additionally, we have conducted new experiments benchmarking token throughput and peak memory usage for 2-bit, 3-bit, and 4-bit affine LeanQuant models for decoding 1K tokens using the exllamav2 kernel on an L40s GPU. The results clearly show that LeanQuant models achieve significant speedups over FP16 models, further substantiating the practical benefits of our method.

[W2] Quantizing to lower bit widths (like 3 or 2 bits) might not yield computational efficiency

We respectfully disagree. While it is true that packed low-bit integers are converted to floating-point numbers for multiplications, this conversion is performed directly within GPU kernels with negligible overhead. Many libraries such as exllamav2, llama.cpp, and Marlin use optimized fused GPU kernels to handle these operations efficiently.

Additionally, LLM inference is primarily IO-bound rather than compute-bound, as noted in prior works like [1]. The main bottleneck is loading weights from memory, and quantization significantly reduces global memory IO operations, leading to faster inference. As shown in the table above, LeanQuant achieves substantial improvements in both token throughput and memory usage for 3-bit and 2-bit models, by leveraging the fused kernel in exllamav2.

[W3] 3-bit or 2-bit weights may still be stored on the GPU in higher bit-widths

We respectfully disagree with this assertion. In practice, 3-bit and 2-bit weights are tightly packed in GPU memory, ensuring reduced memory usage without the overhead of higher bit-width storage. Many LLM inference libraries, such as exllamav2 and llama.cpp, employ efficient packing strategies to store low-bit weights compactly in GPU RAM. This efficiency gain is evident in our experimental results, as shown in the table above, where 2-bit and 3-bit LeanQuant models achieve significant memory savings compared to 4-bit models.

References

[1] Kim, Sehoon, et al. "SqueezeLLM: Dense-and-Sparse Quantization." Forty-first International Conference on Machine Learning.

[2] Guo, Han, et al. "Fast Matrix Multiplications for Lookup Table-Quantized LLMs." Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.

We appreciate the reviewer's thorough review and constructive feedback. Below, we carefully address your concerns.

[W1,Q1,Q6] How is LeanQuant applied to non-uniform quantization, and what are the key differences from affine quantization?

Thank you for this insightful question. LeanQuant solves a layer-wise optimization problem (Equation 1). Before applying iterative loss-error-based quantization to each layer, the quantization grid is learned and fixed.

The input for learning the quantization grid is the same for both non-uniform and affine quantization: a group of model weights, their corresponding inverse Hessian diagonals, and the parameter $p$. However, the key difference lies in the output and the learning process:

• Output Differences: Non-uniform Quantization: The output is a set of centroids, which are learned through k-means clustering (Equation 6). Affine Quantization: The output consists of a scaling factor and a zero-point, learned using enumerative search (Equation 7).

• Learning Objective: In both cases, the objective is to minimize the weighted sum of errors of the quantized weights, where the weights are scaled by the inverse Hessian diagonals. This ensures that the quantization process prioritizes preserving weights that are more important to model accuracy.

• Learning Methodology: Non-uniform quantization uses k-means clustering, which allows centroids to move freely to minimize the objective. Affine quantization employs enumerative search, as the grid points in affine quantization must be evenly spaced, making the process inherently discrete.

These differences in grid learning are detailed in lines 5–9 and 12–16 of Algorithm 1, which outline the distinct procedures for non-uniform and affine quantization in LeanQuant.

[W2,W3,Q2,Q3,Q7] Evaluations on Group-Wise Quantization and Comparison with AWQ

Thank you for highlighting the need for a more detailed evaluation of group-wise quantization and comparisons with AWQ. To address this, we conducted additional experiments on group-wise quantization, including a comparison with AWQ and OmniQuant. The Llama-2 OmniQuant models were obtained from the official repository, while the Llama-3.1 AWQ model was obtained from the official hugging-quants community repository.

The results, summarized in the table below, demonstrate that LeanQuant performs better on average across benchmarks than OmniQuant and AWQ in the group-wise quantization setting. These findings further validate the effectiveness and versatility of LeanQuant. We also provide more evaluations of group-wise quantization in Table 3 and Figure 3 of the paper.

[W4,W5,Q4,Q5] Runtime and Memory Usage Comparison

Thank you for your suggestion to provide a more detailed comparison of runtime and memory usage. Below, we present additional benchmarking results comparing LeanQuant with AWQ and GPTQ. For AWQ and GPTQ, we used the official AutoAWQ and AutoGPTQ packages.

The results indicate that LeanQuant is more memory-efficient and faster than AWQ, but less efficient in runtime compared to GPTQ. However, LeanQuant achieves higher accuracy than GPTQ, offering a trade-off between efficiency and precision. Although LeanQuant is slower than GPTQ, it remains highly scalable, as demonstrated by its ability to quantize one of the largest open-source LLMs to date (405B parameters). We will provide a more comprehensive comparison on memory usage and quantization time in the final paper.

We thank the reviewer for recognizing the strengths of LeanQuant and its comprehensive experimental design. We appreciate your insightful feedback and suggestions and address your concerns and questions in detail below.

[W1] Similar ideas of using "inverse Hessian diagonals" have been introduced in other works.

Thank you for pointing this out. While both LeanQuant and SqueezeLLM employ Hessian-based metrics to quantify the importance of weights, there are key distinctions in the formulation and application of these metrics:

Fundamental Difference in Metrics: SqueezeLLM uses the diagonal of the Fisher Information Matrix, derived from the squared gradient of the network loss. In contrast, LeanQuant leverages the inverse Hessian diagonals of the layer-wise optimization objective (Equation 1), computed directly from activations, eliminating the need for gradient calculations.

Scalability Advantage: The metric used by SqueezeLLM requires gradients computed across the entire model, making it resource-intensive, especially for large models. In contrast, LeanQuant decomposes quantization into a layer-wise optimization problem and calculates the Hessian using only the activations of each layer. This design dramatically improves scalability. For instance, as demonstrated in Table 4 of our paper, SqueezeLLM fails to quantize an 8B parameter model on a 48GB GPU, while LeanQuant successfully quantizes a 123B parameter model under the same constraints.

Comparison of Metrics: To address your suggestion, we conducted experiments comparing the performance of LeanQuant with the SqueezeLLM metric for learning the quantization grid, shown in the table below.

[Q1] Ablation with Other Importance Metrics such as the "Hessian Diagonal" in SqueezeLLM

We appreciate the reviewer’s suggestion to include ablations with other importance metrics. In the table below, we present additional experiments comparing the performance of LeanQuant’s metric with the "Hessian diagonal" metric used in SqueezeLLM for learning the quantization grid. The results show that LeanQuant’s metric outperforms SqueezeLLM’s metric in 3-bit and 2-bit settings while performing comparably in the 4-bit region.

Additionally, we highlight key advantages of LeanQuant in terms of scalability and versatility:

Scalability: SqueezeLLM is memory-intensive, requiring approximately 1.6 TB of GPU memory to quantize a 405B parameter model. In contrast, LeanQuant quantizes the same model using just 65.4 GB of GPU memory, making it far more practical for large-scale models.

Versatility: SqueezeLLM is limited to non-uniform quantization. LeanQuant, however, supports multiple quantization formats, including affine and non-uniform quantization, and performs favorably against competitive baselines in both cases.

[Q2] Lack of Literature Survey on Uniform and Non-Uniform Quantization in Related Works

We thank the reviewer for bringing this to our attention. We appreciate the reviewer’s suggestion and will include a comprehensive literature survey on uniform and non-uniform quantization in the final version of the paper.

We thank the reviewer for their thoughtful and detailed feedback, as well as their valuable suggestions. Below, we address each weakness and question in detail and are happy to provide further clarification if needed.

[W1] The paper assumes the readers are well-versed in quantization.

We are sorry for the confusion caused. We will add visualizations in the final paper to illustrate the concept of quantization grid and affine/non-uniform quantization clearly.

[W1.a] What exactly is a grid?

In the context of quantization, a grid refers to a set of values that represent the possible quantized outputs for the full-precision parameters. This naming is consistent with prior works [1]. To clarify the statement on Line 110, the process of quantization involves assigning each full-precision parameter to its nearest grid point on the quantization grid. For instance, consider a simple 2-bit quantization with grid points $[-1, -0.33, 0.33, 1]$. A floating-point weight of 0.25 would be mapped to 0.33, the closest grid point.

[W1.b] What does line 116 mean?

This statement refers to the most commonly used affine quantization scheme, where the grid points are distributed uniformly within the range of the weights being quantized. The dynamic range of the weights is defined as $[\min(W), \max(W)]$. For instance, if $\min(W)=-1,\max(W)=1$ with 2-bit quantization, the grid points would be evenly spaced as $[-1, -0.33, 0.33, 1]$.

[W1.c, W1.d] A simpler discussion/visualization on uniform and non-uniform grids is requested. Explanation on the quantization grid.

We thank the reviewer for bringing this to our attention. We will add figures and examples to the manuscript to illustrate the concepts more clearly.

[W1.e] Group of Weights

Quantization grids are determined based on the minimum and maximum values within a group of weights. Smaller group sizes result in higher quantization precision because the dynamic range is narrower, allowing for finer granularity in the grid points. The practice of grouping weights is standard in the quantization literature and has been widely adopted in prior works [2,3]. In our paper, we follow this established practice and group weights into blocks of 128 contiguous weights for the results presented in Table 3 and Figure 3.

[W2] Page 6 needs a simpler rewrite if possible.

We apologize for any confusion caused by the complexity of the current version. We will simplify the language and explanations on page 6 in the final version of the paper to improve readability and clarity.

[W2.a] The works seem to be indicating some problems with the grouping of weights using KNN.

Thank you for highlighting this point. To clarify, we are not indicating any issues with the grouping of weights using KNN. For $b$-bit non-uniform quantization, we need to learn a set of $2^b$ centroids in the quantization grid. In this context, we propose a learning objective in Equation 6 to preserve the precision of weights corresponding to outlier inverse Hessian diagonals by using k-means clustering.

[W2.b] The equation on [Line 285] does not have an equation number.

We apologize for this oversight and will add an equation number for clarity in the final version of the paper.

[W2.c] It's not clear what parts are already established and what parts are being proposed by the work.

We appreciate your feedback on this matter. Equations 6 and 7 are novel contributions of our work. They do not replicate any existing approaches, and they are designed to address specific challenges in preserving the precision of weights for outlier inverse Hessian diagonals.

Equation 6: This learning objective minimizes the weighted sum of squared errors between quantized weights and full-precision weights, with weights determined by the inverse Hessian diagonals.

Equation 7: While similar to Equation 6 in its goal, it introduces an enumerative search for the optimal parameters due to the discrete nature of affine grids.

[W2.d] Can this multi-threading technique be applied to existing work and make them more efficient as well?

To the best of our knowledge, existing works do not use a similar approach or formulation for learning quantization grids. Therefore, our fused GPU kernel cannot be directly applied to prior methods. However, we hope that our implementation and multi-threading approach will inspire future advancements in this area.

References

[1] van Baalen, Mart, et al. "Gptvq: The blessing of dimensionality for llm quantization." arXiv preprint arXiv:2402.15319 (2024).

[2] Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers." arXiv preprint arXiv:2210.17323 (2022).

[3] Lin, Ji, et al. "AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration." Proceedings of Machine Learning and Systems 6 (2024): 87-100.

I appreciate the author's effort in resolving all the issues. While I'm going through all the responses, the deadline for uploading the revised version is fast approaching. Is it possible to upload an updated version with all the requested changes?
It would significantly help to read the revised copy (major rewrite promised) to see if there are any lingering issues. Most of the issues were regarding non-clarity, which in the current form of the paper is difficult to resolve.

I fear, there isn't much space to incorporate all the figures and explanations into the main paper, retaining most of the issues of non-calrity in the final copy. E.g. W1) Figure with example [-1, -0.33, 0.33, 1] needs to be added, showing the significance of Equation 6. Also, Page 6 re-write. Will the grouping of weights explanation be added in the final copy?

We sincerely appreciate the reviewer’s thoughtful comments and valuable suggestions. Below, we provide detailed responses to the concerns raised.

[W1] Novelty of the Paper: Comparison with GPTVQ and GPTQ with Grid Range Search

Novelty in research is a multifaceted concept, but it can generally be categorized into two key aspects: empirical novelty, which entails uncovering previously unknown properties or phenomena, and technical novelty, which involves proposing new solutions or methods. We argue that our work demonstrates both types of novelty, as outlined below.

Empirically Novel Observation: High-Magnitude Outliers in the Inverse Hessian Diagonals We introduce a novel empirical observation: the presence of high-magnitude outliers in the inverse Hessian diagonals, which lead to significant quantization errors and model quality degradation in existing iterative loss-error-based quantization methods. This finding uncovers a previously overlooked aspect of LLM quantization, potentially guiding future advancements.

Technically Novel Approach: Accurate, Scalable, and Versatile Quantization Building on this empirical insight, we propose a new quantization method that improves model quality by utilizing loss-error-aware grids. Our approach is versatile, as it adapts to both affine and non-uniform quantization formats, and scalable, efficiently supporting large-scale models, such as Llama 3.1 405B, with moderate hardware resources.

Comparison with GPTVQ and GPTQ with Grid Range Search While GPTVQ extends GPTQ to vector quantization by using multi-dimensional centroids in each codebook, LeanQuant offers broader generalizability. Unlike GPTVQ which is limited to vector quantization, LeanQuant can seamlessly accommodate multiple quantization formats, including both affine and non-uniform grids. This versatility enhances the applicability of the method, making it compatible with popular frameworks such as llama.cpp and TensorRT-LLM, which currently do not support vector quantization.

Additionally, we provide a detailed comparison with GPTQ using grid range search in the response below, and also include comparisons with other recent methods.

[Q1] Comparison and Conceptual Differences with GPTQ Using MSE Range Search

We thank the reviewer for their insightful suggestion. In response, we conducted additional experiments comparing LeanQuant with GPTQ using MSE range search, presented in the table below. LeanQuant achieves superior performance on benchmarks and significantly improved scalability (7x faster quantization).

GPTQ with MSE range search employs an uninformed grid search strategy to minimize the mean squared error (MSE) between full-precision and quantized weights. In contrast, LeanQuant is motivated by the observation of outliers in the inverse Hessian diagonals. By specifically addressing these outliers, LeanQuant effectively mitigates loss errors and preserves model quality. Furthermore, to enable efficient scaling of this loss-error-aware grid learning to large models, we designed and implemented optimized GPU kernels.

[Q2] Did the authors consider the effect of weight value changes during iterative quantization on the k-means centroids?

We thank the reviewer for raising this insightful point. Yes, we have carefully considered this effect and provided a discussion in Section B of the Appendix to explain why it does not pose an issue in practice.

We thank the reviewer for carefully reviewing our paper and providing insightful feedback. Below, we address the concerns and questions raised.

[W1] Similarity to SqueezeLLM in using K-Means Clustering for Non-Uniform Datatypes

While the K-means clustering approach in LeanQuant shares some similarities with the sensitivity-weighted clustering in SqueezeLLM, there are key differences that we would like to emphasize:

Motivation from Inverse Hessian Outliers: LeanQuant is specifically motivated by the outliers in inverse Hessian diagonals, a characteristic unique to iterative loss-error-based quantization approaches. This focus ensures better preservation of weights that may introduce high loss errors during quantization.

Scalability: LeanQuant is designed to be far more scalable than SqueezeLLM. By leveraging layer-wise optimization, LeanQuant can quantize models with up to 123B parameters on a single 48GB GPU, whereas SqueezeLLM fails to quantize an 8B model.

Versatility: LeanQuant is applicable to a wider range of quantization formats, including both affine and non-uniform quantization, while SqueezeLLM is limited to non-uniform formats.

[Q1] Intuition for Why Uniform Grid Point Spacing Improves Accuracy

Thank you for this thoughtful question. The main intuition lies in the distribution of weights:

Weight Distribution Characteristics: Weights tend to be densely distributed near the center and sparsely distributed at the extremes. Standard initializations such as random and k-means++ tend to undersample these extreme values, which may fail to adequately represent these sparsely populated regions.

Uniform Grid Initialization: By initializing the centroids uniformly across the range of minimum and maximum weights, LeanQuant ensures that extreme values are adequately represented in the quantization grid without compromising the precision of non-extreme values.

We plan to include illustrations in the final paper to further clarify this process.

[Q2] Efficiency of K-Means Non-Uniform Quantization Grid Search

The K-means grid search process reported in Table 9 was conducted on the CPU. However, this process is fully parallelizable on the GPU, which would enable significant speedups. We would like to adopt this approach in the near future.