Adaptive Quantization Error Reconstruction for LLMs with Mixed Precision

R1: For comparison with other advanced methods, we utilize consistent evaluation datasets. Additionally, we apply AMLQ to multiple business datasets for text2sql and docqa tasks. The comparative results against GPTQ and AWQ are as follows:

Q1: The quantity and diversity of a calibration dataset indeed affect the quantization results. Our experiments show that when the calibration data size exceeds 128, the performance gap in the quantized models is very small. However, as the volume of calibration data decreases, there's a notable decline in performance, a perspective also discussed in papers such as AWQ. Moreover, since our method is based on GPTQ, the optimal calibration dataset must be sampled from the entire training data, primarily impacting model generalization. From our experiments, this effect on the results is minimal. In this section, we make a fair comparison by referring to the settings from related papers.

Q2: As described by Eq. 2, the normalized error sensitivity is defined as $\delta = \|WX-W_qX\|_2^2 / \|WX\|_2^2$. Fig. 1(a) aims to illustrate that the error sensitivity differs among weight types and layers. For the same type of weight, such as k_proj, there is a significant discrepancy between the beginning, middle, and end. Additionally, there are substantial variations between different weight types, such as k_proj and down_proj, even within the same layer. Fig. 1(b) illustrates the impact of applying quantization error reconstruction methods across various bits per weight (bpw), demonstrating significant performance improvements at lower bit widths. In Fig. 1(c), SVD represents the direct calculation of the rank of $\Delta W = W - W_q$. L^2QER scales $\Delta W$ then calculates its rank. AMLQ determines the rank of the covariance matrix of $\Delta Y$ from Eq. 7-10.

Q3: Fig. 2 shows the search results for bits per weight (bpw) and the required rank of the error reconstruction matrix for each q_proj layer in llama2-13b, which is also related to error sensitivity. Higher sensitivity in intermediate layers necessitates higher average bpw. For example, in layers 28-36, using the highest average bpw results in smaller quantization errors, so only the minimum necessary rank is used for error reconstruction, indicating no need for additional low-rank matrices.

R1: Your perspective is highly valuable. We've strategically designed AMLQ to minimize the total quantization error across linear layers while keeping the overall average bits per weight (bpw) under a predefined target. Through experimental observation, we identify that different layers have varying sensitivities to quantization. Accordingly, we tailor bit allocation to minimize total quantization errors—assigning lower bpw values to shallower, less sensitive layers, and higher bpw values to deeper, more sensitive ones.

R2: We apologize for any confusion. Group size, mixed precision strategy, and proportions are critical for determining the layer's bit width for quantization. For example, dividing a (5120,5120) weight layer into 128-sized groups yields 40 groups. Quantizing 40% at 2 bits, 30% at 3 bits, and the rest at 4 bits averages to 3.3 bpw. The smaller groups require more quantization parameters, leading to a larger bit width. We select group sizes as multiples of 32 to optimize CUDA kernel efficiency, typically requiring smaller groups for lower precision but greater proportions. Therefore, we set three group sizes (32,64, and 128) to match the mixed precision quantization. Tables 1 and 2 validate our method.

Q1: Both our method and LQER use two low-rank matrices to approximate a decomposed matrix, but while LQER decomposes the quantization error matrix $\delta_W$, we decompose $\delta_Y$'s covariance matrix (in Equations 10). Lower matrix rank means less error in approximation, and our experiments show that activation values are low-rank, meaning the rank of $\delta_Y$ is lower compared to the quantization error matrix $\delta_W$. Fig.1(c) in our paper visualizes this low rank, supporting our method's superior error reconstruction.

Q2: Table 1 compares the perplexities on WikiText-2 of AMLQ with other strong baselines, using results from their original papers. LQER has no results below 4-bit and its closed-source code prevents reproduction for 2-3 bit results. Figure 2 shows lower bit widths for shallower layers and higher for deeper ones in the llama2-13b model, aligning with k_proj’s quantization error trends in Figure 1(a). Our greedy binary search might not find the best quantization combination, but better strategies and layer-wise fine-tuning could improve results. We will optimize the search strategy for a balance between effectiveness and efficiency. Overall, AMLQ performs better in most cases.

R1: The primary advantage of our proposed AMLQ lies in its ability to maintain quantization effectiveness and inferencing speed under ultra-low quantization bits while achieving SOTA average scores on downstream tasks. As seen in Table 1, AMLQ significantly outperforms most existing quantization approaches, with only QUIP# ranking higher. However, QUIP# employs layer-wise fine-tuning and an E8 codebook. AMLQ can save 10x quantization time and increase inference speed 3x compared to QUIP#. In Table 2, AMLQ achieves SOTA overall scores across five downstream tasks, demonstrating its superior performance in practical deployment applications.

Q1: (1) In the public evaluation datasets presented in Table 1, most existing quantization methods perform well at 4 bits, with AMLQ ranking among the top. Notably, OmniQuant and QUIP#, which show comparable effects to AMLQ, require backpropagation to update parameters during quantization, making AMLQ more advantageous in terms of quantization time and computing resource utilization. Furthermore, our AMLQ exhibits a significant performance advantage under 2-3 low-bit quantization (shown in Table 1).
(2) We wholeheartedly agree with your perspective. As you mentioned, the GPTQ method is an excellent work from an earlier time, yet it underperforms in ultra-low bit quantization scenarios. In contrast, AMLQ excels in ultra-low bit quantization applications. In Table 1, we showcase the quantization performance at 2-3 bits across various datasets. AMLQ achieve substantial improvements over existing strong baseline models and performed comparably to the QUIP# method. It’s worth noting that QUIP# utilizes layer-wise fine-tuning and an E8 codebook, whereas AMLQ can save 10x quantization time and improve 3x inference speed compared to QUIP#. Furthermore, according to experimental results from the QUIP# paper, AMLQ's performance surpasses that of QUIP#'s non-fine-tuned version in most scenarios.

R1: Our paper addresses two key issues with prior quantization methods. First, we tackle varying sensitivity to quantization across different layers and channels by proposing an adaptive mixed-precision approach. Our method enables a theoretically superior combination of precisions, proven to outperform uniform precision techniques. Second, we counter the significant accuracy drop in low-bit conditions witnessed in previous methods with our novel low-rank quantization error reconstruction. Through theoretical analysis and empirical evidence, we show how output activations are low-rank and how they can be leveraged to mitigate quantization errors. This validates the effectiveness and theoretical soundness of our approach.

R2: Our method innovates for LLMs by capitalizing on uneven quantization sensitivity across LLM layers and the advantageous activation value reordering strategy, similar to that in GPTQ. The adoption of low-rank matrix error reconstruction improves quantization, particularly at ultra-low bits. Our low-rank assumption leads to a unique matrix initialization that requires minimal data, divergent from backpropagation-reliant methods, and is especially effective post-training, tailored for LLMs.

R3: Our AMLQ method offers time and performance superiority to QuiP#, which succumbs to long quantization times and complex design. Despite marginal improvements QuiP# may demonstrate due to its unique codebook design, AMLQ surpasses conventional methods like GPTQ, AWQ, and OmniQuant, showcasing its broad empirical superiority.

R4: Facing a large search space, we adopt a relative optimality approach with our iterative, bisecting greedy strategy. This balances optimality and computational efficiency by systematically narrowing down the layer combinations, guiding us towards a practical solution within the target bits-per-weight.

R5: We will soon provide a comprehensive speed analysis for the Llama2-13B-chat model, with preliminary tests highlighting our solution's inference speed superiority. At 2.5 bits, our method is significantly faster than QuiP# at 2 bits and a non-quantized model at fp16, achieving 46.08 tokens/s, which further increases to 117.7 tokens/s with our specialized inference framework.

Q1: The experiments in Fig. 1 are carried out in llama2-13b. The calibration dataset is sampled from the WikiText2. Due to space limitations, please refer to the detailed explanation of Fig. 1 in Reviewer 4's Q2.