SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

W1: The SBA Utilize KL divergence of block outputs as a precision allocating metric, depend on a certain calibration dataset. Generalization on datasets is not mentioned, so the method maybe effective in the same dataset.

Dear Reviewer ouaC,

We present extensive cross-domain experimental results demonstrating that the choice of calibration dataset has minimal impact on the overall performance of LLM quantization and compression. Therefore, we follow prior high-quality quantization studies, such as GPTQ [1], by selecting the semantically rich and diverse Wikitext2 dataset for calibration, minimizing potential bias introduced during calibration.

In addition to comparing the PPL performance of the quantized model on Wikitext2, we comprehensively evaluated its performance across eight zero-shot benchmarks. As shown in Table 3, our method outperforms GPTQ and OmniQuant in quantizing LLaMA-series models on datasets such as PIQA, ARC, BoolQ, Hellaswag, and Winogrande. Our results consistently lead across these datasets, indicating that calibration dataset selection did not negatively impact performance.

We also note that recent work [2] has explored the effect of calibration data on LLMs. Based on the findings in [2], selecting Wikitext2 as the calibration dataset does not significantly affect final model performance. Furthermore, this slight calibration influence diminishes as model size increases.

[1] Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022).

[2] On the impact of calibration data in post-training quantization and pruning. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

W2: The SQC introduce a novel quantizer by introducing τ to reduce reconstruction during quantization. The 3 - σ rule and 1% ratio is based on a common observation, not a heuristic method. this theoretical basis is not very strong. Dynamic ratio by layer maybe better.

We acknowledge that dynamically setting the salience ratio could potentially provide a more optimal search space. However, this approach comes with significant computational overhead. Using an empirical method such as the 3-σ rule allows for a more efficient quantization process.

We greatly appreciate your valuable suggestions. While a dynamic ratio setting could indeed increase the search space and improve the likelihood of identifying a better-performing ratio, it would also incur greater search costs and computational burden. This would extend the time required to quantize FP16 mixed-precision weights into low-bit integers, potentially undermining the efficiency advantage of PTQ in terms of quantization overhead.

In addition to considering the cost of dynamic ratio settings, we should emphasize that our current ratio selection method already achieves strong performance. As shown in the ablation studies in Figure 5 of our paper, the chosen ratio effectively reduces quantization error and minimizes the information loss of salient weights. This demonstrates that our current approach is both efficient and effective.

That said, building on the current setup, we will continue to explore improved ratio configurations in future work.

W3: Limited comparison with other extremely low-bit quantization methods. For instance, "Ma S, Wang H, Ma L, et al. The Era of 1-Bit LLMs: All Large Language Models Are in 1.58 Bits [J]. arXiv preprint arXiv:2402.17764, 2024." This method is popular and cuser-friendly, as it is supported by Hugging Face.

The paper The Era of 1-Bit LLMs is indeed a valuable contribution to the field of low-bit quantization for LLMs, providing the community with a method for ultra-low-bit quantization and making it openly available on platforms such as Huggingface. However, unlike our work, this paper employs a QAT approach that involves training from scratch, which typically requires significantly more time and GPU resources to quantize the weights.

In contrast, our PTQ method only requires 50 minutes on a single GPU, offering a different and orthogonal approach to exploring ultra-low-bit quantization for LLMs. The two methods are not mutually exclusive. Our work focuses specifically on PTQ-based ultra-low-bit mixed-precision quantization for LLMs, as detailed in Section 1 of our paper. Thank you for your suggestion. We will include a citation and discussion of this paper in our revised manuscript, highlighting the distinctions and connections between our method and QAT.

Q1: Is PTQ practical for ultra-low bit quantization in the case of significant dropout?

We believe that PTQ research in the field of ultra-low-bit quantization is highly valuable, especially in resource-constrained environments. While existing PTQ methods have achieved significant success at conventional bit widths (>3-bit), ultra-low-bit quantization still requires exploration of more effective solutions.

As shown in Table 3 of our paper, our method demonstrates notable improvements over others at ultra-low-bit settings, achieving around 2-point gains on various benchmarks such as PIQA and ARC compared to GPTQ[1] and AWQ[2]. Additionally, as stated in the paper, our structured quantization approach is hardware-friendly, further supporting the effectiveness of our work in advancing PTQ quantization performance at ultra-low bit widths.

Although QAT methods, which involve training on more comprehensive datasets, can achieve better performance, their training time is typically measured in days. In contrast, PTQ offers high efficiency, low resource usage, and the ability to run end-to-end on resource-constrained devices. Based on our findings, we believe PTQ still has significant potential for further exploration in compression.

Through this work, we aim to provide more insights and practical experience in PTQ quantization to the community, contributing to the advancement of this promising technology.

[1]Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers." arXiv preprint arXiv:2210.17323 (2022).

[2]Lin, Ji, et al. "AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration." Proceedings of Machine Learning and Systems 6 (2024): 87-100.

Q2: Is there any details about how exactly does $τ$ expand the solution space of $s$ and $z$?

Please check the reply for Weakness2.

Q3: How do you justify using KL divergence as objective function?

Before applying the practical KL divergence, we first transform the weight outputs into a probability distribution using softmax and then perform the computation. We believe that KL divergence, based on the concept of information theory, provides a measure of the information content between two distributions. This makes it particularly useful in certain cases, offering a deeper understanding of distribution differences, especially in the construction of feature maps in the deeper hidden layers of the model. We compared it with the commonly used second-order loss in the table below and found that KL divergence, as an objective for assigning bit-widths, is more robust and achieves better performance:

Table 1 Comparison between KL divergence and MSE

W1: In Equation 2, the $\alpha$ is not used. The correct equation may be $\hat w_b=\alpha \operatorname{sign}(w_f)$.

In Equation 3, the equation does not hold, please confirm if it is an approximation.

In Equation 4, using KL divergence for outputs is mathematically incorrect as they aren't probability distributions. Variables g1,...,gn are undefined. The definition of $w^{sba}_f$ needs proper notation (The superscript of $w^{b_0}$ cannot denote for bitwidth selection).

In Equation 5, the braces are not use appropriately.

Dear Reviewer jmUu,

We would like to clarify the point regarding Equation 2. As stated in line 185, we have defined $\hat w_b$ as the binarized results, while the $\alpha\operatorname{sign}(w_f)$ you mentioned refers to the float matrix weights after dequantization. We will include the float matrix weights in the final description as you suggested.

Regarding Equation 3, we considered the optimal scenario where the network training converges, and thus we equated the second-order error of the weight output to the right-hand side of Equation 3. This description follows the definitions in [1], where Equation 1 and Equation 2 are defined using $=$. We fully agree with your more rigorous definition, and in the revised version, we will update this to $\approx$ instead of $=$.

In the actual computation, we first apply softmax to the weight output features and then calculate the KL divergence. We will follow your suggestion and present this calculation process in the revised Equation 4. Additionally, the definitions of $g_1, \dots, g_n$ as the structured groups in the weight matrix will be clearly specified in the revised version, along with the modification of the expression for $w^{b_0}$.

Thank you for your valuable suggestions. We will also update the notation in Equation 5 by replacing $\lbrace\rbrace$ with $()$.

[1] QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. ICML 2024.

W2 & Q2: The main problem of this paper is that the algorithm is poorly explained: - Critical algorithm details are hidden in appendix. Implementation details are unclear. - Salience-weighted quantizer calibration is not well-explained, which is hard to understand. In Section 3.2.2, the search space for $s$ and $z$ is undefined. And the role of $τ$ in expanding solution space isn't clearly justified. The authour should clarify how $τ$ expands the solution space and why this is beneficial.

Thank you for your feedback. We have provided a detailed explanation of the use of $\tau$ in Section 3.3.2, lines 343-346. As shown in Equation 1, the general calculation of $s$ and $z$ is derived from the statistical values of all weight elements. However, this is often not the optimal quantization approach. To improve the representation of salient weights, SQC introduces Equation 5 to resolve for $s$ and $z$. Specifically, as defined in line 345, we expand the search range for $\tau$ to $[1 - \lambda, 1 + \lambda]$, dividing it into $2n$ evenly spaced points. Therefore, $\tau$ will fluctuate within this range to satisfy the objective in Equation 5. Under the influence of $\tau$, the search space for ss will be adjusted to $[(1 - \lambda) s, (1 + \lambda) s]$, with $2n$ search elements.

W1: Inference slow down: the paper suffers from a significant problem that the inference speed of the Llama2-7B quantized by SLIM-LLM is even slower than the fp16 baseline (Table 5). Users may as well not do any SLIM-LLM quantization: worse PPL and worse speed compared to fp16 baseline.

Dear Reviewer rUsM,

We would like to clarify that our method primarily focuses on significantly reducing memory usage during LLM inference at ultra-low bit-widths while maintaining inference performance. This enables the deployment of large-scale LLMs in resource-constrained edge scenarios. As shown in Table 5, our mixed-precision quantization method reduces memory usage to 16% of FP16 in the 2-bit quantization experiment for LLaMA-13B. Additionally, compared to other methods, our approach demonstrates significant improvements in PPL performance at ultra-low bit-widths. For instance, in Table 5, our method achieves nearly a 12 point improvement in PPL over GPTQ for 2-bit quantization of LLaMA-13B.

Regarding the inference speed issue you mentioned, weight-only quantization involves complex dequantization operations, which indeed introduce some additional computation and slightly reduce inference speed. However, after thoroughly investigating existing quantization frameworks such as AutoGPTQ, we observed that inference speed often decreases under 2-bit quantization due to the finer-grained weight packing required. We believe that hardware support for ultra-low bit-width quantization remains a promising area of research, and further optimizing the hardware compatibility of ultra-low-bit quantization will be an important focus of our future work.

Q1: The 2-bit PPL is still much higher compared to fp16, it's not practical for practical use yet. 4-bit weight is still the most widely used deployment case. Can the authors provide 4-bit PPL comparison with RTN, AWQ and GPTQ?

In comparison, our method does result in relatively higher PPL at 2-bit quantization compared to FP16. However, it represents a significant improvement over existing methods in optimizing 2-bit quantization for LLMs. As shown in Table 5, our approach achieves better performance under ultra-low-bit quantization than other frameworks, advancing research in this area and demonstrating the great potential and value of ultra-low-bit quantization.

Regarding your suggestion about comparative experiments with 4-bit quantization, we sincerely appreciate your input. We would like to clarify that while SliM-LLM is primarily designed for 2-bit and 3-bit quantization, our method is also compatible with 4-bit mixed-precision quantization. We have added experiments in the revised Table 1 and Table 2 below to address this. And we also tried it on some 4-bit models, we are happy to share the results to you:

Table 1 The results of our proposed method and other methods under 4bit quantization

Q2: (1)Given the additional steps introduced in SBA and SQC, how does the computational cost compare to baseline quantization methods?

(2) Specifically, are there latency trade-offs that should be considered, especially for real-time applications?

(1) In our mixed-precision quantization strategy, SBA and SQC are offline quantization steps that convert the original FP16 weights into ultra-low-bit integer values for storage. Both methods are executed on a resource-limited single GPU, and the quantization process for a 7B model takes approximately 50 minutes, which is about the same level as baseline. This is significantly more efficient compared to QAT-based methods, which often require multiple GPUs and can take a day or longer to complete.

(2) While the cost of quantization is worth considering, it is typically not the most critical factor since it is performed offline and can be scheduled with ample time in advance. In real deployment scenarios, inference efficiency is the key priority. As shown in Table 5 in Section 4.3, our method achieves faster inference speeds for the 13B model compared to FP16 while also delivering better performance than GPTQ[1]. This improvement is achieved through the development of CUDA code tailored to the computational characteristics of SliM-LLM, minimizing the overhead of extra computations. Additionally, we detail in the appendix the methods of our data packaging and distribution process, including how quantized data is aligned and packed into a unified data type, as well as how it is unpacked and dequantized during computation.

[1]Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers." arXiv preprint arXiv:2210.17323 (2022).

Q3: The paper focuses on transformers for text, but given the growing interest in multi-modal LLMs, could this quantization approach generalize to models that handle other data types or non-transformer architectures?

Although our current research focuses on Text-Transformers, as noted in [1-3], most mainstream LLMs and multi-modal LLMs are designed based on Transformer architectures. Their weights are still primarily concentrated in the linear layers of the multi-head attention and feedforward networks (FFN). Moreover, we have observed that while multi-modal LLMs introduce a visual modality to activations, this does not affect the occurrence of outlier channels during inference. Our proposed SliM-LLM is specifically designed to address the structured salience distribution caused by outlier activations. We believe that SliM-LLM holds significant potential for adaptation to multi-modal architectures, and exploring this application will be a key focus of our future research.

[1] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

[2] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in neural information processing systems 35 (2022): 23716-23736.

[3]Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." International conference on machine learning. PMLR, 2023.

Q4: While SliM-LLM claims to reduce overall memory usage, the paper lacks clarity on whether SBA and SQC introduce any hidden memory or storage overheads due to structured bit-width allocation. Further detail here would help assess the method’s efficiency.

We emphasize that the overhead introduced by SBA for structured mixed-precision quantization during storage and inference is negligible. This is a key advantage of structured mixed-precision quantization compared to traditional unstructured approaches. As highlighted in Section 3, while we do introduce a small number of structured parameters to indicate the quantization bit-width of each group, the additional parameter cost is minimal due to the design of our structured quantization method.

For example, for LLaMA-7B with a weight matrix dimension of $4096\times 4096$ and a group size of 128, each group has a size of $4096\times 128$. We only require a 4-bit identifier to store the quantization precision of each group, resulting in an overhead of just 0.000008 bits. Compared to other mixed-precision methods, the parameter increase introduced by our structured mixed-precision quantization is entirely negligible.

W3: The proposed SliM-LLM approach is designed based on empirical observations of outlier channels and salience distributions in models like LLaMA. However, recent architectures such as Gemma2 incorporate normalization layers following both attention and feed-forward (FFN) modules, which could alter the distribution and behavior of outlier channels. These normalization layers might dampen or redistribute the significance of certain activations, potentially influencing the effectiveness of salience-based quantization strategies. This variation raises questions about how well SliM-LLM’s bit allocation and salience-weighted calibration would generalize to such architectures, where outlier handling may not align with observations made in models without inter-layer normalization. A more in-depth analysis would help determine if additional adaptations are required to support architectures that use different normalization schemes, as seen in models like Gemma2.

We observed that various LLM architectures [1][2], including the Gemma2 model you mentioned, exhibit similar behavior in activations, specifically the presence of certain outlier channels. Even though different LLMs may use different layer normalization methods or apply normalization at varying positions, the intermediate activations in many LLMs consistently show outlier characteristics. Our SBA method, as described in the paper, is specifically designed to handle this pattern through mixed-precision quantization. Therefore, we can reasonably infer that it should achieve the expected results across most LLM architectures. Further experiments on Gemma2, summarized in Table 1 below, indicate that the outlier distribution in Gemma2 aligns with that of most LLMs, with certain channels heavily populated by outliers, and Slim-LLM can still effectively quantize these outliers to ultra-low bit-width values under the Gemma2 architecture.

Table 1 PPL Comparison on Gemma2 and Mixtral

[1] Spqr: A sparse-quantized representation for near-lossless llm weight compression. ICLR 2024.

[2] Mitigating the impact of outlier channels for language model quantization with activation regularization. arXiv preprint

Q1: (1) The paper briefly mentions spatial clustering of salient weights, but how does the model respond to cases where salience does not cluster as neatly?

(2) Are there adaptive mechanisms in SBA and SQC to handle such irregularities?

(1) In Section 3.1, we theoretically demonstrate that outlier activation channels in LLMs have a structured impact on the salience distribution of weights. Since outlier activation channels are consistently present in LLMs, this structured distribution phenomenon is generally observed.

(2) SBA addresses the inconsistency in salience distribution across groups by employing a search-based approach to allocate the optimal bit-width configuration, achieving excellent results under ultra-low bit-width settings.

Additionally, as described in Section 3.3.2, we also observed that within a group, there are still a few discretely distributed salient weights. To address this, we introduced SQC in Equation (5) of our manuscript to reduce quantization errors caused by these discrete salient weights during the quantization process. This effectively resolves the issue of unevenly distributed salience. This observation forms the core motivation for the SBA and SQC methods proposed in our work.

W1: The paper lacks a transparent discussion of potential limitations, such as possible trade-offs between compression rate and inference speed, or cases where the approach may not perform as well, such as for models with different architectural peculiarities. Acknowledging these limitations would strengthen the overall rigor of the paper.

Dear Reviewer mxH3,

Thank you very much for your suggestion.

We will include a discussion of the limitations in the revised version. We claim that our main contribution lies in the advantages of memory compression for LLMs and the hardware-friendly nature of data storage. As detailed in Table 5 of our manuscript, we analyzed the actual deployment speed under different memory compression levels. Our mixed-precision method is primarily designed based on recent literature and tailored for most commonly used open-source LLM architectures. However, it is true that it has not been specifically adapted for a few unique LLM models. We will carefully analyze the trade-offs and limitations of our work, particularly regarding compatibility with diverse architectures, and include a reasonable explanation of these limitations in the updated version of the paper.

W2: The experimental evaluation in this paper primarily focuses on the LLaMA series, with additional results on the OPT model series presented in the appendix. However, there is little discussion or empirical validation on architectures beyond these two families, which share many similarities in their transformer structures. This limited scope raises concerns about the generalizability of SliM-LLM across more diverse architectures. The absence of empirical results on these varied architectures leaves open the question of how well the proposed salience-based quantization strategies would perform in models with different salience and outlier channel dynamics. Extending experiments to a broader set of architectures would strengthen the claim of SliM-LLM’s robustness and applicability across LLMs.

We observed that most LLM architectures share similar foundations, being primarily based on Transformer. For our experiments, we selected models including OPT1.3B, 2.7B, 6.7B, 13B, 30B, and LLaMA1~3 in 7B or 13B, designing various experimental setups for configurations. These selections represent a significant portion of LLMs and align with the experimental setups commonly used in recent quantization and compression research.

However, we find your suggestion highly constructive, as it could help enhance the expressiveness and generalizability of SliM-LLM. In response, we have added experiments involving Gemma2 and Mixtral to Table 1 below in these comments.

W1: The authors are suggested to evaluate and report the quantization cost, since the proposed approaches introduce searching overhead.

Dear Reviewer pHSj,

Thank you for your suggestion. In fact, when applying SBA and SQC for PTQ mixed-precision quantization under single-GPU edge conditions, the quantization process for a 7B model takes only about 50 minutes. This is significantly more efficient compared to QAT methods, which typically require multiple GPUs and take a day or more. We will include details about our offline quantization time in the revised version to highlight the efficiency of our approach in optimizing quantization overhead.

W2: While the theoretical expectation for inference speedup from 16-bit to 2-bit quantization is 8x, the actual inference speed of the proposed approach can even be slower than its FP16 counterpart. This makes the proposed approach less convincing. It may be beneficial to explore state-of-the-art low-bit inference frameworks, such as T-MAC, to see if the issue can be addressed.

Our method focuses on reducing memory usage during LLM inference at ultra-low bit widths, enabling the deployment of large-parameter LLMs in resource-constrained edge scenarios. As shown in Table 5 of the paper, our mixed-precision quantization method reduces memory usage to 16% of FP16 when performing 2-bit mixed-precision quantization on LLaMA-13B. Additionally, compared to other methods like GPTQ, our approach achieves significant improvements in perplexity (PPL) at ultra-low bit widths. For example, as presented in Table 5 of our manuscript, our 2-bit quantization of LLaMA-13B achieves approximately a 12-point reduction in PPL compared to GPTQ.

In terms of runtime efficiency, weight-only quantization often involves complex dequantization operations, which may introduce some additional computation and slightly reduce inference speed. Through a thorough investigation of existing GPU-based quantization frameworks such as AutoGPTQ, we observed that at 2-bit quantization, these methods often experience speed reductions due to finer-grained weight packing. Recognizing this, we believe that hardware support for ultra-low bit widths remains a promising research direction. Further optimizing hardware friendliness for ultra-low-bit quantization will also be a focus of our future work.

W3: Figure 4 lacks clarity in its legends and scales, making it difficult to interpret.

We apologize for any inconvenience caused by the insufficient explanations. We will further improve and complete the annotations for all figures in future revisions to enhance readability.

Q1: Since low-bit quantization is popularly used for the LLM deployment on resource-limited user devices, such as laptops and mobile phones. How would you expect the results will be on these devices.

We are optimistic that our method, along with other similar ultra-low-bit-width LLM quantization and compression strategies, could be deployed on laptops and mobile phones in the future. We observe that most PTQ-based studies focus on optimizing compression on resource-constrained edge devices, such as single-GPU setups, aiming to minimize resource requirements during offline quantization and post-quantization inference while ensuring storage and inference efficiency. Some existing works, such as AWQ [1] and GPTQ [2], have already been successfully adapted for deployment on mobile and laptop devices. These methods leverage salience detection, scaling, and structured grouping strategies to achieve hardware friendliness, maintain inference performance, and enable low compression ratios. They are often integrated with optimized frameworks for mobile SoCs, such as ollama and bitsandbytes, which are well-suited for mobile-oriented quantization tasks. Given the hardware-friendly nature of our approach and the choice of group-wise quantization, we believe our method can also be seamlessly integrated into these toolchains, enabling potential deployment on mobile and laptop platforms.

[1]Lin, Ji, et al. "AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration." Proceedings of Machine Learning and Systems 6 (2024): 87-100.

[2]Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers." arXiv preprint arXiv:2210.17323 (2022).