SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

Dear Reviewer H7Hr,

Thank you for your feedback. We will address your questions and recommendations one by one.

Q1: The key aspect of the proposed method ... for uneven weight sensitivity. The proposed saliency measure ... However, no reference is provided.

A: We would like to clarify that in lines 45–46 of the paper, we emphasize that weights "exhibit a structured distribution" a phenomenon that has not been overlooked in previous works. Prior studies mainly focused on compressing weights based on the sparse distribution of salience but did not propose specific strategies to handle structured distribution. This insight motivated us to develop an inference-friendly mixed-precision quantization strategy. Thank you for your suggestion—we will highlight the characteristics of structured distribution more clearly.

Regarding SPQR, we explicitly reference SPQR (Line 199) in Definition 3.2 and also cite other works that adopt the same salience definition[1] [2] [3]. We would like to reiterate that the purpose of our work is not to emphasize the formulation of salience itself, but rather to identify its clustering characteristics within LLM weight matrices. SliM-LLM aims to illustrate, through Definition 3.2, the numerical impact of weight magnitude and activation distribution on the salience of LLM weight matrices and provides in-depth theoretical analysis in Theorem 1 and Appendix G, which further confirms the cause of the observed structured clustering of salience in SliM-LLM.

[1]SparseGPT: Massive Language Models Can Be Accurately Pruned In One-shot. ICML, 2023.

[2]PB-LLM: Partially Binarized Large Language Models. ACL 2024.

[3]LLM-MQ: Mixed-Precision Quantization for Efficient LLM deployment. NIPS, 2024.

Q2: Overall, experimental protocol and choice of baselines ... be added for completeness of evaluation.

A: As summarized in Figure 1, DB-LLM is a QAT-based method, which requires a significant amount of data and extended distillation tuning to complete the quantization process. In other words, it relies on standard backpropagation techniques to adjust the weights. In contrast, SliM-LLM is a totally PTQ method. More importantly, the key advantage of SliM-LLM is its ability to deploy group-wise quantization kernels directly on the GPU, and its seamless integration with AutoGPTQ. DB-LLM is an effective weight-compression method, focusing on memory compression, and is not yet capable of achieving efficient inference and real acceleration, which is the gap we aim to address.

Q3: While the approach yields quite good performance ...... [1, 2], which achieve same or better speed-ups.

A: QuIP# and AQLM are both highly effective 2-bit LLM codebooks-based or vector-based quantization methods that can be improved by additional fine-tuning to restore performance (referenced in our paper). They rely on backpropagation to adjust pre-trained weights to fit the quantization scenario. In contrast, as we emphasize, SliM-LLM is a fully PTQ-based quantization method, with no training on the original weights. For a fair comparison of quantization strategies, we only compare with state-of-the-art PTQ methods, and we have achieved strong low-bit quantization performance within the PTQ framework.

Following your suggestion, we compared the non-trained QuIP# with our SliM-LLM+ at 2-bit. The results above show that even without additional weight tuning, SliM-LLM+ still achieves superior performance.

Regarding the speed-ups you mentioned, we conducted a thorough investigation and reproduction of results. We found that codebook-based and vector-based quantization methods, while effective in compressing weight memory, require complex lookup and decoding operations during real deployment, resulting in inference times that are approximately three times longer than that of fp16 LLMs (https://github.com/Cornell-RelaxML/quip-sharp/issues/63). In contrast, the structured mixed-precision quantization strategy employed by SliM-LLM allows for efficient deployment in AutoGPTQ through a group-wise approach, achieving significant speed-ups.

Q4: Which dataset is used for the double-pointer search - is it the same calibration set used for SliM-LLM+ (i.e 128 samples from Wikitext-2)?

A: Yes, as demonstrated in Section 4, we used the same calibration data setting (128 samples from WikiText-2), and during quantization, we selected samples randomly. The calibration selection method for SliM-LLM+ is identical to that of OmniQuant, both using random sampling of 128 data points from WikiText-2.

Q5: How much does it take to produce a quantized model with SliM-LLM?

A: When applying SBA and SQC for PTQ mixed-precision quantization under single-GPU (RTX 4090) edge conditions, the quantization process for a 7B model takes approximately 25 minutes.

Dear Reviewer iFeq,

Thank you for your valuable feedback and suggestions. We will address your questions and recommendations one by one.

Q1: (1)The analysis of local salience is relatively underdeveloped, and it has also been explored in recent previous works. (2)The authors may add references for works that mentions occurrence of unstructured outliers (salient values) such as [1], [2] of part 2.

A: The variation in weight importance distribution is crucial for LLM quantization, particularly regarding local salience. As noted in prior studies, our key contribution is using the element sensitivity criterion (Definition 1) to analyze significant weight clustering and provide a theoretical explanation for salience clustering.

We did not explore the underlying causes of local salience in detail, as they have been thoroughly discussed in works like [1] [2] and Section 3.3.2. Instead, we built on these insights to develop the Salience-Weighted Quantizer Calibration (SQC) strategy to reduce quantization errors. We appreciate your reference suggestions and will expand the discussion in the revised version.

Q2: Inference efficiency is another important aspect ... efficient inference, appears to discuss a different topic.

Thank you for identifying the typo. This was a layout error, and we will correct it by including the full content of the "Efficient Inference on Device" section in the final version.

Specifically, we extend the CUDA kernel in AutoGPTQ to support experimental mixed-precision inference (details in Appendix B.2). We evaluate LLaMA-7/13B and LLaMA-2-7B under 2/3-bit settings, showing that our approach maintains a high compression rate on GPUs while significantly improving model accuracy, with only a slight inference speed reduction on the A800 due to bit-width alignment. As 1-bit operations currently lack hardware support, additional storage and computation are required. We recognize the potential for further optimization in mixed-precision computing and aim to improve this in future work.

Comments Or Suggestions.

A: (1)Thank you for your valuable suggestions! We will follow your advice and add the complete inference settings in Section 4.3.

(2)The deployment time of SliM-LLM consists of two main parts: SBA bit-width search and SQC quantization parameter determination. As detailed in Section 3.2.2, the SBA dual-pointer search is highly efficient, and SQC computation time is comparable to standard calibration-based quantizers. SliM-LLM integrates seamlessly with existing PTQ strategies and, being training-free in a plug-and-play manner, completes 7B LLM quantization in about 25 minutes.

(3)For the ablation study on group size, we have discussed the differences in detail in Appendix F and provided experimental results in Table 7. We will further highlight this content in the main text to improve the readability of the paper.

Questions For Authors.

A: (1)Thank you for your insightful observations. You are correct about the challenges with models like LLaMA-3. Despite this, SliM-LLM maintains leading quantization performance and remains highly practical.

As noted in prior works [1–4], quantization methods degrade more severely in knowledge-dense models like LLaMA-3, especially at ultra-low bit widths. Our experiments confirm this: in Table 1, LLaMA-3 8B with AWQ and GPTQ yields PPLs of 8.22 and 8.19 under 3-bit quantization, rising to 210 and 1.7e6 at 2-bit. This suggests that as models grow in knowledge density, conventional methods suffer greater quantization losses. In contrast, SliM-LLM achieves PPLs of 7.16 (3-bit) and 39.66 (2-bit), demonstrating its effectiveness.

Additionally, we have included 4-bit quantization experiments, which are more practical, to further highlight SliM-LLM’s advantages on LLaMA-3.

(2)SliM-LLM+ incorporates our structured mixed-precision strategy into OmniQuant’s gradient quantizer. However, as OmniQuant and its comparator, AffineQuant, currently lack support for models like Gemma and Mixtral, we did not report results on these. We have contacted the OmniQuant authors to request updates and are actively working to extend SliM-LLM+ to more models.

[1] Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference. arXiv:2409.20361.

[2] Efficientqat: Efficient quantization-aware training for large language models[J]. arXiv:2407.11062.

[3] Compressing large language models using low rank and low precision decomposition[J]. NeurIPS, 2024.

[4] An empirical study of llama3 quantization: From llms to mllms[J]. Visual Intelligence, 2024.

Dear Reviewer Ut3n,

We sincerely appreciate your insightful feedback and suggestions. Below, we will respond to your questions and recommendations individually.

Q1: The authors are advised to consider testing on more challenging LLM benchmarks, such as GSM8K in the mathematics domain. Compared to MathQA, GSM8K may be more sensitive to the loss caused by quantization.

A: Thank you for your suggestion. We have conducted additional tests to evaluate the performance of our method on more challenging datasets, such as GSM8K. The results are shown in Table below.

The findings demonstrate that, even on GSM8K, our method exhibits less accuracy loss compared to other approaches, highlighting its robustness across different test sets.

Q2: In group-wise mixed-precision inference, the allocation of quantization scales across different channels and the process of dequantization could provide deeper insights into the inference details of this structure. The authors are encouraged to provide further explanations on these aspects.

A: We have explained some details on how to allocate, store, and perform inference with quantization scales in Appendix B.2. This means that, while storing the quantized weights at extremely low bit widths, we also store the corresponding quantization scales for each row, and are able to perform dequantization operations on both scales and quantized integers on the CUDA Kernel, restoring them to the floating-point values required for inference. We will clarify and explicitly refer to this section in the main text.

Q3: There are minor typos, and the abbreviations WM and RM in Table 5 lack annotations.

A: Thank you for your careful and thorough review! We will further check for any writing errors in the paper to ensure better readability. Regarding the issue you raised about the lack of explanations for WM and RM in Table 5, we would like to clarify that WM stands for Weight Memory and RM stands for Running Memory. We will also add the definitions of WM and RM in the relevant section to ensure they are accurately explained. Thank you for your helpful reminder.

Q4: Although SLiM-LLM is a leading PTQ method in the 2-bit and 3-bit settings, which is a popular topic in the research field, it appears that the commonly used 4-bit quantization for industrial applications has not been explored in the paper. Could the authors provide a comparison of 4-bit PPL with RTN, AWQ, and GPTQ?

A: We sincerely appreciate your suggestion regarding comparative experiments with 4-bit quantization. We would like to clarify that although SliM-LLM is primarily optimized for 2-bit and 3-bit quantization, our method is also compatible with 4-bit mixed-precision quantization. In response to your suggestion, we have included additional experiments in the revised versions of Table 1 and Table 2. Furthermore, we have tested our method on several 4-bit models and are pleased to share the results with you below:

*The results of RTN are too worse than GPTQ and AWQ to be listed here.

Dear Reviewer BegC,

Thank you for your valuable feedback. We have summarized your questions and concerns. Due to the character limit for reply, if there are any question that are not detailed, we will further reply in next stage. Thanks you so much.

Q1:(1) There is...diverse deployment conditions. (2)Discussion of trade-offs...conditions.

As shown in Table 5, we analyzed deployment speed under different memory compression levels using a batch size of 1 (align with other works), focusing on extreme compression for single-GPU. We added inference experiments for the compressed LLaMA-7B model on different GPUs, testing various batch sizes.

Slight speed decreases with larger batch, consistent across frameworks like AutoGPTQ and AutoAWQ.

Q2: (1)Most of the core...to be adapted. (2)The usage of...is short. (3) If future LLM...approach?

The LLaMA and OPT series are among the most widely downloaded and applied LLM architectures in the community. They are also commonly used as base models in other works. As you noted, we include more experiments in Table 11 to evaluate models with extra normalization layers. We claim that salience is a relative concept, as LLM training inherently produces such salience[1][2], even in multimodal or fine-tuned models. While architectures like Gemma2 use extra normalization layers, relative salience of features still emerges.

SliM-LLM focuses on compressing LLMs. Follow your suggestion, we expanded multimodal task on the LLaVA-Next 8B model(N:collapse accuracy).

[1]From Attention to Activation: Unraveling the Enigmas of Large Language Models.

Q3: (1)The experiments a...exhibit high salience. (2) The theoretical discussion...distributions.

The proportion of local outlier weights within groups is not an assumption but an observation supported by prior studies. Section 3.2.2 and previous work[1] consistently show that salient weights within groups remain a small proportion. Theorem 1 and detailed proofs in Appendix G explain this phenomenon.

[1] SpQR: A sparse-quantized representation for near-lossless LLM weight compression.

Q4: (1)There is no detailed breakdown ...vary. (2)The paper...large models (e.g., 70B+)?

In Table 7, we provided the quantization performance of four LLMs under different group sizes. SliM-LLM introduces negligible storage overhead, which decreases further as model size grows. For instance, with LLaMA2-70B (group sizes = 64, 128, 256), a single transformer layer (size 8192 × 8192) requires a group matrix of size 8192 × 64/128/256. Using three precision levels, only a 2-bit flag (e.g., 01 = 1-bit, 10 = 2-bit, 11 = 3-bit) is needed per group. The additional storage overhead is $\frac{2}{8192 \times 64/128/256}$—virtually negligible at scale. Other quantization parameters are identical to frameworks like GPTQ.

Based on your suggestion, we will add 70B models results.

Q5: The authors rely on standard...domain-specific conditions.

Our paper evaluates quantized model performance on 13 benchmarks, including Wikitext2, C4, and 8 zero-shot tasks, aligning with other LLM quantization works. Appendix Table 13 includes benchmarks from Humanities, Social Sciences, and STEM, with reasoning tasks like MathQA highlighting SliM-LLM's strengths in math reasoning. We will add results for LLaVA-Next-8B on 4 multimodal tasks in Q2.

Questions For Authors:

(1)As shown in Definition 3.1 and Theorem 1, salience is influenced by weight magnitudes and the Hessian. If all weights exhibit similar salience, the distinction between salient and non-salient weights (Equation 5) becomes negligible. In such cases, τ converges to 1, simplifying quantization and allowing parameters to be derived from basic statistics.

(2)Codebook-based methods can improve accuracy but significantly reduce inference speed—e.g., 2-bit code-book can be 3× slower. SliM-LLM’s structured mixed-precision quantization achieves competitive accuracy while maintaining practical inference speeds.

(3)Frameworks like recent HQQ show promising advancements in GPU kernels for 2-bit quantization. These developments could further improve SliM-LLM's inference speed.

(4)For calibration, SliM-LLM used only 128 random samples from WikiText2, avoiding additional data or adaptive re-quantization.