ICQuant: Index Coding enables Low-bit LLM Quantization

Distribution Assumption

• To verify the uniform distribution property in MoE models, we performed the same Chi-square test on the Llama 4 Scout model (17Bx16E). The results align with other models.

• ICQuant could also apply even if this uniform distribution property does not hold. Please refer to our common response in the beginning for a detailed analysis.

Complexity & Computational Cost

• Our indices decoding can be efficiently fused with matrix multiplication in a single kernel, thereby minimizing overhead and avoiding costly memory transactions. Alternatively, to fully accelerate the inference, outlier indices can be pre-decoded into a compact bitmask before the layer is executed.
• To demonstrate the practical efficiency of ICQuant, we implemented an example 2-bit matrix-vector multiplication kernel. Our implementation supports on-the-fly decoding and dequantization. We benchmark the matrix-vector multiplication subroutine on a standard linear layer, using an Nvidia RTX 4090 GPU. ICQuant can indeed deliver faster inference than all baseline methods.

• Although ICQuant requires additional decoding steps, the inference latency remains lower than baseline methods. Note that SOTA baselines have non-trivial overhead—structured vector quantization schemes like AQLM require loading from a codebook larger than the cache size, while QTIP and QuIP# require additional matrix multiplications and codebook generation.
• Our kernel serves as a proof-of-concept to show feasibility rather than peak performance. For instance, unlike all baselines' implementations, our current kernel uses full-precision (FP32) arithmetic instead of FP16, which does not leverage tensor core acceleration available on modern GPUs. We believe there is significant room for further optimization, yet already the current results are very promising.

Hyperparameters

• We choose 5% outlier ratio for the following reasons:
  • We split the weight range in half to ensure consistent quantization resolution for inliers and outliers.
  • Our empirical analysis (Figure 1, 6) shows that approximately 5% of outliers account for half of the total range.
• To verify the impact of outlier ratios, we evaluate ICQuant on 3-bit uniform quantization of Llama2-7B below, with varying outlier ratios.
  • The perplexity decreases as the outlier ratio increases. This is expected, since a smaller quantization range for inliers benefits the majority of weights.
  • However, the performance gain from separating more outliers saturates after 5%. This aligns with the behavior of Gaussian-like distributions, where the change in range becomes minimal beyond 5%. To balance the trade-off between storage overhead and quantization quality, we stay with 5% for most experiments.

Q1

• Weight clipping cuts all outliers to two predetermined thresholds at both ends, which improves quantization quality by reducing the range of inliers (the majority of weights). However, the effectiveness of clipping is limited due to the large quantization error for outliers. In contrast, ICQuant quantizes the outlier area using the same number of bits, enabling more aggressive shrinkage of the inlier range. When targeting the same inlier range for $b$-bit quantization, weight clipping produces $2^b$ times larger quantization error for outliers compared to ICQuant.
• We compare ICQuant's performance against the results reported in the LLMC paper, using the same benchmarks. Under the same storage cost, ICQuant significantly outperforms weight clipping.

Q2

Please refer to the complexity response above, with results for an example kernel.

Q3

We evaluate ICQuant on the 5-shot MMLU dataset across the Llama2 family. ICQuant achieves near-lossless performance at ~3 and ~4-bit levels. While 2-bit quantization remains robust on the 70B model, smaller models show larger degradation. We also compared against 2-bit AQLM and QuIP#—the only baseline results reported on MMLU. For fair comparison, we applied the same fine-tuning strategy as used in these baseline results. (Values in brackets are post-fine-tuning.) We found that fine-tuning substantially improved MMLU performance, allowing ICQuant to outperform SOTA baselines.

GPTQ & AWQ

The state-of-the-art baselines we compared with (QTIP, QuIP#, AQLM, SqueezeLLM, and OmniQuant) consistently outperform GPTQ and AWQ. As an illustration, we provide the perplexity evaluation (context length = 2048) of the 2.3-bit Llama2 models below. Additional evaluation results can be found in the papers of our baseline methods.

Scaling Laws Paper

• Our paper focuses on post-training quantization, and all evaluations are conducted on fully trained LLMs. We believe that studying quantization across training trajectories is a valuable direction, but it is somewhat orthogonal to our current focus.
• The state-of-the-art baselines (QTIP, QuIP#, AQLM), which are not evaluated in the “scaling laws” paper, do not fail in the low-bit regime. However, these methods all rely on complex vector quantization and expensive fine-tuning. Our experimental results demonstrate that ICQuant can significantly improve the quantization quality of simple scalar quantization schemes (similar to GPTQ and AWQ), achieving the state—of-the-art performance in low-bit quantization without any fine-tuning.

Missing Performance Entries

Prior works evaluate perplexity scores using two different context length configurations. In Table 2, we present ICQuant's performance under both configurations and compare directly with results from the original baseline papers. Some entries are missing because the original papers did not report certain results—for example, QuIP did not provide numbers for the Llama2-7B model.

Experiment Results

• Llama3-70B is known to be particularly challenging to quantize, with many existing solutions failing to achieve good performance even after fine-tuning. ICQuant, however, remains robust in this setting and significantly improves upon the state-of-the-art results.
• For other models, the performance gaps between quantized baselines and non-quantized models (FP16/BP16 in tables) are already minimal (e.g., 0-3% for Llama2 family). Yet achieving such near-lossless performance typically requires complex vector quantization and expensive end-to-end fine-tuning. ICQuant, by contrast, enables simple scalar quantization to reach state-of-the-art performance without any fine-tuning.
• We evaluate ICQuant on 6 different models from Llama2 and Llama3 families, with scales ranging from 1B to 70B parameters, covering all benchmarks used in the state-of-the-art baseline papers.
• Weights in trained LLMs typically follow Gaussian-like distributions, as verified in prior works (e.g., QLoRA). While the outlier position distribution may slightly affect the storage efficiency of our index coding scheme, it does not impact ICQuant’s core advantage of halving the quantization range by storing a small amount of outliers. (Please refer to our common response in the beginning for a detailed analysis of the storage efficiency.)

Novelty

• Outliers pose a major challenge for low-bit weight quantization, leading to many proposed outlier suppression solutions. ICQuant is the first practical framework that can efficiently encode the outlier indices, allowing to quantize outliers and inliers using separate codebooks. It offers distinct advantages:
  • Existing approaches like SqueezeLLM store outlier values and their indices in full precision. Due to the high storage cost per outlier, they can only handle up to 1% outliers. In contrast, ICQuant can process 5x more outliers with the same storage cost. This reduces the 2-bit quantization-induced degradation by 56% on average, for the same quantizer (measured using Table 2).
  • Traditional group quantization simply groups adjacent weight entries without considering their statistics. Irregular grouping is challenging since group labels need to be stored for decoding. ICQuant innovates by fully leveraging weight statistics for both grouping and efficient storage.
• As shown in Section 4.1, ICQuant substantially outperforms existing outlier suppression techniques (including full-precision outlier storage and grouping) under the same storage cost.

Distribution

• The impact of o_proj layers is minimal for two reasons:
  • While their outlier indices deviate from a perfectly uniform distribution, the storage cost of ICQuant's coding scheme remains modest. The table below shows empirical layer-wise costs.
  • O_proj layers make up just 8% of all quantized weights in Llama3-8B.

• We emphasize that ICQuant can be applied to any outlier distribution. We provide a detailed analysis in our common response.

Coding Scheme

Our primary innovation is in leveraging the statistical property of outliers to design a lightweight yet robust index coding scheme. This approach yields significant storage savings and offers a universal benefit for LLM quantization. The simplicity of our method is an advantage, not a limitation—it enables efficient deployment across LLMs at scale.

Permutation

Our random permutation strategy does not add storage or computation overhead. In Figure 7, we use $P_1^\top X$ to denote the reordered activation rather than a matrix multiplication. $P_1^\top$ combines with the previous $W$ and is stored as $P_1^\top W$.

Parameters

The rationale for our parameter choices is as follows:

• We split the range at 50% to ensure consistent quantization resolution of all weights (i.e., inliers and outliers).
• Our empirical analysis (Figure 1, 6) shows that approximately 5% of outliers account for half of the total range.

Experiments

• By separating outliers and inliers, ICQuant effectively gains an extra bit of quantization capacity. In our paper, we apply ICQuant to the non-uniform quantizer from SqueezeLLM paper. Using only ~2 bits, we expect ICQuant to approach SqueezeLLM's 3-bit result, which outperforms all 2-bit SOTA baselines. As shown in the Llama2-7B quantization results below, ICQuant substantially improves upon 2-bit SqueezeLLM, approaching 3-bit performance. The small gap between ~2-bit ICQuant and 3-bit SqueezeLLM exists because the non-uniform quantization biases toward the more concentrated inlier region. This also confirms that increasing the outlier ratio (e.g., to 8.25%) can further enhance performance.

• We plan to release the code upon paper acceptance. If the program chairs can provide a mechanism to do this anonymously, we are happy to share the code if the reviewer deems it necessary.

Q1

Adaptive outlier separation is a promising future direction. In this work, we use a fixed outlier ratio for two reasons:

• Our empirical observations (Table 1, 6) show that layer-wise distribution statistics remain similar across the models we evaluated.
• The choice of outlier ratio reflects a tradeoff between storage cost and model accuracy, and using a global value enables easier control of this balance during evaluation and deployment.

Q2

• ICQuant's core strength—halving the quantization range—is orthogonal to quantization schemes. Moreover, it offers the flexibility to design different schemes for outliers and inliers based on their distinct statistics, which we leave for future work.
• We conducted preliminary experiments using the same fine-tuning procedure as QuIP#/QTIP for 2-bit quantization of Llama2-7B. Fine-tuning further enhances ICQuant's performance beyond the SOTA.

Q3

Please refer to our common response in the beginning for details.

Notation

We appreciate your careful review highlighting these areas that need clarification. Let us address each point:

• We examine the weight statistics within each output channel (i.e., each row of a weight matrix), and we use $w \in \mathbb{R}^{d_{in}}$ to denote a row of weights. This follows the convention in weight quantization literature.

• We define outliers as weights with the largest absolute values (as defined in line 136). This is equivalent to the description in line 94. In trained LLMs, weights typically follow a zero-centered Gaussian-like distribution, as shown in Figure 1(b), with outliers appearing in the two tails of the distribution.

• For the definition of range, we consider the union of active regions rather than the entire span. Using Figure 1(b) as an example, all weights $w$ lie within $[-0.06, 0.06]$, where inliers $w_i \in [-0.03, 0.03]$ and outliers $w_o\in[-0.06, -0.03)\cup(0.03, 0.06]$. The total range of weights is $\text{range} (w) = 0.12$, the range of inliers is $\text{range}(w_i) =0.06$, and the range of outliers is $\text{range}(w_o) = 0.03 + 0.03 = 0.06$. We formally define the range of given weights $w$ as $\text{range}(w) := 2(\max |w| - \min|w|)$. Thus, at line 141, we can state that

  $$\text{range}(w_o) \approx \text{range}(w_i) \approx \frac{1}{2}\text{range}(w).$$

Innovation

• A specific bit having a high probability of being 0/1 does not capture our main findings. It is well known in the literature that a small percentage of weights (outliers) contribute a large amount to the quantization range, which significantly impacts the quantization accuracy.

  • Our main finding is: These outliers are positioned uniformly in each weight channel. Even if this property does not hold, we can enforce it using random permutation, without affecting model performance, inference time, or memory usage.

  • Our main novelty is: A novel encoding scheme that leverages this observation to store outlier positions with minimal bit overhead. Specifically, our method encodes the gaps between outlier indices using a specialized code that takes advantage of their uniform distribution property.

    This leads to big savings in bits as compared to SOTA schemes that separate outliers, which makes it practically feasible to separately quantize outliers and inliers using different codebooks. We prove our result theoretically and verify it on the widely used Llama models.

Evaluation

We did a thorough comparison with existing outlier suppression techniques in Section 4.1. In our main experiments (Section 4.2), we evaluate the quantization performance of ICQuant across 2-4 bits per weight (including 3 bits) on a wide range of models and compare against the state-of-the-art quantization methods. Could you please clarify which 3-bit results you found missing?

Inference Efficiency

Based on the reviewers' feedback, to demonstrate the practical efficiency of ICQuant, we implemented an example matrix-vector multiplication kernel. In the table below, we benchmark the matrix-vector multiplication subroutine performance on a standard linear layer, using an Nvidia RTX 4090 GPU. The result shows that ICQuant can deliver faster inference than all baseline methods.