SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization

1. Clarification on Objective Rankings: The ranking of objectives (S-full > L-full > S-diag > L-diag) is elaborated in Section 4.1, where we demonstrate that L-full outperforms S-diag. This is because S-diag simplifies computation via a diagonal assumption, introducing a certain degree of bias compared to the full form. Practically, Section 5.4.1 demonstrates that L-full outperforms S-diag in perplexity, validating our analysis.

2. Dequantization Efficiency in Mixed-Precision Settings: In mixed-precision implementations, we group channels with identical bitwidths and track their original indices to leverage GPU parallelism. Although dequantization introduces minor overhead compared to uniform-precision methods, this overhead is smaller than the computational cost of sparse tensor operations (e.g., in SqueezeLLM) and is justified by significant accuracy improvements. Our method performs similarly to SqueezeLLM latency-wise and faster than it throughput-wise.

3. Differentiation from Prior Mixed-Precision Work: Existing methods like SqueezeLLM, SliM-LLM, and CLAQ mix only a limited set of predefined precisions. For instance, SqueezeLLM combines INT3/4 with FP16, while SliM-LLM and CLAQ mix INT X with INT X+k and also include INT X-k for compensation. In contrast, our framework adaptively assigns arbitrary bit widths (e.g., mixing 1~4bit) across channels without relying on fixed thresholds or compensation rules. This flexibility broadens the scope of discussion and enables finer-grained error minimization.

4. Expanded Baseline Comparisons: We have added comparisons with QuIP#, a state-of-the-art baseline, which outperforms AWQ and QuIP. We choose the pure-PTQ version of QuIP# (QuIP# with fine-tuning consumes greater computational resources; for example, it requires 50 GPU hours to quantize LLaMA-70B on an 8 GPU node) and the the result is included in the following table:

Note that QuIP# officially supports only integer-bit quantization (e.g., INT2); thus, we compare its INT2 results with our 2.x-bit performance.

5. OmniQuant Implementation Details: Under non-integer bitwidths(eg: 3.x-bits), OmniQuant is evaluated under its group-wise quantization setting (eg: 128 elements per group), as specified in their official implementation. For integer bits (e.g., 3/4-bit), per-channel quantization is used, same as our method. Previous works commonly adopt this setting.

6. LLaMA3 Benchmarking: We focus on LLaMA-1/2 and OPT models to align with established baselines (SqueezeLLM, OmniQuant, and baselines you mentioned), as LLaMA3 results are not yet widely reported in quantization literature. This ensures apples-to-apples comparisons. We will be happy to include LLaMA3 in future work once baselines adopt it.

7. Expanded Evaluation Metrics: Following suggestions, we have added results on ARC-E, ARC-C, and PIQA. Below is the detailed result and our method maintains its consistent superiority. | Method | Precision | PIQA | ARC-C | ARC-E | MMLU | Avg | | --------------------- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | | LLaMA2-7B | FP16 | 77.0% | 42.1% | 64.5% | 45.9% | 57.4% | | INT3 Quantization | | | | | | | | SKIM | INT3 | 76.1% | 40.6% | 61.3% | 42.3% | 55.1% | | SqueezeLLM | INT3 | 75.6% | 38.8% | 61.1% | 41.3% | 54.2% | | INT4 Quantization | | | | | | | | SKIM | INT4 | 76.9% | 42.5% | 64.6% | 45.4% | 57.3% | | SqueezeLLM | INT4 | 77.0% | 41.8% | 63.8% | 45.1% | 56.9% |

1. More Information about the Greedy Algorithm: Through empirical validation, we have demonstrated the effectiveness and efficiency of the greedy algorithm:

• Time Efficiency: The cpp/jit implementation of the dynamic programming algorithm remains slow. For a single block in LLaMA-7B, it requires several minutes of computation. Given that LLaMA-7B has 32 layers with 7 blocks per layer, this algorithm struggles to scale to large models. In contrast, the greedy algorithm, even when implemented in Python, only requires approximately 3 seconds per block, ensuring computational efficiency.

• Practical Effectiveness: To guarantee a near-optimal solution, we employ two different initialization points for the greedy algorithm and select the superior solution. Empirical validation on the first layer of LLaMA-7B shows that the average error between the suboptimal solution from the greedy algorithm and the true optimal solution is below 2%, confirming its capability to deliver sufficiently high-quality results.

2. Additional experiments: We further validated our method by comparing it with more recent works (e.g., adding QuIP# as a new baseline) and extended benchmarking on tasks like PiQA and ARC. These results further validates the effectiveness of our method and detailed results are shown below:

1. Comparison with ShiftAddLLM: Thank you for the suggestion. We have now included a comparison with ShiftAddLLM. Overall, our method is competitive compared to ShiftAddLLM. In fact, ShiftAddLLM uses a smaller group size (128) rather than channel-wise quantization when quantizing LLaMA, so its precision should be classified as non-integer level. ShiftAddLLM does not provide detailed explanations about the precision of the scaling factors or additional overhead, so we compare its performance under the common practice of group size 128. We represented the perplexity of quantized LLaMA-7B on Wikitext2 in the table below. our method performs better at 3-bit, while being comparable at 2-bit. Moreover, by slightly increasing the bit level using the any-bit feature, significant improvements can be achieved. Additionally, beyond perplexity (PPL), our method achieves state-of-the-art performance on some widely-adopted benchmarks (see the table under answer 2), with some benchmarks even approaching FP16 performance. Those empirical results demonstrate that our method achieves comparable or superior performance to ShiftAddLLM. Furthermore, we want to emphasize the superior quantization efficiency of our method. While ShiftAddLLM requires approximately 20 GPU-hours to quantize the LLaMA2-7B, our approach completes the same process in under one hour. This dramatic difference in computational overhead becomes even more significant when scaling to larger models, making our method more practical for real-world applications. | Method | Bit | Group Size | PPL (Wikitext2) | |----------------|-----------|------------|-----------------| | FP16 (LLaMA) | 16 | - | 5.68 | | ShiftAddLLM | 3.0 | 128 | 6.04 | | SKIM (Ours) | 3.25 | Channel-wise| 6.02 | | ShiftAddLLM | 2.0 | 128 | 7.98 | | SKIM (Ours) | 2.25 | Channel-wise| 8.44 | | SKIM (Ours) | 2.4 | Channel-wise| 7.76 |

2. Improvements for >3 Bits: In fact, the improvements shown in perplexity are nonlinear—smaller improvements at lower PPL can still lead to notable performance gains. For example, when quantizing LLaMA2-7B, our compressed model demonstrates significant improvements on PiQA, ARC, and MMLU, as detailed in the table below. Meanwhile, the smaller PPL improvement at 3.x-bit is because, for consistency, we compressed the model to 3.2 bits rather than higher bit levels like 3.25 or above used by some other methods. For 4-bit, we set the maximum available bit to 4 to ensure quantization efficiency, effectively disabling mixed precision. If we were to use higher bit levels for 3.x-bit (e.g., 3.25 or 3.3 bits), PPL could be reduced by an additional 50%-100% (from 6.13 to 6.07 to 6.02), making the improvement more pronounced. Similarly, for 4-bit, if we relaxed the restriction, the gap to FP16 could be reduced from 0.13 to 0.11. This 0.02 reduce is relatively considerable for the INT4 level which is quite close to the original FP16 result. | Method | Precision | PIQA | ARC-C | ARC-E | MMLU | Avg | | --------------------- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | | LLaMA2-7B | FP16 | 77.0% | 42.1% | 64.5% | 45.9% | 57.4% | | INT3 Quantization | | | | | | | | SKIM | INT3 | 76.1% | 40.6% | 61.3% | 42.3% | 55.1% | | SqueezeLLM | INT3 | 75.6% | 38.8% | 61.1% | 41.3% | 54.2% | | INT4 Quantization | | | | | | | | SKIM | INT4 | 76.9% | 42.5% | 64.6% | 45.4% | 57.3% | | SqueezeLLM | INT4 | 77.0% | 41.8% | 63.8% | 45.1% | 56.9% |

3. Clarification on Any-Bit: Previous methods do include some works that support any-bit due to mixed precision, but most fail to demonstrate "continuity" in bit usage. For example, SqueezeLLM allows increasing the precision of certain elements, but if an extra available bit is added to INT3, its performance would be worse than INT4 rather than matching it. In contrast, our any-bit method, based on allocation, can fully utilize the given bit budget, making it unique and practical.