GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models

Thanks for the feedback. Below, we address your concerns one by one.

Response to Weakness 1

We acknowledge the value of broader evaluation and have conducted additional experiments.

Below are WikiText PPL results for LLaMA-3.2 models, using the same settings as in the paper.

These results confirm that GANQ consistently outperforms baselines with superior and more stable performance. While OmniQuant learns the quantization factors and is relatively robust, it still fails in some case (i.e, 3-bit for LLaMA-3.2-1B). RTN and GPTQ show significant sensitivity to weight distribution and degradation, with GPTQ notably unstable on 3B models. To explore this, we test GPTQ with a group size of 128 to mitigate outlier effects (same table head):

Using a group size of 128 improves GPTQ's performance on 1B models but still performs poorly on 3B models, highlighting its sensitivity to weight distribution.

We initially did not include long-context or reasoning-intensive tasks as we evaluated base models without task-specific fine-tuning (e.g., LLaMA-2-7B). Following standard practice [1,2], we used Table 3 tasks to assess general capabilities.

We test LongBench and GSM8K on LLaMA-3.2 1B/3B-Instruct and their quantized versions.

GANQ consistently outperforms baseline. For GPTQ, consistent with PPL results on 3B-Instruct, GSM8K accuracy is near zero. On LongBench, GPTQ failed with a "skip_special_tokens" error, despite identical settings across all methods and use of the official toolkit [5].

We plan to incorporate these results into the revision.

Response to Weakness 2

In Table 5, we compare the efficiency of dequantization-based and LUT-based methods. Following [3], we use GPTQ as a representative baseline for dequantization-based method, as similar methods like RTN and OmniQuant can generally share the same inference kernel and offer comparable efficiency. We will clarify this in the revision.

We acknowledge that the GPTQ efficiency reported in Table 5 is lower than that in the original GPTQ paper. This discrepancy is mainly due to differences in the inference kernel, hardware, and model size, with the kernel likely being the key factor.

As noted in Section 4.1, we followed [1] and used the GPTQ-for-LLaMA [2] inference kernel for GPTQ-quantized models. The reported results reflect actual observations. However, since [2] is implemented in Triton, and our experiments were conducted on an NVIDIA RTX 4090, potential incompatibilities or limited hardware support may explain the slower inference in our setup.

As for metric, as noted in Section 4.1, we followed [1,3], using a batch size of 1 to generate 1024-token sequences and reporting total CUDA time.

To further validate this, we performed an additional test using GPTQ's official CUDA kernel implementation [1] (available only for 3-bit quantization and OPT models). The results (total CUDA time in seconds) are as follows:

The results are consistent. We will incorporate these findings and clarifications into the revision to resolve potential confusion clearly.

Finally, we would like to emphasize that our main contribution is the proposed LUT-based non-uniform quantization method, which improves quantization accuracy. For inference efficiency, it can similarly benefit from advances in LUT-based kernel implementations, as dequantization-based methods do with their kernels.

Response to Weakness 3

As detailed in Section 4.1 (Lines 313–318, right column) and reiterated in Response to Weakness 2, we follow the evaluation setup used in prior works [1,3] to measure inference latency. Specifically, we report the total CUDA time required for the model to generate 1024 tokens, using a batch size of 1 and no initial input tokens. We will make these settings more explicit in the revised paper to improve clarity and reproducibility.

[1] Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers."

[2] Sun, Mingjie, et al. "A simple and effective pruning approach for large language models."

[3] Kim, Sehoon, et al. "Squeezellm: Dense-and-sparse quantization."

[4] https://github.com/qwopqwop200/GPTQ-for-LLaMa

[5] https://github.com/THUDM/LongBench

Thank you for your thoughtful feedback and strong support. We appreciate your recognition of our contributions and provide detailed responses below.

Comment 1: The context length of perplexity.

Thanks for pointing this out. We use a sequence length of 2048 for all models and methods, following prior work, and will clarify this in the revision.

Suggestion 1: More recent models and model families/sizes:

Thank you for the suggestion. To further validate GANQ, we conducted additional experiments on the latest LLaMA-3.2 models. Below are the WikiText PPL results (sequence length 2048, same setting as in our paper).

The additional results confirm that GANQ consistently achieves superior and more stable quantization performance. Our systematic optimization framework for layer-wise non-uniform quantization allows GANQ to adapt effectively to varying weight distributions, unlike baseline methods, which demonstrate greater sensitivity and instability, especially at 3-bit. We will add these findings in the revision.

Regarding the suggestion to evaluate more model families (e.g., DeepSeek) and larger scales (e.g., LLaMA-3 70B), we appreciate your understanding that such experiments are limited by our hardware (RTX 4090 GPU). That says, since GANQ operates row-wise on linear layers and is designed to adapt to diverse weight distributions, we expect it to generalize well to other architectures and larger models. We plan to extend our evaluations to broader model families and larger scales in the future work.

Comment 2: GPTQ CUDA speedup measurements

We acknowledge that the GPTQ efficiency reported in Table 5 is lower than that in the original GPTQ paper. This discrepancy is mainly due to differences in the inference kernel, hardware, and model size, with the kernel likely being the key factor.

As noted in Section 4.1, we followed prior work [1] and used the GPTQ-for-LLaMA [2] inference kernel for GPTQ-quantized models. The reported results reflect actual observations. However, since [2] is implemented in Triton, and our experiments were conducted on an NVIDIA RTX 4090, potential incompatibilities or limited hardware support may explain the slower inference in our setup.

To further validate this, we performed an additional test using GPTQ's official CUDA kernel implementation [3] (available only for 3-bit quantization and OPT models). The results (total CUDA time in seconds) are as follows:

The results are consistent. We will incorporate these findings and clarifications into the revision to resolve potential confusion clearly.

Suggestion 2: The derivation of the closed-form solution of Equation 5

We will add the detailed derivation of Equation 5 in the appendix of the revision.

Let

by the first-order optimality, let

we have

where (\\cdot)^\\dagger denotes the Moore-Penrose inverse.

Corrections

We agree that "perplexity on language datasets" is a clearer phrase and will adopt this wording in the revision.

Response to Question for Authors

Although both GPTQ and our proposed method (GANQ) quantize model weights in a row-wise manner, the strategies differ significantly. GPTQ uses a greedy, sequential element-wise quantization based on the Optimal Brain Surgeon framework [4], quantizing one element at a time and updating subsequent weights to minimize reconstruction error.

In contrast, our method models LUT-based non-uniform quantization as a mixed-integer quadratic program. Rather than sequentially quantizing elements, we employ an alternating optimization for an entire row with a closed-form solution for the T-subproblem and an efficient back-substitution algorithm for the S-subproblem, enabling more systematic and effective quantization than GPTQ.

[1] Kim, Sehoon, et al. "Squeezellm: Dense-and-sparse quantization."

[2] https://github.com/qwopqwop200/GPTQ-for-LLaMa

[3] Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers."

[4] Hassibi, Babak, and David Stork. "Second order derivatives for network pruning: Optimal brain surgeon."

Thank you for the insightful comments and recognition of GANQ's contributions. Below, we address the concerns raised.

Concern 1: Reliability of Cholesky Decomposition and Sensitivity to Preconditioning of $XX^\top$

Response to Weakness 1

Yes, our method relies on Cholesky decomposition to efficiently solve the S-subproblem. When $XX^\top$ is ill-conditioned, we apply preconditioning to ensure the decomposition remains feasible, maintaining the effectiveness of our framework. This is a standard technique in numerical linear algebra.

Response to Questions for authors

Yes, we implement an adaptive preconditioning method for $XX^\top$ and examine its impact on quantization accuracy and efficiency below.

Remark 3.1 introduces preconditioning $XX^\top$ via $XX^\top+\lambda I$, with $\lambda>0$. In practice, we adopt an adaptive method that enforces diagonal dominance, ensuring positive definiteness without manual tuning $\lambda$.

A symmetric matrix $A$ is positive definite if it is diagonally dominant with positive diagonal entries, i.e., $|a_{ii}| \geq \sum_{j \neq i} |a_{ij}|$ for all $i$. Let $\Sigma = XX^\top$, with $\Sigma_{ii} \geq 0$. For each row $i$, we compute the offset vector $\delta$ as follows:

Thanks for the review and helpful suggestions. We appreciate your positive feedback and address your concerns below.

Comment: $m,n,p$ in line 147 and 148 are not introduced.

Response: $m,n,p$ denote the dimensions in a linear layer. In LLMs, $m$ is the output dimension, $n$ the input dimension, and $p$ the total number of tokens processed (batch size $\times$ sequence length). These values vary by models and layers. For example, in LLaMA-7B:

We use 128 samples with a sequence length of 2048, so $p=128\times 2048$. This highlights the large scale of the problem and the efficiency of our method in solving it.

Suggestion 1: The settings of GANQ$^\*$ (with outlier handling) are not explained ...

Response: We describe GANQ$^\*$'s outlier-handling in Section 3.3 and detail it in Algorithm 2 (Appendix A), which separates outliers based on a ratio $r$, with the rest for quantization. As noted in Lines 365-367 (right), we typically set $r=0.5\\%$ for a fair comparison. We will further clarify this in the revision.

Suggestion 2: Some baselines are missing from the profiling comparison ...

Response: In Table 5, we compare the efficiency of dequantization-based and LUT-based methods. Following [1], we use GPTQ as a representative baseline for dequantization-based method, as similar methods like RTN and OmniQuant can generally share the same inference kernel and offer comparable efficiency. We will clarify this in the revision.

Besides, as in [1], we use GPTQ-for-LLaMa [2] as the inference kernel for GPTQ, which currently supports 4-bit acceleration only. Thus, Table 5 includes only 4-bit GPTQ results. While briefly mentioned in Section 4.3, we will clarify it further in the revision and plan to add results with GPTQ kernels that support 3-bit acceleration.

Suggestion 3: GANQ$^\*$ should be compared in profiling ...

Response: We show the results for GANQ$^\*$ below:

While GANQ$^\*$ offers better quantization quality via outlier handling, it incurs slightly higher inference time due to separate sparse matrix operations. Thus, the choice between GANQ and GANQ$^\*$ should depend on the desired trade-off between quality and efficiency.

We note that 3-bit quantization for OPT-6.7B in GANQ$^\*$ leads to longer inference times than 4-bit, likely due to: (1) memory bandwidth reduction being the main speedup factor (as in our response to Questions for Authors below); (2) sparse outlier operations becoming a bottleneck in this case; and (3) 3-bit values misaligning with byte boundaries (INT8), causing entries to span bytes and adding indexing overhead.

We will clearly present and discuss these findings in the revision.

Suggestion 4: The ratio of outliers and quantized weights could be discussed ...

Response: As in Tables 2 and 3, GANQ is already effective without outlier handling, outperforming baselines and close to FP16 models.

As noted in Section 3.3, adding outlier handling can further improve quantization quality but increases inference time and memory, highlighting a trade-off. To illustrate this, we compare both the PPL and efficiency metrics in one table. For example, on LLaMA-7B:

These results demonstrate a clear trade-off between quantization quality and efficiency. We will include this discussion in the appendix to highlight this balance.

Response to Questions For Authors

GANQ's speedup mainly arises from reduced memory bandwidth usage, which align with observation in other model compression methods (Figure 3 in [3]). By storing weights as 4-bit or 3-bit indices, GANQ greatly lowers memory traffic between global memory and CUDA cores compared to FP16.

While GANQ introduces some overhead from table lookups and weight reconstruction, these are minimal. The lookup table are small (e.g., 16 entries, 32 bytes per row for 4-bit quantization), and thus reside in fast constant or shared memory, adding negligible latency. Weight reconstruction is efficiently fused with matrix multiplication in CUDA kernels, avoiding extra global memory accesses.

In summary, GANQ’s modest overhead is far outweighed by memory savings, resulting in faster overall inference, especially on bandwidth-constrained GPUs.

Response to Other Suggestions

Thanks for catching the typo, we will fix it in the revision.

[1] Kim, Sehoon, et al. "Squeezellm: Dense-and-sparse quantization."

[2] https://github.com/qwopqwop200/GPTQ-for-LLaMa

[3] Lin, Ji, et al. "Awq: Activation-aware weight quantization for on-device llm compression and acceleration."

Thank you for the thorough and insightful feedback. We would like to address your concerns point by point below.

Clarification of References ([1, 2]) and Response to Weakness 2

Thank you for highlighting the relevance of references "Fast Matrix Multiplications for Lookup Table-Quantized LLMs [1]" and "LUT Tensor Core: Lookup Table Enables Efficient Low-Bit LLM Inference Acceleration [2]". As their titles suggest, these works primarily contribute optimized LUT-based matrix multiplication kernels and tensor cores. In contrast, our primary contribution is a novel non-uniform quantization scheme tailored for LUT-based inference. Our method produces quantized representations that can leverage and complement the kernels or tensor cores proposed in [1,2]. Indeed, enabling such compatibility was part of our motivation, which we have explicitly mentioned in the Introduction (Lines 93-95, left column), where [2] has been cited.

Due to different objectives, [1] and [2] focus on end-to-end inference speedup using custom kernels, while our work emphasizes quantization accuracy and compatibility with LUT-based kernels.

For inference speed, we currently use the kernel from [3], as noted in Section 4.1. In the future, we plan to adopt newer LUT-based inference kernels like [1,2] to take advantage of their advancements.

For quantization accuracy, to the best of our knowledge, [2] does not propose a new quantization scheme. Instead, it validates the LUT tensor core by precomputing lookup tables with operator fusion. For [1], it utilizes a learned NormalFloat quantization scheme. As noted in their Section 4.2: "Based on our earlier experiments, we selected a group size of 64, which strikes a good balance between quality and speed." This means that every 64 elements share a lookup table. While this improves quantization quality, it increases memory overhead. For example, for LLaMA-3-8B's $W_{q}$ ($4096\times 4096$), a group size of 64 results in 64 ($4096/64=64$) lookup tables per row (each with 16 FP16 values for 4-bit quantization). In contrast, our method is initially designed to support one lookup table per row (i.e., channel-wise), which is much more memory-efficient. Below is the theoretical model compression ratios (0.5 byte for INT4, 2 byte for FP16):

Besides, as noted in Section 3.3, our method supports outlier-handling techniques by extracting outliers into a sparse matrix, balancing compression and quality. For LLaMA-3-8B, our method already outperforms [1] in 3-bit quantization with just 0.5% outlier separation. In the 4-bit setting, increasing the outlier ratio can further improve quality of GANQ, for example, with a 5% ratio, GANQ achieves a PPL of 6.25, which is comparable to [1] using a group size of 64.

These results highlight a key trade-off between quality and memory, with our method offering a much more compact representation while maintaining strong quality.

We will include these comparisons in the revision to highlight our method's unique benefits and compatibility with existing LUT-based kernels.

Response to Weakness 1

Thank you for raising the concern about real-time inference improvements. To clarify, we included inference benchmarks in Table 5. Following prior works [3,4], we used batch size 1 to generate 1024 tokens and reported total CUDA time (in seconds) using the LUT kernel from [3], as detailed in Section 4.1. The results show clear overall inference speedup and peak memory reduction for OPT-6.7B and LLaMA-7B, highlighting the practical benefits of our method over uniform quantization and FP16 baselines.

We acknowledge the importance of testing across different batch sizes and sequence lengths and will expand our experiments accordingly. However, we emphasize that efficiency is largely determined by the underlying LUT-based kernel implementation.

As noted in our response regarding LUT kernels [1,2], our main contribution is a novel LUT-based non-uniform quantization scheme. We plan to integrate optimized LUT kernels in future work to further improve efficiency. We appreciate the suggestion and will highlight the practical benefits more clearly. We believe that both our work and LUT kernel innovations can jointly make valuable contributions to the community.

[3] Kim, Sehoon, et al. "Squeezellm: Dense-and-sparse quantization."

[4] Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers."

Response to Weakness 3

We fully agree that open-sourcing will enhance reproducibility and impact. We will release the complete code upon acceptance.

Typo

Thank you for pointing out the typo, we will correct it in the revised version.