ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals

Thank you, Reviewer Azbj, for putting effort in reviewing our paper. We provide a response to your concerns below.

1. Limited Novelty: While we agree with the reviewer regarding the statement that ResQ is a rotation and mixed precision quantization based approach, we respectfully disagree regarding the novelty aspect of our work. To the best of our knowledge, there is no work which does what ResQ does and we would be interested if the reviewer can point us to the related paper. In Table 4 of main paper, we also show that ResQ outperforms a hypothetical outlier+rotation based approach which is an amalgamation of related baselines QUIK (EMNLP2024) and QuaRot (NeurIPS2024). This highlights optimality of PCA based high precision component extraction which again to the best of knowledge has not been explored before in literature. Additionally, we provide even more comprehensive comparison in the Table 10 below where we compare ResQ and outlier+rotation baseline's Reasoning and MMLU accuracy in addition to Wikitext perplexity for Qwen2.5 models. Emphasizing that ResQ outperforms the outlier+rotation baseline across the stack. Based on these insights we humbly request the reviewer to re-evaluate their stance on novelty of our work.

Table 10: Comparison of ResQ with outlier+rotation approach.

Detailed task-wise results for the above table can be found here in Table 4.

2. CUDA Kernel: Thank you for highlighting this point. We will include additional details about the CUDA kernel in the main paper and plan to release the code upon acceptance. The CUDA Kernel involves, the mixed precision quantization of activations into 4/8-bit components within a single kernel, low precision GEMM of 4-bit and 8-bit operands and a fused dequantization of 4-bit and 8-bit results. Further, the hardware implementation involves quantizing and packing the mixed precision kv cache for memory efficiency. Additionally, we provide several new hardware-related results to address the reviewer's concerns: improved memory usage with ResQ (Table 5, response to Reviewer vpAp), speedup achieved by ResQ on long context lengths (Table 2, response to Reviewer UnBL), inference latency in a distributed setting representative of datacenter workloads (Table 7, response to Reviewer vpAp), and a comparison of ResQ with a baseline quantization kernel (Table 5, response to Reviewer vpAp). We hope these additions comprehensively address the concerns regarding the CUDA kernel.

3. Figure 4: ResQ and SpinQuant project weights and activations by projection matrices in similar manner. While SpinQuant trains the projection matrices and utilizes uniform precision quantization across the layers, ResQ uses PCA basis and random orthogonal rotations as projection matrices and performs mixed precision quantization. The key differentiation in Figure 4 is the layers and activations which are quantized to mixed precision. For ResQ all the layers and activations except down_proj layer is mixed precision.

Thank you, Reviewer x9DL, for your effort in reviewing our paper. We appreciate your recognition of the strength of ResQ's experimental section. Below, we respond to the concerns you raised.

1. Claim regarding quantization error: ResQ's approach of choosing coefficients along principal eigen vectors of activations to keep in 8-bit is indeed optimal to minimize local quantization error. Projection with PCA basis $P$ minimizes quantization error across high/low precision groups while random orthogonal matrix $R$ improves quantization error within high/low precision group. We will further clarify this point in the main paper. While SpinQuant learns rotation matrices minimizing final loss, it still keeps all the components in 4-bit which does not minimize quantization error enough while ResQ intelligently performs mixed precision quantization. Moreover, SpinQuant's training of rotation matrices can additionally be incorporated in ResQ as well at the cost of high calibration time. The 8-bit components can be chosen via PCA basis ($P$) and orthogonal matrix to minimize quantization error within 4-bit and 8-bit quantization groups ($R$) can be learned minimizing output loss. Such an approach further improves performance of ResQ as shown in Table 8 below surpassing SpinQuant by even higher amount.

Table 8: Performance of ResQ and variant of ResQ which trains rotation $R$.

Detailed task-wise results for the above table can be found here in Table 3.

2. Iso-bitwidth comparison: Both ResQ and previous state of the art mixed precision quantization technique, QUIK (EMNLP2024), keep $\frac{1}{8}$ channels in 8-bit which brings average bit-width to 4.5 bits. For other baselines, achieving fractional bit-width is impossible with uniform precision quantization across layers and channels. Additionally, as suggested by the reviewer, we perform comparison at equal bit-width of 4-bit. We create a variant of ResQ which keeps first $\frac{d}{8}$ channels corresponding to lowest eigen value of $P$ in 2-bit, $\frac{d}{8}$ channels corresponding to highest eigen value of $P$ in 6-bit and remaining in 4-bit to achieve average bit-width of 4-bit. As shown in Table 9 below, even at iso-bitwidth of 4-bit, ResQ outperforms SpinQuant and QuaRot highlighting its capabilities.

Table 9: Iso-bitwidth comparison between ResQ, QuaRot and SpinQuant.

Detailed task-wise results for the above table can be found here in Table 5.

Thank you, Reviewer vpAp, for your effort in reviewing our paper. We are delighted that you find our approach impressive. We provide response to your questions below.

1. Details about compute kernel: The compute kernel goes beyond simple combination of INT4 and INT8 kernels. More precisely, the mixed precision quantization of activations into 4/8-bit components is handled by a single kernel call as opposed to two calls to the quantization kernel. Further, the hardware implementation involves quantizing and packing the mixed precision kv cache for memory efficiency. We will provide more details in the camera ready version of the paper and will release the code upon acceptance of the paper. Additionally, we compare the compute kernel with simple combination of INT4 and INT8 kernels as mentioned by the reviewer below (Table 5 below). The inference latency with ResQ is upto 1.3x lesser than simple combination of INT4 and INT8 kernels.

Table 5: Per decoder latency (in ms) of ResQ over simple compute kernel.

2. Memory reduction with ResQ: We show memory usage of ResQ on RTX 3090 (24GB), FP16 baseline and Quarot (INT4) baseline at different sequence length in Table 6 below. ResQ consumes 1.84-3.08x lower memory than FP16 baseline and requires slightly (4-11%) more memory compared to QuaRot. Further, ResQ supports inference of Qwen2.5-14B where FP16 baseline runs into OOM error.

Table 6: Memory of ResQ and baselines on RTX 3090.

3. Inference on NVIDIA A100: Our intention with ResQ was to target inference on consumer devices, which is why we report speedups on an RTX 3090 at batch size 1. While datacenter-scale 4-bit LLM inference is valuable future work, we also provide latency results for Meta-Llama-3-70B on NVIDIA A100 server (Table 7 below). Notably, ResQ enables the 70B model to fit on a single GPU, whereas the FP16 baseline requires three GPUs. In this setting, ResQ runs data-parallel inference, while FP16 uses model parallelism. ResQ achieves up to 4.98× lower latency across various batch sizes and sequence lengths. More sophisticated model parallel inference approaches like pipeline parallelism will only improve throughput of FP16 baseline but will not improve per batch latency.

Table 7: Meta-Llama-3-70B inference latency (in ms) on 3x NVIDIA A100 GPUs.

Thank you, Reviewer UnBL, for your effort in reviewing our paper and acknowledging the strong empirical results and applicability of ResQ. We answer the listed questions below and incorporate your constructive feedback to further strengthen our work.

1. Heuristic theory: We agree with the reviewer that most quantization error analyses, including ResQ, assume normally distributed activations. In practice, LLM activations deviate from normality. However, in ResQ, the activation distribution becomes approximately Gaussian after projection via the orthogonal matrix $U$ (as shown in Lemma 4.1). To support this, we compute the kurtosis of activations before and after projection, shown in Table 1 below. Post-projection activations $XU_l$ and $XU_h$ exhibit kurtosis near 3, indicating Gaussianity. Thus, our theoretical assumptions hold in practice. Table 1: Kurtosis of Activations before and after projection. |Model|Layer|$X$|$XU_l$|$XU_h$| |---|:---:|:---:|:---:|:---:| |Qwen2.5-3B|Attn|91.9$\pm$38.8|3.0$\pm$0.005|2.9$\pm$0.07| ||MLP|179.8$\pm$248.3|3.0$\pm$0.004|2.9$\pm$0.07| |Qwen2.5-7B|Attn|75.6$\pm$60.5|3.0$\pm$0.002|3.0$\pm$0.02| ||MLP|164.7$\pm$243.1|3.0$\pm$0.0|2.9$\pm$0.04| |Meta-Llama-3-8B|Attn|37.9$\pm$50.3|3.0$\pm$0.0|3.0$\pm$0.02| ||MLP|6.6$\pm$1.4|3.0$\pm$0.0|3.0$\pm$0.02|
2. Distributed Inference setting : Please refer to Table 7 in comment to Reviewer vpAp.
3. Long Context Inference: We provide per decoder speedup (similar to Fig. 5 in paper) for longer context lengths upto 20000 on Qwen2.5-32B and Meta-Llama-3-70B models in Table 2 below. With increasing sequence lengths, the improvements of ResQ slightly reduces still achieving 2.33x (2.02x) for Qwen2.5-32B (Meta-Llama-3-70B) speedup at sequence length of 20k.

Table 2: Per decoder speedup of long context inference on RTX 3090.

4. Flexible Subspace Dimension: It is possible to assign different ranks across layers for $U_B$ and $U_C$ projections since they are different for each layer. It is not possible to have different ranks for $U_A$ with the current approach since it is shared across layers. We test one approach of assigning different rank of 8-bit component to $U_B$ and $U_C$ based on the eigen value distribution of key and value. Specifically, for layers with higher eigen values, we keep 15.6% channels in 8-bit while for the rest we keep 9.3% channels in 8-bit achieving 12.5% high precision components in average. Results with such an approach (shown in Table 3 below) show that flexible rank improves the performance on reasoning accuracy but performs worse on MMLU demanding important future exploration. Table 3: Comparison of ResQ with a variant of ResQ having flexible rank of $U_B, U_C$. |Model|Method|Wiki.PPL|Avg.Reasoning acc.|Avg.MMLU acc.| |:---:|:---:|:---:|:---:|:---:| |Llama-3.2-3B|ResQ|8.8|59.0|49.8| ||ResQ-flex.rank|8.7|59.2|49.0| |Meta-Llama-3-8B|ResQ|7.1|63.9|57.2| ||ResQ-flex.rank|7.0|64.3|56.8| |Qwen2.5-3B|ResQ|9.0|61.1|61.2| ||ResQ-flex.rank|9.0|61.9|61.3| |Qwen2.5-7B|ResQ|8.2|65.3|69.0| ||ResQ-flex.rank|8.0|65.4|68.6|

Task-wise results can be found here in Table 1.

5. Codebook-based quant: Codebook-based non-linear quantization methods (e.g., AQLM mentioned by the reviewer) focus solely on weight quantization, whereas ResQ takes a more holistic approach by quantizing weights, activations, and KV cache. To the best of our knowledge, no existing work explores non-linear quantization of LLM activations. This is an interesting direction, and we leave it for future work. Thank you for bringing AQLM to our attention; we will cite the work.

6. Calibration Dataset: To analyse the impact of calibration datasets we evaluate performance of ResQ on 3 different datasets. These include two out of distribution language modeling datasets C4 and PTB and one instruction tuning dataset Alpaca. The results provided in Table 4 below shows no significant performance fluctuations with different calibration datasets.

Table 4: Performance of ResQ with different calibration datasets.

Task-wise results can be found here in Table 2.