PolarQuant: Leveraging Polar Transformation for Key Cache Quantization and Decoding Acceleration

We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal helps address your concerns.

---

W1 & Q2: Figures need refinement in presentation.

Thank you for your helpful suggestion. We will refine Figure 1 and Figure 2 to more effectively showcase our insights and pipeline.

---

Q1: Can PolarQuant combine with the rotation-based weight quantization?

We find your question interesting, but in this work we primarily lies in activation quantization (e.g., KV cache quantization) rather than weight quantization. There are differences between the two domains, both in their core focuses and in the inherent characteristics of weights versus activations. This means we cannot immediately speak to the feasibility of combining PolarQuant with rotation-based weight quantization.

We acknowledge the value of exploring such intersections and need some time to delve into the paper you referenced to gain deeper insights, which may inform and guide the development of our future work. Thank you again for your interest and for bringing this perspective to our attention.

---

Q3: Ablation on the group size.

Thank you for the insightful suggestion! We conduct an ablation study with different group sizes using Llama-3.1-8B-Instruct. The average results across datasets from LongBench are presented below:

We find that PolarQuant maintains consistent performance across different group size settings.

Thank you for the suggestion, we will include this discussion in our revised version.

---

Q4: Breakdown time analysis of key cache quantization in PolarQuant.

Thank you for your thoughtful question. Your input is valuable in refining our paper’s presentation.

We perform a time breakdown analysis of PolarQuant during the generation phase (with a batch size of 1, input length of 256, and generation length of 4096):

From this analysis, we can see that the overhead introduced by the new key quantization is only 0.6% in comparison to the overall generation cost.

We appreciate your suggestion and hope this can clarify your concerns.

---

Q5: Details about the GPU architecture.

We benchmark our experiments on NVIDIA A800-SXM4-80GB. Thank you for the reminder, I will detail the hardware we use in the revised paper.

---

Q6: Why the angles need more precision?

We notice a significant decline in downstream performance when fewer than 3 bits are assigned to the angles. As indicated in Table 5, the (r4, t2) configuration shows a considerable performance drop compared to both the (r3, t3) and (r4, t3) settings.

The angle should have higher precision to ensure that the differences in rotation angles between token positions are as distinguishable as possible, because RoPE introduces positional information based on the differences in these rotation angles.

Specifically, for the lower dimensions of RoPE, after increasing the token position by a group size (g), they undergo multiple full cycles (i.e., the input is rotated by more than 2π). As a result, the quantization of these high-frequency signals in the lower dimensions of RoPE requires higher precision.

Our experimental results in Table 5 also validate the above point.

---

We sincerely appreciate your valuable comments! We hope our response will help address your concerns.

---

W1: Figure 1 needs refinement in presentation.

Thank you for the valuable suggestion to improve the presentation of our paper. We will choose more contrasting colors to better highlight the presence of outliers, thereby enhancing the overall presentation of the paper.

---

W2 & Q1：Why does Table 4 show PolarQuant using less memory than KIVI despite lookup tables?

First of all，here is a misunderstaning that, the lookup table in PolarQuant incurs extra memory overhead.

We want to clarify that the lookup table is actually not cached; instead, it is computed on the fly at block level during decoding (For more details on the implementation, please refer to Appendix A). As a result, therotically there should be no obvious difference in memory overhead for PolarQuant and KIVI when the same bit budget is used.

Second, your observation is insightful that the setups in Table 4 show differences in memory usage, and there are some theoretical discrepancies.

• For the second comparison group (KIVI-2 & Polar-33):

Polar-33, has higher memory usage than KIVI2 (with more than 1GB at 128K), which is due to the fact that we use our customized 4-bit kernel for Polar-33 in our implementation, introducing 1 bit of redundant memory waste.

The 3-bit kernel implementation requires additional integer unpacking and is complicated to implement, so we use our 4-bit kernel as a replacement.

• For the third comparison group (KIVI-4† & Polar-44†):

We assume that some hardware issues might be the cause and are currently investigating.

In Polar_44†, we use our kernel implementation for key and KIVI kernel for value. In KIVI-4†, the dequantization and matmul operations for both the key and value share the same kernel. If the intermediate results by repeated kernel calls are not released, GPU memory usage may accumulate continuously, causing peak memory usage to rise.

We are actively working on pinpointing the memory differences mentioned above. Thank you for your question!

---

Q2: Ablation studies on group size g ?

Thanks for your suggestion! We conduct an ablation study exploring different group size using Llama-3.1-8B-instruct.

The table below presents the performance of PolarQuant and KIVI on LongBench with a 4-bit setting (* indicates better preformance with same group size):

This suggests that PolarQuant performs competitively with KIVI across different group sizes.

With small g, there might not be outliers in each group. Will PolarQuant still maintain its superiority under that circumstance?

We would like to make two points in response:

(1) The quantization parameters are of size 32/g bits. It is clear that smaller group sizes introduce an additional cost.

(2) Considering the overhead of quantization parameters caused by a smaller g and the sufficiently good performance with a relatively large g, we recommend using 128, which is the setting used in our paper.

---

We sincerely appreciate your valuable comments and hope our responses address your concerns.

---

Q1: More diverse model architectures like Qwen3 ?

In our experiments, we focus on popular model architectures, such as Llama and Qwen, as we believe they have demonstrated the broad applicability of PolarQuant across different architectures.

Your observation is very insightful, and we agree that the pre-RoPE normalization holds significant value. Given the time constraints, we have decided to leave the exploration of integrating PolarQuant with Qwen3 for future work. Thank you for your valuable suggestion.

---

Q2: How was the value cache quantization method chosen ?

Our paper focuses on addressing the challenges of quantizing keys, and is orthogonal to specific value quantization algorithms, as demonstrated in the following aspects:

1. In Tables 1, 2, 3, 5, we maintain the value cache in full precision. This effectively isolate the influence of value quantization algorithms and clearly demonstrate the effectiveness of our method in addressing the challenges of key quantization.
2. After establishing the effectiveness of our method, in Table 6, we employ KIVI to quantize values. This step is taken to showcase the compatibility of PolarQuant with value quantization algorithm, thus making our paper more comprehensive. We selected KIVI simply because it is a classic method and a frequently compared baseline in the field.

Taken together, these twofold experiments demonstrate that PolarQuant works well with both full-precision values to low-bit quantized values using classic methods—without performance degradation.

---

Q3: How sensitive is PolarQuant to the RoPE configuration? Could this affect its generalizability across diverse model architectures or custom RoPE implementations?

Thank you for the insightful question. We have experimented with diverse model architectures, each with a unique RoPE setup, including Llama-2-7B-Chat (base frequency 10000), Llama-3.1-8B-Instruct (with rope base as 500000), and Qwen-2.5 (with rope base as 1000000).

In addition, we show that PolarQuant is adaptable to different RoPE variants, which addresses your concern about its generalization to custom RoPE implementations. Specifically, we perform experiments with NTK-aware Scaled RoPE on Llama-2-7B-Chat to demonstrate this.

The experimental results indicate that PolarQuant can adapt well to the customized RoPE configuration, hope this can address your concern.

Thank you for valuable feedbacks! Hope our explanation helps address your questions.

---

W1: The authors do not justify that the overhead of RoPE recomputation in previous work is substantial, particularly since KV cache loading is memory bandwidth bound.

Thank you for the suggestion.

First of all, we contend that the overhead of RoPE recomputation is substantial.

1. Perspective of Theoretical Analysis

KVCache loading in full precision is memory-bound [1, 2], but low-bit KV cache loading with matrix multiplication dequantization does NOT exhibt the same on more advanced GPU architectures.

KVQuant utilizes A6000 to analyze kernel efficiency, whereas our study uses the A800, which is a more advanced GPU and reflects a more practical scenario. We monitor the Memory Throughput and Compute Throughput of the vecquant4matmul kernel of KVQuant on A800. The NCU report indicates that such kernel implementations exhibit a computation-bound.

2. Perspective of Experimental Results

We conduct further experiments to empirically confirm that the overhead of RoPE recomputation is substantial.

On A800, we benchmark the latency of the KVQuant kernel implementation, as well as its end-to-end throughput performance in Transformer. For batch size, we identify the maximum supported batch size for a generation length of 4K, with other experimental settings following those outlined in Section 4.2.

The experimental results are as follows:

Latency comparisons of kernels across context lengths

Throughput comparison

These two aspects will demonstrate that the overhead of RoPE recomputation is substantial.

We hope the reviewer can reconsider the novelty of PolarQuant, given that the overhead of RoPE recomputation is substantial.

1. We highlight the limitations of KVQuant, where in real-world scenarios like on A800, the recomputation of RoPE is substantial, leading to efficiency issues.

2. PolarQuant introduces a novel polar transformation approach to key cache quantization that avoids this recomputation.

Similar to KVQuant, our method utilizes pre-RoPE key properties. However we argue that the polar transformation is an innovative approach to preserve pre-RoPE benefits without the recomputation burden, leading to faster inference across a wider range of applications. We bilieve this is one of our contributions.

Thank you for the insightful suggestion. We hope you will reconsider your evaluation of the novelty of our work. We will include the necessary discussion in the revision to clarify any potential misunderstandings.

References:

[1] SqueezeLLM: Dense-and-Sparse Quantization

[2] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

---

W2: The novelty of PolarQuant's use of lookup table.

We highlight our novelty and contribution by distinguishing (1) the usage of lookup tables and (2) the effectiveness of lookup tables ultization between PolarQuant and KVQuant.

Differences in lookup table usage scenarios

• KVQuant utilizes a lookup table to dequantize the key cache and restores the positional information of key values via RoPE recomputation.
• PolarQuant avoids RoPE recomputation; instead, it directly constructs a lookup table for QK product on the fly to achieve acceleration.

Differences in effectiveness

As highlighted in our response to W1, the cost of RoPE recomputation has become non-negligible on advanced GPUs, emerging as a bottleneck that limits KVQuant’s efficiency. On the other hand, PolarQuant offers improved acceleration.

In summary, our use of the lookup table is neither a follow-up nor an incremental improvement over KVQuant. While lookup tables are a common technique in quantization [1, 2, 3], many aspects of PolarQuant are novel:

• The challenging problem we address.
• The rationale behind our application of lookup tables.
• The specific objects our tables target.

References:

[1] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

[2] PQCache: Product Quantization-based KVCache for Long Context LLM Inference

[3] PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decoupling

---

W3 & Q2: Is Key cache more sensitive to low-precision quantization than Value cache?

Yes. In Section 5.2, we combine 4-bit key quantization (PolarQuant_44) with 2-bit value quantization and observe minimal performance degradation.

Additionally, we test value quantization (KIVI with group size 128) on LongBench while keeping the key cache at full precision and resulting in no performance drop. However, when quantize key in same bit and keep value in full precision, the performance greatly drops.

These results suggest that:

1. Existing quantization methods are already sufficiently effective in value quantization.
2. Key quantization is more sensitive to precision changes than value quantization in current methods. As a result, key quantization presents more challenges than value quantization.

Thank you for the suggestion, we will include this discussion in our revised version.

---

Q1: Overhead of quantizing new keys ?

The following is the detailed time breakdown for the PolarQuant_44 during the decoding phase (with a batch size of 1, input length of 256, and generation length of 4096):

From this, we can conclude that the overhead of quantizing new keys is 0.6% compared to other operations. Quantization is performed every n steps, causing only a slight overhead during decoding, which explains why it is negligible.

---