We sincerely thank you for your thorough feedback. Below, we address the concerns raised and outline revisions to improve the clarity and rigor of our work.

---

1. Key Cache Importance

• The reviewer correctly observes that in certain layers (e.g. Layer 0,1,2,31 of Llama-3.1-8B-Instruct), the proposed intra-layer KV cache precision pair pruning algorithm selects K4V8 rather than K8V4. We will clarify in the revised manuscript that Claim 1 ("key cache is generally more important") reflects an overall trend across layers. KVTuner assigns more bits to value cache than key cache in specific layers. Those layers where higher value bitwidths outperform higher key bitwidths normally have more streaming head patterns than retrieval heads, where key is more robust to low-bit quantization. Therefore, more memory bitwidths should be assigned to value cache, which is more sensitive in this case.

• This phenomenon verifies our observation and theoretical analysis about the strong correlation of KV cache quantization errors and attention patterns in Section 4.4. In addition, the proposed KVTuner is adaptive to the inherent model structure patterns and the proposed layer-wise KV cache quantization precision tuning makes sense.

---

2. Comparison with KIVI

We appreciate the reviewer’s attention to fairness in comparisons. KVTuner offers more flexible accuracy and efficiency tradeoff, which is not available in the static, uniform and non mixed-precisionbaselines KIVI and per-token-asym.

• The accuracy of KVTuner-C3.44 only decreases by 0.52% but the memory usage is reduced by 14% compared with KIVI-4 in Qwen2.5-3B-Instruct. We also push the frontier of nearly lossless KV cache compression (only 0.04% accuracy loss than the BP16 baseline) to 3.44-bit in this model.
• In Table 11 (Page 24~25), we compare more uniform precision including K8V4 (C6), K8V2 (C5), and K4V2 (C3). Llama-3.1-8B-Instruct and Mistral-7B-v0.3 are more robust to low bit key quantization than Qwen2.5-7B-Instruct. However, Qwen2.5-7B-Instruct outperforms others in terms of accuracy.
• In Table 5 and 6 (Page 8), the baselines INT4 KIVI and per-token-asym quantization lead to significant (67% and 26%) accuracy degradation in Qwen2.5-7B-Instruct. However, KVTuner successfully reduces the accuracy loss to 16% with lower 3.92-bit memory usage and 0.18% with similar 4-bit memory usage, respectively. It indicates that the accuracy improvement of KVTuner is noticeable and KVTuner offers more robustness and flexibility to sensitive but powerful models.
• Figure 5 also visualizes the accuracy and equivalent KV cache bitwidth of different layer-wise KV cache precision pairs during KVTuner offline searching. The red circles are uniform precision across all layers. From Figure 5, we can easily observe more Pareto-optimal layer-wise KV cache precision pairs than uniform ones. Especially, the accuracy of the uniform KV4, K4V2, and K2V4 is around 0% in Qwen2.5-7B-Instruct, while the searched equivalent 4-bit and 3-bit configs of KVTuner achieve 80% and 50% accuracy, respectively. It is a huge improvement in terms of accuracy with similar memory usage.
• Latency Baseline Clarification In Table 7, "KV8" refers to 8-bit KIVI quantization. We thus report the total model-level throughput comparison of Llama-3.1-8B-Instruct using the searched config in Table 5. The hardwares are Nvidia RTX 4090 24G. Compared with KIVI-KV8, the throuhgput of KVTuner-C3.25 can be improved by 16.79%~21.25%. |BS, inputLen|KV8(baseline)|K8V4|KV4|K4V2|KVTuner-C4.92|KVTuner-C3.25| |-|-|-|-|-|-|-| |64,128|3836|4193|4567|4697|4240(10.53%)|4652(21.25%)| |8,1024|549|597|632|645|600(9.22%)|641(16.79%)|

---

3. Long context evaluation

We compare KVTuner with the basedlines in the 20 Longbench datasets and the averaged scores are as below. The conclusion is that KVTuner push the nearly lossless long context generation to 3.25-bit.

Qwen2.5-7B-Instruct KIVI

Qwen2.5-7B-Instruct per-token-asym

---

4. Per-channel vs. per-token quantization

KIVI-HQQ also supports key per-channel quantization by tuning the axis_key config. KIVI requires new operators and careful management as discussed in Line 58~69. In contrast, per-token-asym can be easily implemented and is supported in common inference frameworks such as LMDeploy.

---

5. Layer-wise sensitivity analysis

We also study the layer-wise sensitivity to KV cache quantization of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct with both KV per-token-asym and KIVI like quantization modes on the non-math AIGC multiturn softage dataset in Figure 9 (Page 17), 10 (Page 18), 12 (Page 20), and 13 (Page 21). The sensitive layers are consistent with those in math tasks e.g. Figure 8.

Thank you for your thorough review and constructive feedback. We sincerely appreciate your acknowledgment of the innovativeness and feasibility of the proposed methodology and theoretical analysis of attention patterns to KV cache quantization, as well as your recognition of the comprehensive implementation of the experimental investigations. Below, we address your concerns and suggestions point by point:

---

1. Efficiency evaluation clarifications

We acknowledge the lack of clarity in our efficiency evaluation setup. Here are key clarifications:

• Baseline Implementation: In Table 7, the static SOTA baseline KIVI and our solutions are both based on the official KIVI code with flashattention enabled in their kernels during decoding. KV8 and KV4 are KIVI-8 and KIVI-4, respectively. We will correct them in the revised version. We slightly modify the KIVI CUDA kernels to support INT8 and mixed precision of key and value cache. KIVI, per-token-asym, and our KVTuner is a online static method and compatible with FlashAttention. In addition, It uses lightweight post-training calibration and offline bit-width selection. This design choice ensures compatibility with existing inference frameworks. However, other online mixed precision KV cache quantization methods with attention scores based online token importance estimation is normally not compatible with flashattention.

• New GQA efficiency results with KVTuner: We modify the KIVI code to support GQA models including Llama-3.1-8B-Instruct. We thus report the total model-level throughput comparison of Llama-3.1-8B-Instruct using the searched config in Table 5. The hardwares are Nvidia RTX 4090 24G. Compared with KIVI-KV8, the throuhgput of KVTuner-C3.25 can be improved by 16.79%~21.25%. |BS, inputLen|KV8(baseline)|K8V4|KV4|K4V2|KVTuner-C4.92|KVTuner-C3.25| |-|-|-|-|-|-|-| |64,128|3836|4193|4567|4697|4240(10.53%)|4652(21.25%)| |16,512|1102|1205|1275|1304|1239(12.41%)|1296(17.55%)| |8,1024|549|597|632|645|600(9.22%)|641(16.79%)|

• Throughput definition: We follow the same settings and definitions of KIVI. Throughput is defined as the the number of tokens generated per second (measured end-to-end, including quantization/dequantization overhead). For example, if the batch size is 128 and one generation step takes 50ms, the throughput is 128 * 1000 / 50 = 2560 tokens/s.

---

2. Discussion of conflicting literature

We appreciate the opportunity to clarify our position relative to prior works (Atom and KIVI): Atom claims that KV cache is more amerable to quantization than activation matrices in Section 4.4 and utilizes INT4 precision for both key and value. The importance difference of key and value is not clearly discussed in Atom. KIVI implicitly implies that key is more important than value by applying more complex per-channel per-group quantization to key and simple per-token quantization to value.

The conclusion that key cache is normally more important than key cache is validated with extensive empirical studies, which include final perplexity on the wikitext dataset in Table 2, layer-wise attention errors on the GSM8K dataset in Table 3 (Page 4), final model accuracy on the general CEVAL, MMLU, TriviaQA, RACE, and TruthfulQA datasets with both per-token-asym and KIVI quantization modes in Table 11 (Page 25), layer-wise attention score and output errors with the key per-channel-asym and value per-token-asym quantization on AIGC multiturn softage dataset of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct in Figure 10 (Page 18) and Figure 13 (Page 21).

---

3. OOM with 8K sequence

We use 4 batch size with 8K tokens (~4GB). The OOM error for the 8K FP16 setup may arise from an unoptimized memory allocation strategy in our prototype implementation (e.g., redundant intermediate tensors were not freed). The default efficiency testing in the KIVI repo does not enable FlashAttention during prefilling, which may also result in the OOM issue with 8K long sequence.

---

Conclusion

The proposed layer-wise KV cache precision pair tuning naturally suits the layer-wise sensitivity to KV cache quantization, making KVTuner a practical solution to reduce the memory usage and improve inference efficiency of LLMs with various sensitivities. KVTuner successfully pushes the nearly lossless KV cache quantization in complex mathematical and sciencitific tasks to 3.25-bit for Llama-3.1-8B-Instruct and 4-bit for sensitive Qwen2.5-7B-Instruct. KVTuner also greatly narrows the performance difference between the simple per-token-asym and accurate KIVI quantization modes, even when using overall similar low-precision settings.

Many KV cache quantization approaches are proposed recently, but the correlation to LLM attention patterns is not well studied. We theoretically prove that the sparse streaming heads are robust to KV cache quantization than sensitive retrieval heads, which is the cause of layer-wise sensitivity to KV cache quantization.

We sincerely thank you for the thoughtful feedback and constructive critiques. Below, we address each concern and outline planned revisions to strengthen the paper:

---

1. Computational cost

The profiling and layer-wise KV cache precision tuning are completely offline and no online overhead for precision selection is introduced. Only online quantization for the newly generated key and value token using the calibrated precision similar to uniform KV precision is applied.

We only perform 200 rounds of offline multi-objective optimization search with limited data, which is efficient and one-time cost. This is a huge advantage compared with other mixed-precision approaches that incorporate token-importance ranking for KV selection. The two-stage search space pruning only takes seconds. The MOO search with pruned search spaces is the most time consumping part, which performs 200 rounds of the target LLM on the selected 200 prompts. The optuna framework is also quite efficient. The simple huggingface transformers may takes hours in the above settings with single batch inference in Nvidia RTX 4090. The offline tuning time cost is acceptable compared with the saved cost during large-scale online service and model pretraining cost. In addition, better hardwares, torch graph compiling, and multi-batch inference during tuning may reduce the tuning cost to less than one hour.

---

2. Latency experiment

The layer-wise FLOP cost difference is mainly caused by efficiency difference of the KV cache precision pairs. The model-level efficiency reflects the overall effects of layer-wise efficiency of all KV cache precision pairs. The memory movement cost from CPU sto GPUs linearly increases with the KV cache size in most case and attention is normally memory bounded. We also report the total model-level throughput comparison of Llama-3.1-8B-Instruct using the searched config in Table 5 as below. The hardwares are Nvidia RTX 4090 24G. Compared with KIVI-KV8, the throuhgput of KVTuner-C3.25 can be improved by 16.79%~21.25%.

---

3. Long context effectivenss

We compare KVTuner with the basedlines KIVI-8, KIVI-4, our proposed variant KIVI-K8V4, and per-token-asym ones in the 20 Longbench datasets and the averaged scores are as below. KVTuner pushes KV cache quantization for the nearly lossless long context generation to 3.25-bit, outperforming the uniform KV precision.

Qwen2.5-7B-Instruct KIVI

Qwen2.5-7B-Instruct per-token-asym

---

3. Global optimum

Due to the complex and nonlinear dependency of error accumulation, there are no theoretically global optimization solutions for the NP-hard problem. The MOO formulation balances Pareto-optimal solutions rather than seeking global optimality. The two-stage search space pruning aims to help MOO converge and reduce searching cost, comparing Figure 5 and 6.

---

4. Scaling over batch size

When the input and output sequence lengths are fixed at 512 and 128, respectively, the following table presents the throughput (in Token/s) of KVTuner.

---

5. Integration with eviction

We agree that integrating with eviction methods is promising and leave it in our future work. KVTuner is fully compatible with KV cache eviction methods including StreamingLLM, H2O, and SnapKV, because quantization and eviction are two orthogonal approaches.

---

6. MHA

We mainly test on LLMs with grouped query attention (GQA), because most recent and powerful models are GQA based and GQA is a variant of MHA. The layer-wise sensitivity to KV cache quantization is the inherent property of LLM with multi-layer transformers. In addition, key and value in attention layers have different sensitivity to quantization. There are the only two assumptions of KVTuner. We also analyze MLA models such as Deepseek-v2-lite-chat, in which we also observed similar layer-wise patterns.

---

Conclusion

KVTuner’s practical impact lies in its deployability: it requires no inference-time overhead and achieves near-lossless accuracy, making it a compelling solution for production LLM systems and most LLM acceleration hardwares. In addition, we study the underlying mechasim of the higher importance of key cache. We also theoretically analyze that the layer-wise sensivity of attention heads to KV cache quantization strongly correlates with attention patterns, which is novel and may provide more insights to model design and compression.