KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction

Dear reviewer, thank you for your valuable time and effort in reviewing our paper. We have addressed the questions and feedback raised below. We will carefully incorporate the following discussion into our revision.

Q) Weakness 1. positional out-of-distribution issues during the reconstruction process

• By employing chunking in our context reconstruction method (Figure 7), we limit the increase in position ID to the size of each chunk (i.e., 2K tokens) during importance scoring, thus eliminating the mentioned concerns.
• Specifically, for each chunked reconstruction, the position ID starts from the last position ID of the prefilled tokens. Since the chunk size (2K tokens) is less than 2% of the 128K token context length limit for LLaMA-3.1, its impact is negligible and does not introduce OOD issues for our benchmark datasets.
• Additionally, the impact on position ID can be further reduced by decreasing the chunk size without incurring degradation in compression performance.

Q) KVZip is inherently prefill-heavy due to its autoencoding nature, which may limit its applicability in resource-constrained on-device environments.

• We emphasize that KVzip unlocks new application scenarios beyond the conventional online KV eviction setting. Specifically, KVzip produces reusable, accurately compressed KV caches suitable for multiple future queries, thus supporting offline context KV cache compression. This capability is particularly beneficial in real-world use cases such as enterprise document retrieval–augmented systems or personalized user contexts (lines 34–37), where the compression overhead can be amortized over subsequent queries. Existing dynamic KV eviction methods, however, incur significant performance degradation in multi-query scenarios, limiting their applicability in such scenarios (Figure 9).
• Moreover, even in online settings, KVzip's importance-scoring method seamlessly extends to head-level eviction, introducing no additional compression overhead during deployment (see Context-Independent Eviction, Section 4.2). Compared to DuoAttention, a state-of-the-art head-level eviction technique, KVzip provides superior accuracy with substantially lower optimization costs (lines 270–275 and Figure 11).

Q) Could you provide a more detailed explanation of the intuition in Section 3.1, particularly lines 104-106 and 107-111?

• The main insight behind KVzip is ensuring that the LLM's knowledge system (KV cache + model weights) retains the complete context information. We believe this completeness is a necessary condition in a query-agnostic setting, as arbitrary queries may require any detail in the context. Our proposed approach validates this completeness via prompting.
• [lines 104-106] Suppose we reconstruct the original context from the compressed KV cache through prompting. Successfully reconstructing the context implies that we can regenerate the original KV cache by performing a separate prefilling using this reconstructed context. This means we can obtain the original response (without compression) to any arbitrary query by using the compressed KV cache. This forms the intuition behind our context reconstruction-based KV cache compression method.
• [lines 107-111] However, this approach would require a re-prefilling step, which is practically infeasible. Fortunately, we observe empirically that performing direct inference with the compressed KV cache through context reconstruction (Figure 4, right) effectively solves downstream tasks without the need for re-prefilling of the full KV cache.
• These are detailed explanations of the requested lines. If you need any further clarification, please let us know.

Q) In Section 4.2's comparison with DuoAttention, could you specify exactly how head-level scores were measured and computed?

• We first obtain a KV importance score $S$ of shape $L\times H\times n_c$, where $L$ is the number of layers, $H$ is the number of heads, and $n_c$ is the context token length. We compute this score using a single English book sample containing 88K tokens from En.QA in SCBench (line 261). Next, we take the maximum value along the sequence dimension as $\max_i S[:,:,i]$, which serves as the head-level score in our experiments (line 260). The pseudocode in Appendix B describes this computation.
• Note that we use the head-level score derived from a single sample of En.QA for all benchmark datasets, ensuring a fair comparison with DuoAttention.

Q) Have you compared using sum/average aggregation methods instead of max values for attention scores?

• We've tested average aggregation and compared against our max aggregation approach. Note that sum and average aggregation yield identical eviction priorities, making them equivalent.
  | Method \ Compression ratio | 0.1 | 0.2 | 0.3 | 0.5 | 1.0 | |----------------------------|------|------|------|------|------| | Max | 64.3 | 93.4 | 94.3 | 93.6 | 93.1 | | Average | 41.2 | 79.5 | 90.1 | 93.2 | 93.1 |

  Table. Average accuracy (%) of Qwen2.5-7B on SQuAD dataset.

• The results shown in the table above clearly demonstrate that max aggregation outperforms average aggregation. This empirical finding is the primary reason we adopted max aggregation in our algorithm.

• We attribute this result to the sparsity of attention patterns. As illustrated in Figure 16 (Appendix D), LLMs tend to exhibit sparse attention distributions. Consequently, average aggregation can dilute the significance of tokens receiving high attention from specific queries, reducing the perceived importance of KV pairs. In analyzing cases where average aggregation failed, we found that the method often struggled with accurately retrieving words or phrases, negatively affecting various downstream LLM tasks such as QA, coding, passkey retrieval, and reasoning.

Q) Is the repeat prompt robust to alternative prompt designs?

• The choice of repeat prompt was motivated by simplicity, as the specific wording of the repeat prompt has minimal impact on overall performance.

• To validate this, we conducted experiments comparing: (1) the original repeat prompt, (2) a paraphrased version ("Reproduce the preceding context without any changes."), and (3) no repeat prompt (just "\n\n"). Note that we follow the chat template provided by each model (e.g., "<user> [context] [repeat prompt] <assistant> [repeated context]").
  |Repeat prompt type|Accuracy (%)| |---|---| |Original|94.37| |Paraphrased|94.45| |No|94.25|

  Table. Accuracy (%) of Qwen2.5-7B on SQuAD at a 30% compression rate (70% eviction). SnapKV achieves 32.15% in this setting.

• These results show that our method is robust to variations in the repeat prompt. The limited impact arises because the repeat prompt (7 tokens with Llama 3.1) is significantly shorter than the overall context (at least several hundred tokens), thereby minimizing its effect on compression.

• To further clarify this, we analyzed attention patterns. Specifically, we measured the proportion of prefilled KV pairs whose maximum cross-attention scores during reconstruction originated from the repeated context rather than the repeat prompt (see Figure 4).

  • For a 2K token-length context from NIAH, 98.1% of KV pairs had their maximum attention from the repeated context. Among the KV pairs retained after 30% compression, 99.4% derived their maximum attention from the repeated context. These findings confirm the minimal influence of the repeat prompt on KVzip’s importance scores and compression performance.

• It is also interesting to observe that context reconstruction without a repeat prompt achieves comparable performance. This indicates that forwarding the context twice effectively captures the critical attention patterns required for downstream tasks (see Figure 6).

Q) The following recent works related to query-agnostic approaches might be valuable additions to the related work section.

• Thank you for the suggestion! We added the suggested related works in our revision.

Dear reviewer, thank you for your valuable time and effort in reviewing our paper. We have addressed the questions and feedback raised below. We will carefully incorporate the following discussion into our revision.

Q) The choice of the reconstruction prompt—“Repeat the previous context:”—is insufficiently justified and appears overly heuristic. This paper doesn’t explore or compare alternative formulations of the reconstruction prompt.

• This choice of repeat prompt was motivated by simplicity, as the specific wording of the repeat prompt has minimal impact on overall performance.

• To validate this, we conducted experiments comparing: (1) the original repeat prompt, (2) a paraphrased version ("Reproduce the preceding context without any changes."), and (3) no repeat prompt (just "\n\n"). Note that we follow the chat template provided by each model (e.g., "<user> [context] [repeat prompt] <assistant> [repeated context]").
  |Repeat prompt type|Accuracy (%)| |---|---| |Original|94.37| |Paraphrased|94.45| |No|94.25|

  Table. Accuracy (%) of Qwen2.5-7B on SQuAD at a 30% compression rate (70% eviction). SnapKV achieves 32.15% in this setting.

• These results show that our method is robust to variations in the repeat prompt. The limited impact arises because the repeat prompt (7 tokens with Llama 3.1) is significantly shorter than the overall context (at least several hundred tokens), thereby minimizing its effect on compression.

• To further clarify this, we analyzed attention patterns. Specifically, we measured the proportion of prefilled KV pairs whose maximum cross-attention scores during reconstruction originated from the repeated context rather than the repeat prompt (see Figure 4).

  • For a 2K token-length context from NIAH, 98.1% of KV pairs had their maximum attention from the repeated context. Among the KV pairs retained after 30% compression, 99.4% derived their maximum attention from the repeated context. These findings confirm the minimal influence of the repeat prompt on KVzip’s importance scores and compression performance.

• It is also interesting to observe that context reconstruction without a repeat prompt achieves comparable performance. This indicates that forwarding the context twice effectively captures the critical attention patterns required for downstream tasks (see Figure 6).

Dear reviewer, thank you for your valuable time and effort in reviewing our paper. We have addressed the questions and feedback raised below. We will carefully incorporate the discussion into our revision.

Q) The key-value eviction process itself is computationally expensive and requires substantial time overhead.

• We emphasize that KVzip unlocks new application scenarios beyond the conventional online KV eviction setting. Specifically, KVzip produces reusable, accurately compressed KV caches suitable for multiple future queries, thus supporting offline context KV cache compression. This capability is particularly beneficial in real-world use cases such as enterprise document retrieval–augmented systems or personalized user contexts (lines 34–37), where the compression overhead can be amortized over subsequent queries. Existing dynamic KV eviction methods, however, incur significant performance degradation in multi-query scenarios, limiting their applicability in such scenarios (Figure 9).
• Moreover, even in online settings, KVzip's importance-scoring method seamlessly extends to head-level eviction, introducing no additional compression overhead during deployment (see Context-Independent Eviction, Section 4.2). Compared to DuoAttention, a state-of-the-art head-level eviction technique, KVzip provides superior accuracy with substantially lower optimization costs (lines 270–275 and Figure 11).

Q) The authors' evaluation focuses primarily on scenarios with one document, same task type, and multiple different queries. However, the paper lacks evaluation of scenarios involving one document with multiple queries from different task types.

• Thank you for the pointer. In the Appendix, Figure 22 (described in text in lines 555-558), we present experimental results for the multi-task, multi-query setting. The evaluation uses two multi-task benchmarks from SCBench: Mix.RepoQA+KV and Mix.Sum+NIAH [1]. The following highlighted results demonstrate that KVzip consistently outperforms the baselines in a multi-task setting:
  | task | H2O | SnapKV | PyramidKV | KVzip | Full KV cache | |-----------------|------|--------|-----------|-------|---------------| | Mix.RepoQA+KV | 3.1 | 6.0 | 1.9 | 81.1 | 81.5 | | Mix.Sum+NIAH | 45.4 | 66.2 | 64.5 | 67.5 | 68.5 |

  Table. Multi-task, multi-query performance (%) at a 30% compression ratio using Qwen2.5-7B.

• Notably, KVzip is not only query-agnostic but also task-agnostic, as the compression procedure does not depend on task types. This implies that our main experiment in Section 4 using single-task benchmarks spanning 12 different tasks effectively demonstrates the task-generalizability of KVzip, which is why we placed the additional results on multi-task benchmarks in the Appendix. We will highlight these results in the main text in our revision.

[1] Li et al., SCBench: A kv cache-centric analysis of long-context methods. ICLR, 2025.

Dear reviewer, thank you for your valuable time and effort in reviewing our paper. We have addressed the questions and feedback raised below. We will carefully incorporate the discussion into our revision.

Q) The solution is essentially an application of SnapKV to a particular prompt structure.

• We note that our method introduces a query-independent objective, proactively identifying crucial KV pairs for arbitrary future queries. This represents a clear methodological distinction from prior works, such as SnapKV and H2O, which compress based solely on the current query, leading to query-specific overfitting. This distinction significantly enhances practical utility: KVzip maintains inference accuracy at a 30% compression rate, whereas prior methods exhibit severe performance degradation in query-agnostic scenarios (e.g., a 60% accuracy drop on the SQuAD dataset with Qwen2.5-7B at the same compression rate, see Figure 9).

Q) The reconstruction approach is also not new, though I haven’t seen this exact approach in the parameter-free setting before.

• While reconstruction-based objectives are well-established in representation learning, we emphasize that connecting this principle explicitly to parameter-free KV compression is novel, as acknowledged by the reviewer. This novel connection significantly enhances practical applicability in query-agnostic scenarios, enabling near-lossless compression at a 30% rate. In contrast, prior methods incur notable performance degradation even at a much higher 90% compression rate (see Figure 9).

Q) The prefill latency increase induced by KVzip is incredibly high, and is probably untenable in many scenarios.

• We emphasize that KVzip unlocks new application scenarios beyond the conventional online KV eviction setting. Specifically, KVzip produces reusable, accurately compressed KV caches suitable for multiple future queries, thus supporting offline context KV cache compression. This capability is particularly beneficial in real-world use cases such as enterprise document retrieval–augmented systems or personalized user contexts (lines 34–37), where the compression overhead can be amortized over subsequent queries. Existing dynamic KV eviction methods, however, incur significant performance degradation in multi-query scenarios, limiting their applicability in such scenarios (Figure 9).
• Moreover, even in online settings, KVzip's importance-scoring method seamlessly extends to head-level eviction, introducing no additional compression overhead during deployment (see Context-Independent Eviction, Section 4.2). Compared to DuoAttention, a state-of-the-art head-level eviction technique, KVzip provides superior accuracy with substantially lower optimization costs (lines 270–275 and Figure 11).

Q) What inference platform/engine/package was used to determine the numbers in Figure 8?

• We used PyTorch with Hugging Face Transformers and FlashAttention2 on an NVIDIA A100 GPU. This setup follows the convention used in previous works such as SnapKV, PyramidKV, and KVpress.

Q) Can you provide additional “Computational analysis” (Figure 8) for larger models (20B+)?

• Following your suggestion, we conducted computational analysis using Qwen3-32B (FP8) with a context length of 124K tokens on an NVIDIA H100 GPU. (Note: Qwen3’s FP8 quantization does not support the A100 GPU used previously.) We employed identical software packages as used in Figure 8.

• The table below demonstrates that KVzip maintains similar efficiency gains with the larger model, achieving approximately a 2$\times$ speed-up in decoding attention latency at a 20% compression rate.
  | Compression ratio | Decoding attention latency (ms) | KV cache size (GB) | |---|---|---| |1|0.45|32.7| |0.8|0.39|26.2| |0.6|0.34|19.6| |0.4|0.28|13.1| |0.2|0.23|6.5|

  Table. Computational efficiency analysis of KVzip during decoding.

• Regarding compression overhead, KVzip introduces approximately 1.5$\times$ computational overhead, slightly lower than the 2$\times$ overhead observed with the 8B model in Figure 8. Specifically, KVzip’s chunked context reconstruction process requires 1$\times$ MLP FLOPs and 2$\times$ attention FLOPs relative to prefilling. As model size increases, the relative computational contribution of MLP layers grows, thereby reducing the relative overhead of the scoring process.

• Memory overhead from KVzip’s scoring process remains negligible compared to prefilling (0.8% increase in peak memory), consistent with results observed on smaller models.
  |Stage|Compute time (s)|Peak GPU memory (GB)| |---|---|---| |prefill|3m 56s|62.7| |scoring (compression)|5m 25s|63.2|

  Table. Computational comparison of prefilling and KV importance scoring stages. We use a chunk size of 2K for both stages to ensure a fair comparison.

Q) Do you have any additional insights into how the latency of the method can be improved (would a fused kernel help significantly here)?

• We explored various strategies to reduce compression overhead. While a tailored fused CUDA kernel could help reduce latency, the benefit is relatively modest (~10%, line 528 in Appendix) since the primary overhead arises from the FLOPs by the Transformer forward-pass on repeated contexts.
• One such solution in our paper to further reduce latency is context-independent eviction (Line 257), which eliminates additional latency at deployment.
• We also tested the method that interleaves chunked prefilling and compression (i.e., compress the KV cache of the i-th chunk before prefilling the next chunk) to reduce overall attention FLOPs during prefilling. However, this led to decreased compression performance. In our paper, we prioritize achieving optimal KV-cache compression quality and present head-level context-independent eviction as a practical alternative. Exploring further improvements in prefilling latency remains part of our future work.

Q) How was the exact form of the prompt settled on and what alternatives were tried?

• In our code, we used the repeat prompt: "\n\nRepeat the previous context exactly." This choice was motivated by simplicity, as the specific wording of the repeat prompt has minimal impact on overall performance.

• To validate this, we conducted experiments comparing: (1) the original repeat prompt, (2) a paraphrased version ("Reproduce the preceding context without any changes."), and (3) no repeat prompt (just "\n\n"). Note that we follow the chat template provided by each model (e.g., "<user> [context] [repeat prompt] <assistant> [repeated context]").
  |Repeat prompt type|Accuracy (%)| |---|---| |Original|94.37| |Paraphrased|94.45| |No|94.25|

  Table. Accuracy (%) of Qwen2.5-7B on SQuAD at a 30% compression rate (70% eviction). SnapKV achieves 32.15% in this setting.

• These results show that our method is robust to variations in the repeat prompt. The limited impact arises because the repeat prompt (7 tokens with Llama 3.1) is significantly shorter than the overall context (at least several hundred tokens), thereby minimizing its effect on compression.

• To further clarify this, we analyzed attention patterns. Specifically, we measured the proportion of prefilled KV pairs whose maximum cross-attention scores during reconstruction originated from the repeated context rather than the repeat prompt (see Figure 4).

  • For a 2K token-length context from NIAH, 98.1% of KV pairs had their maximum attention from the repeated context. Among the KV pairs retained after 30% compression, 99.4% derived their maximum attention from the repeated context. These findings confirm the minimal influence of the repeat prompt on KVzip’s importance scores and compression performance.

• It is also interesting to observe that context reconstruction without a repeat prompt achieves comparable performance. This indicates that forwarding the context twice effectively captures the critical attention patterns required for downstream tasks (see Figure 6).

Q) The NIAH you selected is quite easy; how does KVzip perform on a more difficult task (multikey, multivalue, uuid retrieval from RULER)?

• Following your recommendation, we evaluated KVzip on RULER. Specifically, we integrated KVzip into NVIDIA’s KVpress framework, which benchmarks state-of-the-art KV eviction methods under a query-agnostic setting with a 4K context length RULER using LLaMA3.1-8B-Instruct.

• The table below compares KVzip to SnapKV on the suggested challenging tasks from RULER. Results indicate that KVzip maintains perfect performance at a 25% compression rate, whereas SnapKV experiences significant performance degradation.
  |task|SnapKV|KVzip|Full KV cache| |---|---|---|---| |Multi key-uuid|1.6|100.0|100.0| |Multi value|25.3|99.9|99.9|

  Table. RULER-4K performance (%) at a 25% compression ratio.

• To further highlight KVzip's effectiveness, we compared our method with various benchmark results from KVpress, which include state-of-the-art methods such as SnapKV, PyramidKV, DuoAttention, and other concurrent unpublished methods. While these results are publicly available on Hugging Face, the link is excluded here due to rebuttal policy constraints. The following table demonstrates that KVzip significantly outperforms current state-of-the-art KV eviction methods:
  |Compression ratio|SnapKV|Best from KVpress|KVzip| |---|---|---|---| |0.5|67.0|91.6|95.5| |0.25|39.9|74.3|95.3|

  Table. RULER-4K average performance (%) over 13 tasks. The full cache performance is 95.5%.

• Lastly, our primary experiments in Figure 9 demonstrate KVzip's capability to handle significantly longer contexts (up to 170K tokens) on SCBench, which also includes representative tasks from RULER.