We thank the reviewer for giving us a comprehensive analysis of our work and highlighting key weaknesses of our manuscript. We address some of the concerns below and hope that the discussion resolves all of them.

Clarification of the claims in Section 3.1 and a motivation for value-guided compression.

Here, we'll try to give a simpler motivation behind our approach of "value-guided compression" of KV caches. Recall that the standard attention output computes $\text{softmax}(QK^T/C)V$, where $C$ is a suitable constant involving the dimension of the vector spaces in question. Most of the current SOTA KV-cache compression methods (SnapKV, H20, AdaKV, to name a few) focus on estimating the importance of a cache entry by only using attention weights, which are computed using corresponding query and key vectors. However, as the attention equation suggests, value vectors may also contribute significantly to the final attention output. Lemma 3.1 of the paper bounds the true Frobenius distance between the attention outputs of the compressed and uncompressed KV caches; as we see, the upper bound depends on how well the value matrix is approximated. This intuition suggests that a good KV cache eviction policy should also incorporate the use of value vectors (and not just the attention weights). Having said this, we want to emphasize on the fact that this does not undermine the importance of key vectors. Infact, we provide ablations to show that a combination of leverage scores of both the key and value vectors provides the best generation results (Figure 3 and Table 17 of our paper). Our ablation results also show that the next best choice is to only use the leverage scores of value vectors, which further highlight the importance of value vectors in designing key-value pair eviction policies.

Regarding the sentence, "for higher compression levels, the second term becomes negligible": note that the matrix $V'$ is the compressed version of $V$, where dropped value vectors are simply zeroed out. At higher compression levels, say above 70%, most of the rows of $V'$ will just be zeros. Inevitably, the Frobenius norm of the resultant matrix will decrease, and will slowly become insignificant compared to the first term of the upper bound of Lemma 3.1.

End-to-end computational efficiency comparison with other baselines. As requested by the reviewer, we run experiments to compare the end-to-end runtimes of CurDKV against other baselines.

We compare four methods: exact CurDKV (where exact leverage scores are computed), random CurDKV (where the random Gaussian projection is used to project the key-value vectors, SnapKV and ChunkKV. We do comparisons on a context length of 128K tokens and generated length of 100 tokens, and we use compression ratios of $30$% and $90$%. We report the total runtime, prefill time and the generation time for all methods. Here are the detailed results:

Table 1: Runtime results for a context length of 128K tokens and a compression ratio of $30$%.

As we see from the above table, CurDKV with a random Gaussian projection outperforms the baselines for a compression ratio of $30$%. This is also intuitive, since the Gaussian projection reduces the effective dimension of the key-value pairs. This, combined with the computationally efficient leverage scores, leads to the overall smallest total time (prefill + generation) for random CurDKV.

Table 2: Runtime results for a context length of 128K tokens and a compression ratio of $90$%.

For a high compression ratio of $90$%, random CurDKV still performs approximately the same as the best baseline (ChunkKV for this case). This, combined with the performance gains of CurDKV, makes it a competitive choice.

An ablation study on the dimension of the Gaussian projection. As rightfully requested by the reviewer, we run an ablation study to determine the effect of the output dimension $r$ of the Gaussian projection matrix. Intuitively, we expect an overall increase in the generation quality as the dimension $r$ increases; this is due to the simple fact that as the dimension increases, the leverage scores of the projected vectors will better approximate the leverage scores of the original vectors. However, we still need to empirically verify this claim, and we do so below. We carry out these ablation studies for compression ratios of $30$% and $90$%. We run evaluations on the tasks in the LongBench benchmarks.

Table 3: Ablation study on the output dimension $r$ of the Gaussian projection matrix for a compression ratio of $30\%$.

Table 4: Ablation study on the output dimension $r$ of the Gaussian projection matrix for a compression ratio of $90$%.

For both compression ratios, we observe that increasing $r$ improves the performance. In Figure 3 in Section 4.2, we have discussed that CurDKV without random projection (Exact CUR) usually performs better than CurDKV with random projection. Having a large $r$ in the approximated leverage score computation approximates the exact leverage score computation. Therefore, the performance increases to that of the exact CUR. Moreover, this behavior is more prominent for larger compression ratios.

128K data line for Figures 4(b) and 5(b).

We thank the reviewer for pointing this out. Due to a limit in the y-axis (the maximum y value for prefill time is taken as 30 sec in the plots, but for 128K, it exceeds 40 sec), the plots for 128K were omitted from both plots. We will correct this in the revised version of the paper.

We thank the reviewer for raising important questions about our claim to develop "value-guided" KV compression methods. We are glad to know that the reviewer found our work strong, easy to understand, and practically effective. We address the concerns raised by the reviewer below and hope that this discussion resolves their queries.

How does CurDKV work in inference-time for longer generation tasks

Below, we detail how our method processes the Key-Value (KV) cache for tokens generated during inference -

1. At each generation step, the model generates a token and computes the corresponding query, key, and value vectors.
2. The newly generated token's key and value vectors are appended to the KV cache of the prefilled (context) tokens.
3. After appending the new token, the method periodically recomputes leverage scores for the cached keys and values (optionally, using the approximation via random Gaussian projections to ensure computational efficiency)
4. Tokens with the lowest combined leverage scores are evicted from the cache to maintain the predefined memory budget. The most informative tokens, identified via high leverage scores, are retained.
5. CurDKV explicitly preserves the first few tokens (attention sinks) as they typically carry critical global context relevant for subsequent generation steps.

We further compare four KV compression methods: exact CurDKV (where exact leverage scores are computed), random CurDKV (where the random Gaussian projection is used to project the key-value vectors, SnapKV and ChunkKV. We do comparisons on a context length of 128K tokens and generated length of 100 tokens, and we use compression ratios of $30$% and $90$%. We report the total runtime, prefill time and the generation time for all methods. Here are the detailed results:

Table 1: Runtime results for a context length of 128K tokens and a compression ratio of $30$%.

As we see from the above table, CurDKV with a random Gaussian projection outperforms the baselines for a compression ratio of $30$%. This is also intuitive, since the Gaussian projection reduces the effective dimension of the key-value pairs. This, combined with the computationally efficient leverage scores, leads to the overall smallest total time (prefill + generation) for random CurDKV.

Table 2: Runtime results for a context length of 128K tokens and a compression ratio of $90$%.

For a high compression ratio of $90$%, random CurDKV still performs approximately the same as the best baseline (ChunkKV for this case). This, combined with the performance gains of CurDKV, makes it a competitive choice.

When we increase the generation length to 4K, the generation time increases for all compression methods due to periodic eviction/retention of key-value pairs.

Table 3: Generation time results for a context length of 128K and generated 4K tokens.

At a 30% compression ratio, exact CurDKV exhibits the fastest generation, whereas, for a 90% compression ratio, random CurDKV achieves the fastest generation. At moderate compression ratios, the Exact CurDKV precisely identifies and retains highly informative keys and values through exact CUR decomposition; therefore, reducing the need for repeatedly decomposing the key-value pairs on generated tokens. At a higher compression rate, more repeated computation of key-value eviction is needed; therefore, under those circumstances, the effectiveness of low-rank projections increases, leading to lower generation time.

On the usage of both the key and value matrices, and the claim of the paper. We appreciate the reviewer for seeking clarification for the main claim of the paper, as we believe it is the main contribution of this work. Most of the existing SOTA literature on KV cache compression methods (SnapKV, H20, AdaKV, to name a few) focus on estimating the importance of a cache entry by using attention weights, which rely only on the corresponding query and key vectors. However, as we show in Lemma 3.1 of our paper, the true Frobenius distance between the exact attention output and the compressed attention output also depends on how well the compressed value matrix approximates the original value matrix; intuitively, this happens because the query and key vectors are absorbed by the softmax operator. This motivates the design of eviction policies which also incorporate value vectors. At the same time, we point out that this analysis does not undermine the importance of the corresponding key vectors. Infact, we provide ablation studies to empirically show that the combined scores of both the key and value produces the best results (Figure 3 and Table 17 of our paper). As our ablations also show, the next best choice is to use the leverage scores of only the value vectors. Overall, through our paper, we aim to establish the importance of incorporating information gained from value vectors in developing key-value eviction policies.

vllm support

We thank the reviewer for raising this important point regarding practical compatibility and integration. CurDKV is fully supported by NVIDIA’s KVPress library, which is specifically designed to implement and benchmark KV cache pruning strategies within Hugging Face Transformers-based pipelines. However, at present, KVPress (and thus CurDKV) does not natively support vLLM. The vLLM inference backend manages KV caches differently and currently has its own separate mechanisms (e.g., KV-Compress, LeanKV) for cache compression. While direct compatibility between CurDKV and vLLM would require additional engineering effort, methodologically, CurDKV can be adapted to work with vLLM by modifying vLLM’s cache eviction and retention mechanisms accordingly.

We thank the reviewer for giving us valuable insights to the potential weaknesses of our work, and we aim to address some of those concerns below. We hope the following discussion clarifies some concerns raised by the reviewer.

CurDKV as a static compression method. It is true that CurDKV is a static compression method, in the sense that once a key-value pair of the context is evicted from the cache, CurDKV never recovers it (even if future tokens shift relevance). However, we'd like to point out that some of the best-performing baselines which we compare against are also static in this sense: H2o, SnapKV, StreamingLLM and AdaKV to name a few. Infact, any reasonable KV cache compression technique must be static in this sense, because the recovery of an evicted key-value pair from the cache must inevitably require either 1) storing the key-value pair in GPU memory or 2) recomputing the key-value pair, both of which may have significant overhead for larger context lengths.

Results on bigger models. Per request of the reviewer, we tested CurDKV on larger models; particularly, we chose the Qwen3 14B and Qwen2.5 32B models, and tested the efficacy of CurDKV on these models against other strong baselines. As in the paper, we report results on tasks from the LongBench benchmark. We report results for compression ratios of $30$% and $90$% respectively.

Table 1: LongBench evaluation results for the Qwen 14B model for a compression ratio of $30$%.

As we see from the above table, SnapKV seems to perform the best on average for the LongBench tasks. However, CurDKV is not far, and consistently performs close to the best baseline.

Table 2: LongBench evaluation results for the Qwen 14B model for a compression ratio of $90$%.

The effect of CurDKV seems to be more prominent for the regime of high compression ratios. For instance, in the above table ($90$% compression), CurDKV significantly outperforms other baselines by a significant margin. Below, we see a more prominent effect for an even larger model (a 34B quantized model).

Table 3: LongBench evaluation results for the Qwen 32B model for a compression ratio of $30$%.

Here again, we see that CurDKV consistently outperforms other baselines by a significant margin.

Table 4: LongBench evaluation results for the Qwen 32B model for a compression ratio of $90$%.

And yet again, we see that for the 32B model quantized model, the gains for CurDKV in a high compression ratio regime are even more prominent. This raises a further interesting question: do value-guided KV caching methods perform better for quantized models? We leave this question for a future work.

On the sensitivity of the safeguard parameter ($\alpha$). As requested by the reviewer, here we do experiments to run a sensitivity analysis on the safeguard parameter $\alpha$. To recall, for KV compression methods that distribute the per-head budget adaptively (aka the AdaKV technique [1]), the parameter $\alpha$ controls the proportion of KV pairs to preserve in each head (see line 28 of the KVPress implementation at https://github.com/NVIDIA/kvpress/blob/main/kvpress/presses/adakv_press.py). The default value of $\alpha$ as proposed by the authors of AdaKV is $0.20$, which we also use in our implementation of AdaCurDKV. Tables 5 and 6 below contain the results of our ablations.

Table 5: Sensitivity analysis of the safeguard parameter $\alpha$ for LongBench tasks at a compression ratio of $30$%. We compare the two adaptive techniques: AdaSnapKV and AdaCurDKV.

Table 6: Sensitivity analysis of the safeguard parameter $\alpha$ for LongBench tasks at a compression ratio of $90$%. We compare the two adaptive techniques: AdaSnapKV and AdaCurDKV.

For both 30% and 90% compression ratios, we observe only a marginal performance difference for different values of $\alpha$, indicating that the adaptive KV compression methods (including AdaCurDKV) are very robust to the selection of this parameter.

References

[1] Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference, Feng et al., 2025.

We thank the reviewer for giving us a comprehensive analysis of our work. We are grateful that the reviewer found our approach of using values to compress KV caches well-motivated. Below, we would like to address all concerns of the reviewer and hope that they answer all their questions.

A clearer description of Algorithm 1. Here we give a clearer description of the algorithm, with a particular focus on how leverage scores are used to compute the decomposition.

1. Consider a set of key and value matrices $K\in\mathbb{R}^{k\times d}$ and $V\in\mathbb{R}^{k\times d}$, i.e $k$ key-value pairs, each of dimension $d$.
2. Project key-value vectors do a vector of dimension $r$ (we use $r = 20$) via a Gaussian projection matrix $G\in\mathbb{R}^{d\times r}$ (lines 2-3 of the algorithm). The significance of this step is to reduce the overall run-time complexity (computing exact leverage scores requires SVD).
3. After this, we compute leverage scores using the new key and value vectors (line 4 of the algorithm). In simple words, these scores are just row norms of the key-value matrices.
4. The key-value leverage scores are then combined (line 5) and normalized (line 6).
5. Finally, the indices with the top-$(k - s)$ scores are computed, and the keys and values for these indices are retained in the KV cache (lines 8-10 of the algorithm).
6. Note that we preserve the first $s$ tokens in the context due to the prevalence of attention sinks. [1]

Note: We've further provided an ablation study involving different values of $r$ for reviewer hDao. TL;DR: we observe only marginal differences when using different values of $r$, and the Gaussian approximation improves as $r$ increases (which is intuitive).

Regarding the maximum length in experiments for the RULER benchmark. As per the reviewer's request, we ran NIAH experiments on the RULER benchmark with a context length of 128K tokens. We've reported the results in the table given below. As in the main paper, we compare CurDKV with the ChunkKV, KNorm, SnapKV and StreamingLLM baselines. We carry out the experiments with compression ratios of $30$%.

Table 1: Results on the RULER benchmark with 128K context length with a compression ratio of $30$%.

Table 2: Results on the RULER benchmark with 128K context length with a compression ratio of $50$%.

As we see from the above results, CurDKV consistently either outperforms other methods or performs in a way sufficiently comparable to the best baseline for a given task, leading to the overall best average score.

Note: For NIAH 128K experiments, we chose 30% and 50% compression ratios for comparing the KV compression methods. At higher compression ratios (>50%), most of the methods' performance drops poorly (almost to 0 for most tasks) due to a lack of the appropriate context length. This behavior is more prevalent in the 128K context than in shorter context NIAH tasks (reported in the paper).

A latency comparison with and without CurDKV. Per request of the reviewer, here we provide a latency comparison of other baselines against CurDKV. We compare four methods: exact CurDKV (where exact leverage scores are computed), random CurDKV (where the random Gaussian projection is used to project the key-value vectors, SnapKV and ChunkKV. We do comparisons on a context length of 128K tokens and generated length of 100 tokens, and we use compression ratios of $30$% and $90$%. We report the total runtime, prefill time, and the generation time for all methods. Here are the detailed results:

Table 3: Runtime results for a context length of 128K tokens and a compression ratio of $30$%.

As we see from the above table, CurDKV with a random Gaussian projection outperforms the baselines for a compression ratio of $30$%. This is also intuitive, since the Gaussian projection reduces the effective dimension of the key-value pairs. This, combined with the computationally efficient leverage scores, leads to the overall smallest total time (prefill + generation) for random CurDKV.

Table 4: Runtime results for a context length of 128K tokens and a compression ratio of $90$%.

For a high compression ratio of $90$%, random CurDKV still performs approximately the same as the best baseline (ChunkKV for this case). This, combined with the performance gains of CurDKV, makes it a competitive choice.

References

[1] Efficient Streaming Language Models with Attention Sinks, Xiao et al., 2024.