R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

This paper tackles the problem of excessive redundant information in the KV cache during long chain‑of‑thought reasoning in large language models by introducing R‑KV, a training‑free, decoding‑time plugin. R‑KV computes, for each cached token, an importance score based on attention weights and a redundancy score based on semantic similarity, then employs a tunable trade‑off coefficient to retain only the most informative and diverse tokens. As a result, it achieves reasoning accuracy comparable to—or even exceeding—that of the full KV cache using substantially less cache space, while significantly reducing memory footprint and accelerating inference.

The paper introduces R-KV(Redundancy-aware KV Cache Compression), a training-free KV cache compression method for reasoning models that produce long, redundant outputs. R-KV works by jointly considering token importance (via attention) and redundancy (via cosine similarity) to selectively prune the KV cache. Evaluated on two distilled models across two mathematical reasoning benchmarks, the method achieves performance comparable to the full KV cache with a significantly smaller memory footprint, leading to substantial memory savings and inference speedups.

This paper addresses a critical challenge in deploying large language models (LLMs) for reasoning tasks: the excessive GPU memory consumption caused by long chain-of-thought (CoT) outputs. These verbose generations often lead to large key-value (KV) caches during autoregressive decoding. While prior work has attempted to reduce memory usage by selecting important tokens based on attention scores, this paper claims that this is insufficient.

Their key insight is that even among high-attention tokens, there is substantial redundancy. Many tokens are semantically repetitive and contribute little to final task performance. Retaining such redundant tokens crowds out more diverse and informative context, ultimately hurting reasoning accuracy.

To address this, the paper proposes R-KV, a redundancy-aware KV cache compression method that considers both:

1. Token importance, estimated via attention scores;
2. Token redundancy, measured through semantic similarity (cosine similarity of key vectors);
3. A joint selection strategy, which balances these two aspects using a tunable parameter λ, to retain a subset of KV tokens that are both important and diverse.

R-KV operates during the decoding stage---where output grows rapidly---and performs cache compression at fixed intervals. It maintains two memory buffers: a cache of size $B_{\text{budget}}$ to store selected key-value tokens, and a buffer of size $B_{\text{buffer}}$ to temporarily hold newly generated tokens.

After generating each $B_{\text{buffer}}$ tokens, the algorithm first preserves the most recent $\alpha$ tokens (observation tokens) to ensure continuity. Then, from the combined pool of $B_{\text{budget}} + B_{\text{buffer}} - \alpha$ candidate tokens, it selects the top $B_{\text{budget}} - \alpha$ tokens according to the joint importance--redundancy score. These tokens, along with the $\alpha$ most recent ones, are retained for the next decoding step, effectively compressing the KV cache while preserving useful and diverse context.

Experiments are conducted on two mathematical reasoning benchmarks: MATH-500 and AIME 2024, using DeepSeek-R1-Distill-Llama-8B and Qwen-14B models. Results show that:

• R-KV achieves near-parity or even 105% of full KV accuracy using only 10–34% of the original KV cache.
• It outperforms the prior state-of-the-art method SnapKV by up to 40% in accuracy.
• It enables up to 90% memory savings, 13× larger batch sizes, and 9× higher throughput, significantly improving inference efficiency under constrained memory settings..

Overall, R-KV is a training-free and model-agnostic method that demonstrates strong effectiveness in both accuracy and throughput.

This paper introduces Redundancy-aware KV Cache Compression for Reasoning models (R-KV), a method that computes and combines importance and redundancy scores for generated tokensthat guide their compression. The focus of R-KV is on decoding rather than prefilling stage with immediate application on the excessively long outputs of reasoning models. The core idea is managing two fixed size buffers for retained (budget buffer) and newly generated tokens and compressing between fixed-length text generation steps. In particular, a number of most recently generated tokens in the text are retained (observation tokens) and the rest tokens to fit in the budget buffer are selected by assigning scores to the remaining generated tokens and previously retained ones (candidate tokens). A token score is a linear combination of its importance (attention) score (where observation tokens serve as queries and a sliding window around a candidate token stabilizes the computation of the attention it receives) and its redundancy score (computed by averaging its key similarities to all other candidate tokens).

In the empirical evaluation of R-KV, authors typically report the accuracy on MATH500 and AIME 2024 datasets for various KV-cache budgets. Full KV-cache setup serves as the bare minimum baseline and SnapKV adapted to the same compression intervals R-KV is another (KV-cache compression) baseline; in most setups, models are Deepseek-R1 distilled ones (8B and 14B parameters). Reported results are impressive: in some configurations, R-KV can maintain the full KV-cache performance by using only 10% of the KV-cache, outperform it (still utilizing a fraction of the KV-cache) or achieve up to 90% memory savings and x4.9 speedup. Interestingly, the authors include a discussion on choosing the coefficients in the linear combination of the two scores and a nice illustration of the excessively hight scores other methods can put on very recent tokens or repetitive token subsets, however of minimal importance for eviction decisions.

Dear Reviewers, ACs, SACs and PCs,

We sincerely appreciate your efforts in maintaining the review process and value the constructive, insightful feedback provided by all reviewers, along with your recognition of our work’s contributions. Below, we summarize the key strengths identified by each reviewer, clarify how we have addressed the raised concerns in our rebuttal, and outline planned improvements for the revised version.

Strengths Recognized by Reviewers

Novelty and Motivation

1. Reviewer evCh — Highlighted the practical importance and timeliness of R-KV for compressing lengthy KV caches in reasoning models, commending our effective use of attention + redundancy scoring to overcome pitfalls of pure attention-based eviction.
2. Reviewer cogD — Appreciated the clear problem statement and principled design of our joint scoring mechanism, emphasizing its relevance to a real gap in existing work.
3. Reviewer vf9G — Praised the strong motivation behind tackling redundancy in reasoning model outputs, calling R-KV a simple yet principled algorithm.
4. Reviewer Dks4 — Noted the novelty of our redundancy-aware perspective, which directly addresses repetitive self-reflections overlooked by prior approaches.

Empirical Strength and Clarity

1. Reviewer evCh — Found our results “impressive” and recognized our discussion on score coefficient selection and pitfalls in other methods.
2. Reviewer cogD — Pointed out that R-KV achieves near-parity or better than full KV cache accuracy while using only a fraction of the cache, delivering substantial throughput and memory gains.
3. Reviewer vf9G — Acknowledged the strong empirical results and practical deployability of our method.
4. Reviewer Dks4 — Appreciated the comprehensive ablations, clear visualizations, and the lightweight, training-free nature of our approach.

Key Clarifications from the Rebuttal

1. Baseline Scope — Addressing concerns from evCh and cogD, we expanded comparisons beyond SnapKV to include StreamingLLM and H2O, demonstrating R-KV’s consistent superiority across various budgets.
2. Model and Task Generalization — In response to cogD, vf9G, and Dks4, we added evaluations on the QwQ model and GPQA dataset (scientific QA), confirming robustness across models and domains.
3. Hyperparameter Specification and Sensitivity — As requested by vf9G and Dks4, we explicitly reported the β, and T values used in our experiments and added sensitivity analysis for β.

Planned Manuscript Revisions

1. Expanded Empirical Evaluation — Incorporate additional baselines (StreamingLLM, H2O), domains (GPQA), and models (QwQ).
2. Improved Presentation — Correct typos and figure inconsistencies, and include further details on efficiency backends and hyperparameter documentation.
3. Extended Analysis — Provide quantitative redundancy analyses, discuss compatibility with other inference optimization techniques, and include sensitivity analyses for hyperparameters.

We believe these additions and clarifications will further strengthen the manuscript, making our contributions clearer, more generalizable, and more compelling. We are grateful for the reviewers’ detailed and thoughtful feedback, which has been invaluable in refining this work.

Best regards, Authors

summary This paper introduces R-KV, a training-free method for compressing the Key-Value (KV) cache in large language models during inference on reasoning tasks. The core idea is to prune the cache by jointly considering both token importance (from attention scores) and token redundancy (from semantic similarity of key vectors). This dual-criterion approach aims to create a more information-dense cache, improving inference efficiency without sacrificing accuracy.

strengths

• The method's core idea of targeting redundancy in addition to importance is novel and addresses a clear limitation of prior attention-only eviction strategies.
• The empirical results are strong, demonstrating that R-KV can match or even exceed the performance of a full KV cache while using only a small fraction of the memory, leading to significant speedups.
• The approach is lightweight, training-free, and model-agnostic, making it a practical and easily deployable solution for accelerating LLM inference.

weaknesses

• The initial evaluation was limited in scope, focusing on a specific family of models and mathematical reasoning tasks, which raised questions about generalizability.
• The paper's initial baseline comparison was narrow, primarily focusing on SnapKV while omitting other relevant methods like StreamingLLM or H2O.
• The manuscript lacked a thorough analysis of key hyperparameters and their sensitivity, which is crucial for reproducibility and practical application.

final descision The paper presents a novel, practical, and effective method for a significant problem. The strong empirical results, combined with a successful rebuttal that broadened the experimental scope, justify its acceptance.