$\text{D}_{2}\text{O}$: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models

Q4 comparison with the smaller full-size model

Thank you for your suggestion. Due to differences in embedding sizes across LLMs of varying scales, the memory required for KV cache storage at the same sequence length differs significantly. This makes it challenging to select a smaller-scale LLM with similar inference time costs and conduct a fair comparison under KV cache compression scenarios.

In most KV cache optimization studies, comparisons are typically made within the same model scale, applying different KV cache compression algorithms at a fixed compression ratio to ensure fairness in performance and throughput evaluation. The experimental setup in our paper follows the standard comparison methodology [1] [2], which ensures consistency and reliability in our results.

[1] Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

[2] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs.

Q5 More analysis on Table 9 and Figure 5

Thank you for your question. Yes, this outcome is reasonable. For example, as shown in Figure 5 for the Llama-3-8B model, our dynamic token-level strategy selectively merges evicted tokens based on a similarity threshold. This process removes redundant information while preserving critical context, enabling the model to focus more effectively on the retained key tokens. As a result, the model can perform more accurate reasoning, leading to better performance compared to the full cache and other baselines under certain KV cache compression rates.

Similar phenomena have also been reported in prior works, such as H2O [1], PyramidInfer [2], and FastGen [3], further validating this observation.

[1] Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

[2] PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference.

[3] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs.

Q6 More experimental results

Thank you for your comment. We followed the StreamLLMs codebase for benchmarking sample generation. In a single A100 GPU setup, the full model often encounters Out of Memory (OOM) issues during ultra-long text inference without KV cache optimization. In the updated paper, we will include the full model's output in the Appendix. If OOM occurs for a sample, we will indicate it accordingly.

For Table 3, using a four-A100 80GB setup, our inference speed falls between SnapKV and CaM, while achieving the best performance among the methods. Thank you for highlighting this.

Q7 Additional citation

Thank you for pointing this out. We will include a citation to this concurrent work in the related work section of the updated paper.

You have offered constructive and valuable suggestions, making our submission more solid and complete. Once again, we sincerely express our best gratitude for your effort and time!

We would like to thank you for taking the time to review our work. Your questions and comments are pretty professional and enlightening. We address the questions and clarify the issues accordingly as described below.

W1 The peculiarity of our innovation and novelty

Thank you for your feedback. We would like to clarify that, unlike the mentioned works, our approach uniquely combines layer-level and token-level adaptive operations for KV cache compression. Extensive experiments demonstrate that D2O achieves both efficient and effective compression compared to SOTA baselines.

a. Compared to CaM [1], which uses accumulated attention scores and a Bernoulli mask for token merging, D2O sorts tokens by attention scores, calculates similarity, and applies an EMA threshold for selective merging. Additionally, D2O merges key caches first and applies weights to value caches, while CaM focuses solely on value caches.

b. Unlike KVMerger [2], which uses a Gaussian kernel to merge all states within each set, D2O selectively merges tokens based on an EMA similarity threshold, filtering out redundant tokens during both prefill and generation phases. Moreover, KVMerger is a concurrent work.

c. Compared to MoD [3], which trains a router to allocate tokens to specific layers, D2O is training-free, plug-and-play, and focuses on accurate layer-wise KV cache allocation instead of token distribution across layers. We will include these related works and comparisons in the revised paper to provide additional clarity.

[1] https://openreview.net/pdf?id=LCTmppB165.

[2] https://arxiv.org/html/2407.08454v1.

[3] https://arxiv.org/pdf/2404.02258.

Q1 Explanation of dynamic allocation strategy

Thank you for your question. We would like to clarify that such a situation will not occur. As detailed in Equation (5) and Appendix A.2, the total number of compressed KV tokens is given by:$\rho \cdot L \cdot S$, where $S$ is the length of the input prompt, $\rho$ is the KV cache compression ratio, and $L$ is the total number of layers. These tokens are dynamically allocated across layers. If the allocation for a single layer exceeds $S$ , we will set the KV cache size for that layer at $S$ , ensuring that no layer's computational load exceeds the cost of processing all tokens. This mechanism ensures both efficiency and scalability in our dynamic allocation strategy.

Q2 Explanation of computational load

Thank you for your question. To ensure batch processing feasibility and maintain uniform computational load, we pad all sequences in the batch to match the length of the longest sequence. For layer-wise KV cache size allocation, we set the KV cache size for each layer in the batch to the maximum allocation determined by the most demanding (the largest cache size allocation) sample in that layer. This ensures that all samples in the batch have consistent KV cache sizes, enabling efficient parallel processing.

It is worth noting that most of the inference datasets used in this paper default to single-sample inference in their benchmark settings. Additionally, while our current implementation has not been adapted to the vLLM framework, we plan to integrate D2O’s KV cache compression strategy with vLLMs in future work.

Q3 The physical rationale behind the dynamic allocation strategy

Thank you for your question. The rationale behind allocating fewer KV caches to layers with high attention score variance is supported by the visualizations in Figures 1 and 9, as well as the attention variance statistics in Figure 4. These analyses indicate significant differences in attention density across layers:

• In lower layers, attention maps are denser, reflecting more uniformly distributed information. Therefore, allocating more KV cache size in these layers reduces the risk of irreversible information loss during compression.

• In higher layers, attention maps are sparser, indicating that information is concentrated on specific tokens. Consequently, fewer KV caches can be allocated to these layers without sacrificing critical information.

This dynamic allocation approach ensures efficient KV cache usage while preserving essential context across different layers.

Dear Reviewer aSVX,

Thank you for your insightful comments and valuable contributions to our work. Your feedback and support have significantly enhanced the quality of our paper. Wishing you all the best in your endeavors!

Bests,

Your friends,

Team 5363

Q1: Clarification of the formula (5)

Thank you for your question. We would like to clarify that this is not a typographical error. The detailed derivation process is provided in Appendix A.2. In the formula, $\rho$ represents the fixed KV cache compression ratio per layer (e.g., 20%), and $L$ is the number of layers. Using the method outlined in the formula, we dynamically allocate the total KV cache budget across layers based on the inverse-softmax distribution. This ensures that each layer receives an appropriate cache size proportional to its attention variance.

Q2: More explanation on calculating AttnScore during token generation

Thank you for your question. To clarify the calculation of $\text{AttnScore}$ in formula (6) during token generation: At step $i$ in the generation phase, the hidden state of the newly generated token has a size of $(\text{batch}, 1, \text{dim})$. After multiplying it with the query weight matrix $W_q$, we obtain $q_i$, an embedding of size $(\text{batch}, 1, \text{dim})$. This hidden state is then used to compute new $k_i$ and $v_i$ embeddings by multiplying it with $W_k$ and $W_v$, respectively. These embeddings are subsequently stored in the KV cache, and the KV cache is updated, extending the key and value token lengths to $L_k + 1$.

Next, $q_i$ is multiplied with the Key embeddings in the KV cache (size: $\text{batch}, L_k + 1, \text{dim}$), resulting in an attention score of size $(\text{batch}, 1, L_k + 1)$. Since we compute cumulative attention scores, this new attention score is added to the scores obtained during the prompt encoding phase, yielding the updated attention score for the generation phase.

We hope this explanation clarifies the calculation process. Please let us know if further details are needed.

You have offered constructive and valuable suggestions, making our submission more solid and complete.We promise to include the above results and clearer expression in the final version. Once again, we sincerely express our best gratitude for your effort and time!

Best wishes,

All authors of Submission 5363.

We would like to express our most sincere gratitude for your time and effort. We value each of your suggestions and provide the following responses:

W1: Clarification of statement in our paper

Thank you for your feedback. We respectfully think that perhaps you have misunderstood us. We would like to clarify a potential misunderstanding regarding our statement in Lines 181-182. When we refer to a "uniform eviction strategy," we mean that methods like H2O and SnapKV allocate the same KV cache size across all layers, regardless of the input or layer-specific representation. In contrast, D2O introduces an adaptive dynamic cache allocation strategy that adjusts KV cache size based on attention variance across layers, allowing for a more efficient and data-driven distribution of memory resources. We will also consider refining our presentation to “uniform layer-wise size” to make the statement clearer and more accurate.

W2: Additional comparison with other methods.

Thank you for your feedback. To address your suggestion, we added comparisons with PyramidKV [1] and PyramidInfer [2] using the Mistral-7B-Instruct-v0.2 model on the LongBench dataset as the following table. The results show that D2O consistently outperforms these non-adaptive KV cache allocation methods across most datasets, demonstrating the effectiveness of our inverse variance softmax-based dynamic allocation strategy. This approach excels in long-context scenarios by preserving critical information across layers.

LongBench (Mistral-7B-Instruct-v0.2) / (20% KV Cache)

Additionally, we also want to clarify that D2O differs from the heuristic approach of PyramidKV or PyramidInfer, which reduces cache size uniformly with increasing layer depth. D2O adapts dynamically to the input data, leveraging attention variance to allocate KV cache size more precisely, leading to better performance.

W3: More analysis on our experimental results

• Thank you for your question. Yes, this is a reasonable outcome. In the Llama-3-8B model scenario shown in Figure 5, our dynamic token-level strategy uses similarity thresholds to selectively merge evicted tokens. This process removes redundant information while preserving key context, enabling the model to focus more effectively on critical tokens and perform more accurate reasoning. As a result, under certain KV cache compression rates, D2O can achieve better performance than the full cache and other baselines. Similar phenomena have also been reported in prior works, such as PyramidInfer [2], H2O [3], and FastGen [4], further validating this observation.

• Thank you for your feedback. We would like to clarify that our evaluation of LongBench was conducted using the official code from KIVI's GitHub repository [5][6]. Our implementation results align closely with the full cache results of Llama-2-7B/13B/ Mistral-7B reported in the KIVI paper, as shown in the table below.

Long-bench implementation result comparison

For our LongBench experiments, we ensured fairness by using the official code and default configurations provided for each baseline method. The differences in reported results may arise from variations in benchmark code versions or experimental setups between studies. For example, we use 20% KV cache size based on the input prompt, whereas they use a fixed context size (e.g., 2048). We will clarify this distinction in the revised paper to avoid confusion.

[1] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling.

[2] PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference.

[3] Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

[4] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs.

[5] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. ICML 2024

[6] https://github.com/jy-yuan/KIVI/blob/main/eval_long_bench.py.

Dear Reviewer FB7s

Thank you for your valuable feedback on our paper. We have incorporated your suggestions and experiments into the main text and the appendix. You are an outstanding reviewer, and under your guidance, our paper will improve significantly. We sincerely appreciate your efforts and wish you success in your future research. Merry Christmas!

Best wishes，

Your friend，

Team 5363

Mistral-7B-Instruct (KV size = 64)

Mistral-7B-Instruct (KV size = 2048)

We observe that discrepancies may exist between KIVI [1] and PyramidKV LongBench setting [2] due to differences in data processing and experimental configurations. However, D2O achieves consistent performance across both settings. Furthermore, results on datasets requiring commonsense and complex reasoning (e.g., GSM8K, CoQA, TruthfulQA), as well as long-context datasets (Needle-in-a-Haystack, Ruler, InfiniteBench), validate the robustness of D2O’s two-level strategy. To ensure transparency, we have added the PyramidKV-based LongBench results to the appendix in the revised paper (Appendix A12).

[1] https://github.com/jy-yuan/KIVI/blob/main/eval_long_bench.py [2] https://github.com/Zefan-Cai/KVCache-Factory/blob/main/run_longbench.py

2. For question 1, regarding the absence of NarrativeQA and HotpotQA in the evaluation, we selected datasets aligned with KIVI’s Table 4 experimental settings, including Qasper, QMSum, MultiNews, TREC, TriviaQA, SAMSum, LCC, and RepoBench-P. While we did not include NarrativeQA and HotpotQA in this specific evaluation, Table 2 in the main paper already provides results for all datasets with multiple LLM backbones.

3. For question 2, due to time and computational constraints during the rebuttal period, we used Mistral-7B-Instruct as the primary backbone for LongBench evaluations. However, we have also evaluated other backbones, such as LLaMA-3.1-8B-Instruct, on additional datasets like Ruler and InfiniteBench (Appendix A11). These results demonstrate that D2O’s effectiveness generalizes across different LLM backbones and datasets.

We sincerely thank you for your suggestions, which have significantly strengthened our paper. We hope our responses and additional experiments adequately address your concerns, and we look forward to any further feedback you may have.

Best regards,

Your friends,

Team 5363

Dear Reviewer FB7s:

Thank you for your thoughtful follow-up comments. We greatly appreciate the opportunity to address your remaining concerns, as your feedback has been instrumental in improving our work. Below, we provide detailed responses to your points and outline the steps we have taken to address your suggestions:

1. Aligning Evaluation using Longbench code from PyramidKV :

Following your suggestion, we have evaluated our D2O method using the PyramidKV codebase for LongBench experiments. We strictly adhered to the settings provided in the official code, including using LLaMA-3-8B-Instruct and Mistral-7B-Instruct models as backbones, and conducted experiments under two KV cache scenarios (KV size = 64 and 2048). For D2O, the total compressed KV cache size was configured as 64×$L$ or 2048×$L$, where L is the number of layers.

For comparison, results of SnapKV, StreamLLM, H2O, and PyramidKV were taken directly from the PyramidKV paper. The evaluation for D2O was conducted using the PyramidKV-provided LongBench settings. From the following Table, the results demonstrate that under extreme compression settings (KV size = 64), D2O outperforms all baselines in average, highlighting the effectiveness of D2O’s combination of layer-level dynamic KV cache allocation and token-level dynamic merging. With higher KV sizes (KV size = 2048), D2O still achieves better performance than baselines on most datasets, confirming the consistent performance of our two-level operation approach.

LLaMa-3-8B-Instruct (KV size = 64)

LLaMa-3-8B-Instruct (KV size = 2048)

We would like to thank you for taking the time to review our work. Your questions and comments are enlightening. We value each of them and have provided detailed responses below:

W1: Additional comparison with recent token eviction approaches

Thank you for the valuable suggestion. Following your recommendation, we conducted additional comparisons with recent token eviction approaches such as SnapKV and SirLLM, as well as token compression techniques like LoCoco. These experiments were performed on the LongBench dataset using the Mistral-7B-Instruct-v0.2 model. As shown in the table below, D2O consistently outperforms these baseline methods across most datasets. This demonstrates that D2O’s combination of dynamic token-level merging and dynamic layer-wise cache allocation effectively preserves context and delivers robust performance in long-context scenarios. We will incorporate the related works and comparative experiments you suggested in the updated version of our paper.

LongBench (Mistral-7B-Instruct-v0.2) / (20% KV Cache)

W2:Additional results on benchmarks with longer contexts

Thank you for your valuable suggestion. In response, we conducted additional evaluations on datasets with longer contexts, including InfiniteBench [1][2] and RULER [3], both featuring instances exceeding 100K tokens. To ensure a fair comparison, we utilized the Llama-3.1-8B-Instruct (128K) model and benchmarked D2O against representative baselines, such as H2O (eviction-based), PyramidKV (layer-wise KV cache reduction), and CaM (value token merging). All experiments were performed on four A100 GPUs with 80GB memory. The results, provided in the tables below, demonstrate that D2O consistently outperforms these baselines across both InfiniteBench and RULER. These findings highlight D2O’s ability to effectively preserve context and maintain strong performance in extreme long-context scenarios, further validating its robustness. We appreciate your feedback, which helped enhance our evaluation.

[1] https://github.com/OpenBMB/InfiniteBench.

[2] https://github.com/thunlp/InfLLM.

[3] https://github.com/NVIDIA/RULER.

Infinite-Bench (Llama-3.1-8B-Insct (128K)) / (20% KV Cache)

Ruler (Llama-3.1-8B-Instruct (128K)) / (20% KV Cache)

You have offered constructive and valuable suggestions, making our submission more solid and complete. We promise to include the above results and analyses in the final version. Once again, we sincerely express our best gratitude for your effort and time!

Best wishes,

All authors of Submission 5363.

Q1: The growth of computational overhead

Thank you for this question. We present results on D2O’s computational overhead with increasing sequence lengths in Appendix A.5 ("Computational Cost Analysis"). In response to your suggestion, we have now included additional experiments comparing latency across different model sizes, specifically Meta-Llama-3-8B, Llama-2-13B, Code-Llama-34B, and Meta-Llama-3-70B, under varying sequence lengths for encoding/prompt encoding. These experiments follow the setup in Table 12 of Appendix A.5. For these tests, we used a single A100 80G GPU for the 8B and 13B models, two GPUs for the 34B model, and four GPUs for the 70B model. As shown in the results, inference time scales with both sequence length and model size, with larger models experiencing increased latency due to computational complexity and communication overhead, even with a fixed-size KV cache optimization.

Overall Generation Duration (s). （KV cache size=256）

Q2: The trade-off between the compression ratio and the output quality

Thank you for the question. We have analyzed the trade-offs between compression ratio and generation quality in Section 5.2 ("Comparative Analysis of KV Cache Compression Ratios") and Figure 5. Our results show that D2O, through its token-level dynamic merging and layer-level KV cache allocation, effectively preserves essential context across varying compression ratios, thereby enhancing reasoning accuracy without compromising performance. This balance highlights D2O’s ability to maintain output quality even as compression increases.

Q3 Dealing with extreme cases

Thank you for raising this important question. To evaluate D2O's performance on sequences longer than 100K tokens, we conducted additional experiments using the InfiniteBench dataset (average sequence length >100K). These experiments utilized the Llama-3.1-8B-Instruct (128K) model and compared D2O with representative baselines, including eviction-based methods (H2O), layer-wise KV cache reduction (PyramidKV), and value token merging (CaM). All tests were performed on four A100 GPUs with 80GB of memory. The results, presented in the table below, show that D2O consistently outperforms the baselines, even for sequences exceeding 100K tokens. This demonstrates that the combination of D2O's dynamic token-level merging and dynamic layer-wise cache allocation effectively preserves context and ensures robust performance in extreme long-context scenarios.

Infinite-Bench (Llama-3.1-8B-Instruct-128K) / (20% KV cache)

Q4 Allocation strategy based on exponential function

Thank you for the question. In Section 5.4, we compare Exponential Decay, Uniform Allocation, and our proposed Reciprocal of Variance. The ablation study shows that the inverse variance softmax effectively allocates KV cache based on attention variance, leading to superior performance across key metrics, demonstrating its adaptability and efficiency.

You have offered constructive and valuable suggestions, making our submission more solid and complete. We promise to include the above results and analyses in the final version. Once again, we sincerely express our best gratitude for your effort and time!

Best wishes,

All authors of Submission 5363.

We would like to thank you for taking the time to review our work. Your questions and comments are precious. We value each of them and have provided detailed responses below:

W1: The peculiarity of our core ideas

Thank you for your comments. We would like to clarify that our method is not a simple combination of existing techniques and introduces several key innovations:

• Differences from Existing Layer-Wise Allocation and Merging Strategies:

Layer-Wise KV Cache Allocation ：Our attention-variance-based dynamic KV cache strategy differs from heuristic approaches like PyramidKV, which reduce cache size with increasing layers. PyramidKV cannot dynamically adjust KV cache size based on the specific data sample. In contrast, D2O leverages attention density from the input data to adaptively and accurately allocate KV cache size according to the specific needs of each layer.

Token Merging Strategy: Unlike previous merging method such as CaM, which focuses on token-level merging, D2O performs a layer-level operation, dynamically selecting cache size based on attention variance across layers. For merging, CaM uses accumulated attention scores and a Bernoulli mask to evaluate token importance, whereas D2O sorts tokens by attention scores, calculates their similarity, and applies an EMA (Exponential Moving Average) threshold to determine merging. Additionally, CaM primarily merges value caches, while D2O first merges key caches and then applies merging weights to value caches. Our experimental results demonstrate the effectiveness of D2O’s dynamic layer-level and token-level strategies.

[1] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling.

[2] CaM: Cache Merging for Memory-efficient LLMs Inference.

• Heuristic Design of the EMA-Threshold Strategy:

As shown in Figure 1, attention patterns in higher layers exhibit a staircase-like pattern, focusing on local windows. Only a few critical tokens exist outside these windows, and indiscriminate merging can introduce redundant information or noise, affecting inference accuracy. Our EMA-threshold strategy dynamically selects essential tokens based on previous and current similarity, preserving critical information and reducing redundancy.

• Distinction from Traditional Attention-Based Pruning:

While attention-based pruning removes redundant representation at the intra-layer level, our dynamic allocation strategy operates at the Inter-layer level, using attention variance to allocate KV cache size across layers. By implementing an inverse-softmax-based dynamic allocation, we achieve superior results over uniform allocation, as shown in our ablation studies. The theoretical analysis in Appendix A.9 and experiment results in the paper further supports the effectiveness of our layer-level allocation over token-level pruning approaches.

W2: Explanation on insufficient analysis

Thank you for your comments.

1. We provide a visualization of attention patterns across different datasets and models in Figure 1 and Appendix A.8 ("Visualization of Attention Weights Across Various Datasets"), offering insights into how attention varies by layer and input data.

2. Regarding the choice of the 3:1 ratio for N:M we include an ablation study in the appendix A.3 (More details of hyper-parameters determination). Additionally, the attention pattern visualizations in Figure 1 and Appendix A.8 illustrate that allocating a smaller cache size to the immediate neighboring window and reserving more capacity for distant contexts allows D2O to retain critical information from longer contexts more effectively.

W3: Explanation on experimental fairness

Thank you for pointing out these concerns.

1. Regarding the comparison with baselines, we ensured fairness by using the default parameter settings provided in their respective papers and GitHub repositories. Since all our experimental scenarios involve generative evaluation, we only use the official generative-style code from H2O as a baseline for comparison. For D2O, we applied a unified set of hyperparameters derived from a hyperparameter search on a subset of datasets, as detailed in Appendix A.3. Furthermore, all methods were compared under the same compression rate to ensure consistency in evaluating performance.

2. For the trade-off between computational overhead and memory savings, we conducted detailed analyses in Section 5.5 and Appendix A.5. In Section 5.5, we evaluated the throughput of our method under varying batch sizes and sequence lengths on an A100 80G GPU. Additionally, Appendix A.5 provides a computational cost analysis, comparing prompt encoding and decoding latencies across different sequence lengths. These analyses validate both the computational efficiency and memory-saving capabilities of our approach, and we appreciate your suggestion to highlight these aspects.

Dear Reviewer LdwY,

We hope this message finds you well.

We have addressed the comments and concerns raised in the reviews to the best of our ability and are keen to receive any further feedback or clarification. With the decision deadline approaching, we would like to kindly check if there are any remaining questions or points you would like us to address.

We greatly appreciate the time and effort you and the other reviewers have dedicated to this process, and we are happy to provide any additional information if needed.

Thank you for your attention, and we look forward to hearing from you soon.

Best regards,

Authors of Submission #5363

Q2 The benchmark for evaluating both generation speed and performance

Thank you for the constructive suggestion. To assess inference time and generation quality during long-generation tasks, we conducted additional tests on the longbook_sum_eng dataset from InfiniteBench [1], which has an average output length of 1.1K tokens. Due to computational constraints, we tested the first 20 samples using the Llama-3.1-8B-Instruct (128K) model [2], comparing D2O with representative baselines, including the eviction-based method (H2O), layer-wise KV cache reduction (PyramidKV), and value token merge (CaM). The experiment was conducted using four A100 GPUs with 80GB.

Lookbook_sum_eng (Llama-3.1-8B-Instruct ) / (20% KV Cache)

As shown in the table, D2O achieved the best performance, with total inference time between H2O and PyramidKV and minimal latency differences.

[1] https://github.com/OpenBMB/InfiniteBench

[2] https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

You have offered constructive and valuable suggestions, making our submission more solid and complete. Once again, we sincerely express our best gratitude for your effort and time!

Best wishes,

All authors of Submission 5363.

We sincerely appreciate your review. Your effort has ensured that our submission received adequate attention and thorough review, and also provided us with lots of suggestions . We value each of your suggestions and provide the following responses:

W1: Comparing the allocation method in D2O with other works

Thank you for highlighting this point. In our view, those works, such as PyramidInfer [1] and PyramidKV [2], primarily rely on heuristically designed functions to allocate KV cache budgets based on the observed trend that attention density decreases with increasing layers. While effective to some extent, their limitation lies in the lack of adaptability to individual data samples, as these methods do not incorporate prior knowledge specific to the input data. In contrast, our proposed D2O employs a dynamic KV cache allocation mechanism that leverages the variance of attention scores across different layers for each specific input. This enables a more adaptive and data-driven allocation of KV cache, setting D2O apart from the heuristic approaches used in prior works. We believe this distinction underscores the novelty and practical advantages of our method. As you suggest, we added additional comparison experiments. The results show that D2O outperforms other non-adaptive KV cache allocation strategies in most of the LongBench datasets. This improvement highlights the advantage of our dynamic allocation strategy, which utilizes an inverse variance softmax function to adjust the KV cache size in each layer. And our adaptive approach is particularly effective in long-context scenarios, where maintaining relevant information across layers is crucial.

LongBench (Mistral-7B-Instruct-v0.2) / (20% KV Cache)

We appreciate your suggestion and have incorporated citations to PyramidInfer [1] and PyramidKV [2] in the Related Work section of the revised version of our paper.

W2 Confusing mathematical symbols and typos

Thank you for pointing this out. We acknowledge that these are typographical errors in the paper and will correct them by adjusting the mathematical symbols accordingly. In the revised version, we will make the necessary modifications.

W3 The clearer experiment setup

Thank you for the suggestion. In the revised version, we will include detailed length information for each dataset in LongBench. Specifically, the length of each dataset from LongBench can be categorized into three ranges: under 4K, 4-8K, and over 8K tokens. The average lengths are as follows: NrtvQA (18409), Qasper (3619), MF-en (4559), HotpotQA (9151), 2WikiMQA (4887), Musique (11214), GovReport (8734), QMSum (10614), MultiNews (2113), TREC (5177), TriviaQA (8209), SAMSum (6258), PCount (11141), PRe (9289), LCC (1235), and RepoBench-P (4206).

Q1 Experiment on the effect of generation length and token merging

Thank you for this question. In Appendix A.4 (Generated Samples of Multi-turn Conversations), our MT-bench experiments show that D2O’s dynamic merging strategy outperforms non-merging eviction strategies like H2O and StreamLLM, even with a fixed KV cache budget. These results indicate that D2O effectively mitigates the impact of increased generation length, preserving essential information more efficiently.

Dear Reviewer JAQq,

We greatly appreciate your valuable comments and suggestions on our paper. We have made every effort to address your concerns and sincerely hope these updates align with your expectations. In response, we have carefully revised our work and added additional experimental validations as necessary. We look forward to receiving your feedback at your earliest convenience.

Wishing you a wonderful Thanksgiving!

Best regards,

Team 5363

Dear Reviewer JAQq,

Thank you for your valuable feedback on our paper. We have carefully addressed all comments in the rebuttal and believe the revisions have improved the paper significantly.

As the rebuttal phase is nearing its end, we kindly ask if you would consider improving the scores, should our clarifications meet your expectations.

We greatly appreciate your time and effort. Please let us know if further clarification is needed.

Best regards, Authors