AttentionPredictor: Temporal Patterns Matter for KV Cache Compression

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

Weaknesses and Questions

W1-1 & Q1: While standard deviations are provided, the paper does not specify the method for error computation or the distributional assumptions. Sections C.5 and 4.6 provide standard deviations but do not specify the error computation method (e.g., closed-form, bootstrap) or distributional assumptions (e.g., normality). Please provide these details.

Thank you for this point. To ensure the statistical significance of our results, we conducted each experiment five times and reported the mean and standard deviation of the outcomes.

The standard deviation was calculated using the common formula for sample standard deviation, which provides a robust measure of performance variance without assuming a specific underlying data distribution. The formula used is:

$s = \sqrt{\frac{\sum_{i=1}^{n}(x_i - \bar{x})^2}{n-1}}$,

where $s$ is the sample standard deviation, $n$ is the number of experimental runs, $x_i$ is the result of each individual experiment, and $\bar{x}$ is the arithmetic mean of the five experimental results.

This method accurately reflects the variability in our results. We will add this clarification to the revised manuscript.

W1-2: Experiments are limited to LLaMA-3.1-8B and QwQ-32B, lacking validation on other architectures.

We appreciate the suggestion. As detailed in Section 4.1 of our paper, the evaluation was conducted on three distinct models: LongChat-v1.5-7b-32k, LLaMA-3.1-8B-Instruct, and QwQ-32B. The results presented in Table 1, Table 4, and Figure 4 demonstrate that AttentionPredictor is effective across these different model families. We agree that further validation is valuable and, we plan to extend our evaluation to other popular architectures like Mistral in our future work.

W2-1: While societal impacts, such as reduced latency and energy consumption, are noted, no deployment data for edge devices is provided.

Thank you for this question. Our work primarily targets large-scale, cloud-based LLM inference systems, where the immense memory footprint of the KV cache for long contexts is a critical bottleneck. While adapting KV cache compression for resource-scarce edge devices is an important research direction, it presents a different set of challenges and is beyond the primary scope of our current paper. We have clarified this target application scope in the manuscript.

W2-2 & Q2: Hyperparameter choices are empirically determined without theoretical justification or sensitivity analysis. Calibration step M and history step H (Table 9) are empirically determined. Please provide a theoretical justification or sensitivity analysis for their impact on performance.

We thank the reviewer for this point. We provide a sensitivity analysis for these key hyperparameters in Section 4.6 and Appendix C.5, with detailed results in Table 9. Our paper provides the following justification for these choices:

• For Calibration Step (M): As discussed in the appendix on lines 638-642, there is a clear trade-off. More frequent calibration improves accuracy by reducing cumulative errors from distribution drift, but at the cost of higher computational overhead.
• For History Step (H): As explained on lines 643-647, a moderate value for H is optimal because it provides sufficient historical information for pattern recognition, while a larger value can introduce redundancy due to the decaying self-correlation of attention scores.

W2-3 & Q3: The communication overhead analysis for cross-token prefetching lacks specific latency data. The cross-token prefetching framework claims to hide latency (Figure 5), but lacks a detailed breakdown of overhead components (e.g., prediction time, data transfer, synchronization).

Thank you for your insightful comment. Our cross-token prefetching framework is designed to hide this latency by overlapping asynchronous tasks with the main LLM computation. The process is illustrated in the timeline in Figure 10 (Appendix B.1). The key components are:

• Prediction Time: The prediction overhead is negligible because our unified CNN model is extremely lightweight at only 21KB.
• Data Transfer & Synchronization: The primary latency comes from fetching the KV cache blocks. This is hidden by executing the transfer in parallel with the computations of other layers of the current token.

Table 4 shows a detailed breakdown. Note that since we perform attention prediction once every two tokens, the average per-token inference latency is further reduced.

Table 4: Computation time breakdown with 1/16 budget. All values are reported in milliseconds per token (ms/token).

Dear Reviewer 8ZdG,

Thank you very much for your positive and constructive feedback. We sincerely appreciate your recognition of our efforts in addressing the concerns raised in the initial review, especially regarding the clarification of the standard deviation calculation, correction of model counts, and the scoping to cloud-based systems. We’re also grateful for your acknowledgment of our empirical analysis and justification of hyperparameter choices.

We are happy to further clarify the follow-up questions you raised:

Response to Question: Regarding Table 4

Could you please elaborate on the system dynamics that cause the Sync/Wait time to decrease as context length increases? Is the observed constant total overhead a fundamental benefit of the cross-token prefetching framework?

Thank you for your question. The decrease in Sync/Wait time with longer context lengths is indeed a fundamental result of our cross-token prefetching design.

As the context length increases, both Attention Prediction and Cache Transfer time grow due to larger attention maps and more KV blocks being prefetched. However, since these steps are fully asynchronous and run in parallel with the main LLM’s computation for the current token (t), the increased latency is absorbed into the LLM’s own inference time. This inference time remains nearly constant because the LLM operates on a compressed cache of fixed budget, making its computational cost relatively stable across context lengths.

The Sync/Wait column captures the remaining idle time after prefetching finishes but before the LLM starts decoding the next token (t+1). As prediction and transfer take longer at larger context lengths, there’s less idle time left, hence the decreasing Sync/Wait value.

This behavior illustrates a core strength of our system: longer prefetching pipelines are accommodated without increasing the overall per-token latency, thanks to tight pipelining with the main inference process.

Copy of Original Table 4: Computation breakdown at 1/16 budget (in ms/token).

We hope these additional clarifications address your remaining questions. Thank you again for your thoughtful review and continued support of our work.

Best regards,

Authors

Dear Reviewer 8ZdG,

We would like to express our sincere gratitude once again for your positive feedback, insightful comments, and constructive suggestions. Your guidance has been invaluable in helping us improve the quality of our work! With sincere dedication, we have carefully addressed your remaining concerns on the overhead of our prefetch system, as detailed in our preceding responses—in response to your invaluable comments.

We are writing to gently remind you that the author-reviewer discussion period will end in less than 12 hours. We eagerly await your feedback to understand if our responses have adequately addressed your concerns. If so, we would deeply appreciate it if you could raise your score. If not, we are eager to address any additional queries you might have, which will enable us to further enhance our work.

Once again, thank you for your kind support and constructive suggestions!

Best,

Authors

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

Weaknesses and Questions

W1: The writing remains a large improvement space. E.g. line 146 mentioned the notion $A^{comp}$, but doesn't tell what is $A^{comp}$.

Thank you for your valuable feedback on our manuscript and for highlighting areas for improvement in our writing. We will revise the text at its first mention.

Revision for lines 146-147: "The input to our predictor is a 2D spatiotemporal sequence $A_H = \langle A_i^{\text{comp}} \rangle_{i=t-H+1}^t$ , representing the H most recent block-wise compressed attention scores, where each $A^{comp}_i$ is generated via max-pooling on the original attention score $A_i$ (as detailed in Section 3.4)."

W2-1: Some explaination are still questionable. E.g. line 171 High query self-similarity. We know that softmax attention is a normalized score. This author only show the high similarity (0.87) of consecutive queries. To support the explaination in line 171, one need to also show the similarity of queries at far distance.

Thank you for this insightful question.

1. Rationale for Consecutive Query Similarity.

Our model predicts the attention at step $t+1$ from a recent history. Since the temporal dimension corresponds to the query sequence (one query per step), the similarity between adjacent queries ($q_t$ and $q_{t+1}$) is the most critical factor for demonstrating step-by-step temporal continuity. This is why we initially focused on the high consecutive query similarity (lag-1) of 0.87.

2. On Attention Normalization.

It is also worth noting that the Softmax function normalizes attention scores across the key dimension for a single query; it does not influence the relationship between different queries across the temporal dimension.

3. Analysis of Query Similarity Across Distances

To provide a more detailed analysis of long-range stability, we measure the average query cosine similarity across various distances (lags). The results are presented in Table 1. The query similarity gradually decreases as the distance increases, which is expected. However, the similarity remains relatively high even at longer distances. This finding suggests that while the influence of the most recent tokens is strongest, the attention patterns exhibit a strong local stability that decays slowly over time. This supports our model's effectiveness in capturing these evolving yet persistent patterns using a recent history window.

Table 1: Query Similarity over Different Distances.

W2-2 & Q1: The author doesn't explain the reason of Seasonal Pattern.

Thank you. The seasonal pattern arises from combining two factors:

1. RoPE's Periodicity: Rotational embeddings are inherently periodic.
2. Contextual Periodicity: In structured data, queries separated by a period $S$ can be similar, so $q_t \approx q_{t+S}$.

The attention score is $A_{t,i} = \sum_m \lVert q_t^{(m)} \rVert \cdot \lVert k_i^{(m)} \rVert \cdot \cos\left( \phi_{t,i}^{(m)} + (i - t)\theta_m \right)$. Each frequency $m$ in RoPE has a period $T_m = 2\pi/\theta_m$. We assume the input's seasonal period $S$ aligns with a dominant frequency's period ($S\theta_{m^*} \approx 2\pi$).

Consequently, the rotational component for the relative distance $(i-(t+S))$ aligns with that for $(i-t)$. When this RoPE-induced alignment combines with periodic queries ($q_t \approx q_{t+S}$) and the same key vector $k_i$, we find that $A_{t,i} \approx A_{t+S, i}$, which explains the seasonal pattern.

W3: the calibration step M is 5, a small number. AKA, every 5 steps the proposed method needs to 1) compute attention scores on the whole sequence; 2) fetch KV cache of the whole sequence. The former requires high computation cost, while the later requires high memory cost.

We thank the reviewer for their insightful comment regarding the computational and memory overhead of the calibration step.

To analyze this trade-off, we conducted an ablation study on the calibration step M. The results on LLaMA-3.1-8B are presented in Table 2, comparing different calibration frequencies (M=5, M=20) and no calibration (w/o M).

Based on these results, we highlight three points:

1. The calibration step is an optional enhancement. Our core method without calibration already outperforms SOTA. This demonstrates the strength of our prediction mechanism, with calibration serving only to further refine accuracy.
2. Our method offers a flexible accuracy-efficiency trade-off. Increasing the calibration interval from M=5 to M=20 reduces the calibration computational overhead by 75% with only a minor performance decrease, still surpassing baselines.
3. The calibration overhead is minimal and can be hidden.
   • Low-Cost Approximate Calculation: The calibration only requires block-level attention scores, so we can therefore use efficient approximation methods, similar to those in Quest, to compute with minimal computational cost.
   • Parallel Offloading: In modern inference systems that separate prefill and decoding stages, this lightweight attention calculation can be offloaded to the CPU. This allows it to run in parallel, ensuring no impact on the main model's inference throughput.

Table 2: Sensitivity Analysis of Calibration Step M on Llama-3.1-8B

W4 & Q2: The evaluation are conducted on narrow tasks, a simple needle-in-a-haystack task and a longbench task. What is the performance of the proposed method on other tasks? e.g. RULER qa tasks, GSM8K, MMLU?

We thank the reviewer for their insightful feedback and constructive suggestions. In our initial submission, beyond the Needle-in-a-Haystack and LongBench tasks, we also included evaluations on:

• The GSM8K benchmark with a long n-shot CoT setting, to assess performance on long-input mathematical reasoning. This is presented in Appendix C.1.
• A long-output reasoning task (AIME), which presents distinct challenges compared to long-input tasks. The results are detailed in Section 4.2 and Figure 4.
• The InfiniBench benchmark, which involves extremely long contexts with an average length of 214K tokens. These results can be found in Appendix C.3.

To further address your comments with a more appropriate benchmark, we are conducting additional experiments on the RULER qa benchmark. We are providing the GSM8K results from our paper here for your convenience.

GSM8K Results (n-shot CoT) As shown in table 3 below (Table 4 from our paper), AttentionPredictor maintains high accuracy on the benchmark across various long-context prompt lengths, significantly outperforming other methods.

Table 3: Evaluation results on the long n-shot CoT GSM8K task with Llama-3.1-8B-Instruct.

RULER qa Results

Following the reviewer's suggestion, we have conducted new experiments on the RULER QA benchmark. The results are presented in Table 4.

Table 4: Evaluation results on the RULER qa benchmark under various context lengths and KV cache budgets.

The results demonstrate that with a highly constrained budget of only 1K tokens, our AttentionPredictor consistently and significantly outperforms the SnapKV baseline across all tested context lengths.

MMLU: We appreciate the suggestion to evaluate on MMLU. However, we believe it is not the most suitable benchmark to demonstrate the effectiveness of our method for two main reasons:

1. MMLU tasks are resolved in a single generation step, which only involves the prefill stage. Since our AttentionPredictor method operates during the decoding stage, it is not applied in these scenarios.
2. Even with a standard 5-shot setting, the average context length for MMLU tasks is only around 500 tokens. This does not align with the long-context scenarios our work targets, where KV cache compression is a critical challenge.

We hope these clarifications and additional results adequately address your concerns. Thank you again for your valuable feedback, which has helped us strengthen our paper.

Dear Reviewer Y2CJ,

We genuinely value the opportunity to clarify and address any remaining concerns.

Response to Question 1: Seasonal Pattern Explanation

Reviewer's Question: What kind of contextual periodictiy is it in practice? Could you provide an example? Is it an assumption? It seems contextural periodicity ($q_t \approx q_{t+S}$) is enough to produce a seasonal pattern. Is it correct?

Thank you for the follow-up questions, which allow us to clarify the mechanism behind the seasonal pattern.

1. Example of Contextual Periodicity in Practice

Contextual periodicity is not a strict assumption but an empirical observation in structured or semi-structured inputs.

A concrete example is processing a structured document like a code, a poem or a table.

• Example: In a long class definition, the structure for declaring properties is often highly repetitive. The queries generated when processing each of these lines can be very similar, creating a periodic context.

      ...
      private bool m_ReadyWait;
      private int m_ReadyCount;
      private bool m_Rematch;
      public bool Rematch{ get{ return m_Rematch; } }
      public bool ReadyWait{ get{ return m_ReadyWait; } }
      public int ReadyCount{ get{ return m_ReadyCount; } }
      public bool Registered{ get{ return m_Registered; } }
      public bool Finished{ get{ return m_Finished; } }
      public bool Started{ get{ return m_Started; } }
      public Mobile Initiator{ get{ return m_Initiator; } }
      ...

2. Is Contextual Periodicity ($q_t \approx q_{t+S}$) Sufficient?

You raise an excellent point regarding the role of contextual periodicity. Our analysis indicates that contextual periodicity ($q_t \approx q_{t+S}$) is indeed the fundamental driver behind the seasonal pattern.

However, this periodicity alone is often insufficient to create the sharp, stable patterns observed in practice, because the positional shift from $t$ to $t+S$ alters the RoPE rotation for each frequency channel. The distinct seasonal pattern emerges when the input's period $S$ resonates with the inherent period of a dominant RoPE frequency channel ($S\theta_m \approx 2k\pi$). In this scenario, the alignment with RoPE's periodicity makes the underlying pattern driven by the context sharp and clearly visible.

Dear Reviewer Y2CJ,

Thank you for taking the time to review our rebuttal and for your follow-up note. While we understand and respect your decision to maintain the original score, we would be truly grateful if you could let us know if there are any remaining concerns or points that you feel were insufficiently addressed.

We are more than happy to provide further clarification or additional analysis if needed. Your feedback is very important to us, and we genuinely hope to improve the work in any way possible based on your suggestions.

Thank you again for your time and consideration.

Best regards,

Authors

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns.

Weaknesses

Motivation and Contributions

W1: Define information formally.

We thank the reviewer for this insightful comment.

1. On "Attention Recovery Rate" as a Metric: We agree minimizing output error is the ultimate goal. We argue that attention recovery rate serves as an effective proxy, as attention output $o = A_t · V$ depends directly on the attention scores. A high recovery rate means the compressed distribution $A_t'$ closely matches $A_t$, leading to an accurate $o' \approx o$. This justifies its use to assess information preservation.
2. The primary contribution is the $AttentionPredictor$, a universal module that predicts the next-token attention distribution. It is distinct from the selection policy and can be paired with various methods, including Top-K or importance sampling.
3. Top-K vs. Sampling & Why Our Method Excels: While Top-K can be biased in some tasks (as noted by MagicPIG), our results show that a precise predictor matters more than an unbiased selection policy. Our AttentionPredictor captures spatiotemporal patterns to better estimate attention, so even with simple Top-K, it outperforms MagicPIG’s complex sampling (Table 1), highlighting the predictor’s key role.

Table 1: Longbench task results with 10% KV budgets.

W2: Provide evidence for the claim that MInference struggles to capture dynamic temporal patterns.

Thank you for the thoughtful question.

1. Temporal patterns are locally stable but evolve globally during long inferences—e.g., shifting diagonal positions. Furthermore, we identified a novel seasonal pattern, where attention periodically shifts between token groups, a dynamic behavior not defined by MInference's categories.
2. Why MInference struggles to capture these dynamics accurately:
   • MInference uses fixed templates per head, making it hard to adapt to such evolving or periodic structures.
   • In contrast, our AttentionPredictor is fully dynamic and learns these transitions from attention history.
3. Table 2 shows a 4.77% higher accuracy, supporting our claim that modeling these dynamics leads to better attention prediction and information retention.

Table 2: Attention prediction accuracy (%) on Llama-3.1-8B-Instruct with 1K KV budget.

W3: How does RoPE lead to diagonal consistency and periodic slashes in attention patterns?

Thank you for the great question.

1. Consistent Attention along Diagonals

Consider the attention scores for two diagonal tokens, $A_{t,i}$ and $A_{t+1, i+1}$. Using RoPE, the score is: $A_{t+1, i+1} = \sum_{m=1}^{d/2}||q_{t+1}^{(m)}||||k_{i+1}^{(m)}||\cos(\phi'^{(m)}+(i-t)\theta_{m})$ Note that $(i+1)-(t+1) = i - t$, so the relative position stays the same along the diagonal. Since RoPE encodes positions via relative angles, the cosine term is unchanged. The other terms in the sum ($||q||$, $||k||$, and the initial phase $\phi$) vary slowly across time due to self-similarity (Sec 3.3), so overall: $A_{t,i} \approx A_{t+1, i+1}$ This leads to stable attention scores along diagonals.

2. Periodic Slashes

RoPE attention scores are sums of cosine functions over relative positions: $A_{t,i} = \sum_{m=1}^{d/2} C_m \cos\big(\phi^{(m)} + (i-t)\theta_m\big)$ Each term is periodic with period $T_m = 2\pi / \theta_m$. If the sum is dominated by a single frequency $\theta_{m*}$, the attention score approximates a single cosine function: $A_{t,i} \approx C_{m*} \cos(\phi^{(m*)}+(i-t)\theta_{m*})$ This causes attention to peak at regular intervals in key positions, forming periodic slash patterns with spacing ≈ $T_{m^*}$.

Efficiency Claims and Measurements:

W4 & Q3: The speed-up is not under general settings. Describe the efficiency experiment setup.

1. Why Offloading Matters: Our cache compression work focuses on the cache-offloading scenario, which is critical for long-context generation when KV cache exceeds GPU memory. Unlike GPU-based eviction/compression, offloading keeps the full cache on CPU, preserving model accuracy. This setting is also adopted in recent works [1–4].

2. Experiment Setup.

   • FlashAttention2: Standard offloading implementation with cross-layer prefetching. At each decoding layer, it load the entire KV cache for the upcoming layers from CPU to GPU to compute the full attention.
   • H2O: Also uses cross-layer prefetching but transfers sparse KV cache, making the comparison a more direct evaluation of the prefetching framework's efficiency.
   • AttentionPredictor: Runs on an offloading setup but prefetches across tokens, improving efficiency.

3. Applicability to General Scenarios. While designed for offloading, AttentionPredictor is adaptable—it can act as a cache eviction policy in memory-constrained setups, predicting less important blocks to evict and speeding up decoding.

[1] InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory. NeurIPS 24.

[2] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference. ICML 25 (Spotlight).

[3] InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. OSDI 24.

[4] PQCache: Product quantization-based kvcache for long context llm inference. PACMMOD 25.

W5 & Q4: The memory impact of cross-token prefetching is not discussed.

Our cross-token prefetching framework actually consumes less GPU memory during decoding compared to a standard cross-layer approach. See Table 3. The reason is that the memory savings from loading only a small, sparse KV cache outweighs the potential overhead from the cross-token prefetching mechanism.

Table 3: Peak GPU Memory Consumption (GB) during Decoding.

W6: Lack Compute breakdown.

Table 4 shows a detailed breakdown of our cross-token prefetching process (the timeline in Figure 10, Appendix B.1). Note that since we perform attention prediction once every two tokens, the average per-token inference latency is further reduced.

Table 4: Computation breakdown at 1/16 budget (in ms/token).

W7 & W10: Lack baseline codes.

We used the official implementations released by the original authors to reproduce the baseline results.

W8 & W9 & W11: Lack statistical analysis.

We ran our method 5 times and the results are reported in Table 5 and Table 6. Our method demonstrates both strong performance and consistency, outperforming other baselines.

Table 5: LongBench results with LLaMA-3.1-8B-Instruct (Budget=1K)

Table 6: AIME-2024 CoT reasoning task

W12: Q8: Lack experimental setup details.

We will provide comprehensive information on the hardware specifications (CPU: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz, RAM: 3.9Ti, OS: Ubuntu 22.04) and software environment (PyTorch: 2.4.0, CUDA versions: 11.8).

Questions

Q1 & Q2: What is meant by static patterns and dynamic patterns?

Thank you for your insightful questions.

1. Static Patterns & H2O:
   • Static patterns are attention behaviors based on fixed rules that don’t adapt during a single, long-output inference.
   • H2O uses cumulative attention scores to select tokens, favoring early, frequent ones. This helps capture fixed-position patterns (e.g., attention sinks) but limits responsiveness to new context later in generation.[5]
2. Dynamic Patterns & SeerAttention:
   • Dynamic patterns evolve throughout generation.For instance, in a sequential pattern, the position, interval, and intensity of the diagonal lines can shift over time (as in Figure 2 in our paper).
   • SeerAttention and similar methods use learnable modules to capture dynamic patterns. However, they predict attention distribution indirectly. SeerAttention, for example, learns from a compressed representation of Q&K to infer the attention distribution, not the distribution itself.
3. Motivation:
   • This indirect approach can misalign with true attention scores. Our AttentionPredictor directly models next-token attention as a 2D time series, achieving much higher accuracy than SeerAttention—especially under tight budgets (Table 5, Appendix C.2).

[5] Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs. ICML 25.

Dear Reviewer gSQC,

We would like to once again express our sincere thanks for your insightful comments and suggestions. Your feedback has been immensely valuable in helping us improve our work. This is a gentle reminder that there are fewer than three days remaining before the end of the reviewer-author discussion phase. We would greatly appreciate hearing from you if you have any further evaluations, questions, or concerns regarding our responses.

If our rebuttal has properly addressed your concerns, we would be grateful if you would consider updating your score. If not, we would be more than happy to continue the discussion and further clarify any outstanding issues.

We hope this message does not inconvenience you in any way. Thank you again for your time and thoughtful engagement.

Best regards,

Authors of Submission 25755

Response to Question 2: Self-similarity of Keys

W3 The other terms in the sum vary slowly across time due to self-similarity.

In Section 3.3, you only compute similarity for Q, while Line 180 mentions that the K matrix remains "relatively stable." It would be helpful to extend the empirical analysis to include self-similarity of keys as well. Thanks for the detailed explanation of Periodic Slashes.

Thank you for your thoughtful question. We would like to clarify two points:

1. Regarding Line 180 and the statement about stable Keys: In this case, we are considering the attention scores between pairs such as $A_{t,i}$ and $A_{t+1,i}$, where both attend to the same position $i$. Therefore, the Keys involved in both computations are exactly the same ($k_i$), and thus no similarity issue arises in this context.

2. Regarding the diagonal "Periodic Slashes" pattern and the stable Keys: This involves comparisons like $A_{t,i}$ and $A_{t+1,i+1}$, where the attention scores are influenced by different Keys ($k_i$ and $k_{i+1}$). In response to your suggestion, we computed the similarity between Key similarity across various distances (lags) and found that they indeed exhibit strong self-similarity. The results are presented in Table 2.

   These high similarity values suggest that Keys vary slowly over time. Our findings are also consistent with prior works such as KIVI [1] and KVQuant [2], which observed that Keys often exhibit fixed, massive channels, indicating limited variation across positions.

   Table 2: Key Self-Similarity at Different Distances

   Distance	1	2	5	10	20
   Similarity	0.82	0.77	0.74	0.72	0.71

   [1] KIVI: A Tuning-Free Asymmetric 2-Bit Quantization for KV Cache. ICML 2024.

   [2] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. NeurIPS 2024.

We hope this additional empirical evidence strengthens the explanation of the self-similarity hypothesis.

Dear Reviewer gSQC,

We would like to express our sincere gratitude once again for your positive feedback, insightful comments, and constructive suggestions. Your guidance has been invaluable in helping us improve the quality of our work! With sincere dedication, we have carefully addressed your remaining concerns on MInference comparison, key self-similarity, overhead in no-offloading setup, dynamic vs. fixed rules, and related work, as detailed in our preceding responses—in response to your invaluable comments.

We are writing to gently remind you that the author-reviewer discussion period will end in less than 12 hours. We eagerly await your feedback to understand if our responses have adequately addressed your concerns. If so, we would deeply appreciate it if you could raise your score. If not, we are eager to address any additional queries you might have, which will enable us to further enhance our work.

Once again, thank you for your kind support and constructive suggestions!

Best,

Authors

We thank the reviewer for the insightful and valuable comments. We are encouraged by your positive feedback on the novelty, generalization, and lightweight nature of our approach. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns.

W1: Clarification on the term "KV Cache Compression".

It is a little confusing that since we get the indices of critical tokens from the attention predictor (see Algorithm 1), does it mean we need to keep all KV caches? If so, it is inappropriate to claim it is a method for KV cache compression.

Thank you for this insightful question. You are correct that we maintain the full KV cache in CPU memory. Our method achieves "compression" by involving only a subset of the KV cache in the attention computation, effectively reducing the active GPU memory footprint, which is the primary bottleneck during inference.

Terms such as "KV cache retrieval" or "attention sparsity" may describe our method more precisely. We will revise the terminology in the final version of the paper to ensure greater clarity.

W2: Details of the CNN Architecture.

While the training process is described in Appendix B.1, more details about the CNN architecture (number of layers, kernel sizes, etc.) would be helpful.

Thank you for the suggestion. To provide a clearer and more quantifiable overview of our CNN architecture, we will add the following detailed breakdown to the paper's appendix. The architecture is in Table 1. This design choice makes the model lightweight and adaptable to varying sequence lengths.

Table 1: AttentionPredictor Structure.

W3: Computational Overhead of the AttentionPredictor.

The computational overhead of the prediction model itself could be better quantified, especially since it runs at every decoding step.

Thank you for raising this important point. To clarify, the predictor does not run at every decoding step. To balance performance and overhead, we invoke the predictor periodically. In our experiments, we update the critical token set only every two steps, which effectively amortizes the prediction cost.

We have quantified the overhead of our predictor model. As shown in Table 2, both the latency and memory footprint are minimal, ensuring that the prediction process does not become a bottleneck. We will include this analysis in the final paper.

Table 2: The computational overhead of the prediction model.

Additionally, Table 3 shows a detailed breakdown of our cache prefetch system. Our cross-token prefetching framework is illustrated in the timeline in Figure 10 (Appendix B.1).

Table 3: Computation breakdown at 1/16 budget (in ms/token).

The "Attention Prediction" latency in Table 3 is greater than the values in Table 2 because it encompasses not only the prediction model inference but also additional matrix operations essential for the prefetching process.

Note that since we perform attention prediction once every two tokens, the average per-token inference latency is further reduced.