We thank the reviewer for the constructive suggestions, and we would like to clarify a few points.

Responses to Weakness

W1: It's not clear if this method can be implemented in real systems like vLLM/SGLang.

A1:

The method could be implemented in both vLLM and SGLang. In the case of vLLM, we have uploaded the implementation (https://github.com/a-anonymous-user/vLLM-KV) from an anonymous account. The eviction process involves the following key steps:

1. Determine the KV Size Budget for Layers: Calculate the memory allocation for the key-value pairs (KVs) for each layer in the cache system. This step involves assessing the total available memory and deciding how much should be assigned to each layer.

2. Select KVs for Eviction: Identify specific KVs to be evicted based on their attention scores.

3. Reorder KVs in the Physical Cache: Rearrange the KVs in the physical cache so that all evicted KVs are grouped into contiguous blocks that do not contain any non-evicted KVs. This reordering is essential for efficient block-level memory management. Free Blocks: Release the blocks containing the evicted KVs, allowing the memory to be reallocated for subsequent scheduling iterations. This step updates the BlockManager but leaves the underlying physical cache structure unchanged.

4. Update the Physical Cache: Adjust the positioning of KVs in the physical cache according to the new ordering determined in Step 3.

To illustrate this process, we provide a vLLM implementation demonstrating these steps. Specifically:

1. Step 1 is managed by https://github.com/a-anonymous-user/vLLM-KV/blob/main/vllm/kvcompress/metrics.py.

2. Steps 2–4 are handled by CompressionScheduler in https://github.com/a-anonymous-user/vLLM-KV/blob/main/vllm/kvcompress/scheduler.py.

3. Step 5 is executed by CacheEngine.execute_cache_moves in https://github.com/a-anonymous-user/vLLM-KV/blob/main/vllm/llm/llm_engine.py, which is invoked in the main engine loop.

Responses to Questions

Q1: How does the speed-up change wrt tensor parallel and pipeline parallel?

A1:

The speed-up offered by PyramidKV is complementary to that achieved through tensor parallelism and pipeline parallelism, as these approaches are not mutually exclusive. PyramidKV can be seamlessly integrated with both tensor parallelism and pipeline parallelism. We have included these details at Appendix D.

Q2: How does the memory allocation and release be implemented? I would assume there will be significant memory fragmentation during the process.

A2:

To perform KV cache eviction, we use torch.gather. Below, we outline the memory allocation and release process of torch.gather:

1. Index Selection: Identify the positions of the elements to extract from the input tensor.

2. Memory Location Calculation: Compute the specific memory locations of the elements to be extracted using the strides of the input tensor across each dimension.

3. Output Tensor Creation: Allocate memory to create a new output tensor and copy the selected elements to their corresponding positions in the output tensor.

4. Memory Management: Since torch.gather is not an in-place operation, it creates a new tensor to store the results, while the memory of the original input tensor is released.

We would like to thank the reviewer once again for the valuable feedback, which can significantly help us improve our work. We hope these clarifications can address the concerns. We hope you find our response compelling enough to consider a re-evaluation of our score. If there are any remaining concerns, we would really appreciate the opportunity to discuss them further during the discussion period.

Thanks for the helpful discussion!

Would you mind sharing the script and setting for benchmarking PyramidKV against vLLM? I'm quite surprised it can be done without big refactoring over the whole CSRC, e.g., cache_kernels.cu and attention kernels. I'll raise my score after I reproduced the result. I'll make sure to finish before Dec 2nd.

We sincerely thank the reviewer for their insightful questions regarding the implementation of KV cache eviction in vLLM [1] and its alignment with the memory management design at vLLM. We would like to take this opportunity to clarify a few key points. While we are currently unable to update the paper, we will make sure that all the following discussions will be incorporated into the discussion section of the revised version of our paper.

Challenges and Limitations

In this section, we discuss the challenges and limitations about the implementation of PyramidKV in vLLM, focusing on aligning the implementation with vLLM’s memory management design. Our primary emphasis is on redesigning the cache architecture to minimize fragmentation, which requires refactoring some components in CSRC, particularly within the cache kernels. We estimate that implementing KV cache eviction will result in a 15% slowdown due to additional overhead. Therefore, KV cache eviction works enhance total throughput by enabling larger batch sizes, but latency for each sample is about 15% slower.

Fragmentation in original vLLM’s kv cache design

In vLLM's original design, each block stores the keys and values generated for all attention heads across all tokens in the block. Consequently, allocating keys and values for one head requires allocating them for all heads. This design can lead to increased memory fragmentation, particularly in PyramidKV, where the number of keys and values varies across layers. Although the number of keys and values per head within a layer remains consistent, these numbers differ between layers. Under vLLM's original design, if one head requires more keys and values than the others, the system must allocate space for the same number across all heads, which would add fragmentation.

Re-design the cache structure to avoid fragmentation

To prevent fragmentation, it is necessary to redesign the KV cache structure. The fragmentation issue can be resolved by adopting a new design where each block contains KV pairs for a single head. This approach eliminates the need to allocate equal space for all heads when one head requires more KV pairs, thereby reducing fragmentation. By restructuring the cache to allocate blocks on a per-head basis, we ensure that additional KV pairs for one head do not force unnecessary space allocation for other heads, effectively mitigating fragmentation.

Scheduling Overhead due to the re-design of cache structure

Increased size of blocks in the new-designed cache structure

Originally, each block contained one token per layer and head (32 layers and 8 KV heads for LLaMA). However, the transformation modifies this structure so that each block contains a key or value for a single head.

Increased size of blocks in the case of LlaMa

This implies that the number of blocks increases significantly. For a model with num_layer transformers with num_head heads for each layer, the number of blocks will be num_layer×num_head times larger. For instance, in the case of llama (32 layers Transformers and each key and value has 8 heads) if there were initially 128 blocks, this would expand to 128×32×8 making the number of blocks 32×8 times larger.

Increased size of blocks results in additional scheduling overhead

This increase is the primary source of additional scheduling overhead, mainly arising from operations on block tables and context-length tensors. These tensors are used to determine the number of new blocks needed and available blocks to allocate during scheduling.

Previously, scheduling was performed while block tables were stored as Python lists, which were dynamically updated during block allocation. After scheduling, GPU tensors were generated from these block tables for use during the attention forward pass. To optimize this process, it is necessary to update the block manager to store block tables directly on the GPU, enabling all block allocation operations to be handled in PyTorch. As a result, the block tables can be sent directly to the model forward pass without intermediate conversions. Moving scheduling operations to the GPU significantly improves speed.

Overall, the additional 15% overhead primarily stems from the increased number of blocks.

IO Overhead

The eviction process is relatively fast, and the I/O overhead may not constitute a significant portion of the total overhead, particularly because PyramidKV does not perform compression at every step. However, if the KV cache eviction were to occur at every step, the I/O overhead could be substantially higher.

We thank the reviewer for the helpful suggestions, and we would like to clarify a few points.

Responses to Weakness

W1: Lack of Novelty: The main contribution of the paper [1]

R1:

Based on the ICLR review guidelines (https://iclr.cc/Conferences/2025/ReviewerGuide) that papers are contemporaneous if they are published within the last four months, we regard [1] as a concurrent work.

Our work differs from PyramidInfer in three key aspects:

1. Decay Strategy: While PyramidInfer employs a geometric decay strategy, our method adopts an arithmetic decay strategy. We argue that the relatively stable and linear nature of arithmetic decay better aligns with the behavior of the attention mechanism. This strategy is derived from empirically observed attention patterns, aiming to closely match them. Notably, our approach also achieves superior results, as demonstrated in the experimental results presented in the table below.

2. Token Selection: PyramidInfer discards tokens in earlier layers, preventing them from being reconsidered in later layers. In contrast, our method allows previously discarded tokens to be re-evaluated in higher layers, recognizing that these tokens may still hold relevance at different stages of the model's processing.

3. Pyramidal Information Funneling Pattern: A key contribution of our work lies in identifying and leveraging the pyramidal information funneling phenomenon within attention mechanisms. Through in-depth analysis, we observe that attention tends to disperse in earlier layers and progressively concentrates on crucial tokens in higher layers. This insight forms the foundation of our arithmetic decay strategy, ensuring that our method aligns more naturally with these intrinsic patterns.

Despite some similarities between the two approaches, these differences lead to significantly distinct outcomes. As shown in the table below, our method consistently outperforms PyramidInfer at 64 KV cache size, highlighting the effectiveness of our design choices. We have included these differences in Appendix J.

Responses to Questions

Q1: The experiments indicate that the proposed pyramid KV cache allocation does not consistently surpass other baselines across all tasks. Do the authors have insights into which types of tasks this pyramidal allocation performs best and where it tends to underperform?

A1:

1. Among the 16 datasets, the tasks where our proposed method performs slightly worse than the baseline are mostly saturated (e.g., HotpotQA, Musique, etc. under the LlaMa-3-8B-Instruct setting with KV Size = 64, as shown in Table 1). In these cases, our method is only marginally inferior to the baseline and remains competitive.

2. Conversely, on tasks with greater potential for improvement (e.g., Qasper, MF-en, TREC, TriviaQA, etc. under the same setting), our method significantly outperforms the baseline. Consequently, the overall average performance of our method surpasses that of the baselines.

3. Notably, these tasks include several In-Context Learning (ICL) tasks (i.e., TREC), our method enjoys the best performance gain at ICL tasks, where the method demonstrates the ability to leverage provided examples to adapt effectively. The importance of ICL tasks lies in their widespread use in real-world applications, where dynamic adaptation to new inputs is critical.

Q2: By summing the budgets across all layers and setting this sum equal to the total budget, can beta be directly calculated instead of being treated as a hyperparameter? Is there a misunderstanding in my interpretation of this allocation scheme?

A2:

1. The parameter $\beta$ is still required to determine the top layer.
2. Once the top layer is identified, the budget of the bottom layer can be calculated by summing the budgets across all layers and equating this sum to the total budget.
3. Finally, the step size can be determined.

We would like to thank the reviewer again for the valuable feedback, which can significantly improve our work! We are glad that our responses address the concerns. If there are any remaining concerns, we would really appreciate the opportunity to discuss them further during the discussion period and address them.

We thank the reviewer for the suggestions, and we would like to clarify a few points.

Responses to Weakness

W1: When the KV budget is retained at 2k, the accuracy of the proposed method does not show significant advantages.

A1:

1. With a small budget, our proposed method's effective allocation strategy better preserves useful attention information. However, with a larger budget, this allocation becomes less critical for covering the necessary information.

2. To further illustrate this phenomenon, we have included an ablation study titled "Attention Recall Rate Experiment" in Figure 8 from Appendix M. The results show that with a small budget (i.e., 64 KV cache size), PyramidKV improves the attention recall rate (the percentage of attention computed using the keys retrieved by the method and the query, relative to the attention computed using all keys and the query.). However, with a larger budget (i.e., 2k KV cache size), the improvement decreases. For 64, 128, 256, 512, 1024 and 2048 KV cache sizes, PyramidKV's average attention recall rate improvements are 1.87%, 0.64%, 0.61%, 0.56%, 0.47%, and 0.36%.

W2: The paper mainly tests models with an 8k context length, lacking accuracy tests for models with sequence lengths above 128k.

A2:

In response, we conducted additional experiments using Llama-3-8B-Instruct-Gradient-1048k with an input length of 128k. The results, summarized in the table below, showcase the model's average performance at LongBench with extended context lengths. We have included details in Appendix N.

W3: In cases of extremely low compression ratios, it is recommended to include comparisons with new technologies such as Minference.

A3:

We would like to clarify that PyramidKV and MInference are complementary approaches addressing different aspects of KV cache optimization. Specifically:

1. MInference focuses on accelerating the generation of KV caches during the pre-filling stage of LLM inference.

2. In contrast, PyramidKV targets efficient KV cache management during LLM decoding.

To evaluate their respective strengths, we compared PyramidKV and MInference on Longbench using a KV cache size of 128. The results demonstrated the superior performance of PyramidKV.

Furthermore, we demonstrate that MInference and PyramidKV can be seamlessly integrated to achieve highly efficient inference while maintaining performance comparable to full attention. The LongBench average results of MInference combined with PyramidKV, evaluated on Longbench with a KV cache size of 128, are presented below:

In summary, we demonstrate that PyramidKV outperforms MInference on Longbench. Furthermore, when integrated with MInference, PyramidKV enhances its performance even further. We have included details in Appendix I.

Responses to Questions

Q1: Besides Llama-like models, do other models also exhibit the Pyramidal Information Funneling phenomenon?

A1:

We further provide additional analysis at Figure 5 and Figure 6 in Appendix C, demonstrating that the Pyramidal Information Funneling phenomenon is also evident in both the Mistral model and the Mixtral mixture-of-experts model. Rather than Llama-like models, Pyramidal Information Funneling phenomenon also exhibit in Mixture-of-Expert models. This supports the universality of the Pyramidal Information Funneling phenomenon across diverse model families. We hope this addresses your concern and underscores the generalizability of our findings.

Q2: When determining the KV Cache budget for different layers, how should hyperparameters be selected to ensure optimal accuracy?

A2:

We conducted hyperparameter testing on the original development sets of 16 datasets in LongBench. The parameter $\beta$demonstrated remarkable stability, showing minimal sensitivity to varying hyperparameter settings, which highlights its robustness. Conversely, $\alpha$ consistently produced superior results when set to 8 or 16. Consequently, these values were adopted for subsequent experiments.

In Appendix H.2 and H.3, we further analyzed the impact of hyperparameter selection on KV cache budget allocation across different layers. The experiments reaffirmed that $\beta$ had negligible influence on the outcomes, underscoring its stability. Meanwhile, \alphaα continued to deliver optimal results at values of 8 and 16.

We thank the reviewer for the helpful suggestions, and we would like to clarify a few points.

W1: The proposed method works really well under extreme condition. However, under not-so-extreme cases, the performance is not comparable to other baselines according to Table 1 in the paper. Is there any explanation to this phenomenon? I think the paper worth a small section of ablation study to explain this phenomenon.

A1:

Thank you for your suggestions and for bringing this observation to our attention. While it may appear that the advantages of our method are less pronounced in moderate cases compared to extreme cases, we would like to highlight two key aspects of our results.

1. Firstly, with a small budget, our proposed method enables more effective allocation, better preserving useful attention information. Second, with a large budget, such allocation becomes less critical, as it is sufficient to cover the necessary information.

2. To further illustrate this phenomenon, we have included an ablation study titled "Attention Recall Rate Experiment" in Figure 8 from Appendix M of the revised version. The results show that with a small budget (i.e., 64 KV cache size), PyramidKV improves the attention recall rate (the percentage of attention computed using the keys retrieved by the method and the query, relative to the attention computed using all keys and the query.). However, with a larger budget (i.e., 2k KV cache size), the improvement decreases. For 64, 128, 256, 512, 1024, and 2048 KV cache sizes, PyramidKV's average attention recall rate improvements are 1.87%, 0.64%, 0.61%, 0.56%, 0.47%, and 0.36%.

W2: In [1], Wu et al. claims that "retrieval heads" exist across models, functioning similarly to the submission's patterns ("massive attention") seen in higher layers—such as layer 30 in Figure 2—to retrieve essential contextual information. Given this, I wonder if retrieval heads are primarily found in the higher layers or if they might also be present in lower layers but are obscured due to the study's averaging of attention scores across heads within each layer. This averaging might be masking the presence of "massive attention" in the lower layers, leading to more-than-enough allocation of KV cache for some heads in the lower layers. Could the authors conduct additional experiments to address my concern?

A2:

1. Retrieval heads are predominantly located in the higher layers, as illustrated in Figure 5 of [1]. Notably, no retrieval heads are observed in layers 0–8 of the LLaMA models.

2. To further investigate, we conducted additional experiments on the lower layers to analyze the attention patterns of the heads, as detailed in Appendix P. Our findings indicate the absence of "massive attention" in any individual head.

W3: There are many variations of NIAH tasks, e.g. haystack formed from repetitive sentences or haystack formed from a long corpus. Can the authors elaborate which setting used in the study?

A3:

We adopt the haystack setting of haystack formed from a long corpus for the NIAH task. This follows the exact setup of [1]. We have included the details in Section 5.3.1.

[1] Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. arXiv preprint arXiv:2404.15574, 2024.

We thank the reviewer for constructive advices, and we would like to clarify a few points.

W1: The observation that the attention scores are more uniform in the first layers but become more skewed in the last layers is NOT new, see [1][2].

[1] InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management [2] MagicPIG: LSH Sampling for Efficient LLM Generation

R1:

1. We respectfully refer the reviewer to the ICLR review guideline (https://iclr.cc/Conferences/2025/ReviewerGuide) regarding comparisons to papers published within four months. Papers [1] and [2] fall within this timeframe and are considered contemporaneous.

2. However, we would like to emphasize the key distinctions between our findings and theirs. While [1] and [2] observe a general trend where attention scores are uniform in the initial layers and become increasingly skewed in the later layers, our analysis uncovers a more intricate evolution of attention patterns across the model's layers. Specifically, our findings reveal:

• Lower Layers: The model operates in a broad-spectrum mode, uniformly distributing attention across the input to capture diverse information.

• Middle Layers: Attention narrows its focus on individual components (See Figure 2. Layers 6 and 12), reflecting a process of refined information aggregation.

• Upper Layers: Attention transitions into a "massive attention" mechanism, heavily concentrating on a few critical tokens to extract key information efficiently for answer generation.

This nuanced perspective provides a clearer and more comprehensive understanding of how attention dynamics evolve throughout the model's layers, highlighting the unique insights offered by our work.

W2: The choice of sampling rate schedule, i.e., how many tokens to sample for each layer, is not discussed clearly? Why use an arithmetic sequence? What are the observations driving this choice? Will also schedule also work?

R2:

1. The choice of sampling rate schedule: For the sampling schedule, the number of tokens allocated to each layer is determined based on the hyperparameter $\beta$. Once the cache budget and $\beta$ are set, the token count for each layer can be calculated using Eq. (1).

2. Why an Arithmetic Sequence? Observations Driving this Choice: Our decision to use an arithmetic sequence is driven by three key factors:

• Alignment with Pyramidal Information Funneling Pattern: Empirical observations reveal a pyramidal information funneling pattern, where lower layers exhibit dispersed attention while higher layers concentrate on fewer tokens. Inspired by this, we adopt the arithmetic sequence design to align with this natural progression.

• Superior Empirical Performance: Through extensive experimentation across diverse datasets, we compared various methods, including the arithmetic sequence and adaptive approaches. Results consistently showed that the arithmetic sequence method outperformed others.

• Computational Efficiency: The arithmetic sequence method introduces minimal computational overhead compared to adaptive approaches, which require dynamically computing cache budgets across layers.

W3: The empirical results are not impressive. As shown in Table 1, PyramidKV does not outperform the baselines in many cases.

R3:

1. Among the 16 datasets, the tasks where our proposed method performs slightly worse than the baseline (i.e., HotpotQA, Musique, SAMSum, and PCount under the LLaMA-3-8B-Instruct, KV Size = 64 setting, as shown in Table 1) are actually saturated. In these cases, our method is only marginally inferior to the baseline and remains highly competitive.

2. Conversely, on tasks with more room for improvement (e.g., Qasper, MF-en, TREC, and TriviaQA under the same LLaMA-3-8B-Instruct, KV Size = 64 setting), our method significantly outperforms the baseline. Consequently, the average performance of our method is substantially better than the baselines.

3. Notably, these tasks include several In-Context Learning (ICL) tasks (i.e., TREC), our method enjoys the best performance gain at ICL tasks, where the method demonstrates the ability to leverage provided examples to adapt effectively. The importance of ICL tasks lies in their widespread use in real-world applications, where dynamic adaptation to new inputs is critical.

Q2: The explantion of using the arithmetic sequence does not provide real content, why an arithmetic sequence? Moreover, why do " adaptive approaches" acutually mean in the author response? How do they work? Regarding the computation efficency, any algebra computation with reasonable computation will be much faster than the actual attention computation, which will not make attention weight scheduling the bottleneck. This is NOT the reason to choose the arithmetic sequence.

A2:

**Why an arithmetic sequence? **

• Pattern Matching: Based on our analysis of the attention patterns, we observe that a relatively stable, linearly decreasing trend aligns closely with the underlying structure of the patterns.

• Empirical Results: We compared the arithmetic sequence strategy against logarithmic and exponential sequence approaches for pyramid KV cache budget allocation, using a cache size of 64 as below. The empirical results consistently showed that the arithmetic sequence strategy outperformed its counterparts, establishing it as the most effective approach for our use case.

We have included this ablation study in Appendix H.1.

why do " adaptive approaches" acutually mean in the author response? How do they work?

We propose two adaptive allocation baselines, which are based on the entropy and Gini coefficient of the attention values at each layer. The weight of each layer is calculated based on its corresponding metric (i.e., entropy and Gini coefficient), and the budget is allocated accordingly. Specifically:

• Entropy-based allocation: Layers with higher entropy receive fewer weights. Each layer's entropy is calculated based on the layer's attention.

• Gini coefficient-based allocation: Layers with higher Gini coefficients receive fewer weights. Each layer's Gini coefficient is calculated based on the layer's attention.

We compared the arithmetic sequence-based allocation against entropy-based allocation and Gini coefficient-based allocation, using a cache size of 64 as below. The empirical results consistently showed that the arithmetic sequence-based allocation outperformed adaptive methods, establishing it as the most effective approach for our use case.

The results show that all of these methods are far less effective than PyramidKV, which uses an arithmetic sequence-based allocation approach. We have included this ablation study in Appendix H.1.

computation efficency

Although algebraic computations with reasonable complexity are faster than attention computations, attention itself remains an unavoidable component of the transformer forward pass. Therefore, the critical issue lies in the additional computation introduced by the proposed method rather than the attention computation, which is inherent to the model. We present a detailed computation time analysis for the proposed method compared with adaptive methods in a llama-3-8b model as below. Allocation time means the time needed to compute for budget allocation.

PyramidKV as an arithmetic sequence demonstrates significant efficiency advantages at computation overhead over adaptive methods.

Based on the above analysis, PyramidKV demonstrates both effectiveness and efficiency advantages over adaptive methods.

We sincerely thank the reviewer once again for their valuable feedback, which has greatly contributed to improving the clarity and rigor of our work. We have carefully addressed each of the raised concerns and hope that our clarifications effectively resolve them. We kindly ask you to consider re-evaluating our score based on our responses. If there are any remaining concerns, we would deeply appreciate the opportunity to engage further during the discussion period. Your insights are invaluable to us, and we really appreciate the opportunity to improve our work.

We thank the reviewer for the insightful questions. And we would like to clarify a few points.

Response to Weakness

W1: I still think the solution is overly simple, the experiment results are not good, and the presentation is not carefully polished

A1:

Response to the solution is overly simple:

Our work focuses on developing simple, efficient, and effective methods that are easily applicable to a variety of models and adaptable to diverse tasks. The simplicity and effectiveness of our approach represent its unique strength, as it can be seamlessly integrated with pre-filling stage acceleration methods (e.g., MInference, as noted by Reviewer UuVr ) and serving frameworks (e.g., vLLM, as highlighted by Reviewer MCVv ). In contrast, adaptive methods that require additional computation may lack this versatility and ease of integration.

Response to the experiment results are not good:

We claim that our method demonstrates generally superior performance compared to the baselines, particularly in extreme scenarios. This strong performance has been highlighted as a strength of our work by reviewers MCVv , A1NK , UuVr , and pUzw. However, we understand the reviewer's concern that our method may underperform relative to baselines on some datasets. To address this, we clarify the following points:

• Among the 16 datasets, the few instances where our method slightly underperforms the baselines typically occur in saturated tasks (e.g., HotpotQA, Musique, etc., under the LLaMA-3-8B-Instruct setting with KV Size = 64, as shown in Table 1). In these cases, the margin of difference is minimal, and our method remains competitive.

• Conversely, on tasks with greater potential for improvement (e.g., Qasper, MF-en, TREC, TriviaQA, etc., under the same setting), our method achieves significantly better results than the baselines.

• As a result, the overall average performance of our method significantly exceeds that of the baselines, especially in extreme scenarios.

Response to the presentation is not carefully polished:

Regarding the reviewer’s feedback about the presentation not being polished, we have carefully revised the paper to address the concerns that the reviewer has about the presentation of our paper. And we would like to sincerely ask the reviewer to further point out the specific issues that are unclear. We are eager to make corresponding improvements. We value the reviewer’s feedback and appreciate the opportunity to improve our work.

Response to Questions

Q1: In the reponse of R1, why the observation is " clearer and more comprehensive" than existing works? The three points basically say the same thing as existing work, i.e., the attention is uniform in the initial layers but more skewed in latter layers.

A1:

We believe that [1] and [2] are both high-quality works on this topic. Despite that [1] and [2] are considered contemporaneous with our work, we humbly assert that our work remains novel and more comprehensive than other studies. Our analysis uniquely examines attention metrics across all transformer layers, from 0 to 30, leading to the discovery of a key phenomenon we term Pyramidal Information Funneling.

[1] conducted a limited investigation into attention patterns, focusing only on the lower layer (layer 0) and a single upper layer (layer 18). While [1] noted that attention becomes more skewed in upper layers, it did not provide a fine-grained observation of attention patterns across all layers. In contrast, our study reveals several novel findings:

• Localized Attention: We observe that attention progressively narrows its focus, targeting specific components within the input sequence.

• Massive Attention Mechanism: In the upper layers, attention heavily concentrates on a small set of critical tokens. Notably, these tokens are not limited to the leading positions, as observed in [1], but also appear at regular intervals across the sequence. The discrepancy arises from differences in input settings, with [1] identifying massive attention only at the initial tokens.

These insights motivated us to propose a token-selection method based on the highest attention scores in the upper layers, rather than solely relying on tokens from earlier positions.

To the best of our knowledge, [2] has not analyzed attention patterns across transformer layers.

Therefore, although [1] and [2] are considered contemporaneous with our work, making a comparison unnecessary, the perspective of our observation is considered novel compared with [1] and [2]. Moreover, although [1] also observed attention patterns, the method we proposed based on our observations is significantly different from [1], further highlighting the novelty of our work.

In summary, our primary objective is to uncover the flow of information across transformer layers and identify effective compression strategies. By analyzing attention patterns at all levels—lower, middle, and upper—we elucidate localized attention mechanisms and the Pyramidal Information Funneling phenomenon. Our observations partially explain how large language models process information from different components. Additionally, we identify that massive activations in the upper layers occur at diverse positions, driving us to propose the method for selecting tokens with the highest attention scores even in the upper layers.

We have included the above discussion and these two contemporaneous works as citations in Appendix C.

[1] InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

[2] MagicPIG: LSH Sampling for Efficient LLM Generation