Thank you for your valuable feedback. We address the questions and confusion below.

Key assumptions — attention sink.

The attention sink phenomenon is well-studied in attention sparsity research. For example, StreamingLLM [1] and DuoAttention [3] discover that only a few useful tokens receive some attention score, while the remaining useless attention score (summing to 1.0) is absorbed by the first few tokens (called sink tokens). If the sink tokens are numerous, they overshadow meaningful ones. In addition, Minference [4] describes this phenomenon as a ^-shape pattern, while more recently LightTransformer identifies lazy layers dominated by attention sink heads.

Besides, a key clarification is that too many sink tokens are actually bad rather than representing important information for each chunk.

Relevant related work is provided below to clarify the phenomenon of attention sink further. We hope this addresses your concern.

[1] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.

[2] J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han, “QUEST: query-aware sparsity for efficient long-context LLM inference,” in Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.

[3] G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han, “Duoattention: Efficient long-context LLM inference with retrieval and streaming heads,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net, 2025.

[4] H. Jiang, Y. Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. Abdi, D. Li, C. Lin, Y. Yang, and L. Qiu, “Minference 1.0: Accelerating pre- filling for long-context llms via dynamic sparse attention,” in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, Eds., 2024.

[5] R. Chen, Z. Wang, B. Cao, T. Wu, S. Zheng, X. Li, X. Wei, S. Yan, M. Li, and Y. Liang, “Arkvale: Efficient generative LLM inference with recallable key-value eviction,” in Advances in Neural Information Pro- cessing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, Eds., 2024.

Positional embedding (PE)

PICI prefills a doc, whose first token is given position 0, to generate their KV cache. Then PICI uses the cache directly without modifying PE. On the contrary, new queries receive position IDs based on their real positions.

Again, too many tokens with PE 0 at the beginning of each chunk absorb excess attention without contributing useful information.

We will add this detail in future versions.

Sliding window

We disabled the optional sliding window for the Mistral model.

Large chunk size

Currently, we chunk large documents into mid-sized segments (e.g., 512 tokens). We will add an analysis of the trade-off between chunking and processing long documents as a whole in future versions.

Source of improvement: from better token selection, or static token selection?

LegoLink’s improvement comes from static token selection, specifically, the first few tokens of each chunk (except for the first chunk). These tokens are preselected for recomputation, offline before execution.

The recomputation gives these "first few tokens" actual position IDs, makes them realize they are not the "first few" tokens anymore (they are in the middle now), and stops them from being sink tokens and absorbing too much attention.

Thank you for your valuable feedback. We address the review questions below.

Missing related work.

Thank you for pointing this out. We will include a subsection discussing RAG applications and their connection to context caching. This addition will clarify PICI's positioning without modifying its intended scope.

AGI.

We acknowledge that "AGI" is controversial and requires a precise definition when discussing it, especially in academic contexts. We will revise the first paragraph.

Needle in a Haystack

Initially, we evaluated PICI on the same four datasets as CacheBlend (WikimQA, SAMSum, MultiNews, and Musique). To further demonstrate LegoLink’s generalizability, we selected the Needle in a Haystack dataset based on its popularity rather than ease of evaluation or improved results. We acknowledge its limitations as a synthetic task where the "needle" is highly distinct from the context. In the revised version, we will clarify our rationale for this dataset selection (popularity instead of methodological suitability) and include passage retrieval results from LongBench, where we expect similar findings.

Storyline and contribution.

The revised story (or the original story we plan to tell) should go as follows: 1) We are the first to formally define the PIC problem and decompose its solution into a four-step framework: document chunking, cache generation, retrieval, and linking. This structured approach establishes a diverse solution space (contribution 1). 2) CacheBlend, the first PIC work, fits in our framework and mainly designs the linking step, while LegoLink enhances its efficiency (contribution 2). 3) LegoLink-0 further improves performance by mainly designing the cache generation step, thereby simplifying the linking step (contribution 3).

This framework establishes a diverse solution space for future research. The key consideration is the distribution of computational costs across different steps. Allocating cost to the chunking or cache generation step resembles a compile-time expense—incurred once but benefiting all subsequent queries. Conversely, placing cost in the retrieval or linking step resembles link-time expense—introducing runtime overhead but allowing for more adaptive and potentially more accurate solutions. In the appendix, we call on future research to find more efficient and accurate solutions utilizing our framework.

The above details are currently in the appendix and will be integrated into the main text.

Text clarity.

We appreciate the feedback and will improve clarity on terms like "token," O-notation, "base" model, and figure positions.

Still strong attention on chunk boundaries (Figure 3)

1. Experiments show that the first token retains relatively strong attention absorption due to being a begin-of-sentence token ("<s>" in LLaMA). This insight motivates LegoLink-0, where we discard "<s>" tokens (and dummy tokens which are also "<s>" tokens), except for the first chunk.
2. In the softmax map, we could still observe that recomputation reduces boundary attention, but attention-score differences on mid-content are harder to observe. We will add a softmax map for cross-comparison.

Thank you for your valuable feedback. We address the review questions below.

More details (overall workflow, cache hit def, indexing, cache swap in/out, too-large cache) of PICI system in the main text instead of the appendix.

Due to page limits, we prioritized the attention sink effect in PIC and our simple yet effective algorithm, placing system details in the appendix. However, we acknowledge this omission and will move system details back to the main text. Below is a summary (from the appendix) related to review questions.

PICI workflow: 1) Users cache frequently used documents via generate_cache API, receiving a cache_id. 2) Users pass queries and cache_ids to PICI in any order (position-independent), and PICI retrieves the cache with minimal overhead (simple cache_id → cache address mapping) and finishes linking as described in the paper.

PICI design philosophy (similar to Gemini): users (or a RAG system) explicitly control caching. 1) Users decide what doc (big or small, frequently used or not) is worth caching and for how long at known costs (xx $/token/hour). PICI never actively deletes users’ PIC cache (more in the next paragraph). 2) Users explicitly pass cache_id to define cache hits. 3) PIC cache resides in GPU memory (no swap in/out).

In contrast, systems like vLLM and SGLang use implicit caching, where cache hits rely on token prefix-matching and swap in/out for memory management. Future work will explore integrating explicit PIC caching with implicit prefix-based caching and optimizing their placement in the memory hierarchy.

Rough-L score --> Rouge-L

Yes, thank you for pointing this out. We will fix it.

I assume that position encodings (PE) for the model are re-used by the chunks of the encoded prefix documents such that the whole process is permutation invariant?

Yes. PICI prefills a doc, whose first token is given position 0, to generate their KV cache. Then PICI uses the cache directly without modifying PE. On the contrary, new queries receive position IDs based on their real positions.

Thank you for your valuable feedback. We address the review questions below.

More details (overall workflow, cache hit def, too-large cache) of PICI system in the main text instead of the appendix.

Due to page limits, we prioritized the attention sink effect in PIC and our simple yet effective algorithm, placing system details in the appendix. However, we acknowledge this omission and will move system details back to the main text. Below is a summary (from the appendix) related to review questions.

PICI workflow: 1) Users cache frequently used documents via generate_cache API, receiving a cache_id. 2) Users pass queries and cache_ids to PICI in any order (position-independent), and PICI retrieves the cache with minimal overhead (simple cache_id → cache address mapping) and finishes linking as described in the paper.

PICI design philosophy (similar to Gemini): users (or a RAG system) explicitly control caching. 1) Users decide what doc (big or small, frequently used or not) is worth caching and for how long at known costs (xx $/token/hour). For now, PICI never actively deletes users’ PIC cache (more in the next paragraph). 2) Users explicitly pass cache_id to define cache hits. 3) For now, the PIC cache resides in GPU memory (no swap in/out).

In contrast, systems like vLLM and SGLang use implicit caching, where cache hits rely on token prefix-matching and swap in/out mechanisms are used for memory management. Future work will explore integrating explicit PIC caching with implicit prefix-based caching and optimizing their placement in the memory hierarchy.

Pre-computing and saving the KV caches for long documents requires larger storage than string.

Yes, it is always a hard trade-off between memory efficiency and computational savings. However, in the PICI design philosophy, users retain the flexibility to enable or disable caching based on their specific needs. Given their deeper understanding of application semantics and intent, users are often better positioned than the system itself to make optimal PIC caching decisions. This flexibility helps prevent unnecessary memory consumption by avoiding the storage of large, unused caches.

Missing citations in Section 2.2.

We will correct the citation problems in Section 2.2, making them as complete as in the Related Work section.

Line 220: O=AVW_O, should be O=AV_{exp}W_O?

Yes.