KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems

Thank the reviewer for the time to review and provide the positive, constructive and valuable feedback. Below, we present our point-by-point responses to each comment.

Q1: Overall Process in Figure 3

Figure 3 is designed to illustrate the overall workflow of KVComm. We summarize the process as follows:

0. Initialization: Before any user requests, all agents precompute and store the KV-caches for all prefix segments defined in their prompt templates.
1. Placeholder Readiness: When a request arrives, each agent checks whether all placeholders’ base KV-caches are available. If not, these are precomputed in parallel. Newly generated placeholder KV-caches then enter the anchor prediction module to check for similar samples in the anchor pool and enable reuse.
2. Reuse or Fallback: Once all placeholder KV-caches are ready, the agent determines if there are reusable KV-cache deviations for each placeholder. If not, standard dense prefilling is used. For placeholders without reusable deviations, the agent computes the difference between their actual and base KV-caches, and stores this deviation in the anchor pool to expand anchor coverage.
3. Offset Approximation: If all placeholders have reusable KV-cache deviations, the agent fetches the matched anchors, estimates the KV-cache deviations using Eq. (6) and (7), and updates the placeholder and prefix KV-caches in parallel.
4. Decoding: The agent then combines the updated placeholder and prefix KV-caches to initiate response decoding.
5. Anchor Update: After generating a response, its KV-cache passes through the anchor prediction module. If a similar anchor exists, the cache is stored in shared memory for future retrieval. The agent then awaits the next request.
6. Fallback Storage: If no similar anchor exists, the response KV-cache is stored in the anchor pool for dependent agents to fill in deviations under their respective contexts, then the agent awaits the next request.

In conclude, all agent interactions occur via the KV-cache Communication Module. When multiple agents share the same user text but have different agent-specific prefixes, we avoid re-running prefilling by treating the KV-cache of the shared text under a new prefix as a context-dependent offset from its base KV-cache. We estimate that offset by interpolating from a small set of anchor examples, align Key positions (RoPE de-rotation / re-rotation), add the estimated Key/Value offsets, then concatenate the adjusted segments and decode.

We will clarify and streamline Figure 3 in the revised manuscript to further improve its readability. If further clarification is needed, we are happy to provide it.

Q2: Explanation of New Technical Terms

We will add a glossary in the revised manuscript for readability and open-source our code for reproducibility. Below are the newly introduced terms in KVComm:

1. Base KV-cache: KV-cache for a prefix/placeholder under its initial context or without external input; serves as the reference for offsets.

2. KV-cache offset/deviation: The difference between a shared text’s KV-cache under a new prefix and its base KV-cache. (Keys require RoPE-based alignment before offsetting.)

3. Placeholder/Prefix offset: Offsets for the placeholder segment (external input) and its adjacent prefix segment (predefined context), respectively, relative to their base KV-caches.

4. Offset variance problem: KV-cache offsets for the same text can vary substantially across contexts, so static reuse is unreliable.

5. Positional alignment/Key de-rotation: RoPE de-rotation/re-rotation to align keys before offsetting.

6. Shareability: Whether a request can skip prefill under anchor criteria; if not, dense prefilling is used.

7. Shared memory/KV-cache: The shared storage that holds base KV-caches across agents.

8. Dense prefilling/generation: Full computation path (prefill + decode) used if reuse is not possible.

9. Anchor (pool): A small set of representative examples, each storing placeholder and prefix offsets, used to interpolate offsets for new contexts.

Q3: Deeper Discussion about the Placeholder and Prefix Offsets

The exact difference between placeholder and prefix offset can be described from the following three aspects:

1. Definition:

   • Placeholder offset is defined as the KV-cache offset for externally injected text, including users, agents, and tools, etc, whose base KV-cache is not preset;
   • Prefix offset is defined as the KV-cache offset for predefined prompt text, including system prompt, placeholder conjunctions, and suffix.

2. Dependence:

   • Placeholder offset arises mainly from changes to the prefix context;
   • Prefix offset is triggered by changes to the preceding placeholder (i.e., injected content).

3. Variance:

   • Placeholder offset show high variance across samples and contexts;
   • Prefix offset are more stable, as their base KV-caches are computed under a fixed system prompt and are less influenced by preceding placeholders.

Due to the rebuttal format restriction, we are currently unable to share visualization results of placeholder and prefix offset distribution in the response. We will add this interesting characteristics into the revised manuscript.

Concrete Example (from Figure 3)

Suppose the prompt template of an agent is defined as follows:

"You are a math expert. You will be given a math problem .... Q: {user_question}. Answers from other agents are: \n Agent 2 as a mathematical analyst: {agent_2_current}, the execution result is: {condition_2_current}.<|assistant|>"

In this case,

• “You are a math expert…” is a prefix segment (fixed context).
• “{user_question}”, “{agent_2_current}”, {condition_2_current}” are placeholders (external content).
• Text following each placeholder (e.g., “Answers from…”) is a prefix segment adjacent to a placeholder.

From this example, we can find that during the precomputation of prefix KV-caches, the subsequent prefix segments are mainly correlated to the first prefix segment, typically containing system prompts, thus their base KV-caches are relatively stable. When the preceding placeholder is injected with specific text, their actual KV-cache will contain the new information. However, the base KV-cache of prefix segments, unaware of this, must be adjusted via offset interpolation using similar anchor cases. Otherwise, the model may receive conflicting signals: the base KV-cache of prefix segments would mislead the model decode by showing no actual placeholder text injected, while the injected placeholder text actually participate in the information fusion. Therefore, this conflict necessitates careful offset estimation for both types. Experimental demonstration of the necessity has been provided in Table 5 of the main text.

Based on the above analysis, hopefully, it would be clearer about the difference between the two offset types and the necessity of them. If further clarification is needed, we are happy to provide it.

Q4: Relationship between the number of anchors, the GPU memory usage, and TTFT speedup

We provide detailed experimental results in Tables 1 and 2, and analyze the relationship as follows:

Memory Cost vs. Anchor Number:

As shown in Table 1, GPU memory usage increases approximately linearly with the number of anchors, for fixed input/output lengths and agent count. This trend holds across all sequence lengths: each additional anchor increases storage for associated KV deviations, leading to higher overall memory cost.

TTFT Speedup vs. Anchor Number:

However, in Table 2, the TTFT speedup does not increase monotonically with more anchors. Specifically, when increasing anchor number from 5 to 10 or 15, the TTFT speedup remains nearly unchanged or even slightly decreases across most settings. This indicates that anchor pool enlargement yields diminishing returns, and excessive anchors may even cause minor efficiency loss due to increased system overhead (e.g., lookup, memory access, or less optimal anchor matching).

Table 1. Memory Cost (GB) under the three-agent setting with different maximun anchor numbers.

Table 2. Mean TTFT speedup under the three-agent setting with different maximun anchor numbers.

Thank the reviewer for the time to review and provide the valuable and constructive feedback. We address each of the comments in detail below.

Q1: Evaluations on Harder Benchmarks

For Math500, we report results using both Llama3.1-8B-instruct (Llama) and Deepseek-R1-Distill-Qwen-7B (DS-QWen). As shown in Tables 1 and 2, KVComm achieves competitive or superior accuracy compared to the baseline method. Notably, even as reuse rate drops with increasing agent number (due to higher context diversity), KVComm consistently matches or exceeds the baseline accuracy, suggesting that referencing anchor information helps improve answer quality. An especially encouraging observation is that for DS-QWen, KVComm attains both a higher reuse rate and superior accuracy on challenging reasoning tasks.

Table 1. Math500 Results – Llama.

Table 2. Math500 Results – DS-QWen.

For AIME, Table 3 reports the results using DS-QWen. KVComm maintains a high reuse rate (70%+) while retaining comparable accuracy to the baseline. The accuracy decrease largely attributes to token-length constraints during decode: as KVComm’s memory cost rises, the available context for generation per agent is reduced. We are actively addressing this with pipelining and efficient offloading techniques to support longer contexts in future work.

Table 3. Performance on AIME using DS-QWen models.

Q2: Comparison with More Methods

Below we report experimental results for DroidSpeak alongside KVComm. DroidSpeak is designed for scenarios where the base and finetuned models process the same entire prompt—thus, context change is not addressed.

Table 4 demonstrates that applying DroidSpeak directly in our experimental setup significantly degrades performance, validating the necessity of context-aware cache reuse. KVComm consistently achieves higher or comparable accuracy across all datasets.

Table 4. Performance comparison with DroidSpeak.

Q3: End-to-End (E2E) Latency Report

We would like to clarify that in the KV-cache reuse literature (e.g., PromptCache, CacheBlend, DroidSpeak, KVLink), most works do not specifically target E2E latency optimization because E2E latency encompasses both prefill and decoding, whereas KV-cache reuse specifically accelerates prefill (TTFT). However, it has been shown (e.g., PromptCache) that in modern serving frameworks supporting prefill/decoding disaggregation (e.g., DistServe, Dynamo), reducing TTFT can significantly improve system throughput and user-perceived latency.

Nevertheless, for completeness, we provide E2E latency for various agent settings (Table 5). While the E2E latency improvement is modest (up to $\sim$ 1.08x in long-prefill/short-decode cases), this is expected since KVComm does not target the decode stage. Still, any E2E improvement directly reflects the benefit of reduced prefill.

Table 5. Latency Breakdown and Speedup vs. Dense Prefill.

Q4: Memory Overhead Report Induced by KVComm

Below we provide the memory footprint for different agentic settings, including varying configurations of input/output length, and anchor numbers. As shown in Table 6, memory usage scales with both sequence and anchor number. Note that it is observed that the deviation tensors stored for each anchor are highly sparse (averaging ~50% elements with absolute value < 1e-1), suggesting potential for lossless compression of anchors and further memory optimization.

Table 6. Memory Cost (GB) under the three-agent setting.

Q5: Finegrained Breakdown for Large Anchor Scenarios

Table 2 in the main text has provided latency breakdowns and speedup, where the anchor loading overhead is counted at the 'Others' part in Table 2. For the large anchor scenario, we take the MMLU benchmark as an example with 4K placeholder tokens. Table 7 shows that anchor offloading can become a major latency bottleneck (up to 1.3 seconds) and that cumulative CPU memory can reach 75GB. While this is significant, it can be mitigated by engineering improvements such as asynchronous scheduling and anchor compression, which are the focus of our ongoing system development.

Table 7. 4K-token Anchor Matching using Llama on MMLU

Q6: Safety Considerations in Cache Sharing

We have reviewed our submission and confirm that safety and privacy of KV-cache sharing are not discussed in this version. While we agree that safety and privacy are of critical importance in practical multi-agent systems, a comprehensive treatment is outside the current paper’s scope. We will highlight this as a promising and important direction for future work.

Q7: Baseline Peroformance on MMLU

We have double-checked our experiments and confirm the correctness of the reported accuracy for the baseline on MMLU. The lower baseline performance under the two-agent setting arises from our agent design: the first agent is a knowledgeable expert who generates relevant keywords, and the second "Final Refer" is tasked to analyze the previous agent’s output and produce a final answer. Some failures are due to the second agent providing only an answer without analysis. All prompt templates are available in Appendix 6.3.2.

Thank the reviewer for taking the time to review our work and providing positive, constructive and valuable feedback. Below, we present our point-by-point responses to each comment.

Q1: Application in Heterogenous Multi-agent Systems

As for the heterogenous systems, we can categorize them into two types for detailed discussion.

For agents consisting of

• Identical Architecuture but Different Model Weights: DroidSpeak has discovered the KV-cache reuse potential for this kind of KV-cache communication, where the base model and domain-specific finetuned model completely share the same prompt. Building on this, we believe that if the clustering characteristics of tokens in the finetuned model remain sufficiently similar to the base model, KVComm could facilitate KV-cache sharing for these cases. However, this may require further technical innovation and thus remains a promising direction for future research.
• Distinct Architectures: In this case, it is impossible to employ KVComm as the reviewer noted.

In summary, KVComm is directly applicable for groups of homogeneous agents and holds promise for agents with identical architectures but different weights, pending further exploration.

Q2: Softmax Overhead in Long-context Anchor matching

In KVComm, the softmax is operated along the anchor number dimension on the negative $\ell_2$-norm distance between the placeholder sample and each anchor, which reduces the Key/Value tensor into the shape of [$m$, 1, 1, seq_len, 1] ($m$: anchor count, seq_len: sequence length). The latency of softmax thus scales with both parameters. And we quantify its latency overhead in Table 1. Here the shape of each KV-cache tensor is [32, 8, seq_len, 128]. We can observe that latency remains reasonable ($\sim$ 18 ms with 25 anchors and 4096 tokens per anchor).

Table 1. Simulated softmax latency (ms) with different anchor counts and sequence length.

These results are from a simulation without competing system workloads. In real multi-agent long-context scenarios, the latency also includes offloading KV-caches to CPU. For example, as shown in Table 2 (using Llama3.1-8B-instruct on the MMLU benchmark), the average softmax latency is 100+ ms when all anchors are offloaded to CPU, while total offloading per agent can reach 1260+ ms for 4K-token contexts. This indicates that the main overhead arises from data movement, not from the softmax operation itself. Such communication overhead is a well-known issue in long-context inference, and can be mitigated with systematic optimization (e.g., pipelining), which is orthogonal to the KVComm mechanism. To achieve perfect acceleration of long-context KV-cache reuse, we are currently committed to integrate KVComm into the modern LLM serving framework such as vLLM and SGLang, which have well optimized these systematic issues yet still lack support for fine-grained, per-layer KV-cache injection across devices.

Table 2. Latency and Memory Cost of 4K-tokens Anchor Matching using Llama3.1-8B-instruct on the MMLU benchmark.

Q3: Experimental Comparisons with DroidSpeak

The reason we did not originally compare directly to DroidSpeak is due to fundamental differences in application scenario:

• DroidSpeak exploits cross-model KV-cache reuse when both base and finetuned models process an identical prompt, so context changes are not involved.
• KVComm targets at the potential of cross-model KV-cache reuse when only a shared segment is reused under different prefix contexts—a more general and challenging setting for multi-agent systems.

Therefore, although both designed based on KV-cache reuse, the two methods aim at solving two different problems in multi-agent systems.

To clarify, we implemented DroidSpeak in our framework and empirically compared it against KVComm, holding DroidSpeak’s reuse rate at 80% for relative fairness. It can be observed in Table 3 that DroidSpeak’s performance drops significantly in our setting, confirming that context awareness is crucial for robust KV-cache reuse.

Table 3. Performance comparison with DroidSpeak on various benchmarks using Llama3.1-8B-instruct.

Q4: Performance on non-RoPE Models

As discussed in Proposition 1, KVComm’s effectiveness relies on the regular positional structure provided by RoPE-based encodings. To validate this, we tested KVComm on OPT-6.7B (absolute position encoding). As shown in Table 4, the reuse rate drops to near zero.

Table 4. Performance of OPT-6.7B on the MMLU benchmark.

This is because absolute position encodings do not preserve local positional relationships when the prefix changes, undermining cache similarity. We will clarify in our revised manuscript that KVComm is designed specifically for RoPE-based models, and leave adaptation to other positional encodings as important future work.

Q5: Discussion about KVComm in Dynamic Agentic Framework

Thank the reviewer for highlighting the potential extension of KVComm. While our current implementation assumes predefined placeholders and templates to facilitate alignment and offset estimation, the core idea behind KVComm—context-aware offset estimation based on embedding similarity and anchor pools—is not inherently limited to static scenarios.

In principle, it can be applied wherever context change patterns recur, provided shared text segmentation is feasible (as with techniques similar to automatic prefix caching in vLLM). Although our current work does not cover fully dynamic, unstructured cases, we recognize this as a meaningful direction and plan to extend KVComm for more dynamic agentic and debate-style benchmarks in future work.

Q6: Reuse Rate Performance under Long-context Scenario

We have provided the experimental results in Table 2 under the 4K context setting. It can be observed that when the cumulative positional deviations amplify, the overall reuse rate drops by 20%, from $\sim$ 70% to 50%. This aligns with expectations and the trend is similar to that of increasing placeholder number in the template, where the cumulative positional deviations will degrade the embedding similarity.

Thank the reviewer for taking the time to review and providing the constructive and valuable feedback. Below, we present our point-by-point responses to each comment.

Q1: More Clarification of KVComm's Contribution

KVComm’s key contributions are threefold:

• Problem Formulation (Novel Motivation). Prior KV-cache reuse methods (e.g., CacheBlend) target single-agent settings such as RAG, where multiple retrieved chunks are concatenated into one prompt and KV-caches are precomputed without external context. In multi-agent systems, however, each agent’s response is repeatedly consumed by successor agents. The resulting KV-cache unavoidably encodes its original prefix context due to autoregressive attention. As evidenced in our work, a naive adoption of previous methods in this case can fail in filtering the previous contextual information in the KV-cache, resulting in performance degradation. We identify and address this multi-context redundancy as a distinct challenge for efficient prefilling in the multi-agent scenario, which to our knowledge, has not been covered by prior work.

• Empirical Insight (Context-aware KV-cache Offset Proximity). We observe that for two similar tokens whose KV-caches are computed under the same prefix, when the prefix changes, their KV-cache deviations remain similar. Unlike RAG-focused findings that KV-caches are largely prefix-robust, our measurements show this robustness breaks down in general multi-context (multi-agent) scenarios (see Figure 1 and Table 1 in the main text).

• New Reuse Paradigm (Prompt-Adaptive). Instead of token-/layer-level selective recomputation (e.g., DroidSpeak, CacheBlend), KVComm performs prompt-adaptive selection at the segment level: a shared text is either fully reused or fully recomputed, based on its embedding similarity to stored anchors. We further design an efficient KV-cache storage/retrieval system that supports fast anchor lookup and long-context multi-agent pipelines.

Therefore, based on the above, KVComm is, to our knowledge, the first training-free, prompt-adaptive cache-sharing framework for efficient prefilling in multi-agent systems.

Q2: Priors Leveraged by KVComm

The priors around prompts we use are consistent to the baseline multi-agent framework design. That is, KVComm follows the baseline multi-agent dependency graph to define each agent’s placeholder set and prefix segment set. IDs (for placeholders/prefix segments) are derived directly from the dependency graph. Details are in Appendix 6.2.2 and 6.3.2 of the supplementary material.

Q3: Definition of Context-Aware Offset

In Figure 2, the context-aware offset is a KV-cache deviation induced by the (changed) prefix context. When the prefix changes, the required offset to align a cached segment’s KV-cache with the new context changes accordingly, hence "context-aware".

Q4: Domains / Assumptions for KVComm

KV-cache reuse requires identical model architecture (tokenizer, computation graph, hidden size, #heads, #layers, etc.); otherwise KV-cache shapes/token spaces mismatch and reuse becomes infeasible without adaptation. KVComm also assumes RoPE position encoding, a common choice in modern LLMs, enabling explicit per-layer positional alignment across varying context lengths.

Q5: Explanation of Performance Gap between KVComm and CacheBlend on HumanEval

HumanEval agents produce code with many syntax separators (e.g., . , ; !), which induce diverse and prefix-sensitive KV-cache distributions. With CacheBlend’s 20% recomputation, many sensitive tokens remain stale, degrading successor agents’ understanding. KVComm instead globally applies the best anchor-based offset to approximate these tokens’ KV-cache under the new context. When we increase CacheBlend’s recomputation ratio to 80%, as shown in the following table, performance significantly increases but remains inferior to KVComm (>81% pass@1):

This suggests coding tasks require near-full KV-cache updates under context shifts, aligning with KVComm’s prompt-adaptive strategy.

Q6: Comparison with Speculative Decoding

At a high level, building up the anchor pool in KVComm might resemble building up a database of past tokens, such as those employed in speculative decoding methods. However, the fundamental concepts and underlying scenarios differ notably between KVComm and speculative decoding methods. Specifically,

• KVComm accelerates prefilling across multiple agents by reusing/aligning existing KV-caches via anchor-based offsets. Each anchor in KVComm is essentially a cached representation of previously encountered segments along with their measured offsets from different prefix contexts. This allows us to approximate and reuse KV-caches dynamically at runtime.
• Speculative decoding accelerates decoding in typically single-agent generation by proposing and validating future tokens. In speculative decoding, the model speculatively predicts based on previously generated tokens or precomputed continuations. If the prediction aligns closely with the actual model's output tokens, it accelerates decoding by accepting the speculative output. Otherwise, it discards the speculative output and recomputes fully.

Q7: Fairness of Comparison with CacheBlend

After carefully double examining the CacheBlend paper and our work, we would like to clarify the fundamental differences in application scenario and the faithfulness to CacheBlend in our implementation, which justify that the experimental comparisonn with CacheBlend is as fair and objective as best we can.

Scenario Difference:

• CacheBlend is designed/evaluated for single-agent RAG, reusing non-prefix chunks with small token-level recomputation. Consequently, the experiments reported in CacheBlend are specially designed for the RAG task in a single-agent setting, which have no experimental results for multi-agent systems as we did.
• Our setting is multi-agent, where responses propagate across agents and prefixes shift frequently. Our scenario involves general coordination among multiple agents rather than single-agent RAG contexts.

Faithful Implementation of CacheBlend:

• We implemented CacheBlend per its paper and official code. To keep the similar reusing rate, we increase the recomputation rate to 20%, which theoretically further improves the performance of CacheBlend.

Comparability of Experimental Results:

• Our experiments intentionally use prompts representative of multi-agent coordination scenarios. Optimizing prompts specifically to match CacheBlend’s single-agent RAG scenario would not only distort the intended application scenario of our approach but also lose the realistic aspects and complexity of multi-agent inference.
• As reported in Table 1 of the main text, CacheBlend performs reasonably on RAG-like tasks (e.g., MMLU with RAG agents; see Appendix 6.3.2), which align well with the claim of CacheBlend, but struggles on math/coding due to the reason analyzed in Q5. Therefore, we have conducted as fair comparsion with CacheBlend as best we can.

Q8: Hardware / Communication Efficiency

KVComm is not restricted to single-GPU. The two assumptions (identical architecture; RoPE) are topology-agnostic. We currently report single-GPU results due to framework API limitations that standard open-source inference frameworks (e.g., HuggingFace Transformers, vLLM, SGLang) focus mainly on prefix-based caching and lack stable public APIs for fine-grained, per-layer KV-cache injection across devices.

Solution to Distributed Communication Bottleneck:

Regarding the potential distributed communication bottleneck, KVComm can deliberately avoid moving entire KV-caches across GPUs/nodes. Instead, we can synchronize only small-sized metadata (anchor indices and offset statistics) and perform KV-cache updates locally, which would remarkably reduce communication overhead.

We are currently integrating KVComm into distributed serving frameworks like vLLM and SGLang by adding minimal KV-cache read/write hooks to fully support efficient multi-agent systems in multi-GPU and distributed environments.

Cache Management Strategy:

Regarding the cache management strategy in KVComm, we designed two three-level cache managers to achieve efficient writing and retrieving anchors' KV-caches and the current shared KV-caches, respectively.

Based on these two cache managers, each agent can quickly retrieve their intended KV-caches and also store their generated KV-caches. Meanwhile, we conduct the process of KV-cache storage and retrieval in an asynchronous manner, thus further improving the efficiency. For clearer understanding, we will opensource the code repository once accepted.

Q9: Paper Structure & Formatting

We will revise Section 2.2 to clarify positioning among KV-acceleration methods, adjust Table formats to avoid misinterpretation, and verify all references' formats.