Sim-LLM: Optimizing LLM Inference at the Edge through Inter-Task KV Reuse

We deeply appreciate your recognition and careful review of our work — it truly means a lot to us. In response to your comments, we provide the following clarifications. Please let us know if any further clarification is needed.

---

• Response to Weakness

In our current design, we prioritized performance by optimizing throughput and latency, which can sometimes result in a slight accuracy loss. This trade-off was made in order to meet the strict real-time processing requirements of edge computing environments, where low latency and high throughput are often more critical than perfect accuracy.

To address this trade-off, we are considering a hybrid model architecture that assigns high-accuracy models to critical tasks while using more efficient models for less critical ones. This would allow the system to maintain performance without sacrificing essential accuracy. Additionally, to further improve accuracy without compromising performance, we are exploring lightweight post-processing techniques to refine initial outputs, aiming to recover accuracy without significant performance degradation.

---

• Response to Q1

Our motivation stems from the observation of task similarities commonly present in broad edge computing scenarios. In cloud environments, the larger scale of nodes and the greater diversity of tasks may pose challenges for identifying such task similarities. However, in specialized domains—such as medicine or law—where large language models are fine-tuned for domain-specific tasks, task similarity can still be high, making Sim-LLM fully applicable in these contexts. Moreover, the presence of high-speed data center networks in cloud infrastructures enables efficient inter-node communication. As a result, even without applying the communication optimizations proposed in Sim-LLM, the system still holds strong potential for performance gains when deployed in cloud settings.

---

• Response to Q2

Given our focus on edge scenarios, the primary model used in our experiments is Llama-7B. In response to your suggestion, we conducted supplementary experiments with Llama-13B, and the results are presented in the table below. As you anticipated, Sim-LLM maintains strong performance with the larger model, consistently outperforming smaller models such as Llama-7B and TinyLlama-1.1B. This observation aligns with the trends illustrated in Figure 11 of Appendix B.1. These findings further demonstrate the scalability and effectiveness of Sim-LLM when applied to larger language models.

---

• Response to Q3

As mentioned in the third paragraph of Section 1 and in the discussion"To address the second challenge" in Section 2, Sim-LLM is inherently compatible with intra-layer KV optimization techniques. While existing methods primarily focus on KV optimization from a layer-wise or token-wise perspective, our work takes a task-wise approach. These perspectives are orthogonal, and therefore complementary. A promising direction for future work is to explore the integration of Sim-LLM with these fine-grained KV optimization strategies to further reduce memory consumption and mitigate accuracy degradation in edge environments.

We would like to express our sincere gratitude to the reviewer for your time and valuable feedback. We are especially thankful for your recognition of our work as “the first” in this direction and for highlighting our “well-founded observation.” Your recognition of the novelty of our work is aligns with the positive assessments provided by other reviewers. Below, we provide clarifications in response to your concerns. Please let us know if any further explanation is needed. ("W" refers to "Weakness" , "Q" refers to "Questions")

---

• Response to W1

In practice, edge servers are typically deployed near users. While not as powerful as cloud servers, enterprise-grade edge deployments often use high-performance hardware. For instance, the NVIDIA Jetson AGX Orin offers 275 TOPS with a 12-core ARM CPU and 64GB DDR5 memory, and the Huawei Atlas 500 Pro provides up to 420 TOPS with a 24-core KunPeng 920 CPU, 128GB DDR4 memory, and 4 Atlas 300I GPUs. In our experiments, each edge server is configured with 4 NVIDIA A40 GPUs (299 TOPS), closely mirroring real-world setups and thus ensuring that our results reflect practical edge computing capabilities.

---

• Response to W2

Although it may seem intuitive that similar tasks share similar KVs, this insight has not been experimentally validated before. In edge environments, task similarity for inference is often unknown, and we are the first to demonstrate it experimentally. Moreover, while most related works optimize inference at the token or layer level, our approach focuses on task-wise KV reuse, distinguishing it from conventional methods. This task-wise reuse effectively extends prior concepts of lossless reuse to lossy reuse, as noted by Reviewer G55H and vsRh.

---

• Response to W3

To address your concerns, we now provide the algorithm box for Sim-LLM. For each task batch, tasks are mapped to LSH buckets for similarity matching. If a match is found, the corresponding KV is reused to accelerate inference; otherwise, standard inference with the sandwich configuration is applied. The hash, embedding, and top-layer KV are stored in the KV_Manager with LRU eviction policy. After each batch, servers update and exchange task prototypes to maintain a global feature table.

---

Input:

• Task batch B = {t1, t2, ..., tn}
• Cosine similarity threshold θ
• LSH bucket size k
• KV_Manager (with cache size C, eviction policy: LRU)
• bottom stage num sandwich_bot, top stage num sandwich_top

Output:

• Inference results with KV optimization

---

Procedure:

1. Preprocessing (Section 3.2)

   For each task t∈B:

   • Tokenize input and generate embedding e_t
     • Compute LSH hash h_t ← LSH(e_t, k)
     • Assign t to bucket LSH_Bucket[h_t]

2. Task Similarity Identification (Section 3.2)

   For each task t∈B:

   • Search LSH_Bucket[h_t] for candidate tasks S
     • For each candidate s∈S:
     • Compute cosine similarity sim(e_t, e_s)
     • If ∃s∈S such that sim(e_t, e_s) ≥ θ:
       • Retrieve cached top-layer KV from KV_Manager[s]
         • Assign t.KV ← KV_Manager[s]
     • Else:
       • Compute full KV for t from scratch

3. KV Cache Management (Section 3.3) For each processed task t:

   • Store (e_t, h_t, KV_t) in KV_Manager
     • If cache exceeds size C, apply eviction policy (LRU)

4. Edge Server Communication (Section 3.4)

   For task t with no local match:

   • Compute task feature f_t and compare with prototypes in global feature table
     • If remote feature match f found:
       • Offload t to corresponding server
   • Periodically update and sync task prototypes across servers

5. Inference Execution (Section 3.3&3.4) a. For each task t∈B:

   • If t.KV exists:
     • Run inference by reusing top-layer KV
   • Else:
     • Run inference with sandwichconfiguration, retaining bottom sandwich_bot layers and top sandwich_top layers KV.

---

• Response to W4

Each server maintains a global feature table containing task prototypes from all edge servers. After each batch, servers update their own prototypes and share them to keep global tables synchronized. These prototypes serve as lightweight indexes for efficient cross-server task matching. The global table is only updated upon receiving new prototypes and does not involve eviction.

In contrast, the KV_Manager stores metadata for newly processed tasks and triggers LRU eviction when the cache limit is reached. We conduct new experiment to confirm the robustness of Sim-LLM. The detailed results can be found in our response to Q3 for Review aP8N.

---

• Response to Q1

We conducted supplementary experiments to evaluate the latency and memory usage of Sim-LLM under a large cache size. The results are presented in Table 1.

As the number of cached tasks increases, memory consumption grows accordingly. To manage this, we implement an LRU eviction policy, which effectively removes less frequently used entries, avoiding memory overflow and maintaining cache relevance.

Although a larger cache may introduce overhead in task matching and KV lookups, we mitigate this using locality-sensitive hashing (LSH) for efficient approximate matching. Additionally, the global feature table accelerates semantic similarity matching across edge servers, keeping overall latency low. For instance, even with a cache size of 4096, Sim-LLM achieves a latency of 45.35ms, compared to 60.61ms for LLaMA.

Table 1 Latency and memory usage under large cache sizes.

---

• Response to Q2

(1) We conducted new experiments to evaluate system performance under different request distributions—Poisson (uniform) and Power-law (bursty)—focusing on both perplexity and throughput. As shown in Table 2, Sim-LLM maintains high performance across varying task patterns. Under bursty workloads, similar tasks occur more frequently, enabling the LRU-based KV_Manager to retain and reuse KVs more effectively, thereby accelerating inference. These results demonstrate that Sim-LLM adapts well to dynamic task distributions.

Table 2 Performance under different task distributions.

(2) Our method does not require periodic retraining, as Sim-LLM relies on cosine similarity combined with LSH for semantic matching, rather than a trainable similarity model. This design significantly accelerates inference without requiring costly architectural modifications and allows the system to dynamically adapt to new tasks, ensuring both efficiency and scalability.

---

• Response to Q3

Similar to many prior works such as [1] [2] [3], our approach assumes that all edge nodes are trustworthy for KV cache sharing. Nonetheless, we acknowledge the security challenges inherent in cross-server communication within edge computing environments. Malicious nodes could threaten data integrity and confidentiality. Existing research addresses these concerns, such as [4], which analyzes potential attacks and defense strategies at the edge; and [5], which enforces secure data sharing via access control mechanisms.

While our work does not focus on securing cross-node communication, existing security solutions can be integrated to mitigate these risks. We consider security issues orthogonal to our core contributions—task acceleration and KV reuse—and thus not directly impacting the proposed methods.

---

• Response to Q4

Temperature sampling controls output randomness by scaling logits before selecting the next token. Higher temperatures increase randomness, while lower temperatures yield more deterministic outputs.

Adjusting the temperature impacts only the quality of the tokens generated during the current step and does not alter previously generated tokens. Consequently, if the existing tokens match a stored similar task, the associated KV cache can still be reused. Therefore, the reusability of early-task KVs remains unaffected.

---

• Response to Q5

The global prototype table stores task prototypes that serve as local indexes for efficient cross-server task matching. Its memory usage depends only on the number of edge servers. To address your concern, we conducted supplementary experiments (Table 3) measuring memory consumption over time. Due to the inherent similarity of task features and fluctuating traffic in edge environments, prototype updates are minimal, and memory usage stabilizes.

Each edge server maintains one such table. After processing each batch, a edge server updates its prototype and shares the update with other edge servers to keep their global prototype tables synchronized. Only when an edge server leaves the network, the item corresponding to the edge server will be pruned.

Table 3 Memory usage over time.

---

Refs

[1] OL-MEDC: An Online Approach for Cost-Effective Data Caching in Mobile Edge Computing Systems. TMC, 2023.

[2] Joint service placement and request routing in multi-cell mobile edge computing networks. INFOCOM, 2019.

[3] Service routing in multi-tier edge computing: A matching game approach. JSAC, 2022.

[4] Edge Computing Security: State of the Art and Challenges. Proceedings of the IEEE, 2019.

[5] Everything Under Control: Secure Data Sharing Mechanism for Cloud-Edge Computing. TIFS, 2023.

We would like to express our sincere gratitude to the reviewer for your time and the valuable feedback. In particular, your recognition of our work as “a novel method”，“a direction underexplored in prior work” and “well-motivated” aligns with the positive assessments provided by other reviewers. Below, we provide clarifications in response to your concerns. Please let us know if any further explanation is needed. ("W" refers to "Weakness" , "Q" refers to "Questions")

---

• Response to W1

We would like to highlight that, in practice, many edge computing environments are already leveraging high-speed communication links between servers [1] [2] [3] [4] [5], including InfiniBand which is a high performance technology aiming to provide low-latency and high-bandwith. In our paper, the edge servers are connected via an InfiniBand network to facilitate the exchange of task prototypes and KV sharing.

To minimize communication overhead due to information exchange between edge servers, when similar tasks are identified on another edge server, only the sequence or the corresponding word embeddings are transmitted to the identified edge server, rather than transmitting the KV cache of the similar task to the current processing task from the identified server back to the original server. Furthermore, regarding the inter-server communication concerns you raised, we conducted supplementary quantitative experiments, as shown in Table 1. These results demonstrate that the communication overhead is almost negligible compared to the inference latency.

Table 1 Communication latency and inference latency with varying numbers of nodes.

---

• Response to W2

In Appendix B.1 Impact of Cosine Similarity Threshold, we present experimental results illustrating how throughput, perplexity (ppl), and downstream task accuracy vary with different cosine similarity threshold. When the threshold is low, throughput performance exceeds that of a higher threshold. This is because most incoming tasks can easily identify "similar" tasks' KVs for reuse, introducing minimal identification overhead and accelerating the inference process. However, this gain comes at the cost of model accuracy. When the threshold is incresd to 0.9, there is a significant drop in generation speed due to fewer tasks being able to reuse KVs, resulting in many tasks performing inference as in the standard Transformer model. Therefore, as a trade-off between generation speed and model accuracy, a threshold of 0.8 is preferred.

The generalizability of this threshold is supported by the observation that many edge tasks share underlying semantic similarities. As noted in Observation 2 (Section 1), KV caches generated by similar tasks are themselves similar, indicating that task features exhibit consistent patterns across different edge scenarios. The selected threshold captures these shared characteristics, making it robust across varied workloads. Moreover, it is resilient to small fluctuations in task features, which are common in dynamic edge environments.

To further validate the robustness of Sim-LLM, we conducted supplementary experiments using different thresholds (0.6, 0.7, 0.8, and 0.9) across four edge servers. The results, shown in Table 2, align with the trends observed in Appendix B.4 Table 2, demonstrating that Sim-LLM maintains consistent behavior under variable threshold settings. As expected, lower thresholds enable more KV reuse and faster inference, while higher thresholds reduce reuse and slow down inference. Nevertheless, due to the semantic consistency of tasks across servers, performance remains stable over a range of threshold values, confirming the adaptability of Sim-LLM in heterogeneous edge environments.

Table 2 Throughput comparison for different sequence lengths and thresholds.

---

Refs

[1] Learning-Aided Computation Offloading for Trusted Collaborative Mobile Edge Computing, IEEE TMC, 2020.

[2] FlowStar: Fast Convergence Per-Flow State Accurate Congestion Control for InfiniBand. IEEE/ACM ToN, 2024.

[3] Tutti: coupling 5G RAN and mobile edge computing for latency-critical video analytics. MobiCom, 2022.

[4] C2DN: How to Harness Erasure Codes at the Edge for Efficient Content Delivery. NSDI, 2022.

[5] Argus: Enabling Cross-Camera Collaboration for Video Analytics on Distributed Smart Cameras. IEEE TMC, 2025.

Dear Reviewer G55H

As we near the end of the author-reviewer discussion phase, we would like to sincerely thank you again for your time and valuable feedback. If you think our clarifications have addressed your concerns, we would deeply appreciate your support in updating the score accordingly.

If there are any remaining questions or points you would like us to clarify, please feel free to let us know. We’re here to support the discussion as best we can.

Best regards,

The Authors

We would like to express our sincere gratitude to the reviewer for your time and valuable feedback. In particular, your recognition of our “solid observations” and the comment that “the overall idea is novel” align with the positive assessments provided by other reviewers. Below, we provide clarifications in response to your concerns. Please let us know if any further explanation is needed. ("W" refers to "Weakness" , "Q" refers to "Questions")

---

• Response to W1

We acknowledge that both cosine similarity and Locality Sensitive Hashing (LSH) are well-established techniques. Their combination, however, offers a scalable and efficient solution for task matching based on semantic similarity, significantly accelerating model inference without requiring costly architectural modifications. Cosine similarity, in particular, has a strong theoretical foundation and has been extensively studied [1] [2]. As discussed in [2], it can be interpreted as the sum of semantic similarities along embedding dimensions after Independent Component Analysis, further justifying its use for semantic comparison.

We adopt top-layer KV reuse is motivated by the interpretation of the Transformer’s stacked architecture as an iterative refinement process, where lower layers tend to encode syntactic information, while higher layers focus more on semantics [3] [4]. In Section 4.3, we further validate this choice through experiments that compare the performance of KV reuse from different layers, demonstrating the superiority of using top-layer KV as the source.

---

• Response to W2

As shown in [5] [6] [7], LLMs exhibit significant layer redundancy—intermediate layers contribute minimally and can be removed with limited performance degradation. This supports our design choice: even when skipping these layers, retaining caches from the bottom and top layers preserves essential task-relevant information. We further validate this through experiments in Section 4.3.

---

• Response to W3

We conducted additional experiments to examine how the KV reuse match rate changes with the number of tasks. As shown in Table 1, the match rate increases as the task set grows. The values in parentheses indicate the proportion of similar tasks in each set. These results suggest that a larger task pool provides more opportunities for KV reuse, thereby enhancing inference efficiency.

Table 1 KV reuse match rate based on the number of tasks and varying similarity proportion of task sets.

Regarding incorrect KV reuse, this situation occurs when LSH distortion happens, i.e., two dissimilar tasks are erroneously mapped to the same bucket, or similar tasks are mapped to different buckets. We first explored the probability of this occurrence, and the results, shown in Table 2, indicate that this probability does not exceed 3%.

Table 2 Probability of incorrect KV reuse due to LSH distortion.

Next, we simulated the impact of a higher error rate on the output quality by forcing tasks to map to the same LSH bucket. The experiments show that only when this probability exceeds 10% does it significantly affect the quality of the output. However, this probability is unlikely to occur in real-world scenarios, indicating that LSH distortion does not overly impact the model accuracy of Sim-LLM.

Table 3 Impact of error rate on model output quality.

---

• Response to Q1

To improve overall inference efficiency, Sim-LLM postpones tasks that fail to match similar ones within a batch. This allows matched tasks to proceed with KV reuse without being delayed by unmatched tasks. If few tasks in a batch are eligible for reuse, the inference time remains comparable to that of standard processing, and no postponement occurs.

To address the reviewer’s concern about the worst-case scenario—where unmatched tasks are repeatedly delayed across batches—we conducted an experiment to measure the probability of continuous postponement. As shown in Table 4, the probability of a task being postponed for 5 consecutive batches is only 1.1%, indicating that such cases are worst-case. Additionally, Table 5 reports the delay overhead introduced by postponement, showing minimal impact on overall inference efficiency (e.g., average inference time remains 0.094s for sequence length 5+128).

Table 4 Probability of tasks being continuously postponed across multiple batches.

Table 5 Delay overhead introduced by postponed tasks with varying similarity proportions of task sets (sequence length is 5+128).

---

• Response to Q2

As shown in Appendix B.1 (Impact of Cosine Similarity Threshold), we analyzed how the similarity threshold affects throughput, perplexity, and downstream accuracy. A lower threshold improves throughput by enabling more tasks to reuse KVs with minimal overhead. However, this comes at the cost of accuracy. Conversely, at a high threshold (e.g., 0.9), KV reuse becomes rare, leading to slower inference comparable to standard Transformers. As a trade-off, we select 0.8 as the optimal threshold, balancing generation speed and model performance.

---

• Response to Q3

(1) We adopt the LRU strategy to maximize reuse of similar KV pairs and thus accelerate inference. LRU retains frequently reused task KVs while evicting the least reused ones.

(2) To address concerns about workload variability, we evaluated LRU under two task distributions: Poisson (uniform workload) and Power-law (bursty workload). Results in Table 6 show that under bursty conditions, LRU maintains strong performance without significant latency increase compared to uniform loads. This is because bursty workloads tend to have many tasks sharing similar features, allowing LRU to effectively retain frequently reused KVs and boost inference speed.

Table 6 Latency comparison for Poisson and Power-law distributions under the LRU strategy.

---

• Response to Q4

We use an InfiniBand network to connect edge servers for exchanging task prototypes and KV sharing, leveraging its low-latency, high-bandwidth characteristics widely adopted in edge computing [8] [9] [10].

After processing each batch, edge servers update and exchange prototypes to maintain a global feature table. To reduce communication overhead, servers typically exchange task sequences or embeddings rather than full KV caches, as noted in the last paragraph of Section 3.4.

Regarding the communication overhead introduced by task prototype exchange and KV sharing, we conducted new experiments. The results, as shown in Table 7, indicate that this overhead is negligible compared to the inference cost.

Table7 communication and inference latency for different batch sizes.

Additionally, we also conducted supplementary experiments to evaluate latency in large-scale deployments. The results presented in Table 8 indict, although communication overhead tends to increase as the network scales up, its overall impact on inference efficiency remain limited.

Table 8 Communication latency and inference latency with varying numbers of nodes.

---

• Response to Q5

Thank you for pointing out the typo. What we intend to convey is the potential integration between our work and approaches using MQA or GQA techniques. We will revise the statement accordingly in the manuscript.

---

Refs

[1] Correlation Coefficients and Semantic Textual Similarity. NAACL, 2019.

[2] Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings. ACL, 2025.

[3] Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation. ACL, 2023.

[4] BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. NAACL, 2019.

[5] ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. ACL, 2025.

[6] LaCo: Large Language Model Pruning via Layer Collapse. EMNLP, 2024.

[7] Llm-pruner: On the structural pruning of large language models. NIPS, 2023.

[8] FlowStar: Fast Convergence Per-Flow State Accurate Congestion Control for InfiniBand. ToN, 2024.

[9] Alarm: An Adaptive Routing Algorithm Based on One-Way Delay for Infiniband. TNSE, 2024.

[10] Enabling the CUDA Unified Memory model in Edge, Cloud and HPC offloaded GPU kernels. CCGrid, 2022.

[11] Tutti: coupling 5G RAN and mobile edge computing for latency-critical video analytics. MobiCom, 2022.

Dear Reviewer aP8N

As we near the end of the author-reviewer discussion phase, we would like to sincerely thank you again for your time and valuable feedback. If you think our clarifications have addressed your concerns, we would deeply appreciate your support in updating the score accordingly.

If there are any remaining questions or points you would like us to clarify, please feel free to let us know. We’re here to support the discussion as best we can.

Best regards,

The Authors