SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching

Thanks for your detailed review! Your answers have fully clarified all my doubts. Therefore, I'm rasing my overall rating to 7 and confidence to 5.

We thank the reviewer for the valuable suggestion. We agree that evaluating SentenceKV across a broader range of model sizes would further demonstrate its generality.

We are currently conducting additional experiments using LLaMA-3.2–3B-Instruct as a smaller model. Preliminary results show that SentenceKV remains effective in reducing memory usage while preserving accuracy even at smaller scales.

If time permits before the camera-ready deadline, we also plan to include results for larger models (e.g., 14B or 32B) to illustrate scaling behavior.

We thank the reviewer for raising this insightful point. We acknowledge the concern that highly relevant but unusually long sentences may occupy a disproportionate portion of the token budget, potentially excluding other semantically useful sentences.

To address this, we apply SnapKV-style prefiltering globally across all input tokens during the prefill stage. Specifically, instead of selecting top tokens from each sentence individually, SnapKV selects the top $r \cdot \tau$ most important tokens from the entire document, regardless of sentence boundaries. As a result, the number of retained tokens per sentence is determined by its relative semantic contribution, rather than its raw length. Even if a sentence is long, it will retain fewer tokens if it is not highly informative.

In practice, this global filtering strategy significantly reduces token retention from long but unimportant sentences. Our empirical analysis of LongBench shows that sentence lengths are generally concentrated around the mean, with very long sentences being rare (anonymous link to analysis). After filtering, token counts per sentence are typically modest and broadly distributed, helping to avoid scenarios where a single sentence dominates the token budget.

That said, we agree that this issue merits further exploration. As part of future work, we plan to detect abnormally long retained token spans ("buckets") during prefill and apply simple mitigation. For instance, we can compute the mean and standard deviation of sentence lengths across the input, and if any sentence exceeds mean + n × std, we can split it into smaller sub-spans. This lightweight check adds minimal computational overhead but can prevent outlier buckets from skewing the retrieval process.

We will add this direction to the discussion section in the revised version.

Thank you for your insightful suggestion regarding the applicability of SentenceKV to long CoT reasoning tasks. We appreciate the opportunity to clarify how our method naturally aligns with this setting.

We agree that reasoning models, especially those fine-tuned with long-form supervision such as DeepSeek-R1, present an ideal setting for evaluating the benefits of sentence-level KV cache management. In CoT generation, each sentence typically encapsulates a self-contained logical step. SentenceKV is well-suited to this structure: it retrieves semantically coherent sentence-sized blocks rather than token spans, allowing the model to focus attention on the most relevant prior reasoning steps without scanning the entire context.

We hypothesize that this yields two main advantages in reasoning-intensive settings: (i) High relevance retrieval: Since each reasoning step is sentence-aligned, our retrieval mechanism brings back precisely the portions of context likely to contain useful intermediate conclusions. (ii) Memory and latency stability: Even in long proofs or multi-turn derivations (as seen in GSM-Infinite), GPU memory remains bounded because only a handful of retrieved sentence embeddings and the current sentence reside in active memory.

We use dense retrieval from sentence embeddings, which generally preserves topical and logical coherence across reasoning chains. Moreover, many CoT models (e.g., DeepSeek-R1, Qwen) are trained on data where local reasoning steps are decoupled, making focused sentence-level retrieval particularly effective in practice.

We will add a paragraph in Section 5 highlighting the natural alignment between sentence-delimited CoT and SentenceKV, and we will cite both DeepSeek-R1 and GSM-Infinite. Due to time and resource constraints, we were not able to extend our experiments to include a CoT reasoning benchmark in the current submission. However, we recognize the importance of this direction and plan to include it in the camera-ready version, along with a discussion of how SentenceKV aligns naturally with the structure of reasoning models.

We thank the reviewer for pointing out these formatting issues. We appreciate the reviewer’s attention to detail in helping improve the presentation quality of the paper.

We thank the reviewer for raising this important point and for prompting a deeper analysis of the reported latency reduction.

First, we would like to clarify that the memory saving from SentenceKV is not from 89.71 GB to 52.71 GB purely due to KV cache reduction. The actual KV cache memory saving is from approximately 33.55 GB (FullKV) to 0.1 GB (SentenceKV), as we previously clarified in Q3. The remainder of the memory usage comes from model parameters and temporary activation buffers. Given that attention computation in long-context LLMs is typically memory-bound, such a significant KV cache reduction justifiably contributes to improved decoding speed.

To further support our claims, we provide below a detailed profiling analysis of SentenceKV's runtime overhead:

We profiled SentenceKV on an H100-96GB NVL GPU using a 64K-token input from the RULER benchmark (niah_single task). All timings were measured with torch.cuda.synchronize() to ensure accurate end-to-end latency accounting. The results are summarized as follows:

• 1. Prefill bucket construction:
  Sentence-level semantic grouping, including attention score aggregation and top-k selection, takes approximately 4 seconds for the full 64K input. This is a one-time cost during prefill and does not impact autoregressive decoding latency.

• 2. CPU–GPU round-trip latency (per decoding step):
  With a token budget of 1024 and a semantic keeping factor of 2, the KV transfer costs are:

  • GPU → CPU (2048 tokens): 0.0191 seconds
  • CPU → GPU (1024 tokens): 0.0038 seconds
  • Total round-trip latency: 0.0229 seconds

• 3. Sentence ranking latency:
  The similarity-based sentence selection adds approximately 0.05 seconds per decoding step (at 64K context), computed via efficient vector–matrix dot products on the CPU.

It is also important to note that the latency values reported in Figure 5 include only the computation latency and communication overhead (i.e., KV offloading and onloading), which directly affect autoregressive decoding. They do not include the sentence ranking latency or prefill-time bucket construction, both of which are either one-time costs or small in scale.

Even if we conservatively include the sentence ranking overhead (~0.05s), the total decoding latency with SentenceKV remains significantly lower than that of FullKV, thanks to the drastically reduced KV size and attention computation scope. This supports our claim that SentenceKV offers a compelling trade-off between memory, latency, and accuracy in long-context LLM inference.

We thank the reviewer for this question. We clarify below what contributes to GPU memory usage during decoding with SentenceKV:

• KV cache (subset only): In Full KV, decoding with a 256K-token context consumes ~33.55 GB for the key and value tensors (per layer × per head). In SentenceKV, we only retrieve and store ~1K top tokens per attention head from CPU to GPU, resulting in a ~0.1 GB KV cache size.

• Sentence-level semantic vectors: These vectors are relatively small in size and contribute minimally to overall memory usage.

• Temporary tensors: Additional buffers used during attention computation (e.g., attention weights, masks) are short-lived and small in size due to the compressed KV.

• Model parameters: The largest contributor to GPU memory is the model itself. For 7B/8B models, feed-forward layers and projection weights are persistently stored on GPU for inference.

Finally, we note that the reported GPU memory usage reflects peak memory, as measured by PyTorch’s torch.cuda.max_memory_allocated(). This includes allocator fragmentation and temporary buffer allocations, and therefore may exceed the theoretical sum of all tensor sizes. The key point, however, is the relative reduction in memory usage, which clearly demonstrates the effectiveness of SentenceKV’s compression strategy.

We appreciate the reviewer’s suggestion. In response, we have added experiments on RULER, a retrieval-intensive benchmark widely used in prior work. These results are now included in the following table and provide a complementary evaluation to NIAH.

To ensure a fair comparison, we also adjusted Quest’s chunk size appropriately for RULER. Unlike NIAH, where we used a chunk size of 32 to match the typical sentence length, RULER contains much shorter sentences on average. Therefore, we set Quest’s chunk size to 16 on RULER, which aligns better with the sentence-based granularity used by SentenceKV.

Besides, we also add another offloading based method - ShadowKV, for a more comprehensive comparison. ShadowKV is a high-throughput long-context LLM inference system that compresses the key cache using low-rank decomposition (SVD) and offloads the value cache to the CPU. It reconstructs sparse KV representations on the fly to reduce memory usage and support larger batch sizes. However, this comes at the cost of increased implementation complexity and prefill overhead.

Below table shows the RULER results with 64k input length.

SentenceKV achieves comparable or better performance on multiple tasks with a much simpler and faster prefill process. We also note that SentenceKV and ShadowKV often achieve similar levels of accuracy, which we consider acceptable and expected given the different design philosophies. ShadowKV performs more fine-grained reconstruction via low-rank SVD and sparse selection, which allows slightly more precise prefill handling—but at the cost of increased prefill time, code complexity, and integration difficulty. In contrast, SentenceKV achieves comparable accuracy while offering a simpler, faster, and more practical implementation, which we believe is more suitable for real-world deployment.

That said, we also observed some performance variation across tasks. In particular, the fixed semantic keeping factor used in our current setup may not always achieve the optimal trade-off between memory usage and retrieval quality.

As part of our ongoing and future work, we are actively investigating the relationship between input characteristics—such as sentence length distribution and semantic diversity—and the optimal semantic keeping factor. Preliminary observations suggest that dynamically adjusting this factor based on input statistics can further improve both efficiency and accuracy.

We believe this addition strengthens our evaluation and confirms the effectiveness of SentenceKV in retrieval-heavy settings.

Yes, the proposed SentenceKV method is applied consistently to all attention layers of the model. This design ensures that semantic-aware KV compression is uniformly integrated throughout the network, preserving coherence and efficiency across all layers.

We thank the reviewer for pointing this out. We apologize for the confusion caused by the previous version of the algorithm description in the Appendix. In fact, Ot is computed for every newly generated token, not just at the end of a sentence.

We have revised the algorithm pseudocode in the Appendix to make this clearer in the updated version. We hope this resolves the confusion and improves the clarity of the algorithm.

We appreciate the reviewer’s comment regarding the efficiency–performance trade-off between SentenceKV and SnapKV. As shown in Table 1 and Figure 4, SentenceKV consistently achieves higher or comparable accuracy across a wide range of tasks and context lengths, while maintaining significantly reduced memory usage.

The primary trade-off lies in slightly increased decoding latency, due to the use of sentence-level semantic grouping and CPU-to-GPU communication. However, we believe this latency overhead is acceptable given the notable improvements in memory efficiency and overall accuracy, which are critical in long-context LLM serving.

It is also important to note that it is not straightforward to fix latency and compare accuracy directly between SentenceKV and SnapKV. Matching latency would require significantly lowering SentenceKV’s token budget, which generally leads to a much more severe degradation in performance than a slight latency increase. In practice, we find that slight latency increases are often more tolerable than aggressive compression.

To further support this, we performed a quantitative analysis on the NIAH benchmark, fixing the token budget at 128 and the semantic keeping factor at 3. In this setting:

• SnapKV achieves ~15 ms decoding latency.
• SentenceKV incurs an additional 0.5 ms for CPU–GPU transfer and ~0.5 ms for similarity-based ranking, totaling ~16 ms latency (a 6% increase).
• However, under these settings, SentenceKV delivers a 25% improvement in accuracy, demonstrating a more favorable trade-off overall.

We thank the reviewer for pointing out this important limitation.

We agree that a more adaptive strategy could further enhance the generalization ability of the method. As part of our ongoing and future work, we are actively investigating the relationship between input characteristics—such as sentence length distribution and semantic diversity—and the optimal semantic keeping factor. Preliminary observations suggest that adjusting this factor dynamically based on such input statistics could lead to further improvements in efficiency and accuracy.

We appreciate the reviewer’s suggestion and will add a more detailed discussion of this limitation and direction in the conclusion section of the revised manuscript.

Dear Reviewer W49T,

We sincerely thank you for the detailed and constructive feedback, which has helped us significantly improve the clarity and completeness of our paper. We would like to summarize the key updates we made in response to your comments:

• We provided a detailed breakdown of SentenceKV's overhead, including prefill cost, CPU-GPU transfer latency, and sentence ranking time, confirming that the additional overhead is small.

• We quantitatively analyzed the efficiency-accuracy trade-off compared to SnapKV, showing that SentenceKV achieves 25% higher accuracy with only a 6% increase in latency.

We hope these updates address your concerns. If possible, we would greatly appreciate any additional feedback you may have, including whether the revisions have resolved your questions. Thank you again for your time and insightful review.

In summary, we have added two offloading-based baselines: InfLLM, as you suggested, and ShadowKV, a recent more advanced offloading method. In addition, we included a more challenging benchmark, RULER, which emphasizes retrieval-intensive reasoning under long-context settings. Across both LongBench and RULER, SentenceKV consistently outperforms InfLLM, and performs on par with or better than ShadowKV while achieving significantly better prefill efficiency. We believe these results provide stronger evidence of the effectiveness and practicality of SentenceKV.

We would sincerely appreciate it if you could consider updating your evaluation and provide any further feedback. Your comments would be very helpful for us to improve the final version. Thank you again for your time and constructive suggestions.

We thank the reviewer again for the valuable feedback. In addition to ShadowKV, we have now implemented and evaluated InfLLM on both the LongBench and RULER benchmarks, with Llama-3.1-8B-Instruct model.

As shown in the updated table below, SentenceKV consistently outperforms InfLLM across both benchmarks. Notably, on the more challenging RULER tasks, SentenceKV achieves better performance due to its more deterministic semantic compression mechanism.

LongBench:

RULER: