ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

Thank you for the supportive comments and for recognizing the novelty of our method and the thorough evaluations. We hope our detailed clarifications below address the remaining concerns.

Q1: Missing clarification on which dataset was used for observations

We appreciate the reviewer's careful reading. The observations regarding the low-rank structure of pre-RoPE keys, the high similarity of post-RoPE keys, and the KV cache hit rate plots (Figures 2a, 2b, 5b, and 5c) are based on sequences drawn from PG-19 [1], a real-world long-context dataset. We chose PG-19 to ensure that our structural analyses reflect natural language characteristics rather than synthetic artifacts.

For Figure 5a, the results are based on sequences sampled from RULER-NIAH, consistent with our downstream evaluation tasks.

Q2: How does the token selection interact with state-of-the-art attention frameworks like FlashAttention? Does it require kernel modifications?

Thank you for the insightful question. In our implementation, token selection is applied prior to the attention computation. At each decoding step, we identify the most relevant KV tokens based on chunk-level attention scores. For efficiency, the chunk-level attention score computation (GEMM + Softmax) is fused into a single kernel. The selected KV entries are then gathered into a compact buffer, which is passed directly to the FlashAttention interface.

Since we only modify the input tensors to attention and not the attention logic itself, no modifications to the FlashAttention kernel are required. This makes ShadowKV easy to integrate into modern inference stacks that already support FlashAttention.

[1] Compressive Transformers for Long-Range Sequence Modelling

Thank you for the thoughtful and thorough review. We truly appreciate your recognition of our findings and experiments. We have thoroughly addressed each of your questions and hope our responses will lead you to consider raising your score.

Q1: Concern about unaccounted latency from key reconstruction and value fetching in the Theoretical Equivalent Bandwidth analysis.

As detailed in Appendix A.6, we provide a full component-wise decoding latency breakdown (in milliseconds) across various batch sizes and context lengths on an A100 GPU, which includes the time cost of both key reconstruction (Recompute K) and value cache fetching (Fetch V). Our system is designed to leverage CUDA multi-streams to overlap these two operations.

As shown in the table, Fetch V and Recompute K exhibit comparable latency. Since they are executed on separate CUDA streams, their durations can be effectively overlapped—meaning only Fetch V contributes to the critical path.

Furthermore, our measured end-to-end throughput gains validate that these design choices translate into practical system-level improvements.

Q2: Some references not discussed.

As stated in Section 3.1 (Line 188), prior low-rank approaches primarily focus on data-independent, offline weight decomposition, i.e., performing low-rank factorization on the model weight matrices using calibration data or during training. These methods either require training or achieve limited compression. The cited works (EigenAttention, HeadsKV, and MatryoshkaKV) belong to this category. Specifically:

• EigenAttention generates lower-rank weight matrices, yielding at most 40% KV cache reduction.
• HeadsKV converts MHA into GQA via low-rank decomposition of weights, which requires training and is only applicable to MHA-based models.
• MatryoshkaKV trains orthogonal projection matrices, achieving around 60% average KV cache compression.

In contrast, ShadowKV identifies the dynamic, sequence-dependent low-rank structure in pre-RoPE keys, and performs online SVD directly on the KV cache during inference. This approach is training-free, adaptive to each sequence, and achieves significantly higher compression (over 6$\times$, 15.6% of the original memory).

Thank you for giving us the chance to elaborate on this point. We will add these mentioned references to the future version.

Q3: Core innovation of the work.

As discussed in Q2, unlike existing low-rank methods that compress weights offline, ShadowKV performs online activation-level SVD on the KV cache in a sequence-adaptive manner, and achieves substantially better compression while preserving accuracy. Prior methods rely on static weight decomposition and often require fine-tuning or model modification.

Moreover, our approach is tightly integrated with a system-level design. We carefully co-design algorithm and system by enabling low-rank key cache reconstruction and CPU-to-GPU value fetching to overlap through CUDA multi-streams. Additionally, we utilize an accurate sparse KV selection mechanism that reduces attention computation and memory movement. This holistic integration ensures end-to-end throughput gains.

Q4: Outlier tokens — are there interpretable patterns? And does outlier count correlate with layers/heads?

Thank you for the insightful question. We clarify that we selected a subset of layers rather than showing all layers in Figure 5c for visualization purposes to avoid clutter. We performed additional analysis to better understand the nature of outlier tokens and their distribution.

We found that certain tokens, such as attention sinks [1], are consistently identified as outliers, with their similarity scores even being negative, highlighting their distinctiveness in the distribution. Across heads, the sets of outliers tend to differ. We did not observe a clear trend or correlation between outlier frequency and specific layers or head indices.

We absolutely agree that this is a rich direction for future exploration, and we plan to conduct deeper investigations into the interpretability and structure of outliers in future work.

[1] Efficient Streaming Language Models with Attention Sinks

Thank you for the thoughtful review and your positive assessment of our work. We appreciate your recognition of the novelty of ShadowKV, particularly its system-level insights and the strength of accuracy evaluations. We address your concerns with additional clarifications and experiments as detailed below.

Q1: Throughput evaluation is somewhat limited. Memory savings do not always translate directly into performance gains.

We appreciate the suggestion. To further support our claim, we include a table below, showing throughput (tokens/s) under varying batch sizes and contexts for both Full KV and ShadowKV with Llama3-8B-1M.

• ShadowKV enables significantly larger batch sizes at long context lengths, where Full KV quickly runs out of memory.
• At small batch sizes (e.g., bsz = 1), ShadowKV may be slightly slower due to reconstruction and fetching overhead. However, as batch size increases, the benefit of memory savings and increased parallelism translate directly into throughput gains.

These additional results help clarify that ShadowKV not only reduces memory usage but also yields consistent throughput improvements. We will include it in the final version and appreciate the reviewer's suggestion, which helped us strengthen this aspect of the paper.

Q2: What is the time breakdown for low-rank key cache reconstruction in a typical scenario?

As detailed in Appendix A.6, we provide a full component-wise decoding latency breakdown (in milliseconds) across various batch sizes and context lengths on an A100 GPU, which includes the time cost of both key reconstruction (Recompute K) and value cache fetching (Fetch V). Our system is designed to leverage CUDA multi-streams to overlap these two operations.

Q3: How effectively can value cache fetching overlap with computation, and under what conditions?

As discussed in Appendix A.6, we leverage CUDA multi-streams to overlap CPU-GPU data transfers (Fetch V) with GPU-side compute (Recompute K), ensuring high GPU utilization during decoding. This overlap is particularly effective in long-context settings where attention computation dominates latency.

As shown in the table, Fetch V and Recompute K exhibit comparable latency. Since they are executed on separate CUDA streams, their durations can be effectively overlapped—meaning only Fetch V contributes to the critical path.

Furthermore, our measured end-to-end throughput gains validate that these design choices translate into practical system-level improvements.

Q4: Experimental settings for Table 3—are they based on standard vLLM?

Our baseline implementation leverages open-source efficient CUDA kernels, including those from both vLLM and FlashInfer [1]. Our inference engine is optimized specifically for large-batch long-context decoding.

The table below presents a decoding latency (tokens/s) benchmark, demonstrating that our baseline achieves comparable or slightly better efficiency than vLLM in full-attention mode. This shows that our baseline is a valid and fair reference point.

Q5: Limitations for integrating SHADOWKV into vLLM and SGLang?

ShadowKV employs custom CUDA kernels for low-rank reconstruction, RoPE fusion, and efficient CPU data prefetching with overlap , built on top of CUTLASS [2]. Integrating ShadowKV into vLLM and SGLang would require backend modifications to support these kernels and its memory pool management strategy. While this integration is non-trivial, it is technically feasible, and we are actively working toward supporting popular serving engines in the future.

[1] https://github.com/flashinfer-ai/flashinfer

[2] https://github.com/NVIDIA/cutlass

Thank you for detailed review and valuable feedback. We appreciate the reviewer's recognition of the novelty and effectiveness of our method. Below, we address the raised concerns below and will incorporate clarifications into the revised version. We hope the reviewer can consider raising your score in light of our response.

Q1: Clarification on Figure 8 and the meaning of "Sparse KV Cache Budget".

The x-axis in Figure 8 refers to the percentage of KV pairs selected at each decoding step for attention computation—i.e., the active sparse KV budget per step, rather than the overall GPU memory usage for KV cache storage. In contrast, the 6$\times$ memory compression refers to the total reduction in GPU memory footprint for the KV cache, achieved through a combination of low-rank key compression and value offloading. The extremely small sparse budget (e.g., 1.5%) is made possible by our accurate TopK selection mechanism. We will clarify this distinction explicitly in the figure caption in the future version.

Q2: Undefined terminology such as KV cache hit rate and $D(H_1, H_2)$.

Thank you for the valuable suggestion. We will add clarifications below in the future version.

• The KV cache hit rate refers to the proportion of selected KV chunks at decoding step $t$ that are also selected at step $t+1$, due to temporal locality in generation. This metric reflects the effectiveness of caching selected sparse KV pairs across decoding steps to reduce redundant key cache reconstruction and value cache fetching.
• As for $D(H_1, H_2)$, it is defined in Section 3.1 (Line 204) as $D(H_1, H_2) = \langle H_1, H_2 \rangle / r$, where $H_1$ and $H_2$ are rank-$r$ projection matrices and $\langle \cdot, \cdot \rangle$ denotes the Frobenius inner product. We agree the definition could be made more prominent, and we will revise the presentation to improve clarity and visibility.

Q3: GPU memory usage during inference — will it spike due to K reconstruction and V fetching?

In ShadowKV, GPU memory usage remains bounded and does not spike during inference. This is due to two key factors:

• At each decoding step, only a sparse subset of KV pairs (e.g., 1.5%) is selected for reconstruction and fetching, not the full sequence.
• These operations are performed layer-by-layer, and the buffers are reused across layers, so peak memory does not accumulate across the entire model.

For example, in Table 3 (e.g., 122K context, batch size of 24), we reconstruct 1.56% of the full KV cache during each decoding step, resulting in <200MB peak additional memory usage—negligible compared to the full KV size. We will add a memory profiling figure to better illustrate this in the future version.

Q4: The paper currently does not mention limitations.

Thank you for this suggestion. We plan to add a limitations paragraph, covering the following points:

1. Dependency on RoPE positional encoding. Our low-rank key compression relies on the pre-RoPE key representation being compressible. While RoPE is used in most state-of-the-art LLMs, the method may require adaptation for models using absolute or learned positional embeddings. We are exploring this generalization as future work.

2. Linear-time sparse KV selection. Our current KV selection requires an $O(N)$ complexity. We view the design of $O(\log N)$-time sparse search methods as a promising direction.

3. SVD-based compression on low-precision hardware. ShadowKV relies on SVD for key cache compression, which is highly efficient on modern GPUs. However, on edge devices with only low-precision compute units (e.g., INT8-only cores), efficient SVD implementation may be challenging. Exploring lightweight alternatives to SVD could improve portability to constrained hardware.

Q5: Generalization to non-RoPE models.

We focus on RoPE-based models because RoPE is the dominant positional encoding scheme in state-of-the-art LLMs (e.g., LLaMA, Qwen, Yi, GLM, etc.). We acknowledge that generalization to other positional embeddings is an interesting direction, and we are currently exploring variants such as ALiBi in ongoing work.

Q6: What if CPU-to-GPU bandwidth is low, or the CPU is slow?

ShadowKV performs well under typical bandwidth conditions and is designed to keep transfer sizes small and overlappable. We clarify that:

• Value cache fetching is PCIe bandwidth-bound. The CPU merely serves as a memory pool for the value cache and is not involved in computation. Hence, CPU speed has negligible impact on performance.
• In our implementation, we use PCIe Gen4 with 32 GB/s bandwidth, which is standard in modern server-grade inference deployments. In practice, the PCIe bandwidth is typically well-matched to the GPU performance. For instance, H200 GPUs are commonly paired with PCIe Gen5 (128 GB/s). Therefore, it is rare to find a high-performance GPU configured with a low-performance PCIe.