Compute or Load KV Cache? Why Not Both?

Thank you for your interest in Cake and for providing insightful feedback. We value your constructive comments and are pleased to address your concerns.

In our original experiments, we adopted the inter-token-latency-optimized configuration in vLLM v0.6.2, where the max number of batched tokens is 512. We agree that exploring different batch sizes offers valuable insight into Cake’s adaptability under memory-bound and compute-bound conditions. Below are our newly added experiment results of Cake’s performance gain over the compute-only and I/O-only prefill on different numbers of batched tokens and different model sizes. We will include such results in our revised version.

Table r1. Speedup of Cake over I/O-only \ Compute-only method under different batch sizes of tokens and different models. Hardware: 1xA100, Seq-len: 16k

As shown in Tables r1, Cake achieves an average Time-to-First-Token(TTFT) speedup of 1.96x over I/O-only methods and 2.25x over compute-only methods across various batch sizes. At smaller batch sizes, KV cache computation is memory-bound, underutilizing compute units. As batch size increases, it shifts toward compute-bound, boosting GPU efficiency. Generally, the speedup over I/O-only methods is higher with larger batch sizes, reflecting Cake’s increasing reliance on computation. These results highlight Cake’s ability to adapt to various compute efficiencies caused by different batch sizes, automatically optimizing TTFT.

Thanks again for your suggestion, and we are happy to address any further concerns.

Thank you for your insightful feedback and kind words about our paper. Below, we address your suggestions regarding the interaction of Cake with Prefill-Decode (PD) disaggregation. We will add the following discussion to the revised version.

• Cake is compatible with Prefill-Decode Disaggregation with a minor modification. The typical workflow of Prefill-Decode Disaggregation involves routing a request to a prefill server to generate the KV cache, which is then streamed to a decode server. Cake can fit into this scenario by splitting its bidirectional process across these servers: the prefill server handles computation, generating the KV cache from the sequence’s start and streaming it to the decode server; On the decode server, Cake simultaneously manages two streams—one is the I/O loading stream, fetching the existing KV cache from disks starting from the last tokens, and the other is the streaming process from the prefill server, starting from the first tokens—until the two processes meet in the middle on the decode server, meaning all KV cache is ready for decoding.
• Thank you for your suggestions to provide more details on chunked prefill. We will add more details in the background Section in our revised version.

Thanks for your insightful feedback, and we are glad to address your concerns.

1. Dynamic I/O, workloads, etc. Cake’s parameter-free design allows it to automatically adapt to dynamic conditions, such as fluctuating network bandwidth or GPU performance, as discussed in Part 3 of Section 4. This is a key strength, as shown in Figure 5 of Section 5.6, we conducted an experiment on dynamic computational power and network bandwidth. Under severe fluctuations, Cake is still able to find the optimal merging point automatically and maximize the utilization of dynamic resources. The reason behind it is that Cake does not rely on any offline profiling or any static parameters. The optimal utilization of compute and I/O is achieved by our bidirectional KV cache generation mechanism, demonstrated in Figure 2. Cake starts KV cache computing and loading from opposite directions, chunk by chunk; the two threads run in parallel asynchronously and merge at a certain point in the middle of the context. A sudden network traffic burst may slow down the I/O transfer, then the two threads will merge at a position closer to the end of the context. Similarly, if a temperature spike reduces GPU frequency and slows down the computing progress, the threads will meet at a position closer to the head of the context. The scheduler doesn’t need to change anything caused by a certain period of exceptional condition, it only needs to interrupt the two threads if the last computed chunk includes a loaded token‘s cache and switch request status to the decoding stage. This design ensures that Cake always achieves the optimal Time-To-First-Token as it fully leverages the available I/O and computing capability, even though they are dynamic.
2. Potential application of CPU-GPU loading. Cake can be applied between CPU and GPU with minor adjustments. Yet, there are two reasons why we don’t suggest applying Cake between host memory and GPU. First, as we show in Figure 3, for a context length of 32k, the KV cache computing speed of an H100 is approximately equivalent to 4GB/s I/O transfer. However, H100 supports PCIe 5.0x16, which provides up to 64GB/s host-to-device transfer bandwidth per GPU. This huge gap makes re-computing less efficient, and loading all the cache is a better option. Second, compared with SSD and remote disks, host memory is more expensive, and its storage is limited. As is profiled by AttentionScore, 80% of cache hits happen on the disk level, and we believe optimizing it will bring broader benefits to the modern LLM serving system.

We appreciate your interest in Cake and your insightful reviews. We are happy to address any further questions about Cake.

Thank you for your insightful comments and feedback on our paper. We are pleased to address your concerns regarding the compatibility and performance of our proposed method, Cake, with other acceleration techniques and system-level optimizations. We will add the following discussion to the revised version.

• Compatibility with other acceleration methods. Thanks to the simple and elegant design, Cake is orthogonal and complementary to the broad acceleration methods you mentioned, including speculative decoding, multi-token prediction, KV cache quantization, eviction, and Prefill-Decode disaggregation, making Cake very flexible to deploy. These methods can be categorized into three main classes:

• • Techniques to Accelerate LLM Decoding Throughput:
• • • Methods like speculative decoding and multi-token prediction are focused on optimizing the decoding stage, e.g., by leveraging parallel generation opportunities to address the inefficiencies of autoregressive decoding.
• • • Cake, however, targets the prefill stage, i.e., generating the KV cache for the decode stage. Since the prefill and decoding stages are distinct, there is no conflict between Cake and these decoding-focused methods. They can be used together to enhance overall inference efficiency.
• • KV Cache Size Reduction Methods:
• • • Techniques such as quantization and eviction aim to reduce the memory footprint of the KV cache. Cake regards the KV cache as simple data, and it does not restrict the size of KV cache or how the KV cache is computed. Therefore, these methods can be seamlessly integrated with Cake.
• • • Actually, we discussed the integration of KV cache compression methods in Section 5.5 of our paper, where we evaluated Cake’s performance with compression techniques, showing that they are orthogonal and can complement Cake’s bidirectional scheduling strategy. This synergy could lead to additional reductions in TTFT, though the extent of improvement may vary depending on specific implementations and resource constraints.
• • System-level Acceleration Methods:
• • • Prefill-Decode Disaggregation. Cake is compatible with Prefill-Decode Disaggregation with a minor modification. The typical workflow of Prefill-Decode Disaggregation involves routing a request to a prefill server to generate the KV cache, which is then streamed to a decode server. Cake can fit into this scenario by splitting its bidirectional process across these servers: the prefill server handles computation, generating the KV cache from the sequence’s start and streaming it to the decode server; On the decode server, Cake simultaneously manages two streams—one is the I/O loading stream, fetching the existing KV cache from disks starting from the last tokens, and the other is the streaming process from the prefill server, starting from the first tokens—until the two processes meet in the middle on the decode server, meaning all KV cache is ready for decoding.
• • • Distributed inference. We have evaluated Cake’s performance with tensor-parallelism using two GPUs, as detailed in Table 3 and Table 4 of our paper. The results demonstrate that Cake can effectively utilize the increased computational capabilities provided by multiple GPUs within a node. By distributing the model across GPUs, the computational power available for Cake’s bidirectional scheduling improves, allowing it to optimize TTFT more efficiently. For distributed inference settings, where multiple inference engines are deployed across various nodes in the data center and requests are routed to different engines for processing, Cake can still be applied within each inference engine. Each engine can independently optimize its own inference latency using Cake, ensuring that the method remains effective even in large-scale, distributed environments.

• About mobile devices: Cake is designated for long-context prefix-caching enabled LLM inference systems and is beneficial as long as the simultaneous KV computation and I/O KV cache loading is possible. The enterprise-level data center is the best use case because of its sufficient GPU computational power, GPU memory capacity, and I/O bandwidth. The idea of Cake could also be applied to mobile devices, but the limited computation power and memory space limit the model size or context length, making mobile computing not the ideal use case. Cake’s deployment on mobile devices is promising but needs future work to address engineering challenges.

• Thank you for your careful reading and pointing out the typos. We will correct them in the final version to ensure clarity.