SCBench: A KV Cache-Centric Analysis of Long-Context Methods

Thanks for your insightful feedback.

1. "how hyper-parameters impact the performance"

It's a great idea. We have supplemented the performance of different efficient methods under various compression budgets on Llama-3.1-8B, as shown in Figure 6 and 7 in updated paper. From the results, we can derive the following insights:

1. Most methods show minimal performance degradation at a 1/2 budget (e.g., A-shape and Tri-shape drop by 5-6 points, SnapKV drops by 11 points). However, as sparsity increases, performance declines significantly. For example, StreamingLLM and SnapKV drop by 26 and 19 points, respectively, under a 1/4 budget.
2. More accurate sparse methods can maintain performance even under higher sparsity. For instance, MInference achieves performance at a 1/32 budget comparable to A-shape and Tri-shape at a 1/4 budget.
3. While some methods exhibit similar performance in single-turn scenarios, they diverge significantly in multi-turn and multi-request scenarios. For example, SnapKV outperforms StreamingLLM in turn-1 but performs significantly worse in turn-2. In some tasks, changing the budget has little impact on turn-1 performance but substantially affects turn-2 and subsequent turns, such as in Long Document QA tasks and summarization.

We will include this analysis in the main text and highlight it in the next version.

2. "bold and underscore in Table 4"

Apologies for the misunderstanding. The bold text highlights the best result among the long-context efficient methods, excluding full attention. We will clarify this explicitly in the table header to avoid any confusion.

Thanks for your effort in reviewing our paper and for recognizing our work.

1. "...list the source of the dataset for each task..."

Thanks for pointing out! We have furhter clarify the source of dataset we used for each task in Table 3. To provide a better recognition for non-original dataset used in SharedContextBench, we also add citations directly in Table 3.

2. "...multi-run scenario?..."

Yes, although our work primarily focuses on evaluating the accuracy of various long-context efficient methods in KV cache reuse scenarios, it can also be applied to multi-run scenarios in system testing. For instance, it is suitable for evaluating prefix caching methods within LLM inference frameworks like SGLang or vLLM.

1. "It is benefical to analyze other other long-context benchmarks"

Thanks for your suggestion. We have added a new table (RTable 2) to provide a more direct comparison of our proposed SharedContextBench against existing long-context benchmarks in terms of long-context capability, request mode, and implementation, highlighting the novel contributions of our benchmark. We have also provided new experimental results of previous benchmarks such as LongBench and InfiniteBench and contrasted them directly with our proposed SharedContextBench, as shown in RTable 3. This will help readers better understand the unique insights our benchmark provides, which prior benchmarks have overlooked:

1. SharedContextBench offers better differentiation between methods, even in summarization tasks;
2. The limitations of KV cache compression methods in handling multi-request and multi-turn modes on SharedContextBench;
3. The improved retrieval accuracy of sparse attention mechanisms, such as A-shape and Tri-shape attention.

RTable 2. Comparison of Long-Context Benchmarks.

RTable 3 (a). Comparison of efficient long-context methods on the summarization task of prior benchmarks and ours.

RTable 3 (b). Comparison of efficient long-context methods on the summarization task of prior benchmarks and ours.

Thanks for your thoughtful reading and constructive comments for our paper. To respond your questions:

1. "...prefix-caching method no tested in the experiments..."

Thank you for your question. Our work focuses on analyzing the accuracy differences of long-context efficient methods in KV cache reuse scenarios. All results presented in our paper are conducted with prefix caching in two kv cache reuse modes. Specifically, we implemented a multi-request KV cache reuse codebase, which we plan to open-source after the review process.

As for the related works on prefix caching methods, they primarily focus on different algorithm such as prefix trees or hash functions to reduce prefix matching overhead and latency. These approaches are orthogonal to the long-context efficient methods evaluated in our paper.
We will include this discussion in our paper to clarify the distinction and prevent any potential misunderstanding.

2. "...explicitly highlight novel insights compared to prior benchmarks..."

Thank you for your suggestion. We have discussed the unique insights of our benchmark in Section 4, which are summarized as follows:

1. sub-O(n) methods such as KV Cache compression achieve misleading high performance on prior benchmarks, but failed in multi-request and multi-turn scenarios which is crucial in real applications; 2) sparse attention especially A-shape attention perform worse on prior benchmarks, but we found it quite robust in multi-turn interactions. This provide insights for further optimization on sparse attention, and motivates us to the new Tri-shape attention.
2. performance drops in multi-turn scenarios: While some efficient methods perform well in single-turn settings like SnapKV, their performance significantly drops in multi-turn interactions. For example, SnapKV and StreamingLLM show substantial degradation in the Zh.QA task, demonstrating that our benchmark more accurately reflects real-world long-context capabilities.

Additionally, in Section 4, we have highlighted other unique insights provided by our benchmark. These insights are instrumental in guiding the design and deployment of efficient long-context methods for practical applications.

3. "...the difference between synthesized data and real life use cases..."

We agree that the synthesized test may be different from real life use cases. However, synthetic test data can be a more efficient way to assess the full length of LLM's long context window. In contrast, it is rather challenging to collecte real test cases that covers the full length of a 200K-length context. In addition, our benchmark also provide many realistic tasks including multi-document summarization, novel QA, and code repo understanding, which align well with real life applications. But we believe it's necessary to include this discussion in our paper. We have put this in the appendix A.

4. "...motivation of the proposed Tri-shape..."

This is a great question. Actually, A-shape’s failure on the first turn and its success on the follow-up queries motivate us to realize the importance of dense computation before the answer generation. That’s what leads us to provide a dense bottom under A-shape’s sparse attention matrix (to form a Tri-shape pattern) that enables the model to shift from sparse to dense computing just before the generation phase. This change closes the performance gap specifically in the first turn, and it’s very important as this improvement benefit the instruction following ability as shown in Table 14. Although Tri-shape only marginally improves turn-1 performance compared to A-shape, it is both simple and effective. We believe the corresponding insights it provides will inspire more innovative approaches in future studies.

Thank you for your effort in reviewing our paper. However, we believe there are some significant misunderstandings about our work. The concerns raised in your review extend beyond the intended scope of this paper. Specifically:

1. Focus on Benchmark Design: Our work primarily focuses on introducing a novel long-context benchmark in KV cache reuse scenarios. By constructing this benchmark, we aim to better capture the limitations and future directions of various long-context efficient methods in real-world applications. The limitations of existing methods, such as KV cache compression, should not be considered weaknesses of our work. Instead, our benchmark serves to reveal these limitations in a systematic and practical way.

2. Beyond the Scope: Some of the points raised, such as "the deeper internal mechanisms of the models" and concerns that the benchmark "may not capture all possible real-world application scenarios," go beyond the intended scope of our paper. Our goal is not to exhaustively cover all possible scenarios but to provide a focused and impactful contribution to the evaluation of long-context methods in KV cache reuse scenarios.

Nonetheless, we appreciate your comments and will endeavor to address your concerns and questions in the following to the best of our ability.

1. "...KV cache compression, demonstrate limited effectiveness in shared context..."

We would like to clarify that our contribution is not any specific KV Cache compression methods that assessed in our paper. Instead, our target is exactly to reveal the limitation of KV Cache compression methods under KV Cache reuse scenarios, that is overlooked by prior research. Therefore, this phenomena should in fact be considerd as our contribution, instead of weakness.

2. _"...StreamingLLM etc are essentially extensions of exiting models..."

Again, this is actually one of the key findings revealed by our proposed SharedContextBench. Our benchmark discovered the weakness of these efficient long-context methods that prevent them to be used together with KV Cache reuse, and we further analyzed the potential origin of these weakness to shed light on future optimization direction. So this should not be considered as our weakness.

3. "...the benchmark fails to conver all possible real-world application, overlooking certain domains knowledge..."

We would like to clarify that it is infeasible for any benchmark to cover all possible real-world application. And the target of our benchmark is not to assess any domain specific knowledg. Instead, our benchmark is specifically to assess the capability and information loss of efficient long-context methods under KV Cache reuse scenarios. So we believe covering the above-mentioned content is out of the scope of our paper, therefore this should not be considered as a weakness of our paper.

4. "...no explore on the deeper internal mechanisms..."

We have provided extensive analysis in Section 4 (page 7,8,9) that pointed out the potential limitation of exiting efficient long-context methods and the origin of their weaknesses on KV Cache reuse. However, further explore the internal mechanisms of any methods covered by our paper is out of the scope of this paper.

5. "...influenced the design of the benchmark tasks selected..."

In SharedContextBench, we designed 12 tasks across four distinct long-context capabilities, each implemented with two KV cache reuse modes. These modes and tasks were carefully designed to cover diverse scenarios and abilities, including multi-document processing, code, math, ICL, and summarization.
To provide a more intuitive representation of performance across different long-context capabilities, we calculate the final metric as the average of the mean performance across the four task categories.

6. "...possible improvements in sub-linear memory methods to enhance multi-turn interactions?..."

We have discuss this question in our Section 4 in detail. We found KV cache offloading methods and Mamba-Attenton hybrid architecture are promising to realize decent performance and efficiency at the same time.

1. "...highlight the source and origin..."

Thank you for the suggestion. We have already indicated the source datasets in Table 3. Out of the 12 datasets, three are from InfiniteBench, one from RepoQA, one from RULER, one from ManyshotICL, one for Lost in the Middle, and the remainding 5 were developed originally by us. To better acknowledge the non-original datasets used in SharedContextBench, we have added direct citations in Table 3 in the updated paper (also show in RTable 1). We would like to emphasize that although some data originated from other datasets, these previous datasets only provided single-turn data. We have regenerated and developed corresponding multi-request datasets.

RTable 1. The source of SharedContextBench.

2. "...question demands information from a previous QA..."

This is a great suggestion. Our SharedContextBench primarily focuses on shared long-context scenarios at the moment, as this represents the major bottleneck in long-context evaluation and is where efficient long-context methods typically employ lossy compression. We agree that adding questions that depend on previous interactions is an excellent idea and can largely enhance the benchmark. We will include this point as a limitation in our current work and mark it as a future direction for improvement.

3. ...discussion regarding other newly proposed long context benchmarks..._

Thank you for your suggestion! We have added a comparison table with other long-context benchmarks in RTable 2. In summary, existing long-context benchmarks all covers multiple dimension of long-context capabilities, while introducing their own novel tasks. For instance, RULER introduces Multi-Key/Value NIAH and Multi-hop Tracing tasks, HELMET proposes Rerank and Cite tasks, and Michelangelo includes Latent List, MRCR, and IDK tasks. These contributions are both essential and impactful.

In SharedContextBench, we primarily focus on the KV cache reuse scenarios, which is prevalent in real-world long-context applications and overlooked by prior benchmarks. To address this, we introduced two novel test modes that require KV cache reuse in SharedContextBench. We also introduced novel tasks including multi-tasking evaluation and Retrieval.Prefix-Suffix to further explore long-context capabilities.

In addition, to provide a more comprehensive assessment of long-context LLM capabilities across different dimentions, we classify our tasks into four types of capability: string retrieval, semantic retrieval, global information processing, and multi-tasking. In future updates to SharedContextBench, we aim to incorporate novel tasks from other long-context benchmarks to further enhance its utility and coverage.

RTable 2. Comparison of Long-Context Benchmarks.