KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing

Thanks for your review. We address your concerns accordingly.

C#1: The paper lacks a comparison with other cache-sharing methods.

As described in our related work section, layer-wise KV cache compression is currently rare, and existing methods require training, while ours is training-free, making direct comparisons less reasonable. We are searching for baselines and will consider introducing them in the future version.

C#2: It should consider the scenario when the KV cache is quantized.

We conducted additional experiments using the GPTQ-quantized Llama2-7B-Chat model:

We also find that KVSharer does not significantly increase the model's PPL within a 25% compression rate, further demonstrating its effectiveness. We will include this result in future versions.

C#3: The paper also lacks a scalability analysis, which is crucial for evaluating how well the proposed method performs as model size and complexity increase.

We have validated our method across multiple model families, such as the Llama2 and InternLM series, and on mainstream model sizes ranging from 7B, 13B, and 20B to 70B. The consistent conclusions can demonstrate the effectiveness of our approach.

Q#1: What is the time scalability of the proposed approach? Will the inference time remain acceptable when scaling up to models with over 400 billion parameters? It would be valuable to provide an estimation or analysis to address this concern.

We have validated our method on multiple model families and mainstream model sizes. Our approach has been proven to significantly accelerate inference speed. Conducting experiments on a 400B model exceeds our hardware capacity, requiring 1000GB or more of memory, which is beyond the reach of most researchers. Additionally, 400B models are rarely used in practice, making such demands uncommon. Our existing experiments already demonstrate the effectiveness and stability of our method.

Thanks for your review. We address your concerns accordingly.

C#1: Results show a noticeable performance drop even at low compression rates (e.g., 12.5%, 25%), which may limit the practicality of the method.

Maintaining over 90% of the model's performance with a 25% compression rate is sufficient for many use cases. Additionally, as shown in our case study on page 16, the responses generated by our model are both fluent and knowledgeable, which can meet the requirements of various scenarios.

C#2&Q#1: Lacks an explanation for why sharing dissimilar KV caches yields better performance, leaving an essential aspect of the method's effectiveness rather unclear.

We are working on providing both theoretical and empirical evidence. However, the KVSharer proposed in this manuscript effectively reduces memory usage while maintaining high performance, and its counterintuitive findings offer insights for future model improvements, highlighting our contribution.

C#3: What happens if KVSharer is unable to find C pairs of layers to share KV caches while satisfying the threshold T?

As noted in the footnote of page 6, this phenomenon does not occur when the threshold is set to a reasonable value, such as our recommended 0.5.

Q#2: Why does the memory usage reduction exceed the compression rate? What is the source of the observed increase in generation throughput?

For memory reduction, we speculate that it might be due to PyTorch's underlying mechanisms utilizing fragmented memory more efficiently. Of course, we will conduct a more in-depth analysis. As for inference speed, we suspect that the acceleration could be attributed to reduced memory read/write operations, even though the computation load hasn't decreased. We are also exploring this further.

Thanks for your review. We address your concerns accordingly.

C#1: Lack of novelty and research depth: no motivation provided in the methodology section.

We respectfully disagree with your viewpoint. Our method is derived from experimental observations, as described in Lines 064-067. While we currently cannot provide a theoretical proof, we have validated our observations through extensive experiments across various models. Moreover, many existing works focus on sharing based on parameter or representation similarity. In contrast, our counterintuitive findings offer a more novel perspective.

C#2: Unreasonable observation without further analysis: The observation sounds unreasonable and the obtained conclusion is just a coincidence and varies across the models and datasets.

We also respectfully disagree with your viewpoint. We have obtained consistent conclusions across a wide range of datasets and models, and our results are certainly not coincidental. While we will attempt to conduct deeper analyses in future versions, we do not accept the doubts about our experimental results.

C#3: Lack Needle-in-a-Haystack experiment.

We will include the experiments on related long-context benchmarks in the future version.

Thanks for the comments. I have raised my score and hope the author could revise the paper based on all the reviewers' suggestions in the future. Thank you.

Thanks for your feedback. We will address your concerns as followed:

Q#1: Whether such a highly aggregated metric(the averaged value of the KV-cache) is informative or not.

Since previous work rarely considered layer-level KV cache compression, as you mentioned, they mainly focused on token- or head-level KV cache representations. However, our work requires comparing the similarity between entire layers. To preserve the layer-level KV cache representation for each sample in the calibration dataset as much as possible, it is reasonable to average the layer-level KV cache representations across all samples without bias toward any particular sample. Moreover, this averaging approach is very common and is frequently used in prior work to generate hidden states or heatmaps for attention maps.

C#1: It would be critical to evaluate the performance of the proposed method over long-context benchmarks.

Thank you for your suggestion. We will include experiments on long-context benchmarks in the future version.

Thanks for your comments. We address your concerns as follows.

C#1&Q1: There is a possibility that the cosine similarity check, rather than dis-similarity ranking, plays a major role. Did the authors investigate the relationship between dis-similarity ranking and the acceptance rate by the thresholding condition.

When preparing the manuscript, we have found that KVSharer selects pairs mostly ranked within the top 30% for dissimilarity, confirming it uses dissimilarity for KV cache sharing rather than cosine similarity check. We will include this analysis in the future version.

C#2&Q#2: No theoretical and empirical evidence that "Euclidean distance for KV cache is a sufficient good metrics". Why did the authors choose Euclidean distance? Could the authors ablate on other distance metrics?

We chose to use Euclidean distance based on our experimental findings, and we also compared it with cosine similarity while preparing this manuscript. We conducted experiments on Llama2-7B-Chat, the following are the supplementary results:

The experimental results indicate that when cosine similarity is used instead of Euclidean distance similarity, the observed pattern also remains consistent: leveraging dissimilarity for sharing performs better than using similarity for sharing. Moreover, since models with the same compression rate achieve better perplexity (PPL) when using Euclidean distance for sharing compared to cosine similarity (as shown in Figure 5 of the manuscript), we chose to use Euclidean similarity as the metric. We will include these analyses in the future version.

C#3&Q#3: Provide comparisons with other layer-wise compression strategies.

The methods in [2], [3], and [4] you mentioned all require post-training to be utilized, whereas our KVSharer is training-free, making a direct comparison less necessary. Additionally, [3] and [4] were public after the ICLR submission deadline, so they are not relevant for comparison in this manuscript. Furthermore, [4] is not strictly a layer-wise compression strategy; it focuses on dropping tokens within the KV cache of certain layers. We have already discussed [1] and [2] in the related work section. However, [1] lacks publicly available code, and we are actively working on reproducing it. The results will be included in the future version.

Q#4: Can the authors provide explanations on how the cos-similarity (representation similarity) threshold is determined/searched?

As noted in the footnote of page 6: During strategy searching, the similarity of the last layer's hidden state between the compressed and original models is usually above 0.8. A 0.5 threshold is set to avoid rare cases of output collapse. Since this is infrequent, we did not conduct an ablation study on T. And we recommend setting the threshold around 0.5.

Q#5: Provide guidance/calculation on how the number of target shared KV cache layers to memory saving and inference speedup.

As mentioned in Lines 319-320, we recommend setting the KV cache compression rate to around 25% to maintain good model performance.

For C#1&Q#1 + C#2&Q#2, I believe adding more detailed results and incorporating them into future revisions is important toward making this paper publication-ready.

For C#3 & Q#3: while I understand that some relevant works have been released only recently, it is essential to benchmark the proposed approach against baseline methods that share similar goals of generation acceleration and memory savings through KV cache compression. In addition to benchmarking KVSharer against itself with different compression ratios (and w/ vs. w/o applying other intra-layer compression techniques).

Regarding Q#5, my question was: "How does this number translate into memory savings and inference speedup?" THe reason I asked was because choosing different compression ratios is a trade off between speedup/memory saving vs. performance. I believe the paper would benefit from a more thorough system profiling. Such profiling should illustrate how system performances are affected by varying compression rates.