Preserving Large Activations: The Key to KV Cache Pruning

The weakness of paper to me are the following:

1. Lack of comparison to prior works in theory. What is fundamentally new about your pruning by activation? Are there something that prevents the prior work from working in the GQA scenario? If there is, I think you should add more in 2.3.
2. Evaluation not rigorous enough (a) You only showed two compression levels. However, there is a huge gap between 256 and 2048. Ideally you want to show a range of compression levels. Also as a side note, you showed 256 and 2048 on table 1 and 128 + 2048 in later tables, what is the rationale? (b) Not sure if the method scales to bigger models. You only showed improvement over small models (<10B). I am not sure whether this works for larger models. (c) You accuracy has dropped significantly compared with FKV on a lot of cases. Will using a smaller model (llama 3.2) outperform running larger model with your compression? 3 (less important). Little explanation on why your method works. The current S1 and S2 calculation seems a little arbitrary.

Insufficient Theoretical Justification: The paper lacks rigorous theoretical analysis to support the claims made. For instance, the importance of large activations in the Value cache (Section 2.2) is asserted without mathematical proofs or empirical studies to substantiate it. For Example, Could the authors provide More detailed theoretical explanation or empirical evidence for why preserving large activations in the Value cache is crucial for maintaining model performance during pruning (e.g. an ablation study comparing performance with and without preserving large Value cache activations)

Methodological Ambiguity: The methodology in Section 3 needs further explanation. For instance, 1.why summation was chosen for S1 and max for S2, 2.why multiplication was used to combine S1 and S2 in Equation 5 providing more rigorous evidence in supporting the approach as stated in Theoretical Justification(see above) 3.Equation (3) computes the importance score S1 by summing attention weights over heads and queries within an observation window. However, it's not clearly justified how this aggregation effectively identifies important Key cache positions. 4.Equation (4) calculates S2 using a max operation on the absolute values of the Value cache, but the rationale behind choosing this operation providing more rigorous evidence method

Presentation Issues: Numerous grammatical errors, typos, and formatting problems make the paper difficult to read and understand such as; 1. Grammatical Errors and Typos: Grammatical errors and typos should be fixed. For example, in the abstract and introduction, phrases like "As context lenghts grows," and "thus sustain large activations in KV cache is essential" 2. Clarity of Explanations: Key concepts and methodologies needs explanation with sufficient clarity. In Section 3.1, the process of grouped-wise recognition of large activations is described in a convoluted manner, making it hard to follow. The pseudo-code in Listing 1 lacks comments and detailed explanations of variables, reducing its utility. 3.Inconsistencies: Making the terminology consistent , such as "Observation Window" and "observing window," and sometimes mixes notations (e.g., using both "KV cache" and "KVs"). Additionally, some sections refer to methods or concepts not previously introduced.

Incomplete Experimental Comparisons: The experiments do not include comparisons with methods that might also consider Value cache activations(e.g. Ada-KV(https://arxiv.org/abs/2407.11550v3) , FastGen (https://arxiv.org/abs/2310.01801), SparQ Attention (https://arxiv.org/abs/2312.04985), or the other activation based eviction strategies mentioned in https://arxiv.org/abs/2407.18003v1.) otherwise consider adding evidence of incompatibility of existing methods with GQA architectures.

Limited Discussion on Computational Overhead: The paper does not discuss the computational cost associated with SlimKV. Consider adding analysis of how the additional computations for importance scoring and clustering affect inference time or resource usage. Inadequate Ablation Studies: The ablation study in Section 4.6 is limited to evaluating the impact of removing S1 or S2. It does not explore other aspects such as sensitivity to hyperparameters (e.g., window size, kernel size) or the effect of different clustering strategies.

Reproducibility Concerns: There is no mention of code availability or detailed experimental settings, which could hinder the reproducibility of the results.

1. The paper does not add enough novelty for ICLR-like submissions.
2. The paper does not discuss the latency improvements and its comparison with baselines.
3. The paper does not clearly show KV-cache reduction (like that of shown in H2O) and accuracy comparison. Hence, It is difficult to understand the efficacy of this approach.
4. The paper does not discuss the overhead of their method in the overall execution.
5. Reading/recomputing attention scores defeats the purpose of using flashattention/flashdecoding like methods to significantly improve the overall throughput. This paper does not discuss how they address such challenges/limitations.
6. In Listing 1, are there golden values across different models and datasets for window_size and max_capacity_prompt? If no, then what is overhead of optimizing these hyperparameters during inference?
7. Performance from fullKV in some benchmarks is too poor. In fact, in most cases SlimKV's performance is very close (~0.5%) to that of other methods.

• The paper fails to explain the variance of the V cache and its importance in evaluating V activations. • The benchmarks are limited, and the discussion on the observation window is missing.