SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator

We would like to sincerely thank the reviewer for the valuable comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

Q1. The authors do not clearly explain or experimentally justify how these separator tokens were chosen.

A1. We take the tokens in the 300B natural language dataset that exceed a certain word frequency threshold as separators, which are [“.”, “,”, “?”, “!”, “;”, “:”, “ ”, “\t”, “\n”].

Q2. The size of the neighboring tokens must be manually selected, requiring a trade-off between efficiency and performance.

A2. In our experiments, the size of neighboring tokens is set to be $m/8$ ~ $m/4$, where $m$ denotes the input sequence length. We find that the performance is still closed to that of full attention in these settings. While the size of neighboring tokens is set to be $m/2$ in StreamingLLM to maintain a similar performance.

Q3. The justification for the chosen set of separator tokens is not fully detailed, raising questions about the robustness of this selection across different tasks or languages.

A3. Indeed, we use the same set of separators in all the experiments, including using lama-3-8B, Pythia-6.9B, Falcon-40B models and conducted on GSM8K, ARC, and MMLU datasets.

Q4. The experiments are largely limited to benchmarks like GSM8K-CoT and MMLU, leaving uncertainty about the method’s effectiveness on more complex, reasoning-intensive tasks.

A4.

As suggested, we evaluate our SepLLM on MATH, which is a more complex benchmark, and the results are as follows. It justifies the superiority of SepLLM in complex tasks.

Q5. In the Related Work, after discussing the shortcomings of existing KV cache compression methods, The paper does not explicitly highlight how its approach overcomes these limitations.

A5. As discussed in Section 1 of our submitted paper, the existing KV cache compression methods are training-free approaches, which is inconsistent with training phase, resulting in approximation error.

Q6. How did you select the fixed set of separator tokens, and have you evaluated alternative sets?

A6. For the selection confusion, please refer to Q1. As suggested, we evaluate our SepLLM with subsets of our previous setting on GSM8K-CoT, which are ["," "." "?" ";"] and ["." "?"]. The experimental results are as follows. It shows that performance drops <0.5 when removing half of the separators. So SepLLM is robust against the separator selection.

Q7. Can you provide insights into how the size of neighboring tokens (n) might be determined adaptively?

A7. As discussed in A2, we recommend the size of neighboring tokens to be $m/8$~$m/8$, where $m$ denotes the input sequence length.

Q8. How does SepLLM perform on more challenging, reasoning-intensive tasks beyond GSM8K-CoT and MMLU?

A8. Please refer to Q4.

We would like to sincerely thank the reviewer for the valuable comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

Should there be any need for further clarification to assist in advancing our scores, please do not hesitate to inform us. Thank you very much.

Q1. "Our positional encoding strategy for streaming settings is the same as the state-of-the-art StreamingLLM" more information on the exact technique would have been helpful, as the correct positional encoding seems to inspire the design a lot. Is the technique used the same as ROPE? Better?

A1. We just adopt RoPE. However, in scenarios involving infinitely long streaming input, we follow StreamingLLM[1] and apply PE's shifting; i.e., we focus on positions within the cache instead of those in the original text. For instance, if the current cache contains tokens [0, 1, 2, 3, 6, 7, 8] and is in the process of decoding the 9th token, the positions assigned are [0, 1, 2, 3, 4, 5, 6, 7], rather than the positions in the original text, which would be [0, 1, 2, 3, 6, 7, 8, 9]. However, for training and downstream tasks of general length (<4K; Sec 3.1), we simply use standard RoPE without PE's shifting. See Sec. 3.3;4.6 and Tab. 8 for details.

Q2. Ablation studies show the importance of additional tokens, but it seems a better idea would have been to show the effect of separator tokens as that is the main claim for improvement.

A2. Indeed, the ablation studies you propose are illustrated in Tab.1-8 and Fig. 5 (for both training & training-free). Without separators, SepLLM degrades to StreamingLLM. As demonstrated by the experimental results across so many settings and tasks, the separators indeed provide significant benefits to the architecture design. See more studies on separators in A6 to Reviewer kB2E.

Q3. How does SepLLM compare against well known KVCache compression techniques such as H2O and SnapKV?

A3. As suggested, we add the H2O, SnapKV and PyramidKV [2] as baselines and the results on GSM8K_CoT and the challenging MATH are as follows (All based on Llama3-8B-Inst backbones).

With similar amount of KV cache, SepLLM can achieve better performance.

H2O and SnapKV are training-free methods that rely on full attention for pretraining and prefilled KV, but their inference-stage KV selection causes training-inference inconsistency and query dependency. SepLLM addresses these issues by introducing a novel language modeling paradigm that uses separators for segment infomation summarization, optimizing from the pretraining to the inference. Additionally, SepLLM naturally aligns with the hierarchical semantic structure of language (e.g., "," for segments, "." for sentences, \n for paragraphs) and reduces KV usage during inference.

Q4. One issue in the ablation studies, the claim of better retrieval was introduced but no result was shown in the main body of the paper.

A4. Indeed, we have already conducted needle in a haystack experiments and the results are shown in Fig. 8-11 & App. E, which demonstrate the retrieval ability of SepLLM.

Q5. Comparison to popular KVCache compression methods such as H20 (Neurips '23) and SnapKV (Arxiv'24)

A5. Please refer to Q3.

Q6. I saw a claim in the introduction that training-free methods are suboptimal but no evidence has been presented in the paper.

A6. As discussed in Q3, training-free methods are based on KV selection, an approximation to full attention, leading to inconsistency between training and inference phases.

Tab.13 can support our claim. Based on the well-trained Llama3-8B-Inst model, we fine-tuned for only 200 steps on the LongAlpaca dataset, and the results outperformed the original training-free SepLLM and even surpassed Vanilla.

[1] Xiao, Guangxuan, et al. "Efficient streaming language models with attention sinks." ICLR 2024.
[2] Cai, Zefan, et al. "Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling." preprint arXiv:2406.02069 (2024).

We would like to sincerely thank the reviewer for the valuable comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

Should there be any need for further clarification to assist in advancing our scores, please do not hesitate to inform us. Thank you very much.

Q1. important evaluations such as long ppl and "needle in a haystack"

A1. We have provided numerous results on perplexity (ppl) for extremely long test datasets. For instance, based on the Llama3-8B model and the PG19 test set, we conducted inference on extremely long sequences of up to 4 million tokens. The results are presented in Table 4 of our paper, which we have copied here for your reference.

Additionally, we have also conducted experiments on Needle In A Haystack, with results and detailed discussions presented in Figure 8-11 of our submitted paper. The experimental results show the long-context retrivie capacity of SepLLM.

Q2. retained one token for every fixed number of tokens

A2. As suggested, we conducted such experiments and the results are shown in the table below.

Here, FixLLM represents the variant you mentioned, where $n$ still denotes the number of neighboring tokens, and every $\Delta$-th tokens are also remained outside the neighboring window. Experimental results indicate that FixLLM is still inferior to SepLLM.

Q3. generalization issue as separators may vary across different languages or datasets.

A3. In our perspectives, separators are general across different languages or datasets. In modern commonly used languages, separators are widely present (including but not limited to English, German, French, Italian, Chinese, and etc, covering over 7 Billion people). While most languages use separators similar to modern English, a few languages still retain the usage of their traditional separators. For example, in Bengali, the period (".") is commonly used as a sentence-ending marker, but the traditional sentence-ending marker "।" (vertical line) is still in use. During training and inference, we only need to specify the separators we want to use, and the commonly used separators in most languages are around 10 in number.

Emprically, we have verified the robustness of our SepLLM across various tasks or datasets, including GSM8K, MMLU, ARC, PIQA and so on. All emprical results can demonstrate the generalization of SepLLM as well.

Q4. low entropy within chunks and high entropy between chunks

A4. We agree that entropy-based grouping is a feasiable apporach. However, entropy-based grouping is not training/inference efficient since we need to calculate the attention mask by some algorithms. In comparison, SepLLM follows the natural modeling of the language and split the segments by separators, which is more efficient and more robust.

Q5. some heads are converted to sliding window attention while allocating the remaining computation to a full window attention? This would represent a standard hybrid baseline.

A5. As suggested, we added the following experiments.

For example, "20/32" means there are 20 sliding-window heads out of a total of 32 heads (the remaining heads using full attention). As can be seen, with similar amount of KV cache, SepLLM is much better than these hybrid baselines.

Q6. any methods to further improve the KV compression rate of SepLLM

A6. One possible method may be the multi-level version of SepLLM. In the current version, we compress the "segment-level" into general seperators. In the multi-level version, we may define "sentence-level" or "paragraph-level" as a higher-level seperation and compress a set of previous seperators into one more condensed KV Cache. Then, the KV compression rate will be improved.