LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

1. "… The latency of API calls may have a high variance. Are the experiments conducted several times, and in different hours? If not, the reviewer suggests improving this and update the evaluation setup."
   Thank you for your suggestion. We appreciate your concern about the variance in API call latency. We want to assure you that the latency reported in the paper is the result of multiple tests. However, to provide a more comprehensive latency comparison, we have added end-to-end latency results of different methods on Multi-document QA, LongBench, and ZeroSCROLLS in Table 1 and Table 2. Our results show that compared to the original prompts, LongLLMLingua can speedup the end-to-end system latency from 1.4x to 2.3x. We hope that these additional results will address your concerns.

1. "… LLMLingua... Citing it with Anonymous authors seems unusual."
   Thank you for your suggestion, we have fixed this issue in the updated version.

2. "It might be beneficial to consolidate the list of contributions into three primary ones."
   Thank you for your suggestion, we will rewrite this section in subsequent versions.

3. "Figure 5 alone seems insufficient to establish the full effectiveness of the framework... Tables 1 and 3 should include E2E runtime data, particularly to offer perspective on the performance of retrieval-based methods."
   Thanks for the insightful feedback. we have incorporated the end-to-end latency results of different methods on Multi-document QA, LongBench, and ZeroSCROLLS in Table 1 and Table 2. The results show that while LongLLMLingua has a slightly higher latency compared to retrieval-based methods like BM25, it still accelerates the end-to-end system latency by 1.4x-2.3x when compared with the original prompts.

4. "'Ground-true' should likely be corrected to 'ground-truth.'"
   Thanks. We have corrected this in the updated version.

1. "the methods contains several aspects... it lacks a detailed ablation study that the influence... from each aspect..."
   Thanks for the helpful suggestions. We have incorporated your suggestions and added detailed ablation study on different datasets in the updated version (Multi-document QA: Table 3; LongBench: Table 6; MuSiQue: Table 5). In general, removing any module in LongLLMLingua such as question-aware coarse-to-fine compression, dynamic compression ratio, document reordering, and subsequence recovery yield performance improvement for LLMLingua across different benchmarks.

2. "For the reordering... whether the reordering will disturb the time-line between each document... whether this operation will disturb or introduce other difficulties for understanding?"
   Thank you for pointing out the potential confusion. Since LLMs tend to be “lost-in-the-middle" as shown in Figure 1b, reordering can help LLMs better capture the most important information and thus benefit the overall performance. In fact, our empirical experiments reveal that LLMs have the ability to understand texts that are organized in an unnatural order. For example, in the timeline reorder task from LooGLE (Table 8), LongLLMLingua outperforms the original prompt by 15.9 points.

3. "Figure3b, ...contrastive perplexity... Would the authors provide more detailed explanation or mathematical proof on the reasons for this?"
   Thank you for the valuable feedback. We added a theoretical derivation of contrastive perplexity in Appendix B. The conclusion is that contrastive perplexity is positively correlated with $p(x^{\text{que}}|x_i, x_{<i})$, signifying the contribution of each token to question $x^{\text{que}}$. The larger the contrastive perplexity, the higher the $p(x^{\text{que}}|x_i, x_{<i})$, the greater the contribution of the token $x_i$, the higher its importance, and the more it should be retained. In addition, we included Figure 5 in Appendix D.1 to illustrate the distribution of document-level contrastive perplexity when the ground-true was placed at different positions (i.e., 1st, 5th, and 15th). It shows that even when key information is not at the beginning of the prompt, using contrastive perplexity can better capture token-level information related to the question.

4. "In the benchmark comparison, it would be better to include the GPT-3.5-Turbo-0613 and LongChat-13B-16k for comparison."
   Thanks for your suggestion. We have added experimental results using GPT-3.5-Turbo-0613(Table 2) and LongChat-13b-16k(Appendix D.5 Table 7) on LongBench. It demonstrates that the compressed prompts achieve higher performance than the original prompts by 4.8 points.

1. "Removing documents and reordering is only relevant in the multi-document scenario... Many use cases (e.g. long-input summarization, code understanding, multi-hop question answering etc.) will not benefit from it and may even be hurt by this procedure."
   Thank you for pointing out the potential confusion. LongLLMLingua is designed as a general coarse-to-fine compression framework that allows us to adjust the granularity of compression based on the task at hand. When dealing with tasks such as long-input summarization where important information is evenly distributed throughout the original text, we can perform coarse-grained compression (i.e., “removing” in the question here) at the paragraph or sentence level rather than document level to improve the key information density. Since LLMs tend to be “lost-in-the-middle", reordering can help LLMs better capture the most important information and thus benefit the overall performance. We believe that the experimental results in long-input summarization (Table 6, LongBench summ.), single-/multi-doc question answering (Table 6, LongBench; Table 8, LooGLE), code understanding (Table 6, LongBench Code), multi-hop question answering (Table 5, MuSiQue), and Timeline reorder (Table 8, LooGLE) well demonstrate the effectiveness of LongLLMLingua. We have clarified this in the revised version.

2. "Subsequence recovery is only relevant in... extractive question answering. However, many real-world use cases... are not extractive rather generative."
   Yes, the benefit of subsequence recovery in certain generative scenarios is a little bit limited than it in extractive tasks. However, in most cases, the generated content may refer a lot to texts in the prompt in terms of entities or phrases, and subsequence recovery can effectively mitigate the information loss here. We conduct ablation study of subsequence recovery on diverse tasks for your information. As shown in Table 6, removing subsequence recovery would lead to a performance drop in Long Bench tasks “SingleDoc (QA)”, “muiltiDoc (QA)”, “Summ. (summarization)”, and “FewShot”.

3. "question-aware compression is only relevant when a question (or more generally, a long and specific instruction) is given. In summarization, or conditional generation this is not the case."
   The question-aware module, both coarse-grained and fine-grained, is designed for aware the changing of information density within the prompt. While it is crucial for tasks that involve following specific instructions (e.g., removing question-awareness would lead to a performance drop up to 35.1 points on NaturalQuestions as in Table 3), it can also be beneficial for summarization tasks and other tasks including multi-hop QA task. For example, Table 6 shows that in the LongBench summarization task, dropping question-awareness does lead to performance drop ~1-2points. And Table 5 shows that in the MuSiQue multi-hop QA, dropping question-awareness does lead to performance drop ~9 points. This suggests that question-awareness such as “please summarize this article” also provide valuable global context for the SLM, both coarse-grained and fine-grained, and have an implicit influence on the perplexity distribution during compression.

4. "the above specifications limit the contribution of the method when contrasting with the existing LLMLingua method" We hope the above responses have already addressed your concerns. The large, consistent performance gain from LLMLingua to LongLLMLingua across various tasks (as shown in Table 1-5 and 7, 8) also well demonstrates the contribution of LongLLMLingua.

5. "the evaluation on ZeroSCROLLS is unsatisfactory, ...the evaluation set contains a small number of examples (~20 per task)... does not have enough statistical power…not provide a breakdown of the results to show that their method is indeed beneficial across the different scenarios."
   Thanks for the suggestions. We use the validate set of ZeroSCROLLS for evaluation because the ground-true test set is inaccessible. We've added the breakdown results of ZeroSCROLLS in Appendix D.2 (Table 4). We further evaluate LongLLMLingua on MuSiQue (multi-hop QA, ~4.8k examples, Table 5) and LooGLE (long dependency QA, ~1.1k examples, Table 8). Experimental results show that compared with the original prompts, LongLLMlingua compressed prompts achieve comparable results on ZeroSCROLLS and outperform the original prompts by a large margin on MuSiQue and LooGLE, well validating its effectiveness in a wide range of scenarios.

6. "The paper also considers several changes as significant contributions, but there are no ablation studies to show their usefulness... An ablation on the ordering of the question and document should be added, as well as for the relevance of the restricting prompt. Moreover, all ablation studies that were performed were done on the NaturalQuestions... I would like to see the ablation study conducted on multi-document scenarios... such as the multi-hop question answering scenario (e.g. MuSiQue [Trivedi et al., 2021]..."
   Thank you for your valuable suggestions. We have incorporated your suggestions and included an ablation study of reordering and restrict prompt on different datasets in the updated version (Multi-document QA: Table 3; LongBench: Table 6; MuSiQue: Table 5). In general, both reordering and the restrict prompt yield improvements for LongLLMLingua across different benchmarks.

7. "The authors... use LLaMA-2-7B-chat as their small LM... An ablation study on the performance of using smaller models should be added."
   Thanks for your feedback. we've added results with GPT2-small (using GPT2-dolly as the small LM) to Table 3, Table 5, and Table 6. The results show that GPT2-small compressed prompts can derive comparable even higher performance than the original prompts.

8. "Typos"
   Thanks for your thorough review. We have fixed these issues in the updated version.

9. "...for the distribution alignment... Was this finetuning done separately for each test case? If so, in the NaturalQuestions case, were the answer positions considered a specific case? Please provide more details on the finetuning procedure."
   Thanks for bringing up the misunderstanding. As mentioned in the section of "Implement details", for all experiments, we use llama2-7b-chat as the aligned version without any further fine-tuning.

10. "In page 5 §4.1... this experiment was only done in the case where the ground-truth document is in the first position? If so, isn’t it possible that the fact the contrastive learning approaches zero as the token position increases is simply a symptom of the “short-term memory” of perplexity...?"
    Thank you for the valuable comments. We would like to clarify that the distribution of contrastive perplexity is not caused by the diminishing effect. We have added a theoretical derivation of contrastive perplexity in Appendix B. The conclusion is that contrastive perplexity is positively correlated with $p(x^{\text{que}}|x_i, x_{<i})$, signifying the contribution of each token to question $x^{\text{que}}$. The larger the contrastive perplexity, the higher the $p(x^{\text{que}}|x_i, x_{<i})$, the greater the contribution of the token $x_i$, the higher its importance, and the more it should be retained. In addition, we include Figure 5 in Appendix D.1 to illustrate the distribution of document-level contrastive perplexity when the ground-true was placed at different positions (i.e., 1st, 5th, and 15th). It shows that even when key information is not at the beginning of the prompt, using contrastive perplexity can better capture token-level information related to the question.

11. "Can you please provide a breakdown on the ZeroSCROLLS test set benchmark?... would be helpful to compare your results with the baseline results..."
    We have added a breakdown of ZeroSCROLLS with the baseline results in Appendix D.2. In general, compared with the original results, LongLLMLingua achieving slightly higher numbers at 3x compression and maintaining the same level of performance at 6x compression.

1. "...tasks, like summarization, the key information can be evenly distributed within inputs. Coarsely dropping a large amount of input may hurt the performance."
   Thank you for bringing up this point. LongLLMLingua is designed as a coarse-to-fine compression framework that allows us to adjust the granularity of compression based on the task at hand. For instance, when dealing with tasks where important information is evenly distributed throughout the original text, we can perform coarse-grained compression at the paragraph or sentence level. Additionally, the framework also allows us to adjust the proportion of content to be dropped during coarse-grained and fine-grained compression. For example, in summarization tasks, we can assign a smaller ratio to coarse-grained compression and a higher ratio to fine-grained compression to avoid dropping too much input during coarse-grained compression. We believe that our experiments in Table 2 (“summ”., i.e., summarization) provide evidence for the effectiveness of LongLLMLingua on tasks such as summarization. We have clarified this in the revised version.

2. " The inference latency can be improved by LongLLMLingua. The models need to evaluate the perplexity and compress prompts every time, which leads to non-trivial latency."
   Thank you for pointing out the potential confusion. Though LongLLMLingua evaluates perplexities every time which may cause some latency, the compression is done by a much smaller language model Llama-7B. Comparing to the latency caused by LLM inferencing on the compressed prompt, LongLLMLingua can still speedup the end-to-end system latency from 1.4x as shown in Table 1 to 2.3x as shown in Table 2 (Figure 5 in previous version).

3. " How is the thresholding of $s_i$ determined to discard tokens of lower importance? Is it determined by the current budget $\tau_k$ of the documents?"
   Yes, we discard tokens of lower importance, i.e., with $s_i$ smaller than the threshold, and the threshold is determined by the current budget $\tau_k$ of the documents.

4. "...Is there an iterative evaluation of token importance?"
   Thanks for the comments. We evaluate token importance in segment-level by iteratively conditioned on all previous compressed segments and calculate document-level threshold based on Eq. 5. We will add clarification in the final version.

5. "If the perplexity varies dramatically across tokens of one sentence... making the sentence inconsistent"
   Thanks for the question. Token perplexities do vary a lot within one sentence. Though removing tokens can render the sentence inconsistent and unreadable to human, it doesn’t appear to affect the ability of language models to comprehend the prompt. This phenomenon has been studied in Figure 4 of LLMLingua paper[1], which demonstrates that GPT-4 is capable of recovering the original details from the compressed, non-human-friendly prompt.

[1] LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. https://arxiv.org/abs/2310.05736