RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation

Thank you for your review and we are encouraged to see that the reviewer found our work to contain extensive experiment and thorough analysis! Please see our response below:

[W1] Regarding baseline methods for extractive compressor, we considered off-the-shelf passage rankers such as DPR and contriever. Our method finetunes contriever to optimize the end performance of LM, showing gains over initial, non-fine tuned checkpoint of contriever. We are the first to study compressing retrieved documents for LMs and propose methods to train such compressors leveraging downstream LM performance. We would like to clarify that we do not claim that our proposed method is the best way to train such compressors, and future work can build upon our method to improve performance of the compressors.

[W2] We thank the reviewer for the suggestion on testing our method on multiple retrievers. We present new results on applying our compressor (trained with contriever-ms-marco retrieved documents) on (1) contriever retrieved documents (Table 1) and (2) DPR retrieved documents on NQ. (Table 2).

We see that both our extractive and abstractive compressor generalize robustly to documents retrieved by another retrieval system (contriever-ms-marco to contriever/DPR), with the best compressor outperforming prepending top 1 documents with 4x compression.

Table 1: RECOMP results on NQ with contriever retrieved docs (FLAN-UL2)

Table 2: RECOMP results on NQ with DPR retrieved docs (FLAN-UL2)

[W3] Regarding inference time, our experiments on NQ shows that both our extractive and abstractive compressor enables 2x throughput. Please see the general response for more detail.

[Q1] Regarding annotation agreement, two of the co-authors annotated faithfulness and comprehensiveness of summaries generated by GPT-3.5-turbo and our t5 compressor for ten questions. The annotators reach an agreement of 75% for factuality and 85% for the comprehensiveness evaluation (both are three-way classification). We will include more agreement statistics in the final paper.

Thank you for your review and we are glad to see that you found our proposed method to be important, novel and effective! Please see our response below:

[W1, Q1] Regarding inference speed, our experiments on NQ shows that both our extractive and abstractive compressor enables 2x throughput. Please see the general response for more detail.

[W2, Q2] Regarding performance improvement of RECOMP, the reviewer is right that our compressor doesn’t outperform prepending full top 5 documents in most of the settings. However, we would like to highlight that (1) the oracle abstractive & extractive compressor outperform no compression in all of the setting, demonstrating that compressors can outperform no compression. As the first work to study compressing retrieved documents, we hope our work inspires future work to propose better methods for training compressors, hence closing the gap. (2) Though not outperforming prepending top 5 documents, our method does outperform prepending top 1 document while prepending less token for both NQ and TQA, this demonstrates that RECOMP allows the model to consider more context, while prepending less tokens, thereby reducing computational cost.

[Q3] As the reviewer mentioned, one advantage of RECOMP is that the model can consider more information that does not fit into the context window. Although we didn’t stress test the model to take a large amount of retrieved documents into consideration, our experiments do show that our compressor, which considers top 5 documents, outperforms prepending top 1 document, while prepending fewer tokens.

Thank you for your review and we are encouraged to see that the reviewer found our work to be innovative and well executed! Please see our response below:

[W1] Regarding limited performance improvement of the compressed context, we would like to highlight that our oracle methods (both extractive & abstractive, in table 1 & table 2) significantly outperformed the normal RALM (without compression). This suggests that compressed documents can outperform normal RALM. Since we are the first to study compressing retrieved documents for LMs, we believe future work can look into how to improve the performance of the compressors, using other training objectives such as reinforcement learning.

[W2] Regarding whether RECOMP is needed with long context LMs, we would like to point out that despite the effort on extending the context window of the language model, recent work [0] has shown that long-context models have difficulty processing information buried in the middle, even though they fit in the context; and that they could be easily distracted by irrelevant context [1]. Thus, RECOMP can not only fit more information to the context, but also present the most relevant information to the LM. Our analysis in Table 3, showing that RECOMP allows the model to extract the right answer more frequently when they are in the context.

[Q1] Regarding inference speed, our experiments on NQ shows that both our extractive and abstractive compressor enables 2x throughput. Please see the general response for more detail.

[Q2] The abstractive model takes the question and the top 5 documents as input, the same as RALM without RECOMP. Although it needs to process the full document sets, the abstractive model (770M) is significantly smaller than the LM, thus much more computationally efficient as we demonstrate through the inference speed measured.

[0] Liu, Nelson F., Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang. “Lost in the Middle: How Language Models Use Long Contexts.” ArXiv abs/2307.03172 (2023). https://arxiv.org/pdf/2307.03172.pdf

[1] Shi, Freda, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Huai-hsin Chi, Nathanael Scharli and Denny Zhou. “Large Language Models Can Be Easily Distracted by Irrelevant Context.” International Conference on Machine Learning (2023). https://arxiv.org/pdf/2302.00093.pdf

Thank you for your review and we are glad to see the reviewer found our work to be important and effective, backed by sufficient and comprehensive analysis! Please see our response below:

[W1, Q1] Regarding inference speed, our experiments on NQ shows that both our extractive and abstractive compressor enables 2x throughput. Please see the general response for more detail.

[Q2]: The reviewer points out a valid concern about the training cost for such compressors. The training costs consist of (1) generating the training data and (2) training compressors on the generated training data. We describe both below.

For extractive compressors, generating training data incur running base LM inference on all the sentences. For an abstractive compressor, the major cost comes from generating training data using LLM such as GPT-3.5-turbo. It costs ~$130 to generate the abstractive summary for the entire NQ training set and we will report detailed costs for the other tasks in the paper.

Training the extractive model takes a couple of hours on a single A40 GPU. Training the abstractive models takes a couple of hours on 4 A40 GPUs.

We would also like to stress that the training cost is a one-time investment, as once the model is trained, it could be deployed for inference for a much larger scale which will make the training costs negligible.