Can long-context large language models understand long contexts?

Q1: Following the promises in the abstract about the human-annotation and the cross-annotator validation, I was very disappointed to see that a large part of the benchmark's ground-truth output was generated using GPT 3.5 / 4. If this is indeed the case, this is disappointing, and the benchmark may be biased toward "questions that are easy for ChatGPT to answer".

A: We thank the reviewer for pointing out this concern. We indeed have spent a huge effort and high cost to make the evaluation as fair as possible in the following steps:

• For short dependency QA, we have manually reviewed all the short QA pairs and carefully refined the answers to make them clear and concise. The model is forced to extract the answer directly from the original document to generate the initial answer. Then we make refinements by filtering out the non-essential contexts and removing redundant descriptions from the model.
• For long dependency QA, there are over 1100 long dependency QA pairs delicately designed by human annotators, despite the high costs and huge effort. Each document for generating QA pairs underwent a meticulous three-step process(Question & answer, Answer & check, Cross-validation & revise) that involved the assignment of two distinct annotators who are unaware of each other’s identities. The annotation adhered to stringent standards including long dependency, diverse problem types, clear & precise questions and deterministic & objective answers. Participants were prohibited from using large language models and tools like ChatGPT for article reading, data generation, and annotation.

3. For summarization, answers are abstracts extracted from the paper.
4. For cloze, we did not use gpt-3.5 to generate both the question and answer directly, and the answers are extracted by a NER model.

This rigorous curation process was undertaken to ensure the high quality of the questions, answers, as well as supporting evidence. This approach aims to achieve questions with a high degree of accuracy, precision, and relevance to the document’s content.

Q2: The text is very unclear in many cases. For example when describing the statistics of the benchmark, the paper says: Extra-long realistic documents. It contains 778 latest gathered and extremely long documents with an average of 16.4k words. There are over 6000 test instances without distribution bias for a more generalized assessment, many of which are exceeding 100k words. So are there 778 examples or 6000 examples? If "many of which exceed 100k words", how many of them? what's the average? Are these two datasets? If not, why are these numbers reported separately?

A: Thanks for pointing out the misleading data statistics. For the polished version of the dataset, we have collected 776 original long documents, 2 of which exceed 100k words while 30% of which exceed 20k. The average words of LooGLE is more than 2 times longer than the existing dataset for long context. Please refer to Figure 4 for the data length distribution. Based on the documents, we further generate over 6000 questions for test and evaluation. Details and numbers of the questions in each task, and their distributions, can be found in Table 2. We have also revised the expressions in the paper to make it precise.

Q3: The results in Section 4.3.1 are very confusing and unclear. For example: In Table 3, it can be noticed that LlamaIndex obtains from the perspective of GPT4 evalution. Instead of memorizing a shortcut of original input with a limited context window, retrieval-based context compression technique augments the LLM by incorporating external memory, allowing relevant information to be retrieved using a specific query. I am not sure what such paragraphs are trying to say. What does it mean that "LlamaIndex obtains from the perspective of GPT4 evalution"? What do the authors exactly mean by "memorizing a shortcut"? Who is memorizing a shortcut?

A: Thank you for your suggestion. We intend to show that the retrieval-based LlamaIndex with external memory does help short QA to some extent and has a competitive performance compared to the winning GPT models, as inferred from Table 3. LLMs inherently lose much information in long inputs with limited context windows (which we call "memorizing a shortcut"). We have rewritten this part and hope that the edited section clarifies these key findings.

Q1: Is the benchmark english-only ? If so, this needs to be mentioned.

A: The current version of the dataset is English-only. Thanks for your reminder and we have included this detail in the paper as well.

Q2: How will the dataset be released ? For instance ZeroSCROLLS only released the inputs and evaluate through a system of leaderboard, will LooGLE be released the same way ? I'm concerned that revealing input/gold outputs pairs would lead to data leakage for future models.

A: Thanks for your comments and it's a great suggestion.Currently, our polished dataset has been released in the HuggingFace Datasets as the dev set, with all the golden truths. In the future, we will keep updating the dataset with newer documents and diverse tasks, providing more examples split for different sets and uses.

Q3: Concerning data collection, were the sourced documents (after 2022) subjected to any machine-generated verification ? I am concerned that ChatGPT-like texts might compromise the fairness of the evaluation.

A: We thank the reviewer for your useful comments For data collection and selection, the sources of original documents (arXiv papers, Wikipedia articles, Movie and TV scripts) are most officially used and widely acknowledged to ensure their high quality. We will keep the data updated and follow your comments on data collection to guarantee the high quality of the dataset.

Q4: Are the open-source models instruction-tuned ? Commercial closed source models like GPT3.5 rely on RLHF or some instruction tuning techniques that enable them to better follow instructions. If the considered open-source baselines are not instruction-tuned, the comparison might be unfair since the prompt used for evaluation is the same and is instruction-based.

A: Thank you for the valuable suggestion and we totally agree with that. Among the four open source models we used in this paper, ChatGLM2-6B was trained with human preference alignment and LLaMA2-7B-32K was instruction tuned for summarization and long context QA. We have highlighted this information in Section 4.1 in the revised paper. Based on your suggestion, we are working on additional experiments for the instruction tuning versions of the other two models. The results for long dependency QA can be seen below:

Q4: Details of Llamaindex is not clear: 7B or 13B, chat model or regular model?

A: We use the official default model text-davinci-003 for LlamaIndex in all the experiments.

Q5: Why not use the chat version of Llama2, which is considered to be skilled at instruction-following and could be potentially better at downstream tasks.

A: We thank the reviewers for your useful comments. To the best of our knowledge, LLaMA2-7B-32K used in this paper is fine-tuned from Meta's original Llama-2 7B model to extend context and also instruction-tuned for summarization and long context QA for better performance. We have highlighted this information in Section 4.1 in the revised paper. We are happy to extend to new models and will keep updating the leaderboard.

Q6: Why LlamaIndex is much better than any other open-sourced models? According to your results in Fig3, retrieval+open-sourced model is better than the long-context version of the same base model, and this conclusion is contradict to the conclusion from other long-context benchmarks (Table 1).

A: Thank you for the comments.

• LlamaIndex is a retrieval-based data pipeline for different models. We use text-davinci-003 as default model in our experiment instead of the open-source Meta Llama.
• From the short QA in Table 3, retrieval-based techniques indeed demonstrate their benefits for short QA. However, they have a limited impact on understanding long context for long dependency tasks, as shown in Table 4.
• Our conclusion in Table 1 is consistent with that in LongBench. However, we have quite a different experiment settings (designed tasks, evaluation metrics, etc.) to drive to this conclusion.

Q7: Writing format: it's better to list url as the footnote. It is supposed to leave a black between the main text and citation brackets.

A: Thanks for your advice and we have already moved all the urls as the footnotes the writing format according to your comments (see Page 4 and 7). Also, we have refined the writing format issue of blank you mentioned.

Q1: The paper states that "by employing scaling techniques like positional interpolation, parallelization, and finetuning on longer texts, open-sourced models have shown improvement in handling longer inputs compared to previous versions." However, the article does not provide performance data of previous version models. This omission makes it challenging to ascertain the improvements is brought by modifying position embeddings or instruction-tuning. For example, the comparision between vicuna-2k and vicuna-16k is a better case to validate this claim.

A: Thanks the reviewer for your constructive advice.

• We have to emphasize that the paper only claims "open-sourced models have shown improvement in handling longer inputs by employing the scaling techniques" instead of gaining performance improvements in downstream tasks.
• However, we also agree with your suggestion and indeed explore this by adding additional experiments. We further compare the varying input lengths of ChatGLM2-6B-32k (32k context window) with ChatGLM2-6B (8k context window) on long dependency QA. We select ChatGLM2 as the testing model since it achieves better performance in our previous experiments among different open-sourced models. Here are some findings:
  1. The original version model ChatGLM2-6B performs better than ChatGLM2-6B-32k across the automated metrics given the same length of inputs. There is a performance decline for the model when extending the longer context window. It can be attributed to the information loss introduced by the scaling techniques of long context models, which calls for further improvement on open-sourced LLMs.
  2. Moreover, for ChatGLM2-6B-32k with varying lengths of inputs, we find that the extension of longer inputs has limited impact on performance. It is because the long dependency QAs in our dataset request for the long dependency comprehension and modeling capability, along with further computation and reasoning. Our dataset propose higher demands on true long context undetstanding, which needs to be desperately resolved and enhanced for further LLMs.

We sincerely thank the reviewer for your acknowledgment and constructive comments on our work. We would like to involve further discussions in the following sections.

Q1: The paper presents Human Evaluations as one of the evaluation technique but never present human evaluation results.

A: We have indeed included human evaluations in 3 parts of the experiments in Appendix C:

1. We manually evaluated the accuracy for 4 long-dependency tasks under 3 settings (without CoT, few-shot CoT, zero-shot CoT) in Figure 10.
2. We provide probable explanations for long QA bad cases to provide insights and directions for model promotion in Appendix C.1.
3. We captured discrepancies in the generated outputs of different models to tackle inherent preferences encountered in long context in Figure 11.

We have also revised the paper to make this part more clearly stated for readers.

Q2: There are several automatic scores have been presented in Tables 3 through 5 and sometimes. These scores not always in agreement; not all the scores are better for the winning model. I found this confusing especially when some conclusions are drawn in the text.

A: Thanks for pointing out. We summarize the main performance in the experiments and provide potential explanations as below:

• For summarization evaluated by both automatic metrics and GPT4-eval, GPT4-8k performs better while longer context window has marginal improvement from Table 4 and Table 5. It is due to the intrinsic features of ArXiv papers with both the introduction and conclusion located at the beginning and in the end respectively. In this way, GPT4-8k with limited window size can still perform well to capture the major sketch of the paper.
• For cloze, short & long QA evaluated by exact/partial match and GPT4-eval, GPT4-32k performs better semantically with more complete inputs with less information loss and stronger context understanding capability from Table 3, 4, 5.
• For QA tasks evaluated by other automatic metrics, GPT4-32k performs worse in some cases from Table 3. It can be attributed to its tendency to generate longer outputs, faces penalties from these automatic metrics. It is more evident in short dependency QA where GPT4-8k can extract the right answer with limited window.

The disagreement between metrics lies in the reason that each of them measures the generated texts from different aspects. For instance, BLEU and ROUGE evaluate n-gram based similarity but compute precision and recall respectively. METEOR incorporates matching in various aspects, including word stemming and variations, syntax, order, and phrase structure, and computes a semantic matching score. BERTScore leverages pre-trained contextual embeddings and computes the cosine similarity from token level. Exact Match entails precise comparison between the predicted entities and the ground truth entities while Partial Match allows for fuzzy matching. Therefore, we use various automatic metrics commonly to obtain comprehensive performance evaluations.

We have also revised the corresponding expressions in the paper to make it clear.

Q3: For the LlamaIndex, what retriever and chunk size are used?

A: We use the official default model text-davinci-003 as the retriever and default chunk size (1024) for LlamaIndex.