M$^4$LE: A Multi-Ability Multi-Range Long Context Evaluation Benchmark for Large Language Models

Thank you for your insightful comments and thorough evaluation of our paper. We are grateful for the opportunity to address your concerns and provide additional clarity regarding our study. Please find our detailed responses below:

Q1: It is clear that the model's performance deteriorates significantly as the length of the context increases, which raises doubts about the effectiveness and appropriateness of such data formation. The random selection of original short sequences appears to negatively impact the performance.

You raised a critical point regarding the construction of long sequences from shorter ones, and the potential impact of this methodology on model performance. We agree that the performance degradation observed in models as context length increases highlights the challenges LLMs face in processing long sequences. However, we argue that this actually supports that our approach is effective rather than a flaw in our data formation approach.

Our method of extending short sequences into longer contexts is designed to rigorously test the five long-context understanding abilities of LLMs without building new datasets from scratch. The performance deterioration observed in some models, especially at longer context lengths, is indicative of their current limitations in handling extended sequences. This is an important aspect of our benchmark, as it successfully distinguishes between models based on their ability to process long contexts. For instance, as illustrated in Figure 2, more competent models like GPT-3.5-Turbo-16K and ChatGLM2-6B-32K exhibit a relatively smaller drop in performance, demonstrating their stronger long-context capabilities.

Additionally, we acknowledge the concern about the random selection of original short sequences. We want to clarify that our approach also aims to mitigate potential pre-training biases inherent in LLMs. Therefore, the difficulty models face with instances formed with randomly selected sequences underscores the effectiveness of our approach in providing a more unbiased assessment of their long-context understanding abilities.

Q2: I kindly request a performance comparison with recent state-of-the-art models, such as Llama2-4k.

Regarding your request for a performance comparison with the latest state-of-the-art models, such as Llama2-4k, we appreciate the importance of including such comparisons in our study. In our initial submission, we evaluated the Llama2-4k models, specifically the 7B and 13B variants, alongside their chat versions.

For Llama2-70B models, we are currently conducting experiments with these models. However, due to the extensive computational resources required for these experiments, we anticipate some delay in obtaining and analyzing the results. We are committed to including this data in our study as soon as it is available and will provide updates accordingly.

We hope our responses adequately address your concerns. We are committed to advancing the understanding of LLMs in processing long sequences and believe our benchmark, M4LE, contributes significantly to this field. We look forward to any further feedback you may have.

Thank you for your valuable feedback and insightful observations regarding our paper. We appreciate the opportunity to address your concerns and clarify aspects of our work. Please find our detailed responses below:

Q1: The construction of the dataset is detached from realistic scenarios.

We acknowledge your concern regarding the realistic applicability of our dataset. It is important to note that M4LE includes 10 tasks that directly target global context understanding abilities pertinent to real-world scenarios, such as summarization, translation, question answering, and classification tasks. These tasks represent a significant portion of our benchmark and are designed to reflect common use cases of LLMs in practical applications.

Furthermore, while some tasks in M4LE may seem detached from typical user interactions with LLMs, they are intentionally designed to evaluate LLM across five key long-context understanding abilities as mentioned in the paper. These five abilities encompass a broad spectrum of long-sequence scenarios and challenges that a language model may face in real-world applications. Therefore, by constructing a diverse benchmark that evaluates these five abilities, we can obtain a measure that closely approximates the performance of models in realistic, long-context scenarios.

Q2: This dataset is evidently more beneficial to models trained on these synthesized short texts, therefore introducing a potential bias.

We understand the concern about potential biases favouring models trained on synthesized short texts. However, this aspect was one of the motivations behind our construction methodology. While the models might be trained on the original dataset, combining the shorter texts presents an instance that is sufficiently distinct from the original ones. For instance, in the WikiText-103 (NLI) task, we observed that even though Wikipedia articles are commonly included in language model training datasets, the performance of most models is poor even at shorter context lengths. This suggests that the benchmark effectively challenges the models beyond their training conditions.

Moreover, the primary objective of M4LE is to assess the ability of LLMs to maintain their performance as context length increases. By focusing on performance trends across varying context lengths, we minimize the impact of any prior training on shorter texts.

Q3: However, when the answer does not appear at the beginning of the document, gpt3.5-16k usually outperforms gpt3.5-4k. I think the potential reasons are as follows: (1) Unfair metrics. This paper mainly uses n-gram matching metrics like ROUGE and F-1 which may not correlate with human evaluation results. (2) Position bias in the dataset construction. Most answers are at the beginning of the document, additional input is noise. (3) The current long context models indeed cannot handle many input tokens but how is the Llama2-4k?

Regarding the evaluation metrics, we use accuracy as the metric for QA, consistent with previous work (Liu et al., 2023). For classification tasks, we use F1 scores, which do not rely on n-gram matching. N-gram metrics are applied in only 15 of the 36 tasks, primarily in summarization and translation. We acknowledge the limitations of these metrics but have focused on performance trends over context lengths to mitigate potential biases. Besides, these N-gram metrics are still commonly reported in much recent literature for summarization and translation tasks.

To address potential position biases, we have carefully constructed instances with relevant supporting paragraphs uniformly distributed within the input context, as outlined in Section 3.1 of our paper.

Finally, regarding Llama2-4k, we used dynamic NTK scaling to extend the effective window size of LLama2 models, which we found to be an effective approach without further fine-tuning as mentioned in the paper.

Q4: I find that in many tasks even a 6B model can outperform gpt3.5 by a remarkable margin. More analysis can make us better understand current long context models.

Thank you for your observations and suggestions! GPT-3.5 outperforms other models in most tasks, except for DuReader and CNNNews where ChatGLM2-6b-32K performs better. We examined some of the outputs generated by different models in various tasks and did not find anything insightful. We believe that other models may occasionally outperform GPT-3.5 due to the inherent noise in extensive experiments. Overall, GPT-3.5 exhibits strong long-context understanding, but certain tasks might favour other models due to random noise factors.

In summary, we believe that M4LE provides a robust and comprehensive benchmark for evaluating the long-context understanding abilities of LLMs, addressing both realistic scenarios and potential biases. We hope our response adequately addresses your concerns and we are open to further discussion or clarification if needed.

Thank you for your valuable time and feedback. As the deadline for author-reviewer discussion ends today, we kindly request that you acknowledge our rebuttal and engage in further discussion if you have any additional comments.

Q8: I will think that the M4LE just simply combine the small datasets together to build long-context datasets, where most of them have no relevance between each other. It means that M4LE is a benchmark mainly tests the retrieval ability over long-context but less consider the comprehensive understanding over the global context.

Thank you for your comment regarding the M4LE. I would like to take this opportunity to clarify the design of M4LE. M4LE does not just test the retrieval ability over long-context. It contains 22 tasks that target the semantic and global understanding ability of the models. These tasks require LLM to understand the full context thoroughly in order to retrieve relevant information to complete the task. For tasks targeting semantic understanding, the model needs to identify a few key text spans after it comprehends the whole context to finish the task. Tasks focused on global understanding require the model to integrate and interpret information from the full context to complete the task. Therefore, by targeting the five different abilities, M4LE is a comprehensive benchmark that evaluates a wide range of long-context understanding scenarios.

Q9: The authors claim that Vicuna and Long-chat exhibit a more pronounced performance drop in Chinese. However, in Figure 4, the drops of these models between these two languages have no obvious difference.

Thank you for your observation and comment! We originally computed the slope of the best-fit lines for both languages and found that Vicuna and LongChat models indeed have steeper slope in Chinese. However, after the updates on the results as mentioned in the global response, we observed similar slopes for both languages. Therefore, we updated the paper to reflect that.

Q10 : How does it alleviate the data leakage problem, which also exists for other benchmarks?

Thank you for your question. Indeed, it is an important problem for benchmarking, as also mentioned by Reviewer "sezj".

Our methodology combines the shorter texts which actually present an instance that is sufficiently distinct to the original ones. For instance, in the WikiText-103 (NLI) task, we observed that even though Wikipedia articles are commonly included in language model training datasets, the performance of most models is poor even when the input contexts are limited to 1k tokens. This suggests that the benchmark effectively challenges the models beyond their training conditions.

Besides, changing the question type and format, for example by asking the models to return the hexadecimal identifier of all the articles that belong to a certain class presents a different challenge for different LLMs, as evidenced by the poor performance of most models in MNDS News (Explicit Multiple) and THUCNews (Explicit Multiple).

We hope these revisions and clarifications address your concerns adequately. Thank you for your insightful feedback, which has significantly contributed to the improvement of our paper.

Thank you for your insightful comments and suggestions. We have carefully considered your feedback and have addressed each point as follows:

Q1: It is better to include some comparison between M4LE and other benchmarks in Table 2. It is also good to include the max. length number in it.

We appreciate your suggestion and have accordingly included detailed statistics in the revised paper under Appendix A.1.

Q2: The evaluation in the paper are weak and uninformative, except that the existing LLMs struggle to process long inputs, and ChatGPT performs the strongest among all. These can be shown from other existing benchmarks as well. The categorization and organization of different ranges of sentence lengths can also be done in other similar benchmarks.

Our paper introduces the first comprehensive benchmark for long-context evaluation, systematically designed around five key abilities that encompass a broad spectrum of real-world scenarios, with contexts extending up to 32K words. Therefore, it is important to examine the performance of different LLMs on this comprehensive benchmark. Additionally, our analysis delves into performance across different languages and investigates the 'lost-in-the-middle' phenomenon, as reported by Liu et al. (2023). We also present further analysis of performance across various task types, as mentioned below.

Q3: M4LE spans different domains and tasks, while the evaluation does not consider these points

Thank you for your valuable feedback. In response, we have added a new section, "Appendix A.6: Performance in Various Task Types", to our revised paper. We also want to highlight that our benchmark, designed to encompass a broad spectrum of real-world scenarios with long inputs, is based on five key long-context abilities. This benchmark, which considers various domains and tasks, ensures comprehensive coverage of different use cases. Therefore, our main paper focuses primarily on analyses related to these five abilities.

Q4: it shows the situations from 3 types of tasks are different. There can be more sophisticated explanations and experiments to furnish further ideas.

We seek clarification on whether you are referring to Figure 3, which shows the effect of placing the supporting document at different locations. While we acknowledge the subtle differences highlighted in Figure 3, our paper's primary focus is on a methodology for assessing the performance variations with context length. Therefore, we think it would be best for future work to perform more extensive experiments (More task types, more datasets per task type, different setup, etc.). As for the performance variations in different tasks of our main experiments, we have included them in the revised paper as stated in the previous question.

Q5: why most models perform worse on explicit multi-span understanding compared to both semantic multi-span and global context understanding

Thank you for raising this interesting question! We briefly discussed this phenomenon in Section 4.3 under "Multiple-span understanding is more difficult, and semantic retrieval is even harder for competent models." While we think that more work is needed to investigate this, we hypothesize that explicit multi-span understanding requires the model to generate hexadecimal identifiers is an extremely unfamiliar task for LLMs. It is very different from most pre-training tasks, and therefore only the most capable LLMs that were fine-tuned on massively diverse tasks and aligned well with human preference can generalize and demonstrate adequate performance at this task.

Q6: only 13B models are evaluated in the paper. It is still unclear whether larger models can perform much better, e.g. LLaMA2-70b.

We are in the process of extending our experiments to include larger models like LLaMA2-70b. Due to the extensive computational resources required for these experiments, we anticipate some delay in obtaining and analyzing the results. We are committed to including this data in our study as soon as it is available and will provide updates accordingly.

Q7: The paper seems to have no detailed report about how to construct the benchmark

Thank you for your suggestions. We have included more details in "Section 3.3 Task Construction". Specifically, we determine the number of instances by dividing the desired context length with the median length of the instances in the testing set. Then, we sample these instances randomly from the testing set.

Thank you for your valuable time and feedback. As the deadline for author-reviewer discussion ends today, we kindly request that you acknowledge our rebuttal and engage in further discussion if you have any additional comments.

Thank you for your insightful comments and suggestions. We have carefully considered your feedback and have addressed each point as follows:

Q1: Additional statistical information of the proposed dataset could be provided, such as the distribution of sentence lengths and the proportion of samples in different languages.

We have now included detailed statistics in the revised paper under Appendix A.1. We believe these additional details will provide a clearer understanding of the dataset's composition and diversity.

Q2: Do the authors have a data cleaning process? How is it done? Regarding our data cleaning process, we adhered to the preprocessing process of the original datasets. Additionally, we exclude instances that were either excessively long or short for our benchmark. The precise methodology and criteria used for this filtering process have been outlined in detail in Appendix A.1 of our original submission.

We appreciate your constructive feedback, which has significantly contributed to enhancing the quality of our paper. We hope that our revisions meet your expectations and look forward to any further suggestions you may have.

Thank you for your valuable time and feedback. As the deadline for author-reviewer discussion ends today, we kindly request that you acknowledge our rebuttal and engage in further discussion if you have any additional comments.