Retrieval or Global Context Understanding? On Many-Shot In-Context Learning for Long-Context Evaluation

Thank you for your valuable feedback. We aim to address your comments below:

1. Task Coverage

In our paper, we currently focus on common in-context learning (ICL) tasks. For the revision, we plan to expand our scope to include additional tasks such as dialogue systems and multi-document summarization. However, our main concern lies in evaluating the outputs of dialogue systems and determining whether such evaluations might be overly challenging. If you have any suggestions regarding suitable datasets or evaluation metrics, we would greatly appreciate your input.

2. Lack of practicality

Thank you for recognizing the significance of our work in the strengths section. In this paper, we identify suitable in-context learning (ICL) tasks for evaluating long-context language models (LCLMs), characterize the primary skills measured by each task, and propose MANYICLBENCH—a benchmark designed to evaluate models’ retrieval capabilities and global context understanding.

Our work enhances the current understanding of many-shot ICL for LCLMs and provides valuable insights into which tasks benefit from many-shot demonstrations. Additionally, we develop methods to characterize each task, offering future researchers a systematic way to categorize tasks and datasets.

Most long-context applications, such as multi-document summarization and code repository understanding, require models to effectively combine retrieval skills with global context understanding. We believe MANYICLBENCH will serve as a valuable tool for evaluating long-context models. Furthermore, the analyses we perform on the benchmark shed light on the current progress of LCLMs and provide guidance for developing more effective long-context training algorithms in the future.

Q1

Since our work primarily focuses on evaluating models' retrieval and global context understanding skills, it serves as a complementary approach to studying information forgetting and context confusion. We can examine how the position of demonstrations impacts the extent of information forgetting by altering their placement within the prompt. Additionally, by including more demonstrations, we stress-test LCLMs in long-context scenarios, investigating whether models become confused when faced with an increased number of contexts or demonstrations.

In future work, we plan to delve deeper into these specific challenges, providing further insights into the limitations and strengths of LCLMs in handling long-context tasks.

Q2

In Table 2, almost all models experience a performance downgrade when extending the context length from 64k to 128k. This degradation is likely tied to the context length extension methods used during model training. An effective training algorithm should ensure that the model maintains its performance across extended contexts.

For instance, Llama-3.1-8B demonstrates stronger retrieval capabilities than Qwen2-7B at context lengths of 64k or smaller. However, when the context length increases to 128k, Llama-3.1-8B suffers a 7% performance downgrade, whereas Qwen2-7B maintains its performance. At the end of Section 6, we discuss how Qwen2 models might employ a more effective approach to extending context size compared to Llama-3.1 models.

We hope we have addressed your concerns and kindly ask you to re-evaluate your assessment. Please feel free to reach out if you have any additional questions.

Thank you for your valuable feedback. We aim to address your comments below:

How section 4 fits into the paper

Section 4 presents a preliminary study we conducted to explore how many-shot ICL enhances model performance across different task types. In this section, we aim to validate whether providing additional shots benefits models on various tasks. Intuitively, all tasks are expected to exhibit a non-decreasing trend as more shots are added. However, in Section 4, we observe that classification and non-classification tasks follow different trends when more shots are provided. We hypothesize that classification tasks with a positive trend primarily require models to retrieve similar examples, while non-classification tasks with a flat or negative trend demand that models understand each example in the prompt.

This observation motivates us to systematically study why some tasks experience performance degradation with more shots while others do not. If the model struggles with non-classification tasks as the number of shots increases, is this due to its inability to retrieve similar demonstrations in the long prompt, or does it stem from a failure to leverage all the demonstrations to understand the task?

Gemini-1.5-Pro Results

We emphasize the importance of conducting experiments on open-weight models to support reproducibility. Closed-source models may become obsolete over time, limiting their utility for long-term research. We include the new Gemini-1.5-Pro results and the updated Table 2 in the following table:

Gemini-1.5-Pro

Similar to other open-weight models on retrieval tasks, Gemini-1.5-Pro begins to show performance degradation beyond 32k. However, it is one of only three models (alongside Qwen-2-72B and GLM-Chat-9B) that demonstrate impressive retrieval capabilities beyond 64k and maintain performance at 128k. On global context understanding tasks, Gemini-1.5-Pro significantly outperforms other open-weight models, showcasing its ability to grasp the global context and effectively utilize all the demonstrations.

Thank the authors for their response and additional experiments! My concerns are partially addressed. After reading other reviewers' comments (in particular, I share the concern of Reviewer pTsV on problem formulation) -- I believe the rating reflected by my original score is appropriate.

5. Classification Tasks Relies on Retrieval

Classification tasks benefit more from demonstrations, as evidenced by our experiment in Section 4. In the classification plot of Figure 1(a), all models exhibit a strong correlation between performance and context length. Prompts with more demonstrations are likely to include more similar examples. This observation is further supported by the retrieval load experiment in Section 5 and the new experiments in Appendix E.

The first experiment shows that removing similar examples hurts the model’s performance more than removing distant examples, indicating that the model relies on similar examples to make predictions. In [2], it is shown that the decision boundary is influenced by the label distribution of the demonstrations. However, the results of the second experiment demonstrate that even when the label distribution remains the same—where the decision boundary is minimally impacted—replacing similar examples with distant examples still degrades performance. This indicates that the model retrieves similar demonstrations to make predictions, further supporting our statement that "classification tasks predominantly test models’ retrieval skills."

The new experiment results are presented in Figure 12 and Figure 13 from Appendix E, along with the table below:

Retrieval Load Ratio Results for Original and Replacement Experiments with BM25 and SBERT

We excluded XLSUM because computing similarity among many news articles is computationally expensive. For SBERT, we excluded geometric_shapes, Dyck_languages, and dialogRE because the inputs of the first two tasks primarily consist of symbols and numbers, while the inputs for dialogRE are too lengthy for the retriever to be effective.

The results from the original experiment still hold under both retrievers. Classification tasks exhibit a higher retrieval load ratio, while non-classification tasks have ratios closer to 1, suggesting that classification tasks rely more heavily on retrieving similar examples.

6. Ordering Effect

As you mentioned, the ordering effect has been well-studied in previous works [1, 3, 4]. We are aware that the order of demonstrations can influence a model’s performance. However, our primary focus is to evaluate the capabilities of long-context models. To minimize the impact of ordering, we shuffle the order of demonstrations across each random seed.

Measure similarity

We only consider x because using y to retrieve examples would make the task overly simple and impractical. The objective of the task is to predict y given x, so initially, we have no prior knowledge of y.

We hope we have addressed your concerns and kindly ask you to re-evaluate your assessment. Please feel free to reach out if you have any additional questions.

[1] Bertsch et al., In-Context Learning with Long-Context Models: An In-Depth Exploration, 2024

[2] Zhao et al., Calibrate Before Use: Improving Few-Shot Performance of Language Models, ICML 2021

[2] Lu et al., Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity, ACL 2022

[4] Li et al., Long-context LLMs Struggle with Long In-context Learning, 2024

Thank you for your valuable feedback. We aim to address your comments below:

1. Definition of the retrieval ability and global context understanding

We define retrieval ability as “the model’s capability to pinpoint relevant and similar examples.” While it is true that almost all NLP tasks require some level of global context understanding, even with a single example in the prompt, our focus is on many-shot in-context learning (ICL). In this context, we define global context understanding as “the model’s ability to comprehend multiple examples across the full input.”

In ICL, the model utilizes its retrieval ability to identify examples similar to the test input and make inferences based on them. Simultaneously, it leverages its global context understanding to process and integrate all the demonstrations in the input. Both skills are crucial for achieving strong performance on downstream tasks such as long-document question answering (QA), multi-document summarization, multi-turn dialogue, and long-context retrieval-augmented generation (RAG).

However, many existing studies primarily evaluate a model’s retrieval ability. Our work offers an alternative approach to assess both retrieval and global context understanding skills, providing a more comprehensive evaluation of model performance.

2. Regarding Q2

We believe that our statement is well-supported by the experimental results in Section 5 and the findings of [1]. Their study compares performance between using random samples and similar samples, demonstrating that sampling similar demonstrations yields better performance than sampling random ones.

While we do not claim that retrieval tasks lack a need for global context understanding, we suspect that these tasks primarily rely on a model’s retrieval skills to achieve strong performance. To further support this assertion, we conducted an additional experiment where we replaced the top k% most similar examples with the most distant examples that share the same labels. This ensures that the label distribution remains unchanged, but the examples themselves differ. Even with an identical set of labels, the model exhibited performance loss on classification tasks, indicating its reliance on retrieving similar examples.

Additional details about this experiment can be found in the Classification Tasks Relies on Retrieval section of the response and Appendix E.

3/4. Using SenteneTransformer as a retriever

We use BM25 because it is one of the most widely used retrievers. To address your question about the generalizability of the results, in Section 5, we plot the results for Qwen2, Llama-3.1, Mistral-Nemo, and GLM, and compute their average. All six models exhibit similar trends. Additionally, we conducted further experiments where we replaced similar examples with distant examples that share the same labels. Using both BM25 and SentenceTransformer (SBERT), we observed consistent results: the replacement has a greater impact on classification tasks.

Additional details about this experiment can be found in the Classification Tasks Relies on Retrieval section of the response and Appendix E.

Thank you for your engagement and for taking the time to delve into our work. We want to clarify some potential confusion you may have and provide further context to ensure our findings and claims are fully understood.

Retrieval and Global Context Understanding

Retrieval can be considered a subset of global context understanding because a model needs to recognize relevant parts of the input to grasp the entire input sequence. However, retrieval and global context understanding are meaningfully distinct. Understanding a small part of the input does not necessarily equate to comprehending the whole sequence.

For example, imagine an input containing 100 unique numbers ranging from 1 to 100. A retrieval task might ask the model to identify the first number greater than 55. In contrast, a global context understanding task could ask for the sum of all numbers greater than 55. While solving the retrieval task involves pinpointing a specific number, solving the global context task requires aggregating all relevant numbers after 55. Success in the retrieval task doesn’t imply the model can perform the more complex aggregation needed for the global context task.

Similarly, in in-context learning (ICL), a model might retrieve a single example that closely matches the query, demonstrating minimal understanding of other examples in the input. However, to solve global context understanding tasks, the model must process and integrate information from all examples, synthesizing them to produce a comprehensive result.

Demostration Ordering

We conducted an additional experiment to investigate how demonstration ordering affects model performance using Llama-3.1-8B. Specifically, we examined the impact of placing the top 10% most similar examples at different positions within the prompt: the front, middle, or end. We then compared the average percent change in performance relative to the original order across input lengths ranging from 1k to 64k.

The table above shows the results of our experiment analyzing how the position of the most similar demonstrations (front, middle, or back of the prompt) impacts model performance across various tasks using Llama-3.1-8B. The performance is measured as the average percent change relative to the original order, with input lengths ranging from 1k to 64k.

From the results, no clear universal trends emerged regarding the effect of demonstration positioning on performance across all tasks. For classification tasks, placing similar demonstrations at the front tended to hurt the model's performance more than placing them at the back. However, for non-classification tasks, no consistent pattern was observed.

Importantly, these findings do not impact the overall conclusions of our experiment or paper, as our comparisons involve models evaluated with the same prompts across three random shuffles, ensuring robustness against ordering effects.

Thank you for your valuable feedback. We aim to address your comments below:

Comparision to InfiniteBench

We acknowledge the contributions of InfiniteBench and will include this work in the related work section of our camera-ready version. We do not claim to be the first to create such a benchmark. Compared to InfiniteBench, our work focuses on identifying suitable tasks and evaluating models' retrieval and global context understanding capabilities. Our benchmark provides two categories of tasks that allow users to conduct fine-grained evaluations of long-context models across varying context lengths—an aspect not covered by previous works. Furthermore, future research can leverage our methods to categorize tasks and datasets, enabling the quantification and characterization of advancements in long-context models.

Gemini-1.5-Pro Results

We emphasize the importance of conducting experiments on open-weight models to support reproducibility. Closed-source models may become obsolete over time, limiting their utility for long-term research. We include the new Gemini-1.5-Pro results on our benchmark and the updated Table 2 in the following table:

Gemini-1.5-Pro

Similar to other open-weight models on retrieval tasks, Gemini-1.5-Pro begins to show performance degradation beyond 32k. However, it is one of only three models (alongside Qwen-2-72B and GLM-Chat-9B) that demonstrate impressive retrieval capabilities beyond 64k and maintain performance at 128k. On global context understanding tasks, Gemini-1.5-Pro significantly outperforms other open-weight models, showcasing its ability to grasp the global context and effectively utilize all the demonstrations.