Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks?

This paper executed a set of experiments to evaluate the capabilities of 17 LLMs to perform retrieval tasks with long contexts. It introduces challenging multi-step threading and multi-threading retrieval tasks to test the models' ability to track information through extended contexts. The results show that increased context length reduces retrieval performance, with accuracy generally decreasing as the context window grows. The authors also demonstrate that many leading LLMs are thread-safe. Additionally, they highlight significant differences in token counts across different tokenizers.

This paper investigates the ability of Large Language Models (LLMs) to handle complex information retrieval and reasoning tasks across long contexts. The authors conduct a series of retrieval experiments with 17 leading LLMs to assess their capability to follow information threads within extensive context windows, revealing that while many models can manage multiple threads without significant performance loss, their effective context limits are often shorter than their technical limits. The study also emphasizes the variability in token counts between different tokenizers, which affects model performance comparisons. A key contribution is the introduction of a task-specific effective context limit metric, which provides a more nuanced understanding of model capabilities in long-context scenarios.

The paper evaluates the performance of Large Language Models (LLMs) as their context limits increase, focusing on their ability to handle complex information retrieval across multiple documents. Through experiments with 17 leading LLMs, the study finds that while many models effectively manage multiple threads of information, their actual efficient context range is often shorter than their maximum allowed, leading to reduced performance as the context window expands. The research also highlights inconsistencies in token counts across different tokenizers. Results and methodologies are made available for further study.

In this paper, the authors introduce simple single-needle retrieval, multiple-needle and conditional-needle retrieval and challenging needle threading and multithreading retrieval. Experiments on haystacks consisting of key-value pairs of UUIDs shows the retrieval precisions of 17 LLMs vary on different context lengths, multiple-needle and multiple threading conditions.

We thank the reviewers for their insightful comments, positive responses, and time spent reviewing our paper. We are glad the reviewers found our paper to focus on an interesting (Q7CU) and intriguing (n6xp) problem with innovative experiments (bsZQ) and task designs that are comprehensive and insightful (wcWb).

Here, we address the weaknesses and questions raised in your reviews. EDIT: We have uploaded a Rebuttal Revision PDF of our paper with an additional ‘Rebuttal Figures’ section at the start of the Appendix (Page 14) containing some figures related to our rebuttal. Note, for the final version, we will incorporate any changes into the main paper and appendix, ensuring the total length is within the 10-page limit.

Use of abstract/synthetic data & applicability to real-world natural language scenarios (Q7CU, n6xp, bsZQ, wcWB):

Thank you for raising this important point. As mentioned in our paper, focusing our evaluation on abstract and synthetically generated scenarios has benefits of enabling fine-grained control and high-quality noise-free generation without potentially expensive annotation. Additionally, the likelihood of data contamination is significantly reduced and we can be confident – especially given the unique nature of UUIDs – that our tasks cannot be performed via memorisation.

However, we concur that there is a gap between our experimental design and real-world applications. To address this, we have carried out a series of natural-language experiments. We focus on the threading and multi-threading tasks as these arguably had the most interesting takeaways. Concretely, we took The History of the Decline and Fall of the Roman Empire book, by Edward Gibbon (see Fig. 1 of our paper) as our “distractor” text. We chose a sentence in the text as a proxy for a UUID pair in the abstract task. Then, we prompted OpenAI’s o1-preview model to provide a list of plausible yet fictional Roman events (that aren’t included in the text). Using these events, we constructed ‘threads’ in the form of linked sentences: Sentence 1: ‘Event A and then Event B.’,... Sentence 2: ‘Event B and then Event C.’,... and so on. And replaced sentences in the text with these thread sentences.

Thus, we have a natural language text containing linked information that is in distribution with the rest of the text. This serves as a natural language proxy for our abstract threading tasks.

Single Threading results:

We repeated the threading task in this new setting by evaluating models on haystacks from 1k to 630k token context lengths with threads from 2-25 steps in length. Please refer to Fig. 14 in the rebuttal revision pdf for selected heatmaps for threads travelling in the forward and backward directions. As in the abstract setting (Fig. 10), following threads in the natural language text proves challenging for the models, with similar poorer performance observed at longer contexts. The preference towards forward travelling threads is clearly also apparent from the heatmaps in this setting. We will update the final paper with these observations.

Multi-threading results:

We conducted multi-threading experiments in the natural language setting using the same approach as the Single Threading experiments above (this time with additional simultaneous threads). We present results for the Thread-Safe experiments repeated in the natural language setting in Figs. 15 and 16 in the rebuttal revision pdf. Each point represents an average over 5 repeats retrieving threads with a length of 3 in 20k LLaMA 3.1 token haystacks. For Fig. 15, threads are randomly-directed whereas for Fig. 16 they are all forward-directed. Unlike the Single Threading experiments – for which the results and insights were largely the same across the abstract and natural text settings – the Multi-threading task in the natural language setting proved much more challenging for the models. Moreover, we find the task to be challenging when retrieving multiple threads that are all forward or all randomly directed. Similar results were obtained when repeating this experiment on smaller haystacks and chain lengths of 2. Thus, our study reveals that the multi-threading results are nuanced – with strong performance in the synthetic setting and weaker performance in the natural text setting. We’ll update the paper with these new insights, which we hope will be valuable for the research community.

We thank the reviewers again for their time, helpful comments and positive responses to our paper. In our rebuttal, we aimed to address all your concerns and consider our paper to be improved as a result.

This paper focuses on experimental validation and discussion of the context understanding capabilities of existing LLMs. The authors designed a series of retrieval experiments, including single needles, and multiple needles with random and clustered distributions. The experimental process is clearly presented and easy to understand, and most reviewers expressed their recognition of this work and gave positive scores. The main concern raised by the reviewers was that the authors only used synthetic datasets for testing, which the authors addressed in the rebuttal phase. I personally find this acceptable, especially since the authors conducted thorough experiments and provided a complete experimental process. Additionally, my suggestion is for the authors to avoid using different x-axis intervals (e.g., Gemini 1.5 Flash and GPT-4 Mini) in the same figure (e.g., Figure 5). To summarize, I recommend accepting this work.

Accept (Poster)