RULER: What’s the Real Context Size of Your Long-Context Language Models?

We thank reviewer 4oYN for their thoughtful comments.

The reviewer raises concern on the template used for all models. We employ the same template to enforce a controlled evaluation setup for all models. While the diversity of the prompts can be a valid concern for evaluating large language models, the robustness to various prompts is an orthogonal research direction which we believe to be out-of-scope of our paper. However, we agree with the reviewer that evaluating models on additional prompts can further strengthen our results.

To answer the questions raised by the reviewer:

1. In the current version of RULER, we purposefully select the synthetic tasks on which the majority of the models perform decently with short input length (<4k) so that the major differences we observe are due to the scaling of context size. We did not include harder tasks in RULER (e.g., those pressure testing Yi-34B in the analysis section) as those tasks will unfairly put some models in disadvantage even at short sequences.
2. As discussed in section 5, the reviewer is correct that the number of distractors indeed also impacts the complexity of a task. Within a constrained domain as in RULER, we believe the task complexity is a function of at least two factors: the number of target output tokens and the signal-to-noise ratio in the context. The number of target output tokens can be changed by changing the number of returned needles, the number of hops in variable tracking, and the number of top-frequent words to be returned in CWE/FWE. The signal-to-noise ratio can be controlled by the number of distractors in NIAH and variable tracking, the frequency of common words in the CWE task and the hyperparameter alpha in the FWE task. We plan to add more detailed relevant discussion in the next version.
3. In our preliminary experiments, we observe very small variances evaluating the same model on different synthetic vocabularies since we fix the size of the synthetic vocabulary and the length of each synthetic word. While these two hyperparameters can also be varied, we keep them fixed in our experiments to have a controlled setup. It can be interesting to investigate how these two hyperparameters impact model performance. We thank the reviewer for bringing this up and will add relevant discussion in our revised version.

Thanks for your considerate response, most of my concerns have been addressed. I would like to see what the additional discussion will provide us in the future version.

We thank reviewer HSfg for their thoughtful comments on our submission.

The reviewer mentions the absence of a comparative study between RULER and real-world applications. We would like to mention that real-world long-context tasks usually have fixed input lengths, thus it can be difficult to perform evaluation like we do with RULER, where we evaluate models with increased context size while maintaining the same and known upper-bound performance. Moveover, existing long-context tasks employ imperfect automatic metrics (e.g. ROUGE for book-level summarization), whereas gold human evaluation for long-context tasks can be expensive and the topic itself warrants a separate research work 1]. We agree with the reviewer that the correlation between performance on RULER and other real-world applications can be fruitful to explore. However, due to the scarcity of high-quality naturalistic long-context tasks that are easy to evaluate, we decide that this particular direction is out-of-scope of our paper and can be a promising direction to pursue in the future.

We agree with the reviewer that additionally profiling inference speed and memory footprint can give valuable information about performance-efficiency trade-off. We will add the support of comparing inference efficiency in our open-source code repository in the future.

Reference: [1] BooookScore: A systematic exploration of book-length summarization in the era of LLMs

We thank reviewer GXoR for their encouraging comments and thoughtful advice.

We agree with the reviewer that there are similar long-context benchmarks and thus have included a brief comparison in section 1 Table 1.

• Compared to existing synthetic evaluation such as passkey retrieval and needle-in-a-haystack, RULER covers diverse task categories such as tracing and aggregation, testing LM capabilities beyond simple in-context recall.
• Compared to existing realistic long-context benchmarks, RULER reduces the impact of parametric knowledge by maximizing the reliance on the contextual information. Evaluating on input that has large overlap with the pre-training data (e.g., book data used by other benchmarks) entangles context utilization with memorized parametric knowledge, whereas testing on fully synthetic data allows us to test a model’s behavior while minimizing the memorization effect.
• Compared to existing realistic long-context benchmarks, RULER also has the key advantage of allowing for flexible context length and varied task complexity. Existing benchmarks mostly adopt fixed input length, which can be too short to test “long”-context language models. In contrast, input can be adjusted to arbitrary length in RULER. Moreover, the number of distractors and the amount of relevant information can also be controlled by task hyperparameters, which lead to different task complexities.

Reviewer mentions a potential gap between the synthetic nature of RULER and the intricacy of real-world long-context applications. We acknowledge the limitation of synthetic tasks and would like to emphasize in the revised version that RULER can serve as diagnostic tests of LM capabilities (e.g., retrieval, tracing, and aggregation), however it cannot replace real-world long-context tasks. We agree with the reviewer that it can be interesting to see the correlation between RULER and other real-world long-context tasks, since many tasks in RULER are heavily motivated by realistic natural language tasks (e.g., variable tracking motivated by coreference chain resolution, and frequent word extraction motivated by summarization) after all. Due to the scarcity of high-quality naturalistic long-context tasks that are easy to evaluate, we decide that this particular direction is out-of-scope of our paper and can be a promising direction to pursue in the future.

We thank the reviewer for bringing these two points up and will add more detailed discussions in our revised version.

We thank reviewer cgKY for their encouraging and thoughtful comments.

We agree with the reviewer that evaluating the latest models optimized for long-context scenarios with RULER can give us more insights. As such, we provide additional evaluations of Gemini-1.5-pro, Command-R-plus, Mixtral-8x22B, Phi3, DBRX, Qwen along with results from our submitted version in the table below. We plan to evaluate Claude 3 and Moonshot after finalizing the evaluation pipeline of these two models.