HELMET: How to Evaluate Long-context Models Effectively and Thoroughly

We thank the reviewer for the helpful feedback, and we are encouraged that the reviewer finds our benchmark to be diverse and reliable, and our insights valuable. The concerns are addressed below:

[W1, Q1] may not fully capture long-context models’ capabilities in highly specific domains like legal or medical texts, limiting its applicability in niche areas… How well does HELMET handle variations in domain-specific tasks, such as medical or financial documents?

HELMET was designed to be a general-purpose benchmark, but it does include a highly specialized domain of law, as Multi-LexSum is a legal document summarization dataset. We show in Figure 9 that general-purpose datasets can also have a high correlation with performance on more specific domains. For instance, Multi-LexSum has a Spearman $\rho = 0.91$ with the $\infty$Bench Sum dataset (books). While it would be interesting to consider other domains as well, a general-purpose benchmark provides a good signal of how the models may perform in more niche areas. We leave such explorations for future work.

[W2] heavy reliance on closed models such as GPT-4 for comparison

Only 3 out of the 21 datasets from HELMET rely on GPT-4 for evaluation (Multi-LexSum, InfiniteBench Sum, and NarrativeQA); the others either rely on automatic metrics or open-sourced models. Thus, most of HELMET are evaluated in entirely open-source settings. Furthermore, we chose to use closed-source models because (1) they are easier to run than large open-source models (e.g., Llama 70B) in practice since they are only API calls and do not require multiple GPUs; (2) using closed-source models for evaluation is a widely-accepted standard practice in the community, such as AlpacaEval2 [1] and WildBench [2]; (3) closed-source models are much better judges than open-source models overall [3][4].

[Q2] Could open-source models trained on synthetic datasets achieve comparable results with additional tuning on HELMET's diverse tasks?

The use of synthetic data in long-context training is an interesting research topic and we believe it could hold huge potential in terms of boosting model performance, However, we do not recommend directly fine-tuning the models on HELMET tasks’ training sets, as this likely will lead to overfitting on specific tasks.

[1] Dubois et al., 2024. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

[2] Lin et al., 2024. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

[3] Zheng et al., 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

[4] Zeng et al., 2024. Evaluating Large Language Models at Evaluating Instruction Following

Dear reviewer,

As we approach the end of the discussion period, we would greatly appreciate your input on the paper. We hope our responses and additional results address your concerns and welcome any further questions or suggestions.

Dear Reviewer,

As the deadline for the discussion period approaches, we hope that we have addressed your concerns of HELMET's application to domain-specific tasks and the use of closed-source models during evaluation. We would greatly appreciate it if you could respond to our rebuttal. Please let us know if you have any other questions or concerns!

[Q3] which types of tasks in HELMET are compatible with the base model without instruction following capabilities?

Base models are compatible with all tasks in HELMET due to the few-shot demonstrations. We show this in Table 8—base models benefit significantly from just two in-context learning examples, and achieve non-trivial performance comparable to other instruction-tuned models. We believe that this practice is a better reflection of the base model’s long-context abilities and will be useful in model development.

We thank the reviewer for the helpful feedback, and we are encouraged that the reviewer finds our evaluation extensive and our findings valuable. We address the concerns below:

[W1, Q1] More analysis considering both the possibility of model issues and benchmark limitations… why attribute these unexpected results to benchmark unreliability rather than potential issues with the larger models themselves? Did you investigate alternative explanations for the performance discrepancies? [W3] more in-depth analysis demonstrating how HELMET improves over them in practice… more direct comparisons of model rankings or performance differences on HELMET and existing benchmarks

Please see our general response.

[W2, Q2] Do you have any results from human evaluation that validates the model-based evaluation metrics? What were the human-model agreement rates? Were there any notable discrepancies between the human judgments and model-based evaluations?

We conducted human validation of our model-based evaluation in Appendix B.5. In the rebuttal, we added a more rigorous and comprehensive human evaluation of our methods, as detailed below.

We first check the key point generation procedure, where we ask the LLM to generate key points from a long summary. We manually check 105 key points originating from 25 Multi-LexSum human-written summaries and find all claims to be factually correct and a major claim in the summary. We did notice one instance with the model excluding a possible key point, but overall, we find GPT-4o to be reliable for key point generation, which also aligns with previous findings [1][2].

Then, we verify if the judge can correctly identify if a key point is supported by the generated summary for the recall score. We randomly sample 10 generated summaries from Multi-LexSum and $\infty$Bench Sum each (generated by Llama-3.1-70B-Instruct and Gemini-1.5-Pro), and manually check the five key point evaluations for each summary (totaling 100 checks). We find a Cohen $\kappa = 0.76$ for $\infty$Bench Sum and $\kappa = 0.72$ for Multi-LexSum, suggesting substantial agreement. Qualitatively, we find that most of the disagreements between humans and the model arise from the partially supported cases. For instance, the key point may include specific details, such as the names of government departments or Court Justices’ names, that are not explicitly mentioned in the generated summary, and the model judge is typically more lenient about the exclusion of these small details while humans are more strict. However, this is also subjective to the preference of the human.

We perform a similar analysis to check if the judge can judge the precision score correctly, and we find $\kappa = 0.91$ for $\infty$Bench Sum and $\kappa = 0.83$ for Multi-LexSum, suggesting near-perfect agreement. We notice a similar trend as before – most judgments are straightforward and agreed upon, but there are some nuanced cases where humans may pay more attention to specific details whereas the model judge is more lenient.

Finally, we manually evaluate the fluency of all the generated summaries from the previous analysis, and we find that we always agree with the model judgment. This is likely due to the clear definitions of fluency in our prompt (shown in Table 10) and the easy nature of the judgment. Thus, we believe that our model-based evaluations are reliable and correlate well with human judgments. We have updated the PDF with human evaluation in Section 2.2 and Appendix B (highlighted in red).

[1] Kamoi et al., 2023. WiCE: Real-World Entailment for Claims in Wikipedia.

[2] Gao et al., 2023. Enabling Large Language Models to Generate Text with Citations

Dear reviewer,

As we approach the end of the discussion period, we would greatly appreciate your input on the paper. We hope our responses and additional results address your concerns and welcome any further questions or suggestions.

We thank the reviewer for the detailed comments, and we are encouraged that the reviewer finds HELMET to be diverse and reliable. The concerns are addressed below:

[W1] High Complexity… ​​demands considerable effort from researchers

We kindly disagree with the reviewer. (1) The complex nature of real-world long-context applications requires a holistic and diverse evaluation, which we spent a significant effort on. We show in Figure 1 that focusing on narrow domains in evaluation leads to inconsistent and unintuitive comparisons. (2) The complexity of designing the benchmark does not hinder its usability. On the contrary, we spent meticulous effort in building HELMET so that developers could evaluate diverse tasks by using just one benchmark. We also built an easy-to-use codebase where all experiments can be reproduced by one command, which we will release with the paper. (3) We also recommended using the RAG subset for fast development, which we supported with rigorous correlation analysis across different applications.

[W2] Low Correlation Among Some Tasks… challenging to assess a model’s overall long-context handling ability

A major argument of our work is the necessity of holistic evaluation of long-context language models across diverse domains—long-context abilities cannot be compressed into one number or tested by a single task, such as NIAH or perplexity (Section 3.2). Thus, HELMET offers a comprehensive evaluation across different axes that will paint a better picture of the model’s long-context performance compared to previous benchmarks.

[W3] High Resource Consumption

Thank you for pointing it out! We recommended using the synthetic recall and RAG tasks during model development for faster iterations (Section 3.1). But we also argue that compute resources required by HELMET are negligible compared to long-context training (e.g., it only takes 1 GPU for running evaluation for an 8B model but takes more than 8 GPUs for days to fine-tune it on 128K). Additionally, we show the correlation across all tasks in Figure 9—we observe that certain datasets may be used during development as a proxy for the entire category. For example, BANKING77 achieves a high correlation with the rest of the ICL tasks while InfBench MC also achieves a high correlation with the Long-Document QA performances. We will include these recommendations in the revision. Furthermore, we optimize our public code repo with tools such as FlashAttention and vLLM to reduce the cost of running HELMET. Consequently, the entire HELMET evaluation of an 8B model at 128K input lengths finishes within 16 hours on one H100 GPU (most datasets take less than 30 minutes to run except for the long generation tasks). The Recall set can be completed in less than 90 minutes for faster development.

Dear reviewer,

As we approach the end of the discussion period, we would greatly appreciate your input on the paper. We hope our responses and additional results address your concerns and welcome any further questions or suggestions.

We thank the reviewer for the detailed and helpful comments. We are encouraged that the reviewer finds HELMET comprehensive and that our findings are insightful. The concerns are addressed below:

[W1] The so called "expected" ranking of LLMs is a bit subjective.

Please refer to our general response.

[W2] Lack of some deep analysis to interesting results… why the json-kv task has higher correlation with re-rank than RAG or LongQA

To clarify, we find, in Figure 8, that RAG overall achieves a higher correlation with Re-rank ($\rho = 0.9$) than JSON KV ($\rho=0.83$). LongQA also observes a high correlation with Re-rank ($\rho = 0.89$; Figure 9). This is intuitive because RAG requires the model to leverage retrieved passages, similar to passage re-ranking.

We agree that there are more interesting possible analyses in this setting, and we dive into them in Appendix E, where we analyze the correlation between individual datasets, challenges in positional embeddings, the lost-in-the-middle problem, and comparison between base and instruction-tuned models. Furthermore, we have since added more qualitative analysis into the failure of specific models; for example, we find some closed-source models, such as Claude, tend to not follow the instructions in the ICL tasks. The updated PDF includes these new analyses in Appendix E.6. We hope the release of HELMET will enable more interesting analyses and model development in the field of long contexts. We plan to release an easy-to-use code repository for the community to use.

[W3] With 8k context, Llama3 should use at least a scaling factor of 32 for 128k testing.

Thank you for the suggestion, we show the results on HELMET at 128K input length for the Llama 3 models with RoPE Theta base set to 8M (scaling factor of 16) and 16M (scaling factor of 32) below. The new results are highlighted in bold:

We find that the model performances do improve, but the absolute changes are still relatively small, and the models degenerate on most tasks, remaining at 0% performance. We will include these results in the final revision.

[Q1] Figure 2 is missing?

Figure 2 is located at the top of Page 6.

[Q2] What is the value for 'depth' in Figure 11?

Depth is the location of the gold passage (or the location of the needle) that contains the answer to the question. From top to bottom, the gold passage moves from the very start of the context (depth=0.0) to the very end of the context (depth=1.0). The depth is evenly spaced out and takes on the values of {0.0, 0.2, 0.4, 0.6, 0.8, 1.0}.

[Q3] It would be better to have results with Gemma series as the tested models.

Thank you for the suggestion, we show Gemma-2 results on HELMET at 128K input length below, where the new results are highlighted in bold:

We test both the base models and the instruction-tuned models (θ denotes changing the RoPE Theta base from 10k to 320k). Since the Gemma-2 models were trained with a context window of 8K tokens, they often degenerate on long-context tasks. We will include these results and analyses in the final revision.

Dear reviewer,

As we approach the end of the discussion period, we would greatly appreciate your input on the paper. We hope our responses and additional results address your concerns and welcome any further questions or suggestions.

5. In practice, these improvements result in HELMET more accurately reflecting model performance. We take a closer look at a subset of open-sourced models’ ranking and performance on HELMET and $\infty$Bench below. The new results added during the rebuttal are highlighted in bold.

Other than the performance discrepancy between Llama-3.1-8B-Inst and Llama-3.1-70B-Inst, which we noted in Figure 1, we also see that $\infty$Bench shows that all Llama-3.2 models degenerate at long contexts (128K tokens). However, on HELMET, we find that the Llama-3.2 models, especially 3B-Inst, rank well against other open-source models. Upon qualitative examination, we find that the Llama 3.2 models are able to produce coherent and useful generations at long contexts with better prompting strategies from HELMET, such as adding in-context learning examples. Thus, HELMET provides a better reflection of how these models would be used in practice over previous benchmarks.

We appreciate the reviewers for the helpful feedback, and we have updated the PDF with the additional discussions in the introduction and Appendix A (new edits are highlighted in red).

[1] Deutsch et al., 2022. Re-examining system-level correlations of automatic summarization evaluation metrics

[2] Goyal et al., 2023 News Summarization and Evaluation in the Era of GPT-3

[3] Chang et al., 2024. Booookscore: A systematic exploration of book-length summarization in the era of llms

[4] Gemini Team, Google, et al., 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.