LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K

Many thanks for your time in reviewing our paper and proposing the questions & suggestions! We answer the questions as follows. Looking forward to any further discussion!

Q1: the contributions of this paper are somewhat limited in scope. While the techniques of confusing fact insertion and keyword/phrase replacement for data augmentation may be useful, they are relatively straightforward and lack significant innovation.

Thank you for the question. We'd like to discuss our contributions in the following two aspects.

(1) Regarding the benchmarking techniques: While confusing fact insertion with keyword/phrase replacement is a straightforward approach in data augmentation, to the best of our knowledge, previous benchmarks do not include this feature and are unable to effectively evaluate LLMs on these aspects. More importantly, KPR allows us to decouple common-sense ability from long-context understanding, which we believe is crucial for accurate long-context evaluation.

(2) Regarding the resulting benchmark itself: To more effectively and fairly assess long-context LLMs, we have invested significant effort in developing this benchmark, which features a sufficiently long context length, challenging evaluation with distracting or conflicting information, and mitigates issues like knowledge leakage and overreliance on common sense. We hope and think the reulting benchmark, LV-Eval, could contribute to the community as a tool to better understand the long-context ability of LLMs.

Q2: to emphasize the contributions of the proposed benchmark, it would be beneficial to compare LV-Eval with more recent long-context benchmarks, such as “XLBench: A Benchmark for Extremely Long Context Understanding with Long-Range Dependencies” and “Loong: Benchmarking Long-Context LLMs with Extended Multi-Doc QA.” Such comparisons would provide a clearer context for evaluating the effectiveness and relevance of LV-Eval within the current landscape of long-context modeling.

Thank you for the valuable suggestion. We note that "XLBench" and "Loong" were developed concurrently with our benchmark. However, we agree that a comparison of LV-Eval with these benchmarks would provide further context for our contributions, and we have included this discussion in the revised version.

There are several key differences between LV-Eval and these two benchmarks:

1. Context Length Balance: Both Loong and XLBench are not length-balanced, meaning that test instances of different lengths do not share the same QA pairs. In contrast, LV-Eval ensures that the supporting documents and QA pairs are shared across different length levels, making it easier to track performance degradation as context length increases.
2. Application Focus: Loong is primarily focused on information extraction tasks, such as sheet analysis, which limits its scope. In contrast, LV-Eval is a more general-purpose benchmark, assessing not only knowledge extraction but also manipulation and understanding in complex long-context scenarios.
3. Benchmarking Techniques: We introduce the KPR benchmarking technique to decouple the evaluation of common-sense knowledge from the evaluation of long-context understanding, addressing a key limitation in existing long-context benchmarks. Additionally, we propose an optimized metric, leveraging manual efforts to label keywords, which makes the scoring more intuitive and interpretable (see Table 5).

We hope this comparison clarifies the distinct contributions of LV-Eval to the community. Thanks again!

Many thanks for your time in reviewing our paper and proposing the questions & suggestions! We answer the questions as follows. Looking forward to any further discussion!

Q1: It's mostly about question-answering, which might not cover everything we need for understanding long texts.

Thanks for this question. LV-Eval indeed cannot assess all types of capabilities of long-context LLMs. Its focus is on evaluating the ability to extract and manipulate knowledge from extremely long contexts, particularly in the presence of distracting or confusing information. This capability is fundamental and essential for many applications that involve long inputs. And our results suggest that this task is challenging for current LLMs.

Q2: Testing some models is pricey, so they couldn't check out all the new ones.

Yes, we believe this is an unavoidable cost if we are to benchmark the long-context capabilities of LLMs. We have provided reference results and hope that more model developers will share their performances using the standard LV-Eval, which would eliminate the need for others to retest the models.

Q3: They're still relying a lot on people to check the tricky parts, which takes time and can be off.

During the construction process, we employ human annotators to handle only the most challenging and critical tasks, such as CFI and KPR, to ensure their quality through manual verification. We hope that the resulting LV-Eval benchmark, built through this effort, will make a valuable contribution to the community.

Q4: There's a chance models could just learn the test, not actually get better at understanding.

We would like to clarify if the reviewer is asking about the scenario where the model is trained directly on the test set. If so, we must acknowledge that there is indeed a risk of benchmark overfitting, which is a common issue faced by most benchmarks.

Q5: How do these results compare to other tests, and does that tell us something about how well they do in the real world?

Thanks for the question. Fir the first question, we can see that current LLMs cannot achieve a high score on LV-Eval, with the highest Moonshot-v1-128k, Qwen2.5-72B achieving only 40% on 16k length level and lower on higher levels. Generally, it's difficult to compare the numbers between LV-Eval results and other long-context benchmarks, as we use different data construction, some benchmarking techniques to make the dataset more challenging, and an improved metric.

In the real world, we found that those LLMs that perform well in LV-Eval are typically good at long-context QA and long-context recall, but other abilities like chat, summary, math, etc., which are also important in real world applications may not be as good as long-context QA.

Q6: How they checked the tricky parts and if there's a way to do it by machine?

Thanks for the question. For the details of annotation process, please refer to Appendix B. In our preliminary exploration, we found it is hard for LLM like GPT-4 to make sure 100% correctness of CFI, although we can use prompt engineering to increase the correctness, but we still need to check each one by human annotation for zero errors.

Q7: What does it mean if models use common knowledge instead of the text, and how can we fix that for future tests?

Thanks for this question. By using common knowledge, the model can answer some questions without referring to the context, i.e., without the need of relying on its long-context understanding ability. For example, without referring to the context that is discussing biological knowledge, an LLM can answer 'red' to the question 'what the color is blood'.

The interference of common-sense knowledge in an LLM can lead to incorrect evaluations of its long-context understanding abilities, which is a critical issue for long-context benchmarks. To the best of our knowledge, we are the first to explicitly address this problem with our KPR method. Specifically, we (1) apply KPR to replace common-sense information in the context with non-common-sense information, and (2) provide explicit instructions to the LLMs to rely solely on the context provided, rather than on their internal knowledge. By doing so, if the model answers the question correctly, we can be confident that it is not relying on common-sense knowledge (since common-sense knowledge would not lead to the correct answer after KPR), but rather on its ability to understand the long context. In summary, our KPR method helps decouple the evaluation of common-sense knowledge from the evaluation of long-context understanding, addressing a key issue in existing long-context benchmarks.

(to be continued ...)

Many thanks for your time in reviewing our paper and proposing the questions & suggestions! We answer the questions as follows. Looking forward to any further discussion!

Q1: The conducted evaluation of LLMs on the LV-Eval benchmark included only 3 closed-source models. It is even more confusing that among those 3 models 2 are very outdated versions of GPT-3.5 and GPT-4. This paper would immensely benefit from inclusion of at least relatively recent closed-source LLMs, such as GPT-4 with 128k context window (which was released almost a year ago to this date), along with Anthropic Claude, which shows remarkable performance in long-context recall. The argumentation of high-cost is slightly confusing - particualrly most recent closed LLMs have been much less expensive compared to their previous versions. This might not apply to some models from Anthropic, but it certainly does apply to GPT-4. The price of GPT-3.5 with 16k context window was 3per1mtokens,forGPT−4iswas30−60 per 1m tokens, while most recent GPT-4o pricing is 2.5$ per 1m tokens.

Thanks for this question and detailed information, we really appreciate it. Despite the cost of closed LLMs having significantly reduced, the full test of gpt on all five length levels still costs 1,768.8 USD (~707.52M tokens). And unluckily, our budget for running model evaluation is running out. We can only say that if we get other reliable results on testing new models or we have got new budgets on testing new models, we will update the performance list in our project page.

Q2: Confusing Facts Insertion relies entirely on GPT-4 internal concept of a "confusing fact". This yields a problem of data leakage - GPT-4 may have an unfair benefit to other models on this benchmark.

Thank you for the great question. We did not consider the impact of this aspect in our study. Indeed, there could be an unfair effect, but we are unsure whether it gives GPT an edge or a disadvantage. This is because, unless explicitly prompted to consider confusing facts within the context, the confusing facts generated by GPT itself may be the ones that are hardest for GPT to distinguish. This is an interesting discussion point, and we've added this discussion into appendix.

Q3: Have you measured an inter-annotator agreement during annotation for CFI and KPR? How many annotators have seen each example?

For each example, two annotators conduct independent annotation, and they will recheck the other's annotation. We did not measure inter-annotator agreement during the annotation process. Since the principles of the annotation task are simple and clear, and the number of annotations is relatively small, we had the two annotators review each other's work to resolve any disagreements after completing their independent annotations. We've made this more clear in the appendix.

Q4: Have you rejected any annotations due to disagreement between annotators?

In our early overseeing of the annotation process, we found that in a very few rare cases, there is disagreement between annotators, but these were mainly due to significant conflicts in the content of the text, making deriving a single reasonable answer difficult. Therefore, we not only rejected the annotations but also removed such data from the dataset.

Many thanks for your time in reviewing our paper and proposing the questions & suggestions! We answer the questions as follows. Looking forward to any further discussion!

Q1: The quality (and usefulness) of proposed benchmark is not fully evaluated. How do state-of-the-art LLMs and human (both domain experts and lay people) perform on this dataset?

Thanks for the great question. We note that the source documents and QA pairs in LV-Eval are primarily derived from established long-context datasets such as Longbench and Loogle. Since the quality of these source documents and QA pairs has already been validated, and our construction process mainly involves synthesizing more challenging contexts from them, we think another turn of full human evaluation on our synthesized benchmark could be omitted. We also acknowledge that this decision is also driven by cost considerations, as conducting a comprehensive human evaluation on the new dataset would be prohibitively expensive. According to these two reasons, we opted not to carry out a full human evaluation on LV-Eval, as is common with synthetic datasets.

Q2: Does this benchmark capture LLM's full capability, other than knowledge extraction/manipulation capabilities for the QA tasks.

LV-Eval does not assess all types of capabilities. Its focus is on evaluating the ability to extract and manipulate knowledge from extremely long contexts, particularly in the presence of distracting or confusing information. This capability is fundamental and essential for many applications that involve long inputs. And our results suggest that this task is challenging for current LLMs.

Q3: In multiple places in the paper, the authors mention human interventions (e.g., Line 269, “ask human annotators to resolve any conflicts in generated facts”), more analysis/discussions on the human annotation qualities are needed.

We discuss the annotation details in Appendix B. Specifically, to ensure annotation qualities, we conduct checks during the annotation process. We hire master students in linguistics and LLM research, and they double check the quality of annotation, e.g., the confusing fact insertion doesn't introduce actual conflicts. Since the number of instances to be annotated is not large (the QA pairs in a dataset remain the same across different length levels), verifying the annotation quality is easy for our hired annotators.

Q4: The second limitation is its construction process is not very clearly discussed. For example, around line 254, the authors mention that “For each length level, we sample distracting documents one by one until the cumulative word count meets the desired length level“, how does the distracting document sampling work exactly?

During construction, for a given target document and its QA pair, we randomly sample irrelevant documents in the same source dataset as the distracting documents. To create context of each length level, we uniformly sample irrelevant documents (without putting back) from the same source datasets, one at a time, and concatenate them into the context. Once the overall length of the context reaches the designated length level, we stop the sampling and return the final context.

Q5: Furthermore, as mentioned in the previous limitation, for each place where human annotation/intervention is involved, more discussions (e.g., # of annotator, annotation guideline, human annotation agreement numbers, etc) are needed. I would recommend authors moving some part of current Appendix section B details to the main text.

Thanks for this suggestion. We have revised Line 211 to include some information (number and academic level of annotators), and we underline the sentence "we employ and guide human annotators to revise the confusing facts, replace the keywords and phrases, and annotate the keywords in GT answers" to make it more clear where we rely on human annotators. Due to space limitations, we were unable to include the full discussion on annotation process in the main text. If the reviewer thinks there are other places where additional information is needed, we are happy to continue the discussion.

(to be continued ...)

Thanks for your follow-up! We hope our following responses can further address the concerns.

Regarding the suggestion of using an AutoRater: Thank you for emphasizing the importance of AutoRater. Based on your suggestion, we have incorporated the AutoRater judge into our evaluation. Specifically, we selected the top 3 LLMs from our study for evaluation with AutoRater. The results suggest that AutoRater generally assigns higher scores than the metrics we proposed. Despite this, the relative model performance is still accurately reflected.

The comparison between the evaluation results using our original metric and the AutoRater version has been added to Fig. A11 and Appendix-line 912 in the revision, including the auto-rating prompts. We will update the codebase to support LLM auto-rater, which incurs an approximate cost of $20 per model evaluation. We truly appreciate your suggestion, as using AutoRater is a valuable choice, albeit with some evaluation cost.

Regarding evaluating the human performance on LV-Eval: Thanks for this suggestion again! We would like to seize the opportunity to further discuss with you and try to reach some understandings. We understand that reporting human performance on a benchmark is helpful for diagnosing the room for improvements for models, and thus adds to the usefulness of the benchmark. However, for our large and long-context LV-Eval benchmark (11 datasets on 5 length levels of 16~256k), evaluating human performance is too time-consuming and labor-intensive.

Additionally, the QA pairs in LV-Eval are primarily knowledge-retrieval tasks, which do not require complex reasoning and are relatively simple for humans to handle. We conjecture that, given sufficient time, human testers could achieve near-perfect accuracy.