LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

W1: Differences between STIC-1 and STIC-2

Thank you for your insightful comments regarding the need for an example to illustrate the differences between STIC-1 and STIC-2. We appreciate your feedback and have included a comparative example in the revised manuscript, specifically referencing results from Table 3 of our experiments, which compare LLaMA3.1-8B and Qwen2 under the short-version setting. STIC-1 and STIC-2 are designed to evaluate instruction adherence at different levels of granularity:

• STIC-1 measures the average success rate of individual instructions within the actual completed portion of a task. For instance, STIC-1 evaluates the correctness of each generated output segment relative to the portion of the task completed, without penalizing for incomplete task segments. This allows STIC-1 to reflect the model’s consistency in following instructions within its generated content.
• STIC-2, in contrast, provides a comprehensive assessment of output completeness. This metric evaluates a task as a whole, counting it as successful only if all specified instructions across the full task length are followed correctly. STIC-2 thus captures the model’s ability to handle long and complex tasks comprehensively, without any partial completion.

Example Comparison

The table below compares LLaMA3.1-8B and Qwen2 to illustrate how these metrics diverge:

In this case, Qwen2 achieves a higher STIC-1 score than LLaMA3.1-8B but a lower STIC-2 score. This difference arises from the models’ varying Completion Rates (CR). Qwen2 typically achieves a 60% completion rate, akin to completing approximately 60 floors of a 100-story skyscraper design task, while LLaMA3.1-8B completes closer to 93 floors.

For STIC-1, Qwen2 scores higher since its evaluation is based only on the 60 floors it successfully generates, compared to LLaMA3.1-8B’s 93 floors. STIC-1 does not penalize Qwen2 for the missing floors, focusing instead on the instruction adherence within the portion generated. In contrast, STIC-2 evaluates the completeness of the entire task; since Qwen2 does not generate the remaining 40 floors, its STIC-2 score is negatively impacted due to this incomplete output.

We trust that this explanation clarifies the distinctions between STIC-1 and STIC-2, and we thank you for the opportunity to expand on these metrics in our revised manuscript.

W2: Abbreviation "CR" in Table 3

Thank you for pointing out that the abbreviation "CR" in Table 3 was not defined clearly. Your understanding is correct: "CR" stands for Completion Rate in Main Task Completion. This metric measures the proportion of the main task completed by the model within the given constraints, providing a quantitative indicator of progress and adherence to the task's full requirements.

We have updated the corresponding section to clearly define "CR" upon its first occurrence to ensure clarity for all readers. We appreciate your attention to detail and your assistance in improving the overall quality of our work.

W4: Missing X-ticks in Figure 2

Thank you for pointing out the missing x-ticks in Figure 2. The x-axis represents the token count of the output length, and we will update Figure 2 to include x-ticks for clarity. We appreciate your attention to detail and your assistance in improving the overall quality of our work.

W5:Evaluation of Correctness

Thank you for your questions and for prompting us to clarify our evaluation process further. We realize our original explanation may have lacked detail, so we are pleased to provide a more comprehensive breakdown of the evaluation pipeline here, as also outlined in Section 2.5 of the paper.

Our evaluation pipeline systematically assesses the ability of long-context LLMs to follow specific, complex instructions. The process can be summarized in three key steps:

1. Generation of Outputs from the Long-context LLM

Given an input task (T) that describes a set of instructions, we prompt the LLM to generate detailed outputs. The output (A) comprises a list of descriptions, represented as: A = {A1, A2, ..., An}

Example: Given the prompt (ref Appendix SCENARIO)

Construct a skyscraper with 100 floors. The floor assignments are detailed as follows:

• Specific floor requirement: Designate Floor 11 for a small art gallery.
• Range floor requirement: Allocate Floors 32 to 39 for corporate headquarters of a major company.
• ...

The LLM generates a response describing each floor in detail, such as:

• Floor 1: ... Lobby ...
• ...
• Floor 11: ... Small art gallery ...
• ...
• Floor 32: ... Corporate headquarters ...
• ...
• Floor n: ...

2. Extracting and Matching Relevant Floor Assignments (Check Set)

From the initial input ("T"), we create a check set containing specific floor assignments to verify if the LLM correctly follows the instructions.

For the example above, the check set includes:

• Floor 11: Small art gallery
• Floor 32: Corporate headquarters
• Floor 33: Corporate headquarters
• ...

We then extract the relevant parts of the LLM output ("A") that correspond to the floor assignments described in the check set.

3. Evaluation Using Llama 3.1-8B instruction Model

For each extracted pair, we use the Llama 3.1-8B model to evaluate whether the output ("Ai") for a given task segment ("Tsi") has correctly fulfilled the specified instruction.

This evaluation task is framed as a simple binary classification problem, which aims to determine if the specific instruction was fulfilled ("yes" or "no"). The prompt used for this evaluation is as follows:

Evaluation Prompts

• Example 1: XXXX Answer: Analysis + #*# Yes
• Example 2: XXXX Answer: Analysis + #*# No
• Context: Long-context model output: "Floor 11: ... small art gallery ..."
• Instructions: Does this context include 'small art gallery'?
• Answer: Please refer to the above example, provide your analysis, and respond with either ## Yes or ## No.

Notably, this binary evaluation is straightforward. We manually labeled 300 data points, and the model's output matched human evaluations for all cases.

By segmenting the long-generation task into smaller units and evaluating each one individually, our approach offers a thorough and systematic method to verify instruction adherence across the full sequence. This ensures that the LLM’s performance on each component of the task can be accurately and efficiently assessed. We hope this detailed explanation clarifies our approach, and we thank you for the opportunity to elaborate on our evaluation methodology.

W6: 16K/32K Length Definition in Table 3

Thank you for pointing out the potential confusion around the “16K/32K” length designation in Table 3. To clarify, the term "16K/32K" refers to the required number of tokens for the model output, aligning with standard conventions when discussing model context lengths. However, for evaluating the actual generated output, we used word count as the measurement unit.

This approach was taken for two key reasons:

• Token-to-Word Conversion Variability: Different tokenizers can vary in how they map tokens to words, typically resulting in an average conversion ratio of approximately 1.5 tokens per word. Consequently, the actual word count of the output is generally around two-thirds of the target token length.

• Practical Focus on Output Content: Since our evaluation prioritizes assessing the quality and completeness of the final output, using word count provides a clearer perspective on the generated content itself. We will revise the paper to make this distinction more explicit, ensuring clarity for readers regarding our choice of measurement and terminology. Thank you again for highlighting this area, as it allows us to improve the precision of our explanations.

Thank you for your careful reading and for delving into the discussion. I really appreciate your example; it's clearer and simpler than the one I provided. Allow me to use it to explain the concepts further.

First, I've slightly modified your example from a 2-level building to a 3-level building to make the explanation clearer:

Suppose we are planning a "3"-level building with 3 constraints:

• TS_1: "floor1 must have a coffee shop"
• TS_2: "floor1 must have a reception desk"
• TP: {TP_1, TP_2, TP_3}, where each TP_i means "floor i must have a washroom"

The model generates:

"floor1: coffee shop, washroom; floor2: washroom."

In this scenario, my check_set is {TS_1, TS_2, TP_1, TP_2, TP_3}. It's important to note that TP actually applies to all three floors, meaning we need to evaluate each TP_i separately.

In our actual evaluation, we strive to ensure that TS, TR, and TP have similar evaluation frequencies.

With the current model output, the completion rate (CR) for the main task is 2/3. Although the task requires outputs for 3 floors, the model only provided outputs for 2 floors.

Now, for STIC-1, we consider how accurately the model has outputted information at the floor level. Since the model output only contains two floors, we evaluate the constraints for these two floors to see if they are fully met. For these two floors, the constraints are TS_1, TS_2, TP_1, TP_2, totaling 4 constraints. The model has correctly fulfilled 3 out of these 4 requirements, which means STIC-1 is 3/4.

For STIC-2, we evaluate the entire check_set, which consists of TS_1, TS_2, TP_1, TP_2, TP_3. The model has fulfilled 3 out of these 5 requirements, so STIC-2 equals 3/5.

The distinction between STIC-1 and STIC-2 allows us to identify the specific reasons for any drop in performance. It helps to determine whether the issue lies in the model's inability to follow instructions for a given output or whether it lacks a complete output in the first place. For example, in the case of a lower STIC-2, is the low score due to having some floor outputs that are incorrect, or is it because there is no complete output for the floors at all? In such cases, we can use CR and STIC-1 together to further evaluate and make judgments.

Once again, we greatly appreciate your comment and the opportunity to clarify the computation of our metrics. We will add this discussion in the appendix to improve the clarity of our paper. If this response addresses your concern, we hope you will consider raising the score. If there are further questions, we are happy to clarify further :)

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

W1:Evaluation of Correctness

Thank you for your questions and for prompting us to clarify our evaluation process further. We realize our original explanation may have lacked detail, so we are pleased to provide a more comprehensive breakdown of the evaluation pipeline here, as also outlined in Section 2.5 of the paper.

Our evaluation pipeline systematically assesses the ability of long-context LLMs to follow specific, complex instructions. The process can be summarized in three key steps:

1. Generation of Outputs from the Long-context LLM

Given an input task (T) that describes a set of instructions, we prompt the LLM to generate detailed outputs. The output (A) comprises a list of descriptions, represented as: A = {A1, A2, ..., An}

Example: Given the prompt (ref Appendix SCENARIO)

Construct a skyscraper with 100 floors. The floor assignments are detailed as follows:

• Specific floor requirement: Designate Floor 11 for a small art gallery.
• Range floor requirement: Allocate Floors 32 to 39 for corporate headquarters of a major company.
• ...

The LLM generates a response describing each floor in detail, such as:

• Floor 1: ... Lobby ...
• ...
• Floor 11: ... Small art gallery ...
• ...
• Floor 32: ... Corporate headquarters ...
• ...
• Floor n: ...

2. Extracting and Matching Relevant Floor Assignments (Check Set)

From the initial input ("T"), we create a check set containing specific floor assignments to verify if the LLM correctly follows the instructions.

For the example above, the check set includes:

• Floor 11: Small art gallery
• Floor 32: Corporate headquarters
• Floor 33: Corporate headquarters
• ...

We then extract the relevant parts of the LLM output ("A") that correspond to the floor assignments described in the check set.

3. Evaluation Using Llama 3.1-8B instruction Model

For each extracted pair, we use the Llama 3.1-8B model to evaluate whether the output ("Ai") for a given task segment ("Tsi") has correctly fulfilled the specified instruction.

This evaluation task is framed as a simple binary classification problem, which aims to determine if the specific instruction was fulfilled ("yes" or "no"). The prompt used for this evaluation is as follows:

Evaluation Prompts

• Example 1: XXXX Answer: Analysis + #*# Yes
• Example 2: XXXX Answer: Analysis + #*# No
• Context: Long-context model output: "Floor 11: ... small art gallery ..."
• Instructions: Does this context include 'small art gallery'?
• Answer: Please refer to the above example, provide your analysis, and respond with either ## Yes or ## No.

Notably, this binary evaluation is straightforward. We manually labeled 300 data points, and the model's output matched human evaluations for all cases.

By segmenting the long-generation task into smaller units and evaluating each one individually, our approach offers a thorough and systematic method to verify instruction adherence across the full sequence. This ensures that the LLM’s performance on each component of the task can be accurately and efficiently assessed. We hope this detailed explanation clarifies our approach, and we thank you for the opportunity to elaborate on our evaluation methodology.

W2:Benchmark Scope and Contributions

Thank you for your feedback regarding the scope of domains and scenarios in our benchmark. We believe that the key contribution of our paper lies not merely in the variety of domains but in establishing a systematic framework for evaluating instruction-following capabilities specifically tailored to long-context generation—a critical yet underexplored area in LLM research.

Our benchmark addresses the essential challenge of instruction adherence over extended text generation, which is foundational to long-form tasks in real-world applications. To rigorously evaluate this capability, we designed four distinct scenarios with two length variations each, resulting in a total of 800 test samples and approximately 14,000 instructions. This setup is substantial and exceeds the size of many existing datasets focused on creative writing [1] and instruction-following [2,3]. Through this carefully structured benchmark, our work introduces a comprehensive, replicable framework that captures key aspects of long-context generation, making it possible to directly compare models on their ability to maintain instruction adherence across extended sequences.

We recognize that our current benchmark does not yet encompass all potential real-world applications, and expanding to more diverse domains is indeed an important future direction. However, we are confident that this initial framework fills a critical gap by focusing on instruction-following in long-form text generation, establishing a strong foundation upon which future benchmarks can build. Additionally, we plan to broaden the benchmark to include narrative coherence, factual consistency, and creative tasks in subsequent iterations and will be introducing a public leaderboard to support ongoing evaluation and continuous model integration.

We appreciate your feedback, which underscores valuable directions for further enhancement, and believe that our work makes a meaningful contribution by creating the first structured and scalable benchmark specifically for assessing long-context generation instruction-following capabilities in LLMs.

References:

[1] Comparison of Evaluation Metrics for Short Story Generation.

[2] Instruction-following Evaluation for Large Language Models.

[3] Infobench: Evaluating Instruction Following Ability in Large Language Models.

Q1: IFEval-style Setting and Code-based Verification

Thank you for this insightful suggestion. We have thoroughly reviewed IFEval, and we acknowledge that it effectively uses code checks to ensure correctness for certain attributes, such as output length and basic properties. This approach is beneficial for simpler, well-defined tasks where outputs can be evaluated through direct comparisons. However, IFEval has limitations when applied to more complex, nuanced characteristics like instruction-following accuracy, particularly for long-form generation tasks that require adherence to sophisticated requirements.

As we mentioned in our response to W1, our approach addresses this gap by enabling evaluation of long-form outputs with a focus on verifying specific requirements from the prompts by LLMs. This focus is crucial for ensuring adherence to complex instructions, as it allows us to assess whether the model meets each component of the prompt accurately and consistently over extended text sequences.

While we recognize the value of code-based checks, especially for tasks that require precise formatting or strict structural adherence, our current benchmark prioritizes the flexible evaluation necessary for long-form generation. That said, we are open to exploring how elements of IFEval or similar methodologies could complement our framework, particularly for tasks where structural constraints are more relevant.

If there are additional aspects of IFEval or other methodologies you would like us to consider further, we would be more than happy to incorporate them into our analysis. We look forward to any further suggestions you may have.

Reference:

[1] IFEval: An Integrated Framework for Evaluating Instruction Following in LLMs.

Dear Reviewer hgz3,

Once again, thank you for your valuable feedback on our paper. We hope our clarifications and revisions have resolved the issues you highlighted. If there are any remaining questions or areas where further clarification would be helpful, we would be more than happy to address them promptly.

As we are nearing the end of the rebuttal period, we kindly request you consider raising our paper's score if our updated responses have addressed your concerns.

Thank you for your time and effort in reviewing our work.

Best regards,

Authors of Submission Number: 3588

Dear Reviewer hgz3,

Thank you once again for your thoughtful feedback on our paper. We appreciate the time and effort you have invested in reviewing our work.

As the Discussion Period is nearing its conclusion, we wanted to kindly follow up to ensure that you’ve had a chance to review our response to your comments. We hope that our rebuttal has addressed your concerns satisfactorily. If there are any additional points or further clarifications required, we would be happy to provide them promptly.

Additionally, if our response has sufficiently addressed your concerns, we would greatly appreciate if you could consider reflecting this in your evaluation, including revisiting your score if appropriate.

Thank you for your time and understanding. We greatly value your feedback and look forward to hearing from you.

Cheers,

Authors of Paper Submission 3588

Thank you for adding the response parsing and evaluation details in the response and the paper. I will maintain my score for the following two reasons (which are somewhat related):

1. LLM-as-a-judge style evaluations are alright for domains such as long context creative writing in benchmarks such as LongBench-Write [1]. However, in evaluating faithfulness in following instructions over long generations, an LLM-based evaluator leaves much to be desired, as it is brittle to generation parameters and the generation/formatting styles of the model under test.

2. I am not convinced that a long context benchmark that isn't grounded in some deterministically verifiable ground truth (like IFEval [2]) is consigned to be simplistic. In fact, in the math and code domain (which most recent LM releases are trained on), we have observed several challenging benchmarks in the last few months, which do a very good job of testing long context understanding and generation while also evaluating based on execution or answer correctness [3,4,5,6].

[1] Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li: LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs. CoRR abs/2408.07055 (2024)

[2] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, Le Hou: Instruction-Following Evaluation for Large Language Models. CoRR abs/2311.07911 (2023)

[3] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R. Narasimhan: SWE-bench: Can Language Models Resolve Real-world Github Issues? ICLR 2024 2023

[4] Nam Le Hai, Dung Manh Nguyen, Nghi D. Q. Bui: REPOEXEC: Evaluate Code Generation with a Repository-Level Executable Benchmark. CoRR abs/2406.11927 (2024)

[5] Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, Timofey Bryksin: Long Code Arena: a Set of Benchmarks for Long-Context Code Models. CoRR abs/2406.11612 (2024)

[6] Lei Wang, Shan Dong, Yuhui Xu, Hanze Dong, Yalu Wang, Amrita Saha, Ee-Peng Lim, Caiming Xiong, Doyen Sahoo: MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs. CoRR abs/2410.04698 (2024)

Thank you for your feedback! Please allow me to elaborate further:

R1: Response to LLM-as-Jugde and LongBench-Write

We recognize the challenges associated with this approach, particularly its sensitivity to generation parameters and formatting styles. However, our work distinguishes itself by addressing these limitations through a structured and systematic evaluation process tailored specifically for long-form instruction adherence.

Our paper introduces a novel benchmark specifically designed to evaluate the ability of large language models (LLMs) to follow detailed instructions over extended outputs. Unlike creative writing tasks, such as those in LongWrite [1], our benchmark emphasizes instruction adherence in practical, structured tasks with significantly longer outputs (average lengths nearly ten times greater). This focus tackles the unique challenges of long-context generation, including consistency, memory retention, and instruction fidelity, which are foundational for advancing LLM capabilities in real-world applications.

To ensure robust evaluation, we adopt a segment-based approach. Instead of relying solely on an LLM-as-a-judge paradigm, we split long outputs into manageable sub-segments (generally 100–200 words) and systematically verify whether each segment meets the given instructions. For example, if the instruction specifies that “Floor 34 must include a coffee shop”, we evaluate whether this requirement is explicitly satisfied in the corresponding segment (Appendix C & Comment(1/2)). This method minimizes dependencies on generation parameters or formatting styles and provides a transparent, reproducible evaluation framework.

While LongWrite uses LLM-as-a-judge, which has faced critiques for its brittleness and susceptibility to formatting (see https://openreview.net/forum?id=kQ5s9Yh0WI), our evaluation methodology is tailored to address these limitations. Furthermore, Long Write’s average output length of 2,772 words represents a fundamentally different scope, focusing on creative writing rather than instruction adherence over ultra-long contexts.

By emphasizing long-context instruction-following, we provide a critical framework for assessing a capability that underpins many real-world applications. This distinction sets our benchmark apart as a foundational contribution to the evaluation of long-context LLM outputs. We hope this clarifies the contributions and robustness of our evaluation.

R2: Response to Long-Context Benchmark and Deterministic Verifiability

Below, we summarize the average token counts for the datasets and benchmarks you referenced:

IFEval [2]: 344 tokens.

SWE-BENCH[3]: 120 words (Table 1)

REPOEXEC [4]: 78.46 tokens (Table 2).

Long Code Arena: A Set of Benchmarks for Long-Context Code Models [5]:

• Library-based Code Generation: generates a single file (largest task).
• Project-Level Code Completion: typically generates single-line code.
• Average file size: 32.5 lines (Table 8).

MATHHAY: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs [6]: Focuses on processing long inputs rather than generating long outputs, combining information-seeking and mathematical reasoning tasks.

While these works demonstrate strong capabilities in long-context understanding, they do not primarily address long-context generation, which we define as producing outputs that exceed 4K tokens, with some tasks in our benchmark requiring significantly longer outputs. Our focus is distinct in evaluating the challenges of maintaining coherence, adhering to instructions, and handling memory over ultra-long text outputs.

This makes our work among the first to systematically benchmark long-context generation. Existing benchmarks, such as IFEval and REPOEXEC, primarily focus on short or moderately long outputs with deterministic evaluation criteria, such as correctness checks for code or mathematical reasoning. These approaches, while valuable, are not directly applicable to the open-ended and complex nature of long-context generation tasks (No ground truth), which require a broader and more flexible evaluation framework.

Given the reasons mentioned above, we believe that using LLMs for fragment evaluation to determine whether an instruction has been completed will not lead to the issues you described. As you noted, existing works like Longwrite are limited to using LLMs for quality evaluation. In contrast, our approach provides greater explainability in evaluation, allowing us to directly identify model output errors instead of relying on vague metrics such as creativity.

Let me know if this works for you or if you'd like further adjustments!

We would like to re-iterate our contribution in this work:

Instruction-following Over Extended Outputs: We tackle the problem of models adhering to detailed and complex instructions across outputs far longer than those evaluated in previous benchmarks. This requires addressing issues such as instruction retention, content diversity, and semantic correctness at scale.

Foundational Benchmark for Long-context Generation: While prior works have laid the groundwork for long-context understanding, we focus specifically on generation challenges, providing a structured framework that enables rigorous evaluation of these capabilities.

By addressing these unique challenges, our work sets the stage for future research in long-context generation. It provides a foundational benchmark for the community to systematically assess model performance in scenarios that demand both scale and complexity.

We hope this clarifies the significance of our contributions compared to the benchmarks you referenced.

[1] Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li: LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs. CoRR abs/2408.07055 (2024)

[2] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, Le Hou: Instruction-Following Evaluation for Large Language Models. CoRR abs/2311.07911 (2023)

[3] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R. Narasimhan: SWE-bench: Can Language Models Resolve Real-world Github Issues? ICLR 2024

[4] Nam Le Hai, Dung Manh Nguyen, Nghi D. Q. Bui: REPOEXEC: Evaluate Code Generation with a Repository-Level Executable Benchmark. CoRR abs/2406.11927 (2024)

[5] Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, Timofey Bryksin: Long Code Arena: a Set of Benchmarks for Long-Context Code Models. CoRR abs/2406.11612 (2024)

[6] Lei Wang, Shan Dong, Yuhui Xu, Hanze Dong, Yalu Wang, Amrita Saha, Ee-Peng Lim, Caiming Xiong, Doyen Sahoo: MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs. CoRR abs/2410.04698 (2024)

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

W2: Structure and Independence of Sub-tasks in Long Content

Thank you for your insightful comments regarding coherence and content interaction in long-form text generation. We recognize that coherence and interaction are essential for the community’s broader goals in long-text generation research. In our paper, we clarify that our current focus is not primarily on evaluating coherence across long outputs but rather on exploring the foundational aspects of long-text generation—specifically, the model’s ability to follow complex instructions over an extended sequence.

Our benchmark is designed to establish a robust baseline for instruction-following capabilities, which we believe is a necessary precursor to effectively handling the more nuanced demands of content coherence and interaction in long-form generation. This focus is similar to the evolution of benchmarks like NIAH, which initially assessed long-context retrieval, leading to more advanced frameworks like RULER and NeedleBench that require deeper understanding and nuanced content interaction.

In tasks like diary generation or skyscraper design, while individual entries may seem independent, the model must follow overarching instructions consistently over time, a quality essential for applications involving long-context generation. These tasks test whether models can remember and adhere to periodic and complex directives over extended sequences without "drifting" or forgetting instructions.

While we acknowledge that content coherence and interactivity are critical dimensions for future work, our benchmark aims to test foundational capabilities first. As the benchmark evolves, we are committed to introducing tasks that address coherence and interdependent content more thoroughly, which will enhance the benchmark’s comprehensiveness. We look forward to advancing our work in this direction and appreciate your feedback, which will help inform our future iterations.

Q1/Q2: Sub-task Splitting and Instruction Satisfaction Evaluation

Thank you for your questions and for prompting us to clarify our evaluation process further. We realize our original explanation may have lacked detail, so we are pleased to provide a more comprehensive breakdown of the evaluation pipeline here, as also outlined in Section 2.5 of the paper.

Our evaluation pipeline systematically assesses the ability of long-context LLMs to follow specific, complex instructions. The process can be summarized in three key steps:

1. Generation of Outputs from the Long-context LLM

Given an input task (T) that describes a set of instructions, we prompt the LLM to generate detailed outputs. The output (A) comprises a list of descriptions, represented as: A = {A1, A2, ..., An}

Example: Given the prompt (ref Appendix SCENARIO)

Construct a skyscraper with 100 floors. The floor assignments are detailed as follows:

• Specific floor requirement: Designate Floor 11 for a small art gallery.
• Range floor requirement: Allocate Floors 32 to 39 for corporate headquarters of a major company.
• ...

The LLM generates a response describing each floor in detail, such as:

• Floor 1: ... Lobby ...
• ...
• Floor 11: ... Small art gallery ...
• ...
• Floor 32: ... Corporate headquarters ...
• ...
• Floor n: ...

2. Extracting and Matching Relevant Floor Assignments (Check Set)

From the initial input ("T"), we create a check set containing specific floor assignments to verify if the LLM correctly follows the instructions.

For the example above, the check set includes:

• Floor 11: Small art gallery
• Floor 32: Corporate headquarters
• Floor 33: Corporate headquarters
• ...

We then extract the relevant parts of the LLM output ("A") that correspond to the floor assignments described in the check set.

3. Evaluation Using Llama 3.1-8B instruction Model

For each extracted pair, we use the Llama 3.1-8B model to evaluate whether the output ("Ai") for a given task segment ("Tsi") has correctly fulfilled the specified instruction.

This evaluation task is framed as a simple binary classification problem, which aims to determine if the specific instruction was fulfilled ("yes" or "no"). The prompt used for this evaluation is as follows:

Evaluation Prompts

• Example 1: XXXX Answer: Analysis + #*# Yes
• Example 2: XXXX Answer: Analysis + #*# No
• Context: Long-context model output: "Floor 11: ... small art gallery ..."
• Instructions: Does this context include 'small art gallery'?
• Answer: Please refer to the above example, provide your analysis, and respond with either ## Yes or ## No.

Notably, this binary evaluation is straightforward. We manually labeled 300 data points, and the model's output matched human evaluations for all cases.

By segmenting the long-generation task into smaller units and evaluating each one individually, our approach offers a thorough and systematic method to verify instruction adherence across the full sequence. This ensures that the LLM’s performance on each component of the task can be accurately and efficiently assessed. We hope this detailed explanation clarifies our approach, and we thank you for the opportunity to elaborate on our evaluation methodology.

Dear Reviewer p7FN,

Once again, thank you for your valuable feedback on our paper. We hope our clarifications and revisions have resolved the issues you highlighted. If there are any remaining questions or areas where further clarification would be helpful, we would be more than happy to address them promptly.

As we are nearing the end of the rebuttal period, we kindly request you consider raising our paper's score if our updated responses have addressed your concerns.

Thank you for your time and effort in reviewing our work.

Best regards,

Authors of Submission Number: 3588

Dear Reviewer p7FN,

Thank you once again for your thoughtful feedback on our paper. We appreciate the time and effort you have invested in reviewing our work.

As the Discussion Period is nearing its conclusion, we wanted to kindly follow up to ensure that you’ve had a chance to review our response to your comments. We hope that our rebuttal has addressed your concerns satisfactorily. If there are any additional points or further clarifications required, we would be happy to provide them promptly.

Additionally, if our response has sufficiently addressed your concerns, we would greatly appreciate if you could consider reflecting this in your evaluation, including revisiting your score if appropriate.

Thank you for your time and understanding. We greatly value your feedback and look forward to hearing from you.

Cheers,

Authors of Paper Submission 3588

Many thanks four reviewing our paper and raising the score! We greatly appreciated your constructive review, which improved our paper!

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

W1: Limited Task Scenarios

Thank you for your feedback regarding the variety of task scenarios in our benchmark. In this initial study, we intentionally narrowed the scope to focus specifically on evaluating instruction-following capabilities within the realm of long-text generation. Our goal was to address a critical component of long-text generation systems that serves as a foundation for more complex and comprehensive tasks.

Evaluating the full performance of long-text generation in diverse applications is indeed a significant challenge that requires ongoing and extensive research efforts. Our contribution to this complex field is to spotlight instruction adherence as a key aspect, which we believe is pivotal for the success of long-text generation systems. By focusing on this foundational element, we aim to provide a structured approach that can serve as a stepping stone for more comprehensive evaluations.

For future iterations, we plan to expand the benchmark to cover a broader range of scenarios, including tasks that test narrative coherence, factual consistency, and reasoning. These additions will allow us to capture more advanced aspects of long-text generation and increase the benchmark’s applicability to a wider array of real-world applications. We appreciate your feedback, as it highlights the importance of broadening our benchmark to evaluate additional capabilities, and we look forward to making these expansions in future releases.

W2/Q1: Scenario-specific Evaluation Metrics

Thank you for your observations regarding the scenario-specific nature of our evaluation metrics. We believe that there is a strong intrinsic connection between the metric and the task it evaluates. Should the proposed metric effectively assess model performance for these tasks, its suitability is affirmed, like FUAR metric to Continual knowledge learning[1], TRUST-SCORE metric to trustworthiness RAG[2].

In this benchmark, we designed metrics specifically to assess instruction-following capabilities in long-text generation, similar to how the FUAR and TRUST-SCORE. For the present scenarios, our metrics effectively measure compliance with provided instructions and offer a clear view of how well models handle extended, instruction-based tasks.

Flexibility for Future Scenarios

While our current metrics are customized to evaluate instruction-following, we recognize that additional metrics would be needed to capture broader dimensions of long-text generation, such as narrative coherence, fluency, and factual consistency (please ref Section 4 ANALYSIS AND LIMITATIONS). Should we expand the benchmark to include tasks with these aspects, we would introduce complementary metrics suited to those evaluation goals. For example, creative storytelling tasks may benefit from metrics focusing on coherence and thematic consistency, while technical report generation could require metrics assessing factual accuracy and data integrity.

We appreciate your feedback, which will guide us in adapting our metrics to accommodate additional tasks and evaluation needs in future releases.

References:

[1] Towards Continual Knowledge Learning of Language Models.

[2] Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse.

W3: Limited Number of Models Evaluated

Thank you for your constructive feedback. In this work, we conducted an extensive evaluation that included models of various sizes and sources, encompassing both open-source and close-source options. To address your suggestion, we have incorporated additional models, including Phi-3-mini-instruct, Phi-3.5-MOE-instruct [1], FILM-7B [2], and Mamba-2.8B [3] into our experiments, as shown in the table below:

All of the above models perform poorly, which we attribute to the lack of SFT training in the long output case. On the other hand, since most of the APIs for closed-source models can only support outputs up to 4K, we are not able to perform a more comprehensive evaluation of closed-source models. In addition, we plan to create a public leaderboard, where we will continuously integrate more models to enable broader comparisons and ensure transparency. This leaderboard will allow researchers to track model performance on long-text generation tasks, making it easier to evaluate newer models as they emerge. If there are specific models you would like us to include, please let us know, and we would be more than happy to add them to our analysis.

References:

[1] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.

[2] Make Your LLM Fully Utilize the Context.

[3] Mamba: Linear-Time Sequence Modeling with Selective State Spaces.

Q2: Other Aspects of Long Text Generation with LLMs

Thank you for your insightful observation regarding the limitations of our current approach. As noted in Section 4 (ANALYSIS AND LIMITATIONS) of our paper, we recognize that our current research focuses primarily on evaluating instruction-following capabilities, while a more comprehensive analysis of content coherence and rationality remains an area for future investigation.

Our work represents the first comprehensive evaluation focused on long-context generation, providing a structured and replicable foundation for assessing large language models (LLMs) in this critical area. By systematically evaluating instruction-following and planning capabilities in extended contexts, our benchmark addresses foundational skills that are essential for any long-form generation task. This approach offers the community a baseline for understanding model performance in long-context settings, allowing future research to build on this framework as evaluation methods and LLM capabilities evolve.

We agree that evaluating other aspects of long-text generation, such as creative tasks like novel writing, presents unique challenges that go beyond structured or planning-based tasks. Designing effective evaluation metrics for open-ended creative content is inherently complex, as it requires balancing subjective qualities like narrative coherence, creativity, and reader engagement—elements that are more difficult to quantify objectively.

We appreciate your feedback, which highlights valuable directions for expanding our benchmark to cover more diverse aspects of long-text generation. By incorporating these additional dimensions in future iterations, we aim to provide a more comprehensive assessment of LLM capabilities in generating long, complex, and creative content.

Dear Reviewer Nybu,

Once again, thank you for your valuable feedback on our paper. We hope our clarifications and revisions have resolved the issues you highlighted. If there are any remaining questions or areas where further clarification would be helpful, we would be more than happy to address them promptly.

As we are nearing the end of the rebuttal period, we kindly request you consider raising our paper's score if our updated responses have addressed your concerns.

Thank you for your time and effort in reviewing our work.

Best regards,

Authors of Submission Number: 3588

Dear Reviewer Nybu,

Thank you once again for your thoughtful feedback on our paper. We appreciate the time and effort you have invested in reviewing our work.

As the Discussion Period is nearing its conclusion, we wanted to kindly follow up to ensure that you’ve had a chance to review our response to your comments. We hope that our rebuttal has addressed your concerns satisfactorily. If there are any additional points or further clarifications required, we would be happy to provide them promptly.

Additionally, if our response has sufficiently addressed your concerns, we would greatly appreciate if you could consider reflecting this in your evaluation, including revisiting your score if appropriate.

Thank you for your time and understanding. We greatly value your feedback and look forward to hearing from you.

Cheers,

Authors of Paper Submission 3588

We would like to take this opportunity to further clarify some of the points related to W1, W2, and Q1. Your insightful comments on benchmarking demonstrate a deep understanding of this research area, and we are grateful for your thoughtful engagement.

We believe you may be familiar with the work on abstract visual reasoning using Tangram shapes. That study introduced a dataset constructed with Tangram pieces to evaluate the abstract visual reasoning capabilities of VLMs. Alongside this, the authors proposed metrics such as Shape Naming Divergence (SND), Part Naming Divergence (PND), and Part Segmentation Agreement (PSA), which were specifically designed for their KILOGRAM benchmark. While the benchmark did not encompass all aspects of abstract visual reasoning, the work’s novelty and contributions were significant enough to earn it the Best Paper award at EMNLP 2022.

Similarly, our work introduces a benchmark and evaluation metrics tailored to a specific instruction-following task within the broader domain of long-form text generation. While our metrics are designed with our benchmark in mind, their primary purpose is to provide accurate performance assessment for this emerging subfield. Generalizability, while important, is a secondary consideration at this stage.

As one of the initial works in this subfield, we kindly ask that you consider our benchmark with an open and inclusive perspective, recognizing its potential to lay a foundation for further advancements in long-context text generation. We deeply appreciate your understanding, your insightful feedback, and your consideration.

ps: happy thanksgiving everyone!

W4/Q5: Evaluating SOTA Models and Architectural Diversity

Thank you for your suggestion. We have conducted additional experiments to evaluate Mamba’s performance under various settings and compared it with existing baseline models, focusing on both inference speed and accuracy. Our findings show that Mamba significantly enhances computational efficiency but fails to maintain accuracy comparable to other models. The results are summarized in the table below:

A limitation of the current Mamba model is its maximum training sequence length of 2,000 tokens, which restricts its ability to handle longer text sequences effectively. This limitation directly impacts its performance in long-context tasks, as observed in our results. While recent efforts, such as LongMamba [1], aim to extend Mamba’s capabilities to longer contexts via training-free receptive field enlargement, the model parameters are not yet available as open-source. We intend to test LongMamba once it becomes publicly accessible.

[1] LongMamba: Enhancing Mamba’s Long-Context Capabilities via Training-Free Receptive Field Enlargement. ICLR-2025 Submission.

Q3: Complex prompts and instructions might need manyshots

Thank you for this insightful question. We chose not to incorporate few-shot or many-shot examples in our benchmark for two main reasons:

Context Length Limitations: Given that our outputs already approach or reach the model’s maximum long-context length, adding few-shot examples would significantly reduce the space available for evaluating long-form generation capabilities. This reduction would compromise our assessment of the model’s performance on extended outputs, as it would limit the effective long-context sequence length available for instruction adherence.

Realistic Evaluation Setting: Zero-shot evaluation aligns better with practical, real-world applications of long-context models, where explicit examples may not always be available in deployment scenarios. Major long-context benchmarks, such as LongBench [1], RULER [2], and NeedleBench [3], also primarily use zero-shot settings. This approach ensures a fair comparison across models without relying on specific prompt design strategies that could introduce variability in performance, allowing us to focus on the intrinsic instruction-following capabilities of each model.

We appreciate your question and hope this clarifies our reasoning. Please let us know if further elaboration is needed.

References:

[1] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding.

[2] RULER: What's the Real Context Size of Your Long-Context Language Models?

[3] NeedleBench: Can LLMs Do Retrieval and Reasoning in a 1 Million Token Context Window?

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

W1/Q1: Main Takeaways and Failure Cases

Thank you for your feedback and for highlighting areas where additional clarity on takeaways and failure cases could enhance our paper. In response, we have taken several steps to provide a clearer overview of model performance and failure analysis. To further illustrate model limitations, we have expanded our error analysis in Appendix F (Original paper error analysis in appendix C), where we provide detailed examples of failure cases. Here, we illustrate issues encountered in the Skyscraper Design task, specifically highlighting where models struggled to consistently follow instruction requirements over extended sequences. Below is a concise breakdown of the errors in this example:

Skyscraper Design Example

Objective: Construct a skyscraper with 100 floors. The floor assignments are detailed as follows:

• Specific floor requirements: Designate Floor 11 for a small art gallery.
• Range floor requirements: Allocate Floors 32 to 39 for corporate headquarters of a major company.
• Periodic floor requirements: Include a sky garden every 15 floors, starting from Floor 30.

The output includes excerpts of floor descriptions and the corresponding correctness evaluation marked with either a green check (✅) or a red cross (❌). Below, we provide a detailed analysis of the highlighted errors:

• Floor 11: Designated for art gallery use, Floor 11 is a sophisticated and flexible space designed to celebrate visual arts..... (✅).
• .....
• Floor 32: Floor 32 serves dual purposes, housing a renowned photography studio and corporate offices. ..... (❌).
• .....
• Floor 34: Transitioning into a leisure space, Floor 34 hosts a small cinema, providing an exclusive entertainment venue within the skyscraper (❌).
• .....
• Floor 60: This floor houses a luxury watch and timepiece atelier, celebrating the art of horology and fine craftsmanship. ..... (❌).
• .....
• Floor 90: Floor 90 offers a dynamic e-commerce and digital marketing center focused on online business innovation and consumer engagement strategies. ..... (❌).

Analysis of Errors

These examples reveal two common failure modes:

• Inconsistent Floor Allocation: Some floors, such as 32 and 34, were incorrectly used for purposes outside the defined range, highlighting the model’s challenges in adhering strictly to range-based instructions.

• Unplanned Floor Use: Floors like 60 and 90 were assigned purposes not outlined in the original instructions, suggesting that the model struggles to maintain instruction adherence over extended sequences.

These findings mark an important contribution as LongGenBench is the first systematic study to examine long-form generation capabilities in extended contexts with a focus on instruction adherence. While some results may not appear surprising in isolation, our benchmark provides a rigorous, comprehensive framework that reveals consistent, quantifiable patterns in model behavior over long sequences. By identifying specific error cases, such as inconsistent allocation and unplanned floor use, we offer concrete insights into the limitations of current long-context LLMs. This structured approach enables the community to better understand and address the inherent challenges in long-form generation, laying the groundwork for future improvements in model architecture and training.

W2/Q2: Prompt Format and Complexity

Thank you for your suggestion and insights regarding prompt format and its influence on model output. We agree that prompt design can significantly impact model performance, as shown in prior research. To address this, we experimented with different prompt formats, including reordering structures, across several models, including LongWriter, Mistral, and Qwen 2. The results are summarized in the table below:

Our experiments indicate that while prompt format variations do impact model performance, the relative ranking of model results remains consistent, even with different formats. This suggests that prompt structure alone does not substantially alter overall model performance trends or relative comparisons across models.

Determining the optimal prompt format for each model is beyond the primary scope of our study. Instead, our focus was on ensuring prompt consistency across all models to evaluate their performance differences fairly. By using the same baseline format for each model, we controlled for prompt-induced variability, allowing observed performance differences to reflect the inherent capabilities of each model rather than prompt design differences.

To provide further insights, we included an analysis of prompt complexity in Appendix G. This analysis examines the impact of different prompt structures on model performance.

W3/Q4: Reasoning Tasks and Adding a Reasoning Axis

Thank you for your suggestion regarding reasoning tasks and the potential addition of a reasoning axis in our benchmark. We agree that reasoning, particularly over extended contexts, is a crucial area for assessing LLMs. In our current benchmark, we address some aspects of temporal reasoning through periodic instructions embedded in the prompts. For instance, instructions such as “run long distance every 7 days” require the model to accurately apply this recurring activity at specified intervals. These periodic directives test the model's basic understanding of temporal structures within long-form text generation and allow us to inspect generated content systematically. This setup facilitates an automated evaluation of temporal reasoning without requiring labor-intensive human review.

While dimensions such as reason, coherence and factual accuracy are undeniably valuable, they become meaningful only after the model demonstrates a foundational ability to follow structured instructions. Similarly, we consider instruction adherence to be a foundational aspect of long-form generation, as models must first reliably follow specified prompts to ensure quality outputs. By focusing initially on instruction-following, we aim to lay a pragmatic and impactful foundation for long-form text evaluation.