LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation

Thank you for your thoughtful comments and feedback! Please find our clarifications and answers to the questions below.

Clarification on Contributions

In summary: The paper describes an analysis of LLMs for effective use of large contexts over different levels of sizes and access of the information in the contexts. The authors curated several datasets across tasks which involve long context like HTML to CSV. The tasks are created with different context lengths.

Thank you for the summary. We would like to clarify that, in addition to evaluating how models access dispersed information in long contexts, a central focus of our benchmark is long-form generation. To support this, we configure tasks with varying maximum generation lengths, as explained in our response to Q1 below. We highlight long-form generation because it is a critical yet relatively under-explored capability in existing benchmarks.

Answers to the Questions

Q1: How do you create the tasks with 3 different difficulty levels? Do they contain the same set of tasks? Do they contain the same base data points (each having other context lengths)?

A: We create three difficulty levels corresponding to different output lengths across the same set of six tasks in our benchmark. To construct these levels, we select subsets of data points that require varying output lengths. As a result, each difficulty level includes different subsets of examples, but they all span the same set of tasks. Please refer to Appendix B for details on the construction process.

Q2: While there is a limit to what the authors could test, I would like to see how models perform with regard to the level of irrelevant or contradictory information in the context

A: This is an interesting point, though we consider such analysis somewhat out of scope for our current focus on long-form outputs and synthesis of dispersed information.

That said, one of our tasks (HTML to TSV) offers a partial investigation of this issue. It includes two settings: 1) extracting all, where the model must identify all relevant items (e.g., “all movies in the page”), and 2) filtering, where the model must extract a subset based on specific conditions (e.g., “all action movies in the page”). Please refer to Appendix B.2 for details on how this task is constructed. In the table below, we report several models’ performance on the HTML to TSV (8K) task, with and without filtering (in our work, we only report the aggregated number). We observe a substantial drop in the filtering setting, suggesting difficulty in handling irrelevant information. We will include this discussion in any future version, but we feel a full analysis of robustness to irrelevant context is beyond the scope of this work.

Q3: Is the Pseudocode task a sequential task? How do you verify that the generated code is in the same order as the pseudocode if you only check unit tests?

A: Thank you for raising this. Our evaluation does not enforce strict adherence to the original ordering, as we focus on the functional correctness of the complete program (following the original setup in Kulal et al., 2019). However, we explicitly instruct models: “Note that the translated code should correspond to the pseudocode line by line.” In practice, we observe that models largely follow the intended order. Minor deviations (such as inserting syntactic elements like braces) can occur, but the translated programs generally follow the original ordering.

We appreciate your thoughtful comments and feedback. Please find our answer below.

Q1: Like with any benchmark: it could be hacked, and the evaluated data could quickly leak in the new revisions of the models.

A: This is a good point, and thank you for bringing it up! We acknowledge the broader challenge of preventing data contamination in benchmarks. At the same time, we believe our long-form generation evaluation offers some natural resilience to this issue: 1) our benchmark evaluates long outputs, which are harder to memorize compared to short-form answers. 2) several tasks involve solving procedures that can be programmatically generated, allowing us to produce ground-truth outputs as needed. In our final release, we plan to withhold ground-truth procedures for tasks where this is appropriate. Thank you again for the helpful suggestion.

Thank you for your thoughtful comments and feedback! Please find our answers below.

Q1: What is the default evaluation mode used to produce the results in Table 3?

A: The default evaluation mode is ICL with procedure. Our primary goal is to assess models’ capabilities in long-form generation given a specific procedure. We include the full procedure in the prompt, to emphasize the challenge of extended generation rather than procedural discovery. We will clarify this more explicitly in any future version. Thanks for the suggestion!

We appreciate your thoughtful comments and detailed feedback. Please find our answers to the questions below.

Q1: The benchmark contains a limited number of examples per setting. Are there plans for a larger hidden test set to deter prompt contamination?

A: Thank you for the suggestion. Expanding the test set is feasible for many of our benchmarks where data can be programmatically generated, with the exception of HTML to TSV and Pseudocode to Code, which require substantial human annotation. We will consider including a larger test set in the final release.

We chose 100 examples per task for two main reasons. First, long-form generation evaluation incurs high computational costs (see answer to Q7). Second, since our benchmark spans 6 diverse tasks, results averaged over 100 examples per task are sufficient to reveal meaningful trends. For example, as shown in Table 3 of the paper, proprietary models exhibit statistically significant performance differences on the 8K token tasks.

Q2: The tasks provide exact procedures in the prompt, conflating instruction-following with reasoning. Real-world noise and open-ended generation are not tested.

A: Thank you for raising this. We provide explicit procedures to highlight the challenges of extended generation. While we acknowledge that this setup overlaps with instruction-following, we evaluate instruction following specifically in the context of reasoning tasks. These tasks often require deducing intermediate steps while maintaining logical coherence across long sequences, and we believe these capabilities are essential prerequisites for complex reasoning.

We do include tasks that contain real-world text (HTML to TSV and Pseudocode to Code). However, our focus is on tasks with reliable and automatic evaluation metrics. While we acknowledge the importance of open-ended generation, such tasks often lack deterministic evaluation, and we consider them out of scope for this work.

Q3: (Unequal prompt budget) Why do reasoning and instruction-tuned models have asymmetric output budgets?

A: Thank you for this question. We believe this is a suitable way for evaluating these two sets of models with distinct characteristics. We allow more tokens for reasoning models to accommodate additional thinking tokens, since otherwise these models do not finish generation and we cannot extract the solutions. By contrast, instruction tuned models do not effectively use additional tokens even if we allow them more tokens, as they terminate most of the time (typically >90% across tasks) before hitting the max number of tokens. We will add this clarification in any future version.

Q4: (Missing baselines) How does long‑context fine‑tuned baselines (LongLoRA) or retrieval/memory‑augmented approaches perform?

A: We would like to clarify that our evaluation includes long-context fine-tuned models. We believe that the models we evaluate represent the current state-of-the-art in long-context processing. Many of them are explicitly trained with long-context capabilities (e.g., as described in the Llama3 and Qwen2.5 technical reports). We also include ProLong, an open-source long-context model. These models achieve strong performance on long-context recall benchmarks (>50.0 at 32K on RULER). By contrast, LongLoRA achieves only 29.4 at 32K on RULER, indicating it is less capable than the models we include.

We do not evaluate retrieval-augmented or memory-augmented approaches because our tasks require using the entire context. These approaches typically involve chunking the input and selecting a subset of the context during generation, which would often result in task failure under our setup. Therefore, we consider these methods to be beyond the scope of our current evaluation.

Q5: How robust is the rule-based evaluation to innocuous formatting deviations?

A: For each task, we implement specific normalization procedures to accommodate slight variations in formatting and wording. For instance, we use standard normalization steps used in QA evaluation (lowercasing, removing punctuation, etc,) for HTML to TSV tasks. We report the full details in Appendix C.

Q6: What are the licensing terms for any third-party data (e.g., HTML pages, IMDB content), and will raw data be fully redistributed?

A: All third-party data is sourced from publicly available, open-source datasets released under redistribution-compliant licenses (e.g., CC 4.0).

Q7: What was the compute cost (GPU hours / USD) of running the full benchmark?

A: For ~10B-scale models, we used a single H100 80GB GPU for approximately 9 GPU hours per model. For ~70B-scale models, we used 4×H100 GPUs with a total compute time of around 110 GPU hours. We will include these details in the revision. Thanks for your suggestions!