LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

Thanks for carefully reading our paper! We address the weaknesses as follows.

• Weaknesses 1: Generalizable of LongWriter to other tasks

Thanks for your suggestion! We acknowledge that it is worth exploring the LongWriter method on other tasks, such as generating extensive codebases. In this work, we primarily focus on ultra-long-form writing tasks, and we hope to explore a broader range of ultra-long generation tasks in future work.

• Weaknesses 2: More elaboration on related works

To the best of our knowledge, prior to this work, the only research focusing on extending the output length of LLMs was Suri [1]. We have compared our model with Suri in the experiments (Table 3), where the results demonstrate that the LongWriter models have significant advantages over the Suri-I-ORPO model in both output quality and length. Thank you for your suggestion, we will include a discussion of [1] in the related work section.

[1] Pham C M, Sun S, Iyyer M. Suri: Multi-constraint instruction following for long-form text generation[J]. arXiv preprint arXiv:2406.19371, 2024.

For future work on long generation, in addition to areas like creative writing and long code generation, we believe there are promising directions such as academic paper writing (which requires being more technical and leveraging RAG and external tool assistance), as well as scenarios where both input and output involve lengthy texts, such as long-form translation, error correction, and content integration.

We humbly believe that our LongWriter work has explored a feasible path for these potential directions, including data construction and model training. We also look forward to seeing more future work on long generation empowering more fields.

Thanks for your valuable review. We address the weaknesses as follows.

• Weaknesses 1: Directly leverage AgentWrite to generate long responses with the target model

Thanks for your suggestion. We've incorporated comparison to directly leveraging AgentWrite to open-sourced models, including AgentWrite + GLM-4-9B-chat / Llama-3.1-8B-Instruct / Llama-3.1-70B-Instruct. Note that we only apply AgentWrite to instructions in LongBench-Write that requires output lengths of 2,000 words or more.

We found that applying the AgentWrite method for open-sourced models can effectively increase the output length, making it closer to the length requirements of user instructions (higher $S_l$). However, in the $[4k, 20k)$ range, the $S_l$ score is still not high enough, indicating that the output length limit of open-sourced models using the AgentWrite method remains insufficient. Meanwhile, the output quality obtained using the AgentWrite approach on these models is slightly lower compared to direct output (slightly lower $S_q$). Overall, LongWriter models demonstrate better ultra-long-form generation capabilities in terms of both length and quality. Additionally, the inference cost is lower--during the "write" phase of AgentWrite, each round of output requires all the history context from previous outputs and requires re-prefilling, leading to significantly higher inference costs compared to single-pass output.

• Weaknesses 2: Declined performance on shorter outputs

First, we need to clarify that in Table 3, GLM-4-9B-chat should be compared with LongWriter-9B-DPO, not LongWriter-9B, because GLM-4-9B-chat underwent DPO training, which significantly enhances the quality of the model's outputs. Compared to GLM-4-9B-chat, LongWriter-9B-DPO shows a 1%-2% decrease in quality score $S_q$ for queries in the $[0, 500)$ and $[500, 2k)$ ranges, while exhibiting a 1%-2% increase for queries in the $[2k, 4k)$ and $[4k, 20k)$ ranges. We believe that such small score fluctuations do not indicate that our method compromises the model's general capability. Variations in the final results can arise from model training, model inference, and GPT-4o evaluation.

Thanks for pointing out the missing reference, we've fixed it.

Here are our responses to your questions.

Q1

Great idea! We tested the GLM-4-9B-chat model on LongWrite-Ruler queries, setting both min_new_tokens and max_new_tokens to match the output length specified in the instructions during inference. Let $x$ denote the model output when min_new_tokens is not set. With the min_new_tokens constraint, the output becomes $[x; y]$, where $y$ is generated because the probability of the eos_token is set to 0 when $x$'s length does not meet the min_new_tokens requirement, forcing the model to continue. We found that in the model's output, $y$ either repeats $x$ (again and again) or consists solely of repetitive words (or even repetitive emojis), adding no meaningful content. We concluded that simply setting the probability of the eos_token to zero is not an effective way to obtain meaningful long outputs from the model.

Q2

In addition to testing the model's output length limit on LongWrite-Ruler (Sec 2), it is worth noting that we also evaluated the output length limit on 120 real user queries from LongBench-Write. As shown in Figure 6 of the paper, despite the varied prompts, the output length of previous models consistently remained below 3k words, while LongWriter models can generate up to 10k words. Notably, as observed in Figure 6, the LongWriter models are largely capable of following the length requirements specified in the instructions. We kindly ask the reviewer where the observation that "even after training the model with longer texts, the model's output still largely fails to meet the required length" was derived from.

• Weaknesses 4: More elaboration on the controlled experiment in Sec 4.2

We apologize for missing information in Sec 4.2 regarding the training details of the baseline model and the controlled experiments. To clarify the notations, GLM-4-9B is a pretrained base model, and GLM-4-9B-chat is its aligned version. Specifically, the GLM-4-9B-chat model is based on the GLM-4-9B base model, first supervised fine-tuned (SFT) with 180k general SFT data (as shown in Figure 5), followed by training with 50k chat DPO data to obtain the final model. In contrast, in our work, during the SFT phase of the GLM-4-9B base model, we mixed 6k longwriter-6k data into the 180k general SFT data, resulting in the LongWriter-9B model. Subsequently, we introduced 4k long-form writing DPO data into the 50k chat DPO data during the DPO training phase, producing the LongWriter-9B-DPO model. Therefore, the GLM-4-9B-chat model and the LongWriter-9B-DPO model form a fair comparison--they both underwent the same SFT + DPO training phases from GLM-4-9B, with the only difference being the inclusion of LongWriter data in LongWriter model's training.

Here are our responses to your questions.

Q1

In AgentWrite, we set the length of each paragraph to be between 200 and 1000 words because: 1. Too short paragraphs (<200 words) result in an excessive number of paragraphs, making the output overly fragmented. 2. Too long paragraphs (>1000 words) often lead to GPT-4o generating paragraphs that fall short of the required word count.

Q2

We do not have a theoretical upper limit for the length of model-generated outputs. From the controlled experiments in Sec 2, we observed that the upper limit of the model's generated length generally aligns with the maximum output length in the SFT training data.

Q3

We explained our choice of using token-level loss instead of sequence-level loss in lines 307-312. To elaborate, we found that when using sequence-level loss, the loss weight assigned to each token in the ultra-long output samples becomes too small, with the majority of the loss weight being allocated to tokens in short sequences. This makes it difficult for the model to fully learn the ability to generate long outputs.

Q4

Here we provide the quality score $S_q$ of GLM-4-9B-chat and LongWriter-9B-DPO on LongBench-Write queries across seven categories (refer to Table 1 in our paper): Literature and Creative Writing (LCW), Academic and Monograph (AM), Popular Science (PS), Functional Writing (FW), News Report (NR), Community Forum (CF), and Education and Training (ET).

We can observe that the LongWriter model shows a decline in quality for narrative or creative writing (LCW, NR), while demonstrating an improvement in quality for technical or formal writing (AM, PS, FW, CF, ET). We speculate that this change in output quality is due to the fact that the data constructed by AgentWrite exhibits higher logical consistency, making it suitable for technical writing. However, the segmented generation of outputs by AgentWrite disrupts the coherence of narrative writing, bringing degradation to creative writing quality. Future work could improve the AgentWrite pipeline to enhance the coherence of outputs for creative writing types, mitigating the segmentation between and within paragraphs during the generation process.

Thanks for your constructive review. We address the weaknesses as follows.

• Weaknesses 1: Human Eval to assess AgentWrite quality

Thank you for your suggestion. We manually assessed the quality of long responses generated by GPT-4o with or without AgentWrite for 58 queries in LongBench-Write that required responses of 2,000 words or more. Based on the quality of the generated responses, we categorized them into three groups: "High Quality" (meets user requirements, fluent, and clearly expressive), "Good Quality" (generally meets requirements but has minor issues in formatting or expression), and "Poor Quality" (fails to meet requirements or has significant issues with formatting or expression). Note that we only consider the quality of the output content, regardless of whether the length meets the requirement.

In our annotations, we found that GPT-4o often produces only an outline for queries requiring ultra-long responses and generates overly general content for professional formats like papers and reports. This led to a noticeable number of "poor quality" cases during our manual checks. However, after incorporating AgentWrite, such "poor quality" cases were significantly reduced. That said, it also introduced issues such as responses appearing somewhat mechanical (e.g., ending almost every paragraph with a "summary" or "outlook") and occasional minor repetition within the responses.

• Weaknesses 2: More comparisons with AgentWrite + open-sourced models

Thanks for your suggestion. We've incorporated comparison with open-sourced versions of AgentWrite, including AgentWrite + GLM-4-9B-chat / Llama-3.1-8B-Instruct / Llama-3.1-70B-Instruct. Note that we only apply AgentWrite to instructions in LongBench-Write that requires output lengths of 2,000 words or more.

We found that applying the AgentWrite method for open-sourced models can effectively increase the output length, making it closer to the length requirements of user instructions (higher $S_l$). However, in the $[4k, 20k)$ range, the $S_l$ score is still not high enough, indicating that the output length limit of open-sourced models using the AgentWrite method remains insufficient. Meanwhile, the output quality obtained using the AgentWrite approach on these models is slightly lower compared to direct output (slightly lower $S_q$). Overall, LongWriter models demonstrate better ultra-long-form generation capabilities in terms of both length and quality. Additionally, the inference cost is lower--during the "write" phase of AgentWrite, each round of output requires all the history context from previous outputs and requires re-prefilling, leading to significantly higher inference costs compared to single-pass output.

• Weaknesses 3: Plan validation in AgentWrite Step 1

Great idea! We are indeed attempting to incorporate validation and refining steps into the AgentWrite pipeline in ongoing works.

Thank you again for your helpful reviews. As the end of the discussion period draws near, we would like to ensure that we have adequately addressed all your concerns. If you have any further feedback, please do not hesitate to let us know.

Here are our responses to your questions.

Q1

Thank you for pointing this out. The 120 test queries in LongBench-Write were filtered and uniformly sampled from user logs based on predefined categories. To ensure privacy, we manually rewrote each query to remove any user-related sensitive information while maintaining the diversity and representativeness of the dataset.

Q2

Thanks for your keen observation. From Table 2, we can see that the output quality score $S_q$ decreased slightly from 91.8 to 91.6 after using AgentWrite. This difference falls within the margin of error for the GPT-4o evaluator, so we consider it not statistically significant, meaning the output quality can be regarded as essentially unchanged. In Table 6 of the appendix, we show the changes in scores across different dimensions of output quality. After incorporating AgentWrite, the model's outputs exhibit a significant improvement in Breadth and Depth, while Coherence and Clarity show a decline.

Q3

Apologies for not emphasizing this in the paper, but in Table 3, the fairest comparison is between GLM-4-9B-chat and LongWriter-9B-DPO. Both models use GLM-4-9B as the base model and were trained with the same data during the SFT and DPO stages, except for the additional data generated using the LongWriter method. As shown, the two models achieve nearly identical generation quality scores (85.5 vs. 85.4), while LongWriter-9B-DPO demonstrates a significant advantage in output length. We do not claim that the LongWriter method improves the model's generation quality. The primary goal of this method is to better align the model's output length with user instructions that require ultra-long outputs. Additionally, as shown in the table for Weaknesses 4, the model trained with LongWriter achieves higher scores compared to directly applying AgentWrite to an existing model. It is also worth noting that in AgentWrite, each segment of output requires all previous history, causing the input token count during inference to accumulate in a Fibonacci-like manner. This makes it significantly less efficient than generating the entire output in a single pass.

Q4

For the NLL loss illustration in Figure 7, we aim to validate the long-range dependencies in LongWriter's long outputs through the continuous downward trend of NLL with token positions, contrary to a simple concatenation of multiple paragraphs (as simple concatenations would not show a consistent decreasing NLL loss curve). Since all baseline models cannot produce such long outputs (8k+ tokens), there is no baseline for the NLL loss study.

Q5

Thanks for your intriguing question on evaluating ultra-long-form generations. As we mentioned in our response to Weaknesses 3, the correlation between GPT-4o and human evaluations is close to the inter-annotator correlation, suggesting that GPT-4o can serve as a reliable proxy for automated output quality assessment. Additionally, in our paper, we adopted pairwise comparison as an evaluation method (Figure 9). We believe these approaches can currently serve as reasonably reliable methods for evaluating ultra-long-form generation. However, we also acknowledge that more robust evaluation methods remain an open area for future research. In our current study, we did not come across any universally effective methods for this purpose.

Q6

Thanks for pointing this out, we've fixed the typo.

Q7

Thank you for your suggestion. We have updated table 4's color scheme and added + and - signs to more clearly illustrate the changes in performance for the readers.

• Weaknesses 4: More comparisons with AgentWrite + other models

Thanks for your suggestion to add AgentWrite + other models as baselines. We've added the results of AgentWrite + GLM-4-9B-chat / Llama-3.1-8B-Instruct / Llama-3.1-70B-Instruct to Table 3. Note that we only apply AgentWrite to instructions in LongBench-Write that requires output lengths of 2,000 words or more.

We found that applying the AgentWrite method for open-sourced models can effectively increase the output length, making it closer to the length requirements of user instructions (higher $S_l$). However, in the $[4k, 20k)$ range, the $S_l$ score is still not high enough, indicating that the output length limit of open-sourced models using the AgentWrite method remains insufficient. Meanwhile, the output quality obtained using the AgentWrite approach on these models is slightly lower compared to direct output (slightly lower $S_q$). Overall, LongWriter models demonstrate better ultra-long-form generation capabilities in terms of both length and quality. Additionally, the inference cost is lower--during the "write" phase of AgentWrite, each round of output requires all the history context from previous outputs and requires re-prefilling, leading to significantly higher inference costs compared to single-pass output.

• Weaknesses 5: Concerns w.r.t. efficiency

We are glad to share that we use the vLLM framework for generation: On an 80GB H800 GPU, the LongWriter-9B model takes approximately 55 seconds to produce an output of 10,000 tokens. Future work could focus on accelerating generation speed through techniques like KV cache compression and sparse attention. Additionally, exploring architecture variants such as Mamba could provide insights into performance and efficiency for ultra-long outputs.

Thanks for reading so carefully on our paper. We address the weaknesses as follows.

• Weaknesses 2: Evaluation consistency of GPT-4o as a judge on LongBench-Write

While GPT-4o serves as a primary evaluator, it has shown high consistency in long-text evaluations. To support this, we report the variance of the $S_q$ scores given by GPT-4o in the test results of Table 3 (based on three evaluation runs).

The reviewer also questioned whether LongWriter improves generation quality. To address this, we provide detailed scores for six dimensions of generation quality ($S_q$) to illustrate the impact of the LongWriter method on the model's generation quality.

It can be seen that LongWriter helps enhance the breadth and depth of the model's output, primarily because the model can generate longer and more detailed content. However, the coherence and clarity of the output are negatively affected, as the model tends to exhibit awkward transitions and occasional pattern repetition when producing longer content.

A similar fluctuation in quality is also demonstrated in Table 6 of our paper: compared to the outputs generated directly by GPT-4o, the long outputs obtained using the AgentWrite method improve breadth and depth but slightly compromise the coherence and clarity of the output. As a result, the LongWriter model trained on such data exhibits similar quality effects when producing long outputs.

Therefore, to further enhance output quality, future work could focus on improving the AgentWrite pipeline to ensure that the generated long outputs are more coherent and clear.

• Weaknesses 3: Evaluation correlation of GPT-4o as a judge on LongBench-Write

For the human comparison win-rate in Figure 9, we invited four annotators to rank the preferences of responses from four models on 120 prompts from LongBench-Write, in the form of preference orderings (e.g., a > b > c > d).

To verify the consistency between GPT-4o and human evaluations of long-output quality, we asked four annotators to independently score the quality of four sets of outputs for 120 prompts from LongBench-Write (resulting in a total of 480 scores). We then calculated the correlation between the human-assigned scores and the GPT-4o scores (GPT-4o) as well as the inter-annotator correlations (Human).

Since evaluating writing quality is a relatively subjective task, the table shows that the correlation between human annotators is not very high. Interestingly, the correlation between GPT-4o and human evaluations is close to the inter-annotator correlation, suggesting that GPT-4o can serve as a reliable proxy for automated output quality assessment.

Thank you again for your helpful reviews. As the end of the discussion period draws near, we would like to ensure that we have adequately addressed all your concerns. If you have any further feedback, please do not hesitate to let us know.