Dear Reviewer

Thank you very much for your valuable comments and questions. We appreciate the time and effort you have put into reviewing our manuscript. Below, we will address your concerns and provide further clarifications.

W1

HelloBench is sufficiently large-scale, has good coverage, and makes significant contributions.

1. Large-scale: HelloBench has a total of 647 samples. In comparison, previous long text generation benchmarks like ProxyQA[1] have only 100 samples and LongBench-Write[2] has 120 samples. Besides, considering checklist-wise evaluation costs, we need 647 * 2,500 = 1,617,500 tokens, which equals $16. Maintaining an acceptable number of samples is also necessary to control evaluation costs.
2. Good Coverage: See G.Q1 in General Response.
3. Significant Contributions: HelloBench makes significant contributions in several aspects. (1) Data: It fills the gap in the benchmark for long text generation by addressing issues such as low coverage, limited data volume, and a narrow range of task types found in the previous benchmark. (2) Evaluation Method: It includes annotated Human Evaluation data, designs high-quality checklists, applies weighted scores to the checklists, and introduces HelloEval, which enhances accuracy and reduces the cost of subjective evaluation. (3) Analysis: we have conducted thorough evaluation experiments, analyzing the weaknesses and error cases of current LLMs from multiple aspects, and providing extensive ablation experiments, offering directions for future research.

W2

HelloBench has NOT VIOLATED Quora's Terms of Service:

1. We did not use automated scripts to scrape data from Quora. Instead, we created an account, sorted posts in popular topics by time, and manually collected the data.
2. According to Quora's Terms of Service Section 4.4, Quora prohibits data collection and scraping for four purposes: creating derivative works, training AI, creating any competitive service, and commercial use. We adhered to these guidelines. Our use is academic, we did not create derivative works or competitive services, and we did not train AI models using Quora's data. Additionally, based on the Terms of Service's Section 4.5, HelloBench’s data collection process is completely reasonable and legal.

W3

The experiment and analysis sections of our work includes:

• S4.2: The performance of LLMs in long text generation.
• S4.3: The explicit length constraints.
• S4.4: The relationship between long input and long output.
• S4.5: The effectiveness of HelloEval.
• S5.2, Appendix K: Analyzing the error modes.

Firstly, we have conducted detailed experiments from multiple aspects, analyzing long text generation capabilities of mainstream LLMs. Secondly, after identifying the models' weaknesses, we aim to understand why LLMs performed poorly by analyzing low-scoring LLMs' outputs and summarizing four main error modes. Finally, we provided directions for future work (see line 503 - line 510 and line 2486 - line 2545). Thus, our work includes a complete research trajectory, including problem identification, evaluation dataset construction, experimental evaluation, error analysis, and future research directions. The overall approach and analysis are comprehensive and in-depth.

W4

S5.2 and Appendix L discusses the conclusions of our experiments and the research directions that these conclusions suggest (see line 2486 - line 2545). We observed that:

1. There is a correlation within the context window, as LLMs are autoregressive models, long input with short output and short input with long output are both related to the model's intrinsic capability to encode long texts.
2. Constructing long text generation data can significantly increase the output length of the model. Examples like LongWriter and Suri[3] illustrate this well.
3. There is a correlation between the length of the generated text and its quality. As the output length increases, the quality tends to decrease.
4. Multi-agent system Methods (like RecurrentGPT[4]) can improve the system's long text generation capability, but these methods are not end-to-end.

As a generative model, LLM faces challenges in long text generation tasks. We believe that research in this field should progress gradually. The current training data, model architecture, and algorithms may not be the most suitable for long text generation tasks. Our work aims to make a significant step forward in this field, guiding subsequent research and promoting the importance of long text generation in the industry, thereby improving the user experience with LLMs.

Q

See W2, Quora allows data crawling in the condition of HelloBench.

Reference

[1] Proxyqa: An alternative framework for evaluating long-form ...

[2] Longwriter: Unleashing 10,000+ word generation from ...

[3] Suri: Multi-constraint instruction following for ...

[4] RecurrentGPT: Interactive Generation of (Arbitrarily)...

Dear Reviewer

Thank you very much for your valuable comments and questions. We appreciate the time and effort you have put into reviewing our manuscript. Below, We will address your concerns and provide further clarifications.

Weakness

We will answer it in two parts:

1. Why use Bloom's taxonomy? Firstly, Bloom's taxonomy is a well-known classification method in the field of education, widely recognized and used by educators. Bloom's taxonomy aligns well with the cognitive levels of LLMs. For HelloBench, Bloom's taxonomy serves as the framework for the entire benchmark. Its main function is to provide researchers with a clear path when constructing the benchmark. Different levels correspond to different sub-tasks, and once sub-tasks corresponding to Bloom's levels are identified, the tasks can then be collected and constructed. A shortfall of previous related work (ProxyQA[1], LongBench-Write[2]) is the incomplete task coverage. With the help of Bloom's taxonomy, we can build a long text generation benchmark with high coverage. Bloom's taxonomy ensures the benchmark is comprehensive and highly covered. Moreover, Bloom's taxonomy minimizes overlap between different data, as each piece of data corresponds to a sub-task, and the overlap between sub-tasks is minimal.
2. Are the sub-tasks aligned appropriately with Bloom's levels? Bloom's taxonomy in HelloBench serves to (1) define the architecture of the entire benchmark, (2) enhance the coverage and completeness of HelloBench (3) reduce overlap between data. HelloBench includes five sub-tasks, which are the most frequently occurring tasks in real-world scenarios (see G.Q1 in General Response). In our work (see line 133 - line 161), we explain the correspondence between sub-tasks and levels and why they are mapped this way. Thank you for your suggestion; we have revised it to "We have selected representative tasks that frequently appear in real-world scenarios and mapped them to the cognitive level."

Thank you again for your suggestion. We noticed that you pointed out only one weakness and did not raise other questions. If you have additional questions, please feel free to ask, and we will promptly address and respond to them.

Reference

[1] Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. arXiv preprint arXiv:2401.15042, 2024.

[2] Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055, 2024.

Dear Reviewer

Thank you very much for your valuable comments and questions. We appreciate the time and effort you have put into reviewing our manuscript. Below, we will address your concerns and provide further clarifications.

Weakness

Firstly, 0.32 is not considered low. In related work, the Spearman correlation in GPTScore[1] ranges from 0.1 to 0.4. In the paper "Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment," the Spearman correlation ranges from 0.1 to 0.5. Spearman correlation is meaningful when used comparatively to show differences between evaluation methods. Secondly, in our experimental setup, we compared three LLM-as-a-Judge based evaluation methods and six traditional metrics (lines 448 - 472). Our HelloEval achieved significantly the highest performance. Additionally, many traditional metrics show negative correlations with human evaluations, further proving that traditional metrics are inadequate for evaluating text generation tasks and underscoring the importance and necessity of HelloEval. Lastly, there is also inconsistency among different annotators (see lines 1877 - 1889), making it even more challenging to achieve high correlation values.

Q1

The license for HelloBench is available in the GitHub repository for this project. Due to the double-blind review process, we have not included the repository link in our paper. Thank you for your suggestion; we have included our license in the appendix of our paper (see line 2612 - line 2634)

Q2

These two parts do not overlap, so there will be no overfitting issue. During the preparation stage, we had five annotators label the human evaluation results of four LLMs, and these results were only used to fit the weighted scores. In the effectiveness experiment of HelloEval, we invited three annotators to label the human evaluation results of seven LLMs. Due to cost reasons, we selected only the summarization task for annotation because only the summarization task has reference answers to compute rouge and bleu metrics. The specific experimental results can be found in Table 23 (see line 2249 - line 2260).

Q3

Continuing to Question 2, we have a total of 7*100*3=2100 annotations. This quantity is large enough to achieve a confident result, and at the same time, by observing the p-value in Table 5, which is very low, it further indicates the confidence of the results. For the plot, we have included the relationship between annotator one's human evaluation scores with HelloEval scores in the appendix of our paper (see line 2222- line 2245).

Q4

The scores of different subtasks are comparable. Firstly, the score ranges of different subtasks are consistent. Secondly, although the checklists for different tasks vary, they generally follow similar evaluation aspects (instruction following, comprehensiveness, easy-to-understand, factuality, overall, see Appendix E).

Q5

The scores obtained through HelloEval are not significantly different from the normal scores. HelloEval focuses on aligning the scores with human evaluation. However, due to rescaling, the benefit curve of the rescaled scores is different from the original 0-100 scale, where each point of improvement represents 1/400 of the overall scale. In terms of positive or negative scores, if the rescaled score is positive, the original score exceeds 75, which means that the scores for each checklist item are generally above 0.75, indicating that your response quality is above the acceptable line. Based on the response to Q4, scores of different subtasks are comparable.

Q6

The main intent of this statement in the paper is to highlight that these data are less likely to leak compared to other data. Although some LLMs might have been passively trained on these test data, at least it ensures that they are not actively trained on them, as LLMs are not trained specifically toward the test data. However, we agree that this statement could be more precise, and we have revised it to: "By doing so, we guarantee that the probability of data leakage is minimized, and the quality of these data is ensured." Thank you for your suggestion.

Reference

[1] Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023.

Thank you for your feedback! If you have any other questions or issues, please feel free to let me know.

Dear Reviewer

Thank you very much for your valuable comments and questions. We appreciate the time and effort you have put into reviewing our manuscript. Below, we will address your concerns and provide further clarifications.

Weakness 1

We have noticed the impressive work of ProxyQA[1] and have referenced it in our paper (see line 521 - line 529). However, we did not include it in the comparison table (Table 1). For now, we have updated and supplemented this in our work (see line 089 - line 096). Regarding your question "which cannot adaptively provide query- and semantic-aware checklists," the checklists in HelloBench are very granular. We have designed specific checklists for each subcategory. Our dataset includes a total of 38 subcategories, so on average, each set of checklists corresponds to around 20 samples.

Weakness 2

Yes, we have added a comparison with ProxyQA in Table 1. In Section 4.5, we compared the performance of HelloEval with several methods (MTBench[2], WildBench[3], etc.) under the same settings (see line 448 - line 472), and HelloEval achieved the best results. When conducting experiments, we also considered whether to compare more evaluation methods, such as ProxyQA. However, these methods are generally coupled with specific datasets. For instance, the combination of proxy questions and meta questions in ProxyQA cannot be applied to HelloBench without annotating a set of proxy questions specifically for HelloBench. We also wanted to compare the correlation of ProxyQA and HelloEval under the same settings, but we found that ProxyQA’s proxy questions are not open-sourced. Additionally, since the evaluated LLMs are not consistent, we cannot compare the ranking consistency between ProxyQA and HelloEval. If you have any good suggestions, we are happy to compare the correlation between ProxyQA and HelloEval.

Weakness 3

Contributions of HelloBench:

1. Evaluation Method: We propose HelloEval, an evaluation method aligned with human evaluation. This method can be plugged into traditional subjective evaluation methods to enhance their alignment with human evaluation, reduce the cost of human evaluation, and improve evaluation efficiency.
2. Benchmark HelloBench: Different from ProxyQA, which focuses on a single task, HelloBench covers a wider range of tasks, and it aims at real user scenarios. HelloBench contains 647 entries, is large-scale, and has undergone multiple steps (cleaning, deduplication, filtering) to ensure high data quality (see line 1223 - line 1259).
3. Experimental Analysis: We have conducted detailed experiments, analyzing the long text generation capabilities of current LLMs from various aspects (implicit word limit, explicit word limit, the relationship between long text comprehension, and long text generation capabilities). Furthermore, after identifying the models' weaknesses, we explore why the models perform poorly by analyzing low-scoring samples and summarizing four main error modes. Finally, we suggest future research directions (see line 503 - line 509 and line 2486 - line 2545). As such, our work presents a complete research pathway, including problem identification, dataset construction, experimental evaluation, error analysis, and suggestions for future work.

Both ProxyQA and HelloBench aim to bring more attention to the field of long text generation and promote improvements in LLMs' long text generation capabilities. Therefore, we believe that both ProxyQA and HelloBench have made significant contributions.

Question

See G.Q2 in General Response

Reference

[1] Tan, H., Guo, Z., Shi, Z., Xu, L., Liu, Z., Feng, Y., ... & Song, L. ACL 2024. ProxyQA: An alternative framework for evaluating long-form text generation with large language models.

[2] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.

[3] Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770, 2024.

Thank you for your feedback! If you have any other questions or issues, please feel free to let me know.