We really appreciate your valuable suggestions and insightful questions. We will address your concerns as follows.

Q1: Did you consider structured generation?

A1: We appreciate the reviewer’s attention to the details of our evaluation. Structured generation, such as JSON output, is relatively new but crucial for applications of LLMs, and we have not considered it yet. While the structured generation is not used, our conclusions are still reliable due to the same generation settings of all evaluated LLMs.

The primary goal of our proposed CriticEval in the current stage is to construct a comprehensive and reliable benchmark for evaluating critique ability. We will supplement discussions about it in the Limitation. Thanks for your valuable suggestions.

Q2: Besides positional bias, did you explore any other biases such as bias to length, style or certain words?

A2: Yes, in addition to positional bias, we have investigated length bias, which is discussed in Section 5.2 (lines 198-200) and Appendix I. Figure 10 (Appendix I) reveals no significant correlation between the count of unique tokens and the Likert scores from GPT-4’s subjective evaluations across three critique dimensions [1,2], likely due to the human-annotated reference critiques used in the prompt.

Regarding other potential biases, such as style and specific word preferences, our human annotations haven't observed substantial influence from these factors.

Q3: Did you consider a few-shot performance?

A3: Yes, we have studied the few-shot prompting strategy. However, our results demonstrate that few-shot prompting reduces performance across various LLMs. For example, when using 1-5 examples in objective feedback evaluations, we observed a significant decline in LLMs' performance as the number of examples increased.

This intriguing phenomenon may be due to the complexity of the critique task, where few-shot examples might impede the LLM’s understanding of the evaluated responses. Consequently, CriticEval currently does not utilize few-shot prompting by default. Given the emerging interest in critique ability research, we look forward to future works investigating advanced inference strategies to improve critique ability of LLMs.

References

[1] AlpacaEval: An Automatic Evaluator of Instruction-following Models

[2] How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources

We really appreciate your valuable suggestions and insightful questions. We will address your concerns as follows.

Q1：... relies on GPT-4 for evaluation. This could introduce a bias favoring models such as GPT-4 ...

A1: Please refer to "Global Response - Overcome Bias of GPT-4 Judge" for more details. In summary, we supply the human-annotated critiques to improve GPT-'s reliability, and experimental results in the meta-feedback dimension have validated its reliability (Section 6.2).

Q2: ... a more diverse set of models or purely human-generated critiques provide a more neutral evaluation framework.

A2: As described in Appendix A.1, human annotators might be influenced by GPT-4' initial critiques. We wish to emphasize that our decision to employ a human-in-the-loop rather than pure human annotation is motivated by two essential considerations (effectiveness and efficiency):

1. Human-written Critiques are Not Comprehensive (Effectiveness)

Our human annotation reveals that annotators might neglect some apparent or severe issues when writing critiques from scratch, consistent with findings in recent studies [1,2]. Neglecting issues usually leads to low-quality critiques. Specifically, experimental results in Table 2 of Global Response - Fine-grained Failure Modes proves that missing issues in critiques lead to low subjective scores.

In contrast, despite the possibility of generating inaccurate critiques, LLMs like GPT-4 offer more comprehensive and detailed critiques [1,2]. By revising LLM's errors by human experts, the final critiques could be more comprehensive and accurate, leveraging the strengths of both human annotators and LLMs [1,2].

As detailed in Appendix G.5, our human annotations exhibit significant revisions (25.22%, 34.83%, and 48.37%) on GPT-4's initial critiques, effectively alleviating potential bias and noise.

2. Annotating Critique Task from Scratch Cost A Lot (Efficiency)

Writting critiques from scratch is a significant challenge [3,4]. For instance, Shepherd [4] incurred an annotation cost of 8\$per sample, leading to over 28,864\\$ and 1,350 work hours to annotate the entire CriticEval, which is unbearable for our project. Thus, we have to employ advanced LLMs to generate draft critiques, followed by human annotation.

GPT-4 is chosen because our preliminary studies indicate that it is the most reliable LLM for producing draft critiques, while other LLMs are much worse (Table 2). Consequently, a diverse set of LLMs introduces more noise in draft critiques, bringing more difficulties to human annotators.

In conclusion, the human-in-the-loop pipeline achieves the trade-off between annotation cost and quality. We promise to add these details to the revised paper to emphasize the motivation of using the human-in-the-loop pipeline. Thanks for your detailed review and insightful question.

Q3: The paper doesn't adequately address the scalability of CriticEval ... reliance on GPT-4 and humans make it challenging to reproduce or extend CriticEval

A3: Please refer to Section "Global Response - Scalability and Cost" for our explanations. Our explanations reveal that the cost of reproducing and extending CriticEval is comparable to established benchmarks like AlpacaEval and AlpacaFarm.

Q4: Insufficient analysis of failure modes ...

A4: Thanks for your very valuable suggestions. We have conducted coarse-grained and fine-grained analyses of failure modes before. Due to space limitations, we only included the coarse-grained analysis in Section 6.6.

To address your concern, we have supplemented our findings in "Global Response - Fine-grained Failure Modes." This supplementary material reveals intriguing phenomena. For example, the most frequent failure modes are missing errors, lacking effective comparison and worse revision than references for feedback, comparison and correction. Besides, inaccurate critiques usually lead to lower subjective scores. The revision that does not follow suggestions in feedback usually leads to the worst performance.

We really appreciate your insightful suggestions and the opportunity to address them.

Q5: Lack of comparison to human performance ...

A5: As described in Q1-A1, collecting purely human-annotated critiques is challenging. Thus, human performance is not recorded in the current submission.

We agree that human performance is valuable, and we are urgently working on annotating human-generated results on the test set. All annotators have an undergraduate level of education. Due to the massive workload of human annotation, subjective annotations are ongoing, and the objective scores have been provided below:

Human performance slightly outperforms GPT-4 on four critique dimensions in the objective split. We promise to add full human performance results in the revised paper.

Thanks for your valuable suggestions.

Q6: How did you evaluate human performance during benchmark construction?

A6: As briefly described in Appendix G.1, for the textual critiques, the supervisor's (authors) review and revise mechanism ensures the quality of human annotation meets our expectations. For the scalar-based critiques, the supervisors conducted a 5% sample inspection. If the error rate exceeds the threshold, annotators are asked to revise their work until the error rate is lower than the threshold. Besides, the inner-agreement among annotators are computed to make sure their judgments are consistent.

References

[1] LLM Critics Help Catch LLM Bugs (OpenAI)

[2] Self-critiquing models for assisting human evaluators (OpenAI)

[3] AlignBench: Benchmarking Chinese Alignment of Large Language Models

[4] Shepherd: A Critic for Language Model Generation

Dear Reviewer kJKF,​

We would like to thank you for the thoughtful and constructive feedback and appreciate that you agree on the strengths of our paper.​

We provided details and analysis to address your concerns during the rebuttal. In this response, we complete the human performance annotation of the subjective tasks on the CriticEval test set, and the overall human-level performance is shown as follows. ​

Note that the cohort and corresponding set of human critiques do not represent the best possible human performance; instead, they represent the capability of annotators selected for this human performance annotation of the CriticEval test set.

As described in Section 5, objective scores (Obj.) for feedback, comparison, and correction are correlation, accuracy, and pass rate. Subjective scores (Sub.) for feedback, comparison, and correction are Likert scores (1-10) generated by GPT-4 with human-annotated critiques as references.

Experimental Results: As shown in the Table above, it can be found that the human level significantly outperforms GPT-4 on the objective task, while it is inferior to GPT-4 on the subjective evaluation.

Then, we conduct the Quantitative and Qualitative Analysis to understand the human performance in subjective evaluation.

Quantitative Analysis

The distribution of failure modes for humans and GPT-4 is shown in the following tables. The numbers in the tables indicate the frequencies of error types in critiques. The descriptions of failure modes are placed in Table 1 of "Global Response - Fine-grained Failure Modes."

Experimental Results: As for the feedback and comparison dimensions, the distribution of E1 (missing issues), E2 (missing suggestions or low-quality suggestions), and E7 (insufficient analysis) in human-written critiques is significantly higher than that of GPT-4. In contrast, the distribution of other errors (most are mistakes in critiques) is much lower.

As for the correction dimension, human annotators usually do not follow suggestions in the provided feedback (E9) and generate additional errors (E11). Through communicating with the annotators, we notice that the primary cause of this issue is that some tasks require domain-specific knowledge. The lack of this knowledge among annotators leads to lower-quality corrections. This phenomenon aligns with the findings of recent work [1].

Conclusion: The human-written critiques are often less comprehensive than GPT-4, significantly reducing the quality. In contrast, the mistakes in human-written critiques are significantly less than that of GPT-4. This phenomenon is consistent with our preliminary study and recent findings [1], further proving the reasonableness of using the human-in-the-loop pipeline to collect comprehensive and accurate critiques (as described in Q2-A2).

Qualitative Analysis

We inspect the human critiques in the subjective evaluation. We notice that human annotators generally write fewer comments than LLMs, and comments are usually general and brief. Besides, many tasks involve domain-specific knowledge that humans may lack, but GPT-4 excels in (albeit with potential hallucinations).

Summary

We provide the human-level performance for the CriticEval test set. The human performance is better than GPT-4 on the objective split, while it is inferior to GPT-4.

We wish to emphasize that: Although our first submission lacks pure human performance, our reference critiques obtained through the human-in-the-loop pipeline could serve as very close candidates for the best possible human performance. During the subjective evaluation, the quality score of reference critique is anchored to 8 (overall 1-10 score range), serving as a relative scoring pivot, which is helpful in analyzing the performance gap between LLMs and human.

We will supplement experimental results and analysis into our revised paper.

We hope these responses address the concerns. We are happy to discuss further comments and suggestions. If the reviewer finds our response adequate, we would appreciate it if the reviewer considers raising the score.

Reference

[1] LLM Critics Help Catch LLM Bugs

Best Regards,

Submission 7061 Authors

We really appreciate your valuable suggestions and insightful questions. Your questions are addressed as follows:

Q1: One important metric for evaluation models, the ability to revise its generation with critique, is excluded ... It is necessary to evaluate each model’s ability to revise existing outputs when provided with critique.

A1: We appreciate the reviewer's insightful comment regarding evaluating the ability to revise generation with critiques. We want to clarify that this capability is a central aspect of our work and has been thoroughly evaluated in our CriticEval benchmark, referred to as the correction ability ($CR$) in our paper. We acknowledge the reviewer's suggestion that "revision" is more precise than "correction" to describe this feature. We will update the terminology accordingly in the revised version of our paper, replacing "correction" with "revision."

To further clarify the evaluation of the revision ability in our study, we highlight the following points:

1. Data Collection and Metric for Revision Evaluation:

• Section 4.3 (line 155) details the process of collecting reference revisions.
• Sections 5.1 (line 189) and 5.2 (lines 194-196) outline the objective and subjective metrics used to evaluate the model’s ability to revise generations.

2. Evaluating Revision Ability: Our evaluation of LLMs’ revision ability encompasses two scenarios: using golden feedback as input and using feedback generated by other LLMs.

• Golden Feedback as Input: Subjective scores are computed by comparing the model’s revisions to human-annotated revisions, and the reliability of this subjective evaluation is substantiated in Section 6.2 (Table 4 and lines 240-246).
• Feedback from Other LLMs as Input: We also conducted experiments to assess how LLMs perform with feedback produced by others, including themselves. As detailed in Section 6.5 (lines 311-326), we evaluated the average performance of the LLMs using three types of feedback: human-annotated, empty, and self-generated. The results indicate that LLMs face challenges in self-improvement, particularly in complex reasoning tasks.

We hope this clarification addresses the reviewer’s concern and demonstrates the comprehensive nature of our evaluation approach. We are grateful for the opportunity to enhance the terminology and presentation of our findings, which we believe will further clarify the contributions of our work.

Q2: ... it excludes critique ability in other multilingual contexts ... and additional data collection on this aspect seems necessary.

A2: Please refer to Section "Global Response - Multilingual Support" for our explanations.

In summary, our work is the first step to constructing a comprehensive and reliable evaluation. The multilingual feature is not considered because we mainly focus on the critique ability over common tasks in the current stage. But, the cost of extending CriticEval to other languages is affordable.

Q3: What is the pattern of score changes when using feedback from other LLMs in Table 5, based on the quality of the feedback from these other LLMs?

A3: We apologize for any confusion caused by Table 5 and the related content in Section 6.3. Since feedback and revision performance are entangled, it is essential to investigate how the quality of feedback affects the performance of LLM's revision ability.

To explore this, we prompted the InternLM2-20B-Chat and Llama2-70B-Chat models to revise responses from CriticEval using three types of feedback with varying quality levels. To enhance clarity, we have restructured Table 5 as below: the first table presents results from the objective split of CriticEval, and the second table displays results from the subjective split.

These results indicate that as the quality of the feedback increases, both the objective and subjective revision performance improves. This trend demonstrates that higher-quality feedback is associated with more effective revisions.

We really appreciate your valuable suggestions and insightful questions. Since some of your questions are similar to those of other reviewers, we will describe them in more detail in Global Response. We also provide some summary of these questions under your review comment to make our explanation clear.

Q1: ... it heavily relies on GPT-4 for evaluations ... The potential biases introduced by using a single LLM for evaluations limit broader applicability.

A1: Please refer to "Global Response - Overcome Bias of GPT-4 Judge" for our explanations. In summary, experimental results in the meta-feedback dimension prove the reliability of GPT-4 as a judge, while other LLMs are much worse. Besides, to further improve the reliability of subjective evaluation, GPT-4 is equipped with our human-annotated critiques, which is a trade-off solution for scalability and reliability of subjective evaluation.

Q2: ... I believe there should be a serious discussion on how such evaluation pipelines and datasets would function for low-resource languages ...

A2: Please refer to Section "Global Response - Multilingual Support" for our explanations. In summary, CriticEval could be extended to other low-resource languages at an affordable cost using translation and further human labor.

Q3: The computational cost and real cost of setting up the CRITICEVAL pipelines should be spelled out.

A3: Thanks for your valuable suggestion. Please refer to "Global Response - Scalability and Cost" for more details about the computation and construction cost of CriticEval. In summary, the construction and evaluation cost in CriticEval is comparable to established benchmarks, like AlpacaEval and AlpacaFarm.

Q4: It is not clear if IRB/Ethical clearance was obtained as the checklist response states that IRB would be easy to obtain ...

A4: Thanks for your highly meticulous review. We apologize for any confusion and misunderstanding of the meanings between the human subjects and the crowdsourcing. CriticEval is annotated by crowdsourcing, and no human subjects are studied in our work. Besides, as described in Appendix B, the hourly wage of crowdsourcing is much higher than that of Amazon Mechanical Turk. Thus, our work does not violate the NIPS Code of Ethics.

Thank you for bringing this to our attention.

Q5: How do you protect against errors in the underlying tasks datasets ... how would this affect your work and how would you mitigate against it?

A5: We agree that some underlying datasets contain inaccuracies that may lead to compounding effects during evaluation. For example, we have observed some incorrect solutions and reasonings for mathematics and coding questions during our human evaluation process.

In our work, to mitigate the effects of such errors, human annotators are instructed to examine each question meticulously, the provided golden answers and the evaluated responses and critiques. They are asked to exclude instances where the golden answers or questions are flawed or incorrect, like wrong solutions to mathematics and coding questions.

We promise to supplement more details and cases in the Appendix. Thank you for raising this critical issue, and we appreciate the opportunity to strengthen our work through your valuable comments.

Q6: Ultimately how many people are involved and computational cost of such an exercise as this affects replication ability of this study to cover some of the limitations you hilighted?

A6: Please refer to Section "Global Response - Scalability and Cost" for more details about our explanations. Our explanations reveal that the cost of reproducing and extending CriticEval is comparable to established benchmarks like AlpacaEval and AlpacaFarm, ensuring replication ability for the research community.