We are truly grateful for the constructive feedback from this review. We are excited to find that the reviewer thinks our work focuses on a really interesting problem. We are delighted to find our paper has a good contribution(3) and presentation (3).

Weaknesses

1. The study seems to be limited to question answering only. LLM has many other applications such as writing reviews and generating essays. Would the proposed methods be applicable to improve evaluations on those applications as well?

• We kindly point out that we evaluated our model on the diverse Vicuna80 dataset encompassing a wide variety of tasks: writing, roleplay, common sense, fermi, counterfactual, coding, math, generic, knowledge, etc. Each task includes over 10 questions.

2. Significance of results in Table 3, 4, and 5? It'd be great if the authors could present t-test results such as p-values of the comparisons.

• We added significance tests for all of the results as shown below:
• Table 3 shows the example-level correlation results for peer rank. Two-tailed t-test results are shown on the accuracy measure for our method compared to the other baselines (statistical significance is indicated with *(p<0.01), **(p<0.002))

• Table 4 describes peer discussion accuracy on LFQA. We note that when PD happens between a model and a different model (e.g. Claude & GPT-4), results are significantly better. Statistical significance is indicated with *(p<0.05), **(p<0.005).

• Table 5 shows the comparison of PD and PR on Vicuna80. Two-tailed t-test results are shown on the F1 measure for our method compared to the baseline.

Questions

1. Is there any data contamination issues for the LLMs being tested?

• The models tested are released before the creation of our Vicuna80 and LFQA datasets, so there should be no or little contamination happening.
• Besides, there are no golden answers to open-ended questions in our datasets and there are possibly large sets of good answers. Thus, there are much fewer data contamination issues as compared to exam-like questions [1, 2].

References

1. Yadav, Vikas, Steven Bethard, and Mihai Surdeanu. "Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering." arXiv preprint arXiv:1911.07176 (2019).
2. Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).

We sincerely thank the reviewer for constructive feedback. We sincerely want to discuss with you about your review and our reply. If you think our clarifications and reporting of additional experiments address your original concerns, could you please consider adjusting the soundness or excitement, contribution, and rating scores? – If there are any other concerns, please let us know and we’ll try our best to address them!

We sincerely thank the reviewer for giving feedback and suggesting further experiments on open-source models. We are encouraged that the reviewer finds our work has good soundness(3), presentation(3), and contribution(3).

Weaknesses

1. Discussing trade-off between PRD cost and benefits, compared to human evaluation.

• We admit that the cost of calling API-based commercial LLMs might be an issue, but it is more fair and is commonly used in evaluating LLMs in recent research literature, such as for evaluating model performance on open-ended questions [1, 2],  summarization [3], dialogue generation [4], etc.
• Moreover, illustrating the decision process through multi-round LLM interactions is an essential part of recent work on model explainability, in topics such as evaluation [5], scalable oversight of model capabilities [6], and code optimization and text generation [7] – we will add this to the final version.
• In our work, we hired domain experts such as food science experts (from Upwork: http://upwork.com) for the human pairwise comparison annotations. For each pair comparison, the annotation takes on average 5-10 minutes. We pay \$2-3 for each question. While in our experiments, GPT-4-based evaluation costs only about \\$0.015 for each question (based on token number). – Thus we believe using LLM for automatic evaluation is promising and cost-efficient. – human annotation details are included in Appendix G.
• Beyond LLM-based automatic evaluation, another bigger scope theme of this  work is about scalable oversight of models [8]. – Enable models to induce self-rankings that are hard for humans to compare.

2. On the same line, this work does not consider open-source LLMs. Knowing what open-source LLM can be used here could be beneficial for the community.

• Open-source LLMs are always an important part of the community. The original paper already includes experiments involving Vicuna-13b which is an original model based on LLaMA [10].
• Additionally, we designed a new experiment for the PR method including new public models: Zephyr-7b-alpha [9], Vicuna-13b (already discussed in the paper),  and GPT-3.5. The results are consistent with Table 2 in the paper.

New results:

• We show that both ranking and Elo change and become closer to the golden ranking when using PR, which is consistent with the experiments of commercial LLMs.

Questions

1. What other options can be used from the open-source scenario?

• Many open-source models can be applied in our method, such as Vicuna, Zephyr[9], LLaMA[10], etc. We conducted a new experiment shown above.

References

1. Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." arXiv preprint arXiv:2305.14314 (2023).
2. Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023).
3. Liu, Yang, et al. "Gpteval: Nlg evaluation using gpt-4 with better human alignment." arXiv preprint arXiv:2303.16634 (2023).
4. Lin, Yen-Ting, and Yun-Nung Chen. "LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models." arXiv preprint arXiv:2305.13711 (2023).
5. Chan, Chi-Min, et al. "Chateval: Towards better llm-based evaluators through multi-agent debate." arXiv preprint arXiv:2308.07201 (2023).
6. Michael, Julian et al. “Debate Helps Supervise Unreliable Experts.” (2023).
7. Madaan, Aman, et al. "Self-refine: Iterative refinement with self-feedback." arXiv preprint arXiv:2303.17651 (2023).
8. Bowman, Samuel R., et al. "Measuring progress on scalable oversight for large language models." arXiv preprint arXiv:2211.03540 (2022).
9. Tunstall, Lewis, et al. "Zephyr: Direct Distillation of LM Alignment." arXiv preprint arXiv:2310.16944 (2023).
10. Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

We sincerely thank the reviewer for constructive feedback. We sincerely want to discuss with you about your review and our reply. If you think our clarifications and reporting of additional experiments address your original concerns, could you please consider adjusting the soundness or excitement, contribution, and rating scores? – If there are any other concerns, please let us know and we’ll try our best to address them!

We thank the reviewer for the feedback.

Weaknesses

1. Motivation and problem

• We focus on the topic of utilizing large language models (LLMs) for evaluating open-ended generation instead of proposing a decoding strategy. The goal is to propose new methods using LLMs for better and fair evaluations that align with human judgments.
  • The basic task formulation is like, given answers A and B, the model is required to evaluate which of them is better.
  • Determining pairwise comparison is also a key step for InstructGPT training [7].
• To get an overview and basic knowledge of the topic, we kindly refer the reviewer to skim a set of related work on LLM-based evaluation [1,2,3,4,5,6,7], where [1] is a very basic model in this direction.

2. Proposed solution

• We propose (1) the peer rank (PR) algorithm that takes into account each peer LLM’s pairwise preferences of all answer pairs and outputs a final ranking of model capabilities;
• And (2) peer discussion (PD), where we prompt two LLMs to discuss and try to reach a mutual agreement on preferences of two answers.

3. Results

• We conduct comprehensive experiments on two benchmark datasets. We find that our approaches achieve significantly higher accuracy and better alignment with human preference.

Questions

1. Is the aim of your method to help LMs produce better answers to questions than other methods currently achieve?

• As outlined in the abstract and introduction, our aim is to introduce improved evaluation methods for large language models (LLMs) that enhance accuracy and better align with human judgments. Our research opens up possibilities for assessing models that are challenging for humans to compare.

Finally, we are eager to answer any further questions from the reviewer to help fully understand the intellectual merits of our work.

References

1. Fu, Jinlan, et al. "Gptscore: Evaluate as you desire." arXiv preprint arXiv:2302.04166 (2023).
2. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023.
3. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023a.
4. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
5. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with language-model-as-an-examiner. arXiv preprint arXiv:2306.04181, 2023.
6. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
7. Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744.

Dear reviewer qoLU,

Thank you for the clarification.

We want to emphasize that we can try our best to improve the writing, and added the significance test for demonstrating that improvements are significant through our the paper.

We sincerely hope that you could provide more fine-grained and targeted reviews on which part of writing should be improved and which part of experiments should be improved (we added the significance test and have shown that all the improvements are significant from t-test), as were in other reviews.

Plus, since your review kinds of deviates from the other three reviewers, can you possibly read the other reviews and confirm that you haven't got any aspect wrong when judging the intellectual metrics of our work?

• Table 3 shows the example-level correlation results for peer rank. Two-tailed t-test results are shown on the accuracy measure for our method compared to the other baselines (statistical significance is indicated with *(p<0.01), **(p<0.002))

• Table 4 describes peer discussion accuracy on LFQA. We note that when PD happens between a model and a different model (e.g. Claude & GPT-4), results are significantly better. Statistical significance is indicated with *(p<0.05), **(p<0.005).

• Table 5 shows the comparison of PD and PR on Vicuna80. Two-tailed t-test results are shown on the F1 measure for our method compared to the baseline.

Dear Reviewer qoLU,

We are sorry that we were not able to help understand of paper. Considering this, can you send a message or write it here to ask the AC/SACs to not consider your review? – so that the decision of our paper’s acceptance/rejection would be more fair?

Questions

1. For PR used in the win rate calculation, how will you determine whether it has converged? I only see the number of iterations in Algorithm 2 in Appendix E. Have you ever analyzed the impact of the number of iterations on the results?

• We examine the impact of iteration quantity by manually contrasting outcomes for N iterations and significantly higher iterations, such as 2 * N or 3 * N. If the variations in each model's rating are less than 0.0001, we conclude that PR has reached convergence.

2. I need more information about the human annotation process. How exactly did the annotation process work? How did you recruit the participants and how about their backgrounds? How much were they paid?

• In our work, we hired domain experts such as food science experts (from Upwork) for the human pairwise comparison annotations. Before annotation, we provided a detailed instruction document to annotators and answered their questions.
• For each pair of comparisons, annotators follow instructions on a UI shown in Appendix G, and the annotation takes an average of 5-10 minutes. We pay $2-3 for each question.

References

1. Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).
2. Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744.
3. Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." arXiv preprint arXiv:2305.14314 (2023).
4. Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023).
5. Liu, Yang, et al. "Gpteval: Nlg evaluation using gpt-4 with better human alignment." arXiv preprint arXiv:2303.16634 (2023).
6. Lin, Yen-Ting, and Yun-Nung Chen. "LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models." arXiv preprint arXiv:2305.13711 (2023).
7. Chan, Chi-Min, et al. "Chateval: Towards better llm-based evaluators through multi-agent debate." arXiv preprint arXiv:2308.07201 (2023).
8. Michael, Julian et al. “Debate Helps Supervise Unreliable Experts.” (2023).
9. Madaan, Aman, et al. "Self-refine: Iterative refinement with self-feedback." arXiv preprint arXiv:2303.17651 (2023).
10. Bowman, Samuel R., et al. "Measuring progress on scalable oversight for large language models." arXiv preprint arXiv:2211.03540 (2022).

We sincerely thank the reviewer for constructive feedback. We are encouraged to find that the reviewer finds our work benefits to further research.

Weaknesses

1. Limited improvements PR. Elo rating becomes closer to human raters' rating, but the ranking of the five LLMs does not change.

• We would politely point out that the Elo score is a more fine-grained and important metric for comparing model capabilities. Recent works [1, 2, 3] all used it as the main metric. This metric is important even if the ranking doesn’t change in the specific case. The Elo rating of "All (Weighted)" is the closest to human rating. It shows our method has a better alignment with human judgment.
• Ranking is more general and coarse-grained and is not informative when the gap of model capabilities is large. Thus it’s more convincing to review the rank when including models that are of similar capabilities (on a challenging set of questions). – we added new experiments including another set of LLMs.

Additional Experiment:

• In the new experiments, we added new public LLM and conducted PR among Vicuna-13b, Zephyr-7b-alpha, and GPT-3.5. We show that both ranking and Elo change and become closer to the golden ranking, which is consistent with the experiments of commercial LLMs. This change demonstrates PR also contributes to adjusting the global ranking.

2. In Table 3, the example-correlation does not increase too much, comparing "All Reviewers (Weighted)" to "GPT-4 & Claude & GPT-3.5". These indicate that the core idea of PR may not be very useful.

• "GPT-4 & Claude & GPT-3.5" is also an important variation of our PR approach of the weighted combination of all reviewers. The “All reviewers (Weighted)” includes two additional models. Both of them significantly outperform the baselines according to the significance test (>2%).

3. In Table 5, the performance boost with PD is not that great

• We conducted a two-tailed t-test and results show that p-value of PD as compared to the baseline is smaller than 0.01, demonstrating significant improvement. – we will add this to the final version.

4. The two proposed methods are weakly related, except that they both introduce "peers" to some extent. The paper doesn't combine the two, which makes me feel cut off.

• We politely argue that PR and PD are closely interrelated and fall under the same theme of providing a more fair (de-biased) ranking of long- and free-form answers. PR is for global ranking and PD is for fine-grained ranking.
• In Table 5 we compare their performance and provide insights on the difference between PR and PD and their benefits.
• Plus, we integrated them into one unified peer evaluation framework, i.e. first using PR to determine the weight of each LLM reviewer, and then conducting fine-grained PD among them, the first prunes out the weak LLMs and the second step enables more explainable evaluation decisions (discussion history). Through the study, we found that it achieves good efficiency and satisfiable results.
• We will add more details in the final version and adjust the corresponding part to make the connection sound more smooth.

5. Introducing the communication between LLMs to obtain a better solution will require computational cost.

• We admit that the cost of calling API-based commercial LLMs might be an issue, but it is more fair and is commonly used for evaluations in recent research literature. Such as for evaluating model performance on open-ended questions [3, 4],  summarization [5], dialogue generation [6], etc.
• Moreover, illustrating the decision process through multi-round LLM interactions is an essential part of recent work on model explainability, in topics such as evaluation [7], scalable oversight of model capabilities [8], and code optimization and text generation [9] – we will add this to the final version.
• In our work, we hired domain experts such as food science experts (from Upwork: http://upwork.com) for the human pairwise comparison annotations. For each pair comparison, the annotation takes on average 5-10 minutes. We pay \$2-3 for each question. While in our experiments, GPT-4-based evaluation costs only about \\$0.015 for each question (based on token number). – Thus we believe using LLM for automatic evaluation is promising and cost-efficient.
• Beyond LLM-based automatic evaluation, another bigger scope theme of this work is about scalable oversight of models [10]. – Enable models to induce self-rankings that are hard for humans to compare.

We sincerely thank the reviewer for constructive feedback. We sincerely want to discuss with you about your review and our reply. If you think our clarifications and reporting of additional experiments address your original concerns, could you please consider adjusting the soundness or excitement, contribution, and rating scores? – If there are any other concerns, please let us know and we’ll try our best to address them!