PiCO: Peer Review in LLMs based on Consistency Optimization

Thank you for recognizing our work. We appreciate your valuable feedback, and we would like to address the issues you raised.

Q1. What’s the meaning of “average loss” in the caption of Figure 5?

A1. We apologize for the confusion caused by this. The "average loss" means the opposite of consistency in Eq.8, i.e., $-$ Consistency $(G,w)$. For better understanding, We will change the average loss in Figure 5 to "peer-review system consistency" in our revision.

Q2. What’s the meaning of Precision@K and RBP@K in Section 3.4?

A2. The Precision@K and RBP@K are two popular metrics for evaluating ranking, which are widely used in recommendation systems. We employ these metrics to measure the alignment between the learned LLM ranking and the human-preference ranking.

Q3. There are other methods that select reviewers unsupervised, such as PRE with auto exam in the original PRE paper. It would be good to compare the method in this paper with it as well.

A3. The comparison results are demonstrated in Figure 4. We compared the performance of selecting reviewers with PRE. By the way, the PRE is not unsupervised, it requires some human annotations as exam questions. The results in Figure 4 show the superiority of our PiCO approach.

Q4. In Table 2, the variance of multiple baselines is 0, which seems a bit weird. It might be better to explain a bit.

A4. The variance of some baselines is 0 in Table 2 because the learned rankings $\hat{\mathcal{R}}$ are the same across different seeds. We will add some explanations for better understanding.

First of all, we appreciate your constructive feedback on our work. However, it seems there might be some misunderstandings about our work, especially regarding the misconception of "using the same models as both reviewers and those used for training and evaluation." We recommend reexamining this section 2 for clarification. Next, we would like to address the issues raised by the reviewer.

Q1. The proposed method uses the same models as both reviewers and those used for training and evaluation. This is akin to comparing performance using the training loss, and the experimental results do not reflect the performance on unseen models.

A1. We highlight this paper aims to tackle the LLM ranking capability issue in an unsupervised scenario, similar to a "combinatorial optimization" problem. The goal is to find an optimal set of capability weights $w$ that maximizes the consistency of the entire peer review system. Therefore, in this "combinatorial optimization" problem, we do not have a training and testing process, we only care about whether the unsupervised optimized LLM ranking satisfies the true human-preference ranking as much as possible.

Q2. I suggest that the authors demonstrate whether the trained PiCO assigns higher ranks to models that are superior to the reviewer models used in its training.

A2. The results are demonstrated in Table 1 (section 2) and Figure 6a (section 3). In Table 1, we use the two popular ranking metrics (i.e., Spearman’s Rank $S\uparrow$ and Kendall’s Rank Correlation Coefficient $\tau\uparrow$) to measure the alignment between the learned ranking $\hat{\mathcal{R}}$ by PiCO and the ground-truth human-preference ranking $\mathcal{R}^*$. We can obtain the following two conclusions from Table 1. First, it can be observed that the Forward Weight achieves better results than the Uniform and Backward ones in all cases, while the Backward one always achieves worse results. It validates that assigning larger weights to those models with stronger capabilities can obtain better results, i.e., the consistency assumption is valid. Most importantly, we start with random weight and optimize the capability weights by our PiCO can obtain better ranking performance. It further validates the proposed approach can assign higher ranks to models that are superior to the reviewer models. More details are demonstrated in lines 186~202. In addition, we visualize the learned $w$ in Figure 6a. It can be observed that some superior models (e.g., GPT-3.5, WizardLM-13B, Guanaco-33B, Vicuna-13B) will be assigned higher weights while some weak models (e.g., Alpaca-13B) will be assigned lower ones. It is closer to the LLM ranking of real human preferences than other methods.

Q3. The proposed method does not experimentally demonstrate whether it can serve as a replacement for existing benchmarks.

A3. Thanks for your valuable suggestions. We select the widely-used benchmarks (i.e., MMLU and GSM8K) to evaluate the model performance ranking $\hat{\mathcal{R}}$, and calculate the Spearman’s $S(\uparrow)$ and Kendall’s $\tau (\uparrow)$ rank correlation with the human preference ranking $\mathcal{R}^*$. The results are as follows,

Reference

[1] Zhou K, Zhu Y, Chen Z, et al. Don't make your llm an evaluation benchmark cheater[J]. arXiv preprint arXiv:2311.01964, 2023.

[2] Zheng L, Chiang W L, Sheng Y, et al. Judging llm-as-a-judge with mt-bench and chatbot arena[J]. Advances in Neural Information Processing Systems, 2023, 36: 46595-46623.

[3] Chen D, Chen R, Zhang S, et al. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark[J]. arXiv preprint arXiv:2402.04788, 2024.

[4] Jain N, Saifullah K, Wen Y, et al. Bring your own data! self-supervised evaluation for large language models[J]. arXiv preprint arXiv:2306.13651, 2023.

[5] Chang Y, Wang X, Wang J, et al. A survey on evaluation of large language models[J]. ACM Transactions on Intelligent Systems and Technology, 2024, 15(3): 1-45.

Q4. The writing lacks academic rigor and professionalism.

A4. We appreciate your valuable suggestions. We will further polish our paper, making it more academic and readable.

We appreciate your constructive suggestions. Next, we will address your issues.

Q1. The paper does not provide detailed information on how the ground truth rankings in the experiments are established, which could impact the transparency and reproducibility of the research.

A1. This paper uses the ground-truth ranking by human-preference annotations from crowdsourced battle platforms. For example, the following ground-truth rankings are collected by ChatBot Arena [1], the ranking can be replicated by the official source code in [2, 3]. The other two datasets are similar to this. We will add this detailed information in our revision.

[1] Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 2023.

[2] https://huggingface.co/datasets/lmsys/chatbot_arena_conversations

[3] https://colab.research.google.com/drive/1J2Wf7sxc9SVmGnSX_lImhT246pxNVZip?usp=sharing#scrollTo=aq3NlxirIyb7

Q2. The effectiveness of the peer review process heavily relies on the design of the questions and prompts. How can we mitigate uncertainty and bias in the evaluations conducted by models?

A2. The consistency of the peer-review process is stable across questions, as we validate the PiCO on multiple crowdsourcing datasets (as shown in Table 1) where the question distribution varies greatly across different datasets. On the other hand, the prompt template can be found in lines 787~847. We changed the expression of the prompt template and repeated the PiCO experiment several times as follows,

The learned final LLM ranking $\hat{\mathcal{R}}$ with multiple system prompts are as follows. The proposed PiCO approach is stable across different system prompts.

In addition, to further mitigate evaluation bias and uncertainty caused by the order of answers, in all our experiments, each battle is evaluated twice in the forward and reverse order as follows.

Q3. It does not discuss specific application scenarios or domains where this method could be particularly beneficial. What is the application scenario of PICO?

A3. This paper performs experiments on three of the most popular crowdsourcing datasets. On these three datasets, we validate that the LLMs themselves can employ the peer-review mechanism for good self-evaluation. Therefore, we believe that this is a universal phenomenon that can be applied to various fields of the dataset, rather than a special case. Recently, we applied PiCO to the evaluation of multimodal large vision-language models (LVLMs) and obtained similar effects. On the other hand, PiCO works in an unsupervised setting, so it will benefit greatly from LLM evaluation in scenarios where human preference annotation is very expensive, as it can get a good LLM ranking without real human annotation.

Dear Reviewer fWmA,

We hope this message finds you well. We have provided detailed responses to the concerns you raised. We would greatly appreciate it if you could review our responses at your earliest convenience. If there are any concerns that you feel have not been fully addressed, we are eager to discuss them with you.

Thank you once again for your valuable time and constructive feedback.

Best regards,

The Authors