PiCO: Peer Review in LLMs based on the Consistency Optimization

Thank you for your thoughtful comments, which prompt us to improve our manuscript.

Q1: What is the benefit of eliminating the weaker ones rather than keeping them around? how did you come up with the threshold of 60% to remove? can you learn this threshold automatically?

A: The benefit of eliminating the weaker ones rather than keeping them around can make the peer-reviewer system more robust. We found that the weaker LLMs also have worse evaluation ability, which will introduce much evaluation noise to damage the peer-review system. We plot the average training loss curve with the increase of eliminated reviewers in Figure R3 of the additional rebuttal material. It can be observed that when eliminating weaker reviewers, the average loss of the whole peer-review system is reduced. It shows that removing those noise evaluations will benefit the whole system. On the other hand, when 60% ($=9$) weaker reviewers are eliminated, the whole system loss achieves the lowest. This phenomenon occurs across three datasets, which indicates that the eliminating threshold is automatically learned. In addition, removing more strong reviewers ($>9$) will also hurt the evaluation process. We appreciate your valuable review, and we will add these results and discussions in our revised paper.

Q2: Are there any scaling issues for 100, 1K, 10K, or 100K models? how about the cost?

A: In our experiments, we validate the scaling issues of the proposed PiCO approach from another perspective. As shown in Table 1 (main text) and R1 (global author rebuttal), we randomly used 100%, 70%, 40%, and 10% of data to perform the experiments on three datasets, the results consistently validate the effectiveness and superiority of the proposed approach. In other words, we can obtain consistent results using only a small portion of the data. From the cost perspective, PiCO has the same complexity as the Chatbot Arena as the model amount scales up. Both tokens consumed will increase, but the human annotation cost of PiCO is significantly lower because it does not require any annotation.

Q3: While the three metrics you use are meaningful, you should also present results for Precision@1 and RBP@3.

A: Thanks for your valuable suggestion. We demonstrated the results of precision and RBP as follows.

The results show that the proposed PiCO approach can achieve better precision and RBP performance in most cases. In addition, as suggested by Reviewer #8Eov, we also construct our experiments under more ranking metrics, i.e., Spearman's rank correlation coefficient $S(\uparrow)$, and Kendall's rank correlation coefficient $\tau(\uparrow)$. These results in global author rebuttal (Table R1) once again validate the superiority of the proposed PiCO approach.

Q4: Fig 5 should be extended to nine graphs (3 metrics * 3 datasets). Discuss that not all "CIN = 1" are created equal.

A: The nine graphs (3 metrics * 3 datasets) are demonstrated in Figure R4 of rebuttal material. The results show that the proposed PiCO approach can achieve better performance than PRE in most cases. In addition, "CIN = 1" can not be always created equal. Therefore, other ranking metrics, such as PEN, LIS, Spearman's, and Kendall's rank correlation coefficients, have been employed to help us validate the effectiveness of the proposed approach from other perspectives.

Q5: What is the benefit of a model evaluating itself? why not explicitly disallow it? w/o a full explanation, this detail unnecessarily complicates the story.

A: Thanks for your valuable suggestion. Evaluating itself or not will not have an impact on the final results. We will simplify the expression of this detail in our revised paper.

Q6: You repeatedly talk about ``higher-level LLM'' w/o defining the concept; is it based on the number of parameters (larger or higher)?

A: In this paper, "higher-level LLMs" indicate those LLMs with stronger capabilities or better performance. It is not entirely related to parameters, even though models with larger parameter quantities often have stronger capabilities.

Q7: Can PiCO (or an extension) also estimate the gap between the performance of two models that are adjacent in the final ranking?

A: Yes. As shown in Figure R2 of the rebuttal material, the learned each LLM's ability $w$ can better estimate the performance gap between different models. The results in Figure 4 also show that re-weighting by the learned weight $w$ can effectively alleviate the whole system evaluation bias, which can better reflect the ability of each model from another perspective.

We appreciate your detailed comments. We would like to address your concerns below.

Q1: Reliance on Consistency Assumption: The effectiveness of the method relies on the consistency assumption. However, a natural concern is, "Does this assumption always hold true in practice?" I suggest the authors further discuss the applicability of their method.

A: The validation of the consistency assumption can be found in Section 3.4, and the results are displayed in Table 2. We compared with four baselines: "Forward Weight Voting", "Uniform Weight Voting", "Backward Weight Voting", and "Random Weight Voting + Consistency Optimization". Here we copy the results of Table 2 as follows,

---

From Table 2, we can summarize the following two conclusions:

• Forward Weight Voting consistently outperforms Uniform and Backward Weight Voting in all cases, while Backward Weight Voting yields the worst results. This suggests that assigning larger weights to models with stronger capabilities results in better alignment, supporting the Consistency Assumption.
• Starting with randomly initialized weights, our consistency optimization algorithm further improves performance across all cases. This demonstrates that adjusting weights based on our algorithm enhances the reliability of evaluations, further validating the Consistency Assumption.

Additionally, as recommended by Reviewer #8Eov, we conducted experiments using additional ranking metrics, such as Spearman's and Kendall's rank correlation coefficients,

---

The results from these metrics are consistent with our previous findings, further confirming the validity of the Consistency Assumption in the LLM peer-review process.

Q2: Complexity of Implementation: The framework involves a complex review process, which requires substantial computational resources to support the LLMs' inference. Can you provide some details on the number of tokens consumed and a comparison of consumption with baseline methods?

A: The review process will not result in excessive token consumption, as each reviewer will only evaluate a small number of question-answer pairs (line 142-143). The comparison of tokens consumed in Chatbot Arena dataset is demonstrated as follows.

---

The proposed PiCO approach has a similar token consumed with other baselines while achieving better evaluation performance. Although Chatbot Arena has a smaller token consumption, it requires 33k human annotations, while PiCO does not require any human annotations.

Q3: Consideration of Multi-Agent Methods: Since the proposed method employs multiple LLMs, I think more recent and advanced evaluation methods based on multi-agent systems should also be considered in the experiments, such as AgentVerse.

A: To the best of our knowledge, the AgentVerse is designed to construct multi-agent systems to better solve tasks and do simulations in real applications. It seems not related to our work about LLMs' unsupervised evaluation. We appreciate your suggestion on considering multi-agent methods. In fact, the compared methods including Majority Voting, Rating Voting, PRD, and PRE are already performed under the multi-agent framework. We will add these discussions about AgentVerse in our revised paper.

Q4: Details of ELO: The ELO system is essential for the proposed mechanism. However, it is only mentioned in the appendix. I suggest the authors add more details about the background of ELO and how it is adopted in the proposed mechanism.

A: We appreciate your valuable suggestion. The ELO mechanism is employed to prevent a model from continuously defeating weaker opponents to achieve a higher ranking. We will move the details of the ELO from the appendix to the main text, and attach more discussions in our revision.

Dear Reviewer Ezxz,

We have provided detailed explanations and additional experimental results (e.g. further validation of the consistency assumptions) to address all your concerns. With just 20 hours left in the discussion period, we would appreciate it if you could review our responses and let us know if there are any final questions or concerns. Your insights have been invaluable, and we thank you sincerely for your time and effort.

Best regards!

Authors

Thank you for your very detailed comments and suggestions for improvement. And we also appreciate your recognition of our interesting idea and assumption. Next, we will answer all your questions one by one.

Q1: Unclear implication of ground truth ranking: What is the physical meaning of the ground truth ranking? What does that imply?

A: The ground-truth ranking $\mathcal{R}^*$ means that the ranking by human annotation, like Chatbot Arena. It constructs anonymous battles between different LLMs and collects real human preferences to rank their ability. The physical meaning is the ranking that aligns with human preference. The goal of this paper is to learn a ranking $\hat{\mathcal{R}}$ in an unsupervised way that aligns with $\mathcal{R}^*$ as much as possible. We will discuss this more clearly in our revision.

Q2: Evaluation metric is strange: What is the motivation of using time series comparison metrics to compare rankings? How does the proposed method work under ranking comparison metrics?

A: Permutation Entropy (PEN) is widely used to evaluate the degree of confusion in an array, while Count Inversions (CIN) and Longest Increasing Subsequence (LIS) can reflect how far (or close) the array is from being sorted. We believe that PEN, CIN, and LIS can well assess the alignment between the learned ranking $\hat{\mathcal{R}}$ with ground-truth ranking $\mathcal{R}^*$ from multiple perspectives. Additionally, we appreciate your valuable suggestion that validates the proposed approach in more standard ranking comparison metrics, i.e., Spearman's rank correlation coefficient $S(\uparrow)$, and Kendall's rank correlation coefficient $\tau(\uparrow)$. We conduct the experiments in these metrics, and the results are demonstrated in the global author rebuttal (Table R1). It can be observed that the proposed PiCO approach also achieves the best performance under the new ranking metrics in all cases. These results once again validate that consistency optimization can learn a better ranking aligned with real human preference. We will add these results in our revision.

Q3: Consistency optimization algorithm is not provided: How do you solve the problem 7?

A: We solve the problem 7 in an iteration way. Firstly, each LLM’s capabilities $w$ are randomly initialized. Then, we calculate the score $G_j$ of each model $j$. Next, we compute the gradient $\Delta w = \frac{\partial\, \text{Consistency}(G, w)}{\partial w}$ and update $w=w+\Delta w$. In the next iteration process, we use the new $w$ to calculate the LLM's score and iteratively update $w$ until convergence. We will add these discussions in our revised paper.

Q4: An optimal solution to the consistency optimization formulation can be useless: How do you prevent the useless but optimal solution? Does that imply the formulation is incorrect?

A: In the consistency optimization formulation, we employ the Pearson correlation to measure the consistency between w and G, i.e., $r = \frac{\sum_{i=1}^{n} (G_i - \overline{G})(w_i - \overline{w})}{\sqrt{\sum_{i=1}^{n} (G_i - \overline{G})^2} \sqrt{\sum_{i=1}^{n} (w_i - \overline{w})^2}}$. If $w$ and $G$ are equal to 0 for all LLMs, this formula will be meaningless. It is worth noting that the $w$ are randomly initialized to a value greater than 0 and less than 1. During the consistency optimization process, it is almost impossible for $w$ and $G$ to be optimized to be both 0. To validate it, we repeated the experiment with different seeds for 1000 times, and plotted the training loss curve and weight distribution in the additional rebuttal material (Figure R1 and R2). The results show that the learning process is stable and the learned $w$ is convergence and greater than 0.

Q5: Query distribution and LLM judgment biases: How could this formulation resist query distribution and LLM biases?

A: In our experiments, we used the most popular crowdsourced datasets, e.g., Chatbot Arena. It constructs anonymous battle platforms, where the user will ask a question $Q_i$ with two Chatbots simultaneously and rate their responses based on personal preferences. We removed those preference annotations and only used these human questions in our experiments. In other words, the query distribution in our experiments is completely the same with the widely-used crowdsourced platforms. Most importantly, we believe that a good evaluation should also consider user query distribution. Usually, we say Model A is better than Model B in terms of user experience, a possible reason is that Model A performs better in most common user queries, rather than uncommon problems. Therefore, we believe that the widely accepted evaluation method is to consider user queries, such as the most popular LLMs evaluation platforms (Chatbot Arena).

Q6: Choices of LLMs for evaluation: What is the principle of the meticulous selection? How does the proposed approach work if strong models such as GPT-4 and Claude are included in the reviewer LLMs?

A: The LLMs selection is based on the Chatbot Arena dataset [1], which includes 33K responses by 20 open-source and close-source LLMs. Among them, the APIs of GPT-4, Claude-v1, Claude-instant-v1, and PaLM-2 are too expensive, so we removed them from the reviewer queue. The official open-source FastChat code [2] has some issues in implementing the RWKV model, so we also remove it from the reviewer queue. Thus the remaining 15 models are used in our experiments.

Q7: Comparison with a simple baseline: How does the proposed method compare with the simple baseline?

A: We compare the Cluade-3 in our experiments, and the comparison results are also demonstrated in the global author rebuttal (Table R1). Compared with Cluade-3, the PiCO can consistently achieve better performance in all cases.

---

References

[1] https://huggingface.co/datasets/lmsys/chatbot_arena_conversations

[2] https://github.com/lm-sys/FastChat

Dear Reviewer 8Eov,

We have provided detailed explanations and additional experimental results (e.g. performance comparison in more ranking metrics and comparison with Claude (Table R1 of global author rebuttal)) to address all your concerns. With just 20 hours left in the discussion period, we would appreciate it if you could review our responses and let us know if there are any final questions or concerns. Your insights have been invaluable, and we thank you sincerely for your time and effort.

Best regards!

Authors

Thank you for your recognition of our interesting observations, good presentation, and great visualization. And now we will explain and discuss the questions you raised.

Q1: How do you define unlabeled questions in line76, as in to what extend is a question unlabeled?

A: The "unlabeled questions" indicate the questions without model responses and human preference annotations. For example, for a question" ($Q_i$): who is the founder of Apple? The response ($A_i^j$) of model $j$: Steve Jobs. The response ($A_i^k$) of model $k$: Bill Gates. Human Annotation ($A_i^j>A_i^k$): model $j$ is more correct than model $k$. Thus the "unlabeled question" only indicate the $Q_i$.

Q2: How do you differentiate high-level from low-level LLMs? Are they interchangeable or more of a dynamic definition while evaluating on different tasks?

A: The ability levels of different LLMs are learned by our consistency optimization, which aims to maximize the consistency of each LLM’s capabilities and scores. It is thus a dynamic process when evaluating different tasks.

Q3: Limitation on bypassing evaluation systems, results in potential fairness and security problems.

A: We appreciate your valuable suggestion. LLMs may indeed train to bypass the evaluation system and deceive reviewers to achieve higher rankings. We will discuss this limitation in our revised paper.