Benchmarking LLMs' Judgments with No Gold Standard

Q1. This is an interesting question. You are correct that in result one (Section 4.1), there are k human reviews for each project proposal in the instructor-annotated peer grading dataset, and we get the average GEM score for each candidate. We choose human-generated references over LLM-generated ones because the incentive mechanism relies on the assumption of conditional independence among reviews. This means each review should be independent of others, given a specific project proposal being reviewed.

However, this conditional independence assumption may not hold for LLM-generated reviews, especially those from the same LLM family. These reviews often exhibit similarities in their generation process, leading to potential correlations. Using correlated LLM-generated reviews as references can bias the GEM scores, potentially leading to inflated evaluations, especially for models from the same family.

In conclusion, while our method does not require gold-standard references in the traditional sense, it necessitates human-generated information to establish a conditional independent baseline.

However, it's important to acknowledge that if human reviewers were to solely rely on AI-generated content, the quality and diversity of human-generated information could diminish, potentially impacting the effectiveness of our approach, as we discussed in Section 6. This also echoes recent research demonstrating that the absence of original human-generated data in AI evaluation can result in model collapse.

Q2. We hypothesize LMExaminer is sensitive to the presentation quality even if it is prompted to evaluate the semantic quality following Review Quality Indicators (RQIs) including four aspects, understanding, coverage, substantiation, and constructiveness. (More details are provided in Section 4 and Appendix E.4.) This hypothesis can be supported by the evidence that LMExaminer significantly increased scores after the meaningless elongation and GPT rephrase.

In Sentence Deletion degradation, since we delete every other sentence, the logical flow can be fragmented. And even if we complete the deleted sentences, the presentation quality may still be worse than the original.

Thank you for pointing out this and we will add this discussion in the revised version.

Q3. We can expect similar results with metrics that are accurate, such as LMExaminer which also shows a positive correlation with human annotation, while other metrics, such as BERTScore, BLEU, and ROUGE, may not have similar results.

However, a good benchmark should be manipulation resistant, otherwise, the LLM developers may gain an unfair advantage in the benchmark, for example, as the LMExaminer is not robust against meaningless elongation, an unfairly high score may be achieved by training or finetuning models to output unnecessary long responses if we use LMExaminer in GRE-bench. Moreover, LMExaminer needs to take information about the task as input, which can be extremely expensive for peer review, considering the length of the papers.

Still, we believe that using peer review as an evaluation task can lead to a promising field, thanks to the continuous influx of rich data that can circumvent data contamination, and the comprehensive ability required to write a peer review. More metrics that are suitable for evaluating judgment quality should be further explored and studied.

Again, we thank you for your thorough questions, especially in the discussion and analysis of our results. We believe these revisions have substantially improved the paper's rigor and soundness. If any aspects of our responses require further clarification or if additional questions arise, we welcome further discussion to ensure the highest quality of our paper.

W1. Thank you for raising this important concern. First, we acknowledge that peer-reviewing scientific papers is a challenging task that encompasses multiple levels of abilities. Our approach deliberately leverages this hierarchical structure to create a meaningful evaluation framework.

The review process requires a hierarchy of capabilities, ranging from lower-level abilities like information retrieval and summarization, to mid-level abilities such as domain-specific knowledge application and methodology assessment, and ultimately to higher-level abilities including critical analysis, hypothesis evaluation, and scientific merit judgment. We acknowledge the previous negative results in using LLMs to fully automate peer review, however, recent work [Liu and Shah, 2023; Shah, 2024] has shown that LLMs can achieve competitive-to-human-reviewer and sometimes even better effectiveness when testing their mid-level capabilities to detect technical mistakes in peer review. For example, in an experiment by Shah [2024], GPT-4 finds about half of deliberately inserted errors in papers, in contrast, only 1 out of 79 human researchers find one error.

Notably, rather than claiming LLMs can fully replace human reviewers, we are using the peer review task to evaluate LLM capabilities at multiple levels. This setup allows us to measure performance across different cognitive hierarchies, where weaker models can be differentiated by their performance on basic tasks like summarization, while stronger models can be evaluated on more complex abilities such as critical analysis. This creates a natural progression for measuring LLM advancement. We leverage these known limitations to create meaningful evaluation metrics that can help us better understand the current state of AI technology.

To make this evaluation more precise, we adjust synopsis granularity to target specific abilities and use GEM-S metrics to evaluate different aspects separately. For example, by conditioning on the abstract, GEM-S (abstract) evaluates the LLM's ability to generate judgments beyond summarizing general topics and keywords, including retrieving author-stated strengths and weaknesses (ASSW) and providing critical feedback. By further conditioning on a summary of these author-stated strengths and weaknesses, GEM-S (ASSW) narrows the focus to the LLM's capacity for critical thinking and delivering constructive feedback.

Through this hierarchical information structure, we can better understand the progression of LLM capabilities in this hierarchy while maintaining a realistic perspective on their current limitations. This understanding is crucial for both the development of AI technology and its appropriate application in academic contexts.

We will add this discussion about hierarchical information structure to our paper in later revisions.

Reference

Shah, N. B. The role of author identities in peer review. Plos one, 18(6), e0286206. 2023.

Liu R, Shah N B. Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622, 2023.

W1. The advantage of GEM is encouraging participants to provide informative responses, even when the quality of peer responses may vary. By rewarding participants based on the mutual information between their response and their peer's response I(X;Y), the data processing inequality guarantees that the less informative version of X pays less than X, regardless of Y’s quality (see proposition 3.1). However, it's important to acknowledge the limitations of Data Processing Inequality. Firstly, it requires Y to be at least somewhat informative. Secondly, the noisy versions of X and Y must be independent conditional on the original X.

Two scenarios can compromise the mechanism:

1. Independent Y: If Y is independent of everything, including X, noisy X and X become indistinguishable, leading to suboptimal outcomes.

2. Violated Conditional Independence: When both Y and noisy X are low-quality AI-generated content, this assumption may not hold, potentially rewarding less informative responses unfairly. This also echoes recent research demonstrating that the absence of original human-generated data in AI evaluation can result in model collapse.

Our results indicate that the human references in ICLR2023 are good enough for the GRE bench based on the GEM metric to effectively distinguish different LLMs. To mitigate these issues in the future, using a diverse range of responses can be useful.

W2. Per your suggestions, we conduct experiments with various preprocessing models and evaluation LMs. Our findings remain consistent across various preprocessing models and evaluation language models. For more details, please see our general reply above.

Q1. Our GEM metric differs from the ELO rating system in the following ways.

1. As the name, Generative Estimator for Mutual Information, suggests, the GEM metric has the same interpretation of mutual information, i.e. how much information knowing one variable can inform us about the other (in bits).
2. The GEM score is static, i.e., we don’t need to match for pair-wise competition, which saves computation power, especially when the number of LLMs goes up.
3. The ELO rating system still needs an examiner to pick the winner. Compared to human examiners, our method saves the money of hiring humans, while LM examiners are not robust against manipulations, as our paper and some previous work suggest.

This inspiring question indicates that there could be an interesting future direction to apply our GEM metric to the ELO rating systems, such as the Chatbot arena, as an examiner. This may create arenas in more scenarios that evaluate subjective judgments, such as peer review, book review, and other qualitative content generation tasks.

Thank you again for your insightful questions and suggestions, which help us further improve our completeness and indicate interesting future directions. We hope our responses have addressed your questions and provided you with a better understanding of our research. Please feel free to let us know if you need any further clarification.

W1. We appreciate your suggestions on the diversity of evaluation tasks.

Currently, we focus on peer review evaluations for several reasons: first, previous probability-based metrics (BARTScore, GPTScore) have been shown to be effective in traditional NLG evaluation tasks like summarization and dialogue. Our GEM can be regarded as a probability-based evaluation metric like previous metrics, so we can expect similar results in these tasks.

Second, tasks like summarization and dialogue have clear human references as gold standards. However, peer review is inherently subjective, with diverse and sometimes conflicting judgments among human reviewers. This makes it a more complex and nuanced evaluation scenario. As our title suggests, we aim to benchmark LLMs’ ability to generate informative judgment without gold standard reference, peer review is an ideal setting.

In addition, our GEM metric is the base of the GRE-bench (Generating Review Evaluation Benchmark), so it’s important to validate GEM’s accuracy and manipulation resistance on peer review tasks, which also implies the effectiveness of the GRE-bench.

In the long term, we are excited about the potential of GEM to evaluate the quality of book reviews, movie reviews, and other subjective content, provided suitable public datasets become available.

Per your suggestion, we have found an open-access dataset of Yelp restaurant reviews. We are working on an experiment validating the GEM metric in the evaluation of Yelp reviews. Specifically, testing whether the degradation will significantly decrease the GEM scores of Yelp reviews. While we aim to include these results in the camera-ready version, we are not sure whether we can finish that by the end of the rebuttal.

As you mentioned in the strength, we have conducted three experiments on both student proposal review and ICLR review, which provide a variety of expertise and task difficulty, we believe that the current positive results effectively show GEM’s accuracy and manipulation resistance being superior to the baselines including the popular GPT4o examiner. Still, we value your suggestion on task diversity which will further strengthen our method’s soundness.

W2. We have conducted experiments with different preprocessing models and evaluation LMs, generally, the results remain consistent across models. Please refer to the general response above for the details.

W3. Line 471, we aim to refer to Figure 4 instead of Figure 5, sorry for the typo, we will fix it in the revised version.

Q1. Thank you for your insightful question. We have conducted experiments to check whether Pr_LLM[x|y]*Pr_LLM[y] = Pr_LLM[y|x]*Pr_LLM[x], and the results revealed a high positive correlation between the two values, although they were not precisely equal. Despite this difference, the empirical results in our paper indicate that this does not negatively impact the desired properties, including the accuracy and manipulation resistance.

Specifically, we randomly select 500 papers from ICLR 2023 and 1874 corresponding reviews in total. For each pair of reviews x,y for the same paper, we use our method to compute log Pr_LLM[x|y], log Pr_LLM[y|x], log Pr_LLM[x], and log Pr_LLM[y]. For those 5328 pairs, we compute the Spearman correlation between Pr_LLM[x|y]*Pr_LLM[y] = exp(log Pr_LLM[x|y] + log Pr_LLM[y]) and Pr_LLM[y|x]*Pr_LLM[x] = exp(log Pr_LLM[y|x] + log Pr_LLM[x]) is 0.943, which indicates that they are highly positively correlated, despite minor differences. Potential noise in the estimation process may account for these differences.

We will include a discussion of this question in our revised version.

Thank you again for your thoughtful feedback, which has helped us further strengthen the completeness of our research. We hope our responses have clarified your concerns and provided a more comprehensive understanding and evaluation of our paper. If any points remain unclear or if you have additional questions, please don't hesitate to let us know.

A common question raised by reviewer 8Skk and CxHL is about the consistency of our GEM metric over various preprocessing models and evaluation LMs. We appreciate this question and regard it as a great opportunity to improve the soundness of our methods. Here are the research questions:

1. Does using an open-sourced preprocessing model (rather than GPT4o) still lead to effective GEM metric and similar GRE-bench results?

To answer this question, we use a Llama3.2-90b model for preprocessing and re-run all experiments mentioned in our paper. We find that most experimental results remain valid except for two manipulation tests. The GRE-bench results also remain highly consistent with the results of using GPT4o preprocessing.

Specifically, GEM shows a significant positive correlation with human annotation in the proposal peer grading dataset (Section 4.1), however, the Spearman’s correlation coefficient between GEM scores and instructor grades drops from 0.433 (GPT4o preprocessing) to 0.274 (Llama3.2-90b preprocessing). In the degradation experiment, GEM with Llama3.2-90b preprocessing also shows significant sensitivity to all degradations and even performs better in “sentence deletion” and “deletion and completion” than using GPT4o to preprocess. In the manipulation experiments, it is robust against “GPT4 rephrase” but fails in the “Llama-3.1 rephrase” and “Meaningless Elongation”.

Based on the GEM metric with llama3.2-90b preprocessing, we re-compute the GRE-bench scores. The Spearman's correlation coefficient between the GRE-bench scores based on the GEM metric with Llama3.2-90B preprocessing and the original scores shown in our paper with GPT4o preprocessing is 0.89. The analogous coefficient for the GRE-bench scores based on GEM-S is 0.85. We will provide all detailed results in the revised version.

Overall, since better models can more accurately preprocess the reports, the GEM metric will be more accurate and manipulation-resistant with superior preprocessing. This also echoes the question from reviewer CxHL “LLMs are released and updated frequently, and I wonder how we can deal with such frequent updates to make the GEM score consistent and reliable over time.” Our answer would be that preprocessing with better LLMs can lead to better accuracy and reliability, and our GEM metrics will benefit from the ongoing advancements in LLM technology.

2. Does using a different evaluation LM still lead to consistent results?

We use Llama-3.1 8B and Llama-3.1 70B as the evaluation LM to estimate the pointwise mutual information, and the results are consistent over these two models.

The Spearman's correlation coefficient between the GRE-bench scores based on the GEM metric computed using Llama-3.1 8B and that same score computed with Llama-3.1 70B is 0.93. The analogous coefficient for the GRE-bench scores based on GEM-S is 0.92. We will further provide all scores in the revised version.

This consistency shows that with our metrics, a small model with 8B parameters that can be deployed on a single NVIDIA L4 GPU, plus proper preprocessing, is still effective in evaluating the quality of judgments generated by much larger models, though larger evaluation LM may lead to better evaluation performance.

Moreover, as reviewer CxHL noted, “LLMs are released and updated frequently”, this observation supports the consistency and reliability of GEM score over different models. We will include these results in our paper, and we do appreciate the valuable suggestions from our reviews.

We acknowledge that there are thousands of potential models we could test for preprocessing and evaluation. While it's impossible to exhaustively test every available large language model, we believe our current experimental results demonstrate the solid effectiveness of the GEM metric. These additional experiments, suggested by our reviewers, have further strengthened the rigor of our methodology.