Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions: Benchmarking Large Language Models

Q6 The reliability of these metrics is somewhat limited as they only assess text similarity between the reference reviews and model-generated reviews.

A6 Thank you for pointing out the limitations of relying solely on text similarity metrics for evaluating model-generated reviews. We recognize that metrics like BLEU, ROUGE, and BERTScore, while widely used, have inherent shortcomings as they primarily measure overlap between the generated and reference text. These metrics can undervalue responses that are diverse, creative, or insightful but deviate from the reference in wording or structure.

Thus, we have adopted a diverse set of evaluation metrics to provide a more comprehensive and balanced assessment. Beyond text similarity metrics, we include validity checks to ensure that the generated outputs align with the task requirements (e.g., whether reviewers provide a score or decision-makers explicitly make an accept/reject decision). Most importantly, we incorporate human evaluations, where expert reviewers assess the quality of the model outputs based on factors such as relevance, clarity, depth, and consistency.

Q7 As described in Section 3, the multi-turn dialogues for paper review consist of only 4 turns.

A7 We appreciate your observation regarding the number of dialogue turns in our framework. While we currently model the peer review process with four key turns—initial review, rebuttal, final review, and decision-making—this setup is designed as an initial step toward capturing the complexity of real-world peer review interactions. Even with this simplified structure, the multi-turn dialogue format marks an advancement over more traditional single-turn review simulations, allowing for a more dynamic and interactive process.

However, we agree that in practice, the peer review process often involves more extensive back-and-forth communication, which can span multiple rounds of discussion, revisions, and clarifications. We see this as a natural extension of the task that could be explored in future work.

Q8 In Section 4.3 (line 268), it is mentioned that the average length of submissions falls within the range of 11,000 to 20,000 words, posing a significant challenge for the inference process of LLMs.

A8 Thank you for raising this important point regarding the challenges posed by the length of submissions. Managing long-context inputs is indeed a critical issue in leveraging LLMs for tasks like peer review.

Recent advancements in large language models, such as LLaMA-3, GLM-4, and Qwen2, have introduced support for context lengths up to 128K tokens, enabling these models to handle full papers and multi-turn dialogues more effectively during inference. For models with shorter context limits (e.g., 4096–8192 tokens), we adopted context truncation strategies to maximize the inclusion of relevant content. Specifically, we prioritized incorporating the most essential sections.

In cases where the paper and dialogue content still exceeded the token limit despite these adjustments, we further reduced the input by prioritizing the title and abstract, along with key dialogue excerpts, as a last resort. While this inevitably leads to some loss of context, our experiments showed that most models were still able to produce reasonable and coherent outputs under such constraints.

To ensure fairness and to account for these limitations, we introduced the hit rate (HR) metric. This metric evaluates whether a model can generate valid responses based on the provided input, even when some parts of the full context are omitted. The hit rate captures the model's ability to engage meaningfully with the content it receives and reflects its robustness in handling long-context scenarios. Models like GPT-4 and fine-tuned Qwen2 demonstrated strong performance on this metric, indicating their capability to process.

We appreciate the time and effort you put into reviewing our work. Your feedback has helped us identify key areas for improvement, including evaluation clarity, task realism, and metric reliability. If you have additional suggestions, we would be happy to address them.

Sincerely,

The Authors

Thanks for your comprehensive answers for my questions. My concerns about some questions are addressed (Q5).

Thank you for your follow-up comment and for highlighting your concerns about the limitations of introducing human annotation in the evaluation process. We fully understand the challenges associated with human evaluations, particularly their non-reproducibility, high time, and cost, and we appreciate the opportunity to clarify how we address these issues in our work.

Human evaluation is widely regarded as the gold standard for evaluating tasks that require nuanced understanding, such as peer review. This is especially true for tasks where automated metrics, like BLEU or ROUGE, fall short of capturing qualitative aspects like insightfulness, accuracy, and persuasiveness. Many existing studies [1-4] rely heavily on human evaluation to validate the effectiveness of LLMs in complex multi-turn dialogues, reasoning, and open-ended generation tasks. In our work, we followed this established practice, introducing human evaluation to supplement the limitations of automated metrics. The primary goal was not to replace automated evaluation but to provide a qualitative baseline for assessing the outputs of LLMs in the peer-review task. This allowed us to reveal critical findings, such as: "GPT-4o is the top-performing model," and "Qwen2 and GLM-4 are the most promising open-source models,". By presenting human evaluation results, we provided a benchmark reference for researchers and practitioners to better understand the current capabilities of LLMs in this domain.

The automated metrics we provided remain the primary tools for scalable evaluation, and future directions include integrating LLM-based evaluators and developing task-specific metrics to improve automated assessment. We appreciate your concern about the balance between human and automated evaluation, and we believe the measures outlined above will address this issue effectively while enabling further improvements in subsequent iterations of this work.

Thank you for your insightful feedback!

Sincerely,

The Authors

[1] Bai, Ge, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin et al. "Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues." arXiv preprint arXiv:2402.14762 (2024).

[2] Dubois, Yann, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S. Liang, and Tatsunori B. Hashimoto. "Alpacafarm: A simulation framework for methods that learn from human feedback." Advances in Neural Information Processing Systems 36 (2024).

[3] Lu, Pan, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. "Learn to explain: Multimodal reasoning via thought chains for science question answering." Advances in Neural Information Processing Systems 35 (2022): 2507-2521.

[4] Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." Advances in Neural Information Processing Systems 36 (2023): 46595-46623.

Q4 It's unclear how these models handle long text during training and generation.

A4 Thank you for pointing out the need to clarify how long texts are handled during training and inference. Addressing long-context inputs is indeed a critical challenge, especially given the token limitations of most current LLMs.

During training, we implemented a chunking strategy to process long documents. Specifically, each paper was segmented into smaller, overlapping chunks that fit within the model's token limit. The overlapping regions between chunks ensured that the model retained continuity and contextual understanding across segments, which is particularly important for tasks like peer review that require long-context comprehension. This strategy is a common and effective solution when training on large documents with models constrained by fixed context windows.

During inference, we tailored the handling of long texts based on the specific capabilities of each model. Recent LLMs, such as LLaMA-3, GLM-4, and Qwen2, support extended context windows up to 128K tokens, allowing these models to process long documents in their entirety without truncation. For other models with shorter token limits (e.g., 4096–8192 tokens), we prioritized including key sections of the paper—such as the title, abstract, introduction, and conclusion. In cases where the text still exceeded the model’s token limit after these adjustments, we truncated the input further as a last resort, retaining only the title and abstract.

This is why we introduced the hit rate metric to evaluate whether LLMs can produce valid responses given the input provided. The hit rate measures the model's ability to engage with the content.

Q5 Presentation inconsistencies exist, such as ChatGLM3 being described as "with a default selection of 6B to 9B parameter" in line 362 but "with 130 billion parameters" in line 1022. Model specifications and access points aren't clearly stated. For instance, YUAN2 only has 2B, 51B, and 102B accessible versions, with no 6B-9B versions, affecting reproducibility.

A5 Thank you for pointing out the inconsistencies in model descriptions, particularly regarding ChatGLM3 and Yuan2, as well as the lack of clarity around model specifications and access points. We apologize for the confusion caused by these discrepancies and will revise the manuscript to ensure clarity and consistency.

In our experiments, we consistently employed models within the 6B to 9B parameter range, except in cases where such versions were unavailable. Specifically:

• For ChatGLM3, we used the 6B parameter version in our experiments. The reference to a "130B parameter" version in line 1022 of the appendix was intended as a general introduction based on the model’s official documentation and does not reflect the version used in our experiments. This misleading description will be removed to avoid confusion.
• For Yuan2, we used the 2B parameter version, as it does not currently offer a model in the 6B to 9B range. We will make this clear in the revised manuscript.

These clarifications will ensure that all reported results are traceable and reproducible. Thank you again for bringing this to our attention!

Q6 The authors mention recruiting five expert reviewers to evaluate language model outputs but don't specify who they are, how they were recruited, their evaluation process, or their expertise. Given the test set covers various ML topics, it's uncertain whether these five expert reviewers are qualified to evaluate such diverse research directions.

A6 Thank you for raising this concern regarding the qualifications and evaluation process of the human reviewers. We acknowledge that providing more details about the reviewers and their expertise is critical for ensuring transparency and credibility. Below, we provide a more detailed response and clarify these aspects.

The five reviewers we recruited are highly experienced researchers who have served as peer reviewers for top-tier AI conferences (e.g., NeurIPS, ICLR, CVPR, and ACL) for at least three years. Their expertise spans a broad range of machine learning and artificial intelligence domains, including but not limited to: natural language processing, computer vision, graph neural networks, AI for scientific applications. These areas cover a wide spectrum of topics present in our test set, and we ensured that each reviewer was assigned outputs related to topics they were familiar with, ensuring a fair and informed evaluation process.

Due to the double-blind policy followed in peer-reviewed research, we are unable to disclose the identities of the reviewers at this stage. However, we commit to providing the full list of reviewers (with their consent) and their affiliations once the paper is accepted for publication.

Q9 Neurips, ICLR is too little scope I feel, while the authors address this weakness in their discussion, this needs to be addressed.

A9 Thank you for your thoughtful feedback. We completely understand your concern regarding the limited scope of our dataset being focused solely on NeurIPS and ICLR.

Previously, we explored the possibility of using Nature Communications, which provides open peer reviews and spans a much broader range of disciplines beyond AI. It would indeed have been an excellent resource for creating a more diverse and comprehensive dataset. However, due to the journal's CC BY-ND (No Derivatives) license, we were unable to utilize its content for constructing a reproducible dataset. This was an unfortunate limitation, as it would have allowed us to expand the dataset's applicability beyond AI and machine learning.

As of now, the most openly available, high-quality peer-review data comes from OpenReview, specifically from NeurIPS and ICLR. These conferences are renowned for their rigorous review processes and provide a rich, freely accessible source of peer-review data. While this is a strong starting point, we acknowledge that these venues represent a narrow scope, focused primarily on AI and ML. Expanding the dataset to include peer reviews from other fields and venues remains an important direction for future work.

We hope this clarifies the rationale behind our current dataset scope and highlights our commitment to addressing this limitation in the future. Thank you for bringing attention to this critical aspect.

Q10 Is rouge a good metric? Often models are quite verbose (which might hurt the rouge scores).

A10 We agree with the reviewer that ROUGE, as a metric, may not fully capture the quality of a model’s responses, especially when verbosity or phrasing differences are present. Verbose yet insightful responses might receive lower ROUGE scores despite being high-quality, as the metric focuses primarily on n-gram overlap.

We do not rely solely on ROUGE. Instead, we employ a diverse set of evaluation metrics, including BLEU, METEOR, BERTScore, and—most importantly—human evaluation, which plays a critical role in assessing the contextual relevance, depth, and coherence of the responses. Human evaluators are particularly effective in identifying high-quality outputs that may deviate from strict lexical similarity with the reference texts. We believe that this combination of automated metrics and human evaluation provides a comprehensive and balanced framework to assess model performance, ensuring that verbosity or stylistic differences do not unfairly penalize insightful responses.

We greatly appreciate your constructive feedback, which has helped us identify areas for improvement and new directions for our work. All your suggestions will be incorporated in the revised manuscript to ensure greater clarity, transparency, and depth. We are confident that these updates will significantly enhance the quality and utility of the paper.

Thank you again for your valuable insights and for recognizing the potential of this work. If you have further suggestions, we would be happy to address them.

Sincerely,

The Authors