Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review

Thank you for the detailed comments.

Q1: Suggestion of put the explanation of metrics in the main text of this paper.

A1: Thank you for your suggestion. We agree that a clear explanation of metrics is crucial for understanding the paper. However, due to the page limit of the main text in ICML, we opted for a trade-off by providing qualitative descriptions in the main text and detailed explanations in Appendix B.1. We plan to provide a more detailed explanation of the metrics in the main text in a future version, which will be updated on arXiv after acceptance.

Q2: Are the results different among various conferences? For example, ICLR and NeurIPS.

A2: We have supplemented the results for ICLR 2024 and NeurIPS 2024. For each conference, 100 papers were randomly sampled for evaluation, and the default LLM GPT-4o-mini was used. The results are shown in the table below. Columns from Recall(S&W) to Jaccard(S&W) represent the analysis of strengths and weaknesses, while columns from F1(Dec.) to G-mean(Dec.) represent the analysis of decisions. "S&W" refers to strengths and weaknesses, and "Dec." stands for decisions. All metrics are better when larger.

ICLR 2024 and NeurIPS 2024 produce similar outcomes, where Agent Reviewers outperform the Single Agent baseline, achieving an average relative improvement of 23.5% on ICLR and 44.7% on NeurIPS, thereby demonstrating consistency.

Q3:Are the results different among various years? For example, ICLR 2024 and ICLR 2023.

A3: We have supplemented the results for ICLR 2023, and ICLR 2024. For each conference, 100 papers were randomly sampled for evaluation, and the default LLM GPT-4o-mini was used. The cutoff year for the shared memory pool (SMP) was set to one year prior to the test data year, to avoid data leakage from SMP. The results are shown in the table below. Columns from Recall(S&W) to Jaccard(S&W) represent the analysis of strengths and weaknesses, while columns from F1(Dec.) to G-mean(Dec.) represent the analysis of decisions. "S&W" refers to strengths and weaknesses, and "Dec." stands for decisions. All metrics are better when larger.

Whether in 2023 or 2024, Agent Reviewers have shown better results than the single-agent baseline, demonstrating the consistency of the gains of our method. It should be noted that for ICLR 2023, due to the earlier cutoff year of the SMP, the number of papers in the SMP decreased from 28,372 to 21,156 (a 25% relative decrease), which may cause the performance decline in 2023.

Thanks for your detailed and insightful comments.

Q1:What kind of LLMs are used in the baseline methods in Table 2?

A1:For the main experiment, all baselines and our Agent Reviewers use GPT-4o-mini for fair comparison (see Table 2 header). Other baseline settings like hyperparameters follow their defaults (Line 307, and detailed in Appendix B.2.). Thanks for your question; we'll clarify in the main text.

Q2:Concerns about potential data leakage of Gemini and Deepseek v3.

A2:We selected the latest ICLR 2024 data as the test set, taking measures to avoid data leakage. In main experiments, we used GPT-4o-mini (knowledge cutoff: Oct. 2023, no leakage). Gemini-exp-1206 (cutoff: Dec. 2023, no leakage) and Deepseek-V3 (cutoff: Jun. 2024, leakage not guaranteed despite efforts) were used only for cross-LLM generalization verification. The risk is explained in Appendix B.4.

We contend Deepseek-V3 has no serious evaluation-affecting leakage due to:1)Overall performance is inferior to GPT-4o-mini(no leakage).2)When asked to detail papers by title, hallucinations are severe.3)Table 12 shows Agent Reviewers outperforms Single Agent with Deepseek-V3, indicating our method value.

Q3:Background knowledge domains and coverage.

A3:We currently focus on review in AI. Background knowledge comes from LLMs' inherent knowledge and the shared memory pool(SMP), which has 283728 AI papers and reviews from ICLR 2017-2023 and NeurIPS 2016-2023 on OpenReview, spanning nearly all AI fields. Notably, our training-free method can be applied to other fields by expanding SMP.

Q4:Open-source model's performance.

A4:Thanks for your suggestion. We've added comparisons with more open-source models (Deepseek-V3-0324, Qwen2.5-VL-72B-Instruct, InternVL-2.5-78B) in the link.

Overall, GPT-4o-mini performs best, Deepseek-V3 close behind. Largest open-source model approaches or surpasses some proprietary ones. Little difference exists between 2 Deepseek-V3 versions.

Q5:Diversity concern & Proposal for different LLMs in different role experiments.

A5:We agree on the importance of diversity in review, tried your suggestion with GPT-4o-mini, Gemini-exp-1206, and Deepseek-V3-0324 as LLMs for 3 domain-specific reviewers. Results at link.

We've seen no better results with a multi-LLM system. We analyze that our system has already enhanced review perspective diversity by endowing domain-specific reviewers with different domain knowledge, so multi-LLMs have limited room for improvement.

Q6:Concerns about low-quality review impact in training data.

A6:Our method is training-free. We initialized the SMP with ICLR 17-23 and NeurIPS 16-23 papers, and retrieved paper info(summaries, processed reviews) as domain-specific background knowledge for reviewers. We agree low-quality reviews can affect performance and mitigate it as follows: 1) Processed reviews in SMP, retaining AC decisions and LLM-aggregated  pros&cons to reduce individual low-quality review impact.2) Retrieved 5 domain-related papers for each reviewers as memory to ensure sufficient high-quality reviews.

Q7:Why focus on title+abstr.+intro.& Plan to extend to full paper content.

A7:We use only title+abstr.+intro. in our current method for two reasons: 1) The full text being about 12 times longer, using it in a multi-agent system with interactions incurs great overhead.2)They're crucial in peer review, even for humans. We agree full text can provide richer info and potentially better performance, but challenges like long context need to be overcome. To extend efficiently, we suggest adding an in-depth reading feature for reviewers. After reading first three parts, they can raise questions and retrieve relevant full-text parts for further analysis before generating comments.

Q8:Insights into why system identifies paper weaknesses less effectively.

A8:Thanks for noting this profound observation. We observed it in all tested methods. We posit it's because reviewers' proposed strengths(str.) maybe similar, e.g., similar to claimed contributions, while weaknesses(wk.) vary greatly, making it harder for LLM-generated weaknesses to match humans'.

We studied the similarity of Agent Reviewers' str. and wk. generated by GPT-4o-mini, Gemini, and Deepseek-V3 for the same article. On average, GPT-4o-mini vs. Gemini: str. 0.664/wk. 0.484; GPT-4o-mini vs. Deepseek-V3: str. 0.738/wk. 0.663. Figure 6 shows most matching str. in "Novelty" & "Research Implications", wk. dispersed.

Q9:Suggestion about real-time online search and memory pool expanding.

A9:Thanks for your comments! We agree and will try these two approaches in future work to offer reviewers more comprehensive knowledge.

Q10:Plans for code and data availability.

A10:We'll open-source all code and data (include memory pool) upon acceptance for reproducibility and are preparing for it.

Thank you for the detailed comments.

Q1: How is the NeurIPS dataset obtained, given that only the reviews for accepted papers are released?

A1: All our data(papers and reviews) stems from the public data on OpenReview, including the NeurIPS dataset. As you noted, 96% of the NeurIPS papers (2016 - 2024) released on OpenReview were accepted. We describe this statistic in Appendix A. Due to the severe imbalance in the NeurIPS data, we do not use it as test data. Instead, we use ICLR 2024 for evaluation.

We use NeurIPS 2016 - 2023 and ICLR 2017 - 2023 data to initialize the shared memory pool, providing domain-specific reviewers with sufficient background knowledge for review. Although nearly all NeurIPS dataset papers are accepted, the ICLR dataset has an acceptance rate of about 37%. Consequently, papers in the shared memory pool are relatively evenly balanced in acceptance and rejection.