ResearchTown: Simulator of Human Research Community

Thank you very much for your insightful and constructive comments. We address each of your comments in detail.

[Novelty+feasibility evaluation with LLM and human] Please check the same tag under Reviewer YZ7h fo novelty and feasibility evaluation.

[Fine-grained evaluation with LLM and human] Please check the same tag under Reviewer YZ7h for fine-grained consistency evaluation.

[Cost and scalability for ResearchTown] Please check the same tag under Reviewer Cny3 for analyzing the computational cost.

[Future Application of ResearchTown] Please check the same tag under Reviewer Cny3 for analyzing social dynamics simulation.

[Baseline score of realistic simulation] To check whether ResearchTown provides realistic simulation, we benchmark similarity in real-world research activity. For paper writing, we reference two concurrent papers [1,2] recognized for presenting nearly identical ideas—yet with different writing styles and experiments—which yield a VoyageAI similarity of 0.8244. This suggests that scores above 0.82 can potentially indicate strong idea overlap. For review writing, we analyze data of reviewers evaluating the same paper. The average inter-reviewer similarity is 0.5900 (strengths) and 0.5904 (weaknesses), reflecting natural variance in human judgment. These inter-similarity score in the real world confirm that ResearchTown’s similarity scores represent realistic simulation.

[Ablation study on LLMs and graphs] For graph structure ablation, Table 1-2 already demonstrates the effect of different types of neighboring nodes during aggregation with different sub-parts of the neighborhood. It can be considered as ablation on graph structures.

Additionally, we provide results on LLM ability variation using Deepseek-V3 (potentially larger than GPT-4o-mini) and Qwen2.5-7B-Instruct (potentially smaller than GPT-4o-mini) on a sample of 100 examples each from PaperBench and ReviewBench.

For paper writing tasks, we find when given the same aggregation setting, the performance improves when the models are larger: Qwen2.5-7B-Instruct < GPT-4o-mini < Deepseek-V3 (evaluated by openai embedding).

For review writing tasks, we test the performance of different models under different aggregation settings. We observe that Deepseek-V3 benefits more from multi-agent + multi-paper settings; GPT-4o-mini tends to be stricter than human reviewers, especially when more context is available.

[More baseline comparison] The TextGNN framework in our work is a general-purpose multi-agent simulation tool where existing LLM-based frameworks can be used to define the message-passing mechanism. Beyond our default setup, we conducted a small-scale experiment extending the AGG-agent setting [Line297–298] into a multi-turn conversation using the SWARM framework. This mimics multiple iterations within a single GNN layer and improves similarity scores from 52.32 to 57.68, evaluated using text-embedding-large-3. We also compare ResearchTown with the Sakana AI scientist framework, which uses five rounds of single-agent reflection. On the same subset of paper writing tasks, Sakana AI Scientist achieves a score of 0.63, while ResearchTown reaches 0.66 using multi-agent simulation.

[Running example] Table14-28 (Line1430-2254) includes multiple running examples from ResearchTown: (1) Table14–17: Paper writing examples; (2) Table18: Review writing examples; (3) Table19–28: Interdisciplinary research examples. We will add more end-to-end running examples, beginning from citations and ending with paper + review outputs, in the revised version.

[Downstream task validation of ResearchTown] We would include more downstream tasks like citation prediction in the modified version of our paper. Based on our HighImpactPaperBench (details in Appendix C.3, Line837-853), we observe that highly cited novel papers like GAN and VAE are harder to simulate the thinking process based on citation and multi-agent, indicating some kind of correlation between citation impact and simulation tasks.

[1] Chen et al. ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning

[2] Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

Thank you very much for your insightful and constructive comments. We ddress each of your comments in detail.

[Fine-grained evaluation with LLM and human] Please check the same tag under Reviewer YZ7h for more human and LLM eval.

[Novelty+feasibility evaluation with LLM and human] Please check the same tag under Reviewer YZ7h for more human and LLM eval.

[More baseline comparison] Please check the same tag under Reviewer 5ZWh for baseline comparison.

[Data leakage concern] For our main results (Table 1–2) and ablation studies (Figures 3–5), as noted in Appendix C.2 (Line 813–815), we use NeurIPS 2024 and ICLR 2024 papers, which are post-dated beyond GPT-4o-mini's October 2023 knowledge cutoff. Thus, data leakage is not a concern. We also mask the full text during the simulation to avoid accidental exposure.

For HighImpactPaperBench (Appendix C.3, Line 840–843), we use high-impact papers from the past decade as an extreme-case test for idea simulation. While some may exist in the LLM’s training data, this benchmark is separate from our main results and serves to explore how LLMs handle well-known concepts.

Our similarity analysis shows that 55% of generated papers score between 0.65–0.75, and 18% exceed 0.75, indicating moderate to high alignment. Only 1% score below 0.45. These scores are comparable to PaperBench (Table 1), suggesting no abnormal inflation. Even famous papers like VAE, GAN, LayerNorm do not receive notably high scores, implying that semantic similarity—not memorization based on citation relationships—drives the results, especially for tool/benchmark papers, which naturally resemble their references more.

[Review writing performance analysis] For the behavior behind Global-Agg of review writing results, we conduct fine-grained analysis on the difference between predicted score $S$ and real-world score $S^*$ on a subset of review writing tasks. The results show that GPT-4o-mini consistently assigns lower scores than the real-world reviewers, especially in the Global-agg setting, where the mean of (S - S*) is -1.47. In contrast, Deepseek-v3 does not show this consistent bias, indicating a better performance on review writing.

Qualitatively, Global-agg reviews with GPT-4o-mini tend to provide more specific and critical assessments, particularly highlighting weaknesses related to novelty and experiment that are often missed in Self-agg settings, resulting in lower scores.

Example 1: Global-agg provides a more detailed description of novelty concern and make the score lower.

• Global-agg: "The proposed method does not present a sufficiently innovative approach compared to existing frameworks. Many cited works, such as P2B and PTTR, already address similar challenges, and the submission fails to articulate clearly how it advances the state of the art.” → Score: 4
• Self-agg: “The novelty of the proposed methods is not clearly articulated. The paper does not convincingly demonstrate how the approach differs from existing methods or why it is a significant advancement in the field.” → Score: 5

Example 2: Global-agg notices more weaknesses compared with self-agg.

• Global-agg: “Experimental Support: The experimental results presented are not robust. For instance, the claim of maintaining output quality is not backed by sufficient statistical analysis or metrics, making it difficult to assess the validity of the findings.” --> Score: 4
• Self-agg: Not mention this weakness. --> Score: 6

[Paper Reading as inserting new node] As described in Algorithm 1 (Line 220–237), the input to the simulation is the paper content itself. To initialize the agent for simulation, we first perform a “paper reading” step, which sets up the agent profile. This is implemented in Line 6 of Algorithm 1. We interpret this as a form of agent node insertion—specifically, initializing the text attributes of an agent node based on an external paper. Thus, although it may not be an insertion in the structural sense, it serves the role of initializing a new agent node in the simulation graph.

[Math notation] Thank you for pointing this out. Our intention was to provide a formal definition analogous to message-passing GNNs, such as in Equation 3 (Line 145–148). However, due to the heterogeneous nature of our agent–data graph, we define different aggregation functions depending on the node types. We acknowledge that this may introduce complexity in notation. We will simplify and clarify the mathematical presentation in the revised version of the paper to improve readability.

Thank you very much for your insightful and constructive comments. We address each of your comments in detail.

[Novelty+feasibility evaluation with LLM and human] Please check the same tag under Reviewer YZ7h for novelty and feasibility evaluation conducted by both LLMs and humans.

[Fine-grained evaluation with LLM and human] Please check the same tag under Reviewer YZ7h for factual consistency and other fine-grained metrics.

[Ablation study on LLMs and graphs] Please check the same tag under Reviewer 5ZWh for experimental results with three different LLMs.

[Review writing performance analysis] For clarification of [Line359-361], please check the same tag under Reviewer t7zy for more analysis about why the performance drop.

[Input and output of ResearchTown] ResearchTown processes either full papers or abstracts, depending on the task, and outputs standardized formats to support consistent and scalable evaluation. For paper writing, it generates a condensed 5Q format [Line893–894]; for review writing, it produces bullet-point strengths and weaknesses [Line910]. Prompt templates are shown in Table12–13 [Line1379–1425], with examples in Table14–28 [Line1432–2253]. As described in Appendix C.3 [Line837–853], this alignment reduces evaluation complexity and enables sub-part similarity scoring [Line898–902]. Input sources vary: only the paper’s abstract is used during reading [Line994–995] and full papers are used for review writing [Line1171–1172]. Aggregation setting details are in [Line688–726].

[Future Application of ResearchTown] In [Line139–144], any research-related content—e.g., images, codebases, models, or social media posts—can be represented as nodes in the agent-data graph, with edge types like “cite paper,” “release model,” or “comment on X post” (examples in Figure 1) defining interactions. By specifying appropriate edge types and agent functions (f_u in [Line134–135]), the framework can be extended to simulate tasks such as code writing, model release, panel discussions, or lectures. While we focus on paper and review writing due to their importance, available real-world data, and simplicity, the framework supports broader applications.

Additionally, ResearchTown can be extended to model social dynamics such as peer pressure, collaborations, and institutional roles via agent-agent relationship edges [Line133–135]. Our current implementation already includes role-based dynamics (e.g., leader vs. participant), and we plan to support richer simulations of institutional and reputational factors in future work.

[Cost and scalability of ResearchTown] TextGNN’s complexity scales linearly with the number of layers under standard GNN inference with full-batch inference. Our implementation uses GraphSAGE-style [1] to support per-paper evaluation, which is slightly less efficient but more practical. In line with findings from models like GraphSAGE, we observe that a 2-layer TextGNN is both robust and sufficient, making the cost affordable and controllable. Instead of deepening the GNN layer, we increase the number of transformation steps within each message-passing layer—interpreted as more agentic conversations—making it another scalable and effective approach. To demonstrate intra-layer scalability, we extend the AGG-agent setting [Line297–298] using the SWARM framework for multi-turn agent interactions, which boosts similarity scores from 52.32 to 57.68. These results show that agentic iteration within layers offers a practical and scalable alternative.

[Details on reviewer selection] To simulate realistic reviewer assignment, we collect over 3,000 unique authors from the author list of PaperBench dataset and generate profiles by summarizing their recent publications. Using the voyage-3 API, we embed each profile with the target paper’s abstract and select the top 5 most similar researchers, excluding the original authors. This method enables high-quality reviewer-paper matching—for example, a social learning paper was matched with reviewers experienced in social science and LLMs.

[Details on review writing inputs] To clarify [Line273–277], in the review writing simulation (Algorithm 1 [Line220–236]), both the paper and review are outputs of the ResearchTown pipeline. While one option is to evaluate reviews based on generated papers, this introduces compounding errors if the paper is inaccurate. Instead, as noted in [Line273], we use the ground-truth paper as input to isolate and more reliably evaluate the review writing stage.

[Missing related work] Thank you for highlighting valuable related works—we will include and discuss them in the revised version around [Line120–125].

[1] Hamilton et al. Inductive Representation Learning on Large Graphs

Thank you very much for your insightful and constructive comments. We address each of your comments in detail.

[Input and output of ResearchTown] Please check the same tag under Reviewer Cny3 for a detailed explanation.

[Creativity of ResearchTown's output] LLMs have been shown capable of generating novel research ideas through large-scale human studies [1]. Multi-agent role-playing and discussion further enhance creativity [2] and originality [3]. Based on these studies, ResearchTown encourages exploration of interdisciplinary ideas via structured prompting and multi-agent role-play design [Line414-426]. Examples are shown on Page27–Page41. We design HighImpactPaperBench in Appendix C.3 [Line837-853] to show ResearchTown’s capacity to generate impactful research [Line355-376].

[Metric calculation in ResearchTown] Our evaluation metrics for both paper and review writing are detailed in Appendix E [Line886–930]. To enable meaningful comparison, we standardize papers into a condensed 5Q format [Line893–895] and reviews into a bullet point format [Line909–911]. As described in Appendix C.3 [Line837–853], we also convert real-world papers and reviews into these aligned formats using strong LLMs. We then compute cosine similarities with voyage-3,text-embedding-3-large and nv-embed-v2, based on Equation 21 for papers [Line898–901] and Equation 22 for reviews [Line912–915]. Detailed sub-part similarity scores are provided in Table 3 [Line935–953].

[Fine-grained evaluation with LLM and human] We extend beyond embedding-based metrics using prompting-based GPT-4o evaluations, covering factual consistency, logical/method alignment, motivation, and context. Each is scored from 1–10.

These results show clearer differences than embedding-based scores, with Global-Agg (paper nodes + agents) performing best on motivation/method alignment. A human study over 40 papers on similarity-based evaluation (20 in-domain, 20 cross-domain) yields Pearson’s r = 0.745 and Spearman’s ρ = 0.735, supporting the validity of embedding/LLM-based metrics.

[Novelty+feasibility evaluation with LLM and human] We conducted small-scale human and LLM evaluations (20 interdisciplinary and 20 ML papers) on novelty and feasibility. Interdisciplinary examples include papers tagged with multiple fields on arXiv (e.g., CS+Economics). Each cell below includes real-world data evaluation vs simulated results evaluation.

LLM vs human novelty/feasibility scores show a moderate Pearson correlation (~0.38), highlighting evaluation difficulty. ResearchTown’s outputs are generally comparable in novelty and more feasible than real papers in the ML domain.

[Text form defined in TextGNN] Each node’s "hidden state" in TextGNN represents a condensed form of a paper or review. Initially, full paper contents serve as node states [Line158-161]. After iterative message passing, paper nodes adopt the standardized 5Q format [Line893-894], condensing information for easier evaluation. Review nodes similarly use bullet points [Line910] as condensed information.

[1] Si, et al. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers.

[2] Lu, et al. LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play.

[3] Zhao, et al. Assessing and Understanding Creativity in Large Language Models.

[4] Lu, et al. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.