Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

Thank you for your insightful review! To address your concerns:

• “it would still be valuable to compare LLM-generated ideas with those from published papers”

To address your comment on evaluating the inter-reviewer agreement level with published papers, we compute the inter-reviewer agreement using the same consistency metric used in the paper for papers accepted by ICLR 2024, which only gets 52.8% consistency. This means that reviewers tend to have even lower agreement for their ranking on the accepted papers (since they might have similar quality in general).

This further validates our point that reviewing is an inherently difficult task and it is expected that the inter-reviewer agreement would be relatively low. As we mentioned in the response to Reviewer HdAN, typical AI conferences such as NeurIPS and ICLR also have low inter-reviewer agreement (66.0% for NeurIPS’21). Moreover, our setting of evaluating ideas without the experiments is expected to involve even higher subjectivity than typical conference reviewing. That said, we believe we are getting meaningful signals from the review results since we are recruiting highly qualified experts for all the review tasks.

• “The limitations observed, such as the lower feasibility of LLM-generated ideas, could potentially be addressed by a more sophisticated agent framework that includes stages for refining and prototyping ideas.”

As we explained in the paper, our focus is on benchmarking the capabilities of current LLMs in comparison to a human expert baseline, rather than on building the strongest agent scaffolding for the idea generation task. We leave it to future work to explore many possible ways to improve upon our current agent design.

• “it would be beneficial if the authors offered best practices or recommendations for using LLMs effectively to generate research ideas”

Thank you for your suggestion! We will include a list of things that we tried and found effective, for example, we discovered that appending previously generated ideas and instructing the model to avoid duplicates can reduce repetition in idea generation; and that reranking is an effective strategy to find higher-quality ideas among all the generations.

Thank you for your insightful review! To address your concerns:

• "Not strong enough motivation for restricting to prompting based research."

We address this in the first point of our general response.

• "The paper does not talk about the usefulness of the generated ideas research ideas."

We made our best attempt to evaluate the usefulness of these ideas by asking expert reviewers to score the ideas on feasibility and expected effectiveness as part of their evaluation. Notably, our effectiveness metric directly asks reviewers whether they think the proposed method would outperform existing baselines, which captures this usefulness aspect of the ideas.

• "Can you please provide some small scale (self-evaluated) tests ablating these parameters to ensure their optimality?"

For ablating the impact of the maximum number of papers to retrieve (N), we vary N among {30, 60, 120, 180} and measure the average relevance score of the top 20 retrieved papers (as judged by the LLM). We show results on the following two topics, where we report the average relevance score of the top 20 retrieved papers (on a scale of 1 - 10), as judged by an LLM. In general, N = 120 as used in our paper achieves good performance across different topics, and the relevance of the top retrieved papers tends to plateau after N = 120.

For ablating the impact of the number of papers to add to the prompt for RAG, we vary k between {0, 5, 10, 20} and measure how it impacts the diversity of the generated ideas on the topic of uncertainty prompting.

Overall, we can see that this hyper-parameter k has minimal impact on the diversity of the generated ideas.

• "A better way to ensure the robustness of results is to look at the distribution of the expertise of the reviewers."

When surveying participants' familiarity (lines 1146 - 1156 in Appendix), we defined familiarity as whether they have authored or read papers on their selected topic, which in a way captures their expertise. This, combined with the qualifications of these participants (as shown in Table 2), serves as evidence that our reviewers in general have strong expertise on their reviewed topics.

• "Interestingness: Can you please clarify the meaning of the term?"

When reviewing ideas, we used the review form as shown in A.7 (lines 1134 - 1270). “Interestingness” in this context (line 115) refers to the novelty and excitement aspects defined there.

Thank you for your insightful review! To address your concerns:

• "The author needs to test more prompts to sufficiently reach such a conclusion for this experiment."

To examine whether the diversity issue exists across models and prompts, we further measure the idea diversity of multiple other base models and prompt setups. We use a temperature of 1.0 for all the generations and measure the percentage of non-duplicate ideas out of 2K generated ideas on the topic of uncertainty prompting.

In the table below, we compare four different base models: Claude-3.5-Sonnet, GPT-4o, o1-mini, and Llama-3.1-405B-Instruct.

We find that different models have very different non-duplicate rates, and overall o1-mini and Claude-3.5-sonnet have the lowest diversity. However, we note that we picked Claude-3.5-sonnet as the base model of our agent because the quality of its generated ideas outperforms other models based on our pilot study, where we randomly sampled 10 ideas from Claude-3.5-sonnet and GPT-4o for a round of pilot expert scoring and Claude-3.5-sonnet scored an average of 5.4 while GPT-4o scored 4.8.

Next, we try several different prompts for idea generation with the Claude-3.5-sonnet backbone: 1) no RAG and not appending previously generated ideas for deduplication; 2) using RAG but not appending previously generated ideas; 3) using RAG with different numbers of retrieved papers included in the prompt (k=5,10,20).

We find that appending previously generated ideas in the prompt and asking the model to avoid repetition can significantly reduce idea duplication. However, including retrieved papers in the prompt has minimal impact on the diversity. We thus conclude that Claude-3.5-sonnet is generally bad at idea diversity across various prompt setups.

• "The paper only verifies the generation quality of the author's simple implementation of an agent, lacking evaluation against other baselines."

As we explained in the paper, our focus is on benchmarking the capabilities of current LLMs in comparison to a human expert baseline, rather than building the strongest agent scaffolding for the idea generation task. We leave it to future work to explore many possible ways to improve upon our current agent design.

• "The research is limited to the NLP field and lacks studies in other domains."

We justify our choice to focus on prompting-based NLP research in the point of the general response. We believe the evaluation framework that we established would be helpful for future works that extend the evaluation to other domains.

• "Observing that some researchers in Table 3 and Table 4 have a familiarity level of 1 with the domain (which could be considered as low-quality annotation), what are the experimental results in Table 5 after removing this data?"

Note that out of all the reviews that we have collected, only two indicated a familiarity score of 1. Removing these two reviews does not impact any conclusions in the paper. Moreover, only one of the idea-writing participants indicated a familiarity of 1 on their topic. Removing their idea in the statistical tests gives the same conclusions.

We show the results of removing the one participant with a familiarity of 1 and the two reviews with a familiarity of 1 below:

Overall, all three main conclusions still hold robustly even if we remove all the ideas and reviews with a familiarity of 1.

You are an expert researcher in AI. Now I want you to help me brainstorm some new research project ideas on the topic of: {topic description}. 

Here are some relevant papers on this topic just for your background knowledge:
{retrieved papers} 

The above papers are only for inspiration and you should not cite them and just make some incremental modifications. Instead, you should make sure your ideas are novel and distinct from the prior literature. You should aim for projects that can potentially win best paper awards at top AI conferences like ACL and NeurIPS. Each idea should be described as: (1) Problem: State the problem statement, which should be closely related to the topic description and something that large language models cannot solve well yet. (2) Existing Methods: Mention some existing benchmarks and baseline methods if there are any. (3) Motivation: Explain the inspiration of the proposed method and why it would work well. (4) Proposed Method: Propose your new method and describe it in detail. The proposed method should be maximally different from all existing work and baselines, and be more advanced and effective than the baselines. You should be as creative as possible in proposing new methods, we love unhinged ideas that sound crazy. This should be the most detailed section of the proposal. (5) Experiment Plan: Specify the experiment steps, baselines, and evaluation metrics.

You can follow these examples to get a sense of how the ideas should be formatted (but don't borrow the ideas themselves):
{demo examples} 

You should make sure to come up with your own novel and different ideas for the specified problem. You should try to tackle important problems that are well recognized in the field and considered challenging for current models. For example, think of novel solutions for problems with existing benchmarks and baselines. In rare cases, you can propose to tackle a new problem, but you will have to justify why it is important and how to set up proper evaluation.
Please write down your 5 ideas (each idea should be described as one paragraph. Output the ideas in json format as a dictionary, where you should generate a short idea name (e.g., "Non-Linear Story Understanding", or "Multi-Agent Negotiation") as the key and the actual idea description as the value (following the above format). Do not repeat idea names or contents.

Thank you for your insightful review! To address your concerns:

• “Although it is relevant and intriguing for the ML community, the paper leans more towards evaluating scientific idea generation, aligning more with applications rather than with core ML research.”

We believe our work is within the scope of ICLR, since “applications” is explicitly mentioned in the Call For Papers page (https://iclr.cc/Conferences/2025/CallForPapers).

In fact, there are at least 10 submissions at this ICLR conference that focus on research idea generation (e.g., “Chain of Ideas: Revolutionizing Research in Idea Development with LLM Agents”, “Review and Rebuttal: Zero-shot In-context Adversarial Learning for Improving Research Ideation”, “Two Heads Are Better Than One: A Multi-Agent System Has the Potential to Improve Scientific Idea Generation”, “CycleResearcher: Improving Automated Research via Automated Review”, “GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation”, etc.).

Despite being evaluation-centric, we believe our work is very relevant to the ICLR community and would be an important contribution to the emerging body of work on LLM for scientific research.

• “The study is limited by its exclusive focus on prompting-based NLP research, which restricts the generalizability of its findings to other fields or other ML directions.”

We address this in the first point of our general response.

• “research ideation remains inherently subjective, as reflected in the study's inter-reviewer agreement of 56.1%”

Subjectivity is the inherent nature of peer-reviewing. As we showed in the paper, typical AI conferences such as NeurIPS and ICLR also have low inter-reviewer agreement (66.0% for NeurIPS’21). Moreover, our setting of evaluating ideas without the experiments is expected to involve even higher subjectivity than typical conference reviewing. That said, we believe we are getting meaningful signals from the review results since we are recruiting highly qualified experts for all the review tasks.

• “The authors used reviews from Openreview, are there any licensing issues?”

According to the OpenReview terms of use (https://openreview.net/legal/terms), all submitters agree that their submissions/comments shall be released to the public under the Creative Commons Attribution 4.0 International (CC BY 4.0) license, which allows free sharing and adaptation.