Predicting Empirical AI Research Outcomes with Language Models

Thanks for appreciating the strength of our paper! We will address each of your questions below, and are willing to expand or provide further responses if necessary.

Concern about data contamination. Lines 72 and 93 suggest that the authors use conference publication dates as the publication dates of papers.

We will clarify this confusion in revision. “Publication date” here refers to the date that the paper first appeared on arxiv (which we extract from its arxiv ID), not the date that it’s published at the conference.

the knowledge cutoff date of GPT-4.1 is June 2024, though the exact date is not specified. To ensure a more reliable exclusion of training data, I believe it would be safer to include only papers published after July 2024.

We pick a subset of our original test set that includes 1,444 ideas published after July 2024. As shown in the Table below, our main findings remain the same: GPT-4.1 achieves 77.3% accuracy (compared to 77.0% on the 1,585 ideas after June).

Concern about the shortcut in the proposed method: simply selecting newer methods, which often outperform older ones. Although I consider the experiments in Section 6 essential for addressing this concern, the results are not compared with baseline methods or expert human performance.

We agree that simply selecting newer ideas is a very likely shortcut. We are happy to include the following result in our revision. Here we report the best possible performance achievable by simply selecting newer ideas. Our fine-tuned GPT-4.1 substantially outperforms this baseline (77.3% v.s. 53.0%), suggesting that it’s not overly reliant on idea newness in its predictions.

The paper lacks detailed statistics, example prompts, samples from the datasets, and raw outputs from the evaluated methods.

Currently, dataset statistics are presented in Table 2, and an excerpt of dataset example in Figure 2. We will include more detailed dataset examples and example prompts (e.g. prompts for query generation, summarization in Section 3) in the final revision. The raw outputs of our fine-tuned models only include “Idea A or B is better”, while the prompted models also generate CoTs before making final predictions; we are happy to include CoT examples in the appendix although they are not used in our method.

Missing references

We will add the following papers about peer-review style research evaluation in our final revision. As discussed in the second paragraph of related work, these papers are based on fully written papers, so still requiring implementing ideas. Moreover, these papers focus on critiques of idea novelty, excitement, and paper presentation, which are often subjective and independent of the ideas actual empirical effectiveness. In particular, a recent paper [5] shows that the ideas with higher LM-judged novelty scores in fact lead to worse empirical outcomes.

[1] The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv 2024.

[2] The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. arXiv 2025

[3] GRAPHEVAL: A Lightweight Graph-based LLM Framework For Idea Evaluation. ICLR 2025

[4] Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents. ICLR2025

[5] The Ideation–Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas. arXiv 2025.

I recommend pairing ideas with a larger gap.

We’d like to clarify that by “gap” the reviewer is referring to the gap in win rates of two ideas on the set of benchmarks, for example, by only labeling one idea as the winner if it beats the competitor on 80% of the benchmarks.

We use different gaps to filter our test set, and report the data size and the performance of our fine-tuned GPT-4.1 in the table below. As expected, increasing the gap leads to higher accuracy, probably because it reduces label noise or picks easier idea pairs.

We agree that the gap in determining labels is an important hyperparameter. In our data collection and training pipeline, that hyperparameter can be easily adjusted. We will clarify in our revision that we don’t impose any choice of this hyperparameter.

could you clarify why you maintain the claim that "GPT-4.1’s cut-off date is June 1st, 2024" (Line 94)

We thought it was June 1st. We will update the main results with July 1st, 2024 as the cut-off date!

While I agree that the threshold is a tunable hyperparameter, I do not consider that the current default of 50% is a reasonable choice.

We will add the results with different thresholds in the paper and emphasize that future work should tune this parameter based on their specific scenarios.

Thanks for appreciating the strength of our paper! We will address each of your questions below, and are willing to expand or provide further responses if necessary.

In Section 3, how does the system determine whether sufficient information has been collected?

We prompt LMs to automatically determine this.

System prompt:

You are an expert researcher in comparing and predicting other proposed research ideas.
You will be given access to a paper retrieval API, which you may use to survey the literature and find relevant papers to help you make your decision.           

You will be given 5 rounds to decide on which idea would work better, but you do not need to use them all.
At any round, you may exit early and decide on the comparison result.

Excerpt of Prompt

First, determine if you need to search for relevant research papers to make an informed assessment 

In Section 3 Step 1, what is a query?

It’s the query sent to the paper retrieval engine. Specifically, we prompt LMs as follows:

First, determine if you need to search for relevant research papers to make an informed assessment (e.g., check if there are similar ideas or supporting empirical evidence). If yes, provide a search query in this format: 
<search>
your query (a natural language sentence, e.g., effectiveness of weight decay in noisy label learning, effectiveness of diffusion models in image style transfer compared to vision transformer, effectiveness of model editing in sequential editing setting)
</search>
Guideline for search queries: 
- At least one of the ideas is entirely new. This means that you won't find exact comparisons in the literature. Instead, search for ideas that could provide indirect or transferable insights, or you can break the new idea into smaller sub-ideas to identify related work for each sub-component.
- Make queries as precise and context-specific as possible to maximize the relevance of the retrieved information.
- Avoid redundancy:
   - Do not repeat previously made queries.
   - Avoid gathering excessive evidence about components you already understand well. Instead, prioritize components that need further investigation.

Here are two examples of generated queries:

• effectiveness of state space models v.s. GRU architectures in sequence modeling tasks
• effectiveness of code-based prompting v.s. Chain of thought prompting for logical reasoning tasks

In l. 130, why is at least one idea completely novel?

Because each test example contains at least one idea published after June 1st, 2024, the knowledge cut-off date of our base model GPT 4.1 (Line 93-95).

What is the role of paper summarization in Section 3?

Adding the whole content of all retrieved relevant papers would 1) exceed the model context limits, and 2) introduce too much irrelevant, non-informative information that might negatively mislead LMs’ predictions (e.g. experiment setup, hyperparameter details, irrelevant results).

Why are the four steps necessary in Section 3 and what is their main purpose?

As explained in Line 122-126, similar to how human researchers draw inspiration from existing literature, we empower LMs with this agentic retrieval engine to improve its predicting ability.

The four steps in Section 3.1 are standard practice for building retrieval-augmented systems [1] [2] [3]. In particular, our paper summarization step is quite different: while prior work directly reuses paper abstracts as summaries, we download each paper’s PDF and prompt LMs to summarize the whole paper with respect to the current query. As shown in Table 3, this substantially boosts the accuracy from 38.8% to 53.0%.

[1] The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv 2024.

[2] The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. arXiv 2025

[3] Can Language Models Generate Novel Research Ideas. ICLR 2025

The limitation section is extremely brief. One could mention for example, that evaluating the outcome of new ideas could drastically increase human reviewing burdens, among many other.

We are happy to include more discussion on the social impact of our work and AI-automated research in general (e.g., the potential increase in human reviewing burdens, as the reviewer suggested) in camera ready with the extra space. Our current discussion focuses on the technical limitations of our approach.

Thanks for appreciating the strength of our paper! We will address each of your questions below. Happy to expand or provide further responses if necessary.

"We only consider the predictions valid when both predictions are consistent; otherwise, inconsistent predictions are considered incorrect by default." --> Could the authors indicate how many times this happens?

The table below reports the frequency of inconsistent predictions (i.e. inconsistency score). Our choice to consider inconsistent predictions as incorrect is in line with prior work ([1][2]) in pair-wise evaluation .

[1] AlpacaEval : An Automatic Evaluator for Instruction-following Language Models. github

[2] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv 2024

The authors state in footnote 1 that Claude 3.7 is not evaluated, but mention Claude 3.7 in the introduction. I believe this should be Claude 3.5?

Yes, it should be Claude 3.5. Thanks for pointing out this typo!

While it is a first attempt at such a dataset, the limitation to binary choice is quite limiting to real-life use cases.

We actually think pair-wise comparison is well aligned with real-life use cases: 1) human researchers often seek to know whether a new idea is better than specific baselines, 2) in an automated research pipeline, a pair-wise reward model can be used to rerank model-generated ideas or directly optimize the idea generation model.

Thanks for appreciating the strength of our paper! We will address each of your questions below, and are willing to expand or provide further responses if necessary.

What are the retrieved papers used for in the final proposed system? Section 3.1 only mentions how to retrieve papers from the database, but it does not mention how retrieval results are used. Are these results simply used as parts of inputs, or are some specific modules proposed for processing this information?

Yes, the retrieved results are simply used as parts of inputs during prompting and fine-tuning.

Will the proposed benchmark be released upon the acceptance of the paper?

Yes, we will release the benchmark.