PaperBench: Evaluating AI’s Ability to Replicate AI Research

Thank you for your insightful feedback! We address your comments below:

1. Concerns About Longevity/Usefulness of Benchmark

Thank you for raising this important point! We are happy to announce that we now have author approval on 20 rubrics and so now the dataset and results are on a 20 paper dataset. We agree that there are contamination risks both as more potentially helpful resources become available online to agents that can browse the web (that aren’t included in the blacklists) and as papers and code repos get included in pre-training data. Building better monitors can help detect occurrences of the former, but won’t be that helpful for pre-training contamination. We hope that in future the PaperBench dataset can be expanded to include new papers that can help the benchmark stay relatively uncontaminated.

2 + 4a + 4b. Human vs. LLM Time Accounting ("Overnight computer runs")

We agree with your critique. We think there are a few factors that mean directly comparing human vs AI agent time can be confusing: for example, AI agent time can be influenced by API inference speed which is variable. Since the version of the paper that was initially reviewed, we have significantly expanded our human baseline experiments in our new manuscript. The updated version shows considerably improved human replication scores (now up to 41.4% from 22.0% previously) via extending the baselines from 1 week to 4 weeks which we believe is a more realistic assumption for a human attempting the task. These new human baselines demonstrate that the PaperBench replications, while challenging, are indeed attainable for skilled humans and the plots suggest that scores would continue to improve if humans were given more time. We discuss this in a revised Section 5.4. In these results (on a 3 paper subset) we find that our tested model, o1, plateaus after around the first hour which justifies our decision to impose a timeout of 12 hours. On these new human baseline results the human line continues to go up and doesn’t show signs of plateauing.

3. We have now included a table in the appendix which stratifies scores based on requirement type. We observe that models perform poorly on Code Execution and Result Analysis requirement types, while scoring better at Code Development nodes.

4c. Why can't LLMs just upload results to their repo?

Thank you for raising this important criticism! In light of this we have changed the way we grade Result Analysis nodes. Now all Result Analysis nodes are judged only by looking at any files created or modified in the reproduction step (as well as docs, reproduce.sh, and reproduce.log). By forcing the judge to only look at newly modified files this better guards against the judge being fooled by results that are included in the preliminary submission (but this could be hacked around by an adversarial AI agent that wants to fool the judge).

5a. Why are scores on some papers higher than others?

Some paper scores are higher than others due to the inherent differences in difficulty between replicating different papers. By having papers of different difficulty in the dataset, we prevent the benchmark from saturating quickly and get a better estimate of agent capabilities since we’re tracking the agent’s performance on a greater variety of tasks. The main benchmark metric is average Replication Score which reports the mean Replication Score across each paper.

Thank you for taking the time to carefully review our paper! The rebuttal is subject to a character limit; we address what we saw as the highest-priority comments below:

Code is not open sourced, or available to reviewers for review.

We will open source our codebase for the camera-ready release. The codebase can be found under the supplementary materials, which has been available during the review process.

No novelty, as using LLM as a judge has been done before

JudgeEval is a necessary auxiliary evaluation in order for us to justify using an LLM-based judge for the grading of paper replication attempts. We’d like to stress that we’re not claiming the use of a judge is a novel idea.

This claim should be tightened, as it is too broad…

We now use updated wording in the Abstract and Introduction which clearly states the intended scope of PaperBench: we claim that it is “a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research”.

…the evaluation aspects of the benchmark and its construction can all be better motivated and improved.

2a. Why use a tree rubric structure, and how weights are determined:

The tree structure allows us to measure partial progress towards a result in a paper (allowing us to determine whether agents are scoring highly on particular sub-trees) and hierarchically decompose the grading task into smaller (and therefore easier) tasks for the Judge.

Rubric weights are carefully set by each rubric creator and are author-reviewed to reflect the relative importance of each sub-task.

2b. Judge LLM Agent only reads top-k code files, where k=10:

Our judge achieved an F1 score of 0.83 on JudgeEval using $k=10$. Preliminary experiments showed larger values of $k$ yielded similar performance, while smaller values lowered the F1 score. Future work exploring alternative judge designs (e.g., longer-context or vision-capable models) would be valuable.

2c. Clarifying Number of Seeds and Runs ("How many random seeds?") In Section 5.2 of our new manuscript, we clarify that we performed evaluations with three independent runs for each evaluated agent to robustly estimate performance and error bars.

However the authors should cite this paper...

Thanks for pointing us towards Bersenev et al.’s work; we will reference it in related work.

Significance: …if the rubrics and any addendums can be self generated for new paper…

We’ve reached the character limit for this rebuttal, so please see our detailed response to Reviewer b2MF’s comment: “Not scalable: each paper requires…”

Why is the human replication rate so low? If so low, how can we expect LLM Agents to solve it?

We’ve since improved the human baseline experiment, extending the best performers after week one to continue their attempt for a total of four weeks, a more realistic timeframe for humans. This led to an increase in the average replication score from 22.0% to 41.4%. In our updated Figure 3, we see a steady increase in the human baseline score over time, which we expect to continue given more time.

Assumption of un-restricted compute, seems difficult for others to use as a benchmark.

We agree that running PaperBench with our setup is too resource intensive for many users. We therefore introduce a variant in our updated manuscript, PaperBench Code-Dev, where models are only evaluated on Code Development nodes. Users therefore don’t need a GPU or an expensive VM, which cuts costs substantially.

Can any consistency be imposed across the rubrics?

We had multiple processes in place to ensure rubrics are created in a consistent manner. We adhered to team-wide guidelines for rubric creation. We also had a core team of reviewers who reviewed every rubric in the dataset, ensuring consistency. Additionally, the author of each paper reviewed the relevant rubric for correctness, often resulting in many iterations of refinement.

Can you run a sanity check where you use your judge on the actual repo produced…

In JudgeEval, we manually graded original author repositories and found replication scores below 100%. As explained in the paper, this is expected since original codebases often have bugs, are incomplete, or lack the required reproduce.sh scripts.

No guarantee or test that the addendum is fully correct…

Each addendum was co-developed with and reviewed by the original authors of each paper – often going through multiple rounds of iterations of refinement – which ensured they were high-quality and accurate.

Explain relative performance amongst LLMs…

Thanks for the suggestion! We’ve updated the paper to briefly analyze why models perform differently from one another. Most models frequently finished early or faced a problem they couldn’t solve. All agents failed to verbalize how best to use the limited amount of time available to them.

—

Thank you for your valuable feedback! Please consider raising your rating if you feel that the paper has improved.

Thank you for your constructive feedback! We address your comments below:

Line 47 mentions the UK AI Safety Institute’s Basic Agent scaffolding without an explicit citation, which should be included to clarify the source and structure of the agent scaffolding.

Thank you for catching this! We’ve added a citation for UK AISI’s Basic Agent scaffolding and have explicitly referenced it throughout the paper.

Very limited annotations and data: a) only 18 papers, and b) only 8 submissions from PhD students to establish human baseline performance.This restricts the benchmark's representativeness and reliability.

Thank you for the feedback! We’ve updated PaperBench to include 20 papers (up from 18), all of which have been finalized with author approvals. The rubric now has 8,316 individually gradable tasks (up from 6,125). We’d like to mention that we think the number of individually gradable tasks gives a more accurate picture of how diverse the tasks in PaperBench are, rather than the number of papers.

We’ve also improved the human baseline experiment, extending the best performers after week one to continue their attempt for a total of four weeks, since we think this is a more reasonable timeframe for humans to replicate a paper. Considering the turnaround time and cost of running such a human baseline experiment, it was infeasible to conduct it with a larger sample size.

Not scalable: each paper requires a manually created rubric and associated weights, raising concerns about scalability due to the intensive manual effort involved.

We agree that each rubric involves a significant amount of effort to create. We found that current models aren’t capable of end-to-end rubric creation (even after extensive prompt iteration) and, even when rubrics were generated end-to-end, they took a significant amount of time to review. Models were, however, useful in assisting humans in writing rubrics. We have released our rubric creation web application in the supplementary materials, which uses models to assist humans in the rubric creation process; we used this tool throughout our rubric-creation process. We’ve also added Appendix A.1 which outlines future directions of automated rubric creation, which may be possible in the near future.

Very expensive for subsequent researchers to utilize the benchmark: grading a single submission costs 150USD and evaluating 18 submissions would cost 2,700 for a single model or method.This high cost is prohibitive for others aiming to benchmark their models, thus limiting the benchmark's broader impact

Thank you for raising this point! We’ve significantly reduced the cost of grading to approximately 66 USD per paper (down from 150 USD) by using a cheaper model (o3-mini-high) for the Judge. Importantly, using o3-mini-high achieves comparable scores to the previous model on JudgeEval. In general, we expect the cost of grading to fall sharply as cheaper and more capable models become available in the future, meaning this barrier to entry will be short-lived.

We also release a lighter-weight variant, PaperBench Code-Dev (which supersedes PaperBench Lite and is even more accessible and affordable), reducing the grading cost further from 66 USD per paper to approximately 10 USD per paper when using our optimized o3-mini based judge. We describe this new lightweight variant in detail in Section 2.6, but in short: it’s a lighter-weight variant of PaperBench designed for situations where you only want to see if the agent can write plausibly-correct code, without the overhead of running it. Grading costs are significantly reduced, since the judge grades fewer nodes, and doesn’t need to place the reproduce.log file in its context (which can be lengthy).

We’ve also created an experimental grading feature whereby we grade rubrics which are “pruned” to a given depth, potentially reducing the cost of grading by 10x or more. We’ve added a new section, Appendix H, which covers this approach in more detail, including experimental results on JudgeEval.

Lack of experiments involving prominent open-source models such as DeepSeek-R1 or LLaMA, restricting the generalizability of the findings.

Thank you for the suggestion! We’ve updated the paper to include a leading open-source model, DeepSeek-R1. We find that DeepSeek-R1 performs better than some closed-source frontier models (e.g., GPT-4o) but worse than others (e.g., Claude Sonnet 3.5).

You mention that 13 rubrics are currently still under review by paper authors, making your presented results an intermediate evaluation. Could you clarify how significantly you expect the final results to differ from the current intermediate evaluation?

We've now completed all author reviews of rubrics, have re-run our experiments and found that our results are broadly consistent with the preliminary results presented originally.

—

Thank you for your valuable feedback! Please consider raising your review score if you feel that the paper has improved.