RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts

We thank the reviewer for the thorough review and insightful comments! We are glad that the reviewer finds the work to be well-written, the evidence to be clear and convincing, and that the results provide insight into the capabilities of current frontier AI on realistic research engineering.

We address the reviewer’s concerns in “Methods and Evaluation Criteria” below:

1. We believe that RE-Bench represents a substantial improvement in the realism and complexity of AI-relevant research engineering evaluations compared to the existing literature (see Table 7 for a detailed comparison)! In designing this benchmark, we decided that 1-day long tasks would be a reasonable compromise between providing valuable new insights and the cost and complexity of running evaluations and getting high-quality human baseline data. However, we recognize that this is a limitation of our work and discuss how time-horizons may affect our results in Section 5. We are excited for future work to build on RE-Bench and develop better ways of measuring longer-task performance!
2. We believe that RE-Bench can measure partial AI performance quite well! We carefully designed our tasks and score functions such that progress is intermediately measurable and it is possible to make easy and quick progress. For example, in the “Optimize a Kernel” task, a very easy improvement is to use built-in Pytorch functions to optimize the slow starting solution. In “Optimize LLM Foundry”, an easy optimization is removing unneeded steps like checkpointing. We thus think that AIs that can reach close to an expert solution for a task are also likely to find easier or partial solutions that make some measurable progress. We would love to see future work investigate how human-AI collaborations might score on tasks like these, and hope that RE-bench can facilitate this.
3. We aimed to be principled in our choice of scaffolds, with Modular representing a simple proven baseline, and AIDE representing a more complex open-source scaffold that achieved top scores on the closest existing benchmark (MLE-bench). We certainly agree that scaffolding is an unsolved problem in AI evaluations and we discuss some of these limitations in Section 5.3. We are excited for future work on better understanding their effects and we hope that RE-Bench can help serve as a valuable testbed for other researchers!
4. While we agree that these results would be interesting, we note that the newer models were released less than 4 months before the ICML submission deadline, which falls under “concurrent work” according to the ICML reviewer guidelines. Therefore, we think that these evaluations would be out-of-scope for our work. However, we certainly are excited for future work that evaluates these, and newer, models on RE-Bench!

Here’s our responses to the reviewer's questions. Note, due to the character limit, we’ve had to be brief in our response.

1. Based on experimentation and qualitative judgements, we expect models substantially older than Claude 3.5 Sonnet (Old), which is the oldest model evaluated in our work, to perform very poorly in RE-Bench.
2. The average token cost for agent runs is \$123, while we paid human experts \\$1855 on average; GPU costs vary but H100s generally cost around \$2/hr (and tasks use 0-6 GPUs). However, we want to note that these environments could be run on less expensive GPUs (like A100s), but the collected run data would not be comparable. Close to no tweaking of the environments, and often none to scaffolds as well, are needed to run newer models.
3. RE-bench has been designed to measure capabilities research primarily.
4. We believe our professional network experts (scoring 0.98 on average) are very strong, with over 5 years of experience in an area highly relevant to the task they are baselining or recent experience at frontier ML research organizations. More information can be found in Section 2.1 or Appendix A!
5. We suspect this result is due to noise.
6. Qualitatively, Claude 3.5 Sonnet (new), for example, seems to be better at tool-use (invoking the right tools correctly, and interpreting their results) than o1-preview, which affects how well models work with different scaffolds.
7. Figure 2 plots time-budget on the x-axis, i.e. increasing amounts of the total time available to agents, and each point is the average performance when using our best-observed way of allocating that time for that AI agent, which often involves doing BoK over many short runs. Whereas in Figure 4, we are plotting wall-clock time on the x-axis for a single run. Scores increase more slowly in this setting, because models often get stuck with bad solutions or incorrect assumptions over time. Frequent resetting seems to increase solution diversity and helps them continue to make progress.

On the suggested improvements for the writing and the typos, we’ll address these in our updated draft of the paper!

We thank the reviewer for their time, attention and thoughtful feedback. We are glad that the reviewer finds the RE-Bench tasks to be nontrivial assessments of research engineering. We appreciate that the reviewer found the paper and presentation of the work to be high quality and the experimental results to be rigorous, well-designed and insightful.

Q1: My primary concern about this work is the [... question omitted due to character limit ...] I have some concern about claiming that the environments are for evaluating the capabilities of agents for open-ended ML "research" engineering.

Thank you for bringing this up! We offer several points in response to this concern:

• The 7 tasks vary significantly in how easily and quickly the ground-truth score can be checked, even though they all allow an unlimited number of invocations of the score command. In environments like ”Restricted Architecture MLM” and “Fix Embedding”, good solutions require long training runs before they can be meaningfully scored (sometimes taking many hours), so in practice agents/humans only have one or a few attempts to train and score a final solution. While in the “Scaling Law Experiment” environment, the agent can never see the ground-truth score of its solutions. In both cases agents can check the score performance of smaller models or shorter training runs frequently, but then have to account for generalization.
• For some tasks in real AI R&D work, the ground-truth score function is actually quite easy to have access to, for example, the runtime is very accessible when trying to optimize code bottlenecks for large training runs. Our task “Optimize a Kernel”, where the goal of the task is to make a GPU kernel run as fast as possible, is a realistic example where it’s very natural to have quick access to the actual score. Therefore, we think it is important to include such environments in our benchmark.
• The reason we give ground-truth feedback in so many of the tasks is to avoid reasonable misunderstandings about how the score will be measured, what kind of generalization is intended, or what kinds of solutions are allowed, which we have previously found can happen very easily even for human experts and can make task results uninformative.
• Agents being fast-iterators is indeed a real phenomenon. We discuss in Section 3.4 that on average, agents run the score function between 25.3-36.8 times per hour, compared to 3.4 times for humans. This seems to help significantly on tasks like “Optimize a Kernel” where rapid iteration against ground truth is possible, but it is not clear that it provides any real advantages on tasks like “Fix Embedding” where proper score assessment requires long training runs and exercising judgment.
• Lastly, we acknowledge that this is indeed a consideration, and we offer a discussion of this in our Section 5.3.

Q2: The use of the time budget can be one of the choices for the "x-axis" for the evaluations and comparisons. Have you considered other choices, such as the number of score evaluations?

Thank you for this question! Here are some points we hope addresses the reviewer’s question:

• Indeed, we include a comparison between humans and agents with non-GPU cost (i.e. pay for humans and token costs for AIs) on the x-axis as Figure 8.
• Due to practical constraints, human baselines had to be conducted with a time-limit of 8 hours (see Section 2.1). We offer additional discussion about how this affects our results in Section 5.2.
• We also wish to highlight that RE-Bench is agnostic to what limit is used for agents or humans. This flexibility means that RE-Bench can easily be used to explore future work on different comparative settings.
• Additionally, agents and humans were asked to optimize performance relative to the time they used, so using time as the x-axis is more representative of their best performances. To properly investigate how score relates to number of score evaluations would require rerunning human and agent results with instructions to minimize unnecessary score invocations.
• Lastly, while not an x-axis, we also include discussion about the number of score evaluations in Section 3.4.

We are pleased to hear that the reviewer found the paper is well-written and clear to read, RE-Bench is novel and original, and the contributions significant. We address the questions the reviewer had below and we’d be happy to provide any further clarifications.

Q1: How were “cheating” or environment-breaking agent solutions handled in scoring?

Any runs that we inspected that had evidence of cheating were excluded from the results entirely. For agent solutions, we carefully inspected the 2 best performing runs for all tasks, and we inspected many more of the top performing runs on “Restricted Architecture MLM” and “Optimize LLM Foundry”. These two tasks often had cheating because agents would often forget or not take into account the restrictions. When running these experiments and developing the environments, we have not seen cheating or environment-breaking agent solutions for any other tasks.

For human runs, we manually inspected score logs and submissions for any signs of cheating or environment-breaking solutions.

Q2: Did any participants use external LLMs while solving, and how did that affect results?

Participants were allowed to use the internet, external LLMs and other tools in order to solve the task. We wanted to compare AI agents against humans with access to their preferred development environment and tools, as that would be more representative of how frontier research is conducted and provide the strongest baseline. To our knowledge, we expect most participants to have used LLM-based coding tools like Cursor, or web-based LLM interfaces. We would be excited to see future work explore how human performance on research tasks varies with access to different types of AI assistance.

Additional discussion around both questions can be found in Appendix A for more details!