Benchmarking Large Language Models as AI Research Agents

Thanks for your valuable feedback and identifying the novelty in the work! We comment on the weakness below:

I have concerns about the relevance of the tasks chosen with respect to "AI Research"...

Please see our overall response in the official comments.

But I don't see anything to constrain the actions of the agent…

Yes we do not constrain the agent from importing pre-trained models if that gives better performance. And we agree in general that the agents mainly do well on older tasks like cifar10 or spaceship-titanic which the LM has seen during pretraining but lack of lot of creativity as seen in some newer and more challenging tasks in our benchmark such as CLRS and BabyLM which are actual open research problems. These older tasks were added to show the spread in the type of tasks agents are capable of doing and some promise in improving current agents for harder novel tasks. We view our work to give people a sense of the current state of agents on ML engineering tasks which involve a lot of open-ended decision making.

The actions were all the actions that an actual human could have taken to solve that particular problem, and hence are open-ended. The easier tasks were added to show the capabilities of current agents as opposed to having a suite a super hard tasks where all agents have a 0 performance (aren’t able to making any meaningful progress in, such as BabyLM). Thus, the 15 tasks in the benchmark are not static and can evolve very easy as agent capabilities increase. We wanted to have tasks which had a decent spread of success rates so that current agents can be compared as opposed to all very easy or very hard tasks only which doesn’t make a lot of sense for a benchmark. Furthermore, our novel framework ensures that adding a task to the benchmark is very seamless (very disentangled from a lot of underlying agent code) which would encourage open-source contributions, and hence would make the benchmark more wholesome and varied in the types of task to be tested. We hope our benchmark can provide valuable foundation for benchmarking more complex types of research in future work. One could simply add the research problem they are working on in our benchmark in <10 mins but it is very unlikely that the agent will show a lot of creativity to make meaningful progress in it (and hence the need for a benchmark to track this).

Thanks for your valuable feedback and identifying the novelty in the work! We answer the questions asked and comment on the weakness about the testing scenario being too simple below:

The testing scenario is simplistic and greatly differs from the demands of the real world to some extent, which limits its ability to accurately reflect the true capabilities of the model.

Please see our general response in the official comments.

Why is there no AutoGen test which should be much stronger than AutoGPT? If the output and input exceed the LLM context length during the process, how can it be resolved? Will the model being tested receive feedback from the evaluation results of the submission? Will it then continue to carry out operations to improve the results?

We thank the reviewer for pointing out several important aspects for clarification. We believe AutoGen was released after we finished most of the evaluation, and adding it will require more budget. We leave testing these new agent frameworks to our future work. When input exceeds the LLM context length, we simply declare it as terminated and only evaluate the progress done so far. We generally try resolve it by summarizing the observations that are too long with another LLM. In this work, we treat the evaluation results of the submission as a final testing step and therefore do not propagate this feedback to the LLM. The AI research agent would have had plenty of chances to receive performance feedback from the eval split of the dataset instead, and the agent is supposed to improve results base on this feedback.

Thanks for your valuable feedback and recognising the usefulness of having a benchmark to keep track of the progress of LLMs to perform open-ended machine learning tasks. We have some comments about the weaknesses highlighted about the paper:

there is relatively little information on how these tasks were selected and how the benchmark was constructed.

Thanks for the suggestion for clarifying how tasks are selected. As noted in section 2.4, “the tasks are chosen such that they range in various difficulties and recency to test the generalizability of the research agent and avoid data contamination”. To avoid the chance of data contamination, we selected Kaggle competitions that were launched only in the last 2-10 months at the time. We also narrowed the types of tasks to empirical machine learning tasks to specifically test the ability of AI research agents on these well-defined evaluable tasks. The tasks were hence chosen so that current agents can show some spread in performance on the benchmark.

It tests LLM ability to complete research in empirical machine learning tasks, but does not test their ability to do other types of research.

Please see the overall response in the official comments. Indeed, our benchmark focuses on the problem of having research agents develop/edit ML models given a task description and a dataset, since the final result can easily evaluated. In contrast, the broader types of research are generally difficult to evaluate objectively and automatically. We leave this as an important future work direction. Please also see our further response in official comments.

It would be more valuable to see some breakdown of which step the model failed at or some other type of automated error analysis.

Our success criteria of improving performance by >10% indeed only shows the broad stroke of the performance of AI research agents. However, we argue that this metric still successfully sets precedence in the evaluation of AI research agents. We have also proposed process-based evaluation, but it is one of the most challenging aspects of the agent evaluation pipeline. As mentioned in the paper, we have attempted to automate this evaluation process with LLMs such as GPT3.5, but the result is too inaccurate to be used. Thus we provide our human error analysis to shed light on the importance of such process evaluation but leave its automation to future work.

To answer your questions about the work:

Did you explore how likely it is that these tasks and their solutions appeared in training data?

The models we used are claude and gpt4, for which we have no information about the training data. So we try to avoid data contamination by selecting recently lauched Kaggle data (within 2-10 months) and current research benchmarks.

Was human annotation performed by the authors? What was the criteria for annotation?

The error analysis is performed by ourselves. The failure modes were carefully chosen by manually inspecting agent logs while also ensuring that those error modes captured some more fundamental limitations of LLM agents. For common failure modes, we outline when is an agent run classified into that mode: Hallucination: We observe that the agent changes part of the code which degrades performance, but still claims that it performed the said task and takes the “Final Answer” action incorrectly. Debugging: We often notice that for our engineering type of tasks, being able to debug code was very important. Very often, agents would get stuck in trying to debug a piece of code it wrote and exceed the max steps permitted. Bad Plan: A run has a bad plan if the initial research plan of the agent itself does not help solve the goal. We will include these detailed descriptions in our appendix.

What aspects of the tasks chosen make them suitable for testing research in general? What kind of research do you expect an agent achieving perfect performance on your benchmark to achieve?

The tasks we chose generally involve building an ML model on a given dataset for as high performance as possible (Note that >10% is an arbitrary threshold we choose to measure the baseline agents, but this threshold could be arbitrarily high instead as a more challenging scenario). Basic improvements could be simply achieved by applying existing knowledge on hyperparameter tuning, data augmentation, architecture changes etc. This allows even naive agents to start demonstrating performance early on. If an agent achieves perfect performance on our benchmark, the agent would have to be very good at ML engineering (which requires software eng skills + knowledge of a lot of state-of-the-art ML methods) to better performance. Furthermore, CLRS and BabyLM in our benchmark are actual open research problems and having an agent solve those would indicate some potential of actually being able to do “research”. To the extreme, the perfect AI research agent should be able to achieve groundbreaking research.

Thanks a lot for identifying merits in our work to have a benchmark to keep track of the capabilities of Agents to perform machine learning tasks. Below, we address some of the highlighted weaknesses:

It is unclear what is meant by "baseline"

As mentioned in 2.1, “The starter code mostly implements a simple baseline model that we can compare with during evaluation”. We will add this clarification to the experiment section and caption.

It would be useful to have a slightly longer description of each of the MLAgentBench tasks in Table 1

We will include clarification of each type of task in the caption, i.e. node classification task refers to classifying node in a networks to different classes; improve speed task refers to improving the run time of a given script; implement tool refers to implementing an LLM based helper script such as performing literature review. We will also add a more detailed per task description in the appendix.

The future work section is too thin for a benchmark paper.

Good point! We will include a separate section detailing future works including: 1) improving agent architecture, such as incorporating literature review, structured memory, better hierarchical actions and even employing multiple subagents to divide up the responsibilities; 2) incorporating more tasks from different domains, such as performing experiments in biology, chemistry, and physics; 3) improving the evaluation pipeline, such as automatic evaluation of agent reasoning process and evaluating agent in more general research scenarios.