MLGym: A New Framework and Benchmark for Advancing AI Research Agents

Q3: Benchmark Saturation

We would like to clarify the metrics used in this work.

1. Performance Profiles and AUP

We use Performance Profiles and Area under Performance Profiles. Both of these metrics measure the comparative performance of the selected models. This metric is not intended to provide an absolute score and does not have an upper limit. The metric upper limit changes depending on the selection of models or agents.

For example, in Figure 2, O1-preview reaching 1 in terms of performance ratio does not signify the ceiling of the benchmark. Instead, it states that at this value of $\tau$, O1-preview can outperform all the other models in the pool, but does not imply that it can achieve the best possible performance on each task in the benchmark.

2. Why a comparative metric?

We have selected a comparative metric for this work due to the open-ended nature of the research tasks. Since we are comparing models on open-ended tasks, we want a metric that is robust for comparison and will not saturate. AUP score can be seen as similar to ELO rating, which is used for measuring comparative performance between models on LMArena. However, ELO has its challenges, and the AUP score has been shown to integrate performance on diverse tasks with different metrics in AutoML.

We have also reported raw performance metrics for each task and model in Tables 5 and 6 in Appendix D.1. Below, we show a snapshot of the raw performance scores from Table 5 (Best Attempt@4) with baseline performance, Claude-3.5-Sonnet (overall best) and Best performance score achieved by one of the models in the evaluation pool.

As shown in this table, no single model consistently achieves the best performance across all tasks. The AUP score is intended to assess the overall consistency of the models amongst the evaluation pool.

3. Flexibility of the Benchmark

The saturation of this benchmark depends on the saturation of the underlying tasks themselves. For this reason, we decided to include a diverse set of tasks that are collectively not saturated. For example, some of the game theory tasks have no theoretically correct solution, and the proper solution depends on the opponent's strategy. Thus, it is easy to increase/decrease the difficulty of the task. Similarly, human researchers are still working towards the Language Modeling problem.

Moreover, we would also like to point out that MLGym-Bench is designed to be a live benchmark; that is, if a dataset used in the task becomes saturated and the agent reaches the best solution (1) that would signify a step change in LLM agents capability (controlling for the data leakage) and (2) the dataset or the task can be easily replaced with a similar but more challenging variant, thus maintaining the difficulty of the benchmark.

Q4: Vision tasks

We acknowledge that the current CV task set is modest. However, this benchmark is not intended to be a comprehensive set of tasks for all fields; rather, it is meant to be a small selection of relevant, diverse tasks spanning a wide range of fields/problems. We intentionally want to keep the benchmark small and well-curated to democratize its use, i.e., anyone can easily evaluate it without needing a lot of compute.

As discussed in Q3.3, MLGym-Bench is supposed to be a live benchmark, and with community involvement, we can increase the difficulty and breadth of the benchmark as research progresses. Moreover, as discussed in Appendix A, MLGym-Bench currently focuses on the Level 1: Baseline Improvement categorization of tasks. However, with the framework, it is easy to simply remove the baseline code and let the agent start with just an evaluation script, thereby increasing difficulty with a simple switch.

Due to the reasons listed above, we believe this benchmark will remain useful for a long time to come and can be easily extended to support a much wider variety of tasks, while also increasing the difficulty.

We thank the reviewer for their thoughtful review and questions, and we hope that we have adequately addressed them. We have also noted the reference mistake pointed out by the reviewer and will update the manuscript to fix it. We are always available to answer any further questions and address any comments the reviewer may have during the discussion period.

Thank you for the detailed author response. Below is concise, point-by-point feedback to help strengthen the revision.

1. Model Coverage & Timely Update

• Benchmark risk: Even if the goal is not to release a full leaderboard, the absence of regularly updated mid-sized models (e.g., 7-34 B checkpoints) can mislead LLM developers about the true difficulty ceiling.

• Actionable suggestion: Add at least a representative slice of open-weight 7–34 B models, or publish a lightweight protocol (task subset / shorter episodes) so the community can contribute results without prohibitive cost.

2. Evaluation Clarity (Sec. 6 vs. Sec. 3.4)

I apologize for any misunderstanding: my concern is not the AUP formula itself but reader confusion caused by two different “Evaluation” discussions:

• Sec. 3.4 (task design) already spends several lines on evaluation scripts.

• Sec. 6 later re-introduces evaluation, somewhat duplicating the narrative.

• Minor suggestion: Merge or cross-reference these sections to avoid the impression that “evaluation = CSV prediction” (Kaggle style) when many tasks require script-based grading.

3. Granularity of Task Descriptions

Because you claim coverage of CV, NLP, RL, game theory, and tabular data, task-level evaluation details matter. Please consider:

• Include a concise per-task table in the main text summarizing input modality, artifact type, grading metric, and compute budget, clear pointers to full evaluation scripts in the repo.

• Alternatively, at minimum, provide qualitative case studies with full agent trajectories for each task in the updated manuscript to offer concrete qualitative examples.

4. Training Facilities & Extensibility

Current design showcases 13 tasks, but it is unclear whether the framework easily scales to dozens more. Concrete ways to strengthen this point:

• Provide a template generator (or wizard) that auto-creates config files, conda envs, and stub evaluation scripts.

• Include at least one “blank” scaffold example (no starter code) to illustrate curriculum flexibility.

5. Multi-Agent Road-Map (Optional but Valued)

While I accept that multi-agent is out of scope, a one-paragraph road map or tiny two-agent demo would reassure readers that the API will not need a fundamental redesign later.

Summary Recommendation

The authors have addressed many issues, but gaps remain in model spectrum coverage, evaluation narrative cohesion, and framework extensibility. Addressing the bullet points above would significantly improve clarity and strengthen community confidence in MLGym-Bench. However, I am reasonably willing to raise my score if the authors provide clear explanations to resolve the remaining confusion.

C2 – “Overly broad and vague definition of AI Research Agents”

1. Definition and taxonomy

We use AI Research Agent to refer to any agent that cycles through idea generation, implementation, and experimentation within a sandboxed ML environment, with the goal of making progress on an AI research problem, such as language modeling. Recent works employ nearly identical language; e.g., “AI Scientist” [5] and “AI Co-Scientist” [6]. Appendix A gives a six-level capability taxonomy (from reproducing research papers to pursuing a long-term research agenda). We will surface this definition and table in the main text.

2. Difference from NAS / AutoML

NAS requires a handcrafted search space and is typically tailored to each domain, and solutions do generalize well in a cross-domain setting [7]; it does not address open-ended problem formulation. By contrast, AI Research Agent runs unchanged across 3-SAT, CV, NLP, RL, and game-theory tasks, operating in the same environment and highly expressive search space consisting of any idea that can be expressed in natural language, algorithm that can be implemented in a programming language, and any experiment that can be run via a bash command. An LLM agent is also much more autonomous, as it needs to validate for itself whether an idea or algorithm is better than the baseline (or the best so far). In contrast, NAS / AutoML typically provides an automatic way of evaluating methods and selecting the best one.

3. Scalability to “real-world” research

MLGym’s modules (Agent | Environment | Task | Dataset) are container-isolated; users can swap in larger clusters or longer budgets without touching core code. Thus, nothing prevents large-scale or resource-intensive experiments. Comparing this with existing ML Engineering/Research agent frameworks, we see AIDE [8] uses a search algorithm but only works on single files, in contrast to MLGym, where the agent abstraction is designed to work on large codebases.

If the reviewer has a concrete task that seems incompatible, we would welcome the specifics so we can address it directly.

[1] Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., Ericheva, E., Garcia, K., Goodrich, B., Jurkovic, N., Kinniment, M., Lajko, A., Nix, S., Sato, L., Saunders, W., … Barnes, E. (n.d.). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts.

[2] Jordan, K., Bernstein, J., Rappazzo, B., et al. (2024). modded-nanogpt: Speedrunning the NanoGPT baseline. https://github.com/KellerJordan/modded-nanogpt

[3] Jordan, K. (2024). Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/

[4] Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., Du, Y., Qin, Y., Xu, W., Lu, E., Yan, J., Chen, Y., Zheng, H., Liu, Y., Liu, S., Yin, B., He, W., Zhu, H., Wang, Y., Wang, J., … Yang, Z. (2025). Muon is Scalable for LLM Training (No. arXiv:2502.16982). arXiv. https://doi.org/10.48550/arXiv.2502.16982

[5] Yamada, Y., Lange, R. T., Lu, C., Hu, S., Lu, C., Foerster, J., Clune, J., & Ha, D. (2025). The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search (No. arXiv:2504.08066). arXiv. https://doi.org/10.48550/arXiv.2504.08066

[6] Gottweis, J., Weng, W.-H., Daryin, A., Tu, T., Palepu, A., Sirkovic, P., Myaskovsky, A., Weissenberger, F., Rong, K., Tanno, R., Saab, K., Popovici, D., Blum, J., Zhang, F., Chou, K., Hassidim, A., Gokturk, B., Vahdat, A., Kohli, P., … Natarajan, V. (n.d.). Towards an AI co-scientist.

[7] Tu, R., Roberts, N., Khodak, M., Shen, J., Sala, F., & Talwalkar, A. (2022). NAS-bench-360: Benchmarking neural architecture search on diverse tasks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Advances in neural information processing systems (Vol. 35, pp. 12380–12394). Curran Associates, Inc.

[8] Jiang, Z., Schmidt, D., Srikanth, D., Xu, D., Kaplan, I., Jacenko, D., & Wu, Y. (2025). AIDE: AI-Driven Exploration in the Space of Code (No. arXiv:2502.13138). arXiv. https://doi.org/10.48550/arXiv.2502.13138