MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

Thank you for taking the time to carefully review our paper! We’ll address your comments below:

My major concern lies in the test splits used in this benchmark, which cannot guarantee alignment with the private leaderboard score and thus leads to unfair comparison with data scientists from Kaggle. I checked several Kaggle competitions used in this benchmark and found that they supported “late submission” to provide the leaderboard score. Why MLE-bench does not choose to fully leverage this feature? Could you check whether the test splits used in MLE-bench align with the realistic leaderboard via this feature? If not, I think MLE-bench can only provide comparison among LLM agents rather than human data scientists from Kaggle.

The “late submission” feature unfortunately places too much reliance on Kaggle infrastructure, and e.g. has rate limits and daily submission limits which are not ideal for the longevity of the benchmark.

As discussed in Section 6 on Limitations, we acknowledge that there is uncertainty in the alignment between the private leaderboard score and MLE-bench grading, but we feel that the effect size will be small given our care in producing similar test splits (See Table 7).

We agree that an experiment to compare agent solutions graded via Late Submission VS MLE-bench would help resolve this question, but it is quite a difficult experiment to run (we can’t grade the same submission.csv for each setting since the test sets differ, so we would have to manually curate solutions for each), and unfortunately we will not do doing this now.

I suspect that the capabilities of LLM agents for winning a medal in MLE-bench also correlates to the year of the Kaggle competition. As such, the evaluation metric solely relying on whether an LLM agent can win a medal may lead to biased conclusion. Could you present complete results to show the effect of the competition starting year for the performance?

That’s a sensible hypothesis! We have added this plot as Figure 9 in the appendix, which finds no strong correlation between agent performance and competition year – though we notably find that agents fail to score models on the most recent comps that occurred in 2023-2024 (which are also most difficult by today’s standards).

Also, could you provide empirical analyses on the effect of the competition complexity (as labeled high/medium/low) for the agent performance?

Yes, we have added this result in Table 9 in the Appendix, showing the breakdown of results by competition complexity. We find that the complexity labels correspond well to the agent’s performance, as expected.

Could you present some successful cases and failed cases of o1-preview in MLE-bench? The trajectories shown in Figure 2 are not complete enough to derive insightful findings.

Thanks for asking! We’re currently looking into releasing our agent transcripts alongside our code release but we’ll need an additional step of approval from our organization for this. We agree this would be a good thing to share and hope to share an update soon.

How does plagiarism detection work? If the current competition is A, the plagiarism is detected in terms of merely notebooks in A or all the notebooks in competitions from MLE-bench?

Good question! For a given competition A, we only check for plagiarism in the notebooks associated with competition A, not all the competitions. We’ve clarified Section A.4 in the paper to describe this more clearly.

As discussed in Line 505-509, I also think the current benchmark is too heavy for potential future research purposes. Maybe a light-weight version of MLE-bench can be considered as future work.

Thank you for highlighting this! We’ve given this careful thought and decided that a good option is to encourage users to make use of the Low complexity split of our dataset for lightweight evaluation (this split contains 22 competitions instead of 71, and skews toward datasets and hardware requirements that are more lightweight). We’ve updated Section 2 of our paper to mention this option, and added Table 9 in our paper to include metrics for each of the complexity splits for comparison. We will also make this option clear in our public messaging around the benchmark later, via channels separate to the paper. We hope that this option will improve the accessibility of our benchmark!

---

Once again, thank you for your thoughts and valuable feedback! We hope we have addressed most of your concerns. Please consider raising your review score if you feel this process has improved the quality of our paper.

The three steps basically align with my expectations except that I thought that the checkpoints of the trained models were saved so that such checks can be easily performed.

I also checked the mentioned variant experiment but I didn’t find any quantitative results.

I still think this is a major concern since public leaderboard and private leaderboard often present misalignment in Kaggle competitions. Even minor differences in the evaluation metrics can lead to different medals. The evaluation here should be rigorous enough to claim whether an agent can achieve a medal. Thus, from my perspective, an experiment (a small subset of competitions is enough) to demonstrate this point is necessary.

Sure, let me try to be clearer! There are two slightly different grading scenarios:

1. Kaggle Grading (KG) The original online Kaggle competition, using the original test set (also used for the Late Submission)
2. MLE-bench Grading (MG) Our test sets constructed for MLE-bench, carved out from the original train set. (Described in Section 2.1 and Appendix A.7)

We are interested to know if there are any differences between the KG and MG grading, such that scores on MLE-bench (measured via MG) may not be comparable to leaderboard scores on Kaggle (KG).

To study the differences, ideally we would like to take a submission $S$, grade it using both KG and MG to obtain $f_\text{KG}(S)$ and $f_\text{MG}(S)$, then measure the difference $f_\text{KG}(S) - f_\text{MG}(S)$. The complication is that since KG and MG have different test sets (KG uses the original hidden test set, MG uses our custom made test set), we cannot use a single submission.csv $S$, since the predictions for each test set would be different.

Instead, we can obtain a program $P$ which, given a train and test set $D$, trains a model and produces a submission.csv for each setting. We can use this program $P$ to obtain $S_\text{KG} = P(D_\text{KG})$ and $S_\text{MG} = P(D_\text{MG})$. Finally, we would measure the difference $f_\text{KG}(S_\text{KG}) - f_\text{MG}(S\text{MG})$.

To complete this experiment, we'll need to

1. obtain a suitable set of programs $P$ for each competition (we'll have to manually source these either from existing Kaggle solutions or find suitable programs from our existing agent attempts),
2. run them to train models and predict solutions, and
3. grade the solutions on KG and MG.

Overall, the experiment is not conceptually difficult but we estimate that this could take a few days to execute well and report results on. It is not an unreasonable amount of time, but it is difficult for our team given our current availabilities.

Finally, we'd like to point out that we have already done a variant of this experiment, by comparing the scores of the Sample Submissions on KG and MG (this allowed us to skip steps 1 and 2 above) as a quality check when implementing our graders. This makes us less worried about this being a serious problem. Please see our reply to Reviewer gB21 here about Sample Submissions for more details.

(If there is a simpler version of this experiment that you had in mind, we'd be excited to hear it!)

First of all, thank you very much for your thoughtful review; we’re thrilled that you think this is a valuable evaluation!

The authors may consider providing a lighter dataset split to improve accessibility, similar to the approach in SWE-Bench.

Thanks for the suggestion! We’ve given this careful thought and decided that a good option is to encourage users to make use of the Low complexity split of our dataset for lightweight evaluation (this split contains 22 competitions instead of 71, and skews toward datasets and hardware requirements that are more lightweight). We’ve updated Section 2 of our paper to mention this option, and added Table 9 in our paper to include metrics for each of the complexity splits for comparison. We will also make this option clear in our public messaging around the benchmark later, via channels separate to the paper.

Table 6 in Appendix A.3 indicates a high false positive rate, which may hinder practical reliability. The rate is sufficiently high that manually checking all flagged submissions would be demanding.

We agree that the classifier is imperfect, and we were careful to separate this from the core of the benchmark so it can be easily swapped out with another tool. We hope that tooling will improve in the near future as models and scaffolds improve.

Figure 5 shows GPT-4o familiarity scores above 0.4 across all problems. Does this suggest that these problems are included in the model’s training set?

Good question! Not necessarily; the familiarity scores are computed from the base model’s log probabilities across all the tokens in the competition pages. Since the pages are written in English, it is expected for the token probabilities to have a baseline level of familiarity simply from common sentence structure and phrases.

the conclusions drawn from the correlation between familiarity and performance could be significantly impacted by confounders, such as problem difficulty.

It is true that we cannot rule out such confounders, though each familiarity level has a mix of competition complexities, so the confounding effects of difficulty should be averaged out.

If the same model training pipeline were applied to both the original training set and the modified training set (a subset of the original), one would typically expect lower performance on the latter due to reduced training data.

Good point! We’ve been careful in constructing our splits (as detailed in A7) to ensure that the training set is not significantly reduced in size. We agree that future experiments like the one you suggested would provide helpful evidence, though we will not pursue it now.

Could the authors clarify the configuration of the example submission, and specifically, what comparisons were made and under which settings?

Sure! The sample submission is competition-specific, but corresponds to a dummy answer (e.g. predicting “dog” for every input in a classification problem). We do the following:

1. Use the same logic for constructing the sample submission as is specified in the competition description to make an equivalent sample submission for our test split.
2. Grade our sample submission locally on our benchmark.
3. Compare the score our sample submission achieves on our benchmark to the score the online sample submission achieves on the Kaggle leaderboard.

Because the sample submission construction logic is identical between the local and online split, verifying that the scores are in line with each other acts as one sanity check on our grading implementation and splits. (Our codebase is now available as Supplementary Material, and contains all the sample submissions used.)

Regarding the difficulty estimation in L145, how reliable is the human estimation process? Could the authors provide additional details on the setup and methodology for these annotations?

The complexity was annotated by an engineer on our team using the definition in Section 2.1, and reviewed by at least one other engineer. We've also added a breakdown of agent results by complexity in Table 9, showing that the complexity labels correspond well to the agent’s performance.

“agents would execute commands that overload the machine’s disk or RAM, resulting in their process getting killed and their run finishing early.” Do the tested agents incorporate any error-handling or reflection mechanisms for such situations?

Great question! For disk space, there is no error-handling mechanism — if the machine runs out of space, the agent will crash. For RAM, the behavior depends on the agent scaffold. For example, OpenHands executes actions in a separate process, and if the process exceeds the available RAM, it occasionally throws a Python MemoryError that the main process can catch and recover from.

In Figure 3, it might be useful to further scale o1-preview

We agree that scaling o1-preview further would likely improve performance, but we won't run further experiments here due to the associated costs.

Thank you for taking the time to carefully review our paper! We'll address your comments below:

The paper would be improved if the authors could provide a lighter version of the benchmark along with the metrics on this subset

Great suggestion! We’ve given this careful thought and decided that a good option is to encourage users to make use of the Low complexity split of our dataset for lightweight evaluation (this split contains 22 competitions instead of 71, and skews toward datasets and hardware requirements that are more lightweight). We’ve updated Section 2 of our paper to mention this option, and added Table 9 in our paper to include metrics for each of the complexity splits for comparison. We will also make this option clear in our public messaging around the benchmark later, via channels separate to the paper. We hope that this option will improve the accessibility of our benchmark!

A clearer analysis of which kinds of tasks agents perform well or badly on (e.g. split scores by complexity level and by task domain).

We’ve added Table 9 and Table 10 in the Appendix to provide this breakdown of scores by complexity and task domain, as requested.

The paper mentions raw per-task scores (Sec 2.2), although these do not seem to be included in the paper.

We’ve uploaded our codebase as a zip file in the supplementary material, and included the full grading reports for all our experiments in the runs/ folder. This will also be included in our public Github codebase release.

The paper mentions that the authors analyzed agent transcripts/logs. It would be useful if these transcripts were provided

Thanks for asking! We’re looking into this, we’ll need an additional step of approval from our organization to release agent transcripts but we agree this would be a good thing to share and hope to share an update soon.

Re: GPU performance: Could you share any insights regarding this? How often/rarely are agents using a GPU if it is provided? Does the majority of medals come from tasks where no GPU is necessary?

We agree that this is surprising. From our experience, we see that agents often write programs that use the GPU if it is available, though this occurs as part of a boilerplate step (e.g. device = torch.device("cuda" if torch.cuda.is_available() else "cpu")) and we don’t see agents focusing on GPU performance. It appears that the majority of medals come from tasks not requiring GPUs, given that the no-GPU setup has a similar number of medals. One hypothesis is that tasks not requiring large GPUs tend to have smaller datasets, which are simpler to work with and easier to iterate on which is why agents naturally do better on these.

Re: Leaderboard metrics: Why did you choose this evaluation over using the leaderboard ranking directly (normalized into [0, 1])?

Good question! Although it’s natural to think so, medals are not simply a quantization of leaderboard percentile. Kaggle has their own carefully calibrated system with specific rules to decide how and when medals are awarded, such that the value of each medal type reflects the same quality of achievement regardless of e.g. how many participants you competed against (see https://www.kaggle.com/progression). We choose to rely on Kaggle’s definition which has been more robustly tested, and also believe that medals makes for salient thresholds, i.e. the Kaggle community is accustomed to comparing how many medals different users have, and Kaggle Grandmasters are defined according to medal count.

Re: Figure 5: Why did you use this new metric instead of either the fraction of medals across seeds (as in other experiments) or the leaderboard ranking?

We ran this experiment with GPT-4o AIDE, which gets a medal only 8.7% of the time. If we used medals on the y-axis, each point would be quantized down and we were worried that the performance would be too weak to get a meaningful signal here. (Note that although this normalized score has more resolution, the floor is not consistently defined across competitions, so we still prefer our medals-based metric as our main metric.)

Depending on the agent, in 20% or more cases the agent is unable to make any valid task submission. What are generally the reasons for this?

To make a submission, the agent has to produce a submission file at /home/submission/submission.csv. This fails to count as a valid submission if the agent does not produce such a file, OR the file does not contain data in the correct format. We’ve seen failures of all kinds, e.g. where the agent fails to do the work necessary to produce a prediction; the agent forgets to write its predictions to a file; the agent writes to the wrong path; the submission does not have the correct number of predictions, etc.

Once again, thank you for your thoughts and valuable feedback! We hope we have addressed most of your concerns. Please consider raising your review score if you feel this process has improved the quality of our paper.

Thank you for taking the time to carefully review our paper! We'll address your comments below:

There seems to be an issue with Figure 2. I can only see a small snippet of the figure.

Thank you for catching this! We replicated the issue in Safari and uploaded a fixed version.

I would suggest the authors provide two versions of the benchmark: one that is more accessible and one that is less accessible.

Thanks for the suggestion! We’ve given this careful thought and decided that a good option is to encourage users to make use of the Low complexity split of our dataset for lightweight evaluation (22 competitions instead of 71, and skews toward datasets and hardware requirements that are more lightweight). We’ve updated Section 2 of our paper to highlight this option, and added Table 9 in our paper to include metrics for each of the complexity splits for comparison. We'll also make this option clear in our public messaging around the benchmark later.

I’d like to see a more clear setup and rules section to make using the benchmark as easy as possible.

Thank you for the feedback! Could you clarify which aspects of the setup and rules were unclear? We’d be happy to address them in the next revision.

I wish the anonymized codebase was made available during submission.

We have now uploaded a zip of the codebase as Supplementary Materials, and we will release this as a public Github repository as well.

I don’t see any presentation of the agent scores as a function of complexity.

Thanks for the suggestion! We’ve added Table 9 which breaks down agent performance by complexity level.

Some of the selection criteria is clear (e.g., completed competition), but others are more qualitative (e.g., well-specified description), so it would be nice to see something a bit more detailed and systematic for those.

Thank you for raising this point! We’ve updated Appendix A1 to clarify what we meant by “well-specified”: The description is detailed and thorough without any major ambiguities about how to implement the competition that might only be resolved in the Discussion tab or external materials. In practice, there were very few borderline cases and it was often clear whether the competition was well-specified or not.

what does “where sensible, we maintain the train/test split ratio” mean? (L157-158).

We’ve updated Section 2.1 to be more explicit about our process for determining train/test split sizes. It was challenging to find a single hard-and-fast rule suitable for all competitions. We therefore followed the rule “take 10% of the original training set for the new test split” except for where it didn’t make sense.

For example, the “New York City Taxi Fare Prediction” competition has 5.42M train samples and 9k test samples. Here, using the 10% rule would give our new test set two orders of magnitude more samples than the original, so we opted to instead maintain a similar train/test ratio to the original.

Similarly, why was the headline metric chosen that way? Is this standard for Kaggle competitions?

Yes, medals are the standard metric used in Kaggle competitions. Basing our headline metric on medals has some advantages: Kaggle has a carefully calibrated system to decide how and when medals are awarded, such that the value of each medal type reflects the same quality of achievement regardless of e.g. how many participants you competed against. For more details, see https://www.kaggle.com/progression. Furthermore, medals have an intuitive interpretation to the general public.

I would have liked to see examples of the generated code, potentially with an additional quality analysis.

Thank you for the suggestion! We’re currently seeking permission to share more examples and will provide an update once we receive a response.

Could the authors please clarify how the complexity for each competition was derived?

The complexity was annotated by an engineer on our team using the definition in Section 2.1, and reviewed by at least one other engineer.

How were the 7 development competitions chosen? (L150-152).

The development competitions were selected for their small dataset sizes, which ensures they’re quick to download and fast to iterate on during development.

Are the restrictions in Section 3 a part of the benchmark? For example, the time limit of 24 hours? (L243).

Good question! The details of our particular setup outlined in Section 3 are not requirements of the benchmark because we don’t want the benchmark to be hardware or resource specific.

Is the plagiarism checker provided as a part of the benchmark for free? (L229-233).

Yes, the plagiarism checker, Dolos, is open source, though we don't redistribute it. Once installed you can call it from our code.

---

Once again, thank you for your valuable feedback! We hope we have addressed most of your concerns. Please consider raising your review score if you feel this process has improved the quality of our paper.