Automated Benchmark Generation for Repository-Level Coding Tasks

We thank the reviewer for their valuable and positive review, highlighting both the novelty of our work and the value it brings to the community. Below, we address their remaining questions:

Can you extend the evaluation of the created benchmarks?
We have added three additional agents/methods (SWE-agent v1, ZeroShot-BM25, ZeroShot-Oracle) and four additional models (Haiku3.5, Llama 3.3 70B, Qwen 2.5 72B, DeepSeek V3) to our evaluation. See the results below

Model comparison with ACR

Agent comparison with GPT4o-mini

While most of the additional results are in line with what we observed so far, we want to highlight that there are some interesting behaviours. In particular, DeepSeek V3 performs much better (2-3x) on SWE than on both other benchmarks, similarly SWE-Agent v1 performs much worse on SWA than on both SWEE and SWE. This highlights the value of diverse benchmarks, enabling a holistic evaluation of new models and methods.

Where do you observe the performance differences mentioned in the Abstract?
E.g., in Table 6, we show that the performance of GPT4o is 40% lower on SWA compared to SWE. We observe even bigger differences in the results reported above.

We thank the reviewer for their detailed review, acknowledging the value of our contribution to the community and the quality of our analysis. Below, we address their remaining concerns.

Can you evaluate the presence and correctness of fail-to-pass tests more rigorously?
We first want to highlight that we already filter the dataset to only include instances where we extracted F2P tests by executing the test suite in the original state with test changes applied and in the fixed state with test changes applied. We have additionally conducted a manual review of the extracted tests (see response to Reviewer FoWC) and find them to be of high quality for 90% of instances. Finally, we have rerun the test execution and extraction 10 times for SWA to identify possibly flaky tests and extract 509 shared instances across all 10 runs (compared to 535 from the original run). We traced the remaining discrepancy down to two main causes: randomly parameterized tests and genuinely flaky tests. We will create a version of both the SWA and SWEE datasets with these filtered out but expect this to have minimal impact on our results and note that similar behaviour is present in SWE-Bench (see e.g., https://arxiv.org/pdf/2407.21787).

Have you evaluated SetupAgent both as a whole and with respect to its individual components?
We evaluate SetupAgent in Section 5.2, discussing its overall performance in Table 3 and conducting an ablation study in Table 4, demonstrating the importance of all its components. We have now additionally conducted a human evaluation, discussed in the response to Reviewer FoWC.

Can you analyse the issues discussed in SWE-Bench-Verified in context of your benchmarks?
Given the similarity in the benchmark generation pipeline, we expect similar findings to those for SWE-Bench(-Verified). However, the two main review/filtering criteria in the creation of SWE-Verified, namely, the completeness of the issue description and the specificity of added tests, only aim to filter out instances that can not be solved without additional information. This is crucial to assess the absolute performance of code generation systems but will affect all evaluated systems to a similar extent and thus have only marginal impact on a comparative analysis. We note that such a comparative analysis is sufficient to select among different systems for a given use case or guide their development. Similarly, we focus our analysis on the differences in the relative performance between models and compare our benchmarks to SWE-Full and not SWE-Verified. Finally, we have conducted a similar human review as for the creation of SWE-Verified on 30 random samples of SWA, which we discuss in the response to Reviewer FoWC. We are happy to include this discussion in the next revision of our work.

Can you discuss the differences between SWE, SWA, and SWEE? Are all needed?
The main difference is the set of considered projects. While SWE(E) focuses on libraries that typically expose functionality to other programs and thus tend to have more stable, well-tested APIs, SWA focuses on applications (e.g., locust or xonsh) that are directly used by humans and often have (graphical) UIs. As we show in Table 6, the performance of some models (GPT4o-min) is quite consistent across these different tasks, while others (GPT4o) varies significantly. We have conducted additional experiments (see response to Reviewer XqFN) where we see significant (3x) differences across benchmark for SWE-Agent v1 and DeepSeek V3. We believe this demonstrates the value of more diverse benchmarks in this highly active field.

Comment on Section 3.1
We introduced the notation in Section 3.1 to formalize our plain English (and sometimes incomplete for brevity's sake) descriptions of different settings. However, we are happy to move this formalization and its usages to the appendix.

Conclusion
We hope to have addressed the Reviewer's concerns, look forward to their reply, and remain happy to answer follow-up questions.

We thank the reviewer for their detailed feedback, highlighting the depth of our analysis, the scalability of our approach, and the value of our work for the community. Below, we address their remaining questions and concerns:

Can you conduct a manual review of generated tasks to assess their quality?
We have conducted a manual review of 30 randomly chosen SWA instances for both task and setup quality.
To assess task quality, we follow the protocol used to create SWE-Verified. That is, we scored issue description quality and clarity on a scale of 0 (well-specified issue with clear success criteria) to 3 (almost impossible to solve correctly without further instructions) and test quality from 0 (test perfectly covers valid solutions) to 3 (tests are too narrow or broad or requiring information not provided in the issue description). For both questions, 0 and 1 are considered passing scores. We observe the following: 23 (77%) instances have a meaningful and sufficiently complete issue description, and 22 (73%) of these additionally have suitable tests (27 or 90% across all instances) to check whether the issue was fixed.
To assess setup quality, we score the extract setup as either functionally equivalent to the described setup or incorrect and the test setup from 0 (functionally equivalent to described setup) to 2 (correct tests only partially or not at all executed). We observe the following: All instances run the correct tests with 77% using exactly the test commands provided in the reference. 22 (73%) instances additionally have a fully correct installation/setup.
We will include an extended version of these results in a revised version of our paper.

How do you ensure the generated benchmarks are meaningful beyond passing (trivial) generated tests?
We first want to highlight that all tests are taken from the original repositories and were thus written by human contributors and considered to be sufficiently valuable to be merged. We further filter out all instances where not at least one such test that failed before the fix and succeeds after. Finally, our manual evaluation above demonstrates that cases where no meaningful tests are present are rare (only 10%).

Can you extend the evaluation of the created benchmarks?
Yes, please see the response to Reviewer XqFN.

How can your benchmarks/approach be integrated into existing agent evaluation pipelines?
To allow for a seamless integration of our benchmarks, we have made sure to create our benchmark instances such that they are compatible with the popular SWE-Bench evaluation harness (only requiring minor changes to load installation and testing commands from the dataset rather than being hardcoded) and will release both our dataset and the modified version of the harness.

How do you plan to motivate the community to use your tool and benchmarks?
We believe that the research community can benefit strongly from our benchmarks as the increased sample diversity will reduce the effect of overfitting and thus level the playing field for new methods that were not tuned to SWE-Bench at a significant cost. Further, we believe the ability to efficiently create new domain-specific benchmarks will be especially important for practitioners working on domains not well represented in SWE-Bench. We believe these will be strong motivators to adapt the benchmarks/method proposed here. Additionally, we plan to launch a benchmark website similar to swebench.com to allow for easy result tracking and comparison.

Can you include a direct comparison to ExecutionAgent?
We first want to highlight that ExecutionAgent is not capable of setting up specific (historic) states of repositories. It can thus not be used directly for benchmark generation and is limited to our repository setting in Table 3. Directly compared on 25 random repositories considered for SWA-Bench and using GPT4o-mini for both methods, SetupAgent (ours) succeeds on 13, requiring 29.2 s on average (12 min total), while ExecutionAgent succeeds on only 9, requiring on average 2700 seconds (18h total). We thank the reviewer for suggesting this experiment and will include it in the revised version of our paper.

Can SetupAgent be expanded to handle more complex setups?
While SetupAgent can already extract complex setup steps, a key limitation is that our dockerized execution environment does not support running docker instances inside this docker environment. We believe this is an exciting item for future work but out of scope here given the involved engineering challenges. We note that such a version of the benchmark would also be incompatible with existing SWE-Bench evaluation harnesses.

Conclusion
We hope to have addressed the Reviewer's concerns, look forward to their reply, and remain happy to answer follow-up questions.

We thank the reviewer for the detailed review and positive feedback, recognizing the relevance and quality of the benchmarks we created. Below, we address their remaining questions:

Can you assess the repeatability of SetupAgent and what it implies for the generated benchmarks?
First, we want to emphasize that benchmarks need only be constructed (and then updated) once to evaluate a large number of code generation methods. Therefore, a lack of repeatability would not have an adverse effect on the benchmark’s value and reliability. Nonetheless, we conducted an additional experiment corresponding to the repository setting in Table 3, rerunning SetupAgent 3 times for 100 random candidate Repositories from SWEE. This led to 26, 27, and 27 successes, where the last two runs install exactly the same instances and the first just drops one. This demonstrates that SetupAgent is highly repeatable. We thank the reviewer for suggesting this experiment and will include it in the revised version of our paper.

How was the sample size for the generated benchmarks chosen ?
We combined our own experience developing code agents and the community's preference for first the Lite and later the Verfied Version of SWE-Bench to choose a sample size of ~500 instances. As the reviewer noted, larger versions can be created easily, however, making evaluation more costly.

Can you add an experiment analysing the effect of the underlying model on SetupAgent?
We first want to highlight that our work focuses on the framework we propose, which we analyse in the ablation presented in Table 4. Further, we specifically chose GPT4o-mini for its great affordability, as it allows us to demonstrate that benchmark generation is now accessible to the wider community. Finally, we note that few cheaper models are available, e.g., even an 8B model hosted by together.ai having the same inference costs, and more capable models being significantly more expensive, e.g., GPT4o having ~15x higher inference costs. We conduct an additional experiment with GPT4o corresponding to 100 SWEE samples in the repository setting in Table 3. We observe GPT4o succeeds for 24 repositories compared to 26 with GPT4o-mini, demonstrating the robustness of our framework to the underlying model. Interestingly GPT4o uses fewer iterations to improve its output (2.1 instead of 3.4 on average), explaining the slightly lower success rate.