SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents

We thank Reviewer ueFW for their critical perspective and for raising interesting questions, which we address below.

Can you discuss the effect of possible data contamination on your results and how it might be mitigated?
We thank the reviewer for raising this point. In short, we see no statistically significant difference in performance (p=37%) for issues from before and after the data cutoff and thus conclude that contamination is not affecting our results significantly. Please see the main response for more details.

Why were significant existing works on LLM-based test generation not included in the paper?
We discuss automated test generation (including using LLMs) in our related work (Lines 71-75). Due to space constraints, this discussion focuses on the most important related works and, e.g., omits [1,2,3,4] which are not applicable to our setting as they only consider coverage and generating passing tests [1,3,4], require a large number of tests and fixes to be generated [2], and the to be tested code to be specified [3,4]. We are happy to include these in the next revision using the additional space. If the reviewer could provide a list of works they believe to be relevant, we would be happy to review and include them.

How can the long-term relevance and maintenance of SWT-Bench be ensured?
We agree that long-term relevance is an important topic, actively discussed in the community. We believe SWT-Bench is particularly well positioned to ensure long-term relevance. In particular, new instances can be easily created from new repositories as well as new issues in already used repositories. However, such a rolling benchmark also has disadvantages (comparability of scores, cost of reevaluation). Regardless of these aspects, we believe our initial results showing the promise of Code Agents to already be valuable irrespective of any long-term maintenance of SWT-Bench.

Can you discuss the relevance of your results on Python and how they might generalize to other languages?
Please see our main response for a discussion on Python’s relevance and how our approach applies to other languages. Further, it is common in the field to focus on a single language, especially when addressing a new task, be that Java [3,4], Kotlin [1], or Python [2,5].

Can you evaluate your benchmark on more than three LLMs?
We first want to highlight that while we only considered 3 LLMs (the, at the time, best proprietary model (GPT4), a model balancing cost and performance (Claude 3 Haiku), and a strong open-weight model (Mixtral 8x22B)), we consider a wide range of different agents based on the strongest available model. Given the challenging nature and thus low resulting performance even of GPT4, we refrained from considering weaker models. We want to highlight that all other reviewers specifically highlighted the quality of our extensive evaluation. Finally, since the original submission, stronger models have been released and we have conducted additional experiments on GPT4o-mini, Claude 3.5 Sonnet, and Mistral Large 2 using SWE-Agent, reporting results in Table 2 of the attached PDF. We observe that results follow general model capability as expected.

Can you discuss the potential societal impact of widespread adoption of Code Agents in software testing, particularly concerning developer employment?
Testing and bug reproduction are often neglected among professional developers due to a lack of extrinsic and intrinsic motivation [6]. We thus believe a more widespread adaptation of Code Agents in software testing has the potential to not only improve code quality but also developer productivity and satisfaction. Regarding the potential to displace human developers, benchmarks such as SWT-Bench show that current Code Agents are still far from matching or even outperforming human developers. Instead, they show that human supervision is still essential to leverage current Code Agents. Finally, while we believe that benchmarks such as ours can help drive progress toward better AI systems and thus increased automation, we do not believe our work specifically to have an outsized societal impact beyond the general developments in Generative AI. We are happy to include this discussion in the revised version of the paper.

Can you discuss the various limitations of your work including the metrics used and potential data contamination?
We first want to highlight that we discuss a range of limitations in “Section 6: Limitations”. Regarding data contamination, we are happy to include the new results presented in the attached PDF and discussed in the global response in the paper. We would like to ask the reviewer to explain which further “various limitations, including … of the metrics used” they have in mind.

Conclusion
We hope we were able to address the reviewer’s concerns and questions. We would further like to ask the reviewer, where they see flaws in the soundness and presentation of our work given that all other reviewers rate them as good or excellent. Finally, we would like to ask the reviewer to reconsider if their score is justified given the practical relevance and high potential impact they attest our work with weaknesses being shared by most other work in the space.

References
[1] Alshahwan et al. 2024, Automated Unit Test Improvement using Large Language Models at Meta
[2] Chen et al. 2022, CodeT: Code Generation with Generated Tests
[3] Schäfer et al. 2023, An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
[4] Chen et al. 2024, ChatUniTest: A Framework for LLM-based Test Generation
[5] Wang et al. 2024, TESTEVAL: Benchmarking Large Language Models for Test Case Generation
[6] Straubinger and Fraser 2023, A Survey on What Developers Think About Testing

We thank the reviewer for their quick reply and hope that we could successfully address all points they did not have follow-up question on.

Effect of Data Contamination on Applicability for GPT4
While the applicability is indeed higher for issues created before the KC for GPT4, both F2P rate and Coverage are lower, which are much stronger indicators for memorization. Applicability solely measures whether the LLM can respect the required diff format, while F2P and Coverage measure the correctness of the generated tests. While memorization would lead to a correct test (higher F2P and coverage) it would not necessarily improve applicability as the test would not have been included in the training data in the right diff format. Further, we want to highlight that only 11 samples after the KC were available, making any conclusions based on these results statistically questionable (as highlighted in our previous reply).

Thank you for the responses, and sorry for the late response due to a flurry of proposals and review dues. I have increased my overall assessment as some of my concerns have been addressed. While others are shown in below.

Data contamination Thanks for your experiments for GPT-3-preview 1106, the experiments do not address my concern. At least, I need to mention the data contamination for all models by directly providing previous inference results and dividing them based on the KC time.

Existing works not included in the paper? Thanks for your response. But I need to mention that omitting CodeT is crazy.

Other languages? The response does not address my concern.

Other LLMs I would recommend providing experiments for OpenCodeInterpreter, DeepSeek-Coder, XwinCoder, CodeLlama, WizardCoder, and Starcoder2 family in open-source LLMs for a new benchmark.

We thank Reviewer pxq3 for their detailed review, insightful questions, and helpful suggestions. We are happy to hear they appreciate the originality, quality, clarity, and significance of our work and in particular our comprehensive and rigorous evaluation across methods and models. Below, we address their remaining questions.

Can you conduct additional experiments to establish statistical significance and/or report confidence intervals?
Please see the main response for a detailed answer. In short, we have added experiments and statistical analysis demonstrating a high statistical significance of our results.

Can you conduct an error analysis, categorizing common errors made by different methods, and provide qualitative examples? Do these error types correlate with issue complexity or repository characteristics?
We have conducted a detailed analysis of common errors for our best-performing method SWE-Agent+ and will gladly add the findings to the paper for the camera-ready version together with concrete examples of their occurrence. We find the most common issues of the agent to be adding tests that do not reproduce the issue (either due to unrelated failure or non-failure) and incorrectly classifying the result as relevant, getting stuck in a loop after making inapplicable edits or encountering syntax errors, failing to execute the test environment after test creation, and adding tests with missing fixtures or missing indentation, rendering the test function non-executable. Further, we want to point the reviewer to Section 5.4, where we already investigate correlations of success with repositories, issue description length, and code repair success.

Can you conduct an ablation study on key hyperparameters of the employed code agents?
In the attached PDF, we provide ablation studies on the temperature used for decoding, the number of interaction rounds for various agents, and the Libro sample size (see Table 3 and Figure 1a,b). In agreement with prior work, we consistently observe the best performance for greedy decoding (t=0). We further observe increasing interaction rounds improve performance until saturation at 5-10 iterations (we use 20 as a default) with the only exception being AutoCodeRover, which still gains performance up to the maximum of 20 iterations we consider. Similarly, Libro’s performance improves as more samples are considered, however saturating also around the 5 samples we use by default. We will include these ablations confirming the choices of our hyperparameters in the updated appendix.

Can you conduct a small-scale study with professional developers to establish a baseline in terms of quality, coverage, and time efficiency?
We believe a human study of statistically significant sample size to be infeasible as a baseline given that many of the issues at hand are complex and challenging to understand, with the original underlying issues taking up to a year to resolve. Further, as all ground truth tests were written by humans, we believe that most professional developers should be able to solve this task given sufficient (substantial) time. We want to highlight that such human baselines are uncommon for code benchmarks, even at the much simpler function-synthesis level, due to the above-mentioned substantial time requirements. Finally, we believe that establishing such human baselines, possibly across skill levels, and focusing on simpler benchmarks could be an interesting work on its own and leave it to future work.

Can you discuss the applicability of your approach beyond Python?
Please see our answer in the main response.

Can you include an analysis of execution times for different methods?
For all LLMs we consider, part of the execution time is directly related to the number of tokens digested and generated (see Tables 7 and 8 of the main submission). For methods that require interaction with an execution environment however, time is usually dominated by setting up such an environment in a clean and reproducible manner (i.e. dockerized). We list results on execution times below and observe that all methods except zero-shot inference take between 3-5 minutes per instance, where we can observe a small trade off due to many-turn interactions in Code Agents versus single-shot execution in LIBRO. Given these small differences however, we believe execution time to be of limited practical relevance as issues can be processed in the background, similar to continuous integration, in response to raised user issues.

Conclusion
We hope to have been able to address the reviewer's concerns, in particular regarding the statistical significance and hyperparameter sensitivity of our approach, remain happy to answer their follow-up questions and look forward to their reply.

We thank Reviewer Ky1F for their detailed review, insightful questions, and helpful suggestions. We are happy to hear they appreciate the potential impact of our work as well as the extensiveness of our empirical evaluation and its analysis. Below we address their remaining questions.

Can the scope of the benchmark be expanded to address other repositories and programming languages?
Our approach can be applied to any programming language and can be easily extended to other repositories, including private projects if Github issues and pull requests were used. However, setting up the corresponding execution environments with historically correct dependency versions and parsing of all test results requires some manual effort. Given the already substantial cost of evaluating the whole benchmark, we decide to leave this to future work targeting specific languages or repositories (please also see our answer in the main reply). Further, we want to highlight that most currently used benchmarks such as HumanEval ([1], 164 instances) and the concurrent Software Test benchmark TestEval ([2], 210 instances), focus on much smaller sets of standalone functions in one or few languages, which are much less representative of real-world issues than the popular Github repositories our benchmark is built on.

Can you add a private test/holdout set to avoid contamination?
Having a fully private holdout set is challenging for agent evaluation, as this would require the party hosting this benchmark to execute submitted methods, which can incur substantial costs. Alternatively, the issue descriptions and codebase details would need to be shared with submitters, which would defeat the purpose of a private holdout set. Besides these practical challenges, the code underlying any such holdout set would still be available online, making it challenging to guarantee that it was not used for training. A previously entirely private repository, circumventing the latter issue, is difficult to obtain. Sticking to online code, a rolling benchmark set of recent instances could be used, similar to live code bench. However, this has the issue that reported numbers can not be compared directly and rerunning all methods on the current version would be necessary when evaluating the benchmark. We thus leave this variation to future work.

Can the failure modes of the test generation be leveraged to improve code repair?
We believe that test generation provides a valuable alternative perspective on the capabilities of Code Agents and help drive developments in the field. It can for example unveil repositories where Agents struggle to leverage even the existing tests, highlighting the need for further improvements in Agent tooling. We find several such cases, especially in the repository django, where executing the test suite itself is non-trivial but would benefit Code Agents for validating their changes. Even when test suites are executed, we find that the model often wrongly considers unrelated failures as related to the given issue. Other failures, such as getting stuck in editing loops seem universal and their resolution would benefit both code repair and test generation. We leave a detailed exploration of these directions for future work.

To what extent does the stochasticity of LLMs explain the variance in failure modes and performance between methods?
We first want to highlight that we use greedy decoding for most methods (except Libro and pass@k) which is deterministic up to floating point errors. Further, since the submission, more powerful cheaper models have been released, allowing us to investigate the variance across multiple runs with GPT4o-mini. There, we obtain the results shown in Table 3 of the attached PDF, showing 95% confidence intervals obtained with 5 runs for a range of temperatures. We observe very low variance, which is interestingly dominated by the flakiness of tests (despite execution in the same docker), present even for greedy decoding (temperature 0).

Can you comment on the generalizability of your results to private codebases that are not available online?
Based on manual inspections of agent traces we find that models typically don’t show signs of memorization. In a few cases, the model tries to assess files that do not exist, pointing to either hallucinations or outdated memorization. Overall, we observe that models make use of the provided tooling to navigate and explore the code base. Therefore we conjecture that our results should transfer well to private codebases. We further confirm this by comparing the performance of ZeroShotPlus on all instances created after the training data cutoff of GPT4 to a random subset of instances created before. We observe no statistically significant difference (p-value of 37% using Students t-test).

Conclusion
We hope we were able to address all of the reviewer's remaining questions and remain happy to answer any follow-up questions they might have.

References
[1] OpenAI 2021, Evaluating Large Language Models Trained on Code
[2] Wang et al. 2024, TESTEVAL: Benchmarking Large Language Models for Test Case Generation

We thank Reviewer i5fN for their review and valuable questions. We are happy to hear they consider the problem we study important and under-explored, our work thorough and interesting and our results impactful. Below, we address their remaining concern.

Is your paper over-claiming and implying that SWT-Bench measures all abilities of human software testers?
While our title may be placative, it only implies that Code Agents perform better than other automated systems on the studied task. In this case, this is the task of reproducing an issue from a description in the form of a test. We fully agree with the reviewer that there is a big gap between the skills SWT-Bench is testing and the full spectrum of abilities required by human software testers. We want to highlight that we do not make any such claims anywhere in our paper. Instead, we explicitly focus on reproducing tests (see e.g. the Conclusion (Line 408) and the Introduction (Line 35)). We will make it even more explicit in the revision that reproducing issues is the only measured skill of multiple software testing skills.

How important is the measured capability of reproducing issues?
While reproducing a described issue or testing a specific functionality is certainly not the sole purpose of software testing, we firmly believe it is a very important one. In particular, it is an aspect of software testing that is difficult to solve without powerful NLP processors such as LLMs. The task of issue reproduction requires translating natural language descriptions of these issues/intended functionalities and into formalized definitions of the desired behavior in the form of unit tests. This is in stark contrast to the task of, i.e increasing coverage of a given (subset of a) codebase, or fuzzing, where the target is already formally defined. Finally, our approach permits test generation for test-driven development, as it can generate tests for unimplemented functionality, i.e. when the raised user issue is a feature request. We thus find it important and promising to study this (relatively under-explored) aspect of software testing.

Conclusions
We hope to have been able to address the reviewer’s concerns regarding the claims made by our paper and would like to respectfully ask them to reconsider their evaluation given their own positive assessment of our work. We are happy to answer follow-up questions and would greatly appreciate it if the reviewer could highlight specific sections where they believe we overclaim.

We thank the reviewer for their response and engaging in the discussion. We are happy to modify the title of our submission (in accordance with the submission guidelines) to follow the reviewer’s suggestion and avoid over-claiming.

We propose to adjust the title to Can Code Agents Reproduce Real-World GitHub Issues?.

As promised in the initial rebuttal, we will also revise the text to make the scope of our work more explicit. We are happy to answer any followup questions or suggestions they might have.

I'd like to thank the authors for the response.

While our title may be placative, it only implies that Code Agents perform better than other automated systems on the studied task.

I think this is where the major discrepancy is: the studied task is GitHub issue reproduction, which is quite far from what people would normally think of the work of software testers. Thus I still think the title is over-claiming and somewhat misleading, and I'm afraid I could not give higher score for this work.

Respectfully,
Reviewer i5fN

We thank Reviewer PhkB for their detailed review, insightful questions, and helpful suggestions. We are happy to hear they appreciate the importance and novelty of the problem we study, the importance of our contributions, the extent of our analysis, and the quality of our exposition. Below we address their remaining questions.

Can you provide more details on how the generated tests improve the precision of code repair agents? And can you conduct more large-scale experiments in this direction
Using SWE-Agent, we generate tests and code patches independently for all instances. We then run the generated code patches against the generated tests and retain only those code patches where the tests fail before the patch is applied and succeed after. Evaluating the resulting patches against the ground truth tests shows a precision of 47.5%, significantly higher than the precision of unfiltered patches (19.2%). How to best leverage tests directly for code generation rather than simply selection is an active topic of research with many competing approaches. We therefore leave this as a promising future work item.

How does your work relate to CodeT by Chen et al.?
CodeT focuses on selecting the best implementation(s) from a large set of proposals (100) by generating large numbers of tests and using execution agreement as a quality metric to select correct ones. However, while this approach is effective for function-level, HumanEval-style problems, we focus on significantly more challenging repository-level synthesis. There, neither sampling nor executing such large numbers of different tests and implementations is feasible due to the substantial computational and monetary (when using APIs) cost. In particular, ZeroShot sampling of 100 tests and repairs would cost around $15k for one evaluation of SWT-Bench-Lite. Therefore, we do not compare to CodeT directly. We will include a corresponding discussion in our related work section.

Can you discuss future directions for improving the performance of LLM Agents on SWT-bench and SWE-bench?
We recommend future work to further explore the relationship between patch generation and test generation to improve the performance for both tasks. CodeT [1] is an interesting prior work in this direction. However, it relies on a large number of candidate implementations and tests to find agreements, which is too costly for real-world repositories. Therefore, future work might consider developing more cost-effective approaches.

We have further conducted a detailed analysis of common errors for our best-performing method SWE-Agent+ and will gladly add the findings to the paper for the camera-ready version together with concrete examples of their occurrence. We find the most common issues of the agent to be adding passing tests that do not reproduce the issue, getting stuck in a loop after making inapplicable edits or encountering syntax errors, failing to execute the test environment during test creation, and adding tests with syntax errors. Future work may consider addressing these issues through specialized modules or additional monitoring.

We are happy to include this analysis, discussion and recommendations for future work in the camera-ready version of the submission.

Conclusion
We hope to have been able to address all of the reviewers questions and remain happy to answer any follow up questions they might have.

References
[1] Chen et al. 2022, CodeT: Code Generation with Generated Tests

We thank the AC for taking such an active role in chairing the discussion and are happy to answer their questions below:

What are golden tests?
Exactly as the AC suggests, these are the tests that were added in the pull request that “resolved” the GitHub issue underlying the SWT-Bench instance. At least one of these tests fails on the original state of the codebase, before the golden code patch is applied, and passes after. They are thus always a valid (gold standard) solution for the SWT-Bench task. We will make this more clear in Line 32 where they are first introduced.

Can you discuss what can cause none of the lines modified by the golden code patch to be covered by the test suite?
The main reason for this is that Python’s trace module by default is unable to measure coverage in subprocesses. If (part of) a test suite is executed using such independent processes, coverage can not be measured accurately and no execution of the patched code is reported. Meanwhile, threading based concurrency is handled correctly. We would like to highlight that this issue comes from the task instance and Python’s trace module, but not the test generation method. The measured coverage is thus still comparable across different test generation methods and computed over the remaining 57% of SWT-Bench instances.

Can you clarify what a valid patch is in Section 5.2?
A valid patch is one that can be successfully/without error applied to a codebase, i.e. that has the correct format. As generated tests are added to the code base in the form of a patch, we reused the term “valid patch”, currently defined implicitly in Line 170. We will follow the AC’s suggestion and rename them to “well-formed tests” and introduce this terminology explicitly to improve clarity. As many models struggle to produce valid/well-formed (i.e. applicable without error) test patches in the default unidiff standard, the rate of valid test patches is a relevant metric in particular when comparing our novel (ZeroShotPlus) with the standard unidiff patch format (ZeroShot).

What is a fail-to-any (F -> X) test and why is it “good” to measure this?
A fail-to-any test is one that fails on the original codebase and either passes or fails on the codebase with the golden patch applied. While some F->X tests might be undesirable as the AC suggests, we consider the rate of F->X tests as a useful metric for assessing a test agent’s overall capabilities for two main reasons. First, creating an F->X test is a prerequisite for generating a fail-to-pass (F->P) test, which is ultimately the desired outcome. Therefore, the rate of F->X serves as an intermediate metric for our final F->P metric. Second, given that the agent has access to the original version of the codebase and the issue report, it can leverage these two inputs to self-verify if a test fails on the codebase before proposing the test. The rate of F->X tests reflects the agent’s ability to perform this self-verification. We will add this discussion in the corresponding section.

Conclusion
We hope to have been able to address the AC’s questions and remain happy to answer any follow-up they might have.