TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark

…details on mutation score calculation…

Thank you for pointing this out. We add more details and references on how the mutants are generated in Appendix D.2.1.

…lack of quantitative analysis results in the main text is a significant weakness…

We agree with your feedback and made these updates in the revised paper version. We refactored the most important parts of our quantitative analysis to be part of the main paper and kept the remaining analysis in Appendix.

…the best practices often suggest to write functions that are independent and de-coupled…

This is true, however to effectively generate unit tests one needs knowledge of the entire file under test. Developers organize unit tests at the file level because it makes sense to have common setup (i.e. mocking objects for a file) followed by tests for individual methods in the file. Additionally, unit test frameworks such as JUnit generally expect generated tests to be at the file level; it is challenging to run individual tests in most frameworks, as people usually execute tests at the file level. Choosing to model our benchmark at the file level allows us to construct tasks that more closely resemble automated quality assurance, for example generating an entire test suite from code under test or adding to an existing complex test suite. LLM approaches that target test file generation outperform those at the method level (see Rao et. al. 2023 in comparison to Nie et. al. 2022), indicating that this file level context is essential for performance.

… Rust…whether these languages fall outside the scope of "real-world unit testing" as claimed by the authors…

We argue that Rust would still fall into our paradigm of generating tests at the file level. In Rust a file would be organized as a series of functions, thus if TestGenEval was extended to Rust, it would measure the ability of LLMs to test functions in a given Rust file. We changed the wording from class under test to file under test to improve clarity.

…using \citep instead of \cite…

Thank you for pointing this out. We agree with your comment and use \citep when appropriate.

I would like to know what is the cost involved in executing TestGenEval? Including length of prompt, required GPU memory, running time and more cost analysis?

We generate ~11M tokens for TestGenEval and ~1.5M tokens for TestGenEvalLite (including both our test suite generation and test completion tasks). Runtime for TestGenEval is typically between 12-24 hours for the full benchmark (depending on model capability), across 16 CPUs and 2-4 hours for TestGenEvalLite across 16 CPUs. Running without mutation score significantly reduces runtime to 2-4 hours for TestGenEval and approximately 10-30 minutes for TestGenEvalLite for the same 16 CPU setup. For all settings, we use the full model context for TestGenEval (typically 128k tokens), however it is possible to run with a 32k context window with minimal performance differences (see Appendix F.4 for full details). Note that executing TestGenEval is significantly less costly than executing SWEBench (fixes for SWEBench often involve understanding multiple large files and running large trajectories, while TestGenEval focuses on generating test suites and test cases for a single file under test).

How do the authors ensure the quality of the dataset, including but not limited to possible leakage of data and semantic correctness of the given code?

To answer the data leakage question, we conducted a thorough experiment in line with existing papers on data leakage (https://arxiv.org/pdf/2404.18824, https://arxiv.org/pdf/2309.10677), measuring both n-gram accuracy and perplexity of SWEBenchLite (lite split of SWEBench and what TestGenEvalLite is based on) compared to popular software engineering benchmarks, and GitBug-Java (a benchmark of recent bugs that should not be leaked).

Below are links to each dataset we compared against:

Defects4J: https://github.com/rjust/defects4j

BugsInPy: https://github.com/soarsmu/BugsInPy

GitBug-Java: https://github.com/gitbugactions/gitbug-java

SWEBenchLite: https://github.com/princeton-nlp/SWE-bench/tree/main/swebench

Perplexity of different models:

5-gram match of different models:

As we can see from both tables, the 5-gram match and perplexity of SWEBenchLite is relatively similar to GitBug-Java (a recent unleaked benchmark). For an older and likely leaked benchmark such as Defects4J, we can observe much greater differences in perplexity and 5-gram match, indicating that data contamination is unlikely to be an issue for TestGenEval at the moment.

For semantic correctness of code, we measure both coverage and mutation score. While it is possible to achieve 100% code coverage by just invoking all lines in the method under test and asserting true, it is only possible to achieve high mutation score if the generated tests can discriminate between buggy and fixed code. The need to check more than syntactic correctness of generated solutions was a major motivation for including mutation score (a strong execution metric) as one of our evaluation metrics.

…Dependence on Prompts and Temperature Settings…

We agree that other settings than 0-shot performance would be relevant (and list this as a limitation in Appendix H). For our tasks, few-shot generation or agents could also be evaluated and we release all our code and containers to allow us to evaluate other methods on our benchmark easily. However, we also believe that comparing popular models for zero-shot generation is very relevant as it is close to the setting users often experience in chat applications and many programming tools.

…Generated tests may fail on the code under test while exposing bugs in the code under test…the oracle problem…

While we agree that the oracle problem exists (and list this as a limitation in Appendix H), we argue it is less significant for our current setup of TestGenEval. We run our benchmark on the “patched” version of SWEBench files, where each of the files has gone through the pull request process and been approved by reviewers. Such code is higher quality than most code on GitHub, and is thus less likely to contain bugs. Our task setup is also still useful in the context of regression testing and also aligns with how developers typically write unit tests. While we might not catch bugs in these files directly, we are still able to catch future regressions with generated tests that obtain high coverage and high mutation score.

Why is the massive Llama 405B selected in the evaluation rather than other better-performing models like Claude and Gemini?

We chose to evaluate both strong open and closed source models. We wanted to include open-source models as they are more customizable and allow for more privacy, which is important for part of the community. Open source models are also not gated behind paid APIs like Claude and Gemini.

For closed-source models, GPT-4o is widely used and recognized as one of the best models for coding tasks. For open-source models, Llama 405B is a strong model achieving similar performance compared to the most widely-used closed-source models such as GPT-4o, Claude and Gemini. Although Llama 405B is large, we could run it on our servers and were not limited by API costs like with closed-source models.

What effort does the benchmark made to ensure the test matches the conventional definition of a unit test?

We follow prior work in generating prompts for unit test generation, and additionally only measure mutation score on the file under test (thus if the model were to write an integration test, it would perform poorly on our benchmark, with low mutation score on the file under test). We filter out tests that do not contain assertions or expect exceptions, marking all such tests as not passing on the code under test. Both our filtering and metrics give us confidence that the model generates tests that match the conventional definition of a unit test. We also perform extensive qualitative analysis, and in all cases we find that the model generates tests that resemble a conventional unit test. We provide a website with all model generations (linked in the paper) if you want to verify our conclusions yourself.

Why isn't any agents/tools evaluated in the evaluation?

This is one of the limitations we mention in Appendix H. To the best of our knowledge, there is no open agent targeting unit test generation and general agents (e.g. Devin) are not publicly available. We consider that creating our own agents for existing models is out of scope for this paper. We wanted to focus on a simpler setting, where we are able to compare models fairly (with lower variance and susceptibility to agent specific hyperparameters). For those looking to build and run testing agents, we open source all code and dockerized containers for evaluation.

Which mutation tool and mutants did you use? And how many mutants do you generate for each example?

We used cosmic-ray to generate mutants and use the default set of mutation operators (https://github.com/sixty-north/cosmic-ray/tree/master/src/cosmic_ray/operators). Executing mutants dominates generation in terms of compute cost (our average source file is 1157 LOC and cosmic-ray takes ~15-30 secs per 1000 LOC). For each example, we generate 1031 mutants on average, with approximately 20% of all files we run mutation testing timing out in the one hour allocated for mutation testing execution. However, our error margin for mutation score is low, with an average mutation score uncertainty of 1.06%, meaning our estimates are accurate even in the case mutation testing times out.

…consider renaming "Pass @ 1" to "Any Pass @ 1"...

We agree that the naming was confusing and updated this in the revised version of the paper.

…not generating tests with asserts…how do you detect them?...do you count…as pass or not pass?

We detect tests without assertion or exception keywords statically, as not all generated tests will throw a runtime error. We qualitatively also looked at these cases to confirm our keyword match was comprehensive. We mark these cases as not passing to not inflate our metrics.

What is the key difference between the newly created benchmark and the evaluation sets used in prior work on ML for test generation/completion?

Both the scale and quality of TestGenEval are significantly larger than any prior dataset. Dinella et. al. 2022 and Tufano et. al. 2020 leverage the ATLAS and Methods2Test datasets, which focus on mapping test methods to their corresponding method under test (or “focal” method). As a result evaluation is at the method level rather than file level. This is much more simplified than the large scale file level test suite generation and test completion tasks that we target in TestGenEval. Repurposing their datasets would require omitting both of our core tasks.

Nie et. al. and Rao et. al. release test generation datasets that could be repurposed, however their datasets generally consist of simpler programs. Both papers construct their evaluation datasets by trying to build projects and filtering out those that didn’t build, biasing their evaluation datasets towards simpler projects. We provide a table to compare properties of each dataset. TestGenEval has both complex tests and reasonably long code methods, along with more data points than other evaluation datasets. Rao et. al. has longer source method lengths due to increased verbosity of Java code (https://dl.acm.org/doi/pdf/10.1145/3571850), but significantly fewer gold test methods, and lower quality test files (both shorter in length and containing fewer tests on average). The higher number of tests and test length of TestGenEval points to generally higher quality repositories with better testing efforts. We argue these more closely resemble real world software testing. This is also evidenced by the very high star count of repositories used in TestGenEval (3,523-78,287 stars).

Outside of the higher quality of TestGenEval data, TestGenEval is set up in a much more reproducible way compared to prior work. We provide docker images for each file pair and an evaluation harness to compute both coverage and mutation score, enabling the community to easily use and extend our dataset. Building around a widely used dataset such as SWEBench also ensures a degree of data quality, as previous issues with testing inconsistencies and indeterminism have already been figured out by the community.

Do you only consider the tests that appeared in PRs in SWEBench?

We consider the code and test files that appeared in PRs in SWEBench, as these files are involved in real world bugs. We argue that testing targeted at these files is more important than testing arbitrary files in a repository, since we are measuring test quality on files that have had historic bugs in them. Furthermore, this paradigm creates a clean mapping between code and test files touched by a patch are directly connected with the code in the patch. This clean mapping ultimately leads to a higher quality dataset that the community can use.

…yours is method-level completion, which sounds more like the "test method generation" task…

Our completion task involves generating entire methods for the file to be tested, as Rao et. al. did for their test method generation task. We updated our phrasing in the revised version of the paper to improve clarity.