Learning to Generate Unit Tests for Automated Debugging

We thank you for your detailed comments and questions. We attempt to address each of them in detail below:

** $R1$ There is no experiment evaluating the proposed UTDebug framework specifically**

Due to the space constraints, we moved this analysis to Appendix E and Table 6 where we ablate both strategies test-time scaling and backtracking one at a time on Qwen 2.5 7B using both randomly-generated UTs and those by UTGen. On average we find that without test-time scaling, debugging performance on MBPP+Fix decreases by 7.7% (absolute). Similarly, removing backtracking in UTDebug yields an average drop of 2.7% in debugging performance. While we point to this in footnote 4 of the main paper, we will use the extra page in the final submission to expand on this.

---

** $R2$ … an approved edit might make the program lose its original correct behavior on ungenerated test cases.**

Your point is well taken, however, based on the empirical results we make the following arguments:

• When conditioning the UT-generator on the target code (as in UTGen and prompted baseline), the model is tasked with figuring out challenging edge cases to test the code’s correctness. The UT output is consistent with the task description, meaning that once generated, a UT can be used for validating the quality of edits and backtracking (L217 - 227) even in subsequent rounds, even if the target code has now been edited. This ensures the edits are more robust in UTDebug as the number of rounds increases.
• Second, if conditioning UTs on a target code was suboptimal for debugging, the random baseline (without any such conditioning) would outperform UTGen. However, we find that UTGen consistently outperforms the random baseline for multiple model families and scales. Additionally, the intrinsic evaluation shows that UTs generated by UTGen are better at predicting if a code is incorrect (by attack rate) and we reaffirm this finding in response to Q5 below.
• Lastly, we agree that increasing the number of UTs would improve the overall debugging process as it provides a higher chance for any errors to be revealed. We test this in Appendix F, where we scale the number of UTs ($n$) generated in each round from 1 to 15 in Fig 4, finding that debugging performance generally scales with increasing $n$. However, we note that the improvements from UTGen over baselines persist even as we increase n=15 unit tests in each round, with an improvement of +10% (absolute) over randomly-sample UTs on MBPP+Fix (Hard).

---

** $Q1$ Can we use more than one test case when prompting the LLM to generate an edit?**

In our preliminary experiments, we found focussing on one failing UT at a time to be just as good if not slightly better than using feedback from all failing unit tests while requiring fewer input tokens. One possible explanation is that due to the multi-round debugging framework we use. Once the code is edited to resolve one failing UTs, in the next round the model generates more challenging UTs conditioned on the edited code, and can continue to debug if there are any errors or failures. Note that using feedback from one failing UT is consistent with prior debugging works such as Chen et al., (2023b) and Zhong et al., (2024).

---

** $Q2$ How did you order the test cases in the experiments?**

In cases where the model generates multiple failing UT, we select the unit test mentioned in the feedback prompt randomly. However, as we explained above, feedback from other failing UTs is indirectly utilized in the form of pass rate to determine if an edit should be accepted or rolled back.

---

** $Q3$ In Table 2, debugging with randomly-sampled UTs sometimes outperformed the prompted failing UTs. Why was this the case?**

As mentioned in Sec 3.3, debugging with model-generated UTs can result in noisy feedback, for e.g., if the output of UT is predicted incorrectly. This, coupled with the findings in Sec 5.1 (L308-314) that models struggle to correctly reason over the output of more challenging UT inputs, could explain your observation. Specifically, in the random baseline, the UTs generated are not conditioned on the target code and therefore, may correspond to relatively simpler inputs whose outputs are easier to predict. On the other hand, in the prompted baseline, the model might generate more challenging UT inputs conditioned on the edge-cases of the target code; if it fails to generate the correct output, the resulting debugging feedback would be less useful. This motivates the need for training LLMs (as we do in UTGen) to generate both failing inputs and the correct UT outputs. These findings are also corroborated by the intrinsic metrics on HE+Fix and MBPP+Fix datasets in Appendix D and Table 5, where the random baseline scores are higher than the prompted baseline.

We thank the reviewer for their helpful comments and feedback, as well as for acknowledging the paper to be “clearly structured”, “addressing a novel research question”, introducing a “helpful benchmark”, and the experimental evaluation to be “thorough” and “comprehensive”.

One of our core contributions lies in characterizing the generation of failing or challenging unit test inputs and predicting correct outputs for them as reasoning tasks for coding LLMs, which motivates our use of chain-of-thought strategies (Wei et al 2022). Please see our detailed responses below:

** $W1$ Insufficient Justification of CoT and Reject Sampling**

We would like to clarify that our research contributions include identifying the tradeoff in unit test generation via intrinsic properties and designing a training pipeline that bootstraps necessary data for training LLMs to be better unit test generators from code-generation datasets. We acknowledge that the broader use of chain of thought or rejection sampling are not novel and have been used in prior work (including data generation) – therefore, we do not claim these as contributions of our work. Additionally, we use rejection sampling for curating the training data because prior to training, models do not always generate failing unit test inputs, thus, requiring additional filtering. We will expand on this discussion in the final version of our paper.

---

** $W2$ Evaluation scope and baselines: Comparison with CodeT baseline**

UTGen differs from CodeT in that CodeT is designed for code generation by selecting a code from the largest consensus set of codes and independently generated UT (without conditioning on code). In contrast, UTGen is designed for UT generation focussing on the intrinsic quality of UTs as well as their utility for downstream tasks like code-debugging and best-of-n ranking by conditioning UT generation on the edge cases of a target code. To directly compare against CodeT, instead of using the code generated from CodeT, we use its unit tests in downstream debugging (L114-117). We would also like to point out that the UT generation procedure in the randomly sampled baseline (L269-272) is consistent with the UT generation procedure in CodeT, but uses self-consistency for predicting the output of the UT.

We run this experiment by scaling up CodeT such that it uses a similar computational budget as UTGen and sample n=3 unit tests for debugging with Qwen 2.5 7B and 32B Code-Instruct models.

Across both models and all three debugging datasets, UTGen outperforms debugging with unit tests generated by CodeT by as much as 7.54% (absolute) on MBPP+Fix and 15.88% (absolute) on MBPP+Fix (Hard) with Qwen 2.5 32B model. Moreover, CodeT (which generates UTs independent of code being debugged) lags behind the prompted UT baseline in 3/6 settings, showing that conditioning UT generation on the erroneous code better helps identify and localize bugs. We find CodeT performs slightly better than the randomly-sampled UT baseline (especially for MBPP+Fix Hard split). We will include these results in the final version of the paper.

---

We hope to have addressed your questions satisfactorily, and will be happy to engage in further discussions based on any follow up questions you may have.

We thank you for your insightful questions and feedback. We are glad that you appreciated the “tradeoff in UT generation” highlighted by our work, “comprehensive evaluation” and that UTGen offers a “cost-effective alternative to commercial LLMs”. We respond to your comments and queries below:

---

** $R3, Q1$ How is the current approach better than CodeT?**

UTGen differs from CodeT in that CodeT is designed for code generation by selecting a code from the largest consensus set of codes and independently generated UT (without conditioning on code). In contrast, UTGen is designed for UT generation focussing on the intrinsic quality of UTs as well as their utility for downstream tasks like code-debugging and best-of-n ranking by conditioning UT generation on the edge cases of a target code. To directly compare against CodeT, instead of using the code generated from CodeT, we use its unit tests in downstream debugging (L114-117). We would also like to point out that the UT generation procedure in the randomly sampled baseline (L269-272) is consistent with the UT generation procedure in CodeT, but uses self-consistency for predicting the output of the UT.

We run this experiment by scaling up CodeT such that it uses a similar computational budget as UTGen and sample n=3 unit tests for debugging with Qwen 2.5 7B and 32B Code-Instruct models.

Across both models and all three debugging datasets, UTGen outperforms debugging with unit tests generated by CodeT by as much as 7.54% (absolute) on MBPP+Fix and 15.88% (absolute) on MBPP+Fix (Hard) with Qwen 2.5 32B model. Moreover, CodeT (which generates UTs independent of code being debugged) lags behind the prompted UT baseline in 3/6 settings, showing that conditioning UT generation on the erroneous code better helps identify and localize bugs. We find CodeT performs slightly better than the randomly-sampled UT baseline (especially for MBPP+Fix Hard split). We will include these results in the final version of the paper.

---

** $R1, Q2$ How good is the perturbed code created by LLM? Any human study to determine the quality of the dataset?**

We clarify that our evaluation datasets (HE+Fix, MBPP+Fix) feature human-written or realistic code-generation errors from smaller LMs, not artificially-perturbed code (L237-250, Appendix A.1). LLM perturbation of gold codes was exclusively used for UTGen training due to limited dedicated datasets (L161-165) and we expand on the training setup in Appendix B (L693-707) as well as the perturbation on Page 21 (bottom). UTGen's demonstrated efficacy in code-debugging and Best-of-N ranking (Sec 5.2 and 5.3), even with initial filtering to control the quality of the perturbed code (L701-704), confirms the high quality of our training data.

---

** $R2, Q3$ Do you have any analysis of the compute cost of the current approach against the baselines like CodeT?**

As explained in L276-277, all unit test generation approaches use the test-time scaling and backtracking features of UTDebug, making them computationally comparable, i.e., similar number of LLM calls, token budgets and run-times. Based on our response to Q1/R3 above, we scale up CodeT’s unit test generation capabilities to also have a comparable computational budget as well as a similar number of UTs sampled. Additionally, we point out that we train UTGen models with a standard academic budget (L708-716), outperforming other trained counterparts like external state-of-the-art 8B RM (Liu et al 2024a) and yielding similar if not better UT-generation capabilities compared to frontier LLMs that are monetarily more expensive. Lastly, we plan on publicly releasing our trained models for the research community to use directly.

---

We hope our response has addressed all of your questions and will allow you to revisit your score. We are happy to answer any followup questions and requests you may have.