Is Self-Repair a Silver Bullet for Code Generation?

Thank you, Reviewer LPE6, for your kind words and suggestions for further experiments! We are happy to hear that you find the paper well written, and the methodology/experiment design both sound and insightful.

As also pointed out by Reviewer PNGf01, you are correct that our contributions are limited to the few-shot/in-context setting. We share your sentiment that it would be very interesting to see how finetuning would affect the conclusions in this paper, but—as you have pointed out—this is out of scope for our experiments (which we emphasize are already very large, due to the branching nature of self-repair). We look forward to future work which can leverage the insights we share here to investigate what role fine-tuning can play in alleviating self-repair’s feedback bottleneck!

As to your questions: You touch on two important ideas, that we certainly share your excitement about.

The first is to use a value network to select which candidate program to repair at each step. Our preliminary experiments did explore some very simple search strategies, such as ranking the candidates by their cumulative log probabilities, but we found no evidence that this was correlated positively with self-repair success rate. However, it is very much possible that using a value network/“confidence model” to rank the candidates (Chen et al., 2021; Inala et al., 2022; Zhang et al., 2022; see Related Work) would yield performance benefits. Although this fell outside of the scope of this paper, we are excited about future work investigating when and how proper tree search can be used in conjunction with self-repair to yield greater performance increases.

Your second question is reminiscent of an RLHF (without the H) training stage, in which the model uses self-sampled feedback (and repairs) to improve its own code debugging & repair capabilities. This is certainly an interesting idea, and one we believe will play an important role in the future as new (publically available, permissively licensed) datasets to train on become more rare. However, it is also a method with clear challenges, and not one we have started experimenting with at this stage!

Thanks again for your feedback. Let us know if you have any further questions; we'd love to continue this discussion through the rebuttal week and beyond!

We thank the reviewer for their feedback on the limitations of our analysis! We agree that there is much left to explore in this space, and are excited about the future work this work will inspire. Without further ado, let’s jump into a discussion of the weaknesses identified by PNGf01.

The scope of our study is limited to self-contained Python programs, which is quite different from real-world software engineering.

Software Engineering (SE) and competitive programming both present their own unique challenges. SE tasks often involve incomplete task specifications as well as (long) contextual dependencies. Competitive programming, on the other hand, is mainly challenging due to the logical and algorithmic complexity, often requiring the use of dynamic programming and graph algorithms to solve the task.

An ideal self-repair model should be able to (1) fix errors in logically complex tasks, (2) handle ambiguous specifications and missing context, and (3) repair without a clear test oracle. The community would benefit from in-depth studies of each capability. In this paper, we isolate the first aspect, providing insights that would be hard to untangle otherwise.

With this in mind, we believe competitive programming becomes a well suited testbed for our analysis:

• Using logically complex, well-specified programming puzzles to benchmark models has a rich history in the literature [0,1,2].
• Recent work shows that even contemporary models are still challenged by intermediate/advanced-level competitive programming tasks [3, 4].
• Competitive programming tasks have relatively complete specifications and unit tests, so we do not have to worry as much about our results being muddled by degenerate cases (e.g. the solution is incorrect simply because the task specification did not provide a sufficient definition of an API).

In our study, we are thus able to hone in on self-repair's efficacy in algorithmically challenging tasks of varying difficulty levels. We use this primarily to isolate the importance of the feedback stage, but it also enables us to discover non-obvious relationships such as the non-straightforward interplay between task difficulty and self-repair efficacy (Section 4.1 and Appendix B). This surprising fact would have been difficult to discover without isolating the effect of the logical/algorithmic complexity of the task at hand.

In future work, we'd like to explore the other two capabilities as well. These are motivated by the question of how future software engineering workflows should best leverage AI code generation tools. One can imagine workflows which emphasize encapsulation to an extent that each individual piece is no more complex than a programming puzzle, but tricky bugs still tend to arise at the interface level. Furthermore, the fact that Test-Driven Development has fallen out of vogue suggests that developers do not like writing unit tests first, which motivates capability (3). Similarly, although natural language may not have played a big role in software engineering historically, minimizing the impact of ambiguity in NL specifications is now necessary in order to enable capability (2).

In summary, as we hope this discussion has shown, we wholeheartedly agree with the reviewer that there is much interesting work left to be done in this space. Furthermore, we will revise the Introduction and Future Work sections to clarify the scope of our contributions, as well as highlight some of the nuances discussed above.

Our study is limited to prompting strategies, and does not consider fine-tuning.

We certainly agree that such an analysis would be of interest to the community! However, we believe that this falls outside the scope of this study, which we emphasize is already significant in terms of its depth (and cost). In addition to the methodological challenges, fine-tuning also comes with practical concerns such as cost and model availability. We thus leave it to future work to investigate whether fine-tuning can alleviate the feedback bottleneck identified by this paper. Besides, prompting may be accessible to a wider audience than fine-tuned models in the current AI ecosystem; studying self-repair in this context may therefore benefit AI practitioners more in practice.

Please let us know if you have any further thoughts or questions - we greatly appreciate your feedback!

---

[0] Li, Yujia, et al. "Competition-level code generation with alphacode." Science 378.6624 (2022): 1092-1097.

[1] Austin, Jacob, et al. "Program synthesis with large language models." arXiv preprint arXiv:2108.07732 (2021).

[2] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

[3] Hendrycks, Dan, et al. "Measuring coding challenge competence with APPS." arXiv preprint arXiv:2105.09938 (2021).

[4] Inala, Jeevana Priya, et al. "Fault-aware neural code rankers." Advances in Neural Information Processing Systems 35 (2022): 13419-13432.

We thank Reviewer Vfe407 for their helpful comments! We are very happy to hear that the reviewer found our contributions insightful, easy to understand and well scoped (with clear discussion of limitations).

We agree with the reviewer that the relationship to Chen et al. (2023b) is important to discuss. After replying to the reviewer’s specific questions below, we will give a detailed account of how our work compares to that of Chen et al (2023b).

---

Individual responses to questions.

Is there a specific reason to restrict feedback and repair samples to 1 in the analysis of feedback boosting (section 4.2)?

This restriction is mainly a practical one: separating the feedback and repair makes the experiment significantly more time-consuming (and costly) to run, since each stage must now be implemented as a separate API call. With this restriction, we can control the cost of this experiment by using a smaller value of $N_f$ and $N_r$ without increasing the risk of statistical artifacts too much. We emphasize that the preceding experiments already showed that this setting is the most effective, and we therefore do not believe this limits the analysis in practice.

Should we expect similar results using the ‘self-debugging’ approach that uses few-shot prompts?

Although our results are already similar (and we do use few-shot prompting; see Appendix F for a complete list of the prompts we use), we agree that our results are–generally speaking–not as strongly in favor of self-repair/self-debugging. This is because of slight differences in the experimental setting being investigated; see the discussion below for a detailed comparison.

---

Detailed comparison to Chen et al. (2023b).

As mentioned in the Related Work, Chen et al. (2023b)’s method is indeed closely related to ours; both use few-shot prompting to encourage the model to first retrospect on the code (in order to understand why it failed) and then perform repair. There are, however, a few differences.

1. The main reason our results differ from those of Chen et al. (2023b) is that we conduct our experiments in a slightly different setting.

In Chen et al.’s study, the self-debugging method has access to a correctness oracle to decide whether to proceed with debugging or not; the baseline does not have access to the oracle. In our work, both the baseline and the self-repair approach have equal access to an oracle (when evaluating pass@k) and are compared with the same sample budget. Thus, the performance improvement over the baseline is more prominent in Chen et al. (2023b)’s work than in ours.

We choose to grant both baseline and repair models equal access to the oracle so that we can better analyze the tradeoff between increased sampling overhead and accuracy gained. Chen et al. (2023b) instead aim to show how oracles and different repair strategies can improve the final accuracy compared to current, standard, non-repair-based approaches. Thus, our findings are not actually contradictory to those of Chen et al. (2023b).

2. Generally speaking, our work focuses on exploring the efficacy of self-repair in depth, while Chen et al. focus on comparing a broader range of different self-debugging strategies in a few different domains. Our studies thus complement each other.

Concretely, our work focuses on what Chen et al. calls the “Unit Test + Explanation” style of feedback, in which the model is provided with external feedback from a unit test suite but then has to generate its own explanation of why the test failed. We study the effect of this textual explanation in great detail through our experiments in sections 4.2 and 4.3, and show that it is the limiting factor in this type of self-repair. We also offer a substantive discussion of the effect of the hyper-parameters, which amongst other things highlights the need to obtain sufficiently diverse initial samples if self-repair is to be successful.

The scope of our study is also limited to what Chen et al. call “Text-to-Python Generation” tasks, while they also consider code translation and SQL query generation tasks. However, we do so in both an easier setting (HumanEval) and a significantly more challenging setting where baseline performance is lower (APPS), while Chen et al. focus on the relatively easy MBPP dataset. As we show in section 4.1 and Appendix B, the relationship between task difficulty and self-repair performance is quite subtle, an insight which we would not have been able to show without analyzing multiple datasets in detail.

---

We once again thank Vfe4 for your helpful feedback on the paper. We hope that this discussion has clarified how our work relates to that of Chen et al. (2023b), and in particular the slightly different experimental settings as well as our emphasis on depth of understanding. If you have any more questions or thoughts, please do not hesitate to share them with us - we look forward to continuing this conversation throughout the discussion period!