We thank the reviewer for insightful suggestions and questions.

1. Paper novelty

While there are several related works on self-debugging, our paper focuses on how to improve the model’s self-debugging capability, which is important but not yet extensively investigated.

We believe "NExT: Teaching Large Language Models to Reason about Code Execution" is a concurrent work with us. Yet there are still differences between the NExT paper and our paper:

1. One of the most notable differences is the proposed RL training with explanation and execution reward. RL training is important as it helps LLMs to learn from the failure generations as well, and the separation of explanation reward and execution reward helps LLMs learn differently about the explanations and fixes.
2. Differences also lie in how we synthesize explanations while they focus on traces reasoning, and how we utilize much larger training datasets such as APPS and CodeContests to improve generalization while they primarily focus on MBPP and HumanEval as the training data.
3. Also, they mainly conduct training experiments on PaLM 2, while we train multiple open-sourced backbones to prove the generalizability of our approach.

We will add a discussion on the differences with this paper in our draft, but we do not think that the contribution and novelty of our paper should be questioned based on this concurrent work.

2. Position in literature

As mentioned above, our paper focuses on how to improve the model’s self-debugging capability via training. Most existing works on self-debugging focus on prompting LLMs to do self-debugging, which does not work well on open-sourced smaller LLMs as we have shown.

A few related works that train LLMs as we cited in the paper:

1. ILF requires human-annotated explanations, while we train LLMs to generate bug explanations by themselves.
2. CYCLE and Self-Edit only train LLMs to generate refinement using SFT. We train LLMs to explain the bug which not only enhances LLMs’ reasoning but also helps developers to understand the wrong code (as our human evaluation shows). We also explored using RL to further improve the self-debugging performance.
3. NExT, a concurrent work from ICML this year that we have discussed above.

We are also different from all these works by designing RL training. The RL training brings improvement to the strong SFT models over several baselines, leading to higher Pass@K, higher refinement rate, and better bug explanation.

3. Self-taught refinement

We provide the experiment results using the synthetic data generation from the model itself for CodeLlama 7B. We highlight some results here. Below is the CodeLlama-7B SFT/RL using the data collected from itself, evaluated on MBPP+ and HumanEval+.

The full results are in our attached author response PDF file, Tables 1 and 2. Results show that self-taught SFT and RL also achieve big improvements. CodeLlama-7B SFT/RL models achieve up to 5% improvement in self-debugging, compared with the baseline prompting method, and the model trained with code generation data only. Though comparing the experiments using data from CodeLlama-34B and GPT-3.5-Turbo, the improvement is less.

4. Comparison to the teacher models

We provide the comparison with the teacher models CodeLlama-34B and GPT-3.5-Turbo on the self-debugging setup in the global rebuttal pdf. Comparing it with Table 2 and Table 10 in the paper, we see that with CodeLlama-34B as the teacher, CodeLlama 7B SFT/RL achieves close to CodeLlama-34B self-debugging performance, while CodeLlama-13B SFT/RL significantly outperforms the CodeLlama-34B teacher (e.g. in HumanEval+ pass@1 56.24% vs 48.51%).

5. Unit Test at test time

The model first generates an initial solution for a given problem description. The initial problem description contains one or more test case examples (example input and expected output).

Then the initial solution is tested against all the test cases provided for the problem. If one test case fails, the model will take the failed test case and the error message to generate refinement.

That is, at the test time, the generation of the initial solution will see some example test inputs and outputs. And the generation of refinement (and explanation) will see the exact failed test cases.

6. Reward Design

CodeBLEU: We find that only using binary execution feedback as a reward does not train the model properly as the reward is too sparse. That is, a completely wrong solution will get the same reward as an almost correct solution, which hurts the RL training. Although the CodeBLEU reward is weak, we find that having it helps stabilize the training by densifying the reward distribution. This is the main reason we introduce the CodeBLEU score in the reward.

Similarity: The formula for R(e) might look strange, but the main goal is to scale the majority explanation similarity (Figure 3 c in paper) to the range of [-5, 5] so that it is in the same value range as the code rewards (Figure 3 d in paper). Our reward design is based on the statistics of the training data.

7. Generalization

To avoid overfitting, we use a large batch size (128) to only update the model for a few thousand steps. We test our models’ perplexity on 10000 samples from BigQuery Python code. The CodeLlama-7B pre-trained model’s perplexity is 1.457, the SFT/RL models' are 1.597 and 1.599, just slightly higher.

We thank the reviewer for the positive comments.

1. Evaluation of APPS and CodeContests

This is a good suggestion and we plan to add the results to the final version if accepted.

Below are the results on APPS and CodeContests. We test StarCoder on the full 5000 APPS test samples. However, due to the large amount of test samples in APPS (5000), we only test the CodeLlama-7B/13B on a subset of 200 samples. We plan to keep running to complete the evaluation and add it to the paper.

On APPS and CodeContests, the prompting baseline also only refines very few solutions and the improvements brought by refinement are marginal. However, with our SFT and RL-trained models, we see significantly stronger self-refinement ability. The final Pass@k is also about doubled from the prompting baseline.

2. Reward Range

Our RL algorithm is based on the PPO algorithm. According to the details in Appendix A3, the rewards of each token are either the code reward, explanation reward, or the KL divergence. From our calculation of the KL divergence distribution on a subset of the training data, the trajectory tokens’ KL divergences have a minimum value of -5.22 and a maximum value of 5.37. Thus, we scale our code and explanation rewards to the range of [-5, 5], which is similar to the KL divergence value range.

Actually, we think as long as the code/explanation rewards are not too large or too small to cause gradient explosion, the setup should be reasonable. For example, [-1, 1], [0, 1], are also common choices of reward range.

3. Scale Effect

We observed a similar scaling effect in Table 4 in Section 4.1.2 where we show the successful refinement rate of different models across prompting, SFT, and RL. We see that from CodeLlama 7B to CodeLlama 13B, the refinement rate improves by around 3% (absolute rate) across the board for all approaches. We think that larger models have better capabilities in general, and the trend applies to self-debugging capability as well. Our proposed training seems to help push the self-debugging performance closer to the model’s limit, and in general, a larger model should have a higher upper bound.

We thank the reviewer for insightful suggestions and questions.

1. Explanation improvement

The reviewer might have some misunderstanding of the explanation evaluation in Table 7. In Table 7, we performed both human evaluation and LLM Judge-based evaluation to evaluate the explanation quality before and after training. It clearly shows that the SFT/RL training significantly improves the explanation quality, by generating more correct and helpful bug explanations. Appendix A6 Table 13 lists the rubrics.

We also provide some case studies on human evaluation in Appendix A6 of the paper to show qualitatively how the finetuned model generates better explanations.

2.Comparison with Instruction-tuning

In Appendix A.4.1 of the paper, we have presented the comparison of SFT on strong code instruction data with SFT on the full data we collected. The code instruction data we used includes the Magicoder dataset [ref], similar to the OpenCodeInterpreter-CL-7B the reviewer mentioned. The results in Appendix A.4.1 show that even when training with strong code instruction data, the model does not obtain self-debug capability out-of-the-box and it only refines up to 3% of its wrong solutions. And this is one of the main motivations of the proposed approach.

3. Evaluation of APPS and CodeContests

This is a good suggestion and we plan to add the results to the final version if accepted.

Below are the StarCoder-15B’s results on APPS and CodeContests. Results of the other two backbones can be found in our attached author response PDF file (Tables 4 and 5). Due to the large amount of test samples in APPS (5000), we only test the other two backbones on a subset of 200 samples. We plan to complete the evaluation and add it to the paper.

On APPS and CodeContests, the prompting baseline also only refines very few solutions, and the improvements brought by refinement are marginal. However, with our SFT and RL-trained models, we see significantly stronger self-refinement ability. The final Pass@k is also about doubled from the prompting baseline.

4. Related works

We thank the reviewer for pointing out missing related works and providing some discussions here.

In the Reflexion paper, an LLM verbally reflects on task feedback signals and generates improvements. It is similar to the prompting method.

OpenCodeInterpreter constructs a multi-turn interaction dataset that integrates execution and human feedback for code refinement using GPT-4, and shows that models on such dataset achieve good refinement performance.

In EURUS paper ([3] mentioned by the reviewer), it curates an UltraInteract dataset as an alignment dataset for complex reasoning tasks using GPT-3.5-Turbo and GPT-4, and includes multi-turn interaction trajectories with the environment and the critique.

Both of the last two papers obtain multi-turn feedback interaction dataset using close models GPT-3.5-Turbo and GPT-4, and perform finetuning. Different from them, our paper shows that even without strong LLMs like GPT-3.5-Turbo and GPT-4, we can generate synthetic data for self-debugging using much smaller open pretrained/instruct models or the same LLM itself from the additional results from this rebuttal, and they are quite effective. We also proposed a novel RL training with explanation and execution reward in our paper.

5. Details on training data

Here is a summary of the APPS dataset and the number of problems that has at least one correct solution (either correct initial solution or correct refinement from GPT-3.5-Turbo) based on difficulty levels:

For the CodeContests dataset, the data is from multiple sources and difficulty levels are not comparable. We give the sample distribution from CodeChef with difficulty levels 1-4:

Problems without any correct initial solutions or refinements are discarded from the training data.

Thanks for the response.

The reviewer might have some misunderstanding of the explanation evaluation ... SFT/RL training significantly improves the explanation quality

From what I can see, the absolute ratings from developers is below 3 on average on a 1-5 scale. While the performance improvement from untrained model baseline is considerable, the absolute scores are still not great.

I suspect a reason why absolute numbers are low is that not all explanations lead to refinement and the average rating might be higher in the second scenario.

The results in Appendix A.4.1 show that even when training with strong code instruction data, the model does not obtain self-debug capability out-of-the-box and it only refines up to 3% of its wrong solutions

Perhaps I was not clear enough -- this work depicts considerable mutliturn improvements from RL training (48% to 57% pass@1 on HumanEval for CodeLLama-7B). My concern is that if the authors start their RL training with a stronger model (say OpenCodeInterpreter-CL-7B ) which already has 70+ pass@1 on HumanEval will the RL training be as effective? For example, OpenCodeInterpreter-CL which trains models with multi-turn SFT dataset only achieves 3% improvements from 72 to 75. This makes it challenging to interpret the performance improvements achieved in this paper.

We thank the reviewer for insightful suggestions and questions.

1. Data collected from the same model itself

We provide the experiment results using the synthetic data generation from the model itself for CodeLlama 7B.

We highlight some results here. Below is the CodeLlama-7B SFT/RL using the data collected from itself, evaluated on MBPP+ and HumanEval+.

The full results are in our attached author response PDF file, Tables 1 and 2. Results show that self-taught SFT and RL also achieve big improvements. CodeLlama-7B SFT/RL models achieve up to 5% improvement in self-debugging, compared with the baseline prompting method, and the model trained with code generation data only. Though comparing the experiments using data from CodeLlama 34B and GPT-3.5-Turbo, the improvement is less.

2. Pass@K+1 versus refinement Pass@K

This is a very interesting point. We evaluate the Pass@2 of the initial solution and Pass@1 after one round of refinement.

Below is the result when the models are not trained (the prompting baseline). We do observe that Pass@2 is better than Pass@1 after refinement. This shows that using a prompting approach to self-debug is not effective.

However, after the models are trained using our pipeline. The refinement Pass@1 is clearly higher than the Pass@2 of initial solutions. This shows that the model’s self-debugging performance is bad without training. The prompting approaches proposed by existing works such as (https://arxiv.org/pdf/2306.09896 and https://arxiv.org/abs/2304.05128) are not as effective on open-sourced LLMs. This experiment further supports our motivation to train LLMs to self-debug and proves the effectiveness of our approach.

3. CodeBLEU in RL reward

We find that only using binary execution feedback as a reward does not train the model properly as the reward is too sparse. That is, a completely wrong solution will get the same reward as an almost correct solution, which hurts the RL training. Although the CodeBLEU reward is weak, we find that having it helps stabilize the training by densifying the reward distribution. This is the main reason we introduce the CodeBLEU score in the reward.

4. Roberta embedding for text similarity

Judging the correctness of code explanation is non-trivial. We use the Roberta model (https://huggingface.co/sentence-transformers/all-roberta-large-v1) that has been massively fine-tuned for text similarity using 1B sentence pairs.

We try to analyze the reliability of this approach as shown in Figure 3(c) in the paper. The similarity can separate the explanations that lead to correct and wrong solutions most of the time.

A potential alternative approach could be using powerful LLMs such as GPT-4 to rate the code explanation, however, this is not scalable enough the handle our RL training data.

5. Code generation improvement

It should be noted that the SFT training includes code generation data (provided in the original MBPP/APPS/CodeContests) + self-debug data. It could be the code generation data improves the initial solution generation. We will make this clearer in our experiment setup.

We compare our approach with fine-tuning using purely code-generation data. The results are in Appendix A4.1 Table 9 in the paper.

6. Evaluation of APPS and CodeContests

This is a good suggestion and we plan to add the results to the final version if accepted.

Below are the StarCoder-15B’s results on APPS and CodeContests. Results of the other two backbones can be found in our attached author response PDF file (Tables 4 and 5). Due to the large amount of test samples in APPS (5000), we only test the other two backbones on a subset of 200 samples. We plan to complete the evaluation and add it to the paper.

On APPS and CodeContests, the prompting baseline also only refines very few solutions, and the improvements brought by refinement are marginal. However, with our SFT and RL-trained models, we see significantly stronger self-refinement ability. The final Pass@k is also about doubled from the prompting baseline.

We thank the reviewer for insightful suggestions and questions.

1. APPS and CodeContests Evaluation

This is a good suggestion and we plan to add the results to the final version if accepted.

Below are the StarCoder-15B’s results on APPS and CodeContests. Results of the other two backbones can be found in our attached author response PDF file (Tables 4 and 5).

On APPS and CodeContests, the prompting baseline also only refines very few solutions, and the improvements brought by refinement are marginal. However, with our SFT and RL-trained models, we see significantly stronger self-refinement ability. The final Pass@k is also about doubled from the prompting baseline.

2. Preference learning (DPO/KTO) vs PPO

Preference learning like DPO or KTO has the advantage of its simplicity without the need for a reward function or reward model. In the setup of self-debugging with execution feedback in this paper, we could construct preference data in such a way: the fix that passes unit tests is preferred over the one that fails unit tests. Such preference data could also work, but might seem less direct as execution feedback/reward is easily obtained from the execution engine, unlike the human preference setup. One advantage of our RL training is that we can assign different rewards to different parts of the sequence, i.e. the explanation reward on the explanation part and the execution reward on the generated fix part.

In this work, our focus is exploring how to train LLMs for self-debugging and bug explanation with the lack of training data. Our proposed data collection pipeline and SFT, RL (PPO-based) training do outperform existing prompting approaches significantly. Exploring alternative RL algorithms could be interesting future work in this domain.

3. Teacher models performance

We provide the comparison with the teacher models in our attached author response PDF file.

Comparing it with Table 2 in the paper, with GPT-3.5 as the teacher, CodeLlama-13B SFT/RL cannot outperform GPT-3.5, which could be because GPT-3.5 is a much stronger teacher model.

Comparing it with Table 10 in the paper, with CodeLlama-34B as the teacher, CodeLlama-13B SFT/RL sometimes outperforms the CodeLlama-34B teacher (e.g. in HumanEval+ pass@1 56.24% vs 48.51%, and in MBPP+ pass@1 56.60% vs 53.19%).

4. Comparison with “Teaching LLMs to Self-Debug”

The method in “Teaching LLMs to Self-Debug” is the prompting baseline that this paper refers to in Tables 2, 4, and 7. We directly compare the results of our proposed method with this baseline in our experiments, and the results show that such a prompting approach cannot work as well on open-sourced LLMs.

The most notable difference is that “Teaching LLMs to Self-Debug” investigates the self-debugging of commercial LLMs by prompting, while our paper investigates how to improve open-source LLMs’ self-debugging capability via training.

Another difference is that our paper tries to generate the explanation of the bug to help humans and LLMs better understand the reasoning, while the paper “Teaching LLMs to Self-Debug” tries to generate an explanation for the code instead of reasoning the bug.

5. GPU hours

The GPU hours for training:

Experiments are conducted on 8 NVIDIA A100 GPUs, each with 40GB of memory

6. Generate 𝑦𝑤′ given 𝑥

If we understand correctly, the suggestion is generating the correct refinement (𝑦𝑤′) given only the problem description (𝑥). We think this is essentially fine-tuning with code generation data.

We compared our approach (framing code generation as an explanation + refinement) with fine-tuning with code generation data only, and the results are shown in Appendix Table 9.

We find fine-tuning for code generation improves the initial solution, achieving comparable or sometimes higher Pass@1 than our approach. But the model’s self-debugging ability is not improved, and the model still cannot benefit from self-debugging. The model trained for code generation cannot successfully self-debug very often and eventually is surpassed by our approach dramatically. We hope this confirms the benefit of “framing code generation as an explanation + refinement task” over simply training a better code generation model.

7. Separate code generation and refinement models

Training a separate dedicated debugging code refinement model is definitely an option. The main motivation of the paper is to improve LLM’s self-debugging capability as one part of the capabilities of LLMs, and not be over-specialized so that the training recipe can directly be incorporated in practice.

8. Generalize to other aspects

We think that the method can be generalized to improve other aspects of code such as readability and secureness. The key lies in how to properly design the reward with respect to each aspect. The readability and secureness are most likely to be determined by a set of static analysis rules. Combining all aspects together to obtain a final reward is necessary and used to provide feedback to the training.