RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning

We thank the reviewer for their valuable comments, and are grateful that they acknowledge the effectiveness of our method and "appreciate the clarity and organization in the presentation of the results".

We'd like to respond to the weaknesses pointed out to the reviewer individually:

• Limited Practical Application: We emphasize that our paper is chiefly concerned with grounding LLMs in execution feedback, which we identify as a weakness in current state-of-the-art models. We argue that addressing this weakness is crucial when deploying LLMs as agents interacting with computer systems, which is an emerging and promising area. Extensions to applications without execution feedback are definitely interesting but clearly outside the scope of our submission. We will however update our recommendations for possible future work to encompass such scenarios.

• Advantage Computation: We train the critic LLM on a per-turn level, i.e., given a prompt $c_t$ (for the response $a_t$) and we train it to predict the reward-to-go at $c_t$. We will extend Appendix 1 with the value function loss (image).

• Effectiveness of Multi-turn Feedback: We believe the experiments in Section 3.3 with random execution feedback support our conclusion that (L495ff) that better initial solutions and an increased diversity of responses are a major contributor to the performance gains, but also that we successfully obtained grounding in execution feedback as it increases the reliability of arriving at correct solutions (pass@1 vs. pass@10 in Figure 4(a)). Furthermore, we observe that the 70B model can improve its scores significantly when provided access to feedback from private tests (Appendix B.2).

  Regarding the additional experiments suggested by the reviewer, we would like to note that we end an episode once public tests are passing; hence, the model would internalize that any re-prompting implies that previous solutions are considered wrong. Adjusting the reward signal as proposed would change the optimal policy to output replies that fail one of the public tests and save the correct response for the final turn. However, we will add a further 8B ablation where we replace execution feedback with "Give it another try", disable early stopping based on public test results, and evaluate the last response only (see global response).

We thank the reviewer for suggesting additional related work. "Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning" is an interesting work and indeed very closely related to "StepCoder" mentioned by reviewer QLXX in that both use ground truth solutions to form a curriculum to help with exploration. We maintain that work on curricula for RL fine-tuning is orthogonal to our submission, though. Our work does not specifically target LLM reasoning capabilities.

We interpret the remark regarding the usage of "some specially designed reward functions" as using execution as an RL reward. For context, we would like to add that agents situated in a simulated environment are the natural domain of RL algorithms. Our reward function is purposefully kept simple, and tuning it could potentially yield further improvements but is outside of the scope of this submission.

Regarding the reported "lack of experiments on more models to verify the generality of the method", we perform experiments on both 8B and 70B Llama 3.1 models, as well as the weaker 8B Llama 3.0 model. While we can sympathize with calls to experimental validation on further models as well as benchmarks we also maintain that the current set of results clearly demonstrates the efficacy of our method, and we hope to inspire further work on grounding LLMs for domains where it is crucial to take execution feedback into account.

We thank the reviewer for their valuable feedback and appreciate them acknowledging that our method addresses a "critical limitation of current LLMs", and highlighting the "significant sample reduction" and "improved performance and efficiency" that we achieve. We respond to the raised weaknesses as follows:

1. We are thankful for the additional related work pointed out by the reviewer, and would like to draw the following comparisons:

• "CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning" is discussed in the related work section (L457ff); however, our manuscript misinterpreted this work and we will revise this part. While they train a single-turn model with RL, they also train two further models: a "critic" to gauge correctness of programs and a "repair" model that maps faulty code to respective ground truth solutions. We agree that this is a particularly relevant approach to our work, and we will include the combination of a single-turn model followed by a dedicated repair model as another baseline (see global response). We also point out that our RLEF method achieves high multi-turn performance with a single model only.
• "Improving Code Generation by Training with Natural Feedback" targets the integration of feedback from humans, which is relevant for user-facing chat applications such as ChatGPT but less so when employing LLMs as agents.
• "Training Language Models to Self-Correct via Reinforcement Learning" (SCoRe) is discussed in the related work (L468ff); this is concurrent work released 9 days prior to the submission deadline. First, their method does not incorporate execution feedback and instead relies on the model to detect its own mistakes. We believe that when execution feedback is available, such as in code generation or in many "agentic" tasks, it should be integrated for improved accuracy. Second, SCoRe requires two separate training stages whereas our RLEF method consists of a single training stage
• The main contribution "RLTF: Reinforcement Learning from Unit Test Feedback" over CodeRL (see below) is a fine-grained reward function; our approach achieves significant gains with a simpler reward function and we regard the tuning of reward functions as orthogonal to our work.
• "Execution-based Code Generation using Deep Reinforcement Learning" is discussed in the related work (L460); they include a dependence on ground truth solutions which are not required for our method
• "StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback" introduces both a curriculum based on ground truth solutions (which we do not require) and masking of code not subject to execution during optimization. We regard both techniques as orthogonal to our work, i.e., they could potentially yield further improvements when applied to our multi-turn setting.
• "B-Coder: Value-Based Reinforcement Learning for Program Synthesis" describes a value-based RL method for LLMs, and uses the public CodeRL model in their experiments.

We will update the related work section of the paper to encompass all references above.

2. We point out that the CodeContests training set and APPS test set have overlaps (cf. https://arxiv.org/abs/2203.07814 Appendix C.6) so that we cannot evaluate our existing models on APPS. Furthermore, APPS does not advertise public test cases explicitly, which would render experiments incompatible with prior work.

3. Our observation that SFT is less effective compared to RL is consistent with other findings from the literature. E.g., https://arxiv.org/abs/2404.10719 showing PPO > SFT, PPO > DPO on CodeContests. We also note that in our SFT experiment we use rollouts of the stock Llama 3.1 70b Instruct model since there is no ground truth data available for iterative code synthesis.

We appreciate the insightful feedback of the reviewer and are happy that they recognize the effectiveness of our method and the thoroughness of our experimental analysis. Regarding the stated weaknesses of our submission, we would like to respond as follows:

1. We maintain that our method is not strictly an empirical extension to previous work and refer to our global response where we outline the planned changes to our related work section. In summary, the novelty of our paper is the combination of optimization with RL over multiple turns and integration of execution feedback in a conceptually simple and highly effective framework, requiring a single model and training pass only. As stated in the global response, we will add results with a CodeRL-like repair model (the same approach used in the later RLTF paper) as an additional baseline.

2. We agree with the reviewer that the quality of public unit tests (i.e., those used for feedback) can have a significant effect on inference-time efficiency. Notably, the public tests in CodeContests do in many cases cover only example inputs and outputs; they are usually already provided in the problem description. In Appendix B.2 we show the effect of using the more extensive private tests for execution feedback. Our original submission contained an outdated result in this section which we will update. For the 70B model we included in Table 1, feedback from private tests improves 1@3 solve rates from 37.5 to 41.2 and from 40.1 to 41.2 on the validation and test set, respectively.

As the reviewer points out, existing and future work on automatic test generation is highly relevant to our work and could help render our method more general, i.e., in cases where no tests are provided by users at inference time. While such use cases are not the main target of our submission, we will add a corresponding note to the Conclusion/Limitations section.

3. (a) We view our design decisions regarding the action space for policy and value learning (L162f) as implementation details that require consideration and might be subject to the specific application when training LLMs in multi-turn dialogs with RL. As such we aimed to clearly explain our choices, but also feel that they are not central to the overall method. We will complete A.1 with details regarding the value function loss (see response to kdLS below).

(b) We justify the use of the n@k metric in 3.1 (L196ff.) and indeed we argue that using standard pass@1 metrics when allowing for multiple generations per solution is a key shortcoming of prior work. The legend of our main results (Table 1) highlights the key fact, namely that k corresponds to the sample budget.

We are further happy to answer the reviewer's questions as follows:

1. Robustness to test cases: as pointed out above, the public test cases of CodeContests typically cover only simple cases. We think that an "optimal amount" of feedback would be highly dependent on the concrete model and benchmark and would thus refrain from giving specific recommendations. In practical applications, there are also further trade-offs to consider such as the number of turns and a model's context length.

2. Effect of turn limit: We would like to point out that we employ a dynamic turn limit in that successful execution of public tests will end the dialog with the LLM. In Figure 4, we show the number of turns can be increased at inference time. We believe that a focus on compute budget is crucial, though, and in Figure 4(b) we show that increasing the inference turn limit to 5 yields compute-optimal results.

3. Hybrid approach to value estimation: We found that in preliminary experiments, optimizing the policy at the turn level (i.e., PPO with `pi(a_t|c_t) = \prod_i=1,n pi(a_{t,i}| c_t, a_{t,1}, ... a_{t,i-1})` for a response a_t consisting of n tokens) resulted in poor performance. With a token-level value reward, we found that the KL penalty biases the model to unreasonably short generations in intermediate turns. Averaging the KL penalty as described in L170ff is an effective remedy and invites modeling with a per-turn reward at the turn level. We will add an ablation to the paper where we use the current per-turn rewards but learn a per-token value function to predict the same return-to-go for each token of a response.

We have now updated the submission with a new version of the manuscript which includes the changes mentioned above. We added the new results (Table 4b in Appendix B), referred to from the end of Section 4.3. The fourth and fifth paragraphs of the related work section have been updated (L460ff, L478ff).