Coarse-Tuning Models of Code with Reinforcement Learning Feedback

• “The experiments do not seem to control for the amount of compute … but it does not apply the same amount of compute as RLCF.”: We interpret the reviewer's concern about "compute" as referring to the amount of resources consumed, including the number of samples, batch size, and training iterations. To ensure a fair comparison, our coarse-tuning baselines (+Mono, +BiPolarRamp, +CodeRL) and RLCF are trained with identical hyperparameters for a given model size. This includes an equal number of samples from CodeNetJava, matching batch sizes, and total training iterations. The variations lie only in the objective function being optimized. The baseline that does not undergo coarse-tuning can be viewed as utilizing 0 compute. However it is outperformed by RLCF across all metrics for all models with just few rounds of training (see Figure 3). This underscores its efficacy as a cost-effective means of enhancing model performance for downstream tasks.

• “The proposed method requires a so-called discriminator D that is part of the grounding function …”: The choice of CodeBERT for the discriminator in the grounding function is based on its suitability for code-understanding tasks. While alternatives like GraphCodeBERT exist, we emphasize that RLCF is a versatile framework where compilers and discriminators can be interchanged for feedback. The use of a pre-trained discriminator, while not strictly necessary for RLCF improvements (as indicated in Table 4 in the appendix), does have a significant impact. As discussed in Section 8.8, this can be attributed to the pre-trained D network’s ability to enter RL-training with the knowledge of how to score responses for a given prompt and compare them. Exploring discriminators that are constructed from other LLMs cooperating with advanced static-analysis tools is an avenue for future research.

• “The results seem to suggest that … distilling "Java code knowledge" from CodeBERT.”: The observed discrepancy between the improvement in Comp@k/Exec@k accuracies and Pass@k is anticipated. Higher improvement in compilation accuracy stems from the compiler feedback that precisely localizes issues in generated responses, while pass@k relies on how well the policy has learned to understand the prompt, which in the absence of test cases to indicate functional correctness has to be estimated via some Grounding function.

  The improvements in pass@k are a result of the grounding function's design in RLCF, where the policy engages in an adversarial competition with the Discriminator. This process aims to generate responses that are equally plausible compared to ground-truth responses, and possibly more plausible. It is essential to note that such an effect goes beyond a simple distillation of CodeBERT's knowledge of Java code; instead, CodeBERT is required to understand how well the generated Java code aligns with the given prompt.

• “I'd suggest running a RLHF baseline with the HF replaced by CodeBert.”: The coordinated feedback from Compiler + Discriminator in RLCF can be indeed interpreted as replacing the Human Feedback component of RLHF. We have discussed such a comparison between RLCF and RLHF in the Related Work section. For a baseline similar to RLHF that only uses CodeBERT for feedback, we ablated the compiler signal and observed a significant divergence in learned policy from its initial distribution. Refer to Section 8.7 in the Appendix for more details and discussion.

• “Are the model descriptions for pre-trained and not pre-trained in Table 4 in the appendix mixed up?”: The terms "pretrain" and "- pretrain" in Table 4 refer to whether the Discriminator enters RLCF tuning with or without learning the Grounding, as clarified in the discussion in Section 8.8. If the terminology is unclear or confusing, we will use different terms for clarity in the context of this ablation study.

Thanks to the authors for the response.

The terms "pretrain" and "- pretrain" in Table 4 refer to whether the Discriminator enters RLCF tuning with or without learning the Grounding

I see, so the - pretrain refers to "subtracting" the pretraining aspect? Normally one would read it as a hyphen.

As discussed in Section 8.8, this can be attributed to the pre-trained D network’s ability to enter RL-training with the knowledge of how to score responses for a given prompt and compare them.

Since a substantial part of the overall improvement of RLCF comes from using a pre-trained D I would expect more analysis and discussion around it. For example how good is it at discerning correct from incorrect programs? The lack of analysis around the discriminator makes it difficult to follow why "... such an effect goes beyond a simple distillation of CodeBERT's knowledge of Java code".

Given that my concerns remain insufficiently addressed, I stick to my initial score.

• “The proposed approach is limited to larger dataset due to the fact that CODENETJAVA does not consider dependencies on user-defined packages or libraries.”: The use of CodeNetJava, which excludes user-defined packages, is a deliberate choice to streamline static analysis during training and facilitate rapid prototyping of RLCF. This dataset choice doesn't limit the applicability of RLCF, as the framework is equally applicable on larger datasets containing user-defined packages. For such datasets, incremental compiler tools like Bazel can be leveraged as they efficiently rebuild modified program parts considered during RLCF-based model training.

• “Is the proposed evaluation and experiments representative enough for other programming languages or models?”: The RLCF framework is designed to be applicable to all languages and models of various sizes. While our experiments focus on Java and a range of model sizes (from 350M to 1.5B), the extension of RLCF to other languages and models is a straightforward task, and we acknowledge this as a potential avenue for future work.

• “How can this approach work with existing pre-trained code-specific LLMs is missing?”: The selected CodeGen (Decoder only) and CodeT5 (Encoder-Decoder) models are code-specific language models (LLMs), and our study demonstrates improvements in generated code with RLCF applied to these models. The RLCF framework is adaptable and can be similarly applied to other existing code-specific LLMs.

• “Is it possible fro authors to compare the proposed method with existing state-of-the-art LLMs like GPT3 or GPT4?”: GPT models, notably GPT-3 and GPT-4, are significantly larger than the models examined in this paper. Unfortunately, their usage comes with associated computational costs, which proved difficult to procure in an academic setting. We can only ask that the reviewer evaluate our proposal on the models that we were able to run.

• “RLTF [1], which is another RL-for-code-generation technique, achieves … ~4% for MBPP)”: Comparing RLTF's reported results on MBPP with RLCF's results on MBJP might be misleading. RLCF operates in a coarse-tuning setting without access to test cases, while RLTF's reported results on MBPP involve fine-tuning with unit-test feedback. Additionally, RLCF addresses a different aspect of the ML-for-Code cycle, leveraging large-scale datasets used for pre-training where unit tests are unavailable. CodeRL, PPOCoder, and RLTF, on the other hand, are designed for fine-tuning with available test cases.

• “For instance CodeRL [2] … (for instance, CodeRL uses a critic to return localized information). Finally, I'm not sure the ablation study for CodeRL is a fair comparison … other than CodeRL” : We appreciate the reviewer's suggestion for experiments in Python, which we will consider for future work. Our Related Work section already includes a comparison with CodeRL (see last paragraph of Section 2). Regarding the ablation study with respect to CodeRL, our implementation of CodeRL follows its specified architecture where we indeed incorporate the critic network. We only removed the reward components dependent on unit-test feedback for a fair comparison with RLCF. Detailed information on the CodeRL implementation and comparison is available in Section 8.6 of the Appendix.

• “Can you provide a comparison of your methods with the 3 works cited above? It would also be good to swap out the learned discriminator with the DFG-based metric from [3] above.”: We have included a comprehensive comparison with CodeRL in the paper. Regarding RLTF and PPOCoder, we emphasize their lack of a grounding function for strict alignment to prompts. While PPOCoder utilizes AST and DFG-based matching between generated and ground-truth responses, we argue that functionally equivalent programs with different structural compositions may receive poor AST and DFG-based matching scores. Large language models (LLMs) like CodeBERT, which learn representations from a diverse set of code, are adept at handling such structural variations and possess contextual understanding of responses with respect to prompts. This enables them to recognize similarities that might be overlooked by DFG matching, which often focuses on local relationships. We will incorporate this discussion of RLTF [1] and PPOCoder [3] in the final manuscript.

• “Can you ablate the localization aspect of the compiler feedback? i.e., just return -1 for compile errors (while maintaining the discriminator).”: Ablating the localization aspect of compiler feedback by just returning -1 for compile errors, while maintaining the discriminator, resulted in significantly more compilation errors per generated response. For a detailed discussion on this ablation, we encourage the reviewer to refer to Section 8.7 of the manuscript.

• "Compiler Limitations ... deemed correct by one compiler but not by others.": We agree that different compilers might enforce different syntactic and semantic checks and could lead to differing analyses of dynamically typed languages like Python. For strongly typed languages like Java considered for RLCF, compilers are all well-tested and quite faithful to the language standard. This would typically not lead to inconsistencies.

• "From the perspective of practical implementation in developer tools the requirement to utilize both a compiler and an additional discriminator model could present significant challenges.": The discriminator and compiler are solely utilized in the model-training phase. In practice during deployment in developer tools, only the RLCF-tuned model would be used, mirroring the deployment setup of tools like Codex.

• "There is a need for more robust RL baselines such as the RLHF": Implementing RLHF, particularly with a binary reward, demands expert human judgment for model-generated samples, posing significant challenges in the context of code. We address these limitations in the Related Work section.

• "The study compares the base pre-trained CodeGen … vs better quality data": We've implemented a 'Mono' Baseline, a supervised variant trained on gold-standard responses from the dataset. This baseline uses samples that successfully compile and align with prompts, providing a comparison to gauge the impact of the RLCF technique on top of high-quality data.

• "The process of generalizing this approach across various programming languages, and different compilers might represent a challenge. If possible, add more tests or discuss it.": The RLCF framework is designed for strongly typed languages with standard compilers, such as Java. Extending it to dynamically typed languages like Python is more challenging as it requires robust static analysis tools. This is why we prototyped the method using Java.

• "Additionally, a comparison with state-of-the-art reinforcement learning techniques … effectiveness of RLCF.": We have compared RLCF against CodeRL, a strong RL baseline, showcasing improvements with RLCF. CodeRL itself relies on a dataset assembled with binary rewards from the compiler. Further details are available in Appendix Sec 8.6 for reference.