We sincerely thank the reviewer for recognizing our contributions and providing detailed feedback that helps refine our work. Below, we address your insightful comments and suggestions.

Ablation Shows PLUM Works For Various (Especially Less-Powerful) Test Generators

We conducted an ablation study using alternative test case generators to evaluate PLUM's scalability and cost-efficiency. The study included open-weight models like Llama-3.1-70B and cost-effective API-based models such as GPT-3.5-Turbo, along with the lowest-cost models from OpenAI (GPT4o-mini) and Anthropic (Claude3-Haiku). The results consistently show performance improvements, even with less powerful test case generators, demonstrating PLUM's robustness to generator variability and its applicability under different resource constraints. For detailed results, please refer to the global response and our revised draft.

Proving PLUM’s Effectiveness on Stronger Models

We evaluated PLUM on stronger models: Qwen-2.5-Instruct-14B and Qwen-2.5-Coder-14B. These models already exhibit strong performance on coding benchmarks, and have undergone more sophisticated alignment techniques.

Our results show that applying PLUM further improves these models. This proves that PLUM can enhance the performance of stronger models and highlights its complementary effect when combined with other alignment techniques. The results have been also included in the revision.

Pseudocode Typo

Thanks for pointing out the mistake! Your understanding is correct that “Solutions passing all test cases are used as the chosen solutions, and those failing at least one are rejected solutions.” We have fixed this in the revision.

Motivation

We appreciate your insightful comments on the motivations for preference learning within the coding domain. While preference learning has traditionally been applied in domains such as human alignment and detoxification, recent work has demonstrated its potential in more complex, outcome-driven tasks across areas like reasoning, mathematics, and coding, where achieving functional correctness is paramount (Dong et al., 2023; Pang et al., 2024; Yuan et al., 2024; Xiong et al., 2024; Lai et al., 2024; Xie et al., 2024; Weyssow et al., 2024).

In the context of PLUM, we focus on encoding functional correctness directly within the preference objective, which offers a new perspective. Our method leverages test-based signals to ensure correctness through a preference-guided approach, thus removing the need for complex reward model setups while addressing potential over-optimization and distributional shift issues. Additionally, our comparative results, including contrastive learning baselines, underscore PLUM's efficacy in reliably improving functional correctness, as it harnesses the advantages of preference learning to a degree not yet fully explored in the code generation domain.

Comparison to DeepSeek-Coder-V2 and CodeRL

CodeRL and PLUM represent fundamentally different methodologies for improving code generation. CodeRL is an iterative code-refinement framework build upon reinforcement learning principles. It evaluates and iteratively adjusts failed solutions with a trained critic network. This involves continuous refinement, leveraging test cases during inference to repeatedly improve solutions until they pass.

In contrast, PLUM operates as an on-policy preference learning framework designed to improve a model's intrinsic ability to generate correct code based on natural language prompts, bypassing the need for external critics or iterative refinement. By leveraging automatically generated test cases as a lightweight, consistent feedback mechanism for training, PLUM enables on-policy training without reliance on offline data, mitigating the distributional shift commonly seen in offline models​.

Finally, the works cited by the reviewer and within our paper neither diminish nor overlap with PLUM’s unique contribution, as they are concurrent or fundamentally distinct. PLUM leverages on-policy preference learning with test cases to directly enhance model performance during training using preference learning algorithms. Our framework is distinct in its simplicity and robustness, bypassing reward models and reinforcement feedback entirely. This approach, combined with automated test case generation for scalable, policy-consistent preference data, establishes PLUM as an original solution that fills an unaddressed gap in nl-to-code generation.

Clarifications on the SFT Baseline

To avoid possible confusion, we clarify that the baseline rows we refer to as “Baseline” in the result tables represent existing models that have been fine-tuned on their respective datasets, not models we further fine-tuned on other data. Therefore, no additional out-of-domain adaptation effects have been introduced. In the rejection-sampling fine-tuning (RFT) baseline, we filtered out responses that did not pass initial test cases, leaving only correct responses. The model is then SFTed on the same set of questions as PLUM paired with these positive responses. By comparing PLUM to RFT, we demonstrate that the performance gains of PLUM are not merely due to filtering out those “out-of-domain” questions, but through learning to distinguish between correct and incorrect responses through on-policy training. We would also like to emphasize that PLUM is designed as an enhancement layered on top of the supervised fine-tuning (SFT) stage. The SFT models used in our study were already extensively fine-tuned on large and diverse instruction datasets. PLUM adds a further improvement step, where the preference learning framework, augmented with test cases, fine-tunes the model to prioritize functionally correct responses more effectively.

Questions to Reviewers

We appreciate the reviewer’s suggestions and reference pointers. However, we would be grateful if the reviewer could provide the titles and/or links to those papers for further clarification, since all references seem to be missing. (Le et al. 2022, Gorinski et al. 2023, Liu et al. 2023, Miao et al. 2024, Zhang et al. 2024, Gee et al. 2024)

Readability / Clarity

Thank you for your suggestion to improve the clarity of the tables and figures. We will reorganize the tables to follow the order of introduction in the main text, ensuring each experiment and its results are intuitively located near its description. Furthermore, for Figure 2, we will include additional annotations to clarify that the purpose of this figure is to contrast non-executable signals against those tested with our on-policy approach. The result actually shows PLUM’s effectiveness in boosting functional accuracy by leveraging test cases, rather than depending solely on signals from compilation failures.

References:

[1] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. URL: https://arxiv.org/abs/2304.06767

[2] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024. URL: https://arxiv.org/abs/2404.19733

[3] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing llm reasoning generalists with preference trees, 2024. URL: https://arxiv.org/abs/2404.02078

[4] Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, Chi Jin, Tong Zhang, Tianqi Liu. Building Math Agents with Multi-Turn Iterative Preference Learning, 2024. URL: https://arxiv.org/abs/2409.02392

[5] Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs 2024. URL: https://arxiv.org/abs/2406.18629

[6] Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P. Lillicrap, Kenji Kawaguchi, and Michael Shieh. "Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning. 2024. URL: https://arxiv.org/abs/2405.00451

[7] Martin Weyssow, Aton Kamanda, and Houari Sahraoui. CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences. 2024. URL: https://arxiv.org/abs/2403.09032

Dear Reviewer RrMw,

Thank you for your valuable feedback on our submission! We have attempted to respond to your comments in our rebuttal and would appreciate any further clarifications or discussions to enhance our work.

Look forward to further discussions! Thanks again for your time and efforts!

Given the lack of clarity on the exact citations on my part, I appreciate the difficulty the authors had in addressing some of my concerns. I am clarifying these citations below, could you please clarify the novelty of your unit-test derived signal on the context of this previous work and why these could not constitute baselines for your approach?

Le et al. 2022 -> CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

Gorinski et al. 2023 -> Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis

Liu et al. 2023 -> RLTF: Reinforcement learning from unit test feedback

Miao et al. 2024 -> Aligning CodeLLMs with Direct Preference Optimization

Zhang et al. 2024 -> CodeDPO: Aligning Code Models with Self Generated and Verified Source Code

Gee et al. 2024 -> Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency

Concurrent Works

We thank the reviewer for providing the pointers. Some of these works are concurrent to ours and should not negatively impact PLUM’s contribution:

• Miao 2024: Aligning CodeLLMs with Direct Preference Optimization

  https://arxiv.org/pdf/2410.18585

  Arxiv date: Oct 24, 2024.

• Zhang 2024: CodeDPO: Aligning Code Models with Self Generated and Verified Source Code

  https://arxiv.org/abs/2410.05605

  Arxiv date: Oct 8, 2024.

• Gee 2024: Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency

  https://arxiv.org/pdf/2406.12502

  Arxiv date: Jun 18, 2024.

Differences From The Rest Of Related Works

The reviewer correctly pointed out the use of test cases for code LMs has been previously explored. The key contribution of this work is a novel on-policy preference learning framework for code LMs, and discuss its differences from previous works below (CodeRL has already been covered in our previous response).

RLTF

RLTF is an extension of CodeRL. Its key is to use the enumeration of various kinds of errors and carefully handcrafted reward values for training the policy Result-wise, the handcrafted RL reward reduces syntax errors and other superficial issues. However, it fails to effectively ensure that the solutions align with NL-specification (Figure 2 of RLTF paper).

In contrast, PLUM excludes syntax errors and other statically invalid programs, and demonstrates that focusing on run-time feedback by execution over test cases could significantly improve code LMs’ NL-to-code capabilities across benchmarks, besides, instead of relying on reward models as in RLTF, PLUM learns directly from the execution results over test cases with algorithms like DPO and KTO.

Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis

They labeled programs with a static-analysis-based test case annotation tool (EvoSuite [1]) to augment the MBPP training split. EvoSuite focuses on ensuring structural coverage—i.e., making every branch of the program runnable—rather than aligning with natural language specifications. The resulting synthetic dataset caused the model to perform worse than the pre-trained checkpoint (as shown in Table 1 of this paper).

In contrast, we developed a scalable method that leverages LLMs to produce high-quality test cases for programming instructions, enabling the collection of on-policy preference data. Leveraging the signal provided by execution against the test cases, our method directly optimizes the model’s functional correctness for NL-to-code tasks. Through extensive experiments, we clearly demonstrate the effectiveness of our approach in improving code LMs.

Additional Remarks:

EvoSuite is limited to strongly-typed languages such as Java, making it incompatible with dynamically-typed languages like Python. Additionally, the extremely low yield rate in Gorinski 2023 —estimated at just 5.57%—renders it impractical for generating test cases. For instance, producing approximately 2,000 examples would require around 360 hours of computation time, assuming one second per test case generation.

Summary

In summary, PLUM as a novel test-case driven preference learning approach provides fresh insights beyond existing works, showcasing the importance of on-policy preference data. Beyond that, its strong empirical performance further strengthens its value.

[1] EvoSuite: automatic test suite generation for object-oriented software. https://dl.acm.org/doi/10.1145/2025113.2025179

I would like to thank the authors for providing their insights on the related work, I certainly appreciate their efforts so far. I think the paper would benefit from the inclusion of this discussion, as it places the presented work in the proper context. I agree that much of that work can be considered concurrent and thus should not diminish the paper's contribution (in my original review I suggested that these should be included in the paper for discussion, not comparison).

However, the novelty of the approach is still impacted by the fact that non-concurrent related work has already established that code models can be improved by exploiting on-policy signal assigned a reward through unit tests (either a continuous reward for RL or a binary for PO). At this point it is also important to note that previous work has also established that LLMs can be used to automatically produce unit tests and that their inclusion (either directly in the input or as a filtering step) can improve performance.

Xiong et al 2024 "The Program Testing Ability of Large Language Models for Code"

Li et al. 2023 "Towards enhancing in-context learning for code generation."

Chen et al. 2023 "CodeT: Code Generation with Generated Tests"

With all that said, to this reviewer, it seems that all individual contributions of the paper (on-policy learning for code, automatic generation of unit tests, filtering of on-policy signal by unit tests) have already been proposed and supported by previous work. Can the authors please help me clarify what is unique to this method, beyond the combination of previously established on-policy methods and signal on code? If the authors claim that nothing is specifically unique, but the proposed variations on these methods perform better, then direct comparisons and ablations are needed.

We thank the reviewer for their discussions.

...non-concurrent related work has already established that code models can be improved by exploiting on-policy signal assigned a reward through unit tests (either a continuous reward for RL or a binary for PO).

We kindly point out that PLUM innovatively leverages test-case-driven preference optimization for code. To the best of our knowledge, no prior works have demonstrated test-case signals can improve LMs’s functional correctness using a reward-model-free preference learning (xPO) technique. As the same reviewer RrMw correctly pointed out in their earlier comments, ‘preference optimization is (was) generally used to align LLM output to human values, e.g. reduce harmful and toxic expressions’, it remained elusive before we proposed PLUM whether and how preference learning techniques can go beyond aligning to human preferences and improve the code LMs’ functional correctness when using test case feedback. PLUM addresses this critical gap and proposes an effective solution, and therefore shall be considered novel and valuable.

Additionally, although ‘RL or contrastive learning over online signal seem more appropriate’ (quoted from the earlier review by RrMw), we demonstrated the strong performance of PLUM, which brings efficient, scalable and effective on-policy improvement for code LMs without the need for reward-models or reward handcrafting.

At this point it is also important to note that previous work has also established that LLMs can be used to automatically produce unit tests and that their inclusion (either directly in the input or as a filtering step) can improve performance.

These referenced works fundamentally differ from PLUM since they use test cases to rerank / refine solutions during inference time, not using these to establish training signals as PLUM did. Indeed, as we have acknowledged, our test case generation approach partially builds upon the insights that LLMs exhibit the ability to generate high-quality test cases based on its understanding of natural language instructions. However, we clarify that the fundamental difference in the use of these test cases distinguishes PLUM as a training framework from the inference time solution-reranking techniques. Therefore, our contribution of proposing a preference learning framework that can build better code LMs from test cases is novel and valuable.

We summarize the works mentioned by the reviewer below.

Xiong et al. 2024: Evaluates and improves LLMs' ability to generate test cases for code, achieving pass rate improvements on HumanEval+.

Li et al. 2023: Proposes AceCoder, an in-context learning method combining test-based analysis and retrieval, significantly enhancing code generation performance.

Chen et al. 2023: Introduces CodeT, an in-context learning method that generates and uses test cases to improve code correctness, achieving state-of-the-art results on multiple benchmarks.

all individual contributions of the paper (on-policy learning for code, automatic generation of unit tests, filtering of on-policy signal by unit tests) have already been proposed

The reviewer correctly notes that PLUM builds on prior insights as all research papers do, which we have acknowledged in the paper. But those works typically address a different problem from PLUM.

PLUM uniquely integrates these components into a unified, scalable framework for on-policy preference optimization of code LMs. Unlike prior work, PLUM demonstrates:

• The effectiveness of on-policy preference optimization for code generation.
• The utility of LM-synthesized test cases in providing high-quality training signals.
• The critical importance of on-policy learning, supported by empirical evidence.

Our contributions fundamentally differ from existing techniques, yielding novel insights and performance improvements that prior work neither achieves nor enables.

Request To Discuss Concurrent Works

We appreciate the pointers from the reviewers. In fact, DeepSeek-Coder-V2 has been discussed in the paper as a concurrent work. It was infeasible for us to include works that were public (on Arxiv) after the ICLR deadline (Miao 2024: Aligning CodeLLMs with Direct Preference Optimization, Arxiv date: Oct 24, 2024. Zhang 2024: CodeDPO: Aligning Code Models with Self Generated and Verified Source Code, Arxiv date: Oct 8, 2024) at the time of submission. That said, we are happy to discuss them in future revisions!

Dear Reviewer RrMw,

Thank you again for the time and effort to review our paper and to initiate discussions that help clarify our contributions!

With 2 days left until the end of the discussion period, it would mean so much to us if you could take a look at the added discussion and materials so that you can re-evaluate our work with those clarifications and update your scores if you find appropriate.

Please let us know any remaining questions / concerns surrounding PLUM's contributions! We are also more than happy to address your further questions and concerns!

As suggested by the reviewer, we applied RLTF on Magicode-DS-6.7B and CodeQwen-1.5-7B-Chat, following their dataset choice (APPS). We appreciate the reviewer's patience, as RLTF (and the precedent RL works) are typically proposed before the prevalence of instruction-tuning for code-LMs!

Results and analysis

The result is presented below:

We notice that RLTF in its original form might not be particularly effective to improve modern instruction-tuned code LMs. In addition to the restrictions brought by the training distributions, sparsity of reward could be another reason. Notably, very recent works like Dai (2024) noted on the sparse reward signals inherent in RLTF and demonstrate its limitations in further improving instruction-tuned code language models through experiments.

Re-emphasizing difference between PLUM and RLTF

Again, we emphasize the distinctions between PLUM and RLTF. RLTF focuses on adapting pre-trained code LMs to specific datasets (e.g., APPS) through reward design. In contrast, PLUM leverages on-policy preference learning to improve instruction-tuned code LMs by directly leveraging the execution feedback, offering a simpler, more scalable, and effective alternative to RL approaches without the need for RL.

Let us know if you have further questions or concerns that we could take the last chance to clarify within the last day of discussion period!

[1] Dai et al. Process Supervision-Guided Policy Optimization for Code Generation. https://arxiv.org/pdf/2410.17621

Thank you for the insightful feedback, which has helped clarify key aspects of our work.

Limitation to Python

We appreciate the reviewer’s comment on the scope of our work, which currently focuses on the Python programming language. The proposed PLUM framework, however, is designed to be broadly applicable across programming languages. Our approach utilizes test cases generated based on natural language instructions and verifies code through test execution rather than relying on any language-specific syntax or semantics unique to Python. The focus on Python was primarily for the availability of well-established training datasets and ease of comparison and reproducibility, as Python is widely used in both code generation and evaluation benchmarks (e.g. HumanEval, LeetCode). We are confident that the methods can be adapted to multilingual settings by training models on diverse programming languages and modifying test generators to meet language-specific requirements.

Dependency on GPT-4

Thank you for this insightful feedback. Our use of GPT-4 as a generator model was based on its superior generation capabilities; however, our framework does not rely on any features specific to GPT-4. The model was chosen purely for practical reasons, and any other language model capable of generating syntactically correct and logically coherent code solutions would also be effective within our framework. Open-source models can replace GPT-4 for test case generation, allowing for scalable expansion while retaining the effectiveness of our approach. Our experiments verify that the robustness of our method stems from the structured preference learning pipeline rather than reliance on GPT-4’s unique characteristics.

Clarification of Model Sizes

We appreciate the constructive feedback on the clarity of the paper. To clarify the reviewer’s concern regarding model sizes, we aim to illustrate that our ablation studies leverage responses from models of varying sizes specifically to demonstrate the comparative benefits of on-policy data. We did not actually train models of multiple sizes, but rather utilized the outputs from both smaller and larger models as part of our on-policy versus off-policy comparisons. This setup is intended to evaluate and emphasize the effectiveness of our preference learning method across different model capacities without requiring separate training instances. Specifically, the models we trained are all 7B-scale language models. CODEQWEN-1.5-CHAT refers to the 7B CodeQwen-1.5-Chat model, Magicoder-DS (-S-DS) refers to the Magicoder models trained from DeepSeek-Coder-6.7B base model, Magicoder-CL (-S-CL) refers to the Magicoder models trained from CodeLlama2-7B base model. Same applies to OCI (OpenCodeIntepreter). We will update the paper to make it clear!

Consistency of Test Cases and Reference Solutions

We appreciate the reviewer’s inquiry into test case consistency with reference solutions. In our approach, consistency between test cases and reference solutions is crucial, but as highlighted in works on software testing, achieving perfect alignment between natural language specifications and test cases remains an open research question. We adopt a practical heuristic by assuming that consistency between generated solutions and passing test cases implies correctness. Our experimental results demonstrate that this assumption holds well across our chosen datasets, which suggests that it is a reasonable approach under current constraints.

We hope that addressing these points will provide additional insights into the PLUM framework, potentially enhancing the reviewers' understanding and appreciation of our contributions and reflecting positively in their assessment.

Dear Reviewer Du4Y,

Thank you for your valuable feedback on our submission! We have responded your comments in our rebuttal and would appreciate any further clarifications or discussions to enhance our work!

Look forward to further discussions!

Thanks again for your time and efforts.

Language expansion

We agree that Python's ease of environment setup significantly contributes to its popularity in execution-based evaluation. Many recent papers like SelfCodeAlign (NeurIPS’24) [1] and more recent recent RLEF from MetaAI [2] also focus solely on python. That said, setting up test environments especially for function-level code is also straightforward, one could build execution environments like MultiPL-E [3] did!

For example, if one wants to test java functions, they could simply install JDK, and prepare the file as shown below:

public class AddTest {
    public static int add(int a, int b) {
        return a + b;
    }

    public static void main(String[] args) {
        assert add(2, 3) == 5 : "Test failed";
        assert add(0, 0) == 0 : "Test failed";
        System.out.println("All tests passed");
    }
}

And run

>>> javac AddTest.java
>>> java AddTest

To get the results!

References

[1] Wei et al. 2024. SelfCodeAlign: Self-Alignment for Code Generation. https://arxiv.org/abs/2410.24198.

[2] Gehring et al. 2024. RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning. https://arxiv.org/abs/2410.02089

[3] Cassano et al. 2022. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation https://arxiv.org/abs/2208.08227

GitHub: https://github.com/nuprl/MultiPL-E

Reliance on GPT-4

We conducted additional experiments to demonstrate the effectiveness of our framework when using other language models (both open-source and proprietary) as test case generators. We kindly direct the reviewer to our global response for detailed results.

We experimented with various test case generators, including both proprietary models with much more affordable API access and presumably less powerful than GPT-4 (GPT-3.5-Turbo, GPT4o-mini, Claude-3-Haiku), and open-weight models (Llama 3.1 70B and 405B).

We have observed consistent performance gains across the experiments. This proves that PLUM does not rely on a single powerful model as its underlying test generator, but is a robust, adaptable and scalable framework that works well under various conditions. In particular, it works under more cost-and-time efficient set-ups, making it a scalable to large data volumes.