Process Supervision-Guided Policy Optimization for Code Generation

Thank you for your review and for highlighting several important points. We appreciate the opportunity to address your concerns and clarify the aspects you found unclear. Below, we provide detailed responses to each of your comments.

My main grievance with this paper is that it has very low reproducibility. Models, training data, and evaluation sets are all proprietary; we don't even learn about the size of the models. It does not seem to be a large model, though, indicated by the "lite" name and by the fact that Table 1 only compares to "mini" models from OpenAI and Google. To add to that, a number of crucial details are missing, e.g., hyper-parameters for SFT and RL.

We fully understand your concerns regarding reproducibility. Due to organizational restrictions, we cannot disclose detailed information about our in-house proprietary models and training data. However, to address this issue and improve reproducibility, we have:

• Statistics on PRM Data for In-House Model Experiments: We have included additional statistics on the PRM data used in our in-house model experiments in Appendix A of the updated paper. These statistics include token counts, code line distributions.
• Training Curves for In-House Model Experiments: We have attached the training curves for all four settings evaluated in our main experiments in Appendix B. These curves clearly demonstrate that compared to the RL baseline, using PRM for both Dense Rewards and Value Initialization yields the most significant improvements.
• Reproduction with Open-Source Models: To enhance reproducibility, we have reproduced our main results using the open-source model Qwen2.5-7B [1]. The new experiments confirm that our method remains effective with Qwen2.5-7B, further validating its general applicability. Detailed hyperparameters and configurations used in these experiments are provided in Appendix D of the revised paper.

We hope these additional details and open-source experiments address your concerns about reproducibility.

[1] Qwen2.5-7B: https://huggingface.co/Qwen/Qwen2.5-7B

Throughout the paper, the authors claim that their PRM improves exploration. I don't see this claim verified experimentally. Instead, the experiments show better generalization to LiveCodeBench and their in-house benchmark, which is not the same. It would be interesting to see if indeed a larger fraction of training problems is solved during training, or whether more diverse solutions are found.

Thank you for pointing this out. We have added supporting evidence in the revised paper. Appendix B now includes training curves for all four RL configurations (with/without DenseReward and ValueInit) of our in-house model. These curves demonstrate that models using PRM with DenseReward solve more problems on the training set compared to the RL baseline, indicating improved exploration during training.

We appreciate your suggestion and hope the additional training curves provide clarity on the role of PRM in improving exploration.

In Table 1, you compare "Ours-SFT" against "Ours-RL", but the text in L366/367 refers to it as "RL baseline"? Relatedly, since you train the PRM on data produced during the RLHF phase, does that mean "Ours-RL" numbers are the product of two RL training stages? Where is the model after RLHF then (i.e., the "RL baseline")?

We apologize for the confusion caused by inconsistent terminology. Here is a clarification:

• Terminology: In Table 1, "RL baseline" refers to the "Ours-RL" configuration where PRM is not used (neither for DenseReward nor ValueInit). This serves as the baseline for comparison.
• Two-Stage Training: The PRM is trained on data collected from the RL baseline checkpoints. After the PRM is trained, we restart RL training using PRM-enhanced rewards (DenseReward and/or ValueInit). This second stage of training produces the results reported for the other three "Ours-RL" configurations in Table 1.
• Where the Model is After RLHF: The RL baseline (or "RLHF model") serves as both a benchmark and the source of checkpoints for PRM data collection. Models trained with PRM restart from scratch, incorporating the PRM rewards into the training process.

We hope these updates address your concerns and improve the clarity and impact of our work. Thank you for your feedback, which has been invaluable in refining the paper. If there are additional points you would like us to address, please let us know.

Thank you for your thorough review and for sharing your detailed thoughts and questions. We appreciate the time and effort you invested in reading our paper and providing valuable feedback. We address your concerns point by point below.

Weakness 1: The paper discusses using PRM for code generation. But I don't see what specific challenges the proposed method addresses regarding code generation.

The motivation for using PRM in code LLMs (Sec.1 para.2 line 035-043) is that the reward signal is sparse. Sparse reward (or more accurately, the use of bandit feedback) is a common challenge in multi-step reasoning tasks such as math reasoning and code, many research have been conducted to address this challenge, e.g., introducing PRM.

• If the paper limits its scope to code generation, the paper should clearly explain what are the specific challenges it address regarding the code generation task. I notice that unit test feedback is mentioned as a cause for the sparse reward, but I don't consider it as a particular challenge specific in the code generation task. In its essence, test cases serve as a means of verification of the holistic response -- same in its role as evaluating the correctness of a holistic response using the gold-answer in math reasoning. The natural question is: Is previous methods to deal with sparse rewards in math reasoning directly applicable to code generation? If not, the current paper does not clearly explain this.

• If the paper targets at proposing new methods for PRM, the paper should clearly state what is new about the methods.

Thank you for recognizing the potential of our approach to generalize to other domains. The simple reason we limited our scope to code generation is that we only conducted experiments in this area and do not wish to claim it can also work for other domains such as mathematical reasoning.

We are aware of existing work on PRMs in the mathematical reasoning domain. Here is our understanding of the difference in the role of process supervision, or PRMs, in each domain:

• In math reasoning, the correctness of the final answer is easy to check, but the correctness of intermediate steps is not. LLMs could output a response with the correct final answer but incorrect reasoning steps. To mitigate this issue, PRMs were introduced to verify intermediate reasoning steps, either to rerank answer responses or to provide guidance signals during decoding.

• In code generation, the correctness of a program is easy to verify with well-designed unit tests. If the program passes all unit tests, there are no intermediate reasoning steps that could be incorrect and need verification. In other words, the code itself serves as a "proof" of solving the problem. Instead, the challenge in code generation is how to efficiently teach LLMs to find a correct "proof" (generate a correct solution) for a problem. In previous work, RLTF introduced the use of unit tests in an online RL framework to let models discover correct "proofs" by themselves. However, a program can only be verified after it is completely generated; thus, the reward from unit tests is sparse and delayed until the end. The role of PRMs here is to provide partial or dense guidance and rewards during the RL process to improve learning efficiency.

The authors suggest that the reviewer have not carefully read their rebuttal.

However, my follow-up responses arise precisely because I have carefully read the rebuttal and found that it does not adequately address my concerns. My intention in providing these responses is to ensure my reviews are responsible and based on a clear understanding of the manuscript.

I hope the authors recognize that the discussion phase is designed to clarify misunderstandings. Instead of assuming reviewers have not read your responses carefully, I encourage the authors to focus on presenting your main ideas more clearly and effectively.

Returning to the reviews, my aim is to ensure my critiques are not based on misunderstandings of the current manuscript. Based on the provided material, I believe my prior reviews are grounded in a correct interpretation of the work. Importantly, regarding the contributions:

• contribution 1: The authors have acknowledged that it is misleading to claim, "We propose..." and clarified that it should instead be "We adopt..." This correction validates my point.
• contribution 2: I want to emphasize that the authors should clearly outline in the manuscript what is new about integrating PRMs into RL compared with findings from MathShepherd and Rest-MCTS.

To avoid any potential complaints about insufficiently careful reviews, I must stress again: These review comments are based on the current manuscript. Since the authors have not revised their manuscript, it remains difficult for readers to properly evaluate the true contributions of the work.

Regarding the statement, "RL with value initialization shows no improvement over SFT (both yielding an overall score of 28.2)," the confusion arises from the lack of clarity in your presentation. In addition, it is impossible for the readers to infer from the descriptions about Table 1 which line corresponds to SFT and RL, because their are no clear references from the description to the table lines. It is only after the authors clarified that 337 corresponds to SFT and 338 corresponds to RL did this become clear. Rather than expecting readers to deduce this, I strongly urge the authors to improve their delivery, e.g., place a clear boundary between SFT and RL, to prevent such ambiguities.

Additionally, regarding my earlier comment, "The paper draws a conclusion that using PRM to initialize the value function of PPO does not work," this interpretation stems from the statement in line 370: "Interestingly, using PRM solely for value function initialization does not provide notable benefits," which is not followed by any discussion of potential benefits.

Based on the clarifications provided in this thread, I find this claim somewhat misleading. If the authors believe there are indeed benefits to using value function initialization, I strongly recommend revising the presentation to more accurately reflect these benefits. Otherwise, the readers should have unexpected interpretations.

Overall, the current paper delivery is poor, leading to misconceptions and difficulties in properly in accurately assessing its contributions. My primary concern is the misunderstanding stemming from the paper's current delivery. Determining whether the contributions are sufficient for publication at ICLR seems infeasible, as the lack of substantial revisions leaves no common basis for meaningful discussion.

Thank you for your detailed review and thoughtful feedback. We appreciate your positive comments on the novelty, clarity, and thoroughness of our work, as well as your insightful questions and suggestions. Below, we address your queries and observations.

For the oracle, is this also using the policy model checkpoints? I see "Our method leverages the model’s own capabilities to generate completions for partial code prefixes and uses automated testing to assess their correctness" which suggests that the policy model itself is used to generate the best-of-K samples. I can appreciate that not requiring a separate oracle model is nice because it is self-contained, but I think this will result in a suboptimal dataset compared to e.g. using a fully-trained RLTF code model as the oracle.

Yes, you are correct. In our experiments, we use the policy model checkpoints as the oracle to generate the best-of-K samples.This self-contained approach simplifies the pipeline and avoids dependency on external models. However, we agree that employing a fully-trained RLTF code model as the oracle could potentially improve the quality of the dataset by providing more robust labels.

While we did not perform an explicit ablation study on this aspect, related work has observed that for larger values of K, an RL policy may not consistently outperform an SFT policy in terms of best-of-K performance [1]. This suggests that further exploration of alternative oracle designs, such as a fully-trained RLTF model, could yield interesting insights and improvements. We appreciate your suggestion and will consider it in future work.

[1] Wang, E., Cassano, F., Wu, C., Bai, Y., Song, W., Nath, V., ... & Zhang, H. (2024). Planning in natural language improves LLM search for code generation. arXiv preprint arXiv:2409.03733.

Reading Section 4.2.2 on RL Training makes me think: Wouldn't a different scheme of data labelling, which labels each line with the "marginal contribution of the line toward success" be more effective than simply rating [0: infeasible, 1: feasible]? Specifically, my intuition is that each added line should improve the success rate of the oracle given K attempts, so my proposed reward is something like "the reward for step M should be the (success_rate_of_oracle_at_step_M - success_rate_of_oracle_at_step_Mminus1)". This captures the idea that each line should increase the likelihood of the program succeeding, and naturally avoids reward hacking by simply adding more lines. Of course, this is easier said than done, I expect the process to be noisy, but this formulation for the dataset seems to be better aligned to the true objective.

Thank you for this insightful suggestion. Estimating the marginal contribution of each line to the overall success rate is indeed a compelling idea, as it could provide a more fine-grained and objective measure of progress. Implementing this scheme would require estimating the value function for each line under the oracle and computing the difference, (e.g.,$V(\text{line}\_M) - V(\text{line}\_{M-1})$).

However, there are practical trade-offs to consider. Using the Monte Carlo method to estimate the value function with K rollouts would require $O(N^2 \times K)$ token generations for a trajectory of length $N$. In contrast, our current binary search labeling procedure reduces this too $O(N \log N \times K)$, significantly lowering computational cost. Given the expense of LLM generation, we prioritized the more computationally efficient approach while achieving robust performance.

(Not a weakness, just a typo) In Section 3.2, I think the authors meant to do citep instead of just cite: "In mathematical domains, LLMs may generate correct answers with faulty reasoning Lightman et al. (2023), making intermediate verification essential" and "While preliminary attempts have been made to incorporate PRMs into RL trainingWang et al. (2024a)"

Thank you for catching this error. We have corrected it in the revised version to ensure proper citation formatting.

We greatly appreciate your thoughtful review and the opportunity to improve our work based on your suggestions. If there are additional points you would like us to address, please let us know.

Thank you for your detailed review and constructive feedback. We appreciate the opportunity to address your concerns and clarify the points raised. Below, we provide detailed responses to your comments and questions.

My primary concerns are reproducibility, the lack of experimental details, and data transparency, which hinder the community from reproducing the results presented in the work and accessing the needed efforts. This paper uses an in-house model, and part of the results are reported on the in-house benchmark. This is not inherently an issue but means that there needs to be more details in the text/contributions in other components.

We fully understand your concerns regarding reproducibility and appreciate your suggestions for improvement. While restrictions imposed by our organization prevent us from releasing detailed information about the in-house proprietary model and dataset, we have taken significant steps to address this:

• Statistics on PRM Data for In-House Model Experiments: We have included additional statistics on the PRM data used in our in-house model experiments in Appendix A of the updated paper. These statistics include token counts, code line distributions.
• Training Curves for In-House Model Experiments: We have attached the training curves for all four settings evaluated in our main experiments in Appendix B. These curves clearly demonstrate that compared to the RL baseline, using PRM for both Dense Rewards and Value Initialization yields the most significant improvements.
• Reproduction with Open-Source Models: To enhance reproducibility, we have reproduced our main results using the open-source model Qwen2.5-7B [1]. The new experiments confirm that our method remains effective with Qwen2.5-7B, further validating its general applicability. Detailed hyperparameters and configurations used in these experiments are provided in Appendix D of the revised paper.

We hope these efforts help bridge the gap in reproducibility and benefit the broader research community.

[1] Qwen2.5-7B: https://huggingface.co/Qwen/Qwen2.5-7B

Description of the In-house model

1. What is the parameter size and architecture?

2. What is the size of the SFT set, if available (number of tokens/data points) mentioned in L241 and L337? Could the authors show an example of the SFT data? Could the authors provide some basic configuration of the SFT (e.g., lr, number of gradient steps)?

Due to organizational restrictions, we cannot disclose details about the parameter size, architecture, or specific SFT dataset of the in-house model. However, for the reproduced experiments with Qwen2.5-7B, we provide detailed configurations in Appendix D.

PRM data and training

1. Could the authors give some statistics on the different strategies of PRM training data, if available (number of tokens/data points, avg. lines of code, the distribution of the % or the line number where the line-level label turns -1 from +1)? These statistics could be a valuable add-on to Table 2 and the plain strategy. Could the authors show one example of the reward produced by the binary search?

Yes, we can definitely share more statistics on the different strategies for selecting PRM training data. Please find these statistics in the Appendix A.

PPO training

1. what is the KL penalty (the value of $\beta$ in Eq(3)) used in the exp.? Some basic hyperparams (lr, steps) would be appreciated.

2. The proposed PRM data collection requires model checkpoints from RL baseline training (L252). Therefore, the whole pipeline is: RL baseline training -> PRM data generation collection -> PRM training -> RL training w/ PRM. It makes the first RL baseline training mandatory and increases the computation cost. Did the author explore alternatives, such as using a base policy (e.g., in-house-SFT) to conduct PRM data generation and collection?

3. While we cannot disclose proprietary hyperparameters for the in-house model, Appendix D of the revised paper includes the hyperparameters used for Qwen2.5-7B.. We hope this information could also be helpful.

4. Yes, your understanding of our proposed training pipeline is correct. It requires first running the RL baseline and using checkpoints sampled during training to collect responses that sufficiently cover the state space. The rationale behind this is that PRM needs to assist RL training throughout the entire process, which necessitates exposure to states that might be visited during training.

In our early experiments, we also included the SFT model in the PRM data collection. However, we observed that this approach negatively impacted the final RL with PRM results, so we decided not to include it. We appreciate your feedback and would like to conduct a more thorough ablation study regarding this during the rebuttal period. However, due to time constraints and resource limitations, we prioritized experiments with the Qwen2.5-7B model and will leave this exploration for future work.