CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning

Thank you for your comments. I will address your questions one by one.

Q1: Lack of Examples

A1: Thank you for your suggestion. We will include more examples and corresponding analysis in future versions of the paper. In the meantime, we have open-sourced the dataset to help provide access to additional examples. However, due to anonymity requirements during the review period, we are unable to share the open-source link at this time.

Q2: How does CodePMP stack up against CoT and search-based approaches in the same experimental setup? What are the strengths and weaknesses of CodePMP relative to these techniques?

A2: Thank you for this insightful question! CodePMP is very suitable to integrated with CoT and search-based approaches. However, since this is a broad topic, we only mention some of our preparations, especially GenPMP and CodePrMP, in the future work section (Section 7). We would like to provide some recent progress, as follows:

• GenPMP models CodePMP in a generative way. It introduces CoT and critic techniques in the context and has brought promising improvement to CodePMP.

• CodePrMP integrates with search-based approaches such as MCTS, which also involves process reward modeling (PRM). By pretraining on process-rewarded code, we aim to enhance PRM's effectiveness, enabling better synergy with MCTS.

Q3: Challenges in Maintaining Data Quality with Evolving LLM Capabilities

A3: This is indeed an important and insightful concern! The issue of "data distribution drift" can lead to reward hacking. One of the key motivations behind designing CodePMP was to improve the generalization ability of the preference model and mitigate reward hacking through sufficiently broad and diverse prompts and responses in the broadly-covered code domain. However, we acknowledge that continuously iterating pairwise data is essential to adapt to evolving LLM capabilities. From the perspective of scalable oversight, when model capabilities surpass human capabilities, we may need to leverage "critic of critic" methods to ensure data quality. We are actively working on this challenge, and we also hope that more peers will pay attention to our work and give us the same valuable suggestions as you.

Q4: Generalization of CodePMP to Non-Structured Reasoning Tasks

A4: Currently, our work primarily focuses on reasoning tasks with a structured, logical basis. However, your point is well-taken, and there is significant potential for extending this work to areas like common-sense reasoning, social interactions, and natural language inference. We are actively open-sourcing our research results to inspire and support the broader community in collaboratively advancing reasoning capabilities across diverse domains.

We hope that our response addresses your concerns and strengthens your confidence in CodePMP’s contribution to the research community.

Thank you for your suggestions.

Q1：In the experiments, only two Qwen2 models are used for the evaluation. Other model family results can be used for the verification of the methods.

A1: Thank you for this suggestion. As noted in the "General Author Response," we conducted additional experiments using Llama3.2 as the generator. Moreover, we provided the verification results of the CodePMP method on the Gemma2 model in Section 5.4 and Figure 8. These results demonstrate the general applicability of CodePMP, significantly enhancing models across different families, including Gemma2 and Llama3.2.

Q2: In the Data Construction section, a description summarizer is used to generate prompts that describe the code. However, there are no details on this part, and the quality of the constructed data pairs is not analyzed. Since this is not a human-annotated dataset, it is crucial to demonstrate the data quality.

A2: We apologize for omitting some details regarding prompt generation and data quality.

• Prompt generation: Specifically, we employed deepseekcoder-6.7b with optimized instructions to generate code prompts from code snippets. In future versions, we will include the prompts and examples in the appendix for clarity.
• Data quality: To ensure the quality of automatically labeled data, we designed a critique prompt to score the label correctness of the generated pairs. On a test set of 100 pairs, we iteratively refined the critique prompt until it achieved 95% agreement between GPT4-o and experienced programmers (human-annotators). Using GPT4-o, we then scored 2,000 generated pairs, and the label accuracy reached 76%. Given that this is preference model (PM) pretraining, we balanced quality and diversity. We are exploring scalable and cost-effective ways to optimize synthetic data quality while reducing potential bias. Encouragingly, as highlighted in our "General Author Response" and the experimental results in the paper, CodePMP has already demonstrated strong generalizability and scalability, with promising room for further improvement in data quality.

Q3: In Section 4.2.2, how are the different numbers of N candidate responses generated? Is the Qwen2 model used to sample these candidate responses? There are no details provided.

A3: To generate varying numbers of N candidate responses: We used Mistral-7B as the generator, sampling each question in the GSM8K and MATH-500 datasets 256 times with a temperature of 0.7 and top-p set to 1. To ensure consistency in the Best-of-N evaluation, we selected the same N candidate responses from these 256 samples for each Reward Model (RM). These candidates were scored using the RM, and the highest-scoring response was selected as the best response for each question. In the primary experiments, Mistral-7B served as the generator for the Best-of-N evaluation. To explore the performance gains of CodePMP with a more advanced generator, we later used Qwen2-Math-7B-Instruct, which excels in mathematical problem-solving. Results are provided in "General Author Response", which shows that CodePMP can even bring significant improvements to Qwen2-Math-7B-Instruct.

Q4: In Figure 4, the author claims that RM with CodePMP initialization consistently outperforms. However, Figure 4(g) seems to show the opposite case, where RM with CodePMP initialization is consistently worse. Please check this in detail.

A4: We sincerely apologize for the confusion caused by the reversed color labels in Figure 4(g). After rechecking, Figure 4(g) still demonstrates that RM with CodePMP initialization consistently outperforms RM without CodePMP initialization. We will fix this issue in future versions. Thank you for pointing it out!

Q5: As shown in Figure 5, increasing the number of code-preference pairs consistently improves accuracy. It seems there is still room for improvement. Do you think more preference data can be obtained without too much work?

A5: Yes, there is substantial potential for further improvement. Open-source platforms like GitHub contain a vast amount of code that can be used to construct code preference data. We have only utilized a small fraction of this resource. We plan to open-source our data generation pipeline to benefit the entire research community.

Q6: Is the PMP data planned to be released? How many code-preference pairs are included?

A6: The 28 million PMP code-preference pairs have already been released. However, due to the anonymity policy, we cannot provide the corresponding link here at this time.

We hope that our response addresses your concerns and strengthens your confidence in CodePMP’s contribution to the research community.

Thank you for your comments. Below, we provide responses to each question in detail.

Q1: Unfortunately, the effectiveness of the RM was not fully verified, e.g., by using the RM for RFT or PPO training.

A1: We follow prior works ([1][2][3]) that primarily use Best-of-N performance as the evaluation protocol for learned RMs. When N is sufficiently large (e.g., N=256 in our experiments), it is to approximate the performance ceiling of the current policy, making it a reliable metric to assess RM capability. While RFT or PPO training can also validate RM effectiveness, they are more resource-intensive as they require policy training. Although we believe Best-of-N is sufficient to evaluate the improvements CodePMP brings to RMs, incorporating RFT or PPO training in future work could enhance confidence in the method.

References:

• [1]Zhang et al. (2024) "Generative Verifiers: Reward Modeling as Next-Token Prediction"
• [2]Cobbe et al. (2021) "Training Verifiers to Solve Math Word Problems"
• [3]Lightman et al. (2023) "Let's Verify Step by Step"

Q2: It remains unclear whether and how code training could help reasoning tasks in natural language. It would be great if the authors could explore the relationship between coding and reasoning tasks for model training.

A2: CodePMP enhances the reward model (RM), which significantly improves Best-of-N (BoN) performance in mathematical and logical reasoning tasks. The stronger the RM, the closer BoN performance approaches the Generator’s capability limit. However, training a robust RM requires large-scale, high-quality labeled data, which is often scarce due to two challenges:

• Lack of prompts: Reasoning-related prompts are hard to collect and often need to be synthesized. Code, which inherently embodies reasoning properties, provides an abundant source of high-quality data, as it is designed to solve specific tasks. Code fragments can be used to synthesize large-scale reasoning-related prompts.
• Difficulty in constructing preference data: Manual labeling for reasoning tasks is inefficient and costly. However, with advancements in CodeLLMs, stronger CodeLLMs can generate better solutions for code prompts compared to weaker ones, enabling automated preference data construction.

By addressing these limitations, we can scale reasoning-related prompts and preference pairs. Pretraining on these preferences provides substantial reasoning capabilities (as shown in Figure 4). Minimal RM fine-tuning further boosts performance on mathematical and logical reasoning tasks. We will include more detailed analysis in future versions.

Q3: No mention of whether the dataset will be open-sourced.

A3: The dataset has already been open-sourced on HuggingFace. However, due to anonymity requirements, we cannot provide the link at this time.

Q4: It would be helpful to verify generalization with more generators.

A4: Following your suggestion, we further used Qwen2-Math-7B-Instruct as the generator, which excels in mathematical problem-solving. The results, included in the "General Author Response," show that CodePMP brings improvements to Qwen2-Math-7B-Instruct as well.

Q5: Figures 4 and 5 are too small to read.

A5: We apologize for the inconvenience. Due to space limitations, we reduced the size of the figures. We will enlarge them in future versions for better readability.

Q6: Why does CodePMP help reward modeling in mathematical tasks? Please provide more analysis in the paper.

A6: CodePMP benefits reward modeling for mathematical tasks because of the intrinsic connection between code data and mathematical problem-solving. Code represents algorithmic implementations, which are essential for solving math problems. Thus, training a discriminator on code preference data improves the model's ability to accurately evaluate mathematical solutions. We will include more examples and analysis in future versions to clarify this further.

Q7: Reference formatting issues, such as on line 429.

A7: We apologize for this issue and will fix it immediately.

Q8: Would you consider using a larger reward model to verify scalability?

A8: As mentioned in the "General Author Response," we followed your suggestion and verified CodePMP’s scalability on Qwen2-72B. The results demonstrate that CodePMP significantly enhances the performance of larger, more powerful models.

We hope that our response addresses your concerns and strengthens your confidence in CodePMP’s contribution to the research community.

Thank you for your response. I am raising my rating to 5 as some of my concerns are addressed. I am not fully convinced of the effectiveness of the final RM model without a result from its application in model optimization (e.g. DPO) though. Besides, If the claim "When N is sufficiently large (e.g., N=256 in our experiments), it is to approximate the performance ceiling of the current policy" is true, the humaneval experiment shows that the ceiling of the current policy is close to Bo1 without CodePMP.

I think the authors propose an interesting idea for RM pre-training though. CodePMP could be a good research step if the authors can add more convincing experiment results. I encourage the authors to further polish the work and make it a solid one.

Thank you for your comments.

Q1: CodePMP creates preference pairs for coding tasks, but coding tasks are not evaluated in experiments.

A1: CodePMP is not specifically designed for coding tasks. Its main goal is to enhance mathematical reasoning and logical reasoning tasks through code preference pre-training. We do include performance comparisons on coding benchmarks in Section 5.5 and Table 2 (page 9). For clarity, we have listed the relevant results here. As shown, the improvement brought by PMP is significant. In the table, "w/o" denotes without PMP, and "w/" denotes with PMP.

Q2: The CodePMP data is constructed by deepseek-coder-instruct. It would be interesting to see whether CodePMP can further improve deepseek-coder-instruct on coding tasks.

A2: We used deepseek-coder-33b-instruct as the backbone of the reward model (RM). As shown in the results below, an RM trained with CodePMP (constructed using code preference pairs synthesized from deepseek-coder-6.7b-instruct and deepseek-coder-1.3b-instruct) outperforms an RM trained without CodePMP in downstream preference fine-tuning tasks. In other words, CodePMP can further improve deepseek-coder-instruct on the coding preference task.

Q3: A more meaningful setting would be to examine whether the Qwen2-based reward model can further improve the performance of Qwen2 or Qwen2-Math on math reasoning tasks.

A3: As noted in our "General Author Response," we further experiment with Qwen2-7B-Math as the generator. The results show that the RM initialized with PMP improves the BoN performance of Qwen2-7B-Math on GSM8K and MATH (Bo4). In contrast, RMs trained directly without CodePMP consistently lead to performance degradation, regardless of the value of N. Additionally, we observed that as the generator is already very strong, weaker RMs tend to misrank outputs (a phenomenon akin to reward hacking). CodePMP alleviates this issue and improves overall performance.

Q4: Majority voting should be included as a baseline in Figure 3.

A4: We have tested the majority voting approach on the GSM8K and MATH datasets. The results are summarized below. As seen in the table, CodePMP outperforms majority voting, with the improvement being more pronounced on the harder dataset (MATH).

Q5: Further analysis is needed to understand why the coding-based CodePMP contributes to improvements in math and logical reasoning.

A5: CodePMP improves the reward model (RM), thereby boosting BoN performance of Generators in mathematical and logical reasoning. The stronger the RM's capability, the closer the BoN performance is to the Generator's capability limit. Training a strong RM requires a large amount of carefully labeled data, and such data is scarce. The limitations are mainly twofold:

• Lack of prompts: Reasoning-related prompts are difficult to collect directly and need to be synthesized. High-quality coding data is abundant, and code inherently possesses reasoning properties as it is designed to solve tasks. Code fragments can be used to generate large-scale code prompts, most of which are reasoning-related.
• Difficulty in constructing preference data: Reasoning tasks are complex, and manual labeling is inefficient and expensive. However, with the rapid development of CodeLLMs, a reasonable assumption is that stronger CodeLLMs generate better solutions and outcomes for code prompts compared to weaker ones. This assumption enables automated preference data construction.

By addressing these two limitations, we can generate scalable reasoning-related code prompts and preference pairs. After pre-training with these preferences, the model gains substantial reasoning capabilities (as seen in Figure 4, where models with PMP show strong performance even without RM fine-tuning). A small amount of RM fine-tuning then further enhances performance on math and logical reasoning tasks. We will add detailed analysis in the future version.

We hope these additional experiments and responses can address your concerns and bolster your confidence in the improvements that CodePMP contributes to reasoning tasks.