Thanks for the reviewer’s response and appreciation of our work!

w1 & Q1: Using one reward model for both reasoning and human preference might weaken the results. More experiments on smaller models with separated reward models

Yes, we have also considered the issue of whether to use a unified reward model (RM) for all tasks or develop a separate RM for each individual task. Our findings indicate that employing a single unified RM achieves nearly the same performance as using a set of task-specific RMs in Best-of-N evaluations. Therefore, for the sake of training efficiency and simplicity, we opted to train a unified RM.

We conducted experiments on mathematical tasks to assess this approach, and the Best-of-N results on the MATH dataset are as follows:

As shown, the unified RM demonstrates performance comparable to the math-only RM and even exhibits a slight advantage, likely due to the inclusion of additional code reward model data in the unified RM. Overall, our results suggest that training a single reward model for multiple tasks is both feasible and does not compromise performance.

w2: The paper mentions GRPO but never gives the expanded version of the acronym.

Thanks for the kind suggestion. GRPO refers to Group Relative Policy Optimization and we have fixed the issue in the updated version (line 205).

Q2: will the reward and policy model be open-sourced? Yes, we will open-source the reward model and also part of the training data. We hope that it could help the community to reproduce our results and contribute to further research in the field.

Thanks for the reviewer’s comments and appreciation of our work!

w1: paper framing

Thanks for your suggestions. As the reviewer has suggested, we have added more explanations in the introduction part in the updated version (line 77-79). In this work, we mainly focus on reasoning-related tasks but also conduct evaluations on general tasks like AlignBench.

w2: Dataset and evaluation choice

it's unclear what the relationship between the training and evaluation datasets is, which means the results are harder to interpret.

In our experiments, certain evaluation sets, such as MATH, GSM8k, and LiveCodeBench, are in-distribution relative to the training data, while others, such as GPQA, are not. And we see similar improvement trends in these benchmarks.

Data distribution and the generalization of the trained model are indeed critical aspects of RLHF training. As described in the experiment section, our training data comprises the following components:

• Mathematics data, including MATH-train, Numina-Math, and a Chinese K-12 dataset.
• Code data, such as competition data from code-contest [1] and TACO [2].
• General chat data, including ShareGPT and our self-collected open-domain chat data.

We evaluate our model on six datasets: MATH, GSM8k, LiveCodeBench, MMLU, GPQA, and AlignBench. For reasoning-related benchmarks, MATH, GSM8k, and LiveCodeBench can be considered in-distribution evaluations, while GPQA serves as an out-of-distribution evaluation due to the absence of related data in our training set. As illustrated in Figure 2, these datasets demonstrate consistent scaling trends. Thus in-distribution and out-of-distribution might not be the direct key factors of scaling.

Regarding MMLU and AlignBench, which do not exhibit clear scaling trends, it’s worth noting that MMLU performance is largely influenced by the pretraining phase and is less affected by RLHF. Similarly, human preference tasks tend to be less scalable compared to reasoning tasks.

[1] Li Y, Choi D, Chung J, et al. Competition-level code generation with alphacode[J]. Science, 2022, 378(6624): 1092-1097.

[2] Li R, Fu J, Zhang B W, et al. Taco: Topics in algorithmic code generation dataset[J]. arXiv preprint arXiv:2312.14852, 2023.

Downstream evaluation may not be a good metric to measure "scaling"

Thanks for pointing this out! We had a similar thought process and tried to seek metrics that could better indicate the "scaling" in RLHF. Since the training loss indicates nothing in RLHF training, we have considered using reward on training or evaluation set. Unlike pre-training in which lower loss generally leads to better performance, as shown in our paper, higher reward does not indicate better performance. Therefore, we finally selected downstream evaluation metrics as the primary measure. Downstream evaluation is an appropriate choice, as it serves as a “golden reward,” providing a reliable signal to assess scaling improvements. As shown in OpenAI o1[1], they also use Pass@1 Accuracy in AIME to show the scaling trends in RLHF training and inference. Therefore, we are hoping this early attempt on this topic could spark more discussions and efforts in it.

[1] https://openai.com/index/learning-to-reason-with-llms/

w3: Agreement / disagreement with previous related works

In the introduction, we list observations about RLHF training and scaling, some of which are first presented in this work and not proposed before:

1. Sampling more responses during training generally improves the policy model’s performance
2. Larger policy models benefit less from RLHF when using a fixed-size reward model
3. Performance improves remarkably in the early stage of training but the return gradually diminishes in later training.

For the remaining ones, they go beyond or disagree with previous findings.

4. [Agree and Go beyond] Previous work also states that larger reward models (RM) show better performance in Best-of-N evaluation. In this work, we show that the larger RM can also benefit RLHF training but the improvement still significantly falls behind the gains in Best-of-N evaluation of the reward model.

5. [Agree and Go beyond] Previous work shows that increasing training responses of each prompt for RM improves its performance, but we show that increasing prompt diversity is more effective than increasing response diversity

6. [Go beyond] Previous work finds that process supervision might be better than outcome supervision. But we show that PRM actually performs better in targeted tasks but struggles to generalize to other tasks.

7. [Disagree] About reward hacking, previous works show that RLHF could suffer from reward hacking and performance degradation, but we find that over-training in RLHF brings less performance improvement but does not result in degeneration.

w4: make a more clear statement about RLHF scaling and pretraining scaling

For pre-training scaling, the general definition is as follows: expanding model size, dataset and training compute in the training can lead to lower training loss and improve model performance.

For RLHF scaling, there are more factors that could affect the final performance, with different policy model sizes, reward model sizes, and sampling budgets. And therefore, in the introduction part, we try to give a definition of RLHF scaling from two aspects: (1) Given a fixed SFT model, how does scaling other factors, including the reward model, sampling budgets, and training data, affect the policy model through RLHF? (2) With a fixed reward model and training strategy, whether a larger policy model can benefit more from RLHF?

Through this expensive study, the main message we would like to share with the community is: Expanding reward model size (supervision signal), sampling budgets and training data can lead to better performance in RLHF, but currently, larger policy models benefit less from RLHF when using a fixed-size reward model.

w5: Smaller issues

add o1 RL training scaling and add error bar in figure 2

Thanks for the kind suggestion. We have addressed the problem in the updated version.

why not using a larger learning rate for a larger batch size

Yes, we have experimented with using a larger learning rate. However, we found that training becomes more prone to collapsing with higher learning rates, resulting in the model’s inability to generate proper responses. RLHF training is highly sensitive to the choice of learning rate: setting it too large can lead to model instability, while setting it too low can cause the learning to become too slow. Therefore, we used a learning rate of 2e-6 across all experiments to balance training stability and performance improvement. This learning rate works across different batch sizes.

why the starting point of larger models is lower in Figure 1(b)

Figure 1(b) indicates the relative improvement of RLHF over the SFT policy. The result that the starting point of the larger model is lower indicates that the relative gain against the corresponding SFT policy from RLHF is lower in larger models than in smaller models. The result corresponds to conclusion 3 that "Larger policy models benefit less from RLHF when using a fixed size reward model"

Q1: more details on the process supervision technique

We applied the method by Math-Shepherd[1] to obtain an estimate of step correctness $Q(p,s)$ by running Monte Carlo rollouts for each step in the solution. Specifically, we divide a sampled solution $S$ of a problem $P$ into steps $S = S_1, S_2, ..., S_k$. For each step $S_i$, we perform $N$ rollouts to obtain $N$ generated solutions based on the same current step prefix $S_{i,j} = \\{S_{1,j}, ... S_{i, j}, S_{i+1, j}, ...,S_{k_j, j}\\}_j^N$, where $k_j$ is the total number of steps for the j-th finalized solution. We evaluate the correctness of the $N$ rollout results $A=\\{a_j\\}_j^N$. The correctness estimate of the current step is based on whether there is a correct solution among the $N$ rollouts at that step and it can be represented as $Q(P, S_i) = \begin{cases} 1 & \exists a_j \in A, a_j = 1 \\ 0 & \text{Otherwise} \end{cases}$ Then, we use the 0/1 binary classification labels of each step in solution $S$ as process supervision signals for PRM training.

[1] Wang P, Li L, Shao Z, et al. Math-shepherd: Verify and reinforce llms step-by-step without human annotations[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 9426-9439.

We conducted additional experiments to investigate whether data distribution impacts the scaling of RLHF. Specifically, we ensure that the reward function closely matched the policy model's output distribution, and that both the training set of RLHF and the evaluation dataset are under the same distribution, thus maintaining an in-distribution experimental setting.

• For the reward model, as described in the experiment part (Line 232), all the training data is sampled from the SFT model and thus the reward function matches the distribution of generated data from the policy model for RLHF training.
• For the RL training, to conduct the in-distribution experiment, we choose to use MATH-train[1] and MATH-test (the same as the MATH dataset in our paper) as the training and evaluation set to ensure they lie in the same distribution.

The following table shows the results of RLHF training on different training dataset.

As shown in the table, training on the MATH-train dataset shows better performance than our original training set in MATH-test evaluation as the MATH-train and MATH-test are under the same distribution. However, they show similar trends that further increasing the sampled responses per prompt shows diminishing returns. Thus, the data distribution might not be the key factor in RLHF scaling.

Thanks a lot for the valuable question and we also gain a lot from these experiments. Please let us know if you have any further questions, and we look forward to the possibility of your updated evaluation of our work.

[1] Hendrycks D, Burns C, Kadavath S, et al. Measuring mathematical problem solving with the math dataset[J]. arXiv preprint arXiv:2103.03874, 2021.

We thank the reviewer for the comments and suggestions. We would like to further clarify the contribution of our work.

1. In this paper, we aim to systematically analyze the key factors that affect the scaling of RLHF and help the community better understand the scaling properties in RLHF training. Similar to our work, previous works [1, 2] investigate the scaling properties for synthetic reward modeling and supervised fine-tuning from the experimental perspective. They also provide practical insights and contributions to research studies.

2. As noted by other reviewers, RLHF scaling has not been deeply investigated yet in the community. While some of the techniques we studied may have appeared in other literature, their scaling properties have not been thoroughly investigated in literature.

There are indeed numerous factors that can be analyzed to understand their impact on the scalability of RLHF, including those discussed in our paper, such as model size and sampling budgets, as well as techniques mentioned in the reviewer’s comment, such as sampling strategies. In this work, our goal is to examine and differentiate between the scalable and non-scalable factors within the current RLHF framework. Specifically, the sampling strategies are not considered scalable factors to the RLHF training process. This provides a systematic understanding of the limitations and potential of the existing RLHF paradigm, in comparison to what has been extensively studied in scaling the pretraining. Based on the insights and findings, we aim to identify promising scaling directions and develop techniques to enhance the scalability of RLHF in the future.

Therefore, we would like to articulate our contributions again in the following:

1. We identified key factors that have the potential to scale the impact of RLHF, including model size, data composition, reward signals, and sampling budget. And to study the scaling properties, we try to define and study the problem from two perspectives: fixed policy model while scaling other factors and scaling the size of the policy model.
2. We conduct systematic studies to understand the impact of these factors. These studies help us identify the limitations and scalable factors.
3. We found some conclusions that disagree with or go beyond previous works’ findings. For example, a larger policy model benefits less from RLHF training. Process supervision can lead to better performance in the targeted tasks but generalize worse.

We aim to provide a better understanding for future research on RLHF scaling and offer actionable insights for practitioners seeking to further scale the training of RLHF. For other techniques stated in the review, such as different sampling strategies,

[1] Gao L, Schulman J, Hilton J. Scaling laws for reward model overoptimization[C]//International Conference on Machine Learning. PMLR, 2023: 10835-10866. [2] Zhang B, Liu Z, Cherry C, et al. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method[C]//The Twelfth International Conference on Learning Representations.

We greatly appreciate your dedicated time and effort in reviewing our work. We kindly remind you to check if the points raised in your review have been addressed in my response. If any remaining concerns or areas require further clarification, we would be happy to provide additional details or explanations.

Thanks for your kind and helpful suggestions!

w1: This paper does not cover all aspects of RLHF scaling.

Thanks for the suggestion! We agree that it is harder to define "scaling" in RLHF than that in pretraining scaling because there are much more factors that could affect the performance. Hence in the introduction part, we try to first give an overview of RLHF and then focus this study on the following aspects:

1. Given a fixed SFT model, how does scaling other factors affect the policy model through RLHF? In this aspect, we investigate the effects of reward model size from 9B to 200B, training data, sampling budget, and supervision signals.
2. With a fixed reward model and training strategy, whether a larger policy model can benefit more from RLHF? In this aspect, we explore the gain from RLHF on different sizes of policy models from 9B to 200B and show our finding that larger policy models benefit less from RLHF when using a fixed-size reward model.

We believe our early attempt on this topic will spark more discussions and efforts to better understand RLHF scaling.

w2: This paper focuses solely on PPO and GRPO and neglects other methods like DPO and KTO.

In this paper, we primarily investigate the scaling properties of on-policy RLHF methods, specifically PPO and GRPO. We updated the submission in the introduction to make it more clear (line 68-70).

1. Prior studies [1,2] demonstrate that PPO significantly outperforms DPO across a variety of tasks. Consequently, our analysis focuses on the scaling behavior of PPO and GRPO. Most works prefer DPO and KTO due to their simplicity, but PPO and GRPO exhibit better performance in downstream tasks.
2. Unlike PPO, DPO and KTO are off-policy methods. Previous work [3] has studied the scaling laws for off-policy approaches and showed that DPO-related methods could suffer from great over-optimization, and are not scalable.

Therefore, this work mainly focuses on PPO and GRPO and explores their potential for scaling.

[1] Ivison H, Wang Y, Liu J, et al. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback[J]. arXiv preprint arXiv:2406.09279, 2024.

[2] Shao Z, Wang P, Zhu Q, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models[J]. arXiv preprint arXiv:2402.03300, 2024.

[3] Rafailov R, Chittepu Y, Park R, et al. Scaling laws for reward model overoptimization in direct alignment algorithms[J]. arXiv preprint arXiv:2406.02900, 2024.

w3: The study is primarily centered on reasoning tasks and does not extend to other important areas like general instruction-following tasks

In this work, we focus primarily on reasoning-related tasks as previous works have shown that post-training shows scaling potential in RLHF tasks[1, 2]. But we also conduct experiments on general instruction-following tasks, as outlined in Section 4.1.

• For training, we curate a dataset comprising both general chat and reasoning data to train the reward model and the policy model.
• For evaluation, we assess our model using AlignBench, a widely recognized benchmark for evaluating the general alignment of large language models (LLMs), and MMLU. The results are presented in Figure 2 and we put the results on AlignBench again as follows:

AlignBench measures performance on general instruction-following and chatting tasks, as well as the effectiveness of Reinforcement Learning with Human Feedback (RLHF) on human preference tasks.

As observed, while RLHF improves performance on these tasks, scaling, including larger reward models, or more sampled responses, does not yield significant benefits for general instruction-following tasks, unlike reasoning tasks.

[1] https://openai.com/index/learning-to-reason-with-llms/

[2] Yuan Z, Yuan H, Li C, et al. Scaling relationship on learning mathematical reasoning with large language models[J]. arXiv preprint arXiv:2308.01825, 2023.

w4: Discussion about potential hypotheses for why RLHF doesn't scale as well as pretraining and experiments that could help isolate the cause is not presented.

In the discussion section (4.4), we discuss the limitations and potential issues in the current RLHF. We conclude that the potential challenges hindering the scalability of RLHF can be attributed to two key factors: inaccuracies in reward signals and the inherent difficulty of training in RLHF. These challenges are evident in two types of gaps: the RLHF improvement largely lags behind the Best-of-N results, and the Best-of-N results largely lag behind Pass@K , as outlined in the following tables.

In comparison, pretraining relies on next-token prediction as the supervision signal, which comes from the text data itself, and also involves abundant of tasks. That might be the reason why the data quality is supreme in pretraining because it determines the quality of supervision signals.

As shown in our experiments, more accurate reward signals(i.e., a larger reward model) can lead to better scaling trends in RLHF. Larger reward models demonstrate significantly improved performance in both best-of-N evaluation and RLHF results on reasoning tasks, as illustrated in Figure 1. Thus more precise reward signals can enhance the scalability of RLHF training. However, the performance of current reward models remains suboptimal, as the Best-of-N results fall considerably short of Pass@N (Correctness of golden answer as the reward). We propose that the inaccuracy of reward signals may be a limiting factor and that refining these signals could play a crucial role in advancing RLHF scalability.

RLHF results

Best-of-8 results:

In addition, we also find some factors that might help RLHF scaling in the future, like sampling multiple responses, and process supervision.

We greatly appreciate your dedicated time and effort in reviewing our work. We kindly remind you to check if the points raised in your review have been addressed in my response. If any remaining concerns or areas require further clarification, we would be happy to provide additional details or explanations.

Thanks for your efforts in reviewing the paper and your valuable question! We hope that our responses could address your concerns. We also conduct additional experiments to investigate the effects of data distribution in the reward model and RLHF training.

1. We compare the performance of training a unified reward model or a specific reward model for each task. We conducted experiments on mathematical tasks to assess this approach, and the Best-of-N results on the MATH dataset are as follows:

2. We conduct experiments to test whether the data distribution could affect the scaling of RLHF. We ensure that the reward function closely matches the policy model's output distribution and that both the training set of RLHF and the evaluation dataset are under the same distribution, thus maintaining an in-distribution experimental setting. Specifically, we conduct RLHF training on MATH-train only and evaluate the performance on MATH-test (the same as the MATH dataset in our paper) as the training and evaluation set to ensure they lie in the same distribution.

We hope that these experiments can help you better evaluate our work. Please let us know if you have any further questions, and we look forward to the possibility of your updated evaluation of our work.

Thanks for your response! As shown in the results above, more precise reward signals can improve performance in RLHF training. However, we observe diminishing returns as we increase the number of sampled responses per prompt from 4 to 16, compared to the increase from 1 to 4. As noted in [1], they found that using ground truth rewards for code/math tasks can sometimes lead to worse performance than using reward models. This is because, for example, in coding tasks, limited test cases may not offer full coverage, making the binary feedback (0-1) noisy and suboptimal. Reward models, in contrast, are more robust and have better generalization capabilities.

Additionally, we are currently conducting experiments using ground truth rewards. Since we need to modify our implementation to enable this training, we will share the results as soon as we have them.

[1] Zhu Q, Guo D, Shao Z, et al. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence[J]. arXiv preprint arXiv:2406.11931, 2024.

We conduct experiments to compare the performance using reward model or groudtruth as reward in RLHF training. We use MATH-train as the training set and MATH-test (the same as the MATH dataset in our paper) for evaluation set. The results are as follows:

As observed, both using the reward model and the ground truth as reward signals show similar trends, with the reward model demonstrating slightly better performance. This aligns with the earlier discussion. The reason for this could be that signals from the reward model are continuous and more robust. However, we also noticed an interesting trend in the experiments where 16 responses were sampled per prompt: when using the reward model, performance peaks around 1/2 to 2/3 of the total training steps (approximately 70 steps). In contrast, when using ground truth as the reward signal, performance continues to improve even after one epoch of training. This suggests that, with more math-specific training data and by sampling more responses per prompt, training with ground truth may offer better scalability. Due to time and resource constraints, we were only able to conduct experiments under the current settings. We will continue our exploration and hope to report further results in the next version of our paper.

We hope that these experiments can address your concerns and questions. Please let us know if you have any further questions, and we look forward to the opportunity to receive your updated evaluation of our work