Author Response for Reviewer kkSM

We thank the reviewer for highlighting these points.

Novelty of this paper

While simple, PF-PPO’s contribution lies in its universal effectiveness for RLHF noise mitigation. Key innovations:

1. Empirical Validation: Extensive experiments across domains (see extended results) demonstrate consistent improvements.
2. Practical Impact: Addressing reward noise—a critical but underexplored RLHF challenge—via a task-agnostic approach.
   We also refer the reviewer to check the new experiment results on various tasks (in “Response to All Reviewers”) and take our effort to demonstrate universal effectiveness of our method into account.

Lack of theoretical derivation

Though heuristic, PF-PPO is motivated by empirical evidence that mid-reward samples harm convergence (likely due to conflicting gradients). Future work will formalize this via gradient variance analysis.

Why sample quality matters in PPO?

PPO relies on accurate advantage estimates. Noisy rewards distort advantage estimates, leading to suboptimal updates. Ablations (e.g., PF-PPO vs PPO-M) show that filtering unreliable samples stabilizes training by ensuring gradients derive from high-confidence data.

Response to All Reviewers

We sincerely thank the reviewers for their constructive feedback and insightful suggestions.

One major advantage of our method is its simplicity, universality, and effectiveness. To further validate the broader effectiveness of our method, we conducted experiments across diverse domains using Doubao-25k (policy and reward model backbone). Tasks included logic reasoning, math, code generation, STEM problems, complex tasks, instruction following, knowledge QA, and language understanding. Each task has distinct evaluation sets and verifiers to assess response correctness. Results (accuracy improvement over vanilla PPO) are shown below. Statistically significant changes (exceeding $\pm 0.5$, based on test case counts) are bolded. These results demonstrate PF-PPO’s consistent effectiveness across tasks.

We thank the reviewers for their rigorous feedback, which has strengthened our analysis. The revised manuscript will incorporate these responses, clarify methodological details, and emphasize PF-PPO’s broader applicability.

Author Response for Reviewer 2Fwc

We appreciate the reviewer’s questions.

Why does BW perform the best?

BW selects extreme samples with high/low rewards, which are most reliably aligned with actual scores (Fig. 1). Mid-reward samples often mix correct/incorrect elements (e.g., non-standard solutions with partial correctness), confusing the policy. BW maximizes the proportion of high-confidence samples, and thus improves learning efficiency.

Why using the coefficient of determination (R2), why it measures the reliability, and how to calculate it in the experiments.

R² measures how well the reward model ranks samples by their true scores/quality. The filtration strategy with high R² ensures remaining samples provide reliable learning signals (i.e., those rewards can well indicate the true quality of the responses). Calculation steps:

• Generate multiple responses per prompt using the SFT policy.
• Filter responses using the given strategy (e.g., BW).
• Compute R² between reward scores and ground-truth scores.

Is the reward model $r_\phi$ mentioned in line 144 the same as the reward model $R_\phi$?

This was a typo, and it should be $R_\phi$ in line 144. We will correct this in the revised manuscript. The reward model in line 144 should be $R_\phi$ (not $r_\phi$). This will be corrected in the manuscript.

Response to All Reviewers

We sincerely thank the reviewers for their constructive feedback and insightful suggestions.

One major advantage of our method is its simplicity, universality, and effectiveness. To further validate the broader effectiveness of our method, we conducted experiments across diverse domains using Doubao-25k (policy and reward model backbone). Tasks included logic reasoning, math, code generation, STEM problems, complex tasks, instruction following, knowledge QA, and language understanding. Each task has distinct evaluation sets and verifiers to assess response correctness. Results (accuracy improvement over vanilla PPO) are shown below. Statistically significant changes (exceeding $\pm 0.5$, based on test case counts) are bolded. These results demonstrate PF-PPO’s consistent effectiveness across tasks.

We thank the reviewers for their rigorous feedback, which has strengthened our analysis. The revised manuscript will incorporate these responses, clarify methodological details, and emphasize PF-PPO’s broader applicability.

Author Response to Reviewer DFFh

We thank the reviewer for the positive and constructive feedback.

W1: Theoretical Motivation

While PF-PPO is empirically driven, we ground our approach in the signal-to-noise ratio principle: ambiguous samples (mid-reward) introduce conflicting gradients during PPO updates, destabilizing training. By filtering these, PF-PPO reduces variance in policy updates. We will formalize this intuition in future work by analyzing gradient variance under different filtration strategies.

W2: Dependency on Reward Model Quality

We believe that our algorithm will yield improvement over different rewards (either generated by reward models or provided by oracles) since filtering out less reliable samples should help across different settings. One example is that, even with perfect rewards (Table 4), PF-PPO improves performance by filtering suboptimal but correct samples (e.g., verbose code), allowing the policy to prioritize concise, high-quality solutions (see Appendix B). Please also refer to the new set of our experiment results shown in the post “Response to All Reviewers” which indicate that our method is robust to rewards for different fields.

Response to All Reviewers

We sincerely thank the reviewers for their constructive feedback and insightful suggestions.

One major advantage of our method is its simplicity, universality, and effectiveness. To further validate the broader effectiveness of our method, we conducted experiments across diverse domains using Doubao-25k (policy and reward model backbone). Tasks included logic reasoning, math, code generation, STEM problems, complex tasks, instruction following, knowledge QA, and language understanding. Each task has distinct evaluation sets and verifiers to assess response correctness. Results (accuracy improvement over vanilla PPO) are shown below. Statistically significant changes (exceeding $\pm 0.5$, based on test case counts) are bolded. These results demonstrate PF-PPO’s consistent effectiveness across tasks.

We thank the reviewers for their rigorous feedback, which has strengthened our analysis. The revised manuscript will incorporate these responses, clarify methodological details, and emphasize PF-PPO’s broader applicability.

Author Response to Reviewer Rkmi

We thank the reviewer for the careful investigation and fruitful discussion.

Generalization to Messy Reward Scenarios

While our experiments focus on code and math tasks with verifiable rewards, we acknowledge the importance of evaluating PF-PPO in scenarios with noisier or subjective rewards (e.g., safety, helpfulness). Our core insight—filtering ambiguous samples to retain high/low-reward extremes—is task-agnostic and relies only on the relative reliability of reward signals. Moreover, our additional experiment results shown in the post “Response to All Reviewers” should also support this claim since the tasks such as language understanding and instruction following are associated with different levels of reward accuracy (e.g., involving subjective rewards).

Response to All Reviewers

We sincerely thank the reviewers for their constructive feedback and insightful suggestions.

One major advantage of our method is its simplicity, universality, and effectiveness. To further validate the broader effectiveness of our method, we conducted experiments across diverse domains using Doubao-25k (policy and reward model backbone). Tasks included logic reasoning, math, code generation, STEM problems, complex tasks, instruction following, knowledge QA, and language understanding. Each task has distinct evaluation sets and verifiers to assess response correctness. Results (accuracy improvement over vanilla PPO) are shown below. Statistically significant changes (exceeding $\pm 0.5$, based on test case counts) are bolded. These results demonstrate PF-PPO’s consistent effectiveness across tasks.

We thank the reviewers for their rigorous feedback, which has strengthened our analysis. The revised manuscript will incorporate these responses, clarify methodological details, and emphasize PF-PPO’s broader applicability.