Discriminative Policy Optimization for Token-Level Reward Models

Thank you for your detailed review and insightful questions on our paper. Below, we address your concerns in detail.

Q1: It seems that for both PPO and REINFORCE, the author report the detailed training hyper-parameter but lacking the details about how the hyperparameters are selected. Also the training details are missing for methods like DPO / SimPO

A1：Thank you for your valuable suggestions. We conducted a thorough hyperparameter tuning process to determine the optimal values for our experiments.

• For REINFORCE-based Methods: We tuned the learning rate within the range [5e-7, 1e-6, 5e-6] and found that 1e-6 consistently yielded the best performance.

• For PPO-based Methods: We explored learning rates in the range [1e-6, 5e-6, 1e-5] and identified 5e-6 as the most effective.

• For Q-RM: We searched for the optimal $\beta$ parameter from [0.1, 0.2, 0.5] and the $\gamma$ parameter from [1.0, 2.0, 5.0], concluding that $\beta = 0.2$ and $\gamma = 2.0$ produced the best results.

• For Baseline Methods (e.g., DPO and SimPO): We followed the recommended configurations from their original papers as well as the default settings from widely used repositories (e.g., Hugging Face’s TRL repository).

We will include these details in the revised version of the paper.

Q2: Pass@N is not a regular metric for math problem solving (because usually the users of LLM are not expected to verify the answer).

A2: Thank you for the thoughtful comment. While we agree that users typically expect a single correct answer, we clarify this concern as follows.

Pass@N measures the probability that at least one of N sampled outputs is correct. This metric is particularly useful for assessing a model's best-case capability under stochastic generation. Pass@N has become an increasingly popular metric for evaluating LLMs on mathematical tasks (e.g., GSM8K, MATH) [1][2][3], as it reflects the likelihood of generating a correct solution within a limited number of attempts. This provides valuable insights into the model’s reasoning ability and output diversity.

To directly address your concern, our study also reports Pass@1 alongside Pass@N. Pass@1 corresponds to real-world scenarios where only a single response is returned to the user, making it a practical measure of immediate correctness.

[1] Kimi k1.5: Scaling Reinforcement Learning with LLMs. Arxiv, 2025.

[2] rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking. Arxiv, 2025.

[3] Large language monkeys: Scaling inference compute with repeated sampling. Arxiv, 2024.

Thank you for your detailed review and insightful questions on our paper. We greatly value your feedback and appreciate your recognition of our work's advancements. Below, we address your concerns in detail.

Q1: Lines 78–80 mention “under certain assumptions.” These assumptions should be explicitly highlighted and explained in the main body of the paper to ensure clarity.

A1: Thank you for your valuable suggestion.

The assumption in question refers to the trajectory entropy of the optimal policy being close to zero, as stated in Proposition 3.2. This condition constrains $\mathcal{V}(\tau)$ within a narrow range, facilitating the effective training of Q-RM using Eq. 17 in the paper. Furthermore, as demonstrated in Proposition 3.3, computing advantage functions with Q-RM is equivalent to utilizing the optimal Q-functions.

We will ensure that this assumption is explicitly highlighted and more clearly explained in the revised version of the paper.

Q2: Including error bars in the experimental results would better demonstrate the robustness of the method and provide more insight into the variability of the outcomes.

A2: We appreciate the reviewer’s insightful suggestion and agree that statistical indicators can offer a clearer perspective on method stability. In our study, we evaluate Q-RM across three task categories: mathematical reasoning (GSM8K, MATH), machine reading comprehension (QA-Feedback), and instruction-following (AlpacaEval 2.0). For GSM8K and MATH, we report Pass@N metrics, which inherently account for statistical variation by measuring the probability of obtaining at least one correct sample from N generated completions. These values, computed over multiple sampled responses (e.g., N=8,16), serve as empirical success rates over stochastic rollouts, indirectly capturing performance variance in a manner analogous to confidence intervals.

Following the reviewer’s suggestion, we have now included standard error for QA-Feedback and AlpacaEval 2.0 as follows.

QA-Feedback:

AlpacaEval 2.0:

More results will be included in the revised paper.

Q3: The paper lacks human evaluation.

A3: Thank you for raising this important point. We would first like to clarify our evaluation strategy.

For instruction-following tasks, we employ AlpacaEval 2.0, which utilizes the LLM-as-a-judge framework. This approach has gained traction as a scalable, cost-effective, and reproducible alternative to traditional human evaluation. Recent studies [1][2][3] have demonstrated that LLM-as-a-judge evaluations strongly correlate with human judgments, particularly when using advanced reference models like GPT-4 as the evaluator. This makes it a practical and reliable substitute for large-scale assessments. Furthermore, AlpacaEval 2.0 incorporates length-controlled win rate (LC win rate) metrics to further reduce bias, enhancing the evaluation’s reliability and interpretability.

We acknowledge the reviewer’s point that human evaluation remains the most direct and interpretable method for assessing LLM performance, particularly for subjective or nuanced outputs. However, due to time constraints during this rebuttal phase, we are unable to complete and present human evaluation results within a few days. We will include these results in the revised version of the paper.

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NIPS 2023.

[2] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. COLM 2024.

[3] Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering. ISSTA 2025.

Q4: The paper does not provide sufficient details about the hyperparameters required to reproduce the method.

A4：Thank you for your valuable comment. We provide detailed hyperparameter settings in Appendix D. For baseline methods, we adhere to the recommended configurations from their original papers and the default settings from widely used repositories (e.g., Hugging Face’s TRL repository). For example, in DPO, the beta parameter is set to 0.1, while in SimPO, beta is set to 1.0 and gamma to 2.0. Moreover, for GRPO, RLOO, and RFT, we generate four responses per instruction, following the approach in [1].

We will further clarify and supplement these details in the revised version of the paper.

[1] Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. ACL 2024.

Thank you for your thoughtful review and valuable feedback. Below, we address your concerns in detail.

Q1: Regarding equivalence claim in Proposition 3.3 and assumptions.

A1: We address it from two key aspects:

• According to Proposition 3.3, the optimal Q-function and the optimal logits from Q-RM differ only by a constant offset. The left-hand side of Eq. 18 is exactly the definition of the advantage function, using $Z^*(s_t, a_t)$ to estimate the advantage is equivalent to using $Q^*(s_t, a_t)$.

• According to Proposition 3.2, we assume that trajectory entropy $\mathcal{H}^*(\tau)$ is close to zero, which constrains $\mathcal{V}(\tau)$ within a narrow range. This allows Q-RM to be effectively trained using Eq. 17.

We will incorporate these clarifications into revised paper.

Q2: Regarding the effectiveness of Q-RM on generalization ability.

A2: Thank you for your valuable suggestion.

In response, we assess Q-RM's OOD generalization by training ORM, DPO-RM, and Q-RM on the MATH pairwise dataset and using OOD dataset Math10K [1] as RL training instructions. Reward models are based on LLaMA-3-70B-Instruct, while policies are based on LLaMA-3.2-3B-Instruct. As shown in the table below, Q-RM consistently outperforms ORM and DPO-RM, demonstrating its effectiveness in handling OOD scenarios. We will incorporate these OOD results in the updated paper.

[1] LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models. EMNLP 2023.

Q3: Regarding the model size of Q-RM.

A3: We address it from the following perspectives:

• Model Size of Q-RM. It is well established that larger models are more effective for ORM compared to smaller ones. Moreover, this work focuses on training a fine-grained reward model for RL training, which presents a greater challenge than ORM. To ensure broad effectiveness across different policy optimizations, a robust backbone for the reward model is essential. Therefore, we selected the 70B model as the backbone for Q-RM.

• Additional Experiments. To further analyze the impact of model size on Q-RM, we use LLaMA-3-8B-instruct as the backbone for both Q-RM and policy, comparing it against ORM and DPO-RM. As shown below, Q-RM remains effective with smaller architectures but excels with the 70B backbone, supporting our choice. Additional results will be added to revised paper.

Q4: Regarding KL loss on $\mathcal{L}^{CLIP}$.

A4: While some works (e.g., [1]) incorporate KL divergence directly into the final training objective $\mathcal{L}^{CLIP}$, it is more common practice [2][3][4] to integrate it into the trajectory rewards. Therefore, in the PPO+Q-RM training framework, we followed this widely adopted approach.

[1] Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv 2024.

[2] REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models. ArXiv 2025.

[3] Delve into PPO: Implementation matters for stable RLHF. NeurIPS Workshop 2023.

[4] https://github.com/huggingface/trl

Q5: Can QRM and policy models share weights?

A5: Yes, Q-RM can share a backbone with the policy model to save memory, but this limits generalizability and reduces learnable parameters, potentially weakening reward signal capture. Thus, we chose separate backbones for Q-RM and the policy model.

Q6: Have you evaluated QRM on RewardBench to quantify its reward modeling accuracy vs. baselines?

A6: Q-RM focuses on finer-grained rewards for RL training, unlike ORM's sequence-level rewards, so we did not originally test it on RewardBench. However, to address this concern, we evaluated Q-RM against ORM and DPO-RM using RewardBench.

As shown in the table below, Q-RM performs worse than ORM and DPO-RM in terms of pairwise accuracy. This result aligns with Q-RM’s objective, which prioritizes fine-grained reward modeling and improved policy optimization over maximizing pairwise accuracy.

Furthermore, high pairwise accuracy in reward models does not necessarily translate to better policy optimization [2]. This is further illustrated in Figure 2(b) of our paper, where Q-RM, despite having the lowest pairwise accuracy, achieves the highest policy optimization performance.

We will include the results and discussion in the revised version of the paper.

[1] What Makes a Reward Model a Good Teacher? ArXiv 2025.

Thank you for your thoughtful review and valuable feedback. Below, we address your concerns in detail.

Q1: Regarding results discrepancy on MATH dataset.

A1: We clarify that the discrepancy in results for Llama-3.2-3B-Instruct on MATH can be attributed to the differences in evaluation frameworks. Nvidia's reported results for Llama-3.2-3B-Instruct are based on Meta's official evaluation framework, as detailed in documentation. While our paper adopts OpenCompass, which is widely used in recent research [1~3]. Below, we outline the key differences between these two frameworks:

• Use of LLM-as-Judge: Meta's evaluation employs an extra LLM to act as a judge, using an equivalence template to determine whether two expressions are equivalent. This introduces an external dependency into the evaluation process. OpenCompass does not rely on an extra LLM for evaluation.

• Use of Latex Parser: Meta's framework utilizes Sympy to parse and verify the correctness of LaTeX equations. OpenCompass does not incorporate such parsing mechanisms, which can lead to lower reported accuracy due to stricter evaluation criteria.

• Prompting and Sampling Strategy: Meta's framework employs zero-shot CoT prompting with top-p sampling, while we adopt OpenCompass to use a zero-shot greedy decoding without additional CoT templates, eliminating the influence of different prompting strategies and randomness on the evaluation.

In summary, we selected OpenCompass for its widespread use in recent studies [1-3], ensuring comparability. In response to the reviewer, we are working to reproduce the results using Meta's framework. Since this requires retraining the policy models, it will take some time, and we will provide an update within 48 hours.

[1] Compression Represents Intelligence Linearly. COLM 2024.

[2] Internlm2 technical report. Arxiv 2024.

[3] Internlm-math: Open math large language models toward verifiable reasoning. Arxiv 2024.

Q2：Regarding Q-RM training with DPO. Is the 70B being distilled into the smaller model? Use same base model for both RM and Policy.

A2: Thank you for your thoughtful comments. We address them from the following perspectives:

• Q-RM is Not Trained Using DPO. We would like to clarify that Q-RM is not trained using DPO. In this work, Q-RM is trained using the proposed discriminative policy, as outlined in Equation 17 of the paper. This approach explicitly decouples reward modeling from language generation. Q-RM and DPO both use preference data but differ in how they model token-level rewards.

• Model Size of Q-RM. It is well established that larger models are more effective for ORM compared to smaller ones, and training fine-grained reward models for RL is even more challenging. To ensure broad effectiveness across different policy optimizations, a robust backbone for the reward model is essential. Therefore, we selected the 70B model as the backbone for Q-RM.

• Relationship with Knowledge Distillation. Our approach follows standard reward modeling in RL, focusing on fine-grained supervision for policy optimization. Unlike knowledge distillation, we don't directly distill the reward model into the policy.

• Additional Experiments. To further analyze the impact of model size on Q-RM, we use LLaMA-3-8B-instruct as the backbone for both Q-RM and policy, comparing it against ORM and DPO-RM. As shown below, Q-RM remains effective with smaller architectures but excels with the 70B backbone, supporting our choice. Additional results will be added to revised paper.

Q3: Regarding the averaging results on GSM8K and MATH.

A3: We agree that GSM8K is simpler than MATH, so averaging their results isn't ideal. Our original intent was to report overall results on math domain. To avoid confusion, we will present separate results for GSM8K and MATH in Table 1 without averaging their performance.

Q4: Regarding the credit assignment in Q-RM.

A4: We acknowledge that credit assignment primarily pertains to the estimation of advantages, and we agree that Q-RM does not directly compute advantages. However, as demonstrated in Proposition 3.3, the rewards generated by Q-RM maintain a linear relationship with the optimal Q-function. This relationship enables the estimation of advantages using Q-RM. We will clarify how Q-RM facilitates credit assignment in the revised paper.

Q5: Regarding the related paper.

A5: Thank you for referencing VinePPO. We recognize its key insight on accurate credit assignment in RL fine-tuning of LLMs and will cite it while discussing its findings in our revised paper.