We sincerely thank the reviewer for the detailed and constructive feedback, as well as for highlighting the strengths of our work.

Below, we address each point raised:

---

Vital role of offline RL algorithms for post-training pipelines

We specifically focused on the offline regime because we believe that a major reason for the cost inefficiency of online RL post-training is the inability to scale by leveraging diverse offline data (i.e., previously collected samples or offline datasets). This is currently one of the main bottlenecks in RL post-training pipelines. For example, recent large-scale RL post-training efforts (e.g., Grok-4) required compute budgets comparable to pre-training, largely due to the sequential and expensive nature of online data generation.

Thus, even though online data collection will likely continue to play a significant role in post-training pipelines, the effective scaling of training requires an offline-capable algorithm that can significantly amortize this data generation cost by reusing the data generated in previous iterations.

---

Policy improvement algorithms cannot be a viable option for offline training

Policy improvement methods (e.g., PPO, GRPO, RLOO) are inherently online algorithms. While slight off-policy training is possible via importance sampling corrections, this approach introduces high-variance gradient estimates, making optimization unstable at strong off-policiness. More importantly, true offline training is still infeasible with importance sampling because it requires knowing the sampling probabilities of each data point under the behavior policy, which is typically unknown for offline datasets collected from arbitrary sources.

Therefore, we would be grateful if the reviewer could provide examples for their claim that "there already exist prior alignment algorithms with these qualitative properties for practitioners" (in the context of offline pointwise algorithms).

In addition, could the reviewer elaborate on "is only narrowly unique in the alignment space"? Given that it covers an essential gap in stable, offline (or highly off-policy) pointwise methods, which can largely impact the scaling of RL pipelines.

---

Results interpretation

In some cases, performance can seem marginally superior to DPO or SimPO for some metrics, but we elaborate on their significance, noting that we primarily rely on AlpacaEval 2, which evaluates downstream generalization on an unseen dataset with GPT-4 as a judge and is a more robust proxy for human preferences than the reward model. Importantly, we focus on its Length-Controlled (LC) score as it both mitigates length bias and shows the highest correlation with human judgments. Then come ArmoRM rewards, which correlate with AlpacaEval2 but can be "hacked" by trained policies, making them less reliable. Among average reward scores, LC-ArmoRM scores remain more informative since they reduce verbosity bias.

With this metric priority in mind,

• On Llama 8B SFT, QRPO consistently outperforms all baselines. On the high-quality Magpie-Air dataset, the LC-AlpacaEval2 gap between QRPO and SimPO is of the same magnitude as the well-established DPO→SimPO gap, which is widely regarded as a substantial improvement in general chat tasks, confirming that QRPO delivers a significant gain.
• On the Mistral Instruct model, we likely observe capacity saturation. After extensive hyperparameter tuning, DPO, SimPO, and QRPO achieve nearly identical scores on UltraFeedback, with differences within the noise level. Importantly, our results for DPO and SimPO significantly surpass those originally reported in the SimPO paper (Meng et. al., 2024), indicating we have pushed all methods considerably further than previous studies. Thus, the observed saturation likely reflects that the limits of Mistral Instruct’s performance were reached, rather than a limitation specific to QRPO.
• On Magpie-Air with Mistral, similar saturation effects appear, with DPO and SimPO exploiting length bias for non-significant AlpacaEval gains. LC-AlpacaEval scores for QRPO remain higher, consistent with its improved robustness to verbosity bias (Fig. 4).

---

Answers to the questions

Q1

First, note that gradient vanishing does not occur simply due to quantile reward saturation ($\mathcal{R}_q(x, y)=1$). Even when $\mathcal{R}_ q=1$, gradients remain non-zero unless the ratio $\pi_{\theta}/\pi_{\text{ref}}=1/\beta$ is exactly reached (from plugging $\mathcal{R}_q=1$ into the optimal policy expression and approximating $Z_q(x)\approx \beta e^{1/\beta}$ for small $\beta$). Thus, gradient vanishing coincides strictly with policy convergence, not reward saturation.

However, widespread quantile reward saturation (many samples achieving $\mathcal{R}_q=1$) can still reduce the magnitude of the reward signal, potentially slowing training. Clarifying when this can occur:

• In the offline setting, quantile rewards are computed from a static dataset that is crafted by the user. If this dataset contains samples that would saturate the quantile, then a preliminary SFT on this data can be performed to reduce the gap/distribution shift of the initial checkpoint/reference policy for QRPO.
• In the online setting, improved policy outputs may frequently exceed reference completions, causing a quantile reward saturation.

However, this saturation coincides with (or follows) convergence to the optimal policy. This is because the optimal policy under the quantile reward remains supported within the reference distribution, due to the “best-of-N” nature of this optimal distribution (Appendix M).

As a result, the policy cannot easily escape the support of $\pi_{\text{ref}}$, and the model is unlikely to produce completions that consistently outperform all reference completions. Hence, full quantile reward saturation typically does not occur before convergence. We also validate this in practice; our experiments never exceeded an average quantile reward of 0.95–0.98, preserving informative gradients.

Moreover, one can still improve beyond the optimal policy given a reference by either lowering $\beta$ or performing several iterations periodically updating the reference model to the most recent trained policy and re-estimating the quantile CDF. This is a common recipe used with policy fitting methods, for example, used with DPO for the Llama 3 models.

---

Q2

We do not consider RLOO and GRPO to occupy the same design space as QRPO, as neither supports fully offline training, which is a key property of QRPO. Nevertheless, it is useful to compare their characteristics in the online setting.

We opted for a rigorous experimental protocol that required 50k GPU hours to complete to show the effectiveness of QRPO in offline/off-policy regimes and support its unique contribution. As the reviewer acknowledges, adding a thorough evaluation of online and semi-online regimes is outside our scope and would require 2 times more resources due to online sampling costs. We therefore provide a theoretical comparison:

1. Theoretical Similarities:

• QRPO’s gradient in the online case is equivalent to a policy gradient with a baseline, making it closely related to GRPO and RLOO.
• The main difference lies in the reward transformation: QRPO optimizes a quantile-based reward, while GRPO and RLOO use the raw signal.
• Both approaches can be brought closer by either applying an inverse-CDF transform (Eq. 12) to QRPO’s quantiles to approximate the raw scale or, conversely, by using quantile rewards in GRPO.

2. Differences: Optimal Baseline in QRPO

• QRPO provides a closed-form baseline, $\beta \log{Z_ q}$, which equals the expected reward under the optimal policy: $\beta \log{Z_q} = \mathbb{E}_ {y \sim \pi^ * (\cdot \mid y)} [r(x,y)], \quad r(x,y) = \mathcal{R}_ q (x,y) - \beta \log{\frac{\pi_ \theta(y \mid x)}{\pi_ {ref}(y \mid x)}}$. This guarantees minimum gradient variance as the learned policy approaches the optimum, while in GRPO and RLOO baseline is typically an estimator with additional variance.
• This baseline is what allows QRPO to work both online and offline, bridging the policy improvement gradient and the policy fitting gradient.

3. Online/Offline Compute Efficiency:

• In the offline case, QRPO is unmatched, the quantile rewards and the reference model probabilities can be precomputed, therefore only requiring to have the training policy during training. These are the same requirements as SFT and represent ~3x less memory compared to GRPO (at least online) which needs the reference model, sampler, and training policy.
• Online:
  • QRPO can precompute $n$ reference completions per prompt once (unless the reference is updated), reusing them throughout training.
  • GRPO and RLOO require fresh completions at every gradient step, leading to considerably higher compute costs due to repeated online sampling.

---

Q3

We distinguish two distinct interpretations of the question: (i) preserving different reward ranges for different prompts, (ii) extracting the information from outliers.

• (i): QRPO allows for a reward range preserving by applying the additional transform on top of a quantile reward (as proposed in Eq. 12). This transformation can be prompt-specific, e.g., it can be a multiplication by the std of the reference model rewards for each prompt.
• (ii): QRPO is indeed insensitive to the outliers with respect to the reference rewards distribution (i.e., it can't distinguish them from normal boundary values). However, one can try to mitigate this issue by picking a special transformation function (eq. 12). For instance, if one needs to account for negative outliers (i.e., safety penalties), one can use a log transformation, thus heavily penalizing any sample with near 0 quantile reward.

Finally, we note that other algorithms based on normalization (e.g., GRPO) also discard the scale information and reduce sensitivity to outliers.

We thank the reviewer for their time and effort in providing detailed feedback on our submission.

Below, we address each point raised:

---

Motivation

We specifically focused on the offline regime because we believe that a major reason for the cost inefficiency of RL post-training is the inability to scale by leveraging diverse offline data (i.e., previously collected online samples or offline datasets). This is currently one of the main bottlenecks in RL post-training pipelines. For example, recent large-scale RL post-training efforts (e.g., Grok-4) required compute budgets comparable to pre-training, largely due to the sequential and expensive nature of online data generation, making such pipelines difficult to scale.

To bridge this gap, there is a strong need for an algorithm that can effectively leverage offline data, while also:

• Operating directly on pointwise absolute rewards, like the SoTA policy gradient methods, and
• Maintaining a simple and stable optimization objective (not requiring costly partition function estimation or auxiliary model-based approximations)

---

Why QRPO is Needed

1. The policy improvement algorithms (policy gradient, PPO, GRPO) are the main reason for this online inefficiency and cannot use offline data. It is well known that applying PPO-like methods with offline data leads to divergence.
2. The current policy fitting (DPO, SimPO, REBEL) algorithms can use offline data, but do not meet the requirements above. They optimize a relative reward signal via pairwise comparisons, which is not a good replacement for pointwise rewards as mentioned in our background section, due to suboptimal updates, probability mass drifts, and limited relative signal with verifiable reward settings.

To the best of our knowledge, QRPO is the first offline method combining pointwise absolute rewards and stable optimization.

---

Theoretical Aspects

No approximation (hence no bias) in $Z_q(x)$

In QRPO, the partition function $Z_q(x)$ is computed analytically and exactly due to the quantile reward transformation. Thus, QRPO introduces no approximation or bias, ensuring a stable and principled optimization without extra modeling overhead.

The only stochastic element is the empirical quantile reward ${\mathcal R}_q$

For each prompt $x$ we estimate

\hat{\mathcal R}_ q(x,y)=\frac{1}{n}\sum_ {i=1}^{n} \mathbf 1 \bigl\\{ \mathcal R(x,y_i)\le \mathcal R(x,y) \bigr\\}, \qquad y_ i \sim \pi_ {\text{ref}}(\cdot|x).

• Unbiasedness. $\mathbb E[\hat{\mathcal R}_q(x,y)|x,y]=\mathcal R_q(x,y)$.

• Consistency and rate. By the Glivenko–Cantelli theorem the empirical CDF converges uniformly:

  $$\sup_r\bigl|\hat F_{n}(x,r)-F_{\text{ref}}(x,r)\bigr| \xrightarrow[n\to\infty]{a.s.} 0.$$

  The Dvoretzky–Kiefer–Wolfowitz inequality gives a finite-sample bound $\Pr \bigl(|\hat{\mathcal R}_q-\mathcal R_q|>\varepsilon\bigr)\le 2e^{-2n\varepsilon^2}$.

Hence, the only error term we add is zero-mean “label noise’’ that shrinks with $n$.

Unbiased quantile noise keeps the optimum intact

The empirical quantile adds only zero‑mean, additive noise to the true target. In an MSE objective, such noise contributes a constant variance term, leaving the arg‑min, and thus the desired policy, unchanged (standard least‑squares result). Because gradients remain unbiased with finite variance, SGD with the usual diminishing step‑size still converges to that same optimum.

We will include this clarification in the paper to be more explicit about the QRPO convergence properties.

---

Answers to the questions

Q1

The partition function is indeed independent of the prompt; however, the normalization is prompt-dependent. The normalization in QRPO comes from the quantile transformation that recenters rewards with respect to the performance of the reference policy for each prompt separately, allowing $Z$ to be prompt-independent.

---

Q2

We interpret the reviewer’s proposed baseline as $(r(x,y) - \beta \log(\frac{p_ \theta (x \mid y)}{p_ {ref} (x \mid y)}))^2$, since otherwise no KL regularization term would be present.

This loss can recover the correct policy gradient in an online setting, however, in the offline setting, it leads to biased solutions. For example, consider a toy environment with:

• Actions $a_1, a_2$ with rewards $r_1 = 1, r_2 = 2$
• Reference policy probabilities $\pi_\text{ref}(a_1)=0.8, \pi_\text{ref}(a_2)=0.2$
• Offline dataset uniformly sampling both actions.

Optimizing Equation (1) yields the true optimum:

$$p^ * (a_1)=0.595,\quad p^ * (a_2)=0.405$$

while different squared-loss formulations give:

---

Q3

Results Interpretation

1. Metrics Priority

• Our primary evaluation metric is AlpacaEval 2, as it measures downstream performance on a dataset of unseen prompts and uses GPT-4 as a judge, providing a more reliable proxy for human preferences than the training reward model.
• Within AlpacaEval 2, we emphasize the Length-Controlled (LC) score, shown to have the highest correlation with human judgments (Dubois et al., 2024) and to mitigate length bias that many algorithms exploit.
• ArmoRM rewards (measured on a test set) are generally correlated with LC-AlpacaEval2 but we've observed that they are sometimes hacked by trained policies in a non-trivial way. While we manually reviewed outputs to filter obvious hacks, reward scores alone are less reliable.
• For ArmoRM rewards, we similarly value the LC (Length-Controlled) score more since it mitigates length bias

2. Significance of Improvements

• On Llama 8B SFT, QRPO consistently outperforms all baselines. On the high-quality Magpie-Air dataset, the LC-AlpacaEval2 gap between QRPO and SimPO is of the same magnitude as the well-established DPO→SimPO gap, which is widely regarded as a substantial improvement in general chat tasks, confirming that QRPO delivers a significant gain.
• On the Mistral Instruct model, we likely observe capacity saturation. Despite extensive hyperparameter tuning (same budget across methods), DPO, SimPO, and QRPO achieve nearly identical scores on Ultrafeedback, with differences within the noise level. Importantly, our results for DPO and SimPO significantly surpass those originally reported in the SimPO paper (Meng et. al., 2024), indicating we have pushed all methods considerably further than previous studies. Thus, the observed saturation likely reflects that the limits of Mistral Instruct’s performance were reached, rather than a limitation specific to QRPO.
• On Magpie-Air with Mistral, similar saturation effects appear, with DPO and SimPO exploiting length bias for slight (non-significant) AlpacaEval gains. LC-AlpacaEval scores for QRPO remain higher, consistent with its improved robustness to verbosity bias (Figure 4).

Qualitative Analysis

To robustly identify the specific behaviors that contribute to QRPO’s improvements, a thorough qualitative analysis (e.g., an extensive blinded case study with a large sample of examples) would be required. Such an analysis is beyond the scope of this work, where we focused on quantitative metrics that are known to correlate well with human preferences. Additionally, since we exhaustively tuned all algorithms, observed differences in responses are subtle and difficult to characterize qualitatively without careful and systematic examination. We would appreciate it if the reviewer could clarify whether we correctly understood this question.

---

Q4

Qualitative Analysis

Similar to what we noted for chat tasks, such characteristics cannot be robustly identified within our scope. We are open to further clarification from the reviewer.

Mathematical Reasoning and Effect of KL Penalty

1. Role of KL Regularization

• Prior work (Liu et al., 2025) observes that a KL penalty stabilizes training and leads to stronger solutions when the reference policy is periodically updated to recent checkpoints.
• Gao et al. (2022) demonstrate that KL regularization primarily acts like early stopping, without shaping the training trajectory. These results support the view that periodic reference policy resets are sufficient to mitigate potential KL-related issues, making QRPO applicable to reasoning tasks as well.

2. Applicability to Reasoning Tasks

• We have not tested QRPO datasets because 7–8B models are limited in reasoning ability, making comparisons between methods potentially uninformative.
• We chose to evaluate QRPO on code generation instead of mathematical reasoning because this is still a highly technical domain demonstrating QRPO's ability to handle tasks requiring structured reasoning and precision, while being easier and more robust to verify, requiring less computational budget (no judge), and more suitable for 7–8B models.

[1] Pal, Arka, et al. "Smaug: Fixing failure modes of preference optimisation with dpo-positive." arXiv preprint arXiv:2402.13228 (2024)

[2] Liu, Mingjie, et al. "Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models." arXiv preprint arXiv:2505.24864 (2025).

[3] Gao, Leo, John Schulman, and Jacob Hilton. "Scaling laws for reward model overoptimization." International Conference on Machine Learning. PMLR, 2023.

We thank the reviewer for the thorough review and appreciate the detailed comments and pertinent questions.

We address the raised concerns below:

W1: online ablation

• We had preliminary ablations in our early experiments where we added online data to offline data in QRPO training on the Magpie-Air dataset, and we observed a 26% greater reward improvement compared to pure offline training.
• We thank the reviewer for the suggestion and will include this preliminary online–offline ablation result in the appendix to provide additional empirical support to the broader impact of QRPO.
• For us, this was rather a consistency check and an expected behavior, as in general for policy fitting methods, online data is expected to help (Guo et. al., 2024).
• We chose to focus on the offline regime (varying distribution shifts) as this is where the main difficulty arises: where existing policy improvement methods (PPO, etc.) completely fail and where QRPO makes its significant contribution compared to the rest of the policy fitting methods (DPO, etc.) that use pairwise data.
• In fact, the inability to scale the RL training by leveraging offline data (offline or off-policy from previous iterations) is a key bottleneck in current RL post-training pipelines, for example, requiring the post-training of Grok-4 RL to consume compute comparable to pre-training, largely due to costly online data sampling.
• Hence, to "bridge the gap" between online and offline algorithms, we focused on validating the ability of QRPO to keep the characteristic of using pointwise rewards from the online-only policy improvement methods and enabling training with offline/off-policy data.
• Our rigorous experimental protocol to evaluate QPRO for offline/off-policy regimes already required more than 50k GPU hours to complete, which is already very large for an academic lab, and we expect practitioners to expand on the preliminary online-offline ablation results that we had (which require 2 times more resources due to online sampling costs) to further highlight the online and semi-online performance of QRPO.

W2: reward distribution discussion

• QRPO does not rely on the specific distribution of the initial reward. The quantile reward has a uniform distribution independently of the shape of the initial reward distribution (see proof in Appendix E; there are no assumptions on the initial distribution shape).
• We've verified that quantile rewards indeed follow a uniform distribution for all models, datasets, and tasks. We will add the corresponding plots to the appendix.
• ArmorRM rewards for the general chat task are nearly Gaussian on both the Magpie-Air and UltraFeedback datasets. However, for some prompts, it is skewed, and this is why it is not correct to assume it to always be Gaussian and use a closed-form expression for the partition function Z(x) of an assumed distribution without applying a quantile transformation.
• On coding dataset, since the reward is limited to the range from 0 to 1 (test case pass rate), the distribution is often non-Gaussian

Q1: sensitivity to data distribution

• We have discussed the online vs. offline in our answer to W1.
• But what we find also interesting, and have experimented with thoroughly in the scaling ablation (Figure 3), is the sensitivity to offline vs. off-policy (that can be treated as relatively close to online):
  • In our offline regime, data comes from a stronger model, or could be from human annotators, resulting in high-quality samples. Since these samples typically lie in the right tail of the reference model distribution (used for quantile transformation), the quantile reward estimates were less sensitive to the number of reference completions used for CDF estimation. Hence, with offline data, a small number of reference completions suffices for a good signal (we used 1 on Magpie-Air), but scaling reference completions yields limited gains.
  • However, in the off-policy regime, however, data is generated from the reference model, so there is no distribution shift between the training data and reference model completions. This makes the noise in quantile reward estimates decrease significantly when increasing the number of reference completions. Hence, with off-policy data, scaling reference data pre-computation leads to larger performance improvements.

Q2: length bias mitigation:

• We hypothesize that the reduced length bias observed in QRPO (and similarly in REBEL) is primarily due to the methods being able to use the scalar ArmoRM reward signal, as this reward model was trained to penalize verbosity and avoid length exploitation.
• Notably, while DPO and SimPO also rely on the same underlying reward model to generate preference pairs, the conversion of scalar rewards into binary preferences appears to discard part of the anti-verbosity signal, making these methods more susceptible to length-hacking.

Q3: RLOO citation: Thank you for catching this oversight. We will ensure that the appropriate citation for RLOO is included in the revised version of the paper.

We appreciate the reviewer’s constructive feedback, and we think their questions improved our work. We remain open to any further suggestions.

[1] Guo, Shangmin, et al. "Direct language model alignment from online ai feedback." arXiv preprint arXiv:2402.04792 (2024).

We thank the reviewer for their time and for providing a thoughtful and thorough review, particularly from the theoretical perspective.

---

Strengths and Weaknesses

Longstanding Bottleneck and Broader Impact

We appreciate the acknowledgment of our theoretical contributions, especially the closed-form expression for the partition function. As highlighted, this addresses a longstanding bottleneck in the field and was also described as "elegant and rigorous" by Reviewer u3X5.

Additionally, Reviewer Ghhc noted the broader potential impact:

“The approach set out in this paper (i.e., converting an arbitrary scalar reward into a bounded, uniform signal whose partition function is a constant) could be reused elsewhere where we face energy-based or KL objectives with nuisance normalizers.”

Empirical Evaluation

Regarding the empirical evaluation, which the reviewer mentioned was outside their primary area of expertise, we believe the comments from other reviewers offer a helpful, complementary perspective.

• Reviewer Ghhc:

  "The experimental work is careful and high-quality, and appears to follow rigorous methodology to appropriately tune baselines, re-use appropriate reward models, and report numbers with uncertainty estimates throughout."

• Reviewer u3X5:

  "Solid empirical evidence supporting the effectiveness of QRPO as compared to the baselines."

---

Questions

Multi-Level Preference Reward Signal

This is indeed an interesting opportunity. Multi-level (multi-dimensional) rewards could be integrated into QRPO using several conventional strategies (linear projection into a single scalar reward, independent training on each reward, followed by model merging, or training with a linear combination of individual reward losses).

However, there is also an opportunity for a novel approach within QRPO. Specifically, we can recover the quantile of a training completion given the reference completions with a non-linear mapping that takes into account all the relative information between the completions. I.e., the completions would not be ranked independently, but all would be ranked relative to each other. For example, we can use a carefully prompted LLM-as-judge with all the reference completions together with training completions to obtain quantile rewards.

We will add this discussion to the paper, which we believe will significantly benefit practitioners in effectively leveraging QRPO and inspire new directions and open questions for future research.

---

We greatly value the feedback and questions from the reviewer, and we remain open to any additional ideas or recommendations that could further enhance the quality and impact of our work.

We appreciate the reviewer’s thoughtful and encouraging review and address the raised points below.

---

W1: Only 7B–8B Experiments

• We agree with the reviewer that given the potential impact of QRPO, experiments with larger models would make the claims stronger and accelerate confident adoption.
• However, providing a thorough scientific experimental protocol such as the one we followed to formulate our conclusions would have been computationally prohibitive with larger-scale models.
• We opted to put our budget and efforts into such a protocol, which at the 8B scale is already significant and allows comparison with previous papers (notably SimPO), rather than conduct a more limited number of experiments with larger-scale models.
• Our protocol includes diverse datasets and models, and appropriately tuned baselines (over 200 trained models for each baseline), and used over 50k GPU hours, which is already very high for an academic project.

---

W2: Calibration of the Reference Policy

• The reference policy is typically taken to be the initial training model, so the calibration can be seen with respect to the initial training checkpoint.
• If the training data is purely offline and strictly much better than the initial checkpoint, then we can perform SFT on this data first to reduce the distribution shift and make the initial checkpoint for QRPO, thus the reference policy. more "calibrated" to allow effective training.
• This is what we have observed in our experiments in Table 6, for example, where models trained on Magpie-Air were more performant by having a preliminary SFT on the chosen answer (whose quantile is around 0.91) before QRPO. (Followed by the models trained offline without the preliminary SFT, and then the SFT models—so the performance is not due to SFT alone).
• When the data is collected iteratively from the trained model, this data can have quantiles that progressively become in the top quantile of the reference model/initial checkpoint.
• However, this saturation coincides with convergence to the optimal policy. This is because the optimal policy under the quantile reward remains supported ("calibrated") within the reference distribution, due to the “best-of-N” nature of this optimal distribution (Appendix M).
• As a result, the policy cannot easily escape the support of the reference model, and the model is unlikely to produce completions that consistently outperform all reference completions. Hence, there is no dramatic miscalibration before convergence.
• We also validate this in practice; our experiments never exceeded an average quantile reward of 0.95–0.98.
• Moreover, when gains may start to diminish, observe that the reference policy can be iteratively replaced by the most recent/performant checkpoint during training, with new reference completions to estimate the quantile reward, which would be better "calibrated". This is a common recipe used with policy fitting methods. For example, iterative DPO was used to train the Llama 3 models.
• We thank the reviewer for raising this point, and we will add this discussion to the paper to make it easier for practitioners to understand how to use QRPO.

We appreciate the reviewer’s insightful feedback and thoughtful points. We also remain receptive to any further suggestions or recommendations that may help strengthen the quality and impact of our work.