Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards

We are very thankful to the reviewers for their time and for their insightful comments.

While the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:

• a lack of diversity in the experimental setup (not enough models, not enough datasets), and
• a lack of comparison to popular RL algorithms.

We want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.

Our initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.

We have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):

Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines. This extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)

In our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. We extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)

We consistently observe the same two phenomena highlighted in Figure 3:

• When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in Table 1).
• As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).

The general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.

Table 2: Comparison of off-VPG and GRPO, off-policy (10000 steps, with evaluation every 200 steps)

We conduct in Table 2 a performance comparison between GRPO and off-VPG. We use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2. For off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.

Under these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.

Note that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.

If the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).

Note that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.

--- End of general response ---

• First, I'm curious why the authors chose to study the surrogate loss function (1). [...] I'm wondering if the analysis will be altered if a KL regularized or clipped objective is used (as in PPO objective).

It was precisely the simplicity of the proposed loss which raised our interest: we wanted to study the potential impact of the simplest lever of action that we could think of, namely our baseline V. Adding a KL regularization or an importance ratio would certainly dramatically impact the theoretical analysis, though we expect that a KL regularization would mainly have the effect of keeping the trained policy closer to th reference policy, as in other algorithms. We leave this for future work. Note however that we have included preliminary comparisons to GRPO in our general response (Tables 2 and 3), where our loss seems to be competitive.

• It seems not very clear to me what Theorem 4.2 implies. In particular, when $V<V^\mu$, the optimal policy seems to be a reweighting of the sample policy $\mu$ by reward minus value $r(y)-V$, but for $V\geq V^\mu$ I feel it very hard to interpret the results.

In the case $V<V^\mu$, the limit policy (which is not necessarily optimal) is more precisely the positive part of a reweighting of $\mu$ minus a constant. The key takeaway is that the probability mass of an action $y$ w.r.t. the limit policy is a somewhat complicated increasing function of $\mu(y)(r(y)-V)$. When $V\geq V^\mu$, the limit policy is generically going to be a singleton (generically w.r.t. the choice of rewards and of $\mu$): large $V$s lead to a loss in diversity in the limit policy.

• What if we do adam or other adaptive algorithm in the experiments? Would the effect of different $V$ be mitigated?

In fact, our LLMs experiments are already conducted using AdamW, as is common when training large transformers (this is stated in Appendix B). Figure 4 shows that the effect is not mitigated.

• The "Intuition" paragraph is indeed intuitive, [...] for this kind of rewards?

All of our theorems hold for continuous rewards (we never assume that the rewards are discrete). We only give the example of $\{0,1\}$ rewards to help the readers' intuition, but the general idea is the same in the case of continuous rewards: letting $V$ be large puts comparatively more emphasis on the trajectories with the lowest rewards, and vice versa.

• Does the same collapse phenomenon observed in on-policy algorithms? I'm also curious on how the proposed method performs comparing to on-policy algorithms?

Regarding the impact of the baseline $\delta V$ on the risk of collapse, we have run some very preliminary experiments in an on-policy setting and observed similar results, but the risk of collapse seems to be more dependent on the choice of model, hyperparameters, and even possibly codebases (as many efficient RL codebases are never 100% on-policy to minimize the downtime of the worker Gpus tasked with generating training trajectories). These are still preliminary experiments from which we do not draw definitive conclusions. We have also run a comparison with GRPO in an on-policy setting, reported in Table 3 below.

Table 3: Comparison between off-VPG and GRPO, on-policy (10000 steps with evaluation every 200 steps, bold values indicate that the run collapsed during training)

Though it it not the main focus of our work, we also conducted a comparison between the off-VPG algorithm and GRPO in an on-policy settings to assess whether off-VPG remains competitive. The results are presented in Table 3. The experiment uses the MATH dataset, with hyperparameters identical to those used in Table 2, except for the update frequency of the actor policy. In this experiment, updates occur at every gradient step, unlike previous experiments where updates occurred every 250 steps.

We run two seeds for each configuration and report the highest score achieved on the test set. All learning curves increase steadily and plateau after approximately 3000 steps. The plateau is consistently higher for off-VPG than for GRPO, across both seeds and models. We also observe a sharp performance collapse during training in 3 out of 4 GRPO runs (highlighted in bold in the table), while no such collapse occurs with off-VPG.

These results suggest that off-VPG remains a robust and effective choice in on-policy settings, with the added benefit of a simple implementation.

We are very thankful to the reviewers for their time and for their insightful comments.

While the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:

• a lack of diversity in the experimental setup (not enough models, not enough datasets), and
• a lack of comparison to popular RL algorithms.

We want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.

Our initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.

We have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):

Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines. This extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)

In our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. We extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)

We consistently observe the same two phenomena highlighted in Figure 3:

• When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in Table 1).
• As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).

The general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.

Table 2: Comparison of off-VPG and GRPO, off-policy (10000 steps, with evaluation every 200 steps)

We conduct in Table 2 a performance comparison between GRPO and off-VPG. We use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2. For off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.

Under these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.

Note that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.

If the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).

Note that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.

--- End of general response ---

• Limited model and task diversity: only one LLM architecture (Llama-3 8B) and one dataset (MATH). and
• Missing baselines: lacks direct comparison to IS-PPO, GRPO, or regularized PG, so relative performance ceiling is unclear.

We have ran additional experiments with 2 other models, one additional dataset, and a baseline (GRPO) (see the general response).

• Statistical under-reporting: LLM results use three seeds without CIs; bandit plots omit error bars.

Figure 3 was only ran with 3 seeds because training $8\cdot 3$ 8B models is quite expensive. Do you think that showing intervals representing the variance of these trajectories would improve the figure? In Figure 4, we show all trajectories for 7 different seeds to prove the robustness of the phenomenon studied to randomness. In the bandit setting, we study the expected version of the algorithm: as such, the gradient descent is non-stochastic. The only randomness comes from the drawing of the rewards associated to each arms, i.e. when definining the bandit task. Combining results corresponding to different tasks (e.g. with error bars) yields a graph that is very hard to interpret. We have observed that the general appearance of the curves is very stable w.r.t. the random seed; should we add variants of these curves for other random seeds in the Appendix?

• Code not yet released, reducing reproducibility.

Our code is shared with another team within our lab; we should be able to release it a month (perhaps even in time for the camera-ready version).

• Lines 596–599: The Lyapunov function is called “$\Phi_t$ is bounded above on the simplex”. Strictly speaking $\Phi_t$ is concave when $b>0$ ; here “above” could be re-worded as “has a finite supremum”.

Thank you for this wording suggestion.

• Lines 628–631: The bound $|\nabla^2 F|\leq 2b$ $\rightarrow$ step-size $<1/b$. Readers may ask why the factor 2 disappears; worth stating that any constant $<2b$ suffices, but $1/b$ is chosen for margin.

Thank you for pointing out this potential confusion. The factor of 2 disappears because we are using the bound on the step size, $< 2 / \| \nabla^2 F \|$. We are not aware of a tighter result.

• In the third statement of Theorem 4.3, does the author mean that by setting the value of $V$ within a certain range, we can ensure that the policy value converges to the optimal value? If so, the use of the phrase “if and only if” is quite confusing.

As you guessed, the statement should be understood as "the limit policy is optimal if and only if $V<V_{0,\mu}$". We agree that the phrasing is awkward, and will change it.

• Proofs rely on a finite-arm bandit with a tabular softmax policy; behavior with function approximation and delayed updates remains open. The behaviour with respect to delayed updates is described in the case of finite-arm bandits by Thm 4.3. We agree that going from a tabular setting to the case of LLMs is a non-trivial change, which our theory does not fully capture (though our experiments show that the intuition from bandits still applies in the case of LLMs). In our defense, there is no agreed upon mathematical framework that convincingly captures the regularity of transformers with respect to the prompts $x$ (e.g. the fact that similar prompts are treated similarly), and such a framework would be needed to reach theoretical conclusions regarding LLMs.

• The monotonic-improvement claim hinges on picking a baseline $V$ that is less than the unknown expected return $V^\mu$, yet in real-world RLHF that return is buried under reward noise, so telling practitioners to “just choose a slightly smaller $V$” simply dumps the toughest task back onto them.

In the case of LLMs, which is our main motivation, this problem is partially averted thanks to the additive renormalization by the prompt-dependent average reward $V^{\mu(\cdot |x)}$, which we estimate with the group-average $\hat{V} = \frac{1}{G}\sum_{i=1}^G r(y_i,x)$: $\mathbb{E}[\frac{1}{G} \sum_{i=1}^{G} (r(y_i,x) - (\hat V + \delta V)) \log(\pi(y_{i}|x))]$, where the expectation is w.r.t. ${x \sim D, \{y_i\}_{i=1}^{G} \sim \mu(.|x)}$. As an estimator of $V^{\mu(\cdot |x)}$ is substracted to the rewards for prompt $x$, the expected return after the renormalization is $0$, which means that the critical value now becomes $\delta V = 0$ uniformly across the prompts $x$s. Our experiments (including the new ones) show that $\delta V =-0.1$ seems like a safe bet, though we do not claim that it is optimal.

We are very thankful to the reviewers for their time and for their insightful comments.

While the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:

• a lack of diversity in the experimental setup (not enough models, not enough datasets), and
• a lack of comparison to popular RL algorithms.

We want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.

Our initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.

We have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):

Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines. This extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)

In our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. We extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)

We consistently observe the same two phenomena highlighted in Figure 3:

• When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in Table 1).
• As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).

The general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.

Table 2: Comparison of off-VPG and GRPO, off-policy (10000 steps, with evaluation every 200 steps)

We conduct in Table 2 a performance comparison between GRPO and off-VPG. We use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2. For off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.

Under these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.

Note that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.

If the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).

Note that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.

--- End of general response ---

• The conclusion that the baselines [...] contribution of the theoretical analysis slightly.

It is true that the observation that baselines introduce biases is not novel; our main contribution here is a theoretical analysis of the effects of such a bias, as well as empirical confirmations that such biases can have a positive impact. Regarding the final conclusion, it is in fact very common to hear that negative trajectories can be just as important, or even more important than positive ones (see e.g. [1]). A common missconception is that penalizing negative samples has the effect of encouraging diversity and exploration, because the probability mass diverted from the penalized action would be somewhat uniformly distributed over other actions. This is in fact not the case, as shown by our results: when penalizing a negative action, most of the probability mass goes to the modes of the distribution $\pi$, as visible in the proof of Thm 4.2, which leads to the lack of diversity described in Thm 4.2 ("the rich get richer"). [1] The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning, Zhu et al.

• The off-VPG is evaluated merely with one model on one dataset. Thus, the experimental support for the algorithm is limited.
• and
• Should the experiments be [...] Lack a more comprehensive experimental validation.

We agree; see the additional experiments in our general response.

• This off-policy algorithm design does not specifically target LLMs and lacks discussions related to existing off-policy RL algorithms.

We see the simplicity of our algorithm (combined with its good performance), which is not specific to LLMs, as an advantage. Are there any particular works that you think would be good additions to our related works section (besides [1])?

• There are relevant discussions [...].

Thank you very much, we will be sure to add this work to our related works section.

• Off-VPG does not account for the relationship between behavior policy and current policy [...] For instance, naive Off-Policy PG [2].

It is common in off-policy RL to correct bias with an importance ratio as follows $\mathrm{E}_{y \sim \mu}[\frac{\pi(y)}{\mu(y)}A(y)]$, where $A(y)$ is some advantage term (e.g. the reward itself). In the context of LLMs, the action $y$ is a long sequence of tokens $t_1,\ldots, t_k$, and the importance ratio $\frac{\pi(y)}{\mu(y)} = \frac{\pi(t_1) \pi(t_2|t_1)\ldots}{\mu(t_1) \mu(t_2|t_1)\ldots}$ is often almost $0$ or extremely large when $\pi \neq \mu$, which makes it impractical. In practice, the ratio is usually clipped on a per-token basis, e.g. in PPO or GRPO. As GRPO is considered one of the sota losses, we compare it to our algorithm in our additional experiments (see the general response).

• Some specific implementation details for Off-VPG are missing. For example, [...] into rejection sampling fine-tuning?

In Definition 4.1, we define the algorithm that we call off-VPG, in which the behaviour policy remains fixed. The behaviour of this algorithm is described in Thm 4.2, and illustrated in the bandits setting in Figure 1. In the paragraph "Policy improvement scheme with off-VPG", we describe the derived policy improvement scheme in which the behaviour policy is updated with the (current) trained policy $\pi$ with a delay (i.e. every N gradient steps on $\pi$, we update $\mu \leftarrow \pi$), and which we also call "delayed updates setting". Thm 4.3 applies to this case, and Figures 2, 3 and 4 illustrate it in the bandits and the LLMs setting respectively.

• There is an absence of comparisons [...] than on-policy techniques." and
• There is no comparison [...] between performance and efficiency?

Our phrasing might have been overly enthusiastic. The goal of this article was not to prove that off-policy RL is better than on-policy RL (e.g. in terms of performance/efficiency trade-off). Our point of view was rather: practitioners often want or have to train off-policy (to benefit from some of the advantages of off-policiness, or because they are constrained by their setup), which makes off-policy RL worth studying. We will rephrase this sentence in a more neutral way. Though a rigorous study of the trade-offs between on-policy and off-policy RL is beyond the scope of this article and left to future work, we note that our preliminary experiments suggest that off-policy training (with off-VPG or with GRPO) with an update frequency of 250 gradient steps for the actor policy leads to final scores that are similar to those obtained with on-policy training (e.g. compare Tables 1 and 2 from our general response with Table 3 in our response to reviewer oGYm)

Thanks for the response. While I will raise the score to borderline reject, my view remains slightly negative because the experiments lack adequate evaluation of off-policy RL methods and comparison with potential on-policy baselines, leaving the claims insufficiently supported.

We are very thankful to the reviewers for their time and for their insightful comments.

While the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:

• a lack of diversity in the experimental setup (not enough models, not enough datasets), and
• a lack of comparison to popular RL algorithms.

We want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.

Our initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.

We have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):

Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines. This extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)

In our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. We extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)

We consistently observe the same two phenomena highlighted in Figure 3:

• When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in Table 1).
• As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).

The general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.

Table 2: Comparison of off-VPG and GRPO, off-policy (10000 steps, with evaluation every 200 steps)

We conduct in Table 2 a performance comparison between GRPO and off-VPG. We use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2. For off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.

Under these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.

Note that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.

If the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).

Note that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.

--- End of general response ---

• In the analyses, are the rewards implicitly assumed to be non-negative?

No, the analyses are invariant to any transformation of the rewards of the form $\alpha \cdot r + \beta$ (and the same holds for the baseline $V$).

• In Theorem 4.3, does the set Y^inf exist always? since the policy can be stochastic.

Theorem 4.3 assumes access to the full population of samples (i.e., we know $\mu$) and the ability to perfectly optimize our surrogate objective (i.e., we can apply ${\cal T}$ exactly). This removes the stochasticity usually referred to as approximation and optimization error. In this setting, even if $\pi$ is stochastic, Y^\inf always exists. We did not address approximation and optimization errors, as they can be handled with classical techniques, which would lengthen the proofs without adding novel insights.

• I do not understand the statement of point 3 in Theorem 4.3 from the current writing, and reading the proof doesn't help. Can you expand more on what you wish to convey?

The phrasing is indeed a bit ambiguous; thank you for pointing it out, we will rephrase it. The third point of Theorem 4.3 states that the limit reward is going to be optimal if and only if the baseline $V$ is smaller than some threshold $V_{0,\mu}$ (which depends on the interplay between $\mu$ and $\pi^*$).

• Do you have a bound on how large V^inf is? Maybe this is related to the misunderstanding above.

We have not sought an explicit bound for V^\inf. A natural lower bound is V^\inf > \max_y \mu(y) r(y).

• How does the convergence rate and V^inf change with V in Theorem 4.3?

This is a very good question that highlights a trade-off: if $V$ is too large, there is a risk of eliminating good arms (indeed, V^\inf is a decreasing function of $V$); conversely, if $V$ is too small, the constant $c$ becomes small (indeed, $c$ is an increasing function of $V$). In other words, the larger $V$ is, the faster the convergence, but the higher the risk of converging to a suboptimal policy.

We are very thankful to the reviewers for their time and for their insightful comments.

While the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:

• a lack of diversity in the experimental setup (not enough models, not enough datasets), and
• a lack of comparison to popular RL algorithms.

We want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.

Our initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.

We have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):

Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines. This extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)

In our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. We extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)

We consistently observe the same two phenomena highlighted in Figure 3:

• When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in Table 1).
• As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).

The general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.

Table 2: Comparison of off-VPG and GRPO, off-policy (10000 steps, with evaluation every 200 steps)

We conduct in Table 2 a performance comparison between GRPO and off-VPG. We use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2. For off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.

Under these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.

Note that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.

If the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).

Note that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.

--- End of general response ---

• First, it omits some [...] off-policy updates, and GRPO?
• The evaluation is also [...] in the literature, such as [1].

Regarding the diversity of datasets and models, as well as comparisons to other algorithms (GRPO), we refer to Tables 1 and 2 of the general response above, where it is shown that our method is competitive and that our results are robust. Thank you for pointing out the APA paper, which we will reference in our revised manuscript. This paper shares goals with ours. However, the approaches differ significantly in terms of methodology, theory, and experiments.

• Finally, some of the theorems in the appendix seem underexplained.

Do you have particular examples in mind? We are happy to give you additional explanations, and potentially add them to the appendix.

• The original motivation [...] efficiency in any way.

In this context, efficiency should not be understood purely in terms of "final score at the end of training". Some of the practical advantages of off-policy RL, such as the capacity to handle asynchronicity between the worker Gpus (in charge of generating trajectories) and trainer Gpus (in charge of updating the weights) or the greater simplicity in implementation (no constant communication between worker Gpus and trainer Gpus), are already known to the community. Others, such as the fact that it allows several passes over the same data points (resulting in more data efficiency), have not yet been analyzed in depth in the context of LLMs, but we plan on rigorously studying these in future work. (For a pure score comparison, consider also Tables 1 and 2 from our general response and Table 3 in our response to reviewer oGYm: off-policy and on-policy final accuracies seem comparable, but these are preliminary experiments)

• For equation (1), how are you avoiding the importance sampling issue?

We do not face this issue, because the objective function that we optimize is NOT the expected reward $\mathrm{E}_{y \sim \pi}[r(y)]$ of the trained policy $\pi$ (though it is not far from it, as the actor policy $\mu$ is kept close to $\pi$ in the setups that we consider). This is what makes theorems 4.2 and 4.3 interesting, as they describe the limit policy of a gradient descent on this unusual off-policy objective function.

• How do you choose the N to [...] respect to N?

The larger the number $N$ of steps between two policy updates, the closer we are to the limit distribution $\pi_{\mu,V}^\star$ (using the notations of Thm 4.2), hence to the situation that our theory describes. In our bandits experiments, we have observed that the training curves in the policy improvement setting remain similarly shaped when we let $N$ vary, showing the robustness of our conclusions. In real world applications, the choice of $N$ is often dictated by practical concerns, such as the ratio between the number of trainer Gpus and worker Gpus available.

• Are the bounds in the bandit RL setting or the multi-step RL setting?

Theorem 4.2 applies in the case where the actor policy $\mu$ is fixed, and a gradient descent is performed on the trained policy $\pi$'s logits. Theorem 4.3 describes the policy improvement case, where the policy $\pi$ is trained on a given actor policy $\mu$, then the actor policy is updated $\mu \leftarrow \pi$ and the process is repeated. '

• Could you provide more motivation [...] is necessary.
• How is delta V trained in practice [...].

In the bandits setting, $V$ is not trained: it is a fixed hyperparameter. The behaviour of the algorithm depends on whether $V$ is larger or smaller than $V^\mu$. In the LLM setting, as you rightly observe, $V^\mu = V^{\mu(\cdot |x)}$ depends on the prompt $x$. We substract a correction $\hat V + \delta V$ to each reward $r(x,y)$. $\hat V$ is an estimator of $V^{\mu(\cdot |x)}$, computed as in GRPO (for example). $\delta V$, just like $V$ in the bandit setting, is a fixed hyperparameter: it is the degree of freedom of our algorithm. The reward corrected by $\hat V$, i.e. $r(x,y) - \hat V$, has expected value $0$ w.r.t. to $\mu(\cdot|x)$ : $\mathrm{E}_{y \sim \mu(\cdot |x)}[r(x,y) - \hat V] = 0$ (here the notation hides the group sampling). Hence having $\delta V <0$ is the same as having $V<V^\mu$ in the bandits setting, and having $\delta V \geq 0$ is the same as having $V\geq V^\mu$ in the bandits setting (see also our answer to a similar question from Reviewer sBrs).

• For most of the bounds in the appendix, [...] to be bounded, right?

We assume that the set of $y$ is finite and that $\mu$ has full support, so the ratio is necessarily bounded. The ratio $\pi(y) / \mu(y)$ does not appear explicitly in our derivations, but the interplay between $\pi^\star$ and $\mu$ can be seen implicitly. For example, if $\pi^*$ differs significantly from $\mu$, and the subtracted baseline $V$ is not conservative enough, good arms may be eliminated too early. This can be seen, e.g., in Theorem 4.2: when $V < V^\mu$, we have