We appreciate your thoughtful review and are glad that you found our approach to be interesting!

We are slightly confused by some of these comments and hope we can clarify our novelty during the discussion period. We would like to emphasize that the core novelty of our work is how we demonstrate that off-policy sampling can reduce sampling error and improve the data efficiency of on-policy policy gradient algorithms. In other words, the core novelty is not only the PROPS update we use to adapt our behavior policy but moreover the fact that we are using a separate behavior policy for on-policy learning in the first place. So while KL penalties and clipping (as in PPO) are not novel on their own, our use of them to enable scalable adaptive sampling with the behavior policy is novel to the best of our knowledge.

Below, we address your comments and questions.

Comparing the PPO objective and the CLIP objective of PROPS

The major difference between these objectives is that CLIP is an objective for behavior policy optimization and the PPO objective is for optimizing the target policy. In CLIP, we weight behavior policy updates by -1 to increase the probability of under-sampled actions, where as PPO weights policy changes by the advantage $A(s, a)$ to increase the probability of good actions with large advantages. Note that we use the PPO objective to optimize the target policy in both PROPS and vanilla PPO.

Are $\theta$ and $\phi$ close to each other?

Your understanding is correct. At the start of each PROPS update, we set $\phi \gets \theta$ (line 3 in Algorithm 2), and then make a location adjustment to $\phi$ that increases the probability of under-sampled actions. You are also correct in saying that regularization helps to keep the behavior policy close to the target policy and that the PROPS update is similar to a trust-region update.

Are $\theta$ and $\phi$ updated at different frequencies?

The behavior policy is updated more frequently than the target policy so that the behavior policy has a chance to reduce sampling error before each target update. In our experiments, the behavior policy is updated every 256 timesteps and the target policy is updated every 1024 - 8192 timesteps.

Why we emphasize the difference between on-policy data vs. on-policy sampling

While we agree that the formulation of the on-policy update does indeed point to on-policy data as the key desiderata, unfortunately, we know of many sources that distinguish on-policy learning by its sampling distribution, including some prominent educational resources:

1. Sutton & Barto’s RL textbook [1] states, “the distinguishing feature of on-policy methods is that they estimate the value of a policy while using it for control.”
2. Lecture 5 of David Silver’s RL course [2] states that on-policy algorithms “learn about policy π from experience sampled from π.”
3. The taxonomy of RL algorithms provided in OpenAI’s Spinning Up package [3] states that with on-policy algorithms, “each update only uses data collected while acting according to the most recent version of the policy.”

Because the view that "on-policy refers to a sampling procedure" is so prominent, we wanted to elaborate on the distinction between on-policy data and on-policy sampling to emphasize how this popular perception is not a reality. We will clarify these points in our revisions and hope to further discuss them with reviewers.

PPO-Buffer and its performance

Your understanding is correct; PPO-Buffer naively reuses historic off-policy data without any algorithmic changes, and PROPS is indeed an adaptive version of PPO-Buffer.

PPO-Buffer outperforms PPO in Walker2d-v4 and Humanoid-v4, which indeed suggests that PPO’s clipping mechanism may be robust to stale data. In fact, OpenAI reused historic data in a similar fashion when using PPO to train the RL agents that ultimately defeated the world champions in the Dota 2 game [4]. However, we note that in most tasks we consider, PPO-Buffer performs comparably to PPO.

Thank you again for your review, and please let us know if our response clarifies your comments and questions! We are happy to discuss any follow-up comments.

References

[1] Sutton, Richard S., and Andrew G. Barto. "Reinforcement learning: An introduction." MIT press, 2018.

[2] Silver, David. "RL Course by David Silver, Lecture 5: Model-Free Control." University College London, Computer Science Department, 2015. https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf

[3] Achiam, Joshua. Spinning Up in Deep Reinforcement Learning. Github, 2018. https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html#a-taxonomy-of-rl-algorithms

[4] Berner et. al. "Dota 2 with Large Scale Deep Reinforcement Learning." Arxiv 2019.

Ablations on KL regularizer and clipping mechanism

We ablate PROPS’s KL regularizer and the clipping mechanism in Appendix C and D and find that both help reduce sampling error and improve agent performance:

• Appendix C, Fig. 9 shows how clipping and regularization affect PROPS’s ability to reduce sampling error with respect to a fixed target policy. In this setting, clipping generally has a stronger effect on sampling error reduction than regularization.

• Appendix D, Fig 12 shows how clipping and regularization affect PROPS’s ability to reduce sampling error in the RL setting with a continually changing target policy. In this setting, both regularization and clipping are crucial to reducing sampling error.

• Appendix D, Fig 13 shows IQM return achieved by PROPS during RL training with and without regularization or clipping. In nearly all tasks, PROPS performs best with both clipping and regularization.

Wall-clock time comparisons

We can include wall-clock time comparisons of PROPS, PPO, and PPO-Buffer in our revisions. We will share our results in a follow-up comment once we have them.

Intuitively, we expect PROPS will take roughly twice as much time as PPO-Buffer; both algorithms perform updates using the same amount of data, though PROPS learns two policies.

Again, please let us know if our response clarifies your comments and questions! We will happily discuss any follow-up comments you may have.

We would first like to thank the reviewer for their comments and questions; we’re glad you found our paper well-written and appreciated our motivating examples. We believe your comments can be easily addressed with small changes to our current paper. We describe these in our response below and kindly ask you to consider raising your rating.

Perceived mismatch between the idea and practical method

We believe this concern arises from a simple misunderstanding of the direction of the ROS/PROPS update, and we will clarify this point in the main paper. The key point is that we are doing gradient ascent on the negative log-likelihood (NLL) rather than gradient descent as done in MLE learning. You correctly point out that MLE learning would have a mismatch with the goal of our work. However, since we use gradient ascent on the NLL, the update has the desired effect of increasing the probability of under-sampled actions.

It is important to note that the ROS and PROPS algorithms do not aim to fully optimize their objectives. Rather, at each iteration, the behavior policy is set equal to the target policy and then the gradient is used to make a local adjustment which decreases the probability of over-sampled actions, thereby increasing the probability of under-sampled actions.

Example in Section 4

You are correct in that PROPS focuses on correcting the state-conditioned distribution of actions as opposed to the distribution of trajectories (since we generally don’t know the trajectory distribution). To convey intuition for how sampling error could impact learning, our example in Section 4 focused on sampling error in the outcomes space. However, we see how this was confusing to multiple reviewers and so we can make a few minor modifications to make the example focus on action probabilities.

In particular, we can add the following action probabilities to Figure 2:

$\pi(left | s_0) = 0.5$,

$\pi(right | s_0) = 0.5$,

$\pi(left | s_L) = 0.02$,

$\pi(right | s_L) = 0.98$

Where, for brevity, we use $left$ = “attempt to move” and $right$ = “remain standing”. These action probabilities preserve the outcome probabilities originally listed.

After 100 visits to $s_0$, the agent may never sample the $left$ branch at $s_L$ and believe that the $left$ branch at $s_0$ always leads to a return of -1 (falling). Having observed a return of +1 (standing) in the $right$ branch at $s_0$, the agent reduces the probability of choosing the $left$ branch at $s_0$, ultimately making the agent more likely to converge to a suboptimal standing policy. We can help correct this sampling error by increasing the probability of sampling the under-sampled $left$ branch at $s_L$.

We hope our response clarifies the soundness of the ROS/PROPS updates as well as the example in section 4. If you have further comments or questions regarding our empirical results, we would be more than happy to discuss further!

Baselines

We considered these baselines but ultimately decided to exclude them from the main RL experiments shown in Figure 5.

We exclude ROS from our RL experiments for two reasons:

1. Since ROS fails to reduce sampling error even when the target policy is fixed (Fig. 3 in the main paper and Fig. 7 and 8 in Appendix C of the supplemental material), we expect ROS to also fail to reduce sampling error in the more difficult RL setting where the target policy is continually updated.
2. ROS is too expensive to run in the RL setting. ROS updates the behavior policy every timestep using a gradient computed over all data in the agent’s buffer, whereas PROPS updates the target policy every 256 timesteps using minibatches sample from the agent’s buffer.

We can clarify both of these reasons in our revisions.

PROPS and GePPO are orthogonal approaches to improve the data efficiency of on-policy learning: PROPS uses adaptive data collection to reduce sampling error, while GePPO reweights data according to historic policies after data collection. We ultimately decided that the comparison distracted from our main focus of determining whether adaptive off-policy sampling could produce sufficiently on-policy data for effective learning. For the purpose of answering the main questions of our study, the empirical evaluation is complete as-is.

Thank you for the in-depth comment on the framing of our work. We completely agree with the historical framing you provide, so we believe a minor change can clarify the narrative. However, before doing so, we would like to explain why we framed the paper as we did and discuss how we can combine your suggestions with our original motivation.

While we agree that the historical trajectory does indeed point to on-policy data as the key desiderata for on-policy policy gradient methods, unfortunately, we know of many cases where on-policy learning is defined by the sampling distribution, including some prominent educational resources:

1. Sutton & Barto’s RL textbook [1] states, “the distinguishing feature of on-policy methods is that they estimate the value of a policy while using it for control.”
2. Lecture 5 of David Silver’s RL course [2] states that on-policy algorithms “learn about policy π from experience sampled from π.”
3. The taxonomy of RL algorithms provided in OpenAI’s Spinning Up package [3] states that with on-policy algorithms, “each update only uses data collected while acting according to the most recent version of the policy.”

Because the view that "on-policy refers to a sampling procedure" is so prominent, we chose a more pointed narrative to emphasize that it is the empirical distribution of data that really matters for accurate gradient estimates. In a sense, our work is pointing back to the historical trajectory you give and saying that we really do want to have samples weighted according to their actual probability and not just generated according to a specific process.

Overall, we believe that there is not a large gap between our view and the reviewer’s view and that we can fine-tune the discussion in our paper to note the historical trajectory given. Specifically, it will be straightforward to 1) add discussion on the historical basis of these methods and say that, yes, clearly on-policy data is the goal, and 2) clarify that we aim to correct a popular misconception in how these algorithms are understood. We hope that we can discuss this point more with the reviewer over the next week to make the narrative as strong as possible.

[1] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.

[2] Silver, David. "RL Course by David Silver, Lecture 5: Model-Free Control." University College London, Computer Science Department, 2015. https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf

[3] Achiam, Joshua. Spinning Up in Deep Reinforcement Learning. Github, 2018. https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html#a-taxonomy-of-rl-algorithms

We greatly appreciate your response regarding our clarification of the motivating narrative! We'll provide another update on that soon. In the meantime, we wanted to use this separate post to respond to your other comments regarding the example in section 4 and a few experimental details.

Example in section 4

You are correct in that PROPS focuses on correcting the state-conditioned distribution of actions as opposed to the distribution of trajectories (since we generally don’t know the trajectory distribution). To convey intuition for how sampling error could impact learning, our example in Section 4 focused on sampling error in the outcomes space. However, we see how this was confusing to multiple reviewers and so we can make a few minor modifications to make the example focus on action probabilities.

In particular, we can add the following action probabilities to Figure 2:

$\pi(left | s_0) = 0.5$,

$\pi(right | s_0) = 0.5$,

$\pi(left | s_L) = 0.02$,

$\pi(right | s_L) = 0.98$

Where, for brevity, we use $left$ = “attempt to move” and $right$ = “remain standing”. These action probabilities preserve the outcome probabilities originally listed.

After 100 visits to $s_0$, the agent may never sample the $left$ branch at $s_L$ and believe that the $left$ branch at $s_0$ always leads to a return of -1 (falling). Having observed a return of +1 (standing) in the $right$ branch at $s_0$, the agent reduces the probability of choosing the $left$ branch at $s_0$, ultimately making the agent more likely to converge to a suboptimal standing policy. We can help correct this sampling error by increasing the probability of sampling the under-sampled $left$ branch at $s_L$.

Please let us know if further clarification is needed on this updated example!

Hyperparameters:

We tune all algorithms using a hyperparameter sweep described in Appendix F (Tables 2 and 3). We perform this sweep separately for all tasks and all algorithms and report results for top-performing hyperparameters.

Off-policy baseline:

We agree that it would be interesting to compare PROPS to an off-policy algorithm. We’ll have to get back to you on this matter; we first want to integrate your feedback on the narrative we crafted.

We would like to thank the reviewer for their positive review! We’re pleased to see you appreciate our contributions towards improving the data efficiency of on-policy learning.

We want to begin by answering your question about the work’s broader impact. The core contribution of our work is that we demonstrate how off-policy sampling can reduce sampling error and improve the data efficiency of on-policy policy gradient algorithms. Our work also emphasizes that on-policy algorithms simply require on-policy data and are agnostic to how the data is generated – a point which is often overlooked in the literature.

Below, we now address your comments and questions.

The distinction between on-policy sampling and on-policy data in the Introduction

We framed the introduction in this way to emphasize that the focus with on-policy algorithms should be on the data, not the procedure for generating the data. In the literature, we find that this point is often overlooked. For instance, Sutton & Barto’s RL textbook [1] states that “the distinguishing feature of on-policy methods is that they estimate the value of a policy while using it for control.”

ROS update

This is a great point that we will clarify in the main paper. ROS is indeed delicate to work with, and a key aspect of our our is work is designing a method (PROPS) which makes ROS more stable in higher dimension benchmark tasks. PROPS, like ROS, does not aim to fully optimize its objective but instead uses the gradient to make a local adjustment to the behavior policy that increases the probability of under-sampled actions. PROPS’s clipping mechanism and KL regularization help ensure that this behavior update remains local.

Ablations on KL regularizer and clipping mechanism

We ablate PROPS’s KL regularizer and the clipping mechanism in Appendix C and D and find that both help reduce sampling error and improve agent performance:

• Appendix C, Fig. 9 shows how clipping and regularization affect PROPS’s ability to reduce sampling error with respect to a fixed target policy. In this setting, clipping generally has a stronger effect on sampling error reduction than regularization.

• Appendix D, Fig 12 shows how clipping and regularization affect PROPS’s ability to reduce sampling error in the RL setting with a continually changing target policy. In this setting, both regularization and clipping are crucial to reducing sampling error.

• Appendix D, Fig 13 shows IQM return achieved by PROPS during RL training with and without regularization or clipping. In nearly all tasks, PROPS performs best with both clipping and regularization.

Should we prefer PROPS over other off-policy methods?

We view on-policy learning with replay as a means to improve the data efficiency of on-policy learning rather than an alternative to off-policy learning. On-policy and off-policy algorithms have different advantages and disadvantages, and the choice of algorithm typically depends on what advantages are most relevant for a given task. For instance, PPO has played a role in several RL success stories [e.g., 2, 3] in part because it is empirically stable and robust to hyperparameter choices.

Example in Section 4

While we agree that an example with a replay buffer would clearly illustrate how historic off-policy data can bias updates, we wanted to emphasize that sampling error can be a problem even when all data in our buffer comes from the current target policy. In fact, in Fig. 15 and 16 of Appendix E in the supplemental material, we show that PROPS can improve the data efficiency of on-policy learning even when the replay buffer contains no historic data (PROPS with b=1).

Suggested edits and typos

Thanks for pointing out all of these! We’ll fix these in our revision:

• We will clarify the limitation of off-policy approaches under large distribution shift.
• We will try to keep figures on the same page on which they are referenced.
• You’re correct that minus should be outside the clip function in PROPS’s clipped objective--nice catch
• We will order references alphabetically.

Please let us know if you have any follow-up questions or comments. We are more than happy to discuss!

References

[1] Sutton, Richard S., and Andrew G. Barto. "Reinforcement learning: An introduction." MIT press, 2018.

[2] Berner et. al. "Dota 2 with large scale deep reinforcement learning." Arxiv 2019.

[3] Vinyals et. al. "Grandmaster level in starcraft ii using multi-agent reinforcement learning." Nature 2019.

We’d like to thank the AC for this explanation! We considered a similar formulation when developing our methods, and we can try to include some discussion on this in a camera-ready version.