[Q2] Experiments

PROPS hyperparameters

how does PROPS perform when using the same hyperparameter values as PPO (and only tunes the PROPS-specific hyperparameters)

In our current Hopper-v4 and Humanoid-v4 experiments, PROPS uses the same batch size and learning rate as PPO, and we see improvements in data efficiency. (Hyperparameters for each experiment are included in the commands directory provided in our supplemental material – we will add these to the appendix).

Does increasing the batch size improve performance? (Do we actually want to reduce sampling error?)

It is not obvious to me that reducing sampling error will necessarily lead to improved performance. Given the non-convex objective of policy gradient methods, noisy updates could even be helpful to avoid / escape local optima.

It is possible that true policy gradient may not help with non-convexity, however, on-policy policy gradient algorithms operate under the assumption that it is desirable to follow the true policy gradient. Policy updates are more accurate when our data more closely matches the on-policy distribution, so under this assumption, reducing sampling error should in principle improve performance.

Did you perform any experimental analysis on this connection (e.g., comparing PPO for the same number of total updates but varying batch sizes)?

We attach two figures for a few representative tasks from our PPO parameter sweep:

Fig. 1 (link): PPO returns vs. the number of updates performed for various batch sizes (10 seeds). Note that in our experiments, we fixed the training budget for each task, so runs with larger batch sizes perform fewer updates. This figure shows that increasing the batch size yields greater returns in fewer updates. Since larger batches have lower sampling error, reduced sampling error does indeed correspond to better performance on MuJoCo tasks.

Fig. 2 (link): PPO returns vs. the number of timesteps (10 seeds). This figure shows that increasing the batch size sometimes yields greater returns in fewer timesteps (e.g. PPO with a batch size of 2048 yields the most data efficient learning on Hopper-v4). In our work, we care about data efficiency, the number of samples required to achieve high returns within a fixed training budget, so we select the batch size whose training curve lies above all others on the return vs. timesteps figure. The most data efficient batch size is not necessarily the most update-efficient batch size.

Sampling error experiments: comparing PROPS with PPO-Buffer vs PPO.

What is the sampling error of standard on-policy PPO (and how does it compare to PROPS for b=1 with the same PPO hyperparameters)? Are similar trends observed?

We made this comparison in our sampling error experiments with a fixed target policy in Section 6.1. More concretely, comparing PROPS with b=1 (when there is no historic data in the agent’s buffer) to vanilla PPO would assess if PROPS reduces sampling error w.r.t a fixed target policy, and that is precisely what we assess in section 6.1.

It seems that “on-policy sampling” refers to PPO-Buffer in the sampling error figures, which does not seem like a fair comparison because PPO-Buffer does not attempt to correct for the use of off-policy data at all.

When computing sampling error in our RL experiments, we compare PROPS with b=2 to PPO-Buffer with b=2 rather than vanilla PPO to assess if PROPS reduces sampling error in historic data (i.e. if PROPS reduces sampling error w.r.t a changing target policy).

Please let us know if our clarification on this point answers your question. We can provide further explanation if needed!

Minor comments

In (1), the combination of an expectation over the visitation distribution and a summation over time inside the expectation is a bit strange.

Thank you for pointing this out! We will rewrite it as an expectation over trajectories in our revisions.

Equation (5) can be written more directly as min(−πϕ/πθ,−(1−ϵPROPS))

Since the PROPS objective (Eq. 5) is equivalent to the PPO objective (Eq. 3) with the advantage A(s,a) is set to -1 for all (s,a), we write the PROPS objective using the same form as the PPO objective. We agree it can be written more simply (e.g. the OpenAI Spinning Up documentation for PPO links to a simplified formulation similar to what you suggest here).

Thank you for your thoughtful review! We’re pleased to see that you found our paper clear and well-written and appreciated our full transparency of experiment details and hyperparameters. Below, we answer your questions.

[W1] Theoretical Analysis

[Proposition 1] is a slight generalization of a result that already appears in Zhong et al. (2022). It is also restricted to finite states and actions, and is based on a different sampling procedure than what is proposed in Section 5.

First, thank you for recognizing the value of Proposition 1!

Proposition 1 considers the sampling procedure used by ROS (Zhong et. al 2022), since PROPS reduces to ROS if we only perform a single update on the behavior policy rather than multiple minibatch updates. To see this, recall that $\phi$ refers to the behavior policy parameters and $\theta$ refers to the target policy parameters, and at the beginning of the PROPS update, we set $\phi \gets \theta$. Then if we perform a single behavior update, we have $\pi_\phi(a|s) / \pi_\theta(a|s) = 1$, and the PROPS update becomes $\nabla E_\pi[-1] = E_\pi[-\nabla \log \pi_\phi(a|s)]$ which is equivalent to the ROS update.

Just as PPO serves as a more scalable approximation of the theoretically motivated TRPO, PROPS serves as a more scalable approximation of the theoretically motivated ROS algorithm.

Proposition 1 only analyzes what happens in the limit of infinite data, but the transient behavior is important in the context of policy gradient algorithms where a finite (and often limited) amount of data is collected between each policy update

We agree that it would be interesting to quantify the rate at which ROS reduces sampling error as a function of sample size, though statistical consistency is an important property to establish, as we always want to make sure we indeed achieve zero sampling error in the limit of infinite data.

It seems that there would likely be an important interplay between the size of policy updates, the amount of data to reuse, and the size of behavior policy updates determined by the PROPS objective

We agree that it would be worthwhile to study the interplay between different PROPS hyperparameters, though our focus was ensuring we gave each method the best hyperparameters. We can add additional ablations studying this interplay in a camera ready version.

[W2] Regarding empirical improvement in data efficiency.

The core goal of our work is to develop nuance in what it means to learn “on-policy” and demonstrate how adaptive off-policy sampling can improve performance in on-policy policy gradient learning. Dramatic improvement on benchmarks is not necessary for providing evidence for the claims we make in this paper.

The experimental results suggest that [KL regularization] prevents PROPS from reusing a significant amount of past data...limiting the data efficiency that can be achieved.

Enabling PROPS to reuse historic data from many previous updates is an interesting challenge for future work. The core challenge is not necessarily that regularization prevents the behavior policy from changing too much. Retaining data from multiple previous updates increases the total amount of data in the agents buffer $D$, so the agent needs to collect a larger number of samples to adjust the distribution of $D$. The core issue is that the agent is not collecting enough data between updates to make $D$ more closely distributed to on-policy. We plan to address this challenge in future work by letting the agent increase or decrease the number of samples the behavior policy collects between target updates according to how closely $D$ matches the current on-policy distribution.

[W3/Q1] Main benefits of PROPS relative to existing methods (off-policy corrections, off-policy algorithms, massively parallel training)

We agree that off-policy correction methods and off-policy algorithms are related, though we believe such a comparison would distract from the core ideas of the paper. Our work’s core goal is to develop nuance in the on-policy vs off-policy dichotomy in the context of policy gradient learning, not to propose a new SOTA on-policy policy gradient algorithm.

While some applications permit just increasing the batch size via parallelization, other applications have stricter sample budgets and consequently a large body of RL research attempts to understand how to develop data efficient learners (e.g. [1-3]). PROPS is more closely related in spirit to the latter line of work. We can emphasize this point in our revisions.

[1] Janner et. al. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS 2019. https://arxiv.org/abs/1906.08253

[2] Chen et. al. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. ICLR 2021. https://arxiv.org/abs/2101.05982

[3] D'Oro et. al. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier. ICLR 2023. https://openreview.net/forum?id=OpC-9aBBVJe

We appreciate the continued discussion! Below, we address your followup comments and questions.

Contributions over Zhong et. al (2022)

Zhong et. al first established this nuance with ROS, though their empirical analysis is very limited: (1) they only focused on policy evaluation in low dimension tasks like Gridworld and Cartpole, and (2) they showed little to no improvement over on-policy sampling in even low-dimensional continuous policy evaluation tasks. Our work shows how to use the concepts introduced by Zhong et. al to develop a new sampling error correction algorithm that can scale to higher dimensional continuous control settings. We ask the reviewer to reconsider this evaluation on the significant improvement compared to Zhong et al.

Theory on how different PROPS hyperparameters related to each other

We provide theoretical intuition on the relationship between the amount of historic data reused, the size of target policy updates, and the size of PROPS behavior policy updates in Appendix A of our revised submission. We discuss the following relationships:

1. When retaining more and more (historic) data, the behavior policy must collect more and more data to make the agent’s data buffer match the on-policy distribution.
2. When sampling error is large, behavior policy updates must also be large.

Sampling error experiments with smaller batch sizes

Figure 1: Sampling error with a randomly initialized target policy

Figure 2: Sampling error with an expert-quality target policy

Thank you for the clarification! We better understand your comment now. PROPS can reduce sampling error with a small buffer size too. In figures linked above, we zoom in on Figures 7 and 8 (sampling error with a fixed target policy) to more clearly show sampling error after collecting a relatively small number of samples. For clarity, we changed the x axis to the number of transitions in the agent’s buffer.

Additional baselines

PROPS should be compared against off-policy correction methods in experiments

Our goal with this work is to improve on-policy policy gradient learning and emphasize a nuance in what it means to learn on-policy, and we designed our experiments to elucidate this nuance. While we agree it would be interesting to compare to off-policy correction methods, our current experiments support the claims we make in our paper. PROPS can in principle be used in addition to off-policy correction methods like Regression Importance Sampling [1], so it could also be interesting to (1) use PROPS to reduce sampling error during data collection, and then (2) use an appropriate off-policy correction method to adjust for any remaining sampling error. However, since our current paper is studying the data collection problem, we leave this extension to future work.

[1] Hanna et. al. Importance Sampling Policy Evaluation with an Estimated Behavior Policy. ICML 2019.

I suspect it will be very difficult for PROPS to match the data efficiency of off-policy algorithms with high update-to-data (UTD) ratios that have shown very strong performance in recent years

We agree that it will be a challenge for PROPS to match the data efficiency of off-policy algorithms – especially those that can use high UTD ratios. However, our goal with this work is to improve on-policy policy gradient learning and emphasize a nuance in what it means to learn on-policy, not to develop an algorithm that outperforms off-policy algorithms. We agree it would be interesting to understand the extent to which adaptive sampling can close the gap between the data efficiency of on- and off-policy algorithms, though this understanding is not critical to supporting the claims we make in this work.

Thank you for the positive review! We are quite pleased to see that you found our paper high-quality, well-written, and valuable to the RL community. Below, we answer your questions about PROPS and our notation.

Clarifying what we mean by on-/off-policy data + Notation changes

A sample is not per se on or off-policy. When we speak about "on-policy" and "off-policy" in estimation, we refer to the generating process (i.e., the distribution) that generates the samples.

We broadly agree with your comment that a sample is not on- or off-policy per se. Nevertheless, we find it common in the RL literature to confuse the generating process and the samples and chose terminology that we believed would call this out and advance a more nuanced understanding. We agree that it might have been better to say “data sampled off-policy.” For brevity, it may make more sense to keep our current phrasing and add text in the preliminaries section where we note that “off-policy data” is shorthand for “data sampled by off-policy distribution.”

Overall, I am conscious that much of the "nomenclature" is borrowed from Zhong et al., thus the authors might have preferred to keep consistency with that.

Much of our notation is indeed borrowed from Zhong et. al, though we think it’s reasonable to adjust our notation based on the points you’ve made in your review.

Equation 2 and 4, should be with the discounted state visitation

Thank you for sharing this reference! In our revisions, we will replace $d_\pi$ with $d^\gamma_\pi$ and note the points raised by Nota and Thomas.

Why we use batched behavior policy updates (rather than updating the behavior policy at every step)

In Zhong et al., the policy is updated at every step in an online fashion: to me, this makes a lot of sense, since we want constantly to "correct" the sampling distribution...However, if I understand well, you propose first to use the target policy to collect the samples and, afterward, to generate a policy that "corrects" the previous behavior. However, such a policy is not well defined: it depends on how many samples will be drawn from such a policy.

Why did you propose this "batch update" rather than an "online update"?

This is a good question, and we are planning to address this point in future work. You are correct that the optimal behavior policy for a given dataset $D$ depends on the number of samples in $D$ as well as how many samples we will collect with our behavior policy. For simplicity, consider a tabular setting. Let $d_\pi(s,a)$ denote the expected fraction of times $\pi$ observes $(s,a)$, let $d_D(s,a)$ denote the fraction of times $(s,a)$ appears in dataset $D$, and let $d_\beta(s,a)$ denote the expected fraction of times our behavior policy observes $(s,a)$. If we collect $n$ additional samples with our behavior policy and add them to $D$, we want $d_b(s,a)$ to satisfy

$$d(s,a) = \frac{|D| d_D (s,a) + n d_\beta(s,a)}{|D|+n}$$

Updating the behavior policy at every timestep would be computationally expensive, so we propose batched updates to make PROPS more scalable. In principle, PROPS will reduce sample error faster if we reduce the number of steps between behavior policy updates, but doing so would come with a computational cost.

---

Thank you again for your positive review, and please let us know if we have addressed your comments and questions! We’ll be happy to discuss further.

We’d like to thank the reviewer for their thoughtful review! We’re glad you found our writing clear and appreciate you recognizing how our work tackles a problem of interest to the RL community! We believe your comments and questions regarding our PROPS algorithm can be addressed with a few minor revisions to our submission.

When is PROPS useful?

When is this really a problem in the real-world? In most standard examples there is ample time to correct the sampling errors with more data and simulation

While some applications permit just increasing the batch size, other applications have stricter sample budgets and consequently a large body of RL research attempts to understand how to develop data efficient learners (e.g. [1-3]). Having said that, we’d like to emphasize that the core contribution of our work is developing nuance in the common usage of on- vs off-policy in the context of policy gradient learning.

[1] Janner et. al. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS 2019. https://arxiv.org/abs/1906.08253

[2] Chen et. al. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. ICLR 2021. https://arxiv.org/abs/2101.05982

[3] D'Oro et. al. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier. ICLR 2023. https://openreview.net/forum?id=OpC-9aBBVJe

Reducing sampling error at future states

there may be cases where we can improve the sampling error in one state, but this will cause worse sampling errors down the road because we will be forced to go into specific states.

This is a great point to discuss; it is possible that reducing sampling error at one state may increase sampling error at future states, and we plan to address this challenge in future work. To ensure we reduce sampling error at the current state and future states, the behavior policy update would need to consider the task’s transition dynamics. While we think it is possible that this challenge could arise, we show empirically in the gridworld domain that PROPS still produces the correct empirical distribution of state-action pairs more efficiently than on-policy sampling.

Wouldn't it make more sense to push this sampling error into the reward instead, and form a suitable policy?

Could you please clarify what is meant by “push this sampling error into reward”?

KL regularization for continuous-action tasks.

is [the KL regularization in Eq. 6] specifically for the continuous case?

Yes, the KL regularizer in Eq. 6 aims to prevent destructively large behavior policy updates in tasks with continuous actions. PROPS will increase the probability of under-sampled actions, but because the tails of a Gaussian policy are generally under-sampled, the PROPS update may push the behavior policy too far away from the target policy in an effort to over-sample the tails.

Why we write the PROPS objective in the same form as the PPO objective.

Equation (5) can be rewritten much simpler to be -max(g, 1-eps). I'm not sure forcing it to look like PPO helps with insight

Why did you chose the form given in Equation 5? why not its log or difference?

Since the PROPS objective (Eq. 5) is equivalent to the PPO objective (Eq. 3) with the advantage A(s,a) is set to -1 for all (s,a), we write the PROPS objective using the same form as the PPO objective. We agree it can be written more simply (e.g. the OpenAI Spinning Up documentation for PPO links to a simplified formulation similar to what you suggest, see here). Below, we provide further explanation for the PROPS objective.

To see why PPO (and analogously, PROPS) use the policy ratio rather than its log or difference, recall that on the first policy update where $\theta = \theta_{old}$, the clipped objective reduces to the advantage A(s,a), and the PPO update thus reduces to the vanilla policy gradient update $\nabla E_\pi[A(s,a)] = E_\pi[A(s,a) \nabla \log \pi_\theta(a|s)]$. Thus, PROPS can be viewed as a policy gradient algorithm that decreases the probability of observed action in proportion to how many times they were observed (i.e. actions that were observed more frequently are pushed down more heavily).

Furthermore, If we only perform a single update on the behavior policy rather than multiple minibatch updates, PROPS reduces to the ROS update it builds upon. In this case, $\phi = \theta$ so that $\pi_\phi(a|s) / \pi_\theta(a|s) = 1$. The PROPS update then becomes $\nabla E_\pi[-1] = E_\pi[-\nabla \log \pi_\theta(a|s)]$ which is equivalent to the ROS update. (Recall that $\phi$ refers to the behavior policy parameters and $\theta$ refers to the target policy parameters, and at the beginning of the PROPS update, we set $\phi \gets \theta$).

---

Please let us know if this explanation clarifies our formulation of the PROPS update and its connection to the PPO update! If you have further questions, we’d be happy to discuss further.

Thank you for your review! We believe we can address your comments and questions by adding a few additional details on our hyperparameter choices in our revisions.

PPO batch size

This paper adjusts the batch size of PPO to "1024,2048,4096,8192" which is much larger than the original batch size (64 samples) of PPO...This drastic increase in batch size is not well-justified.

Please note in our paper, the batch size refers to steps of environment interaction between target policy optimizations, not the mini-batch size used during optimization. What we call “batch size” is called the “horizon (T)” in the original PPO paper, and they set this value to 2048 for MuJoCo experiments just as we do (see Appendix A in [1]). PPO batch sizes between 1024 - 8192 are commonly used in MuJoCo tasks. For instance, the default batch size for PPO is 2048 in popular implementations of PPO in StableBaselines3, CleanRL, and Tianshou:

• StableBaselines3: see Line 85 here
• CleanRL: see line 51 here
• Tianshou: see Line 34 here

We sweep over batch sizes to ensure we accurately represent the performance of PPO with on-policy sampling, and we found that increasing the batch size beyond 1024 often led to large improvements in data efficiency. We attached a subset of our sweep over batch sizes here to support this claim (10 seeds per curve). For instance, on Hopper-v4, PPO achieves far greater returns with a batch size of 2048 compared to 1024.

In fact, it is confusing that their PPO algorithm still achieves the comparable performance of the original PPO with at least 16 times less update due to the batch size increase.

The PPO paper uses the “-v1” versions of the MuJoCo tasks while we use the “-v4” versions, so our results are not directly comparable; the “-v1” versions use an old MuJoCo simulation engine and are outdated and no longer supported (see the Gymnasium documentation here). Also note the original PPO paper uses different hyperparameters (e.g. they perform 3 epochs of policy updates whereas we perform 10).

Writing

The length of this paper can be greatly reduced. Currently, this paper has too many unnecessary examples and sentences.

If there are examples and sentences that you believe could be omitted, could you please refer us to them? We could then or clarify their importance and further discuss whether they can be safely omitted or condensed.

Theory

The theoretical inside of this paper is also weak.

Could you also please clarify the weaknesses in the existing theory we provide? We agree that it would be interesting to theoretically investigate how much PROPS reduces sampling as a function of the number of samples collected, though we believe our work is complete as is and appropriately supports our core claims: PROPS empirically reduces sampling error and improves data efficiency.

Learning rates with Adam optimizer

The learning rate for its proposed behavior policy ranges from 10^{-3} to 10^{-5} for an ADAM optimizer. Such a large difference in learning rate is not common for the ADAM optimizer.

Our choice of learning rates with the Adam optimizer is consistent with what appears in the existing RL research. For instance, the default PPO learning rate in Stablebaselines3, CleanRL, and Tianshou is $3 * 10^{-4}$, and in TRL, a popular package for training LLMs, the default learning rate is $5 * 10^{-5}$:

• StableBaselines3: see line 84 here
• Tianshou: see line 30 here
• CleanRL: see line 47 here
• TRL: see the default learning_rate value in the PPOConfig in the documentation here.

[1] Schulman et. al. Proximal Policy Optimization Algorithms. https://arxiv.org/pdf/1707.06347

Tuned PPO batch sizes used in our experiments

What is the final choice of the batch size for PPO? Only candidates 1024, 2048, 4096, and 8192 are reported.

Below, we list the batch sizes we used in our PPO runs. We can include tuned hyperparameters in appendix with our revisions:

• Swimmer-v4: 4096
• HalfCheetah-v4: 1024
• Hopper-v4: 2048
• Walker2d-v4: 4096
• Ant-v4: 1024
• Humanoid: 8192

---

Thank you again for your review! We believe our response clarifies your comments on hyper parameter choices, but if you have further questions, please ask! We'll be happy to provide further clarification.

We appreciate the followup comments!

PPO batch sizes

For the reported batch size, I feel those numbers are not well-justified.

We tuned our batch sizes to ensure we are comparing PROPS to the most competitive version of each algorithm. It is common in the literature to fix the batch size across all tasks in a given benchmark, though our approach is also valid.

the Ant environment, with its 105-dimensional observation space, is more challenging compared to the Hopper and Walker environments, which have observation spaces of only 11 and 17 dimensions. However, the batch size for Ant is 1024 which is smaller than the batch size for Hopper and Walker.

Understanding why PPO can solve Ant-v4 with a batch size of 1024 is beyond the scope of this work. However, it’s worth noting that the Ant-v4 has only 27 state dimensions; the newer Ant-v5 state has 107 dimensions because it include contact forces (which are excluded in v4, see the Gymnasium version history).

PROPS Hyperparameters

this algorithm also introduces many new hyperparameters for their proposed PROPS algorithm.

Figure: PROPS performance aggregated over a hyper parameter sweep.

In our initial response, we think we overlooked the core comment you made regarding hyperparameter sensitivity. PROPS admittedly introduces additional hyperparameters, but PROPS is not fragile to hyperparameter choices. In the figure linked above, we plot PROPS performance aggregated over the following PROPS hyperparameters from our hyperparameter sweep (we call this “PROPS (aggregate)” in the legend):

• Learning Rate: $10^{-3}, 10^{-4}$ (and $10^{-5}$ for Swimmer)
• KL Cutoff ($\delta_{\text{PROPS}}$): 0.03, 0.05, 0.1
• Regularizer Coefficient ($\lambda$): 0.01, 0.1, 0.3

The PROPS (aggregate) curves closely follow the tuned PROPS curves, indicating that PROPS is not fragile to hyperparameter choices.

We would also like to clarify that we considered PROPS learning rates in \{ 10^{-3}, 10^{-4}}, and then additionally considered $10^{-5}$ for Swimmer-v4 only (we note this in Table 2). We considered PPO learning rates in $\{ 10^{-3}, 10^{-4} \}$, which is standard in the RL literature for MuJoCo tasks.

I am not sure what would be the set of default parameters and what would be the performance of the set of the default parameter.

We suggest the following default hyperparameters:

• Learning rate: 10^{-3}
• Behavior batch size: 256
• KL regularizer (lambda): 0.1
• KL cutoff: 0.05 We’ve added tuned PROPS hyperparameters for all environments in Table 4 of our revised submission, and these suggested default values are the most common values we saw in our tuned hyperparameters.

PROPS with small/large batch sizes

A smaller number of samples may have a mean further from the distribution mean. Thus, this scenario needs more "on-policy" sample correction. However, the smaller number of on-policy samples does not give enough space for PROPS to learn.

PROPS can decrease sampling error even if there are only a few samples in the agent’s buffer. PROPS decreases the probability of sampling actions in the agent’s buffer (thus increasing the probability of sampling under-sampled actions), and if there are only a few samples in the buffer, this simply means that fewer actions will be pushed down (and thus the behavior policy may not change significantly).

We’ll use a single N-arm bandit problem to make this point more concrete. Suppose the agent’s policy places 1/N probability on each arm and the agent’s buffer contains a single sample corresponding to arm 1. To reduce sampling error in the next step, the behavior policy should decrease the probability of sampling arm 1 (since arms 2 through N are currently under-sampled).

A larger number of samples may have a mean closer to the distribution mean. Thus, this scenario needs less sample "on-policy" sample correction and PROPS may not help much.

Your reasoning here is correct; when there is a lot of data in the agent’s buffer, the data’s distribution may closely match the expected on-policy distribution, leaving little room for improvement with PROPS. Our experiments focus on settings where there is only a moderate amount of data in the agent’s buffer and thus sufficient room for improvement with PROPS.