HP3O: Hybrid-Policy Proximal Policy Optimization with Best Trajectory

Question 3: Thank you for your question regarding environments with higher action spaces.

We have conducted experiments in the Humanoid-v2 environment. Preliminary results show that our HP3O method is more effective compared to baseline PPO and other on-policy methods, as claimed in our manuscript.

We will include these results in our manuscript and provide an evaluation across all environments with this high-dimensional humanoid environment. We appreciate your patience and your insightful question.

Weakness 1:

We appreciate your suggestions for adding additional experiments to improve the results.

$•$ Learning Curves of SAC and HP3O+ in Figure 2:

We are in the process of generating these results and will include them in an updated version of the manuscript. The reason we initially did not include SAC in Figure 2 is that our primary focus was on comparing HP3O+ with the PPO variant and other on-policy methods.

Additionally, we will merge Figures 2 and 3 to present a comprehensive comparison, and the additional results for HP3O+ across all environments will be included in Figure 2 as well.

$•$ HP3O+ in Figure 4(a):

We are currently updating Figure 4(a) to include the additional results for HP3O+. The figure will be adjusted accordingly to reflect these new results.

$•$ More Baseline Methods:

We appreciate your suggestion to include additional baselines in our work. We are actively working on incorporating more baseline results in the revised version of the paper. As Reviewer 2 suggested, we will present some of the useful baseline algorithms in our manuscript.

We have faced challenges in implementing certain baselines due to significant differences in experimental setups. Specifically, some of the baseline algorithms were originally implemented in very different settings of the environments, which require extensive code refactoring. As a result, we were unable to include some of these baselines in our experiments. However, we are committed to adding as many relevant baselines as possible, as suggested by the reviewers.

We will update the manuscript accordingly with these additional experimental results, including the additional plots as suggested by the reviewer.

Weakness 2:

Thank you for providing such an interesting idea to plot the curve of episode return against training time. We are currently reproducing such plots by measuring the training time for each method until they reach similar episode returns. Additionally, we will include supplementary tables showing the time usage based on the same return to provide a clearer illustration of the comparison between methods. We will update our manuscript with these additional plots and tables.

Weakness 3:

Thank you for pointing out the reference errors in the manuscript. We apologize for those issues. We will review the manuscript and correct all errors.

Question 1:

Thank you for your question regarding the experimental setup and the use of parallel environments. Our training setup can utilize multiple environments in parallel. In our code, this is easily configurable by adjusting the number of environment parameters, which allows using multiple environments for parallel training.

The reason we conducted our experiments with a single environment setting is to certify a controlled fair comparison across different algorithms, which also minimizes variability that can arise due to differences in the number of parallel environments. However, we acknowledge that parallel environments are widely used in practical applications to speed up training. Our proposed HP3O method is designed to be capable of leveraging parallelization. We leave this as an open choice for users based on real-world hardware restrictions. The parallelization setup in HP3O follows a similar implementation manner as other methods, such as PPO in Stable Baselines3.

We will clarify in the manuscript that our approach is available for parallel environment settings. Thank you again for highlighting this important aspect.

Question 2:

We apologize for the incorrect statement regarding SAC. The phrase should be changed from "only works" to "primarily works" with continuous action spaces. Thank you for pointing out the availability of versions of SAC that can handle discrete action spaces.

In our work, we primarily focus on extending PPO, and the experiments have largely been conducted in continuous action spaces. This choice is due to the fact that continuous environments generally present more challenges to RL agents. Since the HP3O structure fundamentally belongs to a variant of PPO, it is expected to perform well in discrete environments as well. We have also implemented HP3O in a discrete action space environment, such as the CartPole environment. The results are presented in Figure 22 of the appendix and demonstrate that HP3O performs well in discrete settings, showing performance outperformance to PPO.

We will update the manuscript to clarify this distinction and correct the statement about SAC. Thank you again for your insightful comment.

Thank you once again for your valuable feedback. We have addressed the concerns raised as follows:

1. We have added the other baseline results, including both off-policy PPO and Geppo methods.

2. In the supplementary materials, we have included the "Episode Return - Time Cost" curves. From these plots, we observe that both presented models demonstrate greater effectiveness and efficiency. We have discussed these results further in the supplementary section.

3. We have conducted additional experiments for HP3O+, and the revised manuscript now includes these results for all presented environments.

We appreciate all your comments. Please feel free to reach out with any further questions or concerns, and we will do our best to address them.

Weakness 3:

We have conducted some offline investigations on the effect of replay buffer size, taking into account computational time and memory usage. These offline evaluations have helped us understand practical trade-offs between maintaining a larger buffer for more diverse sampling and managing computational and memory costs.

The key factor that impacts training is the batch $\mathcal{B}$ selected from the replay buffer. The size of the buffer plays a crucial role in determining the diversity and recency of the experiences that make up each training batch. A larger replay buffer may increase the diversity of experiences but also requires careful management to avoid incorporating outdated trajectories that could reduce policy stability. Conversely, a smaller replay buffer improves recency but limits the diversity of training data, which may hinder generalization.

We plan to conduct more extensive empirical evaluations on the impact of replay buffer size in future work and include those results. In the meantime, we will expand our discussion in the manuscript to include these practical considerations, highlighting the key role of the replay buffer in balancing performance and stability. In the final version, we will include some ablation studies in the draft.

Question 1:

You are correct $G_t$ - $V_{\phi}(s_t)$ should be replaced by $\hat{A}^{\pi_t}_t$ for consistency with the HP3O+ formulation. Line 16 in Algorithm 1 is used for both HP3O and HP3O+.

We will correct this in the revised version of the manuscript. Thank you again for your careful review and for bringing this to our attention.

Question 2:

There are several reasons why we did not provide fully converged reward plots for all environments. Our primary goal was to conduct a fair comparison by using the same number of training steps for all algorithms across environments, which is a common practice in the RL community. For more details, please refer to our response to Weakness 2.

Question 3:

Thank you for your question regarding the impact of replay buffer size.

There are several factors that influenced our decision not to provide detailed experimental results on the impact of replay buffer size at this stage. Due to computational limitations and resource constraints, conducting such experiments requires a large amount of training time and resources. Currently, we are proceeding with further analysis, and for additional context, please refer to our response to Weakness 3.

Weakness 1:

Thank you for your feedback.

We acknowledge that the use of replay buffers in reinforcement learning is a concept derived from off-policy methods. However, the primary contribution of our work lies in adapting and integrating this concept into the on-policy setting of PPO, which presents several unique challenges. Specifically, the integration required careful handling of the balance between on-policy and off-policy data, addressing issues such as maintaining the benefits of PPO's policy gradient updates while leveraging experience replay to stabilize training and reduce sample variance.

Our approach, HP3O, uses a novel trajectory buffer that retains high-quality trajectories, which enhances the exploration-exploitation balance without compromising PPO's core stability. The key innovation lies in:

$•$ The use of trajectory prioritization to selectively store and replay the best experiences. Notably, when selecting data for updates, we always include the trajectory with the best return, combined with additional random samples from the rest of the buffer. This makes sure that high-value experiences are consistently utilized while still maintaining diversity through random sampling.

$•$ The design of the hybrid on-off policy update mechanism, which adapts PPO's policy update rules to work effectively with the replayed trajectories, mitigating common issues such as policy divergence or instability due to outdated data.

$•$ The incorporation of a variance reduction mechanism that works synergistically with the trajectory replay buffer, ensuring that the additional replay does not introduce harmful biases and instead enhances the training process.

Our approach uses optimal experiences to guide policy learning by always including the best trajectory. As a result, we demonstrate improved performance and sample efficiency while maintaining the theoretical guarantees of on-policy updates. We appreciate your constructive feedback and hope that this addresses your concerns. We will update our manuscript accordingly to include these changes.

Weakness 2:

Thank you for your suggestions regarding the convergence of the training curves. We believe the reviewer intended to refer to Figures 2 and 3.

We agree that in some environments, the training curves do not fully converge, which may make it challenging to assess the ultimate performance. However, for consistency across all algorithms, we maintained the same number of training time steps for each experiment. This allows for a fair comparison of sample efficiency across different methods, even if the algorithms did not always fully converge within the given time frame. Additionally, in some environments, we do present converged training curves, demonstrating the capabilities of the algorithms.

In the reinforcement learning community, it is a common practice to show learning curves at a fixed number of steps for comparative analysis, even if full convergence is not always achieved. Notably, papers on SAC [1], PPO [2], GePPO [3], and Off-Policy PPO [4] follow similar practices, with many of the environments presented in these works employing non-converged curves to provide valuable insights into training dynamics and sample efficiency.

We will update the manuscript to clarify our approach and the reasoning behind presenting non-converged training curves. Thank you again for your valuable feedback.

[1] Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018, July). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (pp. 1861-1870). PMLR.

[2] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

[3] Queeney, J., Paschalidis, Y., & Cassandras, C. G. (2021). Generalized proximal policy optimization with sample reuse. Advances in Neural Information Processing Systems, 34, 11909-11919.

[4] Meng, W., Zheng, Q., Pan, G., & Yin, Y. (2023, June). Off-policy proximal policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 8, pp. 9162-9170).

We have uploaded a revised version of our work based on all the provided feedback. A detailed summary of these updates is included in the revision. Please let us know if you have any further questions or need additional clarification. Thank you.

Weakness 4:

Thank you for your insightful suggestion regarding ablation studies on the buffer size.

We will expand our discussion in the limitations section to provide a more thorough analysis of how buffer size impacts performance and stability.

$•$ Large Trajectory Buffer Size: A larger trajectory buffer size allows us to store a greater number of diverse trajectories. This diversity can enhance generalization and reduce overfitting, as the agent learns from a broad range of experiences. However, in sparse reward settings, maintaining a large trajectory buffer may mean that the inclusion of outdated or less relevant trajectories could introduce instability and slow down learning, as the agent may be exposed to experiences that no longer align with its current policy.

$•$ Small Trajectory Buffer Size: A smaller trajectory buffer retains fewer trajectories, which typically results in the agent learning from more recent experiences that are closely aligned with the current policy. This can improve stability, as updates are based on recent, relevant data. However, a smaller buffer can reduce the diversity of sampled experiences, leading to an increased risk of overfitting and limiting the agent's ability to effectively explore different parts of the environment.

$•$ Sampling Strategy: Our sampling strategy also plays a critical role in managing the trade-off between stability and performance. By ensuring that the best return trajectory is always included in the sampled trajectories, we provide a strong guiding signal that improves policy performance. The sample rate, where we evenly sample from each selected trajectory, helps in maintaining a balance between exploration and exploitation, as well as in utilizing high-quality trajectories effectively.

$•$ Trade-off in Sparse Reward Settings: In sparse reward environments, the need to maintain high-quality trajectories becomes even more crucial. A large trajectory buffer can help capture rare, valuable experiences, but the risk of dequeuing these valuable trajectories before they can contribute meaningfully to learning is higher. Ensuring that the best trajectory is always sampled helps mitigate this issue, but the buffer size still influences how effectively these rare rewards are retained and leveraged.

We hope this expanded discussion provides a clearer understanding of the effects of buffer size on both performance and stability, especially within the context of our trajectory buffer approach. We will continue to explore the impact of buffer size experimentally next and appreciate your suggestion for including ablation studies to further demonstrate these effects. In the final version, we will include some ablation studies in the draft.

Minor:

$•$ Exploration of HP3O and HP3O+ in discrete action spaces, such as Atari games, is an interesting direction for future work. We will ensure that this is added to the future work section in our draft.

$•$ We will correct this minor typo in the revised draft. Thanks for pointing this out.

Question 1:

If the best trajectory is not added to the replay buffer, then HP3O+ cannot be effectively implemented as intended. Consequently, Lemma 3 and Theorem 2 will no longer hold, as these theoretical results are derived based on the HP3O+ framework, which relies on the inclusion of the best trajectory in the replay buffer. The addition of the best trajectory is fundamental to ensuring that the theoretical guarantees of variance reduction and policy improvement, as described in Lemma 3 and Theorem 2, are maintained.

We will clarify this point further in the revised manuscript to emphasize the necessity of including the best trajectory for the theoretical foundations to hold. Thank you again for your valuable question.

Question 2:

Thank you for pointing out this minor mistake in Theorem 2. We really appreciate your careful review. Indeed, the factor of 2 should be canceled out, as the total variation distance is upper bounded by half of the clipping parameter. We will make the necessary revisions to the draft to correct this error. Thank you again for your insightful feedback.

Weakness 1:

Thank you for highlighting this concern regarding the inclusion of more relevant baseline methods.

$•$ GePPO: Initially, we did not present GePPO as a baseline due to a mismatch between the Gymnasium and Mujoco environments used in our setup and those required by the original GePPO implementation. Both environments involve some subtle physical parameter differences, which could lead to unfair comparisons. However, we have now implemented GePPO using its original code version, and we will add the results to our manuscript for a more comprehensive comparison.

$•$ Policy-on-off PPO: We have attempted to implement this baseline; however, we encountered challenges due to inconsistencies between the environments used. In our current study, we have implemented all proposed algorithms in Mujoco and Gymnasium environments to ensure a fair comparison across tasks. However, the original P3O codebase is implemented specifically for the Atari game environment, with a code structure primarily designed for that context. Adapting it to work in Mujoco environments would require extensive refactoring and modification of the original implementation, which was not feasible within the scope of our current work.

$•$ Off-policy PPO: Thank you for pointing out this recent work. We have now implemented this baseline as well and will include the corresponding results in our manuscript.

We appreciate your valuable suggestions and will update the manuscript accordingly.

Weakness 2:

The learning curve for HP3O+ was initially not included in Figure 2 due to computational resource constraints. HP3O+ requires extensive parameter fine-tuning, which is computationally intensive, and scaling these experiments across all environments required considerable resources. At the time, we believed it would be sufficient to present results for two representative environments, as shown in Figure 3, to demonstrate the variance reduction and improved stability of HP3O+ compared to other algorithms.

However, we recognize the importance of providing a more comprehensive comparison. We have now conducted the necessary experiments for the remaining environments and included the learning curve for HP3O+ in Figure 2. These additions aim to provide a fuller picture of the performance improvements brought by HP3O+.

We will add these additional results in the revised version of the manuscript. We hope that these additions strengthen the paper and provide a more complete understanding of the proposed algorithm's capabilities.

Weakness 3: Thank you for pointing this out.

We are currently in the process of implementing the evaluation results for the environment, following the same evaluation protocol used for SAC. Once completed, we will update the manuscript to include these evaluation results. We will also add an additional figure and revise the relevant discussion in the manuscript to reflect these new evaluation analyses.

Thank you again for your suggestion.

Weakness 1:

We acknowledge that the use of the FIFO replay buffer might initially appear incremental. However, we would like to emphasize that this strategy plays a crucial role in significantly mitigating data distribution drift, which is a persistent challenge in off-policy learning. The FIFO trajectory buffer only maintains the recent trajectories, which ensures that the data used for policy updates remains representative of the current policy. This directly addresses one of the primary limitations of conventional off-policy methods, which is the divergence between past experiences and the current policy. Furthermore, we also leverage a trajectory buffer instead of storing isolated data points, which preserves the temporal coherence of sequences of actions and observations. This coherent data structure facilitates more effective policy updates by keeping the context intact, which reduces the high variance commonly associated with on-policy algorithms like PPO. One of the key issues with PPO is its high variance, which arises from the need to collect fresh data at each update step. Our approach blends the strengths of on-policy stability with the enhanced sample efficiency of off-policy learning in a principled manner, which we believe constitutes a meaningful advancement over existing methods.

The proposed HP3O algorithm and its variant have resorted to data randomly sampled from multiple policies in a training batch that is prior to the current policy for the policy update. Thus, there exist multiple updates compared to the vanilla PPO, which only makes one policy update from the current policy to the next policy. In this study, we aim to show how the off-policy sample reuse significantly affects the original sample efficiency PPO has. Though direct sample policy improvement can be significantly beneficial to providing a solid theoretical foundation for the proposed algorithms, a thorough investigation is out of the scope of this study. For instance, to arrive at an $\varepsilon$-optimality for on-policy policy gradient-based algorithms, a few existing works have revealed the exact complexity with respect to $\varepsilon$, but only for MDPs with linear function approximation. The exact sample complexity analysis for the off-policy PPO algorithm with nonlinear function approximation still remains extremely challenging and requires a substantial amount of non-trivial efforts. Thereby, in this paper, we instead disclose the impact of off-policy sample reuse on the tradeoff between sample efficiency and learning stability. We will add a remark on this in the main contents and detailed analysis in the Appendix.

We want to clarify that our primary focus is not on direct comparisons with fully off-policy methods, as our approach still maintains an on-policy nature despite incorporating a trajectory buffer. The trajectory buffer modifies the rollout mechanism to improve sample efficiency, but the overall structure and training process remain aligned with the on-policy characteristics of PPO. This means that while we use a replay buffer, we are fundamentally different from fully off-policy algorithms like SAC and TD3, and thus, our method should be viewed as a hybrid rather than a direct competitor to those approaches.

As mentioned in Remark 1 of the manuscript, we have acknowledged that incorporating PER into our replay buffer is a potential direction for future work. The current implementation of HP3O(+) uses a FIFO strategy for managing the replay buffer, which aligns well with our goal of maintaining a balance between recent experiences and sample diversity without introducing additional computational overhead.

We recognize that methods like DDPG, SAC, and TD3 are effectively able to leverage PER, prioritizing experiences based on their temporal-difference (TD) error. However, integrating PER into our approach requires careful adaptation to ensure stability within the hybrid on-off policy structure of HP3O(+), particularly given the potential biases that could arise from using prioritized samples in an on-policy context. This was beyond the scope of the current study but is a highly promising extension that we plan to explore in future work.

Weakness 2:

Thank you for pointing out the missing cross-references. We will correct all cross-referencing issues throughout our paper. We sincerely appreciate your careful review.

Question 4:

The definition and calculation of the explained variance will be addressed in our supplementary materials. To clarify further:

The explained variance is calculated as follows:

Explained Variance = $1 - \frac{\text{Var}(\hat{y} - y)}{\text{Var}(y)}$

where $\hat{y}$ represents the predicted values from the critic network, and $y$ represents the ground truth returns. Specifically:

$•$ Inputs ($\hat{y}$): These are the predicted state values generated by the value function approximation during training.

$•$ Ground Truth ($y$): This is the cumulative return (discounted sum of rewards) obtained from the environment, calculated for each state within the trajectory.

The explained variance metric provides insight into how well the value function explains the variance in the observed returns. A high explained variance implies that the value function is accurately capturing the variations in the true returns.

We will update the supplementary materials to make the calculation and definition of explained variance more explicit and clear.

Question 1:

Thank you for pointing out this question.

In our paper, at the end of the first paragraph in Section 4, we will include an equation that defines the calculation of the best trajectory and provide a brief description. Specifically, the best trajectory in the replay buffer is defined as the one that achieves the highest cumulative reward. For each trajectory in the buffer, we compute the return as the discounted sum of rewards, and the trajectory with the highest return is selected as the best trajectory ($\tau^*$).

We will further refine the manuscript to provide additional clarity regarding this definition. Thank you again for highlighting this point.

Question 2:

Implementation Without Padding

In our replay buffer implementation, we handle trajectories without padding to preserve the temporal characteristics and natural variations in trajectory lengths across different environments. To achieve this:

$•$ Each trajectory is stored in the replay buffer with its actual length, rather than zero-padding to match other trajectories.

$•$ The replay buffer stores each trajectory as a separate entry, while maintaining its original state-action pairs.

$•$ During minibatch sampling, we simply sample full trajectories, and no padding is applied since the model processes each trajectory in its entirety without introducing artificial data points.

This approach is beneficial for environments where the trajectory lengths vary significantly, as it allows the model to leverage the actual dynamics without diluting them with padded states.

Memory Consumption

Regarding memory consumption, the total memory usage depends on the following:

$•$ Number of Trajectories in the Buffer (Buffer Size): We use a fixed-size replay buffer, which holds $N$ trajectories.

$•$ Average Length of Trajectories ($T$): Since trajectories have varying lengths, the memory usage will depend on the average trajectory length.

$•$ Dimensionality of State and Action Spaces: Let $d_s$ be the dimension of the state, and $d_a$ be the dimension of the action. Each step in a trajectory stores both state and action information.

$•$ Data type size: Denote by $n$ the specific data type size.

The memory consumption ($M$) can be roughly estimated as: $M = NTn(d_s + d_a)$

For example, if $N = 1000$ trajectories are stored, each with an average length of 200 steps, and assuming $d_s = 17$, $d_a = 6$, and using 32-bit floats (4 bytes), the memory requirement would be: $M = 1000 \times 200 \times (17 + 6) \times 4 \text{ bytes} = 18,400,000 \text{ bytes} \approx 18.4 \text{ MB}$

This calculation provides an estimate, but the actual memory usage may vary depending on the environment and the specific implementation of the replay buffer.

We will add more details about this implementation in supplementary material to clarify the structure of the replay buffer and the memory requirements.

Question 3:

Thank you for pointing out this observation regarding the variance of HalfCheetah increasing as training progresses.

The increase in variance during longer training for the HalfCheetah environment can be attributed to several factors:

$•$ Exploration-Exploitation Trade-off: In complex environments such as HalfCheetah, the agent continues to explore different strategies over time. As the policy becomes less constrained, the range of actions becomes more diverse, which can lead to higher variability in the returns, thereby increasing the variance.

$•$ Environment Complexity and Instability: HalfCheetah is a highly dynamic and unstable environment. As the agent learns, it often employs increasingly aggressive and complex behaviors to maximize reward, which can lead to unstable trajectories and subsequently higher variance in performance.

$•$ Replay Buffer with FIFO Strategy: The FIFO strategy in the replay buffer retains only the most recent trajectories. As training progresses, the diversity of these recent trajectories may lead to higher variations in sampled batches, which affects the stability of learning and contributes to increased variance.

$•$ High Sensitivity of the Policy: As the policy approaches higher-performing regions of the solution space, it becomes more sensitive to the sampled experiences. Small differences in the sampled trajectories can lead to significant variations in policy updates, which can amplify the variance over time.

We acknowledge that the increasing variance is a challenge, and potential approaches to mitigate this include prioritized replay strategies or adjustments to balance exploration and exploitation more effectively. We will add this discussion in the revised manuscript to clarify this behavior.

We have now uploaded a newly revised version of our work. Please refer to the Summary for a detailed overview of all the changes we have made. If you have any additional questions or require further clarification, please do not hesitate to reach out. Thank you.

Dear Reviewer XcPF,

We hope this message finds you well. Thank you again for your comments and feedback. We have carefully addressed all of your points in our revised version, and we would greatly appreciate it if you could take another look at the updated manuscript.

While we understand that the discussion period has ended, we sincerely hope you will take our rebuttal and revised draft into consideration when finalizing your evaluation. We have done our best to address all the issues you raised and made significant improvements to resolve your concerns. Your feedback has helped us make the draft more technically solid and sound, and we are truly grateful for your contributions.

Your insights have been extremely valuable to us, and we truly hope the improvements we have made meet your expectations. Thank you so much for your time and consideration.

Best regards,

The authors of Paper 11700

We have uploaded a new revised version that includes additional evaluation results. Section A.15 presents examples of evaluation outcomes for several environments.