Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning

Segment Length (Number) and Value Function Learning

• The segment length, or the number of segments per trajectory, does indeed affect performance. However, we observed that within an optimal range, such as 10-25 segments per trajectory, this parameter is not overly sensitive compared to other hyperparameters. Thus, we use K=25 in all of our experiments. A detailed study on the influence of the number of segments is provided in the updated supplementary material, Section R.5, which was previously reported in Section C.7.
• The approach we use in TCE for learning value functions aligns with that of standard RL methods, such as PPO. In our model, the segment length K influences only the policy update and does not affect the learning of the value function. This method of leveraging value functions is a key factor in why TCE achieves faster learning speeds compared to BBRL[4] in most tasks we reported. However, it also leads to a similar performance decline in the table tennis task, mirroring the challenges faced by other step-based RL methods that rely on value function learning.

Experiment Settings

• Dense and sparse rewards settings
  We attribute the observed phenomenon to two main factors. Firstly, as previously discussed in [4], sparse reward settings often provide a clearer task description, aligning the reward structure more directly with the success evaluation criteria. Secondly, for some of the step-based algorithms, we employed a discount factor of 1.0, as opposed to the usual 0.99. This adjustment mitigates the task's sparsity, potentially enhancing the performance of these methods. However, since the task is constrained by a limited number of time steps, the episode return remains within a finite range. Despite these considerations, some step-based algorithms still exhibit poorer performance compared to their counterparts in dense reward settings.

• Longer training iterations

  Previsouly, we found a few baelines collapsed after a certain training iterations. And for off-policy methods, due to their update-to-data (UTD) ratio of 1, as suggested in references [6, 7], their training often take 1-2 days in the reported tasks. However, the main challenge is not the collection of samples, but the network updates and the overhead involved in data exchange between the policy and the environments. Considering that off-policy methods are designed for enhanced sample efficiency, their failure to learn effectively from a data volume comparable to that utilized by on-policy methods highlights their inferior performance in the tasks evaluated in the paper.

References

[1] Peters, J. and Schaal, S., 2008. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4), pp.682-697.

[2] Kober, J. and Peters, J., 2008. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21.

[3] Peters, J. and Schaal, S., 2006, October. Policy gradient methods for robotics. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 2219-2225). IEEE.

[4] Otto, F., Celik, O., Zhou, H., Ziesche, H., Ngo, V.A. and Neumann, G., 2023, March. Deep black-box reinforcement learning with movement primitives. In Conference on Robot Learning (pp. 1244-1265). PMLR.

[5] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.

[6] Raffin, Antonin, Jens Kober, and Freek Stulp. "Smooth exploration for robotic reinforcement learning." Conference on Robot Learning. PMLR, 2022.

[7] Haarnoja, T., Zhou, A., Abbeel, P. and Levine, S., 2018, July. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (pp. 1861-1870). PMLR.

[8] Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P. and Levine, S., 2018. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905.

================================================

If you have any further questions, please feel free to ask us.

Thank you.

TCE Authors

Dear Reviewer DXh7,

Thank you for your valuable and insightful feedback on our work. We have addressed the major concerns you raised to the best of our knowledge. There are still some open questions that we are actively working on, and we are committed to resolving these in the next few days.

[Update Log, Nov. 18.2023]

Address the major concerns of all reviewers

Upload additional evaluation and ablation study to Supplementary Material

[Update Log, Nov. 22, 2023]

Restructure the paper's storyline, and address the remaining concerns from the reviewers

Novelty and Contribution

We wish to re-emphasize the novelty and significant contribution of TCE, which builds upon the foundational of previous works. TCE represents a distinct algorithmic advancement in Episodic Reinforcement Learning, going beyond mere modifications of existing models. Our work is pioneering in its incorporation of step-based information into the policy update process of Episodic RL [1,2,3,4]—a field that traditionally relied on black box optimization methods. Importantly, TCE maintains the inherent benefits of broad exploration and trajectory smoothness that are characteristic of episodic RL.

Reference of BBRL

BBRL stands for Deep Black Box Reinforcement Learning, as described in [4], an episodic RL method that learns trajectories directly in the parameter space. We will update our paper to offer a proper introduction of it.

MDP and Learning Objective

• Our learning framework maintains the Markovian property, as discussed in section 3.1 of [5]. The Markovian property specifically concerns the state transition probability, $p(s_{t+1}, r_{t} | s_{t}, a_{t})$, indicating that the next state and reward depend solely on the current state and action. It does not impose any assumptions on the policy or the decision process. In our policy, we predict the entire trajectory as a sequence of actions at the start of each episode. This approach to action selection is fundamentally similar to other methods, such as repeating actions over consecutive steps [6] and using temporally correlated noise in action sampling [7].
• We have thoroughly reviewed the paper by [Smith et al]. However, we note that the low-pass filter technique employed in their study is fundamentally different from the approach in our work and other research focused on directly modeling smooth trajectories. One possible failure reason of such a low-pass filter was discussed in the gSDE work [6], as it tends to cancel consecutive pertubations, and will thus lead to poor expoloration.
• Our learning objective as presented in Equation (8) can be viewed as an approximation of the episodic RL’s learning objective in Equation (1), rather than being derived from other Policy Gradient Method. This approximation is achieved by replacing the trajectory likelihood and advantage with their respective segment-wise versions.

Dear reviewer Dxh7,

Since the rebuttal period is almost over, we are eager to know whether our responses have addressed your concerns properly. Here we humbly ask for your feedback at your convenience such that we may still have the chance to discuss with you and chance to improve our final version before the close of the rebuttal window. Thank you!

All the best,

TCE authors

Dear reviewer DXh7,

Thank you for your reply! We sincerely appreciate your approval of our work.

Best regards, TCE Authors

Ablation Studies

We include additional study in the updated supplementary material. Section R.1 highlights the advantages of using full covariance in enhancing trajectory smoothness. Section R.2 examines the capacity of various methods to model movement correlation, with TCE and BBRL Cov demonstrating the greatest flexibility in trajectory sampling. However, it's crucial to note that the ability to model movement correlation is not the only factor contributing to successful performance. The efficiency of policy updates plays a significant role as well, which is a key reason why TCE outperforms BBRL in the majority of tasks we have reported. Similarly, PINK's performance limitations can be attributed to constraints in learning its Q-function, a challenge it shares with SAC. These factors may account for their relatively poorer performance in some tasks.

Formatting Issue

We apologize for the inconvenience, this problem is fixed in the updated version of the paper.

Approach to directly represent each segment

Some studies, such as [5], have employed movement primitives to represent individual segments. However, this approach has been shown to underperform compared to methods that model the entire trajectory, like BBRL, as previously discussed in [4].

References

[1] Peters, J. and Schaal, S., 2008. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4), pp.682-697.

[2] Kober, J. and Peters, J., 2008. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21.

[3] Peters, J. and Schaal, S., 2006, October. Policy gradient methods for robotics. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 2219-2225). IEEE.

[4] Otto, F., Celik, O., Zhou, H., Ziesche, H., Ngo, V.A. and Neumann, G., 2023, March. Deep black-box reinforcement learning with movement primitives. In Conference on Robot Learning (pp. 1244-1265). PMLR.

[5] Bahl, Shikhar, et al. "Neural dynamic policies for end-to-end sensorimotor learning." Advances in Neural Information Processing Systems 33 (2020): 5058-5069.

==============================================

If you have any further questions, please feel free to ask us.

Thank you.

TCE Authors

Dear reviewer psBV,

Thank you for your valuable feedback, which has been instrumental in enhancing our paper. We have addressed the major concern you raised. Additionally, we are actively working on some open questions that emerged from your insights.

[Update Log, Nov. 18.2023]

Address the major concerns of all reviewers

Upload additional evaluation and ablation study to Supplementary Material

[Ongoing works, Nov. 18.2023]

Re-organizing the paper structure and re-doing a few experiments.

Terminology

Episodic Reinforcement Learning (ERL) is a well-established concept, previously applied to learning movement primitives in various studies [1, 2, 3, 4]. This approach shifts the search for solutions from the per-step action space to the trajectory parameter space, thereby promoting broader exploration. In this context, entire trajectories are often treated as single data points, and models are typically trained using black-box optimization methods.

TCE, designed for episodic tasks with a finite horizon, similarly generates a single complete trajectory at the start of each episode, which is then followed until the episode concludes. The key distinction of TCE from earlier ERL approaches lies in its utilization of step-based information from sub-segments of the trajectory, enabling more efficient policy updates.

Contribution Statements

While building upon the foundations laid by previous approaches like ProDMP and TRPL, we believe that TCE stands out as a significant algorithmic advancement in RL. As a novel framework, TCE distinguishes itself by integrating step-based information into the policy updates of episodic RL. This integration not only boosts update efficiency but also maintains trajectory smoothness and supports correlated action exploration, both temporally and across degrees of freedom (DoF). TCE is the first work to successfully merge the benefits of both RL methodologies, marking a novel advancement in the field.

Hyperparameter Selection

We executed a large-scale grid search to fine-tune key hyperparameters for each baseline method. For other hyperparameters, we relied on the values specified in their respective original papers. Below is a list summarizing the parameters we swept through during this process.

• BBRL: Policy net size, critic net size, policy learning rate, critic learning rate, samples per itera- tion, trust region dissimilarity bounds, number of parameters per movement DoF.

• TCE: Same types of hyper-parameters listed in BBRL, plus the number of segments per trajectory.

• PPO: Policy network size, critic network size, policy learning rate, critic learning rate, batch size, samples per iteration.

• TRPL: Policy network size, critic network size, policy learning rate, critic learning rate, batch size, samples per iteration, trust region dissimilarity bounds.

• gSDE: Same types of hyper-parameters listed in PPO, together with the state dependent exploration sampling frequency

• SAC/PINK: Policy network size, critic network size, policy learning rate, critic learning rate, alpha learning rate, batch size, Update-To-Data (UTD) ratio.

Dear reviewer psBV,

Thank you for your feedback. We are glad to note that our efforts have addressed your concerns. To address the remaining issue, we have included an additional ablation study in the updated supplementary material, Section R.7. It conducts a comparison of TCE against its STD variant using a factorized Gaussian policy for the parameter. The results clearly indicate underperformance by the std version, and we attribute this to its limited capacity in modeling movement correlations.

Link to supplementary: https://openreview.net/attachment?id=mnipav175N&name=supplementary_material

Computation cost

In terms of computational load, the main distinctions between cov and std policies are

• Computational Cost in Convex Optimization for Trust Region Enforcement

  As discussed in [1], cov policies require additional computational effort to satisfy constraints like the KL divergence in the trust region enforcement. This is because cov policies include off-diagonal elements representing the 'rotation' of the Gaussian distribution, necessitating more time in the projection process compared to std policies.

• Matrix Inversion During Likelihood Computation

  For a cov policy, inverting the covariance matrix during likelihood computation has a time complexity of $O(n^3)$, where $n$ is the dimension of the parameter vector $w$. In contrast, for a std policy, this operation has a time complexity of $O(n)$.

These factors translate into a 20-50% difference in training time for methods like BBRL when comparing Cov and Std policies. However, in the case of TCE, our focus is on leveraging step-based information for policy updates, where the likelihood computation is w.r.t. the trajectory, as opposed to the parameter $w$. Here, a std distribution in the parameter space maps to a blocked diagonal covariance matrix in the trajectory space, as shown in section R.2, figure 1c. Despite its simpler appearance, this format maintains the same time complexity for likelihood computation as a full trajectory covariance matrix. Thus, the difference in training TCE Std and TCE Cov is minor, arount 10%.

To efficiently manage the likelihood computation in trajectory space, we employ techniques that estimate trajectory likelihood using segment-based counterparts. As a result of this approach, the computation time for TCE is approximately twice that of BBRL.

References

[1] Otto, Fabian, et al. "Differentiable Trust Region Layers for Deep Reinforcement Learning." International Conference on Learning Representations. 2021.

======================================================================

Please let us know if you have further concerns to our work.

Thank you,

TCE Authors.

Dear reviewer ALmz,

We are grateful for the time and effort you have dedicated to reviewing our submission. In response to your valuable feedback, we are currently re-structuring the storyline of our paper to enhance its readability. An updated version will be provided shortly. In the meantime, we have addressed the other concerns and questions you raised to the best of our knowledge, as detailed below.

[Update Log, Nov. 18.2023]

Address the major concerns of all reviewers.

Upload additional experiment and ablation study to Supplementary Material.

[Update Log, Nov. 22, 2023]

Restructure the paper's storyline, and address the remaining concerns from the reviewers

Writing:

• We agree with you that the storyline and the connection to other works need to be improved. The revised version will be updated shortly.

Experiments:

• While we acknowledge that certain baselines, such as gSDE in the box-pushing sparse task and BBRL-cov in table tennis, outperform TCE in specific scenarios, we believe that TCE consistently delivers comparable results across most tasks. Notably, in the MetaWorld tasks, TCE not only achieved the highest asymptotic performance but also maintained the most consistent performance profile. This consistency is characterized by successfully solving a majority of tasks and maintaining a high success rate in those it can solve. Although gSDE shows strong results in box-pushing tasks, its overall performance in the MetaWorld's 50 tasks is moderate, and it fails entirely in the table tennis task. In stark contrast, TCE still manages to achieve a commendable success rate of 65% in the latter.

• Real Scenes

  We agree that comparing these methods in real-world environments would be very useful. However, as mentioned in our paper, many RL baselines have a problem with smooth movements in their trajectories, which could harm real hardware. Following reviewer dLeV's suggestion, we added a new section on trajectory smoothness in the updated supplementary material (Section R.1). This section shows that most step-based RL methods have very low smoothness, even with well-trained policies. On the other hand, using movement primitives has been successful in real tasks, like imitation learning [1] and Sim2Real transfer in RL [2]. Yet, for real-world applications, we believe it essential to first test and confirm the effectiveness of the policy in a simulation environment before transitioning it to real-world scenarios.

References

[1] Gomez-Gonzalez, Sebastian, et al. "Adaptation and robust learning of probabilistic movement primitives." IEEE Transactions on Robotics 36.2 (2020): 366-379.

[2] Klink, Pascal, Hany Abdulsamad, Boris Belousov, and Jan Peters. "Self-paced contextual reinforcement learning." In Conference on Robot Learning, pp. 513-529. PMLR, 2020.

================================================

If you have any further questions, please feel free to ask us.

Thank you.

TCE Authors

Dear reviewer ALmz,

we followed your valuable feedbacks and updated the content of the paper, especially making the structure of Section 3 more concise. As the rebuttal phase is about to finish, we are wondering if our previous responses have addressed your concerns properly. Your feedback will definitely help us reach a more reasonable decision on our submission.

=======================================================================

The main updates in the manuscript that related to your feedback:

• Revise the abstract and restructure the introduction section to explicitly define the concepts of Step and Episodic RL.
• Clarify the term "Open the Black Box" in the title, as well as the research gap that we want to solve.
• Emphasize the main contribution of the paper, and discuss how our framework differs from the existing ones.
• Split the previous Section 2 "Background and Related Works" into
  • a) "Preliminaries" as Section 2, focusing on the methods foundational to our framework.
  • b) "Related Works" as a new Section 4, discussing literature approaches that address similiar research gaps.
• Link Episodic RL to the MDP in Section preliminaries.
• Review and refine the claims at the end of Section 2.1, ensuring they are supported by relevant references.
• For each preliminary method discussed in Section 2, clearly state the specific problem it targets and its relevance to Episodic RL.
• Provide detailed explanations of the mathematical variables used in our framework, including their dimensions.
• Remove redundant content in Section 3 and streamline its structure for better clarity and conciseness.
• Include a discussion on the works closely related to ours, with a special focus on the development of Episodic RL
• Discuss BBRL (Deep Black Box Reinforcement Learning) in detial as it is highly relevant to our work.

=======================================================================

Additionally, we have included an algorithm box to help understand the paper, along with other newly added evaluations and ablation studies in the updated supplementary material.

Link to updated paper: https://openreview.net/pdf?id=mnipav175N

Link to supplementary material: https://openreview.net/attachment?id=mnipav175N&name=supplementary_material

=======================================================================

Thank you!

All the best,

TCE Authors

Dear Reviewer dLeV,

Thank you for your insightful review of our paper. Your feedback has been instrumental in enhancing our work. We have addressed the majority of your concerns and added supplementary material with new experiments and ablations. Currently, we are working on addressing the remaining questions and are in the process of restructuring the paper.

[Update Log, Nov. 18.2023]

Address the major concerns of all reviewers.

Upload additional evaluation and ablation study to Supplementary Material

[Update Log, Nov. 22, 2023]

Restructure the paper's storyline, and address the remaining concerns from the reviewers

Episodic RL & Black-Box Optimization

Episodic Reinforcement Learning (RL), as discussed in references [1,2,3,4], is a distinct branch in the evolution of RL that emphasizes maximizing the return of an entire episode, rather than focusing on the internal dynamics of the environment. This approach shifts the search for solutions from the per-step action space to the trajectory parameter space, thereby promoting broader exploration. In this context, entire trajectories are often treated as single data points, and models are typically trained using black-box optimization methods [5,6,7,8]

The term 'black box' in our title reflects the essence of these methods, rooted in black-box optimization principles, which tend to overlook the detailed step-based information acquired during interactions with the environment. While these methods excel in expansive exploration and trajectory smoothness, they generally require a larger number of samples for effective policy training. In contrast, step-based RL methods have shown substantial improvements in learning efficiency by leveraging this step-based information.

Novelty & Opening the Black Box

Despite the leverage of previous approaches, we believe TCE has made a distinct algorithmic advancement in the field of RL. As a novel framework, TCE uniquely incorporates step-based information into the policy updates of episodic RL, simultaneously preserving trajectory smoothness and facilitating correlated action exploration both temporally and across DoF. Thus, by effectively integrating the strengths of the two RL methodologies, TCE successfully 'opens the black box’.

Writing:

• Figure 1 is an illustration, not an experiment
• We will add a MDP description in the updated paper. However, our learning framework maintains the Markovian property, as discussed in the Chapter 3.1 of [9]. The Markovian property specifically concerns the state transition probability, $p(s_{t+1}, r_{t} |~s_{t}, a_{t})$, indicating that the next state and reward depend solely on the current state and action. It does not impose any assumptions on the policy or the decision process.
• d_mean and d_cov are general notations for discrepancy measures, we specifically use KL-divergence in this work as it has been shown to be effective for policy learning within the parameter space of Motion Primitives [5].

Performance against Baselines

While we recognize that the performance gains in certain environments may be marginal, it is crucial to highlight that our algorithm, TCE, consistently demonstrates superior or at least comparable results in the majority of tasks when benchmarked against robust baselines. A notable example is the table tennis task, where gSDE failed entirely, yet our algorithm achieved a 60% success rate. We acknowledge TCE's underperformance in 5 out of the 50 Metaworld tasks. However, considering the wide variety of task types in Metaworld, this is not unusual. Similar to other baseline methods, which also exhibit variable performance across different tasks, TCE has not been specifically optimized for any individual task. Despite this, it has shown exceptional overall performance across the comprehensive Metaworld benchmark. This not only underscores TCE's robustness but also its adaptability, making it a highly effective method in a broad spectrum of scenarios.

Training horizon of off-policy methods

The original SAC employs an update-to-data (UTD) ratio of 1, as suggested in references [10], a configuration known for balancing training stability with sample efficiency. However, this ratio does not leverage the advantages of parallel processing across multiple environments. The main challenge is not the collection of samples, but the network updates and the overhead involved in data exchange between the policy and the environments. This issue often restricts training to a relatively lower number of samples, typically ≤ 10M steps, as described in [10]. Considering that off-policy methods are designed for enhanced sample efficiency, their failure to learn effectively from a data volume comparable to that utilized by on-policy methods highlights their inferior performance in the tasks evaluated in the paper.

Ablation Studies:

• We include a smoothness evaluation in the updated supplementary material, Section R.1. The evaluation is conducted using three common smoothness metrics in robotics and motion planning. The comparison shows TCE and the BBRL Cov have achieved the highest smoothness among all the approaches.

• In Section R.2 of the updated supplementary material, we present a qualitative evaluation of the action correlation for all methods. This evaluation highlights the varying capacities of these methods in modeling movement correlations. Both TCE and BBRL Cov successfully model correlations across both time and inter-DoF. In contrast, the original BBRL and PINK are limited to modeling correlations within individual DoF. gSDE demonstrates the capability to model temporal correlations, but only over a few consecutive time steps. Notably, PPO, TRPL, and SAC do not engage in action correlation modeling.

  However, it is important to emphasize that the capability to model movement correlation is not the sole determinant of successful performance. The BBRL, for instance, is hindered by an inefficient policy update strategy. Similarly, PINK's performance limitations can be attributed to constraints in learning its Q-function, a challenge it shares with SAC. These factors may account for their relatively poorer performance in some tasks.

• We agree that including tests for PPO and SAC with Movement Primitives (MP) action space is an important ablation study. Previous work [5] has already shown the ineffectiveness of combining PPO with Movement Primitives (MP).

  In the updated supplementary material, Section R.3, we include an additional ablation study where we combined SAC with MP parameters as the action space. The results from this study suggest that this combination is not effective. We believe this ineffectiveness arises from the increased dimensionality of the action space, which significantly complicates the learning process for the Q-function.

• Using p(w) instead of p(y) for likelihood is exactly what BBRL [5] did. We already included that as a baseline, but we agree that more discussion of the method should be included in the experiment section to clarify the differences, and we are currently working on that.

References:

[1] Whitley D, Dominic S, Das R, Anderson CW. Genetic reinforcement learning for neurocontrol problems. Machine Learning. 1993 Nov;13:259-84.

[2] Igel, C., 2003, December. Neuroevolution for reinforcement learning using evolution strategies. In The 2003 Congress on Evolutionary Computation, 2003. CEC'03. (Vol. 4, pp. 2588-2595). IEEE.

[3] Peters, J. and Schaal, S., 2008. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4), pp.682-697.

[4] Kober, J. and Peters, J., 2008. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21.

[5] Otto, F., Celik, O., Zhou, H., Ziesche, H., Ngo, V.A. and Neumann, G., 2023, March. Deep black-box reinforcement learning with movement primitives. In Conference on Robot Learning (pp. 1244-1265). PMLR.

[6] A. Abdolmaleki, D. Sim˜oes, N. Lau, L. P. Reis, and G. Neumann. Contextual direct policy search. Journal of Intelligent & Robotic Systems, 96(2):141–157, 2019.

[7] Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.

[8] Tangkaratt, V., Van Hoof, H., Parisi, S., Neumann, G., Peters, J. and Sugiyama, M., 2017, February. Policy search with high-dimensional context variables. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 31, No. 1).

[9] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.

[10] Haarnoja, T., Zhou, A., Abbeel, P. and Levine, S., 2018, July. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (pp. 1861-1870). PMLR.

================================================

If you have any further questions, please feel free to ask us.

Thank you.

TCE Authors

Dear reviewer dLeV,

We updated the paper manuscript following your valuable feedback.

As the rebuttal period is near to finish, we are eager to know if our recent responses have sufficiently addressed your concerns. Your insights will definitely guide us towards a more informed decision regarding our submission. We appreciate your feedback and thank you for your time.

All the best,

TCE Authors