A Simple Open-Loop Baseline for Reinforcement Learning Locomotion Tasks

Thank you for your comments. We've carefully read your suggestions and would like to provide the following responses.

Although the method requires minimal parameters, a lot of trial-and-error is needed in determining the number of oscillators to use for each environment, which is in direct contrast to RL methods which are general across multiple environments. The need for fine-tuning in each context may limit the method's scalability.

Thank you for your remark. It seems that our initial description may have led to a misunderstanding. To clarify, the number of oscillators used in our method is the same as the number of joints, since they operate in joint space. In practice, extending the method to additional environments is relatively straightforward; for example, it took the authors only 20 minutes to modify the code from Hopper to Walker.

While this result is really nice, this is also very expected. When maximizing the default reward of these environments, the resulting policy always corresponds to cyclical behavior which suggests that a simple cyclical controller could solve the task. Furthermore, these environments are very simple, and this work further intesifies the importance of moving on from this simple environments and focus on harder tasks.

While the success of the open-loop oscillators in the Swimmer environment is anticipated, its effectiveness in the Walker, Hopper or quadruped environments is more unexpected, as one might assume that feedback control or inverse kinematics would be necessary to balance the robots or to learn a meaningful open-loop policy. Another surprising fact is the tiny number of parameters needed compared to DRL algorithms like DDPG, which have almost a million of parameters. The results with RBF policies or ARS with linear policy were already unexpected when they came out [1, 2].

It is true that studies of nature and experiments have shown that periodic control is at the heart of locomotion [3], but it is unclear and surprising how simple a controller can be. As discussed in the last section of the paper, the fact that a single global frequency (per phase for hopper/walker) shared by each joint oscillator is sufficient is also an interesting finding related to another field that studies the intrinsic oscillation of robotic systems (nonlinear modes).

The surprise expressed by R2:vPWP and R3:w6U7 at the failure of DRL in a seemingly straightforward sim-to-real transfer scenario underscores the importance of the insights presented in our paper. By challenging common assumptions and exploring alternative solutions, we hope to contribute to the application of RL to robotics.

We updated the discussion section to underscore more the unexpected results from our paper.

[1] Rajeswaran, A. et al, Neurips 2017. Towards generalization and simplicity in continuous control.
[2] Mania, H. et al. Neurips 2018. Simple random search of static linear policies is competitive for reinforcement learning. [3] Ijspeert, A (2008) "Central pattern generators for locomotion control in animals and robots: a review."

Furthermore, these environments are very simple, and this work further intesifies the importance of moving on from this simple environments and focus on harder tasks.

MuJoCo environments are indeed simple, but have been the basis for advances in continuous control for RL, and are still widely used (for both online and offline RL). Our intention is to raise awareness of the failures of DRL on simple problems like the elastic quadruped, and to open discussion on the cost of complexity/generality, and why many heuristics are needed for simulation-to-reality scenarios.

We would also like to point out that although the task is simple for the elastic quadruped, it is not a trivial environment. The serial elastic actuators are more challenging to control than their rigid counterparts because they introduce more complex dynamics.

Results on more complex environment would make it harder to understand the failures and role of each component, which would go against recommendation by [4].

We updated the discussion section to highlight why results on simple environment can still provide valuable insights for the RL community.

[4] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."

Thanks for your feedback. The following are our replies to the questions, and we have updated the paper accordingly.

central pattern generators have been applied to legged locomotion in different ways in much more realistic and complex setting In this case, it’s hard to see the value of applying CPG to many simple locomotion tasks (Mujoco-v4 in gymnasium). Though Gym contains simple locomotion tasks, no matter whether it’s hard to solve or not, it’s mainly designed for evaluating different RL methods providing a specific manually designed controller for these tasks can hardly address issues in current RL methods.

Thanks for your comment.

Our goal is not to replace DRL methods, nor to claim to be state of the art, but as the reviewer noted, to provide insight into "common pitfalls of existing RL techniques" (see Sections 4.3 and 4.4 for instance), as recommended in section 4 of [1]. The MuJoCo locomotion tasks are indeed simple, but they still require hundreds of thousands of parameters and thousands of trials to solve, and are widely used in the RL community. The complexity of DRL algorithm makes it hard to reproduce, and the computation power required prevent them from being used on an on-board computer. Being aware of the failures of DRL on simple problems like the elastic quadruped, which illustrates the challenges of sim2real, and understanding the cost of complexity is crucial for improving application of RL to robotics.

To clarify the position of the paper, we have updated the abstract, the introduction and discussion sections.

[1] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."

Though this work is compared with several RL methods trying to study the robustness of current RL methods, no recent works are compared. The most recent work compared in this work is SAC. It would be great if this work can provide more comparison with recent RL works if this work is trying to address the issue of existing RL methods, like AWR, V-MPO (just example), or some model-based RL methods.

Thank you for your remark.

We use SAC v3 (with TD3 double critics and automatic entropy tuning [2]), which is an algorithm that provides strong baselines for MuJoCo tasks (similar performance to AWR), is widely used for continuous control problems, and is the basis for recent sample-efficient variants like TQC, REDQ or DroQ [3, 4, 5].

We could replace SAC with its distributional variant TQC [5], but this won't change the findings or the discussion of the paper. TQC has slightly better performance in simulation (still much better than a simple open-loop baseline) and a worse runtime compared to SAC [6]. However, because it shares the same policy structure (among other elements), it has the same shortcomings as SAC for the sim2real experiment (e.g., learning high-frequency control), and that's exactly why we use SAC as a baseline. We believe that SAC is already complex enough (especially with respect to an open-loop model-free baseline), and since SAC and its variants are widely used by the RL community in robotics, providing insight into SAC's pitfalls provides insight into the variants that share the same components.

It is for a similar reason that we use DDPG, to provide better context for the reader (following [1]), despite it being outdated.

We updated the experiments' section to reflect our choice.

[1] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."
[2] Haarnoja, T, et al. (2019) "Learning to walk via deep reinforcement learning."
[3] Chen, X, et al. (2020) "Randomized ensembled double q-learning: Learning fast without a model."
[4] Hiraoka, T, et al. (2022) "Dropout q-functions for doubly efficient reinforcement learning."
[5] Kuznetsov, A, et al. (2020) "Controlling overestimation bias with truncated mixture of continuous distributional quantile critics."
[6] https://github.com/openrlbenchmark/openrlbenchmark and https://wandb.ai/openrlbenchmark/sb3

It would be great to apply these techniques if this work is trying to claim the performance of proposed method in sim2real transfer.

Thank you for your comment.

We intentionally simplified the environment and removed task-specific heuristics in our sim2real experiments to understand why RL fails in this case and to observe the resulting behavior. Our goal is to provide insight and stimulate discussion about the trade-offs between complexity, generality, and performance in current deep RL algorithms.

While we could improve our results by adding complex reward engineering, randomization, or low-pass filtering, our intention is to propose the simplest model-free baseline possible. It would also be unfair to compare our simple open-loop baseline to a highly tuned DRL algorithm that incorporates many implementation details, uses the state as input, and has 10,000 more parameters.

We have updated the sim2real section to make that point clearer.

Is it possible to try more to adjust the parameters of the pd controller and the range of the joint target position generated by the RL policy to improve the results of the RL method? The demo shown by the authors is indeed worse than other RL-based locomotion works recently

Thank you for your remark. The result of the RL controller on the real robot is indeed worse than the recently published RL based locomotion, because we intentionally removed all the task specific heuristics and tried to make the environment as simple as possible. This is one of the main points of the paper, which is to understand why RL fails in this case and what is the observed behavior.

We could improve the results by adding complex reward engineering, randomization, or low-pass filtering (or equivalently tuning the PD controller until it works at the cost of slowness). However, our intention is to provide insight and food for thought about the cost of complexity and generality of the current DRL algorithm by proposing the simplest model-free baseline possible. It would also be unfair to compare the simple open-loop baseline to a highly tuned DRL algorithm, that includes already many implementation details, uses the state as an input and has 10 000 more parameters.

We have updated the sim2real section to make that point clearer.

However, whether those works only used RL or considered the combination of RL and classical control algorithms, they all produced much more impressive locomotion control results. [...] show more videos beyond walking in a straight line?

Thank you for your remark. It appears we had not yet made the paper’s stance clear enough. Our intention is not to replace DRL algorithms or to advance SOTA in performance, but to shed light on their limitations (such as failure on simple problems [3, 4, 5]) and to open discussion on their current complexity and "common pitfalls" (as noted by R1:nvZX, R3:w6U7, and R4:bfuq).

Following the recommendations of Section 4 of [6], we designed the experiments to provide "insights rather than state-of-the-art claims". For example, we show that only ten parameters are sufficient for a model-free approach that doesn't depend on the current state in several locomotion tasks, compared to the hundreds of thousands used by current DRL algorithms. Similarly, our results suggest that DRL algorithms are very brittle when exposed to sensor noise or failure, even on an uninformative sensor (this was confirmed by the experiment we added where SAC is trained with noise on the first sensor).

Current DRL algorithms can indeed solve more challenging locomotion tasks, but despite their generality, they require numerous implementation details and careful task design such as reward engineering, low-pass filtering or domain randomization (mentioned in the "discussion" section). Without these heuristics, DRL succeeds in simulation but fails to transfer on simple tasks like the one presented in the paper. On the same task, a very simple baseline that uses a single frequency for all joints succeeds. We believe that understanding the failure case on simple problems that are commonly used in the RL community is important for advancing the field.

To clarify the position of the paper, we have updated the abstract, the introduction and discussion sections. If the reviewer has other suggestions to make our intentions clearer, we would be happy to incorporate them.

[3] Colas, C et al. (2018) "Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms."
[4] Franceschetti, Maël, et al. (2022) "Making Reinforcement Learning Work on Swimmer."
[5] Matheron, G et al. (2019) "The problem with DDPG: understanding failures in deterministic environments with sparse rewards."
[6] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."

The paper has limited inspiration for the field of machine learning, which is the main topic of this conference.

We acknowledge that our paper provides a robotics perspective for reinforcement learning practitioners. However, we also believe that the insights provided in the paper (and demonstrated by the real robot experiment) are valuable for applications of RL in robotics (which is one of the topics mentioned in the ICLR 2024 paper call [7], see "applications to robotics, autonomy, planning").

[7] https://iclr.cc/Conferences/2024/CallForPapers

why the authers choose speed as the metric of the locomotion task?

We chose this metric because it was the simplest to define the task (only one reward term) and we could compare it to other controllers developed for the elastic quadruped robot.

Can the proposed method still be used if there are more obstacles in the environment, which need some complex behavior (turn left/right)?

As mentioned in the "limitation" section of the paper and in the reply above, open-loop control cannot, by design, address tasks that require responding to variations in the environment. However, the aim of the paper is not to propose a method that can replace RL, but provide a point of reference that is easy to study and that doesn't fail where "vanilla" DRL can be brittle (e.g. with sim2real transfer or when exposed to sensor noise).

Thank you for your remarks. We've carefully considered your suggestions and would like to provide the following responses.

The method is not innovative. In fact, the baseline of many previous RL work was a simple open-loop controller. However, whether those works only used RL [1] or considered the combination of RL and classical control algorithms [2][3], they all produced much more impressive locomotion control results.

Thank you for your feedback.

The authors assume that the reviewer is referring to Kumar et al. (2021), Iscen et al. (2019) and Miki et al. (2022) when mentioning that "the baseline of many previous RL work was a simple open-loop controller". (If we are mistaken, we kindly ask the reviewer to provide us with specific references. We will gladly incorporate them into our related work section.)

We believe that our work has a different scope than these three papers.

In Kumar et al. (2021), the focus is on proposing a new method for sim2real transfer, not understanding RL failures through a comparison with an open-loop baseline, which we were not able to find in that work. Iscen et al. (2019) propose a new approach to combine learning and trajectory generation. The trajectory generator is tailored to the quadruped problem (defined in task space) and is not compared to the learned policy (only briefly mentioned in the experiment section). In Miki et al. (2022), the goal is to learn robust locomotion policies using a height scan in addition to proprioceptive data. The observation and action spaces are inspired by CPG, but no open-loop baseline is used.

In summary, the three papers do not propose a simple baseline for RL tasks but rather focus on bootstrapping learning with robot knowledge, and provide results only for quadrupeds. In contrast, we present a simple baseline that is easy to study, easy to adapt to new problems and evaluate it on a popular benchmark using six different robot morphologies.

We would also like to clarify that the MuJoCo locomotion tasks, which we use in our experiments, were originally designed to evaluate model-based control methods [1], and were later adopted for continuous control RL. To the best of our knowledge, the simplest baseline previously employed for these tasks was a linear policy [2], while open-loop control has not been extensively explored in this context.

Since the MuJoCo locomotion tasks are widely used in the RL community (for both online and offline RL), we believe that providing a very simple baseline would provide valuable insights and better understanding of RL algorithms and their limitations (see Section 4.3 for instance).

[1] Todorov E. et al. (2012) "Mujoco: A physics engine for model-based control."
[2] Mania H. et al. (2018) "Simple random search provides a competitive approach to reinforcement learning"

Thank you for your time and valuable suggestions! Please find below our replies to your concerns and suggestions.

However, I think the paper is missing a discussion on how RL might still play a role when applied to robotics.

We agree that RL has a role to play in robotics (especially for locomotion tasks). Our intention was to convey this in the earlier version. We have updated the discussion section to make this statement clearer.

More generally, the results seem to indicate that if final performance is the key driver being optimised, RL may still be the tool of choice.

RL is the tool of choice for achieving top performance in simulation, as demonstrated by our experiment and discussed in the paper (see Table 2). However, applying RL in real-world scenarios requires numerous additional techniques, such as reward engineering, low-pass filtering, domain randomization, and leveraging privileged information.

Our goal is not to replace DRL algorithms but to highlight their limitations, including their struggles with simple problems. In doing so, we aim to raise awareness within the community about the tradeoffs between the generality of these algorithms and the effectiveness of a much simpler baseline, which has "natural advantages" as noted by the reviewer.

To clarify this point, we have updated the abstract, the introduction and discussion sections.

there are no pattern generating baselines being compared to. Can the authors include another non-learning baseline to understand how much is gained with the specific implementation proposed?

The authors interpret the reviewer's suggestion as an endorsement for the inclusion of a CPG baseline.

Our proposed baseline can be seen as a simplified version of [1, 2], which has similar properties (smooth by design, open-loop, periodic), but is much easier to implement and adapt to new problems. In fact, the CPG presented in these works was our starting point, but it was tailored for quadruped locomotion (using IK and fixed coupling matrix). To be able to adapt it to all MuJoCo locomotion tasks, we simplified it, removed the constraint of a fixed coupling and converged to the simple oscillators presented in the paper.

Additionally, our intention is to provide the simplest baseline for RL locomotion tasks and provide insights for RL algorithms (following [3]), not to advance SOTA in the CPG domain.

We updated the introduction and changed the order of the related work to emphasize the main message of the paper.

[1] Bellegarda, G., & Ijspeert, A. (2022). CPG-RL: Learning central pattern generators for quadruped locomotion.
[2] Raffin, A, et al. (2022) "Learning to Exploit Elastic Actuators for Quadruped Locomotion."
[3] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."

The Contributions list that the baseline can handle sparse rewards. As far as I know none of the environments used for the evaluation use sparse rewards though. Could the authors clarify this point?

Yes, the baseline we propose can handle sparse rewards, which is a benefit of black-box optimization (BBO). Because BBO uses only episodic returns, rather than immediate rewards, the baseline and ARS can effectively tackle problems with sparse or delayed rewards. Since this is a known pitfall of DRL algorithms, we wanted to mention this advantage without making it the main focus of our work. For this reason we did not focus on this aspect during the evaluation but chose to highlight it textually instead.

We added a sentence when presenting the method to make that point clearer.

For the robustness to sensor noise, can the authors include a baseline where SAC is trained with a subset of the noisy parameters? For instance, if SAC were trained to withstand random noise of say 3N, it seems more likely that it would be able to withstand the 5N noise tested in the paper.

Thanks for the suggestion, we added an experiment where SAC is trained with a noisy sensor to the paper.

For Figure 3, shouldn’t the dot for Open Loop sit exactly at 1? It seems that the plot is slightly below 1 which seems off to me.

The plot is indeed slightly below one. This is because we normalize with respect to the maximum performance, and since there is some variance between runs (see performance profiles), the normalized performance of the oscillators is not exactly one, but close to one.

While I like the general evaluation in the Experiments section, I found Table 6 in the Appendix with the actual scores informative. Since there is still space, I would suggest adding that table to the main text - if the authors think it would not affect the narrative flow too much.

This is indeed a conscious choice. The authors follow the best practice recommendations from [4 - NeurIPS 21 outstanding paper, 5] and provide a better statistical representation of performance through the use of IQM, stratified bootstrap, performance profiles, and probability of improvement.

We would like to emphasize that while performance is a critical aspect, runtime and parameter efficiency are another. To maintain the clarity of the paper's narrative, we have chosen not to include the table in the main text. Instead, it is provided in the appendix for readers who are interested in exploring this aspect further.

[4] Agarwal, R, et al. (2021) "Deep reinforcement learning at the edge of the statistical precipice."
[5] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."