We thank the reviewer for his time and for providing feedback to improve our work.

The discussion of related work is rather limited and should be expanded by a control theory angle.

The related works section of our paper lists other benchmark papers and focuses mainly on categorizing and describing past methods that learns policies that produce smooth control behaviors, with few oscillations. Some of the methods investigated in our work do have more of a control theory perspective (esp. [1] and [2]), though not all of them. For example, Liu-Lipschitz [3] is an approach originally presented in the context of 3D imaging.

Because in our work we are interested in measuring the effectiveness of these approaches in a performance x smoothness analysis more so than their mechanisms, we believe an extended discussion from a control theory angle is a bit out of scope. We would invite an interested reader to check the original papers for details, specially for methods proposed from a control perspective.

Established benchmarks such as the DeepMind Control Suite can furthermore often be solved with high-frequency bang-bang control as studied in [...] or general discretization in [...]

Thank you for the reference suggestions. Although these works demonstrate that bang-bang control and discretization can solve the tasks, we did not include these types of controllers in our benchmark since they are the opposite of the type of smooth control we wish to achieve. A major part of our benchmark is done with real-world hardware and bang-bang/discrete type controllers cannot be safely deployed.

Nonetheless, the works listed are interesting benchmarks that evaluate performance in many environments, and we'll include them in the discussion of related works in the final paper.

A very relevant related work that automatically learns regularization trade-offs was presented in [4]

We apologize for missing the related work. The work cited by the reviewer is indeed very relevant. We'll include it in the related works section discussion in the final paper version.

The majority of cumulative returns as well as smoothness scores in Table 1 have overlapping error bands making performance differences difficult to judge.

We argue that our benchmark results show a significant improvement over a "Vanilla" policy versus regularization methods, showing that regularization is indeed effective. It is true that in a single scenario the smoothness between two regularization techniques might not be significant (see Hybrid methods in "Reacher", "ShadowHand", and "Velocity"). However, analyzing the smoothness x performance across all scenarios we can observe that the "Hybrid" methods outperformed others more often than not. While other approaches have significant deviations scenario to scenario, it is clear that the "Hybrids" are the most consistent method across the board.

The current evaluation has insufficient breadth to serve as a benchmark for smooth continuous control. Pendulum/Reacher/Lunar Lander are rather toy-ish examples that are not representative of the challenges the paper aims to analyze

We've added two additional scenarios: "ShadowHand" and "Handstand". These are challenging tasks that require agile and dexterous movements. The Handstand task is also deployed and evaluated with real hardware.

Additionally, we have also measured the performance of every single method in the real-world and included an ablation case of a Vanilla policy trained without domain randomization. Please refer to the comment "Extended Experiments and Results -- Updated Table 1" for the updated version of our results.

The analysis would profit from extension to a broader set of baseline algorithms other than PPO to identify trends in order to serve as an extensive benchmark

We agree with the reviewer that more extensive and broader experiments are better. Still, PPO is one of the most popular algorithm used by the community, specially for sim2real deployment. We believe the scope of the work is wide enough to be relevant to the research community.

---

We also invite the reviewer to check the "Summary of Changes" comment for a more extensive list of major and minor changes we've done thanks to your and other reviewers feedback.

---

[1] - Taisuke Kobayashi. L2c2: Locally lipschitz continuous constraint towards stable and smooth reinforcement learning. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4032–4039. IEEE, 2022

[2] - Ryoichi Takase, Nobuyuki Yoshikawa, Toshisada Mariyama, and Takeshi Tsuchiya. Stability certified reinforcement learning control via spectral normalization. Machine Learning with Applications, 10:100409, 2022.

[3] Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1–13, 2022.

We thank the reviewer for his time and for providing feedback to improve our work.

The authors (and title even) mix the concepts of policy smoothness and reducing high-frequency oscillations, but how these concepts relate and why you get oscillations is not thoroughly defined.

The reviewer raises a good point that our language in the paper is ambiguous. When we use the term "policy smoothness" or a "smooth policy" we simply mean a policy that exhibits/produces smooth control behaviors in the form of reduced oscillations, and not necessarily a policy that produces a smooth mapping function. However, and interestingly, the two do seem to be correlated as most methods that we investigated/refer in our paper do use the latter definition of smooth policy and aim to achieve smoothness, as in "a smooth mapping function", which in turns reduces oscillations, evidenced by our results.

We will clarify the meaning of these terms in the paper and better define "smoothness".

it would be useful if you had tested all the approaches in the real world to see how the approaches generalized.

We've followed up on the reviewer's suggestions and reran all of our real-world experiments with every method reported on the paper. Additionally, we've also measured the cumulative return in the real-world to enable a performance vs smoothness analysis.

Updated Real-World Results - Comparison of All Methods + No DR Ablation + Cumulative Return Computation:

Imitation: https://i.imgur.com/zuWvjqq.png

Velocity: https://i.imgur.com/UG2dvYF.png

Handstand: https://i.imgur.com/vmjOltq.png

• We did not deploy "No DR" ablation for handstand out of concern for robot and actuator damage.

It would also have been useful to see a video of the experiments as is conventional in robotics. Did the robot actually oscillate or is your smoothness metric just picking up a quick motion

Our original video demonstrates a case of an emergent behavior for the Hybrid policy where when the robot is picked up from the ground it will slowly stop moving. In contrast, when running the vanilla policy an out of distribution state such as this generates "wild movement" and jerking motions. We've also updated the supplementary material to include demonstrations of a new handstand policy. We can observe in the video that with our proposed hybrid method the motion is smoother during regular execution as well as when disturbed.

(On a LipsNet modification) This seems like an arbitrary choice that might degrade the performance of a defacto available option. Maybe the implementation is more up to date, or there was a typo in the paper? Can you test and confirm that it performs as well, or why not include both versions?

We've confirmed with the original authors of LipsNet that the version we implemented is indeed the correct and intended way. The authors had incorrectly used a linear activation as a default when open-sourcing their code (The output $K$ of LipsNet must be $> 0$. We've confirmed that the activation function used by the authors in their original experiments matches the version we've used in our work. Additionally, the original authors have since updated their open-source code to match the version we've used after our inquiry.

---

We also invite the reviewer to check the "Summary of Changes" comment for a more extensive list of major and minor changes we've done thanks to your and other reviewers feedback.

We thank the reviewer for his time and for providing feedback to improve our work.

As you have exclusively provided the measure of policy smoothness in real-world situations, I'm curious as to how the real-world performance (reward) measures up against other baselines?

We've followed up on the reviewer's suggestions and reran all of our real-world experiments with every method reported on the paper. Additionally, we've also measured the cumulative return in the real-world to enable a performance vs smoothness analysis. (See below).

In Figure 2, in the disturbance imitation task test, why does the combination of LipsNet + L2C2 underperform compared to the vanilla policy regarding smoothness?

Every policy deployed to the real-world is trained with domain randomization, including environment physics, terrain and actuator properties. The addition of domain randomization by itself results in smoother control behaviors. For a given task, regular execution (i.e. no disturbances) might be smooth enough that there are no significant differences between a regularized policy and a vanilla DR policy. However, when the policy needs to execute a sort of recovery behavior from an unstable state (which could be out of distribution), for example from an external disturbance, we can observe that a policy trained with smoothness regularization is indeed smoother.

For supporting evidence to our point above, we have included an ablation of a "Vanilla (No DR)" vs "Vanilla" which demonstrates this behavior.

---

Updated Real-World Results - Comparison of All Methods + No DR Ablation + Cumulative Return Computation:

Imitation: https://i.imgur.com/zuWvjqq.png

Velocity: https://i.imgur.com/UG2dvYF.png

Handstand: https://i.imgur.com/vmjOltq.png

• We did not deploy "No DR" ablation for handstand out of concern for robot and actuator damage.

---

We also invite the reviewer to check the "Summary of Changes" comment for a more extensive list of major and minor changes we've done thanks to your and other reviewers feedback.

We thank the reviewer for his time and for providing feedback to improve our work.

Could you share training plots at least for Ant and quadruped robot for the reward vs wall-clock time for LipsNet + CAPS and LipsNet + L2C2 vs vanilla MLP?

Unfortunately, because we ran our experiments in parallel, sharing computing load with other processes and experiments, the wall-clock vs reward graph is not truly representative of performance efficiency. We'll consider running a separate experiment sequentially to accurately measure reward vs (real) time.

Are there losses in training (wall-clock time) performance when using these more advanced methods vs MLP or they are computationally comparable to the vanilla MLP?

For pure loss-based methods we have not observed any difference in training time. In regards to the architectural methods, Spectral Normalization (SN-Local) and Liu-Lipschitz are both similar to a vanilla MLP in training time. Because our policy networks are not large, the training bottleneck ends up in the sample collection time in this case.

However, it should be noted that the LipsNet method comes with a significant slowdown. We observe an iteration time approximately 10 times longer than other methods, which is line with what the original authors reported in Appendix D. in [1]. By extension, this also affects the training time for the Hybrid methods that we proposed in our paper.

Overcoming such a slowdown and achieving the same smoothness and performance metrics we measured here could be a fruitful path for future research.

We'll incorporate the points above in the discussion in Section 4 of the paper.

Could you run experiments for more challenging control problems - humanoid, or on of the Allegro (Shadow) Hand dexterous manipulation tasks?

Thank you for your suggestion. We have run our experiments with the ShadowHand task in Isaac Gym. Please refer to the full updated table and comments in our top official comment "Extended Experiments and Results -- Updated Table 1".

We also invite the reviewer to check the "Summary of Changes" comment for a more extensive list of major and minor changes we've done thanks to your feedback.

---

[1] - Xujie Song, Jingliang Duan, Wenxuan Wang, Shengbo Eben Li, Chen Chen, Bo Cheng, Bo Zhang, Junqing Wei, and Xiaoming Simon Wang. Lipsnet: A smooth and robust neural network with adaptive lipschitz constant for high accuracy optimal control. 2023.

Extended Experiments and Results - Real-World

• At the request of multiple reviewers we've greatly extended the breadth of our experiments particularly in the real-world. Where originally we only deployed the best performing method from simulation, we have now ran and benchmarked every method presented in the paper for scenarios where sim2real is applicable.
• Included a new challenging real-world scenario: handstand. The agent is rewarded for balancing on its hind legs and remaining upright.
• Ran all 8 methods for the 3 real-world scenarios.
• Included an ablation for a policy trained without domain randomization (DR).
• Measured the real-world performance for each task with an alternative reward function.
  • The reward function in the real-world is representative of the task and contains similar components to the simulation function. The details will be added to a dedicated appendix section.
• Compiled the results in bar graphs that will supersede Figure 2.
  • Imitation: https://i.imgur.com/zuWvjqq.png
  • Velocity: https://i.imgur.com/UG2dvYF.png
  • Handstand: https://i.imgur.com/vmjOltq.png
• Discussion:
  • The results show that "Hybrid - LipsNet + L2C2" was the best method. It is not the only method that achieves a good combination of performance and smoothness, but it remained consistent for every tested scenario, while other methods faltered.
  • The domain randomization (DR) ablation shows that in addition to (greatly) improving real-world performance DR also results in smoother behaviors, see "Vanilla" vs "Vanilla (No DR)". Note that every method was trained with DR for the real-world experiment test. Still, the inclusion of architectural or loss-based methods as we've studied on this paper can further improve smoothness while maintaining comparable task performance.
  • Note that the "No DR" ablation was not deployed for the handstand scenario. This is a challenging scenario with unstable states and potential for robot collapse and actuator damage. We believe that the ablation for the other two scenarios is sufficient evidence that DR by itself can result in smoother control.
• The discussion points above will be added to "Section 4. Experiments and Results" along with the graph to replace Figure 2.

References

[1] - Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko. Regularizing action policies for smooth control with reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1810–1816. IEEE, 2021.

[2] - Taisuke Kobayashi. L2c2: Locally lipschitz continuous constraint towards stable and smooth reinforcement learning. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4032–4039. IEEE, 2022

[3] - Ryoichi Takase, Nobuyuki Yoshikawa, Toshisada Mariyama, and Takeshi Tsuchiya. Stability certified reinforcement learning control via spectral normalization. Machine Learning with Applications, 10:100409, 2022.

[4] - Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1–13, 2022.

[5] - Xujie Song, Jingliang Duan, Wenxuan Wang, Shengbo Eben Li, Chen Chen, Bo Cheng, Bo Zhang, Junqing Wei, and Xiaoming Simon Wang. Lipsnet: A smooth and robust neural network with adaptive lipschitz constant for high accuracy optimal control. 2023.

Extended Experiments and Results -- Updated Table 1

Cumulative Return (Higher is Better)

Smoothness (Lower is Better)

(*) Low smoothness but subpar performance.

• We transposed the Table for better reading of our results.
• The two newly included scenarios further reinforce the benefit of the smoothing methods compared to nothing/Vanilla. Additionally, the results also follow a similar pattern with the hybrid methods outperforming every other method in smoothness while maintaining a high performance (cumulative return).