End-Effector-Elbow: A New Action Space for Robot Learning

We would like to thank the reviewer for their feedback. We provide an answer to their questions below.

The approach is not general: as pointed out by the reviewer, the idea of constraining one specific joint for E3J is an arbitrary choice and may not always be the most effective choice. In the original submission, we discussed this in Section 4.4, in the Joint Selection and Practical Implementation Choices paragraphs, where we motivated our choice for mainly considering E3J-base and E3J-wrist. In order to further validate our choice, we have now completed the ablation study, where we attempt to constrain other joints of the robot for E3J (find results in the updated manuscript's Appendix). The results show that our intuitions generally proved correct, but there are tasks where constraining other joints than the base or the wrist improves sample efficiency. We also believe E3A could be a more general strategy than E3J, but in practice we found that the agent struggles to directly control the elbow angle more than indirectly controlling the elbow through one constrained joint. As studying what's the best action space for robot learning, in terms of full-body control and efficiency, is one of the main goals of this work, we have selected E3J as the idea that best fits this definition according to empirical evidence.

Relation to existing work: we found the work of Sharma et al, which presents a hierarchical framework to combine task-space constraints, to be relevant to our method, and thus we added a discussion about it in the revised manuscript. Overall, as pointed out by the reviewer, the work presents several differences from our approach, in the settings and the goal pursued: (i) it presents a hierarchical method for selecting actions, while our method focuses on learning flat policies; (ii) it applies task-space constraints, while we combine task-space control with joint constraints, (iii) the goal of Sharma et al's work is to present a modular framework that improves sample efficiency and generalization, while we explicitly focus on studying the most convenient action spaces that enable reinforcement learning or imitation learning methods to solve full-body tasks, whilst retaining the benefits of controlling the agent in task space coordinates. We discuss the work in the Related Work section of the revised manuscript (highlighted in green), along with citing the relevant literature about null-space projections and task-space constraints that have been recommended.

Furthermore, the reviewer made a comment that an interesting direction could be to combine our constraints for controlling the elbow (controlling the elbow angle or one of the robot's joints) with a hierarchical controller (as in Sharma et al). This indeed sounds promising for future research, and so we added it in the Conclusion (see the revised manuscript, in green text).

Benefits of applying the approach in tasks that require less DoFs: in order to compare the learning efficiency of our approach compared to EE on tasks that do not require full-body control, we presented results on 8 RLBench popular tasks, providing aggregated performance in the main paper and detailed results in the Appendix. We did not find any significant difference in performance between EE and E3 in these settings. However, we would like to highlight that 2 out of the 6 full-body tasks are also standard RLBench tasks (meat off grill and take cup out cabinet), requiring in theory <= 6 DoFs. However, the presence of obstacles in reaching the task's main object ends up favouring our method in terms of learning efficiency, as E3 provides the control necessary for consistent obstacle avoidance.

Images/videos of full-body tasks: we added the requested visualizations, with a Figure in the Appendix, and with videos on the project website.

[continues]

(...continues)

• What happens when there exists no IK solution?

This implementation detail is already described in the paper. In the Environment details section, we mentioned:

"Invalid actions are rewarded with 0 and the agent is kept still, to discourage the agent from taking them."

and invalid actions are described in the main text as:

"Note that invalid actions can be caused by the Jacobian-based IK solver failing due to an impossible configuration request. This could be due to: (i) joint configurations that are too far from the current one and thus unreachable, (ii) attempts to collide with objects, or (iii) configurations outside the joint limits."

We also would like to highlight that we have shared our code, so that any implementation detail can be verified for the simulated experiments, and we will open source the code shared in the supplementary material upon publication to foster further research in this direction (thanks to the new two action spaces, the four new tasks and the three new robots we introduced).

Hierarchical control

While the idea of hierarchical control would definitely be interesting to explore for future work, and we discussed this in the paper, as the reviewer acknowledged, the current work is about a new action space, involving no hierarchy. Hierarchical control requires the agent to reason about actions in different stages while our work requires the agent to output a hybrid set of coordinates, in task and joint spaces, in one flat stage. Hierarchical control mostly employs actions that are temporally-consistent and last for a longer time, while our work focuses on atomic actions that should be provided at high frequency for closed-loop control.

For example, in the work mentioned by the reviewer [1], the agent should reason about which controllers should be combined, by selecting multiple object-axis controllers in a prioritized order, which are composed together using null-space projections. In contrast, in our work, there is no priority as the action's dimensions have all the same priority, and thus the policy is "flat".

Another example is primitives [2], where the agent should reason, first, about which primitive to use, and then, on the parameters for applying the primitive to the environment. Our approach jointly controls the robot's configuration by providing targets in task and joint coordinates, rather than prioritizing any of them.

Extending our work to flexibly choose which joint to directly control with E3J is described as a future work direction in the Conclusion, and we believe our work, including all the assets we are releasing (new E3A and E3J action spaces, new full-body tasks to evaluate, new robot models to experiment with), will provide an useful starting point and a solid baseline for future research in this area.

[1] Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation, Sharma et al

[2] Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives, Dalal et al

We are encouraged that the reviewer finds our submission interesting and we hope that the above clarifications along with the revisions to the manuscript address the reviewer's remaining comments.

If you are more satisfied with our revision, we would kindly ask you to update your score to reflect this.

Dear reviewer,

we would like to report that the following experimental studies have been added to the revised manuscript:

• evaluation on different robotic platforms: using the Kuka iiwa 7 and xArm. In this setup, we showed that E3J can be applied in other platforms than the Panda
• evaluation on a robot with more than 7 joints: given the absence of common 8 DOFs, we designed a custom Panda, with additional joints, where the agent can control 8 joints. In this setup, we showed that E3J can be applied to robotic arms with more than 1 redundant joint to control.

The results can be found in the Appendix of the revised manuscript.

We hope the additional evaluations address some of the concerns that have been raised. In particular, the studies should show the more general applicability of our approach. Furthermore, regarding your concerns about the benefits of using a constrained action space in tasks that do not require full-body control, the new results show that our approach (E3J) tends to learn faster than EE, even in standard RLBench tasks, which only require grasping and moving objects to a target (such as take lid off saucepan and stack wine).

Dear Reviewer H84E,

we would like to thank you for participating in the rebuttal and providing additional comments on our work.

Choice of joint for E3J and generality of the approach

We understand the idea of "constraining one joint" of the robot may sound less general. However, we believe this is more of a wording issue and how the method is perceived than an actual issue of the generality of the approach.

When using E3J, the outputs of the agent are organized as (x, y, z, yaw, pitch, roll, joints_controlled_separately), which can also be read as (task space coordinates, joints_controlled_separately). This means the agent provides the same control of the standard EE action space, plus control over additional joints. Clearly, this extends the control of the agent over a larger set of joint configurations, as it implies the agent can achieve EE poses, by setting six joints using the IK, and where there would normally be many arm configurations which achieve an EE pose, the additional controlled joints enable the agent to choose one configuration to move to. We suggest looking at Figures 2 and 3 to gain additional visual insights into how our method works.

We empirically verified that our method provides more extensive control in the ablation introduced with the revision (now under the name E3J controlling different joints), where we showed that controlling any of the redundant joints (which are the oddly numbered ones - 1,3,5,7) allows control over the full-body pose of the robot, as shown in the reach elbow pose task, which requires precise control over the robot's configuration to be solved. The main difference in performance between controlling different joints arises from how convenient directly controlling one joint is, compared to the others, and we found controlling the base shoulder joint as empirically being the most versatile option.

The reason why we used the term "constrain" was due to the underlying implementation, which works by setting the controlled joint position directly and computing the remaining joints' position using the IK, with one less constraint given that the joint is directly controlled by the agent. To avoid any further confusion, we have replaced the term "constraining" with the term "controlling", as from the agent's perspective, the joint is directly controlled (see text highlighted in orange in the revised manuscript).

Implementation details

We ensured that all the implementation details are present on the paper as follows:

• "What are the 6 constraints?"

We apologise for the ambiguity about which constraints are being solved by the IK. We further specified that these are the task space coordinates of the end effector as follows: "solving the IK for 6 constraints that are dictated by the task space coordinates (position and orientation of the end effector)".

• There are multiple ways to solve for IK, which ones did the authors use?

This implementation detail is already described in the paper.

For the simulation experiments, we mentioned:

"We rely on the internal IK solver of CoppeliaSim for EE, E3J and E3A, which, for quickly computing the IK in delta action spaces, employs the pseudoinverse Jacobian method."

For the real-world experiments, we mentioned:

"We use MoveIt and PickNikRobitics’ pick ik with default parameters for solving the IK for EE and E3J. For E3J we first simulate the motion created by the joint that is controlled separately and then compute the IK on the remaining joints."

We also added the following information (in orange text), to provide easier access to how Pick IK works (and how we used it) to the reader:

"We followed the standard "pick_ik Kinematics Solver" tutorial from the pick_ik documentation, which references the default parameters from the pick_ik_parameters.yaml file. We only altered two parameters for the IK: changing both approximate_solution_position_threshold and approximate_solution_orientation_threshold from 0.05 to 0.01, to reduce the tolerance on the EE pose."

"pick_ik uses a local optimizer, which solves inverse kinematics via gradient descent, and a global optimizer, based on evolutionary algorithms"

(continues...)

We would like to thank the reviewer for their feedback. We provide an answer to their questions below.

The work can be seen as an engineering trick: Contrary to viewing the 7 DoF as an inherent opportunity for E3, our new action space is a simple strategic response to this overactuation problem, commonly encountered in robot arms with greater than 6 DoF. The E3 action space applies classical robotics control techniques to Robot Learning (see "Redundant manipulators control" part of the "Related Work" section of the original paper), aiming to replace Joint and EE action spaces currently used in the robot learning literature.

While the E3 action space offers a straightforward solution, its simplicity is its strength: it can be seamlessly applied to many existing and future robot learning works. Additionally, it is clear from our findings that this simple change in action space for robot learning dramatically improves performance on tasks where whole-arm control is imperative. As we confront increasingly complex challenges, such as navigating real-world scenes involving cupboards and other intricate, hard-to-navigate areas, it becomes evident that conventional action spaces fall short in providing the necessary control over the arm's configuration whilst maintaining data-efficient learning.

Previous contributions, which also mostly focus on re-engineering the action space, e.g. by adopting primitives [1] or next-best pose [2], has had a significant impact on the field, leading to higher performance, through only minor modifications of the learning algorithm.

Generalization to $\neq$ 7 DoF arms: In the case of a 6 DoF arm, such as the UR robot family, where the "elbow" has two solutions for end-effector (EE) poses (elbow up or elbow down), E3J method would utilize pose information along with a binary flag indicating elbow orientation. This contrasts with the conventional approach in robot control for UR robots, where the elbow joint is typically constrained to either the up or down position.

Additionally, 6 DoF arms face limitations due to their morphology. Unlike overactuated arms, they may struggle with tasks demanding flexibility. For instance, consider our real-world setup with a cabinet except replacing the Franka Emika Panda arm with a UR5. The UR robot's vertical elbow position would hinder solving the task from this position where the elbow needs to be rotated to the horizontal place in order to reach inside the cupboard. Visual example in simulation available at https://imgur.com/a/n4L5UZH : while the cup position would be reachable by the UR5, the presence of the obstacle and the stiffness of the UR5's elbow do prevent the robot from reaching inside the cabinet.

Why the method talks about the elbow? in most 7 DoF arms, the redundancy of the robot allows control over one more dimension, which can be identified in the position of the elbow. As we mention in the Introduction, E3A allows direct control of the elbow by controlling its angle, while E3J allows indirect control of the elbow by constraining one of the joints to fix the elbow position. Thus, both the methods presented allow control over the elbow redundancy (see Figures 2 and 3 from the original submission for further illustration).

Experiments to validate the choice of the joint: in Section 4.4, we motivated our choice for mainly considering E3J-base and E3J-wrist in the first iteration of the manuscript. In order to further validate our choice, we have now completed the ablation study, where we attempt to constrain other joints of the robot for E3J (see Section A4 and Figure 13 in Appendix). The results show that our intuitions generally proved correct, but there are tasks where constraining other joints than the base or the wrist improves sample efficiency. We also believe E3A may constitute a more general strategy than E3J, but in practice we found that the agent struggles to directly control the elbow angle more than indirectly controlling the elbow through one constrained joint. As finding out what's the best action space for robot learning, in terms of full-body control and efficiency, is one of the main goals of this work, we have chosen E3J as the idea that best fits this definition according to empirical evidence.

We hope our revision answers the reviewers' doubts and we look forward to any further suggestions to improve our work.

[1] Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives, Dalal et al

[2] Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation James et al

Dear reviewer,

we would like to thank you for answering to our rebuttal.

We are glad to have addressed some of your concerns. In addition, we would like to report that the following experimental studies have been added to the revised manuscript, to corroborate the general applicability of our method:

• evaluation on different robotic platforms: using the Kuka iiwa 7 and xArm. In this setup, we showed that E3J can be applied in other platforms than the Panda
• evaluation on a robot with more than 7 joints: given the absence of common 8 DOFs, we designed a custom Panda, with additional joints, where the agent can control 8 joints. In this setup, we showed that E3J can be applied to robotic arms with more than 1 redundant joint to control.

The results can be found in the Appendix of the revised manuscript.

We hope the additional evaluations address the concerns that have been raised about evaluating only on a single 7 DOFs arm and extending the work to arms with higher DoFs.

My concerns have been partially addressed, but this work's technical novelty is till not fully convincing to me. I will raise my score to a positive one.

We would like to thank the reviewer for their positive feedback. We answer the remaining questions as follows.

The evaluation does not consider dynamics/contact forces: for all the approaches we tested, we control the robot by setting target joint positions derived from the agent's output action space: either directly with joint space or through Inverse Kinematics (IK) for EE and E3 action spaces. In regard to the observations/inputs, the agent receives as inputs two RGB images, from the environment and wrist cameras, and a set of proprioceptive states. We did not account for contact forces, however in the RL experiments the agent can infer contacts, by visual or proprioceptive feedback (e.g. the end-effector is blocked) or through the rewards (e.g. the agent is rewarded to stay in contact with some object to complete the task). Nonetheless, we understand this would be useful to know for future readers and thus we added some additional comments on this, when discussing the agent's inputs from the environment, in the revised manuscript (see green text in Appendix A.5).

Extension to higher DoFs: arms with more than 7 DoFs are less common than 6 and 7 DoFs arms, and they are often designed for specific applications, which makes them hard to evaluate in common environments. We discussed this choice of focussing mostly on 7 DoF arms in Section 4.2 (highlighted in blue from the original submission) and, in order to provide a more general idea on how to extend our framework, we provided additional comments (in green) in Section 4.4, discussing how E3J could be adapted for higher DoFs.

We hope our revision responds to the reviewer's questions about our work and we look forward to any further suggestions to improve our work further.

We would like to thank the reviewer for their feedback. Please find our responses to the questions below.

Lack of discussion/comparison to other action spaces: the reviewer mentioned primitives and OSC as potential action spaces to compare with E3. We believe that, while both ideas are very relevant to the robot learning setting we work on, they are also orthogonal to our approach. Primitives, as in [1,2], provide a more abstract level for controlling the agent, through temporally consistent behaviors (often parameterized in task space coordinates). OSC controllers provide an alternative to IK solvers for controlling the robot in task space coordinates. In both cases, when working with a 7 DoF robot, additional control over the robot configuration needs to be provided to the learning agent for solving tasks in constrained spaces.

In our work, we discuss and implement two ways (E3J and E3A) to address such limitations, and we test them in RL and IL settings, in a temporally fine-grained setting and using IK solvers. Nonetheless, we agree with the reviewer that testing our approach using OSC controllers, or defining primitives to control the remaining degrees of freedom of the agent, i.e. the elbow of the robot, are relevant ideas to mention for additional evaluations/future work and thus we discuss them in our revised manuscript (highlighted blue content in the Related Work section about primitives and OSC, new green content about OSC controller comparison in Section 5.1/Analysis).

Is the work a special case of Riemannian Motion Policies (RMP; Ratliff et al)? We acknowledge that RMP represents one more way of controlling the elbow of a 7 DoF robot, and thus we added it to the "Redundant manipulators control" paragraph in the related work section, where we discuss other ideas to control the elbow. However, the nature of our work is very different from RMP: we aim to specifically discuss the issue of controlling the full-body pose for 7 DoFs robots, using data-driven learning techniques such as reinforcement and imitation learning. We discussed and empirically validated two implementations, and we acknowledge in the Discussion section that there are many opportunities to extend our study, along which we now also mention RMPs (see Related Work and Conclusion in the updated manuscript).

Testing on other robots than Franka: we understand that an evaluation on multiple robot platforms could be more insightful and we have started experimentation with two other 7 DoFs robots: XArm 7 with XArm gripper and Kuka iiwa 7 with Robotiq gripper, and we aim to update the manuscript with partial results before the end of the rebuttal period. We will post additional comments when the results are ready.

Specify the method is designed for overactuated manipulation scenarios: The initial submission made this statement in the Abstract, Introduction, Methods and Conclusion sections (see parts highlighted in blue in the manuscript). To further make it clear, we specified this additional times along the manuscript (in green, see the mentions to "overactuacted arms").

We hope our revision responds to the reviewer's doubts about our work and we look forward to any further suggestions to improve our work further.

[1] Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives, Dalal et al

[2] Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks, Nasiriany et al