Reinforcement Learning with Action Sequence for Data-Efficient Robot Learning

[Q6] In and around equations 2 and 4, some of the actions are bold, and some (e.g., lines 132, 141) are not. I was wondering what the difference between the bold and non-bold actions is.

[A6] Thank you for pointing this out. As we mentioned in line 829 of Appendix B, non-bold action corresponds to action at each dimension, and bold-action denotes the action vector consisting of actions from all dimensions. We will update this in the final draft.

---

[Q7] What exactly is $\pi_{K}^{l}$ in Equation 4? Is it a learned policy or does it just denote the maximization over the Q-functions?

[A7] It denotes the policy that outputs action via action inference which we introduced in line 138-144 of the original draft. We will clarify this.

---

[Q8] Why is the one-hot vector $e_{k}$ necessary?

[A8] Thank you for a good question. $e_{k}$ is necessary because we’re first extracting features with the MLP network $f_{\theta}^{`MLP`}$ and then using GRU network to aggregate features. Without $e_{k}$, the MLP would not know where this $\mathbf{a}_{t+k-1}^{l-1}$ comes from. While the training signal from GRU would allow for training MLP networks without explicit index, we find that this empirically works better.

---

[Q9] Where is $a^{l}$ added to the inputs?

[A9] We would like to clarify that our network follows the design of value-based RL algorithms such as DQN, therefore the Q-network doesn’t take $a^{l}$ as inputs, but it outputs Q-values for each $a^{l}$.

---

[Q10] Line 201: $m$ is not defined. Is it a hyperparameter?

[A10] Thank you for pointing this out. $m$ is a hyperparameter used for aggregating the actions in the temporal ensemble, which was originally introduced in [2] as a hyperparameter $k$. We did use $m=0.01$ following [1].

[2] Zhao, Tony Z., et al. "Learning fine-grained bimanual manipulation with low-cost hardware." RSS 2023

---

[Q11] Modify notation $a_{t-i}$ in Line 201 to explicitly denote that this is action calculated at timestep $t-i$

[A11] Thank you for your suggestion. We will make it explicit by denoting it as $\bar{a}_{t}^{i}$ and mentioning that it is an action for timestep $t$ which is calculated at timestep $t-i$. Please let us know if you have any better suggestions.

---

[Q12] Lines 203/204 state that computing the actions only every K steps and applying the K-step action sequence is beneficial for tasks that require reactive control. This is a bit puzzling to me since executing the action sequence open-loop should make the policy less reactive since the agent can only react every K steps, while in the temporal ensemble case, the agent can at least change the actions at every step to some extent.

[A12] This is a great question. You’re correct in pointing out that policy with temporal ensemble can be more reactive in that it can compute actions every time step. However, in practice, we find that temporal ensemble makes the policy less reactive because previous actions from $K-1$ steps affect the action at the current time step. In contrast, without a temporal ensemble, the policy can directly execute the most recent actions predicted by the network, without being affected by the predictions from the previous timesteps. We will add more discussion on this point, thank you for pointing this out.

Dear Reviewer u3LC,

We sincerely thank you for your feedback and insightful comments to improve the manuscript. We address your comment below.

---

[W1] It remains unclear whether such a modification would also lead to similar performance gains in other RL algorithms.

[A-W1] Thank you for pointing this out. We have included the results of DrQ-v2-AS+ which trains both critic and actor with action sequences. This significantly outperforms DrQ-v2+ in the BiGym environment, demonstrating that our idea is applicable to other RL algorithms. We will include more results in the final draft.

---

[W2] Training multiple Q-networks for all the levels and action sequence will be slow. Some comparison of the computational efficiency would be nice.

[A-W2] Thank you for the suggestion. In practice, we do not train all Q-networks separately but train a single network that shares most of the parameters by following the design of [1]. Specifically, the network can share most of the parameters by taking level index and step index as inputs. Nonetheless, it makes training slower as you pointed out, i.e., CQN-AS is approximately 33% slower than CQN as we mentioned in ‘Computing hardware’ of Appendix A. We will further clarify this in the final draft.

[1] Seyde, Tim, et al. "Solving continuous control via q-learning." ICLR 2023

---

[Q1] Is Q-function getting a learning signal for all actions within the action sequence? Wouldn’t Q-function ignore all actions other than $a_{t}$?

[A1] We’d like to first clarify that our Q-network does not take $a_{t:t+K}$ as inputs, but it outputs a $K$-dimensional vector for all $K$ actions within the action sequence, i.e., $(Q(o_t, a_t), Q(o_t, a_{t+1}), \ldots, Q(o_t, a_{t+K-1}))$. Therefore Q-function does not ignore other actions. More specifically, because we explicitly train $Q(o_{t}, a_{t+k})$ to predict $r\_{t+1} + \gamma Q(o\_{t+1}, a’\_{t+k})$, where $a’\_{t+k}$ is the action that maximizes Q-values of action index $k$, for all $k$, each Q-network can learn the effect of outputting action $a\_{t+k}$ at the current timestep. We will further clarify this in the final draft.

---

[Q2] Additional ablation on CQN-AS that computes actions at each timestep and uses the first action

A2. Thank you for your suggestion. We have included the results of CQN-AS (First Action) that computes actions at every timestep but always uses the first action in the sequence. We find that only using the first action performs significantly worse than both CQN-AS and CQN-AS (No Temporal Ensemble). This shows that a mismatch between the training time (train a model to learn Q-values for a sequence of actions) and the test time (only use the first action from the output action sequence) is harmful for the performance. We will include the relevant results in the final draft.

---

[Q3] In some of the plots the methods are not trained for enough steps. E.g., in the Run plot of Figure 5, TD-MPC2 is trained for only 2e6 steps and at this point it seems to be on-par with CQN-AS, but it is unclear whether it will perform better or worse in the long run.

[A3] We’d like to emphasize that comparison against model-based RL methods is not our main focus. For this reason, we used the numbers in the official HumanoidBench repository without running model-based RL methods by ourselves. But we agree that it will be informative for follow-up research, so we will add the results in the final draft.

---

[Q4] Missing standard deviations in Figure 1. Increasing the number of runs from 4.

[A4] Thank you for pointing this out. We have included the aggregate plot that includes the confidence interval. For increasing the number of runs, We could not include results with more runs because it is very costly to run experiments on 53 tasks considered in our experiments. We will increase the number of runs in the final draft.

---

[Q5] How does the Q-network know which bin was chosen at the coarsest level?

[A5] CQN uses the numerical values corresponding to the center of each bin as inputs to the next level’s Q-network, so the network has the information required to understand where it is located within the interval. But it would be interesting to investigate if giving previous actions from all the levels as inputs make any difference. We will clarify this in the final draft.

[A-W1]: Thank you for these additional experiments. These first results look very promising.

[A-W2]: I see, so the Q-network essentially looks like $Q(h_t, l, k) = q_{t+k}^l$ where $q_{t+k}^l$ is a DQN-style vector of Q-values for all actions on the level $l$ for the timestep $t+k$, predicted at timestep $t$? Please consider adding to the manuscript that the Q-functions are parameterized as a single network with the index input and that the network outputs vectors of Q-values. In my opinion, this is not clear from the manuscript.

[A1]: I still do not understand this part. $a_{t+k}$ is the action that is (potentially) executed at $k$ steps into the future. It is updated with the reward for the current step $r_{t+1}$, but $a_{t+k}$ should not have an influence on $r_{t+1}$ (or $h_{t+1}$), right? So, couldn't $Q^k$ just predict the value $Q(s_t, a_t)$ for all outputs, i.e., the regular one-step Q-value, ignoring $a_{t+k}$? I think that should be a valid solution to equation 4.

I feel like the loss should be something along the lines of $(Q^k(h_t, a_{t+k-1}^l, a_{t:t+K}^{l-1}) - \sum_{\tau=t}^{t+k} r_\tau - max_{a'} Q^k(h_{t+k}, a', \pi_K^l(h_{t+k})))^2$. Could you please expand on how exactly the loss from equation 4 works?

[A2]: Thank you for this additional ablation. I am a bit puzzled that this setting does not work at all. I see that there is a mismatch between training on sequences of actions and then testing with single actions, but still I feel that the first action of the sequence should still do something at least somewhat meaningful. Could you expand on what you think are the reasons why this setting does not learn anything meaningful at all for these tasks?

[A3]: Yes, for the final version, it would be very interesting to see the comparison between CQN-AS and the model-based baselines when trained for the same number of steps.

[A4]: Thank you for adding the confidence intervals. Regarding the number of seeds, I can see that computation capacities can be a limiting factor here.

[A5]: I see, so the $a_t^{l-1}$ passed to Q-function $Q^l$ is not a representation of "bin x on level l", but rather the mid-point of the bin chosen on level l, represented in the original action space?

[A6]: This is a bit confusing since the footnote on page 3 states that all actions are assumed to be one-dimensional in the main paper.

[A7]: Thanks for the clarification.

[A8]: I see, so the network could learn to count, but it is easier to simply add the index as an additional input. Please consider adding the explanation to the paper as it could make this part a bit clearer.

[A9]: Thanks for the clarification. As mentioned in [A-W1], I think it would be beneficial for the sake of clarity to mention this in the paper.

[A10]: Thank you for clarifying this.

[A11]: Yes, that would make this clearer.

[A12]: Okay, so let's say K=5, and something that the policy needs to react to happens at timestep 3, this would mean that the "no temporal ensemble" policy would get full control over the actions at timestep 5, while the "temporal ensemble" policy would still be influenced by the previous actions at that point. Only at timestep 8 would this policy be free of the influence of the previous actions. Is this intuition correct? If I understand correctly, the hyperparameter m essentially controls how reactive the "temporal ensemble" policy is. Did you try different values for m to see if that makes the approach more suited for tasks that require reactive control?

[A13]: Yes, changing the statement in that way would make it more accurate.

[A14]: I see, that makes sense. Thanks for the explanation.

[A15]: My main concern here was that you used the "Run" task as an example where CQN-AS achieves competitive performance to the model-based RL baselines, which does not seem to be the case according to Figure 5. So please do not use this task as an example here. I am happy with the updated statement.

Dear Reviewer xAUB,

We sincerely thank you for feedback and insightful comments to improve the draft. We address your comment below.

---

[Q1-2] Intuition on why learning Q-network with action sequence is effective on domains with noisy trajectories. Is the proposed technique generic to other RL algorithms?

[A1-2] We have included the results of DrQ-v2-AS+, which trains both critic and actor with action sequences. This significantly outperforms DrQ-v2+ in the BiGym environment, which shows that our idea of learning RL agents with action sequence is applicable to other RL algorithms. We will include more results in the final draft.

To provide a more intuition on why our idea is helpful, we would like to illustrate an example where a human collects a demonstration of a robot moving forward. Due to noise in the data collection system or human error, the robot may jitter, moving left-forward or right-forward at each timestep, but ultimately proceeds straight. In this case, prior approaches train a Q-network to learn the effect of executing each action $a_{t}$ at the current timestep, but it could be challenging because each action is noisy in a sense that it is not going forward. On the other hand, our approach explicitly trains the Q-value to learn the effect of a series of actions that ultimately move forward, which makes it easier for the Q-network to understand the effect of executing actions. We will include a relevant discussion on the final draft.

---

[Q3] The paper is less self-contained. The paper suddenly uses the term "level", which is never properly introduced in this paper. Therefore, I believe the current manuscript forces readers to visit the previous CQN paper, which is not desirable.

[A3] We agree that some of the definitions are missing and Figure 2 is not fully self-contained. While we provided a detailed background of CQN in the original draft’s Section 2 in an effort to make the paper as self-contained as possible, we will further complement our Background section to make the paper more self-contained. Thank you for pointing this out.

---

[Q4] The paper uses both RL and BC as problems, and I am not so sure it is a good idea. The paper may focus on one of the problems to consolidate the scenario.

[A4] We believe that empirical evaluation on diverse setups with or without demonstrations is one of the main strengths of our paper, as the other two reviewers also highlighted in their reviews. Considering the recent advances in data collection systems and interest in using expert demonstration data for robot learning [1,2,3], we hope that our work can help facilitate future RL research to focus on both setups.

[1] Zhao, Tony Z., et al. "Learning fine-grained bimanual manipulation with low-cost hardware." RSS 2023
[2] Fu, Zipeng, et al. "HumanPlus: Humanoid Shadowing and Imitation from Humans." CoRL 2024
[3] Nakamoto, Mitsuhiko, et al. "Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance." CoRL 2024

Dear Reviewer 2SBg,

We sincerely thank you for your feedback and insightful comments to improve the manuscript. We address your comment below.

---

[Q1] Although the approach aims to handle noise effectively, an ablation study on this can strenghten the claim. For instance, is there a mechanism to manage extreme cases of noisy trajectories, or is this approach limited in such scenarios?

[A1] Thank you for the suggestion. We believe experiments on HumanoidBench represent an extreme case for training RL agents with noisy trajectories, as the initial exploratory data consist of trajectories where humanoid robots exhibit unrealistic behaviors with jerky motions. Thus, we think that improvements on HumanoidBench demonstrate that our algorithm can effectively handle noisy trajectories. We would greatly appreciate it if the reviewer could suggest other scenarios; we would be happy to test the suggested setups.

---

[Q2] Incorporating sequential dependency between Q-values.

[A2] We agree with the reviewer that considering sequential dependencies, as in prior work [1], can be effective for more complex robotic tasks, as we also stated in line 182 of the original draft. However, it would introduce additional complexities requiring many techniques and details to make it work, which is why we leave it as a future work. We will include more discussion on it in the final draft.

[1] Chebotar, Yevgen, et al. "Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions." CoRL 2023

---

[Q3] Intuition on why action sequence works well, especially with coarse-to-fine structure

[A3] As we mentioned in the paper, we expect learning with action sequences makes it easy to learn from data with noisy or multi-modal distributions. However, thinking of the effect of coarse-to-fine structure is really an interesting question. We believe coarse-to-fine structure can be synergistic with learning with action sequence, because coarse-to-fine structure enables the model to quickly learn roughly good actions for all actions in the action sequence, and then to focus on refining actions. This may ease the difficulty of learning Q-networks with action sequences. Thank you for an interesting question, and we will include a relevant discussion in the final draft.

---

[Q4] If the goal is to apply this approach to real-world robotics, the paper can discuss transferability, as these are often major hurdles in robotics. Without this, it’s unclear how well CQN-AS would generalize outside simulated environments.

[A4] Thank you for pointing this out. We expect CQN-AS will also work well in real-world setups, as the backbone algorithm CQN already demonstrated that it can train robots to pick up objects within 10 minutes of real-world training. Moreover, considering that recent behavior cloning methods that incorporate action sequences have shown to be very effective for real-world robot control [2][3], we expect that CQN-AS will be also effective for controlling real robots. We will include relevant discussion in the final draft.

[2] Zhao, Tony Z., et al. "Learning fine-grained bimanual manipulation with low-cost hardware." RSS 2023
[3] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." The International Journal of Robotics Research (2023): 02783649241273668.

---

[Q5] Discussion on the difference to multi-step RL methods

[A5] Thank you for the suggestion. We would like to first note that both approaches are orthogonal, so that we can incorporate multi-step RL objectives into our algorithm, as you also stated in your question. Nonetheless, the main difference of our idea to multi-step RL is that we explicitly learn the values of multiple actions inside action sequences, but multi-step RL still learns a Q-value for each action but using the rewards from multiple time steps. It would be an interesting direction to investigate the performance of the combined approach. For instance, we can consider computing the target Q-values for future actions by using the idea of multi-step RL approaches and then training Q-network with action sequence to predict such target values. We will include relevant discussion in the final draft.

---

[Q6] HumanoidBench Results: The results for TD-MPC2 (Figure 5) appear to be cut off too early in the HumanoidBench experiments ? Is there a reason for this?

[A6] We would like to clarify that it is not our main focus to compare CQN-AS with model-based RL approaches, and thus we used the numbers provided in the official HumanoidBench repository without running model-based RL methods by ourselves. We will further clarify this in the final draft.