Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning

Dear Reviewer yZtr,

We really appreciate your time and effort for reviewing our paper. We provide our responses below.

---

Q1. Empirical evidence that shows why using action sequences is helpful.

A1. In Introduction, we hypothesized that using action sequences can be helpful for training RL agents by making it easier to learn value functions. Our initial observation showed that using action sequences is helpful for predicting return-to-go from robotic demonstrations. We also showed that CQN-AS, which uses action sequences, achieves strong performance, directly supporting our hypothesis. Here, the key insight is that CQN-AS is a critic-only algorithm where actions are selected greedily based on the learned Q-values. The strong performance of CQN-AS is therefore a direct consequence of the critic learning a more accurate value function. We will further clarify this in the final version.

---

Q2. DQN also suffers from over-estimation bias. Further study on the effect of action sequences on over-estimation in RL algorithms could be a useful contribution

**A2. ** As you mentioned, RL algorithms with discrete actions also suffer from over-estimation bias as well. However, we would like to clarify that our focus in this work is on the over-estimation problem that happens when using high-dimensional actions as inputs to the critic network.

In particular, what we find in this paper is that a combination of (i) a critic network that takes high-dimensional action sequences as inputs and (ii) the actor that is trained to output actions using gradients from the critic leads to severe value overestimation problems. We hypothesize this is because the critic network tends to have more function approximation errors from high-dimensional actions and the actor can easily exploit this error. On the other hand, critic-only algorithms can avoid this problem, leading to stable training with high-dimensional action sequences.

To further support our point, we conducted additional toy experiments where we introduced redundant no-op actions for training TD3 and CQN agents on DMC Cheetah Run tasks. Specifically, we trained agents with 6 original actions and 294 no-op actions with [-1, 1] action bounds and used an environment wrapper that slices out no-op actions. In this setup, we found that CQN (critic-only algorithm with discrete actions) is indeed robust to the overestimation problem. In particular, we find that TD3 suffers from severe value overestimation with more no-op actions, while CQN with such no-op actions achieves similar performance with the original CQN agent. This clearly shows that a critic-only algorithm is more robust to training with high-dimensional action spaces.

Based on this observation, one potential alternative is to use expectile regression for Q-learning as in IQL [1], which can avoid value overestimation problems by avoiding the explicit maximization step in TD-learning. We will leave this as a future work and include relevant discussion in the final version.

[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." International Conference on Learning Representations 2022

---

Q3. It is unclear why the properties of demonstrations should make such a difference.

A3. This is because our setup is a very challenging sparse-reward task setup, which makes it extremely difficult to solve the tasks unless we use demonstrations for guiding the training. This necessitates RL algorithms to effectively learn useful value functions from demonstrations and then use information from online trial-and-error experiences to further improve the performance. Therefore the properties of demonstrations become quite important -- if the characteristics of demonstrations (such as noisyness of trajectories) pose challenges in learning value functions from them, RL algorithms would suffer to learn initial meaningful behaviors and thus learn from online data as well in our setup.

---

Q4. Comparison between human-collected and synthetic demonstrations needs more controlled experiments.

A4. Thank you for your suggestion. To support our claim that value learning is easier with smoother and clean trajectories, we conducted preliminary experiments that train a return-to-go prediction model with smoothed trajectories from BiGym’s Move Plate task. In particular, we apply Gaussian filtering to demonstrations with different smoothing magnitudes, and train the model to predict return-to-go conditional on observation-action pairs. While these smoothed trajectories are not actually usable for policy learning as they have lost their precision, we conducted this experiment as a preliminary experiment for understanding the effect of demonstration characteristics on value learning. As shown in the below table, we find that the validation loss decreases with more smoothing, which shows that demonstration qualities can affect value learning, which thus affects RL training as we mentioned in A3.

---

Q5. The title is inadequate as there is no robot in this work.

A5. We respectfully disagree with the assessment of the title. Our use of simulation is in line with the standard and widely accepted practice in robot learning research, where algorithms are developed and validated for robotic agents.

---

Q6. Why do performances start from 0 despite pre-training with expert demonstrations?

A6. We would like to clarify that there is no pre-training. We just initialize the training by initializing the replay buffer with expert demonstrations (and maintaining a separate demonstration replay buffer as well). Therefore all the methods’ performance starts from 0. We agree that this is confusing and we missed the explanation on this. We will include sentences to clarify our setup.

---

Q7. Did all baselines use the same 17 to 60 expert demonstrations?

A7. Yes. All methods use the same number of demonstrations. As we’ve written in Appendix C, we used different number of demonstrations for different tasks, but we used the same number of demonstrations for all algorithms within each task. We will clarify this in the final version.

---

Q8. Differences between CQN-AS without RL and ACT?

A8. As we’ve written in line 163-165, ACT is an Action Chunking Transformer [3] that trains a transformer model to output action sequences. On the other hand, CQN-AS without RL shares the same architecture with CQN-AS and also the discrete action space as well, which allows us to ablate the effect of RL objective.

[2] Zhao, Tony Z., et al. "Learning fine-grained bimanual manipulation with low-cost hardware." Robotics: Science and Systems 2023

---

Q9. Inconsistency with regard to N-step return

A9. This is a good point. We wanted to emphasize that there is no need to set $N$ to be equal to the length of the action sequence ($K$), as that will train RL algorithms with very large $N$-step returns leading to unstable training. Our experiments find that the $N$-step of 1 or 4 (3 was a typo, we will fix this) works better than very large $N$ such as 16. But it looks like our intent was not clear. We will further clarify this and will move it to Appendix per your suggestion.

---

Q10. An empirical comparison to (Saanum et al. 2024) would improve the paper.

A10. Thank you for the suggestion. While we agree that such comparison would improve our paper, but it is quite non-trivial to compare our work to [Saanum et al., 2024] considering that (i) it has never been tested in our main setup that uses sparse-reward tasks and (ii) its implementation is not open-sourced.

---

Minor - Q1. Using Rliable library, adding references to balanced sampling, combining Figure 7(d) and (e)

Minor - A1. Thank you for your suggestions. We will incorporate them.

---

Minor - Q2. Performance with longer action sequences?

Minor - A2. We find that using longer action sequences ($K > 16$) does not improve performance and sometimes degrade performance. We hypothesize this is because training with too long action sequences can be difficult, offsetting the benefit of learning with action sequences. We also note that the best action sequence length can be different for each dataset that has different action frequencies. We will include the relevant results and discussion in the final version.

---

Minor - Q3. Remarks on fragile humanoid hardware

Minor - A3. We will incorporate your point that not all robots are fragile.

---

Minor - Q4. Computational cost of CQN-AS compared to CQN is missing

Minor - A4. As we’ve written in Appendix C, training of CQN-AS is around 33% slower than CQN. At inference time, CQN-AS takes 13% more memory compared to CQN and is 40% slower than CQN. We believe these costs are definitely worthy considering the effectiveness of CQN-AS over CQN as shown in our BiGym experiments. Moreover, to further reduce the cost at inference time, we can consider adjusting the frequency of the temporal ensemble instead of computing action sequences at every time step. We will include the relevant discussion in the final version.

Dear Reviewer YKFp,

We really appreciate your time and effort for reviewing our paper. We provide our responses below.

---

Q1: Some tasks show only marginal gains over baselines. Can the authors provide more detailed per-task results or analysis to clarify where CQN-AS is most effective?

A1. We find that CQN-AS tends to be more effective on relatively long-horizon tasks compared to baselines. For instance, results on RLBench’s long-horizon tasks like Open Oven or Take Plate Off Colored Dish Rack support this. On the other hand, we find that CQN-AS can be weaker than baselines on very dynamic tasks such as reach target due to the use of temporal ensemble. Finally, we find that CQN-AS does not exhibit stronger performance on tasks like on BiGym’s flip cutlery task, where we hypothesize all methods fail due to challenges in perception. We will include more detailed discussion in the final version. Thank you for your suggestion!

---

Q2: Clarification on “17 to 60 demonstrations” per task

A2. As we’ve written in Appendix C, we used different number of demonstrations for different tasks, but we used the same number of demonstrations for all algorithms within each task. This is because not all human demonstrations in BiGym are not successfully replayed in simulation, and therefore we used demonstrations that we were able to replay in simulations, where the available number of demonstrations was different for each task. Specific number of demonstrations used for each task is available in Table 1, Appendix C. We will clarify this in the caption.

---

Q3: Is there an upper bound or threshold beyond which increasing the sequence length causes performance to degrade?

A3. We find that using longer action sequences ($K > 16$) does not improve performance and sometimes degrade performance. We hypothesize this is because training with too long action sequences can be difficult, offsetting the benefit of learning with action sequences. We also note that the best action sequence length can be different for each dataset that has different action frequencies. We will include the relevant results and discussion in the final version.

---

Q4: Comparison to sequence models

A4. While using sequence models like Transformer for RL has a potential to achieve better performance on long-horizon tasks (as shown in offline RL papers like [1,2], we find that GRU is faster to train and leads to stronger performance in our considered setups. We hypothesize this is because it is non-trivial to train Transformer models from scratch using unstable training signals from td learning, which leaves a room for interesting future work.

[1] Chebotar, Yevgen, et al. "Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions." Conference on Robot Learning. PMLR, 2023.

[2] Springenberg, Jost Tobias, et al. "Offline actor-critic reinforcement learning scales to large models." arXiv preprint arXiv:2402.05546 (2024).

Dear Reviewer 5gur,

We really appreciate your time and effort for reviewing our paper. We provide our responses below.

---

Q1: Limited Novelty of CQN-AS.

A1. While the architectural novelty of CQN-AS may be limited, we would like to emphasize our paper has significant contributions: we provide interesting observations from training popular actor-critic algorithms with action sequences, build a new algorithm based on such observations, and provide experimental results and analysis that could be useful for future research.

---

Q2: The paper could be stronger with some potential experiments showing how some of the potential extensions can make CQN-AS perform even better.

A2. As we noted in Section 4.1, CQN-AS struggles to achieve meaningful success rate on some of the long-horizon tasks that require interaction with delicate objects such as cups or cutlery). We believe this can be resolved by incorporating a better perception stack, e.g., incorporating a stronger vision encoder and experimenting with higher-resolution visual observations. While we tried to run experiments per your suggestion, i.e., increasing the image resolution and incorporating ResNet-18, we found that it requires a GPU with at least 48GB memory and is extremely slow to train. We will leave this direction of incorporating larger vision encoders in an efficient manner as a future direction.

---

Q3: Are TD3-AS and SAC-AS really suffering from value overestimation in Figure 2(b)? Can you provide more details of SAC and TD3 baselines in Figure 2(b)?

A3. We would like to clarify that TD3 and SAC with action sequences indeed suffer from severe value overestimation, as shown in the follow table:

Nonetheless, we agree that the figure can be confusing to readers. We will make the figure clearer to understand. For details on SAC and TD3, we used the standard implementation and hyperparameters for training SAC and TD3 for Figure 2(b), for instance, we used the hyperparameters from HumanoidBench official repository for training SAC. For TD3, we used standard deviation of 0.2 for exploration. We will include these details in the final version as well.

---

Q4: Did you use a distributional critic for both CQN and CQN-AS? What’s the effect of the distributional critic on RL with action sequence?

A4: We used C51 for both CQN and CQN-AS. To answer your question, we run additional experiments that don't use C51 for CQN-AS on BiGym’s Saucepan To Hob task, and find that both CQN-AS with distributional critic (86 $\pm$ 19%) and CQN-AS without distributional critic (82.6 $\pm$ 16%) achieve similar performance. This result shows that using a critic-only algorithm with discrete actions is indeed more crucial in enabling RL with action sequences compared to the use of a distributional critic.

Dear Reviewer H7zc,

We really appreciate your time and effort for reviewing our paper. We provide our responses below.

---

Q1: CQN-AS for offline RL?

A1. We find positive results in our preliminary experiments. We combined CQN-AS with Cal-QL [1] and trained it on the dataset that consists of 26 successful demonstrations and 10 failed episodes. We find that CQN-AS + Cal-QL achieves 33 ($\pm$ 6.8) % while CQN + Cal-QL achieves 7 ($\pm$ 14) %, which shows CQN-AS can be indeed effective for offline setup.

[1] Nakamoto, Mitsuhiko, et al. "Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning." Advances in Neural Information Processing Systems 36 (2023): 62244-62269.

---

Q2: Would GRU -> Transformer improve performance on long-horizon tasks?

A2. This is an interesting question. While using Transformer for RL has a potential to achieve better performance on long-horizon tasks (as shown in offline RL papers like [2,3]), we find that GRU is faster to train and leads to stronger performance in our considered setups. We hypothesize this is because it is non-trivial to train Transformer models from scratch using unstable training signals from temporal difference learning, which leaves a room for interesting future work.

[2] Chebotar, Yevgen, et al. "Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions." Conference on Robot Learning. PMLR, 2023.

[3] Springenberg, Jost Tobias, et al. "Offline actor-critic reinforcement learning scales to large models." arXiv preprint arXiv:2402.05546 (2024).

---

Q3: How does Q-learning with discrete actions avoid severe value overestimation?

A3. As you mentioned, Q-learning with discrete actions may still suffer from overestimation due to max operation during training. However, in this paper, we find that a combination of (i) a critic network that takes high-dimensional action sequences as inputs and (ii) the actor that is trained to output actions using gradients from the critic leads to severe value overestimation problems. We hypothesize this is because the critic network tends to have more function approximation errors from high-dimensional actions and the actor can easily exploit this error. On the other hand, critic-only algorithms can avoid this problem, leading to stable training with action sequences.

To further support our point, we conducted additional toy experiments where we introduced redundant no-op actions for training TD3 and CQN agents on DMC Cheetah Run tasks. Specifically, we trained agents with 6 original actions and 294 no-op actions with [-1, 1] action bounds and used an environment wrapper that slices out no-op actions. In this setup, we found that CQN (critic-only algorithm with discrete actions) is indeed robust to the overestimation problem. In particular, we find that TD3 suffers from severe value overestimation with more no-op actions, while CQN with such no-op actions achieves similar performance with the original CQN agent. This clearly shows that a critic-only algorithm is more robust to training with high-dimensional action spaces.

Based on this observation, one potential alternative is to use expectile regression for Q-learning as in IQL [4], which can avoid value overestimation problems by avoiding the explicit maximization step in TD-learning. We will leave this as a future work and include relevant discussion in the final version.

[4] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." International Conference on Learning Representations 2022