Rethinking Decision Transformer via Hierarchical Reinforcement Learning

Q: Limited Empirical Superiority

A: Our ablation studies reveal some interesting findings that might be able to explain why the overall performance of ADT falls short of IQL and HIQL. Our main conjecture is that Transformer as a function approximator is hard to optimize for standard RL algorithms. To verify this, in Section C in the appendix of the revised paper, we first implement an oracle algorithm, which distill the IQL policy using a transformer with supervised learning objective (V-ADT Oracle in Figure 6). The oracle algorithm matches the performance of IQL, suggesting that the transformer architecture is not the bottleneck. We then implement another baseline, named IQL-Trans , by replacing the MLP policy with a transformer policy for IQL. As shown in Figure 3, the performance of IQL-Trans (V-ADT w/o prompt in Figure 3) cannot match the original IQL, further supporting our conjecture. The advantage of ADT over IQL-Trans is mainly contributed to the joint optimization of hierarchical policies (the high-level policy optimizes the prompt and the the low-level policy is optimized based on the prompt), since this is the key difference between these two algorithms. To help readers clearly get this information, we provide the results of V-ADT, V-ADT Oracle, IQL-Trans and IQL in the table below:

Finally, we note that there is still a performance gap between ADT and the oracle algorithm. This motivates the investigation of other techniques to improve transformer-based decision models, which we leave as our future work.

Q: Insufficiency of Information for Replicability

A: We promise that we will release our codes soon.

Q: why using antmaze for these ablations (not MuJoCo), considering its characteristic sparse rewards and diverse objectives?

A: One of the most important ADT's contribution is giving transformer-based models (DT) the stitching ability, referring to the capability to integrate suboptimal trajectories from the data. The Antmaze datasets consist of suboptimal trajectories, requiring the agent use portions of existing trajectories in order to solve a task, is widely considered to be the benchmark for evaluating the stitching ability [1]. Therefore, we choose to ablate the efficacy of each component of ADT in improving the stitching ability using Antmaze datasets.

Reference

[1] Fu, Justin, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv preprint arXiv:2004.07219 (2020).

Q: does the high level provide prompt at every time step?

A: Yes, we describe this in Figure 1.

Q: V-ADT is arguably a hierarchical setting since there is no clear decomposition of a global goal or task

A: The term "hierarchical" is mainly referred to the policy decomposition $\pi(a|s) = \int_{\rho\in \mathcal{P}} \pi(\rho | s) \pi(a|s,\rho) d \rho$ (Eq 3). The prompt is a general concept, includes but not limits to sub-goals used in HRL. Other choices includes returns-to-go (values), option, and skills. For example, return-conditioned prediction [1] can also be included as a special case of hierarchical policy framework.

Although V-ADT does not decompose a global goal into explicit sub-goals, it operates hierarchically by having the high-level policy generate context-specific prompts (values in this case) which guide the decision-making of the low-level policy. This separation of prompt generation (high-level) and action determination (low-level) based on these prompts aligns with the fundamental concept of hierarchy in decision-making processes.

Q: What is the reward function of the high level?

A: There is no reward function of the high-level in V-ADT. The low-level policy of V-ADT makes return-conditioned predictions as in [1]. The key innovation of V-ADT is the value prompt given by the high-level policy is learned from data.

Q: how could you avoid the issues you stated in Sec. 2?

V-ADT can solve the motivating example. The value prompt used by V-ADT represents the maximum achievable value given the offline data. This value prompt then informs the V-ADT low-level policy to produce an action towards achieving the value.

Given this in mind, let's consider the motivating example. The dataset $D$ contains three trajectories $a → b → c$ leading to a return of 0, $d → b → e$ leading to a return of 10, and another trajectory starting from $a$ and leading to a return of 1. As for this particular example, the value prompt for $a$ is 10. To achieve this value, at state $a$, V-ADT takes a step towards $b$, as the other path would lead a worse trajectory with return 1.

Reference

[1] Emmons, Scott, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. "RvS: What is Essential for Offline RL via Supervised Learning?." In International Conference on Learning Representations. 2021.

Q: Integrating decision transformers into a hierarchical setting has already been studied in an earlier paper

A: Thanks for pointing this out. We have cited and discussed this paper and other DT Enhancements in Discussions section in our revised paper. ADT differs from HDT mainly from two aspects: (1) The high-level policy of ADT is more general that could output different kinds of prompts, including sub-goals, target values, latent skills and options. (2) For the low-level policy, we apply advantage weighted regression instead of conditioned prediction as in DT to enable stitching ability.

Q: why the proposed approach outperforms or underperforms the baseline methods

A: Thanks for your suggestion! Our ablation studies are designed to explain the reasons behind the empirical results. But we totally agree that analysis of these results should be discussed more carefully in the paper.

Our ablation studies reveal some interesting findings that might be able to explain why the overall performance of ADT falls short of IQL and HIQL. Our main conjecture is that Transformer as a function approximator is hard to optimize for standard RL algorithms. To verify this, in Section C in the appendix of the revised paper, we first implement an oracle algorithm, which distill the IQL policy using a transformer with supervised learning objective (V-ADT Oracle in Figure 6). The oracle algorithm matches the performance of IQL, suggesting that the transformer architecture is not the bottleneck. We then implement another baseline, named IQL-Trans , by replacing the MLP policy with a transformer policy for IQL. As shown in Figure 3, the performance of IQL-Trans (V-ADT w/o prompt in Figure 3) cannot match the original IQL, further supporting our conjecture. The advantage of ADT over IQL-Trans is mainly contributed to the joint optimization of hierarchical policies (the high-level policy optimizes the prompt and the the low-level policy is optimized based on the prompt), since this is the key difference between these two algorithms. To help readers clearly get this information, we provide the results of V-ADT, V-ADT Oracle, IQL-Trans and IQL in the table below:

Finally, we note that there is still a performance gap between ADT and the oracle algorithm. This motivates the investigation of other techniques to improve transformer-based decision models, which we leave as our future work.

Q: The experimental results presented in the paper appear to lack significance

A: We respectfully disagree with this. V-ADT outperforms DT by 20.5% on MuJoCo datasets, 227.5% on Antmaze datasets, and 42.5% on Kitchen datasets. G-ADT outperforms WT by 14.2% on Antmaze datasets.

The overall performance of ADT falls short of IQL could be attributed to the transformer architecture of the policy. In fact, it is still an open problem if Transformer is more efficient than standard architectures in single task RL. Previous studies show that MLP is competitive with and sometimes more effective than Transformer in offline RL benchmarks (see Table 1 of RvS[1] for example, where RvS-R outperforms DT on Antmaze datasets in the reward-conditioned behavior cloning setting). One conjecture is that high-capacity modles only make sense when we train on large and diverse datasets [2]. We would like to explore this conjecture in future works by applying ADT for multi-modal and multi-task domains.

Q: Can more HRL environment experiments be shown?

A: Our method is not designed specifically for HRL. Instead, we use hierachical framework to rethink transformer-based decision algorithms, and derive joint policy optimization algorithms from it. Our method is essentially an offline RL method and we follow standard benchmarks to prove the effectiveness of ADT.

Reference

[1] Emmons, Scott, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. "RvS: What is Essential for Offline RL via Supervised Learning?." In International Conference on Learning Representations. 2021.

[2] Chebotar, Y., Vuong, Q., Hausman, K., Xia, F., Lu, Y., Irpan, A., ... & Levine, S. (2023, August). Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions. In 7th Annual Conference on Robot Learning.

I thank the authors for the clarification. I also believe the authors have given good justification for the IQL performance gap in my response and to other reviewer responses. I have chosen to adjust my score to marginal acceptance.

Q: The paper is missing key related works

A: Thanks for pointing out this. We have cited and discussed these two papers and other DT Enhancements in Discussions section in our revised paper

Q: The paper is still focusing on single-task learning.

A: Actually, some of the tasks in our experiments include several sub-tasks. For example, the kitchen datasets are collected based on the 9 degrees of freedom Franka robot. The Franka robot is placed in a kitchen environment containing several common household items: a microwave, a kettle, an overhead light, cabinets, and an oven. The environment is a multi-goal reaching problem in which the robot has to interact with the previously mentioned items in order to reach a desired goal configuration. Detailed descriptions can be found at https://robotics.farama.org/envs/franka_kitchen/franka_kitchen/ and https://github.com/Farama-Foundation/d4rl/wiki/Tasks#frankakitchen. We leave multi-task learning on more benchmarks as our future work.

Q: The methodology is hard to follow, and the design choice is not fully discussed.

A: To help understand our methodology better, we provide more detailed descriptions in Section 3 of the revision. As for the design choices, we ablate the effectiveness of the most important components of ADT including the efficacy of the prompt, the efficacy of the RL loss and the efficacy of tokenization in Section 5.4. If you could please point out the specific design choices that need to be discussed, we would happy to respond to them.

Q: The motivating example in Section 2.2.1 does not quite connect to the proposed methodology and leads to confusion. How the proposed method can help solve the example problem in section 2.2.1 (no trajectory a->b->c)?

A: V-ADT can solve the motivating example. The value prompt used by V-ADT represents the maximum achievable value given the offline data. This value prompt then informs the V-ADT low-level policy to produce an action towards achieving the value.

Given this in mind, let's consider the motivating example. The dataset $D$ contains three trajectories $a → b → c$ leading to a return of 0, $d → b → e$ leading to a return of 10, and another trajectory starting from $a$ and leading to a return of 1. As for this particular example, the value prompt for $a$ is 10. To achieve this value, at state $a$, V-ADT takes a step towards $b$, as the other path would lead a worse trajectory with return 1.

Q: For the hierarchical policy, why condition on the current state is sufficient?

A: We think a memoryless policy is sufficient for the Markov environment. However, we certainly agree that applying an auto-regressive model for the high-level policy as did in HDT is also possible. We would like to explore this design in our future work. Thanks for your suggestions.

Q: Does the method’s performance heavily depend on the learned Q and V function from IQL?

A: Yes, V-ADT requires joint optimization of high and low level policies. The output of the high-level policy, which is produced by IQL, significantly affects the performance of V-ADT.

We want to emphasize that this does not mean that V-ADT is simply reinventing IQL by replacing the policy with a transformer. Our ablation shows that leveraging the prompts from a high-level policy significantly boost the performance of V-ADT on some datasets.

Q: How is the proposed method different from the hierarchical decision transformer?

A: ADT differs from HDT mainly from two aspects: (1) The high-level policy of ADT is more general that could output different kinds of prompts, including sub-goals, target values, latent skills and options. (2) For the low-level policy, we apply advantage weighted regression instead of conditioned prediction as in DT to enable stitching ability.