Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making

Thank you for the insightful comments and positive feedback on our work.

W1. The Method of Computing the Reward (RTG) Embeddings.

To compute the RTG embeddings, we adopted the approach used in Ref. [C.1], which involves using a single linear layer to transform a one-dimensional float RTG input into a vector of a predefined hidden size.

W2. Why the Proposed Method is Effective?

The reason why DC is effective can be understood from an inductive bias perspective. Consider the use of MLPs and convolution layers in the computer vision domain. While MLPs are more general and expressive than convolution layers, the local-interaction inductive bias of a convolution layer makes it particularly efficient for image processing. Similarly, although transformers are more expressive than DC, when solving offline RL problems with Markov properties, injecting this Markovianness inductive bias can significantly enhance learning efficiency and performance. Our DC model successfully incorporates the Markovianness inductive bias by employing convolution filters that feature three local windows, thus making it more effective compared with DT. We think that DC is effective for most offline RL tasks with strong Markov property.

W3. Why ODC is Worse than DC on Some Tasks?

As mentioned in [C.2], the ODT/ODC architecture, which incorporates elements of stochasticity for online adaptation, is more complex than the DT/DC architecture. This added complexity might pose challenges in the Antmaze domain which is characterized by its sparse reward nature, potentially complicating the optimization process and leading to lower performance.

References

[C.1] Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." NeurIPS 2021.

[C.2] Qinqing Zheng, Amy Zhang, and Aditya Grover. "Online Decision Transformer" ICML 2022.

Q3. OOD Generalization on Novel Tasks with Multi-Task Learning.

Thank you for your insightful suggestion regarding the exploration of out-of-distribution (OOD) generalization in novel tasks with multi-task learning. We acknowledge that this is a meaningful direction for future research, but we feel that this is beyond the scope of the current paper. At present, the field of offline multi-task/meta-RL, especially concerning OOD generalization, is still in its early stages of development, as indicated by recent literature [B.1,B.2,B.3]. The problem becomes more difficult when it is combined with return-conditioned approaches. These approaches face significant hurdles in multi-task settings, primarily due to the diverse scales and structures of rewards encountered across various complex tasks.

While our current work with the DC algorithm does not directly address these advanced aspects of OOD generalization in multi-task learning, we believe that the insights and methodologies developed here could be useful for future investigation in this area.

Q4. Visualizing the Attention of the Hybrid DC.

We plotted the attention maps of hybrid DC for Atari Breakout and Assault in Section 3 of our anonymous link https://sites.google.com/view/decision-convformer-iclr2024. Since the attention module is used only at the last layer in hybrid DC, we plotted the map of the final layer's attention module. It is seen that the attention map of the last attention layer is quite spread out in the considered Atari games where strong Markov property does not seem to hold. As discussed in our response to Q2, in hybrid DC, since local associations are captured by the previous convolution layers, considering a global perspective in the last layer seems to further enhance the performance of decision-making on domains with weak Markov property.

References

[B.1] Mitchell et al. "Offline Meta-Reinforcement Learning with Advantage Weighting." ICML 2021.

[B.2] Dorfman et al. "Offline Meta Learning of Exploration." NeurIPS 2021.

[B.3] Xu et al. "Prompting Decision Transformer for Few-Shot Policy Generalization." ICML 2022.

Thank you for your positive comments on the effectiveness and strengths of our work. We are grateful for your valuable suggestions to improve the clarity of our paper.

W1. Parts of Writing That Cause Confusion.

We thank the reviewer for pointing out the potential confusion regarding the use of the terms "MetaFormer" and "token mixer block" in our paper. We understand that these terms, particularly for readers unfamiliar with them, might lead to some misunderstanding.

Although we acknowledge the reviewer's suggestion to remove mentioning the MetaFormer framework, the development of DC in our paper is essentially based on the MetaFormer framework. We choose to keep it. To prevent any confusion, however, we will provide a clear explanation in the early part of the introduction. Specifically, we will emphasize two key points:

1. The term "MetaFormer" is used to refer to a general framework rather than a new variant of the transformer model. We will elaborate on its role and significance in the context of our research to offer a clearer understanding for readers.

2. We will clarify that the "token mixer block" in our paper does not refer to the token mixing concept found in the MLP-Mixer. We will provide a precise definition and contextualize its usage within our proposed framework.

Q1. Further Details of the Motivating Examples.

In our motivating examples, rather than using an attention module, we trained $3K \times 3K$ parameters directly in all layers. Here, $K$ denotes the context length, and the multiplication by 3 accounts for the input modalities: RTG, state, and action. We apply causal masking, similar to the masked attention approach, to prevent referencing future timestep information in predictions. Therefore, the learned parameters in each layer have the shape of a lower-triangular matrix with dimensions of $3K \times 3K$.

Q2. Using Attention First then Conv on the Hybrid Architecture.

Thank you for the insightful comment. We provide additional experimental results obtained by running hybrid DC with attention first, followed by 5 convolution layers. As seen in Table R9 below, this reserved version (Attn-Conv) of hybrid DC is slightly better than DT but the gap is marginal compared with the gap between DT and Conv-Attn hybrid DC. From these results, we can see that initially discerning local associations through the convolution module and then observing the global view via the attention module is appropriate for decision-making.

Table R9. Offline performance results of DT, DC with convolution first then attention, and DC with attention first then convolution.

W2. Regarding different performance improvement gaps for different tasks.

First of all, we want to emphasize the performance improvement by DC over DT is significant in the difficult Atari domain, as seen in Table R4 below.

Table R4. Offline performance results of DT and DC on Atari.

Now consider the MuJoCo domain. The different performance improvement gap between DT and DC for different offline tasks in the MuJoCo domain comes from the dataset composition for different offline tasks. The figures in Section 2 of our anonymous link https://sites.google.com/view/decision-convformer-iclr2024 show the return distributions of the stored trajectories in the offline dataset for the MuJoCo domain.

When the return distribution looks like a single delta function as in HalfCheetah-Medium, there cannot be a large gain of DC over DT. Note that both DC and DT are return-condition behavior cloning (BC) methods. Also, when there exists a strong impulse-like cluster at the right end of the return distribution as in Walker2d-Medium, HalfCheetah-Medium-Replay, HalfCheetah-Medium-Expert, Hopper-Medium-Expert and Walker2d-Medium-Expert, this strong right-end cluster enables both methods to detect the expert behavior and clone the behavior as desired by return-conditioned BC, and hence there is not much room for improvement. Please be aware that even in this case, DC consistently outperforms DT. Thus, the major gain comes when the return distribution is spread out without a stong peak at the right end side in the distribution as in Hopper-Medium, Hopper-Medium-Replay, and Walker2d-Medium-Replay, as shown in the figures in Section 2 of our anonymous link https://sites.google.com/view/decision-convformer-iclr2024. As seen in the figures, in Hopper-Medium, Hopper-Medium-Replay, and Walker2d-Medium-Replay, even though there exist trajectories with higher return, DT did not properly execute higher return behavior cloning. DT's performance is lower than the existing higher return in the dataset. On the other hand, DC successfully executed return-conditioned behavior cloning. DC's performance is almost at the right end of the return distribution even when the distributions is spread out. The relationship among return, state and action is better learned. Figure 5 in our manuscript supports this further. As shown in Figure 5, for DT, nullifying the RTG during inference leads to minimal changes, whereas DC is comparatively more influenced by the RTG value.

Consider the Antmaze task. Unlike the MuJoCo domain, Antmaze is a goal-reaching task with a sparse reward, where the agent receives a reward of 1 for reaching the goal region and 0 otherwise. Therefore, to get a high score, the model has to learn to effectively stitch together two beneficial trajectories to reach the goal position. In this context, the DC model, which excels in learning local associations, enables more accurately distinguishing samples and efficiently stitching them, consequently demonstrating superior performance compared to DT.

In summary, DC consistently shows better performance than DT in all standard offline RL benchmarks, including MuJoCo, Antmaze, and Atari domains, and is particularly effective compared to DT for datasets whose return distribution is spread out without a strong high-end peak. To clarify, we will update the manuscript by merging the results from Section 2 of the anonymous page with a response regarding W2. Thank you for pointing this out. We believe this will facilitate clear reading to other readers.

Thank you for the detailed feedback and valuable suggestions regarding our paper. We have incorporated additional analysis and experimental results as recommended, to enhance the clarity and significance of our study. We believe the revision significantly strengthens the manuscript and better shows its significance.

W1. Origin of DC's Gain.

We wish to direct the reviewer's attention to Appendices G.2 and G.3 of our original manuscript, which investigates DC's gain regarding the context length. Specifically, Appendix G.2 elaborates on how different context lengths $K$ and filter sizes $L$ influence DC, while Appendix G.3 provides the performance of DT with respect to context length $K$. For a clearer understanding, we will note in the main text of our manuscript that the content about the context length and window size of DC and DT is in the appendix.

Here, we briefly summarize the results in Appendices G.2 and G.3. Table R1, which is equivalent to Table 18 in our manuscript, shows the performance across different context length $K$ and filter size $L$ for DC on MuJoCo hopper-medium. Since RTG, state, and action form the input at a particular time step, $L$ is a multiple of 3 for DC. Here, $L=3$ corresponds to the case that only the current time step is considered, and $L=6$ corresponds to the case that the convolution filter covers two time steps. The result in Table R1 shows that having a longer context length yields a slight performance enhancement although the gain is not substantial.

Table R1. Expert-normalized returns averaged across five seeds in hopper-medium for different combinations of $K$ and $L$.

Regarding DT with small $K$ focusing more on local information, as suggested by the reviewer, we already performed experiments and the results are in Table 19 in Appendix G.3 in our manuscript, which is shown again in Table R2 below. As seen in Table R2, the performance of DT deteriorates as the context length $K$ decreases.

Table R2. Performance of DT across context lengths K= 20, 8, and 2.

The results in Tables R1 and R2 can be explained from two perspectives.

1. Input length (historical information): it's important to note that, as indicated in [A.1], a transformer like DT, which operates with a global viewpoint, benefits from extensive historical data. The comprehensive historical context allows the transformer to better understand the policy that influenced past actions, thus improving learning efficiency and potentially optimizing training dynamics.

2. Output length (joint action optimization): Reducing the context length $K$ can be detrimental because it loses the benefit from the joint prediction of multiple sequential actions. Note that the objective function of DT and DC is given by $\mathcal{L}_{\text{DT or DC}} := E\_{\tau \sim D} [ \frac{1}{K} \sum\_{t=1}^{K} ( a_t - \pi\_{\theta}(\hat{R}\_{t-K+1:t}, s\_{t-K+1:t}, a\_{t-K+1:t-1}) )^2 ].$

   So, DT or DC are trained not only to predict the last action $a_t$ but also to predict the previous actions $a_{t-K+1}$ to $a_{t-1}$ in a causal manner. So, if we set $K$ as a small number, say, $K=2$, then the training guidance comes from only two terms. On the other hand, if we set $K$ large, e.g., $K=20$, we can exploit the strong training guidance from 20 terms that are intertwined. It seems that this joint optimization yields benefits in training.

Q1. Hybrid DC on MuJoCo and Antmaze.

Below are the results of hybrid DC on MuJoCo and Antmaze.

Table R5. Offline performance results of DT, DC and DC$^\text{hybrid}$ on Mujoco and Antmaze.

As shown in Table R5, hybrid DC does not outperform DC in these domains but still shows relatively superior performance compared to DT. Therefore, for tasks with strong Markov property, convolution layers alone are sufficient. However, for tasks where strong Markov property does not hold, hybrid DC is a good alternative.

Q2. Computational Complexity of the Hybrid DC.

The complexity of hybrid DC can be seen in Tables R6, R7, and R8.

Table R6. The resource usage for training DT, DC, DC$^\text{hybrid}$ on MuJoCo and Antmaze.

Table R7. The resource usage for training ODT, ODC, ODC$^\text{hybrid}$ on MuJoCo and Antmaze.

Table R8. The resource usage for training DT, DC, DC$^\text{hybrid}$ on Atari.

Please be aware that we used the PyTorch grouped conv1D on our implementation, which is known for its inefficiencies within the PyTorch framework, as discussed in https://github.com/pytorch/pytorch/issues/73764. We expect that the training time could be reduced by either employing a different CUDA kernel or opting for an alternative library like JAX.

References

[A.1] Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." NeurIPS 2021.

[A.2] Beltagy, Iz, Matthew E. Peters, and Arman Cohan. "Longformer: The Long-Document Transformer."