D2T2: Decision Transformer with Temporal Difference via Steering Guidance

Weakness 1: Lack of Clarity in Methodology Explanation

The methodology, while innovative, suffers from a lack of clear exposition in its presentation, particularly in Section 3.1. The narrative structure and details within this section do not effectively convey the comprehensive process, resulting in certain ambiguities that could hinder readers' comprehension. A more organized and sequential presentation of the methodology would enhance understanding significantly. Ideally, the process should be delineated into four distinct stages:

1. Pretraining the Value Function (V),
2. Identifying and Labeling the Guided State (G_t) across all states,
3. Training the Decision Transformer (DT) utilizing the labeled trajectories, and
4. Developing the Steering Guidance (SG) predictor $g_{\zeta}$, intended exclusively for the testing phase.

The current discourse in Section 3.1 disproportionately emphasizes steps 2 and 4, the step 3 is introduced in Section 3.2 The explanation of the value pretraining stage (step 1) is relegated to the latter part of Section 3.1 and lacks detailed insight into the pretraining mechanics. The term "optimal value" is used, leading to potential confusion over whether this refers to an "in-sample" optimal value [1] or something else (for example in the Medium dataset). A more explicit, step-by-step breakdown of the training phases, especially the pretraining of the value function, is necessary for readers to fully grasp the proposed method's mechanics and rationale.

Weakness 2: Challenges in Reproducibility

One aspect that could enhance the credibility and impact of this work is improving its reproducibility. While the results presented, especially in the Medium and Med-Replay environments, are indeed impressive, the absence of a comprehensive process description somewhat hinders the independent replication of these findings.

My attempt to delve into the specifics, particularly regarding the pretraining of the value function—presumably through Implicit Quantile Learning (IQL)—for continuous state scenarios, led me to the source code provided by the authors. However, it appears the available code primarily pertains to tabular examples and does not extend to the MuJoCo tasks showcased in Table 3. This gap not only poses a challenge in understanding the specific methodologies applied but also in verifying the remarkable results indicated.

The findings in Table 3 are pivotal to the paper's contribution, and if they are as robust as presented, they deserve to be shared in a manner that the broader research community can thoroughly evaluate and build upon. To this end, I believe the inclusivity of the corresponding source code for the results in Table 3 is imperative. Providing this would significantly strengthen the paper's standing, and I am inclined to view this work more favorably should the authors facilitate a more transparent and accessible approach to reproducing these results.

Weakness 3: Inadequate Recognition of Previous Work

This paper's novelty is evident in its application of guidance techniques to the Decision Transformer (DT); however, it misses adequately acknowledging similar foundational ideas in prior studies, for example [2]. There, the concept of directing offline policy via a learned value function—central to this paper's hypothesis—was also prominently featured.

While the application to DT is innovative, the authors should explicitly recognize the contributions of earlier research to maintain scholarly thoroughness. A concise discussion highlighting how this study both aligns with and diverges would contextualize its unique contributions and strengthen the paper.

[1] Fujimoto et al., Addressing Function Approximation Error in Actor-Critic Methods, ICML 2018.

[2] Xu et al., A Policy-Guided Imitation Approach for Offline Reinforcement Learning, NeurIPS 2022

1. The rationale behind the proposed method appears flawed. On page 3, the author states, "the success of DT relies on the successful estimation of $R_t^*$" and posits that the generation of the optimal action hinges on this estimation. However, doesn't the generation of actions depend on both $R_t$ and $S_t$? If I recall correctly, the original DT paper suggests that actions are derived from the input token $S_t$, reinforcing my conviction that the claim is incomplete.

2. The evaluation results seem inadequate. A more comprehensive assessment would involve examining the generated $\hat{G}$ in a few toy examples and analyzing its influence on action generation in comparison to the default return-to-go $R_t$.

3. The paper requires significant improvements in writing clarity and completeness. Essential background information is omitted; for instance, the paper fails to provide context on steering guidance signals, a crucial concept in the proposed method.

Please also consult the question section for additional concerns.

Some points below:

1. Clarity and flow: There is major room for improvement in the flow of paper. A lot of back and forth has to be done to properly understand the work. E.g. the analysis supporting the claim that DTs do not perform better in stochastic environments ( a central claim to paper) does not have a proper emphasis. It is briefly mentioned in section1, 2.3 and explained in Tailgate driving part of experiments. I think it should be separately emphasized somewhere in the beginning.
2. Coming to point 1, the claim needs to be substantiated by experiments on more tasks. Only Tailgate diving task is used to validate this claim.
3. The proof presented for Proposition 1 primarily re-iterates that the DT in question is well trained and will output the optimal action. Not sure if it contributes new insights or novel perspectives to the existing body of knowledge
4. Typo: Is it $\rho^\pi$ or $\rho_\pi$ in section 2.1 (both versions are used). The output should be $\hat{G_t}$ instead of ${G_t}$ in Fig 1 part (1)?