Value-Guided Decision Transformer: A Unified Reinforcement Learning Framework for Online and Offline Settings

Thank you very much for your careful review of our work. We'll answer your questions one by one in the following, including some misunderstandings and some essential academic questions worth exploring.

W1: The paper does not seem to maintain a standalone dynamic model to predict future states. In this case, when you "predict" next state from "EnvModel" (Alg. 3), you are generating a sequence of $E$-step rollout for each of the candidate RTGs, and always cancel your actions afterwards until the optimal action is selected.
We must acknowledge that the approach of sampling parallel trajectories is difficult to implement in scenarios where future states are completely unpredictable. In such cases, we can only rely on a "single-shot" sampling strategy, i.e., single-step sampling with Evaluation Horizon = 1. As shown in the table below, we find that VDT still maintains a advantage over other methods, which is intuitive: the advantage-weighting and penalty mechanisms provided by the value function can further enhance the model's decision-making ability, even when multi-step parallel sampling is not possible.

On the other hand, although most tasks have or can be trained with a simulation environment, it is indeed important to consider our sampling methods in extreme scenarios where no explicit simulator is available. One intuitive approach is to directly add an environment prediction head to VDT, enabling the model to perform parallel sampling during inference autonomously. Another approach is to train a world model to provide future states for VDT, which has been explored in many previous works and is relatively feasible [1][2]. In general, training a dynamic model to predict states and rewards is often slightly easier than training a decision model. If we do proceed in this way, it could be argued that, to some extent, we are essentially equipping the current task with a simulation environment based on the existing data. Due to the limited time for rebuttal, we regret that we are unable to explore these two designs now. We will include experiments and discussions on both methods in the appendix of the revised version.

[1] Ha D, Schmidhuber J. Recurrent world models facilitate policy evolution. NeurIPS, 2018.
[2] Hafner D, Pasukonis J, Ba J, et al. Mastering diverse control tasks through world models. Nature, 2025.

W2: The implementation of "alignment of RTGs" is not explained. More specifically, how is the RTG token in the last timestep "matching" the reward obtained by the agent? Ideally the RTG in the last step should be 0, which should not match the reward obtained by the agent.
In line 5 of Algorithm 2 in Appendix C, we illustrate the alignment of RTGs, where the RTG at time step $t$ for the current trajectory is calculated as: $RTG_t = \sum_{j=t}^{|\tau|} r_j$, with $|\tau|$ denoting the trajectory length. During trajectory sampling, a trajectory may terminate either due to reaching the interaction limit (1000 steps) or by actually achieving the task goal. Therefore, the reward at the final step (i.e., the reward upon trajectory termination) is more accurately the immediate reward returned by the environment at that step, which may or may not be zero. Even if the goal is achieved, the immediate reward is not necessarily zero. Thus, in the main text, we refer to the RTG of the last token as the reward obtained by the model at the current step. For example, in the maze task, the reward for the final step after reaching the goal is +1. We will further clarify the details regarding the alignment of RTGs in the main text to avoid any ambiguity.

W3: There are some literature in decision transformers missing[1][2].
We have already cited and compared CGDT [2] in the experimental section of the main text. CGDT, designed for offline reinforcement learning, also adopts the DT loss as its foundation, but further introduces the value estimation from an asymmetrically pre-trained Critic as a penalty term, encouraging the policy to select optimistic actions that the Critic evaluates as exceeding the target return. TD3+ODT [1], which largely follows the ODT algorithm pipeline, additionally incorporates the RL gradient from the Critic as a penalty term in the loss. Both [1] and [2] introduce gradient information as penalty terms for offline or online fine-tuning in reinforcement learning, but are only applicable to a single setting. In contrast, VDT leverages advantage-weighted and penalty losses, together with parallel sampling techniques, and is the first to achieve competitive performance in both offline and online fine-tuning scenarios. We will further elaborate on the comparison between VDT and [1][2] in the related work section. In addition, the table below presents a performance comparison of the methods, where it can be seen that VDT consistently achieves strong results across different settings.

W4: The proposed method seems to be sensitive with respect to $\epsilon$.
In Figure 2, we tested four values of $\epsilon$: 0.3, 0.5, 0.7, and 0.9. When $\epsilon$ is greater than 0.5, the value function tends to optimistically estimate the value of actions, while for $\epsilon$ less than 0.5, the value function tends to be pessimistic. Since the range of values used in our ablation study was relatively narrow, it was not easy to fully assess the model's sensitivity to $\epsilon$. Therefore, we further extended the experiments in Figure 2 by expanding the range of $\epsilon$ to [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]. The table below shows the results of varying $\epsilon$ on both offline and online experiments. We found that the model's performance is relatively stable and optimal when $\epsilon$ is in the range of approximately 0.6 to 0.8. This observation is also generally consistent with the findings in the original IQL paper. Beyond a certain range, performance drops rapidly, which is related to the inherent properties of the value function. When $\epsilon$ is less than 0.5, the value function tends to underestimate values, making it difficult to effectively guide the model to achieve performance beyond the expert data. The same issue also arises when $\epsilon$ is set too high.

MW1: On page 25, a space is missing.
We apologize for this typesetting error. We have reinserted the missing space and updated it in the subsequent version.

MW2: the subscripts are out of the circle and almost overlap with other parts in Fig1
We have updated Figure 1 by enlarging the circles to accommodate the subscripts and slightly adjusting the layout of longer subscripts to prevent the circles from becoming excessively large. As images cannot be directly displayed during the rebuttal period, we will present the updated figure in the main text in subsequent versions.

Q1: why is IQL taking comparable amount of parameters to decision transformer-based method and is much slower during online finetuning?
We apologize for this potentially confusing point. To evaluate performance, we replaced the policy network in the IQL algorithm with a transformer architecture whose prediction head outputs probabilities. This led to a rapid increase in the number of parameters and a noticeable decrease in convergence speed, although the performance remained comparable to the original IQL. We aimed to demonstrate that simply replacing the policy architecture in IQL with a transformer does not achieve the same effect as VDT. While the original IQL algorithm has a lower computational cost compared to VDT, this inevitably results in a loss of performance. To avoid such misunderstandings, we will revise the relevant table to include the original IQL algorithm with an MLP architecture, as shown in the table below.

L1: lack of evaluation on image-based environments
We thank the reviewers for this suggestion. We have evaluated the performance of VDT on the image-based Atari dataset, reporting the average performance across three random seeds, as shown in the table below. VDT also demonstrates competitive performance on the Atari dataset, indicating that our method is well-suited for both text and image-based datasets. This success was expected, given the precedents set by DT.

Thank you very much for your careful review of our work. We'll answer your questions one by one in the following, including some misunderstandings and some essential academic questions worth exploring.

W1: Seeing the standard deviation of the baselines would have been very helpful in assessing the contribution of the approach.
We sincerely thank the reviewers for their suggestions. We agree that standard deviations are important for reflecting the stability of each method and can further highlight the advantages of VDT over other approaches. In response, we have now added the standard deviations for all methods in Tables 1 and 2. Some of these values are directly taken from the original papers. For cases where the original papers did not report standard deviations, we reproduced the results using the available open-source code. We calculated the average standard deviation over the same three random seeds as VDT (123, 321, 132). The updated Tables 1 and 2 with standard deviations are shown below. As can be seen, VDT demonstrates better stability compared to most baseline methods, which further illustrates the advantages of our approach. Moreover, as the only method that achieves optimal performance in both offline and online settings, we believe that the VDT pipeline can also provide valuable insights for the offline RL community.

Offline

Online

W2: Relevant baselines
The core idea behind Elastic DT's ability to stitch trajectories lies in adaptive history truncation. During inference, EDT dynamically selects the optimal history length to maximize the maximum achievable return from the current state. Specifically, EDT estimates the maximum return for different history lengths using expected quantile regression and then chooses the history length to feed into the Transformer accordingly. EDT does not use a Q-function to guide action selection; instead, it essentially "forgets" unsuccessful histories, allowing the model to escape from low-return trajectories and stitch onto more optimal trajectory branches. We will further add a comparison of Elastic DT in both the methods and experiments sections of the main text. In addition, we have included a comparison with QDT [2], which combines the dynamic programming capabilities of Q-learning with the sequence modeling strengths of DT, enabling relabeling of return targets and thus improving DT's ability to stitch suboptimal trajectories.

VDT differs in that it explicitly introduces a Q-function to provide value-guided weighting for action selection. At the same time, it employs multi-step Bellman equations and double Q-networks to stabilize value function training, thereby extending VDT to be applicable in both offline and online settings, and enabling parallel decision-making during sampling to leverage the evaluation capability of the Q-function fully. In the experimental section, as shown in the table below, we present a comparison of VDT with EDT and QCS. VDT demonstrates a clear advantage over EDT and QCS across a wide range of tasks.

[1] Elastic decision transformer, NeurIPS 2023.
[2] Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. PMLR 2023.

Thank you very much for your careful review of our work. We'll answer your questions one by one in the following, including some misunderstandings and some essential academic questions worth exploring.

W1: Computational Overhead in Sampling
The trade-off between exploration performance and computational cost is a common challenge faced by all deep learning methods. VDT requires parallel trajectory evaluation during sampling, which introduces some additional computational overhead. However, as shown in our experimental results in Table 4, this overhead is generally manageable due to efficient GPU batching. Compared to other methods, it does not become a bottleneck in terms of actual runtime or computational cost, while providing significant performance improvements. Therefore, we consider this additional overhead to be justified.

Q1: I am curious about the ablation study on how performance improves as we increase the range of RTGs together with Q-value based sampling, e.g. are we able to query higher RTGs than that of in the training dataset?
In our original work, we did not attempt to expand the range of RTGs mainly to avoid introducing excessive manual effort and to maintain the simplicity of our method. Intuitively, expanding the RTG range could further improve the algorithm's performance across different tasks. To investigate this, we conducted experiments in the offline setting on hopper-medium-v2, walker-medium-expert-v2, and pen-cloned-v1. In fact, during sampling on hopper-medium-v2, the maximum RTG used for querying the model in VDT already exceeded the maximum RTG in the training set, which is consistent with the usage in DT and aims to ensure fairness. However, we further explored whether expanding the RTG range could bring breakthroughs. For each task, we designed three additional RTG ranges, referred to as Ext. 1, Ext. 2, and Ext. 3.

---

• hopper-medium-v2 (max RTG in dataset: 3322)
  • Original range: [7200, 3600, 1800]
  • Ext. 1: [18000, 7200, 3600, 1800]
  • Ext. 2: [72000, 36000, 18000, 7200, 3600, 1800, 720]
  • Ext. 3: [72000, 36000, 18000, 7200, 3600, 1800]

---

• walker-medium-expert-v2 (max RTG: 5011)
  • Original range: [5000, 4000, 2500]
  • Ext. 1: [6000, 5000, 4000, 2500]
  • Ext. 2: [8000, 6000, 5000, 4000, 2500, 1000]
  • Ext. 3: [8000, 6000, 5000, 4000, 2500]

---

• pen-cloned-v1 (max RTG: 11271)
  • Original range: [12000, 6000]
  • Ext. 1: [15000, 12000, 6000]
  • Ext. 2: [200000, 15000, 12000, 6000, 3000]
  • Ext. 3: [200000, 15000, 12000, 6000]

---

The purpose of Ext. 1 is to include higher RTGs, while Ext. 2 further increases the upper bound and adds a lowest RTG to provide a lower limit, aiming to maximize the RTG space available to the model without being excessive. Ext. 3 removes the lowest RTG to verify whether it affects sampling performance. To control variables, the evaluation horizon was fixed at 5. The experimental results demonstrate that expanding the RTG range consistently enhances performance, which implies that there is still room for improvement in the strategy used for RTG selection. In addition, the lowest RTG in the range has little impact on sampling performance, suggesting that the model may learn to utilize the largest possible RTG for decision-making without being affected by the lowest value.

Furthermore, we observe that the RTG range and the evaluation horizon are highly correlated, and together they exhibit a synergistic effect that jointly influences the upper bound of performance during the sampling process. A possible explanation is that increasing the evaluation horizon allows the model to leverage better the guidance provided by RTGs. Conversely, if either the evaluation horizon or the RTG range is insufficiently large, this process may be hindered. We present the relevant experimental results on hopper-medium-v2.

Q2: In online tuning stage, how much the performance depends on the initial offline training? how about ablating the different data ratios? how does VDT perform if the offline dataset has very poor coverage while including high-value trajectories?
We believe that the impact of offline training on online tuning can be analyzed from aspects such as the quality and quantity of offline data. For data quality, we have included datasets of varying quality within the same task. For example, in the hopper task, hopper-medium-replay represents the lowest quality, containing trajectories from random to medium policies; hopper-medium is of medium quality; and hopper-medium-expert is of the highest quality, consisting of expert policy trajectories. As shown in the experimental results in Table 1, the performance of VDT trained on datasets of different quality is highly correlated with the data quality.

In the online tuning stage, we fine-tune VDT models that were pre-trained on offline datasets of varying quality. As shown in Table 2 of the main text, we observe that after fine-tuning, the performance of VDT pre-trained on different datasets is further improved, and the overall performance remains linearly correlated with the quality of the offline data utilized. However, in certain scenarios, the VDT pre-trained on the lowest quality dataset (hopper-medium-replay) outperforms those pre-trained on the higher quality datasets (hopper-medium and hopper-medium-expert). We hypothesize that this is mainly because under-trained models may be more sensitive to new gradient signals in some cases, making it easier for online fine-tuning to escape local optima and achieve greater improvements [1].

[1] Levine S, Kumar A, Tucker G, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.

To investigate the impact of offline data quantity on the final online fine-tuning performance, we further subsampled the task datasets in the Mujoco benchmark by randomly selecting trajectories and retaining only 5%, 10%, and 50% of the original data. Notably, the various Mujoco datasets contain an average of approximately 2,000 trajectories, which is relatively small compared to typical vision and language datasets. After subsampling, we can evaluate the performance of VDT under the most challenging data-scarce scenarios. It is worth noting that subsampled expert-level datasets can effectively represent scenarios with limited data coverage but containing high-value trajectories. This is because the significant reduction in data volume inevitably leads to lower coverage, while the inherent characteristics of expert datasets ensure the presence of sufficiently high-quality trajectories.

As shown in the experimental results below, we observe that the performance degradation caused by reducing the data quantity is not substantial. When the data is reduced to only 5% of the original amount, the performance on most tasks remains at around 90% of the initial level. Furthermore, since the number of training steps is kept constant in our experiments, it is possible to further improve performance under reduced data conditions by increasing the number of training steps.

We also investigated the impact of the number of offline training steps on the final performance of online tuning. In the original experiments, we used 10,000 offline pre-training steps, after which the model had almost converged. Keeping all other hyperparameters unchanged, we varied the number of offline pre-training steps to 1,000, 5,000, and 10,000 to simulate VDT models that had not fully converged during pre-training. The experimental results are shown in the table below. It can be seen that the effect of the number of offline training steps on final performance is highly similar to the effect of insufficient data on final performance. Both can be attributed to underfitting scenarios. As the number of offline pre-training steps increases, there is still room for performance improvement, which is consistent with intuition.

L1: The proposed method is the computational overhead of the Q-value guided sampling process during inference.
In Table 4 of the main text, we compare the VDT algorithm with other methods. We observe that the average computational cost of VDT remains lower than that of the commonly used ODT, and this overhead is relatively manageable. When scaling VDT to large and complex environments, the cost can be further mitigated by adopting more efficient batching strategies or model pruning techniques. As discussed in Q1, compared to the potential performance gains brought by VDT through the integration of value functions and parallelized sampling, this additional overhead may be relatively acceptable. More importantly, our approach is currently the only method that demonstrates a significant performance advantage in both offline and online settings. We hope that our work can also provide the offline RL community with some inspiration for designing more general-purpose RL algorithms from a new perspective.

Thanks for reviewing our work attentively. We'll answer your questions in the following.

W1: Connections between the one-step value function and the optimal value function will still be helpful.
We can indeed provide further theoretical guarantees for the strong empirical performance of VDT. To this end, we propose Theorem 2, which establishes an upper bound for the one-step value function and quantifies its difference from the optimal value function.

Theorem 2. For any initial state distribution $\mu$, we have

$V^{ \pi^* }( \mu )-V^{ \pi_{DT}^* }( \mu) \leq \frac{ 2 \gamma }{(1- \gamma )^2} \cdot \mathbb{ E }_{ s \sim d^{ \pi^* }}\left[ \max _{ a \notin \mathcal{ A }_D(s)} Q^{ \pi^* }(s, a)- \max _{a \in \operatorname{ supp }( \beta( \cdot \mid s)) } Q^{\pi^*}(s, a)\right ]$

Here, $\mathcal{A}_D(s)$ denotes the action support of the dataset at state $s$, the set $\mathrm{supp}(\beta(\cdot \mid s))$ denotes the set of actions observed in the dataset at state $s$. Due to the strict word limit during the rebuttal phase, we will include the detailed proof in the appendix. Theorem 2 complements Theorem 1 by further elucidating the quantitative relationship between VDT and the optimal policy $\pi^*$. When the dataset has good coverage, our policy is nearly optimal; even with limited coverage, VDT stays conservatively optimal within the supported regions.

W2&Q3: Empirical evidence of hyperparameterss $\eta$ and $\lambda$ and image-based observations.
We have already conducted an experimental analysis of the range of $\eta$ in Table 9 of the Appendix E. Additionally, we have included an ablation study on $\lambda$ in the offline setting. We found that the value of $\lambda$ does not significantly affect the experimental results (except $\lambda = 0$). This further demonstrates that VDT is highly robust to the choice of $\lambda$.

We evaluated VDT on the Atari dataset with image-based observations, averaging results over three random seeds. Compared to existing methods, VDT achieves superior performance, demonstrating its generalization ability across diverse tasks.

Q1: Why is the expectile loss function an unbiased estimate of the conditional expectation?
We apologize for our spelling mistake. The expectile loss function is an unbiased estimator of the conditional expectile, not the conditional expectation, except when the expectile parameter $\tau = 0.5$.

Q2: Why do we prefer the n-step TD loss over the simpler one-step TD loss?
We use n-step TD loss instead of one-step TD loss because n-step targets better balance bias and variance, leading to faster and more stable learning, especially with sparse or delayed rewards. Prior work has shown the benefits of n-step methods in offline RL [1][2][3]. Our ablation experiments confirm that n-step generally outperforms one-step, but using too many steps can also have negative effects. This is likely because longer returns amplify errors from distributional shift, offsetting the benefits of n-step learning.

[1] Reinforcement learning: An introduction. [2] Fixed-horizon temporal difference methods for stable RL. [3] Adaptive temporal-difference learning for policy evaluation with per-state uncertainty estimates.

Q4: How does this loss function optimize the stability of trajectory modeling and prevent the local optima.
These two advantages correspond to the two terms in the loss function. The advantage-weighted term promotes imitation of high-value actions, reducing the impact of low-value or outlier actions and improving stability—an idea also discussed in the IQL paper. The regularization term encourages the policy to explore actions with higher Q-values, even if they are rare in the data, helping the policy escape local optima within the data distribution. The results in Tables 1 and 2 empirically verify that VDT can indeed learn optimal actions that surpass most previous methods. Furthermore, the visualisation results in Tables 4 and 8 in the appendix also demonstrate, from the perspectives of convergence time and reward curves, that VDT exhibits a certain degree of competitiveness in terms of stability.

Q5: Why is the two-step sampling procedure uniform?
This is because longer trajectories contain more sub-sequences. By sampling trajectories in the first step with probability proportional to their length, and then uniformly selecting a sub-sequence from within each trajectory in the second step, every possible sub-sequence in the entire dataset has an equal probability of being chosen. In other words, the higher chance of picking a long trajectory is balanced out by the fact that there are more sub-sequences to choose from within it, resulting in uniform sampling across all sub-sequences.

Q6: In the online learning phase, does the method use the current ongoing trajectory to update the DT?
If "current ongoing trajectory" refers to the complete trajectory obtained after the model interacts with the environment once, then answer is yes. After each full episode is collected using the current policy, we perform return-to-go alignment on the trajectory and add it to the replay buffer. The DT is then updated using trajectories sampled from this replay buffer, which now includes the most recent trajectory collected from the environment. Therefore, during the online fine-tuning phase, we follow an interact-sample-update loop, which ensures that the model continually incorporates the latest experiences. The training procedure for the online fine-tuning phase is presented in Algorithms 1 and 2 in Appendix C.

Q7: Including some discussions of these two categories of methods in this section could be helpful.
We will add the following content to this section: "Our sampling process is similar to CEM [2] and SfBC [3]. Like CEM, we generate multiple candidate actions at each step, evaluate them with a value function over a short planning horizon, and select the best one—essentially a single-step population-based search. Unlike SfBC, which samples from a behavior policy and selects by Q-value, VDT uses RTGs to guide candidate actions and evaluates them in parallel. This integration of RTG guidance and batch evaluation allows VDT to combine the strengths of population search and candidate selection while maintaining efficient inference."

Q8: How do we predefine the m candidate RTG?
The candidate RTGs vary across different tasks, and the selection of RTGs is largely consistent with that of the DT. Our main focus is on utilizing candidate RTGs effectively, specifically leveraging the guidance provided by different candidate RTGs. As can be seen, in some tasks, we use only a small number of candidate RTGs, and we did not deliberately increase the number or variety of RTGs (although this could potentially lead to performance improvements). The primary reason for this is to ensure a fair comparison and to minimize the need for manual design. Due to space limitations, we present the candidate RTGs used in some tasks below. A more detailed table will be reintroduced in the main text.

Q9: Fig 3 looks like the effect of the evaluation horizon is minor?
Since the evaluation horizon is decoupled from other innovations in VDT, it is indeed possible to omit this component for the sake of simplicity. However, we are still eager to explore opportunities for optimizing the sampling process. Although the performance improvements shown in Figure 3 are not substantial enough to be considered groundbreaking for the field (partly due to the strong baseline performance of the model itself), we believe this direction remains valuable. As mentioned in Q8, we did not carefully select the candidate RTGs. We further expanded the range of candidate RTGs used for hopper-medium, as shown in the table below. Surprisingly, we found that increasing the evaluation horizon leads to different changes in model performance across different RTG ranges. This suggests that effective collaboration between the evaluation horizon and RTG selection can potentially lead to further performance gains. Moreover, due to the parallelism in the evaluation process, we believe that the minor additional computational cost introduced by the evaluation horizon is entirely acceptable.

L1: Including some discussions about the theoretical issue or missing properties could be helpful.
We have reintroduced the analysis of the gap between the single-step value function used in VDT and the optimal value function, providing an upper bound and further improving the theoretical framework of VDT. In addition, following the valuable suggestions from the reviewers, we have refined many details of existing methods, including ablation studies on components and hyperparameters, literature comparisons, and discussions of potential advantages. All of these improvements will be incorporated into the revised manuscript.