Uncertainty-Aware Decision Transformer for Stochastic Driving Environments

Thank you for your time and valuable advice on improving our paper. We are pleased that you acknowledged the motivation and experiments of our paper. Here we provide answers to all of your questions one-by-one as follows (also see the revised paper in the updated pdf):

Q1: Where does the uncertainty in autonomous driving come from? There are some missing information in Figure 1.

R1: Thank you for highlighting the problem, which is crucial for understanding our paper. Specifically, as discussed in the introduction, uncertainty in the autonomous driving environment may arise from the hidden states of other vehicles and partial observability. In other words, the transition $(s,a\rightarrow s')$ along with the reward $r$ is inpredictable (different possible outcome $s'$ and $r$) because the surrounding vehicles can behave stochastically and some information is not observable.

Based on your suggestion, we have made revisions to Figure 1 and its corresponding description. Specifically, we remove the irrelevant traffic lights from the figure. Besides, we clarify that the term 'return' expresses exactly the same meaning as 'reward-to-go' in reinforcement learning, about the cumulative rewards in the episode. Regarding Figure 1(a), it represents a driving scenario composed of a sequence of driving actions, starting with going straight and followed by a right turn. In this scenario, the uncertainty in the environment is relatively low during the straight driving process, while the variance in the distribution of rewards increases during the right turn due to the uncertain behavior of other vehicles in the environment. Therefore, we propose to condition solely on the truncated return from Task I (going straight) rather than Task I & II to mitigate the influence of environmental stochasticity (less stochastic timesteps, lower return variance) for DT training, which still preserves rewards over sufficient timesteps for the optimization of current actions.

Q4: Missing information through the paper: (1) A simplified and intuitive example is preferred in Figure 1. (2) How to show the data collected from CARLA contains uncertainty (e.g. from the uncertain behavior of surrounding vehicles)? (3) Missing information in Figure 2. (4) Missing information in Figure 3.

R4: Thank you for pointing out our oversights in writing. We have made necessary modifications to the corresponding content in our updated version of paper (highlight in green and blue) and here we provide individual responses to all of your questions.

1. As discussed in R1, we have made revisions to Figure 1 and its corresponding description. Regarding Figure 1(a), it represents a driving scenario composed of a sequence of driving actions, starting with going straight (low uncertainty) and followed by a right turn (high uncertainty). We also remove the irrelevant traffic light from the figure and clarify the task and return distribution contents in the figure. For Figure 1(b) and Figure 1(c), we have modified their x-axis labels to provide greater precision, specifically as, 'Normalized Return (Ground Truth)' and 'Timestep', respectively. Typically, the x-axis in Figure (b) represents the ground truth return in original trajectories. More detailed explanations are expressed in the main context of the newly updated paper.
2. Yes, our training data is indeed collected from CARLA. And as we have discussed in R1, the uncertainty can arise from the stochastic behavior of surrounding vehicles and the partial observability. Two respective examples can be referred in Figure 4 in the paper. Typically, in Figure 4(a) we observe an initial increase in uncertainty as the ego-vehicle enters the lane to follow another vehicle, owing to the lack of knowledge about the other vehicle's behavior. This uncertainty gradually decreases back below the threshold when the vehicle stabilizes in the following state. In Figure 4(b), while approaching the green light, the partial observability about the light state change causes the uncertainty to rise quickly. After the vehicle has moved away from the traffic light, the uncertainty immediately drops back below the threshold.
3. The meaning of $h$ is expressed as 'return-span token' in Figure 2, which is further explained below Equation (6). Specifically, the return-span $h_t$ denotes the timestep count involved in the truncated return $R^h_t$. As we discussed in Section 4.3, for states in certain parts, $h$ is set to the number of timesteps to next segmentation position to include the maximum duration that do not contain any uncertain steps; for states in uncertain parts, $h$ is set as empty for meaningless conditions, to conduct behavior cloning. So in Figure 2, we start with $h=1$, which is the final timestep in a certain part. Afterwards, the state is identified as uncertain and an uncertain part (orange background) begins with empty $h$. Finally, another certain part with 80 timesteps starts, with $h=80$, then sequentially decreases to $h=79, 78, ...$ until another uncertain timestep.
4. In Figure 3 the blud rectangles denote surrounding vehicles and the ego-vehicle is the white rectangle. Figure 3(a) illustrates a scenario where the presence of the surrounding vehicles in the adjacent lane prevents lane changing. In this scenario, the TT planner tends to make overly aggressive maneuvers, resulting in collisions with the vehicles in the adjacent lane. On the other hand, the UNREST planner adopts a cautious planning approach and takes timely actions to stop before any collision occurs.

Hope these clarifications could address your concerns and facilitate a more comprehensive understanding of the insights and contributions of this paper.

References

[1] Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning, ICML 2022.

[2] UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning, NeurIPS 2021.

Q2: The motivating example involves the tradeoff between reward and cost. The proposed transformer model may not correctly assess the uncertainty when tuning weights in reward functions.

R2: Thank you for pointing out the problem of our reward design. I believe it is necessary to clarify that all the reinforcement learning algorithms we employ do not exhibit the so-called trade-offs. Instead, their primary objective is to purely seek strategies that maximize rewards.

As you said, we have discussed the reward and cost compositions of our dataset in Appendix D. However, for the RL algorithms, the cost is integrated in the reward function as penalization for dangerous behaviors. Otherwise, the RL agent may frequently conduct aggressive actions that may raise safety issues. So for the example driving scenario in Figure 1(a), the ego-vehicle will learn to yield to surrounding vehicles, thereby avoiding significant negative rewards resulting from collisions.

Furthermore, as you mentioned, adjusting the weights in the reward function will inevitably lead to variations in the estimation of uncertainties. However, we do not consider this to be a problem because the essence of the effectiveness of RL lies in the design of a well-defined reward function. While adjusting the weights in the reward function may harm uncertainty estimation, the more significant concern is that arbitrary adjustments can lead to suboptimal policy learning by the RL algorithm itself. For our approach, since we are identifying uncertainty in returns (rewards), when the reward function can serve as an effective conditioned target for DT policy learning, UNREST will also be capable of effective segmenting sequences for DTs.

The above table records our new testing driving scores on train and new (in brackets) scenarios w.r.t. different cost weights in reward functions. As can be seen from the table, both excessively high and excessively low weights can result in the inability of the DT to train effective policies. However, the introduction of uncertainty estimation in UNREST consistently brings improvements compared to DT across different weight settings.

Q3: Why do we care about the accumulation of uncertainty in driving tasks? If it can be attributed to DTs, this paper may not be applicable to other frameworks.

R3: Thank you for pointing out the problem. Exactly as you said, the uncertainty accumulation is a normal characteristic within the context of autonomous driving environments. However, our attention to this property is not solely due to the DT framework. Typically, it's natural to consider uncertanity accumulation for DTs, since the conditioned reward-to-go includes the uncertainties at each time step from the current state to the end of the sequence. UNREST, however, addresses and reduces this conditioning uncertainty by segmenting the sequences, enabling more effective policy learning in stochastic environments.

But for other RL algorithms, uncertainty accumulation is also a significant factor to take into consideration since the monte-carlo/trajectory statistics are fundamental factors in RL, which utilize simulated trajectory return to update the value functions, or directly select a better action. In these simulated trajectories, the cumulated uncertainty can lead to large variance during backup/forward searching. Our uncertainty estimation is applicable in these frameworks by accurately identifying uncertain timesteps and branching out to perform additional trajectory rollouts at those points. This process helps improve the accuracy of the monte-carlo estimation results，which suggests the general use cases of our uncertainty estimation. For example, [1, 2] propose to learn transition models for monte-carlo simulation. Due to the significant bias associated with single-step rollouts, they have all opted for multi-step beam search to find the most effective and safe actions. By combining our uncertainty estimation method, we can effectively identify specific uncertain states that require branching and rollout. This can greatly helps reduce the number of simulations and improves planning efficiency.

Thank you for your valuable guidance in improving our paper (please see the updated pdf and our response with more discussion and experimental results). Here we humbly ask for your feedback at your convenience such that we may still have the chance to discuss with you and chance to improve our final version before the close of the rebuttal window.

Since the rebuttal period is almost over, we are wondering whether our responses have addressed your concerns properly. Your feedback will definitely help reach a more reasonable decision on our submission. Thank you!

Thank you for the feedback. First and foremost, it is imperative for us to highlight that the application of RL in the context of autonomous driving planning has been increasingly gaining popularity in recent times [3, 4], contrary to your assertion that it is not a promising option (as stated in the first paragraph of our paper). As claimed in [4], RL enables the planners to learn the causal relationship of driving through rewards while IL simply copies experts' actions. In contrast to the performance limitations imposed by expert-based approaches such as IL [2], RL holds the potential to achieve performance that surpasses that of human experts. That's also the reason why our paper and [5, 6] choose offline RL driving planners.

Next, we want to explicate that uncertainty accumulation exists in both rule-based [1] and IL [2] planners as well. For rule-based planners like [1], they generally make decisions by solving an optimization problem over multiple timesteps (model predictive control) and safety constraints. As a result, the uncertainty of environment transitions accumulates with each timestep of the lookup process, necessitating the selection of the most cautious action to ensure safety. For IL planners like [2], due to the relatively narrow scope of expert data in terms of included states, it is common practice to simulate driving trajectories beyond the dataset in order to minimize out-of-distribution (OOD) states during actual testing. In this sense, the uncertainty accumulates over the rollout timesteps, thereby introducing additional trajectories that do not exist in the original dataset to assist in imitation learning.

Finally, based on all of our responses, we would like to express a few points. First, we believe that the "reward-cost tradeoff" and the "scope of uncertainty accumulation" mentioned by you cannot be seriously considered as weaknesses of the model, but rather as potential concerns raised by you. These concerns have now been thoroughly discussed and addressed. Secondly, for the issues you pointed out regarding the presentation in the paper, we have made targeted modifications or highlighted aspects in the original text that you may not have initially noticed, which we believe will provide readers with a clearer understanding of the paper's intent. Noticing the positive feedback from the other two reviewers (xhtH and PUEU), and that reviewer mZKV only highlighted deficiencies in writing, we sincerely hope that you will reconsider your evaluation of the paper (since your concerns are now adequately addressed). Looking forward to your reply.

[1] Comprehensive Reactive Safety: No Need For A Trajectory If You Have A Strategy, IROS 2022.

[2] ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst, RSS 2018.

[3] End-to-End Urban Driving by Imitating a Reinforcement Learning Coach, ICCV 2021.

[4] DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving, ICCV 2023.

[5] Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning, ICML 2022.

[6] UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning, NeurIPS 2021.

Thank you for the valuable advice on improving our presentation. We are pleased that you acknowledged the contribution and soundness of our paper. For your questions, we answer them one-by-one as follows (also see the revised paper in the updated pdf):

Q1: The paper is not well written. (1) Some terms and notations are used without definition or description. (2) Some key contents description is "verbal" and "handwaving", making it ambiguous to understand the algorithm. (3) A brief introduction of DT is needed in the paper. (4) Use more concrete/clear description by notations in the method instead of long words description.

R1: Thank you for pointing out the issues in our writing. We have made necessary modifications to the corresponding content in our updated version of paper (highlight in green and blue). Moving forward, here we provide individual responses to all of your questions.

1. Regarding the issue of notation, we can ensure that all symbols used are clearly defined in the paper. Like the example $\boldsymbol{\tau_{<t}^{ret}}$ you raised in Section 4.2, we have defined in Section 3.2 that $\mathbf{x_{<t}}$ denotes tokens from step 1 to $(t-1)$, and in Section 4.2 we define $\boldsymbol{\tau^{\text{ret}}}=\{(s_t,a_t)\}_{t=1}^T$ to be all state-action pairs involved in the driving trajectory. Besides, we have clarified Task I, Task II, and Normalized Ground Truth in Figure 1. Specifically, Figure 1(a) is composed of two driving tasks: go straight then turn right, and the x-axis in Figure 1(b) denotes the ground truth return in original driving trajectories.
2. For the issue of verbal expressions, since you haven't provided specific examples (we would greatly appreciate it if you could respond with specific points in the paper that were unclear to you, as this could really help in improving our writing), we have taken it upon ourselves to identify and modify certain aspects of the paper. For example, in the introduction, we further elaborate on the content depicted in Figure 1 and the insights derived from it. In addition, we have also provided a clearer introduction to our sequence segmentation and uncertainty-guided planning approach.
3. We acknowledge your point that since DT is a crucial foundation of this paper and warrants a separate section to provide an overview of its fundamental concepts and contents. Specifically, we have written in Section 3.2 of the original paper a description of the training and testing details for DTs, which you may miss in the first time of reading.
4. We apologize for any misunderstanding caused by the informal expression used in our formula. Typically, we use two separate transformers to get $\tilde{x_{s_t}}$ and $\tilde{x_{a_t}}$. And to express the formula more concisely, we have used the notation $\mathcal T$ to replace the long word 'Transformer' in Equation (3) and (7).

Hope these improvements we have made in our writing could address your concerns and help you gain a clearer understanding of the insights and contributions of this paper.

Thank you for your valuable guidance in improving our writing (please see the updated pdf and our response with more discussion). Here we humbly ask for your feedback (since we find you overall like our work except the presentation) at your convenience such that we may still have the chance to discuss with you (e.g. pointing out the specific aspects within the text that you find to be 'verbal' or 'handwaving') and chance to improve our final version before the close of the rebuttal window.

Since the rebuttal period is almost over, we are wondering whether our revised paper and responses have addressed your concerns properly. Your feedback will definitely help reach a more reasonable decision on our submission. Thank you!

Q2: Missing discussion about how to set the uncertainty threshold in experiments. Besides, how to generalize the segmentation approach to scenarios like sequences of nearby intersections.

R2: Thank you for pointing out this important problem of UNREST's practical usage. Due to space constraints, we have postponed the discussion on the selection of the uncertainty threshold to Appendix F, which you may have overlooked during initial reading. Specifically, in Figure 5(a), we plot the histogram of uncertainty value. As we can see the majority of uncertainty estimation associated with state transitions falls within the range of $0$ to $10^{-4}$, aligning with intuitive expectations. In the context of autonomous driving environments with sparse vehicle presence and a limited number of traffic signals, the stochasticity of environment transitions tends to be relatively low. The uncertainty threshold $\epsilon$ is accordingly chosen as 3.0 to cover corner cases with large uncertainties.

Second, to generalize our segmentation mechanism to driving tasks which have large regions of uncertainty that should themselves be segmented, you can refer to results in Figure 7. By adjusting the uncertainty threshold $\epsilon$ and the minimum sequence length $c$, we can flexibly control the distribution of the segmented sequences. For your case, we can simply decrease the values of $\epsilon$ and $c$ (as shown in the third subplot of Figure 7). By using a smaller value of $c$, each uncertain segment can promptly stop when encountering consecutive certain states. Similarly, a smaller value of $\epsilon$ can detect and initiate new uncertain segments more quickly when entering a new uncertain region. So it appropriately helps for further segmentation within large uncertain areas.

Q3: Though with ample empirical evidence for UNREST's success, there isn't a clear theoretical framework to connect it to.

R3: We thank the reviewers for pointing out our oversight in the paper. In fact we have detailedly discuss UNREST's theoretical generalizability in our original paper, as you can find in Proposition 1. Due to the page limit, the proposition in the main context is explained in an informal manner, with the formal statement and proof left to Appendix B. Specifically, the proposition is inspired by [1] and reveals that under natural assumptions in driving environments, UNREST can generalize to achieve high returns with bounded error at less stochastic states if expert demonstrations cover corresponding actions. The relevant content has been highlighted in the new version of our paper, where you can find further details for discussion. Besides, I find the causal/non-causal entropy regularization mentioned by you to be an exceptionally intriguing aspect. It is possible that we can explore this point further to explain how UNREST addresses the issue of overly-optimistic behavior of agents.

Thank you for the positive feedback and constructive suggestions that can help us to improve the paper. We are happy that you acknowledge our work’s motivation, technical quality and presentation. For your questions, we provide additional experimental results and discussions as follows (also see the revised paper in the updated pdf):

Q1: Some parts of the introduction is somewhat difficult to understand.

R1: Thank you for pointing out the issue with our unclear statement. We have modified the introduction in the newly updated version (highlighted in green and blue) and here we answer your problems respectively.

1. About the 'highly representative environment model', it usually means the state transition function $P(s'|s,a):\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ in reinforcement learning. Previous methods [2, 3, 4] attempt to fit the state transition model to deal with transition uncertainty. However, while adding complexity, these methods are applicable only when state transition functions can be learned adequately, which is often not the case for driving because of the uncertainty brought by complex interactions and partial observability. Our method bypasses the challenging high-dimensional state transition function learning by computing uncertainty through conditional mutual information $u_t=D_{\text{KL}}\big(p_{\varphi_a}(R_{t}|\boldsymbol{\tau_{<t}^{\text{ret}}}),p_{\varphi_s}(R_{t}|\boldsymbol{\tau_{<t}^{\text{ret}}},s_{t})\big)$, where the return distributions can be effectively learned with fewer resources.
2. Regarding Figure 1(a) and its corresponding explanation, it represents a driving scenario composed of two tasks (now denoted as Task I & II), starting with going straight (Task I) and followed by a right turn (Task II). In this scenario, the uncertainty in the environment is relatively low during the straight driving process, while the variance in the distribution of rewards increases substantially considering Task I & II as a whole due to the uncertain behavior of other vehicles in the environment during the right turn. Therefore, we propose to condition solely on the truncated return from Task I (going straight) rather than Task I & II to mitigate the influence of environmental stochasticity (less stochastic timesteps, lower return variance) for DT training, which still preserves rewards over sufficient timesteps for the optimization of current actions.

Hope these modifications in our presentation can help you better understand the main insights and contributions of our work.

Q4: How could UNREST generalize to partially observable environments where the drivable area, boundaries, pedestrains and bikes are not fully observed?

R4: Thank you for highlighting this interesting question regarding our experimental design. As you aptly pointed out, the uncertainty in the driving environment can arise from both the latent states of surrounding vehicles and partial observability. In this response, I would like to elucidate the capability of UNREST in effectively handling the uncertainty introduced by partial observability.

Theoretically, we compute the uncertainty by the conditional mutual information $u_t=D_{\text{KL}}\big(p_{\varphi_a}(R_{t}|\boldsymbol{\tau_{<t}^{\text{ret}}},p_{\varphi_s}(R_{t}|\boldsymbol{\tau_{<t}^{\text{ret}}},s_{t})\big)$. So regarding the uncertainty introduced by partial observability, such as pedestrians who were obstructed and not visible in the previous step but are now visible, can result in a significant change in return distribution $p_{\varphi_s}(R_{t}|\boldsymbol{\tau_{<t}^{\text{ret}}},s_{t})$ from $p_{\varphi_a}(R_{t}|\boldsymbol{\tau_{<t}^{\text{ret}}})$ because of the newly captured information.

From the experimental perspective, we have discussed in Appendix D in the paper about our state space, which is similar to that in [2]. Specifically, the ego vehicle can only obtain distances of nearby vehicles within its limited field of view and perceive the boundaries of a limited portion of the target route, which is exactly consistent with your proposed partially observable setting. Building upon this, our experimental results further demonstrate the superior ability of UNREST compared to previous baselines in addressing the uncertainty brought by partial observability. Besides, we also visualize in Figure 4.(b) about the uncertainty curve when the ego-vehicle is passing the green traffic light. As you can see, while approaching the green light, the partial observability about the change of light state causes the uncertainty to rise quickly. But after the vehicle has moved away from the traffic light, the uncertainty immediately drops back below the threshold. This interpretable result also validate the effectiveness of our uncertainty measure.

References

[1] When does return-conditioned supervised learning work for offline reinforcement learning? NeurIPS 2022.

[2] Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning, ICML 2022.

[3] You Can’t Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments, NeurIPS 2022.

[4] Dichotomy of control: Separating what you can control from what you cannot, ICLR 2022.

We express our sincere gratitude for your valuable guidance in improving our paper (please see the updated pdf and our response with more discussion). Here we humbly ask for your feedback at your convenience such that we may still have the chance to discuss with you and chance to improve our final version before the close of the rebuttal window.

Thanks for your time and constructive feedback. We are pleased that you acknowledged the contribution and experiments of our work. Here we provide more experimental results and insights to address your concerns (also see the revised paper in the updated pdf):

Q1: UNREST is only evaluated in a single environment. Would it scale to other stochastic environments like those in related works?

R1: Thank you for pointing out the problem of our experiments. The reason why we only study the driving environment is that, though as a general offline RL algorithm that can be applied to any RL environments, UNREST is specially designed to address two major challenges in autonomous driving: long-horizon planning and environmental stochasticity. To make the segmentation strategy proposed by UNREST effective, two essential factors are required：1）'uncertainty accumulation' property that ensures the truncated return is less affected by the environments than the global return, to provide effective supervison for policy learning; 2) 'temporal locality' property that makes optimizing future returns over a sufficiently long horizon similar to the optimization of global return. We have identified these two properties in the context of autonomous driving, but they may not necessarily apply in other environments. We have also tested UNREST on the conventional D4RL benchmark (approximately deterministic environments, results in Appendix F.5), where it just attains similar results to vanilla DTs. Similarly, SPLT [3] is also only tested on the CARLA simulator. Therefore, we believe that the autonomous driving environment is the most suitable environment to test our proposed algorithm.

In the above table, we test UNREST in the customized stochastic D4RL environments from DoC [4], which add time-correlated Gaussian noise to actions. The results are also added in Appendix F.5, where UNREST achieves highest return among the models.

Q2: The idea of using truncated return is not entirely novel enough.

R2: We appreciate reviewer's attention to the novelty of our work. Here we need to clarify one significant point that the key novelty of UNREST lies in the low-resource occupancy uncertainty estimation method, along with the proposed sequence segmentation and designs of conditions based on it.

• Novelty 1: Different from previous works [2, 3, 4] that attempt to fit high-dimensional state transition functions and introduce complex VAE or adversarial training, we introduce for the first time the utilization of conditional mutual information to estimate uncertainty. As discussed in the paper, the complex interactions, partial observability, and long trajectories challenges training of previous works, where our proposed mutual information can be effectively learned with fewer resources, as validated in experiments.
• Novelty 2: Back to the fundamental objective of this paper, we propose a novel sequence segmentation and designs of conditions for customizing DTs in stochastic driving environments. Specifically, we segment sequences into certain and uncertain parts according to our uncertainty measurements, and learn to behave aggressively and cautiously within them, respectively.
• Novelty 3: We also summarize two essential properties of driving environments: 'temporal locality' and 'uncertainty accumulation' properties, based on which we theoretically prove that replacing conditioned returns in original DT with truncated returns that only consider returns to the next segmented positions can help UNREST generalize to attain high rewards.

Therefore, while it is true that truncated return has been employed in previous works [1], in this paper based on our discovered properties, we introduce a novel specific segmentation method based on our proposed uncertainty estimation techiniques and provide corresponding theoretical justifications. This represents a distinct contribution that was absent in prior literature.

Hope this clarification will assist you in better understanding the key insights and novelty of this paper.

Q3: The algorithm assumes expert-level data to function in uncertain regions. Would it be applicable in offline datasets with suboptimal trajectories? Is there a better strategy the method can follow in uncertain regions than just following the expert?

R3: Thank you for highlighting this crucial issue regarding the scope of applicability of UNREST. In fact, the decision to resort to direct behavior cloning (BC) in uncertain parts within UNREST can be attributed to the compromise made due to the unreliability of returns, which hinders the provision of effective supervision. We opted for this approach for two reasons: (1) Firstly, BC is a lightweight and easily implementable algorithm that simply requires setting the original conditioned return as a dummy token. (2) Secondly, expert data collection is relatively easy in the context of autonomous driving environments. However, it should be noted that the AutoPilot provided by CARLA has multiple fail cases and cannot be considered an optimal expert.

In order to propose a strategy that can be employed in uncertain regions when there exists suboptimal trajectories, we propose a simple yet effective UNREST variant that additionally fit a state value function. The remaining logic is consistent with MARWIL [5], where we utilize neural network estimation of advantages to reweight actions in the BC objective. In this way, UNREST can learn to identify and choose to imitate actions in optimal trajectories. The results are recorded in the following table.

Since the offline data collected by AutoPilot can be suboptimal, we find reweighted BC can effectively down-weight those suboptimal actions and improve UNREST's driving scores at both training and new scenarios. However, due to its significant increase in model complexity, we still recommend the original UNREST as the base planner.

References

[1] Reinforcement Learning Upside Down: Don't Predict Rewards--Just Map Them to Actions, Arxiv 2019.

[2] You Can’t Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments, NeurIPS 2022.

[3] Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning, ICML 2022.

[4] Dichotomy of control: Separating what you can control from what you cannot, ICLR 2022.

[5] Exponentially Weighted Imitation Learning for Batched Historical Data, NeurIPS 2018.

Thank you for the feedback. For your question, as we have stated in the previous response, we simply adopt the same environment setting as [4]. While the original D4RL environments are deterministic by default, we modify them by introducing time-correlated Gaussian noise to the actions before inputing the action into the physics simulator during data collection and evaluation. We find these simple modifications to environments greatly harm DT's performance, where UNREST succeeds in obtaining performance improvements.