Sample-efficient Imitative Multi-token Decision Transformer for Real-world Driving

The major concerns are the novelty and the performance of the proposed method. The authors proposed to combine online and offline RL training with decision transformer, which seems to be a quite straightforward combination of DT and online DT. Another drawback is that, the experiments results are not very strong and seems to be comparable with simple baselines. More recent baselines are missing.

• The author should compare more recent planning methods in autonomous driving [1]. SimDT outperforms the BC and DQN methods in the experiments section. However, I am curious whether a BC method with an appropriately designed network could potentially achieve better results. Additionally, suitable data augmentation [2] may more efficiently and simply alleviate the problem of OOD than RL-based methods. This is particularly relevant in the setting of autonomous driving, where there is an abundance of expert driving data.
• The fine-tuning pipeline requires model rollout in simulation. Due to the simulation domain gap, this may still lead to OOD problems in real-world applications.
• The author should discuss more related work in autonomous driving that utilizes multi-token prediction.

[1] Caesar, Holger, et al. "nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles." arXiv preprint arXiv:2106.11810 (2021).

[2] Bansal, Mayank, Alex Krizhevsky, and Abhijit Ogale. "Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst." arXiv preprint arXiv:1812.03079 (2018).

While the concept of using reinforcement learning (RL) to improve transfer in closed-loop settings is innovative in the field of driving, the results presented in this paper are unconvincing. Additionally, the paper includes several unsupported and potentially incorrect claims. The following issues need to be addressed to improve the validity and contributions of this work.

Major comments (in order of importance)

1. Unsupported claims on performance gains. In the closed-loop evaluation, the authors claim that SimDT improves upon DQN by 45.2% in Off-Road Rate and Collision Rate and achieves a 41% improvement over a Behavior Cloning (BC) model. However, these performance improvements cannot be found in Table 1, and the actual improvements observed in Table 1 are much more modest (e.g., ~0.2% for Off-Road Rate compared to DQN, and about 2% for Collision Rate over BC). Misreporting these performance gains in the abstract and main text overstates SimDT’s effectiveness.
2. Lack of comparison with competitive baselines. A meaningful benchmark for SimDT would include a comparison to the Waymo Open Sim Agent Challenge (WOSAC) leaderboard (https://waymo.com/open/challenges/2024/sim-agents/), which includes the state-of-the-art for closed-loop agent realism and performance on the Waymo Open Motion Dataset. Evaluating SimDT against these established models would provide a clearer understanding of its strengths and limitations relative to current state-of-the-art baselines (as opposed to BC-SAC, which is not SOTA).
3. Missing information on dataset and evaluation. Tables 1 and 2 lack critical details: there is no information about the number of scenes trained and evaluated on, what percentage of the scenarios is used in practice? This makes it hard to interpret the results.
4. Misinterpretation route progress metric. The authors suggest that SimDT’s route progress ratio of 105.63% demonstrates the discovery of more efficient routes. However, a ratio above 100% does not necessarily mean a more efficient route; rather, it may simply indicate that the vehicle overshot the destination (e.g. by driving faster than the logged trajectory) or took a longer path. This metric interpretation, as outlined in the Waymax paper (https://arxiv.org/abs/2310.08710; page 5, Section 3.4), does not support the authors' conclusion and could be misleading to readers.
5. Slightly misleading comparison to expert performance. The authors claim that SimDT’s safety metrics are comparable to those of expert demonstrations, with Collision Rates "within the same magnitude" as expert results. However, the expert Off-Road and Collision Rates are significantly lower at 0.41% and 0.67%, respectively, compared to SimDT’s 3.52% and 2.69%. These differences should be put into context, as small percentage differences can have large practical impacts on safety in driving.
6. Claims on safety and ADE without evidence. The claim that SimDT's focus on safety and kinematic feasibility leads to a cautious driving style with a slightly higher average displacement error (lines 367-368) lacks empirical support.
7. Claims of sample efficiency without supporting information. Although the method is described as "sample-efficient," no information is provided about the training dataset size, RL training time, or computational resources. These details are important for substantiating claims of efficiency and should be included.

Minor comments (that did not impact my score)

• Line 477: "SmiDT" should be corrected to "SimDT."
• “Open-loop” is commonly used to describe settings where no feedback is provided, not specifically related to the behavior of other agents. I would suggest to clarify this to avoid misunderstanding.

1. The contributions seem weak, and the baselines are significantly outdated with the latest methods.

• Authors compare against methods like DQN (Minh. 2013) and BC (Argall, 2009). These are very old methods.
• There have been many new RL algorithms like Rainbow DQN, TD3BC, CQL, and AWAC.
• Many transformer-based approaches like Point Transformer V3 Extreme, MotionTransformer, etc.

Suggestion: Please add the latest baselines. Baselines from the 2024 Waymo Open Dataset Challenge are a good start.

2. Although the paper is well-written, it lacks technical rigor and is hard to follow.

• What exact problem are authors solving? From my understanding, the problem is vaguely introduced only in the introduction.
• The paper does not clearly explain where and how current methods fail.
• Fig 1 is unclear. The purpose of outside black lines is not clear. I assume the blue lines are the new trajectory sampled by the authors' method.

Suggestion: Please add a problem formulation section. Give some examples of how single-token Decision Transformers fail.

3. The method seems credible, but it is heuristically put together.

• "The overall online imitative reinforcement pipeline is essential to achieve the greater data-distributed policy" How does author's method lead to greater data-distributed policy.
• R_{imitation} is not clearly explained.
• Where is R_{imitation} used ? I do not see it in Algorithm 1.
• Switching from offline to online learning seems to have been heuristically chosen. The motivation behind the 0.5 ∗ num scenarios is unclear.

Suggestion: I suggest that the authors rewrite the method section to add technical rigor. Each design decision needs to be clearly explained.

4. The math is not clearly and rigorously defined:

• What are $a$, $s$, $g$, $\pi$ variables? Whether they are scalars, vectors, matrices, or function mapping is unclear.
• It is not clear where loss functions L_a and L_ma are used.

My recommended score for the paper is based on the lack of up-to-date baselines and technical rigor. In my opinion, the paper needs a significant amount of work to be accepted.

1. Overall lack of novelty. There is little novel components introduced in the paper. Multi-token prediction has been explored in NLP and RL; PER is classical and nearly 10 years old; the novelty in combating distribution shift in imitation learning is also unclear.
2. Flaw in experiment design: since the authors’ main argument is that their proposed method has the lowest collision rate, is it possible that this simply comes from the fact that they assigned collision with a very high penalty in the RL? According to equation 5, R_{overlap} = -10 in the method. I don't see any related experiment or discussion to remove this doubt.
3. No significant improvement overall compared to SOTA: given the unaddressed flaw mentioned above, plus the fact that the SimDT cannot consistently outperform SOTA on most if not all of the metrics, I think it is valid to suspect that even the low collision rate performance of SimDT might not have come from the robustness of the algorithm itself, but simply from reward engineering.
4. Lack of experiment for “sample-efficient”: I think this part of the title requires a controlled study (fixed amount of data or training FLOPs) to provide empirical results to justify.

1. The multi-token transformer is not novel at all. In the task of simulation agent, multi-token transformer is a standard practice [1,2,3,4] (should note that the multi-token in sim agents is multiple tokens for agents at the same step, instead of multiple tokens for an agent). My overall idea is that multi-step prediction + recending horizong control is not surprising. In Waymo Sim Agent benchmark [5] and Waymax paper, using receiding horizon control on the "one-shot" model is a standard practice.
2. The combination of hindsight replay, prioritized replay buffer is promising. But they are not suprising and their benefits are expected.
3. Overall, my concern is that the paper lack of novelty. I personally don't prefer the paper putting a bunch of existing practices together and claims we improved the scores, without extensive study on why it works and what insights we can learn.

[1] MotionLM: Multi-Agent Motion Forecasting as Language Modeling

[2] KiGRAS: Kinematic-Driven Generative Model for Realistic Agent Simulation

[3] SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction

[4] Trajeglish: Traffic Modeling as Next-Token Prediction

[5] The Waymo Open Sim Agents Challenge