Diffusion-Based Planning for Autonomous Driving with Flexible Guidance

We thank the reviewer for the constructive comments. Regarding the concerns of the reviewer d65s, we provide the following responses.

W1. It is unclear what novel technical contributions the authors have made in this work.

We appreciate the reviewer for the constructive comment. We have summarized our contributions at the end of the introduction in the revised version. We provide the discussion as follows:

• To the best of the authors' knowledge, this paper is the first to fully harness the power of diffusion models for high-performance motion planning, without reliant on rule-based refinement. This is further demonstrated in the General Response GR1, where we test our model on larger and more challenging benchmarks, still achieving SOTA performance.
• We designed a model specifically for planning, not just combining existing techniques, achieving high-quality trajectory outputs and SOTA performance (Figure 3, General Response GR1). Our DiT-based architecture features a core fusion mechanism between noised future vehicle trajectories and conditional inputs, such as environmental and navigation information. We redefine planning as a future trajectory generation task, jointly producing the ego vehicle's plan and neighboring vehicle predictions. Additionally, we show how the guidance mechanism in diffusion models can align planning behavior with safe or human-preferred driving styles.
• We have collected and evaluated a new 200-hour delivery-vehicle dataset, carefully processing it to ensure compatibility with the nuPlan framework. We will open-source this dataset as a supplement to nuPlan.

W2. The authors should provide a more extensive review of the related literature on diffusion models for autonomous driving.

We thank the reviewer for the suggestion to discuss related works. We follow the recommendation to moved the discussion on diffusion-based methods to the "Related Works" section and have also expanded the discussion on traffic simulation-related works, which is briefly outlined below.

• For [1], [2], and [3] ([1],[2],[3] in the reviewer's comment), which focus on generating diverse driving scenarios, it is important to note that this task differs from planning. Their generation emphasizes diversity in simulation rather than quality or drivability, as the outputs are not directly used for control. Additionally, their method uses the standard UNet structure from diffuser [4] without design considerations specific to autonomous driving planning.

W3. Some works have developed diffusion-based motion planners for nuPlan, they should be included as baselines for comparison .

We thank the reviewer for recommending related works [5] [6] [7] ([4],[5],[6] in the reviewer's comment) as baselines. During the rebuttal phase, we compared them, and our method continued to achieve superior performance (see Table below or more details in General Response GR1). These methods naively apply diffusion loss to existing frameworks or stack parameters without specific designs, making them reliant on post-processing and resulting in high computational costs. Detailed implementation is provided below:

• For [5], we use the Val14 scores for the diffusion policy under the no-LLM condition reported in the paper, as well as the inference time with LLM assistance.
• For [6], this is not a diffusion-based method. While the official checkpoint is provided, the code for closed-loop simulations is not available, making it unusable for our comparison. Moreover, the paper only reports open-loop scores and scores under the reactive setting with refinement, which limits our ability to make a meaningful comparison.
• For [7], the implementation is only available in an open-loop setting. However, we found the corresponding closed-loop scores in its v2 version [8] (released after our submission). Additionally, we used the publicly available 800M checkpoint from [8], removed the rule-based refinement, and re-tested the model, also evaluating the inference time.

We thank the reviewer for the constructive comments and positive feedback on our paper. Regarding the concerns of the reviewer hoYE, we provide the following responses.

W1. Insufficient benchmark and metric comparison

Regarding the comment on the Val14 and Test14-Hard results:

• We appreciate the reviewer's constructive comment. We have added experimental results on the Val14 and Test14-Hard benchmarks. Please refer to General Response or Appendix B for more details. We continue to achieve SOTA closed-loop performance among learning-based methods.

Regarding the comment on the open-loop and reactive results:

• In our original submission, we primarily evaluated performance using closed-loop non-reactive simulation, as recent findings have shown that there is little correlation between open-loop prediction performance and closed-loop planning in both autonomous driving [1] [2] and robotics [3] [4]. Methods optimized for open-loop metrics can face severe distribution shift problems in closed-loop settings. For instance, [1] highlights the misalignment between ego-forecasting (open-loop) and planning (closed-loop), while [4] states that "action MSE on the validation set is another potential metric, but we find it often does not correlate with real-world performance." Due to the unreliability of open-loop evaluations, we chose to report the more reliable closed-loop performance. However, to fully address the reviewer's concern, we have included the open-loop results, as shown in the table below, which align with the above analysis.
• For the reactive metric, nuPlan employs a simple IDM model for neighboring vehicles, which is a relatively weak model and does not accurately reflect real-world traffic characteristics [5]. During the rebuttal phase, we conduct additional experiments in reactive settings, as shown below, and our method continue to achieve sota performance among learning-based baselines.

We really appreciate the reviewer for the constructive comments and positive feedback on our paper.

W1. It would be beneficial to include additional simulation benchmarks from nuPlan, such as the Val14 benchmark and the Test14 hard benchmark

We have added experimental results on the Val14 and Test14-Hard benchmarks, where we continued to achieve state-of-the-art performance among learning-based methods. By incorporating a refinement step, we were able to achieve a score above 94 (Val14 and Test14), surpassing human expert level. Please refer to the new results in the General Response and Appendix B of our revised paper.

We thank the reviewer for the constructive comments. Regarding the concerns of the reviewer J7se, we provide the following responses.

W1. & Q3. Though PLUTO (hybrid method) performs better than Diffusion Planner in the NuPlan dataset, no comparison with PLUTO w or w/o refine is shown for the delivery-vehicle driving dataset

• We thank the reviewer for the valuable feedback. The PLUTO pipeline involves a specific and time-consuming data processing stage. We have made our best efforts to go through the entire PLUTO pipeline using the delivery vehicle datasets; however, it requires significant CPU resources and the provided parallel processing utility often encounters errors, causing interruptions in data conversion. Given the current progress, it is unlikely that we will be able to complete all training by the end of the rebuttal period. However, we will add results in the final version.
• We are able to report PLUTO’s results on the nuPlan dataset thanks to the official checkpoint. Additionally, we have included results from the larger Val14 and more challenging Test14-hard benchmarks to highlight the advantages of our approach over PLUTO. Please refer to the General Response or Appendix B. for more details.

W2. This should clearly outline how the Diffusion Planner addresses each of the limitations mentioned and what specific novel aspects it introduces

We appreciate the reviewer's constructive comments. We have added a detailed statement of contributions at the end of the introduction, which we briefly outline here:

• To the best of authors' knowledge, this paper is the first to fully harness the power of diffusion models with a specifically designed architecture for high-performance motion planning, without reliant on rule-based refinement, as demonstrated in the paper and General Response GR1.
• As shown in General Response GR1,2, our experiments in the paper, and Appendix B, we achieve SOTA closed-loop performance on nuPlan benchmark, generating more robust and smoother trajectories compared to the baselines, even without post-refinement, as illustrated in Figure 3.
• We demonstrate that our model can achieve personalized and flexible driving behavior at runtime by utilizing a guidance mechanism—a desirable feature for real-world applications that is technically challenging for existing transformer-based planning methods.
• We have collected and evaluated a new 200-hour delivery-vehicle dataset, carefully processing it to ensure compatibility with the nuPlan framework. We will open-source this dataset as a supplement to nuPlan.

N1. A legend should be added to Appendix A, Figure 8 N2. Line 195: Conditions C could be mathematically defined

• We appreciate the reviewer’s reminder. The meanings of the elements in Figure 8 have been introduced in Figures 1 and 3 of the main text. In the latest version, we have added the same explanations for Figure 8.
• For the mathematical definition of condition C, we have actually provided the explanation in Section 4.2, including its definition, dimensions, and the formula for how it is integrated into the network.

Q1. Line 466-477, “Due to space limitations, …”, what space limitations is the author mentioning

Sorry for the confusion. We actually meant "space limit." We intended to highlight that various forms of guidance and combinations can be explored in practical applications. Due to the space constraints of our paper, we selected three types of guidance to showcase in the experimental section. We have revised the paper to make this clearer.

Q2. It is unclear in Figure 7 about the choice of hyperparameters in the proposed method. Could the author elaborate on the seen trends in Figure 7.

We apologize for the unclear illustration. For a more detailed analysis of Figure 7, please refer to Section 5.2 (lines 512-517). In Figure 7, darker colors indicate better performance. We find that a low temperature is necessary, and the model demonstrates good robustness across various step counts and levels of low temperature. Ultimately, we select 25 steps for denoising and set the low-temperature sampling parameter to 0.5, as shown in Table 6 in Appendix C.2.

Dear Reviewer,

Could you please look at the author feedback?

Thanks, AC

GR2. Experiments with extra post-processing

To evaluate our methods with similar rule-based refinement, we directly applied an existing refinement module [2] after our learning-based method without any parameter tuning, which adjusts the model outputs. All trajectories were scored as described in [3]. We report the Val14, Test14, and Test14-Hard closed-loop non-reactive (NR) and reactive $R$ results for all rule-based and hybrid methods, as follows.

Our method, with the addition of post-processing, achieved scores that other methods could not reach, even surpassing human performance. This is due to our model's ability to output high-quality trajectories, further enhanced by post-processing. We have visualized the trajectory quality in Figure 3. It should also be noted that, as shown in the previous table in GR1, many high-scoring methods experience significant score drops when refinement is removed. This highlights their over-reliance on refinement, which is consistent with our discussions in the introduction and related work sections.

[1] Yang, Brian, et al. "Diffusion-es: Gradient-free planning with diffusion for autonomous driving and zero-shot instruction following." arXiv preprint arXiv:2402.06559 (2024).

[2] Sun, Qiao, et al. "Generalizing Motion Planners with Mixture of Experts for Autonomous Driving." arXiv preprint arXiv:2410.15774 (2024).

[3] Dauner, Daniel, et al. "Parting with misconceptions about learning-based vehicle motion planning." Conference on Robot Learning. PMLR, 2023.