Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

Weakness 1 and Question 2: Evaluation and visualization of future BEV prediction quality.

Great suggestion! For visualization, we have already presented the predicted future BEV maps in Figure 3 of main paper, as well as in Figures 1 and 2 of appendix. Regarding the quality, we have now measured the mIoU of the predicted future BEV map on the NAVSIM dataset, standing at 38.87. We will include this result in the revision.

Weakness 2 and Question 1: Testing on rare or high-entropy scenarios, longer horizons, complex urban environments, and edge cases beyond NAVSIM and nuScenes.

• Experiment on complex scenarios: Thanks. To conduct the suggested test, we test our model using the scenarios from the test split of NAVSIM with more than 10 agents. As shown in the table below, our model can still achieve strong performance. We have provided a visualization in Figure 1(b) of appendix and will add more in the revision.

• Experiment beyond NAVSIM and nuScenes: For more extensive evaluation, we have further evaluated the more complex Bench2Drive [1] dataset with many rare and high-entropy scenarios such as emergency braking, merging, and overtaking, along with a longer planning horizon and closed-loop testing. As shown in the table below, our model consistently achieves the best performance. We will add this test.

Weakness 3: Iteration number and its impact on performance.

• Explanation: Our model design is, in fact, generic and not tied to a specific optimal iteration count. The ideal number of iterations largely depends on the characteristics and complexity of the target scenario. While more iterations generally lead to better representations, an excessive number can introduce computational burden and uncertainty. This doesn't conflict with our design; instead, our iterative architecture offers the flexibility to improve results by adding more iterations as scene complexity demands.

• Experiment: For instance, 2 iterations proved sufficient for NAVSIM given its comparatively lower complexity. However, for the more intricate Bench2Drive [6] dataset, our ablation study (see table below) clearly shows that 3 iterations yield superior results. This demonstrates the adaptability of our approach.

Ultimately, our framework allows for a flexible trade-off between accuracy and efficiency, letting users decide the optimal balance based on their specific needs and the complexity of the environment they're working with. We will discuss this in the revision.

Question 3: Inference time and suitability for real-time deployment.

Thanks and we will discuss this in revision: Our model achieves an average inference time of 24 ms on the NAVSIM dataset, comparable to DiffusionDrive [1] — a lightweight and deployable diffusion-based model. Meanwhile, our model outperforms DiffusionDrive [1] in terms of planning performance, indicating its suitability for real-time deployment with superior effectiveness.

[1] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving, CVPR 2025.

[6] Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving, NeurIPS 2024.

[7] VAD: Vectorized Scene Representation for Efficient Autonomous Driving, ICCV 2023.

[8] Planning-oriented Autonomous Driving, CVPR 2023.

[9] Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Driving, CVPR 2025.

[10] Bridging Past and Future: End-to-End Autonomous Driving with Historical Prediction and Planning, CVPR 2025.

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

Weakness 1: Differences and connections with WoTE.

• As discussed in Line 104 of the main paper, WoTE [2] employs a BEV world model to generate future BEV, which are then used to select the best trajectory from a set of candidates proposed by the planner. In contrast, our SeerDrive framework establishes feature-level interaction between BEV world modeling and end-to-end planning, enabling uniquely iterative and closed-loop communication between the two components.

• While both WoTE and SeerDrive leverage a BEV world model to support the planning process, their designs are fundamentally different and orthogonal. We will clarify these distinctions in the revised paper.

Question 1: About $\mathcal{T}\_{\text{final}}$.

• Apologies for the lack of clarity. $\mathcal{T}\_{\text{final}}$ refers to a set of candidate trajectories. A probability score is assigned to each trajectory based on the current ego feature, and the final trajectory is selected according to the highest score, consistent with prior work [1].
• As described in Line 200 of the main paper, we also specify the number of trajectory modes. This clarification will be highlighted in the revised version of the paper.

Question 2: Ablation study on the performance of $\mathcal{T}\_{\text{a}}$, $\mathcal{T}\_{\text{b}}$, and $\mathcal{T}\_{\text{final}}$.

Thanks. We conduct an ablation study to separately evaluate the performance of $\mathcal{T}\_{\text{a}}$, $\mathcal{T}\_{\text{b}}$, and $\mathcal{T}\_{\text{final}}$, as shown in the table below. The results show that $\mathcal{T}\_{\text{final}}$ significantly outperforms both $\mathcal{T}\_{\text{a}}$ and $\mathcal{T}\_{\text{b}}$, indicating that each contributes meaningfully to the superior performance of the final trajectory.

Question 3: The use of the anchored endpoints to initialize the future ego feature.

• Explanation: As described in Line 202 of main paper, the future BEV corresponds to the final planning step. Thus, we initialize the future ego feature using the anchored endpoints, allowing it to encode prior knowledge of the ego vehicle’s future state at the final planning step.

• Clarification: Note that the future ego feature is not updated iteratively — only the current ego feature undergoes iterative refinement.

• Experiment: For more extensive evaluation, we have now analyzed different initialization strategies for the future ego feature. As shown below, using anchored endpoints as a prior yields the best performance. We will add this in revision.

[1] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving, CVPR 2025.

[2] End-to-end driving with online trajectory evaluation via bev world model, ICCV 2025.

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

Limitation 1: The world model predicts the final future BEV at horizon H.

• Explanation: We predict the final future BEV at the horizon $H$, corresponding to the last planning step, as the future scene at the final frame serves as the primary reference with the most information for determining the final direction of the trajectory. This provides an effective and efficient reference for planning.

• Experiment: Regardless, as suggested, we have now analyzed this design choice by additionally predicting a sequence of intermediate BEV representations and concatenating them as references for the planner. As shown in the table below, incorporating intermediate predictions does not lead to significant performance improvements, while it increases model complexity. Thus, directly predicting the final future BEV at the last planning step is a more effective and efficient choice.

Limitation 2 and Question 2: Uncertainty in future predictions and planning.

Apologies for lack of clarity.

• To tackle the uncertainty caused by the multi-modal characteristics of future trajectories, our framework performs prediction and planning while explicitly considering the multi-modal nature of the future. As described in Line 143 of the main paper, the predicted future BEV feature is multi-modal and shares the same modality as the ego feature used for planning. Accordingly, we also predict a multi-modal future BEV representation. For supervision, we adopt a winner-takes-all strategy: the best trajectory and the corresponding future BEV are selected using the same probability generated from the current ego feature, and the loss is computed based on the selected mode.

• To address the uncertainty that may lead to errors in both prediction and planning, we propose the Iterative Scene Modeling and Vehicle Planning module. This design allows for iterative refinement of both future BEV prediction and trajectory planning, resulting in more accurate and consistent outcomes.

• We will revise the paper to include a clearer and more detailed description of this design.

Limitation 3 and Question 3: The choice of loss weights.

• Explanation: Thanks, and we will clarify. The loss balancing among multiple objectives is not specially hand-tuned. Instead, we follow the characteristics of the NAVSIM dataset and prior works [1][2], by scaling the current BEV loss by a factor of 10 and reducing the future BEV loss weight to 0.1. For nuScenes, we use uniform loss weights, setting all to 1.

• Experiment: As suggested, we have now conducted a sensitivity analysis, as shown in the table below. The results indicate that SeerDrive is not particularly sensitive to the choice of loss weights ($\lambda_1$, $\lambda_2$, $\lambda_3$).

Limitation 4: Gains over prior work.

• Explanation: We do not directly build upon previous methods such as DiffusionDrive [1] or WoTE [2], which are orthogonal to our approach. Nevertheless, our framework achieves state-of-the-art performance. Specifically, unlike DiffusionDrive [1], we do not adopt a diffusion-based policy for planning, and unlike WoTE [2], we do not employ multiple future BEV predictions for evaluation. We believe that incorporating such designs into our framework could potentially lead to further performance improvements.

• Gains compared to the baseline: As demonstrated in Table 3 of the main paper, our proposed design improves the PDMS from 87.1 to 88.9, with a notable gain of +1.8. Additionally, as shown in Table 2, we conduct ablation studies on the nuScenes dataset, which clearly validate the effectiveness of our proposed modules through consistent performance improvements.

• Complexity: Despite the iterative inference and dual-task training, our model does not introduce additional complexity. The average inference time is 24 ms and the parameter size is 66M, which is comparable to DiffusionDrive [1] and WoTE [2].

Question 1: Quantitative metric for predicted future BEV.

Great point! The mIoU of the predicted future BEV map on the NAVSIM dataset is 38.87. We will clarify.

[1] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving, CVPR 2025.

[2] End-to-end driving with online trajectory evaluation via bev world model, ICCV 2025.

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

Weakness 1: Comparison with existing methods.

• Thanks for this comment. As discussed in Line 104 of the main paper, LAW [3] leverages world modeling as auxiliary supervision during training. In contrast, our method establishes feature-level interaction between world modeling and end-to-end planning, uniquely enabling iterative and closed-loop communication between the two processes. This design facilitates deeper integration, leading to significant improvements in planning performance.

• We summarize in the table below to highlight the differences between our and prior works. We will further clarify in the revised paper.

Weakness 2: About ablation studies.

• We respectively disagree. As shown in Table 3 of the main paper, we conduct ablation studies on NAVSIM to evaluate the effectiveness of the proposed modules. By introducing Future-Aware Planning and Iterative Scene Modeling and Vehicle Planning, the PDMS (the most important metric) improves from 87.1 to 88.9, with a notable gain of +1.8, establishing a new state of the art (see Table 1 in main paper). Without these modules, the performance (87.1) is inferior to that of DiffusionDrive [1] (88.1), WoTE [2] (88.3), and Hydra-NeXt [5] (88.6).

• As shown in Table 2 of the main paper, we also conduct ablation studies on nuScenes, which demonstrate clear performance gains from our proposed modules. In addition, we have now included another ablation experiment on nuScenes by removing all the proposed modules. The results below show a consistent and significant performance drop, further validating the effectiveness of our design.

Weakness 3: Case studies and discussions.

• Thanks for this suggestion. In addition to the discussion provided in Line 262 of main paper, we will further add an in-depth discussion as follows:

The BEV world model adopts a transformer architecture specifically tailored to our framework, making it both effective and efficient for planning. However, it does not benefit from the generalization capabilities of foundation models. On the other hand, using off-the-shelf foundation models as world models often suffers from slow inference speed and challenges in joint optimization with the planner. Therefore, developing a tightly integrated paradigm of planning and world modeling represents a promising direction for future work.

• We will add in-depth limitation analysis to guide future work in revision.

• Regarding failure cases, beyond those presented in Line 20 of the appendix, we will include additional examples along with more detailed analysis in the revised version of the paper.

Question 2: Parameter size and inference cost.

Thanks, and we will provide it in the revision as follows:

The model size is 66M, and the average inference time is 24 ms on the NAVSIM dataset, under the same configuration as in the main paper.

[1] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving, CVPR 2025.

[2] End-to-end driving with online trajectory evaluation via bev world model, ICCV 2025.

[3] Enhancing end-to-end autonomous driving with latent world model, ICLR 2025.

[4] Navigation-guided sparse scene representation for end-to-end autonomous driving, ICLR 2025.

[5] Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training, arXiv 2503.12030