Enhancing End-to-End Autonomous Driving with Latent World Model

Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.

Response to W1

It seems there may be a misunderstanding regarding the primary purpose of our work. While prior studies, such as GAIA-1, ADriver-I, and Drive-WM, have indeed explored self-supervised world models for autonomous driving, their focus is primarily on future image generation. These works employ diffusion models to generate realistic future scene images, which are then utilized to facilitate reasonable driving decisions. In contrast, our work emphasizes using a world model to enhance end-to-end driving by improving scene feature learning. To the best of our knowledge, we are the first to reveal that leveraging the world model to perform self-supervised future prediction tasks can help models learn better scene representation, thus improving end-to-end driving performance. While it is possible to test whether an image-based world model could similarly enhance scene feature learning, there are some issues.

Integrating an image-based world model into an end-to-end driving system is highly inefficient. Existing image-based world models rely on diffusion processes to generate high-resolution images, which can take several seconds to produce a set of six-view images for a future scene. This process demands immense GPU memory and time. To address this, we propose a latent world model that predicts latent features instead of high-resolution images for future scenes. This approach is significantly more efficient in terms of time and memory usage. Our experiments demonstrate that the latent world model effectively enhances end-to-end driving performance.

In summary, the novelty of our work lies in leveraging a world model to enhance feature representation learning, thus improving end-to-end driving performance. We greatly appreciate the reviewer's feedback and have revised the introduction section to clarify our motivation and prevent misunderstandings.

Response to W2

To address this concern, we conduct the experiments in the following table and find that performance gains diminish over longer time horizons. Specifically, we conducted an experiment predicting latent features 10 seconds into the future. The results indicate a significant decay in performance, which completely diminishes the gain brought by the world model. This experiment demonstrates that it is indeed challenging to predict the distant future.

However, a 3-second horizon is not long enough to eliminate the performance gains brought by the world model. The reasons are discussed below. Low-level information in images can undergo dramatic changes within a relatively small horizon. However, high-level driving features tend to remain relatively stable over short horizons. For example, the road layout surrounding the ego vehicle is generally stable within a short horizon. To validate this, we provide the feature L2 distances. The L2 distances between the features at 1.5s, 3s, and 10s into the future and the current features are 0.0263, 0.0420, and 0.0752, respectively. These results demonstrate that the feature changes within 3 seconds are significantly smaller than those within 10 seconds. As a result, we observe substantial performance gains over horizons such as 1.5s and 3s, which makes their performances seem similar.

We thank the reviewer for highlighting this critical aspect of our work. We have incorporated these experiments and insights into the main paper as they provide a deeper understanding of future latent prediction.

Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.

Response to Q1

We introduce the shape of Visual Latents as follows. For the perception-based framework, the Visual Latents need to support tasks such as detection and map construction, resulting in a relatively large shape. Specifically, the BEV feature map has dimensions of [25, 25, 256], where h = w = 25 and C = 256. Flattening this feature map results in Visual Latents of shape [625, 256]. In contrast, the perception-free framework does not require perception tasks, focusing on learning high-level planning features. Consequently, the shape of the Visual Latents is smaller. The shape is [36, 256]. With 6 views, each view contains 6 feature vectors, and each feature vector has 256 channels. We further conduct an ablation study on the number of Visual Latents, summarized in the table below. The results indicate that increasing the number of Visual Latents results in a higher L2 displacement error but a lower collision rate. We appreciate your insight into the importance of Visual Latent shapes.

Response to Q2

Thank you for the insightful feedback. We considered conducting experiments with multi-trajectory supervision using the CARLA simulator. However, collecting future frames for multiple trajectories through the CARLA API was very time-consuming. This is due to CARLA's limited rendering efficiency, which results in slow image generation, especially when producing multiple sets of future frames for different trajectories. As a result, it becomes challenging to complete data collection, model training, and testing within a two-week timeframe. We sincerely appreciate the reviewer's valuable suggestion and plan to investigate this direction further in future work.

Meanwhile, as an alternative, we can provide a similar experiment that involves predicting multiple future latents, as detailed in ''Response to Q3.1''. We believe that their impacts on performances will be similar, as both of them introduce more supervision to enhance feature learning.

Response to Q3.1

To address this concern, we utilize the latent world model to predict multiple future frame latents, with the results presented in the following tables. Due to the increased time required for extracting and predicting multiple future latents, we conduct this experiment using only the front-view camera to facilitate faster training. In detail, the future frame latents are predicted in an auto-regressive manner. For example, we first predicted the latent for 1.5 seconds into the future, then used this predicted latent to further predict the latent for 3 seconds into the future. The latent world model shares the same weights throughout this process. The results demonstrate that predicting multiple future latents further improves performance, as it introduces additional supervision. Thank you for your suggestion. We have incorporated these experiments into the revised paper.

Response to Q3.2

Building on ''Response to Q3.1'', we conduct further experiments to predict multiple future frame latents while incorporating multiple input frame latents to leverage temporal information more effectively. Specifically, we adopted a two-stage training paradigm for improved convergence, inspired by SOLOFusion [1]. In the first stage, we trained the model using single input frame latents for 12 epochs. This corresponds to the model denoted as "1.5s, 3s (autoregressive)" in the table in ''Response to Q3.1''. In the second stage, we fine-tuned the model for an additional 6 epochs, now using two input frame latents, from the current and 1.5s ago. The results are summarized in the table below.

The baseline (first row) represents the model fine-tuned using only a single current latent. In contrast, the second row corresponds to the model fine-tuned with two input frame latents. The latter achieves significantly better performance. This highlights the crucial role of temporal information in autonomous driving, as discussed in our introduction. Thank you for your insightful suggestions! We have incorporated these experiments into the revised paper.

Response to Q4

Yes, as discussed in ''Response to Q3.1'', we employ an auto-regressive paradigm to predict multiple future latents. This approach involves step-by-step prediction, where each predicted latent is used as input for the subsequent step. The latent world model is used with shared weights throughout the process.

[1]: Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection

Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.

Response to Q1

The view selection strategy can be adapted to different numbers of cameras using the following designs:

1. when the total number of cameras changes:

   We only need to adjust the number of selection queries when the total number of cameras changes. The reasons are as follows. First, we do not need to change the main pipeline as both our feature extractor and latent world model are view-independent. View-independent means we extract each view’s feature and predict the future feature of this view independently. As a result, the overall framework is not affected by the change of the total number of cameras. Therefore, we only need to adjust the number of selection queries according to the total number of cameras. For example, we have 6 cameras now. We select the front camera and also select another camera out of five cameras, so the number of selection queries is C(5, 1)=5, where C(n, k) represents the number of combinations of selecting k items from n items. When we have 7 cameras, the number of selection queries will be C(6,1)=6.

2. when the selected number of cameras changes:

   We address this issue by increasing the number of selection queries to encompass all possible configurations of selected cameras. For instance, with six cameras, there are 2^6 = 64 possible configurations, where each camera can either be selected or not. To account for all these configurations, we leverage 64 selection queries, with each query corresponding to a specific configuration. The corresponding 64 ground truth rewards can be generated using the original pipeline. By training the model to predict rewards for all these configurations, it can provide predicted rewards for any combination of selected cameras at each time step. This approach enables the framework to adapt seamlessly to scenarios where the number of selected cameras varies over time. Moreover, the framework can determine the optimal number of cameras to select for the next time step based on these predicted rewards.

Response to Q2

Thank you for pointing this out. The revised sentence is: “For the closed-loop benchmark, we use ….”

Response to Q3

To further enhance the system with additional supervision, we incorporate a detection task for bounding box supervision and a map construction task for map supervision into our framework. The results are in the following table. It shows that these additional supervision tasks (detection and map constructions) can further improve the system.

Response to Q4

The performance when the view selection strategy is included is in Table 7 of the main paper. We also provide a copy of Table 7 as follows for your convenience. The second row shows the performance when the View Selection strategy is included.

Response to Q5

There might be a misunderstanding here, as the settings for "Six Views" and "Front + GT Views" in the table represent two different setups. The "Front + GT Views" experiment leverages GT waypoint information as a pilot study to explore the performance upper bound, whereas the "Six Views" experiment does not utilize GT information. Specifically, the "Front + GT Views" experiment involves selecting the front camera and pairing it with one of the remaining five cameras, creating five combinations. Each combination generates a set of waypoints, and among these five sets, the one closest to the GT waypoints (based on their distances) is selected. This process inherently utilizes GT information, which explains its superior performance compared to the "Six Views" setting.

Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.

Response to W1

We provide the latency results in the following table. The latency of our model is tested on a single RTX 3090 GPU. For our perception-based framework, we adopt VAD-Tiny as the baseline. Since the latent world model is only utilized during training, the inference latency remains identical to VAD-Tiny. For the perception-free framework, our method achieves a latency of 51.2 ms, corresponding to ~20 FPS, which meets the requirements for real-time deployment on edge computing devices. Thank you for your insightful comment. We have incorporated the latency results into Table 1 in the revised version.

Response to W2

We provide the results under various weather and lighting conditions in the table below. Our findings demonstrate that the model exhibits robustness to diverse weather and lighting scenarios. We sincerely thank the reviewer for encouraging us to explore this aspect, which makes our work more comprehensive. We have incorporated these results in the revised version.

Response to W3

These references significantly enrich the depth of our work, and we have incorporated them into the related works section in the revised version. The corresponding sentences are as follows: DriveVLM [1] utilizes Vision-Language Models (VLMs) to improve comprehension of complex scenes and to optimize planning processes. EMMA [2] integrates multiple driving-related tasks into a unified language framework, employing task-specific prompts to generate the respective outputs. VLP [3] advances autonomous driving by enhancing the foundational memory of the source model and improving the contextual awareness of self-driving systems. OmniDrive [4] presents a comprehensive approach that ensures robust alignment between agent models and the requirements of 3D driving scenarios.

[1] DriveVLM: The convergence of autonomous driving and large vision-language models

[2] EMMA: End-to-End Multimodal Model for Autonomous Driving

[3] VLP: Vision Language Planning for Autonomous Driving

[4] OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning

We sincerely appreciate your timely and insightful feedback. Your experiments on latency and challenging environments have been particularly constructive, and they have provided valuable guidance for improving our work. We are dedicated to further exploring challenging environments in our future work.