Thank you for your insightful review and recognition of PWM's strengths and contributions. Detaield responses are as follows.

W1: Concerns of computational efficiency and inadequate multi-sensor input support.

Efficiency: In our response to Reviewer n8oM's comment (W3), we compare the inference costs between state-of-the-art autonomous driving methods. Our inference latency is comparable with MLLM-based competitors with leading driving performance. Detailed results will be added to the final version of our paper.

Multi-sensor / multi-view support: While our current work does not explicitly focus on multi-view or multi-sensor fusion, we consider the autoregressive framework inherently extensible and well-suited for integrating multiple modalities. Although this is not the primary focus of our current study, we regard it as a promising direction for future research. Notably, concurrent works such as Gaia-2 have demonstrated the feasibility of multi-view autoregressive generation, which we find both insightful and inspiring for extending our method.

Q1: Pretraining vs. Forecasting in Performance Gain.

To better disentangle the effects of video pretraining and future frame forecasting, we conduct additional ablation studies on the NAVSIM dataset (containing 100k training samples). As shown in the following table, pre-training is indeed crucial for our model to learn world modeling capabilities. With our proposed forecasting mechanism, we can observe an additional driving performance gain, which indicates that the contributions of forcasting and pre-training are orthogonal to the final performance.

Q2: The choice of token quantity and impact on performance.

In our implementation, the tokenizer used in the baseline model encodes a 128×224 image into a compressed feature map of shape 8×14, resulting in 112 tokens. In our work, we adopt a more compact representation by further compressing the spatial dimensions of the future frames, resulting in a 4×7 grid of tokens (i.e., 28 tokens) per frame.

We acknowledge that exploring the effect of token count on downstream planning performance is a meaningful direction. However, such an investigation requires retraining the tokenizer, followed by full-scale pretraining and finetuning for each token configuration, which is computationally expensive. As a result, we have not yet completed this ablation.

However, we note that prior work [1] has studied the effect of token count in visual tokenizers designed for generation tasks. An interesting observation as shown in Figure 4 of [1] is that increasing the number of tokens improves generation quality. When these tokens are directly used for perceptual tasks (e.g. ImageNet-1K linear probing) the performance tends to degrade. This suggests a potential trade-off: using more tokens helps capture texture-level information, while using fewer tokens yields more compact and semantically concentrated representations, which may be more beneficial for high-level tasks such as planning.

[1] Yu Q, Weber M, Deng X, et al. An image is worth 32 tokens for reconstruction and generation[C]. Advances in Neural Information Processing Systems, 2024, 37: 128940-128966.

Thank you for your detailed analysis and valuable feedback; we greatly appreciate your professional insights. Detaield comments are as follows.

W1: About Performance of Our Method

In our experimental setup, the collision rate metric is relatively crucial. The Nuplan benchmark paper [1] introduces the L2 metric to measure "shape similarity" in imitation learning, but it has a weak correlation with safety. On the other hand, the collision rate metric directly measures the safety of behavior and has a significant impact on the deployment and robustness of autonomous driving systems. In Table 1, we compare our method with the current state-of-the-art autonomous driving algorithms. Our method achieves the most advanced performance in terms of the collision rate metric, while simultaneously achieving state-of-the-art results for the L2 metric.

Our method uses less input information. In the evaluation shown in Table 1, we achieved results comparable to the state-of-the-art methods while using less input information. Specifically, our method only utilizes a single front-view visual input, whereas other methods in the table rely on multi-view visual inputs or even radar inputs.

We highlight the optimal results and enhance the clarity of presentation. We will classify the results based on the amount of input information to help avoid any doubts regarding the performance of our method.

W3: Closed-loop Evaluation

Our experiments on the NAVSIM dataset and simulator demonstrate that the closed-loop results can be effectively validated. NAVSIM is based on real-world data and uses a simulator to unfold the predicted future trajectories into a short-term (typically 4 seconds) small-scale BEV (Bird’s Eye View) simulation. This design is similar to a “Pseudo-Simulation” approach. The PDMS (Predictive Driver Model Score) rating system of NAVSIM is inspired by closed-loop scoring systems, such as the CLS in nuPlan. This rating system, through multi-dimensional comprehensive evaluation, ensures that the open-loop assessment results are highly consistent with the complete closed-loop simulation metrics. We compared the open-loop evaluation of NAVSIM (PDMS) with the real closed-loop simulation (CLS), and the results showed that PDMS achieved a Pearson and Spearman correlation coefficient of approximately 0.8 (significantly higher than the traditional prediction error-based score of ~0.7) [2].

Undoubtedly, it is essential to conduct experiments on widely recognized simulation platforms such as CARLA. But due to time and engineering constraints, the relevant experiments and results are still under development. We will provide the complete experimental results in the near future.

W2: Typos

Thank you for your pointing out the typo. The expression α > β in the main paper is correct. In Appendix B.1, it should be α ≤ β.

[1] Karnchanachari N, Geromichalos D, Tan K S, et al. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving[C]. 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 629-636.

[2]Dauner D, Hallgarten M, Li T, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking[C]. Advances in Neural Information Processing Systems, 2024, 37: 28706-28719.

We sincerely appreciate the reviewer’s thoughtful and constructive suggestions. Detailed responses are as follows.

W1: Inference cost

The inference cost of autonomous driving systems is indeed a crucial metric for evaluating the safety and practicality of the system. In our response to Reviewer n8oM's comment on w3, we provided a table comparing the inference costs between modern state-of-the-art autonomous driving methods. Our inference latency is comparable with MLLM-based competitors with leading driving performance. Detailed results will be added to the final version of our paper.

W2: Differences from previous work

DrivingGPT and Doe-1 employ an autoregressive approach that alternates between multiple modalities for training and inference. This contrasts significantly with the staged training paradigm proposed in our work, which first focuses on world knowledge learning without actions, followed by end-to-end planning task learning with future predictions. The work of VaViM and VaVAM centers around integrating an action expert model with a video generation model for application in autonomous driving. During the pretraining phase, VaViM and VaVAM utilize an autoregressive training method for video generation, but their primary focus is on representation modeling and learning. In the fine-tuning stage, the video generation capability is not explicitly leveraged in planning tasks, and their inference process does not involve predicting future states. In contrast, our approach requires future state prediction during inference, which is then used as the basis for planning action predictions.

W3/Q1: Interleaved variant

We understand your intent as conducting an ablation experiment to verify the interdependencies between the world model's future predictions and the autonomous driving planning tasks. In response, we provide in Table 4 the impact of different levels of future state predictions during fine-tuning on downstream planning tasks. It is evident that at the appropriate level of future state prediction (10 frames), the planning task achieves optimal performance. We analyze this result as follows: at lower levels of future state prediction, the model is unable to predict future states accurately or sufficiently, leading to a decline in planning task performance; at higher levels, the uncertainty of the model's future state predictions becomes too high, causing incorrect guidance for the planning task.

W5: Discusson on MLLM

Thank you for your suggestion. Undoubtedly, the development of MLLMs forms a crucial foundation for our work. We will include a dedicated section on related work in the main paper.

Q2:Bypass visual prediction

Thank you for your question. Our framework proposes the paradigm of using a world model to predict future states, while simultaneously performing planning tasks based on the predicted future states. This approach has not been fully explored in the field of autonomous driving, which is the primary motivation behind our PWM proposal. The paradigm you mentioned, which does not explicitly predict future states but directly outputs actions, has been discussed in works such as VaViM and VaVAM. Additionally, in our work, Table 4 presents the performance of PWM in planning tasks without visual predictions in the first row.

W4: Typos

Thank you for your careful feedback. We will correct these typos in the main paper.

Dear authors,

I appreciate your efforts in addressing my concerns. After reading the rebuttal and other review comments, I find that most of my issues have been solved, and I have no further major concerns. In the revision, please rephrase the content regarding W2 according to your response, it will be clearer for readers. Thank you!

I will maintain my initial rating, as I believe PWM does present a valuable, unified, and scalable framework for autonomous driving research.

We are glad that the strengths (technically sound, consistent performance improvements, thorough ablations, practical efficiency) of our work can be acknowledged by the reviewer. We thank the rveiwer for the constructive suggestions. Detailed responses are as follows.

W1/Q1: PWM vs. Drive-OccWorld

1. Motivation: Drive-OccWorld aims to design an independent occupancy-based world model to support control interactions. In contrast, our Policy World Model focuses on integrating the world model and the policy into a unified architecture, which can leverage its pretrained world knowledge to inform and guide decision-making. This integration enables planning to benefit directly from future predictions generated by the world model itself.

2. Theoretical Analysis: As shown in Fig.1-2 of the Drive-OccWorld paper, their occupancy prediction model and trajectory planner are two stand-alone model, which belongs to the type-(a) paradigm shown in our Fig.1 (a), albeit with occupancy grids instead of video inputs. Our approach aligns with the type-(c) paradigm in our Fig.1 (c), where the PWM model can not only simulate the future states as a world model but also acts as a policy model for decision based on the forcasted future.

3. Experimental Performance: We conduct additional experiments on nuScenes, where all methods are evaluated under the same settings: without ego-state inputs and using the BEV-Planner evaluation protocol. Drive-WM and OccWorld results are taken from the OccWorld paper. OccWorld achieves the best L2 error, possibly benefiting from its strategy of sampling multiple candidate trajectories, which could increase the likelihood of overlapping with the ground-truth. In contrast, our PWM predicts a single trajectory without candidate selection. Despite this, PWM achieves the lowest collision rate, indicating that our unified model produces safer and more consistent predictions.

W2/W5/Q2: Clarification on Our Unified Generation-Planning Framework. We appreciate the reviewer’s observation regarding the design of our generation-planning framework. However, we would like to clarify a key misunderstanding: in our unified framework, future state prediction and trajectory planning are not handled as independent stages nor via separate models. Instead, we enable the world model itself to act as an executor for planning, where both the prediction of video frames and the prediction of actions are accomplished by a single model. Specifically, as illustrated in Figure 2 of the paper, our lightweight action decoder does not infer trajectories directly from predicted videos. Instead, it serves to map the action embeddings output by the Policy World Model into a low-dimensional trajectory space. These action embeddings are generated conditioned on the model’s autoregressive prediction context, which includes predicted textual descriptions and future forecasting embeddings.

W3: Concerns About Use of OpenDV-Youtube Several existing methods also incorporate external datasets to enhance image generation or captioning capabilities before comparing against traditional baselines such as UniAD and VAD. For example, GenAD and VaVAM utilize the OpenDV-Youtube dataset, Doe-1 leverages BDD100k, and DrivingGPT includes data from the nuPlan dataset. We use the OpenDV-Youtube dataset primarily to equip the model with future video prediction capabilities, which is a core component of our approach. This dataset is only used during the action-free pretraining phase and is not directly involved in optimizing planning performance. Meanwhile, the pretraining process for video generation training does not require annotations, which is a key advantage of our method in terms of annotation costs and scalibility. Moreover, we believe a key value of world models lies in their ability to model environmental dynamics in open-world scenarios. Achieving this goal naturally requires learning from rich and diverse visual data to capture temporal regularities, much like how language models benefit from large-scale pretraining. Our approach follows this trajectory to advance the development of planning-centric world models.

W4: Concern regarding the necessity of the multimodal component. The direct performance gain from textual generation in our current implementation is indeed limited. However, textual generation is not intended for performance improvement but rather serves broader goals—aligning with trends (e.g., DriveLM, Omnidrive) using multimodal models to boost interpretability and human-AI interaction in autonomous driving. Motivated by these trends, our method includes a language generation component to demonstrate the model’s ability to produce high-level textual summaries of driving scenes and decision contexts, aiding explainability and future human-in-the-loop applications. We will provide more explanations in the final version.

Q3: More ablation results without extra data (OpenDV-YouTube) pre-training. We would like to clarify the meaning of different configurations in Table 3. Specifically, “Pretrain” refers to training on the Open-Youtube dataset, while “L_DFL-p” and “L_DFL-f” denote whether the Dynamic Focal Loss is applied during the pretraining and fine-tuning phases. Accordingly, L_DFL-p becomes inapplicable without pretraining, as its application is predicated on the existence of a pretraining stage. Therefore, the first two configurations listed in Q3 (with Pretrain = ×, L_DFL-p = √) are not feasible in our framework. To address the concern regarding the necessity of extra data, we provide an ablation where we disable pretraining (Pretrain = ×, L_DFL-p = ×) and only apply L_DFL-f (L_DFL-f = √) during fine-tuning on the nuScenes dataset. This configuration shows that even without large-scale pretraining, applying L_DFL during fine-tuning alone improves future frame prediction and leads to measurable gains in planning performance. These results support the effectiveness of our loss design independent of external data scale.

Q4：Compute and latency details. We compare the model size and inference latency (including token decoding) of our method with both traditional lightweight models (e.g., UniAD, DiffusionDrive, LAW, Drive-OccWorld) and multimodal large models (e.g., DriveLM-Agent) on a single NVIDIA 4090 GPU using the nuScenes dataset. As expected, multimodal large models such as DriveLM-Agent and Ours generally have larger model sizes than lightweight baselines, given their use of large-scale language or vision-language backbones. In terms of inference latency, in addition to differences in model scale, variations also stem from different output requirements. For instance, Drive-OccWorld involves iterative reconstruction of future occupancy grids before planning, which adds substantial computational overhead. LLM-Driver and DriveLM-Agent require autoregressive generation of relatively long natural language outputs prior to decision making. Our method, while generating shorter textual outputs, also produces compact future-frame embeddings.

Q5: Failure‑case analysis. Due to restrictions on uploading external links or supplementary PDFs on the rebuttal platform, we provide a textual description of failure cases where future-frame hallucinations have misled the planner. These failure cases can be mainly attributed to Limited Field-of-View Induced Hallucination. As our method operates on monocular observations, certain scenarios—such as intersections or turns—pose a challenge. The limited field of view may prevent the model from capturing distant or occluded dynamic agents (e.g., oncoming vehicles), leading to hallucinated clear paths and subsequently unsafe planning decisions. Future work will enhance dynamic capture via advanced modules and adopt multi-view inputs to reduce hallucinations and improve safety.

Thank the authors for their further clarification. I now have a clearer understanding of the claimed ‘collaborative’. In my opinion, it's a trade-off. Future RGB frame predictions in an action-free pattern can be pre-trained on large-scale datasets. However, when performing end-to-end planning, I think considering the impact of future selected actions, such as $P(D_{t+i}|D_{1:t+i-1}, A_{t:t+i-1})$, is a natural idea and may lead to better planning performance.

I suggest the authors include this discussion in the revised version, especially expanding the end-to-end planning section (currently only 11 lines, should include at least optimization objectives and state-action interaction process) to better demonstrate the designed collaborative state-action prediction, considering its significance and contribution.