AdaWM: Adaptive World Model based Planning for Autonomous Driving

Weakness

We thank the reviewer for your suggestions and we address the raised concerns as follows.

A1. (Introduction) We appreciate the reviewer's feedback on the introduction's structure and conciseness. In the revision, we follow your suggestion to streamline the introduction by condensing the background on embodied AI and world models into a concise context-setting paragraph. We also refine the Motivating Example section to more directly connect to our technical contributions of adaptive world models for autonomous driving.

A2. (Related work) We restructure the related work discussion to explicitly highlight three key distinctions of our approach: (1) Unlike existing methods that rely on static models, AdaWM enables real-time adaptation to changing dynamics through our novel LoRA-based world model updates; (2) Current methods like UniAD and VAD cannot handle distribution shifts during deployment, while our approach continuously refines its understanding through online interaction; (3) AdaWM addresses the fundamental challenge of determining when and what to adapt during online interaction by introducing alignment-driven finetuning, which wasn't explored in previous research.

A3. (Baselines) We appreciate the reviewer's comments about baseline comparisons. We want to clarify a crucial distinction in our experimental design: supervised learning methods like UniAD and VAD fundamentally cannot be fine-tuned during online interaction due to their reliance on ground-truth labels, which are unavailable during deployment. This limitation is precisely why we employ reinforcement learning, which makes continuous adaptation possible through environmental interaction without requiring explicit supervision. We follow the standard comparison as conducted in [R1,R2] and provide the VAD and UniAD's performance to demonstrate the advantages of online adaptation. For a fair evaluation of adaptation strategies, we do include comprehensive comparisons with relevant RL-based fine-tuning approaches, including model-only fine-tuning, policy-only fine-tuning, and alternate fine-tuning as ablation baselines (Table 2). These comparisons directly showcase the advantages of our selective adaptation mechanism over existing RL fine-tuning methods. We have provided a more detailed description on the comparison in our revision.

• [R1] Li et al. "Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2).", 2024

• [R2] Jia et al. "Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving", 2024

Questions

A1. (Fintuning) We appreciate the reviewer's question about fine-tuning algorithms. We want to clarify that AdaWM's core contribution lies in its adaptive mechanism for identifying and selectively updating either the world model or policy, rather than in the specific fine-tuning method used. Our approach is agnostic to the choice of parameter-efficient fine-tuning algorithm. We chose NoLa (a LoRa type of finetuning method, line 321 [R1]) for its simplicity and computational efficiency, but other methods like QLoRA could be readily integrated into our framework. This is because AdaWM's key innovation is in determining when and what to adapt based on mismatch detection, while the specific method of parameter updates is modular. Due to the scope of our study, we have focused on demonstrating the effectiveness of our selective adaption strategy rather than comparing different finetuning algorithms.

A2. (Policy Mismatch) Thank you for raising this important point about policy mismatch measurement. We want to clarify that our state distribution metric actually measures the state visitation distribution, which is a direct consequence of policy behavior and it captures the states that the agent visits under its current policy. This is fundamentally different from treating states as mere inputs to the policy. In reinforcement learning, the state visitation distribution $\rho^{\pi}(x)=\sum\_{t}P(x=x\_t\vert \pi)$ is intrinsically linked to policy behavior as it reflects the long-term consequences of action selection. This approach provides a more comprehensive view of policy behavior than just comparing action distributions at individual states, as it captures the cumulative effect of the policy's decisions over time. This is particularly crucial in autonomous driving, where the sequence of visited states directly reflects the vehicle's trajectory and driving behavior.

• [R1] Koohpayegani, Soroush Abbasi, et al. "NOLA: Compressing LoRA using Linear Combination of Random Basis." ICLR'24

We hope that we clarified these very helpful comments. We would be glad to engage in discussion if there are any other questions or parts that need further clarifications.

I have carefully reviewed your responses and revisions, and I believe your work is valuable. However, the presentation of your method could lead to confusion and misinterpretation, which may undermine both the validity of your experiments and the perceived novelty of your approach. As a result, I have temporarily adjusted your score from 3 to 5.

There are still two main points in your paper that require improvement in terms of writing:

1. Logical Coherence: I understand that you use UniAD and VAD to demonstrate the superiority of your method. However, as you mentioned, your core contribution is the adaptive fine-tuning strategy, AdaWM. There are at least two reasons why AdaWM might outperform all baseline methods: (1) Fine-tuned models may perform better on new tasks than those that have only undergone pre-training. (2) Adaptive fine-tuning may be more effective than other fine-tuning approaches. The first point has already been established by Julian et al. in [R1], and while you may reiterate this conclusion in the context of autonomous driving, you still need to demonstrate (2) to fully prove AdaWM's superiority. This is why I previously requested that you supplement your experiments with Table 2. I suggest clearly presenting these two points in the Experiment section and providing separate results to support them, which should help address concerns about the fairness of your baseline selection.

2. Strength of Evidence: In the Related Works section, you simply listed and summarized various papers and their contributions without highlighting their shortcomings in comparison to AdaWM. This omission could make it difficult for readers to grasp the significance of your proposed method. I recommend summarizing the deficiencies of existing methods based on the two points, (1) and (2), I mentioned earlier. Additionally, for the related works on RL fine-tuning or World Model fine-tuning, it would be helpful to explain why these methods are not included in your baselines (e.g., they may not be applicable to autonomous driving tasks) or to clarify how they fall into categories such as alternate fine-tuning, model-only fine-tuning, or policy-only fine-tuning.

[R1] Ryan Julian, Benjamin Swanson, Gaurav Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. In Conference on Robot Learning, pp.2120–2136. PMLR, 2021.

We greatly appreciate Reviewer's thorough review and constructive feedback. We hope that our response has addressed your concerns on our manuscripts and we are more than happy to answer any further questions if needed.

We thank reviewer kwJr for your careful reading and valuation feedback. We address the raised concerns as follows.

A1. (Adaptability) Our work follows previous works and chooses CARLA leaderboard v2 to evaluate the performance. In particular, leaderboard v2 contains large variety of scenarios including freeway and various densities in urban driving. Our experiments are conducted on routes that contains both urban intersections (ROM03, RTD12) and highway scenarios (LTM03, LTD03) and our results in Table 1-3 show consistent performance improvement across these varying complexities.

A2. (TV Distance) We clarify that in our implementation, the total variation distance computation is lightweight, requiring only element-wise operations on probability distributions and our implementation optimizes this calculation through vectorization. The computational overhead is negligible ($<1$ms) compared to the planning cycle ($100ms$).

A3. (Trade-off) We thank the reviewer for pointing out the tradeoff between accuracy and efficiency, especially in the online planning tasks. In our work, AdaWM achieves such balance by following two designs: 1) We adopt NoLa (a LoRa type of finetuning method, line 321 [R1]) which decouples the number of trainable parameters from both the choice of rank and the network architecture. This flexibility allows for fine-tuning with fewer parameters while maintaining accuracy. 2) AdaWM conduct finetuning based on performance feedback. Specifically, at each step, AdaWM estimates the mismatch of model and policy and determines the dominate root cause for the performance degradation and hence guide the finetuning to reduce the gap.

A4. (Testing Scenarios) We thank the reviewer for raising the corner cases and we clarify that the testing scenarios are from CARLA leaderboard v2 tasks (Bench2drive dataset), which focus on the corner cases. For instance, tasks RTD12 requires the ego vehicle is performing an unprotected left turn at an intersection, yielding to oncoming traffic. In the case when the extreme distribution shifts happen and the pre-trained polices do not generalize well (such that from the case with traffic signal to the case of without traffic signal), AdaWM is capable of identify the root cause first (through mismatch identification) followed by alignment-driven finetuning (ref. the comparison of performance RTM03 and LTD03 in Table 1).

• [R1] Koohpayegani, Soroush Abbasi, et al. "NOLA: Compressing LoRA using Linear Combination of Random Basis." ICLR'24

We hope that we clarified these very helpful comments. We would be glad to engage in discussion if there are any other questions or parts that need further clarifications.

Update in the revision: We thank the reviewer for their valuable advice on improving our work. In the revised Appendix G, we include five additional experiments across diverse scenarios in the Bench2Drive (CARLA v2) dataset, including two freeway scenarios: highway cut-in, highway exit, yielding to emergency vehicles, blocked intersections, and vehicle-turning pedestrian routes. The results (Tables 7–9) further validate the effectiveness of alignment-driven fine-tuning, demonstrating consistent evaluation performance for AdaWM as presented in Section 3.

Weakness

A1. (Prediction Quality) We thank the reviewer for your comments and we update the visualizations of the model's prediction quality in Appendix F. Meanwhile, we provide the planning visualization (in gif) in the supplementary files.

A2. (Baselines) We appreciate the reviewer's suggestion regarding comparative methods. Our work specifically examines real-time adaptation for world-model based approach, where Think2Drive [R1] (based on DreamerV3) currently holds state-of-the-art performance on the CARLA leaderboard v2 (Bench2Drive [R2]), making it the most relevant baseline method. Meanwhile, we follow the well-established works [R1,R2] and include the results on UniAD and VAD for comparison and we use the checkpoints provided by Bench2Drive for those methods for a fair comparison.

More importantly, both methods use fundamentally different methodological approaches and such comparison aims to show the advantages of online adaptation in RL-based methods against traditional supervised learning paradigms (which can not adapt during online interaction when the label is not available). This comparative framework allows us to demonstrate the distinct advantages of adaptive planning while maintaining scientific rigor by benchmarking against current top-performing systems.

A3. (Online Adaptation) In response to the reviewer's concern, in the following we elaborate on the distinction between simulation and learned world models in our approach. Our method's world model is learned through interactions with physical engine (specifically CARLA) and represents the agent's internal understanding of dynamics, which is similar to how humans develop mental models through experience. Moreover, the learned world model is different from the simulation and can adapt to new scenarios and handle out-of-distribution issues through online updates, making it fundamentally different from using fixed simulation-based approaches. More importantly, the world model's adaptive nature actually provides an advantage for handling domain gaps since it can continuously refine its predictions and adapt to distribution shifts based on real-world observations, unlike static policies that rely solely on training data. This online adaptation capability is particularly crucial for bridging any simulation-to-real gaps that may emerge during deployment.

Questions

A1. (Finetuning Method) The different adaptation strategies are chosen based on their specific roles. For instance, the learned world model contains core knowledge on the latent representation and should be preserved during finetuning. To this end, the model parameters are decomposed into two lower-dimensional vectors: the latent representation base vector and weight vector. LoRa is suitable for such product structure and can enable efficient updates of dynamics predictions while preserving such core knowledge. Meanwhile, the convex combination approach for policy allows rapid behavior adaptation through reweighting existing skills rather than learning entirely new ones. We thank the reviewer for your suggestions and we have clarified the rationales for our choice in the revision (Section 2.2)

A2. (BEV) We clarify we follow the standard approach as in [R1,R2,R3,R4] (including the bench2drive dataset) to use privileged BEV instead of a BEV former, which is beneficial evaluate the core decision-making capabilities of our planning algorithm.

A3. (Training Tasks) UniAD and VAD baseline are trained on open source Bench2Drive dataset [R1,R2], where all four tasks, ROM03, RTD12, LTM03 and LTD03 are included.

• [R1] Li et al. "Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2).", 2024

• [R2] Jia et al. "Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving", 2024

• [R3] Chen et al., "Learning to drive from a world on rails." ICCV. 2021.

• [R4] Gao et al. "Enhance sample efficiency and robustness of end-to-end urban autonomous driving via semantic masked world model." IEEE Transactions on Intelligent Transportation Systems, 2024.

We hope that we clarified these very helpful comments. We would be glad to engage in discussion if there are any other questions or parts that need further clarifications.

We sincerely thank the reviewer for engaging in the discussion and providing us the detailed feedback. We address the raised concerns as follows.

A1 Driving Smoothness: We appreciate the reviewer's attention to driving smoothness and checking our supplementary material.

Reward Structure: Our reward function comprehensively includes multiple critical factors and the smoothness is not the only factors to consider in terms of the driving performance. We follow the standard approach [R1,R2] and consider six factors: driving smoothness, safety (collision or not), target reaching, velocity, orientation, and traveling time. While we fully agree that smoothness is important for passenger comfort, autonomous driving requires careful balancing of all these factors simultaneously.

Navigation in the Challenging CARLA Scenarios: The trained agents must learn to navigate complex trade-offs in real-world scenarios. An agent could achieve perfect smoothness by driving very slowly, but this would severely compromise travel time and target reaching objectives. In particularly challenging scenarios like unprotected right turns in dense traffic (one of our CARLA leaderboard v2 test cases), the agent must sometimes make decisive movements when safe gaps appear, requiring temporary trade-offs between smoothness and successful task completion.

Our evaluation specifically focuses on highly challenging scenarios from the CARLA leaderboard v2, including complex situations like unprotected right turns with dense traffic and yielding to emergency vehicles. The observed trajectories reflect our agent learning to balance these competing objectives in these demanding scenarios, where perfect smoothness might need to be temporarily compromised to achieve critical safety and task completion goals.

• [R1] Knox, W. Bradley, et al. "Reward (mis) design for autonomous driving." Artificial Intelligence 316 (2023): 103829.

• [R2] Kiran, B. Ravi, et al. "Deep reinforcement learning for autonomous driving: A survey." IEEE Transactions on Intelligent Transportation Systems 23.6 (2021): 4909-4926.

We thank the reviewer for engaging in the discussion. Below are our detailed responses to each concern:

A1. Smoothness.

While our primary focus is on handling challenging CARLA scenarios (e.g., long tail events such as unprotected traffic, pedestrian on the driveway.), we acknowledge the importance of smooth motion in basic driving conditions.

We note that this issue is common when testing in the challenging scenarios, in which other baseline methods are performing even less smooth behaviors comparing with AdaWM. This qualitatively shows that AdaWM has the capability of reducing the jittering. We have included additional qualitative experiments results (in .gif) in simple driving scenarios. The supplementary materials now include new driving records demonstrating smooth performance in basic conditions (ref. Stable Performance Demo/adawm-right_turn_spectator.gif,adwm-going straight,Dreamerv3_RightTurn.gif)

In particular, for the simple scenarios, our framework allows for adaptive behavior refinement through continued training in simpler environments, enabling the system to effectively optimize its performance to obtain "smooth behaviors".

A2. Transferability

We acknowledge that bridging the gap in sim-to-real is an important direction and highly nontrivial, and there are separate line of works on addressing this issue [R1], which is beyond the scope of our work. Our work's scope is to provide insights into handling complex driving scenarios that are rather essential for autonomous driving. To this end, we choose CARLA as our testing bed, which is the current state-of-the-art benchmark for autonomous driving algorithms due to its high-fidelity physics simulation that closely mirrors real-world dynamics. The environment provides realistic sensor models and environmental conditions. Our approach follows well-established validation protocols from previous works in the field and demonstrate the performance of our proposed AdaWM.

• [R1] Yang et al. "Drivearena: A closed-loop generative simulation platform for autonomous driving." arXiv preprint arXiv:2408.00415 (2024).

A3. Benchmark beyond CARLA

We thank the reviewer for suggesting nuPlan as a comparison benchmark. We would like to clarify our choice of CARLA over nuPlan [R2] was aligned with our research objectives.

We follow the SOTA method Think2drive and Bench2drive on close-loop autonomous driving to set-up our experiments on CARLA leaderboard v2 tasks. CARLA enables end-to-end evaluation of direct sensor processing and provides comprehensive testing of interactive behaviors in complex scenarios. It is specifically designed to evaluate long-tail events and serves as the primary benchmark for end-to-end autonomous driving systems.

• [R2] Hallgarten, Marcel, et al. "Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?." arXiv preprint arXiv:2404.07569 (2024).

We hope our responses have sufficiently addressed the issues raised. If there are any remaining concerns or areas requiring further clarification, please don’t hesitate to let us know. We would be happy to provide additional explanations or make further adjustments as needed. Thank you again for your time and effort.

Dear Reviewer,

As the rebuttal period is nearing its conclusion, we would like to kindly follow up to see if there are any additional questions or areas that need further clarification. Thank you once again for your time and valuable feedback.

Authors

A2. (1) Benchmark results. We would like to point out that AdaWM is indeed conducted on the scenarios defined in learderboard v2 and Bench2Drive (Appendix E). For comparison, we include the results reported by Bench2Drive in our revision.

(2) BEV. We clarify we follow the standard approach as in [R1,R2,R3,R4] (including the bench2drive dataset) to use privileged BEV instead of a BEV former, which is beneficial evaluate the core decision-making capabilities of our planning algorithm. While we acknowledge that UniAD and VAD tackle the full autonomous driving stack including perception, both methods do not accept ground truth BEV as it learns hidden representations for BEV rather than using ground truth BEV. Moreover, the main scope of our study is on advancing planning robustness under distributional shifts, indicating that the perception component is only tangential to our primary research objectives. Meanwhile, as evidenced by our comparison with state-of-the-art method Think2Drive, which also use a privileged BEV for their analyses and comparisons, their method suffers from performance degradation despite using privileged BEV. Hence, using a privileged BEV enables a more rigorous evaluation of our core technical contributions in planning adaptation, as it prevents perception errors from obscuring the performance differences in planning strategies.

• [R3] Chen et al., "Learning to drive from a world on rails." ICCV. 2021.

• [R4] Gao et al. "Enhance sample efficiency and robustness of end-to-end urban autonomous driving via semantic masked world model." IEEE Transactions on Intelligent Transportation Systems, 2024.

(3) Evaluation of AdaWM. We first clarify that AdaWM is a world model based approach for planning, while addressing the performance degradation issue during online interaction. We acknowledged that it is feasible to use our proposed framework for finetuning VAD and UniAD, but non-trivial treatments are needed to modify the original methods' structure, as outlined as follows:

• VAD and UniAD are designed as end-to-end networks that directly map sensor inputs to driving actions through supervised learning, without explicitly modeling environmental dynamics, not mentioning the latent dynamics model. This world model is essential for our reinforcement learning-based adaptation, as it enables counterfactual reasoning about different actions and their consequences, allowing the system to learn from simulated experiences during online adaptation.

• More importantly, the supervised learning objectives used in VAD and UniAD are also inherently different from RL approach, where they optimize for prediction accuracy against human demonstrations (i.e., label is needed), whereas our method optimizes for expected future returns through environment interaction (i.e., no label is available).

• Moreover, while theoretically one could attempt to retrofit VAD or UniAD with a world model (such as through knowledge distillation), this would require fundamental architectural changes that would essentially transform them into different methods altogether.

Weakness

We thank the reviewer EsQt for your careful reading and valuable comments. We provide further clarifications below.

A1. (1) Pretraining details. The world model and the planning policy are pretrained based on Dreamer v3 (Hafner et al. 2023). The training details can be found in Appendix E in our revision, and we also outline the key steps as follows:

• World Model Pretraining: As stated in line 195 and 1008, the world model is implemented as a Recurrent State-Space Model (RSSM) to learn the environment dynamics, encoder, reward, continuity and encoder-decoder. This is achieved by optimizing the representation loss, dynamics loss and prediction loss.
• Policy Pretraining: The policy is derived by using actor-critic with 16 steps look-ahead. As stated in line 393, the pre-training is conducted on 1 single V100 for 12 hours.

(2) Tasks configuration. We include the details of the task configuration in Appendix D (line 938). The tasks considered in our work use the same scenarios defined in CARLA leaderboard V2 and Bench2drive. In particular, we use the starting point and the end point appointed by the CARLA leaderboard 2.0 benchmark. In our revision, we provide all the tasks' configurations details in **Appendix D **(line 938).

(3) Metrics. We first clarify that the success rate (SR) used in our comparison is the same as the completion rate, i.e., percentage of the trails that do not have any failure incidents. In our revision, we also updated the terminology to avoid confusion (ref. line 402). We also appreciate the reviewer mentioning the driving score metric in CARLA benchmark. However, we choose not to use Driving score (DS) due to its limitations on evaluating the algorithm performance ( see [R1,R2] below). Specifically, DS is known to have high variance due to its multiplicative nature, where a single infractions disproportionately impact the overall performance evaluation. This high variance makes it less reliable for comparing algorithmic improvements. Meanwhile, we acknowledge Weighted Driving Score (WDS) attempts to address some of these issues, it also has limitations, such as a hand-crafted weighting schemes may not reflect real-world priorities.

For instance, the definition of WDS is $\mathrm{WDS}=\mathrm{RC} * \prod\_i^m \text{ penalty }\_i^{n\_i}$, where RC means route completion rate and penalty and $n_i$ is determined by the number of infractions and scenario density, and these factors are tied to the testing dataset and may not accurately reflect real-world conditions. In our work, we choose TTC, a critical metrics for evaluating human driving safety. In our experiments, we use success rate (SR) to evaluate the overall performance, while TTC reflects safety-critical aspects of driving behavior.

(4) Reward. We emphasize that reward serves as a learning signal and it is a standard way to use the accumulated reward to evaluate the learning performance. We also note that SOTA approaches ([R1, R2]) on this scope also use same accumulated reward for their model learning. For instance, in [R1], the reward is defined as $r=r\_{\text {speed }}+\alpha\_{\text{tr}} r\_{\text {travel }}+\alpha\_{\text {de }} p\_{\text {deviation }}+\alpha\_{\text {st}} c\_{\text{steer}}$. We follow those well-established works and use the reward to incorporate many factors including safety (relevant to TTC) and target reaching (relevant to success rate).

Moreover, we evaluate the final driving performance using widely accepted metrics like TTC, which are distinct from the training reward. As expected, the learning performance is directly related to the driving performance, and it can be seen in Tables 1,2,3, (e.g., Dreamer V3's performance) the learning performance is directly related to the driving performance and the driving performance indeed can suffer from performance degradation when the agent is facing a new scenario.

• [R1] Li et al. "Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2).", 2024

• [R2] Jia et al. "Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving", 2024

Questions

A1. (Reward) We first clarify that reward is used for signaling the learning performance by considering multiple critical factors. We follow the standard approach [R5,R6] and consider six factors: driving smoothness, safety (collision or not), target reaching, velocity, orientation, and traveling time. The degradation of the learning performance indicates that certain properties learned before is degraded, e.g., the safety and/or the driving smoothness is compromised. Furthermore, we evaluate the driving performance using standard metrics such as Time-to-Collision (TTC) and route completion rate (success rate), which are widely accepted in the autonomous driving community and is different from our training reward. In Tables 1-3, it can be seen that the driving performance degradation is indeed related to the training reward degradation.

Moreover, we note that our reward design follows the standard approach [R5,R6] and incorporates several safeguards for robustness. First, our reward function is designed based on fundamental physics-based metrics that remain meaningful across different scenarios and distributions. Second, we empirically validate this robustness through our experiments across diverse scenarios.

• [R5] Knox, W. Bradley, et al. "Reward (mis) design for autonomous driving." Artificial Intelligence 316 (2023): 103829.
• [R6] Kiran, B. Ravi, et al. "Deep reinforcement learning for autonomous driving: A survey." IEEE Transactions on Intelligent Transportation Systems 23.6 (2021): 4909-4926.

A2. (Parameters $C\_1$ and $C\_2$) Parameter $C\_1$ and $C\_2$ are obtained by re-organizing the items on the RHS of the inequality in Theorem 1 and we add the detailed derivation Appendix B.

A3. (Switching strategy ) The purpose of the switching strategy is to address "when should we finetune the dynamics model and planning policy", where the priority of the finetuning is crucial for effective learning. It can be seen in Figure 1 that the finetuning strategies play a pivotal role in the learning process during online interaction. For instance, when model and policy are finetuned alternately (one model finetune followed by one step of policy finetune), the learning process can be hindered greatly since the major root cause for the performance degradation is not addressed effectively during finetuning (e.g., the model mismatch). To compare the different finetuning strategies impact on the leanring progress, we update the comparison Tables 1-3 in our revision.

A4. (Finetuning) We thank the reviewer for pointing out this interesting direction. In response to the question in Weakness 3, we clarify that our current work focuses on demonstrating AdaWM's effectiveness on world model-based policies as they naturally align with our adaptation framework's design principles. VAD and UniAD follow fundamentally different architectural paradigms (end-to-end supervised learning), thus special design is needed to learn a world model that can represent the agent's understanding of the environment, which inevitably will change the VAD and UniAD structure.

Moreover, our approach is general and model-agnostic, where the mathematical formulation and adaptation mechanism can theoretically be integrated with any world model-based driving system. The current evaluation focuses on demonstrating this fundamental effectiveness of real-time adaptation, which is a capability that supervised learning-based methods like VAD and UniAD inherently cannot achieve due to their offline training nature.

We hope that we clarified these very helpful comments. We would be glad to engage in discussion if there are any other questions or parts that need further clarifications.