Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)

Thanks for your acknowledgement and kind advice. Regarding your concerns, we give responses below:

Weakness 1,3，Question 1, Discussion on Transferability in Real-world Autonomous Driving

• We appreciate the reviewer’s concerns regarding the Transferability in Real-world Autonomous Driving.

• About the Transferability of RL

  • Recent breakthroughs of reinforcement learning (RL) in domains such as LLMs[1], VLMs[2] and AlphaFold[3], has shown that RL's unique advantages compared to SFT/imitation learning. While real-world deployment of RL remains challenging, the community has made significant efforts in high-fidelity simulation generation[4] and reconstruction[5,6], though their readiness for large-scale RL deployment remains limited.

  • Our work builds upon the trend of RL, serving as a preliminary study of applying RL to end-to-end autonomous driving. Due to the unavailablility of real-world-level authentic simulator, we use CARLA but we argue that the scientific problem studied and discoveries of the early exploration in this work are general and worth-sharing for the community.

• About the Transferability of Raw2Drive

  • Raw2Drive utilizes a dual-stream world model to effectively decouple perception and decision-making. On the decision side, trajectory prediction is conducted based on structured perception outputs (e.g., BEV or compact representations)—a widely adopted practice in both academia and industry. On the perception side, the emphasis is on enhancing the accuracy of these BEV-based representations.

  • By leveraging the world model, our paradigm enables a fully differentiable pipeline while preserving a separation between perception and planning.

  • Furthermore, with recent progress in high-fidelity scene reconstruction[4] and simulation technologies [5,6], a practical deployment path is to pretrain the world model via imitation learning and fine-tune the policy with reinforcement learning—an increasingly feasible strategy in real-world and industrial settings.

[1] 2024, Technical Report, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

[2] 2025, Arxiv, VLM-R1: A stable and generalizable R1-style Large Vision-Language Model

[3] 2021, Nature, Highly accurate protein structure prediction with AlphaFold

[4] 2025，ICCV, HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

[5] 2025, ICCV, BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting

[6] 2025, Arxiv, ReSim: Reliable World Simulation for Autonomous Driving

Weakness 2, Question 3, Explanation of Stochastic State

• In our model, the stochastic latent state plays a key role in capturing uncertainty in world model predictions. It consists of multiple categorical variables(num=32), each represented as a one-hot vector(dim=32). These variables are parameterized by logits output from the world model. During training, the stochastic state is sampled using a Gumbel-Softmax estimator to maintain differentiability. At inference, the mode (argmax) is used to select one-hot values. The final stochastic state is the concatenation of these one-hot vectors, serving as the discrete latent representation of the environment dynamics.

Weakness 3, Results about Real-time Inference

• Thank you for the suggestion. We have added latency analysis for each module. In end-to-end autonomous driving, the perception backbone (e.g., surround-view image encoder) typically dominates the overall latency. As shown below, our world model and policy are highly efficient (each under 2ms), while the raw sensor stream is mainly bottlenecked by the vision encoder (e.g., BEVFormer).

Weakness 4 and Question 4, Discussion on Stability of World Model Training(Privileged vs Raw Sensor)

• We appreciate the reviewer’s concern. The observed training instability mainly affects the Raw Sensor World Model, which learns from high-dimensional, partial visual inputs. In contrast, the Privileged World Model is trained on structured BEV states that are low-dimensional, semantically aligned, and less ambiguous—leading to much faster and more stable convergence without perceptual noise or occlusion.

• As shown in Figure 7 (Appendix F), removing rollout guidance causes the Raw Sensor model’s loss to oscillate and fail to converge. We further provide a visualization in Appendix G.2, where adjacent frames appear visually similar but correspond to drastically different rewards. This semantic ambiguity makes it hard for the Raw Sensor World Model to capture consistent dynamics, reinforcing the need for privileged guidance during training.

Question 2, Discussion on Privileged Input in non-simulated settings

• Thank you for the question. In non-simulated settings, privileged inputs are typically obtained from perception outputs—either generated by state-of-the-art detection models[7][8] or manually annotated—both of which are common practices in industry. While ground-truth semantics or HD maps may not be perfectly available, this is also the case in many real-world trajectory prediction setups[9], where models are trained on perception outputs that are imperfect but sufficiently reliable.

Our framework follows this convention and can incorporate such perception-derived privileged inputs accordingly. This makes our privileged stream design practical and compatible with real-world deployment pipelines.

[7]2024, CVPR, UniMODE: Unified Monocular 3D Object Detection

[8]2024, ICLR, LaneSegNet: Map Learning with Lane Segment Perception for Autonomous Driving

[9]2025, CVPR, Truncated Diffusion Model for Real-Time End-to-End Autonomous Driving

Thank you again for your valuable comments. We hope our responses have effectively addressed the concerns raised. If there are any remaining concerns, we would be grateful for further clarification.

We sincerely thank you for your valuable comments, which have greatly helped us improve the quality and clarity of our work. We fully agree with your insightful perspective on the sim2real gap—a critical challenge for RL-based driving systems. While CARLA remains the only viable closed-loop simulator suitable for RL research at present, our work focuses on policy learning and introduces a dual-stream world model design that is conceptually decoupled from the underlying simulator. As generative or reconstruction-based simulators continue to evolve, we believe that exploring end-to-end RL within these emerging frameworks will become increasingly practical and impactful. We will incorporate these valuable discussions into the revision. Thank you again for your thoughtful feedback!

Thanks for your acknowledgement and kind advice. Regarding your concerns, we give responses below:

Weakness 1, Question 6, Discussion on Transferability and Explainability in Real-world Autonomous Driving

• We appreciate the reviewer’s concerns regarding the Transferability and Explainability in Real-world Autonomous Driving.

• Recent breakthroughs of reinforcement learning (RL) in domains such as LLMs[1], VLMs[2] and AlphaFold[3], has shown that RL's unique advantages compared to SFT/imitation learning. While real-world deployment of RL remains challenging, the community has made significant efforts in high-fidelity simulation generation[4] and reconstruction[5,6], though their readiness for large-scale RL deployment remains limited.

• Our work builds upon the trend of RL, serving as a preliminary study of applying RL to end-to-end autonomous driving. Due to the unavailablility of real-world-level authentic simulator, we use CARLA but we argue that the scientific problem studied and discoveries of the early exploration in this work are general and worth-sharing for the community.

• Regarding explainability(Question 6), as shown in Figure 8, we reconstruct BEV representations instead of expensive RGB images. This not only provides supervision for training the world model but also enhances interpretability by offering a structured and semantically meaningful intermediate representation.

[1] 2024, Technical Report, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

[2] 2025, Arxiv, VLM-R1: A stable and generalizable R1-style Large Vision-Language Model

[3] 2021, Nature, Highly accurate protein structure prediction with AlphaFold

[4] 2025，ICCV, HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

[5] 2025, ICCV, BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting

[6] 2025, Arxiv, ReSim: Reliable World Simulation for Autonomous Driving

Weakness 2, Additional Ablations and Discussion on Alignment

• Thanks for your suggestions. In fact, Table 7 presents ablations on the three types of latent states(Abstract-State Alignment).

• About Spatial-Temporal Alignment

  • In addition, we add results on Spatial-Temporal Alignment (based on Dev10). Spatial Alignment ensures consistency between image representations and BEV representations—removing it is similar to “driving blind.” Temporal Alignment maintains the consistency of future predictions across time. Both components are essential and complementary for stable world model rollout.

• About Encoder Sharing
  • Notably, because the privileged and raw sensor streams have different inputs, their encoders cannot share parameters.

Weakness 3, Clarification on World Model Alignment during Online RL Updates

• We appreciate the reviewer’s concern, but we respectfully disagree with this point. During online RL updates, there is no misalignment between the world models.

• In the first stage, similar to standard model-based RL approaches [7], the Privileged World Model and Privileged Policy are updated alternately. In the second stage, as shown in Figure 4, the Privileged World Model is frozen, ensuring a stable supervision signal for optimizing the Raw World Model. Therefore, no misalignment occurs during training.

[7] 2025, Nature, Mastering Diverse Domains through World Models

Weakness 4, Clarification on the Comparison with AdaWM

• In fact, as shown in Table 1, we include a comparison with AdaWM[8]. It is important to note that AdaWM is not an end-to-end autonomous driving method. Its benchmark - ROM03, a derivative of CARLA v1, which lacks complex corner cases(CARLA v2, such as Bench2Drive) and is not open sourced yet.

[8] 2025, ICLR, AdaWM: adaptive world model based planning for autonomous driving

Questions 1, Clarification on the Necessity of Representation Alignment

• As described in Sec. 3.1 and Sec. 3.2, the dual-stream world model processes two distinct input modalities: privileged input consists of time-sequenced BEV semantic masks, while raw sensor input includes multi-view images and IMU data. Due to their fundamentally different input formats and encoder architectures, simple fine-tuning between the two streams is not feasible.

• As shown in Figure 7 (Appendix F), removing rollout guidance causes the Raw Sensor model’s loss to oscillate and fail to converge. Since the world model is used for future prediction, representation alignment is essential to ensure consistency between the two streams during rollout.

Question 2, Clarification on World Model Stability During Policy Fine-Tuning

• In our dual-stream design, the first stage follows a standard model-based RL paradigm[7], where the Privileged World Model and Privileged Policy are updated alternately to ensure stable training. In the second stage, as shown in Figure 5, both World Models are frozen during raw policy fine-tuning. As a result, the predictive accuracy of the world model remains unaffected, and there is no degradation due to policy distribution shift.

Questions 3, Clarification on Single Encoder to Serve both the World Model and RL Agent

• This is a common practice in model-based RL[7,8,9]. During training, the RL agent interacts with the world model instead of the simulator, significantly reducing simulation overhead and improving training efficiency. One of the key roles of the world model is to predict the next observation, which is then used by the RL agent for decision-making. Therefore, using a single shared encoder for both the world model and the RL agent is not only reasonable but necessary to maintain consistency in representation and ensure effective policy learning.

[8] 2020, ICLR, Dream to Control: Learning Behaviors by Latent Imagination

[9] 2021, ICLR, Mastering Atari with Discrete World Models

Questions 4, Discussion on Fairness of Comparison with IL Methods and Interaction Nums

• As noted in Sec. 4.1, we adopt the same 1000 training routes from Bench2Drive to ensure fairness. However, as an online learning paradigm, RL naturally collects a larger number of interactions—both successful and failed—to iteratively refine its policy. This trial-and-error process is a core advantage of RL over offline imitation learning.

• In Bench2Drive-Base, the 1000 routes yield a total of 250k samples. To further analyze interaction efficiency, we also include additional ablations. Similar to early work such as MaRLn [10], which trained and tested on the same set of routes(albeit with limited generalization), we show that while training on fixed routes is feasible, generalization remains a challenge.

• Notably, Raw2Drive enables better generalization to unseen scenarios by learning to predict future dynamics—this requires more interactions. As the number of interactions increases from 250k to 750k and 1M, we observe performance gains. Both successful and failed trajectories are crucial for learning a robust world model. However, we acknowledge that training time also increases accordingly.

[10]2020, CVPR, End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances

[11]2023, CVPR, Planning-oriented Autonomous Driving

Questions 5, Discussion and Ablation on Generalization Across Input Modalities and Sensor Configurations

• Thanks for your suggestions. please refer to the answer: Reviewer #ps71 (Questions b, Additional Results on LiDAR-Based Study)

Thanks again for your time and advice!

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Weakness 1, Discussion and Clarification on Novelty and Student-Teacher Paradigm

• Thank you for the helpful suggestion. Our core focus is reinforcement learning in end-to-end autonomous driving. With recent progress in model-based RL [1,2], our work introduces two key novelties:

  • Successfully achieve end-to-end RL To our best knowledg, Raw2Drive is the first successfully achieve end-to-end RL with performance significantly better than the end-to-end IL of the same period.
  • Dual-stream World Model as a Bridge: We use a dual-stream world model to decouple perception and decision-making. This design simplifies the joint optimization problem, which is often difficult to converge in end-to-end autonomous driving.
  • Dual-stream World Model Alignment: Some student-teacher methods such as CaT [3] and DriveAdapter [4] are based on imitation learning. While they achieve strong performance in CARLA v1, they struggle in corner cases under the more challenging Bench2Drive (CARLA v2) benchmark. Moreover，Unlike standard teacher-forcing strategies, as shown in Figure 6 in manuscript, future prediction in driving involves both stochastic (e.g., other agents’ intentions) and deterministic (e.g., ego state) components over time and space. We propose Spatial-Temporal Alignment and Abstract-State Alignment to ensure consistency between the two streams during rollout.

• We will revise the final version of the paper accordingly, adding citations to better emphasize these contributions and improving the discussion of related methods.

[1] 2020, ICLR, Dream to Control: Learning Behaviors by Latent Imagination

[2] 2021, ICLR, Mastering Atari with Discrete World Models

[3] 2023, CVPR, Coaching a Teachable Student

[4] 2023, ICCV, DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving

Questions a, Clarification on Raw Sensor Inputs

• Thank you for the question. As stated in Sec. 3.2 (Line 97), the raw sensor input includes multi-view images and IMU data. For visual simplicity, the IMU input is omitted from the figure but is fully incorporated in the model implementation. We will update in the final version.

Questions b, Additional Results on LiDAR-Based Study

• Regarding the LiDAR sensor, this highlights another advantage of our method-extensibility. Our dual-stream world model decouples perception and decision-making, where the perception module focuses on enhancing the accuracy of BEV-based representations. To further validate this extensibility, we include additional experiments using LiDAR inputs. Specifically, we replace BEVFormer[5] with BEVFusion[6]. Incorporating LiDAR leads to some performance improvement, particularly in reconstructing occluded objects, but the gains are marginal as perception accuracy is already near saturation. Moreover, most occluded objects are distant and have limited impact on immediate decision-making.

• Due to the latest NeurIPS submission policy, we are unable to include visualization links, but we will provide them in the final camera-ready version.

[5] BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

[6] BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

Questions c, Analyzing the Performance Gains of MBRL over IL

• Thank you for raising this important analysis. As noted in the Introduction, imitation learning (IL) relies on expert data and often struggles with out-of-distribution generalization. It is also prone to issues such as causal confusion (e.g., in traffic light scenarios, where the model may mistakenly associate “other vehicles move” with “I should move”) and compounding errors during long-horizon rollouts. In contrast, reinforcement learning (RL) learns through trial and error, optimizing behavior directly from reward signals.

• Recent breakthroughs in RL across domains such as LLMs [7], VLMs [8], and AlphaFold [9] have demonstrated RL’s distinct advantages over supervised fine-tuning or imitation learning, particularly in discovering policies that go beyond demonstrator behavior.

• In summary, while IL provides a strong initialization from expert behavior, it is fundamentally limited by demonstration quality and generalization capacity. Our MBRL framework leverages privileged guidance and learned world models to go beyond imitation, enabling robust and reward-driven policy learning that better handles long-horizon reasoning and distributional shifts. We will incorporate this analysis in the final version to clarify the advantages of MBRL over IL.

[7] 2024, Technical Report, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

[8] 2025, Arxiv, VLM-R1: A stable and generalizable R1-style Large Vision-Language Model

[9] 2021, Nature, Highly accurate protein structure prediction with AlphaFold

Limitaions 1, Discussion on Method Limitations

• Thank you for the valuable suggestion. As a reinforcement learning framework, our method requires trial-and-error interaction, which incurs additional computational cost compared to imitation learning that directly learns from expert demonstrations. Moreover, due to the lack of highly realistic real-world simulators, we conduct experiments in CARLA. While CARLA has limitations in fidelity, we believe the core scientific questions and early-stage insights presented in this work are broadly applicable and valuable to the community. We will explicitly discuss these limitations in the final version.

Limitation 2, Discussion on Transferability in Real-World Autonomous Driving

• We appreciate the reviewer’s concerns regarding the practicality of transferability in real-world autonomous driving.

• About the Transferability of RL

  • While real-world deployment of RL remains challenging, the community has made significant efforts in high-fidelity simulation generation[10] and reconstruction[11,12], though their readiness for large-scale RL deployment remains limited.

  • Our work builds upon the trend of RL, serving as a preliminary study of applying RL to end-to-end autonomous driving. Due to the unavailablility of real-world-level authentic simulator, we use CARLA but we argue that the scientific problem studied and discoveries of the early exploration in this work are general and worth-sharing for the communit.

• About the Transferability of Raw2Drive

  • Raw2Drive utilizes a dual-stream world model to effectively decouple perception and decision-making. On the decision side, trajectory prediction is conducted based on structured perception outputs (e.g., BEV or compact representations)—a widely adopted practice in both academia and industry. On the perception side, the emphasis is on enhancing the accuracy of these BEV-based representations.

  • By leveraging the world model, our paradigm enables a fully differentiable pipeline while preserving a separation between perception and planning.

  • Furthermore, with recent progress in high-fidelity scene reconstruction[4] and simulation technologies [5,6], a practical deployment path is to pretrain the world model via imitation learning and fine-tune the policy with reinforcement learning—an increasingly feasible strategy in real-world and industrial settings.

[10] 2025，ICCV, HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

[11] 2025, ICCV, BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting

[12] 2025, Arxiv, ReSim: Reliable World Simulation for Autonomous Driving

Thank you again for your valuable comments. We hope our responses have effectively addressed the concerns raised. As the rebuttal period is ending soon, we’d sincerely appreciate a score update if the concerns are resolved. If there are any remaining concerns that prevent a higher score, we would be grateful for further clarification.

Thank you for your valuable comments, which have greatly helped us improve the quality and clarity of our work. We fully agree with your insightful perspective on the sim2real gap—a critical challenge for RL-based driving systems. While CARLA remains the only viable closed-loop simulator suitable for RL research at present, our work focuses on policy learning and introduces a dual-stream world model design that is conceptually decoupled from the underlying simulator. As generative or reconstruction-based simulators continue to evolve, we believe that exploring end-to-end RL within these emerging frameworks will become increasingly practical and impactful. We will incorporate these valuable discussions into the revision. Thank you again for your thoughtful feedback!

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Weakness 1，Question 3, Discussion on Practicality of RL Training and Transfer

• We appreciate the reviewer’s concerns regarding the practicality of RL training. Recent breakthroughs in reinforcement learning (RL) in domains such as LLMs[1], VLMs[2] and AlphaFold[3], have shown that RL's unique advantages compared to SFT/imitation learning. While real-world deployment of RL remains challenging, the community has made significant efforts in high-fidelity simulation generation[4] and reconstruction[5,6], though their readiness for large-scale RL deployment remains limited.

• Our work builds upon the trend of RL, serving as a preliminary study of applying RL to end-to-end autonomous driving. Due to the unavailability of real-world-level authentic simulator, we use CARLA but we argue that the scientific problem studied and discoveries of the early exploration in this work are general and worth-sharing for the community.

[1] 2024, Technical Report, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

[2] 2025, Arxiv, VLM-R1: A stable and generalizable R1-style Large Vision-Language Model

[3] 2021, Nature, Highly accurate protein structure prediction with AlphaFold

[4] 2025，ICCV, HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

[5] 2025, ICCV, BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting

[6] 2025, Arxiv, ReSim: Reliable World Simulation for Autonomous Driving

Weakness 1，Question 2, How many interactions were needed to train this end-to-end architecture?

• As noted in Sec. 4.1, we adopt the same 1000 training routes from Bench2Drive to ensure fairness. However, as an online learning paradigm, RL naturally collects a larger number of interactions—both successful and failed—to iteratively refine its policy. This trial-and-error process is a core advantage of RL over offline imitation learning.

• In Bench2Drive-Base, the 1000 routes yield a total of 250k samples. To further analyze interaction efficiency, we also include additional ablations. Similar to early work such as MaRLn [7], which trained and tested on the same set of routes(albeit with limited generalization), we show that while training on fixed routes is feasible, generalization remains a challenge.

• Notably, Raw2Drive enables better generalization to unseen scenarios by learning to predict future dynamics—this requires more interactions. As the number of interactions increases from 250k to 750k and 1M, we observe performance gains. Both successful and failed trajectories are crucial for learning a robust world model. However, we acknowledge that training time also increases accordingly.

[7]2020, CVPR, End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances

[8]2023, CVPR, Planning-oriented Autonomous Driving

Weakness 1，Limitations 1, Discussion on How to use this paradigm in Real-World?

• Thanks for your suggestions. Raw2Drive utilizes a dual-stream world model to effectively decouple perception and decision-making. On the decision side, trajectory prediction is conducted based on structured perception outputs (e.g., BEV or compact representations)—a widely adopted practice in both academia and industry. On the perception side, the emphasis is on enhancing the accuracy of these BEV-based representations.

• By leveraging the world model, our paradigm enables a fully differentiable pipeline while preserving a separation between perception and planning.

• Furthermore, with recent progress in high-fidelity scene reconstruction[4] and simulation technologies [5,6], a practical deployment path is to pretrain the world model via imitation learning and fine-tune the policy with reinforcement learning—an increasingly feasible strategy in real-world and industrial settings.

Weakness 2, Question 1, Contribution and Novelty of Raw2Drive

• Thank you for the helpful suggestion. Due to page limitations, we moved the related work section to the appendix. Nonetheless, as shown in Table 1 of the Introduction, we compared prior work, including both RL-based and world-model-based approaches. Our core focus is reinforcement learning in end-to-end autonomous driving. Early work such as MaRLn [7] (CoRL 2017) encountered significant challenges due to the difficulty of learning directly from raw images, which made policy optimization unstable.

• With recent progress in model-based RL [9, 10], our work introduces two key novelties:

  • Successfully achieve end-to-end RL To our best knowledg, Raw2Drive is the first successfully achieve end-to-end RL with performance significantly better than the end-to-end IL of the same period.
  • Dual-stream World Model as a Bridge: We use a dual-stream world model to decouple perception and decision-making. This design simplifies the joint optimization problem, which is often difficult to converge in end-to-end autonomous driving.
  • Dual-stream World Model Alignment: Unlike standard teacher-forcing strategies [11, 12], future prediction in driving involves both stochastic (e.g., other agents’ intentions) and deterministic (e.g., ego state) components over time and space. We propose Spatial-Temporal Alignment and Abstract-State Alignment to ensure consistency between the two streams during rollout.

• We will revise the final version of the paper accordingly to better highlight these contributions.

[9] 2020, ICLR, Dream to Control: Learning Behaviors by Latent Imagination

[10] 2021, ICLR, Mastering Atari with Discrete World Models

[11] 2021, ICCV, End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

[12] 2022, NeurIPS, Teacher Forcing Recovers Reward Functions for Text Generation

Weakness 2, Clarification on the Bench2Drive Metric

• In fact, as noted in Sec. 5.1 of [13], their evaluation metric is based on Bench2Drive, which is consistent with ours. Furthermore, [13] modifies the base penalty coefficients for CP (crossing red lights) and CV (vehicle collisions), reducing them specifically to address issues introduced by their early stopping strategy.

[13]2025, Arxiv, Hidden Biases of End-to-End Driving Datasets

Question 2, More introduction of RL method

• We follow DreamerV3 [14], a representative model-based RL method, which employs the REINFORCE[15] algorithm for policy training. As detailed in Our Appendix B and E, we employ a latent imagination-based actor-critic framework. Unlike standard actor-critic methods [16] that rely on real environment rollouts, we train the actor and critic entirely within a learned latent space using trajectories imagined by the world model. The actor is optimized to maximize expected long-term reward under these imagined rollouts, while the critic learns to predict value estimates in the same latent space, enabling sample-efficient and stable training.

[14] 2025, Nature, Mastering Diverse Domains through World Models

[15] 1992, Machine Learning, Simple statistical gradient-following algorithms for connectionist reinforcement learning

[16] 2017，NeurIPS, Proximal Policy Optimization Algorithms

Question 3, Clarification on NAVSIM simulator

• Thank you for the question. NAVSIM[17] is a non-reactive simulator designed for benchmarking perception and planning, but not for training interactive policies. It does not support closed-loop interactions or provide observations after interaction, making it unsuitable for reinforcement learning, which relies on trial-and-error learning with visual feedback. We will clarify this limitation in the revised version.

[17] 2024, NeurIPS, NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

About Paper Formatting Concerns

• Thank you for pointing this out. We apologize for the oversight and will carefully correct all typos and formatting issues in the final submission.

Thank you again for your valuable comments. We hope our responses have effectively addressed the concerns raised. As the rebuttal period is ending soon, we’d sincerely appreciate a score update if the concerns are resolved. If there are any remaining concerns that prevent a higher score, we would be grateful for further clarification.

Thanks for your reply.

Further Clarification on Concern 1

• Thank you for the suggestion. First, we would like to clarify that NAVSIM[1] does not support end-to-end reinforcement learning (RL). As an online RL agent must learn from trial and error, NAVSIM[1], being non-reactive and pseudo-closed-loop, cannot provide RGB frames outside of the dataset(e.g., collisions or going off-road). Thus, it is not suitable for training end-to-end RL.

• In contrast, CARLA remains the most widely adopted closed-loop simulator for end-to-end RL. Numerous prior works have conducted experiments solely within the CARLA, such as TCP [2], InterFuser [3], ReasonNet [4], ThinkTwice [5], TransFuser++ [6, 7], DriveAdapter [8], LMDrive [9], ReasonPlan [10], SimLingo [11], ORION [12], and ETA [13]. In our work, we thoroughly validated our method in two standard closed-loop benchmarks, Leaderboard 2.0 and Bench2Drive. If the reviewer is aware of other benchmarks that support end-to-end RL, we would humbly appreciate a reference for our consideration.

• Additionally, as generative[14] or reconstruction-based simulators[15, 16] become more mature, it may be feasible to explore end-to-end RL in such frameworks. However, our current focus is on policy learning, which is decoupled from the fidelity of rendering or simulation.

• Finally, we would like to emphasize that Table 5 of Bench2Drive includes only imitation learning baselines, except for ours. This highlights our core contribution: successfully enabling end-to-end RL that outperforms IL methods.

[1]2024, NeurIPS, NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

[2]2022, NeurIPS, Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline.

[3]2022, CoRL, InterFuser: Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

[4]2023, CVPR, ReasonNet: End-to-End Driving with Temporal and Global Reasoning

[5]2023, CVPR, Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving

[6]2021, CVPR, Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

[7]2023, TPAMI, TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving

[8]2023, ICCV, DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving

[9]2024, CVPR, LMDrive: Closed-Loop End-to-End Driving with Large Language Models

[10]2025, CoRL, ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Driving

[11]2025, CVPR, SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment

[12]2025, ICCV, ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation

[13]2025, ICCV, ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Models

[14]2025，ICCV, HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

[15]2025, ICCV, BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting

[16]2025, Arxiv, ReSim: Reliable World Simulation for Autonomous Driving

Further Clarification on Concern 2

• Question a, Clarification on UniAD being trained on 250k “interactions”

  • Bench2Drive has become a widely adopted benchmark, with official implementations of imitation learning methods such as UniAD, VAD. The “250k interactions” mentioned in UniAD correspond to the official Bench2Drive dataset, which includes 1000 routes totaling approximately 250k steps.

[17]2023, CVPR, Planning-oriented Autonomous Driving

[18]2023, ICCV, VAD: Vectorized Scene Representation for Efficient Autonomous Driving

• Question b, Clarification on “How was your ‘250k’ policy trained”

  • While UniAD and VAD are trained using the high-quality 250k expert trajectories provided in Bench2Drive, our method generates 250k data points through online reinforcement learning interactions. These trajectories include failure cases such as collisions and traffic violations, making them significantly noisier and lower in quality compared to expert data. However, this is precisely the strength of reinforcement learning—it learns from trial and error.

  • As described in Appendix B (Details of Training Pipeline), our training consists of two stages. We first train a privileged WM and a privileged policy using privileged inputs; Then, we leverage this privileged WM to guide the learning of the raw sensor WM and policy, enabling effective RL policy training from raw sensor data.

• Question c, Clarification "directly training an end-to-end policy via IL from Think2Drive as an expert policy"

  • As shown in Table 5, all compared methods—TCP-traj*, ThinkTwice*, and DriveAdapter* (* denotes expert feature distillation)—are imitation learning (IL) approaches similar to those discussed in [19], where an RL expert policy is distilled into an IL end-to-end model. Additionally, UniAD is a standard IL method that learns directly from expert demonstrations.
  • In contrast, our core contribution lies in successfully enabling expert-guided RL end-to-end model through a dual-stream WM design and alignment mechanism, achieving superior performance compared to IL-based approaches.

[19]2022, Science robotics, Learning robust perceptive locomotion for quadrupedal robots in the wild