World-Model based Hierarchical Planning with Semantic Communications for Autonomous Driving

Last Update: in response to the reviewer's comment on Dec 2nd

We thank the reviewer for reviewing our paper, and we address the raised concerns as follows.

[W1] The reviewer inquired about the motivation to learn the lower-level policy instead of directly tuning a PID controller to track waypoints.

We thank the reviewer for providing a different perspective, and we clarify that an end-to-end lower-level policy is favorable and we summarize the reasons as follows:

a) Decision Making on a Rich Context. The lower-level policy operates on rich latent representations encoding complex state information including traffic dynamics derived from image inputs, not just positions or speeds used by a PID controller. This allows HANSOME to learn adaptive behaviors for collision avoidance and speed modulation directly based on visual inputs with surrounding traffic. For example, the higher-level policy may plan the same “go straight” intention in different situations, but the surrounding traffic makes a huge difference in expected behaviors of the lower-level policy. The lower-level policy can decide to speed down to follow leading vehicle or keep its current speed when there is no leading vehicle. It goes beyond the capability of a traditional PID controller solely based on position and speed. In contrast, our lower-level policy can handle such cases since it plans based on richer information of the surrounding environment in an end-to-end manner. Our demos in supplementary materials (e.g., collision avoidance) also showcase such capabilities of the lower-level policy that behaves differently given different traffic situations.

(b) Joint Optimization for Optimality. Joint training of high and low-level policies enables more coordinated behavior adaptation. The high-level policy can learn to generate waypoints that account for the low-level policy's capabilities, while the low-level policy simultaneously learns to better interpret and execute the high-level intentions. This tight coupling would be lost with an independently tuned tracking controller. Moreover, it has been widely demonstrated that end-to-end policies are highly promising [1,6,7,11,12]. HANSOME’s lower-level policy also does not involve physical parameter tuning like Ki, Kp, and Kd in PID controllers.

[W2] The reviewer inquired why we need a simulator during policy rollout, and the reviewer suggested “the motivation for learning a world model is to learn a realistic simulator”.

We clarify that during policy rollout, the tracking error from Eqn. 1 the reviewer referred to (typically referred to as reward) is directly predicted by the world model without the assistance of the simulator. Policy training occurs entirely in the world model's imagination (inference), not in simulation. We utilize the simulator to compute ground truth reward, but that is for the purpose of training the world model to predict the reward. Then, the agent learns its policy solely based on the learned dynamics and reward model. This aligns with the standard practice in world model works including DreamerV1-V3 [2][3][4] and MBRL works [5].

[W3] Benchmark

We greatly appreciate the reviewer's suggestion regarding Waymax benchmark. We would like to clarify our choice of CARLA as our primary testing environment for the following crucial considerations.

First, Waymax challenges have mainly focused on predictive tasks, while HANSOME focuses on closed-loop interactive scenarios for policy learning and evaluation. CARLA has been widely used in the well-established works on imitation learning or reinforcement learning to evaluate their policies on customized scenarios (like MILE[6], SEM2 [7], TransFuser [8], InterFuser [9], ISO-Dream [11]). Moreover, we note that the 100k scenarios in Waymax mentioned by the reviewer are logged records driven by IDM. Therefore, the Waymax is suitable for predictive tasks like predicting future trajectory, occupancy flow, or semantic segmentation [10].

We hope the clarifications can address the reviewer’s concerns and we are willing to address any further questions.

We thank the reviewer for the careful review and feedback. We will provide more clarifications in response to the reviewer’s concerns.

[W1] Testing Scenarios

Thank you for the suggestion. We add a new challenging scenario “Roundabout” with dense and aggressive traffic on top of the scenarios we already have (DenseTraffic, LeftTurn and RightTurn, ObstacleBypass). We keep the same model configurations and hyper-parameters, and used 3 random seeds to compute the confidence interval. Director failed to learn a reasonable policy with similar training budgets as other models around 600k steps, therefore we marked it as N/A. Overall, HANSOME shows optimal performance on all tasks.

“-C” indicates the baseline models benefits from enhanced observability through semantic communications for fair comparison.

We clarify that it is non-trivial to design testing scenarios involving RL based online learning approached with communications.

We discussed the considerations of task design from line 868-873 and 910-925, and discussed how prior works customized their tasks for evaluation. For example, some prior world model based RL works test the agent to drive to maximize total rewards within 1000 steps [11, 13] in CARLA town03 with 20 or 100 vehicles. HANSOME is online RL learning and our tasks are more challenging in the sense that traffic is even denser (300 vehicles in DenseTraffic) and the task goal is not to drive as much as possible but to complete a driving skill. Collision avoidance is way more challenging at intersections since the traffic flow is set to be “aggressive” and they can ignore the ego vehicle like irrational human drivers. The ego agent has to understand other agents’ intentions and predict whether they will interfere with the ego agent to find a proper time to proceed. Furthermore, for our work that features multi-agent communications, it is also necessary for us to design and develop tasks that rigorously evaluate the impact of semantic communications. But these tasks are not readily available, and this is why we have meticulously designed such challenging tasks with dense traffic or challenging objective in a highly interactive environment CARLA. The ego vehicle has the find proper timing to merge lanes, or take turns, or flexibly bypass obstacles ahead, avoid collision, and complete tasks.

[W2] Availbility of Intentions in Real-World Settings.

In response to the reviewer's concern, we add experiments to show that HANSOME are robust to environments where semantic intentions are unavailable.

[W4] How does the model understand language or traffic rules

HANSOME agents understand intentions through a cross-modal encoder-decoder. The decoder tries to predict other vehicles’ future trajectories from latent representations. This can be seen as an auxiliary task to force the latent representations to learn what is means by each intention, while the agent can still learn purely from visual inputs without additional text prompts nor prior knowledge. We did not use LLMs nor text prompts. HANSOME is a reinforcement learning based framework like [2, 3, 6], where the expected agent behaviors are defined by reward functions. We do not explicitly provide traffic rules to the agent. Instead, we include reward terms such as these penalizing out-of-lane or collisions.

We thank the reviewer qDEw for their thoughtful review and questions, which have greatly helped us improve our paper. We're glad the most of the reviewer's concerns were addressed! Based on the reviewer’s feedback, we have made the following revisions:

• Handling Unpredictable Driving Behaviors: We show that the agent can handle unpredictable driving behaviors by introducing aggressive background vehicles in our urban driving tasks. This is addressed from lines 405, 474, and Appendix D.4.
• Practicality of Intentions: We discuss the impact of the availability of intentions as a practical issue in real-world settings (line 322). We demonstrate that HANSOME does not assume full availability of intentions while still achieving reasonable performance (Table 3).
• Adaptivity of AdaSMO: We provide further discussion on why AdaSMO is adaptive and how it is inspired by adaptive learning process of human (line 1144, line 356, line 367, and Appendix D.1).
• Comparisons with More Literatures: We included more comparisons to HRL, MARL in Related Works and Appendix A, highlighting the novelty of HANSOME compared to these fields.

We are happy to address any additional questions on the final day of the discussion period. We hope that our responses and revisions have resolved the remaining concerns. If so, we kindly request the reviewer to consider updating the score before the discussion period ends today.*

[W4] Benchmark Selection

We thank the reviewer for the suggestion. HANSOME requires interactive and communicative scenarios to rigorously evaluate the benefits of semantic communications, while there are not such benchmarks readily available for closed-loop autonomous driving in highly realistic simulators like CARLA.

Although it might take considerable effort to incorporate semantic communications to benchmarks like Leaderboard and address many potential challenges with online RL learning in multi-agent systems, we believe extending HANSOME to these even more complicated and challenging scenarios is a very meaningful step that we will pursue in future works.

In recent studies, imitation learning approaches conduct more experiments on Leaderboard by having expert demonstrations [13]. We discussed in Appendix A that given the challenge to learn RL policies in online environemnts, previous world model based RL research like SEM2 [3] and Iso-Dream[12] tend to test on simpler driving cases with less traffic or easier tasks (20 or 100 vehicles in Town03). Think2Drive [10] (Feb 2024) is so far the only world model RL approach that trains DreamerV3 on Leaderboard scenarios, while it is not communicative and it uses BEV inputs with pre-determined routes. Given the challenge of both online RL learning and communicative learning in multi-agent systems, we will pursue this significant step for our future extensions, and we appreciate the reviewer's suggestion. Our tasks are challenging since they involve several driving skills, dense and aggressive background traffic. For example, we have 300 vehicles in DenseTraffic to test the agent's collision avoidance; in LeftTurn, RightTurn, and Roundabout, the dense traffic flows are set to be aggressive and they do not actively avoid collision (like irrational human drivers); ObstacleBypass is spefically designed to test the flexiblity of hierarchical planning.

In response to the reviewer’s suggestion, we have added another challenging scenario Roundabout with dense traffic. We follow the same model settings/hyper-parameters discussed in Appendix F where baselines follow their original implementations and share hyper-parameters for common components. We did not do hyper-parameter tuning for any models like on other tasks. Director is inactive exploring during training and it fails to complete the task with the same 600k training step budget when other models already learns the skill. It is not as sample efficient as other models and might take much longer time than other models to converge, and therefore we mark the results as N/A.

We will include the table in the response to Q1 below.

[Q1] Baseline Performance on LeftTurn and RightTurn

Thank you for the suggestion to further justify the effectiveness of hierarchical planning and AdaSMO. We added the experimented suggested by the reviewer and the table shows success rates of HANSOME and baseline models on LeftTurn, RightTurn, and a new task Roundabout. HANSOME shows optimal performance on all the tasks.

We thank the reviewer for the careful review and insightful feedback. We address the raised concerns as follows.

[W1] More Comparison with Related Works

We thank the reviewer for sharing the concern. We will provide detailed comparison against the research fields suggested by the reviewer and will revise the paper.

1) Comparison with LLM-Based Methods (e.g., DriveLM [6] and DriveVLM [7])

LLM based methods such as DriveLM [6] and DriveVLM [7] are open-loop methods. The objective involves optimizing NLP-based metrics, such as comparing scene descriptions/analysis with ground-truth annotations (driving captions, visual Q&A) using GPT. The “hierarchical planning” in these works does not involve multiple policies – it proposes action descriptions and converts descriptions to waypoint tokens. Waypoint tokens are obtained from trajectory statistics in training data since having LLMs produce numerical results is challenging [6]. To evaluate LLM action proposals, they compare the converted trajectories with ground-truth trajectories. We can see the planning is achieved through LLM proposals that are mapped to waypoints, and they do not have an actual controller to track waypoints or evaluate the planner since it is open-loop. This is different from HANSOME that focuses on developing a closed-loop policy and evaluating policy performance in simulation. The lower-level policy follows accurate waypoints proposed by the higher-level planner. HANSOME higher-level policy produces semantic intentions without prior knowledge from LLMs. Both levels are learned from scratch within world models’ imagination without strong priors from LLMs. HANSOME models are more lightweight (30M parameters, significantly fewer than LLMs) and therefore more practical for real-time inference.

2) Comparison with Other Hierarchical Planning Works

it is challenging and highly non-trivial to address non-stationarity during hierarchical RL training because multiple RL policies simultaneously change [9]. Imagine a higher-level policy plans intentions which cannot be achieved by the lower-level policy learning from scratch, and therefore the higher-level only observes short paths and cannot learn complex path planning skills. A vanilla two-policy training will yield poor performance as shown in Fig. 5. Conventional approaches of hierarchical RL leverage hindsight replay and re-labelling techniques to address non-stationarity. One of the contributions of HANSOME is AdaSMO that does not need any extra re-labelling process. AdaSMO is inspired by human learning, where humans learn each basic skill proficiently before integrating all skills for more complex tasks. AdaSMO adaptively adjusts higher-level policy entropy to scalarize the two-level learning. As shown in our ablation of AdaSMO and Fig. 5, we provided a novel perspective in solving hierarchical RL, and we successfully applied this hierarchical RL method in challenging autonomous driving tasks.

3) Comparison with Multi-Agent RL and V2X Research

Although multi-agent RL or V2X communities have studied information sharing [1][11], it is challenging to apply information sharing to highly realistic simulation environments for closed-loop evaluation in addition to multi-agent communications. MARL typically suffers from high-dimensionality and non-stationarity [1], the common testbeds can be grid worlds like predator prey. We simplify the problem and leverage ego-centric learning for autonomous driving to make learning with communications tractable in multi-agent systems as discussed in Appendix B. Moreover, our closed-loop evaluation emphasizes more on policy performance. V2X research such as cooperative perception [11], can focus on, e.g., prediction errors of occupancy flow, that does not require policy rollouts. In summary, HANSOME aims at initiating the first step towards practical world model learning for autonomous driving with communications in multi-agent systems. The distinct challenges of closed-loop evaluation motivate us to pursue practical and novel solutions like semantic communications, ego-centric learning, and AdaSMO to mitigate issues of high-dimensionality and non-stationarity in multi-agent systems.

4) Comparison with TCP [8] and Works that Predict Vehicle Motions

TCP is based on supervised imitation learning, while HANSOME is online RL based on world models. HANSOME needs to cope with non-stationarity during hierarchical RL learning. Though HANSOME supports two-level planning like TCP, and support predicting background vehicle future trajectories, it is neither the purpose of this work. The contribution of HANSOME is the seamless integration between the two – a higher-level policy that not only guides its own lower-level but also can generate messages to enhance others’ future prediction and planning.

We thank the reviewer for carefully reading our responses and appreciate the reviewer’s constructive feedback. Based on your feedback, we have revised the paper to improve the clarity (the updates are highlighted in blue fonts).

The revision includes the new experiments (Table 5) suggested by the reviewer, dicussions according to the reviewer's feedback regarding for reward function and scalarization scaing factors (Sec 3.1, Appenix D.1), why HANSOME is not replacement but complement to route planning of Google Maps (, and comparison to LLM, MARL, and V2V research (Appendix A). Again, we appreciate the reviewer's insightful feedback which helps a lot for us to improve the presentation and clarity.

Response to Reviewer Concern

Concern: “Communication is not mandatory and the method can still achieve comparable performance with perturbations.”

We clarify that:

1. HANSOME does not mandate communications, as it is more practical to assume that some agents do not support communications while others may use communication prior to standardization in real-world mixed autonomy settings.
2. Semantic communication is beneficial for traffic safety and efficiency, and HANSOME is designed to take advantage of this when semantic communication is available.

Our experiments take both scenarios into account.

Performance Comparison

In response to the reviewer’s observation that “the method can still achieve comparable performance” when training with disabled intention sharing, we reorganize the performance comparison. The results show an 18%-34% reduction in collision rates relative to the benchmark using visual only. Efficiency is also improved by 11%-20%, indicating the development of more confident policies.

Collision Rate Comparison of Different Communication Settings

Norm. Speed Comparison of Different Communication Settings

Updated this response on Nov 27 with the paper revision

We thank the reviewer for the feedback. Here are our responses to the main concerns.

[W1] Does HANSOME needs communications to make progress? Or is it applied to real world distributions where all agents are not controlled by a uniform policy?

Thank you for the question regarding the application of HANSOME.

Communication is not mandatory but it improves safety and efficiency.

As discussed from lines 86-90, HANSOME does not mandate communications to make progress. Shared intentions are an optional input to our cross-modal encoder-decoder. As stated in the paper, shared intentions help the encoder-decoder predict the future trajectory of background vehicles. In absence of such information, the prediction just becomes less accurate, but HANSOME can still make planning decisions based on ego vehicles’ own observations. It is similar to visual information sharing, which contributes to a more complete BEV in case some vehicles are blocked by others.

To further testify to this point in response to the reviewer’s comment, we add another set of experiments to demonstrate HANSOME’s robustness to environments where intention communications are unavailable. The models are all trained with around 500k steps budget with the same model configurations/hyper-parameters.

Handling Non-Uniform Policies.

The communication protocols in HANSOME are agnostic to agent models or policies since agents do not need to communicate their full policies but rather share their intentions (such as “left turn”). As the messages do not involve any agent-specific representations, HANSOME does not impose any restrictions on the lower-level policy. Therefore, our setup is practical in the environment with heterogeneous agents. Furthermore, we point out that during the training/testing, background vehicles include 1) rule-based drivers, including aggressive drivers (line 474-476, line 536-538) 2) other HANSOME agents (as discussed in Appendix B Ego-Centric Learning), which have non-uniform policies. Our experiment show that HANSOME is capable of navigating through these environments with mixed policies.

Real-world Application with Heterogeneous Agents

HANSOME supports heterogeneous agents by using semantic language as a universal communication interface, unlike conventional multi-agent RL approaches [2, 3, 4] that require homogeneous agents with shared latent spaces. Our experiment results demonstrate that this semantic communication significantly improves traffic safety and efficiency. While V2V communication protocols [6, 7, 8, 9] provide the technical foundation for inter-vehicle communication, our contribution focuses on demonstrating that semantic language offers a simple yet effective communication paradigm for heterogeneous agents with different underlying policies.

[W2] Comparison experiments.

We clarify that our work focuses on world model based planning and we compare our proposed method with SOTA methods such as DreamerV3, Director [10, 11, 12].

We discussed the selection of baselines and tasks from line 396 and line 409, and discussed how prior works customized their tasks for evaluation. For example, some works have the agent drive to maximize total rewards within 1000 steps [11, 13] in CARLA town03. Our work focuses on multi-agent communications (different from the works on general robotics planning or LLM based planning), and it is necessary for us to design and develop closed-loop tasks that rigorously evaluate the impact of semantic communications. But these tasks are not readily available. In this regard, we have meticulously designed challenging tasks with dense traffic and challenging objectives in a highly interactive environment CARLA. The ego vehicle has to find proper timing to merge lanes, or take turns, or flexibly bypass obstacles ahead, avoid collision, to complete tasks.

We hope the clarifications can address the reviewer’s concerns and we are willing to address any further questions.