AD-H: Autonomous Driving with Hierarchical Agents

Limited baseline comparisons：

• Should include more recent works like CarLLaVA and other hierarchical approaches

Insufficient ablation studies:

• No analysis of impact of different granularities of mid-level commands

Limited novelty

The core idea of hierarchical decomposition is relatively straightforward and similar approaches have been explored in diverse domains, including autonomous driving and robotics control.

The experimental validation could be more comprehensive:

• Only one main baseline (LMDrive)
• Limited analysis of failure cases

Limited Real-World Validation:

• All experiments are conducted in simulation (CARLA)
• No discussion of potential sim-to-real transfer challenges

Computational Requirements:

• The system requires two large models (7B parameter MLLM + 350M parameter controller). Runtime analysis and latency considerations are not discussed

Dataset Construction:

• The rule-based annotation process for mid-level commands could introduce biases
• More details needed on annotation quality control measures

1. The innovation of hierarchical planning is limited: DriveMLM has investigated the LLM-based mid-level lateral and longitudinal decision commands by setting decision states in Carla.
2. This paper uses LLaVA-7B and OPT-350M as the high-level planner and low-level controller, respectively. Can the 7B planner provide accurate mid-level instructions? Can the 350M controller output a well-planned trajectory based on ground truth instructions?
3. Separate evaluation experiments for each module should be added to better assess the performance of each module and the effectiveness of mid-level instructions in enhancing the controller.
4. Figure 3 demonstrates the self-correction capability of the MLLM controller, but what is the source of this capability? More examples should be provided to analyze the self-correction ability. Additionally, when training the MLLM controller, we aim for strong instruction-following capability. Can the current model balance instruction-following with self-correction effectively?
5. In Table 5, I understand that the ADH dataset used to train LM-Drive does not include mid-level instructions and only uses the resample method. The superior performance of ADH compared to LM-Drive may stem from the mid-level instructions in the dataset. An experiment should be added where mid-level instructions are input along with instructions to train LM-Drive for comparison, analyzing the advantages of using a low-level controller versus having LLaVA-7B generate trajectories directly.
6. Regarding model-related questions: After training the MLLM planner and controller separately, was joint training conducted? Are the decision frequency of the planner and the planning frequency of the controller consistent?
7. The hyperlinks for tables and figures are set incorrectly, such as in Tables 5 and 6, and Figures 4, 5, 6, and 7. There is also a hyperlink error in lines 312–313.

1. Evaluation of Consistency between Language and Action: While the hierarchical framework is well-conceived, further evaluation of how consistent the language command and executed actions are, would be beneficial. In other words, do the final control actions comply with the mid-level commands? This is essential for fully validating the model’s real-world safety and applicability.

2. Labeling of Mid-Level Instructions: While the authors clearly introduced the hierarchical design, the methodology for labelling mid-level instructions lacks sufficient detail. For example, how do the authors define the high-level and mid-level instructions, and what is the principal difference between them? What is the design principle for mid-level instructions? While code snippets for generating mid-level commands are attached in the supplementary, descriptions of these design principles would better help readers grasp the gist of the design.

3. Inclusion of Point Cloud in the Controller Only: As shown in Fig 2, the choice to include point cloud data only within the MLLM controller, but not in the MLLM planner, is interesting but unexplained. More context regarding this decision would clarify whether it impacts the controller’s performance or if it’s just to balance computational resources.

4. Ablation Study on the Motion Resampling: In Section 6.1 and Table 5, the ablation study findings appear counterintuitive, as resampling is expected to improve performance but seems to lower certain metrics in LMDrive. An explanation regarding the discrepancy, and an additional experiment with AD-H trained without resampling, would possibly strengthen the study. It's fine that the resampling increases the performance, since that is also part of the contribution in this paper. And clarifying the performance contributions of each component would improve the message to the community regarding AD-H’s core strengths.

5. Minor Presentation Issues: A few minor presentation improvements are recommended, including correcting the typo of reference failure on Line 313 and marking the second-highest metrics in tables with an underscore to improve clarity.

In short, the paper, overall, makes the most minuscule contribution to society. Here are some key problems in the fundamental logic and their experiment design:

1. The authors suggest that by decoupling the perception-planning and controlling stage, one can better leverage LLMs' emergent capabilities. However, their justifications are rather feeble - the hierarchy helps free the LLM from generating low-level control signal decoding and focus on perceptive reasoning and planning. From a methodology perspective, the decoupling of planning and control can introduce additional bias during message passing from upstream to downstream. For example, if the LLM upstream yields an arbitrary command such as "slow down to ensure safety," how can the authors make sure that the target speed generated by the downstream controller can guarantee safety? Meanwhile, during fine-tuning, how can you determine whether the controller should be further fine-tuned to improve control quality or the upstream LLM should be fine-tuned to improve command generation?
2. On the other hand, the authors attribute the limitations of previous methods to a cause that they deviate from the inherent language generation paradigm. However, existing works have proposed several methods as well as benchmark datasets to exploit and evaluate the reasoning capabilities of the LLMs in autonomous driving systems. For example, in the paper "LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs" from CVPR 2024, the authors propose and provide a collection of atomic command building blocks with high-level objectives and use LLM to fill in blanks in these commands to achieve control. The authors should investigate and compare their work to this paper and other works that similarly showcase their method's superiority.
3. This paper's experiment design is rather poor. On the one hand, the authors built a dataset for self-consistently evaluating their proposed method but only compared their results to one of the existing methods. This is far from sufficient to justify their claimed benefits. Meanwhile, it is hard to convince the audience that the dataset is not built with a bias in favor of the proposed method over the competitors. Lastly, the primary concern from decoupling planning and controlling arises from extra time and memory requirements, which the authors didn't investigate further in their experiments.

• The author did not demonstrate failure cases. On the LangAuto benchmark, the best route complete rate is 53.2%, what are the cases where it did not complete?