A Language Agent for Autonomous Driving

Thanks for your valuable comments!

• Comparison with DiLu: Thanks for pointing this out! We have discussed the comparisons with DiLu in the related works (please refer to the last paragraph of appendix A). Our method is fundamentally different from DiLu in many aspects:

(1) Architecture: Compared to DiLu, Agent-Driver has a more comprehensive architecture and has a better capability to handle realistic driving challenges. Our tool library includes a comprehensive perception and prediction toolkit which covers detection, prediction, occupancy, and mapping. Our memory is divided into a commonsense memory that stores driving knowledge and an experience memory that stores past driving experiences. Our reasoning engine is more structured and contains reasoning, task and motion planning, and self-reflection. All the above-mentioned components were not proposed or included in DiLu.

(2) Experimental evaluation: Compared to DiLu which was evaluated in a virtual and simple highway environment, Agent-Driver is evaluated on a real-world dataset nuScenes, and a more challenging close-loop simulator Carla. Evaluation in these realistic and challenging driving scenarios better reflects the effectiveness of our method.

We will add more discussions of DiLu in the related works. Thanks for your comments!

• Latency issue: In addition to the limitations we have discussed in the paper, we'd also like to point out that recently more and more papers [1-2] have been actively working on accelerating the inference of LLMs, which greatly reduces the latency of LLMs. In the meantime, due to hardware development, autonomous vehicles' onboard computational power also evolves rapidly. Given these two factors, we believe the inference restriction will be resolved in the near future. We will add more explanations to the limitation section. Thanks for your advice!

• Loadable LLM model instead of API calls: We have provided experiments using Llama2-7b as a core LLM in Section 3.5 and Table 3. The results show that using a loadable LLM like Llama2 can lead to a decent performance.

We hope these responses properly address your concerns. We remain fully committed to addressing any additional concerns that may arise. Thank you again for your valuable comments!

[1] Liu et al. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. [2] Lin et al. Awq: Activation-aware weight quantization for llm compression and acceleration.

Thanks for your valuable comments!

• Multimodal Domain Integration: Thanks for your comments! While leveraging a unified multi-modal LLM to handle all driving challenges is a promising research direction, a multi-modal LLM that directly maps sensory inputs to actions generally lacks sufficient robustness, as it cannot fully utilize important intermediate representations like detections, predictions, and maps to avoid collision. On the other hand, we choose a fundamentally different approach, where we still leverage those modular perception modules as tools and introduce LLM as a scheduler and reasoning engine, which could benefit from both these accurate perception results and the reasoning and planning ability of LLM. Which approach is better remains an open question, and we're open to insightful discussions.

• Language-related Benchmarks: Thanks for your suggestions! Compared to the multi-modal driving agents that focus more on language-based driving scene understanding or driving question-answering, our Agent-Driver has a different focus: we aim to address the planning and decision-making problems in real-world driving. Hence, we choose to evaluate our approach on the broadly used driving challenges like nuScenes and Carla. Language benchmarks, such as BDD-X for VLM driving agents, are out of the scope of the current paper. Also, our system does not have a VLM available for BDD-X evaluation. However, we agree that scene description and understanding are very important to autonomous driving, and we plan to integrate a multi-modal LLM as a tool for scene understanding in a follow-up work. Thanks for your suggestion.

• Environmental setup: Thanks for your suggestions! For closed-loop experiments on Carla, we leveraged the perception modules in LAV [1] and kept the other parts of our system the same. We also followed the same training setting and evaluation protocols in [1] for a fair comparison. We will add more details about the closed-loop experiments to our paper. Thanks for your suggestions!

• Commonsense memory and experience memory. Sorry for the confusion. We will change the terminology to semantic and episodic memories and provide further explanations in our paper. Thanks for pointing this out!

We hope these responses properly address your concerns. Feel free to let us know if you have further concerns and we would be thrilled to address them. Thank you again for your valuable comments!

[1] Chen et al. Learning from All Vehicles.

Thanks for your valuable comments!

• What is the difference between the reasoning engine and the implementation in GPT-Driver? There are several key differences between our reasoning engine and the one in GPT-Driver:

(1) We decoupled the chain-of-thought reasoning, task planning, and motion planning as separate steps to the LLM. Compared to generating both reasoning and planning in a single pass as in GPT-Driver, we empirically found that such separation eases the learning and prediction process, and yields a better result.

(2) We use in-context learning for instructing the LLM to perform reasoning and task planning, which encourages better diversity and generalization ability. While in GPT-Driver, it uses fine-tuning.

(3) We additionally introduce a self-reflection stage to further rectify the motion planning results, which leads to fewer collisions and better safety.

• Information flow in the current method: We employed multiple rounds of conversations and used separate messages at each stage to the LLM. Compared to sending all messages as a whole to the LLM, we empirically found that separating the messages makes the LLM focus on one task at each stage, which eases the learning process and yields a better result.

We hope these responses properly address your concerns. We are committed to addressing any further concerns. Thank you again for your valuable comments!

Thanks for your valuable suggestions!

• Costs: inference time using API calls takes 500 ms on average, which also includes the network communication latency. We note that many papers are actively working on accelerating LLM inference, and vehicles' onboard computational power also evolves rapidly. Therefore, we believe this limitation will be resolved soon. We will add more explanations to the limitation section.

• Where does the method struggle: similar to other driving methods, our method is limited by the perception results from the tool library. If the tool library provides noisy results, the reasoning engine might not make appropriate decisions. Figure 2 in Appendix F.3 better illustrates this issue. Future work to address this limitation includes leveraging multiple perception models to provide increased robustness. We will add more explanations to this section.

• How much does CoT help in long-tail / challenging driving scenarios: we provided Figure 1 in Appendix F.1 as a qualitative study showing the effects of CoT in challenging scenarios. The figure shows that when driving near a construction area, CoT helps generate more suitable driving trajectories that effectively avoid approaching this construction area.

• Progress and comfort as evaluation metrics: in addition to the L2 and collision metrics, we have also included Route Completion and Driving Scores as metrics for driving progress and comfort in Carla experiment (please refer to Table 1 and section 3.2 of the main paper). The Route Completion denotes the progress of driving on a route, and the Driving Score additionally takes comfort and safety into calculation.

• What does "invalid" mean in sec3.6: as the reasoning engine needs to generate waypoint coordinates for motion planning, we denote "invalid" as the waypoint outputs contain non-numerical values. We note that fine-tuning enables robust alignments and our reasoning engine can always generate valid waypoints with a few fine-tuning data. We will add more explanations to this section.

• In-context learning: we will add more clarifications in the main paper regarding the concept of in-context learning and fine-tuning. For fine-tuning, we can use more samples, as it is not restricted by the context window like in-context learning.

We hope these responses properly address your concerns. Thank you again for your valuable suggestions! We are committed to providing additional responses should you have any further comments.