PADriver: Towards Personalized Autonomous Driving

1. The necessity of making PADriver a MLLM for HighwayEnv is unclear.

The most important contribution of this paper is the proposed PADriver, which is an MLLM trained for HighwayEnv. As mentioned in Line044, the main reason is that only providing text description is not sufficient to capture enough scene information. Therefore, they use vision inputs with BEV rasterizations.

However, in fact, the BEV image from HighwayEnv does not provide any additional information than all the agent's locations in the scenario, which can be easily extracted as float numbers and provided as either text input or vectorized low-dimensional features. Therefore, it is unclear to me why is using vision input necessary and what benefits it provides for HighwayEnv. It seems to me using text inputs or simply agent-location vectors will be sufficient.

In short, I agree that vision input might be important for more realistic environments like CARLA. However, as this work only conducts experiments on HighwayEnv, the purpose of using MLLM is unclear.

2. The necessity to train the MLLM PADriver instead of directly prompting SOTA MLLMs is unclear.

Related to the last part, PADriver is trained with a large amount of driving data generated by rule-based policies as well as human annotations. However, because the BEV rasterization in HighwayEnv is a very easy and intuitive representation of the scenario, it might be very likely that current MLLMs (e.g., GPT4o) might be already able to solve the PAD problem with relatively good performance.

DILU has already proved that pure-text GPTs are already able to do well in HighwayEnv with text input, I believe with proper prompting and input formatting, it is very straightforward to modify DILU's prompts and use GPT4o to take in the image input and output personalized driving behaviors. To show the necessity of training an MLLM to solve this issue, I believe including GPT4o's performance is important.

3. The use of personalized driving prompts is very limited.

Although the paper has an emphasis on "personalized driving prompts", the kind of personalization is only limited to "slow, medium, and fast" and do not include very detailed personalized prompting like "always try to drive on the leftmost side", "try to keep a distance of 10m from the car in front", "do not exceed acceleration of XX level".

The reason why the detailed personalized prompts are expected is that the PADriver is an MLLM, it is very natural for it to take in free-form and high informative language descriptions of the personalized needs. The current support for 3 levels of speeding can be simply input to the model with a one-hot vector with 3 dimensions, without any text involved.

As personalized driving is the core focus of this paper, I think this limitation becomes an important drawback of this work.

1. The current implementation of three driving styles (slow, fast, normal) is limited to speed variations. Expanding the range of driving styles could better reflect the "Personalized" aspect emphasized in the title.

2. The experiments are solely based on Vicuna-7B-1.5. It would be beneficial to test other LLM weights, such as LLaMA3, LLaVA, and Vicuna-13B-1.5, to evaluate performance across different models.

3. The driving personalities are currently represented through statistics. Providing more visualizations or trajectory plots of different driving styles could offer a more intuitive understanding of the results.

4. The need for images in the framework is not fully justified. An experiment removing images from the inputs would clarify if textual descriptions alone can sufficiently describe the scenarios.

5. The related work on end-to-end autonomous driving methods is insufficient. Many recent imitation learning based works are missing, such as Interfuser[1], ReasonNet[2], DriveAdapter[3], etc,.

6. Line 191 has a format problem. The format of the subsubsection is inconsistent in the paper.

[1]: InterFuser: Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

[2]: A New Paradigm for End-to-end Autonomous Driving to Alleviate Causal Confusion

[3]: ReasonNet: End-to-End Driving with Temporal and Global Reasoning

1. The visual environment in this simulation is very simple, which means the proposed method would not be practical in real-world applications. Additionally, it seems this paper mainly addresses motion planning. Based on these, the visual environment should be more complicated if the visual encoder is used, such as changing the shape and color of the simulated "cars". Otherwise, the visual encoder should not be used and only use langugae modality for motion planning.

2. The information between different modalities should not include each other. As shown in Figure 2, it doesn't make sense that the system prompt directly points out color information in the visual images.

• The previous methods focus on individual modeling of personal auto-drive could provide a different value as data-driven personal design of the driving, which is not a drawback. Author should put more focus on the method comparison, as general preference personalization and individual personalization are two different directions of the applications.

• Prompt setup is the key for the framework, and it lacks of the detail contents of it, e.g., traffic rule, as a general pre-set prompt, what the principle to design it, it seems if the rule is define by rule-base or limited items, does it enough to cover the different scenarios in the real world.

• Slow, normal, fast how it is utilized in the pipeline is not clear, it seems the mode is defined in the personal prompts, but the example prompt can not find the clue of the definition of it.

• The train process is not clear, which loss and which part is the train target is not mentioned. It suggests using the math symbol and equations to describe the data and train process.

• Proposed dataset is unbalanced for use, and the train on the top of it may cause the result to be preferred to give the action which is the major in the dataset to acquire high scores.

• The most concerns are, scene understanding and danger level estimation as the base information for the action decision are not evaluated to show how accurate it is, as the two important parts of the model.

• Table 3, ablation study should keep the Mode as the same; Comfort metrics need to give clear definition;