Learning a Bi-directional Driving Data Generator via Large Multi-modal Model Tuning

1. The model's performance in more diverse and complex driving scenarios beyond the tested environments may require further exploration and validation.
2. The evaluations appear to be inadequately conducted. The authors assert the existence of 19 potential answer classes; however, they report only the quantitative results from the overtaking prediction task. Furthermore, the zero-shot testing with GPT-4o is the sole baseline selected for comparison. There are also no numbers supporting the claim that training trajectory description and trajectory generation at the same time would be a more favorable approach.

1. Unmatched motivation and proposed solutions. To my understanding, the stated motivation in the abstract is the lack of multi-modal data and the high cost of obtaining labeled data for training, but the proposed solution is a model that can match the performance of LLMs but can be adopted in a resource-constrained setting. There is a gap between them. In the context of generating multi-modal data, why do we need a resource-constrained model? Do the authors try to generate data during driving, or deploy the generation model in the vehicle? If not, why is there a need to design such a model?

2. Insufficient contribution. As the authors said between lines 88 and 97, the existing LLMs cannot fully comprehend the complicated multi-modal connections between trajectories and languages due to the lack of readily accessible world-knowledge. Hence, we would expect that with the proposed solutions, the generation performance should at least outperform the existing LLMs, despite the model size. But so far, the annotation performance is only compatible, and hence the proposed solution is not as effective as the authors claimed.

3. Poor readability in terms of images and texts. The images are not aligned with the text around it. For example, Fig. 3 is too far away from the text describing it. Fig. 4 is 2-pages away from the corresponding text. Readers may get confused by the images and find it difficult to find the text and hence fail to follow.

4. Unclear method description. Maybe I miss something. In the trajectory generation part, the loss is the auto-regressive loss between the generated trajectories and the actual ones. If the authors aim to fine-tune the model based on this task, then we are assuming that the model is the core component controlling how the trajectories are generated. But as we include human languages or prompts here, is it possible that the inputs are affecting the generating performance? Do we consider any loss in terms of the discrepancy between the generated trajectories and what the prompts ask for?

1. The examples shown in Figure 5 are quite confusing. First, the generated trajectories violate the vehicle dynamics and are out of road in most times. The first point of the generated trajectory in the middle figure is behind the last point of the history trajectory. The first point of the generated trajectory in the right figure is too far away from the last point of the history trajectory. Second, it is hard to identify Spinout, Stay-behind, and Overtake in these figures. In summary, I think the generated trajectories have low quality and low realism.
2. I feel the evaluation of this paper is quite limited. It seems that this paper only focuses on high-performance multi-car racing scenarios, as mentioned in the abstract. Even though, I think it is still important to show quantitative results of the average performance of the proposed method. However, the only numerical evaluation now is the overtake classification task shown in Figure 4. I think it is necessary to show the evaluation of realism, diversity, and instruction following. In addition, scenario generation has been a widely investigated area, which means it is easy to find comparable baseline methods, for example, LCTGen [1] and ProSim [2].
3. There is no evidence to show the benefit of using the generated scenarios for downstream tasks. The only example is the overtake classification task. But I am not sure how large the value is of identifying if a scenario is overtaking or not. I think it is more important to show that the generated scenarios help with the training and testing of autonomous agents in terms of performance and safety.

[1] Tan, Shuhan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Kraehenbuehl. "Language conditioned traffic generation." arXiv preprint arXiv:2307.07947 (2023). [2] Tan, Shuhan, Boris Ivanovic, Yuxiao Chen, Boyi Li, Xinshuo Weng, Yulong Cao, Philipp Krähenbühl, and Marco Pavone. "Promptable Closed-loop Traffic Simulation." arXiv preprint arXiv:2409.05863 (2024).

1. While the motivation and methodology are sound, the experimental setup seems too simplistic to fully validate the capabilities of Bi-Gen. The authors mention that there are only 19 possible answers in their question-answering task, which is more akin to a classification problem. This limited setup may not adequately demonstrate the LLM's ability to freely annotate trajectories in an auto-regressive manner. More open-ended annotation would be valuable.
2. The dataset used for training and evaluation is relatively small, with only 877 trajectories collected. Moreover, the participants were racing against fixed trajectories rather than human players or other agents, which limits the diversity and complexity of the driving behaviors captured. A larger and more varied dataset would provide a more robust evaluation of Bi-Gen.
3. Given the large capacity of LLMs, it is possible that Bi-Gen is overfitting to the training data. The authors do not provide sufficient qualitative results to assess the model's generalization capabilities. It would be beneficial to compare Bi-Gen with a baseline method that uses classification-based annotation and recurrent trajectory generation, rather than solely comparing it with GPT-4o.
4. The binary classifier used to validate the quality of the generated trajectories may not be a strong indicator of performance if the data distribution is too simple. It is unclear whether the higher accuracy achieved by the classifier is due to the quality of the generated trajectories or the simplicity of the data distribution.
5. The paper heavily relies on the supplementary material to provide important details about the methodology and results. Some of this information should be included in the main paper to improve clarity and completeness. Additionally, there is redundant information in the main paper, such as the repeated mention of the model pipeline components (system prompt, map, opponent trajectory, and ego-trajectory).

1. The experiments focus on a racing domain with specific trajectory types, which may not generalize well to broader driving scenarios or other real-world applications without additional testing. I want to know if it's possible to extend to multi-agent scenarios, for example Waymo or Nuplan scenarios.
2. While the use of lightweight encoders and TinyLlama enhances efficiency, it might limit the model's capacity to capture finer details in complex, multi-modal interactions compared to larger models.
3. Bi-Gen’s performance relies on well-defined question-answer and generation prompts, which may limit its adaptability to novel or unexpected queries in deployment.
4. The paper does not explore the impact of different architectural choices (e.g., encoder and decoder sizes, tokenization approaches), which would strengthen understanding of the model's design trade-offs.
5. Following point 4, the tokenization approaches to the map and trajectory are unclear. See questions.