Trajectory-LLM: A Language-based Data Generator for Trajectory Prediction in Autonomous Driving

We sincerely thank you for your valuable comments. Below, we respond to your questions point-to-point.

Q1: I did not see which specific large language model was used; the only information provided in the system diagram shows the LLM has an encoder-decoder structure, which does not seem to align with GPT-like models. I urge the authors to elaborate on the design of the large language model.

Answer: We respectfully clarify that the specific LLM used in this paper is Llama-7B (see Section C of the supplementary file). Furthermore, we have provided an anonymous GitHub repository (https://github.com/anonymous-github-Traj-LLM/Traj-LLM) for the complete implementation of Traj-LLM to help readers to follow our work.

We sincerely thank you for pointing out this confusion about our framework's encoder-decoder structure. Based on the fundamental Llama-7B, Traj-LLM follows the decoder-only structure like GPT. In Figure 2 of our framework, the encoder embeds the texts of vehicle interactions and behaviors into hidden features. It actually acts as the text embedding layer. To eliminate the confusion, we have modified the encoder in Figure 2 and its definition in the relevant formulations to the text embedding layer.

Q2: As a generative model designed for trajectory prediction tasks, the experimental section is insufficiently thorough. Firstly, it only selects the WOMD dataset and does not test on other representative datasets, such as Argoverse. Secondly, it only uses mAP as the metric, neglecting other common metrics such as distance error and miss rate. Therefore, the generalizability of the proposed method remains questionable.

Answer: Thanks for your suggestion. We have added the metrics of distance error and miss rate to the results on the Waymo dataset (see Table 4 of the paper and Tables 5-7 of the revised supplementary file). We have also added the trajectory prediction performances on the Argoverse dataset, where the vehicle trajectories generated by Traj-LLM also improve the downstream prediction model (see Table 7 of the revised supplementary file).

Q3: Training a generative model on a third-party dataset and then using the generated data for model training—does this yield effects equivalent to training on the original third-party dataset? I recommend the authors include an ablation study comparing the experimental results of training with the original dataset.

Answer: We respectfully note that the experimental results in Table 5 of the revised supplemental file compare the training on the original and generated data. As shown in the first row (the original L2T dataset) and the fourth row (the data generated by Traj-LLM), training on the generated data yields better results. For a fair comparison, the amounts of original and generated samples for training are the same. This performance improvement stems from the data diversity brought by the generation process of Traj-LLM, which helps the downstream prediction model to witness vehicle trajectories with novel shapes. We remark that the data diversity of our generated trajectories has been evidenced in Table 1 of the paper.

Q4: Can the proposed model be used for simulating traffic scene? I suggest the authors consider this application area, rather than limiting the focus solely to trajectory prediction tasks.

Answer: Thanks for sharing this inspiring idea. Without needing to modify the main model architecture, a trivial way of extending Traj-LLM to simulate the traffic scene is to input the interactions between every pair of vehicles to generate all vehicles' trajectories. In this paper, we have experimented with using Traj-LLM to yield the trajectories of up to 5 interactive vehicles in the same scenario, which is already the basic traffic scene.

The above trivial solution may need complex text descriptions of vehicle interactions, especially for traffic scenes containing many vehicles. A possible solution is to prepare higher-level descriptions of traffic scenes in short texts for training Traj-LLM, rather than using the complex text descriptions of all vehicle interactions. This could be investigated in our future work. It also remains a problem of efficiently combining the scene- and vehicle-level descriptions in a short text, which Traj-LLM parses to control vehicle interactions to make them consistent in the same scene to avoid collision or out-of-map.

We sincerely thank you for your valuable comments. Below, we respond to your questions point-to-point.

Q1: The proposed dataset L2T is collected in simulation platform and manually use the wheels to drive and collect the data. The realism of the proposed dataset is questionable. Further justification and motivations are needed for creating a new dataset with good realism instead of curating existing open real-world dataset to create such paired dataset.

Answer: We sincerely appreciate your feedback on our dataset. We respectfully clarify that many vehicle trajectories in datasets like Waymo and Nuscenes lack or only have weak interaction. This phenomenon is mainly attributed to the high risk of traffic accidents associated with collecting vehicle trajectories with high-intensity interactions on real roads. On the other hand, vehicle trajectories with high-intensity interactions play a crucial role in training autonomous driving systems, as they help the systems to reasonably handle many vehicle interactions that are prone to causing traffic accidents.

Rather than taking the risk of collecting data on the real-world road, we resort to simulating the vehicle trajectories with high-intensity interactions in the virtual world. We construct a new simulator to increase the realism of vehicle trajectories collected in the simulated environment. Our simulator integrates the re-developed hardware and software to efficiently collect and annotate the vehicle trajectories in the L2T dataset. We have added more details about our simulator to Section A.1 of the revised supplementary file.

The hardware component of our simulator mainly includes a steering wheel, a gear shifter, gas and brake pedals, and several displays, as shown in Figure 7(a) of the supplementary file. The displays visualize the first-person view of the road environments from the left, right, front, and back car sides, as shown in Figure 7(b). Here, the road environments are simulated by LGSVL, a simulation engine for autonomous driving. Our hardware simulates a realistic car cockpit, helping users to drive on the virtual road to produce vehicle trajectories as real as possible.

The software component is mainly built on top of the LGSVL engine. LGSVL can record all trajectories of vehicles traveling on the virtual road. We re-develop this engine to enable the connection among multiple sets of the above hardware, thus allowing multiple users to control different cars on the same virtual road. This software facilitates the simulation of vehicle interactions like those in the real world. In Figure 8, we show the interactive cars in the virtual road from the third-person view.

In Section A.3 of the revised supplementary file, we have compared the data distribution of our L2T dataset and other real-world datasets like Waymo. In Figure 12, we compare the distributions of trajectories' shapes represented by the waypoints' coordinates. The distributions are similar, showing that the vehicle trajectories in the L2T dataset are realistic.

Q2: The main benchmark is performed on the proposed L2T and open dataset WOMD is only used in ablation study while existing literature mainly compared on other open datasets. Without benchmarking on the same open dataset, it would be hard to justify the technical contribution of the proposed model.

Answer: Thanks for your constructive comment. As you requested, we have compared different trajectory generation models on the real-world Waymo dataset to measure the realism and diversity of the generated trajectories. The experimental results are in Table 8 of the revised supplementary file, where Traj-LLM outperforms other methods. These results show that Traj-LLM's generated trajectories also deliver realism and diversity compared to real-world data rather than the simulated L2T dataset only.

Apart from the downstream trajectory prediction on the Waymo dataset in Table 4 of the revised paper, we have added the trajectory prediction performances on the Argoverse dataset, where the vehicle trajectories generated by Traj-LLM also improve the state-of-the-art prediction model (see Table 7 of the revised supplementary file). These results show that the vehicle trajectories generated by Traj-LLM deliver generalization across different datasets like Waymo and Argoverse.

Dear Reviewer v7Vc,

We thank you again for your valuable comments, which significantly helped us polish our paper. Could you kindly let us know if the responses have addressed your concerns and if further explanations or clarifications are needed? Your time and efforts in evaluating our work are appreciated greatly.

Best,

Authors of Paper ID 4151

Dear Reviewer v7Vc,

Thank you again for your review. We are pleased to see that your questions have been answered. We will follow your suggestion to add more detailed information to our paper.

Best,

Authors of Paper ID 4151

Q4: The paper does not specify what kind of input was used to train the LLM model. Did they use generated prompts, or did they provide feature embeddings?

Answer: Llama-7B is the fundamental LLM in our implementation (see Section C of the supplementary file). A training sample for LLM can be regarded as a triplet (interactions, behaviors, trajectories). In this triplet, vehicle interactions and behaviors are text descriptions (see examples in Figure 4(c) and (d)). During the training, LLM takes input as the short text of interactions, learning to parse the interactions and outputting more detailed text of behaviors. The output behaviors and the ground truth provided in the training sample are used to calculate the loss. Next, LLM takes input as the behaviors to output the trajectories with numerical coordinates, which are used to compute the loss with the ground truth in the triplet sample.

We sincerely thank you for your valuable comments. Below, we respond to your questions point-to-point.

Q1: The clarity of the paper can be improved. In the introduction, the motivation and objectives should be clearly stated, along with a summary of their accomplishments.

Answer: We sincerely thank you for your valuable suggestion. In the revised introduction section (see the third paragraph), we have compared the language interface based on the large language model (LLM) to the graphical and programmable interfaces for generating vehicle trajectories. The language interface with LLM can leverage the short text of vehicle interactions to generate vehicle trajectories with detailed motion parameters. It means that users can rapidly achieve a large amount of trajectory data, thus saving data collection efforts. This efficiency motivates us to study LLM as the trajectory generator.

The objective can be found in the last sentence of the fourth paragraph of the revised introduction. In this paragraph, we have discussed the shortcomings of the existing language interfaces. Without the guidance of human beings' driving logic, these language interfaces conduct "interaction-trajectory" translation, which means the text description of vehicle interactions is directly translated into trajectories. They likely generate vehicle trajectories with unreasonable driving behaviors. Moreover, they likely fail to generate diverse trajectories according to the unseen maps, lacking the data diversity centered on the trajectory generation task. Thus, the primary objective of this paper is to involve the human driving logic to guide the translation between the text description of vehicle interactions and trajectories.

In the last paragraph of the revised introduction, we summarize our contributions as:

• We advocate a new paradigm of the language interface with the“interaction-behavior-trajectory.
• We propose a novel language interface, Traj-LLM. We also contribute a new L2T dataset containing 240K traffic scenarios with vehicles' interactive trajectories and rich text descriptions.
• We use the vehicle trajectories generated by Traj-LLM, which is trained on the L2T dataset, to finetune the trajectory prediction models, whose performances are effectively improved on the public Waymo and Argoverse datasets. These results can further inspire relevant research on the language-based trajectory generator.

Q2: It is unclear how the trajectories are generated. Are they based on real data, generated through a simulator, or did the authors develop their own simulator?

Answer: Our simulator integrates the re-developed hardware and software to efficiently collect and annotate the vehicle trajectories in the L2T dataset. We have added more details about our simulator to Section A.1 of the revised supplementary file.

The hardware component of our simulator mainly includes a steering wheel, a gear shifter, gas and brake pedals, and several displays, as shown in Figure 7(a) of the supplementary file. The displays visualize the first-person view of the road environments from the left, right, front, and back car sides, as shown in Figure 7(b). Here, the road environments are simulated by LGSVL, a simulation engine for autonomous driving. Our hardware simulates a realistic car cockpit, helping users to drive on the virtual road to produce vehicle trajectories as real as possible.

The software component is mainly built on top of the LGSVL engine. LGSVL can record all trajectories of vehicles traveling on the virtual road. We re-develop this engine to enable the connection among multiple sets of the above hardware, thus allowing multiple users to control different cars on the same virtual road. This software facilitates the simulation of vehicle interactions like those in the real world. In Figure 8, we show the interactive cars in the virtual road from the third-person view.

In Section A.3 of the revised supplementary file, we have compared the data distribution of our L2T dataset with other real-world datasets like Waymo. In Figure 12, we compare the distributions of trajectories' shapes represented by the waypoints' coordinates. The distributions are similar, showing that the vehicle trajectories in the L2T dataset are realistic.

Q3: The methodology section is difficult to follow and needs rewriting for better clarity.

Answer: We sincerely appreciate your feedback on our presentation. We respectfully invite you to specify which part of the methodology section is difficult to understand. This will help us to reorganize the content and make it more understandable. Additionally, we have provided an anonymous GitHub repository (https://github.com/anonymous-github-Traj-LLM/Traj-LLM) for the complete implementation of Traj-LLM to help readers to follow our work.

Again, we thank Reviewer baVM for the review. In the last round of review and rebuttal, we have further explained the input data and the overall workflow of our approach (see our answer to Reviewer baVM's Q4). We have also respectfully invited Reviewer baVM to specify which part of the methodology section is difficult to understand (see our answer to Reviewer baVM's Q3). Given that Reviewer baVM's question Q3 about the missing details of the proposed approach is unclear, and now Reviewer baVM has repeated this question again, we kindly ask for Reviewer baVM's specific comment on which part of the methodology section is difficult to understand. This definitely will be critical for improving our paper. Here, we express our greatest gratitude to Reviewer baVM's help!

We are pleased that Reviewer baVM found the additional information in the supplementary material useful. However, it is infeasible to include all the details in the main paper, which has a 10-page limit. Again, we want to emphasize that we are always willing to modify and provide details as long as the reviewer can give us specific comments. If required, we can exchange the necessary contents of the supplementary file and less important contents of the main paper.

Dear Reviewer baVM,

Again, we want to express our sincere thanks for the insightful feedback you have provided. Though the discussion deadline is approaching, we hope to receive your valuable comments on our paper presentation. Undoubtedly, your suggestion is critical for the success of this research work.

Thanks again for your time and effort in reviewing our paper.

Best,

Authors of Paper ID 4151

Q3: The authors have experimented with the proposed method in open-loop settings; it would be good to see how the proposed method performs in the closed-loop evaluation.

Answer: Thank you for sharing this inspiring idea. Here, we respectfully clarify the central purpose of this paper, which is to find a cost-effective way of collecting a large amount of vehicle trajectories with high-intensity interactions. Rather than collecting these trajectories by taking the risk of traffic accidents, we employ LLM, the most advanced language-understanding technique, to parse the brief text of vehicle interactions to generate the trajectories. These generated trajectories play as the data for training the downstream autonomous driving models like trajectory prediction. From the view of the application of generated trajectory data for the downstream model training, LLM generates trajectory data in the open- or closed-loop does not matter. This is because the data generation and downstream model training are separate, which means the downstream model training is unaware of the open- or closed-loop generation. Thus, this paper focuses more on increasing the realism, diversity, and controllability of the trajectory generation process, which are more relevant for the performance improvement on the downstream model.

We agree with the reviewer's view of making LLM work in the closed-loop setting. This setting may allow LLM to generate vehicle trajectories that represent interactions under the circumstances with unpredictable changes, thus helping to train more robust downstream models. However, letting LLM work in the closed-loop setting means LLM has to interrupt its generation based on the current environment as input and start a new reasoning using the changed environment. This process results in problems such as the waste of computations performed before the interruption and the slow speed of data generation. In contrast to applying LLM to understand human dialog, the above problems relevant to autonomous driving become particularly prominent when the road environment is complex and ever-changing. They must be solved perfectly before implementing the closed-loop LLM to generate trajectory data.

Q4: It is good to see that authors have also dedicated a section in supplementary material for the limitation of their work, but it would be good to see what the future directions will be based on their work.

Answer: Thank you for pointing out this loss of information. We have modified our discussion of the method's limitation in Section E of the supplementary file. Traj-LLM relies heavily on a standalone polyline encoder to embed the map information into the hidden feature, which is concatenated with the interaction and behavior features for generating the vehicle trajectories. It means that Traj-LLM does not comprehensively parse the map information. This differs from the text of vehicle interactions and behaviors that are well parsed by Traj-LLM and embedded into features. In the future, we will follow two directions to solve the above problem:

• We can investigate how to transform the map's polylines into text properly. Traj-LLM, which has been pre-trained on a large amount of textual data, may be better able to parse the textual map.
• We can also study how to make Traj-LLM a large visual-language model. In this manner, the model can regard the map as an image representing not only the lane lines but also other environmental factors like trees, buildings, etc. The model trained on the map images can parse the maps with complex environmental factors, generating the vehicles that fit the maps.

Q5: In the paper, the author has used random locality attention; it would be good to do an ablation study on its effect on driving behavior and also the trajectories. Also, if another form of attention, for instance, cross attention, is used, what will affect behavior and trajectories?

Answer: Thanks for your valuable advice. In the third row of Table 3 in Section 5.2, we remove the random and locality properties to degrade the random locality attention to a vanilla cross-attention, yielding lower realism, diversity, and controllability than the random locality attention. We have modified the text of Section 5.2 to clarify the above point. We have also conducted the ablation study by removing random or locality, reporting the results from the second to the sixth rows of Table 3. These alternatives degrade the quality of trajectory generation. The experimental results demonstrate the effectiveness of enabling the random and locality together.

We sincerely thank you for your valuable comments. Below, we respond to your questions point-to-point.

Q1: I acknowledge that the authors have put great effort into creating the dataset, but it was unclear whether they have used existing datasets, for instance, Waymo and Nuscenes, to build their dataset.

Answer: We sincerely appreciate your feedback on our dataset. We respectfully clarify that many vehicle trajectories in datasets like Waymo and NuScenes lack or only have weak interaction. This phenomenon is mainly attributed to the high risk of traffic accidents associated with collecting vehicle trajectories with high-intensity interactions on real roads. On the other hand, vehicle trajectories with high-intensity interactions play a crucial role in training autonomous driving systems, as they help the systems to reasonably handle many vehicle interactions that are prone to causing traffic accidents. Thus, we do not use these existing datasets to build our dataset, where our dataset focuses on high-intensity vehicle interactions.

Rather than taking the risk of collecting data on the real-world road like the existing datasets, we resort to simulating the vehicle trajectories with high-intensity interactions in the virtual world. We construct a new simulator to increase the realism of vehicle trajectories collected in the simulated environment. Our simulator integrates the re-developed hardware and software to efficiently collect and annotate the vehicle trajectories in the L2T dataset. We have added more details about our simulator to Section A.1 of the revised supplementary file.

The hardware component of our simulator mainly includes a steering wheel, a gear shifter, gas and brake pedals, and several displays, as shown in Figure 7(a) of the supplementary file. The displays visualize the first-person view of the road environments from the left, right, front, and back car sides, as shown in Figure 7(b). Here, the road environments are simulated by LGSVL, a simulation engine for autonomous driving. Our hardware simulates a realistic car cockpit, helping users to drive on the virtual road to produce vehicle trajectories as real as possible.

The software component is mainly built on top of the LGSVL engine. LGSVL can record all trajectories of vehicles traveling on the virtual road. We re-develop this engine to enable the connection among multiple sets of the above hardware, thus allowing multiple users to control different cars on the same virtual road. This software facilitates the simulation of vehicle interactions like those in the real world. In Figure 8, we show the interactive cars in the virtual road from the third-person view.

In Section A.3 of the revised supplementary file, we have compared the data distribution of our L2T dataset with other real-world datasets like Argoverse and Waymo. In Figure 12, we compare the distributions of trajectories' shapes represented by the waypoints' coordinates. The distributions are similar, showing that the vehicle trajectories in the L2T dataset are realistic.

Q2: If this is not the case, and they adopt some manual feedback from the driver, how can they evaluate those behavior's annotations?

Answer: Section A.1 of the revised supplementary file introduces how to evaluate the annotation quality. We recruit 30 drivers with rich experience to explain and document vehicle interactions and behaviors with driving logic. We conduct a two-level check to evaluate the annotation quality. We develop an automated Python script at the first check level to verify whether the trajectories align with the text interactions and behaviors. These scripts incorporate some basic checking rules, such as the rule that the overtaking between two vehicles must involve acceleration, and the behavior parameters within the acceleration must be equivalent to the motion parameters of the trajectories. We have uploaded these scripts to the anonymous GitHub repository (https://github.com/anonymous-github-Traj-LLM/Traj-LLM/tree/master/auto_behavior_detection).

At the second check level, we divided the scenarios that have passed the first check level into 10 groups. Three drivers check the annotations of each group of scenarios. We randomly select 20% of scenarios from each group. If half of the selected scenarios have false annotations, all scenarios in this group must be re-annotated; otherwise, only the scenarios with false annotations are re-annotated.

Thank you for providing the detailed response. Well the authors have mentioned in their response as "We respectfully clarify that many vehicle trajectories in datasets like Waymo and NuScenes lack or only have weak interaction. This phenomenon is mainly attributed to the high risk of traffic accidents associated with collecting vehicle trajectories with high-intensity interactions on real roads." However, these statements appear contradictory. Waymo and NuScenes are widely adopted datasets, and extensive research has utilized them for motion planning and pedestrian interaction. In the subsequent paragraph, the authors mention using a simulator to collect their dataset. While it is true that simulators provide inherent safety advantages, given the availability of well-established interaction datasets, the question arises: what is the specific impact or added value of this dataset? One more concern that is already raised by one of the reviewer is the evaluation of proposed method on other open-loop dataset for instance NuScenes. Also one minor or on a presentation side comment is that authors should put the necessary (that helps reader to understand) information in the main paper rather than in supplementary material. I think supplementary material is just to complement your paper.

Q1: Well the authors have mentioned in their response as "We respectfully clarify that many vehicle trajectories in datasets like Waymo and NuScenes lack or only have weak interaction. This phenomenon is mainly attributed to the high risk of traffic accidents associated with collecting vehicle trajectories with high-intensity interactions on real roads." However, these statements appear contradictory. Waymo and NuScenes are widely adopted datasets, and extensive research has utilized them for motion planning and pedestrian interaction.

Answer: We are sorry for the misunderstanding raised by our previous statement. In Section 4 of our paper, we explained the interactive characteristics of the L2T dataset, describing them as "These high-intensity interactions should be completed in a short range of less than 10 meters." In contrast, within real-world datasets such as Waymo and NuScenes, most interactions are maintained at a safe distance (>15m), which we call weak interactions. A detailed explanation is in the next answer to Q2.

Q2: In the subsequent paragraph, the authors mention using a simulator to collect their dataset. While it is true that simulators provide inherent safety advantages, given the availability of well-established interaction datasets, the question arises: what is the specific impact or added value of this dataset?

Answer: As previously mentioned in the Answer of Q1, the L2T dataset is characterized by short-range, high-intensity interactions. In Figure 13 of the supplementary material, we have visualized this discrepancy. It shows that the shortest interaction distances in L2T are more concentrated within the 0-10 meter range, whereas WOMD concentrates within the 15-25 meter range. It demonstrates that interactions in L2T are stronger than WOMD. The characteristic of longer interaction distances in WOMD primarily stems from cautious and risk-averse drivers in the real world. Autonomous driving systems often require a greater capacity to handle high-intensity interactions, which is one of the significances of our dataset. In Table 4-7 of our paper, we demonstrated that adding L2T or generated data from Traj-LLM into the training of motion prediction models can improve their performance on WOMD and Argoverse.

On the other hand, L2T also has rich annotations of traffic scenarios. Different from existing datasets, L2T not only contains high-level annotations about interaction relationships (bypass, overtake, yield) and driving targets (go straight, turn left, turn right), but also includes detailed driving behavior annotations such as cruise and change lane. In the framework of Traj-LLM, we conduct the“interaction-behavior-trajectory" translation and achieve better performance in Realism, Diversity, and Controllability. We hope that L2T can contribute to exploring the enhancement of performance and interpretability of adding textual information in autonomous driving systems. It is also one of our future directions.