Language-Driven Interactive Traffic Trajectory Generation

Response to Reviewer awjY

$**Question1:**$ Since the training of the proposed method also still relies on pre-collected ground truth data, I think the authors need to compare it with CTG and CTG++. Another reason is these two methods are also based on language conditions.

$**Answer1:**$ The setting of CTG and CTG++ is different from our method setting of language-conditioned trajectory generation. CTG and CTG++ require past trajectory observations to generate traffic trajectories, as outlined in their problem formulation (Sec.III-A in CTG and Sec 3.1 in CTG++). Without past trajectory observations, these models are unable to produce plausible trajectories. This is the primary reason why these two models were not selected when comparing our model to the baseline models.

$**Question2:**$ How diverse are the trajectories generated by the proposed method? Will they be similar to the training data?

$**Answer2:**$ Because of the generalization ability and multiple possibilities of the LLM's response, diversified scenarios can be generated from the same language input. Figure S4-1, S4-2 and S4-3 in the rebuttal pdf demonstrate one such case. For the same language input, the LLM will generate satisfying responses with variations, reflecting the realism and diversity of the generated scenarios by our model. Additionally, because of the LLM's generalization ability and multiple possibilities of LLM's response, the generated trajectories may differ significantly from the training data.

$**Question3:**$ In some visualization results, there are some collisions in the generated vehicle trajectories, such as the red vehicle in the overtaking in Figure 5, and the orange vehicles in Figures 7(b) and 8(b). What is the reason for this?

$**Answer3:**$ The vehicles involved in collisions in these figures mentioned are from scenarios generated by baseline models (LCTGen). This is due to two main reasons: i) the baseline method does not consider vehicle interaction and generates vehicle trajectory independently, which can lead to conflicts or collisions between different vehicle trajectories. ii) the intermediate code design in the baseline method does not account for the global movement trend, thus there may be conflicts between the vehicle end state and map boundary. In comparison, our method i) considers interaction-aware numerical codes to jointly generate interaction-aware vehicle trajectories to avoid unreasonable collisions; ii) introduces the global trend in the vehicle code V to provide a more reasonable overall trajectory moving states, thereby avoid the collision between the vehicle end state and map boundary.

$**Question4:**$ What is the size of the trajectory data generated by the proposed method? The visualization results show that the number of vehicles is not large. Can a more complex trajectory be generated?

$**Answer4:**$ We generate a 5-second trajectory at 10 fps for each vehicle. For the requirement of generating more complex traffic scenarios, LLM can smoothly assign appropriate vehicle codes to each vehicle and analyze inter-agent interactions to produce reasonable traffic scenarios. When the number of agents increases, our model is still able to generate scenarios that conform to linguistic descriptions. Figure S5 shows our model's ability to generate complex scenarios with more than 5 vehicles. Even as the number of vehicles in the scene increases, our model maintains its performance in scenario generation, handling cases with up to ten vehicles well, whereas the baseline method often results in significant vehicle collisions.

Thanks for your reply.

Response to Reviewer xSeP

$**Question1:**$ It seems the proposed framework is computationally expensive; the authors should provide the details about the efficiency of the model.

$**Answer1:**$ The overall resources needed for model training and inference are not expensive. Only the code-to-trajectory decoder needs to be trained as the LLM in our model encoder does not need training. It takes about 12 hours for 100 epochs on 4 NVIDIA GeForce RTX040 GPUs for the decoder training process. During the inference phase, it takes about 30 seconds to generate a scenario, which includes about mainly 30 seconds to receive the response from the LLM and about 0.2 seconds to generate corresponding trajectories.

$**Question2:**$ Is the language description encoder used in this paper GPT-4? There is no comparison of the effects of different large language models as the encoder.

$**Answer2:**$ We use GPT-4 as the language description encoder in our paper, but many other LLMs also can be incorporated into our work. Here we further show the result of applying two other representative LLMs as encoders, including the Llama3-3.1-70b and Mistral-large. See Figure S2, which illustrates the visualization results provided by both Llama3 and Mistral-large for different types of interaction. We see that applying other LLMs also achieves compliant scenario generation.

$**Question3: **$ Controllability is an important contribution of this paper, which should be validated more intuitively (visual) or quantitatively.

$**Answer3:**$ Our method can control the specified vehicle trajectory generation in the scene. As controllability relies on the language input, here we validated more intuitively through visualization. Figure S3 shows the generation results under four different control descriptions applied to ego vehicles. We observe that our method can control specific vehicle trajectories through different linguistic descriptions.

$**Question4:**$ Could the proposed model handle the different lengths of the generated trajectories? or it only generates the fixed-length trajectories?

$**Answer4:**$ The length of the generated trajectories is pre-defined before the model is trained. In our paper, we generate 5-second trajectories at 10 fps in experiments. The setting of fixed-length trajectories is common in related tasks like trajectory generation [1,2] and prediction [3,4]. [1] Learning to generate diverse and realistic traffic scenarios, ICRA 2023 [2] Language-conditioned traffic generation, CoRL 2023 [3] Motiondiffuser: Controllable multi-agent motion prediction using diffusion, CVPR 2023 [4] MotionLM: Multi-Agent Motion Forecasting as Language Modeling, CVPR 2023

Response to Reviewer YJzW

$**Question1:**$ Clarify how the model performs when dealing with emergent or less common interaction types, like parallel driving or platooning? Would the model generalize well to such scenarios, or would it require retraining?

$**Answer1:**$ Our model can generate compliant scenarios when dealing with less common interaction types. The left part of Figure S1 in the rebuttal pdf shows the experimental results. We present the results for rare interaction types including mentioned uncommon cases of parallel driving (Figure S1-1), platooning (Figure S1-2), and another uncommon case involving pulling over (Figure S1-3). We see that for less common interaction types, our method effectively translates the relevant behaviors to generate compliant scenarios. Our model is able to generalize scenarios with emergent or less common interactions without requiring retraining. The main reason is that the LLM used in our encoding process possesses strong generalization abilities. The large language model can understand linguistic descriptions of uncommon interaction scenarios and convert them into appropriate numerical codes for decoding. Then the decoding process, which also has generalization ability due to training with massive numerical codes, would translate these codes into trajectories and generate compliant scenarios.

$**Question2:**$ How does InteractTraj handle ambiguity in language descriptions, like phrases like "the car behind" in dense traffic scenarios? Does the model prioritize proximity or other factors when resolving such ambiguities?

$**Answer2:**$ We leverage the reasoning ability of the large language model to manipulate linguistic inputs to solve possible ambiguity problems. We do not introduce additional prioritized proximity or other factors. Ambiguous situations mainly fall into two categories: those where the reference to the object is unclear, and those where there is a contradiction within the language instruction itself.

• For the first type, LLM would understand language descriptions and convert them into numerical codes to satisfy one of the meanings of phrases. We present the result of the mentioned phrase "the car behind", see the right part of Figure S1 in the rebuttal pdf. We see that when the reference to "the car behind" is ambiguous (Figure S1-4), the LLM would randomly designate one vehicle as the "behind car". However, with clearer descriptions like "the car behind ego car" (Figure S1-5), the LLM would accurately handle the descriptions.
• For the second type involving self-contradictory requirements, the scenarios generated would partially meet the instructions' criteria. Figure S1-6 illustrates this with an example of language instruction "A car is turning left and moving in a straight line." LLM may opt to generate an intersection in the scene to fulfill the left-turning requirement while disregarding the requirement for straight-line movement. To better tackle language ambiguity, an optimal solution involves introducing LLM-human interaction to iteratively verify language descriptions. This approach will be explored in our future work.

$**Question3:**$ Currently, the focus is on generating trajectories for vehicles only. This limitation could be addressed by extending the model to encompass a broader range of entities.

$**Answer3:**$ At present, our work concentrates on trajectory generation for vehicles since vehicles are the most common subject, specifically creating interactive scenarios by modeling vehicle interactions. To incorporate various types of traffic participants, such as pedestrians and cyclists, we could utilize specialized heterogeneous networks as the backbone for the decoder network. Additionally, we plan to design specialized dynamic models, such as the bicycle kinematic model, tailored for different subjects. This comprehensive systematic design will be pursued in future work.

$**Question4:**$ The authors admit that map generation is restricted by the available map library. The system could be expanded to generate more diverse and flexible maps, possibly through generative models.

$**Answer4:**$ We use the current map library method since i) it ensures the obtained map is totally realistic and ii) the built map library could already support plenty of trajectory generation cases. Although a map generation method could generate more flexible maps, it may encounter the problem of being unrealistic. Meanwhile, in this work, our main solving problem is how to generate traffic trajectories with interactions. Thus, we hope the map is ensured to be realistic to better evaluate our trajectory generation design. In future work, we will switch the focus to map generation. To expand our method with more efficient map generation methods to obtain maps that conform more closely to linguistic inputs, we plan to use masked prediction for map expansion given partial map observations or use diffusion models for map generation.

$**Question5:**$ While the model performs well with common interaction types, its ability to handle more complex or rare interactions has not been thoroughly evaluated. The authors could consider testing the model's performance in scenarios involving intricate multi-agent interactions.

$**Answer5:**$ We have analyzed and shown some qualitative visualizations of complex or rare interaction cases in the responses A1 and A2. We will include more results of complex and rare interactions in the revised version. The quantitative evaluation of intricate multi-agent interactions is restricted by the existing benchmarks since there are no direct datasets full of intricate interactions labeled with language descriptions. In the future, when direct language-to-scenario datasets or specific datasets focusing on intricate interactions are available, we will conduct a more thorough evaluation of intricate multi-agent interaction cases.

Thank you for taking the time to address my comments