WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving

We appreciate the reviewer's insightful and constructive feedback. We address your comments and concerns below.

Q1. Human Assessment Scale.

We are grateful for the reviewer's suggestion on providing stronger evidence of validations. To explore the validity of our assessment, we further enlarge our assessment to ~2.2k Q&As. With that being done, the total correct rate comes from 91.99% in the paper to 91.80%, which does not fluctuate much. This suggests that our sampling for assessment is valid and effective. Besides, we hope to confirm that our dataset improvement and human verifications are continuously ongoing. We will update the dataset and the validity when newer data become available.

Q2. Real-world risks of imperfect (intention) languages.

We thank the reviewers for raising this question. We hope to address that, since language cannot thoroughly describe exact positions and velocities etc., language models are more commonly being used as an auxillary driving model to provide explainability and introduce high-level knowledge (like traffic rules in our case) or explainability. Therefore, its failing in infrequent cases would generally not cause accidents due to its auxillary nature. Furthermore, our intentions part serve as the summary of interactions between ego agent and all other agents, which is a very comprehensive task than other classes, therefore we believe that 87.5% is satisfying for an auto-labeled dataset. To further improve our dataset's quality, we will also update our dataset with human verifications to further ensure its reliability to 99+%.

Q3. Downstream impacts on real-world Autonomous Driving tasks.

We highly appreciate this question. To justify WOMD-Reasoning dataset's ability in downstream tasks, we perform a trajectory prediction task using outputs of our Motion-LLaVA model fine-tuned upon WOMD-Reasoning. Multipath++ is used as the trajectory prediction baseline model. As the interaction part is the most significant part of our dataset, we introduce the interaction part of language output into Multipath++, by cross-attending the T5 embedding of them with corresponding agents' features. Each experiment is runned with 3 seeds to ensure reliability. The results are shown in the following table

We observe a significant performance enhancement using language outputs from motion-LLaVA fine-tuned upon WOMD-Reasoning. This strongly supports WOMD-Reasoning's ability to fine-tune LMs to help downstream tasks like predictions or explainabilities of driving behaviors. We will add final results of these experiments into our manuscript in our next version.

Q4. More quantitative analysis.

We thank the reviewer for suggesting more downstream useful evaluations to prove WOMD-Reasoning's abilities. In Q3 we provide evidence proving the effectiveness of WOMD-Reasoning in trajectory prediction tasks. Since WOMD is a highly-interactive real-world driving dataset, prediction evaluations on WOMD offer evidences of the usefulness of WOMD-Reasoning on real-world applications.

Q5. Validation of Motion-LLaVA - "facts" measured.

We thank the reviewers for requesting details on the definition of facts in Motion-LLaVA evaluations. Our Facts questions can be categorized into 3 types:

1. Environment-related questions focus on the scenario’s surroundings, covering details such as the presence and type of intersections, the number of lanes on the ego vehicle’s side, and the existence and location of crosswalks or stop signs.
2. Ego agent-related questions assess the ego vehicle's characteristics, including its speed, motion status, lane position, and directional attributes.
3. Surrounding agents-related questions focus on other agents in the scenario, covering their type, speed, motion status, and position relative to the ego vehicle and intersection.

We hope this clarifications help, and we will include this in our next version.

Q6. Was Motion-LLaVA fine-tuned with motion prediction task or not?

We are grateful for the question. While Motion-LLaVA is not fine-tuned with motion prediction task, we additionally use Motion-LLaVA outputs to illustrate the effectiveness of WOMD-Reasoning data in motion prediction tasks, the table is shown in Q3. We will add final results of these experiments into our manuscript in our next version.

Q7. Typo in Line 149

We thank the reviewer for pointing this typo out. We will fix this in our next version.

We appreciate the reviewer's insightful feedback. Please see our response below.

Q1. Details on the BEV input used for LLaVA & Evaluations on real-world scenarios.

We appreciate the reviewer's suggestions:

1. For the BEV input, it comes from plotting WOMD data. Road elements are plotted with line segments, while trajectories of agents are plotted with arrow-shaped boxes to indicate position and orientation. Trajectories are labeled with their Agent IDs. Additionally, an XY scale bar is added to help to interpret the position and the velocity.
2. For real-world evaluations, we first address that WOMD is a large real-world dataset. Therefore, testing our model on WOMD's cases can reflect the real world situations. To provide more evals on real-world tasks, we test introducing language outputs of Motion-LLaVA into trajectory prediction model Multipath++, seeing ~7% improvements on minFDE_6 and MR_6 in real-world WOMD cases. Details can be found in our answer to reviewer BaWk's Q2.

Q2. Generalization to other datasets or cases w/o motion inputs.

We appreciate the question on generalizations.

• For other motion datasets, our translation program convert motion data into raw language suitable for feeding LLMs to produce data in our dataset's format. With minor modifications in the translation program's IO part, it can be used for any motion dataset. Our translation program is included in the supplementary materials and will be open-sourced.
• For non-vectorized motion inputs, we believe that the well-developed perception models can convert non-vectorized inputs into vectorized motion data. Furthermore, with our simulated visual modal, users can directly train VLMs with WOMD-Reasoning to take vision inputs.

Q3. BEV preparations and limitations

We thank the reviewers for suggesting the comparison between BEV and motion vectors. The BEV preparations are detailed in Q1. BEV and motion vectors are equivalent representations of the same scenario. However, BEV images convey the coordinates and velocity info more implicitly, which poses challenges to extract these info. Therefore, our Motion-LLaVA choses motion data as inputs.

Q4. Simplifying the visuals.

We appreciate the suggestions on simplifying the visuals to improve readability. We will fix this in the next version.

Q5. Data efficiency.

We thank reviewer's suggestion on showing the data efficiency of WOMD-Reasoning by fine-tuning Motion-LLaVA with different data sizes. The results of the suggested experiments are listed below:

For the factual questions:

For the interaction reasoning questions:

We find that with more WOMD-Reasoning data involved, the fine-tuned model first gains the ability to answer factual questions, which hits a good GPT score with only 0.5M data. As the dataset grows, the model gradually learns to answer interaction reasoning questions. These justify the size of WOMD-Reasoning, as well as its data efficiency in helping models to answer hierarchical driving-related questions.

Q6. Comparisons with different backbone models.

We thank the reviewer for suggesting on using more LM baselines. We provide this comparison in our response to reviewer BaWk's Q1. Results show that w/o fine-tuning on WOMD-Reasoning, all baselines can hardly answer driving-related questions, supporting our dataset's motivation. We then fine-tune LLaVA and LLaMA-Adapter on WOMD-Reasoning, which both significantly benefit from the fine-tuning. Finally, we find that fine-tuned Motion-LLaVA works better than other fine-tuned models, supporting its superiority.

Q7. Processing of Agent IDs when fed into the network.

We appreciate the question on agent IDs. These IDs are neither categorical inputs nor embedded. Instead, they are used as textual identifiers within the prompt, enabling the LLM to distinguish agents through natural language understanding. The prompt format is: Ego agent: <motion>\nAgent #0: <motion>\n...\nNow, please answer: {Question}, where <motion> represents the encoded motion data from each agent's own viewpoint.

These IDs only serve as range allocations, i.e. #0-#99 indicate vehicles, #100-#199 indicates bicycles and #200-#299 indicates pedestrians. We transform WOMD agent IDs to these local IDs to avoid downstream models overfitting to specific agent's behavior. Besides this, specific agent IDs do not carry any additional information.

We thank the reviewer for the positive comments and insightful suggestions. Please see our responses below:

Q1. Dataset accuracy: Eval criteria, diversity and annotation reliability.

We thank the reviewer for questions on validations, please see our responses below:

1. Evaluation Criteria: We perform 3 evals on WOMD-Reasoning and Motion-LLaVA fine-tuned on it:
   • For human assessment of WOMD-Reasoning, the criteria is detailed in Q7.
   • For evaluating the outputs of Motion-LLaVA, we use language metrics (ROUGE, BLEU, METEOR, CIDEr, and SPICE) along with a GPT Score which assesses semantic understanding beyond mere text similarity shown in Table 11.
   • For evaluating WOMD-Reasoning in real-world applications, we add a trajectory prediction task, whose prediction metrics are minFDE_6 and MR_6.
2. Distributional Diversity WOMD-Reasoning's scenario diversity is backed by that of WOMD, a large-scale highly-interactive dataset. Also our paper's Figure 2 shows that among 63k scenarios covered in our dataset, there are ~120k yields, ~74k lights, ~74k stops, showing our dataset’s extensive coverage of traffic rule-induced interactions beyond just near-end events.
3. Human Annotation Reliability To justify the size of our human eval, we scale it up from ~1.6k to ~2.2k Q&As, and the correct rate remains stable (91.99% -> 91.80%), justifying the current eval scale.
4. More Validations We are tuning our data generation pipeline and incorporating WOMD's recent upgrades to offer new versions of the dataset with even better accuracy soon. We have also completed several human verification demos. We are proceeding to the human labeling.

Q2. Quantitative tests in real-world driving.

We appreciate the questions on testing on the real world. First we address that testing Motion-LLaVA results on WOMD's val set can reflect real-world interactions, as WOMD is a real-world dataset. To provide more quantitative evals, we introduce language outputs of Motion-LLaVA into a trajectory prediction model Multipath++, seeing ~7% improvements on minFDE_6 and MR_6 in real-world WOMD cases. Details are included in our answer to reviewer BaWk's Q2.

Q3. Dataset's reliability and biases.

We appreciate the question on reliability. We analyze the eval criteria, distribution diversity and human eval in Q1. Besides, in our response to reviewer BaWk's Q2, we show that WOMD-Reasoning helps a trajectory prediction model to enjoy ~7% improvements, which supports its overall reliability.

Q4. Error analysis.

We appreciate the suggestions on error analysis. In our evaluations, we find that 1) Some scenarios are too complicated to describe in text. 2) Some errors still come from automatic translation program despite efforts. 3) We also see occasional error by the LM due to imprecise attention (e.g., confusing “following” when a car is behind but not in the same lane). Therefore, we are continuously updating and human-verifying the dataset to further improve its quality.

Q5. Evidence on traffic rule-compliance planning.

We appreciate the suggestions. As the first step, we add a trajectory prediction task in our answer to reviewer BaWk's Q2, which proves overall effectiveness of our dataset on real-world tasks. We will follow the suggestions to quantify traffic rule-compliance metrics to give more specific evidence.

Q6. Literatures Comparison.

We thank the reviewers for these important references. We discuss them below and will cite them in our next version.

• DriveLM(ECCV 2024) We have cited and compared to it in Table 1 and in the introduction (line 28, right column). We will update its version in our next version.
• WiseAD(ArXiv 2024) We see that WiseAD seems to be a driving model work rather than a new dataset. Their Tab. 1 lists the datasets they use for training. We hope our dataset will help encode traffic rule knowledge in models like WiseAD.

Q7. Criteria for evaluating Q&A pairs, and for reasoning.

We appreciate questions on the eval criteria of Q&A pairs. Generally, we ask evaluators to judge whether the correct answer is included in the dataset's answers. Practically: For facts, the answer must be totally correct. (e.g. if number of lanes is asked, that number must be correct.) For interaction reasoning, the correct interaction or intention keyword must be included. (e.g., if human sees Agent #0 is required by rules to yield to Agent #1, then keywords Agent #0 yields to Agent #1 must exist) These standards help us to maximize the eval accuracy while minimizing the human labor. We are performing human verifications to further boost data quality.

Q8. Motion-LLaVA v.s. non-language-based models.

We thank the reviewer for the comparison request. However, instead of comparing, the outputs of Motion-LLaVA can be used as an addition to help real-world trajectory prediction models to perform better, like the ~7% improvement we observe (see our response to reviewer BaWk's Q2).

We thank the reviewer for the positive comments and insightful suggestions, and we address your comments and concerns below.

Q1. More LM baselines.

We are grateful for the suggestion. While our main goal for fine-tuning Motion-LLaVA is to show the effectiveness of WOMD-Reasoning, we agree that testing and fine-tuning more baselines would make our claims more reliable. Therefore, we test the quality of answers provided by a few vanilla and WOMD-Reasoning fine-tuned baselines, including LLaMA-Adapter, VITA and Qwen, all using 7B version. Results are shown below:

We observe that w/o fine-tuning, all models can hardly answer driving-related questions, supporting the motivation of building WOMD-Reasoning. We then fine-tune LLaMA-Adapter on WOMD-Reasoning, which also significantly benefit from the fine-tuning. Due to limited resources, fine-tuning other baselines are still ongoing. Besides, we find that fine-tuned Motion-LLaVA works better than fine-tuned LLaVA or LLaMA-adapter, proving its well-designed structure in utilizing WOMD-Reasoning info.

Q2. Downstream Applications

We thank the reviewer for this suggestion. To justify WOMD-Reasoning's ability in downstream tasks, we perform a trajectory prediction experiment to see the influence of using language outputs of Motion-LLaVA fine-tuned on WOMD-Reasoning. Multipath++ is used as the trajectory prediction baseline. As the interaction part is the most significant part in our dataset, we introduce the interaction part of language outputs into Multipath++, by cross-attending the T5 embedding of them with corresponding agents' features. Each experiment is run with 3 seeds for reliability. Averaged results are shown below,

We observe a significant boost from language outputs, which strongly supports WOMD-Reasoning's ability to help downstream tasks like predictions. It can also help understanding the explainability of driving behaviors.

Q3. More comparisons.

We thank the reviewers for these important papers to compare to! We will cite and compare to them, and we list a brief comparison here in our Table 1's form:

The stats further support that WOMD-Reasoning is the largest language Q&A dataset for driving by its release, with uniqueness in analyzing traffic rule-induced interactions.

Q4. The method part applies a motion encoder to replace the image encoder in LLaVA. Is there any adaptation?

We appreciate the question on the design of Motion-LLaVA. Yes, we thoroughly adapt LLaVA to better encode motion info for answering driving-related questions, which are discussed in Appendix A.7. Besides introducing a motion encoder from Multipath++ to replace the vision encoder in LLaVA, we have also made following adaptations:

1. We use the encoder of a pre-trained Multipath++ as motion encoder. Motion prediction pre-training ensures the quality of the feature extracted from motion data, which is ablated in our Table 13.
2. We remove the projector alignment stage in LLaVA, and keep the motion encoder unfrozen in fine-tuning, to help the model better encode info for answering driving-related questions. The benefit of this is ablated in Table 14.
3. In Motion-LLaVA, we design a prompt for feeding motion embeddings from the Multipath++ encoder into LLM. The format is: Ego agent: <motion>\nAgent #0: <motion>\n...\nNow, please answer: {Question}. Here, <motion> represents the encoded motion data from each agent's ego-centric viewpoint. The Agent IDs are randomly assigned and serve only to differentiate agents in downstream Q&As.