iMotion-LLM: Motion Prediction Instruction Tuning

1. Lack of concrete task definition. As this paper proposes and studies about a novel task Text-Guided Intention Trajectory Prediction, it is very important to give a mathematical definition of this task with clear explanations. For example, the format of the instruction is not clearly stated for a prediction. The lack of task definition makes the paper difficult to follow.
2. The application of this task in real world is not clear. It is confusing how the input instructions can be acquired in the test phase in real-world applications such as autonomous driving. Further, the necessity of such input instructions for trajectory prediction in autonomous driving is also not clear, although I suggest that the instructions can be useful for planning and simulation.
3. Evaluation metrics. For IFR, it is diffucult to follow since the format of the input instruction is not clear. For DVS, it is not the case that greater diversity means better prediction results. Although DVS can evaluate the diversity of predictions, it does not reveal the accuracy of each prediction.
4. Some statements are not supported by evidence or references. For example, "Although previous models can predict multi-modality trajectories, the predicted paths are not diverse enough and mainly focus on one driving behavior" in L48.

1. The most significant concern of me is: the authors claim this is a "text-guided trajectory prediction" work, but how can guidance and prediction exist at the same time? In trajectory predictions, we aim to analyze the history trajectory and propose all possible future trajectories, to help a safe planning. When a specific text prompt is involved, a future modal is specified so the prediction is no longer covering more modals. I would suggest the authors considering topics like "simulator" or "scene generator", which I believe would be more accurate and more significant.
2. If prediction is considered the topic for this work, results in Tab. 3 and Tab. 4 seem not supporting a positive effect of the proposed method. Without instructions, there seems not to be any metric improvement. While metrics do improve with instruction, it is noticeable that the instructions contain future trajectory modal information, which involves severe information leak. So these improvements seem ambiguous to me. I would suggest that if the authors continue to claim their method as a prediction model, metric improvements might have to be achieved with careful model designs, to prove the benefit of the proposed language model.
3. Some examples provided in Fig. 1 and Fig. 5 seem not to be correct to my common knowledge: i) Fig. 1, U-turn, it should be rejected because right U-turn is certainly not permitted in traffic scenes going on the right side. It has nothing to do with whether there is a lane on the right. ii) Fig. 1, AS and Other feasible direction, their narratives do not correspond, especially on Agent-2 and the Stop Sign's positions. iii) Fig. 5, Stationary, I don't see strong reasons to reject it, nor do I feel that the reasons provided ban the car from stopping. These make the figures confusing. My advice is to pick some clearer cases to avoid misunderstandings.
4. A full example of InstructWaymo can be provided as a figure / table to show its contents more clearly and to avoid bulk texts.
5. I hope the authors can clarify on the reason for measuring IFR for INF, and why IFR for INF is the lower the better. Why to test the model's prediction ability with INF instruction during inference, which has no ground-truth in the training stage?
6. On presentations: generally the paper might need to be thoroughly revised to provide a clearer logic on what it is focusing on and what significant improvement it achieves. For example, Fig. 3 should be improved for the streamline and the look. Currently, it looks a little bit messy at the first glance. Various colors mix together, and those purple dotted lines look overwhelming. I would suggest to re-organize this figure to better present the pipeline.
7. Typo: Line 460, it should be "baseline", not "basline".

1. Poor Writing and Structure: The paper suffers from poor organization and writing quality, which significantly impacts clarity. The overall flow and logic are difficult to follow. A structured rewrite, with a stronger focus on motivation and high-level contributions, would improve readability.

   • The abstraction and introduction lack structure and logic. Among all examples, the most obvious one is that, evaluation metrics are introduced even before the proposed model itself, leading to a confusing narrative.
   • The methodology sections (Sections 3, 4, and 5) are even harder to follow. For example, the beginning of Section 3 contains too many details, making the paper more like a technical report rather than a scientific paper. After reading, it still remains unclear what constitutes an “instruction” in the context of this paper. For example, is it limited to one word as in the categories listed in Table 1? A full list or a representative list of actual used instructions would be helpful. The authors should describe the design principles and structures, and leave details to the appendix.
   • Besides, numerous unexplained terms appear abruptly. For example, the definition of “caption” on line 212 causes a lot of confusion, and the reviewer only finds an implicit definition of it later in Line 306.
   • In Section 4, a large portion of the text is describing another paper, namely GameFormer, which shows excessive detail and unclear focus. I know the proposed method is based on the structure of GameFormer, but this description should be condensed in one paragraph, instead of half a page.

2. Unclear Motivation for Instruction-Conditioned Prediction: The paper appears to aim at addressing instruction-conditioned prediction, yet the introduction focuses instead on prediction diversity. The authors do not clearly present the motivation, insights, or advantages of condition-based prediction. Later, the authors suggest that conditioning motion prediction offers benefits such as (1) enhancing prediction diversity and (2) generating challenging, novel traffic scenarios. However, these points are not convincingly substantiated:

   • How the proposed method achieves diverse predictions is unclear. Simply enumerating possible language instructions to prompt the model seems neither meaningful nor efficient.
   • Generating realistic traffic scenes requires multi-agent modeling and interaction, whereas the proposed method generates predictions for only a single agent at a time, which limits the model's ability to simulate complex multi-agent scenarios realistically.

Additionally, the motivation and contributions of this work are introduced late and somewhat buried (line 312). A revision to clarify and emphasize true contributions would greatly improve the paper’s coherence and impact. The current writing and presentation make it challenging for reviewers to fully appreciate the technical contributions of this work.

3. Experiment Design Concerns: Several issues in the experimental setup raise questions about the validity of the findings:
   • Customized Data Split: The use of a custom data split with only 2,311 samples from Waymo raises concerns about statistical confidence. Justification for not using the full dataset, possibly due to computational constraints, and how these 2,311 samples are selected, would be helpful.
   • Baseline Evaluation: In Table 2(a), the evaluation for GameFormer on instruction-following and diversity metrics is unclear. Since GameFormer does not take instructions as input, how is its instruction-following performance assessed? Are predictions generated without instructions and then compared against the instructions? If so, further explanation on the significance of this baseline would be helpful.
   • Unclear Baseline Descriptions: Again in Table 2(a), the baseline of conditional GameFormer lacks sufficient descriptions. How GameFormer is conditioned on discrete directions, by using ground truth directions as additional inputs? More transparency is needed to understand how these comparisons are made.
   • Incorrect Naming in Tables: In Table 2(a), the last two rows have identical method names. Besides, in one table, we observe multiple methods are labelled as 'ours', which could be confusing.
   • Incomparable evaluation: Tables 3 and 4 use different data for evaluation (test set VS. customized val subset), which undermines comparability. I understand this metrics are used to get a sense of SOTA methods's performance, but using different data split could hinder that purpose. A better solution could be use SOTA methods' checkpoints and evaluate them on your customized data split

1. Language-conditioned trajectory prediction and generation has evolved in years and there are many previous works that the authors to compete in terms of the performance, such as Trafficgen: Learning to generate diverse and realistic traffic scenarios [ICRA 2023], Language Conditioned Traffic Generation [CoRL 2023], etc. The authors should have discussed about their model design choices compared to the previous literature.
2. Although mentioned in their introduction, the discussion of safety-critical scenarios is limited in the main paper. I feel like it is a very important topic since the incorporation of human intention should be measured or evaluated with the compatibility to prevent danger in real-world cases. I suggest the authors to follow RiskBench: A Scenario-based Risk Identification Benchmark [ICRA 2024] paper to at least have some discussions on how the current model could benefit risk localization and anticipation in traffic scenario.

Lack of Clear Motivation: The paper claims advantages such as enhancing safety and the ability to anticipate and adapt to potential hazards. However, it does not demonstrate how the proposed method contributes to these areas. The motivations provided could be achieved with existing multi-modal trajectory forecasting models. While the paper propose a novel approach, this novelty does not seem to contribute to its motivations.

Overly Restrictive Data Augmentation: The rule-based approach relies on absolute directions and speed changes, which may not generalize well to complex or curvy roads where "going straight" involves turning. The categories used do not align with common driver decisions, such as lane changes, limiting the applicability of the method for general scenes and interpretability.

Questionable Feasibility Parameters: The feasibility conditions (e.g., minimum and maximum speed changes, maximum range of 60 meters) are hard to interpret and seem arbitrary.The method for lane association to assess feasible directions is not clearly explained.

Lack of Comparison with Baselines and Ablations: The proposed architecture is complex, involving multiple components and customized inputs/outputs, but lacks clear justification for its design. However, it shows weak ablation studies to justify the numerous design choices. There is only one comparison with GameFormer on the core application and two ablations in table 2.(a). One ablation is keeping the GameFormer architecture but adds a conditioning on instructions, the other is the proposed approach without the feasibility detection. The results are mixed without a clear best choice between these three ablations. The proposed approach shows a significant degradation of instruction following recall on actual scenario and other feasible scenario (bad) while showing a lower compliance to infeasible instructions (good). It is unclear if the actual scenario instruction following to infeasible instruction following ratio shows a significant gain because it is 4x smaller than the unconditionned GameFormer baseline. Two other baseline models are used (MTR and MotionLM) but using different metrics (minADE and minFDE) that are not core to the paper (all three baseline outperform the proposed approach on these metrics).

Rigid and Arbitrary Metrics: The diversity and instruction-following metrics are based on rigid categorizations, which may not generalize across diverse road scenarios. Related works on measuring multi-modality and diversity in trajectory predictions are not adequately discussed to assess the novelty of the proposed metrics. The metric definitions uses thresholds that match exactly the instruction generation and might not be useful as a novel metric proposition for other methods that do not rely on these instructions.

Misleading or Inaccurate Claims: The statement that previous prediction models lack comprehension of different driving behaviors is inaccurate, as many models are designed to capture diversity and consider various driving scenarios. Some design choices and their justifications are unclear or unjustified within the context of the paper.

• More experiments are needed. The paper only considered the GameFormer model as the motion prediction model. It will be interesting to add more models as baseline motion predictors. E.g., Add instruction guidance to the MTR [1] that has strong performance in motion prediction task.
• In table 4, the baseline GameFormer without guided instruction is always better than iMotion-LLM (even considering actual scenario instruction). It seems the generalizability of the instruction-based model becomes worse.
• In the main results, the OF/INF drops a lot after adding feasibility detection, which means the feasibility detector could reject more feasible instructions.

[1] Shi, Shaoshuai, et al. "Motion transformer with global intention localization and local movement refinement." Advances in Neural Information Processing Systems 35 (2022): 6531-6543.