DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

Q1: Claim for SOTA performance.

Thank you for your attention. Regarding the Contemporaneous Work LOF [1], it was released on arXiv on June 20, 2024, after we submitted our paper to NeurIPS in May 2024. We will add a discussion on LOF to provide a thorough comparison in the revised paper.

Q2: Heavy computational load for long-horizon state queries.

To address this issue, we propose balancing efficiency and performance by utilizing one state query to represent several time steps, as shown in the left part of Table 6. This can effectively reduce the computational load.

Q3: Computational cost compared to other methods.

Thank you for this valuable suggestion. We have added a table below to further compare computational cost. The experiments are conducted on Argoverse 2 using 8 NVIDIA GeForce RTX 3090 GPUs.

[1] FutureNet-LOF: Joint Trajectory Prediction and Lane Occupancy Field Prediction with Future Context Encoding. arXiv preprint: 2406.14422, 2024.

[2] SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Driving. IEEE Robotics and Automation Letters, 2024.

[3] Query-centric trajectory prediction. CVPR, 2023.

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

Q1: About loss function.

Thank you for your careful consideration of our work. As indicated in the right part of Table 5 in our paper, each loss component in our model contributes effectively to the training process, ensuring that no single loss term dominates the optimization. This approach aligns with recent works, such as QCNet [1] and MTR [2], which also employ a simple sum of all loss terms.

Q2: A Typo at Line 154.

Yes, we apologize for the typo. It should be $Q_s$. We have revised it in the manuscript.

Q3: About $T_s$ and $T_f$ between Lines 135-136.

If $T_s$ and $T_f$ differ, it results in the cases mentioned in the left part of Table 6, where we reduce the number of state queries to make the model more efficient. For example, in Argoverse 2, with $T_f=60$ and $T_s=30$, this corresponds to the case in the third row of the left part of Table 6, where one state query represents two contiguous time steps. There is no need for multiple predictions, and the directional intentions remain unchanged.

[1] Query-centric trajectory prediction. CVPR, 2023.

[2] Motion transformer with global intention localization and local movement refinement. NeurIPS, 2022.

Dear Reviewer V1LD

We appreciate the reviewer's time for reviewing and thanks again for the valuable comments.

Best wishes

Authors

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

Response to Weaknesses.

1. Results: Results on WOMD (Waymo).

We provide results on WOMD using the settings in UniTraj [1], as shown below. The results of other methods are also from UniTraj.

2. Analysis: Better measure query decomposition.

Thank you for this valuable suggestion. We measure the outputs of state queries and mode queries with $minADE$ and $minFDE$, as shown below. We can see that the $minADE_1$ and $minFDE_1$ of the trajectories from state query outputs are better than those from mode query outputs. This means state dynamics are encoded in state queries. Additionally, there are six output trajectories from mode queries, indicating that directional information is predominantly stored in mode queries. The final outputs take advantage of the strengths of both. We will add this analysis to the revised paper.

3. Analysis: About RNN-based model and the GRU result.

As shown in the left part of Table 5, we conducted an ablation study on an RNN-based model to compare GRU with Mamba. If we directly replace Mamba with GRU to process state queries, it is difficult to achieve convergence, leading to rather poor results, as shown below. Perhaps RNN-based models can process state queries in some other way; this could be a research problem on its own.

4. Reproducibility: Architectural description.

Thank you for your question. The layer sizes are provided in Table 7 in the Appendix. For normalization, we use nn.LayerNorm, and for activation, we use nn.GELU. Additional details can be found in the implementation section at Lines 184 and 459.

Response to Questions.

1. A precise formulation of the auxiliary losses.

As in Line 143, we use an MLP to decode state queries into a single future trajectory $Y_f$ and calculate the loss with ground truth $Y_{gt}$ to obtain $L_{ts}$.

$L_{ts} = {\rm SmoothL1}(Y_f, Y_{gt})$

As in Line 150, we use MLPs to decode the future trajectories $Y_f$ and probabilities $P_f$. So $L_m$ is shown below:

$Y_{best}, P_{best} = {\rm SelectBest}(Y_f, Y_{gt})$

$L_{m} = {\rm SmoothL1}(Y_{best}, Y_{gt}) + {\rm CrossEntropy}(P_f, P_{best})$

2. Training time.

About 9 hours in total. Our settings are as mentioned in Line 184 and Line 463.

3. The model in ID3 (Table 4).

Yes, you are right. Decoupled queries are just concatenated and fed directly to the MLP decoder, and we use two auxiliary losses to optimize the two types of queries.

4. Model size vary across your ablations (Table 4).

The table below shows the model size variations across our ablations in Table 4. Additionally, we perform an ablation on the depth of Attention and Mamba layers, as shown in the right part of Table 6. We can see that even a single layer can achieve decent performance. This indicates that the performance improvements are due to architectural choices rather than merely increasing model capacity.

5. For inference speed.

Thank you for this valuable suggestion. In most scenarios ($>$ 90%), the inference speed is between 25 to 45 ms. In some complex scenarios, it can go up to 70 ms, while in some easy scenarios, it is only 20 ms.

6. Multi-agent setting.

We use queries for each agent to predict their trajectories in the ego agent's coordinate system, so we do not re-encode the scene in each agent-centric frame. This approach avoids making the model costly.

7. For Line 488 about ZOH in Mamba.

ZOH can discretize continuous signals; the specific formula can be found in the Mamba [2] paper.

[1] UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction. arXiv preprint:2403.15098, 2024.

[2] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint:2312.00752, 2023.

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

Response to technical descriptions.

Q1: About Eq. (2) and related symbols meaning.

For the meaning of {$t_1,…,t_{T_s}$}: as in Line 135, these represent real-time differences of temporal states. For example, in Argoverse 2, we aim to predict future trajectories over 60 time steps. Therefore, if we use 60 state queries, $T_s=60$, and {$t_1,…,t_{T_s}$} corresponds to {$1,2,…,60$}.

For $T_f$ and $T_s$: As stated in Line 114, $T_f$ refers to the future time steps. As stated in Line 135, $T_s$ represents the state steps used for initializing state queries. As stated in Line 233, in our default setting, $T_s=T_f$. We utilize one state query to represent several time steps in the left part of Table 6. In these cases, $T_s$ differs from $T_f$.

Q2: For the motion mode $Q_m$.

As stated in Line 146 and Line 151, we have $K$ mode queries to represent $K$ trajectories. We decode the $K$ future trajectories and their corresponding probabilities, using losses to optimize specific features for different motion modes, as done in QCNet [1], MTR [2], and other works.

Q3: About loss function $L_{ts}$ and intermediate features.

As stated in Line 143 and shown in Figure 2, we use an MLP to decode state queries into a single future trajectory $Y_f$ and calculate the loss with ground truth $Y_{gt}$ to obtain $L_{ts}$.

Thus, the state queries $Q_s$ serve as intermediate features, similar to the mode queries $Q_m$. The role of these two types of intermediate features is to form the hybrid queries $Q_h$ to decode the final outputs.

Q4: About detailed information about the model’s parameters and settings.

• Training settings: Line 184, Line 459 (Appendix).
• Dataset and metric settings: Line 175, Line 180.
• Model size (5.9M) and inference speed (38ms): Line 242.
• The number of layers in each component: Table 7 (Appendix).

Response to contributions.

Q5: Table 1 and Table 3 do not include baseline methods such as SEPT, SEPT++, QCNeXt.

For Table 1, our method (ranked 3rd) is better than SEPT [3] (ranked 4th) on the official leaderboard of Argoverse 2. SEPT is a self-supervised method utilizing all sets (including the test set) for pretraining. It is orthogonal to ours, and we believe that SEPT can also be integrated into our method for further improvements. Additionally, SEPT++ has not been released yet.

For Table 3, our model is not specifically designed for the multi-agent setting, unlike QCNeXt [4], which is a method specifically designed for competition. We apologize for the insufficient comparison, which we have addressed in the revised paper.

Q6: Compare with previous works (e.g. Motion Mamba) for time series data.

Although both Motion Mamba [5] and our method utilize Mamba modules, the motivation and tasks are completely different. Our critical idea is decoupling motion forecasting, and Mamba is an effective tool to implement this aim. Additionally, the structures are also completely different. We will add a discussion to highlight the distinctions and contributions of our work compared to Motion Mamba in the revised manuscript.

[1] Query-centric trajectory prediction. CVPR, 2023.

[2] Motion transformer with global intention localization and local movement refinement. NeurIPS, 2022.

[3] Sept: Towards efficient scene representation learning for motion prediction. ICLR, 2024.

[4] Qcnext: A next-generation framework for joint multi-agent trajectory prediction. arXiv preprint:2306.10508, 2023.

[5] Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective ssm. arXiv preprint:2403.07487, 2024.

Dear Reviewer pCB4,

We appreciate your time for reviewing, and we really want to have a further discussion with you to see if our response solves the concerns. We have addressed all the thoughtful questions raised by the reviewer (eg, technical descriptions and contributions), and we hope that our work’s impact and results are better highlighted with our responses. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors