UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

1. Authors could provide a better way to evaluate the model.

R: We appreciate the reviewer's understanding of this challenge. Jointly training a model across multiple datasets and tasks for motion prediction incurs substantial computational costs. For this foundational work, we focused on successfully addressing multiple tasks within a single dataset. Expanding our evaluation across diverse datasets is a valuable direction we've left for future work.

Further, we have now conducted experiments on transferability from Waymo to nuPlan (Val14), and the results are presented below. Our findings indicate two key points:

• As expected, zero-shot performance is less than satisfactory due to inherent data distribution discrepancies (especially for motion token vocabulary) between the two datasets.
• Crucially, our model still achieves decent performance after fine-tuning, even if it is slightly inferior to methods specifically tailored for the nuPlan dataset.

These results verify the generality and adaptability of our solution across different autonomous driving scenarios. We will include this transferability test in the revised paper.

2. What is the size of the model?

R: Thanks, we will clarify: We follow the lightweight Transformer design with factorized attentions and the model size is approximately 7M.

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

1. Weakness 1: This two-phase procedure significantly increases the model's complexity and computational cost for a specific motion task.

R: A great comment regarding our two-phase procedure! However, all our fine-tuned models originate from the same jointly trained model, and we introduce no additional parameters during the fine-tuning stage (except for the prediction task). This means it barely affects overall model complexity. For overall training cost, this approach can often be more efficient because the joint training, typically the most expensive part, is shared across all tasks, and subsequent fine-tuning is comparatively cheaper.

For full transparency, we've provided a statistics table detailing our model size and computational time below.

2. Weakness 2 and Q2: The models compared in this paper differ from the top models published on the 2025 Waymo Open Dataset Challenges website.

R: We regret that we couldn't participate in the award selection for the 2025 Waymo Challenges. However, for a direct comparison, the reviewer can find the relevant leaderboard by appending "2025/sim-agents" to the Waymo Challenges website (we're unable to provide external links directly in this response).

At the time of our NeurIPS submission, our method achieved an RMM of 0.7851, which is directly comparable to the first-place approach TrajTok (RMM 0.7852) in the 2025 Waymo Challenges.

Additionally, it's worth noting that there is no official motion prediction track in the 2025 challenges. Therefore, our motion prediction comparisons were conducted against the 2024 leaderboard to ensure a relevant benchmark.

3. Q1: What are the methodological differences between UniMotion and these existing works?

R: Thanks, we will clarify: The key methodological difference lies in our scope: the metioned works primarily concentrate on optimising the planning task. In contrast, UniMotion proposes a unified motion framework designed to handle all motion tasks (prediction, planning, and simulation) within a single architecture. Therefore, our overall methodologies are largely orthogonal.

Regarding the specific design for planning fine-tuning, our method adopts an agent trajectory prediction stage followed by ego planning generation, which can be regarded as an extension of existing approaches where the planner reacts to predictions.

4. Q3: More discussion is needed on how different motion tasks in the context of ADS, which have quite different targets and input/output requirements, share "knowledge".

R: Thanks, we will further elaborate as below.

In fact, these tasks share significant commonalities in their underlying input and output information. For instance, all these tasks typically process agent trajectories and high-definition (HD) maps as input, and ultimately aim to output future trajectories for scene agents. This shared informational foundation is precisely what enables our unified model. The main differences, as discussed in Section 3.1, lie in the varying demands placed on these future trajectories — whether it's for diverse motion generation or accurate long-horizon prediction.

Furthermore, while the specific objectives of tasks like LFR and NTP appear distinct, they actually share crucial underlying commonalities. LFR directly supervises the model with long-range logged trajectories, and critically, the initial motion direction embedded within these trajectories simultaneously serves as the maximization objective for NTP. This inherent consistency and synergistic relationship ensures stable joint training, allowing the model to effectively leverage shared knowledge across different motion challenges.

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

1. Weakness 1 and Q2: The experiments only involve a single dataset, which could raise questions on the generalization capability of this method.

R: Excellent suggestion! We have now conducted experiments on transferability from Waymo to nuPlan (Val14), and the results are presented below.

Our findings indicate two key points:

• As expected, zero-shot performance is less than satisfactory due to inherent data distribution discrepancies (especially for motion token vocabulary) between the two datasets.
• Crucially, our model still achieves decent performance after fine-tuning, even if it is slightly inferior to methods specifically tailored for the nuPlan dataset.

These results verify the generality and adaptability of our solution across different autonomous driving datasets. We will include this transferability test in the revised paper.

2. Weakness 2 and 3: The method delivers sub-optimal results in at least one of the metrics, and the paper could benefit from some quantitative or qualitative analysis on some of the failure cases.

R: Thanks. UniMotion consistently ranks among the top two across various metricc; For some results with relatively large differences, we provide more analysis below.

Tab. 1 minADE: Different from prediction task, simulation task focuses more on generative ability of motion models. Hence, we adopt the tokenization-based NTP designs, which select trajectory segments from clustered trajectory token vocabulary, hence showing stronger generative performance but inferior in accurate prediction over the regression-based methods.

Tab. 3 red light: DIPP adopts a tailored planning cost function that involves many rules, such as velocity, comfort and red light. To ensure generality and consistency, we just supervise training with logged trajectories, and this metric can be further improved by involving traffic rules.

Besides, we have already provided more analysis of specific failure cases in the supplementary material, which the reviewer could refer to.

3. Weakness 4: A qualitative comparison with other methods should be provided in order to demonstrate the strength of the proposed method.

R: Great suggestion. Our intention in Figure 4 was to showcase the ability of our method in handling various motion tasks, while overlooking the comparison with other methods. We will supplement this part accordingly.

4. Q1: Why use decoder-only Transformers for all tasks？

R: Great point, we will clarify: We opted for a decoder-only Transformer structure because it's better suited for multi-step generative tasks like simulation, where input and output lengths can vary. Conversely, the encoder-decoder structure, while excellent for sequence-to-sequence prediction tasks, doesn't generalize as well across the diverse demands of our multi-task framework. Adopting a decoder-only architecture during joint training ensures stronger generalization capabilities across all our tasks.

For the prediction fine-tuning stage, we do introduce an additional multi-modal decoder to handle richer multi-modal outputs, which can be seen as a encoder-decoder structure.

5. Q3: Could the authors provide some theoretical discussion on why joint training works well?

R: Thanks, we will elaborate on the theoretical underpinnings of our joint training approach: As detailed in Section 3.1, our primary objectives are NTP and LFR. Intuitively, this joint training paradigm fosters the learning of shared, rich intermediate representations across multiple tasks. This enables tokens within the model to attend simultaneously to both their immediate surroundings and more distant scene contexts, leading to a more holistic understanding of the environment.

Further, despite their apparent differences, the two task objectives possess crucial commonalities. Specifically, while LFR provides direct supervision on long-range logged trajectories, the initial motion direction embedded within these trajectories concurrently serves as the maximization objective for NTP. This underlying consistency and synergistic relationship between LFR and NTP are fundamental to ensuring stable joint training and preventing mode collapse, allowing the model to learn effectively from both detailed local cues and broad long-range dependencies.

I appreciate the authors' response. I think the authors have addressed all my concerns by providing new experiments and new analysis. Therefore, I've decided to maintain my positive score. P.S. I've written the above comment in the final justification part earlier, which I didn't realize was invisible to the authors.

We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:

1. Concerns about the fairness and reliability of performing ablation on 20% data.

R: Thanks for the reviewer's rigorous concern. This is mainly due to the constraint of resource available. Ablation on 20% data is acceptable and reasonable as exemplifed in previous perception models [1] and recent motion-related models [2, 3]. As the Waymo dataset offers a much larger volume of data than other datasets (such as Argoverse or nuScenes), even 20% of the data can reflect the overall distribution well. We thus followed this practice.

Besides, we have now conducted the ablation for Tab. 5 on the full data. The results below show consistent observation as that with 20% data.

2. Run-time performance during testing should be reported, particularly for the planning module.

R: Apoligies for lack of run-time analysis. We have now provided the inference time on RTX 3090 for all tasks as below. Please note, for the planning latency, we perform auto-regressive generation to yield complete trajectories for offline evaluation, while in real-world scenarios, per-step planning (online) happens.

We will add this to the revised version.

Simulation one-step inference latency

Prediction inference latency

Planning inference latency

3. The paper would benefit from a more detailed analysis of comparison with state-of-the-art methods to better support its claims.

R: Thanks, and we will further elaborate: For Tab. 1, different from prediction task, simulation task focuses more on generative ability of motion models, and is not particularly concerned with accurately predicting the logged trajectories (not involved in Realism Meta Metric). Hence, we adopt the tokenization-based NTP designs, which select trajectory segments from clustered trajectory token vocabulary, hence showing stronger generative performance but inferior in accurate prediction over the regression-based methods.

For Tab. 3, DIPP adopts a tailored planning cost function that involves many rules, such as velocity, comfort and red light. To ensure generality and consistency, we just supervise training with logged trajectories, and this metric can be further improved by involving traffic rules, like [4, 5].

Thanks. We have revised the formatting errors in Equation 1.

[1] Embracing Single Stride 3D Object Detector with Sparse Transformer. Lue Fan et al. CVPR 2022

[2] BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction. Zikang Zhou et al. NeurIPS 2024

[3] Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models. Zhejun Zhang et al. CVPR 2025

[4] Rethinking Imitation-based Planners for Autonomous Driving. Jie Cheng et al. ICRA 2024

[5] PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Driving. Jie Cheng et al. arXiv preprint