SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction

Q1: ”The paper explores motion prediction in GPT-style network, which has been done in many works such as StateTransformer.” + “The scale law ability of models based on transformer structure has been proven in many papers, resulting in limited novelty of the method proposed in this paper”

A1: Thank you for your valuable feedback. We will reference related work in the introduction of the revised manuscript and provide additional clarification.

1. While StateTransformer is indeed an autoregressive model, similar models like MVTE also exist. These autoregressive models utilize diffusion or distribution regression frameworks but do not implement discrete tokenization for map and agent input features or employ a cross-entropy based next token prediction paradigm.

2. Furthermore, prior methods like StateTransformer have limitations in validating scaling laws because both training and testing datasets are based on the same dataset. This increases the likelihood of overfitting to specific datasets, resulting in better performance. In contrast, validation of scaling laws and generalization abilities in LLM[1] and vision domain[2] often requires completely independent testing datasets. Our paper states in the introduction: "Generalizability means achieving satisfactory results across diverse datasets through zero-shot and few-shot learning, while scalability involves improving model performance as dataset size or model parameters increase." Therefore, our experiments on scalability and generalization are the first to be validated across multiple datasets. Additionally, as shown in the results in our note to all reviewers Q2, we believe that the combination of cross-entropy autoregressive prediction and discrete tokenization for both map and agent features is crucial for achieving scalability and generalization across datasets, which represents our key contribution.

Q2: "The result in "Table 4: Absorption study on each component of SMART" indicates that "RVT", "NAT", and "NRVT" are harmful for models trained on WOMD."

A2: We would like to clarify that the ablation study in Table 4 shows that while discrete tokenization may lead to some information loss, this loss can negatively impact performance on a single dataset. However, it significantly enhances generalization across different datasets, which aligns with our expectations. Moreover, techniques like "NAT" and "NRVT" can effectively improve model performance after discretization of inputs. We also emphasize in the conclusion the importance of lossless discretization for map inputs in future research.

Q3: "can the author provide results on NuPlan and compare with the latest works?"

A3: Thank you for the valuable suggestion. Below are the results of our experiments, which are fully aligned with the Val14 benchmark in PLUTO [3]. Due to character count limitations, we regret that we cannot cite every baseline method. The results in the table demonstrate the performance of our multi-agent SMART model when directly transferred to the NuPlan challenge, with a particular focus on pure data-driven methods. Notably, SMART achieves the highest Planner Score (90.17), Progress (99.52), and Drivable (99.33) among the Pure Learning methods, outperforming other models in several metrics.

To ensure these results are reproducible, we will update the code for Nuplan challenge accordingly. However, as noted in the conclusion of the original paper, "As a motion generation model, the ability of SMART to migrate to planning and prediction tasks still needs to be verified, and this is our top priority for future work." The primary reason for not including this experiment in the paper is that, unlike sim-agent tasks, the characteristics of planning tasks have not been fully considered, such as the need for more ego vehicle information input and a greater emphasis on driving safety.

[1] Achiam, Josh, et al. "Gpt-4 technical report." arXiv preprint arXiv:2303.08774 (2023).

[2] Bai, Yutong, et al. "Sequential modeling enables scalable learning for large vision models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Cheng, Jie, Yingbing Chen, and Qifeng Chen. "PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Driving." arXiv preprint arXiv:2404.14327 (2024).

Thanks for clearly addressing my concerns, I would like to maintain my current Rating.(6) and raise the "contribution" score.

We sincerely thank the reviewer for providing thoughtful review and constructive suggestions. We answer the questions as follows.

Q1 : What's the difference in decoder design compared with Trajeglish? Please clarify.” + “Minor in methodology differentiating with Trajeglish[1] and MotionLM[2], other auto-regressive decoder for motion prediction or sim agent.”

A1： We appreciate your feedback, and we will include a discussion of these differences in the revised manuscript.

• In addition to the discrete tokenization of road and the decoder-only transformer structure emphasized in the paper, there are also structural differences in the decoder. As described in Section 3.2 of our paper: “In our work, we leverage a factorized Transformer architecture with multi-head cross-attention (MHCA) to decode complex road-agent and agent-agent relationships along the time series.”. This approach separates spatiotemporal attention into two dimensions, unlike Trajeglish and MotionLM, which compresses the spatiotemporal dimensions into a single sequence. We will add the differences to the Related Work in the revised version.
• This design difference significantly affects inference efficiency. In our model, all vehicles make simultaneous decisions for the next time step, while MotionLM and Trajeglish sequentially output each agent's predictions at the same time. For example, generating the future scenes of 32 agents over 8 seconds at a 2 Hz frequency requires only 16 steps of autoregressive inference with our model. In contrast, the latter requires 32 * 16 = 512 autoregressive inference steps. This efficiency is a key reason why our model can complete multi-agent interactive simulations within 20 ms per step. We will include similar explanations in the discussion of the inference section in the revised version.

Q2: ” Minor in the experimental details, such as clearer auto-regressive sampling process, sim performance for different scale, etc.”

A2: Thank you for your feedback. In the initial version of the manuscript, a description of the inference process is provided in Appendix A.1, line 495, and we conducted performance evaluations of different scale models in Table 9, line 545. We acknowledge that the previous version's brief description of the inference process may have led to your confusion.

1. We will add the following clarification in the revised version of Appendix A.1: Specifically, our model operates in 2Hz, and we interpolate the results to 10Hz for evaluation. We use 1 second of history along with the current frame as conditions and predict the information for the next 8 seconds. We can further factorize the traffic simulation task as:

​where $S_t \equiv s_{t}^{1},\ldots,s_{t}^{N_{A}}$ is the set of all states for all agents at timestep $t$. As indicated by the formulas, our approach samples the future motion $S_t$ for all vehicles in the current scenario simultaneously for the next time step. In contrast, previous methods like Trajeglish and MotionLM sequentially sample each vehicle's future motion for the next time step. The main reasons for our approach are twofold: first, it significantly increases the model's inference efficiency by a factor of $N_{A}$, where $N_{A}$ is the number of agents in the scene. Second, in real traffic interactions, it is challenging to define a reasonable sequence of vehicle interaction at the same time, as vehicles typically plan their intentions concurrently.

2. Regarding the sampling process, we do not employ complex strategies. As stated in line 502, "To balance realism and diversity, we use top-5 sampling at every step during the simulation. Given the focus of this article on the generalization and scalability of the model, we have achieved strong results in specific scene generation without extensively exploring detailed sampling tricks.”

Q3: "Lack of RoadNet evaluation on the road-vector NTP task” +“Whats the performance of SMART without this augmentation? "

A3: We have included relevant explanations in our note to all reviewers Q1 and will make appropriate modifications in the revised manuscript.

Q4: “What is the defination and learning process for road-vector NTP?“

A4: Unlike sequential agent motions, road vectors form a graph. To address this issue, we extract the original topological information of roads and model the road vector tokens with sequential relationships based on their predecessor-successor connections. This approach requires RoadNet to understand the connectivity and continuity among unordered road vectors. The loss function for a single tokenized road polyline is defined as:

where $p_{\gamma}(r_{i}^{j+1}|r_{i}^{1:j})$ denotes the categorical distribution predicted by the RoadNet parameterized by $\gamma$, $J$ represents a complete polyline that has not yet been split into road vector tokens, $\text{r}^{1:j}$ Representing the road token embedding of the predecessor, and $\text{r}_{i}^{j+1}$ is the next predicted road vector token. This loss function ensures that RoadNet learns to predict the correct next road vector token given the preceding tokens, thereby capturing the spatial continuity and connectivity within the road network. The loss function formula and explanations will be included in the revised version of the paper.

Q5: A thorough check for notations and writing"

A5: We apologize for the oversight. We will correct the noted issues in the revised version, including the suspected ChatGPT sentence in line 508 and the notation for modality to clearly differentiate it from "Value."

We sincerely thank the reviewer for providing thoughtful review and constructive suggestions. We answer the questions as follows.

Q1: How much does road vector next token prediction contribute to SMART's performance?

A1: We have included relevant explanations in our note to all reviewers and will make appropriate modifications in the revised manuscript.

Q2: In Table 1, why is SMART's minADE higher than that of Trajeglish? This doesn't seem to match the results in Table 6 of the appendix.

A2: We appreciate your attention to this detail. The minADE in Table 1 reflects the correct evaluation results from the WOSAC 2023 leaderboard. We mistakenly reported the minADE metric from WOSAC 2024 in Table 6. We appreciate your understanding and have corrected this in the revised manuscript.

Q3 :"Table 4 shows the efficacy of certain architecture choices on generalizing from one dataset to another. How do these architecture choices affect the model's ability to learn from multiple datasets? Do these choices matter in the setting where you have access to NuPlan, WOSAC, and the proprietary dataset” +“Without comparisons to other methods, it is difficult to interpret the significance of the scaling law and zero-shot generalization experiments. “

A3: Thank you very much for your feedback. We agree that including relevant results can enhance the reliability of our main contributions. We have added these results in our note to all reviewers Q2. We regret that due to the high GPU cost of training large-scale models across datasets, we were unable to conduct ablation comparisons for each sub-module during our validation of scaling laws. We only compared SMART W/O, SMART, and the earlier replicated MVTE method. Nevertheless, the existing results demonstrate that discrete tokenization is an effective method for bridging dataset gaps. Additionally, autoregressive models utilizing cross-entropy classification loss are crucial for scalability.

Q4: “Why does dataset size limit your architecture size to 100 million parameters? Is this a matter of overfitting? If so, I think that it would be interesting to make note of this in the appendix of your paper.”

A4: This conclusion is based on our observations during training. Specifically, we found that for models with 10 million parameters, performance improvements plateau when the total training token count reaches around 0.1 billion. For 30 million parameter models, this plateau occurs after training on approximately 0.7 billion tokens, while the 100 million parameter model continues to show improvements with full data. Due to constraints in training resources and dataset size, we did not pursue training larger models. Regarding the potential for overfitting with larger models, our experience suggests that when a model with a higher parameter count is trained on a single dataset for multiple epochs, its generalization ability tends to decline. We will include this discussion in the appendix of the revised manuscript.

Q5: “ There may be an unintended sentence "Here’s the revised version of your text with improved precision and grammar:".

A5: We apologize for the oversight. This sentence was unintentionally included and will be removed in the revised version of the paper.

Q6: “The paper's results can be made stronger with more statistically significant experiments. For example, in Table 9, it is unclear whether the differences between SMART 8M, 36M, 96M are statistically significant or within the bounds of noise, given how close the numbers are“

A6: We appreciate your suggestion regarding the need for more statistically significant experiments. Given the extensive test dataset of 44,920 samples in the actual WOSAC evaluations, statistical noise is minimal, with metric fluctuations typically around ±0.02. Furthermore, the models with different scales are already performing very close to the maximum possible score on the Waymo sim agent metrics (where the ground truth maximum score is 0.80), meaning the differences in performance metrics have become less pronounced. Consequently, the performance improvements between SMART 8M, 36M, and 96M models may not appear as significant due to these near-maximal scores.