BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction

We sincerely appreciate your constructive suggestions, which greatly help improve the quality of our manuscript. In the following, we attempt to address some critical concerns you raised.

1. Scope down claims slightly

(1) Sample efficiency: We evaluate our models trained with different proportions of training data. Our model is able to achieve very decent performance when trained on merely 20% of data, which is attributed to the high sample efficiency of our approach.

(2) Decoder-only: We agree that the notion of "decoder-only" is somewhat semantic. We will revise the introduction of our paper to emphasize our approach's capability of parallel training instead of claiming "the first decoder-only agent simulator."

(3) Next-patch prediction: In our approach, each token in the decoder has fused the information of multi-step states and takes charge of generating multiple subsequent states; since we define multiple consecutive states as a patch, we think it appropriate to use the term "next-patch prediction."

---

2. Reproducibility

More details on implementation will be supplemented in the revised version. Below are the ones that you are concerned about:

(1) Map tokenization: We sample points along the polylines every 5 meters and tokenize the semantic category of each 5-meter segment via learnable embeddings. The shape of the embedding table is [17, 128], indicating 17 categories and a hidden size of 128.

(2) RNN head: A single RNN+MLP outputs all the mixture components.

(3) Hidden size: All hidden sizes are 128.

---

3. Manuscript quality

(1) Figure: Thank you for the suggestions. We will adjust the figures (e.g., removing the redundant cartoon in Figure 3 and deleting Figure 4) to leave some space for qualitative results. We will also expand the captions to describe more details.

(2) Reference: Thank you for spotting the issues. We will clean up the references carefully.

(3) Notation: Indeed, the notion of scene-level patches is not that important in our model. We will change Equation 2 into an inline equation.

---

4. Forecasting results

BehaviorGPT considers the joint distribution of ALL agents in the scene, which is misaligned with the objective of existing motion prediction benchmarks. For example, the Waymo Motion Prediction Benchmark and the Argo 2 Single-Agent Motion Forecasting Benchmark are for marginal prediction; the Waymo Interaction Prediction Benchmark and the Argo 2 Multi-Agent Motion Forecasting Benchmark concern the joint distribution of a subset of agents in the scene. Indeed, we can customize our model architecture and training objective to obtain high scores on motion forecasting benchmarks, but that is out of the scope of this work. Thus, it is more reasonable to compare our approach with joint multi-agent motion prediction models that consider the joint distribution of all agents. We also noted that QCNeXt, a joint multi-agent prediction model that has won 1st place in the CVPR 2023 Argo 2 multi-agent motion forecasting challenge, is also on the WOSAC leaderboard. We found that QCNeXt performs much better on minADE (1.08 vs our 1.54), but its closed-loop performance lags far behind most simulation-oriented models. It seems to be a trade-off between open-loop and closed-loop performance.

---

5. Model introspection

(1) A2A attention: Without A2A attention, the model cannot capture agent interactions.

(2) #Layers: We try to fix the hidden size as 128 and vary the number of attention layers. We found that increasing the depth of the models can benefit the performance.

(3) Patching: We try to increase the replan frequency by discarding a portion of the predicted states at each simulation step. The test set results produced by the model with a patch size of 10 are shown as follows. From the table, we can see that increasing the replan frequency from 1 Hz to 2 Hz can even improve the overall performance, which may benefit from the enhanced reactivity. This phenomenon demonstrates that the performance gain is not merely due to the lower replan frequency, as the model with a patch size of 10 beats that with a patch size of 5 even harder if using the same replan frequency (i.e., 2Hz). However, we found that an overly high replan frequency harms the performance, as indicated by the third row of the table. Overall, we conclude that using a larger patch indeed helps long-term reasoning, but a moderate replan frequency is important for temporal stability.

---

6. Output introspection

Please refer to the PDF file in the general response.

---

7. Alternative validation

Validating the usefulness of data-driven simulators for motion planning/prediction is definitely something worthy of doing. This will be our next step!

---

8. Inference

The average latency per simulation step is less than 10 ms on an NVIDIA L20 GPU (seq length: 9 secs).

---

9. Number of patches

9-sec sequences are used for training, while 1-sec trajectories are used as initial history during inference.

We sincerely thank your valuable feedback. To resolve your concerns, we try our best to conduct some scaling experiments under the constraints of limited time and computing budget.

Q1: Can BehaviorGPT scale with data and computation?

A1: We answer this question regarding the quantity of training data, the hidden size, and the number of decoder layers.

(1) Quantity of data: We evaluate our models (5M parameters, hidden size = 128, #Decoder layer = 4) trained with different proportions of training data provided by the Waymo Open Motion Dataset. As shown in the table below, our model is able to achieve very decent performance when trained on merely 20% of training data, which is attributed to the high data efficiency of our approach. Increasing the proportion of training data from 20% to 50% further improves the performance across various metrics. Moreover, training on 100% of the data continues to make the model more powerful. Judging from the trend indicated in the table, we believe that feeding more data for model training will continuously improve the overall performance.

(2) Hidden size: We vary the hidden size to obtain models with different parameters. The experiments use 20% of the training data and 2 layers of the Transformer decoder. As depicted by the table below, increasing the hidden size consistently improves the performance.

(3) Number of decoder layers: We also try to fix the hidden size as 128 and vary the number of decoder layers, obtaining models by training on 20% of the data. Based on the experimental results below, we can conclude that increasing the depth of the models can benefit the performance.

It is a pity that we do not have more computing resources to experiment with even larger models, and we welcome researchers with sufficient computing resources to examine the scalability of our architecture after we contribute our code to the open-source community.

---

Q2: What about the inference latency?

A2: The average latency per simulation step is less than 10 ms on an NVIDIA L20 GPU (seq length: 9 secs).

We sincerely appreciate your thoughtful comments. We will revise the statements, notations, and figures according to your suggestions. In the following, we attempt to address your critical concerns by giving more analyses and clarifying the implementation details.

Q1: Comparisons with timeseries/spatial-temporal Transformers.

A1: Thank you for pointing out these relevant works. We will discuss them in the revised version.

---

Q2: Scaling experiments.

A2: Experimenting with models as large as Trajeglish is beyond the reach of our computing resources, but we try our best to conduct some scaling experiments under the constraints of limited time and computing budget. We conduct two groups of experiments, including (1) varying the hidden size when fixing the number of decoder layers to be 2 and (2) varying the number of decoder layers when fixing the hidden size to be 128. Although the experimental results show that larger models can achieve better results, the gain in performance also seems to plateau at the scale of 5-7M parameters. we can expect from the trend that continuing to enlarge the model would not bring too significant improvement.

---

Q3: Tokenization of the map.

A3: Each map token spans 5 meters, and a semantic category is assigned to the 5-meter segment. We will clarify this in the revised version.

---

Q4: Significance of small models.

A4: We understand that today is the era of large models, but we must point out that many industrial applications, such as autonomous driving, are still using small models (some false advertising on large models might somewhat mislead consumers). As also indicated by the Reviewer gNhc, autonomous driving companies need to run simulations at massive scales before real-world deployment, so simulation efficiency determines how fast we can upgrade the autonomous driving system. In industry, pursuing 1% more realistic simulation with 10x larger models may not be a wise strategy. Thus, we believe small models can also make a great impact on real-world applications.

---

Q5: Does an extremely large model need the patching design?

A5: Since it is impossible for us to train a super-large model, we are unable to reach a conclusion. However, we have shown that the patching design helps small models beat up very large models, demonstrating the value of the patching mechanism--if autonomous driving companies can attain satisfying simulation quality with a small model, there is no reason for them to spend way more money on training and deploying an extremely large model.

---

Q6: Data efficiency.

A6: We evaluate our models (5M parameters, hidden size = 128, #Decoder layer = 4) trained with different proportions of training data. Our model is able to achieve very decent performance when trained on merely 20% of training data, which is attributed to the high data efficiency of our approach.

---

Q7: Why are mixing coefficients only predicted once per patch?

A7: Because we desire the mixing coefficients to represent the likelihood of multi-step behavior.

---

Q8: Why does training take so much time?

A8: Besides model size, the training time is also determined by the computational complexity. On the one hand, our model is trained on 9-second sequences at 10 Hz. On the other hand, the agent simulation task is more than a timeseries problem, as most traffic scenarios involve hundreds of agents. Last but not least, the Waymo Open Motion Dataset is one of the largest datasets in autonomous driving, involving 574 driving hours and a 1.4 TB download size.

---

Q9: Does training time change significantly with larger patches?

A9: We did not notice a significant change in training time when using a patch size of 10, which may be due to the light weight of the RNN head (1 GRU layer with a hidden size of 128).

---

Q10: Why only model the space dimension autoregressively?

A10: While it is natural to model the time dimension autoregressively, it is difficult to determine the order of agents in the chain. Image autoregressive models (e.g., PixelCNN) also face this problem, where the order of pixels in the chain may significantly affect the performance.

---

Q11: Long-term vs short-term.

A11: Please note that scenarios are generated autoregressively rather than in one shot. Without good short-term performance, it is unlikely to have good long-term performance owing to compounding errors.

---

Q12: How about the extrapolation ability to longer sequences?

A12: We tried training a model on 5-second sequences and generating 9-second sequences during inference. The results below demonstrate the extrapolation ability of our approach.

Thank you for the insightful comments! We fully understand your concerns and hope to explain our motivation as well as some subtle details regarding your doubt.

Q1: Does decoder-only architecture really matter?

A1: Indeed, it is possible to achieve very good results with an encoder-decoder model, but our paper intends to convey the message that "decoder-only models are sufficient to do very well." In other words, it may not be necessary to design some intricate modules to encode the history and employ some well-designed DETR-like decoders for future prediction as done by the most advanced trajectory prediction models, such as Wayformer, MTR, and QCNet. Given that using a much more concise decoder-only architecture can attain super strong results, there is no reason for us to pursue a more complicated solution due to the principle of Occam's razor. Besides the conciseness, combining decoder-only Transformers with relative spatial-temporal positional embeddings has advantages in engineering, as we can utilize the key-value cache technique to reuse past computations. By contrast, typical encoder-decoder models need to encode the history and run the full decoder at each simulation step, which is quite inefficient.

Moreover, we also found that decoder-only models have very high sample efficiency. Compared with encoder-decoder models where each scenario is treated as one pair of history and future, BehaviorGPT models each time step as the current one and requires each state to model subsequent states’ distribution during training, which is equivalent to constructing training samples from every possible history-future pair of the time series. To illustrate the sample efficiency of BehaviorGPT, we evaluate the models (5M parameters, hidden size = 128, #Decoder layer = 4) trained with different proportions of training data. The table below shows that using merely 20% of the data for training has already enabled the model to surpass many SOTA. The high sample efficiency allows small models to achieve incredible performance, which is also an essential reason for using decoder-only architecture.

---

Q2: If one added the patching idea to other models or lowered the replan frequency, would those also perform much better?

A2: We believe the patching idea can be applied to other autoregressive models (e.g., Trajeglish). But one thing we want to point out is that a lot of top methods on the WOSAC are adapted from SOTA motion prediction models (e.g., MTR). These methods do not necessarily comply with the closed-loop requirements of agent simulation and often produce 8-second trajectory points in one shot. Thus, these open-loop methods have already been operating at an extremely low frequency (i.e., 0.125 Hz, which is forbidden). However, our approach outperforms them by a large margin when operating at a moderate replan frequency.

To help readers understand how the replan frequency may affect the simulation results, we varied the replan frequency of the same model (patch size = 10) during inference, which can be achieved by discarding a portion of the predicted states at each simulation step. The test set results are shown as follows. From the table, we can see that increasing the replan frequency from 1 Hz to 2 Hz can improve the overall performance, which may benefit from the enhanced reactivity. This phenomenon demonstrates that the performance gain is not merely due to the lower replan frequency, as the model with a patch size of 10 beats that with a patch size of 5 even harder if using the same replan frequency (i.e., 2 Hz). However, we found that an overly high replan frequency harms the performance, as indicated by the third row of the table. Overall, we conclude that using a larger patch indeed helps long-term reasoning, but a moderate replan frequency is important for temporal stability, which may be neglected by prior works.

---

Q3: Is it appropriate to use the terminology "patching"?

A3: Thanks for your comment, and we agree that the notion of a "patch" is more suitable for describing 2D or 3D stuff. In fact, BehaviorGPT uses a multi-agent formulation to simulate all agents' states simultaneously. In this sense, our formulation involves multiple dimensions, including the agent dimension, the time dimension, and the state dimension (3D position + 2D velocity + 1D yaw angle). Thus, we think it appropriate to use the notion of a "patch" in our case.

---

Q4: What about the motion forecasting results?

A4: As mentioned above, many approaches on the WOSAC leaderboard are adapted from typical marginal motion forecasting models, so they can be tested on the Waymo motion prediction benchmark without any effort. In contrast, BehaviorGPT is a generative model that considers the joint distribution of all agents in the scene, which is misaligned with the objective of the Waymo motion prediction benchmark. Thus, it is more reasonable to compare our approach with other joint multi-agent motion prediction models. In fact, the WOSAC leaderboard also evaluates minADE, the most commonly used metric for motion prediction. We also noted that QCNeXt, a joint multi-agent prediction model that has won 1st place in the CVPR 2023 Argoverse 2 multi-agent motion forecasting challenge, is also on the WOSAC leaderboard. We found that QCNeXt performs much better on minADE (1.08 vs our 1.54), but its closed-loop performance lags far behind most simulation-oriented models. It seems to be a trade-off.

We sincerely thank you for the valuable feedback. In the following, we answer your questions to resolve some critical concerns.

Q1: Details of final models and ablation studies.

A1: We list the detailed configurations of each group of experiments as follows, which will be added to the revised version.

---

Q2: Relevant ideas in related fields.

A2: Thank you for mentioning the relevant ideas in related fields, such as the concurrently proposed multi-token prediction in LLMs and the KNN design of HPTR, which will be discussed in the revised version. However, using patch-wise tokens in decoder-only Transformers is a new attempt to our knowledge.

---

Q3: Implementation Details.

A3: All hidden sizes are set to 128. All attention layers have 8 attention heads with 16 dimensions per head. We train our models for 30 epochs with a batch size of 24 using the AdamW optimizer. The weight decay rate and the dropout rate are both 0.1. Using the cosine annealing scheduler, we decay the learning rate from $5 \times 10^{-4}$ to 0. Our main results are produced by a single model with 2 decoder layers and 3M parameters. Other details that you are interested in are discussed below.

(1) Is it correct that you have as many RNNs as modes, with N_mode different parameters?

The parameters of the RNN are shared across all modes.

(2) Sampling strategy.

We randomly sample one behavior mode from each agent's next-patch distribution until we complete the 8-second trajectories of all agents. To obtain 32 replicas of rollouts, we repeat this process using different random seeds.

(3) Post-processing tricks.

We do not apply any other post-processing tricks.

(4) Is it correct that your input tokens are continuous?

Yes, our input and output tokens are all continuous.

(5) Do you run the patch generation in a receding horizon fashion?

We try to increase the replan frequency by discarding a portion of the predicted states at each simulation step. The test set results produced by the model with a patch size of 10 are shown as follows. From the table, we can see that increasing the replan frequency from 1 Hz to 2 Hz can improve the overall performance, which may benefit from the enhanced reactivity. However, we found that an overly high replan frequency harms the performance, as indicated by the third row of the table. The results in Table 1 are based on 1-Hz replan frequency.

(6) Why is the training so slow?

Compared with typical encoder-decoder models, the sequence length in our approach is much longer. For example, most motion forecasting models on the Waymo Open Motion Prediction Benchmark utilize 1-second history to predict 8-second future trajectories, while our decoder-only Transformers simultaneously utilize 1-second, 2-second, ..., and 8-second history to predict the next patch during training.

(7) Why do we use RNNs for the patch generation?

We hope to comply rigorously with the closed-loop requirements of agent simulation via autoregressive RNNs. We have not tested a simple MLP head yet.

---

Q4: What are the limitations?

A4: (1) Currently, BehaviorGPT underperforms in kinematics-related performance (e.g., linear/angular speed likelihood), which can be enhanced by incorporating a kinematic model (e.g., bicycle model); (2) BehaviorGPT does not support controlling agent behavior with prompts (e.g., language, goal points). Future work on agent simulation may consider controllable generation; (3) We have not verified whether BehaviorGPT will facilitate the development of motion planning.

---

Q5: Scaling experiments.

A5: We try to conduct some scaling experiments under the constraints of time and computing budget.

(1) Hidden size: We vary the hidden size to obtain models with different parameters. The experiments use 20% of the training data and 2 layers of the Transformer decoder. As depicted below, increasing the hidden size consistently improves the performance.

(2) Number of decoder layers: We also try to fix the hidden size as 128 and vary the number of decoder layers, obtaining models by training on 20% of the data. Based on the experimental results below, we can conclude that increasing the depth of the models can benefit the performance.

We sincerely appreciate your thoughtful comments and constructive suggestions. In the following, we attempt to address your critical concerns by giving more analyses and clarifying the implementation details.

Q1: What makes BehaviorGPT outperform SOTA with fewer parameters?

A1: Indeed, the most notable design choices that set BehaviorGPT apart from other autoregressive behavior models lie in three perspectives, including (1) the patch-based tokens, (2) the decoder-only Transformers based on relative spacetime representation, and (3) the use of continuous distributions. Let us understand these design choices better with more in-depth analyses.

(1) Patching mechanism: Our paper argues that the Next-Patch Prediction scheme can enhance models’ capability of long-range reasoning. In Table 3, we show that using a larger patch (e.g., increasing the patch size from 5 to 10) can improve the performance. However, some may doubt that the performance gain is merely due to the lower replan frequency (e.g., the 1 Hz for a patch size of 10 compared with the 2 Hz for a patch size of 5) as discovered by Waymo [30], which may enhance the temporal stability of trajectories. To clarify this, we try to increase the replan frequency by discarding a portion of the predicted states at each simulation step. The test set results produced by the model with a patch size of 10 are shown as follows. From the table, we can see that increasing the replan frequency from 1 Hz to 2 Hz can even improve the overall performance, which may benefit from the enhanced reactivity. This phenomenon demonstrates that the performance gain is not merely due to the lower replan frequency, as the model with a patch size of 10 beats that with a patch size of 5 even harder if using the same replan frequency. However, we found that an overly high replan frequency harms the performance, as indicated by the third row of the table. Overall, we conclude that using a larger patch indeed helps long-term reasoning, but a moderate replan frequency is important for temporal stability, which may be neglected by prior works.

(2) Decoder-only Transformers with relative spacetime representation: Decoder-only Transformers with relative spacetime representation can utilize training data more efficiently. Compared with encoder-decoder models where each scenario is treated as one pair of history and future, BehaviorGPT models each time step as the current one and requires each state to model subsequent states’ distribution during training, which is equivalent to constructing training samples from every possible history-future pair of the time series. To illustrate the sample efficiency of BehaviorGPT, we evaluate the models (5M parameters, hidden size = 128, #Decoder layer = 4) trained with different proportions of training data. The table below shows that using merely 20% of the data for training has already enabled the model to surpass many SOTA. The high data efficiency allows small models to achieve incredible performance.

(3) Choice of distributions: Unlike standard autoregressive models, we choose continuous distributions for modeling since we prefer more end-to-end solutions. To model traffic scenarios with categorical distributions, we must first conduct discretization, which involves many essential implementation details (e.g., choosing the proper vocabulary size). Given that the code of Trajeglish is not publicly available, it is inappropriate to recklessly conclude something like "using continuous distributions is better" by comparing our model with a discretized variant that is not well-tuned. However, we indeed found that the choice of distributions is crucial. Prior to using Laplace distributions, we used Gaussian distributions for parameterization and noted that the model could not converge normally (see Figure c in the general response). Moreover, we observed that directly optimizing the full mixture models is better than using the winner-take-all training strategy (Realism score: 0.7396 vs 0.7146). These are the most critical things to make a continuous autoregressive model work well.

---

Q2: What about the implementation details?

A2: All hidden sizes are 128. All attention layers have 8 attention heads with 16 dimensions per head. We train our models for 30 epochs with a batch size of 24 using the AdamW optimizer. The weight decay rate and the dropout rate are both 0.1. Using the cosine annealing scheduler, we decay the learning rate from $5 \times 10^{-4}$ to 0. Our main results are produced by a single model with 2 decoder layers and 3M parameters. To obtain 32 replicas of rollouts, we randomly sample a behavior mode from each agent's next-patch distribution using different random seeds.

---

Q3: What is the significance of not using the GT when unrolling the RNN?

A3: We have tried a non-autoregressive GRU that does not rely on the predicted states when unrolling the next states, which can achieve lower minADE (1.5203 vs 1.6554) but performs worse on higher-order kinematic metrics (speed likelihood: 0.3517 vs 0.3544, acceleration likelihood: 0.2630 vs 0.2873).

---

Q4: What are the limitations?

A4: (1) The kinematic performance can be enhanced by incorporating a kinematic model (e.g., bicycle model); (2) BehaviorGPT does not support controlling agent behavior with prompts (e.g., language, goal points); (3) We have not verified whether BehaviorGPT will facilitate the development of motion planning.