GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS

Thank you for recognizing the significant speedup provided by our simulator and its potential to enable research in complex real-world scenarios. Below are our clarifications:

Sim functionality and mixing agent behaviors

The paper claims compatibility with existing datasets but only demonstrates map loading, leaving other functionalities unclear. For example, the imitation learning experiment or mixing agent behaviors—some from datasets and others from RL agents during training.

We highlight that GPUDrive fully supports mixing agent behaviors; you can combine replay agents, scripted agents, pre-trained agents, and RL agents in a single rollout without sacrificing parallelism. $\color{green}{\text{We have added this and plans for future work in this regard to Section 5 of the paper.}}$

Domain randomization

I believe one major advantage of parallel environments is that it allows you to do randomization across different environments (worlds). However, the paper lacks detail on whether GPUDrive supports this capability.

Domain randomization is certainly a valuable feature for any simulator. While it is already possible to implement some types of domain randomization, such as randomizing goals or initial positions, we believe the primary challenge of driving all agents to the observed goals remains unsolved for now. We plan to explore domain randomization in future work and appreciate the reviewer’s suggestion.

Customization

While GPUDrive offers a Python interface, I am curious how easy it is to customize those key elements in the environment given that the observation, reward, and dynamic functions are written in C++.

Thank you for pointing this out. We believe GPUDrive is easy to extend despite being in C++ for two key reasons:

1. We offer extensive Python bindings that cover most expected observations, reward functions, and dynamics, allowing users to customize many components without having to touch the C++ code.
2. C++ remains a popular language, particularly in the robotics and autonomous vehicle communities, and many of our users have the necessary familiarity to extend the framework. Additionally, the framework handles parallelism, so even a basic understanding of C++ is sufficient for implementing new rewards or dynamics.

Algorithms

Experiments only evaluate IPPO, despite the paper claims that it targets a mixed-motive setting.

The primary goal of the paper is to propose a fast multi-agent simulator, with the RL experiments and code included to help users get started. We note that IPPO is a valid solver for mixed games and performs well empirically. In many multi-agent problems, it has been observed that PPO is an effective solver (See e.g., MAPPO [1]).

If the reviewer is asking whether the "independent" aspect of IPPO is compatible with a general-sum game, the answer is yes—independence refers solely to decentralized training.

Questions

In Figure 3, the speedup appears nearly linear. However, it would be helpful to examine scaling performance by adding more environments to identify saturation points and gain insights into system limitations.

We apologize for the confusion, please note that this is a log-log plot. The plot shows that as we increase the number of worlds (x) the throughput increases at the same rate (y).

In Figure 5, do you use the CPU-parallel version of Nocturne?

Yes! For a fair comparison, we plot both the single-CPU (striped blue) and the parallelized Nocturne using PufferLib (dotted blue, 16 CPUs). Note that the PufferLib version has been carefully designed to outperform naive Python multiprocessing.

Conclusion

Thank you for your valuable feedback! We hope we have addressed the reviewer's comments and are happy to further discuss any of these points. We kindly ask that if we have addressed the reviewer's concerns, they consider increasing their support for our paper.

References

[1] Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., & Wu, Y. (2022). The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35, 24611-24624.

Hi,

We appreciate the rebuttal and discussion. We have attempted to clarify some of the points below. If we can restate your objections to the paper they appear to be the following:

• The simulator is written in C++. We assume since this was not brought up in the response that we are in agreement that this is not an issue since most simulators are actually written in C++ or other languages and just have python bindings.
• The sim does not currently support domain randomization. While we agree that domain randomization would be an interesting feature to have, we are unclear why it is a critical feature. The simulator already comes with 450000 unique scenes and is not readily solved by extensive tuning of an RL algorithm. Is it possible that there was confusion and the existence of 450000 unique scenes was not clear? If so, we appreciate that being pointed out and will rewrite it.
• Using IPPO instead of MAPPO. Note that our usage of an RL algorithm is not to make claims about algorithms but purely to point out how quickly scenes can be solved in the benchmark. If it would lead the reviewer to increase their score, we can run MAPPO. We are familiar with the distinction between IPPO and MAPPO and it is unclear to us why MAPPO would make a difference here since the empirical differences between the two are small. For example, in the MAPPO paper, centralized value functions only really helped in 3 and 4 player Hanabi.
• The simulator does not provably contain mixed games. We want to caution that there are two possible uses of the word mixed and we are not sure which one the reviewer is referring to. In the first case, mixed agents i.e. multiple policies, the simulator does support this and we have updated the text already to clarify it. If the reviewer is referring to the simulator containing general sum games, whether it does or does not is a function of what reward functions the user chooses to use. Under the default reward of goal reaching, the game is general sum and also contains conflicts of interest since this is a standard occurrence in driving (for example, at intersections).
• "The plot shows that as we increase the number of worlds (x), the throughput increases at the same rate (y)," to be unclear. Does this imply that the simulator has no limit on parallelization? If there is a limit (assume you use commercial GPUs), linear scaling would eventually plateau as the number of environments increases. Our point, which we did not make clearly, was that the simulator experiences sublinear scaling but speed is still improving with more environments. We cannot run more than 1000 or so environments as we exhaust all the GPU memory at that point.

Hopefully, those points clear up some of the issues?

Finally, we want to re-address the point about user difficulty in programming the simulator. We claim a mono-language implementation in Python is neither sufficient nor necessary to meet our dual goals of productivity and performance. From a language perspective, the GPUDrive architecture can be decomposed as follows:

1. A learning layer (Python)
2. A bridging layer (C++)
3. CUDA kernels.

Why is it hard to do it any other way?

1. CUDA kernels must be written in CUDA-C++ because NVIDIA does not support JIT compilation for any language other than C++, and JIT compilation is upstream of maximizing performance on NVIDIA GPUs. For instance, by opting for JIT compilation instead of the more classic Ahead-Of-Time compilation we enable “Runtime LTO”. We refer the reviewer to JIT LTO for an explanation of the performance benefits of this feature.
2. Though written in C++, we argue the Bridging layer does not decrease productivity. A simplified view of the PyTorch architecture is that it exports Python bindings to CUDA kernels. Just as with GPUDrive's architecture, the Python bindings are bridged to CUDA via C++. We observe this makes PyTorch no less productive for end-users.

We could of course attempt to rewrite the simulator using an array-based programming language like PyTorch or Jax. However, implementing complex training environments using state-of-the-art simulation methods requires complex data structures and non-trivial control flow (traversing acceleration structures, collision solvers, state machines, conditional logic in functions) is cumbersome in array-based abstractions. For this reason, Jax and Torch-based environments rarely contain all these features.

Dear reviewer

Thank you for your thoughtful comments and for noticing GPUDrive is built on top of a real-world driving dataset (WOMD), which enhances sim realism. Please find our responses below:

Complex behaviors

While the focus of this paper is on providing a novel simulator, it would be very interesting to see some more complex behavior over longer time-horizons to fully capture the capabilities unlocked by the simulator (e.g., training a single agent policy with higher velocity limit to weave through a simulated traffic scene, etc.)

We think this is a great suggestion and an interesting direction for future work. As mentioned, the key contribution of the paper is the simulator itself; we look forward to users or future papers showing these behaviors!

... showcasing such behavior would likely require addressing the “Absence of a map” limitation raised in the paper, in order to formulate a more sophisticated reward function. An important question would then be how easily this could be integrated, and how much the absence of such a feature could hurt adaptation of the simulator.

The reviewer correctly points out that some features are challenging to implement without a map. While the current simulator task of reaching goals without collisions can be achieved without an explicit map, more complex scenarios would benefit from one.

That said, we do incorporate map-like features, such as lane lines, which assist with imitation learning and allow for reward functions like staying lane-centered. Full map connectivity is available in the dataset, and we are actively working to $\color{orange}{\text{integrate full map connectivity into the simulator}}$.

References

The discussion of batched simulators could be extended to include references [1-3], where [1] has driven many results in single-agent robot learning, while [3] considers heterogeneous multi-agent settings

Thank you for pointing this out! $\color{green}{\text{We integrated these references in the discussion of batched simulators.}}$

Typos

Thank you for catching these!

Figure 3 mentions performance on an RTX 4080, while line 711 states RTX 8000

$\color{green}{\text{We fixed that typo, it should be RTX 4080}}$.

Line 308: “number valid number”

$\color{green}{\text{Noted and fixed.}}$

Questions

How easily can the simulator be updated to efficiently provide map-like utilities that allow for lane-keeping rewards (re mentioned limitations)?

Some required features are already present in the data structure; they can be implemented by adding a reward for staying near lane lines. However, for a more robust implementation of this reward, it would likely be necessary to support maps.

Do you support loading multiple different policies for individual agents? Could they have different sampling rates? How would these aspects affect efficiency?

We do support loading multiple policies per agent, and they could have different sampling rates as long as they are integer multiples of the simulator time-step. Since agents not taking action at a step would simply be stepped, this would only increase the speed of the simulator step.

How do traffic jams affect throughput (re BVH)? This could be an interesting experiment to add.

Thank you for the suggestion! We currently do not have an answer, but we agree that studying the impact of traffic jams on throughput (in relation to BVH) would be an interesting avenue for future research.

In video scene_53.mp4, agent 4 displays rather jerky behavior when moving towards its goal - could you elaborate on the underlying reasons?

Great observation and thank you for checking out the videos! These agents are trained solely with a goal-reaching reward; they have no incentive not to drive jerkily. If we added jerk penalties (which can easily be done), this would probably disappear.

In video scene_43.mp4, agents 1 and 10 seemingly disappear without reaching their goals - could you elaborate on this behavior?

Yes, we configured the simulator to remove vehicles from the scene when they collide with a road edge or another agent, although this behavior can be easily adjusted in the config file. The video shows that the pre-trained policy, which achieved 95% performance over 1000 scenes (colliding 5% of the time), still leaves room for improvement. We are actively working on $\color{orange}{\text{improving both the effectiveness and diversity of the simulation agents}}$.

Conclusion

Thank you for taking the time to provide such a thorough review! We believe we have addressed all of your questions and welcome further discussion. If all your concerns have been resolved, we would be thankful if you could consider increasing your support for our paper.

Dear Reviewer,

Thank you for your valuable feedback! We appreciate your recognition of GPUDrive's scaling benefits and mentioning that we present the wide range of simulator features in a “comprehensible way”. Please find our response to your comments below:

Reactive Sim Agents

The paper does not provide simple IDM (intelligent driving models) agents that can be sometimes practical to have basic reactivity to the ego agent.

We agree that reactivity of simulation agents is crucial. While we do not include simple IDM agents, we provide pre-trained reactive simulation agents trained through self-play in the simulator that offer this basic reactivity. The behavior of these agents can be seen in the videos at https://sites.google.com/view/gpudrive/. We are actively working on expanding the $\color{orange}{\text{diversity and effectiveness of available sim agents}}$.

End-to-End Training Throughput

The authors mention that the current work is limited in properly utilizing the generated samples for optimal training.

We assume the reviewer is referring to the end-to-end training performance in Section 4.2. We acknowledge the concern regarding sample utilization. We have since addressed this issue and $\color{green}{\text{added an improved IPPO implementation, improving the training speed 10X: from 50K to 500K AFPS. Please see the updated Figure 5 in the paper.}}$

Implementation in C++

Just a thought: The implementation is in C++ and it provides a binding interface with Python environments. It would have been nice to have a mono-language (primarily Python based) tool as the model training and other related pipelines are mostly in Python.

We claim a mono-language implementation in Python is neither sufficient nor necessary to meet our dual goals of productivity and performance. From a language perspective, the GPUDrive architecture can be decomposed as follows:

1. A learning layer (Python)
2. A bridging layer (C++)
3. CUDA kernels.

Why is it hard to do it any other way?

1. CUDA kernels must be written in CUDA-C++ because NVIDIA does not support JIT compilation for any language other than C++, and JIT compilation is upstream of maximizing performance on NVIDIA GPUs. For instance, by opting for JIT compilation instead of the more classic Ahead-Of-Time compilation we enable “Runtime LTO”. We refer the reviewer to JIT LTO for an explanation of the performance benefits of this feature.
2. Though written in C++, we argue the Bridging layer does not decrease productivity. A simplified view of the PyTorch architecture is that it exports Python bindings to CUDA kernels. Just as with GPUDrive's architecture, the Python bindings are bridged to CUDA via C++. We observe this makes PyTorch no less productive for end-users.

We could of course attempt to rewrite the simulator using an array-based programming language like PyTorch or Jax. However, implementing complex training environments using state-of-the-art simulation methods requires complex data structures and non-trivial control flow (traversing acceleration structures, collision solvers, state machines, conditional logic in functions) is cumbersome in array-based abstractions. For this reason, Jax and Torch-based environments rarely contain all of our features while meeting our performance targets.

Questions

• Pedestrian and Cyclist Behavior: Both pedestrians and cyclists are controlled within the simulator. We use the same dynamic models as for vehicles, with smaller bounding boxes for pedestrians to reflect realistic behavior.
• Ethical Statement: $\color{green}{\text{The ethical statement has been added as Section 7 in the current version}}$.
• Log Scale for Fig. 5: We have the log plot. $\color{green}{\text{Please see the updated Figure 5 in the paper}}$.

Conclusion

Thank you again for your thoughtful comments! We hope we have addressed all of your questions and welcome any further discussion. If all concerns are resolved, we kindly ask that you consider increasing your support for our paper.

Dear Reviewer,

First of all, thank you for your kind words about the clarity and structure of the paper and code. We are excited about the potential of GPUDrive to enable RL at scale on a single GPU and make the algorithmic design process interactive. Please find our responses to your comments below:

Figure 2

Figure 2 needs a better caption and an explanation

We agree that the caption and explanation could be more detailed. We have $\color{green}{\text{updated the place and caption and added a more comprehensive explanation}}$ in the current version (see Figure 1).

Integrating Multiple Datasets

Designed to fit one exact dataset. A section explaining the effort required to integrate other datasets is desirable.

We appreciate the suggestion to explain the effort required to integrate other datasets. We are actively working on this and have $\color{green}{\text{added a new paragraph in the current version (see Section 5)}}$ to address this and will point to our data processing code after the rebuttal period.

Questions

• Benchmarks and Dataset Limitations: Yes, we are actively $\color{orange}{\text{integrating additional datasets}}$, such as the Nuscenes (https://www.nuscenes.org/); this simply requires putting the files into our JSON format. Note that most large, diverse datasets have similar limitations as they are collected from the sensors of a single vehicle. While no dataset is perfect, we believe that working with human data, even with its limitations, provides significant value.
• Stable Baselines and PPO Speed: We acknowledge the limitations of the Stable Baselines 3 (SB3) IPPO implementation. In response, we have implemented an $\color{green}{\text{improved version of IPPO, achieving an end-to-end training throughput of 500K, a 10X speedup}}$ compared to the previous SB3 version.
• Multi-GPU Support: We do not currently have multi-GPU support as this is a property of the training code and not the simulator. It is straightforward to add multi-GPU support by wrapping the model in torch DDP but we do not mention this in the work. We can include it if it feels important to the reviewer.

Conclusion

Thank you for helping us make this work better! We hope we have addressed your comments properly and welcome further discussion.

Dear reviewers,

As the discussion period is coming to an end, we kindly ask for your engagement with our rebuttal. We have put significant effort into addressing your concerns and would greatly appreciate any further feedback or discussion.

Thank you all for the time and thoughtful comments so far!