Robust Autonomy Emerges from Self-Play

We thank the reviewer for their thoughtful review, and list a detailed reply for all questions raised.

To contextualize this rebuttal we would like to clarify one of the main differences between Agent and Environment Simulation (WOSAC) and learning a driving policy (this work). The goal of WOSAC (and motion forecasting more generally) is to learn how human traffic generally behaves. If a vehicle collides or goes off-road (in the real world, or in a log due to sensor noise) a good agent and environment simulator will simulate this event. The WOSAC benchmark contains 3% collision events and 12% off-road events. A competent driving policy should never collide, and rarely go off-road.

For example, the best Agent and Environment Simulations (SMART etc) should reproduce the logged trajectory almost perfectly to maximize their score. A good policy will not because certain noisy trajectories lead to collisions and off-road events.

It was never our aim to claim to be state-of-the-art on WOSAC, nor should a competent driving policy be expected to. We show that in interactive driving simulations (CARLA, nuPlan, Waymax) our policy is a competent driver which gets to its goals safely, robustly, and efficiently. However, it does not necessarily achieve this in a human-like manner, nor does it imitate the sense noise of the waymo perception stack. This leads to a lower WOSAC score. The only mention of WOSAC in the main paper is on line 429. We primarily mention it as it is interesting, but not critical to the claims of the paper, that we achieve a non-trivial WOSAC score.

Additionally references

Thank you. We will add them in the final version.

For the Waymax benchmark, the authors propose a Score metric which is not present in the original Waymax paper as their summary metric...

It is unfortunately quite easy to trade individual metrics off in the Waymax benchmark. Yes BC and Wayformer perform better in Log ADE (following the demonstrated trajectory). They clearly should as this was their training objective. However, they fall short across all other simulation objectives with a collision rate of 4+%.

As noted in the Claims and Evidence section, WOSAC should be evaluated on the newer 2024 leaderboard.

We are working on this, and hopefully will have numbers for the final version.

It is unclear how much hyperparameter tuning of C_reward was needed to achieve state of the art performance on each dataset. / How much tuning (if any) was done to select the reward hyper parameters for each benchmark?

All benchmarks use the same reward hyperparameters (the mean of the randomized range). We did not tune reward parameters as we considered this overfitting to the target benchmark (and thus would invalidate our zero-shot generalization claim). There is a single difference between all benchmarks, which is the radius of the target region. Carla requires more precise navigation (the agent needs to follow the given route points closely.) than Waymax and nuPlan and uses a tighter target radius. We will add these values to the appendix in the final version.

How does the performance of SoTA WOSAC policies compare against your policy on the Waymax benchmark? I would specifically like to see the performance of SMART which is available open source here.

SMART is certainly performing better at Agent and Environment Simulation (WOSAC). The aim of the WOSAC results was to show the policy learns human-like driving, and not to claim state-of-the-art on this benchmark. As such, we sharply outperform SMART on collision scores, our primary metric of interest, but underperform on distance to logged trajectories, which our paper did not target and is not always relevant for a self-driving car.

Have any other prior art used the baseline implementations presented in the Waymax paper as a benchmark?

As developed, the tools developed in the waymax paper are primarily intended for the development of human-like simulation agents. However, as the data is interesting and can be used to measure robustness, other works have started to use it in similar manners to those used here [1, 2, 3].

[1] Cornelisse, D., Pandya, A., Joseph, K., Suárez, J., & Vinitsky, E. (2025). Building reliable sim driving agents by scaling self-play. arXiv preprint arXiv:2502.14706.

[2] Xiao, L., Liu, J. J., Ye, X., Yang, W., & Wang, J. (2024). EasyChauffeur: A Baseline Advancing Simplicity and Efficiency on Waymax. arXiv preprint arXiv:2408.16375.

[3] Charraut, V., Tournaire, T., Doulazmi, W., & Buhet, T. (2025). V-Max: Making RL practical for Autonomous Driving. arXiv preprint arXiv:2503.08388.

Will you open-source this simulator to allow others to reproduce your results?

We would very much like to open-source this work, but cannot commit due to approval requirements beyond our control.

We thank the reviewer for their thoughtful review, and list a detailed reply for all questions raised.

GPUDrive

We are aware of the work, and have been in close communication with the authors of GPUDrive from the start of their project. We will cross-cite each other's work for the camera-ready paper.

... requires accurate high-definition (HD) maps to achieve robust planning, which are not available for most areas

True, training requires HD-maps. Unclear how big the reliance on HD maps is during deployment. Our policy generalizes across several different vector map formats (Carla, nuPlan, Waymo). It does not directly see the vector representation, but rather an intermediate representation. A performant sense stack should be able to construct this representation.

The paper adopts the Frenet coordinates, which potentially encourage the vehicles driving on the center of lanes

This is a misunderstanding. The Frenet frame provides a policy with a rotation invariant representation of its local world. However it does not limit its driving ability. The policy choses to go into the opposing lane if required (see Figure 4). After all, PPO optimized the policy on such a huge amount of experience that it seems inconceivable that biases like lane-centering would survive only because of the input representation.

... Sec. F.3? ... Figure A4?

Our training environment is augmented with erratic drivers and severe dynamics noise. Sec F.3 evaluates robustness under ordinary conditions. In this regime, our agents achieve over 3 million kms of driving between incidents of any kind. This corresponds to approximately 99.99999997% of transitions in the simulator being collision-free. To our best knowledge, this is one of the first examples in the literature where such level of robustness is demonstrated for a stochastic policy trained with RL.

The policy drives better with a higher decision making frequency (Fig A4) as it has to commit to "shorter" actions (lower reaction time).

Will clarify in final paper.

... Could the authors clarify which other rewards can potentially be used for more delicate designs?

Our reward encodes road rules, but not behavior or expected interactions between agents. We do not use:

• Penalties shaping expected proximity to other agents
• Expected speeds around other agents
• Forward progress

These terms trade short term performance with generalizability.

... Lines 258-268 ... how the policy autonomously adapts to various scoring criteria if they are contradictory.

Our eval is zero-shot. We use a single reward condition for all benchmarks, there is no autonomous adaptation.

Would using a single policy ... result in unstable training, ...?

It does not. Ultimately, the major reward the agent receives is to reach a goal. The randomized components of the reward are secondary. This is also reflected in the behaviors learned. The agent first learns to move towards the goal and collect close-by goals, only after that will it pay attention to red lights, driving direction etc. The training curves are stable. We will add them to the appendix.

Routing information / potential label leakage?

We do not use the expert trajectory in any way (no leakage). For nuPlan, we use a point in the final lane segment as the goal (no routing information, our policy learned to route locally during training). For waymax, we use the final location of the log ego vehicle projected forward 20m along the lane to ensure that the agent continues driving if it arrived early. In neither setup, our agent sees more information that baselines.

2024 WOSAC leaderboard?

Working on it, unfortunately wont finish in time for this rebuttal.

Clarifications on Figure A3 and randomization

The metrics for nuPlan and Waymax are very compressed. Both benchmarks have a certain number of scenarios that are unsolvable and thus a non-zero optimal policy. Carla has fewer scenarios that are completely unsolvable, and its metric degrades less gracefully. Reward randomization does help for all methods (nuPlan 93.08 -> 93.17; Waymax 98.79 -> 98.92).

Detailed number of data used.

Over 99% of trajectories are used during training and need to be simulated to completion to calculate rewards and advantages for each timestep. Figure 1 is an accurate comparison of sizes. Gigaflow is about five orders of magnitude larger than prior public attempts.

We filter 80% of time-steps (see section 2.2). This filtering happens after simulating rollouts. Even accounting for this filtering, RL training sees at least four orders of magnitude more data than a single pass over any publicly available data would provide.

Typos

Thank you. Will be fixed.

Supplementary videos

Sorry, we forgot to upload them: https://drive.google.com/drive/folders/1hlDrPDQIu56-iq039fwOa18g8aUGSKYl?usp=sharing

Open-sourcing

We would very much like to open-source this work, but cannot commit due to approval requirements beyond our control.

We thank the reviewer for their thoughtful review, and list a detailed reply for all questions raised.

The intuition and motivation to use one policy for all agents is under-discussed in this work. Furthermore, the ablation study of comparing shared policy and separate policies for different types of agents is not reported in this work.

N separate policies are technically hard to implement efficiently at scale for two reasons: Computational cost grows, as there is a significant amount of routing required from agent states to potentially non-uniformly batched inference and training steps / sequences Separate policies likely train slower: Each separate policy sees 1/N amount of experience. Prior evidence suggests that embedding a population in a single policy can accelerate learning [1].

The amount of policies required to cover our continuous reward parametrization can potentially be quite large.

[1] Liu, Siqi, et al. "Neural population learning beyond symmetric zero-sum games." arXiv preprint arXiv:2401.05133 (2024).

Although this work shows promising results of learning robust policy via self-play, it is notable that all policies presented in this paper operate on highly abstracted vector space without sensor inputs, which is different from real-world driving agents that reason over their online sensor data.

The reviewer is right that scaling this work to sensor simulation or real-world deployment will require additional work, and presents an exciting avenue of future work. One thing to note is that we are the first work to show that a single policy can generalize well across several different vector maps (Carla, nuPlan, Waymo). The main reason for this is that our policy does not directly see the vector representation, but rather lane boundaries etc. These inputs can likely be inferred by a sense stack, although getting a good noise profile will be part of future work.