ReSim: Reliable World Simulation for Autonomous Driving

Thank you for the careful and constructive review. We answer each of your questions as follows, and will update these discussions in the revision.

Effect of expert and non-expert simulation data alone.

Our intuition of including both expert and non-expert simulation data is to cover the wide range of possible behaviors. Non-expert simulation data is generated by predetermined rules (such as slight turns and hard turns), without much consideration of its likeness to human behavior. Thereby, we also include expert simulation data to compensate for the loss of human-like maneuvers such as driving smoothness. We agree that explicitly ablating their effects would be insightful and will consider it in future experiments.

Inobvious novelty in model design, yet good data contribution.

As you pointed out, our primary contribution is a data-centric methodology that incorporates heterogeneous data from multiple sources to train a reliable driving world model. The essence of the proposed method is our tailored training recipe (Sec. 2.2), which includes a new dynamics loss, improved noise sampling, and a multi-stage learning scheme for effective learning on heterogeneous data, which could be insightful for future work in this direction.

Do baselines use the same video model?

We primarily use two models as our baselines, namely ReSim (w/o sim.) and Vista.

• ReSim (w/o sim.) baseline uses the same base model (CogVideoX [1]), and training procedure as our full model; the only difference is the exclusion of the CARLA data. This provides a fair ablation of the contribution of the simulated data.

• Vista is a widely used baseline, which is developed upon the Stable Video Diffusion [2] with different training data and procedures. The comparison against Vista demonstrates the overall improvement of our proposed system.

We will clarify their architectural distinction in the revision.

---

[1] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

[2] Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets

Thanks for your thoughtful review. We address each of your concerns below and will include all discussions in the revision accordingly.

Details on how to handle the domain shift among different corpus.

We mitigate the domain shift between real and simulated data through three key strategies:

• Dominance of Real Data: Our training corpus is overwhelmingly real-world data (4M+85K real samples vs. 88K simulated). This grounds the model's visual understanding in reality, while the small CARLA dataset is primarily used to teach the dynamics of rare events.

• Progressive Multi-stage Learning: We first train the model exclusively on real data. We then introduce mixed data while freezing the model's backbone and fine-tuning only a lightweight LoRA adapter, which prevents catastrophic forgetting of real-world visuals.

• Improved Dynamics Modeling: Our Dynamics Consistency Loss (Sec. 2.2, lines 116-131) encourages the model to learn abstract motion patterns instead of just visual details, making the dynamics learned from simulation more transferable to the real world.

We will elaborate on this part in the revision.

Intuition behind different data scales.

The intuition of different data scales is detailed in the following:

• OpenDV (4M): We use all training samples from filtered OpenDV following Vista to maximize scenario generalization.

• NAVSIM (85K): We included all samples from the NAVSIM training split to provide expert trajectories for learning controllability.

• CARLA (88K): We collect this data of a similar size to NAVSIM to provide balanced action-controlled samples from the real-world and simulated domains. Despite its small scale, this data effectively improves controllability and reward estimation.

We did not perform tuning their ratio due to computational constraints. We believe adding more labeled expert and non-expert action data would further improve controllability, which is left as future work.

Data debiasing to enable controlling diverse actions.

According to Sec. 2.1 (in lines 99-101), we debias the data collection process for non-expert actions via explorative driving strategies. For each data sample, one of the following strategies is randomly chosen for the ego vehicle:

• Slight Turns: The vehicle steers slightly left or right towards a randomly chosen angle between 10-20 degrees and then continues forward.

• Hard Turns: The vehicle performs a sharp left or right turn towards a random angle between 40-60 degrees.

• Forced Lane Changes: The vehicle executes a hard lane change to the left or right.

• Tailgating: The vehicle disables its brakes and applies full throttle to induce a rear-end collision with a vehicle ahead.

Each strategy also includes randomization of its internal parameters (e.g., turning angle, speed) to encourage action diversity. We will add these details to the revision.

How much noise can the algorithm tolerate during training and inference?

We hypothesize your question is related to the noise in diffusion process. We employ an Unbalanced Noise Sampling strategy (Sec. 2.2, lines 132-142) that biases training towards higher-noise diffusion timesteps (timesteps 500-1000 out of 1000). This forces the model to learn the underlying data structure and dynamics from highly corrupted information, rather than taking "shortcuts" on low-noise examples that are slightly corrupted and can be easily denoised. This approach improves robustness and dynamics, as shown qualitatively in Figure 9.

Thanks for your valuable feedback. We respond to each of your concerns below and will incorporate the discussions into the revision.

Quantitative results on the ablation study of unbalanced noise sampling and dynamics consistency loss.

We provide a quantitative ablation on the nuScenes dataset to validate the effectiveness of our proposed methods.

The results show that both unbalanced noise sampling and dynamics consistency loss improve video prediction fidelity. Note: to make an efficient comparison, these ablation models were trained with less compute than our final model in the main paper.

How to pick the evaluation trajectory for the human study?

For the human study, we created challenging test cases by pairing 20 random scenarios from the unseen Waymo validation set with mismatched, non-expert trajectories. For each sample, the associated trajectory is sampled from the same validation set, but with a different driving command. For instance, we might associate a "driving straight" scenario with a "turning left" trajectory as its non-expert action. We will clarify this process in the revised paper.

More details on CARLA data collection.

Our CARLA data collection follows a structured "expert-takeover" process. First, the expert PDM-Lite policy drives for the initial period (1s). Then, control is switched to one of the following exploratory strategies to generate diverse, non-expert actions for 4s:

• Slight Turns: The vehicle steers slightly left or right towards a randomly chosen angle between 10-20 degrees and then continues forward.

• Hard Turns: The vehicle performs a sharp left or right turn towards a random angle between 40-60 degrees.

• Forced Lane Changes: The vehicle executes a hard lane change to the left or right.

• Tailgating: The vehicle disables its brakes and applies full throttle to induce a rear-end collision with a vehicle ahead.

Each strategy also includes randomization of its internal parameters (e.g., turning angle, speed) to encourage action diversity. We will add these details to the revision. We will include the above details in the revision.

We appreciate your insightful review and have carefully considered your comments. We will revise the manuscript based on the discussions below.

Visual artifacts brought by CARLA data.

Thanks for the question. We typically do not see CARLA artifacts because our model is grounded in real-world visuals. The model is first pre-trained on the massive, real-world OpenDV dataset, establishing a strong visual generalization. When simulation data is introduced in later stages, the model learns to predict dynamics while preserving the realistic visual style learned from the input context.

However, in extreme night scenes with low visibility, which are more frequently shown in our CARLA data than in our real data, the model can occasionally struggle. One possible way to mitigate this issue is to increase the sampling rate of real-world night cases during training, e.g., repeating this data multiple times. We will add the discussion and more visualization to our failure mode analysis.

Slight overclaim in "overcome".

Thanks for this suggestion. We will revise "overcome" to a softer claim such as "alleviate".

Clarification on controllability in Figure 1b.

In Figure 1b, the control trajectory, which is applied to the following future sequence, is a slight left curve to the sidewalk. ReSim correctly follows this non-expert action, while the baseline continues driving straight. We amplified the curve in the diagram for illustration only.

For your reference, the actual control trajectory is "[(7.52, 0.09), (15.20, 0.48), (22.97, 1.17), (30.58, 2.13), (38.25, 3.39), (45.80, 4.93), (53.25, 6.76), (60.62, 8.86)]", representing the relative displacements of 8 positions over the next 4 seconds. The first element of each coordinate indicates the forward displacement, and the second represents the lateral displacement.

We will update Figure 1b in the revision to more precisely depict the control trajectory.

Trajectory difference with challenging actions in a longer horizon.

To validate the controllability of ReSim on non-expert actions with large drift from recorded action, we construct a subset by associating each scenario with a challenging, non-expert action sampled from other scenes. The average end point distance of this subset is 24 meters, and the evaluation horizon is 4s. The quantitative results below show that including simulation data improves the controllability of non-expert actions with large drift.

Note: The Inverse Dynamics Model (IDM) used for this metric is trained only on expert data, so its accuracy on these hazardous videos is limited, as you mentioned. Vista is not included as it cannot be controlled on a 4-second horizon, which exceeds its training window.

How does the model handle the behavior of other agents?

Since our work focuses on controlling ego vehicle only, the behavior of surrounding agents is learned implicitly from all training corpus, including OpenDV, NAVSIM, and CARLA, and cannot be explicitly controlled during inference time. Indeed, our CARLA data also introduce "non-expert" behaviors for surrounding agents since they are governed by various behavioral strategies such as being aggressive. Scaling up this simulated data to further improve stochastic modeling of all agents is a promising direction, and we will add this point to our future work section.

Terminology of "non-expert".

You are correct that our "non-expert" actions are generated using randomized strategies to ensure action diversity . We use this term to align with robotics literature [1, 2], where "non-expert" commonly contrasts with "expert demonstrations" and refers to a broad set of suboptimal or exploratory behaviors. We will clarify this in the revision.

---

[1] Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning

[2] Reward-Conditioned Policies