SimGen: Simulator-conditioned Driving Scene Generation

Before diving into the details, we reclaim the benefits of our simulator-conditioned pipeline.

Simulators can reconstruct scenarios from public datasets, real-world driving logs (e.g., Waymo, nuPlan), and program-generated scenes to obtain data with diverse layouts and annotations. Also, simulators allow for editing the motions and states of traffic participants with rules or interactive policies [1], enabling the generation of risky scenarios from driving records. Finally, simulators provide control over all objects and spatial positions, facilitating the controllable generation.

W1: Neglection of surround-view cameras.

We agree that multi-view generation is meaningful for driving systems, but it is diagonal to the problem we tackle here. Our goal is to achieve appearance and layout diversity in generating novel scenarios, while multi-view generation emphasizes consistency between adjacent views. They are two interrelated yet distinct. Rather than just overfitting on nuScenes [2], SimGen learns from both datasets and web data to ensure content diversity. The front-view setup provides SimGen with a unified IO for utilizing data with various sensor setups. We plan to achieve multi-view consistency via cross-view attention [2] and learning from 360-degree images [3] in our future work.

W2: Necessity of conditioning on simulators.

Conditioning on the simulator has significant advantages.

1. The MetaDrive simulator provides control over all objects and their 3D locations in a traffic scenario. It also provides access to 100K real-world traffic scenarios imported from Waymo and nuPlan. Besides, it can effortlessly generate scenes encompassing risky behaviors with its physics engines.

2. While mesh and occupancy can be utilized for conditional generation [4], existing annotators struggle to reconstruct scenes from web data. Thus, the model's generalization capability is limited to public datasets. Differently, semantics and depth can be rendered from simulators, and pseudo-labels can be obtained from datasets and web data. Therefore, SimGen achieves superior generation diversity while preserving controllability by corporating the simulator.

3. It also paves the path for agent-based interaction to unify the generated perception with downstream decision-making and simulation applications.

W3: Implementation of incorporating simulators.

1. Incorporating simulators is not trivial. Beyond implementing interfaces, we customize and extend the functions within MetaDrive, including importing attributes from real logs, modifying asset shapes, customizing camera parameters, designing instance cameras, etc. All codes will be made publicly available.

2. The usage of simulators cannot be replaced by traffic simulations like LimSim [5], as spatial conditions help enhance the quality of generated images. It's not feasible to export the first-view conditions (e.g., the semantic shapes) from a traffic simulator. While projected 3D boxes offer basic control over object positions and sizes, simulator assets closely resemble real objects in terms of 3D shapes and provide details like vehicle doors, windows, wheels, pedestrian limbs, etc. Moreover, annotating 3D boxes from web data is challenging, restricting models based on traffic simulation to generate scenarios within a limited amount of fully annotated data.

W4: The generation is restricted to simulator assets.

The simulator's assets include 12 foreground objects, such as vehicles, pedestrians, cones, traffic signs, etc., not limited to just roads and cars. For a specific category, SimGen can generate diverse appearances based on texts, not confined to the assets. Tab. 15 (9th row) shows SimGen generates different types of vehicles (vans, trucks, and buses), differing from the predefined asset. Besides, we report the per-class results of Tab. 4-5 in Supp Tab. 2-3. Tab. 2 shows SimGen's excellent controllability, with only a -4.8 decrease in the bus category. Tab. 3 validates SimGen's benefits across all nuScenes categories, including motorcycle (+2.9), cone (+1.6), and barrier (+1.7).

Questions.

1. Aligned with semantic labels of Cityscapes, SimGen can generate 19 categories, covering all classes in nuScenes and 97% pixels in daily driving scenes. Fig. 14 shows the diverse categories that SimGen can generate, including poles (r1c2), buses (r3c1), motorcycles (r3c3), barriers (r4c4), etc. Besides, the Supp Tab. 3 confirms SimGen generates all categories in nuScenes.

2. Part of DIVA dataset, DIVA-Real, shares similarities to OpenDV as both use YouTube data. But DIVA targets different tasks and has different data-preparing pipelines. Task objective: DIVA is designed for controllable image generation. To support multimodal conditions, DIVA offers depth and segmentation labels. These labels enable SimGen to learn and control the layouts and behaviors of generated scenarios. OpenDV is collected mainly for predictive video generation, so there is no need to collect these labels. Automated pipeline: The collection of DIVA is fully automated, facilitating data scaling while ensuring quality. In contrast, OpenDV requires manual quality checks, which is less efficient.

3. In Supp Tab. 9, as the CFG scale increases from 5 to 14, pixel diversity rises from 21.3 to 28.2 before reaching saturation. The rise is due to changes in image contrast and sharpness; as the scale hits a threshold, the image distinctions diminish.

[1] CAT: Closed-loop Adversarial Training for Safe End-to-End Driving.

[2] DrivingDiffusion: Layout-Guided Multi-view Driving Scene Video Generation with Latent Diffusion Model.

[3] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion.

[4] Wovogen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generation.

[5] LimSim: A Long-term Interactive Multi-scenario Traffic Simulator.

W1: Content presentation suggestions for this paper.

Thank you for the suggestion. We follow your advice to reorganize the Related Work section and move the content in Appendix D.3 to the main paper.

Q1: The solution of the statistical shortcut.

Integrating input conditions through an adapter is a common practice in training image generation models with multi-modal conditions [1], which serves as our foundation. Randomly dropping conditions within the adapter helps the model balance various condition modalities, preventing overfitting to specific ones. The adapter merges information from different modalities to reduce uncertainty caused by randomly dropping. Furthermore, we mitigate conflicts between multi-modal conditions by masking out simulator background information, preventing the model from learning associations between simulator backgrounds and nuScenes-style images.

Q2: The usage of the safety-critical data originating from Waymo.

The collection of safety-critical data is a part of our contributions. The DIVA-sim dataset includes simulated driving videos and annotations of hazardous behaviors initialized from Waymo, ensuring diverse scene generation and advanced simulation-to-reality research. Annotations such as object attributes, lane lines, trajectories, etc., can be derived from the simulator. In addition to showcasing how SimGen can generate safety-critical scenarios qualitatively based on DIVA-Sim, the labeled driving scenes can also be utilized in map segmentation and object detection tasks for data augmentation. Furthermore, agents learned from safety-critical data have the potential to achieve superior driving safety [2]. We leave these parts as our future work.

Q3: Future work to extend SimGen to multi-view image generation.

Thank you for the suggestion. Regarding extending SimGen to multi-view image generation, we could consider leveraging 360-degree images from Google Street View [3], which have recently been used to train diffusion models for generating realistic multi-view street scene images. Additionally, we might incorporate cross-view attention mechanisms to ensure consistency across different views, similar to the approaches demonstrated in DrivingDiffusion [4] and Panacea [5].

[1] UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild.

[2] CAT: Closed-loop Adversarial Training for Safe End-to-End Driving.

[3] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion.

[4] DrivingDiffusion: Layout-Guided Multi-view Driving Scene Video Generation with Latent Diffusion Model.

[5] Panacea: Panoramic and Controllable Video Generation for Autonomous Driving.

W1: Despite achieving superior results in single image generation, the 3D consistency has not been evaluated.

We agree with the reviewer that achieving 3D consistency will be an important future direction. However, it is not the main focus of this submission. We have preliminary attempts at video generation and design the temporal consistency in Sec. 4.3 and Appendix D.2. Fig. 5, 12, and 16 provide visualization examples, and Tab. 11 indicates that SimGen performs comparably to other driving video generation models. Nevertheless, ensuring and evaluating 3D geometry consistency in generated driving scenes poses a challenge, with all existing work (e.g., Panacea, DrivingDiffusion, GenAD) struggling to achieve this. Even powerful video generation models such as Sora and SVD have similar difficulties.

Q1: Without videos demonstrating 3D consistency, the proposed method is less practical for autonomous driving applications.

Although SimGen cannot guarantee 3D consistency, the capability to generate diverse driving images and videos with annotations is crucial to many applications. Tab. 5 demonstrates how SimGen can generate diverse driving scenarios for data augmentation in map segmentation and 3D object detection models. Fig. 5 and 12 showcase its applications in generating hazardous driving scenarios and closed-loop evaluation. Other applications like lane detection and BEV detection can also benefit from synthetic image-annotation pairs.

Q2: I am confused about the conditions used in SimGen. Figure 3 indicates that DIVA synthetic images are not used in the generation process, whereas Figure 4 shows them being used as conditions. Could you clarify where the synthetic images are actually utilized?

As illustrated in Fig. 3 and Tab. 2, a scene record is fed into the simulator to generate DIVA synthetic images, which are further grouped into SimCond (simulated depth and segmentation) and ExtraCond (rendered RGB images, instance maps, and top-down views). SimCond is transformed into RealCond by the CondDiff module and then, along with ExtraCond and text, is fed into the ImgDiff module to generate real images. More descriptions on this part will be included in the revision.

Limitations.

We have discussed the social impacts in Appendix A.

W1: Unclear usage of DIVA-Sim dataset in training.

As stated in L158, Tab.2, and Appendix C.3, DIVE-Sim is only used in the training of ImgDiff. ImgDIff takes RealCond from DIVA and nuScenes and ExtraCond from DIVA-Sim as inputs, without including SimCond, and outputs real images. Consequently, there are no optimization issues. We will add more descriptions for Tab. 2 in the revision.

W2: Lack of scalability analysis for data augmentation experiments.

In Supp Tab. 8, we assess the various proportions of synthesized data as an augmentation strategy to boost the perception model (BEVFusion). The results indicate that performance improves with an increase in synthesized data, from 47.7 to 51.9 on $\text{AP}_{3D}$. This validates SimGen’s data scaling capability for data augmentation in driving applications.

W3: Lack of high-level system workflow for readability and better understanding.

We will add a figure to illustrate the system workflow in the revision.

Thank you for your feedback, and I will keep my score as strong accept considering the quality and potential impact of this work.

W1: DIVA Dataset quality.

Supp Tab. 1 includes the evaluation of the annotation quality. To evaluate the VLM, we utilize the DIVA-Real and nuScenes datasets and employ the widely used ROUGE-L metric (84.4 and 85.2) [1] to assess the similarity between the annotated data and pseudo-labels. The pseudo-labels are derived from GPT4V and manually checked to ensure reasonableness. For the evaluation of the depth estimator and segmentator on the nuScenes dataset, we leverage the metrics of absolute relative error (0.118) and mIoU (82.4), respectively. The results indicate that our annotated data exhibits a sound data quality level. Besides, we follow your advice to include a column indicating whether the annotations are human-labeled or pseudo-labeled in the revision.

W2: SimGen performance.

For a fair comparison, we have set up the SimGen-nuSc within the nuScenes dataset in Tab. 3. SimGen-nuSc surpasses DrivingDiffusion in terms of image quality and diversity by -0.3 and +0.4 respectively under the same training data. If other methods were trained on DIVA, there might be a performance boost similar to that from SimGen-nuSc to SimGen (+6.1). However, methods like BEVGen and MagicDrive require explicit labels like 3D boxes and lane lines for training, preventing them from utilizing the web data. Moreover, training SimGen on DIVA only and testing on nuScenes would be unfair as it raises domain generalization issues in those works.

W3: The utility of the generated scenarios.

The DIVA dataset is an essential part of our contribution. Aside from the improvements resulting from the framework (demonstrated by SimGen-nuSc), Tab. 5 also shows that SimGen trained on the DIVA dataset brings higher improvements and outperforms baselines by a large margin. In Supp Tab. 2-4, we report complete data for SimGen's performance in 3D object detection tasks. Tab. 3 witnesses the benefits of SimGen on all categories in nuScenes, including motorcycle (+2.9), cone (+1.6), and barrier (+1.7). The results indicate that SimGen can generate a diverse range of objects (10 categories) covering common driving scenarios, thereby confirming its utility in practical applications.

W4: Dataset diversity.

In Supp Tab. 5-6, we provide a comparison between DIVA and nuScenes. nuScenes data is only from Boston and Singapore, while DIVA includes data from South America (8.5%), Europe (16.9%), Asia (14.6%), and Africa (3.1%). In nuScenes, 88.4% of the data is during the daytime, and 80.5% is on normal days. In contrast, DIVA collects data at dawn (16.3%) and dusk (10.1%), covering cloudy (28.6%), snowy (10.2%), and many other weathers (3.1%). Also, 75.3% of nuScenes data is keeping forward and turning, while DIVA includes lane changing (28.1%), intersection passing (6.6%), and U-turns (1.2%), reflecting more complex traffic layouts. The results show that DIVA has a more diverse and balanced data distribution than nuScenes. In addition, Tab. 7 reports statistics on corner cases, including 9 kinds of behaviors of other vehicles that lead to safety-critical driving scenarios. The statistics report that the proportions of crash back and cutting in are 41.2% and 19.0%, respectively.

W5: Safety-critical scenarios.

A safety-critical scenario is a situation where one or more vehicles collide with the ego vehicle, which is rare to collect in real-world datasets like Waymo. We utilize CAT [2] to generate risky behaviors from logged scenarios to ensure reality and feasibility, which uses a data-driven motion prediction model that predicts several modes of possible trajectories of each traffic vehicle. Please refer back to [2] for a detailed description of safety-critical scenarios.

Q1: Detailed discussion on CondDiff.

We have tried the conventional Sim2Real method which focuses on photorealism enhancement [3]. As discussed in Appendix A and C.1, the pixel-to-pixel transformation only brings limited appearance diversity. It is like applying an image filter or style transfer to images rather than generating new content, with the outputs strictly constrained by input conditions. If input conditions derived from a simulator lack background information (e.g., trees, buildings, etc.), the output images may completely ignore generating any background content. Moreover, such a model struggles to alter visual appearances via text prompts in generating novel scenarios.

Q2: Diffusion steps for Sim2Real transformation.

Appendix C.2 shows the diffusion steps ($t_s$) for Sim2Real transformation is 0.5.

Q3: Comparision of SimGen to GenAD.

SimGen aims for controllable image generation in driving scenes, so it is not specifically designed for video generation. We only have a preliminary attempt to achieve temporal consistency. GenAD incorporates spatiotemporal optimizations and is designed for video generation, but it has much less flexibility and controllability in content generation compared to SimGen.

Q4: Explanations on Sim2Real transformation.

Fig. 11 visualizes Sim2Real transformation under varying text prompts rather than a failure case. The real conditions generated by the model may be related to the training data distribution. Despite "street at night" not mentioning trees, the model still envisions the presence of far trees in the scene and generates a corresponding semantic map. Due to the scarcity of pedestrian pixels in the simulated condition, the model may overlook this condition when the text prompt is "Mountains" where there are supposed to be no pedestrians, resulting in their disappearance.

Minor issues.

We will reorganize the sentences and correct the typos in the revision.

[1] ROUGE: A Package for Automatic Evaluation of Summaries.

[2] CAT: Closed-loop Adversarial Training for Safe End-to-End Driving.

[3] Enhancing Photorealism Enhancement.