Glad: A Streaming Scene Generator for Autonomous Driving

We thank reviewer MnsB for the valuable time and constructive feedback.

---

W1: On the lack of supporting evidence.

A1: Thank you for your suggestions regarding qualitative analysis. We have provided demonstrations for various scenarios: lane changes can be seen in the last column of Fig. 6 in the manuscript (achieved by manually changing ego position), sharp turns are shown in Fig. 9 in the Appendix (focusing on corner cases), and examples of varying traffic densities are available in the supplementary material.

---

W2: About the ablation with temporal transformer layer.

A2: Thank you for your suggestion. Following the implementation of Panacea, we have incorporated a temporal layer into our framework for an ablation study. As shown in the table below, using the temporal layer for propagating temporal information yielded an FID score of 17.91 and an FVD score of 183. While the FID is slightly worse than that of latent variable propagation (LVP), the FVD is marginally better. This suggests that the temporal layer more effectively models temporal propagation but has limitations in spatial modeling of single frames. Overall, LVP achieves a better balance in both spatial modeling and temporal propagation. Moreover, the approach using the temporal layer, which requires modeling multiple frames simultaneously, is considerably more expensive in terms of memory consumption and training time. Therefore, the LVP method is more efficient. We have updated these ablation results in the Appendix.

---

W3: About the performance gain by generating new trajectories.

A3: We have taken your suggestion seriously. However, there are some problems when we consider conducting the experiment. First, there is no open-source toolkit available to edit or generate new trajectories for the nuScenes dataset. Generating additional data require the manual annotation of a large number of trajectories, which would be extremely costly. Second, simply generating new trajectories may pollute the existing data, as the new trajectories may not follow the physical world—e.g., the trajectory might go beyond the road or make impossible turns. A simulator environment for trajectory generation that complies with physics is urgently needed for current video generation models for autonomous driving. There are some relevant works, such as HouseLLM[1], focused on the generation of indoor layouts. We eagerly anticipate the extension of similar work to the domain of autonomous driving in future research.

---

W4: About the training data.

A4: We apologize for the lack of detail about the training data, which we have now included in the manuscript. Our two-stage training process utilizes the same data and processing rules as Panacea. We only utilized key frames (2HZ) that have detailed 3D bounding box annotations. In the first stage, we conduct resizing and cropping operations solely on single-frame images. Considering the native resolution of $1600\times900$, we randomly resize the images to a proportion of 0.3 to 0.36, and then crop a $512\times256$ region. In the second stage, we further organized the nuScenes dataset into a video stream format to facilitate the fine-tuning of streaming video generation.

---

Q1: About the simulation capabilities.

A5: Thank you for your concern. To demonstrate the simulation capability, we use the end-to-end autonomous driving model VAD[2] as an example, retraining it at $512\times 256$ resolution. We perform inference using this retrained model on the validation set generated by Glad and include the recorded process in various scenes as a video in our supplementary material. It's worth noting that simulators based on video generation models are still in their early stages. While generative models excel at creating new scenes compared to fully NeRF-based simulator methods, NeRF-based models offer superior reconstruction of existing scenes with lower computational costs. Recent work has explored combining these approaches, using generative models to create unseen or low-quality scenes that NeRF-based models can then reconstruct more effectively. We believe generative models show significant promise for developing non-rule-based simulators.

---

We appreciate your thoughtful review and we hope to address your concerns. Please let us know if you'd like any further discussion.

---

[1]Zong, Ziyang, Zhaohuan Zhan, and Guang Tan. "HouseLLM: LLM-Assisted Two-Phase Text-to-Floorplan Generation." arXiv preprint arXiv:2411.12279 (2024).

[2]Jiang, Bo, et al. "Vad: Vectorized scene representation for efficient autonomous driving." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Dear Reviewer,

Thank you for your valuable suggestions, especially for introducing ScenarioNet to us. We have taken a close look at ScenarioNet, which collects trajectories from several public datasets and reconstructs them within a simulator. This approach has greatly inspired us, prompting us to attempt the generation of augmented data using trajectories from other datasets as new trajectories. Specifically:

We selected trajectories from the Argoverse 2[1] (AV2) dataset to serve as our new trajectories due to its capability to encompass a full 360-degree view. Subsequently, we employed Glad to infer over the trajectories of the AV2 dataset to obtain new generated scenes, which were then integrated with the nuScenes dataset to enhance performance. We chose Far3D[2] as our perception evaluation method, as it offers training support for both the AV2 and nuScenes datasets. The experimental results are as follows:

As shown in the table, we observed that utilizing new trajectories for data generation leads to performance improvements (about 1% NDS). Notably, Far3D employs a stronger backbone, VoV-99, which has been pre-trained on the DD3D 15M dataset. Moreover, the camera poses and scene distributions in AV2 differ from those in nuScenes. In the future, if specialized tools for trajectory generation become available, creating in-domain trajectories could potentially yield even better results.

We hope to address your concerns. Please let us know if you'd like any further discussion.

Best regards.

---

[1] Wilson, Benjamin, et al. "Argoverse 2: Next generation datasets for self-driving perception and forecasting." arXiv preprint arXiv:2301.00493 (2023).

[2] Jiang, Xiaohui, et al. "Far3d: Expanding the horizon for surround-view 3d object detection." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.

Thank you for taking the time to review our paper again and providing your valuable feedback. We appreciate your insights and the opportunity to address your concerns. In response to your comments, we would like to offer the following clarifications:

---

Summary: On the limited novelty and suboptimal results.

A1: Thank you for serving as this paper's reviewer again. Regarding the limited novelty, even though our Glad is simple, it proposes a new paradigm. Compared to current methods that depend on temporal attention layers, Glad shows that the importance of the noise prior for video consistency has been ignored. By the way, our latent variable propagation is easy to follow. We think the contribution of our Glad is to tell the research community to pay more attention to noise priors. As for the suboptimal results, the mainstream benchmark of current video generation is FVD. The FVD lacks a uniform platform or requirements; in our practical experience, there are many tricks to reduce the FVD score, such as adjusting the resolution of the video, the length of the video, and the FPS of videos. Finally, we want to highlight that Glad is an initial attempt to steer research towards paying more attention to the noise prior.

---

W1: About the simple training tricks for our proposed modules.

A2: Thank you for your feedback. Although the streaming data sampler is concise and intuitive, it is indeed a new training paradigm. It is designed for our frame-by-frame video generation. For ADriver-I, MagicDrive, and Drive-WM, they are not truly frame-by-frame. For example, ADriver-I inputs 4 frames to generate the future 4 frames, MagicDrive generates 7-frame clips at once, and Drive-WM also adopts 3D-UNet for multi-frame generation at once. Not only that, the sampler is also efficient for alleviating error accumulation, and as shown in Tab.5(b), this sampler can reduce the GPU memory requirement. For the previous works that also adopt first/former frame to preserve the context, Text2Video-Zero just backpropagates the Gaussian noise to maintain the inner-frame consistency, which is not for preserving the context, and it also integrates Cross-Frame attention. ControlVideo appears to just use original Gaussian noise. Video ControlNet explores optimizing the initial Gaussian noise to reduce drastic changes in texture, which is not designed for frame-by-frame video generation.

---

W2: On the comparison with previous methods.

A3: We have expanded comparisons with both earlier and contemporary works, such as Vista and DriveWM. For DriveDreamer-2, the notably low FVD performance was also of interest to us. Delving into the experimental details, we found that its training and testing settings differ from the mainstream methods. In contrast, our approach has maintained the same training and testing settings as Panacea. The specific differences are as follows:

DriveDreamer-2 has annotated the sweep data from the nuScenes dataset to create a 12Hz training and testing dataset, while other methods like Panacea and MagicDrive have only used 2Hz keyframe data. This means DriveDreamer-2 has utilized 6 $\times$ more training data than these methods. For FVD testing, a sequence of 16 frames is typically required. Since DriveDreamer-2 tests at 12Hz, it only needs to generate a 1.33 seconds video clip. In comparison, Glad and Panacea, must maintain a video length of 8 seconds. Moreover, DriveDreamer-2 uses real-scene images as the initial frame. With a video length of just 1.33 seconds, there is minimal scene movement, making it easier to maintain consistency with the initial frame. These factors may contribute to the low FVD score of DriveDreamer-2.

---

We appreciate your thoughtful review and we hope to address your concerns. Please let us know if you'd like any further discussion.

---

We thank reviewer wGv7 for the valuable time and constructive feedback.

---

W1: On the motivation concerns.

A1: We are sorry about the lack of clarity in our motivations. Regarding the restricted video length claimed in our paper, we do not intend to question the ability of current methods for long video generation. Instead, we focus on the efficiency and potential to effectively conduct long video generation between two different paradigms. More specifically, Drive-WM generates long videos by first generating frames 1 to N, and then frames 2 to N+1. This means generating N frames at once but only keeping one frame, which is inefficient. Panacea also adopts the same sliding window strategy. Even though some other methods support generating long videos by producing frames 1 to N, then N+1 to 2N, and so on, our Glad approach offers more flexibility and does not determine the hyperparameters (such as overlap length, condition length, etc.) when generating long video. It is particularly advantageous when only a few frames need to be generated or when input conditions suddenly change. As proved by reviewer XvyP, our work was finished in May 2024. The work Delphi mentioned was posted on arXiv one month later, which should be considered as contemporaneous work with ours.

Regarding the unseen scenes claimed in our paper, conventional long video generation methods require complete future trajectory information in advance, which is impractical in real-world scenarios. When an ego vehicle deviates from its planned trajectory during driving, the new trajectory becomes "unseen" to the model. In other words, these scenes are considered "unseen" because they emerge in future timesteps rather than a predefined offline new path. While existing long video generation methods can handle offline new trajectories, they lack the flexibility of our frame-by-frame generation paradigm in adapting to temporal uncertainties. Our approach excels at handling dynamic trajectory changes since it doesn't require pre-planning the entire sequence and can adjust to new trajectory inputs immediately.

---

W2: On the technical novelty concerns.

A2: We are glad to discuss with you the difference between other methods and our Glad. We think the frame-by-frame video generation ability comes from the paradigm difference. Let's explain the reason. First, for generating video frame-by-frame, both Drive-WM and Panacea have a limitation. For video generation, these models must output at least 2 frames and keep the last one. In contrast, Glad only needs to generate 1 frame everytime. On the other hand, temporal-layer based video generation models (such as Panacea) have specific training requirements. In the training stage, they must learn how to model long temporal sequences. This means that the length of training video clips is important and heavy computation is required. The Streaming Data Sampler is concise and intuitive, but it is a new training paradigm. It is designed for frame-by-frame video generation. Not only that, the sampler is also efficient for alleviating error accumulation. As shown in Table 5(b), this sampler can reduce the GPU memory requirement.

---

W3: On the empirical validation concerns.

A3: We kindly remind that the object tracking results evaluated by StreamPETR have been shown in Table 2. Regarding the validation of long sequence videos, the current FVD only supports a maximum of 16 frames per video, so we only show the results for 16-frame videos. To address those concerns, we are pleased to provide 40-frame videos from the nuScenes validation dataset in the supplementary material. For the word "online" in our paper, it does not mean on-the-fly rendering; it represents frame-by-frame video generation. We will clarify this concept in the revised paper. The generation speed for diffusion-based video generation models is still a main bottleneck. Recent works, such as SANA[1], propose an efficient high-resolution image synthesis method. Considering that the acceleration of diffusion process is not the main focus of this paper, but due to the properties of our Glad, we can directly convert these efficient image synthesis methods for video generation.

---

W4: On the presentation concerns.

A4: Thanks for your remind, we have correct the mistake in the manuscript.

---

We appreciate your thoughtful review and we hope to address your concerns. Please let us know if you'd like any further discussion.

---

[1]Xie, Enze, et al. "SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers." arXiv preprint arXiv:2410.10629 (2024).

We thank reviewer 8mSK for the valuable time and constructive feedback.

---

W1: Confusing writing.

A1: We apologize for the confusing writing. Your summary is precise. We have rewritten method Section to highlight the main idea in this work and updated the manuscript. We would be pleased if you could read our paper again and propose some suggestions.

---

W2: Missing key reference.

A2: We greatly appreciate your recommendation of TRIP[1]. It is an important exploration of noise prior for video generation, and we have refered to it in the manuscript.

Although both TRIP and Glad employ the term "noise prior," there are significant differences between the two methods. Glad directly uses the latent features from the previous frame as the noise prior, effectively replacing the Gaussian noise in the diffusion forward process. In TRIP, the noise prior is more accurately described as "reference noise." By obtaining this reference noise, the model only needs to predict the deviation from it in the backward process. The prior approach in TRIP is similar to the "anchor point prior" used in 2D detection.

Furthermore, TRIP predicts the noise prior for the entire video clip from the initial frame, which can be inaccurate over large time spans. In contrast, Glad passes the noise prior frame by frame, which is more reasonable.

---

W3: Serious issue in paper organization.

A3: Thank you for your suggestion. Section 3 is somewhat redundant and, as you previously mentioned, it has resulted in a lack of focus in the current structure. Therefore, we move Section 3 to the Appendix.

---

W4: Poor literature review.

A4: Thank you for your reminders. We have added DriveWM's results to Table 2 in the manuscript.

MagicDrive is a single-frame method, and its perception performance is assessed using the single-frame model BEVFusion. Therefore, it is somewhat unfair to compare the detection performance to Glad in Table 2, given that it is considerably lower. By the way, we have updated the FVD result of MagicDrive in Table 1.

As reviewer XvyP has noted, this paper was drafted in May, making Vista a contemporary work of ours. The settings for FID and FVD are different from ours, as Vista crops the images to $224 \times 224$ while we use $512 \times 256$. However, considering the relevance of Vista to our research, we have also included discussion of it within the manuscript.

---

Q1: Balancing consistency and motion in generated video.

A5: We are glad to discuss the balance between frame-to-frame consistency and quick motion with you. In the field of video generation, consistency is vitally important. Current evaluation metrics such as FVD also mainly focus on consistency. On the contrary, quick motion is important for practical application value. Quick motion issue primarily appear in the foreground objects of generated videos. During rapid movement, objects may appear to jump between frames. We think the object-centric evaluation metrics like MOTA might help measure the consistency of these objects across frames.

To maintain consistency in this work, for simplicity, we only make an effort through the input condition. Specifically, we keep the unique color of each object's bounding box consistent throughout a video. Despite its simplicity, our method achieves comparable AMOTA performance to approaches that directly inject subject IDs, such as SubjectDrive[2]. Quick motion remains a significant challenge in video generation, presenting substantial opportunities for future improvements.

---

We appreciate your thoughtful review and we hope to address your concerns. Please let us know if you'd like any further discussion.

---

[1]Zhang, Zhongwei, et al. "TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models." CVPR. 2024.

[2]Huang, Binyuan, et al. "Subjectdrive: Scaling generative data in autonomous driving via subject control." arXiv preprint arXiv:2403.19438 (2024).

We thank reviewer B154 for the valuable time and constructive feedback.

---

W1: The error accumulation issue of Glad.

A1: Error accumulation is a significant challenge in video generation, particularly for long video sequences. We also faced this issue at the outset, but ultimately resolved it by constructing an appropriate noise prior. The analysis and resolution process are as follows:

As shown in Fig. 8(a) in the Appendix, in our initial approach, we constructed the current noise prior by extracting the latent features from the real image of previous frame. Under this strategy, the model was trained solely on the transition between two consecutive frames. During testing, errors accumulated gradually with each additional frame, ultimately leading to a collapse in image quality.

To address this issue, as depicted in Fig. 8(b), we introduced noise into the latent features from previous real frame. This strategy was designed to encourage the model to reconstruct the current frame from features that included errors. However, the noise we introduced not align with the actual patterns of errors observed during testing. Consequently, the model not achieve the desired improvement.

Building on these attempts, we have adopted the current construction strategy, as shown in Fig. 8(c) in Appendix. Instead of directly using the real features from the previous frame to construct the noise prior, we employ the denoised previous features. On one hand, the denoised features contain errors compared to the features extracted from the real image. On the other hand, the model is still required to accurately reconstruct the current image during training. This interplay has endowed the model with the capability to recover the current frame from erroneous inputs. By extending the training frames, the error propagation in training more closely mirrors that of the testing phase. Ultimately, it enhances the model's ability to correct accumulated errors.

In the Table below, compared to other strategies, the noise prior $\mathbf{z}^{n-1}_{denoised}$ achieves the best performance in all evaluation metrics. Therefore, in our Glad approach, we utilize the denoised latent feature.

---

W2: About the training data. It is unclear to me how many training data used for Glad and other methods in Table 1, such as Panacea and Oracle.

A2: For the training of our generative model, Glad used the same training dataset as Panacea. The term "Oracle" is not a generative algorithm. We apologize for any misunderstanding and will provide an explanation for it.

In Table 2, we evaluate the authenticity of our generated data by employing a pre-trained StreamPETR on the validation set. The "Oracle" row represents the results of StreamPETR when applied to the real nuScenes validation set. The other rows indicate the performance of StreamPETR on synthetic validation set. The closer the perceptual results to the "Oracle," the higher quality of the generated data.

---

W3: Detail of the detection performance. It would be interesting if the authors could show the detailed detection performance of 10 classes on the nuScenes.

A3: Certainly, the detailed detection performance of the 10 classes can reflect the generation quality of each category. Since Panacea does not provide results broken down by category, we compare our results with the Oracle results from StreamPETR. It can be observed that common categories such as Car and Traffic cone have higher generation quality compared to rare categories like Construction Vehicles and Trailer. The "CV" and "TC" are the abbreviation of the Construction Vehicle and Traffic Cone.

---

Q1: Lack of performance improvement when using more video frames.

A4: This is an insightful question, and we also noticed the performance saturation at 8 frames in the ablation study (see their Table 5) of StreamPETR. We have explored the reasons for this phenomenon and attribute our preliminary conclusions to the inherent properties of the nuScenes dataset.

As we know, current controllable video generation models are mainly object-centric, focusing on objects with annotations like 3D bounding boxes. We analyzed the object consistency in nuScenes, and the results are shown in the Fig.7 in Appendix. As we can see, almost 96.2% of the objects are persist for less than 8 frames, while only 3.5% are persist between 8 frames and 16 frames. These results mean that in a generated video of 16 frames, the objects in the last frames are entirely different from those in the first frame, which leads to performance degradation.

---

Q2: Performance on generated data.

A5: We have examined our previous experiments and present the results in the Table below. When trained solely on synthetic data, Glad outperformed Panacea by 4.6% mAP and 4.7% NDS.

---

Q3: Generation in corner cases.

A6: To generate corner cases, we utilized the CODA[1] dataset, which is specifically designed for real-world road corner cases. CODA encompasses three major autonomous driving datasets, including 134 scenes from nuScenes. We focused on this subset of 134 nuScenes scenes. After filtering to identify which scenes belonged to the validation set, we retained 30 valid scenes. These 30 scenes serve as our corner case examples, and we present selected generation results in Fig.9 in Appendix. We showcase five corner cases that demonstrate our generation capabilities: a car making a sudden turn in the opposite lane (case 1), traffic cones along the roadside (cases 2 and 3), unusual roadside obstacles (case 4), and unconventional road signs (case 5).

---

We appreciate your thoughtful review and we hope to address your concerns. Please let us know if you'd like any further discussion.

[1] Li, Kaican, et al. "Coda: A real-world road corner case dataset for object detection in autonomous driving." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.