PerLDiff: Controllable Street View Synthesis Using Perspective-Layout Diffusion Model

Q1: I suggest not using the word BEV annotations. Both 3D box annotations and map annotations exist within 3D space rather than BEV space.

A4: Thank you for your helpful comments. We have revised the term "BEV annotations" to "3D annotations" in our revision.

Q2: Section 3.2 is hard to read. I suggest to include a dedicated diagram to effectively illustrate the PerL-based cross-attention mechanism. The bottom of Figure 2 need to be refine.

A5: To facilitate a better understanding of PerLDiff, we have provided a detailed explanation of PerL-based cross-attention (Object) in Figure 8 of Appendix C.4. Compared to other methods, we utilize the Perl masking map as a prior, allowing each object condition to precisely control the corresponding pixel features. This results in more accurate positioning and orientation of objects in the generated images.

Q3: Why is the attention mask not multiplied into A_s in Equation 5? Adding a binary mask into the feature seems unconventional.

A6: We conducted experiments based on Equation 1, as shown in Table 7 (Mu*Mask). The results indicate that all metrics are lower than those of PerLDiff. We believe that the performance decline is due to the degradation of the cross-attention maps. Specifically, the cross-attention maps are highly sensitive to changes in the Perl masking map. As illustrated in Figure 9 of Appendix C.5, the attention maps learned by the Mu*Mask method overly influence the background regions, ultimately leading to a decline in image quality.

$$\mathcal{A}_s = *softmax*(\lambda_s \cdot \mathcal{M}_s \times \mathbf{Z}\mathbf{H}_m^{T}/\sqrt{d}), \mathcal{A}_b = *softmax*(\lambda_b \cdot \mathcal{M}_b \times \mathbf{Z_s}\mathbf{H}_b^{T}/\sqrt{d})$$

Q4: Given that using perspective-Layout, why not use controlnet framework for more straightforward interaction?

A7: We adopted the ControlNet architecture, training view cross-attention within Stable Diffusion. As illustrated in Table 8 (ControlNet + Mask), the performance of models based on ControlNet is inferior to that of PerLDiff. Additionally, we found that PerLDiff employs a network architecture similar to GLIGEN, allowing for faster convergence on smaller datasets like NuScenes, compared to the ControlNet architecture, as shown in Figure 10 of Appendix C.6.

Q5: Can this method be applied to occupancy control?

We hypothesize that your concern pertains to the efficacy of the data generated by PerLDiff for use in occupancy-based models. To address this, we tested the generated validation dataset using TPVFormer (CVPR 2023). As shown in Table 9, our results indicate a smaller discrepancy from the real data compared to MagicDrive and BEVControl, suggesting that the data generated by PerLDiff can also enhance occupancy-based segmentation networks.

Dear Reviewer 14qK：

Thank you for your review and comments. We provide the following discussions and explanations regarding your concerns.

W1: The reason for selecting ConvNext as the feature extraction network requires further exploration, particularly given the training objective of image classification on the ImageNet dataset. This raises questions about the appropriateness of utilizing ConvNext in this context. Additionally, it is essential to conduct an ablation study comparing ConvNext with other image backbone architectures, such as the CLIP image encoder, to provide a comprehensive understanding of their relative performance.

A1: We initially selected ConvNeXt due to its ability to effectively extract image features while maintaining a relatively low parameter count (27.82M). Furthermore, it has demonstrated efficacy in downstream segmentation tasks, particularly in extracting edge information from images [1].

Moreover, we conducted an ablation study on the road map using the CLIP image encoder and the autoencoder from Stable Diffusion. The results indicate that the performance of the feature extraction network has negligible impact on PerLDiff ($\pm0.02$ NDS between CLIP encoder and ConvNext encoder).

[1]. Liu, Zhuang et. al., A convnet for the 2020s. In CVPR2022.

W2: The formulation of this work is not novel to me; rather, it represents a combination of several existing methodologies. The framework closely resembles that of Panacea and MagicDrive, with the key modification being the introduction of an attention mask.

A2: Our research is not merely a combination or replication of existing methods. While approaches like BEV Control, MagicDrive, and Panacea use BEV Layout as a condition and cross-attention for data generation, a fundamental question emerges: do these methods truly achieve conditional control? As Figure 1 in the paper shows, these methods sometimes fail in this regard. We propose that this failure arises because simple cross-attention does not guarantee a connection between control conditions and the generated images. As evidenced in Figure 3, simple cross-attention can produce disordered and blurry outputs, resulting in poor control performance.

Based on this observation, we meticulously developed a PerL-based cross-attention mechanism that utilizes a PerL masking map—comprising road and box components—as geometric priors to facilitate high-precision, object-level image generation. This mechanism extracts conditional features from 3D annotations and explicitly renders them into perspective layout masking maps, serving as geometric priors.

To validate the effectiveness of our approach, we conducted comprehensive experiments on the KITTI and NuScenes datasets, which demonstrated significant performance improvements over existing methods. Additionally, we performed data augmentation experiments to illustrate that our method can generate synthetic data for autonomous driving, offering a potential solution to address corner cases in this field.

W3: Lack of quantitative comparison with Panacea [a], which similarly uses the perspective layout.

A3: Since our PerLDiff focuses on ensuring the controllability of objects in the generated images, we conducted comparisons with image generation models for validation (Penacea is a video generation method). Table 6 presents a comparison of various metrics, illustrating that our PerLDiff model can surpass the performance of Panacea even in the absence of temporal modeling.

Dear Reviewer 14qK,

We would like to gently remind you that today is the last day for reviewers to post messages to the authors, and we have not yet received any additional feedback from you since November 25. If our responses and revisions can help address your concerns, we kindly request that you consider updating your rating.

Your insights are invaluable to us, and we would greatly appreciate it if you could share your thoughts before the end of the day.

Thank you for your time and consideration. We look forward to your response.

Best regards,

The Authors of Paper 52

Dear Reviewer gVD5:

Thank you for your acknowledgment and constructive comments. We provide discussions and explanations about your concerns as follows.

W1: The first question is whether the proposed method can ensure temporal consistency for the generated images. The paper doesn't discuss this problem and displays the image in different timestamps. However, the detection and segmentation models used, particularly StreamPETR, require temporal inputs. If the synthesized images do not achieve good temporal consistency, how can they improve the performance of temporal-based methods?

A1: We acknowledge that PerLDiff currently does not incorporate any temporal blocks for video generation. It is worth noting that, based on our experimental results, the key for temporal-based detection models is the positions and categories of objects in each frame; detailed information about objects, such as color and brand, is not crucial. As illustrated in Figure 16 of Appendix D, when provided with continuous frame inputs, the generated images ensure that the positions and categories of objects, as well as the road map, are accurately consistent with the specified conditions between adjacent frames.

W2: In Section 3, it appears that all input data comes from 2D space (with boxes and roadmaps projected onto images and a general scene description). Another question is whether the proposed method can effectively disentangle the camera's intrinsic and extrinsic parameters compared to BEVGen and BEVControl.

A2: We did not fully understand your comments and would appreciate your feedback on our response below. All annotations are projected from 3D space using both intrinsic and extrinsic parameters. BEVGen, BEVControl, and our PerLDiff all incorporate these parameters to process object coordinates due to the task setup.

W3: Cross-view attention: Could the authors explain the differences between view cross-attention and text cross-attention in PerlDiff compared to BEVControl and MagicDrive?

A3: View cross-attention and text cross-attention are common settings. For view cross-attention, as described in Lines 301-304, MagicDrive, BEVControl, DrivingDiffusion, Panacea, and PerLDiff employ the same mechanism. The key idea is to obtain features from adjacent views to ensure consistency. For text cross-attention, BEVControl and MagicDrive freeze the weights of Stable Diffusion, while PerLDiff trains the text cross-attention block to enhance scene control.

W4: Object rotation controllability: Could the authors clarify how the proposed method achieves object rotation controllability? In Section 3, it seems that the box information is represented as eight corners in the image space, where the rotation information is not explicitly presented.

A4: PerLDiff controls the rotation of objects through the input coordinates, as illustrated in the figure below. As you pointed out, we utilize the eight corner points arranged in sequential order, with the first four points representing the front of the vehicle. This effectively encapsulates the angle information of the objects. Additionally, we use the object's Perl masking map as a geometric prior to ensure that the generated object angles align correctly.

                                                                        up z
                                                      front x           ^
                                                           /            |
                                                          /             |
                                         	         <1>-------------<2>
                                                        /|            / |
                                                       / |           /  |
                                       		     <5>------------<6>   <3>
                                                      |  /          |  /
                                                      | / origin    | /
                                      left y<-------- <8> ----------<7>
                                          

Q1: Could the authors provide the details of the experimental setting on the nuScenes dataset? One point of confusion is that, compared to the training set, the validation set is not very large. Why does adding it to the training data lead to such a significant improvement in model performance?

A5: PerLDiff is trained using the training set from nuScenes, which consists of a total of 28,130 samples across six views. We set the batch size to 16 (2 per GPU) and the learning rate to 5e-5, training for a total of 60,000 iterations. In Table 3, for the training of all BEVFormer models, we adopted the official configuration with a batch size of 1 and 24 epochs, only adjusting the resolution to 256x384. For training all StreamPETR models, we utilized the official configuration, employing a batch size of 2 and running for 24 epochs. Additional training details are provided in Appendix B.

The improvement observed in the test set performance after incorporating the validation set into the training process (comprising 28,130 training frames and 6,019 validation frames) can be attributed to the relatively limited size of the original training dataset. Including the validation set, which increases the data by 21.4%, is a common practice when dealing with small datasets. For further reference, a screenshot of relevant GitHub issues is provided in Figure 18 of Appendix D.

Q2: What is the meaning of * in Table 3?

A6: In Table 3, it should be “Syn. val*” represents the synthetic validation set generated by BEVControl. We have corrected these typos in the revision.

Dear Reviewer gVD5,

We sincerely thank you for your insightful review and hope that our response effectively addresses your previous concerns regarding this paper.

In response to the issue you raised: in Figure 16 of Appendix D, we have supplemented the visual analysis and provided an explanation of the key elements for temporal-based detection models. Additionally, in our comments, we included a detailed explanation of your other questions regarding our work, such as "Object rotation controllability" and the "details of the experimental setting."

As the discussion period comes to a close, we kindly invite you to share any additional comments or concerns about our work. We would be more than happy to address them. In the meantime, we hope you might consider revisiting your evaluation and potentially increasing your rating.

Thank you for your thoughtful feedback and consideration! We truly appreciate it!

Best regards,

The Authors of Paper 52

W3: Is there any ablation study on the ConvNext for encoding the road map? How about other choices for the road map encoding? And why the ConvNext is frozen?

A3: Following your suggestion, we conducted an ablation study on the road map using the CLIP image encoder and the autoencoder from Stable Diffusion. The results reveal that the performance of the feature extraction network has negligible impact on PerLDiff ($\pm0.02$ NDS between CLIP encoder and ConvNext encoder). The reason for freezing ConvNext is twofold: it reduces the number of parameters to be trained and, since models like ConvNext already possess strong feature extraction capabilities, additional training is unnecessary.

Q1: The term "train + Syn. val*" in Table 3 is not clearly defined.

A4: In Table 3, it should be “Syn. val*” represents the synthetic validation set generated by BEVControl. We have corrected these typos in the revision.

Q2: In Table 1, why the FID metric of PerLDiff is worse than the baseline BEVControl? Moreover, why the performance with BEVFormer is better than the one with multi-modal BEVFusion?

A5: Regarding the FID metric, our results are comparable to those of BEVControl. While PerLDiff incorporates prior constraints to ensure accuracy in object detection, this may adversely affect the details in the background of the images. As illustrated in Figure 14 of Appendix D, PerLDiff produces background details that do not align with those of real images.

As for the performance of BEVFusion, we apologize for the confusion. We utilized the BEVFusion (Camera Only) model, similar to MagicDrive, which indeed shows performance metrics weaker than those of BEVFormer. Additionally, we present the results of BEVFusion (Camera + LiDAR) in Table 4, where the performance metrics are significantly higher than those of BEVFormer.

Q3: In Table 2, why PerLDiff is significantly better than the BEVControl? I want to know the true reason behind the method perspective.

A6: Table 2 presents the comparative performance results of monocular 3D object detection using various synthetic KITTI images. There are two main reasons for the observed differences. First, the limited size of the KITTI dataset, which contains just 3,712 training images, impedes the learning process of traditional methods that do not utilize PerL masking map. Second, monocular 3D object detection is highly sensitive to accurate depth prediction. Depth is derived from the 2D projected size and the estimated 3D size through perspective projection. Our method produces more precise object sizing, thereby enhancing detection performance.

Dear Reviewer sheB:

Thank you for your acknowledgment of PerLDiff’s idea and constructive comments. We provide the following discussions and explanations regarding your concerns.

W1: The authors use a mask-based representation in the PerL-based Cross-Attention mechanism. However, this type of representation lacks sufficient 3D priors, which may contribute to the orientation generation issues for vehicles mentioned in the limitations. Did the authors consider using the depth information of multiple vehicles with ControlNet to enhance object controllability? Specifically, it would be insightful to explore the benefits of Depth + ControlNet vs Mask + PerL-based Cross-Attention in improving object controllability.

A1: Firstly, in addition to the PerL masking map, we also utilize the sequential eight corner coordinates of each object's bounding box as a controlling condition, representing sparse 3D prior knowledge. Secondly, in accordance with your suggestion, we implemented depth maps as controlling conditions. As demonstrated in Table 1 (Depth + ControlNet), their effectiveness is inferior to that of PerLDiff. Finally, one motivation for PerLDiff is to mitigate the issue of data scarcity. It is often impractical to obtain and manipulate the depth maps of multiple vehicles from images for synthetic data generation.

W2: Has the PerL-based Cross-Attention been tested on other baselines beyond BEVControl to examine its potential improvements in controllability and perception tasks? For instance, would applying it to a baseline like MagicDrive yield similar gains?

A2: As illustrated in Figure 8 of Appendix C.4, MagicDrive utilizes text-based cross-attention from Stable Diffusion to implicitly learn a unified feature that concatenates text, camera parameters, and bounding boxes in the token dimension. Based on your suggestion, we integrated the object mask into the token dimension corresponding to the bounding box. As shown in Table 2 (MagicDrive + Mask), results indicate improvements on BEVFormer, with NDS (e.g., 29.77 vs. 28.79 for MagicDrive) and mAOE (e.g., 0.73 vs. 0.81 for MagicDrive) demonstrating the effectiveness of PerLDiff in enhancing the performance of MagicDrive. Note that MagicDrive utilizes a single attention map for managing text, camera parameters, and boxes in the cross-attention process. Consequently, our ability to make improvements is constrained by the limited scope available for modifying the attention map within this architecture.

Dear Reviewer sheB,

We sincerely thank you for your insightful review and hope that our response effectively addresses your previous concerns regarding this paper.

Regarding the issue you are concerned about: we have included additional experiments regarding "Depth + ControlNet" in Table 1. In Table 3, we provided further ablation studies to demonstrate the impact of different choices of road map encoders. In Table 4, we explain the reasons for the performance gap with BEVFormer and BEVFusion. In Appendix (Sec. C.4), we supplemented additional experiments on MagicDrive and diagram to clarify our motivation. Additionally, in Figure 14 of Appendix (Sec. D), we provided visual examples to explain the FID metric.

As the discussion period draws to a close, we kindly invite you to share any additional comments or concerns about our work. We would be more than happy to address them. In the meantime, we hope you might consider revisiting your evaluation and potentially increasing your rating.

Thank you for your thoughtful feedback and consideration! We truly appreciate it!

Best regards,

The Authors of Paper 52