SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs

Thank you for the detailed feedback and insightful comments. We appreciate your time and great efforts in reviewing. We now reply to each comment below.

(1) Reweighing strategy

We agree with the reviewer that the reweighing strategy represents a minor level of innovation. As a result, we do not view it as a key contribution or novelty in our paper. Our contribution primarily focus on precise 3D controllability for image generation. The reweighing strategy serves as an engineered component designed to improve image quality and condition-generation alignment.

(2) Object-level editing

In our paper, we provide object-level editing in Figure 3 by placing new objects and deleting existing objects. Furthermore, from a broad perspective, all images shown in our paper represent object-level editing from an empty canvas.

(3) Align experiment setting in 3D detection using BEVFusion

To provide a comprehensive evaluation, we evaluate our method on 3D detection. In Gen setting in our paper, our method achieves mAP in 22.3 and NDS in 31.3, higher than MagicDrive’s 20.8 and 30.2. This denotes we achieve a more effective 3D controllable generation than prior work.

(4) Compare reweighing method with GeoDiffusion

We compare our Depth-aware Foreground Reweighing with reweighing method in GeoDiffusion. Analogous to the setting in Table 3, our method achieves 25.50 in mIOU, which is more effective than GeoDiffusion’s 25.07. Our reweighting method is formulated as an inverted cosine annealing, while GeoDiffusion formulated it as an exponential function. Both of them applied it to depth value.

(5) Noticeable view inconsistencies

In our paper, while contributing to 3D controllable image generation, we acknowledge its limitations.

In most of our cases, our method achieves satisfactory consistency in the intersecting area between the views or frames. However, it is important to acknowledge that our method is not perfect. There is still room for improvement in our consistency, as certain cases have instances of artifacts, like Fig 14 - row 2 column 2-3. We discuss a variety of potential reasons for this problem.

1. Reason 1: Missing occupancy data. In the context of autonomous driving applications, taller buildings are irrelevant to the planning. Consequently, objects exceeding 5 meters in height are excluded from the occupancy data. This leads to the content above this threshold, such as cloud and building, being randomly generated by a diffusion model. In these cases, randomness potentially contributes the inconsistencies in appearance.
2. Reason 2: Imperfect attention. The second reason lies in the imperfect cross-view and cross-frame attention module. View consistency is compromised due to a limited overlapping region between views, resulting in relatively weak supervision. Meanwhile, achieving frame consistency is hindered by the small amount of video clips in the nuScenes dataset, posing a challenge to deliver satisfactory results. As the scale of our data and model is limited in the current context, the visual results are inevitably limited. In the future, we plan to harness the advanced video foundation model such as (open) Sora and its alternatives to improve consistency.

Certain artifacts may not impact downstream tasks. In general, artifacts above the height threshold may not significantly impact downstream applications, as evidenced by our data augmentation experiments. For tasks such as 3D detection or occupancy detection, the upper regions of images are often cropped out, as they exceed the perception boundary, and are not necessary for planning.

Finally, we would like to emphasize that our primary contribution does not lie in frame-consistent or view-consistent generation. Our work primarily focuses on precise 3D controllability for image generation. In our implementation, the frame and view attention modules are analogous to our baseline method MagicDrive. Although we do not show significant improvement in consistency, the performance of our model is sufficient and adequately robust for downstream application, as demonstrated through our data augmentation experiments. We will continue to enhance the multi-view and frame generation in future work.

---

If there are any further concerns, we would be happy to address them in further discussion. Thank you!

Thanks for your positive comments and constructive suggestions. In the following, we reply to individual questions and comments raised by the reviewer:

(1.1) Inconsistency between views and frames.

In our paper, while contributing to 3D controllable image generation, we acknowledge its limitations.

In most of our cases, our method achieves satisfactory consistency in the intersecting area between the views or frames. However, it is important to acknowledge that our method is not perfect, and there is still room for improvement in our consistency, as certain cases have instances of artifacts. We now discuss potential reasons for this problem.

We first note that our condition of occupancy provides a consistent geometry (shape), but does not provide texture consistency between views or frames. In our video demos of Fig.11 row 5 columns 2-3, the car displays a consistent shape but with texture shifting between views. This case is primarily due to insufficient texture consistency, which remains an unresolved open problem. The underlying reason lies in the imperfect cross-view and cross-frame attention module. Their performance heavily depends on the available training data. View consistency is compromised due to a limited overlapping region between different views, resulting in relatively weak supervision. Meanwhile, achieving frame consistency is hindered by the small amount of video clips in the nuScenes dataset, posing a challenge to deliver satisfactory results. The scale of our data and model limited in the current context, is insufficient to enable these attention modules to achieve perfection in our scenario. In the future, we plan to draw on experiences from video generation and harness the advanced video foundation model such as (open) Sora and its alternatives to improve consistency.

Finally, we would like to emphasize that our primary contribution does not lie in frame-consistent or view-consistent generation. Our work primarily focuses on precise 3D controllability for image generation. In our implementation, the frame and view attention modules are analogous to our baseline method MagicDrive. Although we do not show significant improvement in consistency, the performance of our model is sufficient and adequately robust for downstream application, as demonstrated through our data augmentation experiments. We will continue to enhance the multi-view and frame generation in future work.

(1.2) FVD evaluation

Good idea! The FVD metric serves as an effective measure for assessing video fidelity. As our baseline method MagicDrive does not provide FVD evaluation. We compare our method with DriveDreamer and DrivingDiffusion. Our achieved FVD score of 251 surpasses DriveDreamer's 340 and DrivingDiffusion's 332, demonstrating better video quality.

(2) Results for hard cases

It is a good point. We regard the categories such as bicycle and moto as the hard case and corner case that pose significant challenges for the generative model. As they have articulated structures and complex topologies, generating images with bicycles or moto sometimes becomes a mess in our current capacity. Moreover, bicycles and motos belong to the long-tailed class in the training set. The image generation network inherently suffers from data imbalance issues. As a result, the imperfect data with bicycle and moto will hurt the performance, while larger and easier objects such as buses and drivable surfaces tend to yield positive improvement. In the future, we plan to explore the application of scaling laws to scale of our dataset and model size to mitigate this issue and yield better results on bicycles and motos.

(3) Additional experiment

During rebuttal, we add additional experiments. Due to limited resources, including lab equipment and the very long training time of FB-Occ (4 days x 2), we provide small-scale experiments that train FB-Occ using 30% training iterations to reduce training time, while keeping others unchanged. In the setting of data augmentation, our method achieves an mIOU of 34.9, higher than ControlNet’s 33.2 and ControlNet+depth’s 33.5. These experiments denote that our method provides better effectiveness than the ControlNet baseline.

(4) Discussions regarding 3D occupancy as a 3D geometry condition

1. Defect in occupancy. We agree that the coverage area of occupancy input is limited. In the context of autonomous driving applications, taller buildings are irrelevant to the planning. Consequently, objects exceeding 5 meters in height are excluded from the occupancy data. This leads to the content above this threshold, such as cloud and building, being randomly generated by a diffusion model. In these cases, randomness potentially contributes the inconsistencies in appearance.
2. Open-vocabulary image generation. Good suggestion! We will explore open-vocabulary generation to broaden our application domain in the future. In this setting, we will achieve a more fine-grained control than a fixed vocabulary.
3. Instance-level condition. Adding instance-level conditions will alleviate this problem. One possible method is to incorporate instance masks containing instance IDs such as 0,1,2. This way can be effective in distinguishing overlapping objects but not 3D-aware to capture the underlying 3D shape of objects.

---

If there are any further concerns, we would be happy to address them in further discussion. Thank you!

We greatly appreciate the careful proofreading of our paper. We would like to thank the reviewer for the valuable feedback and for appreciating our clear writing. In the following, we reply to individual questions and comments raised by the reviewer:

(1) W1. Geometry Constraint

In our understanding, the reviewer posits that the absence of explicit geometric constraints precludes the diffusion model from generating strictly consistent images, a perspective we respectfully find to be inaccurate. By implicitly learning geometry-aware image generation from 3D conditions, our diffusion model is trained to produce images that are aligned with the provided conditional input. While the consistency between the condition and the image may not be strictly guaranteed like a 3D representation (mesh or NeRF), the underlying label-image consistency is sufficiently robust. As evidenced by our data augmentation experiments, the generated image is already effective and valuable for downstream tasks.

As for view and frame generation, we additionally learn cross-view attention and cross-frame attention for consistent generation as described in our paper Sec. 3.4. The two attention modules are designed to enable the target view to access information from its neighboring views, specifically from the left and right views, or history frames. This feature flow facilitates the synthesis of coherent and contextually relevant views, thereby enhancing the overall consistency of the generated content.

(2) W2, Q1. Hyperparameters

The assumption that more parameters, are more difficult to tune may be inaccurate. In practice, we find that the hyperparameters m and n in Eq. 6 do not need to be carefully tuned. n is the training iterations that do not need adjustment. m is the maximum weight coefficient. Empirically, we’ve tried m=2 and m=3, both of them yield nearly equivalent performance. The hyperparameter is heuristically designed and shared across different datasets and categories.

(3) W3. Video Demonstration

We have provided generated video in Figures 11, 12, and 16. To clearly demonstrate and compare the consistency across different views and frames, we present the videos frame by frame. It is important to acknowledge that there is still room for improvement in our consistency, particularly as we fine-tune our cross-frame attention with a limited and small dataset. In the future, we plan to harness the advanced video foundation model such as (open) Sora or its alternatives to improve consistency.

Finally, we would like to emphasize that our primary contribution does not lie in frame-consistent or view-consistent generation. Our work primarily focuses on precise 3D controllability for image generation. In our implementation, the frame and view attention modules are analogous to our baseline method MagicDrive. Although we do not show significant improvement in consistency, the performance of our model is sufficient and adequately robust for downstream application, as demonstrated through our data augmentation experiments. We will continue to enhance the multi-view and frame generation in future work.

(4) Q2. Appearance change

In various figures throughout our paper, such as Figure 5, 7, and 11, the objects and contents maintain consistent identities across different frames and viewpoints. This consistency is learned by cross-frame and cross-view attention within a classifier-free guidance framework. We acknowledge that there are minor artifacts in certain cases due to imperfect attention modules. In our implementation, the two attention modules are analogous to MagicDrive. Although we do not show significant improvement in view consistency, the performance of our model is sufficient and adequately robust for downstream application, as demonstrated through our data augmentation experiments.

---

If there are any further concerns, we would be happy to address them in further discussion. Thank you!

We thank the reviewer for the valuable comments. As the deadline for the rebuttal disscussion is approaching, we would greatly appreciate it if you could take the time to review our rebuttal at your earliest convenience.

Please let us know if there is any additional information we can provide or any further clarification needed. Thank you very much for your time and consideration. We look forward to your review.

We would like to thank the reviewer for the positive and detailed feedback. Below, we reply to individual comments and questions raised by the reviewer:

(1) W1. The reliance on occupancy introduces a minor burden but with enhanced controllability

We agree with the reviewer regarding the reliance of our method on more sophisticated occupancy data. Nevertheless, acquiring such data is not overly challenging. Occupancy data can be semi-automatically annotated using semantic point cloud and 3D bounding boxes (refer to [a]), which stands for an acceptable burden. On the other hand, condition on occupancy enables finer 3D controllability, so that we can edit object shapes and irregular background elements like drivable area and terrain. This level of control is challenging to achieve with previous 3D bounding boxes or Bird's Eye View (BEV) layouts.

[a] SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving

(2) Q1. Similar structure in Fig.1: Controllability ensures our condition-image consistency

We provide several analyses of this effect:

1. A similar structure in Fig.1 is an expected result. The first thing need to note is that, if we use the same random seed and condition input, the generated images in multiple runs will be the same. As we only modify a partial of occupancy input by placing traffic cones, the majority of occupancy data is unchanged. Besides, in implementation, we use the same random seed in two figures. Consequently, the generated images display similar street structures. In general, our diffusion model learns 3D controllable generation through a CFG paradigm [b], aligning with the frameworks used in text-to-image and ControlNet. Using pairs of conditions and images, our method learns controllable generation in a classifier-free manner.
2. Empirically, it has been observed that our method does not overfit the nuScenes training set. As demonstrated in Figure 5 of our paper, when the same occupancy input is provided, our method produces a variety of images with consistent attributes and diverse appearances. This evidence indicates that our method does not overfit training images, as it maintains the ability to generate diverse yet aligned outputs.
3. By employing a larger training dataset, our approach is expected to improve generalizability. We consider the nuScenes dataset, which comprises 200,000 images, to be of substantial scale. Utilizing this dataset for training, our model is more likely to learn the intrinsic relationships between the input conditions and the generated images, thereby mitigating the risk of unexpected consistency of overfitting.

[b] Classifier-Free Diffusion Guidance

(3) L1. User-friendly editing: disentangle the occupancy into foreground and background

To ease the editing, we provide a strategy that disentangles the foreground control and background control in occupancy data. If we want to edit a car’s trajectory, we can keep the background occupancy unchanged and select the car’s first frame occupancy using the 3D box. During the following frames, we remove foreground occupancy and simply place our foreground target’s occupancy in a certain location using trajectory. By doing so, we only add minor steps by using occupancy but provide more precise 3D control, which makes it a favorable choice for conditioning.

---

If there are any further concerns, we would be happy to address them in further discussion. Thank you!