We sincerely thank the reviewer for the thoughtful and constructive questions. Please find our detailed responses below:

W1. Video Generation Quality

We appreciate your thoughtful feedback. This phenomenon is expected given the nature of the scene. The effect observed in certain scenarios—such as the discontinuities in lane line rendering in the upper part of Figure 11—primarily arise from the presence of dashed road markings, which inherently result in intermittent visual patterns when extracting consecutive frames. We kindly refer you to the supplementary videos, which further illustrate the temporal consistency and overall quality of our video generation results.

W2. Complexity of Object Editing

A key contribution of our work is enabling controllable editing for a wide variety of geometric shapes. As shown in Figure 7e, our framework can generate complex and irregular objects, such as excavators, demonstrating the model’s capability to handle challenging object geometries. Since we are unable to add new visual results during the rebuttal period, we will expand our experiments to systematically assess the generalization ability of our method on more complex and irregular object editing tasks in future work.

W3. Data Augmentation

We appreciate your insightful observation regarding Table 9, which indicates that an excessive amount of synthetic data can negatively impact model performance. Our analysis attributes this phenomenon to the distribution gap between synthetic and real-world data; over-reliance on synthetic samples may introduce distributional noise and reduce generalization. To address this, we recommend carefully balancing the ratio of synthetic to real data in practical applications. Additionally, strategies such as data filtering and adaptive sampling can be employed to enhance the quality of the training set. In the revised manuscript, we will provide further experimental analysis on the effects of different data mixing ratios and discuss methods for improving data quality.

It is important to note that the primary focus of our work is on controllable generation, rather than the synthetic data itself. We believe our contributions in this area are substantial. Nevertheless, we recognize the value of synthetic data for downstream tasks and plan to explore its broader applications in future work.

W4. Additional Related Work

We have provided discussions of methods such as UniScene[1] in the related work section. We appreciate the reviewer's comparison with UniScene. However, it is important to note that UniScene operates as an image-to-video (i2v) model that requires an additional initial frame as input (the Zc conditional reference frame in Eq.8 of the UniScene paper). Because of this difference in required inputs, comparing FID scores is neither reasonable nor fair (GT leaking: simply repeating the initial frame n times would yield decent FID scores). As a follow-up work to ours (UniScene was released after our arXiv preprint), the condition design of UniScene actually aligns with our ControlNet+depth experiment in Table 1, which uses semantic+depth for control. Under aligned settings in Table 1, our MPI approach achieved significant improvement over UniScene's semantic+depth approach (25.5 vs 17.5 mIoU).

Below we provide a comparison of mIoU and FID between our method and Semantic + Depth under aligned settings:

As shown in the table, our MPI-based approach achieves a better mIoU and FID compared to Semantic + Depth, demonstrating the effectiveness of our method in geometry control.

We would also like to point out that UniScene adopts a lossy representation (single plane), which inevitably leads to some information loss during the condition generation process. In contrast, our approach leverages a more faithful and information-preserving representation, resulting in improved controllability and fidelity in the generated scenes.

Thank you again for your valuable suggestions, which will help us further improve and refine our work.

We sincerely appreciate the reviewer’s valuable comments and in-depth discussion.

We fully understand and agree with the reviewer’s concerns regarding the effectiveness of mixing synthetic and real data during downstream task training.

Due to limitations in computational resources and time during rebuttal, we are difficult to conduct a systematic and comprehensive study on this issue. However, we have carried out preliminary explorations and plan to further address this topic in future work. Specifically, we propose the following directions for improvement:

(1) dynamic data sampling, where the ratio of synthetic to real data is adaptively adjusted according to the training stage of the model to mitigate the impact of distributional differences;

(2) bilevel core set selection, which aims to identify and select the most valuable synthetic samples for downstream tasks, thereby improving data utilization efficiency.

We hope these approaches have the potential to further narrow the performance gap between synthetic and real data, and we will conduct more systematic empirical analyses in our future work.

W1. Regarding the comparison between multiple image planes (MPIs) and standard voxel projection

We greatly appreciate your suggestions. We agree that the main advantage of MPIs lies in their ability to encode occluded regions. Regarding the statement that "ControlNet+depth can be regarded as a degradation of SyntheOcc reduced to a single plane," we acknowledge the architectural differences between the two approaches. We have conducted additional experiments to align the MPI encoder and re-weighting components, with the following results:

Through these experimental configurations, we can systematically analyze the contribution of MPIs to occluded region modeling and the impact of architectural differences on performance. We found that both the intermediate representation of MPIs and the network structure of the MPI encoder contribute to improvements in the final results.

W2. Regarding the supplementation of related work

We thank you for your corrections. We will supplement more literature on 3D occupancy prediction and diffusion-based image generation in future versions, including seminal works such as MonoScene (CVPR'22), and add citations to these works in the references section to improve the literature review.

W3. Regarding the practical effectiveness of data augmentation and computational resource consumption

We acknowledge that the current improvement is limited and training costs are high. We believe that SyntheOcc's main contribution lies in proposing a new controllable generation paradigm, with data augmentation being only one of its potential applications. In the future, we will explore more efficient training and inference schemes to enhance its practicality.

W4. Regarding the comparison in Figure 7a

We found that simply adding depth information cannot fundamentally solve this problem. The possible reason is that the occupancy resolution is relatively small and cannot precisely reflect depth relationships, while MPIs can completely preserve the length and width information of objects, thereby maintaining correct object poses in the generated results. Since rebuttal cannot provide new visualizations, we will include this comparison in future versions.

W5. For video generation, we used a VLM for annotation, specifically employing the cogvideox annotation tool cogvlm2-llama3-caption for video labeling. We will supplement this explanation in the revised manuscript.

W6. Regarding the fairness of the "Gen Val" setting in Table 1

We thank you for your attention. In fact, we used different models to generate different validation sets. For example, we used MagicDrive and ControlNet to generate corresponding validation sets, and then evaluated them with FB-OCC. Under these circumstances, the comparison is fair.

W7. Regarding the parameter settings for the "Progressive Foreground Enhancement" and "Depth-aware Foreground Reweighing" modules

We acknowledge that these modules have some hyperparameters, but we found that making small adjustments to the hyperparameters within a certain range has minimal impact on the results.

To further validate the impact of parameter changes on model performance, we conducted ablation experiments on the weight parameter m in the two modules. Due to computational resource constraints, we experiment with a few parameters. The table below shows the mIoU performance of the model on the validation set under three different m values in Eq. 8:

As m increases, we observe that the mIoU experiences a slight decrease; however, the overall performance of the modules remains relatively stable across different m values. This indicates that our method is robust to changes in this hyperparameter, and small variations in m do not significantly affect the final results.

We thank you again for your valuable suggestions, and we will supplement and improve the above issues in future versions. We welcome further discussion!

Thanks authors for their detailed response and the considerable effort invested in conducting these experiments, especially within such a short timeframe.

Does the MPI's advantage lie in the ability to encode occlusion?

I am convinced by the provided results, which confirm the benefits of MPI for modeling occluded scenery, given the significant performance degradation when retaining only visible voxels in the MPI. Thank you for providing experiments that isolate the architectural effects. I am convinced by the results. However, the authors should still modify the incorrect statement: "ControlNet+depth can be regarded as a degradation of SyntheOcc reduced to a single plane."

Poor related works section

I trust that the authors will significantly improve the related work section to better connect and contextualize their work within the existing literature. Not directly relevant, but the paper reminds me of SceneRF (ICCV'23), where they adapt "MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis," -- which is a MPI-based method -- for self-supervised occupancy prediction. This might be worth mentioning as it provides additional context for MPI applications in 3D occupancy prediction tasks.

Regarding the practical effectiveness of data augmentation and computational resource consumption

I agree with the authors that "SyntheOcc's main contribution lies in proposing a new controllable generation paradigm, with data augmentation being only one of its potential applications." However, this might be worth mentioning in the limitations section.

Figure 7a is unconvincing due to comparison with ControlNet lacking depth information

The authors stated that "adding depth information cannot fundamentally solve this problem. The possible reason is that the occupancy resolution is relatively small and cannot precisely reflect depth relationships." This explanation is not convincing, as one could use raw depth without discretizing into occupancy. Given that the authors have a "ControlNet+depth" baseline in Table 1, it remains unclear why this wasn't used directly for Figure 7a instead of only "ControlNet."

The "Gen Val" setting in Table 1 creates potentially unfair comparisons

The authors clarify: "we used different models to generate different validation sets. For example, we used MagicDrive and ControlNet to generate corresponding validation sets, and then evaluated them with FB-OCC. Under these circumstances, the comparison is fair." Please add this crucial information to the experiments section, as it is currently missing from the paper.

Regarding parameter settings for "Progressive Foreground Enhancement" and "Depth-aware Foreground Reweighing" modules

When varying hyperparameters, the mIoU changes very slightly (25.56 → 25.54 → 25.53). Does this indicate that these modules have minimal impact? If hyperparameters have negligible effects, what justifies introducing these components?

Additional concerns from other reviewers

I find several concerns raised by other reviewers quite compelling:

• The comparison between 3D MPIs vs. full 3D occupancy volume conditioning (e.g., UniScene) raised by reviewers H8Lu and 9Zfe. While comparison with UniScene is not mandatory given its very recent publication at CVPR'25, the discussion should still be addressed.
• While SyntheOcc increases dataset diversity, Table 9 demonstrates that excessive synthetic data can degrade model performance (reviewer 9Zfe)

I would like to see the discussion from other reviewers before making my final decision. Overall, I remain positive about this paper.

We sincerely thank the reviewer for the detailed evaluation and valuable suggestions. We address each of the raised concerns as follows:

W1. Control property of 3D semantic occupancy

Our method mainly relies on the semantic occupancy annotations provided by the nuScenes dataset, which currently do not include finer-grained traffic element information. If more detailed annotations become available in the future, our method can seamlessly support more precise scene control.

Our objective and contribution is not to achieve lane-level control, but rather to propose a lossless intermediate representation that extracts geometric and semantic features from occupancy. Currently, the semantic occupancy based on nuScenes annotations does not include lane-level annotations. However, if such annotations were available, our methodology would be fully capable of performing lane-level controllable generation. This property is not a disadvantage of our methodology, but rather a characteristic of the dataset itself.

W2. More test cases

Thank you for your suggestion. As we are unable to add new experiments during the rebuttal period, we kindly ask you to refer to our supplementary materials. In video1.mp4, the two videos in the upper right corner (e.g., row 2, column 3) demonstrate ego vehicle turning and rotation scenarios, where our model successfully generates plausible videos. As we can not add new figure during rebuttal, we will include additional spatial consistency validation cases, such as lane changing and cut-in scenarios, in future versions to further demonstrate the practical potential of our method.

W3. FVD comparison

We used the same FVD evaluation code as MagicDrive to compute FVD metrics between the generated videos and the ground truth videos. The specific implementation can be found in the open-source repository of MagicDrive. We have also provided our generated videos in the supplementary materials for reference, so the generation quality can be intuitively observed. In future versions of the paper, we will include more detailed descriptions of the evaluation process and protocols to further improve the transparency and reproducibility of our evaluation.

Once again, we thank the reviewer for the constructive feedback, which will greatly help us improve and refine our work. We welcome further suggestions and discussion.

We sincerely thank the reviewer for the thoughtful and constructive questions. Please find our detailed responses below:

W1. On the “lossless representation” claim

We appreciate the reviewer's careful consideration of our "lossless representation" claim. We would like to clarify that our approach maintains the same resolution as the input occupancy grid, where dmin and dmax correspond directly to the occupancy voxel boundaries. Furthermore, our use of surround-view cameras enables 360-degree coverage, ensuring comprehensive capture of all occupancy within the camera frustum. Regions outside the camera field of view are naturally excluded from generation requirements. We acknowledge that our original terminology may have been imprecise and will revise our manuscript to more accurately reflect this perspective.

W2. Benefit of using 3D MPIs

We appreciate the reviewer's comparison with UniScene. However, it is important to note that UniScene operates as an image-to-video (i2v) model that requires an additional initial frame as input (the Zc conditional reference frame in Eq.8 of the UniScene paper). Because of this difference in required inputs, comparing FID scores is neither reasonable nor fair (GT leaking: simply repeating the initial frame n times would yield decent FID scores). As a follow-up work to ours (UniScene was released after our arXiv preprint), the condition design of UniScene actually aligns with our ControlNet+depth experiment in Table 1, which uses semantic+depth for control. Under aligned settings in Table 1, our MPI approach achieved significant improvement over UniScene's semantic+depth approach (25.5 vs 17.5 mIoU).

Below we provide a comparison of mIoU and FID between our method and Semantic + Depth under aligned settings:

As shown in the table, our MPI-based approach achieves a better mIoU and FID compared to Semantic + Depth, demonstrating the effectiveness of our method in geometry control.

We would also like to point out that UniScene adopts a lossy representation (single plane), which inevitably leads to some information loss during the condition generation process. In contrast, our approach leverages a more faithful and information-preserving representation, resulting in improved controllability and fidelity in the generated scenes.

W3. On inference runtime analysis

We appreciate the reviewer’s suggestion regarding inference efficiency. Our experiments indicate that the average image generation time is 1 second per frame, while video generation (49 frames) requires approximately 50 seconds. These results were obtained on an NVIDIA A100 GPU.

We will incorporate these experimental results into the revised manuscript to provide a more comprehensive assessment of the practical applicability of our method.

Q1. Application of Eq. 8 for reweighting loss in Eq. 9

A1. Yes, the reweighting is implemented via masking, utilizing the ground truth (GT) to guide the denoising objective.

Q2. “Gen” validation set in Table 1

A2. Yes, the “Gen” validation set refers to samples generated by our proposed method. We evaluate the alignment between the input occupancy condition and the generated images, and further assess perception accuracy on this generated validation set using a pretrained detector.

We thank you again for your valuable suggestions, and we will supplement and improve the above issues in future versions. We welcome further discussion!

Thank you very much for your valuable feedback and for taking the time to review our manuscript. We are pleased that we have carefully addressed the concerns and questions you raised in your review.

As you’d mentioned an update to your overall rating, we wanted to gently remind you that you do have the option to adjust your review rating accordingly (edit button). We’ve noticed that the system still shows the original rating of 3 at this time, and it seems no changes to the rating have been made just yet.

Thank you again for your contribution to improving our work.