DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes

We sincerely thank Reviewer URnX for the valuable comments provided during this review. The point-to-point responses are as follows.

---

W1: "The Dynamic Object Inpainting Results seem to make the ground have holes (Fig. 1). Could you please explain this phenomenon and discuss any potential limitations or artifacts of this inpainting approach?"

A: Thanks for your comment. We would like to clarify that the "holes" on the ground you mentioned are part of the input ground truth occupancy to the inpainting. While our inpainting results may occasionally exhibit small holes on the ground, we believe this reflects adherence to the distribution of the training data.

---

W2: "It would be helpful to explain the distinction between 'LiDAR generation' and 'semantic occupancy generation'."

A: Thank you for your valuable feedback. To avoid ambiguity, we have updated the "LiDAR" in the title of our paper to "Occupancy". Occupancy usually represents concatenated dense LiDAR scenes, and we have added more necessary explanations in our revised paper.

---

W3: "It’s recommended to compare model sizes across different methods."

A: Thanks for your suggestion. Below, we have included a comparison of model size between our method and OccSora. Our models are significantly smaller than that of OccSora. For other comparisons refer to Table 2 to Table 5 below.

Table 1: Model Size Comparison

---

W4: "It’s suggested to include a comparison of training time and GPU requirements against the baseline method."

A: Thanks for your comment. Below, we have included a comparison of GPU memory usage, training and inference time between our method and OccSora. While our method can be slightly slower and consumes more memory compared to OccSora, we achieve much better performance.

Table 2: Autoencoder on CarlaSC

Table 3: Autoencoder on Occ3D-Waymo

Table 4: DiT on CarlaSC

Table 5: DiT on Occ3D-Waymo

---

Last but not least, we sincerely thank Reviewer URnX again for the valuable comments provided during this review.

Dear authors,

I'm glad to observe the improvement of this paper. Actually, although I'm familiar with the street scale problem, I'm not so professional in the dynamic problem.

I take time to review your paper and give responsible, but not so much breaking feedback that I'm sure won't harm this paper.

Since I'm not so professional in dynamic problems, I may not be able or qualified to judge whether to give a stronger acceptance recommendation based on my knowledge.

Considering this, I lower my confidence.

Update:

Sorry that I forget to choose the authors to be visible in my previous reply:

I acknowledge the authors' reply. My concerns have been addressed. I keep my positive rating.

But I'm still deciding to lower my confidence after consideration for responsible feedback.

We sincerely thank Reviewer a6Q2 for the valuable comments provided during this review. The point-to-point responses are as follows.

---

W1: "... confused by the title ... “occupancy” generation ..."

A: Thank you for your valuable feedback. To avoid ambiguity, we have changed the term "LiDAR" in our title to "Occupancy". We add the necessary explanations in the revised paper.

---

W2: "... method can support long sequential modeling of up to 128 frames, but the factors limiting sequence length (e.g., GPU memory) are not discussed. Additionally, the inference running time is not mentioned."

A: Thanks for your comment. Actually, our long sequential modeling is not limited by GPU memory. It is achieved through auto-regressively generating HexPlanes conditioned on the previous HexPlane. Since we perform long-sequence generation auto-regressively, HexPlanes are generated one at a time, and GPU memory consumption remains similar to that of unconditional generation. We include statistics such as GPU memory usage and inference running time in the table below. While some of our models can be slightly slower and consumes more memory compared with OccSora, we achieves much better performance.

Table 1: Autoencoder on CarlaSC

Table 2: Autoencoder on Occ3D-Waymo

Table 3: DiT on CarlaSC

Table 4: DiT on Occ3D-Waymo

The tables have been included in Appendix Section B.4. Theoretically, our model can generate sequence with arbitrary length, but less stable generation may occur when generating extremely long sequences. Hence, we report "up to 128 frames" in our paper.

---

W3: "Can the proposed method support outpainting similar to SemCity? ... Furthermore, the inpainting examples in the paper are confined to small regions."

A: Thanks for your question.

• Outpainting. Our model can support outpainting, and outpainting results have been included on our website and Appendix C.7 to demonstrate its ability to expand scenes.
• Inpainting. We would like to clarify that the region of inpainting depends on the given mask is not restricted to small regions. To illustrate this, we provide more inpainting results with much larger inpainted regions, observed from a different perspective, on our website.

---

W4: "... to provide quantitative comparison with SemCity in static scenes. The caption of Table 2 points out the authors compare the method with SemCIty, but I do not see results in the table."

A: Thanks for your comment. "SemCity" in Table 2 is a typo and has been removed in the revised PDF. Table 2 in our paper is intended to compare 4D generation results, therefore we did not include SemCity, which is a 3D generation method. We take your suggestion and conduct a quantitative comparison of our method against SemCity on the CarlaSC dataset. The results are as follows:

Details can be found in the Appendix. B.5.

---

Last but not least, we sincerely thank Reviewer a6Q2 again for the valuable comments provided during this review.

We sincerely thank Reviewer nfdL for the valuable comments provided during this review. The point-to-point responses are as follows.

---

W0: "The pretrained networks used to calculate IS, FID, and KID should be motivated more thoroughly, especially in the 2D case ... Unless this 2D CNN is trained to process semantic color maps as inputs ..."

A: Thanks for your suggestion. We would like to clarify that the backbone used to calculate metrics in 3D is pretrained using occupancy data. In the 2D case, SemCity and OccSora use metrics such as IS, FID, and KID, but they do not provide any details of the pretrained networks they use. We use ImageNet-pretrained networks, following the common practice of most 2D diffusion methods. To alleviate your concern, we train InceptionV3 and VGG16 using rendered semantic color maps and re-evaluate our preformance. The evaluation results are provided below and have been updated in Table 2 of our paper.

---

W1: "One conceptual limitation is the fact that the method does not explicitly model uncertainty when forecasting a future scene conditioned on the present. Is this something that can be modeled by sampling multiple futures from the latent space?"

A: Thank you for asking, but we are not sure about the "uncertainty" you mentioned. We assume it refers to the uncertainty in the generation process. Since we are using diffusion, it models generation uncertainty by randomly sampling from gaussian distribution for generation. In other words, different samples can lead to different generations, ensuring the generation uncertainty.

---

W2: "... simulation and motion forecasting. Presenting some results in this area could strengthen the paper..."

A: Thank you for suggesting. This work mainly focuses on generating 4D scenes, which is an early study in this field, holding potentials for various downstream tasks. To address your concern, we conduct an experiment on occupancy forecasting as a comparison with OccWorld. Occupancy forecasting takes 2 seconds of occupancy context as input and predicts the next few seconds. The results are provided in the table below. Our method outperforms the previous state-of-the-art in most metrics.

More details can be found in Appendix Section B.2.

---

W3: "No source code seems to be promised."

A: Thanks for your comment. We commit to release our source code. We will update the codebase soon.

---

W4: "In the current stage, the approach is LiDAR-only. This is a minor limitation, though, and I am primarily mentioning it for completeness."

A: Thanks for your suggestion. Our work mainly focus on generating 4D occumancy, which is challenging but holds the potential of various downstream tasks. We hope our work can inspire future studies involving additional modalities.

---

W5: "Minor suggestions: L245: "first generate" -> "first generates"; L866: Missing parentheses around the PyTorch citation."

A: Thank you for pointing out the typos in our manuscript. We have corrected them in the updated paper.

---

Q0: "How do you calculate metrics like FID in 2D? What specific features do you use for the computation? BEV? What is the network used in these metrics originally trained on? ... If ... are pre-trained on natural images, are they a good fit for comparing ... semantic images?"

A: Thanks for your question.

• 2D Metrics. We follow SemCity and use torch-fidelity to calculate the 2D metrics.
• Feature. We render our features using a 45-degree angle perspective with the xy-plane above the scene instead of the BEV perspective (it lacks z-axis information). Reference images illustrating these viewpoints have been added to our website and our Appendix A.2 for clarity.
• Network. We use ImageNet pre-trained networks for evaluation. Based on the feedback, we have fine-tuned the feature extractor specifically for semantic color maps and updated the results in our paper. The revised metrics can be found in Table 1 above.

---

Last but not least, we sincerely thank Reviewer nfdL again for the valuable comments provided during this review.

We sincerely thank Reviewer D4sh for the valuable comments provided during this review. The point-to-point responses are as follows.

---

W1: "... the dense feature volume and projection module of autoencoder are still very heavy, which limits its efficiency and scalability."

A: Thanks for your comment. We would like to clarify that the memory used by dense feature volumes with 16 frames on CarlaSC is only 8MB, and our projection modules are relatively small, each having only 4,352 parameters. While this operation is heavier than the average pooling used in SemCity, it significantly improves model performance, as demonstrated by our ablation study.

---

W2: "The sample of the dataset is quite limited, which may lead to overfitting and memorization ... lacks clarification of the division of training and test sets, as well as experiments and comparative results ..."

A: Thanks for your suggestion.

• "Overfitting & Memorization". It is possible to observe the memorization issue due to the use of diffusion models. But we want to highlight that our model can generate novel 4D lidar sequences, especially when we are performing conditional generations, even if we are using a relative small dataset for training the model.
• "Division of Training & Test Sets." We use the default training and testing splits on CarlaSC, following the setup in SemCity. For Occ3D-Waymo, we use the validation split for testing, as the test split is not provided with the dataset.
• "Lack Experiments." More generation results, especially conditional generation results, have been added to our website to prove our generalization ability and generative diversity. We would like to highlight that our method can perform conditional generations on HexPlane, command, and layout, which are not possible with previous works such as OccSora.

---

W3: "... a lack of some more simple and practical conditions such as images, text or single-frame point clouds that are easily accessible."

A: Thank you for your constructive recommendation. We provide single-frame point cloud conditional generation result on our website. We fine-tune our HexPlane conditional generation model to condition on single-frame point clouds. This is achieved by encoding a static scene as a HexPlane condition and training our HexPlane conditional generation pipeline as-is. Results can also be found in Appendix Section C.8.

---

Q1: "... XCube using advanced 3D sparse structure ... more explanation and comparison for the 3D backbone."

A: We appreciate the mention of XCube, which uses sparse 3D convolution in its encoder and dense 3D convolution in its decoder. Our encoder uses a light 3D convolution before the main Projection Module and dense 3D convolution in the decoder. The use of advanced 3D sparse structures provides limited benefits to our autoencoder, as the encoder does not utilize many convolution operations. We plan to explore more efficient operations in the future to further optimize our 3D backbone. More discussion can be found in Appendix Section A.3.

---

Q2: "Does the model have the generalization ability and generative diversity? ... provide more (conditional) generation results..."

A: Thanks for asking. Yes, our model could demonstrate strong generalization ability and generative diversity. More conditional generation results have been uploaded to our website to demonstrate that our model can be controlled to generate novel scenes.

---

Q3: "... incorporate more practical conditions such as images, text or single-frame point clouds..."

A: Thank you for your advice. We have included single-frame point cloud conditional generation result in our website. We also include the result in our Appendix Section C.8.

---

Q4: "Given that the title is LARGE-SCALE LIDAR GENERATION ... Or to avoid ambiguity."

A: Thank you for pointing out this. To avoid ambiguity, we include necessary explanations in our paper, and we replace the term "LiDAR" with "Occupancy" in our title. In fact, occupancy represents concatenated dense LiDAR scenes.