PanoDiffusion: 360-degree Panorama Outpainting via Diffusion

We appreciate the reviewer's valuable feedback and suggestions. We would like to provide a detailed response to each comment.

Q1: The proposed RGB-D framework is constructed by using two parallel LDM to reconstruct the depth and RGB images separately. This structure might result in a larger and more complex model architecture than using a shared or depth-conditional LDM.

We do not use two parallel LDM's. As is claimed in Section 3.3, using two parallel LDMs to reconstruct Depth and RGB images separately will incur excessive computational resources. To avoid this, our PanoDiffusion uses two VAE encoders, but only one LDM, with depth info as concatenated channels. Therefore, the computational cost does not increase significantly compared to the original conditional LDM (see Q6).

Q2: Apart from the visualization results, how about the evaluation results of using concatenation or cross-attention in the so-called depth-conditional diffusion model?

We've additionally tested depth-conditional diffusion models that use the usual concatenation and cross-attention conditioning on all types of masks. The results show our PanoDiffusion consistently outperforming the traditional depth-conditional LDM formulation.

Q3: How is the cross-attention operation used in the depth-conditional method? Whether the authors try other advanced multimodal fusion methods to better combine the features from RGB and depth?

For depth-conditional LDM's, we pretrained a VQVAE for depth, and kept the encoder. The encoded depth is fed into the LDM as a condition in the usual two options, either via concatenation, or by cross-attention. Additionally, to improve robustness, we allowed the LDM to update the depth-condition encoder weights during training. However, these methods led to unsatisfactory outpainting results, often resulting in blurry content.

We did not investigate more advanced multimodal fusion baselines, because our main task is RGB outpainting without any depth input at inference time.

We appreciate the reviewer's valuable feedback and suggestions. We would like to provide a detailed response to each comment.

Q1: There is not many noval changes to the LDM framework. The authors only add a pretrained depth map encoder as a guidance.

The more obvious approach is to train an LDM with depth masks as condition, particularly during training. We avoided this, and instead chose to use a RePaint-based structure, because we want to free the model from presuming mask distributions. This is critical for our two-end alignment design, as masks will continually change due to rotation. As LDM or Repaint cannot individually handle this challenge properly, it provides ample motivation for us to fuse these two components, naturally combined with our novel rotational denoising. Furthermore, we advanced our model to a bi-modal structure, where depth maps prove to have significant auxiliary effect in generating well-structured results. We also consider that our discovery that the asymmetric condition, where having a training setting (use of depth and without masking) that is substantially different from the inference setting (exclusion of depth and with masking), still leads to improved performance, is an interesting novel finding.

Q2: The authors should add more visual results for comparison in the supplementary materials.

Perhaps it was overlooked, but we had previously included more visual results for comparison in our Appendix, previously Fig. 12, now 13, including two groups of examples for each mask type. Additionally, we had also provided some examples where we project different panorama images to 3D scenes in our supplementary video.

Q3: How about the training and inference time compared to other frameworks?

For training, we compared average training time (minutes) for one epoch of PanoDiffusion against baseline models on the same devices, using the same batch size 4 and the same training dataset. For inference, we compared the time (seconds) required to infer a single image.

The results show that while our model is not the fastest, it remains within a reasonable range. It's also noteworthy that, compared to the original LDM framework, our bi-modal structure achieves a significant improvement in the quality of outpainting. This improvement comes without a proportionate increase in resource consumption – we observed only a modest increase of 13.8% in training time and 25% in inference time.

We appreciate the reviewer's valuable feedback and suggestions. We would like to provide a detailed response to each comment. For Q3 and Q4, much of the replies here are also included in our response to reviewer SqKp.

Q1: I would be interested in seeing the panoramic images of both the proposed method and competing method in a viewer like threejs.

Thank you for your suggestion! In fact, we've already completed a demo that transforms panorama images into interactive threejs for users to check the results, and we plan to release the examples upon the acceptance of our paper. As proof, we provided some examples at 00:12 in our supplemental video for a preview, which are outpainted panoramas of our PanoDiffusion.

Q2: There are many ways to represent 360 degree panorama images. The authors should clarify that using the equirectangular projection is a choice they are making.

Thank you, we have clarified the choice of using equirectangular projection in the dataset section of our revised paper.

Q3: Do the authors only rotate the camera in the horizontal direction? Or are vertical rotations allowed as well?

For both training and inference stages, we only consider rotations in the horizontal direction. As panorama images typically follow equirectangular projection, the distortion increases non-uniformly towards the top and bottom poles. Introducing angle rotations in the vertical direction would lead to substantial changes in the projection results, and require more complex preprocessing, which will increase both training and inference costs. We have added two examples where the camera has a 90-degree vertical rotation in our Appendix 1.3 Fig.~10. Conversely, distortion is uniform horizontally -- image manipulation here only involves horizontal cropping and splicing, without the need for reprojection. Since the main objective of PanoDiffusion is to generate coherent and realistic panorama images, we have focused more on the spatial wrap-around consistency in the horizontal direction only. In the future, we will consider better approaches to support wrap-around consistency in all directions.

Q4: During inference, did the authors try 180 degree shifts instead of 90 degree shifts? Or some other rotation?

During inference, we have explored the effect of different rotation angles, including 180°, 90° and 45°, on the outpainting results. The results show that the wrap-around consistency of outpainting results is improved across all settings. Compared to 180°, 90° leads to better consistency. However, diminishing the angle further to 45° did not lead to additional improvements. We believe this is reasonable, as the model is expected to generate coherent content when the two ends are in contact for enough denoising steps. Therefore, smaller rotation angles than 90° and longer connections do not necessarily lead to more consistent results. We have incorporated this part into the Appendix of our revised paper.

Q5: What’s the difference between Fig. 2 and Fig. 3?

Fig. 2 is intended to provide an overview of our method. We had previously deliberated on integrating the circular shifts, masking operations and iteration loop into Fig. 2, but eventually felt it became excessively cluttered. So we created Fig. 3 to illustrate those operations in more detail, while excluding them from Fig. 2.

More specifically, Fig. 2 presents an overview of the entire model pipeline, where we only show the input and output of each stage and omit the specific details of circular shift and adding noise. This is to make our two-stage bi-modal structure clearer and to highlight the difference between the inputs we use for training and inference.

For Fig. 3, we instead focus on explaining how the camera rotation mechanism is implemented. We detailed the flow of the rotation process, including the incorporation of noisy images with different transparency to represent different noising steps.

We have clarified this difference in Fig. 2 caption in our revised paper.

Q6: In the supplemental video, do the shown 3D Scenes at 00:12 use the generated depth maps?

The 3D scenes at 00:12 do not use the generated depth maps. We only use the outpainted panorama images of PanoDiffusion and project them onto spherical coordinates using threejs, which enabled their display in an immersive 3D format.

We appreciate the reviewer's valuable feedback and suggestions. We would like to provide a detailed response to each comment.

Q1: During the training stage, does the random angle rotation only apply to horizontal direction?

During the training stage, we restrict random angle rotation solely to the horizontal direction. As panorama images typically follow equirectangular projection, the distortion increases non-uniformly towards the top and bottom poles. Introducing random angle rotation in the vertical direction would lead to substantial changes in the projection results, and require more complex preprocessing, which will increase training costs. We have added two examples where the camera has a 90-degree vertical rotation in our Appendix 1.3 Fig.~10. Conversely, distortion is uniform horizontally - image manipulation here only involves horizontal cropping and splicing, without the need for reprojection.

In terms of training efficacy, without providing camera viewpoints as part of the input, adding vertical rotations does not improve model performance. Instead, it can complicate the learning of equirectangular projection patterns, reducing overall effectiveness. Since the main objective of PanoDiffusion is to generate coherent and realistic panorama images, we focus more on data augmentation in the horizontal direction - the spatial wrap-around consistency.

Q2: In order to test the generalization capability, can you provide more results of running the proposed method on other datasets?

We conducted additional tests using a set of 2305 panorama images from Matterport3D [1] dataset. These tests were performed without re-training or fine-tuning PanoDiffusion, which was trained on the Structured3D dataset.

Matterport3D differs from Structured3D with greater complexity and diversity, including multi-story and outdoor scenes. Despite these challenges, PanoDiffusion outperforms the baseline models and is able to outpainting reasonable panorama images in most of the cases. We have provided some qualitative results in the Appendix of our paper for your reference.

Q3: This paper claims the camera rotations as one of the contributions, please provide an ablation study on how the camera rotation actually works effectively.

We have previously provided some ablation results on this. Please see Fig. 8 and Table 3, where we compared PanoDiffusion with and without the use of rotations.

To further shed light on the reviewer's question, we additionally explored the effect of the rotation angle, including 180°, 90° (chosen for our final result) and 45°, on the outpainting results. The results show that the wrap-around consistency is improved across all settings. Compared to 180°, 90° leads to better consistency. However, diminishing the angle further to 45° did not lead to additional improvements. We believe this is reasonable, as the model is expected to generate coherent content when the two ends are in contact for enough denoising steps. Therefore, smaller rotation angles and longer connections do not necessarily lead to more consistent results.

[1] Chang A, Dai A, Funkhouser T, et al. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158, 2017