DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion

We thank Reviewer Yygs for his constructive suggestions and appreciate comments like "interesting task", "useful for practical applications such as VR environments", "new dataset quite meaningful", and "effectively adapts to panoramic images". We will gladly incorporate all the feedback in the revised version.

1. The main concern of reviewer Yygs is the quantitative or qualitative evaluations of the realism of the generated multi-view panoramas. We adopt the FAED metric to evaluate the realism. Experiments show that our method achieves lower FAED than MVDream and Text2Light, but slightly higher than PanFusion.

However, our CLIP Score achieves the highest results. Note that our method is significantly faster than existing methods, making scalable panoramas possible. The reason is that PanFusion adopts several complex modules in single-view panorama generation, which can improve FAED to a certain extent, but the speed is too slow. A fast pre-training module makes fine-tuning faster and better during multi-view generative network training.

2. Transform the panorama to the perspective views: As suggested, we converted the generated panorama into perspective images and conducted qualitative (shown in global response) and quantitative comparisons. Experiments show that our method achieves the lowest FID, while our method is higher than MVDiffusion in IS and slightly lower than PanFusion. It should be noted that MVDiffusion directly generates perspective images and then stitches them into panoramas. It is not in the ERP format and does not have the top and bottom parts. Our panorama generation speed is faster.

3. Diversity of DiffPano: We show some qualitative panorama results in the global response, which shows our method has good diversity. We also show some outdoor panoramic generation results to demonstrate generalization. Generalization and panorama effects are a trade-off. When we pursue better geometric consistency in panorama generation, generalization may indeed decrease. This requires a balance. A potential solution is to use more panoramic images to train native text-to-panorama generation diffusion models, rather than using LoRA to fine-tune from SD, which can be a future work. However, training such a model from scratch requires lots of GPU and dataset. Using LoRA to fine-tune the SD model may be a more practical and cost-effective approach.

4. One-stage vs Two-stage: We conducted ablation experiments on one-stage and two-stage training. Experiments show that due to the introduction of the epi module, the multi-view consistency of the two training methods is relatively good. Comparing the consistency is not enough to measure the difference between the two methods. The IS of the two-stage training method is lower, and the diversity is reduced to a certain extent, which is slightly worse than the one-stage training. However, it should be noted that the images after one-stage training will have ghosting, but the two-stage will not. We show the qualitative comparison results in the global response (the FID value of the two-stage method will also be higher).

5. Missing citation of Cap3D: First of all, the objects of text annotation are different. Cap3D annotates objects, while we annotate scenes. We first annotate the perspective view of the cube map. When integrating different perspective views, the instructions input to LLM are specially processed for the scene to prevent redundancy and other problems. We show the difference in text description before and after LLM integration in Figure 2 of this article. Our LLM text prompt is as follows:

You will be given six text prompts that describe the appearance of a room. Your task is to summarize these six given text prompts into a single text prompt that describes what the room looks like. This single summary text prompt in your response should strictly begin with <begin> and end with </end>. Simplify the room types if there are multiple room types in the prompts. The final text prompt should be less than 77 tokens and do not omit any furniture or any appearance details in the given prompts.

As suggested, we will cite Cap3D in the revised version.

Dear Reviewer Yygs,

Thank you again for your review. We hope that our rebuttal could address your questions and concerns such as the quantitative or qualitative evaluations of the realism of the generated multi-view panoramas, the qualitative examples of the perspective projections of the generated panoramas, the generalizability of DiffPano, and the citation of Cap3D. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.

We have provided an anonymous link to the AC to forward the link to the reviewers, due to the requirements during the rebuttal stage. This anonymous link contains some generated outdoor panoramic samples.

It would be appreciated if you could raise your score on our paper. We thank you again for your effort in reviewing our paper.

Best regards,

DiffPano Authors

We thank all reviewers for their constructive comments and recognition of our work ("first work, pioneer work, interesting task", "novel and technically sound", "plausible and visually pleasant panoramic results"), and ACs and SACs for their hard work. However, Reviewer K8Nd seems to have misunderstood the core content of this paper, the problem it solves, and the core innovation. We believe this review may be pure LLM generated. We hope that ACs, SACs, and all reviewers can read our paper and the rebuttal and then have in-depth discussions on K8Nd's misunderstandings to ensure that reviewer K8Nd can eliminate the factual errors and re-evaluate our work to obtain a fair review. We sincerely thank you.

Here, we point out the factual error. Reviewer K8Nd argues that our approach lacks innovation and is a technique that has been studied in previous approaches. The goal of this paper is to perform scalable multi-view panorama generation based on text (which can be expanded from one room to another). As far as we know, we are the first to propose text-to-multi-view panorama generation. Previous methods are limited to single-image panorama generation. The core reason is the lack of large-scale multi-view panorama datasets, and single-view panorama generation work cannot guarantee the multi-view consistency of the generated panorama. A new framework suitable for multi-view panorama generation is needed. To this end, we first construct a large-scale panoramic video-text dataset. Then, we design a multi-view panorama generation framework. First, we propose a diffusion model for text-to-single-view panorama generation (fine-tuned on the newly proposed panorama dataset using LoRA). To achieve multi-view consistency, we derive a spherical epipolar constraint suitable for panoramas and embed it into the network as an attention layer to realize the spherical epipolar multi-view panorama generation framework, which can achieve diverse and consistent scalable panorama generation results. Our method is different from previous methods in terms of both technique and dataset (as recognized by Reviewer 2LuU and Reviewer 7HQE), but Reviewer K8Nd did not realize the innovativeness of our technique.

We hope that our rebuttal and explanation can correct the reviewer K8Nd's misunderstanding so that our work can be properly reviewed and obtain a satisfactory review result. We sincerely hope that ACs and SACs can have in-depth discussions with all reviewers to review our work more fairly.

We thank Reviewer K8Nd for his constructive suggestions and appreciate comments like "interesting idea", "new way to enhance 3D Vision", "well-structured", and "clear explanation". However, the reviewer seems to have misunderstood the core content of this paper, the problem it solves, and the core innovation. Here, we point out the factual error.

This paper aims to perform scalable multi-view panorama generation with text (which can be expanded from one room to another). As far as we know, we are the first to propose text-to-multi-view panorama generation. Previous methods are limited to single-view panorama generation. The core reason is the lack of large-scale multi-view panorama datasets, and single-view panorama generation cannot guarantee the multi-view consistency of the generated panorama. A new framework suitable for multi-view panorama generation is needed. To this end, we first construct a large-scale panoramic video-text dataset. Then, we design a multi-view panorama generation framework. First, we propose a diffusion model for text-to-single-view panorama generation (fine-tuned on the newly proposed panorama dataset using LoRA). To achieve multi-view consistency, we derive a spherical epipolar constraint suitable for panoramas and propose the spherical epipolar multi-view panorama generation framework, which can achieve diverse and consistent scalable panorama generation results. We will gladly incorporate all the feedback in the revised version.

1. Summary: As mentioned above, the reviewer seems to have misunderstood the core mission, problem, solution, and core innovation of this paper. Other reviewers (2LuU, 7HQE, Yygs) can understand the content of this paper relatively well. We suggest that the reviewer re-review this paper to make a reasonable evaluation. We are glad to resolve any doubts of the reviewer. This paper first introduces a novel task for text to multi-view panorama generation. So we need to reconstruct a large-scale panoramic video-text dataset for this task and design a novel multi-view panorama generation framework with novel spherical epipolar attention. This method can generate diverse and consistent scalable panorama generation.

2. Lack of Technical Contribution: The paper focuses on text-to-multi-view panorama generation, rather than single-view panorama generation. It is the essential difference between this paper and the existing single-view panorama generation method. We propose a novel panorama video construction and caption pipeline and create a large-scale panorama video-text dataset. Then we design a novel multi-view panorama generation framework with novel spherical epipolar attention.

3. Indoor scene panorama generation: Although this paper focuses on the multi-view panoramic generation of indoor scenes, we also show the results of some outdoor scene generation of this method in the global response, which shows a certain generalization. In the future, we will explore outdoor scene reconstruction to construct large-scale outdoor panoramic video datasets to further improve the generalization of multi-view panoramic generation.

4. LoRA-based fine-tuning: It should be emphasized that Lora-based fine-tuning is a method for text to single-view panorama generation (which is detailed introduced in Section 4.1), and this paper focuses on text-to-multi-view panorama generation (which is detailed introduced in Sec. 4 and 5 and Appendix C). Here we add the details of Lora-based fine-tuning: We first fix the stable diffusion model 1.5 and use Lora for fine-tuning. The fine-tuning dataset is selected from our newly proposed video panoramic-text dataset to form 8508 high-quality text-panorama pairs. The learning rate is 0.0001, the batch size is 32, and 8 A100 80GB are used to train 100 epochs.

5. Any specific challenges or limitations encountered when simulating panoramic images? Our synthetic panoramic video-text dataset is based on Matterport3D, which is reconstructed from the captured real images. Based on this dataset, there will be no domain gap problem at least when generating indoor panoramas. We also try to perform outdoor panora generation, which is shown in the global response. Experiments show that our method has a certain degree of generalization in outdoor panorama generation. Of course, if we can add a more high-quality outdoor dataset, it may make our model more generalizable. We leave it as future work.

6. Dataset quality of the proposed panoramic video-text dataset? releasing date? In the global response, we show the comparison between our dataset and PanFusion dataset. Compared with the PanFusion dataset, our dataset has a clear top and bottom, and the text description is more detailed. Since the complete dataset is very large, we plan to release the train/test dataset and open source a complete panoramic video construction and caption pipelines upon acceptance, so that the community can create large-scale panoramic video-text datasets according to their needs. We will also release the complete framework upon acceptance so that the community can use and follow our work.

7. Closer to benchmark and dataset track? No, as mentioned above, we propose a novel task of text to multi-view panorama generation and propose novel panoramic video construction and caption pipelines to create a large-scale panoramic video-text dataset. Then we design a novel panorama generation framework, consisting of a single-view panorama generation module and a novel spherical epipolar multi-view panorama generation module. It's not just a dataset; the dataset is just one of our core contributions. This paper is suitable for the main paper track.

8. Scalability and generalizability: We demonstrate panoramic video generation and outdoor panorama generation, which shows that our method has good scalability and generalization ability.

9. Typos: We will polish the writing and proofread the revised version, including the typos.

Dear Reviewer K8Nd,

We hope that our rebuttal addresses your questions and concerns. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.

It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Best regards,

DiffPano Authors

We thank reviewer 7HQE for his constructive suggestions and appreciate comments like “pioneer work in scalable multi-view panorama generation", "diverse and rich panoramic video-text dataset promote the community of panoramic generation", "compact and practical framework", "well-structured and easy to follow", "plausible and visually pleasant panoramic results". We will gladly incorporate all the feedback in the revised version.

1. Text descriptions between previous panoramic datasets (PanFusion) with our proposed panoramic video-text dataset: PanFusion uses BLIP2 to directly generate text descriptions for panoramas. It only has four or five words and is very concise. The CLIP Score (CS) cannot reflect the accuracy of the text description, and the PanFusion dataset has the problem of blurry top and bottom. In contrast, our panoramic video dataset construction pipeline first generates text descriptions for perspective images using BLIP2, then uses LLM to summarize, which can obtain more detailed text descriptions. At the same time, the top and bottom of our panoramas are clear, and the dataset is larger (millions of panoramic keyframes ). We also provide the camera pose of each panorama, the corresponding panoramic depth map, etc. We show the difference between the PanFusion dataset with our dataset in the global response.

2. Consider cylinder convolutions or padding to maintain left-right continuity: As suggested, we compare cylinder padding without data augmentation and the data augmentation method used in this paper to improve the left-right consistency. Experiments of 1092 samples show that our simple data augmentation scheme can obtain higher CLIP Score (CS) and Inception Score (IS), as well as lower Fréchet Inception Distance (FID). To further verify the consistency, we propose a new left-right consistency evaluation method. First, we simply cut the generated panorama into left and right images, and then use the image-matching algorithm to calculate the correspondence. The number of correspondences is called as Left-Right Matching Pairs (LRMP) metric. Experiments show that our method can obtain more correspondence. | Methods | IS↑ | FID↓ | CS↑ | LRMP↑ | |:-------:|:----:|:----:|:-------:|:-------:| | Circular Padding | 3.81 | 68.31 | 23.51 | 143 | | Data Augmentation | 4.07 | 61.43 | 23.58 | 149 |

3. Specific metrics for panorama, such as Left-Right Consistency Error (LRCE): The Left-Right Consistency Error (LRCE) metric proposed by PanoFormer evaluates the accuracy of panoramic depth estimation by comparing the gap between the leftmost and rightmost horizontal gradients of the predicted panoramic depth map and the ground-truth panoramic map. In the panorama generation task of this paper, we do not have the ground-truth panoramic depth, and this method does not generate panoramic depth. The LRCE metric is not applicable to the work of this paper. Instead, we propose a Left-Right Matching Pairs (LRMP) metric to calculate the left-right consistency by using matching algorithms. We follow Text2Light to report Fr´echet´s Inception Distance (FID) and Inception Score (IS) on panoramas to measure realism and diversity. Additionally, the CLIP Score (CS) is used to evaluate the text-image consistency. While FID is widely used for image generation, it relies on an Inception network trained on perspective images, thus less applicable for panorama images. We adopt a variant of FID customized for panorama, Fr´echet Auto-Encoder Distance (FAED) in PanFusion, which is used to better compare realism. The quantitative results are as follows: | Methods | FID↓ | IS↑ | CS↑ | FAED↓ | Inference time↓ | LRMP↑ | |:-------:|:----:|:----:|:-------:|:-------:|:-------:|:-------:| | Text2Light | 76.5 | 3.60 | 27.48 | 97.24 | 80.6s | 127 | | PanFusion | 47.62 | 4.49 | 28.73 | 6.36 | 15.1s | 133 | | Ours | 48.52| 3.30| 28.98| 10.70| 5.1s| 149

Experiments show that our method achieves the best CLIP Score, the best left-right consistency, and the fastest inference speed, which makes scalable panorama generation possible. Although our method is slightly weaker than PanFusion in FID, IS, and FAED metrics, it cannot obtain clear images due to the blur at the top and bottom, and the complex network makes its inference speed too slow.

4. Extended to panoramic video generation? Yes, our method can be extended to panoramic video generation. This paper mainly studies the method of text-to-multi-view panorama generation. We have recently studied the method of image-to-multi-view panorama generation. To generate panoramic videos, we can first generate multi-view images of different rooms with large pose changes conditioned on the text, and then run the image-to-multi-view panorama generation conditioned on the generated multi-view panoramas, so that it can be expanded to generate longer panoramic videos. We will add this part to the revised version. We show some panoramic video effects in the pdf of the global response. It is recommended to open it with Adobe reader software so that you can see these videos.

5. typos: We will polish the writing and proofread the revised version, including the typos.

Dear Reviewer 7HQE,

Thank you again for your review. We hope that our rebuttal addresses your questions and concerns, such as the motivation for constructing the panoramic video-text dataset, circular padding vs data augmentation, and specific metrics to evaluate the panorama model and panoramic video generation. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.

We have provided an anonymous link to the AC to forward the link to the reviewers, due to the requirements during the rebuttal stage. This anonymous link contains the difference between PanFusion's dataset and our proposed dataset, panoramic video generation, etc.

It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Best regards,

DiffPano Authors

We thank the reviewer 2LuU for his constructive suggestions and appreciate comments like "well-written", "easy to follow", "first work for text to mv panorama", "valuable panoramic video-text dataset", "novel and technically sound approach of spherical epipolar-aware diffusion", "effectiveness in generating scalable, consistent, and diverse panoramic images". We will gladly incorporate all the feedback in the revised version.

1. The main concern of reviewer 2LuU is the generalizability of DiffPano and whether it can generate outdoor scenes. To our knowledge, there is no outdoor multi-view panoramic video dataset. Although DiffPano is currently only trained on the indoor multi-view panoramic dataset, as suggested by the reviewer, we try to generate outdoor panoramic scenes. In the global response, we generated some outdoor multi-view panoramas of DiffPano, which show that DiffPano has good generalization. Of course, it is not perfect. We can explore the 3D reconstruction of outdoor scenes and put the 3D scenes into the panoramic video-text data pipeline to create a large-scale outdoor panorama dataset to improve the generalization of DiffPano further. We leave this as future work.

2. Reviewer 2LuU is also concerned about “why generating multi-view ERP panoramas is meaningful”. A single-view panorama can only represent local areas, missing many parts of the scene. A multi-view panorama is controllably generated with camera poses, which can generate the missing parts of a single view. We visualized the difference between multi-view and single-view in the global response.

3. As for the use cases of this work, we support single-view panorama generation or multi-view panorama generation with text prompts and camera poses. We additionally provide a generation framework with image conditions, which can further achieve the effect of panoramic video generation, as shown in the global response. We will add it to the revised version.

4. Reviewer 2LuU would like to know about the downstream applications of multi-view panoramas. Multi-view panoramas have many applications, such as room roaming in VR (reviewer Yygs recognized this application), interior design preview, embodied intelligent robot exploration, etc. For example, in VR applications, existing single panorama generation can only obtain local information about scenes, leading to limited roaming scapes, while multi-view panorama generation can achieve unlimited roaming from one room to another.

5. Typos: We will polish the writing and proofread the revised version, including the typos.

Dear Reviewer 2LuU,

Thank you again for your review. We hope that our rebuttal could address your questions and concerns, such as the generalizability of DiffPano, why generating multi-view ERP panoramas is meaningful, and the downstream applications of multi-view panorama images. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.

We have provided an anonymous link to the AC to forward the link to the reviewers, due to the requirements during the rebuttal stage. This anonymous link which contains outdoor panorama generation and single-view vs multi-view panoramas.

It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Best regards,

DiffPano Authors

Dear Reviewer 2LuU,

Thank you again for your review. We hope that our rebuttal could address your questions and concerns, such as the generalizability of DiffPano, why generating multi-view ERP panoramas is meaningful, and the downstream applications of multi-view panorama images. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.

We have provided an anonymous link to the AC to forward the link to the reviewers, due to the requirements during the rebuttal stage. This anonymous link contains outdoor panorama generation and single-view vs multi-view panoramas.

It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Best regards,

DiffPano Authors

Dear Reviewer 2LuU,

Thank you again for your review. We are glad that our rebuttal has addressed your concerns. We will discuss more in the revised version of the use case of the multi-view panorama and include more related work(such as recent single-view panorama generation work[1, 2, 3, 4, 5, 6]) in panoramic 3D scene generation.

[1] Zhou, Haiyang, et al. "HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions." arXiv preprint arXiv:2407.15187 (2024).

[2] Li, Renjie, et al. "4K4DGen: Panoramic 4D Generation at 4K Resolution." arXiv preprint arXiv:2406.13527 (2024).

[3] Wu, Tianhao, Chuanxia Zheng, and Tat-Jen Cham. "Ipo-ldm: Depth-aided 360-degree indoor rgb panorama outpainting via latent diffusion model." arXiv preprint arXiv:2307.03177 (2023).

[4] Wang, Jionghao, et al. "360-degree panorama generation from few unregistered nfov images." arXiv preprint arXiv:2308.14686 (2023).

[5] Lu, Zhuqiang, et al. "Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 13. 2024.

[6] Wang, Hai, et al. "Customizing 360-degree panoramas through text-to-image diffusion models." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

Compared with single-view panorama generation, our method can generate consistent multi-view ERP panoramas, which can be transferred to the perspective view for immersive VR roaming with unlimited scapes. There is related work [7] on using single-view panoramas for VR roaming. We can use this work for VR roaming.

[7] DreamSpace: Dreaming Your Room Space with Text-Driven Panoramic Texture Propagation, VR2024.

Additionally, we would like to discuss the use of multi-view panoramas for interior home design. Given a floor plan of a house, users can customize different rooms throughout the house according to their needs. Then given a textual description of the tour from different rooms (similar to the specified trajectory and textual description in our panoramic video-text pipeline), our method can be used to generate multi-view panoramic videos. In this way, users can preview their envisioned home design.

More importantly, the single-view panoramic image generated by previous methods mainly supports 3DoF roaming. Our method generates multi-view panoramic images that support 6DoF roaming, serving as inputs for 360-degree Gaussian Splatting[8] or 360-degree NeRF[9, 10]. Our method also has a great potential value in relightable 360-degree NVS with the combination of 360-degree multi-view inverse rendering method [11].

[8] Xiong, Haolin. SparseGS: Real-time 360° sparse view synthesis using Gaussian splatting. MS thesis. University of California, Los Angeles, 2024.

[9] Huang, Huajian, et al. "360Roam: Real-Time Indoor Roaming Using Geometry-Aware ${360^\circ}$ Radiance Fields." arXiv preprint arXiv:2208.02705 (2022).

[10] Chen, Zheng, et al. "PanoGRF: generalizable spherical radiance fields for wide-baseline panoramas." Advances in Neural Information Processing Systems 36 (2023): 6961-6985.

[11] Li, Zhen, et al. "Multi-view inverse rendering for large-scale real-world indoor scenes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

We hope our reply could address your questions. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.

It would be appreciated if you could recommend accepting our paper in the reviewers-pc discussion. We thank you again for your effort in reviewing our paper.

Best regards,

DiffPano Authors