CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation

We thank Reviewer FHAm for the detailed review. In the following, we try to assess all questions and concerns.

Regarding the first point:

Thank you for requesting a deeper analysis of how discontinuities are addressed. This makes sense. We had in fact visualized the overlapping predictions in different views in Figure 4 (e.g., the duplicated fireplace in the right and back views). But we agree this should be made clearer in the text, and the illustrations. We also want to refer to the appendix, where the faces are visualized in Figure 10 and 11. The overlaps are clearly visible there and illustrate how predictions are managed in these regions.

Discontinuities at the boundaries of the overlap do not matter, these parts are discarded. The important point is that the pixels within the overlap area are in perfect agreement, such that cropping yields seamless transitions between adjacent cube faces. In our experiments we find that the model reliably predicts these overlaps as it should, we have not observed any issues with discontinuities.

To clarify, the VAE is not fine-tuned at all, it remains frozen throughout. It is exclusively the diffusion loop in latent space that is responsible for generating correct overlaps. Full attention is critical in this context, as it captures global contextual relations between pixels in different views, ensuring consistency.

Regarding the second point:

You are right, we should have included those ablations and have now added them in the main paper and the appendix. In particular, we qualitatively demonstrate the impact of GroupNorm and of overlapping predictions (quantitative performance metrics are not sensitive to the visually obvious differences, see detailed explanation in our response to Reviewer 1).

1. Fig. 7 in the main paper for the added ablation study is good. Could you merge Fig. 6 and Fig. 7 in the appendix to compare Non-Sync, No-overlap, Yours, and Ground Truth in each row? It is hard to interpret without putting them together.

2. I lowered my score because the rebuttal fails to address my concerns:

   • Overlap prediction. How does the prediction look in its original uncropped version? How does the VAE handle it (e.g., reconstruction quality)? How does the prediction compare to the ground truth?

   • Novelty. Is this work the first to use cubemap for panorama generation? I noticed you cite RoomDreamer, which introduces cubemap for 3D synthesis as well. Please provide a full history of using cubemap for panoramas and compare your work to theirs in the related work section.

   • Reproducibility. Do you use any pretrained LDM? In Line 192, you mentioned adopting an architecture similar to SD, so I assume you introduced a new architecture and trained it from scratch. Why not build upon SD and use the pretrained weights as much as possible? Please also include other important implementation details that might help reproduce your work.

We thank Reviewer mPic for the detailed review. In the following, we try to assess all questions and concerns.

Regarding the first point:

Our test data (e.g., all panoramas in Figs. 1 and Figure 8) was totally distinct from the training data, but we agree that we should have included examples that are more obviously out-of-domain. As requested by the reviewer, we have now included more exotic styles like a panoramic extension of Van Gogh's "Starry Night", and a panorama based on "Alice in Wonderland" in the appendix. Please refer to Figure 9. We thank the reviewer for the suggestion, we feel that it has strengthened the paper and are happy to include the samples in the main paper in an updated version.

Regarding the second point:

Thank you for the feedback. Regarding Tabe 1, there seems to be a misunderstanding. In our work we use intermediate image conditioning, i.e., we first run a conventional text-to-image model to generate one (the "front") face of the cube map, then feed both the generated front face and the text prompt into the panorama generator. As such, both Oursimg+txt and Oursimg+multitxt effectively operate in text-only mode. That approach is consistent with previous work such as CAT3D.

The color scheme served only to highlight the best and second-best scores across all columns, as often done for large tables in machine learning papers. However, we appreciate the suggestion for improved clarity and have updated Table 1 accordingly.

The stylistic panoramas look great!

We thank Reviewer MXUG for the detailed review. In the following, we try to assess all questions and concerns.

Regarding the first point:

This is indeed an important aspect of our paper, thank you for giving us the opportunity to clarify:

Differences to MVDiffusion: MVDiffusion introduces a correspondence-aware attention mechanism, which, while innovative, has notable limitations. Specifically, it operates only within a very limited receptive field (a local neighborhood of 9), and ia initialised with all zeros. This means that attention weights must be trained from scratch, consequently they are prone to overfitting and do not generalize all that well. In contrast, the attention mechanism of CubeDiff spans all six frames at their full spatial extent: each latent can attend to every other latent in the feature map, leading to a a much more comprehensive contextual understanding. What is more, for CubeDiff we only adjust the token length while preserving the channel size, which means that it can exploit pretrained weights learned from internet-scale data. We point the reviewer to the ablation studies in the MVDream paper, which show that inflating the 2D self-attention layer to 3D significantly reduces overfitting and improves generalization, overcoming a main design limitation of MVDiffusion.

Comparison with MVDream: MVDream explores three different architectural strategies, including the addition of a separate 3D self-attention layer on top of the existing 2D self-attention layer. However, this additional layer must again be trained from scratch rather than leverage pretrained weights, leading to overfitting. The ablation studies in the MVDream paper demonstrate that, to inflate the 2D self-attention layer to 3D context, it is better to extend the token sequence length, such that pretrained weights can be used. For CubeDiff we adopt this approach. Given that our choice is informed by the earlier findings of the MVDream authors we chose not to repeat this specific ablation study.

Regarding the second point:

Thank you for bringing PanFusion to our attention. We agree that it is an important baseline for panorama generation from text prompts, and have therefore added it to our study. As shown in the revised Tables 1, CubeDiff surpasses PanFusion's performance in 9 out of 10 metrics, including the proposed FAED metric. We also include qualitative comparisons in the revised paper and on our anonymous website to visually demonstrate the superior image quality and generalization of CubeDiff.

We note that CubeDiff differs from PanFusion in several key aspects. PanFusion employs a two-branch pipeline with both a panorama model and a perspective model. To allow cross-branch communication, the authors develop a custom attention module that links equirectangular and perspective projections, which once again must be trained from scratch. It is an important design choice of CubeDiff to set up the architecture such that the model can build on broadly applicable, "foundational" attention weights learned through large-scale pretraining.

Regarding the third point:

Thank you for pointing out the FAED metric, and for the question regarding the FID and KID metrics, this is indeed an important detail.

FID, KID, and CLIP-FID Evaluation: We followed a similar protocol as existing works and measure them on 10 random perspective views extracted from each generated 360° panorama (excluding the input image). In this way, the metrics are computed on the expected image format and allows for a fair comparison across models.

FAED Evaluation: We evaluated FAED using PanFusion’s autoencoder pretrained on Matterport3D. As shown in Table 1, CubeDiff consistently outperforms other models in FAED, highlighting its ability to generate realistic, high-fidelity panoramas.

Regarding the fourth point:

We agree that we should have better supported the decision to employ synchronized group normalization and overlapping predictions. We have now included qualitative results in the manuscript to demonstrate the impact of these components.

Quantitative comparisons for these components are challenging: their main benefit is to prevent subtle color shifts and discontinuities at the border of cube faces, but existing metrics (including FAED) are not sensitive to these artifacts; whereas they stand out for human perception and markedly degrade visual quality. Hence, visual inspection offers a more meaningful insight into the effects of these components. More results can be found in the appendix.

Overall:

We are glad the reviewer recognizes the novelty and potential of our cubemap-based approach. CubeDiff’s inflated attention mechanism is key to its performance, enabling spatial coherence and leveraging generalizable, pretrained attention weights. We revised the manuscript to emphasize these points and better explain CubeDiff’s strengths. We hope our revisions address the reviewer’s concerns.

We thank Reviewer oYCw for the detailed review. In the following, we try to assess all questions and concerns.

Regarding the first point:

We agree that we should have better supported the decision to employ synchronized group normalization and overlapping predictions. We have now included qualitative results in the manuscript to demonstrate the impact of these components.

Quantitative comparisons for these components are challenging: their main benefit is to prevent subtle color shifts and discontinuities at the border of cube faces, but existing metrics (including FAED) are not sensitive to these artifacts; whereas they stand out for human perception and markedly degrade visual quality. Hence, visual inspection offers a more meaningful insight into the effects of these components. More results can be found in the appendix.

Regarding the second point:

We concur that the pieces of the puzzle -- cube maps, full multi-frame attention, extending the token sequence length to keep extensively pretrained weights, synchronized group norm -- were all there. But we are adamant that the simplicity of our design is a strength rather than a weakness. Our goal was efficient and high-quality panorama generation, and we feel CubeDiff achieves that without cluttering the network diagram with arbitrary custom modules and branches. We are quite pleased that we found a way to assemble the right components into a clean architecture without bells and whistles, which, in hindsight, looks obvious.

Regarding the third point:

We respectfully disagree with the assertion that the comparisons are unfair. The goal was in-the-wild deployment, i.e., a zero-shot setting where the characteristics of the input image are not know in advance. All models, including CubeDiff as well as the baselines (except Diffusion360 and OmniDreamer), were evaluated in that same setting, i.e., on out-of-distribution data that none of them had been specifically trained for. In our view, this is an entirely fair comparison: the target is a method that works out of the box for most reasonable inputs, and the evaluation protocol emulates that setting. This form of evaluation is accepted standard practice in computer vision, and in machine learning at large.

Besides the fact that the responsibility to choose suitable training data lies with the respective authors, re-training baselines on the same training set is not feasible, since some do not make their training code public. Retraining only a subset would, if anything, make the comparison more skewed.

Hi Reviewer oYCw and Reviewer FHAm,

thank you for catching this error. We have updated the website to display the correct conditioning images, and we appreciate your attention to detail. We believe that the OOD examples strongly demonstrate the effectiveness of our method.

Regarding your other points, we want to assure you that we are actively working on addressing them. We have been dedicating significant effort to this rebuttal, focusing on providing the additional experiments, evaluations, and results reviewers have asked for, e.g. evaluating another baseline and even an additional metric for alle baselines and our method. Currently, we are training the models you asked for with limited data and will share the results later today.

We kindly ask for your patience, as training and evaluating these models take some time. Thank you again for your thorough feedback and understanding.

Best regards The Authors

Hi all,

Thanks for the really valuable discussions! First of all, I really like this work — it is very simple and seems highly effective. However, to be honest, I feel the authors seems failed to spend enough time on tackling some critical issues during rebuttal. That being said, I still vote for the acceptance of this work due to its simplicity and effectiveness.

I fully agree with reviewer oYCw regarding the comparisons. First of all, the initial version lacks too many ablation studies. From the current version after the rebuttal, we still do not see the critical ablation study that demonstrates the strengths of this work and highlights the most important components of its contributions. There should be an ablation study the same architecture same training but without cubemap generation (which appears to be the most important component). Additionally, the authors should provide a baseline for an extremely simple version of the cubemap model without any other novel designs (e.g., groupnorm, overlap prediction, 3D attentions). This would serve as a baseline to show the importance of your other technical details and how it outperforms the simpler baseline.

Regarding the training difference with MVDiffusion and other works, it makes sense. I feel adding ablation without cubemap generation can address this concern, and I feel this is the reason why reviewer asked this question: We do NOT know the quality improvement comes from the most important component (Cubemap), or from the training difference. The ablations provided now are just minor things. I strongly these ablations during rebuttal or at least in camera ready if the paper accepted).

Lastly, I really want to know the history of using cubemaps for panoramas, as I mentioned in my review but did not get an answer. This would help establish the novelty of your work.

Hi authors,

I want to follow up on your updated OOD results in your anonymous project page: the conditioning image seems to be a panorama by itself rather than a perspective image, is it intentional or something went wrong there? Also, I wonder if you have further updates on the first and third points in our discussion.

I didn’t mean to start an argument—perhaps I miscommunicated. We are not trying to block your paper or ignore your efforts. I simply want to ensure that the value of your work is clearly visible through well-presented ablation studies and a detailed account of its history and your unique contributions. It was not very clear, but now it is getting better and better.