Spherical-Nested Diffusion Model for Panoramic Image Outpainting

Many thanks for your positive opinion and valuable comments!

• Tab.1 - 4 are in https://anonymous.4open.science/api/repo/tabl/file/T.pdf?v=5327f2ef
• Fig.1 - 9 are in https://anonymous.4open.science/api/repo/figu/file/F.pdf?v=2d1be253

1 Supplementary Material: Further Analysis on Fig. A. 3

The randomness in our SpND model consists of two components: the structural prior $\mathbf{P}\_\text{ERP}$ from spherical noise and the diffusion noise $\mathbf{Z}$. Fig. A. 3 in Supplementary-A illustrates the impact of these two components, in which spherical noise minimally affects the semantic content of generated images, maintaining structural consistency. In contrast, diffusion noise drastically changes the output, creating distinct visual features. This indicates spherical noise serves as a structural prior, preserving geometric constraints, while diffusion noise controls semantic content generation. These observations support our formulation of $\mathbf{P}\_\text{ERP}$ as a mechanism to regularize the latent space without disturbing semantic attributes.

2 Relation to Broader Scientific Literature: Relation to Non-Isotropic Diffusion Models

The noise in our SpND method follows non-isotropic Gaussian distributions, with the non-isotropic property arising from the ERP operation applied to spherical noise. In contrast, Ref. [1] introduces a critically-damped Langevin diffusion process to enhance network convergence by adding an auxiliary velocity variable, while Ref. [2] generalizes score-based denoising diffusion models with non-isotropic Gaussian Free Field (GFF) noise, improving generation quality on CIFAR10. However, these methods, with their particular non-isotropic settings, are not designed for panoramic image outpainting. For instance, GFF noise in Ref. [2] defines the covariance between sampled values $f(\theta, \phi)$ and $f(\theta', \phi')$ as: $\left(\text{Cov}(f(\theta,\phi), f(\theta',\phi')) \approx -\log\|(\theta,\phi)-(\theta',\phi')\| \right) + C,$where $C$ is a constant. In contrast, for our spherical noise, when $\phi=\frac{\pi}{2}$, the covariance simplifies to: $\text{Cov}(f(\theta,\frac{\pi}{2}), f(\theta',\frac{\pi}{2})) = \sigma^2.$Thus, the constant covariance $\sigma^2$ contradicts the logarithmic dependence required by Ref. [2], demonstrating that the non-isotropic noise in our SpND method cannot be generalized by existing models. This highlights the novelty and contribution of our approach in panoramic outpainting.

3 Weakness 1: More Visual Results

Yes, outpainting tasks heavily rely on perceptual evaluations. We have added more visual examples for all comparisons discussed in our experiments, including the main comparisons, ablation study and view-image outpainting experiments, in Figs. 5-8 in the link. The results contain ground truth, masked input, all baseline methods, and our SpND method for both the Matterport3D and Structured3D datasets, which consistently demonstrate the superior performance of our SpND method.

4 Weakness 2: Evaluations on Distorted Masks

To further evaluate our method on different masks, we conducted additional experiments on masks near the poles, where spherical distortion becomes evident. Specifically, we projected a $256 \times 256$ image onto the ERP format at $\theta=90^\circ, \phi=60^\circ$ to create a highly-distorted mask near the pole. We then retrained our SpND method, along with the second and third best baseline methods (PanoDiff and PanoDiffusion). Results reported in Tab. 4 and Fig. 9, confirm the superior performance of our SpND method, highlighting the advantages of spherical noise and the SDC operation in accommodating panoramic deformation.

5 Questions For Authors 1: Mask Location and Evaluation Angle

Yes, in our original experiments, the images were masked only around the center during training, corresponding to Fig. 4-(a) in our manuscript. For the masked input, the region containing image content spans an elevation angle range of approximately $[-45^\circ, 45^\circ]$, within a full image that has an elevation angle range of $[-90^\circ, 90^\circ]$. We further evaluated our SpND method's ability to generalize to various non-centered masks, as detailed in Rebuttal 6 below.

6 Questions For Authors 2: Generalization to Varying Masks

Yes, our SpND model can generalize to varying mask formats. We re-trained our SpND method with the entire input masked and show the subjective results in Fig. 4. While performance was reduced, our SpND method achieved an FID of 29.65 and successfully performed panoramic generation, confirming its generalization ability. Additionally, our SpND method is highly flexible with multiple input views and we refer to our Rebuttal 4 above and our Rebuttal 3 to Reviewer Qs7j.

7 Questions For Authors 3: More Possible Applications

Many thanks! For more possible applications, we refer to our Rebuttal 1 to Reviewer GNkK.

Many thanks for the positive comments and insightful suggestions.

• Tab.1 - 4 are in https://anonymous.4open.science/api/repo/tabl/file/T.pdf?v=5327f2ef
• Fig.1 - 9 are in https://anonymous.4open.science/api/repo/figu/file/F.pdf?v=2d1be253

1 Claims and Evidence 4: Elaboration on One-to-Many Mapping

The one-to-many mapping naturally arises from the ERP format of the spherical content. Pixels near the polar regions exhibit strong correlations in the ERP domain, reflecting spatial coherence in this projection. The consistent denoising procedure ensures coherent style and content by propagating the entire image across different views. Controlled by $\zeta$, the one-to-many correspondence reaches its optimum when the correlations between noise and feature/image are similar, as explained in Supplementary-A. When $\zeta$ is too small, blurry artifacts occur due to excessive pixel correspondence. Conversely, when $\zeta$ is too large, spatial coherence decreases. This was verified through two new ablation studies, which is detailed in Rebuttal 2 to Reviewer QS7j.

2 Methods and Evaluation Criteria 2: Applicability of RePaint to Panoramic Generation Methods

Outpainting can be achieved through text-to-panorama and diffusion-based generative methods, with each inference step integrated by the RePaint operation for denoising the full panoramic image. However, MVDiffusion generates $8$ perspective views via a multi-view diffusion model, which are then aggregated into a full panorama using Rodrigues rotation. This makes it infeasible to access each inference step for denoising, preventing the straightforward application of RePaint. In contrast, PanFusion accesses the full latent panoramic features at each inference step, enabling outpainting via RePaint. We modified PanFusion by adding RePaint and present new results in Fig. 3 in the link. As shown, PanFusion's outpainted panoramas exhibit repetitive patterns and poor quality due to latent feature rotation to preserve spherical continuity. In comparison, our SpND uses a masked image as input and consistently achieves the superior panoramic outpainting.

3 Methods and Evaluation Criteria 3: Pixel-space Discrepancy by RePaint

Yes. The discrepancy may exist in our SpND method. Most diffusion-based outpainting methods, particularly for panoramic images, operate within the latent space and suffer accuracy degradation in the input mask. For methods using RePaint, we calculated the PSNR$\_\text{cent}$ for the repainted area against the original image, with results reported in Tab. 3 in link. As shown, our SpND method achieves the highest accuracy and quality, which naturally results from our intrinsic SDC operation and spherical noise.

4 Experimental Designs or Analyses 3: Clarifying FID and FID$_\text{hori}$, with Enhanced Metrics

The FID metric is widely used to evaluate generated image quality, and we preliminarily applied it to assess the performance of outpainted panoramic images. The FID$_\text{hori}$ metric evaluates the FID values of 8 perspective view images of size $512 \times 512$, following MVDiffusion. We also incorporated the panorama-specific FAED metric to evaluate the quality of panoramic images. Notably, FAED requires depth information, which is infeasible for outpainting. Therefore, we followed the PanFusion method to compute FAED without explicit depth input. The results, reported in Tab. 3, demonstrate that our SpND method outperforms state-of-the-art outpainting techniques.

5 Relation to Broader Scientific Literature: Applicability to Panoramic Generation Tasks

Yes, we believe injecting non-i.i.d. noise into existing panoramic generation methods could further improve detail retention and the quality of generated panoramic images, including text-to-image methods such as MVDiffusion and PanFusion. Our SpND method can also generalize to generate images from text descriptions using a fully masked input. We evaluated this during the rebuttal and we achieved an FID score of 29.65 by retraining our method without other modifications. Subjective results are shown in Fig. 4.

6 Essential References not Discussed: PanFusion and AOG-Net

Yes, we have analyzed PanFusion by using the RePaint strategy, by Rebuttal 2 above. AOG-Net uses an autoregressive pipeline for 360° outpainting, utilizing feature remapping for panoramic content. We conducted a comparative analysis in Fig. 5 and Tab. 3 against AOG-Net using the Matterport3D dataset. From these, we conclude that our SpND method, by enforcing panoramic deformation through intrinsic convolution and spherical noise, consistently outperforms others in generating high-quality outpainted panoramic images.

7 Other Comments or Suggestions: Training and Inference Procedures

Yes. During inference, we applied iterative RePaint steps based on the trained SpND model, following other outpainting methods such as PanoDiffusion.

We wish to thank the reviewer very much for the positive opinion and insightful suggestions.

• Tab.1 - 4 are in https://anonymous.4open.science/api/repo/tabl/file/T.pdf?v=5327f2ef
• Fig.1 - 9 are in https://anonymous.4open.science/api/repo/figu/file/F.pdf?v=2d1be253

1 Weaknesses: Our Broader Impact.

Yes. Although belonging to the broader fields of panoramic image processing, the panoramic outpainting task still possesses rich applications, especially for nowadays VR/AR applications such as autonomous driving, medical imaging, cultural heritage preservation, media and entertainments, to name but a few. The capability of completing panoramic content from masked views also allows for efficient compression and transmission of large-volume panoramic images, one of the key requirements to achieve meta-universe. Moreover, the newly proposed techniques, such as the spherical deformable convolution (SDC) operation and spherical noise, may also contribute to other panoramic-related tasks, including panoramic generation (as we have newly evaluated in Rebuttal 5 to Reviewer vstk) and content understanding. We believe that this direction is worth investigating with rich juicy unrevealed, in which our method paves a possible way to the field.

2 Other Comments Or Suggestions: Typos

Many thanks! We will modify ''gird'' to ''grid'' and also carefully polish our wordings, by correcting grammatical errors and typos.

3 Questions For Authors: Clarification on Equation (9)

Many thanks for the careful considerations. Yes, the outputs of our spherical net are integrated into the primary U-Net architecture, as depicted in Fig. 3 of our manuscript. Therefore, Equation (9) was misleading and is corrected by:

$$\epsilon\_{\boldsymbol{\psi}} = f\_\text{SD}\left[\mathbf{Z}\_{t}, \mathbf{T}, f\_\text{PDN}\left[\mathbf{F}\_\text{pri}, \mathbf{T}\right]\right]$$

where $f\_{\text{SD}}$ denotes the pre-trained diffusion model and $f\_{\text{PDN}}$ denotes the spherical net, which is embedded into the diffusion model. Moreover, $\mathbf{T}$ is the embedding of the input prompt and $\epsilon\_{\boldsymbol{\psi}}$ is the output of our SpND model. In this way, the spherical net can extract panoramic features that guide the pre-trained diffusion model for panorama outpainting.

Many thanks for the valuable comments!

• Tab.1 - 4 are in https://anonymous.4open.science/api/repo/tabl/file/T.pdf?v=5327f2ef
• Fig.1 - 9 are in https://anonymous.4open.science/api/repo/figu/file/F.pdf?v=2d1be253

1 Weakness 1: Novelty of Our SDC Layer

To the best of our knowledge, the novelty of our SDC layer lies in the first successful attempt of proposing a new basic convolution operation to address the panoramic deformation, the key to achieve high-quality panoramic outpainting. Our spherical noise setting also contributes to our overall novelty in addressing this task. In contrast, existing methods, including those for other tasks such as panoramic image segmentation and depth estimation, typically address the panoramic deformation via soft loss regularization or feature re-sampling strategies. More specifically, Ref. [1] developed existing rectilinear architectures via pre-defined rectilinear-panoramic projection, to achieve 3D object detection and monocular depth estimation. Ref. [2] employed spherical resampling solely for the value feature within Transformer, to enhance panoramic image depth estimation. Ref. [3] added learnable positional encoding cues with existing features that are still processed by standard 2D convolution, for panoramic image outpainting. Ref. [4] regularized the network by spherical contrastive loss to effectively capture panoramic content for the segmentation of 360° images. However, Refs. [1, 2, 4] are incapable of generating panoramic content and thus unsuitable for the panoramic image outpainting task. More importantly, all the existing methods rely on either extra feature re-organization (e.g., Refs [2, 3]) or sphere-assisted losses (e.g., Refs. [1, 4]) to accommodate panoramic images. In contrast, our SDC layer essentially alters the basic convolution operations, which is fundamentally different from the existing strategies adopted for both outpainting and other panoramic-related tasks. Indeed, the fundamental improvement of our SDC layer on the convolution operation is much effective and straightforward to enforce geometry cues in addressing panoramic deformation, thus benefiting remarkable improved quality of outpainted images.

Ref. [1]: Eliminating the blind spot: Adapting 3D object detection and monocular depth estimation to 360 panoramic imagery.

Ref. [2]: Panoformer: Panorama transformer for indoor 360° depth estimation.

Ref. [3]: Cylin-Painting: Seamless 360 panoramic image outpainting and beyond.

Ref. [4]: Panoramic panoptic segmentation: Insights into surrounding parsing for mobile agents via unsupervised contrastive learning.

2 Weakness 2: Quantitative Evaluations on Non-i.i.d. Noise in ERP Format

Yes. Our Non-i.i.d. noise setting caters for the panoramic format during the outpainting, and we further conducted quantitative evaluations as suggested by the reviewer. More specifically, we further ablated two settings by varying sample density $\zeta$, i.e., via $\zeta=15$ and $\zeta=45$, and report the results in Tab. 1 in the link. As shown in the table, our SpND method achieves optimal performance at $\zeta = 45$ and $\zeta = 30$, consistent with the analysis in Supplementary-A, where the ERP noise closely aligns with the ground-truth features/images. Compared to $\zeta = 45$, $\zeta = 30$ offers the highest efficiency. Choosing $\zeta = 15$ slightly degrades outpainting performance but still outperforms existing baselines. These results confirm the rationale and robustness of our proposed non-i.i.d. ERP noise, derived from i.i.d. spherical noise.

3 Weakness 3: Limited Capability Regarding Input Versatility

We agree with the reviewer that outpainting from multiple input views can further improve the practicality and applicability of our method. Indeed, our model can readily accommodate multiple partial-view and overlapping inputs by adjusting the training masks accordingly. More specifically,

• For the multi-input scenario, we newly evaluated our SpND method based on two viewpoints centered at $(\theta_1=-90^\circ, \phi_1=-15^\circ)$ and $(\theta_2=90^\circ, \phi_2=15^\circ)$ within the equirectangular projection (ERP) format, thus denoted as Dual.
• For the overlapping scenario, we newly evaluated our SpND method based on two viewpoints centered at $(\theta_1=20^\circ, \phi_1=0^\circ)$ and $(\theta_2=90^\circ, \phi_2=15^\circ)$ with overlapped regions, also using the ERP format. We denote this as Overlapping.

We report the results in Tab. 2, together with the subjective results in Figs. 1-2 in the link. From this table and figure, we can conclude that our method can generalize to multiple overlapping and partial-view inputs. Note that due to limited rebuttal period, the model was trained by adequate convergence, and performance improvements can be expected with further training. Even though, our SpND method still achieves superior performances for the panoramic image outpainting task.