Conditional Panoramic Image Generation via Masked Autoregressive Modeling

Thank you for recognizing our strengths: 1. Clearly written paper. 2. Reasonable design choice. 3. Strong performance. 4. Thorough ablation studies. We provide more clarifications below.

Q1: The difference with StitchDiffusion regarding the proposed dual-space circular padding strategy.

A1: At a high level, both methods aim to mitigate the inconsistency between the left and right ends of the panorama. However, our proposed dual-space circular padding is fundamentally different from StitchDiffusion[1].

1. From methodology, StitchDiffusion focuses on the denoising model, but our method focuses on VAE. StitchDiffusion pads the latent and then denoises twice on the stitch area, which increases the huge computation cost. On the contrary, the extra cost brought by our method is negligible (please refer to our answer for Reviewer Pih7's Q2).

2. From motivation, StitchDiffusion forces the model to fuse seam areas during inference. On the contrary, our pre-padding makes the model learn semantic consistency during the training phase, and our post-padding achieves pixel-level consistency in the inference stage.

3. From the results, we argue that simply focusing on pixel consistency cannot solve the problem of semantic inconsistency, which is demonstrated in Fig.8 of our main manuscript. Moreover, Semantic inconsistency (houses of different heights and colors) is observed in Fig.7c of StitchDiffusion's manuscript.

Q2: The reason for choosing the outpainting baselines.

A2: We select two peer-reviewed works with publicly available code, AOG-Net[2] and 2S-ODIS[3], as the baselines for the Matterport3D dataset. We choose PanoDiff[4] and Diffusion360[5] as OOD baselines because the model weights trained on the SUN360 dataset are provided in the official repo. Following the reviewer's suggestion, we further reproduced PanoDiff and compared it with our 1.4B model on panorama outpainting task. The training dataset is Matterport3D and the text prompt is set to empty. Our model has advantages in terms of FID metric.

Q3: Comparisons with the methods whose codes are unavailable.

A3: We summarize the code and model availability for the mentioned methods in the table below:

As shown, both DiffPano[6] and CubeDiff[7] do not provide publicly available code or pretrained model weights, which makes direct quantitative comparison challenging. Despite this limitation, we have carefully reproduced the DiffPano pipeline according to their paper and compared with our 1.4B model on text-to-panorama task. The training dataset is Matterport3D. The results are as follows, and our model has advantages in terms of FID metric.

Q4: Visualizations of perspective projections.

A4: In Tab.2, our method outperforms baseline methods on the FID-h metric. For FID-h, we horizontally sample 8 perspective images with 90-degree FoV for each panorama and then calculate their FID. As additional visualization results are not permitted in the rebuttal stage, we will add them in the next draft.

Q5: The overuse of abbreviations seems to harm its readability.

A5: We apologize for the inconvenience. The abbreviations for tasks and metrics are introduced in L78-79 and L207-213, respectively. We will try to make the explanations more explicit in the next draft, including using full names where possible, or providing a brief introduction to the abbreviations in the captions.

Q6: Visual overview of the proposed framework. How the input and output of the "Transformer f" are structured.

A6: Thanks for your suggestion. Our "Transformer f" follows NOVA[8] and is described in Sec.3.3. Briefly, it takes patchified visual tokens (from the VAE encoder), masking indicators, and text embeddings as inputs. It outputs a conditional signal to drive the subsequent denoising network MLP. We will add a visual overview of the framework in the next draft.

[1] Customizing 360-Degree Panoramas through Text-to-Image Diffusion Models, WACV 2024.

[2] Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation, AAAI 2024.

[3] 2S-ODIS: Two-Stage Omni-Directional Image Synthesis by Geometric Distortion Correction, ECCV 2024.

[4] 360-Degree Panorama Generation from Few Unregistered NFoV Images, ACM MM 2023.

[5] Diffusion360: Seamless 360 Degree Panoramic Image Generation based on Diffusion Models, Arxiv 2023.

[6] DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion, NeurIPS 2024.

[7] Cubediff: Repurposing diffusion-based image models for panorama generation, ICLR 2025.

[8] Autoregressive Video Generation without Vector Quantization, ICLR 2025.

Thank you for recognizing our strengths: 1. Unified framework. 2. Detailed experiments and ablation studies. We provide more clarifications below.

Q1: Inference time comparison with other model.

A1: We compare PanFusion[1] with our 0.3B model using one NVIDIA A100 GPU with a batch size of 8. The AR steps are 64, and the denoising steps for MLP are 25.

Q2: Inference time regarding circular padding.

A2: In the inference stage, the circular padding only participates in the VAE decoding phase, which transforms latents into pixels. This means that circular padding does not bring about additional computational cost in the transformer and MLP, which account for the majority of inference time.

The table below shows the inference speed for different padding ratios with AR step=16.

Please note that:

1. The proportion of time occupied by circular padding can further decrease as AR steps increase.

2. From Fig.8 in the main manuscript, the synthesized results with a padding ratio of 0.0625 are close to convergence.

Q3: Is there any plan to improve the outdoor generation result?

A3: Thanks for your suggestion. Yes, we plan to curate and annotate a large-scale, high-resolution outdoor panoramic dataset with diverse scenes and lighting conditions. This dataset will help reduce the domain gap and provide richer context for model training. Additionally, we are considering incorporating domain-specific data augmentation and leveraging pretraining on related outdoor datasets to enhance generation quality further. We believe these steps will benefit outdoor scene synthesis in future work.

[1] Taming Stable Diffusion for Text to 360 Panorama Image Generation, CVPR 2024.

Thank you for recognizing our strengths: 1. Innovative approach. 2. The paper is well-written, and the experiments, including the chosen baselines and metrics, are plausible. 3. Strong spatial consistency. 4. Strong Generalization. We provide more clarifications below.

Q1: Lack photorealistic quality compared with CubeDiff and 3D Worlds.

A1: First, as the code and models of CubeDiff[1] and 3D Worlds[2] are unavailable, we carefully check the visualization results in their manuscript and project page. Incorrect generation results also exist. Examples include the unrealistic desk (CubeDiff project page, Living Room), twisted table (CubeDiff project page, Alice 2), peach tree on the wall (3D Worlds, Fig.1), and blurred and distorted walls (3D Worlds, Fig.1).

Second, CubeDiff and Generating 3D Worlds focus on the outpainting task, in which the ground truth is projected into the center of the panoramic image, which can easily produce a more realistic feeling to human eyes.

Third, the training data volume. We use the Matterport3D dataset, which contains 9k images for training. On the contrary, CubeDiff collects around 48k images from multiple data sources, and 3D Worlds is trained on DL3DV-10K, which includes 51.2 million frames.

Q2: Blurry regions are present at the poles.

A2: Blurry regions at the poles is an inherent property of the Matterport3D dataset. In other words, all ground truth images in the Matterport3D dataset have blurriness near the poles. This phenomenon is also visible in the visualization results of PanFusion[3].

Q3: Details and ablations of the denoising MLP.

A3: We follow NOVA[4] for the construction of the denoising MLP. The MLP has multiple stacked blocks. Within each block, the input is first processed by AdaLNZero to get gating coefficients. It is then transformed using a projection layer consisting of two linear layers with silu activation. The result is then LN-normalized and multiplied by the gating coefficients, and finally combined with a residual connection. We use the Euler solver with 25 denoising steps.

Here is the ablation study for different denoising steps with our 0.3B model. Few steps lead to integral approximation errors, while too many steps lead to cumulative errors.

Q4: Whether the encoder and decoder are trained on ERP panoramas.

A4: We use the default VAE encoder and decoder from NOVA without any specific fine-tuning. During the training phase, VAE parameters are frozen, only transformer and MLP parameters are trainable.

Q5: The mechanism of patchification in L156.

A5: We employ the patchification operation similar to ViT [5]. Specifically, after obtaining the latent feature maps from the VAE encoder, we apply a 2x2 convolution with a stride of 2. This operation converts spatial feature maps into a sequence of tokens, serving as an input for the subsequent transformer encoder.

Q6: Discussions on the advantages and disadvantages of autoregressive modeling compared to multi-view diffusion models for panorama generation.

A6: Advantages:

1. Computational Efficiency: Multi-view diffusion models require generating several perspective images (e.g., six views of 512x512 to cover a 512x1024 panorama), resulting in around 9 times computational cost due to self-attention.

2. Global Consistency: Coordinating multiple views demands complex attention mechanisms to ensure coherence, and struggles to maintain a consistent global understanding — sometimes leading to duplicated or inconsistent content.

3. Fewer Seam Issues: Multi-view approaches must handle more seams at the boundaries between views, which increases the difficulty of achieving seamless stitching and can introduce visible artifacts. AR modeling generates the panorama as a whole, inherently avoiding these seam problems.

Disadvantages:

Since our model directly generates the entire panoramic image, it needs to use panoramic data to drive the pre-trained model (trained on perspective images) to adapt to the panoramic domain knowledge, which might be less efficient than multi-view modeling because it directly handles perspective images.

Moreover, research in AR-based panorama generation is still limited, and optimal solutions for this task remain to be discovered. We hope our work can inspire further progress in this direction.

We will fruther include these comments in the refined draft.

[1] CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation, ICLR 2025.

[2] A Recipe for Generating 3D Worlds From a Single Image, Arxiv 2025.

[3] Taming Stable Diffusion for Text to 360 Panorama Image Generation, CVPR 2024.

[4] Autoregressive Video Generation without Vector Quantization, ICLR 2025.

[5] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021.

Thank you for recognizing our strengths: 1. The proposed dual-space circular padding and translation consistency loss are novel and well-motivated. 2. The text is well-structured. The tables and figures are clear and easy to follow. 3. Strong performance. We provide more clarifications below.

Q1: Failure cases analysis may improve the paper.

A1: Thanks for your suggestions. Since visualization results cannot be provided in the rebuttal stage, we will add the failure cases analysis in the next draft. According to the visualizations in the main manuscript, the generated images may present unrealistic results in some parts, and there is still a gap compared with real photos, such as some details and textures. This is a common problem of generative models. We believe that scaling data can further improve the generation quality, and we leave it as future work.

Q2: Comparison with Omni2.

A2: As the data, code, and models of Omni2[1] are unavailable, we compare Omni2 with our PAR from two perspectives: dataset construction and modeling. For dataset construction, Omni2 requires complex data engineering, including the collection of over 60k images for various tasks, the application of GroundedSAM and LAMA, with multiple rounds of manual selection for data filtering. On the contrary, we use Matterport3D without specific data preprocessing.

From a modeling perspective, multi-view generation (Omni2 utilizes six perspective views of a cube) presents the following challenges:

1. Higher computational cost. Using six 512x512 perspective views of a cube instead of a single 512x1024 panorama results in approximately nine times the computational overhead due to the self-attention mechanism.

2. Poorer global understanding. Due to the existence of multiple views, the model is prone to duplication and requires more complex mechanisms to achieve harmony across the entire scene.

3. More seam issues. Multi-view approaches must handle more seams at the boundaries between views, which increases the difficulty of achieving seamless stitching and can introduce visible artifacts.

Q3: Training and inference time.

A3: Training time: It takes about 18h to train our 0.3B model, and 2 days for the 1.4B model on a single node with 8 NVIDIA A100-80G GPUs.

Inference tine: We benchmark the inference speed for the 0.3B model with different autoregressive steps using one NVIDIA A100 GPU with a batch size of 8. The denoising steps for MLP are 25.

Q4: CLIP score compared to PanoLlama.

A4: Our model is slightly inferior to PanoLlama in terms of CLIP score, which we argue is due to the gap in panoramic geometry and CLIP's domain knowledge. In Fig.3, the generated results of PanoLlama are more similar to perspective images, while ignoring the panoramic geometry. Since CLIP is pre-trained on massive perspective images, it might have a preference for perspective views.

[1] Omni2: Unifying Omnidirectional Image Generation and Editing in an Omni Model, ACM MM 2025.