Response for Reviewer jUcp

Thanks for the positive feedback and thoughtful suggestions. We will address the concerns and questions as below.

Q.1 The visual quality is relatively poor. Given that recent video generation models are capable of producing higher-quality outputs, it would be better if the authors could further explain these limitations.

Thanks for the comment. The visual artifact primarily stems from our backbone, AnimateDiff, which is known to produce temporal flickers in many cases. Despite this limitation, our method improves upon AnimateDiff in both visual quality and temporal consistency, as shown in Table 1. To further address this issue, we plan to upgrade the backbone to a more advanced video generation model (Wan2.1/2.2), with technical modifications from UNet with DiT. This update is expected to significantly reduce visual artifacts and improve overall temporal coherence. We will update our results with the upgraded backbone in the final version.

Q.2 It is unclear whether the baseline models were trained using the same dataset collected by the authors. If not, the comparisons may not be entirely fair, and this should be clarified.

Thank you for the thoughtful suggestion. The results of the baseline models reported in Table 1 are not trained on YouTube360. However, we have included the performance of 360DVD trained on YouTube360 in Table C of the supplementary material.

We agree that it is important to clarify the training data used for each model and to also report baselines trained on the same dataset. We add the results of baselines+LoRA trained on our data (marked with *) in the table below. Most of the metrics improve with our training data, especially EPE, which reflects 360 motion correctness. We will move the results from Table C to the main paper, clarify the training data source in Table 1, and include additional results of other retrained baselines in the final version.

Q.3 Add comparison of training and inference costs to better assess the model’s efficiency and practical applicability.

Thanks for the valuable advice. We provide an inference comparison table in terms of runtime and VRAM below. The experiments are conducted on single NVIDIA A800 card. We report the average runtime based on tests using 16-frame video clips. The memory footprint is evaluated with nvidia-smi.

From the inference cost comparison, we observe that compared to another V2V model Follow-Your-Canvas (FYC), our model is faster in inference speed and consumes less VRAM. The main reason is that FYC requires multiple round of outpainting to ensure outpaint quality, and they also train & infer in float32. Thanks to our dual-branch structure, Imagine360 achieves good visual quality in a single pass of inference. AnimateDiff and 360DVD have faster inference speeds, which is within expectation, as they are text-guided and image-guided frameworks. In addition, by employing multiple memory optimization techniques (bfloat16,vae slicing,cuda.empty_cache, intermediate tensor cleanup, etc.), results show the dual-branch structure incurs only a small and affordable increase in memory usage, while achieving better generation quality and fast inference speed compared to the baselines.

In terms of training cost, AnimateDiff is trained on WebVid10M dataset using 16 A100 GPUs for 5 epochs. Follow-Your-Canvas (FYC) is trained on Panda70M using 8 A800 GPUs for 50K steps. 360DVD is trained on WEB360 (which includes both RGB and optical flow videos) using a single GPU with ~50GB of VRAM for 100K steps. In contrast, our model is trained on a significantly smaller dataset, containing only ~9K panoramic RGB videos, using 8 A100 GPUs for 20K steps. Despite requiring generally less training data and compute, our method achieves superior 360 video generation quality compared to all baselines, demonstrating both data and compute efficiency.

Q.4 Figure 3 overlaps with surrounding text, causing readability issues.

Thanks for pointing this out! We will revise the paper and fix the formatting issues in the final version.

Response to Reviewer opro

Thanks for the constructive comments and inspiring suggestions. We sincerely apologize for not including more qualitative examples due to the NeurIPS rebuttal policy, but we will add them in the final version of the paper. The comments are addressed in detail as follows.

Q.1 The technique contribution is limited. This paper is a simple combination of [1-3] and additionally adds a perspective video as input. As shown in Table 2, the most significant design is the dual-branch denoising outpainting, which has been proposed by [1]. And the rotation-aware design is essentially camera poses augmentation.

Thanks for the comment. We respectfully disagree with the claim that our method is a simple combination of [1–3] with an added perspective video input. These works address different tasks: [1] is text-to-image, [2] is text-to-360° video, and [3] is image-to-video. None of them solves the task of generating 360 videos from perspective video signals. Moreover, handling video-conditioned 360 video generation is nontrivial and requires careful modeling of dynamic input camera motion and 360 motion structure. To address these challenges, we introduce two new components: (1) the rotation-aware designs to improve robustness to changing camera trajectories, and (2) antipodal attention masking to model reversed camera motion in 360 views. As shown in Table 2, both modules lead to clear gains, especially in 360 motion metrics like EPE and OmniFVD.

While we follow [1] in adopting a dual-branch denoising structure, our pipeline is a smart, cohesive integration of relevant generative components, tailored for perspective-to-360 video generation. The proposed modules are key to addressing the unique challenges of this new setup. Additionally, we contribute a clean and high-quality 360 video dataset with temporal interval annotations, which we believe will benefit the community.

We already included a side-by-side comparison with PanFusion in Appendix C, highlighting our differences in task domain, pipeline design, and temporal modeling. We will move this comparison to the main paper for a clearer comparison. We hope the reviewers and AC will consider the novelty of the task, the framework design, and the dataset contribution when evaluating our work.

Q.2 The antimodal mask design needs further verification. In Table 2, using the antimodal mask design brings minor improvements. Also, SUV case in supp. contain strange motion patterns. Suggest providing input and antipodal views for better comparisons.

Thanks for the suggestion.

We would like to clarify that the antipodal mask is designed to address a unique challenge in 360 video generation, that antipodal regions should undergo reverse motion under camera translation. The purpose of this module is not to directly improve image quality or video smoothness, but rather to provide a geometry-aware bias that facilitates global motion modeling on the sphere. In Table 2, we see modest gains on VBench IQ and MS, which is as expected, since these metrics capture overall appearance and temporal smoothness. We also see non-trivial gains on OmniFVD (257.6 → 204.0) and EPE (2.7169 → 2.5583), which reflect 360 motion correctness, the intended benefits of antipodal modeling, indicating improved motion for 360 video generation.

We also acknowledge that antipodal motion remains a challenging issue. While our method does not fully solve it (SUV case), it provides clear improvements. This is supported by quantitative gains on large evaluation data (Table 2). We strongly agree that visualizing the input and antipodal views is a good idea, and we will include more qualitative video results in the final version to further illustrate the effectiveness of this module.

Q.3 Why not directly use MonST3R to predict camera trajectories from enormous monocular videos, and use these predicted camera poses to replace the rotation-aware design during training? By doing this, we can reduce the domain gap, and the predicted camera poses are more realistic.

Thanks for the insightful suggestion. Our current approach employs random trajectory sampling during training to expose our model to diverse trajectory patterns and improve robustness. Sampling from a camera trajectory library collected by MonST3R on massive real videos seems a promising alternative, as it better reflects the real-world camera motion distribution and may reduce the domain gap. While collecting large-scale real videos and extracting camera trajectories from them is hard to accomplish with our current resources in a short time, we will continue to implement and verify this solution through experiments. The results will be reported in the final version.

Response to Reviewer vJmB

We thank the reviewer for acknowledging our pipeline as "a clever adaptation of recent generative modeling ideas to the spherical video domain" and our data contribution as " addresses the training data bottleneck", "valuable". The concerns and questions are addressed in detail as follows.

Q.1 The dual-branch design mostly follows PanFusion, which makes the contribution of this part a bit weak.

Thanks for the comment. As detailed in Appendix C, we provide a side-by-side comparison between our work and PanFusion. Our work differs significantly in several key aspects, including the task domain, the overall pipeline design, and the introduction of temporal modeling.

While we adopt a dual-branch denoising structure, our method goes beyond PanFusion through a cohesive integration of antipodal-aware attention, rotation-aware camera modeling. These designs are specifically tailored for the novel task of perspective-to-360° video generation, which has not been systematically explored before. The effectiveness of each component is validated through ablations (Table 2), and qualitative results demonstrate good generalization to in-the-wild scenarios.

In addition to the method, we also contribute a clean and high-quality 360 video dataset with temporal interval annotations, which we believe will be valuable for future research.

We will move the PanFusion comparison to the main paper to make the differences more prominent. We hope the reviewers and AC will consider the novelty of the task, the thoughtful framework design choices, and the data contribution when evaluating the overall contribution of our work.

Q.2 The dual-branch architecture and rotation-aware modules likely add computational overhead. Add a discussion or table showing runtime comparisons or inference speed for understanding the practical trade-offs.

Thanks for the advice. We provide a runtime breakdown table and an efficiency comparison table in terms of runtime and VRAM consumption below. All experiments are conducted on a single NVIDIA A800 GPU. We report the average runtime based on tests using 16-frame video clips. GPU memory usage is monitored using nvidia-smi.

In the runtime breakdown table, we observe that the external camera estimation module from the rotation-aware designs indeed takes up a considerable amount of time. While optimizing the inference speed of the off-the-shelf camera pose estimator is beyond the scope of this work, we will seek to upgrade this module with a faster and more robust camera estimator.

The dual-branch architecture, on the other hand, eliminates the need for multi-round inpainting as required in Follow-Your-Canvas (FYC), thus reducing overall inference time. With memory optimization techniques applied, our method achieves a peak VRAM usage of 25.59 GB, which is lower than FYC’s 27.99 GB.

Q.3 Add a discussion or stress test to strengthen the model's robustness claims.

Thanks for the constructive suggestion. We agree that demonstrating robustness under challenging, real-world cases is important. Our current quantitative evaluation (Table 1) already includes a wide range of diverse, in-the-wild scenarios, such as randomly sampled videos from RE10K and 360-1M, as well as videos generated by CogVideoX using random prompts from GPT-4o. To further strengthen our robustness claims, we will curate a set of harder corner cases and include a dedicated stress test and analysis in the final version.

Q.4 A discussion on video flickering and how this could be improved.

Thanks for the comment. The video flickering primarily stems from our backbone, AnimateDiff, which is known to produce temporal flickers in many cases. Despite this limitation, our method improves upon AnimateDiff in both visual quality and temporal consistency, as shown in Table 1. To further address this issue, we plan to upgrade the backbone to a more advanced video generation model (Wan2.1/2.2), with technical modifications from UNet with DiT. This update is expected to significantly reduce flickering and improve overall temporal coherence. We will update our results with the upgraded backbone in the final version.

Q.5 Why choose MonST3R for camera pose estimation. How robust it is and what if using more robust models like MegaSAM etc?

Thanks for the comment. We chose MonST3R because it was one of the most robust camera pose estimation models when we primarily developed our method. Empirically, we find that it produces accurate and stable pose estimations in most test cases. As camera pose estimation continues to advance, MonST3R can be seamlessly replaced with more robust models, such as MegaSAM, helping our method to handle even more in-the-wild scenarios. We will upgrade MonST3R to MegaSAM and provide updated results and codebase in the final version.

Responses to Reviewer 6m1K

We thank the reviewer for the positive feedback and constructive suggestions. We address the comments as below:

Q.1 No clear comparison to baseline on efficiency and resource requirement.

Thanks for the advice. We add a runtime efficiency breakdown table below and an efficiency comparison table in terms of runtime and VRAM consumption. All experiments are conducted on a single NVIDIA A800 GPU. We report the average runtime based on tests using 16-frame video clips. For all methods listed in the comparison table, only inference time is measured. GPU memory usage is monitored using nvidia-smi.

From the efficiency comparison, we observe that compared to another V2V model Follow-Your-Canvas (FYC), our model is faster in inference speed and consumes less VRAM. The main reason is that FYC requires multiple rounds of outpainting to ensure outpaint quality, and they also train and infer in float32. Thanks to our dual-branch structure, Imagine360 achieves good visual quality in a single pass of inference. AnimateDiff and 360DVD have faster inference speeds, which is within expectation, as they are text-guided and image-guided frameworks. In addition, by employing multiple memory optimization techniques (bfloat16,vae slicing,cuda.empty_cache, intermediate tensor cleanup, etc.), our dual-branch structure does not increase VRAM usage by a lot.

In the runtime breakdown table of our method, we observe that the external camera estimation module takes up a considerable amount of time. Although optimizing the inference speed of off-the-shelf camera pose estimator is beyond the scope of this work, we will seek to upgrade this module with a faster and more robust camera estimator. Both efficiency tables will be added to the final version with a detailed discussion.

Q.2 Missing details of EPE and OmniFVD metric computation.

Sorry for the confusion and thanks for pointing this out. The EPE metric is adapted from optical flow to evaluate motion accuracy. We compute optical flow for both the generated video $V_{\text{gen}}$ and the ground-truth $V_{\text{gt}}$ using PanoFlow, and define EPE as the pixel-wise Euclidean distance between the two flows: $\text{EPE}(V_{\text{gen}}, V_{\text{gt}}) = \| V_{\text{gen}} - V_{\text{gt}} \|$.

For OmniFVD, we follow OmniFID by projecting the 360 video into six cubemap views, compute FVD on each view, and report the average as the final OmniFVD score. We will clarify these details in the final version.

Q.3 When MonST3R’s camera pose estimation fails or is inaccurate, does it manifest as noticeable artifacts? Can you discuss possible solutions and illustrate the robustness of the system?

Thanks for the comment. Empirically, we find that MonST3R produces accurate and stable pose estimations in most test cases. In rare and challenging corner cases, estimating precise camera poses remains an open problem and is beyond the scope of this paper. Nevertheless, as camera pose estimation continues to advance, MonST3R can be seamlessly replaced with more powerful models, enabling our method to handle even more in-the-wild scenarios robustly.

Q.4 The [41] and [42] both refer to the same citation in the reference section.

Thanks for pointing this out. We will revise the paper and fix this in the final version.

Dear Reviewers,

Thank you again for your thoughtful and constructive reviews.

The authors have now submitted their rebuttals. I encourage you to read the responses carefully and engage in discussion. Please aim to complete the discussions by August 6th.

Your AC