VideoPanda: Video Panoramic Diffusion With Multi-view Attention

Re: Question3 “However, 360DVD uses the text-to-image model SD1.5, while the proposed method uses the SVD video generation model, which inherently offers an advantage in video smoothness. Did the authors conduct a relatively fair comparison, such as applying SVD to 360DVD or using SD1.5 with the proposed method?”

(Please see point 2&3 in our response to all reviewers)

To clarify we did not use SVD as the base model for our experiments. In the main paper we used the VideoLDM model (not SVD) which was presented in the work Align your Latents [Blattmann et al., 2023a)] as cited in our paper. This model is based on SD1.5 and is quite similar in design to AnimateDiff.

One main benefit of using multiview perspective generation compared to the equirectangular format used by 360dvd, is that we can more naturally generate content in the sky and ground views (above or below +/-60 degree elevation) that typically are highly distorted in the equirectangular format. We show this in the appendix contained in the supplementary material (please see Appendix Figure A5). It is also very visible in the OOD text-conditional samples when viewing the sky or ground views using the provided VR viewer: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

We thank the reviewer for their insightful comments and suggestions. We would also like to point the reviewer to the additional results and evaluations we have prepared for the rebuttal and presented in the summary response to all reviewers above including:

1 Expanding and Improving OOD Evaluations

2 Results on new base model (CogVideoX-2B)

7 OOD Evaluation: Quantitative Metrics

Updated qualitative results can be seen on our updated webpages below:

OOD Text-cond VBench prompts: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

OOD Col-cond AI generated videos: https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html

We address each of the points mentioned in the review below:

Re: Weakness1 “Essentially, this paper can be seen as an application combining SVD and MVDiffusion, with the spatial and temporal blocks derived from SVD and the multi-view block from MVDiffusion.” “it would be beneficial for the authors to explain how their approach differs from these methods and whether they have made specific design choices to address challenges unique to this application”

Please see point 5 in our response to all reviewers in which we extensively comment about the novelty of our work and relate them to similar works.

We would also like to highlight our multi-task training strategy for letting the same model handle flexible conditioning (see point 5c) and the randomized matrix strategy that shows we can greatly improve the quality of test time generalization to longer sequences with less computational demands (see point 5d).

Re: Weakness1 “Regarding the training strategy, the proposed random matrix strategy is essentially a compromise due to limited computational resources; theoretically, using more views or frames in training would yield better results. From the experimental results in Table 3, it can be seen that improvements in FID and FVD scores are achieved at the cost of PSNR (rows 2 and 3)”

(Please also see point 5D in our response to all reviewers)

We agree with the reviewer, as we state in the paper, line 476 the random matrix strategy is a technique to be able to handle larger time horizons for a given compute constraint. This improves the test-time generalization we observe from our model being able to handle longer time horizons. PSNR decreases a bit but it is also not a perfect metric for the generative setting where the solution space is inherently ambiguous and a mode covering solution may obtain a better PSNR than a plausible looking generative distribution of samples. We hypothesize based on the qualitative comparison result in Figure 7 in the paper that the effect of random matrix is to greatly improve the visual fidelity of the generation (bottom row) whereas fixed matrix training when extended in horizon at test time generates oversmooth content in the unseen views (top row). This is also consistent with the improvements in FID and FVD, we see.

Re: Weakness2 and Question1 (Multi-task strategy) “As for the multi-task strategy [...] it would be helpful for the authors to clarify how their approach differs from these methods to demonstrate the novelty of their method”, “In Table 3, after applying the multi-task training strategy, several metrics, such as FVD and PSNR, show a decline. Could the authors provide an appropriate explanation for this?”

(Please also see point 5C in our response to all reviewers)

For multi-task training, the purpose is to enable the same model to flexibly handle text, image and autoregressive generation. Such a model has more capabilities than any single model and is more expedient than training a separate model for each setting. Quality wise the metrics in Table 3 do decline a bit but they remain very similar. Indicating that we can use one model for all tasks while retaining a similar level of quality on an individual task.

>Re: Question2 “In section 4.6, the authors tested autoregressive long video generation, producing videos up to 61 frames. Have the authors attempted to generate even longer videos? As video length increases, does generation quality continuously degrade? If so, what feasible strategies might help mitigate this issue?”

Thanks for the suggestion and it is an interesting exploration to consider. We are working on this result and will share it shortly!

Thank you to the authors for their detailed response. The newly added experiments, including the use of a stronger video generation model (CogVideoX) and training with more data, have indeed improved the quality of panoramic video generation. However, my two primary concerns remain unresolved:

1. Method Novelty: The authors acknowledge that their multi-view attention is essentially the same as that in MVDream, arguing that applying it to panoramic video generation constitutes novelty. However, in my view, generating panoramic videos is very similar to multi-view video generation. A single frame of a panoramic video is essentially a composite of images from multiple views, making it fundamentally a multi-view image.

2. Fair Comparisons: As Reviewer cZZr also pointed out, the authors use more powerful video generation models like VideoLDM/CogVideoX as their base, whereas the comparison method (360DVD) uses the image-generation model SD1.5. This inherently disadvantages 360DVD in terms of temporal consistency and motion smoothness. As such, the comparison is relatively unfair, making it difficult to determine whether the performance gains are due to the proposed method itself or the superior capability of the base generation model.

We thank the reviewer for their insightful comments and suggestions. We address each of the points mentioned in the review below:

Re: Weakness1 “Poor qualitative results. I feel that the overfitting is evident in the effect”

(Please also see point 1&2 in our reply to all reviewers)

To address these concerns we have greatly expanded and improved our OOD evaluations for the text-conditional setting and show many more qualitative examples now for both settings on our updated websites. We have also trained a version of VideoPanda using the more powerful base model CogVideoX-2B and have found greatly improved generalization and visual quality.

Video condition: We would like to point out that in the video-conditional setting, our model demonstrates a stronger degree of generalization (beyond the WEB360 domain) as we have demonstrated with evaluation on the out of distribution videos. To further support this, we have expanded our OOD evaluation for the video-conditional setting and show many more examples on our updated website.

Please check some qualitative results from our OOD Video-cond AI generated videos https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html

These OOD evaluation videos were sourced from generated samples from SORA/Runway/Luma as downloaded from their respective websites. In total there are 50 videos used for the evaluation. We will add these additional details regarding the selection process of the evaluation set in the supplementary materials.

Text condition:

Our text-conditional OOD evaluations are now using the full VBench suite of prompts (1700+ prompts) with great diversity and coverage of different dimensions and categories. In the text-conditional case, bad results for OOD prompts are caused in part by our base model not understanding the prompt. We found that switching to a stronger base model (CogVideoX-2B) simply fixes it.

Please check some qualitative results from our OOD Text-cond VBench prompts: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

We have also computed quantitative metrics (Clip Score, FID and FVD to HDVila videos) for our OOD text-conditional evaluation on VBench prompts. Please see our comment to all reviewers “7 OOD Evaluation: Quantitative Metrics”.

Re: Weakness1 “The watermarks are being generated, and on closer inspection you can see the airpano.”

(Please see point 6 in our reply to all reviewers)

It is not necessarily the case that generating a watermark indicates that such samples can’t be different from WEB360. For example, quite a few of our OOD text-conditional samples contain the watermark despite being drastically different from content present in WEB360. Just to name some: “0) In a charming Parisian café, a panda sits…” and “1) A joyful Corgi with a fluffy coat and expressive eyes” that can be seen here: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

Note that there are also some examples of our model not generating the watermark such as: “(7) A vibrant individual, dressed in a colorful outfit with a..” and left column of *8) A serene individual, dressed in a flowing white shirt and dark trousers” and “(19) A focused individual, wearing a dark apron over a white shirt, stands”. The results look reasonable in these cases.

This could be solved by careful data filtering, or some other techniques (like the DomainAdapter of AnimateDiff [Guo et al., 2023]) to remove the watermark. But this is not the main focus of this paper.

Re: Weakness2 “Some of the diagrams in the paper don't have good consistency when zoomed in.”

We thank the reviewer for bringing up this concern. Are you referring to the panorama results?

Part of the seams between view boundaries were a result of our suboptimal stitching code for forming the panorama from the multiple views, we adopt the feathering approach used in the MVDiffusion implementation that smoothly blends images based on distance to image center which removes many of the more subtle seamlines that were apparent before.

In some OOD cases, the model could still exhibit some less subtle inconsistencies between views but we show that these reduce when using a stronger base model (CogVideoX-2B). Please refer to our new and greatly expanded OOD evaluation video samples in both the video-conditional and text-conditional cases:

OOD Video-cond AI generated videos: https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html

OOD Text-cond VBench prompts: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

We thank the reviewer for their insightful comments and suggestions. We address each of the points mentioned in the review below:

Re: Weakness1 “First, most of the results present ambiguous semanic structure or broken scene, such as the [autoregressive generation] showcase "anime girl standing on a boat", and the [video-conditioned with different prompts] showcase "A view of the seashore with the sea coming up against a cliff.”

(Please also see point 1,2 and 4 in our reply to all reviewers)

Video-conditional case: our model does sometimes exhibit imperfect semantic structure. These cases are generally heavily out of distribution examples and it performs better in more typical cases. We have added many more qualitative examples of our OOD video-conditional generation to our new website page here: https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html

We note that this is a data issue rather than a fundamental issue in our model.

We have trained our same model with more panorama video data we collect that contain more varied elevations whereas WEB360 is heavily biased towards zero elevation. The additional data helps for videos with non-zero elevations as can be seen when comparing the left and middle column of (7) “A view of the coastal town, the historic stone structure with a dome, the terraced buildings, and the vast blue sea.” and (8) “A view of a rugged off-road vehicle driving.” show extreme examples of non-standard elevation.

Further, we found that using a more powerful base model, CogVideoX-2B can greatly improve the handling of OOD content which we show on the same website in the rightmost column: VideoPanda (CogVideoX-2B + NewData) In particular (5) “A Japanese animated film of a young woman standing on a ship and looking back at camera“ features an extreme closeup of the face. The new model based on CogVideo understands the concept of placing the scene on top of a boat better.

Text-conditional case: We would also like to highlight that we have greatly expanded our text-conditional OOD evaluation and use all VBench prompts, presenting many qualitative results here: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

Despite not being perfect, our model generally creates scenes that are proper panoramas whereas 360DVD features heavy distortions in sky and ground views. Our model is also generally better aligned to the prompt. We corroborate these qualitative findings with our new quantitative evaluations on these OOD cases where we perform much better on Clip Score and FID/FVD to HDVila videos. Please see comment (7 OOD Evaluation: Quantitative Metrics) to all viewers above for specifics.

Re: Weakness1 “Besides, almost all results present obvious seamline across views. It seems the newly introduced multi-view attention does not work as expected.”

Part of the seams were a result of our suboptimal stitching code for forming the panorama from the multiple views, we adopt the feathering approach used in the MVDiffusion implementation that smoothly blends images based on distance to image center which removes many of the more subtle seamlines that were apparent before.

In some OOD cases, the model could still exhibit some less subtle inconsistencies between views but we show that these reduce when using a stronger base model (CogVideoX-2B). Please refer to our new and greatly expanded OOD evaluation video samples in both the video-conditional and text-conditional cases:

OOD Video-cond AI generated videos: https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html

OOD Text-cond VBench prompts: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

Re: Weakness1 and Question4 “A possible attempt is jointly finetuning the base model with LORA, which may help the model better adapt to the panorama video distribution.” “Would it be helpful to the generation quality (such as the ambiguous/broken scene semantics) if the base model is tuned with LORA?”

Thank you for the great suggestion. We note that we are currently training our CogVideo model without any freezing. We found that the more powerful base model is better able to preserve its general knowledge even without needing freezing. However, it could be further improved by using a combination of freezing and LoRA as you suggested We are running this experiment now and excited to report on it as soon as we have ablated this. Thanks again for the suggestion!

Re: Weakness2 “In contrast, although showing over-smooth textures, the semantics of 360DVD are more natural and the scene of content are more identifiable than the proposed method.”

(From point 3 in our reply to all reviewers)

We hope our new text-conditional OOD evaluations can help address some of the concerns about how our model compares to 360DVD. We evaluate both methods head-to-head on the VBench prompts. We want to point out that the 360DVD results are almost always not generating proper panorama videos. Additionally, we found that the prompt alignment and scene consistency for VideoPanda can all improve simply by applying VideoPanda to a more powerful base model (CogVdeoX-2B).

One main benefit of using multiview perspective generation compared to the equirectangular format used by 360dvd, is that we can more naturally generate content in the sky and ground views (above or below +/-60 degree elevation) that typically are highly distorted in the equirectangular format. We show this in the appendix contained in the supplementary material (please see Appendix Figure A5). It is also very visible in the OOD text-conditional samples when viewing the sky or ground views using the provided VR viewer: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

Such errors are hard to visually spot from the equirectangular projection native to 360DVD but very important for the actual task of panorama video generation. Please have a try!

This finding is also supported by our quantitative evaluation on the OOD prompts. Please see the comment (7 OOD Evaluation: Quantitative Metrics) to all reviewers for more details regarding the evaluation. We present the results in the table below:

Note that the original model we presented in our paper (VideoPanda using VideoVLDM base model) is clearly better than 360DVD on all metrics. VideoPanda significantly outperforms 360DVD on elevated views, highlighting its superior ability to generate the ground and sky views, which are distorted in the equirectangular representation used by 360DVD. Additionally, the superior performance gained by using the CogVideoX-2B base model with VideoPanda is supported by improvement across all metrics.

> Re: Weakness3 “The comparative examples with 360DVD are almost static scene.” “It is highly required to evaluate on cases with moving objects (such as "moving car on the street", "astronaut riding a horse on grass", etc.)”

(Please also see point 1 and 3 in our reply to all reviewers)

We have now included the entire VBench suite for our OOD text-conditional experiments and it contains many prompts with dynamic objects. Please see the qualitative comparisons here: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html. We do find that the generations from our model with the VideoLDM base model tends to generate videos with lower amount of motion. However, this is related to the base model and VideoPanda on CogVideoX-2B trained on the same data is able to generate scenes with fairly dynamic content.

As 360DVD conditions on optical flow, it can directly force some motions in the scene but the motions can often be incoherent such as objects morphing in and out of existence. Viewing results in VR also highlights other strange motions in 360DVD including the ground morphing in a spiral direction (0) “In a charming Parisian café, a panda” or inanimate objects sliding in different directions (17) “A serene scene unfolds with a delicate porcelain teacup” when comparing VideoPanda (left column) vs 360DVD (right column).

Re: Question1 “How many samples are used for the evaluation in Table 2? How do you collect the prompts? Do they cover good diversity?”

Table2 is our comparison to MVDiffusion on the in-distribution video-conditional setting. Table1 is our comparison to 360DVD on in-distribution text-conditional setting. Both of these in-distribution evaluations used the same data source.

• The number of ground truth panorama videos used for in-distribution evaluation is 100.
• These videos were selected from online panorama videos. To ensure, the selected videos do not overlap with WEB360, additional videos we sourced from airpano were limited to ones uploaded beyond the creation date of WEB360 and from another channel, “National Geographic”.
• The prompts for the clips were obtained from captioning the input view extracted from the panorama with CogVLM.
• In terms of diversity, the clips are similar in distribution to the WEB360 data.
• We sampled one seed per prompt for doing the evaluations in the paper but for the new OOD evaluations we sample 3 seeds for each method.

We have greatly improved our OOD evaluations including quantitative metrics and using VBench prompts that cover a good diversity of different dimensions and categories. Please see the comment (7 OOD Evaluation: Quantitative Metrics) to all reviewers for more details regarding the OOD evaluations.

> Re: Question2 “It is hard to understand why multi-task training is better than single task training. This is contrast with some common feeling, for example all the T2V or I2V modes are processed with fixed frames, because varied number of tokens in the attention computation may hinder the performance on a specific frames and resolution. I would like to see more elaboration over this.”

(From points 5C & 5D in our response to all reviewers)

We want to clarify that in the paper the “multi-task training” refers to training the same model to be capable of handling different types of conditioning (text-, video- and autoregressive). This model is more capable than a model only trained for one of the settings. Performance-wise the multi-task model is on par with the single task video-conditional model albeit slightly worse as shown in Table 3 of the paper. Please also see point 5C in our response to all reviewers for more detailed explanation.

The use of non-fixed frames is the “random matrix” strategy we present in the paper. It is a computational technique to improve the model’s ability to generate full multiview videos of longer time horizons at test time for a given computational budget during training.

Our random matrix strategy is explained in line 476 in the main paper. We wish to generate 8 views with 16 frames but because we are generating more views than single view training, it can not fit in GPU memory and also slows down training. We could still train with 8 views and 6 frames instead which does fit in memory. We call this strategy “fixed matrix”. Although simple, the mismatch in training frames (6) vs inference number of frames (16) results in blurry outputs which we show in the top row of Figure 7. We apologize for an oversight in the caption of Figure 7. It should say “fixed matrix” rather than “full matrix”.

Instead of the reduced fixed matrix training, we notice that one can still fit 3 views at 16 frames. By randomizing which 3 out of the 8 views we choose and randomizing other combinations that can fit in memory such as 4 views at 12 frames, we can strengthen the models ability to generalize to sampling more frames at test time. This strategy is called “random matrix” and its improvement is illustrated in the bottom row of Figure 7. We quantitatively evaluated this strategy on the in-distribution video-conditional task in Table 3, where all quality metrics improve quite a bit but reconstruction PSNR is slightly lower which could be related to slightly better global color consistency for the fixed matrix model that is sees more training iterations featuring all views.

Note that this technique is still applicable for extending the horizon even with larger compute and memory budget (for example even if we use Context Parallel to train with more frames, it doesn't make random matrix useless as it can still improve the performance when the test time temporal window is extended even longer for the same compute cost).

Re: Question3 “The artifacts of full matrix shown in Figure. 7, actually also happen in the result of the proposed method (i.e. random matrix), such as in .. "A view of a panda's face ...“.”

Although we do see that oversmoothing could happen for more OOD results, the random matrix strategy largely reduces these cases. We quantitatively evaluated this through ablating random matrix in the paper. Please see the results we present in Table 3 where the random matrix strategy is the middle row and greatly reduces the FVD and FID score compared to the last row which is not using it.

Re: Weakness1 and Question 3 “Results generated by the method often contain the airpano watermark, whereas the few results without watermarks on the webpage exhibit strange artifacts in other regions.” “Can the authors provide more quantitative and qualitative results without watermarks?”

(Please see point 6 in our reply to all reviewers)

It is not necessarily the case that generating a watermark indicates that such samples can’t be different from WEB360. For example, quite a few of our OOD text-conditional samples contain the watermark despite being drastically different from content present in WEB360. Just to name some: “0) In a charming Parisian café, a panda sits…” and “1) A joyful Corgi with a fluffy coat and expressive eyes” that can be seen here: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

Note that there are also some examples of our model not generating the watermark such as: “(7) A vibrant individual, dressed in a colorful outfit with a..” and left column of *8) A serene individual, dressed in a flowing white shirt and dark trousers” and “(19) A focused individual, wearing a dark apron over a white shirt, stands”. The results look reasonable in these cases.

This could be solved by careful data filtering, or some other techniques (like the DomainAdapter of AnimateDiff [Guo et al., 2023]) to remove the watermark. But this is not the main focus of this paper.

Re: Weakness2 “FID and FVD do not necessarily require paired prediction-ground truth data”

(From point 7 in our reply to all reviewers)

We agree that FID/FVD are distributional measures and thank the reviewer for their suggestion. We now compute FID and FVD for the expanded OOD text-conditional evaluations using the VBench prompts. Since these prompts can be fantastical, there are no ground truth video set that accompany them, instead we use the popular image dataset MS-COCO for FID reference set and video dataset HDVila for our FVD reference set. Please see the comment (7 OOD Evaluation: Quantitative Metrics) to all reviewers for more details regarding the evaluation. We present the results in the table below:

Note that the original model we presented in our paper (VideoPanda using VideoVLDM base model) is better than 360DVD on all metrics. VideoPanda significantly outperforms 360DVD in the 60 degree elevation views, highlighting its superior ability to generate the ground and sky views, which are distorted in the equirectangular representation used by 360DVD. Additionally, the superior performance gained by using the CogVideoX-2B base model with VideoPanda is clearly seen by the quantitative evaluation above, with massive improvements observed across all metrics.

Re: Weakness3 and Question3 “There are no ablation studies on the multi-view attention mechanism. The paper does not clearly explain the differences between the proposed multi-view attention and existing methods, such as MVDream.” “There are no ablation studies on the multi-view attention mechanism”

(Please also see point 5B in our reply to all reviewers)

We describe our multi-view attention mechanism in Section 3.1, starting at line 190. There is no inherent difference to multi-view attention mechanisms used by other works such as Cat3D [Gao* et al., 2024] and MVDream [Shi et al., 2023b)]. In the paper, we do not claim the multi-view attention mechanism as a novel contribution. Instead we are the first to utilize it for the generation of panoramic videos and natural handling of input video conditioning.

Please let us know if you would like us to try any specific ablations. Thanks!

We thank the reviewer for their insightful comments and suggestions such as computing non-paired FVD. We address each of the points mentioned in the review below:

> Re: Weakness1 “The model appears to be overfitted to the WEB360 dataset.”

(Please also see point 1&2 in our reply to all reviewers)

To address these concerns we have greatly expanded and improved our OOD evaluations for the text-conditional setting and show many more qualitative examples now for both settings on our updated websites. We have also trained a version of VideoPanda using the more powerful base model CogVideoX-2B and have found greatly improved generalization and visual quality.

Video condition: We would like to point out that in the video-conditional setting, our model demonstrates a stronger degree of generalization (beyond the WEB360 domain) as we have demonstrated with evaluation on the out of distribution videos. To further support this, we have expanded our OOD evaluation for the video-conditional setting and show many more examples on our updated website.

Please check some qualitative results from our OOD Video-cond AI generated videos: https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html

These OOD evaluation videos were sourced from generated samples from SORA/Runway/Luma as downloaded from their respective websites. In total there are 50 videos used for the evaluation. We will add these additional details of the evaluation set in the supplementary materials.

Text condition:

Our text-conditional OOD evaluations are now using the full VBench suite of prompts (1700+ prompts) with great diversity and coverage of different dimensions and categories. In the text-conditional case, bad results for OOD prompts are caused in part by our base model not understanding the prompt. We found that switching to a stronger base model (CogVideoX-2B) simply fixes it.

Please check some qualitative results from our OOD Text-cond VBench prompts: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

We have also computed quantitative metrics (Clip Score, FID and FVD to HDVila videos) for our OOD text-conditional evaluation on VBench prompts. Please see our comment to all reviewers “7 OOD Evaluation: Quantitative Metrics”.

> Re: Weakness1 “The test videos are directly sourced from the WEB360 dataset or selected from similar scenes on the airpano channel”

We want to explicitly state that the test videos we used for our in-distribution evaluations were NOT sourced from WEB360. To ensure, the selected videos do not overlap with WEB360, additional videos we sourced from airpano were limited to ones uploaded beyond the creation date of WEB360 and from another channel, “National Geographic”. For example the panda video you refer to on the website is from nat geo: https://www.youtube.com/watch?v=0XrH2WO1Mzs. Although they have related content type, the video is different.

As for the ice and mountains (in "100666"), this is from the in distribution text-conditional evaluation. As we mentioned in the paper (line 354), the in distribution text-cond evaluation uses the captions of the in-distribution evaluation videos. In this case, this text prompt corresponds to a caption generated from a clip in this “National Geographic” video: https://www.youtube.com/watch?v=XPhmpfiWEEw&pp=sAQA. We agree that for the in-distribution evaluation, there can be similarities between the collected videos and videos in WEB360 which could result in similar text prompts to ones in WEB360 in some cases and we therefore supplemented these with our OOD evaluations.

Please kindly refer to our OOD text-condition and OOD video input evaluations above. OOD Text-cond VBench prompts: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html OOD Col-cond AI generated videos: https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html

Re: Weakness4 and Question1 “The paper lacks experiments on conditions with varying fields of view (FOV) and view directions.”“Can VideoPanda generate videos conditioned on different FOVs and view directions?”

(From point 4 in our reply to all reviewers)

We agree that handling varying FOV and view elevations is an interesting and important problem for wider practical application. Even with our current model, it is able to handle camera variation to some extent due to natural elevation changes in the training videos. Although WEB360 contains very limited elevation changes, the new dataset we collect contains more and we highlight examples of these cases in the video-conditional setting here: https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html

The additional data helps the understanding of different elevations as can be seen in the middle column for example (7) “A view of the coastal town, the historic stone structure with a dome, the terraced buildings, and the vast blue sea.” and (8) “A view of a rugged off-road vehicle driving.” show extreme examples of non-standard elevation. Using the CogVideoX-2B as base model also further boosts the visual quality and handling of OOD content.

That being said, handling of other FOV is approximate and extreme elevation changes can also be confusing to our model. We also commented on this limitation in the conclusion of the main paper, line 529: To handle such extreme camera variations, we could combine our method with CamFreeDiff [(Yuan et al., 2024)] which has been applied in the image panorama generation setting. We leave this important exploration to future work.

3 Comparison to 360DVD

We hope our text-conditional OOD evaluations can help address some of the concerns about how our model compares to 360DVD. We evaluate both methods head-to-head on the VBench prompts. We want to point out that the 360DVD results are almost always not generating proper panorama videos.

One main benefit of using multiview perspective generation compared to the equirectangular format used by 360dvd, is that we can more naturally generate content in the sky and ground views (above or below +/-60 degree elevation) that typically are highly distorted in the equirectangular format. We show this in the appendix contained in the supplementary material (please see Appendix Figure A5). It is also very visible in the OOD text-conditional samples when viewing the sky or ground views using the provided VR viewer: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

Viewing results this way also highlights the strange motions in 360DVD including the ground morphing in a spiral direction (0) “In a charming Parisian café, a panda” or inanimate objects sliding in different directions (17) “A serene scene unfolds with a delicate porcelain teacup” when comparing VideoPanda (left column) vs 360DVD (rightmost column) here: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html.

Such errors are hard to visually spot from the equirectangular projection native to 360DVD but very important for the actual task of panorama video generation. Please have a try!

4 Video-conditional setting: Handling variations in camera elevation in input video

Even with our current model, it is able to handle camera variation to some extent due to a few natural elevation changes in the training videos.

However WEB360 contains very limited elevation changes in the training dataset, usually showing smooth landscape fly-overs. We hence collect a new dataset of panorama videos and show that it can enhance the models understanding of variations in elevation. Furthermore, similarly to the text-conditional setting, we also trained VideoPanda using the more powerful CogVideoX-2B base model which boosts the visual quality and enhances the handling of OOD content further. We highlight these new results in the OOD Video-conditional results here (in particular, on the left column is the result from the model in our paper, the middle column is the same model but trained with the new data instead of web360 and the right column is further switching the base model to CogVideoX-2B): https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html

The additional data helps the understanding of different elevations as can be seen in for example (7) “A view of the coastal town, the historic stone structure with a dome, the terraced buildings, and the vast blue sea.” and (8) “A view of a rugged off-road vehicle driving.” show extreme examples of non-standard elevation.

6 Watermark

Generation of watermarks are a dataset bias. When training on web360 (almost all the training videos have the airpano watermark), it’s expected that the model will generate the same watermark in inference.

It is not necessarily the case that generating a watermark indicates that such samples can’t be different from WEB360. For example, quite a few of our OOD text-conditional samples contain the watermark despite being drastically different from content present in WEB360. Just to name some: “0) In a charming Parisian café, a panda sits…” and “1) A joyful Corgi with a fluffy coat and expressive eyes” that can be seen here: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html

On our newly collected dataset used to support the handling of camera elevation variations (see point 4 above), we notice there are other types of watermarks typically appearing on the bottom views. Also a lot of these videos are shot from a camera held by a person, mounted on their head or mounted on a vehicle of some sort, which is often also adapted by the model. This could be solved by careful data filtering, or some other techniques (like the DomainAdapter of AnimateDiff [Guo et al., 2023]) to remove the watermark. But this is not the main focus of this paper.

5 Motivation and Novelty

As we mentioned in the paper, our main contributions are as follows:

A: Propose Video-conditional Panorama Video Generation Task

We are the first to introduce the video conditional multiview generation task on open domain. We believe the capability of a model to synthesize multiple views of a dynamic scene is key to address many important problems in vision, graphics and robotics. To support this goal, panoramic videos are a very promising yet underutilized source of true multiview spatiotemporal data that contains dynamic objects.

There are many existing tools for forming image panoramas of static scenes from image collections or videos captured by handheld cameras and phones. These collate and compose the images. However, this approach is not applicable for producing panorama videos of dynamic scenes as the different views will be temporally inconsistent with each other. To capture panoramic videos, customized camera rigs need to be used.

Generating panoramas conditional on input videos is hence an important step to democratize the creation of panoramic content. Such content can be used for creating VR experiences and as backgrounds for visual effects and other graphics applications. Another interesting use case is scene reconstruction, where diffusion models have made recent contributions to supply high fidelity priors to resolve inherent ambiguities. Multiview image modeling capabilities for this use case are being enhanced by leveraging pretrained video models but are mostly still restricted to static scene cases. Such capabilities may also aid robotic applications where we may wish to playback a recorded sequence but allow the agent to modify its behavior such as viewing direction to some extent.

B: Using Multiview-Attention for the Panorama Video Generation

The multi-view attention mechanism we use is not inherently different from existing ones like MVDream or CAT3D, but we introduce it to the video panorama generation setting and show we can get good results whilst keeping the multiview-attention to operate only per-frame and let the rest of the layers keep interpreting each full video separately.

C: Multi-Task Training for Flexible Conditioning

We introduce multi-task training to enable a single model to handle text-conditioning, video-conditioning and autoregression which saves training time and is more convenient. While dropping conditioning during training is a common technique as is used in models like GLIDE, Imagen, and SVD, It is mostly done to enable classifier-free guidance rather than flexible conditioning for a single model. The use of joint video and text conditioning in our model is similar to other video outpainting settings and also related to the task of Instruct-Pix2Pix. In our case, our single model is able to handle full generation from text as well as conditional on a video, and autoregression.

D: Randomized Matrix Training Strategy for Longer Temporal Horizon

We introduced the randomized matrix training strategy that improves the model’s ability to generate full multiview videos of longer time horizons at test time for a given computational budget during training.

Our random matrix strategy is explained in line 476 in the main paper. We wish to generate 8 views with 16 frames but because we are generating more views than single view training, it can not fit in GPU memory and also slows down training. We could still train with 8 views and 6 frames instead which does fit in memory. We call this strategy “fixed matrix”. Although simple, the mismatch in training frames (6) vs inference number of frames (16) results in blurry outputs which we show in the top row of Figure 7. We apologize for an oversight in the caption of Figure 7. It should say “fixed matrix” rather than “full matrix”.

Instead of the reduced fixed matrix training, we notice that one can still fit 3 views at 16 frames. By randomizing which 3 out of the 8 views we choose and randomizing other combinations that can fit in memory such as 4 views at 12 frames, we can strengthen the models ability to generalize to sampling more frames at test time. This strategy is called “random matrix” and its improvement is illustrated in the bottom row of Figure 7. We quantitatively evaluated this strategy on the in-distribution video-conditional task in Table 3, where all quality metrics improve quite a bit but reconstruction PSNR is slightly lower which could be related to slightly better global color consistency for the fixed matrix model that is sees more training iterations featuring all views.

Note that this technique is still applicable for extending the horizon even with larger compute and memory budget (for example even if we use Context Parallel to train with more frames, it doesn't make random matrix useless as it can still improve the performance when the test time temporal window is extended even longer for the same compute cost).