Response of Reviewer v3P6’s Review

Q1: Clarification of training Strategy.

The two training phases are applied to the same CVSM parameters in a single training pass. (The CVSM in Fig.3a and Fig.3b are the same modules.) Specifically, we blend the data from Webvid10M or RealEstate10K together. Then in each training iteration, we randomly pick one data batch from our hybrid dataset and train our model in the corresponding phase.

Q2: Problem in Derivation from Eq. (16) to Eq. (17):

The transformation from Eq. (16) to Eq. (17) can be expanded as:

$\frac{**1**(k\in S)}{1-\bar\alpha_t} ( \sqrt{\bar \alpha_t} \int q(**v**_0^S | **v**_t^S) **v**_0^k \ d**v**^S_0 - **v**_t^k)$ $= \frac{**1**(k\in S)}{1-\bar\alpha_t} ( \sqrt{\bar \alpha_t} \int q(**v**_0^k | **v**_t^S) q(**v**_0^{S/k} | **v**_0^{k}, **v**_t^S) **v**_0^k \ d**v**^S_0 - **v**_t^k)$ $= \frac{**1**(k\in S)}{1-\bar\alpha_t} ( \sqrt{\bar \alpha_t} \int q(**v**_0^k | **v**_t^S) **v**_0^k \int q(**v**_0^{S/k} | **v**_0^{k}, **v**_t^S) \ d**v**^{S/k}_0 \ d**v**^k_0 - **v**_t^k)$ $= \frac{**1**(k\in S)}{1-\bar\alpha_t} ( \sqrt{\bar \alpha_t} \int q(**v**_0^k | **v**_t^S) **v**_0^k \ d**v**^k_0 - **v**_t^k)$

We will add the intermediate steps in our revision to make it more accessible.

Q3: Missing references of related works.

Thanks for pointing it out. MAS and MVHuman are both interesting recent works that focus on 3D human/animal generation, which is a similar but different task from ours. We will add the references and discussion in our revision.

Response of Reviewer EujL’s Review

Q1: Lack of ablation study.

We would like to emphasize that our cross-video augmentations are applied to the pairs of videos, which serve as the training input throughout our experiments; It is infeasible to apply such augmentations to monocular video generation methods such as MotionCtrl and CameraCtrl since they always only take one video as training input. To verify our design choices, including the effectiveness of our cross-video augmentations, we supply ablation study results on our method in Tab.3 in our supplementary, where our full model outperforms all ablative variants by a large margin, demonstrating the effectiveness of our design. For example, our full model achieves 25.2 scores at the Rot. AUC 5%, and scores of other variants are 16.8 for the model without epipolar attention; 17.9 for the model without WebVid10M training; 22.0 for the model without homography transformation;

Q2: Qualitative Analysis of camera trajectories.

In our paper, we show both quantitative and qualitative comparisons regarding camera trajectories. In Tab.1, the AUC scores evaluate the accuracy of camera poses optimized from our video predictions. In Fig.4, we show the results of our model and all baselines given the input camera trajectories (shown on the left). We think these experiments demonstrate the capability of our model to follow the input cameras.

Q3: More details for inference (how to generate multiple views).

Thanks for pointing this out. To generate more than two videos with our model, we designed a sampling algorithm as described in Section A.3.2. In general, the algorithm first initializes M noise maps from Gaussian (corresponding to the videos of M camera trajectories). In each denoising step, multiple pairs among the noise maps are selected, and the network is run on each selected pair for noise prediction. The predictions are then averaged over all pairs and applied to each noise map individually for denoising.

Due to the page limit, we moved most of our inference details to our supplementary (Section A.3.2). If accepted, we will add more details to the main paper with the additional page quota.

Q4: Is it possible to edit videos using our method (i.e., generate another video given a source video)?

We think this is a very interesting and doable direction. While our model is trained to generate two videos simultaneously, it can also be adapted into a model conditioned on a source video. To do that, we can train it with the same training dataset but feed the model with one video as a reference and let it predict the other one.

We also notice that a concurrent work from Hoorick et al. [1] also made a good attempt at this setting. We believe this demonstrates the great potential of multi-view video generation models to be applied in many applications in the future.

Q5: Novelty of our paper.

We believe our work has several significant distinctions from previous works. To our knowledge, we are the first work attempting the task of general multi-view video generation; We introduce the cross-view synchronization module (CVSM), which is inspired by previous works on epipolar attention, but has a very different approach aiming for the synchronization across videos in the task of multi-view video generation; And we propose a novel training-free algorithm to sample more consistent views from our model at inference.

[1] Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis, Hoorick et al., 2024.

Has the author's rebuttal addressed your concerns?

Please provide your feedback on the rebuttal and make your final decision.

AC

Response of Reviewer MCcD’s Review

Q1: Clarification of Eq.(4) (how the attention masks are generated).

Thanks for pointing this out. The attention masks are calculated as follows: For each pair of pixels coordinated at x_1 and x_2 in two frames, respectively, the attention mask from x_1 to x_2 is calculated by the epipolar distance between x_1 and x_2 (i.e. the shortest distance between x_1 and the epipolar line of x_2 in x_1’s frame). We will fix the typo and add the clarification in our revision.

Q2: Limitations of homography warping and other possible dataset candidates.

Thanks for raising this important question. Technically, the homography warping phase is introduced to enhance the model’s capability to produce synchronized motion across multi-view videos, which is an action taken out of necessity due to the lack of large-scale generic 4D datasets. Indeed, as the reviewer suggested, homography warping has many downsides, such as distortion of shape, incorrect perspective, and lack of view-dependent appearance and depth information. However, despite the limitations, homography warping serves as a pseudo-bridge attempting to close the gap between the training of static RealEstate10K videos and dynamic WebVid10M videos. Further, we argue that the perspective issue can be addressed by the pseudo epipolar lines: They are similarly distorted as the warped geometry, hence canceling out its effect on the model. In other words, it pushes the video model to strictly follow the given epipolar lines, making it eventually generate decent videos when the lines become correct. The combination of homography warping and epipolar-based inductive bias both effectively improves the transferring of learned correspondence information between our two training stages, as indicated in Tab.3 in our supplementary.

In our work, we choose to follow CameraCtrl and MotionCtrl and use RealEstate10K, which has less diversity in camera trajectories since it only consists of monocular videos. But we believe our model can be further improved by integrating more datasets into the training, such as 3D/4D object datasets (Objaverse, OmniObject3D, MvImgNet), 3D landscape datasets (MegaScene, Acid), and small-scale 4D scenes (Multi-view video datasets). On the other hand, there exist many real-world dynamic motions that are hard to capture in multi-view, such as the motion of fireworks, waves, or wild animals. In these cases, monocular videos are still the only available large-scale resources. We believe there is great potential for exploring future works based on our method with additional datasets, and will add more discussions in the limitation section of our paper.

Q3: How do you combine the models trained in different phases?

As our model inherits the ability from AnimateDiff to generalize on different LoRAs, during inference we select the LoRA that best satisfies the needs in different settings. For the quantitative comparisons, we use LoRA trained on RealEstate10K (provided by CameraCtrl) for RealEstate10K scene generation and LoRA trained on WebVid10M (provided by AnimateDiff) for WebVid10M scene generation; For qualitative comparisons, without specification, we use the RealisticVision LoRA for its superior performance in generating high-quality images and videos. On the other hand, LoRA from CameraCtrl for temporal layers is applied in all experiments.

We thank the reviewer for the reply and respectfully disagree with some of the claims.

(Q1, Q3) Our model does not have enough ability to generate large-scale camera changes:

We first argue that our current model can handle large camera motions to some extent, as shown in our results. For example, sample #3 in the first generation results in our demo video (0:01) are controlled by very different cameras; In the first demo of our 6-view video generation (1:35 in the demo video), the differences between the view on the left-top and the one on the bottom-right are also significant. These examples represent some of the most diverse and extreme camera trajectory variations possible that we can think of, within a 16-frame sequence, demonstrating that our method performs well even when the camera moves in opposite directions (e.g., flying in vs. flying out, rotating left vs. rotating right).

Second, CVD is grounded in the currently limited infrastructures of open-source video generation and available datasets, functioning as a proof-of-concept that maximizes the potential of existing resources. This small-scale setup naturally affects the model’s capability to support large camera motions for two reasons: The capabilities of our method are inherently constrained by the performance of our base models, namely CameraCtrl. Consequently, the extent of motion and the range of camera paths we can manage are limited by the pretrained AnimateDiff and CameraCtrl models. Since the camera path conditioning is derived from RealEstate10K’s camera paths in CameraCtrl (and other models such as MotionCtrl, VD3D), our method cannot accommodate dramatic camera changes beyond what is covered in the RealEstate10K domain. As an extension of existing camera-conditioned video generation techniques, our approach necessarily inherits these limitations. At the time of our paper’s development, open-source video generation models were generally limited to producing only a few frames. For example, AnimateDiff, the base model used in our work, is capable of generating only 16-frame videos. Within this frame limit, accommodating large motions and significant camera movements is particularly challenging. Although CVD is relatively small in scale, we are inspired by our results and see significant potential in a scaled-up version incorporating systems like SORA or DreamMachine. We believe our work will inspire and motivate future research and development efforts in multi-view and multi-camera video generation.

(Q2) Our model lacks generalization by using different LoRA for different experiments:

We want to clarify that it is a common practice for diffusion models to apply different LoRA in different tasks. Both CameraCtrl and AnimateDiff are trained with specific LoRA to adapt the training data distribution, but are combined with unseen LoRAs in their qualitative results. While we apply different LoRAs for the spatial attention layers (which are pretrained models from previous works) for each task, our CVSM module remains the same across all experiments. We also want to emphasize that, although during the training our model has never seen RealVision’s LoRA module, it can naturally adapt this LoRA during the inference. These indicate that our CVSM can be generalized in various tasks.

On the other hand, even though we have shown that our model generalizes well to unseen LoRAs, this does not mean our method does not work well without a LoRA. To support this, we also provide additional results without using any appearance LoRA (along with results in Table 1. in our paper):

Model w/o LoRA:

Rot. AUC: 21.9 / 34.6 / 49.4

Trans. AUC: 2.7 / 7.4 / 16.3

Prec.: 44.0

M-S: 17.6

Full Model:

Rot. AUC: 25.2 / 40.7 / 57.5

Trans. AUC: 3.7 / 9.6 / 19.9

Prec.: 51.0

M-S: 23.5

MotionCtrl [56]+SVD:

Rot. AUC: 12.2 / 28.2 / 48.0

Trans. AUC: 1.2 / 4.9 / 13.5

Prec.: 23.5

M-S: 12.8

Our model without any LoRA slightly underperforms our full model but still performs much better than the baselines.

We also notice that merging different LoRAs into a single model is an active area of research (e.g. Work from Gu et al. [1] and Po et al. [2]). Therefore, while our setting matches standard practice in the field, we believe it would be interesting to explore CVD in combination with very recent advances, like MoS or orthogonal adaptation, which we will clarify in the discussion.

[1] Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models, Gu et al. 2024 [2] Orthogonal adaptation for modular customization of diffusion models, Po et al., 2024

Thank you again for your time and consideration.

Response of Reviewer WFsL’s Review

Q1: How are the LoRA modules used in the inference step?

As our model inherits the ability from AnimateDiff to generalize on different LoRAs, during inference we select the LoRA that best satisfies the needs in different settings. For the quantitative comparisons, we use LoRA trained on RealEstate10K (provided by CameraCtrl) for RealEstate10K scene generation and LoRA trained on WebVid10M (provided by AnimateDiff) for WebVid10M scene generation; For qualitative comparisons, without specification, we use the RealisticVision LoRA for its superior performance in generating high-quality images and videos. LoRA from CameraCtrl for temporal layers is applied in all experiments.

Q2: Wrong notation in Eq. 4. (x_i should be x_1).

Thanks for pointing this out. We will fix this typo in our revision.

Q3: Clarification on how the attention masks are generated and used.

Our attention masks are added as an inductive bias to the self-attention modules, encouraging them to extract information from the epipolar geometry. For this reason, our attention masks function as a 0/1 masking on the full self-attention score, where only the pixels on the corresponding epipolar lines have attention weights, as shown in Fig.6 of our supplementary. This process does not directly indicate correspondences (since we never possess ground truth pixel correspondences during training and inference) but rather introduces the epipolar line inductive bias about where to find the correspondences. As shown in our ablation study in Tab.3, having this inductive bias is very helpful in terms of aligning the content across the two frames. Empirically, we find the hit rate of a pixel’s corresponding pixels being picked up by the attention module (at some diffusion step) close to 100%.

Q4: Comparison between our model and running monocular video diffusion model multiple times.

Indeed, running a powerful monocular VDM multiple times with the same condition (ideally the same keyframes) is a rather reasonable method for multi-view video generation. Yet, this approach faces a major issue: It will suffer from generating inconsistent motion and different content that is unseen in the keyframes. The reason is that monocular VDMs are not trained to introduce consistency across views. For example, if we let a monocular VDM generate multiple firework videos given the same condition (text prompt or starting frame), it will very likely generate different firework patterns in each pass. Our model instead jointly predicts all videos together with their information being shared in the cross-view modules, thus it can produce more view-consistent results. In our experiments (Tab.2), we compare our model with the baseline “CameraCtrl+SparseCtrl”, which can be considered as the monocular VDM approach. Our model outperforms the latter by a large margin.

Response of Reviewer 3sRF’s Review

Q1: The black regions might affect the performance of video generation.

During our training, we removed the L2 loss in the unseen pixels (black regions) of the cloned video for data integrity. We show examples of homography warped videos in our attached PDF (Fig. A2) and will clarify the loss in black regions in our revision.

Q2: Add FVD metric evaluation.

Thanks for pointing this out. We evaluate our method and all baselines using the FVD metric, and here are the results:

• CameraCtrl: 277
• AnimateDiff+SparseCtrl: 327
• CameraCtrl+SparseCtrl: 430
• Ours: 285

Similar to our experiments on the FID metric, our method is on par with CameraCtrl and performs better than other baselines.

Q3: How is the 3D reconstruction obtained using the generated multi-view videos?

We use NeRFactor [Tancik et al. 2023] to reconstruct 3D from our multi-view videos. We will clarify more details in the revised version.

Q4: How is Tab.3 obtained? Why does training on WebVid10M improve Rot. AUC Trans. AUC as compared to training on RealEstate10K only?

This is an insightful question! For the evaluation results in Tab.3, each model generates video pairs given the same text prompt and camera trajectories. For each video pair, we calculate the SuplerGlue [Sarlin et al. 2020] matches between each frame pair (i.e. two frames captured at the same time from the video pair). Then, we run RANSAC on these matches to calculate the relative camera poses between the two frames in each frame pair and compare the camera poses with the ground truth to get the AUC score.

We believe there are two reasons why our full model outperforms the model trained on RealEstate10K. Firstly, we think the credit goes to our epipolar attention. In the WebVid10M training stage, while there are no camera poses available, we managed to use pseudo-gt epipolar lines (i.e. lines calculated from homography matrix H. The line of pixel x in the warped frame goes through the pixel Hx) to describe the spatial relationship between video frames. This enhances the model’s ability to generate videos that satisfy the given line conditions. Hence, in a camera-control setting, the full model is more constrained to the epipolar lines and generates videos that align better with the camera poses. Secondly, since RealEstate10K mostly consists of static indoor scenes, models trained on RealEstate10K may suffer from data bias and may not perform well on general scenes, thus resulting in poor evaluation performance in this experiment. We will clarify this in our revision.

Has the author's rebuttal addressed your concerns?

Please provide your response to the rebuttal and make your final decision.

AC