SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints

Thanks for your constructive comments! We address the concerns below:

The challenge identification part is unclear, especially in clarifying the limitation of existing methods.

Thanks for the constructive comments. We have revised the related parts (line:68~78) in the introduction section to clarify the limitations of existing methods, and we believe the flow and logic of the description get better. We also introduce the necessitate of the research below.

Our task of generating multi-camera videos from diverse viewpoints is an under-explored problem that also enables some unique applications.

1. Film Production: In real-world filmmaking, switching back and forth between multiple angles is a commonly used technique to create a storytelling atmosphere. For example, in a conversation between two people, the camera might switch from focusing on one speaker to a wide shot of the entire scene, maintaining temporal continuity. Generating synchronized multi-view videos can meet this need. For instance, a creator can splice together the video from view_i during the time period t_0 to t_1 with the video from view_i+1 during the time period t_1 to t_2 to create the desired sequence. In contrast, monocular video with a dynamic camera can not achieve this effect because a single camera cannot instantly switch between two vastly different perspectives.
2. Used as Training Data for Downstream Tasks: Synchronized multi-view video generation can be used as training data for various downstream tasks. For example, understanding a scene from multiple viewpoints is often necessary for visual robotic manipulation [1, 2, 3]. Additionally, in 3D human pose estimation [4, 5, 6], multiple viewpoints are required to accurately estimate the 3D posture of a person. To this end, we designed SynCamMaster as an exploration of using diffusion models for open-domain synchronized multi-view video generation.

[1] Akkaya, Ilge, et al. "Solving rubik's cube with a robot hand." arXiv preprint arXiv:1910.07113 (2019).

[2] James, Stephen, et al. "Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation." CVPR 2022.

[3] Seo, Younggyo, et al. "Multi-view masked world models for visual robotic manipulation." ICML, 2023.

[4] Wang, Jinbao, et al. "Deep 3D human pose estimation: A review." Computer Vision and Image Understanding, 2021.

[5] Dong, Junting, et al. "Fast and robust multi-person 3d pose estimation from multiple views." CVPR 2019.

[6] Mitra, Rahul, et al. "Multiview-consistent semi-supervised learning for 3d human pose estimation." CVPR 2020.

Unclear exposition about the model formulation and the input/output of some key components.

Thank you for your reminder! We have made the following updates:

1. Added a more detailed illustration figure and corresponding introduction of the base text-to-video generation model in Appendix A.
2. Modified the pipeline figure (Fig. 2) in the main text to highlight the correspondence between Fig. 2(a) and Fig. 2(b).
3. Introduced the loss function and optimization mechanism in Section 3.1.

How does SynCamMaster determine the rotation center and radius when generating rotating camera viewpoints?

SynCamMaster is a multi-camera video generation model that uses text prompts and camera poses as conditions. If we want to generate videos from rotating camera viewpoints, we only need to define the camera poses (R,T) of all viewpoints in such a configuration and feed them into the model as conditions. More specifically, we regard the camera coordinate system of the first camera as the global system, where we first define a circle with a specified center and radius, then we can place the cameras on the circle by specifying their translation vector T and ensure all the cameras orient to the center by specifying their pose matrix R.

Evaluate the control accuracy by estimating poses from the generated videos and comparing with the given poses.

Thank you for your valuable suggestion! We use the state-of-the-art image matcher GIM [1] to obtain the estimated pose and evaluate the camera accuracy in our paper (introduced in lines 370-373), and the results are shown in Table 4. The reason we do not use COLMAP is we found that COLMAP typically requires more cameras and higher image resolution to complete the registration. Even when we used ground truth images rendered by Unreal Engine with 4 cameras at a resolution of 384x672, the registration often failed. Therefore, we chose GIM as an alternative.

[1] Shen, Xuelun, et al. "GIM: Learning Generalizable Image Matcher From Internet Videos." arXiv preprint arXiv:2402.11095 (2024).

Dear Reviewer bsPL,

Thanks again for your elaborate review and the prompt reply! We address your concerns below:

We would like to explain that the rotation center and radius are not determined by the model itself, but are uniquely defined by the relative extrinsic matrix $[R|T]$ of the camera. We provide a detailed explanation using an example of two cameras positioned on a circular arc, both facing the center of the circle, with an azimuth angle difference of $n$ degrees:

Firstly, according to the principles of imaging, the rotation center lies on the principal ray (the ray that passes through the camera's optical center and the principal point on the image plane). Secondly, for any two camera positions with an azimuth change of $n$ degrees, we can derive the rotation radius $r$ from their relative translation vector $T$ using the formula $r$=norm(T) / 2 / sin(n/2). Therefore, the rotation radius is uniquely determined by the relative pose $[R|T]$. In other words, given the same video input and the same relative pose, the model should output a video with a consistent rotation radius.

To verify this, we conducted the following two sets of experiments:

1. We performed two inferences with SynCamMaster using the same input reference video, relative extrinsic matrix, and text prompt, but with different random seeds. The results are shown in Figure A on the project page (https://syncammaster.github.io/SynCamMaster/#Fig_A, at the bottom of the page if the link doesn't jump). It can be observed that the output variations due to different random seeds have consistent relative angles, with differences only in the outpainting areas where the new view does not overlap with the input view.

2. Given the relative extrinsic matrix $[R|T]$, input reference video $V_i$, and text prompt $P_t$, we used SynCamMaster to generate a new view video $V_o =$SynCamMaster$(V_i, [R|T], P_t)$. Then, using the generated new view video as input, we generated a video with $[R|T]^{-1}$ and $P_t$ as conditions, i.e., $V_{i}' =$SynCamMaster$(V_o, [R|T]^{-1}, P_t)$. The results are shown in Figure B (https://syncammaster.github.io/SynCamMaster/#Fig_B). It can be observed that the final output $V_{i}'$ is almost identical to the input reference video $V_i$ in terms of viewpoint, which verifies our explanation.

We hope we have correctly understood your concerns and addressed them. We would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues.

We thank the reviewer for the insightful comments, and we address the concerns below:

Evaluate the generated videos by 4D reconstruction.

Thanks for the advice! We add the 4D reconstruction results with 4DGS [1] to our project page (https://syncammaster.github.io/SynCamMaster/). Specifically, we condition SynCamMaster on four cameras with 10° difference in azimuth and the text prompt to synthesize multi-view videos. Then, we use the four view camera parameters and our generated videos as input to 4DGS [1], and follow their open-source code (https://github.com/hustvl/4DGaussians) to train the 4D Gaussian splatting. As exhibited, the rendered novel views are overall consistent with fewer artifacts. It's worth noting that it's quite challenging to reconstruct 4D with sparse views (4 views in the case) at low resolution (384x672 in the case), therefore the performance can be further improved by 1) utilizing a T2V model that is able to generate higher resolution videos; 2) train SynCamMaster with more viewpoints (e.g., >10) on larger memory GPUs.

[1] Wu, Guanjun, et al. "4d gaussian splatting for real-time dynamic scene rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Robustness to out-of-distribution poses.

Thank you for your valuable suggestion! We have added results on our project page (https://syncammaster.github.io/SynCamMaster/) with the input of a 3-meter difference in adjacent views, therefore view1 and view4 are 9 meters apart (greater than the maximum interval of 5.5 meters in the training data). The results demonstrate that our model has promising generalization capabilities, effectively generating views with significant visual differences, which meets the requirements of most real-world applications.

Evaluate the inter-view consistency by cycle consistency check.

Thank you for the advice! Since SynCamMaster is designed to generate multi-view videos from text and camera poses simultaneously, we cannot generate some views first and then condition on the generated videos to create new views for a cycle consistency check. As an alternative, we assess 3D consistency in Tab. 1&2 by:

1. Using automatic metrics following previous work SV4D [1], including FVD-V and CLIP-V.

2. Calculating the number of matching pixels estimated by GIM [2] with confidence greater than the threshold, denoted as Mat. Pix.

Additionally, we add qualitative results on 4D reconstruction on our project page.

[1] Xie, Yiming, et al. "Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency." arXiv preprint arXiv:2407.17470 (2024).

[2] Shen, Xuelun, et al. "GIM: Learning Generalizable Image Matcher From Internet Videos." arXiv preprint arXiv:2402.11095 (2024).

About dataset release.

Yes, we will release the multi-view video data rendered by the UE engine.

How were the data mixture probabilities (0.6, 0.2, 0.2) chosen, and what role do they play in the model’s performance?

Thanks for the advice! We added the analysis below in Appendix D:

Discussion on the Data Mixture Strategy In the paper, we jointly train our model on multi-view video data, multi-view image data, and single-view video data with probabilities of 0.6, 0.2, and 0.2, respectively. We also explored the impact of different mixing ratios on the generated results. We found that a higher proportion of multi-view image data disrupts the temporal continuity of the videos. On the other hand, using too much single-view video data causes the model to favor synthesizing views with small relative angles, affecting the camera accuracy. Therefore, we sample multi-view image data and single-view video data with small probabilities.

Thanks for your feedback!

Yes, the reconstruction quality is primarily affected by the sparsity of the views. Generating more views could further improve the 4D reconstruction results. To verify this, we have updated the project page (https://syncammaster.github.io/SynCamMaster/) with the reconstruction results using 4-view videos from the Plenoptic Dataset [1]. Since the Plenoptic Dataset is recorded using synchronized camera setups to capture multi-view videos, it can be considered as "ground truth videos" to evaluate the performance of 4DGS [2] under sparse-view settings. Specifically, we first perform the center crop on the videos from the Plenoptic Dataset to match the aspect ratio of SynCamMaster. Then, we resized them to the same resolution as SynCamMaster to use as input for 4DGS. It can be observed the reconstruction results are competitive with reconstruction from videos generated with SynCamMaster, indicating that the performance is mainly limited by the number of available views. The reconstruction performance can be further improved by training SynCamMaster with larger memory GPUs to generate more views.

Additionally, the 4D consistency of SynCamMaster outperforms baselines by a large margin. It is evidenced by the number of matching points in Tab. 1, where we utilize image matcher GIM [3] to calculate matching points for the same frame across different views and average them over the temporal dimension.

[1] Li, Tianye, et al. "Neural 3d video synthesis from multi-view video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Wu, Guanjun, et al. "4d gaussian splatting for real-time dynamic scene rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Shen, Xuelun, et al. "GIM: Learning Generalizable Image Matcher From Internet Videos." ICLR 2024.

We thank the reviewer for the insightful comments, and we address the questions below:

Why not adopt more explicit camera pose representation, like ray positional encoding?

Thanks for the advice! Following SPAD [1], we attempted to transform camera poses into ray positional encoding and input them into the camera encoder. However, we observed that there was no significant difference in model performance between the two representation methods:

We also present the qualitative results in Fig. 10. As shown, there is no significant difference in the generated results between the two representations. We assume this is because the UE data (rendered multi-camera synchronized videos) has the same camera intrinsics, and in this case, both representations contain consistent information.

We add the results in Appendix D.

[1] Kant, Yash, et al. "SPAD: Spatially Aware Multi-View Diffusers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Explain the UE style issue of Fig. 6(a).

We believe there might be some misunderstandings regarding Fig 6(a). In Fig 6(a), we used text prompts from the validation set, which correspond to the prompts for UE-rendered scenes. Therefore, the model outputs videos that are similar to the rendered scenes, which is correct.

Regarding how we prevent the model from overfitting to UE rendering scene features, we: 1) incorporated more diverse real-world image (DL3DV-10K) and video (with stationary camera) data, as introduced in section 3.3; 2) during training, we froze the base T2V generation model and only trained the newly added modules to maximize the model's generalization ability. In our experiments, we found that these two designs effectively prevent the model from "only generating UE-rendered scenes" and enable it to generate multi-view videos in diverse scenes based on text prompts, as shown at https://syncammaster.github.io/SynCamMaster/.

The rotation angle in Fig. 6 (b) appears noticeably less than 90 degrees. Additionally, the precision of camera control is low, as indicated in Table 4.

First, we would like to clarify that the first and second rows of Fig 6(b) denote the results of baseline methods that do not use diverse multi-view image data for training. So, these models exhibit poor generalization in open-domain settings, causing low accuracy of camera control (noticeably less than 90 degrees). In contrast, the third and fourth rows, as the results of models trained with diverse multi-view image data for joint training, demonstrated a substantial accuracy improvement compared to the first and second rows. Additionally, we have indeed found that generating videos with large angular changes is very challenging for the model, resulting in greater errors compared to generating videos with smaller relative pose differences. We believe this is primarily due to the limited scale of available multi-view video data (500 scenes, 36 cameras each), which is a challenge for this task.

The proposed multi-view synchronization module is essentially a cross-attention layer, which lacks explicit modeling of 3D information.

Thank you for your suggestion. We attempted explicit modeling of 3D information by implementing epipolar attention in the view-attention layers. Specifically, when implementing epipolar attention, for the token at spatial position $(x, y)$ in view $i$, it only aggregates features from all tokens within view $i$ and tokens on the epipolar line in other views. In contrast, each token attends to all tokens in all views in the full attention setting. We found that although epipolar attention has low rotation error, it can result in inconsistency with the text prompt semantics (shown in Fig. 11) compared to the full-attention design. The results are presented in Appendix D.

On the other, we found that it is sufficient to learn the spatial correspondence in a data-driven manner without explicit 3D modeling, which is consistent with the findings of recent work [1].

[1] Jin, Haian, et al. "Lvsm: A large view synthesis model with minimal 3d inductive bias." arXiv preprint arXiv:2410.17242 (2024).

Dear Reviewer zv9N:

Thanks again for the constructive comments and the time you dedicate to the paper! Many improvements have been made according to your suggestions and other reviewers' comments. We also updated the manuscript and summarized the changes above. We hope that our responses above and the revision will address your concerns.

Since the discussion is about to close (Dec. 2nd), we would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues.

Thanks a lot, and with sincerest best wishes

Submission 1274 Authors

Thanks for providing encouraging comments on our paper! We are encouraged to see that SynCamMaster is acknowledged to present an effective approach to multi-camera video generation, with innovative data construction and training strategy, as well as high-quality visual results. We address your concerns below:

Lack of Comparison with MotionCtrl.

Thanks for the suggestion! We add qualitative comparisons with MotionCtrl on our project page (https://syncammaster.github.io/SynCamMaster/).

We thank the reviewer for providing encouraging comments on our paper. We provide clarifications to the concerns below:

The proposed approach can only generate a single video at a novel viewpoint each time.

We would like to claim that SynCamMaser is designed to generate multiple videos from different viewpoints simultaneously, rather than generate one video each time. Specifically, during inference, we denoise $n$ noisy latents simultaneously, and their features interact through the designed multi-view synchronization module, resulting in the simultaneous output of $n$ videos. In our paper, we validated the effectiveness of our method under three settings of $n=2, 4$. We believe it meets the needs of most practical applications and can further increase $n$ if sufficient GPU memory is available.

Show more visual results of different cases.

Thank you for the advice! We have updated the project page (https://syncammaster.github.io/SynCamMaster/) with more video results in the Demos and More Results sections.

Around the line #189, should $\phi_t$ be $\psi_t$? And what do $\psi_t^{-1}$ and $\psi_t'$ mean, respectively?

Yes, $\phi_t$ should be $\psi_t$ in line 189, we have corrected the typo in the main text, thanks for pointing it out! $\psi_t^{-1}$ represents the inverse function of $\psi_t$, and $\psi_t'$ represents the first derivative of $\psi_t$.

More details about the real-world monocular videos used in this paper.

Thank you for your suggestion! We introduce the construction process of real-world monocular videos in lines 296-303 of the main text, accompanied by a detailed description in lines 869-879 of the appendix.

In the line #140, does $f$ indicate the number of frames?

Yes, we add the explanation in line 142.

Is there any plan to release the code?

Yes, we have transferred the proposed method to the open-source video generation models CogVideoX-2B and CogVideoX-5B [1], and the performance is generally comparable to that in our paper. We will open-source the code implemented on these CogVideoX models and release the multi-view video data rendered by the UE engine used in the paper.

[1] Yang, Zhuoyi, et al. "Cogvideox: Text-to-video diffusion models with an expert transformer." arXiv preprint arXiv:2408.06072 (2024).

We thank the reviewer for the detailed feedback on our paper!

Clarification on the motivation of our task.

Our task of generating multi-camera videos from diverse viewpoints is an under-explored problem that also enables some unique applications.

1. Film Production: In real-world filmmaking, switching back and forth between multiple angles is a commonly used technique to create a storytelling atmosphere. For example, in a conversation between two people, the camera might switch from focusing on one speaker to a wide shot of the entire scene, maintaining temporal continuity. Generating synchronized multi-view videos can meet this need. For instance, a creator can splice together the video from view_i during the time period t_0 to t_1 with the video from view_i+1 during the time period t_1 to t_2 to create the desired sequence. In contrast, monocular video with a dynamic camera can not achieve this effect because a single camera cannot instantly switch between two vastly different perspectives.
2. Used as Training Data for Downstream Tasks: Synchronized multi-view video generation can be used as training data for various downstream tasks. For example, understanding a scene from multiple viewpoints is often necessary for visual robotic manipulation [1, 2, 3]. Additionally, in 3D human pose estimation [4, 5, 6], multiple viewpoints are required to accurately estimate the 3D posture of a person. To this end, we designed SynCamMaster as an exploration of using diffusion models for open-domain synchronized multi-view video generation.

[1] Akkaya, Ilge, et al. "Solving rubik's cube with a robot hand." arXiv preprint arXiv:1910.07113 (2019).

[2] James, Stephen, et al. "Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation." CVPR 2022.

[3] Seo, Younggyo, et al. "Multi-view masked world models for visual robotic manipulation." ICML, 2023.

[4] Wang, Jinbao, et al. "Deep 3D human pose estimation: A review." Computer Vision and Image Understanding, 2021.

[5] Dong, Junting, et al. "Fast and robust multi-person 3d pose estimation from multiple views." CVPR 2019.

[6] Mitra, Rahul, et al. "Multiview-consistent semi-supervised learning for 3d human pose estimation." CVPR 2020.

Evaluating 4D consistency of our results.

Thanks for the advice! We add the 4D reconstruction results with 4DGS [1] to our project page (https://syncammaster.github.io/SynCamMaster/). Specifically, we condition SynCamMaster on four cameras with 10° difference in azimuth and the text prompt to synthesize multi-view videos. Then, we use the four view camera parameters and our generated videos as input to 4DGS [1], and follow their open-source code (https://github.com/hustvl/4DGaussians) to train the 4D Gaussian splatting. As exhibited, the rendered novel views are overall consistent with fewer artifacts.

It's worth noting that it's quite challenging to reconstruct 4D with sparse views (4 views in the case) at low resolution (384x672 in the case). To verify this, we have updated the project page (https://syncammaster.github.io/SynCamMaster/) with the reconstruction results using 4-view videos from the Plenoptic Dataset. Since the Plenoptic Dataset is recorded using synchronized camera setups to capture multi-view videos, it can be considered as "ground truth videos" to evaluate the performance of 4DGS [1] under sparse-view settings. Specifically, we first perform the center crop on the videos from the Plenoptic Dataset to match the aspect ratio of SynCamMaster. Then, we resized them to the same resolution as SynCamMaster to use as input for 4DGS. It can be observed the reconstruction results are competitive with reconstruction from videos generated with SynCamMaster, indicating that the performance is mainly limited by the number of available views. The reconstruction performance can be further improved by training SynCamMaster with larger memory GPUs to generate more views.

On the other, the 4D consistency of SynCamMaster outperforms baselines by a large margin. It is evidenced by the number of matching points in Tab. 1, where we utilize image matcher GIM [2] to calculate matching points for the same frame across different views and average them over the temporal dimension.

[1] Wu, Guanjun, et al. "4d gaussian splatting for real-time dynamic scene rendering." CVPR 2024.

[2] Shen, Xuelun, et al. "GIM: Learning Generalizable Image Matcher From Internet Videos." ICLR 2024.

Evaluation data and metrics should be explained.

For the evaluation data, we use prompts synthesized by our finetuned LLM across various scenarios. We will open-source these prompts to facilitate future comparative research. Additionally, we include experiments using the open-source VBench prompts, and the results are exhibited in Tab. 7 in Appendix E. For the evaluation metrics, we add a detailed introduction to the metrics in Section 4.1.

We thank the reviewer for the elaborate review! We will address your concerns below:

Response to the comment that Sora possesses multi-camera video generation capabilities.

We have not found any indication in Sora's technical report that it has the capability to generate "multi-view synchronized videos". We would greatly appreciate it if you could point us to any documentation or information that demonstrates Sora's ability in this regard.

Emphasize how SynCamMaster distinguishes itself from existing methods since the inter-view synchronization module is similarly designed.

As you mentioned, SynCamMaster has a similar module design with some methods in related fields. However, we would like to emphasize our distinct contributions:

1. We explore a new task, specifically the generation of open-domain multi-view synchronized videos.
2. We identify the importance of dataset construction for this task and propose a mixed dataset composed of multi-view images, multi-view videos, and general videos, as illustrated in Fig. 3.
3. We design a progressive training strategy with gradually increasing relative angles between views and employ joint training to enhance generalization and video quality.

Details about how the 12 extrinsic parameters are processed to align with the spatial feature.

Thanks for the advice! In each block, we employ a MLP (namely the camera encoder) to project the 12 parameters into feature embedding of $d$ dimension (the spatial features dimension). The corresponding description can be found in lines 214-215 in the main text.

What is M.V. Images in Tables 1 and 4.

Thanks for pointing it out! It refers to the multi-view image generated by SynCamMaster. We add the corresponding explanation at lines 423-425 in the main text.

Provide the video comparison with existing methods.

Thank you for your reminder. We have added video results comparing SynCamMaster with baseline methods on the project page (https://syncammaster.github.io/SynCamMaster/).