Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention

We thank the reviewer UoYX for the detailed comments and actionable suggestions. However, we are not able to agree with the reviewer’s over-concerning about our contribution. Thanks to our novel architectural design and our joint training strategy, our proposed framework outperforms all prior or concurrent camera-controlled video generation methods in terms of object motion quality or camera pose accuracy. Detailed responses are provided below.

Q1. The video content presented demonstrates low dynamics (object), which may limit the ability to fully showcase the capabilities of the proposed method.

We have included generated videos with large object motion in the supplementary. We have also added some of them to the main draft as suggested (Fig. 11). We invite the reviewer the evaluate these results in a comparative context. As clearly demonstrated in the supplementary videos, baseline methods fail to produce any motion, remaining purely static. In contrast, our method produces vivid object motion while maintaining camera pose control accuracy.

Q2. The contribution of this work appears to be incremental. The proposed method shares significant similarities with CVD, suggesting that the innovations may be more evolutionary than revolutionary.

We’d like to remind the reviewer that CVD is a concurrent work, as mentioned by the ICLR 2025 policy. We have already included their results for comparison in our supplementary materials. It can be clearly seen that our method produces significantly more accurate camera control and much clearer reconstructed 3D scenes. Below we provide detailed analysis and comparisons.

• Task : CVD is a concurrent work designed for text-to-video generation, while our focus is image-to-video generation. We have compared qualitatively and shown great improvement in camera pose accuracy and 3D reconstruction quality.
• Arch: The CVD framework is limited to leveraging only two views. In contrast, our proposed cross-view attention adopts a fundamentally different approach, allowing us to flexibly train and test with an arbitrary number of views. This capability enables joint training on both monocular and multi-view video datasets and facilitates generalization to four views during testing. By sampling four views simultaneously in a single pass, our approach is more efficient and accurate compared to CVD, which requires enumerating all two-view pairs at each diffusion step.
• Data: We make the first attempt to jointly train on real multi-view and monocular videos, instead of using homography augmented data in CVD.The reliance on pseudo multi-view data without realistic camera motion in CVD makes it challenging to achieve complex camera motions. Furthermore, the black regions introduced by homography augmentation in CVD often manifest as black artifacts in their generated outputs, as shown in our supplementary materials.

Q3. The trajectories showcased in the supplementary demo are relatively simple, which raises questions about the model's ability to generalize to more complex scenarios. Further demonstration on a broader range of data would help to substantiate the method's effectiveness.

We have included more complex cameras in the supplementary materials in Figure F. We’d like to mention that neither the camera poses nor the testing images are seen during the model training, indicating outstanding generalization ability of the proposed model.

Q4. The camera conditions are normalized to a unit scale, can this approach still achieve variable speed camera control similar to MotionCtrl?

The camera speed control in MotionCtrl is implemented by scaling the camera translation along trajectories. However, because our framework normalizes the translation of the entire trajectory, it does not support changing the scale of the trajectory in the exact same way. This limitation is acknowledged in our limitations section.

It is important to note that the scale implementation in MotionCtrl is not metric-scale. Their trajectory scaling relies on cameras estimated by COLMAP, which assigns an arbitrary scene scale for each separate COLMAP run. As a result, MotionCtrl does not achieve valid speed control. Moving forward, we aim to explore the use of metric-scale data to enable more precise camera speed control.

That said, our current framework does support changing the speed of camera motion. By setting the camera position of the final frame to a fixed distant location, we can arbitrarily adjust the speed of the cameras for the remaining frames, achieving variable-speed camera control.

Dear Reviewer UoYX,

Thank you for your valuable feedback. To address your concerns about the low object motion in our results, we have provided a new Fig. 14, which offers a direct comparison between our method and our base model, SVD. As highlighted by Reviewer T6Cg, the object motion generated by our approach is comparable to that of SVD. Furthermore, Cavia introduces critical capabilities, such as precise camera control and motion synchronization for multi-view video generation, which are not supported by SVD. This comparison demonstrates that the level of object motion in our results reflects the inherent characteristics of SVD, which we selected as our base model for its status as the best-performing image-to-video model available at the time of our study. Moving forward, we are eager to integrate our proposed innovations into more advanced base models as they become publicly accessible.

While methods like Sora from OpenAI are recognized for producing larger object motion in standard monocular video generation, they are not designed for camera-controllable multi-view video generation. Our extensive experiments validate that Cavia significantly outperforms existing research works in this domain.

We greatly appreciate your thoughtful evaluation and hope that the additional qualitative comparisons and the favorable assessments by other reviewers (T6Cg, EfvF) provide a clearer understanding of our contributions. We kindly request that you consider revisiting your assessment in light of this new evidence. We have put significant effort into building Cavia and believe our work offers substantial value to the research community, particularly in advancing controllable multi-view video generation.

We thank the reviewer FTwe for the reviewing time and constructive suggestions. However, we respectfully disagree with the reviewer’s concerns regarding the scale of object motion. We encourage the reviewer to evaluate our results in a comparative context. In the supplementary videos, it is clearly shown that the other camera-controlled video generation frameworks are only able to generate pure static videos, whereas our approach demonstrates vivid and dynamic object motion.

Q1. most of the examples presented in the paper involve minimal object movement.

We have examples with large object motion in the supplementary. We invite the reviewers to evaluate our results in a comparative context. While the results presented in the paper do not showcase extremely large object motion, it is evident that baseline methods are restricted to producing purely static outputs, regardless of the content. In contrast, our proposed method addresses this limitation in controllable multi-view video generation by presenting vivid and dynamic object motions.

Q2. For example, I would like to see examples like this: "on a street with vehicles and pedestrians, a horse is running, with one camera capturing a global view from behind the horse and another camera tracking from the side". Showing more examples like this would be more convincing and demonstrate the value of controllable multi-view video generation.

We agree that the proposed prompt is more encouragingadvancing the field of controllable multi-view video generation. However, it exceeds the capabilities of our current base model, Stable Video Diffusion (SVD). When working on this project, no superior open-source image-to-video models were available. Looking ahead, we are eager to integrate our proposed innovations into more advanced base video models as they become publicly accessible.

Q3. Most of the cases in the paper, are close to the style of Objeverse dataset. Is it because the fulltuning destroyed the capabiliy of SVD in generating realistic style and scene? If so, this should be claimed as a limitation, especially compared to those adapter plugin methods like MotionCtrl and CameraCtrl.

We have provided results with complex scenes (e.g. street views, natural scenic and indoor scenes) in the supplementary. It is important to note that our evaluation methodology fundamentally differs from prior works that focus exclusively on the Objaverse dataset. Unlike these approaches, all of our evaluations involve rich backgrounds, requiring the model to accurately comprehend the geometry of the scenes.

Regarding the MotionCtrl and CameraCtrl approaches, the image-to-video models released by their authors generate only purely static outputs. This limitation is undesirable for controllable multi-view video generation, where object motion plays a critical role. In comparison, our method demonstrates the ability to generate vivid object motion, guided by precise camera instructions, even in complex scenes.

Q4. From the results in Fig. 8, it appears that while the introduction of monocular video increases generalization ability, such as maintaining the content of the input images, it also affects the accuracy of multi-view camera control. In the second row of (b), the shooting direction for the fish and the fox did not turn as expected.

Thanks for raising this good question. We’d like to clarify that the differences in Fig. 8 (a) and (b) are not related to the accuracy of multi-view camera control.

• For the fox example, the variation in the fox’s movement across different sampled videos arises because our framework does not control object motion. However, examining the viewpoint change around the fox’s feet reveals that the camera motion remains consistent.

• For the fish example, the behavior described as "did not turn as expected" occurs because our camera control is not metric-scale. Instead, the scene’s scale is implicitly determined by the model. It is clearer shown in the dynamic video that the Fish(b) exhibits a smaller translation scale compared to Fish(a). Therefore, the coral reef appears to rotate differently.

Due to the lack of a high-quality open-source metric-scale depth model as well as high-quality object motion annotations, we leave such explorations for future work. Nonetheless, as shown in Tab. 3, the introduction of monocular video does not negatively impact the multi-view camera control accuracy.

Thank you for sharing your thoughts and highlighting the importance of our multi-view video generation task. We appreciate your feedback and would like to address your concerns to clarify any misunderstandings:

• Generalization ability:
  • We would like to emphasize that MotionCtrl and CameraCtrl demonstrate results exclusively on in-distribution camera trajectories derived from RealEstate10k. In contrast, we additionally present out-of-distribution results, utilizing trajectories from DL3DV-10k, CO3Dv2, monocular videos, and even hand-crafted trajectories. This represents a significant improvement in generalization to diverse camera trajectories compared to prior work.
  • The Objaverse dataset serves as a critical data source for supporting our multi-view video motion generation capabilities. Our model leverages this dataset, along with a carefully designed data mixture and training strategy, to produce results with realistic backgrounds, demonstrating superior generalization to diverse real image content. In contrast, SV3D, IM-3D, and VideoMV exhibit poorer generalization, generating artificial backgrounds composed solely of uniform colors.
• Data source:
  • Previous camera-controllable methods are limited to generating static results, a challenge further compounded in multi-view scenarios due to the absence of high-quality multi-view video datasets. A concurrent work, CVD, attempts to address this limitation by augmenting pseudo-multi-view data from single-view videos. However, as shown in our supplementary materials, this approach exhibits poorer consistency compared to our method.
  • We propose leveraging Objaverse objects to ensure precise camera controllability and synchronized object motion across multi-view videos. Unlike previous methods such as SV4D, 4DGen, and DreamGaussian4D, which are limited to producing foreground elements, our approach demonstrates exceptional geometric consistency for both foreground and background. Furthermore, our results maintain synchronization when object motion is present.
• Object motion strength:
  • While SVD allows a wide range of motion strengths as a hyperparameter, we fixed the motion strength to 127 (which is the recommended value) across all experiments, including the fine-tuning of Cavia. This ensures that Fig. 14 provides a fair comparison of motion strength capabilities before and after fine-tuning, highlighting Cavia as a robust and effective multi-view video generation framework. We also emphasize that previous camera-controllable video generators based on SVD were limited to producing static results, underscoring Cavia’s contribution to the field.

We hope this additional evidence clarifies your concerns. To the best of our knowledge, no other research work achieves performance comparable to Cavia, underscoring its significant contribution to the research community. We appreciate your high expectations and rigorous standards for Cavia. However, we wish to emphasize that the factors you prioritize most, such as object motion and generalization capabilities, are precisely the areas where Cavia excels. Our approach outperforms all related works by a substantial margin in the areas you mentioned.

We thank the reviewer EfvF for the favorable assessments and detailed comments. Our responses to your concerns are listed as follows.

Q1. What are the "general" videos come from? Compared to the RealEstate10K dataset, does the "general" videos have larger camera motion? And does the dynamic complexity is limited?

The ``general’’ videos used for evaluation are sourced from our monocular video dataset. Based on our observations, these videos typically exhibit smaller camera motion compared to the RealEstate10K dataset. This difference arises because most YouTube videos are not characterized by simultaneous large object motion and extensive camera motion. However, these videos contain rich object motion, in contrast to RealEstate10K, where the majority of scenes are static.

Q2. In Table 2, there should be some quantitative comparisons with the CVD model.

Unfortunately, the CVD model has not been publicly released, limiting our comparisons to qualitative analysis. We have provided the comparisons in our supplementary materials (Fig. C and Fig. D). It can be clearly seen that our generated videos enjoy better consistency and improved pose accuracy. Moreover, our generated videos can be reconstructed into clearer and more coherent 3D scenes.

Q3. In the provided videos, the object dynamic degree is limited, can you provide a comparison on the object dynamic degree between CAVIA and the base video generation model SVD?

Thanks for raising this good question. We also observe that two view generation reduces the object motion to some extent. This can be seen in our ablation study (“Ablation on cross-view attention”). Specifically, we find that our model variant without the cross-view attention exhibits larger object motion compared to the one with cross-view attention.

We hypothesis that this behavior may be attributed to the model’s capacity. Since the cross-view attention requires our model to generate synchronized object motions, it tends to produce only those motions that can be confidently synthesized from multiple viewpoints. This task is inherently more challenging than monocular camera-controllable video generation and even more so than arbitrary video generation without camera pose, as seen in Stable Video Diffusion (SVD).

We have also provided a side-by-side comparison between Cavia and SVD in Fig. 14. The frames generated by our method exhibit rich object motion that is comparable to, if not better than, that of SVD. For example, our penguin, giraffe, and bird examples are behaving realistically, while SVD presents limited object motion.

Dear Reviewer EfvF,

As the author-reviewer discussion deadline is approaching, could you please take a look at the authors' rebuttal (if not yet) and see if it addressed your concerns or if you have any further questions?

Best,

Authors

We thank the reviewer T6Cg for their time and actionable suggestions. We have updated the presentation of sections 3 and 4 as suggested. Regarding the concern about our method being minimal, we would like to emphasize that our contributions include both advancements in model architecture and the curation and analysis of data. For other questions, we also provide detailed responses as below.

Q1. The visual examples provided in Figure 5 lack noticeable differences, which undermines the claimed improvements. What factors contribute to the significant metric-based improvements under the 2-view setting despite the subtle visual changes?

We include examples with more noticeable differences in the supplementary and have incorporated them into the updated draft (Fig. 11). We have also included side-by-side comparisons between our method and the base model (SVD) in Fig. 14 to validate the object motion strength we generate. Our method outperforms the camera-controllable baseline methods by a large margin in two aspects:

• Object motion: When examining dynamic videos rather than individual static frames, it is evident that the baseline methods generate predominantly static results. In contrast, our method demonstrates vivid and realistic object motion.

• Pose accuracy: In camera-controllable video generation, achieving precise control over camera pose is critical. While humans are adept at perceiving the general direction of camera motion, they often struggle to assess its accuracy. To address this, we systematically evaluate the pose accuracy of the generated frames using COLMAP and SuperGlue. Our results show that our method achieves the most accurate camera pose control among all compared approaches.

Q2. Poor Presentation: (i) The method section is minimal. (ii) The paper includes non-standardized expressions, such as "(B V F C H W)" in Line 222. (iii) Section 4 is ambiguously presented, making it difficult to comprehend.

We have updated our sections 3 and 4 as suggested by the reviewer. Please kindly note that our proposed framework involves innovations both in terms of the model architecture and the data construction. We provide a detailed analysis of the joint training strategy that drives our model's performance, as well as the data curation workflow that supports this strategy.

Q3. Given the extended token length in the cross-frame attention module, does this component impose a significant demand on GPU memory?

Our proposed Cross-frame Attention requires extra GPU memory compared to the vanilla 1D temporal attention used in Stable Video Diffusion. However, it’s not a significant demand. During training, our model comfortably fits into an A100 GPU with 80G memory.

Dear Reviewer T6Cg,

As the author-reviewer discussion deadline is approaching, could you please take a look at the authors' rebuttal (if not yet) and see if it addressed your concerns or if you have any further questions?

Best,

Authors

Thanks for the prompt response. Can you please clarify which part is confusing about the method illustration? We will be happy to revise the draft accordingly.

In Fig. 11, we provide 2-view videos generated by Cavia. For each set of two rows, the first row and the second row refer to the two different views.

In Fig. 14, we compare the motion strength of our method against SVD, which serves as our base model. While our approach performs on par with SVD in terms of motion strength, it additionally enables precise camera control and supports motion synchronization for multi-view video generation. In contrast, SVD is specifically designed for generating standard monocular videos without any control mechanisms. We appreciate your recognition that our method achieves motion strength comparable to our base model.

By saying Cavia outperforms the baseline methods by a large margin, we refer to the other camera-controllable video generation methods. Previous camera-controllable video generation methods are all limited to imprecise camera controls and pure static scenes.

Since the deadline for updating the PDF has passed, we have documented these points and will upload a revised version in the future.

Dear reviewers,

Thank you again for the valuable feedback that significantly enhanced our submission. As the author-reviewer discussion deadline is approaching, we wish to emphasize that Cavia excels in the very aspects you prioritize most (e.g. model design, object motion, and generalization capabilities) when compared with related research works. The development of Cavia demanded substantial effort, and we firmly believe it represents a significant contribution to the research community. Notably, no other work has achieved comparable performance in camera-controllable multi-view video generation. Therefore, we respectfully request that you revisit your assessment and evaluate our work fairly, within a comparative framework aligned with the standards of the research community.

Thank you,

Authors of Paper 2215