VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control

We deeply appreciate the reviewer's detailed and positive assessment of our work. Below, we would like to clarify several concerns.

Novelty concern (part 1/2): Plucker camera embeddings have already been explored in CameraCtrl.

We respectfully want to emphasize that our work not only developed the use of Plücker coordinates for camera control independently, but our core technical innovation lies in the transformer-centric design. We developed a new projection scheme that effectively integrates camera information into the representation space of video tokens, which is crucial for joint spatio-temporal processing in transformer architectures. This technical challenge required novel solutions distinct from those used in traditional UNet-based approaches, as transformers handle information flow and processing in fundamentally different ways.

Novelty concern (part 2/2): Using RealEstate10K has already been explored in MotionCtrl.

We believe there is a misunderstanding: we do not argue that the use of RealEstate10K is a novel component of our work. The key novelty of our paper lies in exploring and designing a lightweight and precise camera conditioning mechanism for transformer-based diffusion models.

How would the method perform on top of DiT?

That’s a great question! We incorporate our approach into a vanilla DiT model trained in the latent space of CogVideoX [1]. We include these results as part of the revised paper in Table 10 and Table 11. We furthermore include visual samples into the rebuttal.html.

Instead of building on top of read attention in FIT, we incorporate the ControlNet conditioning on top of the vanilla attention mechanism of the actual tokens. We provide evaluations in the two Tables below and observe that the vanilla DiT version further improves quality and camera accuracy on out-of-distribution prompts (MSR-VTT). We believe that our proposed method of spatio-temporal Plucker tokens and aligning them with video patch tokens through a ControlNet-type of conditioning mechanism is agnostic to the transformer architecture and will serve as a starting point for follow-up works.

[1] Yang et al., Cogvideox: Text-to-video diffusion models with an expert transformer, arXiv 2024

MotionCtrl backbones should be frozen.

We already trained and evaluated a variant of MotionCtrl where the backbone is frozen in Sec. B.3 with comparisons in Tables 5, 6, and 7, as part of the appendix. While the quality and motion is less degraded, this variant further degrades the camera control since such design lacks enough expressivity to follow a viewpoint conditioning signal.

The amount of trainable parameters is unknown.

We use 230M trainable parameters for VD3D. For fair comparisons, we used for both MotionCtrl and CameraCtrl experiments the identical number of parameters to also match GPU memory usage. We train all variants with the same batch size and same number of iterations. We included this information in Appendix C.

Use R’_f and t’_f to represent normalized extrinsics.

We agree with the reviewer on this observation and thank them for the valuable suggestion. We updated Section 3 to accommodate this improvement.

Baseline implementation details about MotionCtrl/CameraCtrl.

Correct, MotionCtrl uses a simple MLP to map from input cameras to camera embeddings that are concatenated to the video patch tokens. CameraCtrl uses a more complex camera encoder including temporal attention and 2D convolutions to map from input cameras to camera embeddings.

What is the key ingredient of motion preservation?

The key to preserving motion lies in our carefully designed conditioning mechanism. Through extensive experimentation, we found that two components are crucial: our Plücker-based projection scheme to obtain camera tokens and the way we integrate them into video tokens of a diffusion transformer via cross-attention followed by zero-initialized convolution to preserve the model initialization. This approach effectively balances the challenging trade-off between camera control precision and motion quality. While enforcing camera control typically degrades scene dynamics, our solution achieves precise camera control while preserving natural motion.

We deeply appreciate the reviewer's detailed and positive assessment of our work. Below, we would like to clarify several concerns.

This paper is a bit of an A+B paper.

We respectfully disagree with the characterization of our work as a simple combination of existing methods. As demonstrated both quantitatively (Table 2) and qualitatively (supplementary materials), merely combining transformer-based video diffusion with standard conditioning approaches yields poor results and imprecise camera control. Developing an effective camera conditioning scheme for spatiotemporal transformers required extensive experimentation to achieve three critical goals: (1) successfully incorporating camera information into the video token representation space, (2) enforcing this naturally weak conditioning signal during synthesis, and (3) maintaining scene motion and visual quality throughout the process.

Figure 3 needs to be improved.

We appreciate this valuable suggestion. We've made several improvements to both Figure 3 and the notation in Section 3. First, we simplified the notation used to describe the method, and then we restructured the diagram to create a clearer connection between its visual elements and the variables used in the method description, making it more intuitive to read. Finally, we improved the layout to make the figure more visually appealing.

Evaluation on diverse camera trajectories and different trajectories starting from the same scene.

We appreciate these valuable suggestions regarding trajectory evaluation. While our original submission included quantitative results for out-of-distribution camera trajectories in Table 8, we have now substantially expanded this analysis. The new evaluation tests 11 distinct out-of-distribution trajectories across 1000 different scenes for both RealEstate10K and MSR-VTT, shown in the Table below. We visualize 16 prompts for these trajectories, resulting in 176 new visual results in total in the rebuttal.html as part of the supplementary file, showing additional examples where multiple trajectories are used for the same scene. This comprehensive analysis is now available in Table 9 of the revised manuscript.

We deeply appreciate the reviewer's detailed and positive assessment of our work. Below, we would like to clarify several concerns.

Will the method generalize to other transformer-based diffusion models?

That’s a great and justified question, and to answer it, we implemented our VD3D method on top of a pre-trained text-to-video DiT model in the latent space of the CogVideoX [1] autoencoder. We include these results as part of the revised paper in Table 10 and Table 11. We furthermore include visual samples into the rebuttal.html.

Instead of building on top of read attention in FIT, we incorporate the ControlNet conditioning on top of the vanilla attention mechanism of the actual tokens. We provide evaluations in the two Tables below and observe that the vanilla DiT version further improves quality and camera accuracy on out-of-distribution prompts (MSR-VTT). We believe that our proposed method of spatio-temporal Plucker tokens and aligning them with video patch tokens through a ControlNet-type of conditioning mechanism is agnostic to the transformer architecture and will serve as a starting point for follow-up works.

[1] Yang et al., Cogvideox: Text-to-video diffusion models with an expert transformer, arXiv 2024

Evaluation on out-of-distribution trajectories.

In addition to the quantitative evaluations of random out-of-distribution camera trajectories presented in Table 8 of our original submission, we have now conducted a more extensive analysis. The new evaluation tests 11 distinct out-of-distribution trajectories across 1000 different scenes for both RealEstate10K and MSR-VTT, shown in the Table below. We visualize 16 prompts for these trajectories, resulting in 176 new visual results in total in the rebuttal.html. This comprehensive analysis is now available in Table 9 of the revised manuscript.

ControlNet and Plucker camera embeddings were explored before.

Our work addresses fundamentally different challenges from ControlNet in several aspects. First, ControlNet was specifically designed for UNet architectures, while we develop conditioning mechanisms for transformer-based models, which require entirely different approaches due to their distinct computational nature. Second, while ControlNet successfully handles strong spatial signals like edge maps or depth masks that largely determine image structure, we tackle the challenging problem of camera pose conditioning— a much weaker signal that must influence both spatial and temporal aspects of video generation. The integration of such conditioning into joint spatio-temporal transformer computation presents unique technical challenges that require novel solutions.

CameraCtrl is not concurrent.

We respectfully want to clarify that our research, which began in February, developed its approach independently. Moreover, while CameraCtrl is currently under review at ICLR, we have provided comprehensive comparisons with their method throughout our paper. These comparisons demonstrate that their UNet-based approach does not translate effectively to transformer architectures, which required us to develop fundamentally different solutions for camera control in the context of joint spatio-temporal processing.

We thank the reviewer for their thorough feedback. Below, we address the raised concerns.

Overfitting on RealEstate10K camera trajectories.

We respectfully note that all our evaluations were performed on the test-set RealEstate10K camera trajectories. Regarding the out-of-distribution cameras, as the reviewer rightfully pointed out, we provide quantitative results with random camera trajectories in Table 8 in Appendix B. With the current update, we include the results for non-random, user-defined trajectories in the rebuttal.html as part of the supplementary, which will be incorporated into main.html for the final version. We also provide new quantitative evaluations for these trajectories below and in Table 9 in the revision of the paper. These new trajectories also involve camera movements with significant directional changes including rotations and translations. We observe that our method generalizes to camera trajectories with variable rotations and translations.

Plucker embeddings have already been explored in CameraCtrl.

We respectfully note that our work developed the use of Plücker coordinates for camera control independently, and our key technical contribution extends beyond just the idea of using them— we developed a novel projection mechanism that effectively integrates camera information into the representation space of video diffusion transformers with joint spatio-temporal layers. This specialized approach is essential, as transformer architectures process information fundamentally differently from UNet-based models and require unique solutions for effective camera control.

PixArt-δ has already explored ControlNets for a transformer-based diffusion model.

We respectfully note that ​while PixArt-δ developed a ControlNet-based mechanism for transformer-based diffusion, it is limited to spatial conditioning with spatial signals. Spatial conditioning signals explored by ControlNet and PixArt-δ (e.g., canny edges, depth/normal/segmentation maps, etc.) are very strong and almost fully describe the structure of the output image. In our paper, we explore conditioning mechanisms for camera poses, which is a weak spatio-temporal signal with an intricate temporal component. Moreover, we explore it for transformer-based diffusion models with joint spatio-temporal computation which adds an extra level of complexity because these dimensions are entangled. We thank the reviewer for their valuable pointer and added the relevant discussion in Section 2 of the paper.

Details on the pre-training data of the backbone model.

We appreciate the reviewer's attention to the matter of training data in the backbone model upon which our work is built. However, we want to clarify that we obtained only the upstream model weights from this model, which were published in a paper at CVPR 2024. Our paper focuses entirely on our novel 3D camera control methodology, which we developed on top of this previously established model.

In this case, we believe the 3D camera control methodology should be the focus of the review as is common when new contributions are made on top of existing work. We note that many important papers have been built upon closed models, for instance:

• DreamFusion (ICLR’23 Best Paper) builds upon Imagen (NeurIPS’22 Best Paper)
• Cat3D (NeurIPS’24 Oral) builds upon an unnamed upstream LDM model.
• Magic3D (CVPR’23) builds upon e-Diff-I
• VidPanos (SIGGRAPH Asia’24) builds upon Lumiere

To ensure consistency in the criteria used to evaluate our contribution, we would appreciate a similar approach to be taken with the review of our paper.

Pre-training data can be potentially contaminated with test data.

As to the potential contamination of pre-training data with test data, we confirm that the pre-training data does not overlap with neither train nor test sets of RealEstate10K or MSR-VTT, ensuring the validity of the results reported in Table 2.

Reproducibility concern.

In our original submission, we provided a detailed description of our methodology in Section 3, comprehensive experimental details in Section 4, and complete training and architectural specifications in Appendix C. This constitutes a solid foundation for future projects to reproduce and build upon our work.

To further enhance reproducibility, we include the source code of our camera-controlled FIT block — the key component of our method — as supplementary material. Additionally, we are committed to providing any technical details that reviewers find necessary for complete reproduction of our results.

We have added a Reproducibility Statement as Section 6, summarizing our commitment to reproducibility and the available resources.

We deeply appreciate the reviewer's detailed and positive assessment of our work. Below, we would like to clarify several concerns.

Evaluation on diverse camera trajectories

We respectfully note that we provide quantitative evaluation for random (i.e., not RealEstate10K) camera trajectories in Table 8 in the appendix of the original submission. With the current update, we include 176 new visual results for non-random, user-defined camera trajectories in the rebuttal.html as part of the supplementary, which will be incorporated into main.html for the final version. We also provide new quantitative evaluations for these trajectories below and in Tab. 9 in the revision of the paper. These new trajectories also involve camera movements with significant directional changes including both rotations and translations, as suggested by the reviewer. We observe that our method generalizes to input camera trajectories with variable rotations and translations.

Novelty concern

We demonstrated in Table 2 and through qualitative results in the supplementary materials that existing ControlNet-like architectures do not directly translate to diffusion transformers with joint spatiotemporal processing. Extensive experimentation was required to develop a setup that achieves precise camera control while maintaining motion quality and visual fidelity. Importantly, Latte's work is orthogonal to ours—while they focus on developing a diffusion transformer backbone, our contribution lies in creating camera conditioning mechanisms for such models and addressing the inherent challenges that arise. We appreciate this relevant reference and have incorporated Latte into our related work discussion in the revised paper.