Boosting Camera Motion Control for Video Diffusion Transformers

1. Unclear role of conditioning dimension and CMG.

The architectural shift from U-Net to DiT introduces significant changes with multiple factors influencing performance. To better understand these effects, we conducted a series of experiments to isolate and analyze their impact.

• We established that higher camera embedding dimension leads to better result, as shown in Table 2d vs. 1b for DiT-CameraCtrl using the same RT representation and without CMG. Conditon-to-channel ratio (CCR) has now been added to Tables 1 and 2 to further clarify this relationship, highlighting that higher conditioning dimensions yield better outcomes.
• DiT-CameraCtrl is actually working despite not as well without CMG, as compared to DiT-MotionCtrl which practically has no control over camera pose. In fact, DiT-CameraCtrl achieves controllability (Table 1: 2c, rotation and translation errors) close to its U-Net counterpart (Table 1: 2a), aside from a much lower motion magnitude.

We verified that increasing the conditioning dimension and introducing CMG independently contribute to performance improvements. Although the exact cause of the limited motion in DiTs remains unclear, our proposed CMG, combined with higher conditioning dimensions, achieves a motion magnitude boost of over 400% (Table 1: 2c, Supplementary Video 1) while simultaneously enhancing controllability by reducing rotation and translation errors.

2. CMG is not for DiT only, it has the potential for U-Net

Indeed, it is a potential strength of our method. The proposal of CMG was motivated by challenges we observed in DiT, which is why the paper title reflects this focus. Our experiments in Table 1 suggest that existing camera conditioned schemes already yield reasonably effective controls in U-Net, but they are not as successful when implemented in DiT, which is why we focus on DiT. We observed that the degradation of camera control could be due to the overall less pronounced motion generated by OpenSora1.0 (both 1b and 2b in Table 1 have little motion). Our CMG effectively enhances the motion as pointed out in Table 1 and Figure 3, in turn enabling better camera control. While being conceptually applicable in both architectures, we focus on the challenging case (DiT) first and consider the U-Net experiments future work.

3. What is the purpose of sparse camera control?

Sparse camera control has practical applications in streamlining the video creation process. For instance, this approach allows directors to focus on defining key moments, such as the starting and ending camera poses, while the model intelligently interpolates the intermediate poses to create smooth and coherent camera motion. This enables creative professionals to concentrate on high-level artistic direction rather than intricate technical details.

Overclaiming to be the first camera control for space-time video DiT.

Introduced in L95-100, video DiT architectures can be broadly categorized into two types: space-time (2D+1D) and spatio-temporal (3D). Space-time (2D+1D) architectures first encode individual 2D frame into single embedding, our method injects camera pose to each video frame before batching multiple frames for temporal attention. On the other hand, spatio-temporal (3D) architectures, like VD3D uses, simultaneously encode multiple 2D images across different time slots into a single latent embedding, which lack individual image embedding for camera pose injection using methods such as CameraCtrl, MotionCtrl and ours. Hence, camera control for these 2 architectures are inherently incompatible. To sum it up, VD3D was the first in spatio-temporal (3D) DiT while ours was the first in space-time (2D+1D) DiT, collectively making contribution towards improving camera control for DiTs.

Missing experiments of DiT-CameraCtrl with RT representation.

The experiment result is already in Table 2:(2d) which shows improved result from DiT-MotionCtrl (Table 2:(1b)) by projecting the same RT representation into higher dimensional embedding. Conditon-to-camera ratio (CCR) have now been added to Tables 1 and 2 to further clarify this relationship, highlighting that higher conditioning dimensions yield better outcomes, affirming our hypothesis.

No moving object in study.

Thank you for the valuable feedback. The evaluation of camera pose control in our study relies on estimating camera poses from videos using structure-from-motion techniques, such as COLMAP, which was also used in the baseline methods (MotionCtrl, CameraCtrl) and concurrent works (Vd3D and Camco). Since moving objects can degrade pose estimation accuracy, the current standard practice of all aforementioned methods is to train and evaluate camera pose control separately from dynamic object motion. The primary goal of our work was to improve camera pose guidance. Although dynamic objects were not the focus of our experiments, we do not anticipate that our method negatively affects object motion, as demonstrated in "Supplementary Video 4," where we showcase video generation with both static and dynamic scenes. We acknowledge the need for a more comprehensive analysis of object motion and video quality, and we plan to improve this aspect in future work.

How does different setting affect video quality.

Our method finetunes only the temporal attention of DiT, hence no video quality disparity was observed for different camera representation and conditioning methods. Camera Motion Guidance (CMG) is applied only during inference, and we conducted a sweep of the CMG scale (Table 3). As expected, (L438-443) increasing the CMG scale leads to blurrier videos due to higher camera motion, resulting in a decline in FID. We found that a CMG scale between 4 and 5 achieves the optimal balance between camera control and video quality.

Insignificant novelty and no new method being proposed.

VD3D reported that existing camera control methods fail for spatio-temporal transformers, and our study demonstrates similar limitations for space-time transformers. Collectively, these findings confirm that existing methods are ineffective for DiT architectures. Despite trying different combination of methods and camera representation in our study, the camera controls were still suboptimal and the small motion magnitude renders DiT-CameraCtrl inpractical for application. It only became effective with the introduction of Camera Motion Guidance (CMG) which brings 400% improvement in camera motion.

In hindsight, adapting CFG for CMG might seem straightforward; however, none of the existing camera control methods (MotionCtrl, CameraCtrl, VD3D, CamCo) implemented this approach. We argue that being able to fully lock the viewpoint is also one important aspect of camera control, which is often neglected. We confirmed methods above often fail to do so when inputing static cameras, making them not very useful in practical video editing workfows. Our CMG explictily teaches the network when to keep the camera still during training. While it is simple in design, its impact is significant, representing an original contribution in guiding camera motion in video generation.

Is there a typo in L193? Should DiT and U-Net be exchanged order in this sentence?

Thanks for pointing out, we have fixed that in the revised paper.

Adjusting dimensions to vary condition-to-channel ratio (CCR).

DiT-CameraCtrl encodes the camera pose to match the embedding dimensions of DiT, ensuring a fixed 1:1 CCR that cannot be adjusted. In contrast, using the DiT-MotionCtrl method of concatenation inflates the input channel dimension across all linear layers, significantly increasing GPU memory usage and risking out-of-memory errors. In Table 2, even without CMG or the "better" Plücker coordinate, we demonstrate that projecting RT camera poses into higher-dimensional embeddings improves the results for baselines (Table 2: 2d vs. 1b). To further clarify this relationship, we have now added CCR to Tables 1 and 2, highlighting that higher conditioning dimensions yield better outcomes.

Not tested on U-Net

Our experiments in Table 1 suggest that existing camera conditioned schemes already yield reasonably effective controls in U-Net, but they are not as successful when implemented in DiT, which is why we focus on DiT. We observed that the degradation of camera control could be due to the overall less pronounced motion generated by OpenSora1.0 (both 1b and 2b in Table 1 have little motion). Our CMG effectively enhances the motion as pointed out in Table 1 and Figure 3, in turn enabling better camera control. While being conceptually applicable in both architectures, we focus on the challenging case (DiT) first and consider the U-Net experiments future work. However, we agree with the reviewer that the applicability on U-Net remains to be confirmed so we have removed the related statement in the revised paper.

Only one backbone DiT model is evaluated.

At the time of our project, we only found two open-source space-time (2D+1D) video DiT models for experiments: OpenSora, which we used, and Latte. OpenSora adopts a generic DiT architecture from Latte, making our analysis broadly applicable to both and to other standard DiT video models. Another open-source model, OpenSora-Plan, employs a spatio-temporal (3D) architecture hence unsuitable. VD3D, a concurrent camera control work based on spatio-temporal DiT, evaluated their camera control method on a closed video model, SnapVideo with non-standard DiT architecture which is not released.

How to obtain sparse pose?

In training, we randomly drop camera poses from some frames (L287-288), making the model to learn to interpolate the missing poses to generate smooth motion. In inference, we subsample evenly from the complete poses. Table below shows the frame index (out of 16 frames) used for each sparsity ratio in Table 4.

What is the purpose of sparse camera control?

Sparse camera control has practical applications in streamlining the video creation process. For instance, this approach allows directors to focus on defining key moments, such as the starting and ending camera poses, while the model intelligently interpolates the intermediate poses to create smooth and coherent camera motion. This enables creative professionals to concentrate on high-level artistic direction rather than intricate technical details.

Similarity to VD3D and no experimental comparison

The concurrent work VD3D based on a proprietary video model with non-standard DiT architecture and did not release any code, rendering reproduction and experimental comparisons unfeasible.

Introduced in L95-100, video DiT architectures can be broadly categorized into two types: space-time (2D+1D) and spatio-temporal (3D). Space-time (2D+1D) architectures first encode individual 2D frame into single embedding, our method injects camera pose to each video frame before batching multiple frames for temporal attention. On the other hand, spatio-temporal (3D) architectures, like VD3D uses, simultaneously encode multiple 2D images across different time slots into a single latent embedding, which lack individual image embedding for camera pose injection using methods such as CameraCtrl, MotionCtrl and ours. Hence, camera control for these 2 architectures are inherently incompatible.

Insignificant novelty.

VD3D reported that existing camera control methods fail for spatio-temporal transformers, and our study demonstrates similar limitations for space-time transformers. Collectively, these findings confirm that existing methods are ineffective for DiT architectures. Despite trying different combination of methods and camera representation in our study, the camera controls were still suboptimal and the small motion magnitude renders DiT-CameraCtrl inpractical for application. It only became effective with the introduction of Camera Motion Guidance (CMG) which brings 400% improvement in camera motion.

In hindsight, adapting CFG for CMG might seem straightforward; however, none of the existing camera control methods (MotionCtrl, CameraCtrl, VD3D, CamCo) implemented this approach to guide or improve camera motion control. On the other hand, we argue that being able to fully lock the viewpoint is also one important aspect of camera control, which is often neglected. We confirmed methods above often fail to do so when inputting static cameras, making them not very useful in practical video editing workfows. Our data augmentation and CMG explicitly teaches the network when to keep the camera still during training. While it is simple in design, CMG's impact is significant, representing an original contribution in guiding camera motion in video generation.

What is the purpose of sparse camera control?

Sparse camera control has practical applications in streamlining the video creation process. For instance, this approach allows directors to focus on defining key moments, such as the starting and ending camera poses, while the model intelligently interpolates the intermediate poses to create smooth and coherent camera motion. This enables creative professionals to concentrate on high-level artistic direction rather than intricate technical details. Table 4 demonstrates that utilizing sparse camera poses in conjunction with our data augmentation pipeline results in only modest degradation in camera accuracy. This promising outcome lays the groundwork for future research aimed at further enhancing sparse camera control methods.