I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strength

We appreciate Reviewer whzD for the insightful feedback. Below are our responses to your comments.

---

Dense control greatly increase computational cost.

• Adapter Scale: Our adapter structure is notably smaller than the base model. The increased computational time with the adapter is minimal, only about 5.7% more.

• Sparse vs. Dense Control Signals: Sparse control doesn't reduce the dimensionality of the tokens. MotionCtrl, for instance, uses a simple repeat method to expand signals to have (H, W) dimensions. CameraCtrl employs a plucker embedding to transform sparse signals into a dense format with shape (BatchSize, FrameNumber, ChannelNumber, Height, Width) to be compatible with attention mechanisms. Therefore, dense control does not inherently add to computational demands.

• Training Complexity: Dense control signals do not increase the difficulty of training. On the contrary, by providing more motion information, they theoretically reduce the complexity of training, as they offer more direct guidance and cues to the learning process.

• Sparse Feature Points from SFM: Using sparse feature points from SFM for control sounds appealing as it represents detailed information with fewer points. However, this can be challenging during user inference due to difficulties in providing such point inputs.

---

Choice of second-order representation for motion strength

Static parts in the world coordinate system undergo rigid/linear motions in the camera coordinate system. This linear motion is ideally represented by a linear term. Naturally, we classify the motions not accounted for by this rigid motion as being attributable to the intrinsic motion of the objects themselves. This is why we use a second-order representation to represent the object motion.

---

For complex video generation models, handling depth and solving for Rλ and tλ are expensive, limiting the algorithm's practical applicability.

• During the testing phase: There is no need to solve for Rλ and tλ nor to use the L-BFGS algorithm. The only requirement is invoking the Unidepth model, which incurs a negligible cost (whin 2s) relative to the overall time (about 180s) and memory consumption of the diffusion inference process.

• During the training phase: Although the cost of solving offline optimization for the control signal appears high, it is relatively minor compared to the comprehensive data processing required for training the base model. Although previous methods (e.g. MotionCtrl, CameraCtrl) utilized inexpensive open-sourced data, we have demonstrated that it does not meet the requirements for our training needs. Further, once our annotated data are generated, this data can be repurposed across different base models, which is cost-effective.

---

More comprehensive explanation (settings, time & memory analysis) for data pipeline implementation

We use the pytorch implementation of L-BFGS (https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html), where we set:

Other parameters in Algorithn 1 :

• $N_{max}$, $\epsilon$, $\alpha$ are set as 10, 5, 0.3
• we only sample a grid with resolution 70 $\times$ 70 when solving L-BFGS instead of using all pixels

Tips on initialization of R and t in L-BFGS : For the first time we solve {R,t} with L-BFGS, we directly derive the values from 3D point clouds as the initialization values. Subsequently, we always use the previously solved values of R and t as the initial values for the next iteration.

Time complexity and memory requirement : We label data on V100 GPUs, 32G memory is sufficient. The processing time varies for different samples. On average, it takes about 40s to obtain a training sample, of which about 15s is for 3D tracking, and the rest of the time is used to solve partitions. We used 48 V100s to process the data in parallel, so the processing time was not a big obstacle. If resources are limited, I think we can reduce the resolution of the grid size in Algorithn 1.

---

Additional Results and Evidence.

As suggested, we have added additional visual results in our anonymous page for rebuttal(https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md), including the pixel-level control, the multiple motion strength and the generation of multiple dynamic objects.

We would like to thank Reviewer UqgM for the positive review. We are gratified to see the recognition of the main innovative aspects of our paper and the acknowledgment of our key contributions. Going forward, we look forward to delving deeper into the topic of video control, with the aim of further enhancing our capabilities in video motion control.

---

Suggestions on combining multiple camera movements

We appreciate the suggestion. As pointed out, our method indeed supports the combination of multiple camera movements, not limited to only zoom-in/out, rotation, or translation. In our anonymous page for rebuttal(https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md#combinations-of-multiple-camera-movements) we have added results demonstrating various combinations of camera movements. These results vividly illustrate that our approach maintains effective pixel-level control and adjustable motion intensity even with complex camera movement combinations. We plan to feature these results on accessible platforms like our project page, where the video effects can be showcased.

We would like to thank Reviewer k3TV for the valuable feedback. We are delighted to see the acknowledgment of the rationality in our definition of motion strength and are pleased with the praise for our results. Regarding some concerns, we will address them below and hope it will facilitate a better understanding of our paper.

---

Which video generation model is used?

As Line 340 of the paper (4 EXPERIMENTS -> 4.1 SETTINGS -> Implementation details) states, we employ a image-to-video version of Magicvideo-V2 as our base model.

Furthermore, as suggested by Reviewer iTMq, we conducted additional training on a DiT-based model, Seaweed (https://jimeng.jianying.com/ai-tool/video/generate). The results can be viewed on our anonymous rebuttal page https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md#experiment-on-another-base-model.

---

The main body primarily discusses motion strength?

We consider camera control and motion strength to be equally important and address both aspects in parallel; moreover, our approach also shows improvements over previous methods in terms of camera control:

• From the title, we emphasized both "Precise Video Camera Control" and "Adjustable Motion Strength," which we think are both critical.
• In section 3.1 on video representation, we refer to camera control using the rigid terms and to additional movements using higher order terms.
• In section 3.2, we constructed two control signals: one for point trajectory, addressing camera control, and the other for motion strength, addressing the control of motion intensity. These two are designed to work in conjunction.
• In sections 4.2.1 and 4.2.2, we discuss the good camera control capabilities and the adjustable feature of motion strength, respectively, which are both the primary focus points of our control framework.

---

The camera control capability of the proposed method is comparable to those of SOTA papers, like MotionLORA.

As noted in Lines 047, 144, MotionLoRA suffers from some key limitations:

• Limited Flexibility: MotionLoRA methods are confined to a single fixed movement mode. For instance, a "zoom-in LoRA" cannot be used to control zoom-out movements. Even if multiple LoRAs are trained, they do not support the direct implementation of complex combinations of camera movements.

• Coarse Control Scale: MotionLoRA cannot precisely control the scale of camera motion, offering only coarse control. This limitation can impact the effectiveness of the method in scenarios requiring fine-grained camera adjustments.

Contrastingly, our proposed method supports arbitrary, combinable motion modes, offering significantly higher flexibility. Moreover, our approach allows for precise control over movement scales, enhancing accuracy in camera movement adjustments. This makes our method more suited for diverse and demanding applications where detailed and multifaceted camera control is essential.

---

The point trajectory representation relies on an additional input: 3D points.

As noted in Line 321 and Fig.2, we utilize the Unidepth method to automatically convert the input image into a 3D point cloud. This means that users are not required to provide an external 3D point cloud manually. Instead, they simply need to input the camera control and motion strength signals.

---

The range of 2D coordinates? The projections are largely similar across different viewpoints？

We suspect these concerns stem from our previous lack of visual demonstration for the projected trajectories. To better understand this step, we have provided visualizations of the projected point trajectories on our anonymous rebuttal homepage (see the ground truth preview). These visualizations clearly show that, following user-inputted camera control signals, there is a discernible and coherent trajectory of the projected points. Although these projected pointclouds are "largely similar," they exhibit noticeable movement, which addresses the concern about similarity across viewpoints.

Regarding the range of 2D coordinates, we did not implement normalization of these coordinates (as shown in the ground truth previews), although some points may fall outside the image.

We would like to thank Reviewer iTMq for the valuable feedback. The comments and suggestions have greatly helped us in improving the quality of our work. Please see below for our responses to your comments.

---

More qualitative and comparative visual results.

We have provided additional qualitative and comparative visual results in our anonymous rebuttal page https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md.

---

Small training dataset.

• Indeed, we did not utilize an extensive dataset for our experiments. A significant reason might be that the task of camera control does not necessarily require massive amounts of data. Initially, we began training with around 2K data samples and achieved satisfactory camera control effects. Subsequently, we expanded our dataset to 30K data points and conducted comparative experiments. We believe that the task does not require much data because it primarily involves learning the rules of pixel motion rather than generating content, which allows for good generalization.

• We consider data quality to be more crucial than quantity. Although our dataset comprised only 30K training sampels, it was carefully selected to include videos with significant camera motion and substantial subject movement, ensuring the relevance and effectiveness of the data, as stated in Line 351 in the paper.

• Our data pipeline is fully capable of handling large amount of data. In fact, since the submission of our paper, we have expanded our dataset to 1670K training samples and are currently training on more complex base models to achieve consumer product-level results.

---

Writing Suggestion: "rigid and non-rigid motions" vs "static and dynamic parts"

This is a good suggestion. All static objects in the world coordinate system essentially undergo a rigid motion in the camera coordinate system. The initial rationale for using the terms static/dynamic was to maintain consistency with the terminology used throughout the paper. We acknowledge the need for clarity, and we plan to address this distinction more precisely in the revised version of the paper.

---

Writing Suggestion: "point trajectory" at Line 170

We will modify the expression to indicate that we perform the point trajectory projection process.

---

Effect of breaking the scene motion into ‘global’ and ‘local’?

We display the visual effects of the ‘global’ and ‘local’ division results on our anonymous rebuttal page https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md#visualization-of-dynamic-mask. We can see that our method successfully classifies subjects exhibiting substantial motion into dynamic segments with a high degree of accuracy.

Following the suggestion, we have visualized some examples that feature seas or flowing water. This has proven to be an insightful experiment. From the visualized samples, it is evident that the classification of water parts into the dynamic mask is not perfectly distinct (e.g. the 3-rd sample). This imperfection might stem from the nature of water as a fluid—its movements are challenging to define, and its motion trajectories are difficult to track precisely. However, as long as there is a reasonable accuracy rate in the training data, it should sufficiently serve the training purpose.

---

No experiments to support the claim of compatibility with various base models.

• Theoretical Basis: The adaptive format inherently advantages compatibility with a variety of base models. There is no fundamental barrier to integrating our control signals into any base model for training purposes. This has also been demonstrated in previous works, such as MotionCtrl.

• Experimental Validation: We conducted experiments on a new base model (Seaweed (https://jimeng.jianying.com/ai-tool/video/generate)) to prove this compatibility. Due to limited time, the experiment is carried out at a resolution of 256x256, primarily to validate the feasibility of our approach. The results of these experiments can be viewed on our anonymous homepage https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md#experiment-on-another-base-model.

---

How are the users supposed to provide the dense point trajectories?

As stated in Line 321 and Fig.2, we utilize the Unidepth method to automatically convert the input image into a 3D point cloud. Users can adjust 6 sliders(6 DOF, moving along xyz-directions & rotation around xyz-directions) in the interactive interface to decide the camera pose, then the camera matrix is used to compute the point trajectory.