Thanks very much for the review! Here are our replies.

Note: Please refer to the one-page PDF for the mentioned figures and tables with an "R." in their names.

Novelty of our method

Thanks for the suggestion. We appreciate that most reviewers have acknowledged the novelty of our method and we will add the missing citations in the final version.

Due to the response word limit, please see the general response for comparison to related methods.

Ablations of the latent shift filling method

To address the reviewer's question regarding additional ablation studies, we have conducted experiments to explore different methods for filling the missing region in the shifted noised latent. We present the qualitative results in Figure R.1 (f) and the quantitative results in Table R.1 (c).

In our experiments, we tested several approaches:

1. Random Init: Filling the hole with random values. This method resulted in severe artifacts due to the disruption of the natural horizontal and vertical distribution of latent pixels.
2. Random Sample: Randomly sampling values in the latents. Similar to Random Init, this approach also produced significant artifacts and degraded the visual quality.
3. Loop: This method involves using the moved-out-of-scene values to fill the hole, creating a looping background effect. While this technique preserves video quality better than random initialization or sampling, it limits the flexibility of camera movements by producing only looping effects. Thus, it is not suitable for more varied camera controls.
4. Reflect: This is the method used in Text2Video-Zero, where the missing region is filled by reflecting the surrounding content. However, in our Text-to-Video (T2V) situation, this approach resulted in the method collapsing, failing to maintain the desired quality.

The quantitative results in Table R.1 (c) corroborate these observations, showing that Random Init and Random Sample lead to significant artifacts, while Loop provides better quality but lacks flexibility. The Reflect method, although effective in other contexts, did not perform well in our scenario.

In conclusion, our straight-sample method consistently maintained high visual quality without introducing significant artifacts or limiting camera movement flexibility.

Details about attention suppresion

Thank you for your insightful question. In our approach, we manipulate the dot product between the query matrix $\mathbf{Q}$ and the key matrix $\mathbf{K}$. The final attention map is then calculated using a Softmax function across all tokens.

The amplification of the subject tokens is during the earlier denoising steps.

In the later steps, we suppress all the tokens, including the subject tokens all to $-\infty$. This does not completely ignore any tokens due to the nature of the Softmax function, which normalizes the scores across all tokens.

Our qualitative results, as presented in the paper and the accompanying one-page PDF, demonstrate that the generated videos do incorporate other aspects of the prompt, such as verbs or background elements. This shows that while the subject tokens receive amplified attention, the rest of the prompt tokens are still considered, influencing the overall output.

Video metrics

Thanks for the feedback. Due to the word limit, please refer to the general response.

Flow error metrics calculation

We exclude the subject region when calculating flow error, so the flow error only accounts to camera motion control.

The flow error $E_{flow}$ for each frame can be expressed as:

$$E_{flow} = \frac{1}{|R|} \sum_{p \in R} \| \mathbf{F}_p - {\mathbf{C}} \|,$$

where $\mathbf{F}_p$ is the predicted flow vector at pixel $p$. ${\mathbf{C}}$ is the ground truth camera motion condition. $R$ represents the region outside the subject bounding box. $|R|$ is the number of pixels in region $R$.

Evaluation datasets

Thank you for your feedback. We acknowledge that current open-sourced, state-of-the-art T2V models, such as LaVie and Zeroscope, exhibit limitations in generating a wide variety of object types. For example, Zeroscope struggles to generate humans with its pre-trained weights. In our study, we have followed the precedent set by previous research [1,2,3] in selecting our evaluation dataset, which includes categories like pets, plush toys, cartoons, and vehicles. We believe this selection provides a sufficiently diverse set of objects to comprehensively evaluate our proposed techniques within the capabilities of the existing T2V models.

We emphasize that our approach, characterized by training-free intrinsic subject and camera motion control methods, offers strong potential for generalizability. We are confident that, with the advent of more advanced T2V models, our methods will demonstrate even broader applicability and effectiveness across a wider range of object classes.

Out-of-bbox problem

Our approach involves controlling the approximate position of the subject primarily during the earlier denoising steps. This method can sometimes result in portions of the subject extending beyond the defined bounding box in the final output. This is a deliberate design choice, as overly restrictive bounding box conditions would lead to rigid, unnatural outputs that may appear as squared or boxed regions, which are not desired.

Model efficiency

Training:

Fine-tuning a subject takes about 10 minutes (Line 263 in paper).

Inference:

Naive T2V pipeline: around 15.0s per video

ours:

+ subject control: 15.3s per video

+ camera control: 20.6s per video

+ camera & subject control: 21.5s per video

[1] Ruiz, Nataniel, et al. "Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation." CVPR 2023.

[2] Wei, Yujie, et al. "Dreamvideo: Composing your dream videos with customized subject and motion." CVPR 2024.

[3] Wang, Zhao, et al. "Customvideo: Customizing text-to-video generation with multiple subjects." arXiv 2024.

Thanks very much for the review! Here are our replies.

Note: Please refer to the one-page PDF for the mentioned figures and tables with an "R." in their names.

Separate results for subject motion control, camera motion control, and both

Thank you for your feedback and suggestion. It's a valid point to consider the performance of our method across different types of motion scenarios.

In the main paper, we've already presented quantitative results for scenarios involving only camera motion, as shown in Table 2. Additionally, we provided results for cases where both subject and camera motion are controlled, as detailed in Table 1. Following your recommendation, we conducted an additional experiment to evaluate our method specifically for scenarios involving only subject motion. The results are presented in Table R.1 (a).

For a comprehensive analysis, we have compiled the results for all three cases—camera motion, subject motion, and both—into a single table. This allows us to better understand how controlling these aspects influences the performance. Notably, we observed that when both subject and camera motions are controlled, there is a slight decrease in some metrics. For instance, with the Zeroscope model, there's a 0.1 drop in the R-CLIP score and a 0.01 decrease in the T-Cons metric when both motions are controlled, compared to controlling only subject motion. Similarly, the flow error increases slightly when subject motion control is added to camera motion control. However, there are also instances where the R-DINO and CLIP-T metrics show a slight improvement under combined control.

In conclusion, while there is a partial performance drop when incorporating both control aspects, we believe this trade-off is reasonable and acceptable within the context of the motion-aware customized video generation task. It's natural for some quality loss to occur when adding more control signals, especially with training-free methods. However, the overall quality remains above an acceptable threshold, ensuring that the generated results are still meaningful and effective. Finally, considering the novelty of the proposed subject and motion control methods and the performance gain compared to existing approaches, we believe our work makes a substantial contribution to the field.

Examples of controlling both camera and subject motion

Thank you for your feedback and for taking the time to review the supplementary video. We appreciate your concerns regarding the number of qualitative results and potential cherry-picking. We would like to clarify and address your points.

We clarify that all the qualitative cases compared with baselines are sampled without cherry-picking. We use different subject and camera motions to better demonstrate the model capability. We will release all the qualitative results in the final version.

As indicated in the quantitative results, our method is designed to effectively control both subject and camera motion. For a more comprehensive understanding, we have provided additional qualitative examples in the revised manuscript.

In particular, Figure R.1(g) illustrates the "dog running on the beach" video with eight distinct camera movements. This example demonstrates the capability of our method to manage complex scenarios involving simultaneous subject and camera motion. The dog's movement from the top right to the bottom left of the frame, combined with various camera motions, showcases the robustness of our approach in maintaining both motions' integrity.

Additionally, Figure R.1(d) shows another challenging scenario where the camera pans to the left while zooming in on the subject by enlarging the bounding box. This example further emphasizes our method's ability to handle complex motion dynamics.

We believe these examples provide clear evidence of our method's effectiveness in different motion scenarios, including cases involving simultaneous subject and camera motion. We will consider your suggestion to include more detailed segregated results to further validate the method's performance across different types of motion.

Thank you again for your valuable input. We hope these examples address your concerns and demonstrate the robustness of our approach.

More detailed ablations of our method and more comparisons

In the one-page PDF, we also conduct many other qualitative and quantitative experiments to thourghly analysis our method, and compare with more baselines.

To be specific, Figure R.1 (a), shows we can achieve the zoom-in and zoom-out effect by gradually enlarging or shrinking the subject bounding box.

Figure R.1 (c) involves a scene dominated by a large candy house, nearly occupying the entire frame. Despite this, our technique effectively pans the camera to the right. This outcome suggests that our method can handle camera movements even with large foreground objects.

Table R.1 (e) and (f) shows quantitative experiments comparing with more baselines, including TrailBlazer, Directed Diffusion, Boximator, Motion-Zero, MotionCtrl, and Text2Video-Zero.. Our results generally outperforms the alternatives.

Figure R.1 (f) and Table R.1 (c) explores different methods for filling the missing region in the shifted noised latent.

We hope the extra results will provide a broader and deeper sight about our method.

Thanks very much for the review! Here are our replies.

Note: Please refer to the one-page PDF for the mentioned figures and tables with an "R." in their names.

Metrics for static image crop of the subject

We apologize for any confusion. We believe your suggestion involves cropping a video to focus on the subject, according to the bounding box, and then calculating a metric that reflects the "animation quality" within that cropped video.

This is a valuable suggestion. In response, we developed a new metric called "Region CLIP-text." This metric first crops the video to isolate the subject as guided by the bounding box. Then, it calculates the similarity between the cropped video and a text prompt describing the animation, such as running, jumping, etc. We believe this metric better reflects the quality of the subject's animation by focusing specifically on the subject's actions.

The results, presented in Table R.1. (f), compare our method with other approaches in terms of subject motion control. The "Region CLIP-text" metric shows a trend similar to CLIP-T but with a stronger emphasis on the subject region. It more accurately captures the alignment of the subject's animation with the descriptive text. Our method outperforms the baselines, demonstrating the effectiveness of our subject motion control technique.

Regarding a desired video metrics, please refer to the general response due to the word limit.

Capability of camera motion control

We acknowledge that the proposed latent shift module primarily facilitates control over 2D camera movements, such as up and down panning. However, we observe that we can achieve the zoom-in/out effect through gradually enlarging or reducting the bounding box for the subject, with subject motion control.

This behavior is demonstrated in Figure R.1 (a), where examples illustrate the zoom-in and zoom-out phenomena corresponding to the bounding box adjustments. Additionally, Figure 14 in the main paper includes an example that showcases the enlargement of an object through the progressive increase of the bounding box.

By combining this technique with 2D camera motion control, we can provide a more versatile camera control experience.

As illustrated in Figure R.1 (d), we demonstrate an example where the camera pans left while simultaneously zooming in. This showcases our system's ability to achieve a more complex, coordinated camera movement, highlighting the potential for more dynamic and flexible camera control in future developments.

The assumption of "shrunk image"

Thank you for your insightful feedback. Models such as ModelScope, Zeroscope, LaVie, and VideoCrafter all adhere to the principle where the latent can be considered a "shrunk image," preserving the same visual geographic distribution as the generated output.

Regarding newer models, we anticipate that they will likely follow a similar architecture, especially given the examples of video editing capabilities demonstrated by Sora using SDEdit technology. The requirement for the latent to act as a "shrunk image" appears to be a foundational aspect of such technologies. Therefore, we believe our techniques will remain applicable and useful even as new latent encoders are developed.

Mask the diffusion loss vs mask the training images

Thank you for your insightful suggestion. We conduct an ablation study to compare the effect of masking the background regions in the training images directly in pixel space, as opposed to masking the diffusion loss.

As demonstrated in Figure R.1 (e), masking the training images significantly impairs the model's ability to learn the subject. The video example clearly shows that this approach results in a substantial degradation in the model's performance. Furthermore, the quantitative metrics, as presented in Table R.1 (b), reflect a marked decrease in performance when the training images are masked.

We hypothesize that this decline is due to the masked images creating an unnatural distribution, which disrupts the learning process. Therefore, we conclude that masking the diffusion loss, rather than the training images, provides a more effective approach for maintaining the integrity of the learned representations.

Numbers in Table 3

It is not a copy-paste error. This result demonstrates that using class-specific videos is comparable with not using any videos.

Differences between c, d, e, and f in Figure 3

Figure 3 shows an ablation about learning loss techniques. They are all sampled from a customized Zeroscope model on the white dog. (c) means the model is trained without using subject region loss and without video loss. (e) means the model is trained with subject region loss but without video loss, etc.

Control the camera motions through text prompt

Thank you for your insightful suggestion. We conduct an ablation study and presented the results in Figure R.1 (h). We add simple text prompts such as "camera pan right" and "camera pan up right" to control the camera motion. The generated videos do follow these instructions to some extent, particularly for simpler motions like panning right. However, we find that these text prompts are often too vague, lacking the ability to specify important details such as camera motion speed or distance. Consequently, this approach yields sub-optimal results compared to our proposed method.

Quantitative results supporting this observation are shown in Table R.1 (d). While text guidance results in slightly higher CLIP scores and temporal consistency, the flow error—which measures the precision of camera is significantly lower than what we achieved using our method. This demonstrates that our approach provides more precise and stable camera motion control than can be achieved with text prompts alone.

Dear reviewer:

Thanks for your comments.

We will update our draft following your comments with the analysis on camera zoom in/out, text-based camera control, and the exploration of masking in pixel space.

Best regards!

Authors of #9629.

Thanks very much for the review! Here are our replies.

Note: Please refer to the one-page PDF for the mentioned figures and tables with an "R." in their names.

Details about video data used for training

Thanks for the question, during training, we randomly download 500 videos from the Panda-70M dataset. The Panda-70M dataset is a large-scale text-video dataset that contains 70M videos from YouTube. The videos are annotated with captions. We call them common videos because we do not constrain them to have any specific class or the customized subjects. The reason to use video data is to preserve the video generation ability of pre-trained T2V models.

The customization ability is learned through 4-6 images of the same subject.

For more details on the training data, please refer to Sec. 4.1 Implementation details, specifically lines 257-258.

Camera motion control on extreme cases

Thank you for your valuable feedback. We conduct analysis to address the limitations you mentioned, focusing on extreme scenarios involving significant camera movements and large foreground objects.

Large Camera Motion Speed

We test our camera motion control technique with varying levels of camera movement. The results are shown in Figure R.1 (b). Initially, with camera movement [c_x, c_y] = [-0.5, 0.45], indicating a movement of half the video width to the left and nearly half downward, our technique successfully controls the camera motion.

Even when we double the camera motion speed, the model continues to output accurate results. However, at triple the speed, [c_x, c_y] = [-1.5, 1.35], the method fails, showing only a downward tiling effect.

These findings indicate that while our method can manage camera movements spanning the entire video width, it does have limitations under extremely high-speed conditions.

Camera Motion Control with Large Foreground Objects

Results are shown in Figure R.1 (c). The experiment involves a scene dominated by a large candy house, nearly occupying the entire frame. Despite this, our technique effectively pans the camera to the right.

This outcome suggests that our method can handle camera movements even with large foreground objects. We believe this effectiveness stems from the method's reliance on latent shifts, which moves both foreground and background elements simultaneously. Therefore, the presence of large foreground objects does not significantly impair the performance.

Comparisons with more subject and camera motion control methods

Thanks for the suggestion. Due to the response word limit, please see the general response for comparison to related methods.

Zoom-in/out

We acknowledge that the proposed latent shift module primarily facilitates control over 2D camera movements, such as up and down panning. However, we observe that we can achieve the zoom-in/out effect through gradually enlarging or reducting the bounding box for the subject, with subject motion control.

This behavior is demonstrated in Figure R.1 (a), where examples illustrate the zoom-in and zoom-out phenomena corresponding to the bounding box adjustments. Additionally, Figure 14 in the main paper includes an example that showcases the enlargement of an object through the progressive increase of the bounding box.

By combining this technique with 2D camera motion control, we can provide a more versatile camera control experience.

As illustrated in Figure R.1 (d), we demonstrate an example where the camera pans left while simultaneously zooming in. This showcases our system's ability to achieve a more complex, coordinated camera movement, highlighting the potential for more dynamic and flexible camera control in future developments.

Differences of losses with Mix-of-Show

Mix-of-Show: This method focuses on training an ED-LoRA (Efficient and Decentralized Low-Rank Adaptation) model for individual clients. The primary aim is to merge multiple LoRA weights using a multi-concept fusion technique to handle multiple concepts effectively.

Our Method: We aim to eliminate the overfitting problem to the background while learning from a few subject images. Our approach includes specific loss functions designed to address this issue and improve the representation of subjects within videos.

1. Subject Region Loss:

Mix-of-Show: Does not explicitly focus on subject region loss. Instead, it leverages the general capabilities of the ED-LoRA model to handle multiple concepts.

Our Method: We introduce a subject region loss specifically designed to minimize overfitting to the background. This loss ensures that the model learns to focus on the subject itself, rather than the surrounding context, thereby improving subject-specific customization.

2. Subject Token Cross-Attention Loss:

Mix-of-Show: The approach does not include a specific loss function that guides the subject token with the subject's position in the video.

Our Method: We propose a subject token cross-attention loss to explicitly guide the subject token with the subject's position in the video. This helps in maintaining the spatial consistency of the subject across frames, ensuring that the subject is accurately represented in its correct position throughout the video.

To summarize, our method distinguishes itself by introducing specialized loss functions that address specific issues in customized image generation, such as overfitting to the background and maintaining subject position consistency. These targeted losses are not present in the Mix-of-Show approach, which focuses more broadly on decentralized training and merging of multiple LoRA weights.

We will cite Mix-of-Show and incorporate this discussion in the final version of our work. If further specific aspects of our loss functions need to be compared with those in Mix-of-Show, we would be happy to provide a more detailed comparison. Thank you for your attention to this matter.

Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments and a PDF file. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider the raising score after we address all the issues.

If you still have more questions and concerns, please comment here. We will reply it as soon as possible.

Thank you

Thanks very much for the review! Here are our replies.

Note: Please refer to the one-page PDF for the mentioned figures and tables with an "R." in their names.

The definition of hyper-parameters

We use different sets of hyper-parameters for the Zeroscope and LaVie models, as detailed in Appendix A.3 of the paper. The selection of these hyper-parameters, including the value of parameter alpha, is primarily based on experimental performance. Once determined, we maintain these hyper-parameters consistently across all our experiments to ensure comparability of results.

Furthermore, we conduct a thorough ablation study of these hyper-parameters, including alpha, as presented in Figure 12 (a) and (b) of the main paper. This study analyze the impact of different settings, and the findings are discussed in detail in Appendix A.4. We hope this clarifies the basis and consistency of our parameter choices.

Enlarging and shrinking subjects through subject motion control

Thank you for the question. Our subject motion control technique indeed accommodates changes in the bounding box size, such as gradual enlargement or reduction. When the bounding box is enlarged, the method appropriately scales the subject, resulting in a zoom-in effect in the generated video. Conversely, shrinking the bounding box causes the subject to appear smaller, producing a zoom-out effect.

This behavior is demonstrated in Figure R.1 (a), where examples illustrate the zoom-in and zoom-out phenomena corresponding to the bounding box adjustments. Additionally, Figure 14 in the main paper includes an example that showcases the enlargement of an object through the progressive increase of the bounding box.

The 3D controlling ability of camera motion control technique

We acknowledge that the proposed latent shift module primarily facilitates control over 2D camera movements, such as up and down panning. However, it is important to note that our work is pioneering in utilizing this approach for camera control. While our current method does not inherently support 3D camera motions like moving into the scene or full rotations, we have integrated subject motion control techniques to achieve zoom-in and zoom-out effects. By combining these techniques with 2D camera motion control, we can provide a more versatile camera control experience.

As illustrated in Figure R.1 (d), we demonstrate an example where the camera pans left while simultaneously zooming in. This showcases our system's ability to achieve a complex, coordinated camera movement, highlighting the potential for more dynamic and flexible camera control in future developments.

The expalanation to control the position of subjects as subject motion control

Thank you for your feedback. In our paper, when we refer to controlling subject motion, we are specifically addressing a limitation observed in recent text-to-video (T2V) models. Many of these models fail to generate realistic motion that involves actual positional changes of the subject within the scene. For example, given a prompt like "a dog running in the forest," some models, such as LaVie, produce animations where the dog appears to be running in place, without any change in its overall position in the video.

Our approach focuses on overcoming this limitation by explicitly controlling the subject's position, thereby enhancing the perceived realism of the motion. For instance, we ensure that the dog not only appears to be running but actually moves from one side of the scene to the other. This positional control is crucial for creating a more convincing representation of motion.

While it is true that our method does not allow for granular control over specific subject movements like walking, dancing, or jumping, we believe that our focus on controlling positional changes significantly contributes to the overall quality and realism of motion in generated videos. Additionally, specific subject movements are already guided by text prompts, which direct the model to generate appropriate actions. Therefore, we maintain that our claim of controlling subject motion is justified, as it addresses a critical aspect of motion that many existing models overlook.

Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments and also more results are shown in PDF file. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues.

Thank you

Dear Authors,

Thanks for your detailed responses. I have read through the rebuttal and the rebuttal has solved most of my concerns. Enlarging and shirking subjects performance is good. However, the reviewer thinks the method is a bit limited to coarse camera control and subject position control, and cannot control subject motions such as walking, jumping, etc. Therefore, the reviewer decides to keep the original score.

Best regards,

Reviewer aAio