AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

Q1: Lack of technical novelty, the proposed domain adapter is based on LoRA and the motion module is following standard feature inflation and the architecture is following timesformer.

As motivated in Sec. 1, our primary goal is to develop an effective pipeline to animate personalized T2I models. The technical novelty of our work is not in the modules themselves, but in the unique way they are combined and utilized, leading to an innovative solution that is greater than simply putting them together.

Since our goal is to turn a T2I model into an animation generator, feature inflation is the most efficient practice. Extensive experiments in Sec. 5 and high-quality visual results in Fig. 1,4,9,10 also validate the effectiveness of our architecture choices.

Additionally, it is not ad-hoc to use these two modules. In Sec. 5.3 and Sec. 4.2, we investigate different options for the motion module, e.g., a convolution version, and it turns out the current simple design is strong and efficient. In terms of the domain adapter, as discussed in Sec. 4.1 and Sec. 5.3, it’s worth noting that we didn’t use it for domain personalization as in previous work. Instead, we only use it during training to reduce the gap between video training data and high-quality training images of T2I. This not only results in high-quality video generation but also offers a new usage of domain adapters.

Q2: I do have some concerns about the generalization of the motion module. It is not clear to me how reliable to use LoRA for domain adaption, and if the motion module does not generalize well, should we further fine-tune the motion module on the task domain data?

To be clear, the domain adapter (implemented as an image LoRA) aims to bridge the gap between the training videos and the images (for base T2I models), which would be discarded after training. Sec.5.3 and Fig.6 demonstrate that this design indeed helps alleviate the negative effects of the video dataset.

The motion module is proposed to learn the motion pattern regarding the video dataset. For instance, videos in the training set usually involve how humans should act and how cars should move. In this case, the generalization ability largely depends on the motion that the training set covers. After this learning, personalized video generation is enabled with a user-specific domain by inserting a personalized image LoRA.

One straightforward solution for some specific complex movements that might not appear in the training set is to collect the larger-scale corresponding videos into the training set and tune the motion module. However, this is costly and impractical. We, therefore, propose MotionLoRA as a complementary method that could learn a specific motion pattern based on the general motion prior (learned by our motion module). As mentioned in Sec. 4.3, training such a MotionLoRA usually requires 20-50 reference videos and takes around 1 hour, which provides a more practical solution.

Combining them all, our AnimateDiff could not only alleviate the effect of training videos but also capture their motion prior. More importantly, it could quickly and efficiently adapt to other new/complex motion patterns that the aforementioned training set cannot cover.

Dear reviewer,

We kindly request your feedback on whether our response has addressed your concerns. If you have any remaining questions or concerns, we are happy to address them. Thank you for your time and consideration!

Q1: About the dependency of AnimateDiff on the underlying T2I models.

Current AnimateDiff indeed relies on the underlying T2I models. Due to such dependency, our method could inherit their appealing generated appearance and might hardly avoid some failure cases caused by T2I models. On the other hand, AnimateDiff could be further improved with the advancement of T2I models. We implement AnimateDiff on Stable Diffusion XL (SDXL), which enhances the visual quality and text conditioning on T2I. The comparison of two different T2I models is presented in Appendix D.2 and Fig. 20,24, showing that our AnimateDiff could also be compatible with other advanced T2I models, namely, avoid struggling with the limitations caused by a certain T2I model. We have already updated the related discussion in Appendix D.

Q2: Domain Adapter Visualization.

We retrain the motion module with the adapter entirely removed from the pipeline and present the visualizations in Appendix C.2 and Fig.18,22. Obviously, when the domain adapter is entirely removed from the training pipeline, the visual appearance in the training dataset, i.e., the watermarks, appears on the synthetic animations (1st row). Similarly, the watermarks appear when the adapter has its full impact (2nd row). In contrast, by fitting the visual distribution to a separate domain adapter and dropping it at inference, our full pipeline (3rd row) achieves favorable quality with no watermarks.

Q3: Visual Quality of AnimateDiff in Base T2I Models.

Although animating the original base T2I model is not our focus, we present the generated videos upon the original T2I in Appendix C.1 and Fig.17,21. It is obvious that the synthesized frames remain smooth over time. However, due to the lack of enhancement from personalized T2I models, the produced image corresponds closely with the video dataset (containing watermarks sometimes). Appendix D.2 and Fig.20,24 showcase the results from enhanced base T2I models (like SDXL). Since it is challenging to collect a larger-scale and higher-quality dataset than WebVid, we recommend users leverage the power of personalized image models since it is obviously more flexible and practical.

Dear reviewer,

We kindly request your feedback on whether our response has addressed your concerns. If you have any remaining questions or concerns, we are happy to address them. Thank you for your time and consideration!

Q1: In certain animation results, the motion amplitudes are minimal, and flickering between frames is noticeable. A thorough analysis of these factors should be included in the limitations section.

Thanks for pointing it out. You are right about the future improvement of motion prior. Our experimental results in Appendix C.3 and Fig. 19,23 demonstrate that the generated motions could benefit from scale-up training, producing videos with better motion amplitude, diversity, and temporal consistency. This also implies that the current model design is capable enough of capturing larger motion. In addition, enlarging the video-text paired dataset is indeed a potential avenue for further advancing our approach. However, it is important to acknowledge that this task can be challenging and costly at this moment. We have already updated the related discussion in Appendix D.

Q2: Qualitative comparisons are not enough. Please provide visual results for more cases like in Figure 5.

We have updated more qualitative results in Fig.9,10,11 on pages 16,17. Importantly, 60 samples are involved in our user study in Tab.1 for quantitative comparison, demonstrating the effectiveness of our approach.

Q3: For image animation, the identity preservation is poor, in other words, the characters in the animation results are not very similar to the one in the input image.

There might be a need for clarification since our method aims to generate personalized videos given a text prompt. Although animating a given image might be achieved in a certain way, it is not very related to this submission. Overall, the temporal consistency could also be improved through scale-up training, as shown in Appendix C.3 and Fig.19,23.

Q4: Why does the scale of the domain adaptor in Figure 6 appear to be visually linked with camera movement?

There might be a misunderstanding. In terms of the domain adapter, it’s worth noting that we didn’t use it for domain transfer during inference as in previous work. Instead, we only use it during training to reduce the gap between video training data and high-quality training images of T2I. This not only results in high-quality video generation but also offers a new usage of domain adapters.

The pose change in Fig. 6 is not due to the camera movement but because of the varying image distribution. When adjusting the scale of the domain adaptor from 1.0 to 0.0, the generated appearance goes to the personalized image domain from the WebVid domain.

Dear reviewer,

We kindly request your feedback on whether our response has addressed your concerns. If you have any remaining questions or concerns, we are happy to address them. Thank you for your time and consideration!

Q1: Some modules are not that novel from a technical perspective. E.g., the transformer block is also explored in GEN-2. LORA and adapter are also commonly used in various personalization papers.

As motivated in Sec. 1, our primary goal is to develop an effective pipeline to animate personalized T2I models. The technical novelty of our work is not in the modules themselves, but in the unique way they are combined and utilized, leading to an innovative solution that is greater than simply putting them together.

And it is not ad-hoc to use these two modules. In Sec. 5.3 and Sec. 4.2, we investigate different options for the motion module, e.g., a convolution version, and it turns out the current simple design is strong and efficient. In terms of the domain adapter, as discussed in Sec. 4.1 and Sec. 5.3, it's worth noting that we didn't use it for domain personalization as in previous work. Instead, we only use it during training to reduce the gap between video training data and high-quality training images of T2I. This not only results in high-quality video generation but also offers a new usage of domain adapters.

Q2: The paper somewhat overlooks a thorough discussion of the method's limitations.

We have added a section in the revised submission to discuss the limitations of our method (see pages 22-23, Appendix D). We address the concerns in [W2] in the following Q3-Q5.

Q3: About the learning dynamics of the motion module when pre-trained on the WebVid dataset.

AnimateDiff utilizes data priors to learn motions in the training videos dataset WebVid-10M, which contains mainly real-world footage such as outdoor scenes and human activities. By training upon it, the motion module could learn common real-world motion patterns, e.g., ocean waves, the movement of cars and boats, and human body/hand motions, as shown in Fig. 1. We have already updated the related discussion in Appendix D.1.

Q4: About the learning of MotionLoRA and whether the motion patterns can be inferred from text clues.

In AnimateDiff, whether a kind of motion can be triggered via text prompt depends on whether the training dataset accurately labeled such motion. We investigate the WebVid-10M dataset and find its motion labels very coarse. For instance, the zoom-in and zoom-out videos in WebVid are mostly labeled with a single tag, “zooming.” As a result, using text hints such as “zoom in” results in both zoom-in and zoom-out generation. On the other hand, more complex motion patterns such as “Rolling” do not have a specific text label, and thus, these motion patterns cannot be triggered via simple textual clues.

This observation suggests that MotionLoRA does not learn new motion patterns entirely from scratch but refines and enhances the pre-existing motion priors (regardless of whether they can be triggered by text) obtained during pre-training. It thus enables the motion module to express the desired motion priors at inference. We have already updated the related discussion in Appendix D.1.

Q5: About the performance on different motion categories.

In practice, we find the pipeline works well on non-violent motions such as fluid (e.g., ocean waves, fog, etc.), rigid objects (e.g., cars, boats), and simple human movements (e.g., walking, facial expressions), as shown in Fig. 1,4,9. However, if the pipeline is instructed to generate complex or rare motions rarely seen in the training dataset, e.g., dance or camera movements with drastic scene change, the model will likely generate static animation or unnatural deformations. We have already updated the related discussion in Appendix D.1.

Dear reviewer,

We kindly request your feedback on whether our response has addressed your concerns. If you have any remaining questions or concerns, we are happy to address them. Thank you for your time and consideration!