DisPose: Disentangling Pose Guidance for Controllable Human Image Animation

We greatly appreciate your valuable feedback. Therefore, we have incorporated several additional experiments into the paper to showcase the effectiveness of DisPose. These newly added experiments are highlighted in $\color{red}{red}$ within the revised version of the paper.

Q1: Trainable parameters and inference time.

We compare the trainable parameters and inference time of the different models in the table below. For a fair comparison, the size of the generated video is set to 576x1024. Our method requires fewer trainable parameters based on the baseline model. During inference, our method estimates the motion field for the reference image, which increases inference time a little. We have updated these results in Table 7 of the manuscript.

Q2: The performance comparison on CD-FVD.

We show the performance comparison on CD-FVD[1] using VideoMAE-v2-SSv2 features in the following table. We have updated these results in Table 1 of the manuscript.

Q3: More side-by-side video comparisons.

We provide side-by-side video for cross-pose and cross-identity on the project page (https://anonymous.4open.science/r/DisPose-AB1D). Compared to the baseline model, our method improves overall quality, background consistency, and character consistency.

Q4: The out-of-domain real human identity, e.g. old people.

1. We provide an out-of-domain real human identity comparison for the elderly on our project page. Notably, since our method is trained with fewer parameters using fewer videos, our out-of-domain generalization ability is also affected by the baseline model.

2. For the generalization of facial identity, face-swapping[2] as a post-processing is an efficient method.

3. Moreover, extracting identity features from existing personalization generation[3,4] methods and then inserting them into existing models can also improve out-of-domain generalization.

[1] On the Content Bias in Fréchet Video Distance. CVPR2024.

[2] https://github.com/s0md3v/roop

[3] Instantid: Zero-shot identity-preserving generation in seconds. 2024.

[4] Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.

Q5: Settings for comparison with MimicMotion.

MimicMotion and ControlNeXt are trained based on SVD and do not use ReferenceNet. we used exactly the same settings when comparing them. To the best of our knowledge, training ReferenceNet for SVD-based models would be beneficial to preserve a consistent appearance, but it would increase the model parameters and training costs.

Q6: Incorrect citation in L78.

Apologies for our oversight, we have updated the manuscript.

We greatly appreciate your valuable feedback. Therefore, we have incorporated several additional experiments into the paper to showcase the effectiveness of DisPose. These newly added experiments are highlighted in $\color{red}{red}$ within the revised version of the paper.

Q1: The detailed disentanglement operation.

1. The motion field guidance and key point correspondences are not input by the user, rather we disentangle them from the user's input of the driving pose.

2. As shown in Figure 2(c)(d) of the manuscript, the detailed operation consists of extracting the motion fields and keypoint correspondences.

3. Section 4.1 introduces the disentanglement operations for sparse and dense motion fields. Specifically, we first estimate the skeleton pose by DWpose. Then, we compute the trajectory matrix (Ps) and apply Gaussian filtering to obtain the sparse motion field (Fs). Next, we predict the reference-based dense motion field (Fd) via conditional motion propagation (CMP).

4. Section 4.2 presents the detailed operation of keypoint feature extraction. Specifically, we first extract the DIFT features of the reference image using the pre-trained image diffusion model and then obtain the keypoint embedding according to the keypoint coordinates in the reference pose. We initialize the keypoint correspondence map (Fp) according to the trajectory coordinates of the driving pose.

Q2: Reasons for differences in the calculation of Pd for training and inference.

There is a gap between the inference and the training optical flow.

1. During inference, we do not propose extracting the forward optical flow directly from the driving video, as it ignores the gap between the reference character and the driving video. As shown in Figure 13(a) and Figure 14(a) of Appendix D, directly using the forward optical flow as motion guidance is clearly inconsistent with the reference image.
2. When there is a large difference between the reference image and the driving video, it is impossible to get the corresponding motion field by the existing optical flow estimation model, as shown in Figure 14(b) of Appendix D. Therefore, we have to compute Pd differently during inference.

Q3: Reasons for sampling before prediction.

Although the dense motion field we proposed in Section 4.1 can adapt to different body variations during inference. However, there are two limitations of this dense motion field: (1) there is a gap with the motion field extracted from real videos, and (2) the low computational efficiency is not suitable for use during training. Considering that pairs of training data have no body changes, to utilize accurate control signals during training and to improve computational efficiency, we approximate the optical flow during inference by sampling sparse optical flow before prediction, as shown in Figure 13(c) of Appendix D.

Q4: The effectiveness of multi-scale point correspondence.

1. First of all, the key points of the human body correspond to the most distinctive appearance characteristics. We capture the appearance characteristics of the reference character by extracting the diffusion features corresponding to the keypoint coordinates in the reference image. Notably, our keypoint correspondence is not just a set of coordinates; it is a set of feature maps that represent key appearance information.

2. Moreover, Figure 7 in the manuscript shows that keypoint correspondence can localize to the correct semantic region during cross-pose and cross-identity, which justifies that keypoint correspondence can provide information about appearance content. Table 3 in the manuscript demonstrates that removing keypoint correspondences affects various metrics.

3. To move a step further, we provide qualitative experiments in Figure 11 of Appendix D, which prove that keypoint correspondence contributes to appearance consistency.

Q5: Description of the Point Encoder from Section 4.2.

To seamlessly integrate into existing models, we introduce keypoint correspondences into the intermediate features of the hybrid control network encoder. Thanks to your suggestion, we have updated Section 4.2 of the manuscript.

Q6: More realistic generation results.

As discussed in Section 8 of the manuscript, we provide the qualitative results of the synthetic characters to avoid leakage of personal privacy. Thanks for your thoughtful comments. We provide real-life results on the project page (https://anonymous.4open.science/r/DisPose-AB1D). Please do not distribute or divulge these personal portraits.

We greatly appreciate your valuable feedback. Thus, we have added generated results to the project page to showcase the effectiveness of DisPose.

Q1: Real-life human results.

As discussed in Section 8 of the manuscript, we provide the qualitative results of the synthetic characters to avoid leakage of personal privacy. Thanks for your thoughtful comments. We provide real-life results on the project page (https://anonymous.4open.science/r/DisPose-AB1D). Please do not distribute or divulge these personal portraits.

Q2: Optical flow may introduce error information from the noise background.

1. First of all, we would like to clarify our setting. In this paper, we focus on controllable human animation. Following prior work (Animate Anyone[1]), given a static image and a driving video, we aim to generate the corresponding animation video. It belongs to the field of image-to-video generation. Specifically, it is required to preserve a static background, which is consistent with the reference image. Therefore this task does not support accepting dynamic backgrounds. On the other hand, there are some "similar" video generation works that allow taking noisy backgrounds, e.g., MIMO[2], and MotionEditor[3], which implement controllable character video generation as a video editing task, i.e., the input for video editing is a reference video and reference characters. They aim to replace the characters in the reference video using the reference characters.

2. Moreover, our method does not introduce background noise. Extracting the optical flow directly from the driving video may capture incorrect background noise. In contrast, our motion field estimation is based entirely on the skeleton pose through motion propagation, which makes the extracted motion field only relevant to the character's motion and avoids introducing noise in the background.

[1] Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation. CVPR2024.

[2] MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling. arXiv2024.

[3] MotionEditor: Editing Video Motion via Content-Aware Diffusion. CVPR2024.

Q3: Evaluating models to preserve subject identity while transferring test poses.

We appreciate your thoughtful consideration. Considering the similarity of topics in the Tiktok dataset, we did not use the videos in the Tiktok dataset to train our model. To further compare the generalization ability of the models, we provide comparison experiments for cross-pose and cross-identity on the project page. Compared to the baseline model, our method improves overall quality, background consistency, and character consistency.

We greatly appreciate your valuable feedback. Therefore, we have incorporated several additional experiments into the paper to showcase the effectiveness of DisPose. These newly added experiments are highlighted in $\color{red}{red}$ within the revised version of the paper.

Q1: More full-body and side-view generation results.

Thanks to your valuable suggestions, we provide the generated full-body and side-view videos on the project page (https://anonymous.4open.science/r/DisPose-AB1D), demonstrating our method's generalizability.

Q2: The impact of hybrid ControlNet architecture.

We show the impact of hybrid ControlNet architecture in the following table. Specifically, we design two variant architectures, (1) Exp1: inserting the motion field into the denoising network instead of the hybrid ControlNet as shown in Figure 12(a) of Appendix E.1, and (2) Exp2: removing the hybrid ControlNet and inserting the motion field guidance and keypoint correspondence into the denoising network as shown in Figure 12(b) of the Appendix E.1. We have updated these results in Table 4 of the manuscript.

Exp1 shows that the motion field needs to be jointly optimized with U-Net to provide the correct representation. Exp2 shows that complex motion information and appearance features cannot be modeled with only two shallow encoders.

Q3: The impact of CMP.

We provide the ablation analysis of CMP in the following table, which shows that CMP can improve the consistency of the generated video. We have updated these results in Table 5 of the manuscript.

Q4: Reasons for enhanced visual details and visual improvements.

We provide additional ablation analyses in Appendix C. The region-level guidance provided by our motion field guidance facilitates the enhancement of consistency across body regions. The proposed keypoints correspondence improves generation quality by aligning DIFT features of the skeleton pose, e.g., facial consistency.

Q5: More standard metrics.

To further evaluate the generated results, we provide performance comparisons for image-level metrics in the table below. Compared to the baseline model, our method achieves significant improvements in all metrics. We have updated these results in Table 6 of the manuscript.

[1]MusePose: a Pose-Driven Image-to-Video Framework for Virtual Human Generation. 2024.

[2]MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance. 2024.