Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion

Thank you for your feedback. We appreciate your recognition of our method's innovation and applicability. Here are our responses to your comments.

Q1:However, it's unclear how the head pose and facial dynamics transfer and the perceptual loss improve the disentanglement, is there any detailed explanation for this?

Our transfer loss is primarily inspired by DPE[1] and VASA-1[2]. DPE introduces a bidirectional cyclic training strategy akin to CycleGAN's A2B2A' and B2A2B' pathways to achieve the disentanglement of pose and expression. Our approach is similar but constructs bidirectional, non-cyclic pathways. Specifically, we randomly sample two frames $I$ and $J$ from a video clip; these frames are characterized by their expression and pose attributes: $exp_i$, $pose_i$ for frame $I$, and $exp_j$, $pose_j$ for frame $J$. After applying attribute transfer, we obtain images $I_{exp_i,pose_j}$ and $J_{exp_j,pose_i}$. If the attribute transfer technique is perfect, we should obtain two portrait images with identical expressions and poses. To achieve this goal, we apply a perceptual loss to make the synthetic results appear more realistic.

[1] Pang Y, Zhang Y, Quan W, et al. Dpe: Disentanglement of pose and expression for general video portrait editing

[2] Xu S, Chen G, Guo Y X, et al. Vasa-1: Lifelike audio-driven talking faces generated in real time

Q2:Alongside the ablations on CFG scales (Table 2) and the Adaptive Norm (Figure 7), it would be beneficial to include a quantitative analysis of each proposed module—specifically, the perceptual loss and the emotion-control module.

Thank you very much for the reminder.

(1)About the transfer loss, the purpose of this loss is to achieve more precise disentanglement of facial attributes. Due to time constraints, we calculated the APD (Average Pose Distance) metric of Playmate on two datasets (HDTF and our dataset), as shown in the table below.

(2)About the emotion-control module, we have compared our method with several approaches, as shown in the table below, demonstrating the effectiveness of our expression control and achieving superior generation quality in terms of emotional expression.

Q3:How does the method handle scenarios in which both the audio and the driving video provide conditions for localized lip movement? Is there a mechanism to balance or integrate these dual inputs?

Since in most cases users expect to use audio to drive lip movements, when both audio and a driving video are available as inputs, we default to using audio for driving lip movements and expressions, while using the driving video to control the pose. In fact, we support multiple driving modes. For example, audio can drive lip movements, while the driving video controls the expression and pose, and vice versa. As shown in video1 and video2, the first row displays the reference image, the first column shows the pose control mode, and the remaining cells present the generated results. The lip sync is synchronized with the audio input, whereas the pose control is achieved through various driving mechanisms (e.g., driving video, preset mode, fixed pose).

Q4:Although the supplementary website offers visual comparisons, a user study could further substantiate the effectiveness of the proposed Playmate framework in comparison to baseline methods.

Thank you for your suggestion. Due to time constraints, we conducted a user study involving 10 participants who rated videos using the MOS (Mean Opinion Score) rating method, on a scale of 1 to 5, across four metrics: Lip Sync (LS), Video Definition (VD), Naturalness (N), and Visual Appeal (VA). As illustrated in the table below, Playmate has a notable advantage in the VD and VA metrics. While the LS and N metrics are slightly lower than Sonic's, they still outperform those of other methods, showcasing Playmate's strong competitiveness.

We are grateful for your review and valuable comments, and we hope our response fully resolves your concerns.

Q1:precise motion decoupling... (a) confusing and (b) lacking evidence...

About the motion decoupling. We emphasize motion decoupling because it is the foundation for Playmate to achieve various flexible driving combinations. When multiple control conditions exist, we can provide multiple driving modes. For example, a driving video can be used to control the pose, while audio is used to drive lip movements. This is also the most significant distinction between Playmate and methods like Sonic and EMO. Sonic only takes a reference image and audio as inputs, with the generated results (lip, pose, and expressions) all driven by the audio, without allowing users to specify different driving methods. We have uploaded multiple pose control videos and emotion control videos on our anonymous project website(video1 and video2). The lip sync is synchronized with the audio input, whereas the pose control is achieved through various driving mechanisms(e.g., driving video, preset mode, fixed pose).

Q2:state-of-the-art lip synchronization/superior video quality...Sonic achieves superior performance in this regard...

Thank you very much for the reminder. Although our quantitative metrics for lip synchronization on the two test sets are marginally lower than Sonic's, they still outperform other compared methods. Furthermore, across the remaining four comparison metrics, our method consistently exceeds the performance of all other competing approaches. We have uploaded more comparison videos with Sonic to our anonymous project website(video3, video4, video5 and video6). These videos show that, qualitatively, our lip synchronization is not far behind Sonic's. Moreover, in terms of video clarity, we are significantly better, especially in areas like the teeth. Regarding the differences from Sonic, let us briefly explain here. Sonic is a purely audio-driven algorithm that generates all features of the portrait, including lip movements, expressions, pose, etc., based on audio. This indicates that its driving flexibility is limited. In contrast, we achieve multiple controllable portrait driving methods by constructing a precise attribute disentanglement space, providing users with various flexible driving options. The implementation difficulty of this decoupling and then driving approach is higher than that of a simple audio-driven method. This is also the reason why we emphasize "Flexible Control" in our paper title.

Q3:About the user study.

Thank you for your suggestion. This question was also raised by Reviewer SMci. Please refer to our response to Reviewer SMci's question Q4.

Q4:While quantitative results show good performance on image/video metrics...This conflicting language is misleading to the actual performance of the model.

Thank you very much for the reminder. We will correct these issues in the revised version and will also carefully revise the full text.

Q5:Further, analysis that video quality is superior to that of other methods, while...

For details on the comparison with Sonic, see the response provided in Q2.

Q6:the strength of the novelty of the work however is not proven...

Thank you for your skepticism. In this response, we have added multiple tests(including responses to other reviewers), and uploaded more visual results on anonymous website, hoping that these results will dispel your doubts and concerns in this regard.

Q7:About the Adaptive Normalization.

Regarding Adaptive Normalization, our approach focuses on adapting to the dimensions of facial attributes. We apply distinct means and standard deviations for pose and expression, which provides additional prior information and reduces the learning complexity for the model. This facilitates more flexible control over the generated outputs. In the inference stage, we have the flexibility to derive these means and standard deviations from various available sources, thereby enabling more precise and controllable driving effects.

Q8:At inference, which private mean/std is used...

This question was also raised by Reviewer 54Zx. Please refer to our response to Reviewer 54Zx's question Q3.

W1:The writing, particularly around claims and introduction of...

We will correct these issues in the revised version and will also carefully revise the full text.

W2:the strength of the novelty of the work however is not proven in experiments in my opinion, and...

Due to the character limit on response, please refer to our response to Q6.

First, we would like to thank the reviewer for your careful reading and providing numerous constructive comments! Below we address the concerns mentioned in the review.

W1:The attached website does not include head pose control videos.

Thank you for pointing this out. We have uploaded multiple pose control videos and emotion control videos on our anonymous project website. For the pose control videos(video1 and video2), the lip sync is synchronized with the audio input, whereas the pose control is achieved through various driving mechanisms(e.g., driving video, preset mode, fixed pose). Regarding the emotion control videos, we have compared our method with several approaches(video3 and video4). Additionally, we conducted quantitative comparisons on emotion control accuracy and video quality, please refer to our response to Reviewer SMci's Q2.

W2:Line 215 includes some spelling mistakes. “to enhance”.

Thank you very much for the reminder. We will correct these issues in the revised version and also carefully revise the full text.

Q1:The training stability of the transfer loss.

Regarding the training stability, since we use $\mathcal{M}$ from LivePortrait as pre-trained model, which inherently possesses face attribute disentanglement capabilities, fine-tuning with transfer loss ensures both training stability and rapid convergence. We have uploaded sample image(image1) on website to demonstrate disentanglement between head pose and facial dynamics.

Q2:Evaluate the accuracy of pose conditional control.

In the field of image animation, Average Keypoint Distance (AKD) and Average Pose Distance (APD) are commonly used to evaluate pose control performance. We calculated the APD metric of Playmate on two datasets (HDTF and our dataset), as shown in the tables below.

Additionally, we have uploaded sample pose visualization videos on our anonymous project website(video5 and video6), demonstrating that Playmate achieves good generation quality even in pure pose-driven scenarios.

Q3:In the inference stage, how does denormalization operate?

In the inference stage, for expression, the mean and standard deviation are the same used during training, calculated from all training data. For head pose, the mean and standard deviation are optional and can be computed from user-provided driving videos. If not provided, they can either be set to default parameters(e.g., computed from forward-looking videos) or calculated from randomly selected videos in the dataset.

Q4:About the collected dataset.

Our own dataset involved about 39k clips, featuring more complex scenes, diverse styles, higher clarity, and greater challenges, which leads to worse overall performance for most approaches. We'll try our best to release the collected dataset, but we hope you understand that both we and the community need to exercise caution when releasing rich datasets. This caution stems from concerns regarding potential risks, particularly those related to individual privacy and likeness rights.

Q5:The poses do not showcase too much variability.

Thank you for your interest in this section. In fact, we support multiple pose driving methods. As shown in our newly uploaded sample videos, under the same audio, different pose control conditions result in different poses generated by Playmate. We support various methods to enhance pose variability, such as using videos with significant pose variations as driving videos, or manually adjusting the rotation value of the pose.

Q6:It works surprisingly well with only two DiT blocks inserted before the MLP head for emotional control, any explanations?

Thank you for your affirmation of our emotion control module. We believe its effectiveness is due to two main reasons: (1) Precise latent space construction and attribute disentanglement, enabling the model to effectively correlate emotion control with emotion features after decoupling attributes like pose and expression, leading to effective learning; (2) A two-stage training approach, where we first stabilize audio-driven training, then train the emotion control module separately while keeping most weights fixed, reducing training complexity. Training all parameters simultaneously might prioritize other labels, rendering the emotion control signals ineffective.