Response to Reviewer xzB4

Thanks for such a detailed review. Here are my responses.

According to the provided reviews, we believe there are a few misunderstandings on the task setting that need to be clarified.

Task & Setting Issues

  (1) Unlike the approaches concentrating on holistic body movement synthesis (e.g., AnimateAnyone and MagicPose), our ShowMaker focuses on half-body 2D talking avatar creation, which aims to upgrade the traditional 2D talking head avatar task to the next level with richer movements (e.g., hand gestures and upper body movements). In this scenario, the training data usually contains frontal images with very few side-view cases, therefore learning to distinguish the front/back of the human is not necessary (Question 3).

  (2) Our approach follows the person-specific setting as Make-Your-Anchor rather than the one-shot animation setting used in AnimateAnyone. We provide our train/test data information in Table 1 in the pdf file. Under the person-specific setting, we focus on self-driving and cross-driving performance on trained IDs instead of out-of-domain data (e.g., cartoon-style images (Question 5)). Considering this person-specific setting and our task scenario, we conduct experiments on the TALKSHOW dataset and our collected dataset instead of the TikTok dataset (Weakness 3).

  (3) The dual-branch design with a ReferenceNet is an appropriate backbone for generative tasks, but we did not involve them in our contributions. Our technical contributions are summarized below (Weakness 2 and Question 6):

    (a) We propose an effective hand recovery module for fine-grained synthesis by using a positional encoding strategy and a carefully designed key point-based texture codebook. Our approach shows obvious superiority in the hand synthesis over other dual-branch baselines.

    (b) We propose a Face Recapture module to improve facial details and identity-preserving ability, especially under the cross-driving setting with pose and shape gaps. The results in our supplementary video and the pdf file also show the temporally consistent facial texture and identity.

In addition to the task and setting issues, there are also some experimental issues.

Experimental Issues

• Q1. Comparison with TPS from CVPR2022 and MagicPose from ICML 2024. (Response to Weakness 1)

  A1. Thanks for your suggestion. We compared the TPS method in our paper, please see Table 1 and Figure 3 in the main paper for more details. In terms of MagicPose, since it is a concurrent work that was accepted by ICML in May 2024 (very close to the submission date), we involve another work MagicAnimate (similar to MagicPose) in comparison.

  Here, we finetune MagicPose on our training data and provide the comparison results in Figure 4 in the pdf file. Similar to AnimateAnyone and MagicAnimate, MagicPose suffers from inaccurate generation and texture degradation in both facial and hand regions. Please zoom in and see the details in the red boxes. The results demonstrate the necessity of our proposed Face Recapture module and Fine-Grained Hand Modeling.

• Q2. About longer visualization. (Response to Question 1)

  A2. In our supplementary video, we have provided demo videos of 10-15 seconds, which is comparable to the length of videos in the TikTok dataset. During our exploration, we managed to generate videos longer than 30 seconds with high-fidelity and temporally consistent texture. Since we cannot provide external links, we will add longer-generated videos to our supplementary video.

• Q3. Pose gaps between the reference and driving images & Non-frontal images. (Response to Question 2, Question 4)

  A3. We provide two examples in Figure 5 in the pdf file to illustrate how our ShowMaker performs when dealing with challenging cases with pose gaps or non-frontal images.

    (1) For pose gaps, we first provide an example by using a reference image from a male and driving signals from a female. As shown in Figure 5 (left part), there is a huge gap in both pose and shape. Please also refer to A4 for Reviewer jdWr, we adopt our pose alignment strategy to first align the driving signals with that of the reference image for better synthesis and the results show the effectiveness of our approach under huge pose gaps.

    (2) Non-frontal images. Since most of the videos in our dataset are in the front view, so we took some time to look for a side-view reference image in the test set, and the generated result is shown in Figure 5 (right part). The results demonstrate that our network design can preserve human identity and hand texture even when processing non-frontal reference images.

Response to Reviewer R54L

Thanks very much for your valuable suggestions and for pointing out the typos in the manuscript. Here are my responses.

• Q1. About the comparison with talking head synthesis.

  A1. Thanks for your suggestion, we will add the talking head comparison in the demo to the main manuscript to reflect that our task is an extension of the talking head.

• Q2. About the structure of the pose encoder.

  A2. For a detailed description of the Pose Encoder, please refer to Figure 2 in the pdf.

• Q3. About some typoes.(Response to Weakness 2 and Questions 1, 2)

  A3. Sorry for the confusion. The "Reference Pose" in Figure 1 (d) should be removed. It should be repeated F times instead of L times in line 140 on page 3. We have corrected the known writing errors and will carefully revise the manuscript.

• Q4. The reason for choosing the VAE encoder as the texture extractor in the Face Recapture Module?

  A4. The VAE pre-trained on large-scale datasets has excellent compression capabilities and can capture local texture information of the face region well. Combined with global facial identity features, it can comprehensively represent facial information. Additional results are shown in Figure 3 in the pdf file, it can be seen that our method can generate satisfactory facial details due to the Face Recapture module.

• Q5. About the shapes of $G_h$ and $G_f$. The advantages of adopting a discrete codebook chosen to represent hand information.

  A5. Sorry for our oversight. The shape of $G_h$ is $B\times F\times 2\times 512$ and the shape of $G_f$ is $B\times F\times h\times w\times 768$. Representing hand information with a discrete codebook is robust. Moreover, the designed codebook consists of a set of orthogonal basis directions, which can fully represent hand structure and texture. (Please refer to A4 for Reviewer HRx2 and A3 for Reviewer jdWr for more motivation.)

Response to Reviewer HRx2

Thanks very much for taking time out of your busy schedule to review our manuscript. We reply as follows.

• Q1. The experimental settings. (Response to weakness 1 and question 2)

  A1. Our goal is to extend the talking head task to generate an expressive half-body conversational speaker. Therefore our experimental setting is not a one-shot animation. Following the settings of Make-Your-Anchor, the IDs of the training set and the test set are consistent, and the test set and training set do not overlap.

    For detailed dataset information, please refer to Table 1 in the pdf file. Due to space limitations, we will place the additional experimental details in the appendix.

• Q2. About "One ID" and Animate Anyone and TPS.

  A2. Since Make-Your-Anchor provides only one pre-trained model of Seth without the training code before our submission. We failed to reproduce its experiments on other subjects. Therefore, we compared our approach with it only on the Seth data ("One-ID") for fairness.

    Please also note that the training and test sets of Animate Anyone and TPS are consistent with ours.

• Q3. About some color changes in the results.

  A3. We suppose that the color changes mentioned by the reviewers may come from lighting changes in the facial region. Please refer to the videos in our supplementary videos and additional results in Figure 3 in the pdf file, our results show temporally consistent texture in the facial region due to the Face Recapture module.

• Q4. About introducing region images as a baseline for comparison.

  A4. Thanks for your suggestion. Actually, we follow the idea of ​​using the regional image to enhance the results. In our Face Recapture Module, we use cropped and aligned face images to enhance the facial details in the generated results. However, when dealing with hand regions, there are two major problems:

    (1) Since they only occupy a small proportion of the image, it is difficult to capture enough textured details within the cropped hand images.

    (2) Hand poses, expressed in more degrees of freedom, are more complex when compared with head poses. The hand poses from the reference image and driving signals may be very different, which may alleviate the contribution of texture from the cropped hand image.

    Therefore, we map the driving hand key points to a high-dimensional space and adopt a discrete codebook for hand enhancement (refer to A3 for Reviewer jdWr).

    We still follow your suggestion to conduct an experiment and the results are illustrated in Figure 1 (the second row). Here, the cropped reference hand image is sent to the VAE encoder for feature extraction and then fed into the hand attention module. The results show that the hand modeling strategy we proposed enables the best synthesis of hand details and structures among all these variants.

Response to Reviewer jdWr

Thanks very much for your careful review comments. We'll answer your questions one by one.

• Q1: About Fig 1 (d).

  A1. Thanks for pointing out this mistake. We will remove the “Reference Pose” from Figure 1 (d) and refine the whole figure in the revised manuscript.

• Q2: About Eq (5) and Eq (6).

  A2. Thanks for your suggestion. We will replace Eq (5) and Eq (6) with a unified definition of the attention operation and describe the different inputs used in Eq (5) and Eq (6).

• Q3. The effect of position encoding on generation results.

  A3. Please refer to Figure 1 (the third row) in the pdf file, the comparison results demonstrate that position encoding brings a significant performance improvement in both textural and structural details of human hands. The key points of the hands are distributed in a low-dimensional space and are relatively clustered, directly using the coordinates of the hand key points as inputs of the Hand Modeling module cannot well capture the structure and texture information from the hand.

• Q4. Alignment strategy in cross-driving experiments.

  A4. For the cross-driving setting, we adopt a pose alignment strategy that roughly aligns the driving signals from other subjects with the ones of the target person. The key points of our pose alignment strategy are as follows:

    (1) When preparing the training data, we first produce the DWPose results from each frame and set the center of shoulders as the cropping center. Then we crop the original video frame by using an adaptive cropping size, where the cropping size is designed as a fixed ratio of the shoulder width. This operation forces the human body to lie in a roughly consistent position.

    (2) In the inference stage, we further try to bridge the gap between body shapes by scaling the driving poses to match the reference pose. The scaling ratios of width and height are defined as $w_r/w_d$, $h_r/h_d$. Here, $w$ represents the shoulder width, and $h$ represents the height of the human body. We will involve a detailed description of $w$ and $h$ in the appendix.

    The above strategy enables satisfactory pose alignment between different subjects under our scenario.