CyberHost: A One-stage Diffusion Framework for Audio-driven Talking Body Generation

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1. About the hand heatmap estimator

Specifically, we added several convolutional layers in each hand RAM module to serve as the hand keypoint heatmap estimator, with weights shared across timesteps. The right part of Equation 6 represents the supervision signal for the hand heatmap estimator. Since we only train our RAM modules in stage 2, the hand heatmap estimator is also updated during stage 2 along with the loss in Equation 6.

Q2. About the Pose Encoder

Thank your very much for your reminder, we apologize for any confusion caused by our unclear description. The Pose Encoder in the Body Movement Map and the one in the Pose-Aligned Reference Feature share the same model architecture, except for the first layer, but do not share any model parameters. Based on your suggestion, we have modified the corresponding symbols in Figure 2 and added clearer descriptions in the relevant paragraphs of the main text to distinguish between these two pose encoders.

Q3. About the model initialization

We apologize for any confusion caused by our unclear description. The base denoising U-Net and reference net are both initialized with SD1.5 pretrained weights. We inserted the same temporal layers as AnimateDiff into the basic U-Net architecture to support sequential video prediction. To accelerate convergence on video data, the temporal attention layer weights are initialized with AnimateDiff's pretrained weights.

Q4. About the dimension of Latent bank

Thank you for your reminder. In the region attention module, we set n, m, and d to 512, 64, and 512, respectively. Consequently, this results in a spatial latent of shape (1, 512, 512), and a temporal latent of shape (1, 64, 512). We have clarified these settings in Section D of the revised appendix.

Q5. About the Review System

Thank you for your interest in this section. Conducting safety reviews of generated content is crucial for the deployment of any AIGC task. For CyberHost, we are planing to develop a review system to ensure the safety of user-uploaded images and audio. For images, techniques such as pornography detection and celebrity recognition can be employed. For audio, Automatic Speech Recognition (ASR) can convert audio to text, which can then be analyzed for potential issues using Large Language Models, thereby preventing the video generation with offensive audio.

Q6. About the detection of hand clarity score

(1) About the selection of detection methods, first of all, we have tried to use the IQA expert model [1, 2] and the MLLM model [3, 4] to detect the blurriness of the crop hand image, but found that the effects are not good. (2) In the video, we show the impact on the generation results when different hand clarity score conditions are input. It can be seen that it will directly affect the generation quality and dynamic degree of gestures. As illustrated in Figure 13 of the paper, the hand clarity score reflects the clarity of the hands in the training data, thereby allowing the identification of training videos with varying quality of hand visuals. It enables the model to distinguish between high-quality and low-quality hand images during training, thereby guiding the model to generate clear hand visuals during inference by inputting a high hand clarity score. This is also supported by the ablation experiment in Table 1.

[1] Su S, Yan Q, Zhu Y, Zhang C, Ge Xin, Sun J, Zhang Y. Blindly Assess Image Quality in the Wild Guided by a Self-Adaptive Hyper Network

[2] Wu H, Zhang E, and Liao L, and et.al. Towards Explainable Video Quality Assessment: a Database and a Language-prompt Approach.

[3] Wu H, Zhu H, Zhang Z and et.al. Towards Open-ended Visual Quality Comparison

[4] Wu H, Zhang Z, Zhang W and et.al. Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels

Summary

Following your comments, we added discussions on the hand heatmap estimator, pose encoder, model initialization, the dimension of the latent bank, review system, and hand clarity score in the revised appendix. We think this enhances the paper and better explains the method. Thank you again for your valuable comments and positive recommendation for our paper.

Thank you for authors for the rebuttal.

I have reviewed the comments, including those from other reviewers and responses. Most of the concerns I previously had have been addressed, which led me to raise my score. Furthermore, additional studies conducted during the discussion (rebuttal) period provided valuable insights.

Related to Q1 hand heatmap estimation, one suggestion for future work would be to analyze the timesteps, as different timesteps may contain distinct feature information.

Q5. About the spelling mistakes

Thank you for your reminder. We have made the corrections in the corresponding section of the original text.

Q6. About the training details

Thank you for reminder. We apologize for any lack of clarity in our paper. Both the video-driven body reenactment model and the multimodal-driven video generation model require separate training processes. During the training process for video-driven body reenactment, we need to replace the body movement map with the full-body skeleton map. For training multimodal-driven video generation model, we draw the skeleton map of both hands onto the body movement map. We have clarified these settings in Section D of the revised appendix.

Q7. About the effectiveness of predicting regional masks in all layers

Thank you very much for your suggestions. They have been very enlightening, and we conducted verification experiments based on your direction. Firstly, we calculated the cross-entropy between the predicted hand mask from each layer and the hand region in the generated results on a batch of test data to measure the effectiveness of each attention block. For specific details, see Figure 10 in revised appendix. Based on the accuracy of the predicted masks from each layer, we created corresponding top-1, top-5, and top-10 experiments (where "top" refers to lower cross-entropy) to verify whether predicting masks and applying RAM in fewer layers could yield better results. Unfortunately, as shown in the table, better results were not achieved. We hypothesize that although the top-n layers have more precise hand structure prediction abilities, removing other layers with relatively poorer mask prediction capabilities still reduces the overall expressive power of the model.

Summary

Following your comments, we added discussions on the implementation details, the effectiveness of hand clarity score, training details and the effectiveness of predicting regional masks in all layers in the revised appendix. We think this enhances the paper and better explains the method. Thank you again for your valuable comments and positive recommendation for our paper.

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1. About the implementation details of CyberHost

a) About Audio Feature: To fully leverage multi-scale audio features, we concatenate the hidden states from each encoder layer of the Wav2Vec model with the last hidden state, resulting in an audio feature vector of size 10752. To capture more temporal context, we use a temporal window of size 5 and construct an input audio feature of shape (5, 10752) for each video frame.

b) Orthogonal Latents Bank: We applied the Gram-Schmidt process to the learnable latents bank during each forward pass to ensure orthogonality.

c) We directly use MSE loss as the loss function to supervise the mask prediction process. During training, the hand mask loss and face mask loss are weighted by 0.005 and 0.001 respectively in the total loss. Thank your very much for your reminder, all of these details are fully elaborated in Section D of the revised appendix

Q2. About the hand clarity score

In the video, we show the impact on the generation results when different hand clarity score conditions are input. It can be seen that it will directly affect the generation quality and dynamic degree of gestures. As illustrated in Figure 13 of the paper, the hand clarity score reflects the clarity of the hands in the training data, thereby allowing the identification of training videos with varying quality of hand visuals. It enables the model to distinguish between high-quality and low-quality hand images during training, thereby guiding the model to generate clear hand visuals during inference by inputting a high hand clarity score. This is also supported by the ablation experiment in Table 1.

Q3. About the Pose-aligned Reference Feature

1. Effect Demonstration: In the video, we show the difference between the model with Pose-aligned Reference Feature and the model without Pose-aligned Reference Feature for the same examples. It is evident that for more complex initial poses, the Pose-aligned Reference Feature significantly improves the reasonableness of the generated results and reduces issues with synthesizing key parts like hands.

2. Pose-aligned Reference Feature: This feature helps the model better understand the reference image, as the generated result needs to maintain the ID information of the reference image, including the hands. Logically, the model must know the hand pose in the reference image to capture its texture. This means pose detection should be completed before image generation. However, for the pose detection task, the hand gestures seen in the current training data may not be sufficient to support highly robust detection, leading to "recognition errors" and subsequent generation errors. Using an expert model for detection can avoid this issue by providing additional information to prevent "recognition errors," thereby reducing "generation errors." As seen in the video, without the Pose-aligned Reference Feature, the model confuses the reference image's arm with the clothing, resulting in an arm being stuck to the clothing in the generated output. The introduction of the Pose-aligned Reference Feature significantly improves such issues, enhancing the final generation quality.

Q4. About whether a one-stage model has an advantage over a two-stage model if an infinitely precise mesh detector exists.

First, let's assume that this mesh predictor can capture details including muscle movements. We need to discuss the upper limits of one-stage and two-stage methods. For a one-stage framework like Cyberhost, the input is a single image and audio, and the output is a video. There are no strict limitations on the input format. With appropriate training data design, the single image can include a single person, multiple people, or even people and pets. The generated video can then depict singing, conversations, or interactions between people and animals. This high degree of input flexibility stems from its nature as a one-stage audio-to-video generation framework. In contrast, methods based on intermediate representations currently struggle to achieve such extensibility. From a feasibility standpoint, using high-quality mesh reconstruction methods like [1,2] involves significant computational time, space constraints, and equipment costs, which limit the scale of available training data. This makes it difficult to benefit from data scaling up.

[1] Baradel* F, Armando M, Galaaoui S and et.al. Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot

[2] Pavlakos G, Shan D, Radosavovic I and et.al. Reconstructing Hands in 3D with Transformers

Thank you once again for your valuable suggestions and positive feedback.

We apologize for any misunderstanding regarding the intent of question 3 earlier. To clarify, we predict independent region masks at various layers and utilize them to guide the corresponding regional attention for local feature learning. As illustrated in Figure 10, the middle and later layers of the U-Net achieve higher mask prediction accuracy. Due to the inherent feature dependency in the network's forward pass, the earlier layers are unable to leverage the masks predicted by the subsequent layers. Consequently, if masks are solely predicted at the later layers, it precludes the possibility of directly applying regional attention across all layers.

However, inspired by this idea, we could also consider using the region mask predicted by the most accurate layer at timestep t+1 for all layers at timestep t. Based on our video visualizations, the masks predicted at timestep t+1 tend to be slightly less accurate than those at timestep t. Therefore, implementing this optimization approach necessitates careful consideration of more complex comparative relationships.

Nonetheless, the exploration of the effectiveness of different network layers for local feature learning is indeed a compelling topic. We intend to pursue further research on this aspect in the future.

Q4. About the dataset used in [2] and the beat consistency metric

Thank you for your reminder. Due to time constraints, we conducted visual comparisons with previous methods on only a few examples from this dataset in the video. Despite these methods being specifically designed for video clips of a few characters, whereas our approach targets audio-driven video generation for open-set images, we still achieved relatively natural driving results. Besides, in the audio-driven scenario, we supplemented the beat consistency metric and compared it with the baseline methods. Since our approach does not directly generate keypoints, we uniformly used DWPose to detect 2D keypoints of the generated videos and calculated the beat consistency metric using the speed of these 2D keypoints for comparison. However, we found that this metric has limited discriminative power in evaluating performance for upper-body speaking scenarios; even Ground Truth videos did not achieve a relatively high score. We hypothesize that this may be due to our more open data collection sources, where the recorded characters do not intentionally exhibit intense emotional movements.

Please note that we have included a discussion on the comparison results on the Speech2Gesture[6] dataset in the revised appendix.

∗ denotes evaluate on vlogger test set.

Q5. About the training particulars of video/multimodality-driven generation

We apologize that we did not describe this clearly. For the new driving method, the model needs to be retrained. We have supplemented the detailed settings and training details in the revised version.

Q6. About the typos and writing

Thank you very much for the reminder. We have corrected these issues in the revised version and also carefully revised the full text.

Summary

Following your comments, we added discussions on the structural effectiveness of RAM, the effectiveness of hand clarity score, the comparison with ANGIE and the discussion of comparison results on Speech2Gesture[6] dataset in the revised appendix. We think this enhances the paper and better explains the method. Thank you again for your valuable comments for our paper.

[1] Liu X, Wu Q, Zhou H, Du Y, Wu W, Lin D, Liu Z. Audio-driven co-speech gesture video generation.

[2] Qian S, Tu Z, Zhi Y, Liu W, Gao S. Speech drives templates: Co-speech gesture synthesis with learned templates.

[3] Li R, Yang S, Ross DA, Kanazawa A. Ai choreographer: Music conditioned 3d dance generation with aist++

[4] Ahuja C, and Lee D, Won N and et.al. Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures

[5] Siarohin A and Woodford O and Ren J and et.al. Motion Representations for Articulated Animation

[6] Shiry G, Amir B, Gefen K and et.al. Learning Individual Styles of Conversational Gestures

We are grateful for your review and valuable comments, and we hope our response fully resolves your concerns.

Q1. About the spatial and temporal latents

Thank you very much for your suggestions. Based on your suggestion, we conducted more ablation experiments on the Region Attention Module to verify the effectiveness of its structure. As shown in the table, eliminating the spatial bank significantly affects the quality of the generated videos, while the 3D Latents Bank, with a 55-fold increase in parameters under the current configuration, did not provide further performance gains. We hypothesize that the spatial and temporal latents bank, which more closely aligns with the behavior of U-Net by decoupling the temporal attention from spatial attention, consequently leads to more efficient learning of local features. Additionally, due to time constraints, we only conducted ablation experiments on different sizes of the spatial latents bank in the RAM. As shown in Figure 9 in the appendix of the revised version. The results indicate that increasing the size of the latents bank beyond the current configuration provides only marginal performance gains. To investigate the feature representations learned by the spatial latents and temporal latents, we visualized their respective feature maps used for the residuals. As shown in the video, the spatial latents focus more on all the texture regions of the hand, whereas the temporal latents emphasize the temporal smoothness information at the edges of the hand.

Q2. About the detection of hand clarity score.

(1) About the selection of hand clarity detection methods, first of all, we have tried to use the IQA expert models [1, 2] and the MLLM models [3, 4] to detect the blurriness of the crop hand image, but found that the effects are not good. (2) In the video, we show the impact on the generation results when different hand clarity score conditions are input. It can be seen that it will directly affect the generation quality and dynamic degree of gestures. As illustrated in Figure 13 of the revised paper, the hand clarity score reflects the clarity of the hands in the training data, thereby allowing the identification of training videos with varying quality of hand visuals. It enables the model to distinguish between high-quality and low-quality hand images during training, thereby guiding the model to generate clear hand visuals during inference by inputting a high hand clarity score. This is also supported by the ablation experiment in Table 1.

[1] Su S, Yan Q, Zhu Y, Zhang C, Ge Xin, Sun J, Zhang Y. Blindly Assess Image Quality in the Wild Guided by a Self-Adaptive Hyper Network

[2] Wu H, Zhang E, and Liao L, and et.al. Towards Explainable Video Quality Assessment: a Database and a Language-prompt Approach.

[3] Wu H, Zhu H, Zhang Z and et.al. Towards Open-ended Visual Quality Comparison

[4] Wu H, Zhang Z, Zhang W and et.al. Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels

Q3. About the comparison with method ANGIE.

Thank you for the reminder. Due to time constraints, we conducted a visual comparison with ANGIE[1] on PATS[4] dataset. In the video, we used Cyberhost to infer the demo video provided on their homepage. It is important to note that ANGIE is trained and tested simultaneously on the four individuals in the PATS dataset, whereas CyberHost focuses on driving open-set images. Despite not specifically training on the PATS dataset, our generated results exhibit natural motion patterns and significantly better video clarity. Although ANGIE establishes a unified framework to generate speaker image sequences driven by speech audio, the overall emphasis of this work is on the audio-to-motion generation aspect, utilizing a pre-existing image generator model from MRAA[5] for the visual synthesis. While our proposed CyberHost aims to optimize both motion generation and image generation simultaneously, providing a more unified algorithmic framework. We have included a discussion on the comparison with ANGIE[1] in the revised appendix.

We are grateful for your review and valuable comments, and we hope our response fully resolves your concerns.

Q1. About the clarity and detail of generated results.

Thank you for the reminder. Although Cyberhost cannot achieve the same clarity as the reference images, it has reached the best performance in terms of image quality metrics such as SSIM，PSNR and FID in Tables 1 at the current resolution. Additionally, it leads in video quality metrics like FVD. This paper focuses on building a one-stage audio-driven talking body video generation diffusion framework. The experimental results validate the feasibility of this approach and show significant advantages in motion naturalness and video quality compared to existing methods. To further improve the results, some data cleaning strategies can be adopted. We can enhance output quality by removing low-resolution data and further increasing resolution. Training with multi-resolution data in stages can also help. Enhancing fine detail generation is a common issue, and we aim to address this more effectively in future work based on the Cyberhost framework. In the video, we also present several test results from our high-resolution model version (640x480), which exhibit superior performance in maintaining identity and preserving details. This result is only to demonstrate that the proposed cyberhost can generate finer image quality, and its potential can still be enhanced by high-quality data, not limited by the framework itself.

Q2. Regarding whether the codebook can effectively guide the model to generate correct hand shapes and facial features.

First, we apologize for any unclear representations in our method. We assume that the "codebook" mentioned by the reviewer refers to our designed latents bank. We address this question from two perspectives: (1) Method Design: During training and testing, we use predicted regional masks to focus region latents on specific areas like hands and face. The video shows that predicted masks accurately capture these regions, ensuring the region latents functions correctly. (2) Visualization Analysis: In the video, by removing face and hand region latents attentions on a trained full model during inference, we observed that the corresponding regions were significantly affected while other parts remained largely unchanged. This, along with the quantitative and qualitative results in Table 1 and the supplementary video, demonstrates that region latents are crucial for generating local areas.

Q3. About the visualization of the ablation study for the "Motion Codebook" and "ID Descriptor".

In the video, we provide the visualization of the ablation study for the "latent bank" (which may refer to the motion codebook) and the "ID Descriptor".

Q4, About the detection of the hand position during inference

Although we only train the hand mask prediction at low timesteps, we found that it generalizes well to different timesteps during inference. In the video, we provide a visualization of the predicted region masks. It shows that in the early stages of denoising, when the hand generation is unclear, the predicted hand mask is also rough. As the denoising steps increase, the prediction becomes more stable, which should be sufficient for the region latents attention to be effective.

Summary

Thank you for your suggestions. We will include the ablation results of the Latents Bank & ID Descriptor and the visualization of the predicted hand mask on our project website in the future. We think this enhances the paper and better explains the method. Thank you again for your valuable comments for our paper.

Thank you once again for your valuable feedback.

We have included more experimental results and visualizations based on your comments. We hope you find the response satisfactory. As the discussion phase nears its end, we look forward to any additional feedback to ensure all concerns are addressed.

Best regards

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1: About the discussion of failure cases or limitations.

Thank you very much for your suggestion. We have included an analysis of the current model's limitations in the video. CyberHost still cannot handle the following situations:

For the input image: Exaggerated body proportions in cartoon images often result in unrealistic outputs; Complex backgrounds resembling hand colors, such as skin-toned backgrounds, are prone to generating errors; Additionally, challenging human poses, such as raising both hands above the head, complicate the generation of natural motion.

For the input audio: scenarios with complex background sounds and varying tones, such as singing, make it difficult to generate accurate lip movements and naturally synchronized motions.

Possible approaches to address these issues include: 1) Collecting a more diverse dataset of character videos, including anime scenes and data featuring complex human poses. 2) Replacing Wav2Vec with a more advanced audio feature extractor and implementing background noise reduction strategies. We have also provided a more detailed discussion in the appendix of the revised version.

Q2. About the scalability to full-body generation

Thank you for your reminder. Full-body scenes present significant challenges, which we plan to address in future work. In the video, we have provided examples of the current results in full-body scenes. From a methodological perspective, Cyberhost can be expanded to full-body scenarios. But right now, there are two main reasons why the performance isn't that good: (1) The VAE struggle to effectively model faces and hands due to their smaller proportion in full-body frames. This can be fix by increasing the training resolution and model capacity. (2) The current training data mostly focuses on upper-body scenes, so we need to collect and train on full-body scene data. Full-body scenarios are challenging but also very useful, and this is exactly what we hope to address in our future work.

Q3. About the user study for subjective evaluation

Thank you for the reminder. Due to time constraints, we conducted a user study with five independent, professional evaluators. Below is a summary of the questionnaire results. We used the MOS (Mean Opinion Score) rating method, with specific criteria detailed in the appendix of the revised version. As can be seen, Cyberhost shows performance advantages across multiple dimensions compared to the baseline methods. Evaluation results involving more participants will be included in the final version.

* denotes evaluate on vlogger test set.

Q4. About the release of the training dataset

Thank you for your suggestion. At this time, we are indeed unable to provide a definite plan for making the dataset publicly available, as this decision requires further consideration and internal data review processes. However, we have included a detailed data filtering process and data composition analysis in the revised version, and we hope this will be helpful.