MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting

• L197: I do not see the two adjustments as novel. 1) “bypass the complex and time-consuming diffusion process” means you opt for a GAN-based model. People try diffusion-based methods for their better generative capacity, not for their speed. In another hand, GAN-based Lip-Sync methods have been widely explored [1,2,3,4], which are not discussed in this paper. 2) Using an occluded lower half face input is also a common practice in Lip-Sync, I do not see it is an adjustment in any way.

• L294: “However, achieving precise synchronization is challenging due to different sampling rates between the two modalities. While our multi-scale fusion architecture addresses this issue to some extent, it does not entirely resolve it.“ Cannot understand why the sampling rate between the two modalities affects audio-visual synchronization? "multi-scale fusion" was also explored in previous works like wav2lip, [1,2,3,4].

• Fig. 2 (b) I couldn’t understand what was meant by “information volume” and “information modulator” until realizing that they are simply terms referring to the temporal length and SyncLoss. It seems these are just “fancier” expressions for familiar concepts.

• I would try to refine the figures. Sometimes find too small the text and face.

• One contribution claimed by the authors is to improve the lip-sync accuracy. However, from the evaluation in Table 1, I see only a compromise between lip-sync and generative quality.

• From the supp files, the mouth region appears noticeably more blurred than other areas of the face.

[1] RADIO: Reference-Agnostic Dubbing Video Synthesis

[2] Dubbing for Everyone: Data-Efficient Visual Dubbing using Neural Rendering Priors

[3] Audio-driven Talking Face Generation with Stabilized Synchronization Loss

[4] StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator

• Method Description
  1. Please explain how the channel dimension is transformed from 2c to c after concatenating the reference image and the source image along the channel dimension.
  2. What does the $t$ in $I^t_{ref}$ represent? If it does not refer to the diffusion model, should $t$ be removed to avoid confusion?
  3. What should be done if SIS selects multiple reference images? According to Figure 1, there should be only one reference image for each source image.
  4. The inspiration for AAM lacks logical coherence and supporting references. What are the differences between AAM and directly using a 0.2-second segment, and how do they compare in terms of performance?
• Performance Evaluation
  1. This paper removes the denoising manner but does not analyze the impact of this removal, particularly on image quality and lip sync. There are already related papers, such as DiffTalk [1].
  2. The baselines compared in the paper are relatively weak. It would be beneficial to include a comparison with stronger baselines like IP_LAP [2], which is known for its stable performance.
  3. The paper conducts experiments solely on the HDTF dataset, which features relatively simple scenes with limited head and mouth movements. It would be advisable to consider testing on more complex datasets, such as VoxCeleb and VFHQ.
  4. The title of the paper includes "real-time," but the comparative evaluation only addresses quality and does not provide any information about inference speed.

[1] S. Shen et al., "DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 1982-1991.

[2] W. Zhong et al., "Identity-Preserving Talking Face Generation with Landmark and Appearance Priors," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 9729-9738.

1. The work does not provide baseline comparisons with many of the existing efforts that have been published recently such as V-Express, Audio-driven Talking Face Generation with Stabilized Synchronization Loss - ECCV 2024, StyleSync - CVPR 2023, Resyncer - ECCV 2024. Besides there are efficient high speed talking face works that use Gaussian Splatting such as FlashAvatars that provide higher quality with higher inference speeds. In view of this, it is not clear that the proposed method provides improved results or insights over the existing works
2. The proposed work does not provide significant insights into high-quality talking face synthesis. The efforts, while showing good results are similar to many existing works in terms of architecture and loss functions. The main improvements appear to be in terms of adaptation of sampling and synchronization strategies
3. The proposed method would not adapt to individual idiosyncracies - such as enunciation of vowels or style of speaking. As such, this would limit the method from being adopted for high-quality applications such as entertainment
4. The method uses a per-frame prediction strategy. This can be limiting as identified by the authors. Particularly when it comes to talking faces with people moving pose while talking, it can be challenging. The results do not demonstrate the extent to which this is limiting by evaluating the method on challenging scenarios.

The SD-like framework is not novel to me given lots of previous works [1][2]. Beyond this, the authors propose two strategies to further improve the performance: Selective Information Sampling & Adaptive Audio Modulation. SIS is reasonable to me but the second leverages lip-sync loss which is first proposed in 2020, making the contribution of this work lighter.

It is not clear that how do you determine the k in SIS. Both Pose-Aligned Image Set and Lip motion Dissimilarity Image Set use the same hyperparameters, what if there is no overlap between them?

The results show that the proposed method achieve better performance on FID but worse on lip sync. Also, in table 2, SIS improves the FID but at the cost of lip sync quality compared to Lip Motion Dissimilarity Set.

[1] EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

[2] Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation