Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation

1. While it is true that cross-identity re-enactment does not have a groundtruth, a quantitative analysis based on metrics such as FVD or FID (preferably FVD as it models spatiotemporal correspondence), identity loss (face identity consistency)/CSIM, ARD, AU-H that have been used in prior research [A-D] would be expected.

2. While the qualitative results shown are promising in comparison to the baselines provided, there exist visible artifacts such as entanglements with the background (3rd cross-id video in supplementary), warped head shape in cross-id reenactment (eg: 2nd and 3rd cross-id video in supplementary), etc. While these limitations are reasonable given the complexity of the problem, discussing them in a limitations section is recommended.

3. The paper has a few clarity issues which I have highlighted in the section below.

[A] Guo, Jianzhu, et al. "Liveportrait: Efficient portrait animation with stitching and retargeting control." arXiv preprint arXiv:2407.03168 (2024).

[B] Ma, Yue, et al. "Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation." arXiv preprint arXiv:2406.01900 (2024).

[C] Yin, Fei, et al. "Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[D] Doukas, Michail Christos, et al. "Free-headgan: Neural talking head synthesis with explicit gaze control." IEEE Transactions on Pattern Analysis and Machine Intelligence 45.8 (2023): 9743-9756.

1. Insufficient Explanability. The introduction of the article explains the reasons for using the appearance compensation module but lacks an explanation of the reasons for using codebook learning, adding relevant explanations would help readers better understand the motivation and rationale behind choosing codebook learning as a solution to the problem of accurate pose and fine-grained facial details in Talking-head generation.

2. The methodology lacks innovation. The core modules, MCC and ACC, exhibit a structure akin to that presented in reference [1], and the approach for optimizing codebook learning remains unchanged. The paper merely applies these methods across multiple scales.

[1] Zhou S, Chan K, Li C, et al. Towards robust blind face restoration with codebook lookup transformer[J]. Advances in Neural Information Processing Systems, 2022, 35: 30599-30611.

(1) The novelty of using multi-scale features. The usage of mutli-scale features are quite straightforward in many computer vision applications. Regarding talking head generation, previous works such as [1] has proposed to use multi-scale appearance features to form a tri-plane for talking head generation. Despite of different final representations (implicit field v.s. key-point based), the idea of introducing multi-scale features is similar and there seems no discussions in this paper.

(2) The motivation of introducing codebooks for compensate motion and appearance is not clear enough. A straight forward method, at first glance, would be directly output residuals for compensation without using codebooks. One guess (from my POV) is that the learned codebooks constrained the operation space when performing compensations with input flow maps / appearance features, preventing arbitrary residuals which may destroy the output. However, this is purely my guess and there are no discussions of the motivations behind the usage of these codebooks. It would be helpful if the paper could explain why codebooks are preferable to directly outputting residuals, and what specific advantages the proposed method offers for talking head generation.

(3) The result quality has not significantly outperform previous works. Per table 1, the proposed method does not outperform other baselines on CelebV-HQ dataset quantitatively. In figure 5, one could easily observe that the synthesized image has unconvincing identity preservation when performing cross-identity reenactments (e.g., in the first row and the second row the generated image has a wider face that similar to the driving person instead of the source; in the last row the generated image has a round chin while the source image has a pointed chin). Overall, these results raise my concerns that whether the current result is good enough to validate the effectiveness of the proposed method. Providing a more in-depth analysis of the method's performance particularly on the CelebV-HQ dataset where it failed to outperform other baselines would be helpful.

References:

[1] Li, Weichuang, et al. "One-shot high-fidelity talking-head synthesis with deformable neural radiance field." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

(1) When we check out the experimental results, we see that the method in this paper isn't much better in terms of indicators. For example, on the VoxCeleb1 dataset, for PSNR and AKD, the baseline (Table 2) is already almost as good as or even better than TPSM and MCNet (Table 1). Looks like this paper has a pretty strong baseline.

(2) The experimental results can't really show that the codebook works well in both appearance and motion. From the visualized results, the method in this paper does well in clarity compared to existing ones. But from Table 2, it seems that MMC is more important for the improvement in PSNR and FID. The result of MAC isn't obvious. For motion, MMC can make some improvement, but it's not much better than other methods (like having a worse AKD than MRFA under same-identity reconstruction).