Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation

[Q1]: This work is a combination of existing works. The autoregressive animation pipeline has already been explored in EMO. The noise simulation trick is commonly used in the autoregressive generation models to alleviate error accumulation. The 4K resolution generation is an extension of VQGAN in audio-driven portrait animation. Therefore, this work is a combination of prior works from the engineering perspective, which is not novel or significant enough.

[A1]: Our work is built based on existing works, and advances the portrait image animation area in duration and resolution. In particular, our work can generate 1 hour video with identity maintained and background consistent, which is not reached by previous approaches.

In terms of methods, (1) we agree that noise simulation trick is comonlly used, but it still is not able to generate long (e.g., 1 hour) video. Our dropping pixel scheme is simple, not explored for portrait image animation, and works surprisingly well. (2) we extend codeformer to video superresolution, by adding the temporal alignment.

In summary, we believe that our work benefits the portrait image animation area, especially generating high-quality and consistent long videos.

[Q2]: Compared with Hallo, the animation framework remains unchanged and is quite similar to prior works like EMO. The reference unet borrows ideas from human image animation but the authors didn’t cite them properly. Please consider cite [1,2] in the final version.

[A2]: The updated manuscript cites two significant works (Line 157-158) that greatly contribute to our research: Animate Anyone and MagicAnimate.

Animate Anyone introduces a novel framework for character animation based on diffusion models. It features ReferenceNet, which utilizes spatial attention to preserve intricate appearance details, a pose guider for controlling character movements, and a temporal modeling approach to ensure smooth inter-frame transitions. Meanwhile, MagicAnimate presents a diffusion-based framework that enhances human image animation by improving temporal consistency and identity preservation. This is achieved through an innovative video diffusion model and appearance encoder.

Both frameworks demonstrate state-of-the-art quantitative results on benchmark datasets and effectively animate arbitrary characters with impressive results.

[Q3]: The long video results shown in the supplementary do not contain too many translation and rotation of the talking head, which cannot demonstrate the temporal consistency for long-term animation. It is recommended to conduct such an experiment with diverse translation and rotation of the talking head to prove the effectiveness of this proposed method.

[A3]: Following the recommendation, we conducted experiments of talking head with diverse translation and rotation. Hallo and ours indicate the statistics testing on HTDF data-sets. We obtain the diverse translations and rotations of the talking head (results of the last row) by applying a higher coefficient on pose, as introduced in Hallo.

We observe that with diverse translation and rotation of the talking head, our method maintains comparable performance in image and video quality metrics (FID and FVD) and lip synchronization accuracy (Sync-C and Sync-D), compared with the original reported results of ours.

(Ours* represents the talking head with diverse translation and rotation.)

[Q4]: The ablation of HRE is not comprehensive enough. Comparisons with the 512x512 model cannot show the necessity of developing such a VQGAN-based superresolution in the pipeline. It is recommended to compare the proposed method with previous superresolution models, such as ESRGAN.

[A4]: We follow the kind suggestion and compare the proposed method with previous superresolution models on samples of CelebV, specifically ESRGAN. The corresponding statistics are updated in Table 6 of the revision. The results show that both ESRGAN and our method enhance visual details, as measured by MUSIQ. However, our proposed high-resolution enhancement (HRE) achieves superior video quality, as reflected by FVD scores.

(HRE represents the proposed high-resolution enhancement)

[Q5]: The design of Gaussian noise augmentation is simple and easy to understand, which was implemented by adding the Gaussian noise to the augmented latent representations. This was used as one of the conditional inputs. Could you provide more insights on why such an operation works? Ensuring temporal consistency is difficult. The paper seems to offer a much simpler and non-complex solution. Therefore, it would be more necessary to provide stronger validations on such a design.

[A5]: The effectiveness of Gaussian noise augmentation lies in its ability to simulate the error accumulation process in motion frames. By introducing Gaussian noise to augmented latent representations, the model learns to recover from motion-induced errors and generate consistent and realistic portrait videos. As demonstrated in Figure 8 of the revised paper, without Gaussian noise augmentation, errors tend to accumulate in the background over time. With Gaussian noise augmentation, the model effectively suppresses these errors, resulting in clearer and more consistent video background.

[Q6]: The set of conditional inputs used in this work includes four parts, which are the audio, textual, noise-augmented motion frames, and reference image encoding. Based on the flowchart given in Figure 3, the cross-attention conditions occur in different layers of the encoder and decoder of the U-Net. Could you justify how to select the corresponding layer for the designated condition? Are they all implemented via cross-attention?

[A6]: We have updated Figure 3 in the revised submission to clarify the role of input conditions in video generation.
Specifically, the input audio is encoded using Wav2Vec and projected through an MLP to generate key and value representations, while the latent features are encoded as queries. Cross-attention is then applied to incorporate the audio conditioning. For the reference image features, $z_{\text{ref}}$ is concatenated with the latent features $z_{\text{t}}$ and integrated via self-attention. Additionally, the textual prompt is encoded using CLIP and incorporated through adaptive layer normalization. Noise-augmented motion frames are encoded by reference net, and the output features are concatenated with the latent features $z_{\text{t}}$ and integrated via self-attention in the motion module.

[Q7]: The high-resolution enhancement is still unclear to me.

[A7]: In Section 3.2 of the revised submission (Line 274-282), we follow the kind suggestion and clarify the high-resolution enhancement process. Following option #2 in Figure 4, we first decode the low-resolution latent features and use a low-quality encoder to map them into a continuous latent space. We then apply spatial and temporal attention mechanisms and perform code index prediction to select corresponding features from the codebook. Finally, these selected features are fed into a high-quality decoder to generate the super-resolution video.

[Q8]: The other issue might be related to the over-claim of the proposed approach by saying the "hour-long". The evaluation was done on minute duration (e.g., 4 minutes in evaluation). More evidence is required for this "hour-long" claim.

[A8]: We follow the suggestion and provide the "hour-long" video demo via the following anonymous link: https://1drv.ms/v/s!AueTKT2aVehrgQd654p4-jhuA3Gt?e=bsnPeK .

[Q9]: The textual prompt implementation looks just like a regular conditional injection. This paper claims it to be one of the strengths of this work.

[A9]: The method of textual prompt condition injection is regular, but we can generate animation videos with diverse expressions with the help of textual prompt (please see Figure 11), which is different from previous works.

[Q10]: The paper needs to add some more comparisons such as the EMO work. This has been mentioned in the Introduction but appears to be not compared in the experimental section.

[A10]: Based on the videos and corresponding audio (arround 1 minutes) provided on the official EMO website, we conducted a comparative analysis. We found that, given the same reference image and driving audio, both methods exhibit comparable lip synchronization capabilities, while our approach achieves superior image detail quality, measured by MUSIQ. Our approach builds upon EMO and other notable Unet-based diffusion models for portrait image animation, and further enhances long-duration and high-resolution (please see the table below) video generation.

[Q1]: The question here is if we generate random masking and apply that to the time step t-1, would the same masking positions be enforced with the t-2 to t-N? Or the t-2 to t-N time steps still use a random masking operation. Could you clarify that and also show the motivation or the merits of each design?

[A1]: In our implementation, the t-2 to t-N time steps use the same masking operation in our work. We follow kind suggestions and add a comparison shown in the following table to clarify the metrics of each design. The comparison indicates that using random masking at each time step and using the same masking positions across all time steps yield comparable results in terms of visual quality and lip synchronization metrics. The motivation of each design both aims to have the generated video's facial identity appearance rely more on the reference image and less on the motion frames.

[Q2]: The other question is that it makes sense the omission of patches can still preserve the coarse structure. The motion dynamics on the other hand are not affected by the masking operations. Could you provide more details on why this is the case?

[A2]: The paper indicates that motion dynamics—including lip movements, facial expressions, and poses—primarily rely on coarse structural information. For instance, partial contours of the eyes, nose, lips, and overall facial outlines are sufficient to represent these dynamics. In contrast, facial identity appearance requires all the fine-grained features. As the reviewer mentioned, the omission of patches can still preserve the coarse structure. Therefore, by appropriately omitting patches (with a drop rate of 20%–30%, as determined in our following findings), the essential motion dynamics information can still be effectively retained.

[Q3]: Another intriguing part is related to the choice of patch size for dividing the latent image. Based on the results given in Table 3, the best patch size is 1*1, which means that a single pixel is dropped. This seems to contradict the intuition of using a patch. I don't mean 1 pixel is not a patch, but it does not conform to the normal choice of patch size. Could you offer more insights on why 1 pixel is considered to be more effective in this task? This would be very important.

[A3]: Our experimental results show that using bigger patch sizes (1–4 pixels) with appropriate patch drop ratios (20%-30%) yields satisfactory and comparable performance on lip sync metrics (Sync-C and Sync-D) and image and video quality metrics (FID and FVD). We add additional statistics to the following table, and update Table 3 of the updated revision. Specifically, for motion frames, dropping smaller patches allows the generation process to more effectively leverage facial identity information from the reference image, while preserving essential contour details related to facial motion—such as the eyes, nose, and mouth.

[Q4]: Regarding the preceding generated frames, which approach was used? Was that achieved by generating the second frame based on the first one, and then continually generating the third frame based on the first two frames?

[A4]: On the first time of inference, we duplicate the reference image to create two identical motion frames. These frames are used to generate a video sequence of 16 frames. After that, we take the last two frames of this sequence as new motion frames to generate the next set of frames. We do this operation iteratively until generating all frames.

[Q1]: This paper proposes three strategies to improve the performance of audio-driven face image animation; however, the motivation for each strategy is not well explained. For example, in Section 3.2, a codebook prediction approach is used to enhance temporal coherence. It would be helpful if the authors provided more insights into why temporal coherence is insufficient without this codebook prediction approach.

[A1]: The motivation for the codebook prediction approach in Section 3.2 is to enhance the super-resolution of facial visual details. We acknowledge that making frame-level independent predictions without temporal constraints can lead to inconsistencies across the temporal dimension in the generated 4K images. To mitigate this issue, we introduce codebook prediction with temporal alignment enhancement, aiming to ensure visual consistency over time following super-resolution. As demonstrated in the table below, the application of codebook prediction (without temporal alignment) based super-resolution (second row) results in obvious improvement in image detail quality, measured by MUSIQ metric. However, we observed that temporal coherence between video frames remained insufficient, indicated by suboptimal FVD metrics. By incorporating temporal alignment into the codebook prediction process (third row), we maintain enhanced super-resolution results while also substantially improving temporal coherence across frames.

[Q2]: From lines 238 to 241, the author uses a masking strategy to mask frames in the video. The author claims that this approach improves consistency. However, I’m a bit puzzled. In video generation, the masking strategy is generally beneficial for generating long videos autoregressively, so why would masking improve consistency? Could you please give more insights?

[A2]: Before applying the masking strategy, reference images contained face identity appearance information, while motion frames included both facial motion and identity appearance details. As the generated videos become longer, noise and distortions in the face identity appearance from the motion frames can accumulate. This accumulation leads to the face identity appearance in the generated video gradually deviating from that of the reference image, resulting in inconsistency (see first row of the following table). Please see the first row of the following table (Table 5 and Figure 9 of the updated revision). By introducing the masking strategy, we encourage the generated video to rely primarily on the reference images for face identity appearance, minimizing dependence on the motion frames for appearance details. Since the motion frames are intended to provide motion information only, masking their appearance features ensures that the face identity appearance in the generated video comes almost entirely from the reference image. This approach enhances the consistency of the face identity appearance throughout the generated video. Please see the last row of the following table.

[Q3]: Some descriptions in the HIGH-RESOLUTION ENHANCEMENT section are a bit confusing. In the first half of this section, it seems that a Transformer is used to model the video in a continuous latent space. However, Figure 4 actually shows a codebook. Since it appears the authors are using a codebook, I suggest reorganizing the first half of this section to reflect this.

[A3]: Yes! We have followed the kind suggestion and reorganized Section 3.2 to clarify our approach, as reflected in the revised manuscript ( Line 274-282). In Section 3.2, we first decode the low-resolution latent features. Then, we use a low-quality encoder to map these features into a continuous latent space. After applying spatial and temporal attention mechanisms, we perform code index prediction. The predicted indices are then passed into the codebook for super-resolution operations, as illustrated in Figure 4.

[Q4]: I observed flickering in some of the videos in the experimental results, especially with the man wearing a checkered shirt. Could this be due to limitations in the pre-trained VAE? Given that there is temporal alignment in the HIGH-RESOLUTION ENHANCEMENT section, why hasn’t this temporal alignment helped reduce the flickering effect?

[A4]:This is due to the limitation of the pretrained VEA. The temporal alignment can partially reduce the effect, but can't handle it satisfactorily. Because the latent feature encoded by VAE loses some information due to perceptual compression, making it difficult to accurately reconstruct the original content during decoding. This limitation results in flickering in the reconstructed video frames, which is particularly noticeable when handling checkered details such as the patterns on the man’s shirt. As shown in the table below, we demonstrate the statistics of the checkered shirt video with and without temporal alignment.

[Q5]: For patch augmentation, could the authors analyze the “effective augmentation ratio”? Since the purpose of patch augmentation is to encourage the model to disregard apearance information and focus on motion, an analysis of which patches contain motion information and which contain appearance information, along with a calculation of the effective augmentation ratio for the video clip, would provide valuable insight into the effectiveness of this design.

[A5]: To analyze the "effective augmentation ratio" we conducted experiments with different patch sizes and drop rate settings, as presented in Table 3 of the updated paper. Our results indicate that using patch sizes of 1–4 pixels with appropriate patch drop augmentation ratios of 20%–30% yields comparable performance on lip sync metrics (Sync-C and Sync-D) and image and video quality metrics (FID and FVD). The paper indicates that motion dynamics include lip movements, facial expressions, and poses. Therefore, patches that include contours of the eyes, nose, lips, and overall facial outlines contain motion information. Meanwhile, appearance information—which comprises both facial identity related details and background texture information—is inherently present in all patches.

[Q6]: On the HDTF dataset (i.e., Table 1), the proposed method outperforms most methods except Audio2Head. Could the authors provide an explanation for this?

[A6]: While Audio2Head outperforms our method in the Sync-C and Sync-D metrics—which assess the synchronization between audio and lip movements—this is due to its use of explicit facial keypoints, enabling more precise, parameterized modeling of lip movements. However, reliance on these keypoints may lead to generated outputs that lack realism in facial expressions. In contrast, our approach demonstrates a significant advantage over Audio2Head in the FID, FVD, and EFID metrics, reflecting superior visual quality and more realistic facial expressions.

[Q7]: The paper should include further explanation of the evaluation metrics used to help readers understand which capabilities are being measured and improved.

[A7]: The explanation of the evaluation metrics are provided as below. Specifically, FID measures the similarity between the distribution of generated images and real images. It evaluates the quality of generated images by computing the Fréchet distance between their feature distributions in a high-dimensional space. FVD is the video counterpart of FID, designed to evaluate the quality of generated videos. Sync-C and Sync-D metrics evaluate the synchronization between audio and lip movement in generated videos by caiculating the similarity between audio feature and vision feature extracting by syncnet. E-FID evaluates the expressiveness of the facial expressions in the generated videos by compute the FID score of generated video expression parameters and ground-truth video expression parameters.

[Q8]: In the third image on the bottom right side of Figure 6, there appear to be “water droplet-like” artifacts. Could the authors explain the cause of these artifacts?

[A8]: We have replaced the incorrect figure with the proper one in the updated revision.

[Q1]: For high-resolution enhancement, the paper lacks a comparison with video VQ models (e.g., MagVITv2) and does not include quantitative evaluations of the super-resolution results from the SR encoder-decoder.

[A1]: We appreciate your insightful suggestion. In Table 6 of the updated revision, we have included a comparison with video VQ models including MagVITv2 and include quantitative evaluations of the super-resolution results from the SR encoder-decoder. It's important to note that various VQ models and super-resolution methods are viable; we have selected one representative approach for this study. We are willing to follow the MagVITv2 formulation for future research.

(HRE represents the proposed high-resolution enhancement)

[Q2]: Since textual prompt serves as an additional conditioning factor, the authors should clarify how this information is labeled, details of training behavior (e.g., random dropping for CFG). How prompts are incorporated during training data preparation.

[A2]: To obtain the textual prompt in training data preparation, we use a vision-language model, named MiniCPM, to generate initial textual prompts. After that, we obtain the final textual prompt using Llama 1.5 to extract the key words containing expression and emotion from the original full caption by MimiCPM and format it. The textural prompt has the following format: {{human}{expression}}. As for details of training behavior, we train our model using almost 160 hours of videos without the textual control module first. After that, we add the textual control module to our model and only train this textual control module using about 40 hours of videos with textual prompt. During training, we use a probability of 0.05 to random drop the textual prompt condition.

[Q3]: For textual prompt control, the paper does not provide a detailed description of how prompts are used in comparisons with baseline methods. How is the textual condition used during inference?

[A3]: The detailed description of how prompts are used is as follows and updated in Line 405-411 of revision. During inference, there are two different ways to apply the textual condition. First, we use null textual prompt as the condition, when comparing with other baseline methods. Second, we use the regular textual prompt describing human emotion and set the CFG scale of textual prompt to 3.5 when discussing the effectiveness of textual prompt.

[Q4]: How CFG is applied during inference？It would also be useful to include an analysis of the impact of conflicts between non-textual and text-audio conditions.

[A4]: During inference, we set the CFG scale for the textual prompt, reference image, and audio all to 3.5. We also conducted an experiment to analyze the impact of conflicts between conditions. By comparing the first and third rows of the following table, we observe that a higher reference image CFG scale results in better image quality, as measured by FID. Similarly, comparing the second and third rows shows that a higher audio CFG scale improves lip synchronicity, as measured by Sync-C and Sync-D. Furthermore, comparing the third and the fourth rows reveals that a higher textual prompt CFG scale leads to more vivid expressions, as measured by E-FID. To balance all conditions and generate high-quality video, we set the CFG scale for the textual prompt, reference image, and audio all to 3.5.