Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1: About the ablation of inter/intra- clip layers

First, we apologize for not clearly explaining this in the paper. These results are actually already included, corresponding to "w/o inter-clip temp." in Line 463 and “1 temp.+ 20 MF” in Line 466. After removing the inter-clip temporal layer, to retain the information of motion frames, the remaining single temporal layer becomes the setting for "inter-clip temporal layer only." The two results mentioned above correspond to different numbers of motion frames (4 and 20). It can be observed that regardless of the setting, there is a significant performance drop compared to the full model. For the setting where the inter-clip temporal layer is completely removed and motion frames are not retained, the synthesized long videos exhibit noticeable misalignment every 12 frames, rendering the results unusable. Therefore, we did not evaluate this setting.

Q2: About TSM functionality and worse results with longer motion frames.

From Lines 466-469, it can be observed that applying the dual temporal layer improves overall performance as the number of motion frames increases (by adjusting TSM parameters for longer coverage). This indicates that the TSM helps establish longer-term motion dependencies, enhancing final performance. In contrast, Line 478 states that increasing the number of motion frames leads to performance degradation when using a single temporal layer. This is because, with a single temporal layer, the temporal information needs to handle 20 frames of non-noisy latent features from the last clip and 12 frames of latent features from the current timestep. When the former dominates, it affects the temporal modeling of the noisy latents in the current clip, thereby impacting the generation task. In contrast, the dual temporal layer can distinguish between these two scenarios: one temporal layer models the inter-clip temporal information dominated by non-noisy latent features from the last clip, while the other models the intra-clip temporal information dominated by the noisy latents of the current clip. This distinction enhances performance.

Q3: About different roles of the two proposed modules.

The proposed modules directly impact motion, so intuitively, removing any module will lead to a decline in motion modeling capability and overall expressiveness.

1. From the supplementary materials, it can be observed that removing the audio2latents module results in a more noticeable decline in eyebrow and eye movements (independent movements rather than overall motion), which are the parts that its auxiliary spatial condition can control.
2. We provide results from the challenging test set CelebV-HQ in video-abalation, where the model without the proposed modules struggles to handle, especially without the inter/intra-clip temporal layers, better demonstrating their functionality. It can be observed that models without the inter/intra-clip temporal layer are more prone to random movements and temporal inconsistencies, while models without the Audio-to-Latents module tend to produce more expressionless results. Overall, both proposed modules enhance motion modeling, but with different focuses: the inter/intra-clip temporal layer improves temporal naturalness and continuity, while the Audio2Latents module enhances subtle expression generation. This distinction is more evident in complex examples.

Q4: About the motion frames setting during inference

During inference, if motion frames need to be filled, this is achieved by duplicating the reference image multiple times. However, during training, motion frames are almost never exactly the same. Therefore, for the first generated segment, if zero initialization is not used, there is about a 20% chance (based on rough estimates) that the generated result will have varying degrees of color difference compared to the original image. Other than this, there is almost no difference. Additionally, zero-initializing motion frames for the first segment can save one motion frame VAE encoding time (although we later found this impact to be minimal, we retained this setting).

Q5: About whether the head is controllable and movement and expression variances.

Because Loopy is an end-to-end audio-only model, we cannot precisely control head movements. However, the training of audio2latents allows some influence over head movements and expressions. In video-control_demo, we show results of replacing audio features with specified exp and var values in the audio2latents module. As input values change, the generated results vary accordingly, similar to controlling expression types.

Summary

Based on your comments, we analyzed the implementation and effectiveness of dual temporal layers and TSM in our responses, and added a discussion on the motion frame inference setting in the revised version. Thank you again for your positive comments and valuable suggestions.

Q6: About the discussion of the limitations of the proposed method.

Thank you for your reminder. Our intention in using an open-set test set is to explore the capability boundaries of the Loopy model. In video-limitation we provide an analysis of the current method's limitations. Regarding input images, Loopy struggles with images containing large proportions of upper limbs, blurry or occluded human figures, animals, and multiple people, often resulting in incongruous or uncanny valley effects. Regarding input audio, Loopy performs poorly with long periods of silence, chaotic background noise, and extended high-pitched sounds. Overall, Loopy's synthesis quality suffers from long-tail effects, showing robustness issues and awkward movements for uncommon input types.

Q7: About evaluating the method's performance reasonably

Yes, this is indeed a perplexing issue for us as well. We have strived to provide a fair comparison by training our model on HDTF. Even with only 1/10 to 1/20 of the training data compared to other methods, Loopy still achieves leading performance on several key metrics. On the collected dataset, Loopy also shows significant improvements through comparative experiments and provided videos, demonstrating the effectiveness of the proposed modules both qualitatively and quantitatively. From an evaluation perspective, misaligned datasets pose a challenging problem for evaluation. Retraining methods on large datasets involves significant time and setup costs, especially for diffusion models. We currently believe that demonstrating module effectiveness through comparative experiments and showing qualitative and quantitative improvements in final results to achieve state-of-the-art performance is a suitable approach.

Q8: About the relationship between improvement and self-collected data

Based on the considerations from the previous response, we provide a simple baseline trained on the collected dataset. This baseline removes components such as audio2latents, dual temporal layers, and TSM, making it very close to echomimic, except for spatial condition modules like the landmark encoder and weighted loss, which are functional components or independent optimization points. This comparison better illustrates the effectiveness of our method. The significant performance drop observed when all proposed modules are removed underscores their importance.

Regarding the audio-visual synchronization metrics, our method leads even without the proprietary dataset (HDTF only), outperforming other methods that use 10-20 times more training data. This indicates that our performance advantage is not solely due to the amount of data. In fact, the HDTF training set includes very high-quality speech videos, and we found that audio-visual synchronization is relatively easy to learn. The increase in data volume primarily enhances the overall synthesis quality, such as image quality and the learning of complex motion patterns, which also benefits from the efficient utilization of data by the proposed modules.

The poorer performance of the compared methods could be due to various reasons. For example, echomimic exhibits large movements but poor lipsync, while hallo shows the opposite. It is possible that these methods did not effectively leverage the long-term motion dependencies in the data, and the introduction of spatial conditions caused motion pattern misalignment between training and testing, resulting in imbalanced model performance and suboptimal overall results.

Q9: About the independent control of head movements and expressions

Certainly. In video-control, we provide the results of replacing the audio features input with specified exp values and var values in the audio2latents module. As the input values change, the generated results also exhibit corresponding variations.

Summary

Based on your comments, we have added an analysis of the computational efficiency impact of TSM, addressed the limitations of loopy, and provided a simple baseline to highlight our module's effectiveness. We appreciate the reviewer's suggestions on head movement and expression detection using FLAME, which will aid our further research. Thank you for your positive rating and valuable feedback.

We are grateful for your positive review and valuable suggestions, which are also constructively helpful for our subsequent research, and we hope our response fully resolves your concerns.

Q1: About the effects of movement and expression and implementation details.

We first apologize for any lack of clarity in our previous explanation. Your understanding is correct. the term "audio" here refers to the concatenated features from the multiscale outputs of wav2vec, as described in Line 280. For the effects of movement and expression, in Table 3, we conducted a abalation experiment by removing the audio-to-latents module, which resulted in a noticeable decline in performance. Additionally, in video-abalation and the supplementary materials, it can be seen that the audio-to-latents module enhances the expressiveness of movements, such as the eyebrows and eyes. In video-control_demo, we also demonstrate the ability to control movement and expression independently, illustrating their relationship with portrait motion. Based on this, strong movement-related signals can help the model better learn the relationship between audio and portrait movement.

Q2: Regarding expressions and movement detection.

Thank you for your excellent suggestion. First, we provide some video-detection_demo to demonstrate that the current landmark-based calculations of var and exp can capture most of the variations in facial expressions and head movements. Based on these observations, we initially chose the landmark-based detection method in our experiments, which currently generates satisfactory results. Second, it is evident that the current landmark detection does have some motion coupling issues. FLAME-based detection can bring intuitive improvements. We are considering incorporating training strategies with detection confidence scores, which might further enhance the role of auxiliary spatial conditions during training. We will investigate this further in future work. Thank you.

Q3: About the computational efficiency after introducing TSM

The introduction of TSM may result in increased time consumption in two areas: (1) The VAE computation before the denoising network increases from 4 frames to 20 frames. However, this time can be omitted because the first segment uses zero-initialized motion frames, and subsequent segments use VAE latents predicted by the previous denoising net, resulting in no additional computation compared to the baseline. (2) In the denoising network, the temporal attention increases from 12+4 to 12+20.

Below, we present the time comparison for generating 12 frames in 25 denoising steps, as well as the time consumption and corresponding RTF (real-time factor) for generating a 15-second video. The experiments were conducted on an unloaded A100. The introduction of TSM only increases the RTF by 3%, which we believe is acceptable given the improvement.

Q4: About more analysis on the performance in Table 1 and 2.

In Tables 1 and 2, Loopy achieves the top performance on most metrics. Although it ranks second on some lipsync and EFID metrics, the differences are minimal (lipsync and expression matching evaluations are inherently challenging and subject to some degree of fluctuation or error). As for GLO and EXP, which represent the absolute magnitude of movements, bigger values are not necessarily better. These metrics are provided to illustrate the preference differences in movement magnitude among different methods, with some methods tending towards stillness to avoid errors. The metrics DGLO and DEXP represent the deviations of GLO and EXP from the corresponding values in the ground truth (GT) videos, and Loopy still leads in most cases. Since objective metrics may have some inaccuracies, we also include subjective evaluations in Figure 5 to validate Loopy's superiority across multiple dimensions. The results show that Loopy has an overwhelming advantage.

Q5: About the writing issues.

Thank you very much for pointing out these issues. We have made the revisions in the revised version, reviewed the writing again, and corrected some inappropriate uses of punctuation, such as ":".

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1: Regarding the impact of different audio characteristics (e.g., background noise, varying loudness) on the model’s performance

Thank you for your meaningful suggestion. Below, we provide an analysis of the model's performance with audio inputs of varying disturbance levels (adding different Gaussian noise and applying random volume scaling). Due to time constraints, we tested the first 20 examples from the HDTF test set. It can be observed that even with significant background noise (which is already quite disruptive for humans, 0.02 noise), Loopy still maintains good audio-visual synchronization and performs comparably to existing methods under normal input conditions. However, for noise that masks human voices, Loopy's performance noticeably declines. In terms of loudness disturbances, Loopy is relatively robust and can still achieve good results. Therefore, Loopy demonstrates good robustness to relatively common abnormal samples. In our response to Q2, we have also provided videos and further analysis of the limitations.

Q2: Regarding the imitations of the proposed method and failure cases.

Thank you for your reminder. Our intention in using an open-set test set is to explore the capability boundaries of the Loopy model. In video-limitation we provide an analysis of the current method's limitations. Regarding input images, Loopy struggles with images containing large proportions of upper limbs, blurry or occluded human figures, animals, and multiple people, often resulting in incongruous or uncanny valley effects. Regarding input audio, Loopy performs poorly with long periods of silence, chaotic background noise, and extended high-pitched sounds. Overall, Loopy's synthesis quality suffers from long-tail effects, showing robustness issues and awkward movements for uncommon input types.

Q3: Regarding the influence of the training data and other baselines trained on our collected dataset.

To validate our method's effectiveness, we demonstrated its impact in two ways:

(1) In Table 3, results trained on HDTF-only show our method outperforms other baselines on key metrics like audio-visual synchronization (sync-c/sync-d) and overall video quality (FVD), even without additional collected data.

(2) The ablation analysis in Table 3 and the supplementary videos highlights the improvements of the proposed modules from both quantitative and qualitative perspectives.

Training on the same dataset is a valuable suggestion. Due to limited time and different stage-1/2 training parameters of different methods, we trained a simple baseline on the collected dataset using our pre-trained stage-1 weights. This baseline removes components like audio2latents, dual temporal layers, and TSM, making it similar to echomimic, except for spatial condition modules like the landmark encoder and weighted loss. As shown in the table below, the simple baseline performs significantly worse than the full model, with substantial declines in most metrics, even falling short of the full model trained only on HDTF. This, combined with comparative experiments, clearly demonstrates that the improvements in our method primarily stem from the proposed modules.

Q4: Regarding the public availability of the dataset

At this time, we are indeed unable to provide a definite plan for making the dataset publicly available, as this decision requires further consideration and internal data review processes. We have included data filtering details and data composition in the revised version, and we hope this will be helpful.

Summary

Thank you for your valuable suggestions. In the revised version, we have added discussions on the limitations of Loopy and comparisons with the simple baseline. We believe this will make the paper more complete. Thank you again for your suggestions and positive feedback on our work.

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1: Regarding the performance degradation caused by the removal of the audio-to-latent module.

First, we apologize for any lack of clarity in our previous explanation. In Table 3, "w/o A2L" refers to the removal of the A2L module while still retaining the injection of audio features into the denoising network via cross-attention ( indicated by the audio attention layer in Figure 2 and Line 283). Removing the A2L module means the model retains only the original audio-video pixel mapping relationship, reducing its capacity to model audio-portrait relationships compared to the full model. Although the A2L module highlights head and facial expression changes, the audio2latents can still provide additional audio control information about the portrait. Mapping to latents also facilitates better learning of the relationship between audio and movement, such as lip movement, which can be considered a combination of several basic features (similar to blendshapes). Consequently, this leads to performance degradation, with the most significant impact observed in the audio-related metrics, sync-c and sync-d, especially on the challenging open-set test set.

Q2: Regarding the weighting scheme of audio ratio and ref ratio

Thank you for the reminder. Our initial parameter settings were based on visual observations of a few cases. Below, we provide the experimental results where we fix either the audio ratio or the reference ratio and vary the other parameter. Due to time constraints, we tested the first 20 examples from the HDTF test set for lipsync and FVD. It can be observed that increasing the audio ratio gradually enhances the accuracy of lip synchronization, reflecting the relevance of the audio, until it eventually saturates. On the other hand, increasing the reference ratio improves the overall video synthesis quality, but an excessively high reference ratio can actually degrade the quality. This may be because the reference image not only affects identity preservation but also influences the degree to which the generated result follows the reference image. An excessively high cfg can lead to reduced motion amplitude and even color changes. A reference ratio of around 3 and an audio ratio of around 5 appear to be a well-balanced choice.

Q3: Regarding further optimization and adaption to real-time settings.

Very valuable suggestion. This is exactly what we aim to explore in our future work. Currently, it takes 18 seconds to complete 25 steps of denoising to generate 12 frames on an A100 GPU, resulting in a real-time factor (RTF) of 43.1 for a 25 FPS output. By leveraging computational parallelism, we can offload audio and reference CFG inference, as well as VAE, to other GPUs for an estimated 3x speedup. Additionally, using few-step distillation methods to reduce the steps to within 4 (estimated 6x speedup), we could potentially achieve a computation speed of around 2 seconds. Combining this with acceleration techniques like TensorRT and caching mechanisms, or using advanced GPUs like the H100 or slightly lowering the frame rate, real-time performance might be achievable. Although feasible, actual implementation may not be so smooth and poses many challenges that we aim to address in future work. Thank you.

Q4: Regarding the imitations of the proposed method and failure cases.

Thank you for your reminder. Our intention in using an open-set test set is to explore the capability boundaries of the Loopy model. In video- limitation, we provide an analysis of the current method's limitations. Regarding input images, Loopy struggles with images containing large proportions of upper limbs, blurry or occluded human figures, animals, and multiple people, often resulting in incongruous or uncanny valley effects. Regarding input audio, Loopy performs poorly with long periods of silence, chaotic background noise, and extended high-pitched sounds. Overall, Loopy's synthesis quality suffers from long-tail effects, showing robustness issues and awkward movements for uncommon input types.

Summary

Following your comments, we added discussions on multi-CFG impact, real-time inference potential, and method limitations in the revised appendix. We think this enhances the paper and better explains the method. Thank you again for your valuable comments and positive recommendation for our paper.

Thank you to the AC and reviewers for recognizing our work. We have made modifications to the camera-ready version according to the instructions.

For testing, most current diffusion-based audio-driven portrait synthesis methods use self-collected datasets of similar scale from mixed sources. In addition to differences in training sets, there is no clear consensus on test set selection in most methods. To address this and provide a fairer comparison, we have included the results of the open-sourced Hallo trained on our dataset and provided the split and processing of the test set on the project page. We have also included a detailed description of the dataset.

For writing, we have carefully reviewed the entire paper in the hope of further enhancing the quality of the camera-ready version.

Finally, we express our sincere gratitude to the conference organizers for their efforts and thank them for the valuable feedback that has significantly improved the quality of our work.