Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation

We appreciate the reviewers’ positive feedback, including recognition of our well-motivated methodology, the novelty of L-RoPE, and our impressive quantitative and qualitative results. Below, we address the specific concerns raised by the reviewers individually.

Q1: More details of our dataset.

Thank you for raising these important questions regarding our dataset. We appreciate the opportunity to clarify the data-related aspects of our work. All data used in our experiments were collected from publicly available sources on the internet. We only downloaded data from platforms that explicitly permit data download for research purposes, and we strictly avoided any sources that prohibit such use. Our data collection process follows the best practices established by previous works [1]-[3], ensuring that our methods are consistent with the standards in the community. All data sources are under the CC BY 4.0 International license that allows for academic research and non-commercial use. We comply with the platform’s terms of service and applicable privacy regulations during the data collection and usage process. Our dataset comprises approximately 2,700 unique subjects, with approximately 71% male and 29% female. The distributions of age and race are presented in the following tables. We have made efforts to ensure diversity in age, race, and gender to the best extent possible based on the available metadata and annotations. We acknowledge that perfect demographic balance is challenging due to the nature of publicly available data, but we have taken steps to maximize representativeness. We strictly followed the NeurIPS ethics guidelines regarding privacy, consent, and copyright. All data is used strictly for research purposes and will not be commercialized.

For data preprocessing, we closely follow the procedures described in [1] and further filter out samples exhibiting large facial movements or unsynchronized speech and mouth motion [4]. This ensures the high quality and reliability of our dataset.

We acknowledge the importance of reproducibility and are committed to supporting the community. We will release the MTHM test dataset, along with the corresponding code and model weights, to the research community to encourage reproducibility and facilitate further research and development in audio-driven human animation.

[1] Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content

[2] Openvid-1m: A large-scale high-quality dataset for text-to-video generation

[3] MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions

[4] Out of time: automated lip sync in the wild

Q2: Clarification of our evaluation.

There appears to be a misunderstanding regarding our paper. The "MultiTalk single" entries in Tables 1 and 2 report results from our pre-trained single-person model on single-person datasets, without using L-RoPE. Similarly, "MultiTalk multiple" refers to our fine-tuned model, also evaluated without L-RoPE. In both cases, L-RoPE is not involved in single-person inference; it is only required for multi-person generation. We present both "MultiTalk single" and "MultiTalk multiple" in the table to demonstrate that L-RoPE does not degrade single-person performance, and our strong results in single-person scenarios are not due to L-RoPE.

Regarding differences between "MultiTalk single" and the FantasyTalking and Hallo3:

(1). Model architecture: Our method uses Wan2.1 as the video backbone, while Hallo3 uses CogVideo. Although both are DiT-based, there is a significant performance gap between different foundation models. Hallo3 also incorporates an identity preservation module, which we do not. Compared to FantasyTalking, we employ a different audio encoding method, and FantasyTalking introduces a Face-Cross-Attention mechanism, which our model does not have.

(2). Training strategy: We use partial parameter training and a multi-task training strategy, which are different from those in Hallo3 and FantasyTalking.

On the issue of data fairness, it is common practice in this field to compare methods using their official open-source models, as each work is trained on its own collected dataset. For example, Hallo3 uses 1,200 hours of YouTube data + 2,346 hours of film video, FantasyTalking leverages large-scale internet data, OmniHuman-1 uses 18.7K hours of data, and Hunyuan-Avatar uses 1,250 hours of collected data. Given these differences, we compare directly with the available open-source checkpoints, following standard practice.

Q3: Explanation of Table 3.

In our L-RoPE design, we assign distinct video and audio label ranges to different persons (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). Table 3 investigates the impact of using different label ranges for each person. The results indicate that varying the label assignments yields comparable performance results, which demonstrates that L-RoPE is robust to the choice of label ranges and not sensitive to this hyperparameter.

We acknowledge that Table 3 does not provide a direct comparison between models with and without L-RoPE. However, such an ablation is not feasible in the multi-person setting, as it is not possible to generate audio and lip synchronized videos for two individuals without the L-RoPE mechanism.

Apologies, I have just used a VPN and found out this is location-dependent. I will discuss with the ethics reviewers in the other thread

L-RoPE

Thank you for your question regarding the necessity of the L-RoPE mechanism for generating audio and lip-synchronized videos for multiple individuals.

After removing L-RoPE, the model is unable to achieve accurate synchronization between each audio stream and the corresponding person’s lip movements. The reason is that, without L-RoPE, there is no effective mechanism to bind each audio stream to the correct visual region in the attention map. As a result, the attention weights for each audio stream are distributed more uniformly across the entire video, which prevents the model from learning distinct audio-person associations. This case corresponds to Figure 3(a).

Beyond simply removing L-RoPE, we also experimented with alternative approaches, as shown in Figure 3(b) and (c):

Strategy (b): This approach also fails to establish effective audio-person binding, resulting in poor synchronization.

Strategy (c): This mask-based method can achieve audio-person association in simple cases with minimal motion, but it fails when there is significant movement or complex interactions.

We present the quantitative results for each of these approaches on the full MTHM dataset in the following table, which further demonstrates the effectiveness of L-RoPE for robust multi-person video generation. We will include these clarifications and results in the revised manuscript.

Ablations Study for Training Strategies

Yes, we confirm that the provided ablation studies for training strategies were performed on the full MTHM dataset. Thank you for your attention to this detail.

We appreciate the reviewers’ positive feedback, including recognition of our introduction of a new multi-person task and the affirmation of the L-RoPE. Below, we address the specific concerns raised by the reviewers individually.

Q1: The difference between L-RoPE and mask-based audio injection strategy.

Thank you for your question regarding the correspondence between audio streams and persons, and the potential use of mask-based approaches. Our method does not require obtaining a mask for each frame of the generated video. Instead, we only need to establish the correspondence between each person with their audio stream in the reference image, which is a much simpler and more robust requirement.

As you mentioned, the mask-based method is the same as in Figure 3c of our paper. The mask-based method explicitly binds a specific mask region to an audio stream, requiring a predefined mask for each frame throughout the video. However, this approach faces significant limitations: if the generated person moves outside the predefined mask area, the correspondence between speech and mouth motion can become unsynchronized, leading to degraded performance.

In contrast, our proposed L-RoPE method leverages visual similarity to dynamically associate each human appearance with its corresponding audio input, without relying on fixed spatial masks for the whole video. This design enables strong generalization and robustness, especially when the generated characters undergo large motions or pose changes. As a result, L-RoPE maintains accurate audio-visual synchronization even in challenging scenarios, which is a key advantage over mask-based and single-person methods. We further validated the effectiveness of L-RoPE through additional experiments comparing these two strategies, as presented in the table, where the mask-based method corresponds to (c) and L-RoPE corresponds to (d). The proposed L-RoPE achieves superior results.

Q2: Model collapse and large motion case.

Thank you for your comments and suggestions regarding the stability of adaptive person localization. We agree that the stability of similarity-based localization depends on the quality of the underlying video foundation model. With a less robust model, there is indeed a risk of latent collapse. However, in our work, we use Wan2.1, which is currently one of the most stable open-source video generation models. In our extensive experiments, we have not encountered latent collapse or related issues.

Regarding large motion, Figure 7 in our paper already demonstrates cases with significant movement. Additionally, we have conducted further tests involving both camera motion and substantial person movement, and our method is able to handle these scenarios effectively. We will include more such examples in the final version of the paper to better illustrate the robustness of our approach.

Q3: Training strategy and conflict between different conditions.

Thank you for your comments regarding the training strategy and potential conflicts in multi-task control. We agree that the distribution of the training data can affect the generation results. However, our training dataset does include a substantial number of samples featuring hands, bodies, and camera movement. Moreover, our findings are consistent with those reported in [1], which indicate that full parameter sharing in multi-task training can result in image quality degradation and distortions.

Regarding the concern about potential conflicts between text control and audio control, we have conducted extensive testing and have not observed any conflicts. The model is able to follow text-based instructions effectively while maintaining accurate audio-driven lip synchronization.

[1] OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation

Q4: Generalize to more speakers and overlap scenarios.

Thank you for your thoughtful questions regarding the generalization of the L-RoPE approach to scenarios with more than two speakers and overlap scenarios. Our L-RoPE framework demonstrates strong generalizability across varying numbers of speakers. In the two-speaker setting, we assign distinct, non-overlapping ranges of video and audio labels to each individual (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). For scenarios involving three or more individuals, we conducted additional experiments to verify that this labeling scheme can be extended by assigning new, non-overlapping ranges (e.g., video labels 40–44 and audio label 42 for a third person). This flexible approach enables the model to accommodate a greater number of speakers simply by expanding the label ranges for both video and audio streams.

Importantly, the L-RoPE extension strategy remains effective even when the number of persons during inference differs from that during training. By assigning dedicated label ranges, each person’s lip movements are accurately aligned with their respective audio streams. Furthermore, we observed that fine-tuning the model with data containing the same number of persons as inference can further improve the synchronization between audio and lip movements.

For overlap scenarios, when individuals’ faces are visible—even if their bodies are partially occluded—our method consistently associates the audio stream with the correct person. In scenarios where a person’s face is fully occluded and not visible in the frame, the corresponding audio does not drive lip motion or appearance for the occluded individual. Notably, if a person is temporarily occluded and subsequently reappears, our model is able to accurately re-associate the audio stream with the correct individual upon their return.

Visualization results for scenarios involving more speakers and overlap will be provided in the supplementary materials of the updated version.

Thanks for the authors' response. This resolves some of my doubts, but there are still the following questions.

Q1: For the mask strategy, it is not necessary to provide it for every frame, as the task focuses on humans or characters, and tracking can be used to achieve this. The method proposed in this article still needs to establish an initial relationship between each character and their audios, which does not fundamentally differ much from providing a mask for the first frame; in addition, the advantage mentioned for large movements in characters, I have not seen any video demos.

Q3: Opening full parameters training will lead to a decrease in image quality dose not make sense, this is most likely a problem with your training strategy and data quality; by the way, regarding the mentioned control conflict issue, let's assume a scenario: a character is putting something into their mouth with their hand (controlled by text), and they are also speaking (using audio). Assuming only the audio branch is trained, the text branch will also perform control. What is the reason there is no conflict in this situation if only train the audio branch?

Given the above, I choose to keep my initial rating.

We appreciate the reviewers’ positive feedback, including recognition of our clear and detailed writing, superior generation results, and strong capabilities in multi-person and interactive video generation. Below, we address the specific concerns raised by the reviewers individually.

Q1: Definition of the new task and its distinction from single-person talking face task.

Thank you for your valuable feedback. We appreciate your suggestion to provide a more formal definition of our proposed task and to clarify its distinction from existing single-person talking face generation tasks.

Audio-driven multi-person conversational video generation is defined as follows: Given a reference image containing multiple persons and corresponding audio streams (with a one-to-one correspondence between each person and their audio), the goal is to synthesize a video sequence in which all persons appear together in the same frame, and each person’s lip movements are temporally synchronized with their respective audio input. Unlike previous single-person talking face generation tasks, this new task requires the joint modeling of multi-person interactions, spatial consistency, and audio-visual synchronization within a unified, end-to-end generative framework, requiring only a single diffusion process.

In contrast to approaches that generate videos for each person independently and subsequently composite them, this new task demands integrated modeling of multiple individuals, offering several key advantages:

1. Higher computational efficiency: Only a single inference process is required, substantially reducing computational costs.

2. Global consistency: The unified framework enables better control over the overall coherence of the generated content, such as coordinated camera movements, lighting, and scene dynamics.

3. Enhanced interaction modeling: This approach is inherently more suitable for capturing interactions among individuals, enabling natural and contextually appropriate reactions (e.g., when one person is speaking, others can display attentive or responsive behaviors).

We will revise the paper to include a formal definition of the new task and add a dedicated section to compare it with existing single-person talking face tasks. This will help clarify the novelty and technical challenges of our work.

Q2: Generalize to more speakers.

Thank you for your thoughtful questions regarding the generalization of the L-RoPE approach to scenarios with more than two speakers and varying numbers of persons at inference time. Our L-RoPE framework demonstrates strong generalizability across varying numbers of speakers. In the two-speaker setting, we assign distinct, non-overlapping ranges of video and audio labels to each individual (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). For scenarios involving three or more individuals, we conducted additional experiments to verify that this labeling scheme can be extended by assigning new, non-overlapping ranges (e.g., video labels 40–44 and audio label 42 for a third person). This flexible approach enables the model to accommodate a greater number of speakers simply by expanding the label ranges for both video and audio streams.

Importantly, the L-RoPE extension strategy remains effective even when the number of persons during inference differs from that during training. By assigning dedicated label ranges, each person’s lip movements are accurately aligned with their respective audio streams. Furthermore, we observed that fine-tuning the model with data containing the same number of persons as inference can further improve the synchronization between audio and lip movements.

Visualization results for scenarios involving more speakers will be provided in the supplementary materials of the updated version.

Q3: Identity degradation in the generated videos.

Thank you for pointing out the issue regarding identity preservation in our generated videos. We acknowledge that identity degradation may occur in generated videos, particularly when the video duration is excessively long or when there are substantial changes in facial motion angles. This limitation arises because the video foundation model we employ also suffers from the identity degradation problem. On one hand, as video foundation models continue to advance, the issue of identity degradation is expected to be mitigated. On the other hand, in future work, we plan to investigate explicit identity injection techniques to further enhance identity preservation.

Thank you again for your valuable feedback.

Q4: Dataset release and experimental reproducibility.

Thank you for your comments and concerns regarding the dataset and reproducibility. All data used in our experiments were collected from publicly available sources on the internet. Our data collection process follows the best practices established by previous works [1]-[3]. For data preprocessing, we closely follow the procedures described in [1] and further filter out samples exhibiting large facial movements or unsynchronized speech and mouth motion [4], and do not have any unique data processing procedures.

We acknowledge the importance of reproducibility and are committed to supporting the community. We will release the MTHM test dataset, along with the corresponding code and model weights, to the research community to encourage reproducibility and facilitate further research and development in audio-driven human animation.

[1] Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content

[2] Openvid-1m: A large-scale high-quality dataset for text-to-video generation

[3] MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions

[4] Out of time: automated lip sync in the wild

Q5: Broader impact of the paper.

Thank you for highlighting the importance of discussing the broader impact and ethical considerations of our work. This paper introduces an effective approach for audio-driven multi-person conversational video generation to the community. However, this technology also raises significant ethical concerns. Beyond the risk of generating fake videos of celebrities, there are broader implications, including the potential for misuse in creating deepfakes for misinformation, defamation, fraud, or harassment. Such synthetic videos could be used to impersonate individuals, manipulate public opinion, or violate privacy. These risks are not unique to our approach but are shared across the broader field of human animation and generative media.

Thank you for your suggestion. We will include the discussion of broader impact in the main manuscript in the updated version.

Q6: Inference time.

Thank you for your valuable suggestion regarding the evaluation of video generation time. Although our method introduces an additional audio condition, the computational time required to pass through the DiT backbone remains the same as in Wan2.1. On an NVIDIA A100 GPU, each inference step in our method takes approximately 35 seconds, resulting in a total generation time of about 1,430 seconds for a 40-step inference. Furthermore, all acceleration strategies available for Wan2.1—such as TeaCache and model distillation—are also applicable to our approach. For example, when employing a distilled model (such as lightx2v), the total inference time is reduced to approximately 170 seconds per video.

Q7: Audio injection results.

Thank you for the valuable comments. We conducted an additional ablation study to investigate the impact of different audio injection strategies. The results are summarized in the table below, with each row corresponding to an audio injection strategy as illustrated in Figure 3. Strategies (a) and (b) fail to bind multi-stream audio to the corresponding video latent regions. Strategy (c) employs a hard mask-based audio binding approach, which is capable of associating multi-stream audio with different persons; however, its effectiveness is limited to videos with minimal motion. When a person exhibits extensive movement, this strategy also results in failure cases. In contrast, our proposed L-RoPE method (d) achieves the best results across all tested scenarios, demonstrating the superiority of our approach. We will provide these results in the supplementary materials of the updated version.

We appreciate the reviewers’ positive feedback, including recognition of our introduction of a new multi-person task, the reasonableness of our method, the clarity of our writing, and the promising results. Below, we address the specific concerns raised by the reviewers individually.

Q1: Ablation study for multi-stream audio injection.

Thank you for the valuable comments. We conducted an additional ablation study to investigate the impact of different audio injection strategies. The results are summarized in the table below, with each row corresponding to an audio injection strategy as illustrated in Figure 3. Strategies (a) and (b) fail to bind multi-stream audio to the corresponding video latent regions. Strategy (c) employs a hard mask-based audio binding approach, which is capable of associating multi-stream audio with different persons; however, its effectiveness is limited to videos with minimal motion. When a person exhibits extensive movement, this strategy also results in failure cases. In contrast, our proposed L-RoPE method (d) achieves the best results across all tested scenarios, demonstrating the superiority of our approach. We will provide these results in the supplementary materials of the updated version.

Q2: Generalize to more speakers.

Thank you for your thoughtful questions regarding the generalization of the L-RoPE approach to scenarios with more than two speakers and varying numbers of persons at inference time. Our L-RoPE framework demonstrates strong generalizability across varying numbers of speakers. In the two-speaker setting, we assign distinct, non-overlapping ranges of video and audio labels to each individual (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). For scenarios involving three or more individuals, we conducted additional experiments to verify that this labeling scheme can be extended by assigning new, non-overlapping ranges (e.g., video labels 40–44 and audio label 42 for a third person). This flexible approach enables the model to accommodate a greater number of speakers simply by expanding the label ranges for both video and audio streams.

Importantly, the L-RoPE extension strategy remains effective even when the number of persons during inference differs from that during training. By assigning dedicated label ranges, each person’s lip movements are accurately aligned with their respective audio streams. Furthermore, we observed that fine-tuning the model with data containing the same number of persons as inference can further improve the synchronization between audio and lip movements.

Visualization results for scenarios involving more speakers will be provided in the supplementary materials of the updated version.

We appreciate the reviewers’ positive feedback, including recognition of our introduction of a new multi-person task, impressive qualitative results, the novelty and effectiveness of our method, thorough experimental validation, and superior performance. Below, we address the specific concerns raised by the reviewers individually.

Q1: Robustness under occlusion and overlap scenarios.

Thank you for your question regarding the robustness of our adaptive person localization method in highly occluded or overlapping scenarios. When individuals’ faces are visible—even if their bodies are partially occluded—our method consistently associates the audio stream with the correct person. In scenarios where a person’s face is fully occluded and not visible in the frame, the corresponding audio does not drive lip motion or appearance for the occluded individual. Notably, if a person is temporarily occluded and subsequently reappears, our model is able to accurately re-associate the audio stream with the correct individual upon their return. The corresponding visualization results will be provided in the supplementary materials of the updated version.

Q2: Handling more complex instructions.

Thank you for your insightful comment regarding the instruction-following capabilities of our model. We conducted additional evaluations to assess the robustness of our method on more complex and compositional prompts:

Multi-step narratives: Since Wan2.1 can only generate 81 frames per chunk, it is challenging to complete multiple actions within a single chunk. Hence, we decompose multi-step narratives into several actions, generating each chunk with prompts specifying a few actions. Through leveraging the long video generation mentioned in the paper, our model is able to represent multi-step narratives effectively.

Emotional expressions: We use the same reference image, and specify emotional (e.g., angry, sad, happy) via the text prompt. Our model successfully generates videos with the corresponding emotional expressions.

Simultaneous actions: In multi-person scenarios, we assign different actions to each person via text prompts. The resulting videos demonstrate that both individuals can perform their respective actions simultaneously.

Overlapping speech: By providing separate audio streams for each person, our model generates synchronized lip movements for both speakers without interference, thus achieving overlapping speech.

Indirect commands: Our model currently struggles with indirect commands, likely due to the base video model’s limited ability to understand such instructions.

Due to rebuttal format constraints, we are unable to include success and failure visualization results here. However, we will provide these examples in the updated version.

Q3: Generalization to more speakers and variable person numbers.

Thank you for your thoughtful questions regarding the generalization of the L-RoPE approach to scenarios with more than two speakers and varying numbers of persons at inference time. Our L-RoPE framework demonstrates strong generalizability across varying numbers of speakers. In the two-speaker setting, we assign distinct, non-overlapping ranges of video and audio labels to each individual (e.g., video labels 0–4 for person 1 and 20–24 for person 2; audio labels 2 for person 1 and 22 for person 2). For scenarios involving three or more individuals, we conducted additional experiments to verify that this labeling scheme can be extended by assigning new, non-overlapping ranges (e.g., video labels 40–44 and audio label 42 for a third person). This flexible approach enables the model to accommodate a greater number of speakers simply by expanding the label ranges for both video and audio streams.

Importantly, the L-RoPE extension strategy remains effective even when the number of persons during inference differs from that during training. By assigning dedicated label ranges, each person’s lip movements are accurately aligned with their respective audio streams. Furthermore, we observed that fine-tuning the model with data containing the same number of persons as inference can further improve the synchronization between audio and lip movements.

Visualization results for scenarios involving more speakers will be provided in the supplementary materials of the updated version.

Q4: Performance gap between real and synthetic audio.

Thank you for your valuable question regarding the performance gap between real and synthetic (TTS-generated) audio. The observed gap can be primarily attributed to a domain discrepancy: our model is trained exclusively on real audio, which typically contains rich emotional cues and natural prosody. As a result, the generated videos exhibit more expressive and realistic facial behaviors when driven by real audio. In contrast, most current TTS-generated audio lacks emotional variation and nuanced expressiveness, leading to video outputs that appear less vivid and natural. A promising approach to address this issue is to augment the training data with TTS-generated audio, thereby enabling the model to better generalize to synthetic audio inputs. More detailed analysis and experimental results on TTS robustness will be provided in the updated version.