Audio-Sync Video Generation with Multi-Stream Temporal Control

We thank the reviewers for their insightful and constructive feedback.

In accordance with the rebuttal policy, we are unable to include new qualitative results (e.g., generated videos) or external links. We assure the reviewers that, should the paper be accepted, the camera-ready version will be thoroughly updated to include:

1. All additional experiments and qualitative results are discussed in this rebuttal.

2. A public link to an open-source demo showcasing our MTV framework as a practical video generation tool.

We now address the specific points raised by the reviewers.

W1: Copyright

We thank the reviewer for this important question. We take data sourcing and copyright considerations very seriously and have followed established academic practices.

1. Data sources:

As detailed in our paper (Sec. 3, Lines 109-112), our DEMIX dataset is constructed from two main sources:

Existing Public Datasets: We use several well-established benchmarks, including CelebV-HQ, MovieBench, Condensed Movies, and Short-Films 20K.

Publicly Available Videos: We also collect data from YouTube, mirroring the widely-used academic datasets like VGGSound.

2. Release justification:

For videos from existing datasets, we will follow their original protocols by providing the video IDs and other necessary metadata for users to retrieve the data from the original sources.

For videos we collected from YouTube, we will provide a list of the corresponding URLs and timestamps.

For our contributions, we will directly provide our generated annotations, including text descriptions and the demixed audio tracks.

We do not redistribute the original video/audio content. The dataset is compiled strictly for academic research.

3. Licensing:

To ensure clarity, we will adopt a dual-licensing approach similar to VGGSound:

Our code, annotations, and data-processing scripts will be released under the Apache 2.0 License.

We will explicitly state that the copyright of the underlying video and audio content remains with their original owners.

W2: Layer-adaptive feature extraction

We sincerely thank the reviewer for this insightful suggestion and for proposing an interesting alternative design.

Our current design (audio features are injected into all DiT blocks) follows the established practice in state-of-the-art talking face generation models (e.g., FantasyTalking). Since this approach has been proven effective and robust, we adopt it as our primary design.

To explore the reviewer's suggestion, we are implementing a layer-wise approach. We find that directly adapting the standard DiT blocks of CogVideoX results in out-of-memory errors, even using eight A800 80G GPUs. Therefore, we are proceeding by modifying our own MTSControlNet to perform layer-wise injection, where specific blocks of audio features are injected into corresponding blocks of the DiT backbone.

Due to the time constraints of the rebuttal period, this experiment is currently in progress. We will post the results and our analysis as soon as they are available.

W3: Adaptability to additional T2V backbone

Our proposed MSTControlNet can be seamlessly integrated into various T2V backbones (e.g., OpenSora and Wan) without architectural changes, as they share a similar DiT-based structure. Due to time limitations, we only implement and test the integration with Wan14B. Specifically, our interval feature injection and holistic style injection modules are added after each text cross-attention layer.

For inference, MTV (Wan14B backbone) requires 180s to generate an 81-frame, audio-synchronized video at a 480P resolution on 8 NVIDIA A100 GPUs. We report the new quantitative results below.

Q1: Copyright

See W1 please.

Following up on our previous response, we have now completed the experiments regarding your insightful suggestion on layer-adaptive feature extraction. We thank you again for motivating this valuable ablation study.

As we state previously, directly adapting standard DiT blocks results in out-of-memory errors. Therefore, we implement the suggestion by modifying our MST-ControlNet to inject features from its different layers into the corresponding blocks of the DiT backbone. Since the CogVideoX backbone has 42 DiT blocks, we test configurations where the number of MST-ControlNet blocks (and thus the number of injection points) is set to N=4, 21, and 42.

As result presented below, these metrics reveal a complex trade-off:

• For audio-related metrics (Audio-C and Sync-C/D), the metrics generally improve as more blocks are used, indicating stronger audio synchronization.

• For the temporal consistency of generated videos (Temp-C), the metric degrads as the number of blocks increases.

• For overall quality and text alignment (FVD and Text-C), the metrics fluctuate without a clear trend.

As a result, increasing the number of blocks in a layer-wise injection approach improves audio synchronization, but this comes at the cost of video fluency. It appears that forcing the model to adapt to audio features at too many layers overly prioritizes synchronization at the expense of the video's natural temporal flow. We hypothesize this is because audio features from earlier MST-ControlNet layers are less refined, which means the backbone's early layers do not receive sufficient high-quality audio cues.

Notably, although we test these different layer-wise configurations, our original proposed architecture still achieves the best overall performance. This suggests that for our task, layer-wise injection may not be the optimal approach.

We thank the reviewers for their insightful and constructive feedback.

In accordance with the rebuttal policy, we are unable to include new qualitative results (e.g., generated videos) or external links. We assure the reviewers that, should the paper be accepted, the camera-ready version will be thoroughly updated to include:

1. All additional experiments and qualitative results are discussed in this rebuttal.

2. A public link to an open-source demo showcasing our MTV framework as a practical video generation tool.

We now address the specific points raised by the reviewers.

Weakness

W1: Errors in source separation

We agree that the upstream source separation is not always perfect. However, all of our reported results are generated from these imperfectly separated audio tracks, not from idealized clean inputs. As shown in Tab. 2 and Fig. 3, our framework achieves state-of-the-art performance across six standard metrics, directly demonstrating its inherent robustness in real-world scenarios. This stems from our demixing filtering strategy (Lines 117-122). We explicitly filter out extremely poor-quality samples during training, which enables our model to learn precise audio-visual synchronization, while the remaining unclear separations serve as a form of data augmentation that enhances robustness.

To completely bypass this separation challenge, we further developed a demo. It uses Qwen3 to interpret user prompts into audio descriptions, synthesizes them into perfectly clean audio with ElevenLabs, and then generates a synchronized video with our MTV framework. This presents a complete text-to-video pipeline to avoid potential source separation errors entirely.

W2: Additional datasets and backbone

We thank the reviewer for this valuable suggestion.

1. Evaluation on new datasets

To demonstrate the generalization capabilities of our MTV framework, we follow TempoTokens to conduct additional experiments on both the Landscape and AudioSet-Drum datasets. To make a fair comparison, we fine-tune all baseline methods (MM-Diffusion, TempoTokens, and Xing et al.) on our DEMIX dataset using their official training schedules, and evaluate them on these separate datasets. As shown in the tables below, our MTV framework still achieves significantly better performance. Since neither dataset includes human talking, the lip synchronization metrics (Sync-C and Sync-D) are not applicable for this evaluation.

Comparison on Landscape datasets

Comparison on AudioSet-Drum dataset

2. Adaptability to additional T2V backbone

Our proposed MSTControlNet can be seamlessly integrated into various T2V backbones (e.g., OpenSora and Wan14B) without architectural changes, as they share a similar DiT-based structure. Due to time limitations, we only implement and test the integration with Wan14B. Specifically, our interval feature injection and holistic style injection modules are added after each text cross-attention layer.

For inference, MTV (Wan14B backbone) requires 180s to generate an 81-frame, audio-synchronized video at a 480P resolution on 8 NVIDIA A100 GPUs. We report the new quantitative results below.

W3: Human assessments

To better evaluate our method from a human perception perspective, we conducted three subjective user study experiments as requested. We presented videos generated by our method and all baselines to participants and asked them to choose the best one based on the following criteria:

• Semantic consistency: "Which video best matches the text description?"

• Motion fluency: "Which video displays more realistic and temporally coherent motion?"

• Overall preference: "Which video do you prefer overall, considering all aspects?"

For each study, we randomly select 50 text descriptions from the test set, and the evaluations are conducted by 25 volunteers. The table below shows the percentage of times each method is chosen as the winner. Our method is consistently favored by human observers and has achieved the highest scores across all three subjective criteria.

We hope that our new experimental results have fully addressed the reviewer's concerns, and we would be happy to provide further clarification on any remaining points during the author-reviewer discussion period.

Q1: Varying levels of demixing quality

We follow the reviewer's suggestion to conduct an analysis of our model's performance under varying levels of demixing quality. We create three quality-based subsets (Level 1 being the highest quality, Level 3 the lowest), with each subset containing 20 samples. The quality level for each sample is determined by a consensus rating from 5 human volunteers. We also evaluate a "Random" subset, which is randomly sampled from the entire test set for baseline comparison.

As the results below show, our model is robust to the quality of the demixed audio. Its performance on metrics like FVD and Text-C shows only minor fluctuations across the quality levels, while audio-visual correlation metrics (Audio-C and Sync-C) are even slightly better on lower-quality inputs. This may suggest that our model effectively focuses on audio cues that persist even when the demixed audio is not perfectly clean. Notably, with any-level quality inputs, our model remains superior to all baselines evaluated on the Random subset. Therefore, our multi-stream approach provides robust gains overall.

Q2: Additional datasets and backbone

See Reviewer 453Q-W1.

L1: Resolution and duration

In fact, the resolution of our MTV framework ($480 \times 720$) is the highest among all comparison methods. For instance, MM-Diffusion is limited to a $256 \times 256$ resolution, while TempoTokens and Xing et al. can only generate $384 \times 384$ videos. Additionally, as demonstrated in our previous response (W2), we have adapted our MSTControlNet to the Wan14B backbone, enabling our framework to generate high-resolution $720 \times 1280$ videos.

Our framework is fully capable of generating long videos. Numerous examples of long videos are already present in our submission. For instance:

• In the main paper, the video in the 2nd row of Fig. 1 is 132 frames, and the video in the 2nd row of Fig. 5 is 146 frames.

• The supplementary material contains even longer examples, including videos of 110, 116, 156, and up to 234 frames.

L2: Key design choices

The architecture of our stream interaction block is inspired by the dual-stream DiT block of the HunyuanVideo, a proven effective architecture. Beyond this, we already provide extensive ablation studies in the paper:

• We analyze the architecture of our MTSControlNet in Sec. 5.2 (Lines 250-255).

• We study the optimal number of blocks in Appendix A.2.

We further conduct an additional ablation study ("Linear"), where we replace our interval interaction block with a stack of linear layers (following Hallo3). As the results shown below, our proposed interval interaction block achieves superior performance on this task.

We also conduct an additional ablation study ("Mix") to validate our training strategy, where we train our MTV framework on the full DEMIX dataset, without the multi-stage training strategy.

As the results below show, this ablation significantly degrades lip synchronization performance. We believe this is because lip-sync is a highly sensitive task that requires precisely aligning audio with a small visual region (i.e., the mouth). Without first learning this fundamental alignment on a simpler subset of facial data, the model struggles to learn this correspondence when faced with the complexity of large motions and multiple people.

Dear Reviewer, as the discussion period is ending soon, we would be grateful for any feedback on our rebuttal and are happy to answer any further questions.

Dear Reviewer 453Q,

With the discussion period ending soon, we would be grateful for any final thoughts on our rebuttal. We are happy to clarify any remaining points.

Thank you for your time and consideration.

We sincerely thank the reviewer for their positive feedback and for raising their score.

We will incorporate all of these additional experiments and clarifications into the final version.

We thank the reviewers for their insightful and constructive feedback.

In accordance with the rebuttal policy, we are unable to include new qualitative results (e.g., generated videos) or external links. We assure the reviewers that, should the paper be accepted, the camera-ready version will be thoroughly updated to include all additional experiments and qualitative results discussed in this rebuttal.

We now address the specific points raised by the reviewers.

W1: Global style control

We agree that for long-sequence videos, mood distribution may not be uniform. However, we would like to clarify that our approach follows the common and well-justified practices of both affective computing and video generation.

• Affective computing

Our work focuses on generating video clips, typically under 10 seconds. In such durations, assuming a consistent visual mood is a standard approach. For example, the LIRIS-ACCEDE [1] dataset (a general video dataset for affective content analysis) consists of video clips between 8-12 seconds, each annotated with a global Arousal-Valence value. The authors explicitly justify this choice, stating that using these durations "greatly minimizes the probability that annotations are a weighted average of consecutive emotions".

Furthermore, this principle of applying a global label is also adopted in the highly influential DEAP dataset [2] for music videos, where even one-minute clips are assigned a single set of ratings (Valence, Arousal, Dominance, etc.) for the entire duration.

• Video generation

In our MTV framework, music features serve as a representation of the overall aesthetic, which is analogous to a global style for the generated video. Enforcing such global style consistency is a central challenge and a standard practice in recent video editing models to ensure temporal coherence.

For example, FateZero [3] "to warp the middle frame for attribute and style editing" to guide the entire sequence, while Render-A-Video [4] applies "cross-frame attention to all sampling steps for global style consistency". Notably, both methods utilize a single text description as a global style command for all frames, which is conceptually similar to how our framework processes a single music feature stream as a global guide.

[1] Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing, 2011.

[2] LIRIS-ACCEDE: A video database for affective content analysis. IEEE Transactions on Affective Computing, 2015.

[3] Fatezero: Fusing attentions for zero-shot text-based video editing. ICCV, 2023.

[4] Rerender a video: Zero-shot text-guided video-to-video translation. SIGGRAPH Asia, 2023.

W2: Cross-attention

The core novelty of our multi-stream temporal ControlNet lies in demixing the audio into speech, effects, and music tracks. This allows user-provided audio to precisely control the generated video's lip motion, event timing, and visual mood, respectively. To achieve this, we design an interval stream for specific feature synchronization and a holistic stream for overall aesthetic presentation. As a result, our MTV framework achieves significantly better performance than the comparison methods (Tab. 2)

Additionally, the injection module is more sophisticated:

• The interval stream employs interval interaction blocks to model the interplay of speech and effects within each time interval using self-attention, which maintains coherence with inferred semantic features. These interacted features are then injected via the interval feature injection mechanism. Unlike standard cross-attention that applies a condition to all frames, our approach injects each audio feature only into its corresponding video time interval, enabling precise temporal synchronization.

• The holistic stream uses a holistic context encoder to extract music features as video style embeddings. To ensure visual consistency, an average pooling is applied to merge all intervals. Subsequently, we estimate scale and shift factors from these holistic music features and inject them into all intervals via the holistic style injection formula (Eq. 3). Notably, no cross-attention is applied in this stream.

W3: Visual mood

We have adopted the Audio Consistency (Audio-C) metric, which uses ImageBind embeddings to calculate the cosine similarity between video and audio features. A higher alignment score indicates the generated video features are more semantically consistent with the audio. We further consider the FVD as an indirect indicator, which measures the distribution gap between generated videos and ground truth videos. A significant mismatched mood would result in a poorer FVD score.

We further conduct an additional ablation study, where we remove the music track. As shown in the table below, this leads to a degradation in both FVD and Audio-C scores, demonstrating the important role of the demixed music track.

If the reviewer finds any appropriate metric, we would be glad to add its quantitative results.

Q1: Alternative audio encoders

In our empirical experiments, we find that using different audio encoders produces similar performance. This finding is consistent with previous Visual-Animation [6] (see its Appendix 6.7).

We conduct an additional ablation experiment where we replace Wav2Vec with Beats for both the effects and music tracks. As shown in the table below, this ablation achieves comparable performance, suggesting that our current choice of Wav2Vec is a robust and effective one for this task.

[6] Audio-Synchronized Visual Animation. ECCV, 2024.

Q2: Mixed speech and sound effects

The speech and effect tracks are not mixed into a single input. Specifically, we apply a self-attention mechanism across the features of both the speech and effect tracks. This allows them to model their interplay at each time interval. However, they remain as two separate tracks after this step. Subsequently, these two tracks are injected into the video features independently using two separate masked cross-attention modules.

We recognize that if two audio control signals are applied without any interaction, their independent features could create conflicting instructions and interfere with sensitive tasks like lip synchronization. Therefore, our goal is to construct a coherent control representation while retaining the distinct nature of each track. This is inspired by real-world scenarios where visual events are driven by a combination of sounds (e.g., a person speaking while their footsteps are heard). By modeling the interplay, our network can learn to produce a unified semantic feature that guides the larger video context (e.g., both lip movements and body posture).

We further conduct an additional ablation study, where we remove the shared self-attention module that models the interplay and instead apply self-attention to each track individually. As shown in the table below, this ablation leads to a degradation in both lip-sync scores. Therefore, our interplay self-attention module is the solution to potential interference, instead of a source of it.

Q3: Multiple speakers

In our specific implementation, we introduce an additional speech track for a second speaker during the multi-character training stage. The weights of the second speech track are initialized with those of the first speech track, which is pre-trained on a single-character subset. To handle multiple speakers, we perform speaker diarization using Scribe to distinguish active speaker segments during data processing (Line 124-126). After that, we crop the segments for the first and second speakers and feed the cropped audio into two individual speech tracks. This allows our MTV framework to handle speaking overlaps between multiple speakers.

To map a speech track to its corresponding face region, we create a template to structure text descriptions so that the correspondence is adaptively learned (Line 203-210). These text descriptions explicitly state which speaker is active and their turn-taking order.

As our model uses audio to control the video generation, the generated video is inherently aligned with the provided audio. Therefore, temporal misalignment does not occur.

Q4: Three components

We demix audio into three components: speech, sound effects, and music. This methodology aligns with the standards of the CDX'23 challenge, and we adopt its leading models. In practice, this framework separates human voice into the speech track and instrumental accompaniment into the music track. All remaining audio components, including crucial modalities like ambient sounds and reverb, are thereby categorized as the effects track.

To evaluate the information integrity of this process, we sum the three demixed audio tracks and calculate the L1 distance against the original audio. The table below presents the mean L1 distance, which demonstrates the high fidelity and information completeness of our separation.

Dear Reviewer, as the discussion period is ending soon, we would be grateful for any feedback on our rebuttal and are happy to answer any further questions.

[W3]-1. Audio-C

We agree that the drop in the Audio-C score for the "W/o music track" ablation is not significant. We believe this is expected and still demonstrates the effectiveness of our multi-stream design.

The Audio-C metric primarily measures high-level semantic correspondence. When the music track is removed, the video's aesthetic mood may be less aligned, but the speech and effects tracks still ensure the primary events and lip movements are correctly synchronized . As the core semantic content remains correct, this maintains a reasonable Audio-C score.

The resulting video is not "wrong", but rather adopts a more average presentation style. As a result, the score does not degrade significantly.

[W3]-2. Emotional alignment.

The affine transformation (holistic style injection) is intentionally not responsible for generating the dynamic facial expressions. This task is handled by our interval stream, which drives interval-wise facial muscle movements based on the speech track. Instead, the goal of the music track is to apply a consistent visual mood (e.g., color tone) globally across the frames.

For instance, the speech track can drive a talking face with a smile, while the music track determines whether that smile appears in a warm context or gloomy one. This design allows our framework to align the global aesthetic presentation with the specific facial expression without conflict.

[W3]-3. Sync-C/D

We have verified that our Sync-C/D scores are calculated correctly. The perceived difference in scores stems from the significantly higher difficulty of our audio-sync cinematic video generation task compared to the talking head task in the cited papers.

To provide a proper baseline, we present the scores on our test set below, including the Ground Truth (GT):

As shown, the GT score on our complex DEMIX dataset is already much lower than the 5.x-6.x scores reported on clean talking-head datasets. This is because:

1. Dataset complexity. Unlike standard talking-head datasets (e.g., VoxCeleb2) that use stable and front-facing heads, our DEMIX dataset contains "in-the-wild" cinematic videos. These clips often contain small or blurry face regions (e.g., Fig. 1, 1st row), and non-speaking clips where audio-visual speech correlation is naturally low (e.g., Fig. 1, 4th row, which has a Sync-C of only 1.26). These real-world challenges affect the SyncNet evaluation model itself (which is used to calculate Sync-C/D scores), lowering the scores even for ground truth videos.

2. Task difficulty. Methods like SyncTalk++ and SyncTalkFace are designed to animate a provided reference image or video, focusing solely on the head. In contrast, our framework generates the entire scene from only a text description, including the background, body, and face, without any visual reference. Therefore, a direct comparison of these scores can be misleading, as our framework solves a much broader and more challenging problem that existing talking-head methods do not support.

Thank you for thoughtful feedback on our work. We appreciate the opportunity to provide further clarification. We address each of your remaining points below and welcome any further discussion.

[W1 & W2]-1. Cross attention

We thanks for the reviewer's suggestion. We have already begun running this ablation study, where music features are fused via cross-attention instead of our proposed holistic style injection. The experiment is currently in progress, and our preliminary observations support our original design choice.

Interestingly, thanks to the powerful T2V backbone, we are not observing significant flickering in the generated video. The backbone itself is robust enough to maintain generation stability. Instead, the issue we have identified is that injecting music features at each timestep leads to a visual mood that feels disconnected from the overall feeling of the music. We believe this is because this per-interval approach lacks a global understanding of the audio's emotional tone. Instead, our original design using an average pooling layer effectively captures and provides this overall aesthetic presentation.

We expect to post the final quantitative results from this ablation study here as a follow-up comment within the next 28 hours. We are sharing these preliminary findings now to ensure there is sufficient time to address any other questions you may have before the discussion period concludes.

[W1 & W2]-2. Expressive generation

To clarify, since the audio is a user-provided input, our model's goal is to generate a video that is expressively synchronized with it, not to generate the audio itself.

Our framework is explicitly designed to "prevent music features from hindering the video generation" by decoupling the control streams (see Fig. 2 (b)). Specifically, the music track is processed in the holistic stream to establish the overall aesthetic presentation (i.e., the scene's mood), while the speech and effects are handled by the interval stream for interval-wise specific feature synchronization (i.e., lip motion and visual events). These two streams are not in opposition. For example, a character can be talking with an expressive smile (driven by the speech track) within a scene that has a globally gloomy mood (driven by the music track).

Additionally, the foundational generation quality relies heavily on the pretrained T2V backbone (e.g., CogVideoX), as evidenced by the visual quality improvement when we integrated with Wan14B. The role of our MST-ControlNet is to ensure that the strong expressiveness of the backbone is accurately aligned with the audio.

We will include these results in the final version.

Following up on our previous response, we have now completed the experiment regarding the reviewer's insightful suggestion of injecting music features using cross-attention. We present the quantitative results below:

These results lead to the following conclusions:

1. Video temporal consistency (Temp-C) shows only minor fluctuations, demonstrating that the T2V backbone effectively maintains generation stability regardless of the injection method.

2. Lip synchronization metrics (Sync-C/D) also show only minor fluctuations, providing the evidence that the music track is effectively decoupled from the speech and effects tracks, which independently control audio synchronization.

3. Overall video quality (FVD) shows a slight degradation, which suggests the visual mood generated by the cross-attention method deviates from the desired cinematic aesthetic.

4. The general audio-video alignment (Audio-C) also drops, indicating that the cross-attention approach provides less accurate audio-video correspondence overall.

In conclusion, this ablation study confirms that while cross-attention is a valid mechanism, which may not be the optimal approach to inject music features.

Additionally, we believe that our approach does not damage the emotional expression of the video. Compressing the music feature via average pooling provides a global understanding of the audio's emotional tone, which is ideal for establishing the overall aesthetic of a short video clip. As we detailed previously, this global visual mood does not interfere with the facial movements. These interval-wise facial expression are driven independently by the speech track through our interval stream. This design ensures a clear decoupling between global visual mood and specific facial movements.

We thank the reviewers for their insightful and constructive feedback.

In accordance with the rebuttal policy, we are unable to include new qualitative results (e.g., generated videos) or external links. We assure the reviewers that, should the paper be accepted, the camera-ready version will be thoroughly updated to include:

1. All additional experiments and qualitative results are discussed in this rebuttal.

2. A public link to an open-source demo showcasing our MTV framework as a practical video generation tool.

We now address the specific points raised by the reviewers.

Weakness

W1-1: Unclear source separation

We agree that audio demixing tools are not always perfect. However, we would like to clarify that all of our reported results are generated from these imperfectly separated audio tracks, not from idealized clean inputs. As shown in Tab. 2 and Fig. 3, our framework achieves state-of-the-art performance across six standard metrics, directly demonstrating its inherent robustness in real-world scenarios. This stems from our demixing filtering strategy (Lines 117-122). We explicitly filter out extremely poor-quality samples during training, which enables our model to learn precise audio-visual synchronization, while the remaining unclear separations serve as a form of data augmentation that enhances robustness.

We additionally create three quality-based subsets (Level 1 being the highest quality, Level 3 the lowest), with each subset containing 20 samples. The quality level for each sample is determined by a consensus rating from 5 human volunteers. We also evaluate a "Random" subset, which is randomly sampled from the entire test set for baseline comparison.

As the results below show, our model is robust to the quality of the demixed audio. Its performance on metrics like FVD and Text-C shows only minor fluctuations across the quality levels, while audio-visual correlation metrics (Audio-C and Sync-C) are even slightly better on lower-quality input. This may suggest that our model effectively focuses on audio cues that persist even when the demixed audio is not perfectly clean. Notably, with any-level quality inputs, our model remains superior to all baselines evaluated on the Random subset. Therefore, our multi-stream approach provides robust gains overall.

Finally, to completely bypass this separation challenge, we further developed a demo. It uses Qwen3 to interpret user prompts into audio descriptions, synthesizes them into perfectly clean audio with ElevenLabs, and then generates a synchronized video with our MTV framework. This presents a complete text-to-video pipeline to avoid potential source separation errors entirely.

W1-2: Different pre-existing models

We create two new baselines by swapping key pre-trained components to evaluate how much of the performance gain comes from our proposed method.

1. Wan14B with umT5. Since Wan14B shares a similar DiT-based structure with CogVideoX, we can integrate our proposed MTSControlNet into it without architectural changes. Specifically, our interval feature injection and holistic style injection modules are added after each text cross-attention layer. The quantitative results below show this baseline achieves better performance on video and text metrics (FVD, Temp-C, Text-C) due to the stronger capabilities of the Wan14B and umT5 models, while achieving comparable performance on all audio-related metrics (Audio-C, Sync-C, and Sync-D). This demonstrates that our MTSControlNet maintains its high performance on audio alignment and lip synchronization, independent of the specific T2V backbone.

2. Beats. Since Wav2Vec is a common setting for speech encoding (e.g., Hallo3), this baseline only replaces Wav2Vec with Beats for both the effects and music tracks. As shown in the table below, this baseline achieves overall comparable performance, suggesting that our current choice of Wav2Vec is a robust and effective one for this task. This finding is consistent with previous work in Visual-Animation [1] (see its Appendix 6.7).

Notably, pre-trained T2V generators are typically tightly coupled with their specific text encoders. Therefore, replacing only the video generator or its text encoder separately would require fine-tuning the entire backbone, which requires prohibitive computational resources and is not feasible within the rebuttal period.

[1] Audio-Synchronized Visual Animation. ECCV, 2024.

W1-3: Reproducibility

We will release the source code and pre-trained models upon acceptance. Should the reviewer have questions about any specific implementation details during the rebuttal period, we are glad to post the corresponding code directly in the rebuttal response.

W2: Complicated scenario

The primary purpose of our text template is not to describe every detail of the scene, but to effectively specify the active speakers, their appearance, and the scene in which they are present. Based on our empirical observation that most one-shot scenarios have fewer than three active speakers, we initially set the upper bound of active speakers to two.

While this number can be increased to three or more with fine-tuning, the true bottleneck for complex scenes lies in the token limit of the text encoder (i.e., T5's 512-token limit). This inherent constraint of the T2V backbone prevents users from providing long descriptions for every single person, regardless of whether a structured template is used. As a result, for scenarios with complex backgrounds (e.g., a crowd of people), these details can be concisely described and included in the "scene description" portion of our template (e.g., in a bustling city square full of people). We will include such cases in the final version.

We further conduct an additional ablation study by fine-tuning our model using only free-form text descriptions instead of our structured template. As the results below show, the video quality is degraded, demonstrating the effectiveness of our text template design.

W3: Comparison fairness

In our initial submission, we evaluate TempoTokens and Xing et al. using their publicly released weights. To make a fair comparison, we fine-tune all baseline methods (MM-Diffusion, TempoTokens, and Xing et al.) on our DEMIX dataset using their official training schedules, and re-evaluate them on our DEMIX dataset. The updated results are presented below. Our proposed MTV framework still maintains a significant lead, although TempoTokens and Xing et al. show notable improvement in lip synchronization after fine-tuning.

We further follow TempoTokens to conduct evaluations on both the Landscape and AudioSet-Drum datasets. As shown in the tables below, our MTV framework still achieves significantly better performance. Since neither dataset includes human talking, the lip synchronization metrics (Sync-C and Sync-D) are not applicable for these evaluations.

Comparison on Landscape datasets

Comparison on AudioSet-Drum dataset

Q1: Retrained baselines

See W3 please.

Q2: Finetuning MM-Diffusion

For fine-tuning MM-Diffusion, we strictly follow the official training schedule: We first train the coupled U-Net, and subsequently train its super-resolution model. Both stages are performed on our full DEMIX dataset.

We do not apply our training schedule to comparison methods. As this is the contribution of our MTV framework, applying it to other methods would prevent a fair evaluation of our improvement.

Q3: Imperft audio demixing

See W1-1 please.

Q4: Complicated scenarios

See W2 please.

Dear Reviewer, as the discussion period is ending soon, we would be grateful for any feedback on our rebuttal and are happy to answer any further questions.