Response to Reviewer tzZJ

We sincerely thank the reviewer for the thorough technical review and valuable feedback. Your observations have significantly helped us improve both the technical accuracy and clarity of our manuscript. We address each concern systematically below:

---

Q1: "Flow Matching" terminology confusion

Thank you for this crucial observation. We found that there was indeed a technical error in our manuscript's mathematical presentation.

You are right that our method description created confusion between Flow Matching and diffusion terminology. While we correctly stated using "Flow Matching as our training objective" (Line 127), Equation (2) incorrectly presented diffusion loss notation instead of proper CFM formulation. This created unnecessary confusion about our approach.

Our method is indeed a standard Conditional Flow Matching (CFM) approach following Lipman et al.[1]. The DiT model predicts velocity field $v_θ(x_t, t)$ to regress conditional velocity (not noise $ε$), with training objective being standard CFM loss for velocity field learning, and generation process through integration of learned velocity field: $dx/dt = v_θ(x_t, t)$.

The proper CFM loss function for Equation (2) should be:

$L_{CFM}(\theta) = E_{t, x_t , V_{cd}, A_{ab}, V_{ab}}[\|v_\theta(x_t, V_{cd}, A_{ab}, t) - u_t(x_t|V_{ab})\|_2^2]$

where $v_θ(x_t, V_{cd}, A_{ab}, t)$ is the learned velocity field predicted by DiT with conditioning on input video $V_{cd}$ and target audio $A_{ab}$, $u_t(x_t|V_{ab})$ is the conditional velocity field typically defined as $u_t(x_t|V_{ab}) = (V_{ab} - x_t)/(1-t)$ for the linear interpolation path $x_t = (1-t)x_0 + tV_{ab}$, and this differs from diffusion models that predict noise $ε$.

In the revision, we will correct Equation (2) to proper CFM mathematical formulation, clarify throughout that DiT predicts velocity fields rather than noise, and ensure consistent Flow Matching terminology across the manuscript.

[1] Lipman Y, Chen R T Q, Ben-Hamu H, et al. Flow matching for generative modeling[J]. arXiv preprint arXiv:2210.02747, 2022.

---

Q2: Limited discussion of lip shape leakage in main paper

Thank you for your feedback. We appreciate the reviewer’s careful attention to the issue of lip shape leakage. As described in detail in Supplementary Section 5 (lines 148–170), we analyze the phenomenon, its typical causes in mask-based approaches, and our proposed solutions in OmniSync. We agree, however, that this important topic should be explicitly addressed in the main paper as well. In our revision, we will add a dedicated paragraph to the main text to clearly define lip shape leakage, explain its prevalence in mask-based methods, and describe how our mask-free training paradigm and dynamic spatiotemporal CFG jointly alleviate this problem.

We appreciate your suggestion to quantitatively assess lip shape leakage. Although no standard open-source implementation is available for KeySync's LipLeak metric, we have followed the methodology described in their paper and utilized the LipScore codebase to compute LipLeak scores on our entire evaluation set. The results are summarized below:

Compared to most previous works, our method significantly reduces lip shape leakage. Even compared to KeySync, which is specifically designed to minimize lip leaks, our scores are comparable.

---

Q3: Face detection limitations for stylized content

Thank you for your valuable comment. We agree that for stylized but anthropomorphic subjects—such as statues and paintings resembling human faces—existing face detection and alignment techniques can still function reliably, which is also reflected in KeySync's demos. However, the true limitation arises in scenarios involving non-human or highly non-anthropomorphic characters (e.g., animal faces, fantasy creatures), where face detectors often fail to localize landmarks accurately or at all.

As demonstrated in Figure 3 (second row, "wolf girl") of our paper and further visualized in our supplementary videos, existing methods relying on face alignment frequently struggle or fail in such challenging cases, resulting in poor lip sync and severe artifacts. In contrast, our mask-free approach does not depend on explicit facial keypoints, enabling robust lip synchronization even when standard detectors fail. Thus, while KeySync and similar approaches may work for "easy" stylized humans, they cannot generalize to the full diversity of AI-generated content—an important advantage highlighted by our method.

---

Q4: Dataset specification and pseudo-paired data extraction

Thank you for your valuable feedback. We apologize for the confusion regarding the dataset selection and the construction of pseudo-paired data. To clarify:

As described in lines 234–236, for early timesteps, we use the MEAD dataset, which consists of controlled lab recordings with minimal pose variation—ideal for stabilizing structural learning. For the middle and late timesteps, we utilize a diverse 400-hour YouTube dataset to ensure generalization to varied real-world scenarios.

By "pseudo-paired data," we refer to pairs of video segments from MEAD where the head pose and identity remain nearly identical and only the lip movement differs, enabling us to simulate paired conditions without requiring perfect alignment or explicit pair annotations.

This timestep-dependent sampling strategy is a key innovation of our method. For full details on the dataset choices and the practical process for constructing pseudo-paired data, please refer to Supplementary Section 54–110. We will revise the main text to explicitly highlight these points for greater clarity.

---

Q5: Missing citations for temporal and spatial CFG approaches

We will add comprehensive citations acknowledging prior work, including temporal CFG approaches from Video Stable Diffusion[1] and "Analysis of Classifier-Free Guidance Weight Schedulers"[2], as well as spatial mouth attention mechanisms from Diffused Heads[3], KeyFace[4] and KeySync[5].

[1] Blattmann A, Dockhorn T, Kulal S, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets[J]. arXiv preprint arXiv:2311.15127, 2023.

[2] Wang X, Dufour N, Andreou N, et al. Analysis of classifier-free guidance weight schedulers[J]. arXiv preprint arXiv:2404.13040, 2024.

[3] Stypułkowski M, Vougioukas K, He S, et al. Diffused heads: Diffusion models beat gans on talking-face generation[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024: 5091-5100.

[4] Bigata A, Stypułkowski M, Mira R, et al. Keyface: Expressive audio-driven facial animation for long sequences via keyframe interpolation[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5477-5488.

[5] Bigata A, Mira R, Bounareli S, et al. Keysync: A robust approach for leakage-free lip synchronization in high resolution[J]. arXiv preprint arXiv:2505.00497, 2025.

---

Q6: Enhanced evaluation metrics and KeySync comparison

Thank you for your valuable suggestions. Due to space limitations in the main paper, we were unable to report LSE-D previously. As requested, we now provide LSE-D results for all compared methods below. Additionally, following your advice on evaluation metrics, we have also computed LipScore using the official KeySync implementation. As shown in the table below, our method not only achieves competitive LSE-D (7.33), but also leads on the LipScore metric (0.5668), outperforming KeySync with a relative improvement of 15.7%. This further demonstrates the strong perceptual quality and accurate synchronization achieved by OmniSync.

---

Q7: Text prompt analysis and exploration

Thank you for your valuable feedback. We appreciate your interest in the impact and limitations of text prompts in our framework. In Section 6 of the supplementary material, we provide qualitative analysis and ablation studies demonstrating how video description prompts affect lip clarity and movement amplitude. Specifically, our results show that incorporating descriptive prompts such as "clear facial and tooth movements" leads to more expressive and visually distinct lip synchronization, with enhanced dental visibility and mouth articulation.

---

Q8: Figure 2 notation confusion

Thank you for your insightful comments. You are correct: in Figure 2, the "C" should more appropriately be a "+" to indicate addition, not concatenation. We apologize for this oversight and will revise the figure to use the correct notation in the camera-ready version.

Regarding the "Flow Matching Noise Generator," it indeed adds noise to the original frames, but unlike standard random noise addition, the process is controlled based on the flow-matching principle.

---

Q9: Dataset and code release plans

We are actively working to release our 400-hour dataset with extraction scripts, subject to institutional data policy approval. We commit to providing comprehensive technical specifications including DiT backbone architecture, training configuration, and evaluation mehtod to ensure reproducibility.

Response to Reviewer X2Zu

We sincerely thank the reviewer for the evaluation and questions. We address each concern systematically below:

---

Q1: Threshold selection and theoretical understanding

Thank you for your questions regarding the theoretical understanding and parameter sensitivity, especially concerning the selection of $t_{threshold}$ in our timestep-dependent sampling strategy.

We agree that the timestep threshold is crucial for balancing the use of pseudo-paired and diverse data. In practice, $t_{threshold}$ represents the diffusion timestep where the transition occurs from structure/pseudo-paired to diverse/speech-driven sampling. Early diffusion steps predominantly control global structure (pose, identity), while middle and late steps drive finer details (such as lip shape and texture). We provide detailed explanation of this mechanism in the supplementary materials (Sec. 2 Training Procedure Details), noting that excessively high thresholds cause misalignments (structure not adapted), while overly low values lead to lip shape leakage.

Parameter Sensitivity: Our experiments indicate that performance is relatively robust within a reasonable interval around the chosen threshold (850 in our setting, out of 1000 steps), with ±5% variation not significantly affecting results. However, extreme values did degrade performance substantially.

Generalization to Stylized Content: Due to our method's training on real datasets and its strong generalization capability, it maintains excellent performance on stylized content. However, during training, we still set the optimal threshold based on realistic content characteristics to ensure robust learning foundations.

---

Q2: Computational efficiency analysis

We appreciate your concern about the computational cost and efficiency of OmniSync compared to prior GAN-based methods, such as Wav2Lip and VideoReTalking. It is true that diffusion-based methods generally require multiple iterative denoising steps, which can impact inference speed.

Optimization Strategy: To address this challenge, our approach incorporates a progressive noise initialization mechanism: rather than starting the diffusion process from pure noise, we initialize from the input frame at an intermediate noise level, enabling us to reduce the required denoising steps during inference to just 30-40 steps.

Performance Benchmarks: We tested our method's inference speed on an A100 GPU, where generating a 5-second video requires 3.5 minutes. Through implementing a series of acceleration techniques including TeaCache and sequence parallelism, and benefiting from our flow-based noise initialization that eliminates the need for complete 50-step denoising, we achieve excellent results with reduced computational overhead. After these optimizations, generating a 5-second video requires only 1 minute.

This represents a significant improvement in practical deployment efficiency while maintaining our quality and generalization capabilities.

---

Q3: Training data requirements and pseudo-paired data strategy

We agree that a key aspect of our method is leveraging pseudo-paired data (i.e., with minimal pose variation) during early diffusion timesteps to stabilize learning of head pose and identity.

Data Requirements: We utilized approximately 40 hours of MEAD data for pseudo-paired training. We experimented with using in-the-wild data for training but found that the model struggled to converge effectively, confirming that controlled data with strong pose consistency is necessary for stable learning of structural features.

Generalization Capability: Despite this controlled training requirement, our model demonstrates strong generalization and robustness, proving that our pseudo-paired data strategy is well-suited to the task requirements. The model successfully handles diverse real-world scenarios and stylized content that were not present in the training data.

Future Enhancement: Your suggestion is excellent—we can leverage our current trained model to generate additional pseudo-paired data for incorporation into future training iterations. This self-supervised data augmentation approach could further enhance our model's performance and robustness across even more diverse scenarios.

Thank you for this feedback, which opens promising directions for future improvements.

Response to Reviewer zvtB

We sincerely thank the reviewer for the positive evaluation and insightful questions. We address each concern systematically below:

---

Q1: Inference speed and acceleration strategies

Thank you for the question and suggestion. We acknowledge the importance of reporting inference speed for practical deployment considerations.

Current Performance: On an NVIDIA A100 GPU, OmniSync generates a 5-second video in approximately 3.5 minutes using the default 50 denoising steps. By incorporating lightweight acceleration techniques such as sequence parallelism and test-time caching (e.g., TeaCache), and leveraging our flow-matching-based progressive noise initialization to reduce the number of denoising steps, we can shorten inference time to approximately 1 minute for a 5-second video without noticeable quality loss.

Future Acceleration Directions: Recent advances such as consistency models and distillation-based methods present promising directions to further reduce inference latency. Integrating these techniques is feasible and forms part of our ongoing research; we expect significant acceleration with minimal quality degradation.

Thank you for highlighting this important practical aspect—we will include detailed performance benchmarks in the revised manuscript.

---

Q2: Center-biased crop fallback frequency and impact

Thank you for this valuable question regarding our fallback mechanism's behavior and performance impact.

Frequency Analysis: On the AIGC-LipSync benchmark, our center-biased crop fallback is triggered in approximately 7.29% of cases, primarily occurring with highly stylized or non-human characters where facial landmark detectors fail to provide reliable keypoints.

Performance Impact: Qualitative inspection demonstrates that this fallback mechanism does not introduce noticeable artifacts or degrade lip synchronization quality. This robustness stems from our mask-free and reference-free design philosophy, which reduces dependency on precise facial landmark localization compared to traditional mask-based approaches.

We will clarify both the trigger frequency and performance impact of this fallback mechanism in the revised paper. Thank you for this constructive suggestion.

---

Q3: Timestep-dependent sampling clarification

Thank you for raising this clarification regarding our training data sampling strategy.

In our framework, both $V_{ab}$ and $V_{cd}$ are always sampled from the same original video to ensure consistent identity and detailed facial appearance preservation. The segments may differ in head pose or lip movements, but using entirely different videos for $V_{ab}$ and $V_{cd}$ would break identity consistency and hinder the learning of fine facial details. This same-video constraint is crucial for maintaining the temporal and identity coherence that enables our method's performance on diverse content types.

We appreciate you pointing out this potential confusion and will revise the manuscript text to make this requirement clearer and more explicit. Thank you for this valuable feedback.

Response to Reviewer h8zb

We sincerely thank the reviewer for the constructive feedback and thoughtful evaluation. We address each concern systematically below:

---

Q1: Clarity of facial dynamics preservation

Thank you for your valuable feedback. We appreciate the opportunity to clarify these points.

Unlimited-duration inference: Our mask-free framework achieves this capability because, unlike previous diffusion-based methods that rely on a single reference frame and mask-based inpainting, we use the current frame itself as reference at each timestep. This design fundamentally eliminates the accumulation of alignment or identity drift errors over time. Consequently, our model can perform frame-wise editing for videos of arbitrary duration without introducing compounding artifacts or identity inconsistencies. We have experimentally validated this on sequences up to 18,000 frames (10 minutes) without quality degradation.

Natural facial dynamics preservation: Our method leverages a progressive denoising process guided by both temporal (audio-driven) and spatial (mouth-centric) cues, ensuring that only lip regions are modified in alignment with the audio, while preserving all other facial features and dynamics present in the source video. Our ablation studies demonstrate that this approach maintains both identity and natural motion, which can be objectively observed in our results.

We acknowledge the importance of precise language and objective presentation. We have carefully ensured throughout the paper that our method's strengths and limitations are presented accurately. We will refine the descriptions for enhanced clarity in the revision.

---

Q2: Implementation details and code availability

We appreciate your emphasis on reproducibility. While we will make effort to release the source code of our project, rebuttal stage restrictions currently prevent us from providing additional links or an anonymized repository.

To ensure reproducibility, we provide comprehensive implementation details:

Model architecture: Our system adopts a DiT-based latent video diffusion framework with 12 transformer blocks (hidden dimension: 1024, MLP dimension: 4096, 16 heads, RMSNorm, learned 3D positional embeddings), and uses a 3D VAE encoder/decoder (6 layers, 4 latent channels, 1/8 spatial reduction, KL=0.05) for latent representation. Audio is conditioned via a pretrained Whisper encoder (1024-dim features, segmented to frame-length, linearly projected for DiT cross-attention), and additional text conditioning is provided by T5-large (768-dim embeddings) for video descriptions.

Training configuration: Training is conducted for 80,000 steps with batch size 64 (gradient accumulation=2, effective batch=128) over 64 NVIDIA A100 GPUs (~80 hours). We use AdamW optimizer (lr=1e-5, weight decay=0.01, betas=0.9/0.999). Data consists of both the MEAD dataset (for early-timestep pseudo-paired structural supervision) and a 400-hour collection of YouTube videos (for diverse mid/late-timestep audio-lip mappings). Training uses a timestep-dependent sampling strategy ($t_{threshold}$=850, $T$=1000): for $t$>850, pseudo-paired ($V_{cd}$, $V_{ab}$) are sampled from MEAD to ensure pose/identity stability, while for $t$≤850, arbitrary ($V_{cd}$, $V_{ab}$) pairs from YouTube broaden lip shape and context diversity.

---

Q3: Gesture generation capabilities

Thank you for raising this point. Our method is specifically designed and optimized for audio-driven lip synchronization while preserving all original head pose and non-lip facial dynamics present in the input video.

The gesture and interaction-related actions observed in Figure 1 are not generated by our model; rather, they are present in the source videos and faithfully retained by our approach. We intentionally selected challenging cases featuring hand or object occlusions to demonstrate our method's robustness in complex real-world scenarios.

This design choice highlights a key advantage of our mask-free approach: maintaining natural scene dynamics even under partial occlusions, where traditional mask-based methods would fail.

---

Q4: Language precision and key results

We appreciate your suggestion regarding result presentation. We will revise the abstract and introduction in the revision to more prominently feature key quantitative results.