VividFace: A Robost and High-Fidelity Video Face Swapping Framework

Thank you for your helpful comments on the paper. We have addressed them as the following:

Q1: The conditioning input is redundant. The motion frame is encoded three times, and the masked source image overlaps with the face region input. This may introduce inefficiency without clear justification.

Thank you for pointing this out. We would like to clarify the computational overhead introduced by our conditioning inputs:

1. Face Region Input:

The face region input introduces minimal computational overhead. Specifically, it only increases the number of input channels in the first convolutional layer from 12 to 13, adding an extra computational cost of approximately $2.5 \times 10^{-6}$. Therefore, subsequent model complexity and inference speed remain virtually unaffected.

2. Motion

Thank you very much for this helpful comment regarding redundancy in motion-frame encoding. To clarify, each motion frame is encoded only once, prior to the iterative denoising process. Given that our pipeline typically involves around 30 denoising steps, the extra computational overhead introduced by encoding motion frames is approximately 3%, which we consider negligible in practice. For clarity, we provide the full implementation details in the supplementary materials (infer.py, lines 480–523). Additionally, we provide simplified pseudocode below for your convenience:

# Encode motion frames once per inference
motion_features = encode_motion(motion_frames)

# Iteratively denoise latent representation (~30 steps)
for step in range(denoising_steps):
    # Update latent using motion features and additional conditioning inputs
    latents = denoise(latents, motion_features, other_conditions)

However, we fully acknowledge your valuable suggestion. In future iterations, we will explore further simplifying the conditioning inputs to assess whether the current redundancy indeed contributes positively to model performance or represents a constraint of our current architecture.

Q2: Figure 2 is confusing. The blue block is ambiguously labeled (sometimes the temporal module, sometimes latent representation z0). Missing tensor dimensions for latent representations (blue, yellow, gray blocks), making it harder to reproduce.

1. We selected similar shades of blue for different concepts (temporal modules <light blue> and latent representation <dark blue>). We will revise Figure 2 using clearly distinct colors to improve readability.

2. Regarding missing tensor dimensions, we have included complete, runnable code in the supplementary material for full reproducibility. However, we recognize explicit tensor dimensions in figures facilitate understanding, and we will add these explicitly in the final revision.

For reference, the tensor shapes are:

Yellow: $B \times T \times 3 \times H \times W$

Pink: $B \times T \times 3 \times H \times W$

Gray: $B \times T \times 3 \times H \times W$

Light Blue: $B \times T \times 1 \times H \times W$

Thank you for helping us improve the clarity of our method's presentation.

Q3: The methods works well when source/target faces are similar (e.g., Figure 1, all people have high checkbones) but struggles with large identity gaps (e.g., Figure 6). Results in Groups 1–6 either: Use overly similar identities, or Fail to preserve identity when faces differ significantly.

You raise an insightful observation regarding performance gaps with large identity differences:

• We conducted an extra quantitative experiment analyzing identity preservation under varying degrees of identity similarity. We confirm that our approach, like other face-swapping frameworks, performs best when the identities involved share similar facial structures and appearance attributes.

• To explicitly quantify this, we compared our method against other baselines across a variety of identity similarity levels. We observed that, although performance naturally degrades with increased identity differences, our method still consistently achieves state-of-the-art identity preservation compared to alternatives.

• Unfortunately, due to the rebuttal policy, we cannot upload additional visual comparisons at this stage. We will include these analyses and visualizations comprehensively in our supplementary materials upon revision.

We acknowledge that swapping faces across significantly different identities (e.g., cross-gender or large age gaps) remains challenging and a critical area for future research. We are actively working to address these challenges further.

Q4: Fair Comparison Concerns: Comparisons with SimSwap/others use different examples. Testing on identical pairs would strengthen validity.

We appreciate this important concern regarding the fairness of our comparisons:

• To clarify explicitly, all quantitative and qualitative evaluations in our submission—including Figures 6, 10–15, and Tables 1 and 2—are performed using identical source-target pairs across all methods, ensuring a fair and direct comparison of model performance.

• If your concern relates specifically to evaluating methods on source and target faces from the same identity (self-swapping), we note that such self-face swaps generally constitute trivial tasks with near-perfect results across all methods.

Q5: Have you experimented with simplifying the conditioning pipeline (e.g., reducing motion frame encodings)? Does redundancy improve results, or is it an architectural constraint?

Yes, we conducted ablation studies to explore the impact of simplifying the conditioning pipeline, including reductions in motion frame encodings and face-region inputs. Our findings indicate that:

• Motion encoding contributes positively to temporal consistency. While its removal does not drastically degrade performance, we observed a modest drop in temporal metrics (e.g., FVD32, FVD128), suggesting that this redundancy provides valuable contextual cues rather than adding unnecessary complexity.

• Face-region conditioning also plays a crucial role in maintaining visual realism. Excluding it led to slight decreases in identity preservation and expression accuracy, underscoring its importance in high-fidelity generation. These results imply that the current level of conditioning is beneficial and not merely an artifact of architectural choices. We will include comprehensive ablation details and discussions in the supplementary materials.

Q6: Is the identity preservation issue in Figure 6/Group 1–6 due to dataset bias (e.g., AIDT lacks diverse pairs) or a fundamental trade-off in the framework?

The core challenge arises from the nature of face swapping, which inherently lacks ground-truth supervision when swapping between different identities. High-quality face swapping across substantially dissimilar identities (e.g., significant differences in facial geometry, or across gender/age) remains universally challenging. While our AIDT dataset offers considerable diversity, it primarily features same-gender identity pairs to stabilize training and ensure data quality.

When swapping between highly dissimilar faces (e.g., square to round face, male to female), the swap may technically succeed, but the result can still appear perceptually unrealistic or unconvincing to human observers. This reflects a broader challenge in the field: visual plausibility may degrade even when the system technically preserves identity features.

We acknowledge that this limitation stems both from the training data’s constrained diversity and from an inherent trade-off common to most existing frameworks. Nevertheless, our evaluations clearly demonstrate that our method achieves state-of-the-art identity preservation, though we agree that further improvement is necessary and ongoing.

Q7: Could you include comparisons on the same source/target pairs as SimSwap or other methods to ensure fair evaluation?

As explained above, all evaluations in our manuscript consistently use identical source-target pairs across all methods (including SimSwap) to ensure a fair comparison. If your concern is specifically related to self-swapping cases (i.e., swapping within the same identity), we would like to clarify that such evaluations often produce trivial outcomes with near-perfect accuracy and therefore offer limited comparative insight.

Thank you for your helpful comments on the paper. We have addressed them as the following:

Q1: The authors claim they are the first video face-swapping network. However, several video face-swapping methods have been proposed recently, e.g., DynamicFace and HiFiVFS.

Thank you for pointing out the concurrent works DynamicFace and HiFiVFS. At the time of our submission, both were available only as arXiv preprints, without official publication or publicly released code. We acknowledge that our original claim could have been clearer, and we will revise the manuscript to explicitly state that our work is concurrent with these recent efforts. We will also include appropriate references and discussion of these methods in the related works section.

Q2: Unsatisfying results. The generated videos still contain several artifacts, \eg, large opening mouth, hair on the forehead.

We appreciate your observation regarding certain artifacts, such as exaggerated mouth openings or imperfections around the hair region. These are indeed challenging areas in video face synthesis. While we acknowledge the presence of such issues in some cases, our results demonstrate state-of-the-art performance compared to existing video face-swapping methods—particularly under difficult conditions involving occlusions, large pose variations, or complex motion.

One contributing factor to these artifacts is the imperfect disentanglement between identity and attribute features. In certain instances, entanglement can cause the model to misrepresent fine-grained attributes (e.g., mouth dynamics or hair texture) while trying to preserve identity, leading to visual inconsistencies.

Q3: The comparison cases are not sufficient. Most of the test cases are man-man or woman-woman swapping. Including more cases under different genders might be better to show the superiority of the proposed method.

We appreciate your concern regarding the diversity of comparison cases. To clarify, our evaluation set (VidSwapBench) already includes diverse identity combinations, and the current cases were not intentionally biased toward similar-looking pairs.

Moreover, we do include several cross-gender examples, such as the upper-right example in Figure 1, and Figures 6(b) and 6(c). These examples illustrate our model's superior generalization across genders compared to other methods.

However, we agree that cross-gender face swapping is inherently more challenging. Currently, our dataset (AIDT) consists predominantly of same-gender pairs to facilitate stable training. Based on your suggestion, we will include additional cross-gender face swapping examples in the supplementary materials to provide a clearer picture of our method's strengths and limitations in more diverse scenarios.

Based on your valuable feedback, we have conducted additional experiments specifically targeting cross-gender scenarios. In particular, we curated two new test sets:

• Table 1: Woman → Man (source image is female, target video is male)
• Table 2: Man → Woman (source image is male, target video is female)

Each set contains 1,000 distinct image–video pairs.

| Cross-Gender Swap (Woman → Man) |

| Cross-Gender Swap (Man → Woman) |

Q4: Although 3DMM can effectively preserve the motions of the target video, it suffers from incorrect estimations under large headposes. Also, estimation 3dMM is quite slow.

1. Efficiency. We acknowledge your concern about the reliability and efficiency of 3DMM estimation, particularly under extreme head poses. To clarify, we utilize the learning-based 3D reconstruction method Deep3DFace's (official repo)[1], which is notably fast and efficient compared to traditional optimization-based approaches. Specifically, this approach achieves approximately 8ms (from input image, model forward, rednering) per image on our NVIDIA A100 SXM4 (80GB) GPU at batch size = 1. With larger batch sizes (e.g., bs=32), processing speed further improves to <1ms per image, making it practical for video applications.

2. Reliability at large poses. Empirically, learning-based 3D reconstruction is more stable than 2D-only cues under large yaw/pitch because it predicts a full 3D face with semantic correspondences, enabling consistent PnP pose solving and disambiguation of depth/scale across frames. Public results from representative methods—Deep3DFaceRecon[1], 3DDFA-V2[2], and SynergyNet[3]—specifically target stability and accuracy under full-pose conditions; 3DDFA-V2 (ECCV’20) and SynergyNet (3DV’21) report strong accuracy and real-time speeds in unconstrained, large-pose settings. Nevertheless, we acknowledge that even advanced learning-based 3DMM methods can still struggle with extreme poses. We plan to explore more robust 3D reconstruction approaches or complementary pose-estimation methods to improve accuracy further in future work.

[1]: Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set, https://github.com/sicxu/Deep3DFaceRecon_pytorch

[2]: Towards Fast, Accurate and Stable 3D Dense Face Alignment.

[3]: Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

Q5: Can you generate videos with arbitrary length while maintaining temporal consistency?

Yes, our method supports generating videos of arbitrary length while maintaining good temporal consistency. To achieve this, we sequentially generate clips, utilizing the last three frames of the previously generated clip as motion context for the next clip. In practice, we have verified consistent quality in sequences of approximately one minute (about 1,000 frames). For significantly longer videos, a slight degradation might occur due to accumulated error, but restarting the process periodically mitigates this issue.

Q6: What is the inference speed of this method?

Our pipeline achieves approximately 2 frames per second (FPS) on an NVIDIA A100 GPU at a resolution of 512×512, providing high fidelity and temporal coherence. Although this is slower than GAN-based approaches (e.g., SimSwap at 256² or FSGAN at 256²), our diffusion-based framework is significantly faster than comparable diffusion-based methods such as:

Thank you for your helpful comments on the paper. We have addressed them as the following:

Q1: While the paper's focus is on generation quality, a practical limitation of diffusion models is their inference speed. The paper acknowledges this in the limitations section but does not provide a quantitative comparison of inference time against the baselines in the main experimental tables. For a video task, metrics like frames-per-second are highly relevant and would provide a more complete picture of the trade-offs involved.

Thank you for raising this important point. While our work emphasizes generation quality and robustness, we agree that inference efficiency is a crucial factor for practical deployment. Our current framework achieves approximately 2 frames per second (FPS) on an NVIDIA A100 GPU at 512×512 resolution, which ensures high fidelity and strong temporal consistency. In comparison: DiffSwap (CVPR 2023) reports ~0.028 FPS, DiffFace achieves ~0.68 FPS (3× slower than ours), REFace (WACV 2025 Oral) runs at ~1 FPS (2× slower).

While our speed is indeed lower than GAN-based methods such as SimSwap (resolution: 256²) or FSGAN (resolution: 128²), which trade quality for speed, our method significantly outpaces other diffusion-based frameworks. We will include these FPS comparisons in the revised version to provide a clearer picture of the quality-efficiency trade-off.

Q2: I notice that the eyes of the generated face are not the same as the target face, and the eye contact is not as vivid as human beings. This is a challenging problem. But humans are very sensitive to this.

We appreciate this insightful observation. Indeed, humans are highly sensitive to gaze direction and eye contact fidelity. In the face swapping formulation, eye identity (e.g., shape, iris) is preserved from the source face, while gaze direction and expression are inherited from the target video through the attribute encoder. We find that eye identity is generally maintained reasonably well from the source in the VividFace.

Although our attribute network captures high-level facial movements—including pose and coarse eye dynamics—it does not explicitly enforce fine-grained gaze alignment or vividness. This can lead to subtle deviations in pupil orientation or liveliness. Improving eye realism is a meaningful and challenging direction. In future work, we plan to explore:

1. Explicit gaze estimation and alignment losses to better match pupil direction,

2. Eye-specific perceptual or adversarial losses focused on iris and specular highlights,

3. Localized eye-region enhancement modules, including cropping-based super-resolution.

Thank you for your helpful comments on the paper. We have addressed them as the following:

Q1: Diffusion-based method limits the practical applicability

We acknowledge that diffusion models are slower during inference, which can limit real-time deployment. However, recent advancements—including latent-space sampling, optimized U-Net architectures, and fast sampling techniques (e.g., DDIM)—have significantly improved inference speed. Moreover, diffusion-based methods have already been adopted in industrial-grade tools such as PiKa and Runway, indicating their increasing practical viability.

In this work, our primary focus is to advance generation quality and robustness in challenging video face swapping scenarios, such as those involving occlusions or large pose variations. Efficiency-oriented improvements, such as model acceleration and architectural streamlining, are promising future directions. We are also actively exploring the integration of DiT-based backbones to further enhance speed without compromising quality.

Q2: It is better to separate the training and inference stages in Fig. 2

Thank you for pointing this out. We agree that separating the training and inference stages would improve clarity. While we have created a figure to explicitly illustrate the inference pipeline, we regret that we are unable to include additional images in the rebuttal due to policy constraints. We have revised Figure 2 accordingly in the manuscript.

Q3: Source and target in Lines 200–207 are inconsistent with Fig. 2

We apologize for the confusion. This was due to an error in the ordering of ‘source’ and ‘target’ terms. Thank you for catching this—both the manuscript and figure have been updated to ensure consistency.

Q4: Although robustness enhancement is adopted, why the performance in pose and expression only comparable to SOTA methods?

Thank you for the valuable question. As you rightly pointed out, most widely-used benchmarks such as FFHQ (reported in Table 2) and VidSwapBench (reported in Table 1) contain relatively few instances of extreme poses or severe occlusions. While these challenging cases are rare in these benchmarks, they are often the most disruptive in real-world applications. Our robustness strategies are explicitly designed to handle these difficult scenarios.

Although this focus may lead to only modest gains on standard quantitative benchmarks, we would like to emphasize that the improvements are much more apparent in qualitative results—particularly under extreme conditions, as illustrated in Fig. 6 and 7. We believe these results highlight the practical value of our approach beyond standard metrics, and we truly appreciate the reviewers' attention to this aspect.

Q5: Is the encoder-decoder is firstly trained on video data and secondly training on image data? Then random selection of previous video and image data is conducted for hybrid training?

Our training for the VAE follows a unified single-stage strategy, where image and video data are randomly interleaved across batches. To ensure stable gradient synchronization, each batch contains only a single modality—either images or videos. The spatial components of VidFaceVAE are initialized from a pretrained 2D SD-VAE, and the model is then jointly trained on both modalities. This enables effective spatial-temporal representation learning while preserving diversity in appearance and motion. We believe this offers a clean and efficient training paradigm, and we sincerely appreciate the reviewers for prompting this clarification. We have revised the manuscript to better reflect this design.

Q6: Performance gap between baseline and hybrid training is small

While the performance improvement may appear modest in aggregate metrics, our method demonstrates consistent gains across FVD, ID retrieval, pose, and expression accuracy (Table 3, Exp. 1), indicating stable and reliable improvements across diverse aspects. Notably, models trained solely on static images tend to suffer from inferior temporal consistency and more noticeable visual artifacts in video generation. To further support this, we evaluate our method on VBench[1]—a comprehensive and challenging video benchmark—and observe clear improvements in consistency, flicker reduction, and motion smoothness compared to both static training and existing state-of-the-art methods.

[1]: VBench: Comprehensive Benchmark Suite for Video Generative Models