X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention

Thank you for your insightful comments and appreciation of the effectiveness of our method.

Re: Temporal v.s. Single-Frame Motion Extraction

Thank you for your thoughtful question. We conducted similar experiments where latent motion sequences were extracted using multiple driving frames instead of a single image. Specifically, we implemented a pipeline with a temporal motion encoder trained end-to-end using 16-frame video clips. The results of this ablation study are summarized below:

As shown in the table, the differences across all metrics are minimal. However, training with video clips significantly reduces training efficiency, as the batch size for multi-frame training is reduced by a factor of 16 compared to single-frame training. Consequently, we opted for the more efficient single-frame pipeline.

Regarding why temporal encoding yields negligible gains for portrait animation, we believe this is due to the structured nature of facial video data and the capability of our motion encoder, which learns directly and implicitly from smooth RGB image signals. Facial videos exhibit strong structural consistency and clear motion patterns, enabling fine-grained motion extraction on a frame-by-frame basis. This ensures temporal continuity without requiring multi-frame inputs. Additionally, in our self-driven training setup, the diffusion generation network accurately reconstructs the target image using a single driving frame, often disregarding motion cues from neighboring frames.

While temporal motion extraction may not provide significant benefits for our current task, we recognize its potential for future applications. We aim to explore this design further, particularly for generating dynamic temporal effects (e.g., simulating hair motion) or addressing more complex video datasets, such as full-body driving tasks.

Dear Reviewer tmTm,

We thank you for your helpful comments on our paper. We have taken your concerns into consideration and addressed them carefully. We have also conducted additional experiments to support our claims, and we have included our responses in our rebuttal. We hope that our rebuttal has convinced you. Has this response clarified our method and experiments? We kindly ask the reviewer to reassess the paper in light of our response, if it clears things up. If not, we would be happy to provide alternative explanations, or to provide clarifications on any other part of the paper. Thanks again for your time reviewing our paper.

Thank you for your constructive comments. We have provided detailed responses to your questions as follows.

Re: Definition of Zero-Shot

In our work, zero-shot refers to the model’s capability to generalize to unseen portrait identities and driving motions without requiring fine-tuning during inference. Specifically, although our model is trained exclusively on real human videos in a self-driven manner, it can effectively generalize to out-of-distribution reference images, such as stylized or non-human characters (e.g., 3D cartoon portraits), and in-the-wild facial motions, for cross-identity driving. This definition of zero-shot aligns with those employed in many prior works, including X-Portrait (SIGGRAPH 2024), GAIA (ICLR 2024), DiffPortrait3D (CVPR 2024), and Zero-1-to-3 (ICCV 2023).

Re: Definition of End-to-End Training

X-NeMo represents an end-to-end fully differentiable and trainable pipeline from video to video, without relying on predefined intermediate motion descriptors to condition the generator as in previous methods. While our full pipeline is trained in three stages for training efficiency and warm-up pretraining, it supports fully end-to-end optimization with all modules trainable.

Specifically, in the first stage of image pretraining, we optimize both the backbone UNet and the reference net, which specializes the generative network for portrait generation. In the second stage, we simultaneously optimize all network components except for the temporal modules. Here, we achieve what we refer to as end-to-end learning of the motion encoder and image generator. The temporal module is trained independently in the third stage, solely to ensure cross-frame coherence, following the standard practices of prior works (e.g. AnimateDiff, AnimateAnyone, MagicAnimate).

Our concept of end-to-end learning emphasizes optimizing the implicit motion representation directly within the generation process. This paradigm allows the motion encoder to fully exploit the motion diversity and richness inherent in the training video collections, without depending on external motion detectors. As a result, the learned motion embedding space captures fine-grained motion semantics effectively.

Re: Effectiveness and Novelty of Augmentation for ID Disentanglement

Thank you for your comments. We agree that applying color and spatial augmentations alone does not entirely remove the identity information encoded in the driving image. However, these augmentations effectively encourage the generative backbone to rely on the reference network for appearance and identity-related information.

Moreover, our motion representation is designed as a 1-D, low-dimensional latent vector, which inherently restricts the information flow of high-frequency appearance and identity features from the augmented driving images. This compact representation focuses on capturing motion information, leading to reduced training errors and better disentanglement. This has been evidenced in our quantitative (Tab 1, 2) and qualitative results (Fig 1, 3 and video demo).

While we acknowledge that data augmentation is a classical strategy, our approach uniquely leverages spatial augmentations within a diffusion-based framework for portrait reenactment. This is a key advantage of our method. In contrast, prior approaches relying heavily on spatially aligned control signals, such as with ControlNet or PoseGuider, are more sensitive to spatial augmentations, which can degrade both robustness and accuracy.

Re: Definition of App-Encoder in Fig.2

Thank you for pointing this out. As mentioned in line 356, the App-Encoder is a feature extractor network that encodes the reference image into an appearance latent embedding, $f_{\text{app}}$, which is utilized by the GAN branch to generate images. We will update the caption of Fig. 2 to include this description for improved clarity.

Re: FVD metric

Thank you for your suggestion. We have added comparisons of the FVD metric under the cross-reenactment setting with other baseline methods:

The results demonstrate that our method achieves significantly lower FVD values, indicating a high level of temporal consistency and visual fidelity in the generated videos. This also aligns with the qualitative comparisons shown in the supplementary video.

We appreciate your detailed feedback and would like to further address your concerns:

Re: Definition of End-to-End Training

Thank you for your comment. We will re-clarify the definition of "end-to-end" in the final version of the paper to avoid potential misunderstandings.

Here, our use of "end-to-end training" emphasizes two key aspects:

1. Direct optimization of the implicit motion representation within the generation process, from video to video, distinguishing our approach from prior methods that rely on external motion detectors. That being said, our method enables directly learning new facial motion distributions from RGB face videos, without reliance on explicit motion representations.

2. After the initial three-stage training process (intended as warm-up), we train the entire pipeline end-to-end, with all trainable modules optimized simultaneously.

We will ensure our revised description is more precise and reflects these points clearly in the final manuscript.

Re: Effectiveness and Novelty of Augmentation for Disentanglement

Thank you for your thoughtful feedback. We appreciate the opportunity to clarify our augmentation strategy and its role in our framework.

Regarding your concern that “color and spatial augmentations cannot isolate the identity of the reacted human. The claim ‘...' is in dispute. Those identity-agnostic information like pose and facial expression is remained in the augmented images”, we suspect there may have been some misunderstanding about how augmentations are applied in our method. As illustrated in Fig. 2 of the paper, augmentations are applied exclusively to the driving images, not the reference images. The purpose of these augmentations is to disrupt identity-related information in the driving images (e.g., skin color, facial shape, and feature distribution) while preserving identity-agnostic information such as pose and facial expression, which are subsequently extracted by the motion encoder. During training, the inconsistency between the augmented driving image and the target image encourages the generative network to rely more heavily on the reference network to extract appearance- and identity-related information from the reference image.

We acknowledge that augmentation alone is insufficient to achieve full disentanglement of identity and motion. As detailed in Sec. 3.3, augmentation is only one component of our multi-faceted strategy for disentanglement. Other key components include 1-D low-pass latent motion embedding, Dual Head Latent Supervision and Reference Feature Masking, which are critical in enabling the motion encoder to extract fine-grained, identity-agnostic motion features.

We do not position the augmentation strategy as a technical contribution in the paper. While augmentation is conceptually similar to those used in GAN-based portrait animation approaches for distanglement, our framework uniquely combines it with latent motion representations under a diffusion-based generative model, yielding significant improvements in disentanglement. The core insight behind our approach is that “generation facilitates understanding”. By leveraging the superior generative capacity of diffusion models, our method enhances the motion encoder’s ability to embed the disentangled representation of facial images. This is evident in our comparative experiments (Fig. 5 and Tab. 1), where the GAN-based PD-FGC model trained on the same dataset shows inferior performance in both identity disentanglement and expression fidelity compared to our method.

Nevertheless, we acknowledge that our combined strategies do not guarantee 100% disentanglement between identity and motion. However, our approach demonstrates state-of-the-art disentanglement for portrait animation, as evidenced by diverse qualitative evaluations and quantitative metrics, substantially outperforming prior works in this area.

Dear Reviewer qbkt:

Thanks for your valuable comments. We have carefully considered your concerns and questions, and we have made every effort to address them thoroughly in our rebuttal. Since the discussion phase is halfway around, we would like to know if our responses have adequately addressed your concerns.

We kindly ask the reviewer to reassess the paper in light of our responses, if they clear things up. If not, we would be happy to provide alternative explanations or clarifications on any part of the paper.

Once again, we thank you for your time and effort in reviewing our paper and appreciate your feedback.

Thank you for your thoughtful feedback and for recognizing the visual quality and comprehensiveness of our experiments. We provide our responses to your comments as follows.

Re: Temporal Evaluation: Spatial-Aligned v.s. Attention-Based Motion Injection

Thank you for raising this insightful question. In our submission, we have conducted an ablation study (referenced as “w/o cross-attn” in Table 2), where the motion latent is transformed into a 2D control map via an upsampling decoder and applied to the UNet through ControlNet. To further substantiate our claims, we include FVD metric for evaluating video fidelity and temporal consistency. The results are as follows:

As these results indicate, our attention-based motion injection mechanism achieves superior FVD scores, highlighting better temporal coherence and visual fidelity.

Using spatially aligned motion injection, however, leads to severe ID leakage issues, as illustrated in Fig. 3(a) of the paper. As analyzed in the paper, spatially aligned motion injection inherently learns to formulate the image-wise structural guidance, including identity-specific characteristics, before the denoising backbone. This violates our design principle that the motion control path should remain agnostic to identity-specific attributes. Such inconsistency is particularly noticeable when driving with continuous exaggerated expressions or large head rotations, further degrading temporal consistency.

Moreover, the baseline with spatially aligned motion injection does not inherently provide stronger spatial priors, as the 2D control map is still derived from the 1D motion latent. In contrast, our attention-based motion injection mechanism effectively mitigates the identity leakage and leverages fine-grained latent motion embeddings to generate smooth and coherent animation results.

Re: Superior Temporal Consistency Achieved by X-NeMo (vs. AniPortrait and X-Portrait)

Thank you for your valuable feedback and for acknowledging the temporal consistency achieved by our method. While LivePortrait is a GAN-based method, we focus here on the differences between our approach and AniPortrait and X-Portrait.

To ensure temporal consistency, our method incorporates strategies similar to those used in prior works, such as the training of temporal modules, the prompt traveling strategy during inference, and the use of an SVD VAE decoder for multi-frame decoding. However, X-NeMo achieves superior temporal consistency primarily due to the expressive capabilities of our motion latent descriptor, which is encoded directly from the original driving frames.

Our end-to-end training framework allows the motion encoder to extract fine-grained and temporally smooth motion control signals. In contrast, AniPortrait and X-Portrait rely on pretrained motion control signals, such as explicit landmarks (AniPortrait) or implicit landmarks (X-Portrait). These signals are prone to instability and jitter across frames, and are not sufficient to fully capture the complex facial expressions present in the training dataset (e.g., NerSemble). Consequently, the conditional diffusion generators in these methods tend to retain higher levels of noise-induced variability, leading to instability during the reenactment process.

Thanks for the explanation which partially address my concerns. However, the exact contribution source for the performance is still in question. First, I do not think FVD is good metric to evaluate the temporal coherence as it is a video-level metric to measure the overall video quality. What I concerns are ① the periodic "flicking" artifact (only occurs in the overlapped frames between two clips) introduced by the clip-based inference strategy adopted almost by all existing video generation methods, and ② the "stitching" artifact (i.e., not smooth and consistent enough across frames and even within an image) almost appearing in all other methods except for X-Nemo.

One possible reason of such good performance may come from the strong single-frame face reenactment performance, which can thus lead to a strong video-level performance. However, I am not agree with what the authors' claim on the contribution source of the performance. The authors mention "X-NeMo achieves superior temporal consistency primarily due to the expressive capabilities of our motion latent descriptor", and thank to their "end-to-end training framework allows the motion encoder to extract fine-grained and temporally smooth motion control signals". From the method and experimental analysis, I guess the contribution sources are the GAN head, augmentation and stepwise training strategy.

First, the verification of the effectiveness of the GAN head and augmentation can be drawn to some extent in Table 2. The GAN head and augmentation are the most significant factor influencing AED and APD, which reflects the motion accuracy including the expression distance and head pose distance.

Second, the authors' claim of the end-to-end training is in question. X-Nemo is trained in several stages, in which the first stage trains the U-Net and reference net while the second stage trains the motion encoder and motion-attention layers. I think the separation of appearance encoding and motion encoding (previous works almost learn the two in one step), as well as the unification of motion encoding and motion injection (also named as "end-to-end training" by the authors), together with augmentation and the GAN head providing another supervision to force the model to entangle the appearance and motion such clearly.

I acknowledge the contribution of this paper, but deep analysis are still required to refine this paper, especially more clear explanation about the contribution source of the strong performance. Current writing will mislead others in understanding which are the important techniques in training a good portrait animation model. I suggest to accept this paper. Of course, better version is required. If the authors promise to provide a revised version according to the discussions, I will raise my score.

Dear Reviewer 82UL:

We appreciate your valuable comments on our paper. We have carefully considered your feedback and provided detailed response to your concerns regarding temporal consistency and tongue motion. We hope that you find our rebuttal satisfactory.

As the discussion phase is still in progress, we would like to invite you to review our rebuttal and share any further comments or questions that you may have. We value your feedback and are committed to improving the quality of our paper accordingly.

If our rebuttal has successfully addressed your concerns, we kindly request you consider updating your score. Your updated evaluation would be greatly appreciated.

Thank you for your time and attention.

A qualitative comparison with clear clarification—such as selecting frames from the overlapping clip, zooming in, or employing other effective presentation techniques—could effectively demonstrate the superiority of X-NeMo in terms of temporal consistency if there is no suitable metrics for assessing temporal consistency. Of course, It is important to avoid terms like "end-to-end training", as they have established meanings that are widely recognized.

I look forward to your updated version with the promised refinements, as I believe it will provide valuable insights to the community. I am willing to increase my score to accept the strong performance and design of the method.

Thanks for your recognition of the contributions of our work. We are pleased to hear that, after our previous discussion, it seems that your concerns have been well addressed. We sincerely hope that you will raise your score to support our work. We are currently refining our work based on feedback from the last round of discussion:

1. We have updated the paper with revisions indicated with blue text:

• Revised several statements about the 'end-to-end training' in the original manuscript.
• Re-clarified the training stages.
• Added a section in the Appendix to explicitly clarify the contribution of each component to the overall model performance.

2. We are conducting an ablation study on the stepwise training process, which means we need to retrain a variant that simultaneously optimizes all parameters from scratch. However, due to the relatively small batch size, training this variant from scratch is very time-consuming. This experiment is ongoing, but given the rebuttal time constraints, we apologize that we may not be able to present the results immediately within the discussion period. We promise that we will include this ablation study in the final version of the paper.

Thank you for your constructive feedbacks and insightful comments. We are glad to see that you acknowledged the effectiveness of our method and appreciated our visual quality. We provide our detailed responses to your questions as follows.

Re: Effectiveness of Our Method under OOD Motions

Our motion encoder, trained on a diverse video set of rich and complex facial expressions, effectively captures structured spatial variations and correspondences. As demonstrated in Figures 4 and 5, as well as in our supplementary videos, our method exhibits strong generalization capabilities, handling novel facial expressions and head movements sourced from in-the-wild movie clips and expression videos.

To further address your concern, we added a new figure in the paper and included additional examples (Figure 13, first two rows) of OOD facial motions—such as extreme mouth distortion, eye-rolling, and lip puckering. Despite these exaggerated expressions being outside our training distribution, our method successfully replicates and transfers fine-grained expression details across subjects.

That said, as noted in the Limitations section (Figure 7), our current model may struggle with highly exaggerated expressions that are entirely absent from the training data. However, the scalable nature of our end-to-end framework offers a clear path for improvement by incorporating additional correlated training videos. This flexibility contrasts with many prior approaches that rely on explicit motion representations (e.g., facial landmarks or synthetic cross-identity images), where the motion extraction often fails already under extreme expressions.

Re: Cross-Identity Reenactment with Distinct Identity Features

We have demonstrated a range of qualitative examples where the source and driving characters differ significantly in facial shapes, proportions, styles, and head orientations. Notable cases include 3D/2D cartoon characters driven by real human motion (Figure 1) and examples from our supplementary video, showcasing pronounced differences in head poses (timestamps: 1:06, 1:37, 2:24) and facial features (timestamps: 2:30, 4:33, 5:45). Furthermore, Figure 5 illustrates that while prior baselines show substantial drift in identity characteristics, our method achieves the highest fidelity in preserving identity resemblance.

To further support our claims, we added a new figure in the paper and included additional visual evidence (Figure 13, last two rows: "Panda" and "Thanos"), which highlight extreme identity attribute differences. Thanks to our identity-disentangled design, our method consistently delivers high-fidelity reenactments with minimal identity leakage from the driving video.

Re: Non-Human Reenactment

Thank you for your insightful question. Our method’s independence from pre-defined intermediate representations grants us the flexibility to train directly on non-human video datasets after pre-processing. We believe that our approach is well-suited for video datasets focused on foreground object motion. In such scenarios, general object detectors like YOLO or SAM could be employed to isolate and crop the foreground objects for processing.

However, as we have not yet collected or processed such a non-human video dataset, we regret that we cannot provide empirical results to validate this approach at this time. That said, we see significant potential in extending our framework to accommodate a broader range of video categories, and we intend to explore this direction in future work.