AdaFace: A Versatile Face Encoder for Zero-Shot Diffusion Model Personalization

Thank you reviewer jpRS for your thorough and detailed review of our paper. We appreciate the time and effort you have invested in providing your critical feedback. While some of your comments are indeed challenging, we believe they are valuable for improving the quality and rigor of our work.

1. Multi-timestep distillation.

While it's true that the student model accumulates larger errors with more steps, the effect of aligning the output with the teacher model will reduce accumulated errors. This is because we assume the teacher model is the gold standard at any number of timesteps. Therefore, aligning the student output with any timesteps will help reduce the intermediate errors in all steps. Similar techniques have been adopted in other topics, such as optical flow estimation (Section 3.3 of [a]).

2. Dynamic model expansion.

The dynamic model expansion involves duplicating the Q (query) and V (value) projection layers (as opposed to tokens) in the attention layers. The weights of the extra copies of Q, V projections are perturbed with Gaussian noises to introduce variability. We know that each vanilla attention layer has only one Q/K/V projection. After expansion by a factor of $N$, there are $N$ Q,V projections and 1 K projection. Consequently, each input token is mapped to $N$ queries and values, but only 1 key. The number of output tokens is determined by the number of keys, and therefore, the number of output tokens does not change.

3. Relevance to video generation.

We respectfully disagree with the notion that video generation is not relevant to our topic of subject-driven image generation. As the community shifts focus from image generation to video generation, we believe that the ultimate goal of subject-driven generation is to develop general methods applicable to both images and videos.

A long-standing issue with current video generation models, such as Gen3 and Luma Dream Machine, is that subject consistency deteriorates as the video becomes longer. Dedicated subject embeddings could serve as a long-term memory to maintain subject consistency in very long videos. As a preliminary exploration, we hope our method may inspire more research works along this direction.

Due to the limitations of AnimateDiff, we are currently only able to showcase our method on 1-second videos. However, we aim to demonstrate the broad potential of our method of generating longer videos with more powerful open-source pipelines.

4. Comparison with PuLID.

We clarify that we do not claim superior performance compared to PuLID. Instead, our focus is to provide the community with a face encoder that performs comparably to existing SOTA methods, with the unique advantage of being plug-and-play with most existing pipelines, such as animate generation. Specifically, inherited from Arc2Face, AdaFace is capable of generating highly realistic images. In contrast, existing SOTA methods, PuLID included, tend to produce more artistic images, as demonstrated in Figure 7 and the attached figure PDF.

To gain an intuitive sense of how our model performs compared to others, we invite you to try our online demo hosted on Huggingface.

5. Ablation study of the proposed components.

We acknowledge the absence of some ablation studies. Due to limited response time and our small team's low computational resources, it has been unrealistic to train ablated models within this short period. We will add these ablated results to a newer version of the paper as soon as they are available.

[a] RAFT - Recurrent All-Pairs Field Transforms for Optical Flow, ECCV 2020.

Thank you for taking the time to carefully examine our claim of "seamless integration with video generation pipelines." We appreciate the opportunity to further clarify and differentiate our approach from similar techniques.

1. While methods like DreamBooth, LoRA, and more recently, MagicMe [a], can achieve subject consistency in video generation, these approaches require cumbersome subject-specific fine-tuning, which limits their practicality for long videos featuring multiple subjects. These limitations have partly driven the development of recent zero-shot subject-driven generation methods.

2. More importantly, subject-driven video generation, often referred to as visual story generation, aims to create long sequences of videos where specific subjects transition through different scenarios. For example, in StoryDiffusion [b], a video story is described where a man reads a newspaper, drives to a forest, and encounters a tiger. Achieving such complex narratives requires more than just frame-to-frame consistency; it necessitates a mechanism for maintaining subject identity and appearance across diverse and lengthy sequences.

3. Traditional video generation methods can maintain subject and background consistency across adjacent frames due to large-scale pre-training on videos, where adjacent frames are typically consistent. This training allows the model to implicitly learn and preserve such consistency. However, when extended to long sequences, these models often suffer from subject distortion or semantic drift due to the absence of dedicated subject representations. Our method, while not yet perfect, represents a step towards zero-shot dedicated subject representation learning for visual story generation, addressing these challenges more effectively. In this sense, our method is also significantly different from the layout-based method you mentioned, which primarily focuses on maintaining spatial consistency over a short window of videos, rather than ensuring subject identity over longer sequences.

For a more detailed discussion on the nuances of these techniques, we invite you to refer to the ID-Animator paper [c] (one of our baseline methods), particularly Section 2.3.

We hope this clarification helps in understanding the unique contributions of our work in the context of video generation. We respect your perspective and appreciate your feedback, which has been instrumental in refining our presentation.

[a] Magic-Me: Identity-Specific Video Customized Diffusion. arXiv:2402.09368.
[b] StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation. arXiv:2405.01434.
[c] ID-Animator: Zero-Shot Identity-Preserving Human Video Generation. arXiv:2404.15275.

Thanks reviewer jmG6 for your constructive feedback. Your insights have been incredibly valuable in improving the quality and clarity of our work.

Responses to weaknesses:

1. In terms of novelty.

   • While ELITE is the first to propose a text-space embedding method for personalization, it requires training individual global K, V projections for attention layers, which affects its compatibility with existing diffusion pipelines. In contrast, Adaface does not modify the existing diffusion pipeline, and the subject embeddings are applied similarly to ordinary text tokens without special treatment. This compatibility allows Adaface to be used with AnimateDiff for generating subject-themed videos seamlessly.
   • SuTi adopts a different approach to personalization by training millions of "expert" models in advance and mapping the input subject to these expert models for zero-shot generation. We are not aware of similar compositional distillation techniques as proposed in SuTi. However, we do note in lines 170-177 that there are methods adopting similar techniques, with an emphasis on our unique contributions.

2. Extra discussions.

   • Ablation study of the proposed components. We acknowledge the absence of some ablation studies. Due to limited response time and our small team's low computational resources, it has been unrealistic to train ablated models within this short period. We will add these ablated results to a newer version of the paper as soon as they are available.
   • Training face encoders from scratch is highly computational demanding. For example, InstantID was trained on 48x80GB NVIDIA H800 GPUs, which is inaccessible to most research teams. Arc2Face, the teacher model of AdaFace, was trained with 8*A100 GPUs for several weeks. In contrast, AdaFace was trained with 2xA6000 GPUs for less than 1 week. Moreover, it could learn from multiple teacher models to combine their advantages.

3. Editability of Multiple embeddings. The reduced editability of ELITE with multiple embeddings is likely due to overfitting. However, our face encoder is trained on hundreds of thousands of face images, thus significantly alleviates this issue. Additionally, our compositional distillation techniques further enhance editability. Our results align with recent models such as IP-Adapter, InstantID and PuLID, which adopt multiple subject embeddings while maintaining good editability.

4. Comparison with IP-Adapter-FaceID. The consensus in the AI art community is that IP-Adapter-FaceID produces far less authentic images compared to other methods like InstantID or PuLID. Therefore, we did not include a comparison with IP-Adapter-FaceID.

5. Multi-stage training. To the best of our knowledge, multi-stage training is widely adopted by many diffusion models. Given their inherent complexity and the numerous factors involved, multi-stage training helps steer these models towards achieving strong performance across various aspects.

Once again, thank you for your valuable suggestions to improve the clarity of our presentation. We will incorporate them into the updated version of the paper. In addition, we invite you to try our online demo, hosted on Huggingface, to gain an intuitive sense of how our model performs compared to others.

Thanks reviewer 6DX2 for your constructive feedback. Your comments are valuable in improving the quality and clarity of our work.

Responses to weaknesses:

1. The performance issue of the Huggingface online demo.

Thank you for trying out our demo. We would like to clarify that our first model's performance on some subjects was suboptimal due to poor settings of hyperparameters and the diffusion pipeline. We have since updated these parameters and the pipeline, which should now significantly improve performance using the same face encoder checkpoint. We invite you to try our updated online demo, which more reflects the performance of our model.

Moreover, we would like to emphasize that we do not claim new "state-of-the-art" performance in our paper. Instead, our focus is to provide the community a face encoder that performs comparably to existing SOTA methods, with the unique advantages of being highly realistic, as well as plug-and-play with most existing pipelines, such as animate generation.

2. Possible data contamination of the celebrity subjects.

We selected a few popular athletes of the Paris Olympics 2024 for extra qualitative evaluation, as presented in the figure attachment. We will add more subjects as examples into an updated version of the paper as you suggested.

3. The ID fidelity is upper bounded by the teacher model.

This is an inherent limitation of model distillation. Nevertheless, we can mitigate this limitation by doing distillation on multiple teacher models to combine their strengths, for example, on both Arc2Face and PuLID, to gain good performance on generation of both realistic and artistic images.

4. Ablation study of the proposed components.

We acknowledge the absence of some ablation studies. Due to limited response time and our small team's low computational resources, it has been unrealistic to train ablated models within this short period. We will add these ablated results to a newer version of the paper as soon as they are available.

Thank you reviewer Uvbt for your favorable evaluations. Your comments are valuable in improving the quality and clarity of our work.

1. Limited examples. We have added more examples of Paris Olypics Atheletes in the attached PDF file.
2. Ablation studies of proposed components. Due to limited response time and our small team's low computational resources, it has been unrealistic to train ablated models within this short period. We will add these ablated results to a newer version of the paper as soon as they are available.
3. Comparison with ConsistentID. On simple prompts, ConsistentID usually performs well. However, on more complicated prompts, such as "playing guitar, ocean waves, cyberpunk street market, neon signs, diverse crowd, futuristic gadgets, vibrant colors, urban style", ConsistentID totally ignores the specified style words. Please see the attached PDF file for this example. Since our evaluation prompts are all simple ones, they are unable to reflect the limitations of ConsistentID on long, complex prompts. Therefore, we do not include quantitative evaluation results of ConsistentID.