Thanks for your acknowledgment and careful reviews. Our responses are as follows:

Q1: Discuss the limitations and generalization ability of the method.

The limitations have been discussed in Section 6.1 of the paper. In terms of generalizability of PuLID, the paper has validated its effectiveness on the following grounds:

1. Section 6.5 presents the compatibility of the PuLID model on some widely-used community base models (both accelerated and non-accelerated models), exhibiting the broad applicability of our method.

2. We evaluate our method on two datasets. DivID-120, proposed in this paper, encapsulates diverse skin tones, ages, and genders, while Upsplash-50, proposed by LCM-Lookahead, comprises non-celebrity photos newly uploaded this year, hence unlikely to be a part of the training dataset. These datasets offer a wide range of input types. We will release DivID-120 to facilitate the evaluations for ID customization methods.

3. The qualitative results showcased in Figures 1, 4, 6, 9, 10, 11 demonstrate various style transition results, including 2D, 3D, anime, pixel, line art, cyberpunk, cg, fantasy, cartoon, etc., alongside a spectrum of facial edits. Most of these scenarios were not included in the training prompts, further underscoring our model's generalization abilities.

We also provide more qualitative results in Figure 1 of the Rebuttal PDF.

---

Q2: How do the proportions of the three losses affect experimental results, especially ID loss?

• Impact of diffusion loss: In practice, we keep the weight of the diffusion loss constant (i.e., 1) while adjusting the weights for ID loss and alignment loss. In extreme experiments where we exclude the diffusion loss, the detailed facial fidelity slightly degrades, particularly with regard to eyewear and hairstyles. This is because the ID loss doesn't directly constrain these details, hence the reconstruction loss from the diffusion loss helps in preserving them.

• Impact of ID Loss and Alignment Loss: The weights of these two losses determine the trade-off between ID similarity and editability. Increasing the weight of ID loss enhances ID similarity but reduces editability. Conversely, increasing the weight of alignment loss boosts editability but lowers ID similarity.

---

Q3: Is there a significant improvement in computing efficiency with fast sampling method, compared to the traditional multi-step (e.g., 30) diffusion model.

Please refer to GQ1. When compared to the traditional diffusion model's 30-step inference, using a 4-step fast sampling can make the training process twice as fast; if compared with a 50-step inference, it would be three times faster.

---

Q4: Concerns about the universality of this method.

As explained in GQ1, our method can be adapted to base models without acceleration plugins, demonstrating the method's generalizability and universality.

---

Q5: Is it possible to extend this method and replace ID with other things, such as (clothing)?

It is theoretically feasible; please refer to GQ3 for more details.

Thanks for your constructive and positive comments. Our responses are as follows:

Q1: Concerns about the internal training dataset.

1. Dataset or method – primary source of improvement? Table 2 in the main paper illustrates that a baseline, naively trained on the internal dataset, underperforms in ID fidelity and editability. Conversely, the introduction of the PuLID training paradigms delivers significant enhancement. Therefore, this substantiates that the improvement mainly comes from the method, rather than the dataset. We will revise the table analysis to emphasize this point.

2. Fairness in comparing methods trained on different datasets? Existing SOTA methods often utilize internal and private training datasets. For instance, InstantID uses a 60 million-entry dataset, compared to our 1.5 million, and PhotoMaker uses an internal ID-group dataset, more challenging to gather than a non-ID-group dataset.

3. Lower dataset quality requirements with our method. We would like to highlight that our method is less dependent on high-quality training datasets. Particularly, the ID loss and alignment loss in the Lightning T2I branch only require uncaptioned facial regions. Further, the unique feature of PuLID, which minimize the contamination to the original model behavior, ensures that the results are unbiased towards the quality of the training dataset, effectively reducing dataset quality requirements.

---

Q2: More quantitative metrics.

Thanks for your suggestion, here are the results on additional metrics:

Speed is tested on an A100 GPU in the ComfyUI environment at a resolution of 1024x1024. Our method, PuLID, and IPAdapter have similar speeds, while InstantID is slower due to its inclusion of a ControlNet. PhotoMaker is the fastest because it uses LoRA in attention layers, rather than new to_k and to_v layers like adapter-based methods, thus facilitating faster inference speeds by fusing LoRA before testing.

FGIS measures the similarity of DINO embeddings in the facial region. While our method outperforms other methods on this metric, we would also like to emphasize that the embeddings extracted by the backbone trained on face recognition are more aligned with human perception and are widely recognized.

FID measures the distance between the distributions of generated images with and without ID insertion. Our method significantly outperforms others on this metric, indicating that our method causes less disruption to the base model after ID insertion.

Face Div. measures the diversity of generated images by calculating the LPIPS scores between pairs of faces in the generated images. Our method is comparable to PhotoMaker and IPAdapter in this regard. However, it's worthy to note that we achieved such diversity under the condition of much higher face similarity.

Please note that we do not provide the DINO similarity for the full image region as we believe that measuring DINO similarity across the entire image is less accurate and reasonable compared to focusing on the facial region.

---

Q3: Reasons for PhotoMaker's Underperformance in the Experimental Results.

The observed underperformance of PhotoMaker is anticipated and aligns with findings from concurrent research, namely LCM-Lookahead. We speculate that this may be attributed to the narrow scope of celebrities in PhotoMaker's training dataset, limiting its effectiveness when applied to non-celebrities, as discussed in L41-L43 of the main paper.

---

Q4: Ablation on different fast sampling methods and different steps.

Please refer to GQ2 for detailed comparison.

Thanks for your helpful advices. We respond to your core questions as follows:

Q1: Lines 133 and 195 both mention that Q is an image feature of UNet, please explain how that image feature was obtained.

In each cross-attention layer of UNET, the UNET image features are projected into query features via a linear layer. We denote these features as Q. This is reflected in L132-L134 of the main paper. We will revise the description to make this clearer.

---

Q2: Is L_align-layout necessary? Is there any connection to your solution to the problem of ID information polluting the prompt?

Yes, it is necessary. We consider layout and composition of the generated images as part of the original model's behavior. Therefore, if the layout and composition change after ID insertion, we perceive it as a disruption to the original model's behavior. The L_{align-layout} is helpful in maintaining the original model's layout and composition, as demonstrated in Figure 3 (b) of the paper, we also provide more qualitative results in Figure 2 of the Rebuttal PDF. For a quantitative ablation, we provide the table below:

As observed from the table, if L_{align-layout} is removed in stage3, despite achieving the same level of face similarity as stage3 with all losses, both CLIP-T and CLIP-I decrease. While the decline in CLIP-T is relatively mild, the decrease in CLIP-I is more significant, as it tends to reflect consistency in layout and composition to a certain extent.

---

Q3: How the approach relates to the CFG scale and whether CFG is valid for this task

During training, our Lightning T2I branch does not require CFG, hence the CFG scale is 1. However, our method can incorporate CFG during training. When the SDXL-Lightning acceleration is not employed (refer to GQ1), a CFG scale of 7.5 is used.

As for testing, various CFG scales are compatible with our approach. For the base models utilized in our paper, we adopt the recommended CFG scales, such as 1.2 for SDXL-Lightning, 7 for RealVisXL, and 2 for Juggernaut-XL-Lightning.

Thanks for your positive and valuable comments. Our responses are as follows:

Q1: Since the model requires an efficient text-to-image diffusion model (in terms of number of inference steps), it can hinder the application of the introduced method with other base models.

Please refer to the detailed explanation in GQ1. The fast sampling plugin is not indispensable for our method, which indicates that our method can be adapted to other base models without an acceleration plugin.

---

Q2: How much memory is required for training?

We provide comprehensive data in GQ1 and GQ2 and discuss the memory issue in the Limitations section of the paper (L464-L465). As most comparative methods did not make their training codes open-source or disclose their memory usage, we cannot make a direct comparison.

---

Q3: How does the model perform if we set the number of diffusion step to (1, 2, 6)? (and how much memory footprint)

Please see GQ2 for a detailed comparison. Note that due to memory constraints, we only update the gradients at 6 of the 8 steps.

---

Q4: Does this method work for subject other than human as input (e.g., dog, cat, etc)?

Yes, it is theoretically feasible. Please refer to the discussion in GQ3.

Dear Reviewer bfEK,

We thank you for the precious review time and valuable comments. We have provided corresponding responses, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work. Thanks :-)

Best,

PuLID Authors