DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation

Thank the reviewer for the constructive comments and suggestions. We have revised our paper according to all the reviewers' suggestions and uploaded the revised version, where the revised contents are colored in blue for easy reading. The reviewer can download the revised version for more qualitative results and analysis. We address your concerns point by point as follows.

1. Reference and comparisons(W1/Q1):

   • We have added all the mentioned references in the related work in our revised version. Thanks for making our related work more complete.

   • We have also compared the mentioned baselines together with the baselines suggested by Reviewer UqfN in Appendix A.5 in our revised version. Both the qualitative(Table 7) and quantitative results(Figure 13) show the superiority compared to the baselines. For easier reading, we also present the quantitative results in the following table, which shows that our proposed DisenBooth has the best image fidelity and text fidelity.

   • Note that E4T[1] is a pretrained method and its open-source version is only for human subjects, which faces severe out-of-domain problem and cannot achieve satisfying performance on DreamBench. This phenomenon also indicates that although there are some non-finetuning methods for fast customization, effective finetuning to adapt to new out-of-domain concepts is still important. Although we tried to compare with [2] [3] in the suggested references, unfortunately we found these two works are two pretrained non-finetuning methods by Google, and there are no open-source codes and checkpoints for the pretrained models. We cannot compare with them because we have no access to the pretrained models.

     	Custom	SVDiff	Break-A-Scene	E4T	ELITE	DisenBooth(ours)
     DINO	0.675	0.668	0.639	0.241	0.606	0.675
     CLIP-T	0.314	0.318	0.313	0.262	0.291	0.330

   • Following the reviewer's advice, we have compared with ELITE quantitatively in our revised version and also in the table above. In our first version, we just want to compare our method with the finetuning-based method for fair comparison consideration, while ELITE is a pretrained and non-finetuning one.

2. Qualitative results on real human figures(W2):

   Following the reviewer's advice, we provided the visual comparison with the baselines on the human subjects in Appendix A.9 and Figure 16 in our revised version. The results further show that our proposed method has a clear advantage over the baselines in both preserving the human identity and conforming to the textual prompt.

3. How InstructPix2Pix is used for comparison(Q2):

   We use an example to show how we use InstructPix2Pix for subject-driven text-to-image generation as follows. Assuming that we want to customize the backpack and we have an input image of it, we want to generate an image of the backpack under the Eiffel Tower. What we do in the experiment is send the input image to InstructPix2Pix, and then use the instruction ``put the backpack under the Eiffel Tower'' to generate an image. In the DreamBench setting, the backpack has 6 input images, we will repeat the generation process for each input image and calculate the average DINO and CLIP-T score of all the generated images.

4. Paper Structure(Q3):

   We have tried to move Appendix A.1 to section 5.3, but we find it will exceed the page limit of ICLR. To emphasize this important part, in our revised version, we give the detailed cross-reference at the last of section 5.3 to make the readers easily find this ablation. Thanks for your suggestions.

5. Typo(Q4):

   We have fixed this typo. Thanks for your careful reading and pointing it out.

Thanks for the valuable suggestions from the reviewer. We think the revised content makes our experiments more comprehensive and our paper more clear.

Dear reviewer, thank you again for the review and we hope that our response and the uploaded revised paper have addressed your concerns. As the author-reviewer is about to end, we would greatly appreciate your feedback and please feel free to let us know if you have any other questions.

Thank the reviewer for the response. This is a really fruitful rebuttal process, where the paper is further polished with more clear writing and the results are further strengthened with the suggested experiments. Thanks for your suggestions and response again.

Thank the reviewer for the constructive comments and suggestions. We have revised our paper according to all the reviewers' suggestions and uploaded the revised version, where the revised contents are colored in blue for easy reading. The reviewer can download the revised version for more qualitative results and analysis. We address your concerns point by point as follows.

1. Comparison with more methods(W1/W3):

   Following the reviewer's advice, we compare DisenBooth to the mentioned baselines together with the baselines suggested by Reviewer QDoi in Appendix A.5 in our revised version, where the reviewer can see both quantitative (Table 7) and qualitative comparisons (Figure 13) on the DreamBench. Both the qualitative and quantitative results show the superiority compared to the baselines.

   For easier reading, we also present the quantitative results in the following table, which shows that our proposed DisenBooth has the best image fidelity and text fidelity. Note that E4T is a pretrained method and its open-source version is only for human subjects, which faces severe out-of-domain problem and cannot achieve satisfying performance on DreamBench. This phenomenon also indicates that although there are some non-finetuning methods for fast customization, effective finetuning to adapt to new out-of-domain concepts is still important.

   	Custom	SVDiff	Break-A-Scene	E4T	ELITE	DisenBooth(ours)
   DINO	0.675	0.668	0.639	0.241	0.606	0.675
   CLIP-T	0.314	0.318	0.313	0.262	0.291	0.330

2. More experiments for one single image finetuning(W2):

   Following the reviewer's advice, we have conducted experiments of tuning with only 1 image on more subjects (1/3 of the DreamBench dataset), and we give both quantitative results(Table 5) and qualitative visual results(Figure 11) in Appendix A.3 in our revised version. The results show that our proposed method can still disentangle under this 1-shot challenging scenario. For easier reading, we give the average quantitative results in the following table, which shows the superiority over existing methods.

   	Pix2Pix	TI	DreamBooth	DisenBooth(ours)
   DINO	0.613	0.538	0.610	0.617
   CLIP-T	0.302	0.288	0.297	0.321

Thanks for your suggestions, and we believe the revised contents further support our work and improve the quality of our paper.

Dear reviewer, thank you again for the review and we hope that our response and the uploaded revised paper have addressed your concerns. As the author-reviewer is about to end, we would greatly appreciate your feedback and please feel free to let us know if you have any other questions.

4. Performance with objects in corners and covering fewer pixels(W4/Q4):

   Following the reviewer's suggestions, we conduct some experiments when the objects only cover fewer pixels and are not located in the center. The experimental results are presented in our revised paper in Appendix A.8 and Figure 15. The results show that:

   • Our method can still effectively learn the identity-preserving and identity-irrelevant information.
   • However, similar to the multi-subject scenario, when the subject covers very few pixels, although the generated subject is overall similar to the given subject, some visual details are different. Therefore, in this scenario, a possible solution to preserve the details is to first crop the original image and then use a hyper-resolution network to obtain a clear image of the subject. After that, we can conduct finetuning with DisenBooth.

Thanks for your suggestions, we believe the review and the revised content can show more potential of DisenBooth and further improve the quality of the paper.

Thank the reviewer for the constructive comments and suggestions. We have revised our paper according to all the reviewers' suggestions and uploaded the revised version, where the revised contents are colored in blue for easy reading. The reviewer can download the revised version for more qualitative results and analysis. We address your concerns point by point as follows.

1. Lora Finetuning v.s. Full Finetuning(W1/Q1):

   Following the reviewer's suggestions, we have conducted experiments on finetuning the U-Net on 10 subjects and compared it with the results of Lora finetuning. The experimental results are provided in the following Table. DisenBooth-full in the Table means we finetune the whole U-Net in DisenBooth.

   	DreamBooth	DisenBooth-full	DisenBooth(ours)
   DINO	0.671	0.664	0.666
   CLIP-T	0.321	0.333	0.336

   From the results, we can see that the performance of finetuning the whole U-Net is comparable to Lora finetuning, and additionally, both DisenBooth-full and DisenBooth can alleviate the impact of overfitting and achieve higher CLIP-T(textual fidelity). However, considering that finetuning the whole U-Net for each subject will averagely consume 122 min while finetuning lora only requires 38 min on our V100 32G GPU. Therefore, we recommend finetuning with lora for efficiency.

2. Customization for multiple subjects in a scene(W2/Q2):

   The reviewer proposes a very interesting scenario to customize multiple subjects in a scene, and we conduct a preliminary experiment under this scenario in Appendix A.7 in our revised version, which the reviewer can refer to for qualitative generation results(Figure 14) and analysis.

   During customizing multiple subjects in a scene, we will use a prompt to describe the multiple subjects for the identity-preserving branch, and leave the identity-irrelevant branch to learn other features. The preliminary exploration results show that our proposed method can also be applied to this scenario and generate overall satisfying images, demonstrating the potential of DisenBooth for broader applications.

   Also, there are some challenges. Although the generated subjects are overall quite similar to the given subjects, some visual texture details of the generated subjects are a little different from the input subjects. The detail differences may result from the following reason: since the identity-preserving branch needs to customize more subjects, and the pixels for each subject become fewer, it will be hard to notice every detail of each subject. Although beyond the scope of this work, we believe customizing multiple subjects in a scene is an interesting topic and there are more problems worth exploring in future works.

3. Using masks on pixels or on features(W3/Q3):

   Following the reviewer's advice, we compare the results of using pixel-level masks (explicitly integrating masks into images before encoding) and feature-level masks (masking in the adaptor) on 10 subjects in the following Table.

   	DreamBooth	Pixel Mask	DisenBooth(ours)
   DINO	0.671	0.666	0.666
   CLIP-T	0.321	0.327	0.336

   We can see that both applying pixel-level masks (Pixel Mask) and feature-level masks (DisenBooth) can improve the CLIP-T score compared to DreamBooth, alleviating overfitting the identity-irrelevant information problem. However, it shows that applying the feature-level mask brings better performance. The reasons are as follows:

   • Masks at the feature level are filtering information at the semantic level instead of pixel level, enabling it to capture more identity-irrelevant factors such as the pose or position of the subject. Therefore, the identity-preserving branch does not have to overfit these factors, thus achieving better CLIP-T score and better text fidelity. However, using the masks on the pixels can only disentangle the foreground and the background, ignoring other identity-irrelevant factors such as pose.

   • The mask at the feature level is learnable and jointly optimized in the finetuning process, while adding an explicit mask to the image is a two-stage process, whose performance will be a little worse.

   We add the quantitative results (Table 4) and also qualitative analysis (Figure 10) in Appendix A.2 in our revised version to better explain the reasons (colored in blue). Generally speaking, if the users do not want to change other identity-irrelevant factors except for the background, such as the pose of the subject during generation, both ways can be applied. Otherwise, applying the mask at the feature level as our method may bring better performance.

Dear reviewer, thank you again for the review and we hope that our response and the uploaded revised paper have addressed your concerns. As the author-reviewer is about to end, we would greatly appreciate your feedback and please feel free to let us know if you have any other questions.

Thank the reviewer for the constructive comments and suggestions. We have revised our paper according to all the reviewers' suggestions and uploaded the revised version, where the revised contents are colored in blue for easy reading. The reviewer can download the revised version for more qualitative results and analysis. We address your concerns point by point as follows.

1. More details about identity-irrelevant branch:

   In the identity-irrelevant branch, we want to extract the identity-irrelevant information of the input image $x_i$. Therefore, we first resort to the pretrained CLIP image encoder $E_I$ to extract this feature, and consequently we obtain a feature $f_i^{(p)} = E_I(x_i)$. However, this feature obtained from the pretrained model may contain the information about the identity. To filter out the identity information from $f_i^{(p)}$, we design a learnable mask $M$ with the same dimension as the feature, whose element values belong to (0,1), to filter out the identity information. Therefore, we obtain a masked feature $M*f_i^{(p)}$, by the element-wise product between the mask and the pretrained feature. If one dimension of the mask is learned to 0, it means the same dimension of the pretrained feature will be discarded by multiplying 0, thus achieving the goal of filtering.

   Additionally, considering that during the Stable Diffusion pretraining stage, the text encoder is jointly pretrained with the U-Net, while the image encoder is not jointly trained during Stable Diffusion pretraining, making the image feature and text feature in different space. We use the MLP with skip connection to transform $M*f_i^{(p)}$ into the same space as the text feature $f_s$, i.e., $f_i = M\*f_i^{(p)} + MLP( M\*f_i^{(p)} )$, which is the equation (4) in our paper as shown in the following. We call the mask and the MLP together with skip connection as the Adapter.

   Thanks for your suggestion, we have revised Sec. 4.1 to make it more clear. Also, the effectiveness of the mask to filter out the identity information is validated in Appendix A.2 in our submitted version. If there is anything unclear, please feel free to let us know and we are glad to answer further questions.

2. The number of used images:

   For fairness, we keep the number of images used by all methods the same, which is exactly the same as the dataset provided by the paper of DreamBooth. Additionally, we also provide the results of tuning with 1 image for all the methods in Appendix A.3, which further shows the superiority of our proposed method.

3. Whether finetuning text encoder causes more overfitting:

   The answer is yes. Following the reviewer's advice, we try to explore the influence of tuning the text encoder for both DreamBooth and our DisenBooth, by finetuning on 20 subjects of the DreamBench. We report the average quantitative results in the following Table,

   	DreamBooth	DreamBooth+text	DisenBooth(ours)	DisenBooth+text
   DINO	0.703	0.728	0.694	0.719
   CLIP-T	0.322	0.286	0.333	0.301

   where "+text" means finetuning the text encoder. We can see that both finetuning text encoder for DreamBooth and DisenBooth will increase the DINO score, meaning the generated images will be more similar to the given input images. However, the CLIP-T score (textual fidelity) will be decreased, indicating more serious overfitting problem. Despite the overfitting, we find that compared to DreamBooth, our proposed DisenBooth still shows clear superiority in reducing the impact of overfitting, surpassing DreamBooth in CLIP-T with a clear margin.

   We believe this exploration is important, and we have added the results to Appendix A.4 in our revised version (Table 6), and also together with the corresponding qualitative results (Figure 12).

Thanks for your suggestions, we believe the review and the revised content make the paper more clear and further improve the quality of the paper.

Dear reviewer, thank you again for the review and we hope that our response and the uploaded revised paper have addressed your concerns. As the author-reviewer is about to end, we would greatly appreciate your feedback and please feel free to let us know if you have any other questions.