Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach

We thank the reviewer for the insightful comments and suggestions, below we address some concerns and answer some questions.

Q: From Equation 10 to Equation 11.

A: Equation 11 is derived from Equation 8 and Equation 10. In Equation 8, we obtain an inaccurate prediction for timestep $t-1$, which is denoted as $\widetilde{x_{t-1}}$. In equation 10, we inject the structure information in $\widetilde{x_{t-1}}$ into the current noised sample $x_t$ to obtain an updated sample $x_t$. As we discussed in the paper. Simply feeding the conditions independently into the model will lead to structure conflict. Thus we need to first obtain an “inaccurate” next-step sample, which is harmonized in terms of structure information . It is called “inaccurate” because which it is conditioned on $\gamma S^*$. Using this $\widetilde{x_{t-1}}$ as a prediction for $x_{t-1}$ leads to generation with some details lost, as shown in Figure 12. But this “inaccurate” sample $\widetilde{x_{t-1}}$ does not have structure conflict anymore. Thus we propose to use its structure to infer a corresponding updated sample $\widetilde{x_{t}}$ for current time step $t$, then perform sampling based on this $\widetilde{x_{t}}$ instead of $x_t$.

Q: Related works.

A: Thanks for the suggestion, we will add the discussion.

We thank the reviewer for the helpful reviews, below we address some questions and concerns.

Q: Some related works are missing:

A: Actually, [3] (E4T) is the most important baseline we have already discussed in our paper, our method obtained better results as presented in the paper. We will include [1] and [4], and add comparison. Although [2] is not an encoder-based method, we will add a discussion.

Q: The important term "independent conditions" is not clearly defined.

A: By "independent conditions", we simply mean the two conditions $S^*$ and $C$ are independent given the generation, i.e, $p(S^*,C|x) = p(S^*| x)p(C|x)$.

Q: Question about identity preservation.

A: We want to emphasize that we never claimed the better fine-grained details can be obtained by a sampling method. What we claimed is that details can be better preserved if the model is trained/fine-tuned without regularization. However, baseline sampling fails to generate desired images with respect to text in this case. Figure 9 shows that Fusion Sampling can generate images aligned with text, under the setting without regularization, no matter before or after fine-tuning. Better details are certainly obtained by fine-tuning. However, without Fusion Sampling, we can not enjoy the setting without regularization which has the best detail preservation.

That's why we propose to fine-tune without regularization and perform Fusion Sampling after fine-tuning. As we show in the paper, compared to related encoder-based method [3] which fine-tunes the model with regularization and performs baseline sampling, better results are obtained by our method.

Furthermore, we are able to perform more flexible generation with Fusion Sampling: we can choose to emphasize either better creativity or better details, which is why the results of our method in Figure 5 and Figure 7 are represented by line instead of point like other methods. And our line is above all the points, indicating better results are obtained by our method.

Q: About experiment results.

A: Customized generation is subjective and hard to evaluate. That's why we performed both quantitative evaluation and human evaluation. In quantitative evaluation, pre-trained models are used to compare extracted features from original and generated images. Better quantitative results are obtained by our method. In human evaluation, workers from Amazon Mechanical Turk are asked to compare results from different methods. All the workers have performed at least 10,000 approved assignments with an approval rate ≥ 98%, thus we believe the human evaluation results from the workers can objectively illustrate the effectiveness of our proposed method.

[1] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. ELITE: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848, 2023.

[2] Chen, Wenhu and Hu, Hexiang and Li, Yandong and Ruiz, Nataniel and Jia, Xuhui and Chang, Ming-Wei and Cohen, William W. Subject-driven Text-to-Image Generation via Apprenticeship Learning. NeurIPS 2023.

[3] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or. Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models. arXiv preprint arXiv:2302.12228 (2023).

[4] Xuhui Jia, Yang Zhao, Kelvin C.K. Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, Yu-Chuan Su. Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models. arXiv preprint arXiv:2304.02642, 2023.

We will add more experiment results to illustrate the effectiveness of the Fusion Sampling in the Appendix soon.

There might be some misunderstanding, we would like to clarify more on the "encoder-based" method. SuTI directly adopts the architecture from Re-Imagen [1], which is a retrieval augmented diffusion model. Different from E4T, ELITE or our method, it does not introduce an extra encoder network outside the diffusion model, but directly use the "encoder part" of the UNet. We did not call it encoder-based method, just like we would not call diffusion model itself as an encoder-based method. Furthermore, SuTI focuses on apprenticeship learning so that the final model can imitates the behaviors of different experts, rather than focusing on design encoder mapping to map input image into embeddings like E4T do. We are happy to add SuTI as related work for discussion, but want to emphasize the difference between it and other works like E4T and ELITE.

[1]. Re-Imagen: Retrieval-Augmented Text-to-Image Generator. Wenhu Chen, Hexiang Hu, Chitwan Saharia, William W. Cohen.

I thank the authors for the rebuttal. The response addresses most of my concerns. However, I agree with other reviewers that the motivation of the need for "regularization-free" is not comprehensively analyzed. Perhaps the authors should emphasize more on the problem of overfitting to $S^*$ instead of detail preservation. Also, the effectiveness of the Fusion sampling method has not been demonstrated convincingly in the current version. Therefore, I lowered my rating accordingly.

We thank the reviewer for the insightful comments and suggestions, below we address some concerns and answer some questions.

Q: About line#9 in the algorithm.

A: As we discussed in the paper. Simply feeding the conditions independently into the model will lead to structure conflict. Thus we need to first obtain an “inaccurate” next-step sample, which is harmonized in terms of structure information . It is called “inaccurate” because which it is conditioned on $\gamma S^*$. Using this $\widetilde{x_{t-1}}$ as a prediction for $x_{t-1}$ leads to generation with some details lost, as shown in Figure 12. But this “inaccurate” sample $\widetilde{x_{t-1}}$ does not have structure conflict anymore. Thus in line #9 of the algorithm we propose to use its structure to infer a corresponding updated sample $\widetilde{x_{t}}$ for current time step $t$, then perform sampling based on this $\widetilde{x_t}$ instead of $x_t$.

Q: About m=1 in the algorithm.

A: In all the experiments reported in the paper, we use m=1, which can already lead to better results than other methods. We believe that it is enough to show that m=1 works well in practice. We add ablation study on m>1, in the revised Supplementary Material. From which we can see that increasing m does not lead to noticeable improvement. Considering the fact that increasing m will lead to longer sampling time, thus we believe using m=1 in practice is a better option.

Q: Do all the baseline methods use data augmentation?

A: We didn’t use data augmentation in the pre-training stage, which is the same as other method. Data augmentation during fine-tuning is part of our proposed method. We will provide ablation study of comparing baseline sampling and Fusion Sampling, under different settings including with data augmentation, without data augmentation, and different level of regularization during fine-tuning.

Q: Implementation details.

A: We conduct all the experiments on Nvidia A100 GPUs. The pre-training time for encoder on CC3M dataset cost around 1 week on 8 A100 GPUs. The fine-tuning time on a single A100 GPU is around 25 seconds for 50 steps.

Q: Is it possible to combine the proposed method with other regularization methods?

A: Good question. Yes, the proposed method can be combined with any regularization methods as the Fusion Sampling will not influence the training, it only changes the inference. However, whether it can lead to improvement over a model without regularization depends on the regularization method itself. Currently, applying Fusion Sampling on a model with L2 regularization does not lead to observable improvement. Because some fine-grained details will be lost by applying regularization, and can not be recovered by simply using Fusion Sampling.

Q: Trade-off between creativity and detail preservation.

A: Good question, we provide some results in the revised Supplementary Material. We can easily control the generation, obtaining different level of creativity and detail preservation, by tuning a single hyper-parameter $\gamma$.

Q: Can the method capture style?

A: Good question, we provide some results in the updated Supplementary Material, from which we can see that the proposed method can successfully capture the style of the input image and combine it with user-input text to create new generations.

We thank the reviewer for the insightful comments and suggestions, below we address some concerns and answer some questions.

Q: About regularization used in verifying the claim "regularization can result in detail lost".

A: In experiments, we tried regularization $\lambda \Vert S \Vert^2$ with $\lambda$ selected from [1e-5, 1e-4, 1e-3, 1e-2, 1]. Figure 3 is presented to illustrate the influence of regularization on generation, small regularization like 1e-5 will result in failure in generating creative contents, while larger regularization will lead to identity loss. In customization, we are looking for a point where the model can preserve the details while having the capability to generate creative contents. However, simply tuning regularization hyper-parameter can not solve this problem.

Q: Why feeding C independently into the UNet can address this issue.

A: When we feed both $S^*$ and $C$ into the model, the model will ignore $C$ because $S^*$ is too strong. But if we feed $C$ along to the model, for sure the model will notice $C$, as there is no other condition here. What’s more, feeding conditions independently gives us the flexibility to control the influence of each condition, through a hyper-parameter $w_i$ in classifier-free guidance. We can control how much should the results be conditioned on $S^*$ and $C$. As we show in the paper. Feeding conditions independently can not fully solve the problem, that’s why we need another fusion stage, to avoid some structure conflict in the generated results.