DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image Generation using Limited Data

Thanks for your precious time for reviews. Here we provide corresponding responses and results to cover your concerns.

1.$**Cross-domain correspondence approach**$: We provided the discussion between our approach and this prior work in Appendix J. In the revised manuscript, we illustrate the difference in Sec 3.1 as follows:

We are inspired by CDC, which builds pairwise similarity loss based on features in certain layers of the generator in StyleGAN2 to preserve cross-domain similarity. DDPMs are trained to predict noises with multiple steps in synthesizing images. It is difficult to find proper features to represent generated samples. As a result, we directly build image-level pairwise similarity with the completely denoised samples, which makes our approach compatible with DDPMs. Besides, our method is designed for domain-driven generation to preserve the basic distributions of subjects, which take up most parts of the generated samples, instead of building one-to-one correspondence across domains like CDC. We use various noise inputs for source and adapted samples in T2I generation but still achieve the preservation of subject categories.

More detailed discussion of our approach is added in Appendix J.

2.$**Outperforming CDC**$: As discussed above, our approach is different from CDC. Besides, CDC is based on StyleGAN2 while our approach is designed for DDPMs. These models are different. We cannot simply get an equation like “StyleGAN2 + CDC == DDPM + CDC“ even if the source models share similar metrics of FID and Intra-LPIPS. As shown in the visualized results, unexpected blurs and deformation are inevitable in the adapted samples of GAN-based approaches. However, such problems do not exist for DDPMs. Directly fine-tuned DDPMs get smooth results with limited diversity and lack high-frequency details. With our approach, adapted DDPMs achieve high-quality results with rich details.

3.$**Subjects in source and target domains**$: Our approach is designed for domain-driven generation by preserving the subjects in source samples and learning the domain knowledge from training samples. We have revised this sentence as: “Subjects in source and adapted samples share the same category but can be different from subjects in training samples.” to make it clearer. For example, given 10-shot Van Gogh houses paintings, we can set the source samples with the prompt “a dog” and the adapted samples with the prompt “a [V] dog”. The subjects in source and adapted samples are different from the subject “a house” in training samples. We have revised the whole paper to remove the confusing description of “source domains” and “target samples” to avoid similar misunderstanding.

Several common expressions used in our paper:

“source samples”: samples produced by the pre-trained model, we aim to preserve the subjects in them

“target domains”: domains (e.g., styles) extracted from few-shot training data

“adapted samples”: samples produced by adapted models, which are designed to preserve the same category of subjects in source samples and be adapted to target domains

4.$**Details and Diversity**$: The high-frequency enhancement in our approach preserves more diverse details in adapted samples. It is designed from two perspectives: preserving more details provided by source models (which share the same format of pairwise similarity loss) and learning more details from limited data for finer quality. More details preservation contributes to generation diversity as well. For example, we have more diverse detailed structures of houses in the samples of full DomainStudio compared with DomainStudio w/o high-frequency enhancement, which tends to synthesize simpler houses (see Fig. 9 in the revised manuscript).

5.$**Other Methods**$: As far as we know, we are the first to tackle synthesizing samples in specific domains using limited data with DDPMs.

Please let us know if our responses solve your concerns. If you still have any unclear parts about our work, please let us know as well. Thanks for your review.

Dear reviewer w5yc,

We thank you for your valuable reviews. We have revised the manuscript and provided detailed response to cover your concerns. Today is the last day for discussion between authors and reviewers. We are looking forward to your reply.

Thanks for your review!

Thanks for the so many valuable suggestions provided in your review. We have revised our paper following your valuable suggestions.

1.$**Quantitative Results**$: We have moved the key part of the quantitative evaluation to the main paper. More details are provided in the appendix.

2.$**Ablation Analysis**$: We have moved the representative parts of the ablation analysis to the main paper. More details are provided in the appendix.

3.$**Introduction**$: We have revised the introduction part to give a clear definition of “domain-driven generation” and the necessity of the few-shot domain-driven task. Then, the main idea of contribution is introduced. We directly use the name for related works like DreamBooth and Textual Inversion.

4.$**Related Work**$: We have revised related work to provide much more higher-level analysis to identify the differences between our approach and prior works.

5.$**Fig.1**$: The same subject refers to the same category of subjects. We have revised Fig.1 and the whole paper to provide a clearer definition of our task to avoid misunderstanding.

6.$**Citation**$: We have revised all our paper's citations and changed them to \citep. Thanks!

7.$**Lightweight**$: It is a writing mistake, and we have fixed this issue in the revised manuscript.

8.$**High-frequency Regularization**$: We find Sec. 3 is not convincing and removed this part in the revised manuscript. We include the ablation analysis to demonstrate the motivation of each part in our approach.

9.$**Fig.5 and Fig.6**$: We have revised Fig.5 to make it fit with Fig.6 better and revised their captions to describe the contributions.

Looking forward to more discussion and suggestions on our paper from you. Thanks again!

Thanks for your precious time for reviews. Here we provide corresponding responses and results to cover your concerns.

1.$**LoRA**$: We provide our approach on text-to-image generation with modern few-shot fine-tuning methods based on stable diffusion like textual inversion and dreambooth. We have added additional comparison with LoRA using qualitative (Fig. 36 in Appendix N) and quantitative (Table 6 & 7 in Appendix D) results in the revised manuscript. Similar to DreamBooth, it’s also hard for LoRA to achieve realistic results in domain-driven generation. Thanks for your advice.

2.$**Ablations**$: We provided ablation analysis of our approach in Appendix E. Following the advice from Reviewer 4uc5, we have moved this part to the main paper in the revised manuscript. More detailed qualitative and quantitative ablation analysis of each term is provided in Appendix E.

Please let us know if our responses solve your concerns. If you still have any unclear parts about our work, please let us know as well. Thanks for your review.

Dear reviewer Qxcc,

We thank you for your valuable reviews. We have added experiments of LoRA to show the improvement of our approach in domain-driven tasks. Besides, detailed ablation analysis have been added to the main paper as well. Today is the last day for discussion between authors and reviewers. We are looking forward to your reply.

Thanks for your review!

8.$**Additional Baselines**$: We have added the qualitative and quantitative results of AdAM and RICK, which are designed to deal with incompatible source/training samples, like FFHQ $\rightarrow$ AFHQ CAT. We run the official implementation code to get qualitative results and evaluate the FID and Intra-LPIPS results. We got similar visual effects as shown in their paper but failed to reproduce the FID results as well as reported in their paper. As for Intra-LPIPS (generation diversity), unconditional DomainStudio achieves better performance. We add FID results in their paper to the revised manuscript. Despite its outstanding FID results, RICK still fails to avoid generating unnatural deformation and blurs like prior GAN-based methods. Our approach shows apparently better visual effects, as shown in the visualized samples provided in the revised manuscript (Fig. 5 and Appendix N). We also fix the statement in abstract to “better visual effects than state-of-the-art GAN-based approaches”.

9.$**Raphael's paintings**$: We conduct additional experiments with weaker regularization and get better results in target domains on FFHQ $\rightarrow$ Raphael’s paintings and LSUN Church $\rightarrow$ Landscape drawings. Corresponding visualized and quantitative results have been updated in the revised manuscript.

10.$**Experiments necessity**$: As discussed above, taking the 10-shot landscape drawings (see scene 2 in Fig.1) as an example, it is hard to find proper text prompts for such data. There can be thousands (even more) of drawing styles in the training data of large-scale text-to-image models.

Please let us know if our responses solve your concerns. If you still have any unclear parts about our work, please let us know as well. Thanks for your valuable suggestions.

6.$**Task Definition**$: This paper tackles a domain-driven generation task, which is defined as retraining the category of subjects in source samples and adapting them to target domains extracted from few-shot training data. It is designed to synthesize samples in specific domains (e.g., styles). We have provided detailed definition in the revised manuscript and show several application scenarios, some of which cannot be handled by existing approaches in Fig. 1 in the revised manuscript. When the source samples share the same class of subjects with training data, it shares the same target as prior unconditional few-shot image generation tasks (e.g., FFHQ $\rightarrow$ Sunglasses). When different, DomainStudio is designed to preserve subjects in source samples and learn domain knowledge only from training samples.

7.$**Task Necessity**$: Our work tackles the task of synthesizing samples in specific domains given few-shot training samples. It is necessary due to two reasons. Firstly, it is difficult to describe domains with text prompts accurately for T2I models due to the ambiguities in natural language. Besides, even if pre-trained models have captured typical domains owing to the presence of corresponding samples in their training data, they may still produce biased results influenced by the out-of-distribution effects. For example, there exist different styles in Van Gogh's paintings. Given a text prompt of ``Van Gogh style", T2I models may choose a certain style randomly or mix some of them, leading to unsatisfactory outputs. Therefore, DomainStudio is designed to synthesize samples in specific domains learned from the given few-shot training samples. We have added this part to the introduction of the revised manuscript to make our task setting more convincing.

Thanks for your precious time for reviews. We provide corresponding responses here and revise our paper to cover your concerns.

1.$**Section3**$: We agree with you that Section 3 is not surprising and placing this part in the main paper makes our task confusing. We have provided a clear definition of the domain-driven generation task in the abstract and introduction of the revised manuscript and removed this part (Sec.3) in the revised manuscript.

2.$**Fig.2**$: In the original Fig.2, we directly use image-level information (RGB values) to compute cosine similarity. Fig.2 has been removed in the revised manuscript. Instead, we use ablations of our approach with qualitative and quantitative (FID and Intra-LPIPS) results (which are originally provided in Appendix D) to discuss the diversity and high-frequency details that are improved by our approach compared with directly fine-tuned models.

3.$**Fig.3**$: In the original Fig. 3, we use DreamBooth as a comparison to demonstrate that it is not qualified for domain-driven tasks or it cannot learn the domain knowledge with an identifier [V], which is achieved by our approach. DreamBooth w/o prior preservation loss corresponds to directly fine-tuned Stable Diffusion. Without the proposed relative distances preservation and high-frequency details enhancement, DomainStudio degrades to DreamBooth on text-to-image generation. Overall, we agree with you that Fig.3 makes readers confusing and removed it in the revised manuscript.

4.$**Related Works**$: We put the discussion between our approach and CDC in Appendix J. In the revised manuscript, we illustrate the difference in Sec 3.1 as follows:

We are inspired by CDC, which builds pairwise similarity loss based on features in certain layers of the generator in StyleGAN2 to preserve cross-domain similarity. DDPMs are trained to predict noises with multiple steps in synthesizing images. It is difficult to find proper features to represent generated samples. As a result, we directly build image-level pairwise similarity with the completely denoised samples, which makes our approach compatible with DDPMs. Besides, our method is designed for domain-driven generation to preserve the basic distributions of subjects, which take up most parts of the generated samples, instead of building one-to-one correspondence across domains like CDC. We use various noise inputs for source and adapted samples in T2I generation but still achieve the preservation of subject categories.

We have added the discussion between high-frequency details enhancement in our approach with related works, including MaskedGAN and FreGAN, as an added part of related work (Sec. 2) in the revised manuscript. Our approach differs from the prior GAN-based methods in many aspects.

5.$**Contributions**$: We have revised the contributions of our work in the revised manuscript. The contributions of our work come from the task definition, the domain-driven generation approach compatible with both unconditional and T2I models, and the effectiveness demonstrated with a series of experiments.

Dear reviewer mvse,

We thank you for your valuable reviews. We have revised the manuscript and provided detailed response to cover your concerns. Today is the last day for discussion between authors and reviewers. We are looking forward to your reply.

Thanks for your review!

Thanks for your detailed response. We still need to clarify several points about your concerns.

1.$**Task Definition**$:

1. FSIG tasks aim to produce diverse samples following the distributions of few-shot training samples. As illustrated in the revised related works, they share the same target with our work when the source and target domains share the same category of subjects, e.g, FFHQ --> Sunglasses. The subjects mentioned in our paper refer to the category of subjects, which has been illustrated in the revised manuscript. "Subjects learned from training samples" doesn't means that they are trained to preserve the 10 subjects in training samples. We have revised the table in Fig.1 in the manuscript to make it clearer.

The target of FSIG is different from our work when subjects in source and training samples are different. For example, when FSIG adapts source models pre-trained on LSUN Church to 10-shot landscape drawings, they tend to produce samples without churches since there are no churches in training samples, which is the category of subjects in source samples (visualized results provided in Fig. 28 in Appendix N). However, our approach preserves diverse churches in adapted samples and adapts them to target domains. Similar samples can be found in Fig. 7 of CDC, it tends to produce samples of haunted houses even if the source model is pre-trained on cars or horses. AdAM and RICK work on source/target domains with different categories of subjects and get samples of cat faces using the source model pre-trained on FFHQ (human faces).

2. As illustrated above, our approach shares the same target with CDC and EWC when the source and target domains share the same kind of subjects.

3. As illustrated in the paper, our approach can be combined with DreamBooth to realize both domain-driven and subject-driven tasks at the same time. Detailed methods and more experiments are provided in Appendix G. It can be seen as an extensive application of the proposed domain-driven method. As for the example, our approach preserves the key features of the target dog, e.g., the same eyes, noses, and ears, and adapts it to the ink painting style.

2.$**Task Necessity**$:

We have provided the necessity of this task from two aspects: the ambiguities in natural language and the out-of-distributions effect. Recent works from Google tackle similar tasks as this paper using other text-to-image models [1-2] instead of diffusion models and depending on human feedback. Besides, we have provided an example in the rebuttal above. Given 10-shot landscape drawings, we cannot provide an accurate description with text prompts for Stable Diffusion to synthesize the same style. For another example, in Fig.7 of the revised manuscript, we cannot find a hard text prompt directly for comparison as well. We just use some examples from Van Gogh as examples. Besides, there also exist different styles in Van Gogh's different works. Our approach is designed to learn the style from given samples directly. Moreover, taking DreamBooth as an example, they synthesize results of a Corgi dog given several examples in their paper. However, we cannot say that the subject-driven task is unnecessary since we can directly ask the Stable Diffusion to synthesize samples with a hard text prompt "a Corgi dog" instead of ''a [V] dog".

[1] StyleDrop: Text-to-Image Generation in Any Style, arxiv 2306.00983

[2] Learning disentangled prompts for compositional image synthesis.arXiv:2306.00763

3.$**Results from AdAM and RICK**$:

Firstly, we directly run the official implementation code with the provided hyperparameters to make sure the reproduced results are correct. We fix the noise inputs for different GAN-based methods for fair comparison and provide visualized results in Fig. 5 and additional results in Appendix N. The FID results are directly copied from their paper. As for the Intra-LPIPS results, we synthesize images with fixed noise inputs for fair comparison as well. As illustrated in the revised paper, AdAM and RICK still produce low-quality blurs and deformations of human faces, which can be found in Fig. 25 in its supplementary as well. We have added an direct comparison between our results and the visualized samples provided in the publications of AdAM and RICK in Fig.25 of our supplementary to show the advantages of our approach. Our approach achieves apparently better visual effects with high-quality details and avoids unnatural blurs or artifacts which can be found in GAN-based methods, including AdAM and RICK. In our opinion, the visual effects should be the first consideration for generative tasks. The metrics aren’t always super well-aligned with human perception of quality. Besides, the proposed approach is qualified for domain-driven tasks which existing GAN-based methods cannot handle, e.g., LSUN Church $\rightarrow$ Landscape drawings.

Looking forward to your responses. Thanks for your precious time.

I really appreciate the effort of the authors to address my concerns. However, I think there are still some major problems:

I. Task Definition in Figure 1, is more intuitive and not accurate enough.

• How can we claim that few-shot image generation (FSIG) learns subjects from the training samples given that there are only 10 subjects? note that these methods can generate thousands of subjects after adaptation.

• The definition provided for the domain-driven image generation (preserve subject from prior knowledge, learn domain/style from training samples) is in fact the goal of most of the FSIG approaches that aim to generate images from close domains. For example, EWC [a] and CDC [b] motivate their approach with this goal.

• Scene 4 in the figure is quite self-contradictory. First, the authors mention that the goal is to preserve the subject from the prior knowledge, but here they aim to personalize the subject from training samples. Am I missing something? In addition, the reported results do not preserve the subject at all. The subject in the training samples is a dog with the Chow Chow breed, and the generated ones are from other breeds.

I think the authors need to think a bit more about the exact definition of the task that they have proposed, and the definition is still not clear and acceptable.

II. I still am not convinced about the necessity of this task, as I mentioned in the main review. I was able to produce quite convincing results about the desired domain using just a proper text prompt (without any prompt engineering and just a hard text prompt). I suggest authors also think about this aspect more carefully and have a more systematic study/comparison with the hard prompt version.

III. Regarding experimental results, I understand that it might be hard to replicate the results for SOTA approaches given the time limitations in rebuttal. However, the reported results in these papers, both quantitatively, and qualitatively (for example figure 25 in AdAM's Supp. for FFHQ $\rightarrow$ Sunglasses) seem to be much better than the provided results in this paper.

Overall, I appreciate the effort from the authors and I believe the revised version has improved a lot. However, there are still some major problems that need to be thought about and addressed carefully. Hope my comments to be constructive for the next version of this work.

Dear all reviewers,

There are about two days left for discussion. We have provided the rebuttal for your reviews. Could you please look at our responses and let us know what concerns are still not solved or if you still have any unclear parts about our work? We are looking forward to your responses.

Thanks for your review!