Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation

Dear Reviewer,

Thank you for taking the time to read and review our paper! We address your comments in detail below.

I would advise the authors to provide clear insights through experiments and offer some specific suggestions.

In response to your advice for clearer insights and specific suggestions through experiments, we have introduced a new table (Table 1) in the revised manuscript. This table is aimed at providing a straightforward guide for readers in choosing suitable algorithms for various fine-tuning scenarios. For instance, based on our comprehensive experiments, we suggest that LoHa may be more effective for fine-tuning multiple "easier" concepts, whereas LoKr with full dimension appears better suited for targeting single "harder" concepts. This insight is reflected in the revised conclusion of our paper to provide more specific guidance.

In the meantime, we want to emphasize that while our summarized findings and suggestions provide a useful overview, they should be considered with caution. The complexity of fine-tuning text-to-image models, as discussed in detail in appendix C, means that a simplified summary could hardly capture the full picture. In this regard, our detailed text discussion offers a more nuanced explanation of the various ways in which a hyperparameter can influence the results, and is indispensable for a more comprehensive understanding of the fine-tuning process.

I cannot evaluate this paper because I believe it is proper for a benchmarking and dataset track, not the main track.

We appreciate your perspective regarding the suitability of our paper for a benchmarking and dataset track. While our work does include benchmarking, it is important to highlight that these are not its only contributions. A major part of our research introduces and elaborates on new fine-tuning method such as LoHa and LoKr. Moreover, the introduction of the library should also be regarded as an independent contribution. Generally speaking, our work aims to identify the specific fine-tuning methods that are more appropriate for a task of interest. Many works in this line of research (discussed in detail in Appendix A) have been previously accepted by the main track of the conferences, such as

• Zhiheng Liu et al. Cones: Concept neurons in diffusion models for customized generation. In ICML, 2023.
• Yuchao Gu et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. In NeurIPS, 2023.
• Zeju Qiu et al. Controlling text-to-image diffusion by orthogonal finetuning. In NeurIPS, 2023
• Nupur Kumari et al. Multi-concept customization of text-to-image diffusion. In CVPR, 2023.
• Nataniel Ruiz et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.

Finally, we do not believe ICLR possesses a separate "dataset and benchmark" track. In fact, on the call for paper page of ICLR 2024, datasets and benchmarks is clearly indicated as a valid subject area, even though in our case, as explained, benchmarking serves more to support and validate the novel contributions, rather than being the sole focus of our work.

Dear Reviewer,

Thank you for your feedback and for raising important points regarding our paper. Please find below our responses to your questions.

This paper mainly focus on LoRA-based finetuing strategies, can it be expanded to other parameter-efficient finetuning methods such as [1] and [2]?

While our paper is primarily focused on LoRA-type methods, it is completely possible to include other methods in our library and experiments.

In fact, IA3 that we mentioned in the paper provides a different strategy via fine-tuning a multiplicative term, which is later generalized by orthogonal fine-tuning (OFT) [1] that you mention. Although we only got aware of this work after submission of the paper, we have already included it in the latest version of LyCORIS. This is also indicated in the revised version of the paper.

On the other hand, as far as we understand, DiffFit [2] is tailored to class-conditional models and can otherwise be regarded as a more restrained case of IA3 with several additional components, notably the bias and normalization layers, being fine-tuned. We can thus readily configure the training to get something that is really close with earlier version of our library.

Finally, the choice of the focus on LoRA, LoHa, LoKr, and native fine-tuning in the paper are due to their wider use in the community, but we are also considering conducting the same experiments for other methods, and notably OFT.

The conclusion about the performance of different finetuning methods is not clearly presented in the experimental section. Maybe some tables can more straightforwardly represent your final conclusions.

Following your suggestion, we have added a new table (Table 1) in the revised version of our paper to demonstrate the influence of algorithm choice in the fine-tuning process. This addition should enhance clarity and accessibility for readers, providing a more direct understanding of our findings. Thank you for this valuable suggestion!

In addition to the table, the SHAP beeswarm charts readily included in our original paper serve as another effective tool for illustrating the influence of hyperparameters. However, it is crucial to note that many subtleties and context-specific insights discussed in our text cannot be fully captured by these visualizations alone. The readers should thus consider both the table and the charts in conjunction with the detailed textual explanations to gain a comprehensive understanding of the influence various factors on the fine-tuning process.

Dear Reviewer,

Thank you for your insightful comments and questions. Please find our responses to your comments below.

HuggingFace has also released the PEFT library, which supports a wider range of pre-trained models and includes the methods mentioned in the paper. Therefore, what are the advantages of the LyCORIS library compared to PEFT?

We appreciate your question about the advantages of the LyCORIS library compared to HuggingFace's PEFT. PEFT indeed provides a wide collection of parameter-efficient fine-tuning methods. However, there are two key distinctions that set LyCORIS apart.

• LyCORIS is specifically designed to cater to users with less expertise in AI and Stable Diffusion, making it more accessible to a wider audience. In contrast, PEFT is tailored more towards expert users.
• In terms of fine-tuning of text-to-image models, LyCORIS covers more methods than PEFT. In fact, while PEFT initially supported LoRA, our library has incorporated other methods, with some of them, notably LoHa and LoKr, later got implemented by PEFT.

The paper conducted a multitude of experiments and comparisons on existing methods and various hyperparameters, leading to certain conclusions. Based on these findings, could there be a more optimal algorithm or design compared to previous ones?

We conducted extensive experiments and comparisons on existing methods and various hyperparameters. While these studies have yielded important insights, we agree that asserting the existence of a universally "optimal" algorithm is challenging. The effectiveness of any given approach largely depends on specific datasets and tasks. For example, as we indicate in the conclusion of the revised paper, our experiments seem to suggest that LoHa would fit better the case of fine-tuning for multiple easy concepts while LoKr could be more suitable for the case of fine-tuning for a single difficult concept. Besides this, our findings offer guidance on hyperparameter choices, while acknowledging the dynamic and context-dependent nature of performance in this field.

For this kind of paper that builds benchmarks based on a certain field, I would recommend the author to submit to a journal.

Your suggestion to submit our work to a journal is appreciated. While benchmarking is a significant contribution of our paper, it is not the sole one. Our work also introduces innovative approaches like LoHa and LoKr, which represent important advancements in the field. Therefore, while a journal submission is certainly a viable path, we believe our paper’s diverse contributions warrant consideration in its current format and venue.

Dear Reviewer,

Thank you for your encouraging comments and valuable feedback. We have addressed each of the points raised in your review as follows.

The results of this framework combined with ControlNet can be presented in this paper.

We appreciate your suggestion in combining our framework with ControlNet. This is an interesting idea. StabilityAi indeed implemented LoRA for ControlNet during the writing of this paper [1]. However, their dimension is relative large (256 and 128) due to the heavy training workload and task drift. The introduction of other methods implemented in LyCORIS for ControlNet may then be helpful in this regard. While these have not yet been included in our library, we acknowledge their potential and are committed to exploring this integration in future work.

[1] https://huggingface.co/stabilityai/control-lora

Efficiency (time and GPU memory cost) of different approaches are not provided and analyzed.

To address this point, we have included a summary of the time and GPU memory costs of different configurations in Table 5 of the revised version of our paper. It is worth noticing that both the actual VRAM usage and training efficiency can vary greatly depending on the implementation and the used hardware. For example, we have modified the implementation of native training in LyCORIS after the experiments, and this should lead to faster training compared to what we report in the table.

It will be better if the authors provide the results on other versions of stable diffusion, such as SD2.0 and SDXL.

Our library has added the support for other versions of Stable Diffusion, such as SD2.0 and SDXL. In the paper, we choose to focus on SD 1.5 due to its wider use and larger user base in the community. Moreover, due to page limitations, we are constrained in the number of experiments we can include. Nevertheless, we believe the experiment based on SD 1.5 already provides a substantial assessment of our framework's capabilities.