Locating What You Need: Towards Adapting Diffusion Models to OOD Concepts In-the-Wild

Part 3: Using 1000 data is not feasible (Q2)

Using all the 1000 data for training is not feasible for adapting OOD concepts. The experimental results of using all data are as follows:

As we see in this table, performing adaptation over the full dataset can lead to severe performance loss compared to using carefully selected ones. On Textual Inversion (TI), it even becomes worse than random sampling. This is due to that the full dataset contains lots of "bad" samples, which exhibit too many disruptive elements -- in turn -- introducing wrong details that mislead the concept adaptation.

Part 4: Generalizing CATOD to other customization methods. (Q3)

Thanks for the advice! CATOD can be easily generalized to ELITE and LyCORIS since it is designed as a general framework for the adapters. We show the results for applying CATOD to ELITE/LyCORIS on concepts "axolotl" and "emperor penguin chick" as follows:

From these results, we can also see that CATOD gives a consistent boost to ELITE and LyCORIS compared to other sampling strategies. Despite that CATOD is compatible with ELITE and LyCORIS, they do not exhibit a notable performance gain compared to LoRA.

Thank for the good words! We are happy that you enjoyed the paper!

Part 1: Why CMMD is a good metric (W1, Q1)

As reported in many recent works [1,2,3], the most popular image-matching metrics like Inception Score, Precision/Recall, and FID (Frechet Inception Distance) may disagree with human raters, thus ill-suited for evaluating recent text-to-image models. In comparison, CMMD uses CLIP embeddings and Maximum Mean Discrepancy (MMD) that correctly ranks the image sets based on the severity of the distortion. The limitations of FID and previous metrics can be listed as follows:

Also, we are glad to report our experimental results with FID, Precision, and Recall as follows based on CATOD adapted through LoRA:

[1] Jayasumana, Sadeep, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. "Rethinking fid: Towards a better evaluation metric for image generation." arXiv preprint arXiv:2401.09603 (2023).

[2] Grimal, Paul, Hervé Le Borgne, Olivier Ferret, and Julien Tourille. "TIAM-A Metric for Evaluating Alignment in Text-to-Image Generation." In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2890-2899. 2024.

[3] Lee, Tony, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, et al. "Holistic evaluation of text-to-image models." Advances in Neural Information Processing Systems 36 (2024).

Part 2: About the diversity evaluation (W2)

CATOD maintains the diversity to produce OOD concepts with different angles or poses in generative results. To validate the diversity of our generative results, we further provide a quantitative evaluation with LPIPS~[1,2]. The results are shown in the Table below:

In this table, we can see that CATOD gives out a diversity improvement up to 0.17 in the LPIPS score compared to CLIP-based sampling, and outperforms CLIP over most categories. From this, we conclude that CATOD also preserves diversity. Since the training set contains samples with different angles, it is also easy to produce objects with different angles. We will surely add a visualization of our selected samples and generative results with different poses/angles in our final revision.

Thank you for your comments and suggestions! We hope that our rebuttal addresses your concerns.

Part 1: About the diversity evaluation (W1, Q1)

CATOD maintains the diversity to produce OOD concepts with different angles or poses in generative results. To validate the diversity of our generative results, we further provide a quantitative evaluation with LPIPS~[1,2]. The results are shown in the Table below:

In this table, we can see that CATOD gives out a diversity improvement up to 0.17 in the LPIPS score compared to CLIP-based sampling, and outperforms CLIP over most categories. From this, we conclude that CATOD also preserves diversity. Since the training set contains samples with different angles, it is also easy to produce objects with different angles. We will surely add a visualization of our selected samples and generative results with different poses/angles in our final revision.

Part 2: Ablating initial size/quality in CATOD (W2, Q2)

It is intriguing to ablate CATOD over the initial training set size and quality! In detail, we retest CATOD with different numbers of initial samples (10,20,50) and different quality (high-quality and random sampling), with 100 images sampled at last and 10 images selected per cycle. These experiments are tested with adaptor LoRA. The results are shown as follows:

From these results, we may draw the following conclusions:

1. With HQ initial samples, the initial batch size does not have a significant compact, since a change in this size is not more than 0.3 in the CLIP score and 0.08 in the MMD score. However, this initial size has a significant impact with randomly initialized samples with at most a 5.71 loss in CLIP score and 0.49 in CMMD. We account this phenomenon for that randomly initialized images contain more bad-quality ones that mislead the adaptation.
2. The quality of initial samples does have an impact on generative results since we can see a consistent performance loss when changing HQ initial samples to randomly initialized samples. With the initial size increase, this impact tends to be even more significant.

The full results will be added in our revisions.

Part 3: Ablating the text-to-image encoder (W2)

Thanks for the advice! It is straightforward to extend our experiments to other text-to-image encoders (like SD 1.5 and SDXL). We show some of these results with LoRA in the tables below:

This table shows that our proposed CATOD is also compatible with different text-to-image encoders with notable performance. We will add these ablations to our final revisions.

Part 4: How the weight of different scores changes (W3, Q3)

Following Eq. (11) in the main context, the overall aesthetic score $\gamma_{aes}(A)$, overall concept matching score $\gamma_{con}(A)$​ and the corresponding weights on concept "emperor penguin chick" can be listed as follows:

During the first three cycles, concept-matching takes precedence over aesthetics because the OOD concepts are not yet fully learned by the adapter. As the cycles progress, concept-matching is quickly achieved. In the last two cycles, the adapter stabilizes and focuses more on enhancing aesthetics.

Part 7: Why CMMD is a good metric (Q12)

As reported in many recent works [1,2,3], most popular image matching metrics like Inception Score, Precision/Recall and FID (Frechet Inception Distance) may disagree with human raters, thus ill-suited for evaluating recent text-to-image models. In comparison, CMMD uses CLIP embeddings and Maximum Mean Discerpancy (MMD) that correctly ranks the image sets based on the severity of the distortion. The limitations of FID and previous metrics can be listed as follows:

[1] Jayasumana, Sadeep, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. "Rethinking fid: Towards a better evaluation metric for image generation." arXiv preprint arXiv:2401.09603 (2023).

[2] Grimal, Paul, Hervé Le Borgne, Olivier Ferret, and Julien Tourille. "TIAM-A Metric for Evaluating Alignment in Text-to-Image Generation." In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2890-2899. 2024.

[3] Lee, Tony, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang et al. "Holistic evaluation of text-to-image models." Advances in Neural Information Processing Systems 36 (2024).

Part 8: Other minor questions

Q1 (About Figure 1): Yes, the CMMD in Figure 1 is calculated before adapting. This is used to show how out-of-distribution a concept is before adapting.

Q2 (About Figure 2): Yes, the high-quality samples in the third part of Figure 2 are selected by the CATOD framework.

Q4 (About Eq. 6): Apologies for the confusion! Since both the aesthetic and concept-matching scores range from 0 to 10, Equation 6 monotonically increases according to both variables, with both sine functions ranging from 0 to 1. We will ensure that the scores $\gamma_{aes}$ and $\gamma_{con}$ are clearly stated to range from 0 to 10 in future revisions.

Q6 (About the epochs to use): Training for 20 epochs is common for adaptors such as LoRA, DreamBooth, and other related methods, as they typically require much less data (usually around 100 samples per concept). Training for too many epochs can lead to the underlying model replicating the images as exact copies.

Q7 (About algorithm 1): We don't need to select new samples at each training epoch. The selection is performed in lines 2-6, prior to training. The algorithm demonstrates the process of a single Active Learning cycle, while training CATOD involves several cycles. The selection is conducted at the beginning of each cycle.

Q8 (About the schedule): R includes 5 learning rates: $5\times 10^{-4},2.5\times 10^{-4},7.5\times10^{-5},5\times 10^{-5},2.5\times 10^{-5}$. Following Q7, we calculate the indicator $\gamma(A)$ after every epoch. When this indicator falls below the previous evaluation, we reduce the learning rate. If the learning rate cannot be lowered further, we conclude that the model has converged and the training is complete.

Part 9: Some typo errors (According to Q3)

Thanks for pointing out the mistake! In lines 173 and 174, the sentence should be "if $r_k$ is the closet representation for $r_x$".

Thank you for your comments and suggestions. We have carefully addressed your concerns as follows:

Part 1: About the key technical contribution (W1)

The key technical contributions of this paper are twofold: (1) our method for estimating the impact of each sample on the model without training, corresponding to the aesthetic/concept-matching scorer described in Section 3.2; (2) our approach to effectively utilizing these scores and balancing the trade-off between the two impact factors (Eq. 6 and Eq. 7). This is achieved through Active Learning, a vital paradigm that enables dynamic trade-offs cycle-by-cycle.

Part 2: A more detailed explanation for the theories (W2, Q5)

We are pleased to provide a detailed explanation of the connection between our proposed method and the underlying theories!

As introduced in our background Section (Sec.2), the LDMs consist of two core components: An encoder $\mathcal{E}$ learns to map images $x$ to a latent code $z=\mathcal{E}(x)$, and a diffusion model trained to produce code conditioned on texts. Let $c\_\theta(y)$ be a model mapping the conditioning text $y$ into latent vectors, $A$ the adapter embedded into the diffusion model, the LDM loss is given by: $L\_{LDM}(x,A)= \mathbb{E}\_z\sim\mathcal{E}(x),t,\epsilon\sim\mathcal{N}(0,1)\left[\Vert \epsilon-\epsilon\_{\theta, A}(z\_t,t,c\_{\theta, A}(y)) \Vert\_2\^2\right]$ in Eq.1, where t is the time step, $z\_t$ the latent noised to time $t$, $\epsilon$ the noise sample, $\epsilon\_\theta$ the denoising network. When optimizing the LDM loss, there are two important factors we should consider: the adapter $A$ that adapts the underly model to OOD image $x$, and the conditional text $y$ for the images $x$ to match. As we claimed in the main context that disrupting $x$'s can lead to misunderstanding of concept $y$, our objective is in Eq.2:

$A^*,X\_{T}^*=\mbox{argmin}\_{A,X\_T} L\_{\mbox{LDM}}(x,A,y)$.

Since the optimal set for $A, X_T$ is initially unknown and cannot located at once, is common to use an iterative paradigm that alternatively optimizes $A$ and $X_T$. We have the following conjugate forms:

$\mathbf{X}\_T\^{(t)} = \mbox{argmin}\_{\mathbf{X}\_T\subset D\_{pool}}\mathbb{E}\_{x\sim \mathbf{X}\_T}L\_{LDM}(x,A\^{(t-1)},y)$ (Eq.3 in main context)

$A\^{(t)} = \mbox{argmin}\_{A}\mathbb{E}\_{x\sim \mathbf{X}\_T\^{(t-1)}}L\_{LDM}(x,A,y)$ (Eq.4 in main context)

For the first equation (Eq.3), we optimize the training $X_T$ towards both conditional text $y$ (image-text matching) and loss reduction (aesthetic preserving), which in turn leads to the motivation of decoupling aesthetic/concept-matching scores. Theorem 4.3 verifies the necessity of this decomposition while Section 3.2 implements this decomposition. The second equation (Eq.4) just performs an ordinary adaptation on select samples.

Since the iterative paradigm in Eq. 3 and Eq. 4 may lead to convergence to local optima, we also need a dynamic weighted scoring system to trade-off between aesthetics and concept-matching, to alleviate the potential bias towards only one of them (leading to the practical implementation in Sec 3.3). To this end, we have explained the motivation of our theory and how we organize our CATOD.

Part 3: The pre-calculated scores are not feasible (W3)

Since Out-of-Distribution (OOD) concepts are usually unseen by both the underlying text-to-model and the aesthetic/concept-matching scorers, it is likely that the preset scorers provide inaccurate results. To address this, we must use the currently available high-quality samples to correct these inaccuracies. Following the iterative paradigm outlined in Part 2, the scorers and adaptors are alternately adjusted to ensure accurate results. Additionally, we update the scorers after selecting high-quality samples in each cycle. To demonstrate this experimentally, we conduct another ablation experiment with pre-calculated scores, as shown in the following table:

We observe a noticeable performance loss when the scores are pre-calculated.

Part 4: Other architectures (Q9).

Thanks for the advice! It is straightforward to extend our experiments to other text-to-image models (like SD 1.5 and SDXL). We show some of these results with LoRA in the tables below:

This table shows that our proposed CATOD is also compatible with different text-to-image encoders with notable performance. We will add these ablations to our final revisions.

Part 5: The impact of initial size/quality in CATOD (Q10)

Thanks for the advice! It is intriguing to ablate CATOD over the initial training set size/quality. In detail, we retest CATOD with different numbers of initial samples (10,20,50) and different quality (high-quality and random sampling), with 100 images sampled at last and 10 images selected per cycle. These experiments are tested with adaptor LoRA. The results are shown as follows:

From these results, we may draw the following conclusions:

1. With HQ initial samples, the initial batch size does not have a significant compact, since a change in this size is not more than 0.3 in the CLIP score and 0.08 in the MMD score. However, this initial size has a significant impact with randomly initialized samples with at most a 5.71 loss in CLIP score and 0.49 in CMMD. We account this phenomenon for that randomly initialized images contain more bad-quality ones that mislead the adaptation.
2. The quality of initial samples does have an impact on generative results since we can see a consistent performance loss when changing HQ initial samples to randomly initialized samples. With the initial size increase, this impact tends to be even more significant.

The full results will be added in our revisions.

Part 6: Evaluating the quality of ID concepts after adapting OOD concepts. (Q11)

In brief, our method will not cause much degradation in non-OOD concepts. To prove this, we list a comparison of the performance of the following non-OOD concepts (corresponding to Q1) over SD 2.0 when fine-tuning with LoRA. The results are shown in the table below:

From this table, we can see that our proposed CATOD only leads to a performance loss of at most 0.15 on the CLIP score and at most 0.12 on CMMD on the non-OOD concepts after fine-tuning. This proves that our method does not distort non-OOD concepts when adapting native models to OOD concepts.

Thanks for your comments!

Part 1: About the number of samples to use for training and validation.

In brief, since most recent adaptors require only a small number of training samples (usually around 100), our validation set is also set to be of the same scale as the training samples.

As mentioned in the paper, adaptors are designed to reduce training costs when introducing new concepts. Specifically, they require significantly fewer samples compared to direct fine-tuning, typically no more than 1000 samples, and often around 100 samples. The following table provides a comparison of the most commonly used adaptors (including our proposed CATOD):

Since previous adaptors often fail to accurately depict out-of-distribution (OOD) concepts due to inconsistent image quality, we also ensure that our validation set is of high quality to provide accurate validation.

[1] Gal, Rinon, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. "An image is worth one word: Personalizing text-to-image generation using textual inversion." arXiv preprint arXiv:2208.01618 (2022).

[2] Ruiz, Nataniel, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. "Dreambooth: Fine-tuning text-to-image diffusion models for subject-driven generation." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500-22510. 2023.

[3] Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).

[4] Yeh, Shih-Ying, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. "Navigating text-to-image customization: From lycoris fine-tuning to model evaluation." In The Twelfth International Conference on Learning Representations. 2023.

Part 2: About the number of categories to use for the adaptors.

Thank you for reminding me to investigate the capacity of adaptors in learning categories with OOD concepts! Extending CATOD to encompass even more categories is straightforward. Using the category "Insect" as an example, we explored the impact of adding three additional OOD concepts. Specifically, we conducted experiments with the following additional categories: "Chlumetia transversa," "Mango flat beak leafhopper," and "Rhytidodera bowrinii white." The table below shows the average performance in the "Insect" category as we add new concepts:

From this table, we can see that while CATOD can be easily generalized to more concepts, it experiences a performance decline as the number of categories increases. This decline is because recently proposed adaptors were originally designed to accommodate only a few concepts. However, since CATOD is designed as a general framework, it can be easily adapted to work with any future adaptors that are capable of handling a larger number of concepts.