Mining your own secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models

Dear Reviewer w18g,

Thank you for your comments and suggestions. We have tried our best to address your concerns below.

1. W.1: A summary algorithm to improve clarity: Thank you for your suggestion. We have accordingly updated the pdf with a summary Algorithm 4 in Appendix A that shows how DSC and EWC work together during optimization using Algorithms 1, 2, and 3 as references.

2. W.2: On the necessity of DSC: Thank you for your comment. First, from an empirical viewpoint, we note that DSC + EWC + DC outperforms EWC + DC on all our setups across majority of the metrics. The performance gain of the former over the latter varies across setups with DSC + EWC + DC surpassing EWC + DC significantly on the setups of Custom Concept (Table 1), Textual Inversion (Table 3), and Celeb-A (Table 4). Our additional experiments on Stable Diffusion v2.0 (mentioned in the rebuttal to Reviewer 76RN) also show consistent performance gains of DSC + EWC + DC over EWC + DC. Second, from an intuitive viewpoint, EWC is not enough if we want to retain knowledge from some specific previous task LoRA . This is because with EWC, we maintain a task-shared Fisher information matrix that continuously updates the importance scores of LoRA parameters based on the training data from each new task. On the other hand, with the DSC framework, we can be more flexible at controlling which task’s knowledge do we want to distill more. This can simply be achieved by replacing the uniform sampling for teacher-2 with a weighted sampling strategy such that the desired task’s LoRA has a higher chance of being sampled as teacher-2 (on line 5, Algorithm 3 in the main paper). As such, DSC can be clearly seen to be adding performance and flexibility gains to our consolidation framework.

3. W3: On the computational complexity and comparison with C-LoRA: First, we argue that C-LoRA is problematic in Sec 3.1, and hence is not a suitable choice for parameter-space consolidation – our results across all dataset setups show that C-LoRA exhibits catastrophic forgetting across several tasks. Second, per your suggestion, in Table 10 (Appendix I) in the updated pdf, we have compared the wall clock time for a single iteration of C-LoRA with our proposed algorithms using different values of the subset of seen concepts k on the latter. In line with our theoretical training time complexity analyses from Sec. 4.3, we find that as the number of tasks grow, the training time gap per iteration between C-LoRA and our proposed methods bridges. For instance, on the 50th training task, the time required per training iteration of our proposed algorithms is almost half of that of C-LoRA's, e.g., for k = 2, C-LoRA requires 3.8s while EWC + DC requires 1.36s and DSC + DC requires 1.8s. For the ease of readability, we provide the Table 10 comparison results of training time per iteration (wall clock time in seconds) below too:

4. Q.1: On the updates of $p_\theta[c^i]$ in Algorithm 1: Thank for your comment. $p_\theta$ is indeed a probability distribution over n+1 concepts (n personalization concepts and one pre-trained concept). However, as mentioned in Line 5 Algo. 1, for one consolidation iteration, we only seek to update the probability values for k out of n+1 concepts, where k is the subset of seen concepts that are relevant for computing DC scores (shown in lines 4 and 9 in Algo. 2 and Algo. 3, respectively). As a result, even though Line 5 in Algo 1 initializes probability values for all n+1 concepts with a very small non-zero score of $ω$ = 1e-10, it is only the k concepts for which the probability scores are later updated (to a larger value) on line 6.

Hence, initialization of DC scores dictionary serves two-fold purposes. First, initializing the probability scores for all n+1 concepts ensures that the model gets to see a consistently ordered probability distribution for all n+1 concepts across iterations. Second, using a very small yet non-zero value for $ω$ helps us avoid any potential numerical instability that could result from taking the logarithm of the zero-valued probability scores while computing the cross-entropy losses for EWC (Algo. 2 , line 10) and DSC (Algo 3, line 12).

Dear Reviewer 76RN,

Thank you for your comments and suggestions. We have tried our best to address your concerns below.

1. On the novelty of our work: Thank you for your remark. While our approach does build upon techniques like Elastic Weight Consolidation, the novelty of our work lies in the innovative application of diffusion scores as a regularization term for continual personalization in diffusion models—a connection not previously explored. From a broader standpoint, our approach advocates that going beyond the previous applications for test-time classification [1,2], the pre-trained class-specific representation knowledge of diffusion models can further be reinforced through thoughtfully designed finetuning frameworks which in turn can help enhance the performance of downstream generation tasks. Given that our thorough empirical analyses validate the effectiveness of our method and that no other work has explored the utility of diffusion classifier scores for enhancing continual personalization of diffusion models, we believe that our approach is ipso facto novel [3].

2. Experiments with Stable Diffusion 2.0: As we follow the setup for C-LoRA paper, which in turn leverages the Custom Diffusion paper setup [4] for training, we choose Stable Diffusion v1.4 to reproduce their setting as closely as possible. Also, we note that v1.4 and v1.5 remain architecturally very similar in that they both leverage the CLIP ViT-L/14 text encoder and a U-Net capable of processing 64x64 latent representations corresponding to 512x512 images. As Stable Diffusion v2.0 improves upon these (with better text encoder and support for higher resolution images), we report the results for v2.0 on our Custom Concept setup here as well as in Appendix Table 7 in the updated pdf:

References:

[1] Li et al. "Your Diffusion Model is Secretly a Zero-Shot Classifier”. ICCV 2023.

[2] Clark et al. "Text-to-Image Diffusion Models are Zero-Shot Classifiers". NeurIPS 2023.

[3] "Novelty in Science," Medium blog post by Michael Black, https://medium.com/@black_51980/novelty-in-science-8f1fd1a0a143

[4] Custom Diffusion results are based on Stable diffusion v1.4: https://github.com/adobe-research/custom-diffusion?tab=readme-ov-file#results

2. Experiments with Stable Diffusion v2.0 (Continued): We report the results for v2.0 on our Waterfall landmark setup here as well as in Appendix Table 8 in the updated pdf:

From the above results, we note that our variant using Diffusion Classifier (DC) scores with EWC and DSC performs consistently the best on both the dataset setups (highlighted in bold) with Stable Diffusion v2.0.

3. Results on all Stable Diffusion versions and on more parameter-efficient finetuning methods: We would like to state that text-to-image diffusion is a rapidly evolving field, as is the field of parameter-efficient finetuning itself. Given this fast-paced environment, it is challenging to provide an exhaustive comparison with every single version of the Stable diffusion model and every recent parameter-efficient finetuning method. While we strive for comprehensive analysis, we hope the reviewer understands that an all-encompassing comparison may not be practically achievable within the scope of this work. That said, we have reported the results with VeRA [1] at the end of Sec. 4.3 in the main paper, since it has a more flexible support within the Diffusers library than LyCORIS and DoRA. To summarize, our analyses demonstrate the effectiveness of our proposed method for two Stable Diffusion versions: v1.4 and v2.0, and for two parameter-efficient finetuning adapters: LoRA and VeRA.

4. Further experimental results on multi-concept generation: Thank you for your comment. Per your suggestion, we have updated the paper with the results for sequentially trained Custom Diffusion (CD) on multi-concept generation (see Fig. 17b in Appendix G.2). We have also provided the quantitative scores for multi-concept generation in Table 6, and have detailed (in blue font) on how we compute these scores (given that we do not have ground truth images that incorporate multiple concepts) in Appendix G.2. In line with our main paper findings (Sec. 4.1) on single-concept generation, we see that CD has forgotten the custom category attributes (color, background, etc.) and instead generates multi-concept images that exhibit the model’s pretrained knowledge about the subject (e.g. humans posing with the plushie tortoise in front of the waterfall). For the ease of readability, we mention the quantitative results of Table 6 below (bold refers to the best scores):

References:

[1] Kopiczko et al., "VeRA: Vector-based Random Matrix Adaptation". ICLR 2024.

11. Q.4: Specific examples of $c_0$, $c_n$, and $V_n$: We mention the details for $c_0$ being an abstract pretrained concept “person” in our implementation details in App. H.1. The examples for n-th task concept $c_n$ are comprehensively listed in App. D, where we provide the concept names for all our datasets. Finally, Sec. 3 “How do we structure our CL framework for DC scores?” mentions that $V_n$ is the word vector for the n-th task concept $c_n$ following the Custom Diffusion paper.

4. W.4: Selection of k (number of sampled tasks) for Figure 16: We first note that our consolidation stages for n-th task leverage only the n-th task training data, i.e., in a replay-free setup. As such, we must use the n-th task LoRA and its corresponding n-th task concept $c_n$, i.e. k = 1. Next, following Custom Diffusion and C-LoRA, we also leverage the readily available pretrained concept $c_0$ that each LoRA has learned, i.e., k = 2. For k = 3 onward, we find that increasing k up to a certain point helps improve the generation quality as the DC scores carry more discriminative information with more classes. Namely, we find that the generative quality saturates with k > 5 since not all previous concepts carry useful discriminative information for classification. Hence, an intermediate value of k helps strike a good balance between discriminative information and inter-class confusion. Lastly, although we perform this selection of k on the six task setup of the Custom Concept dataset, we find k = 5 to be working well on the 10 tasks setup of Landmarks v2/Celeb-A, as well as on the 50 tasks setup (Sec. 4.2) of Custom Concept.

5. W.5: Editorials: Thank you for pointing this out. Our notation assumes each concept to be a pair of a class label and its respective input images. Therefore, we refer to concepts and class labels interchangeably throughout the manuscript (we will mention this explicitly in the final manuscript). On this note, we have updated line 103 to read “ .. a text prompt encompassing $c$” rather than “ .. a text prompt $c$” (highlighted in blue). We have also updated line 245 as “.. the one-hot ground truth for concept $c_n$ rather than “.. the one-hot ground truth $c_n$”. Lastly, per your suggestion, we have changed the superscripts for $c$ to be subscripts instead.

6. W.6: Equation 5 fix: Thank you for the suggestion, we have updated Eq. 5 accordingly.

7. W.7: On the use of 'secret' in the title: The keyword ‘secret’ emphasizes that conditional diffusion models, while primarily designed for generative tasks, inherently encode class-conditional representations that can be used to estimate the likelihood of data belonging to different classes. Consequently, we propose to use this classification score of text-to-image diffusion models as a regularization basis for continual concept learning. We also note that previous works have used the word 'secret' to emphasize on the classification ability of diffusion models [1].

8. Q.1: On the limitation of C-LoRA: Thank you for your comment. We believe that there has been a misunderstanding regarding the behavior of L_forget in C-LoRA. You are partly correct in that a straightforward interpretation of L_forget being close to zero indeed appears like no forgetting is happening. However, our limitation discussion in Sec 3.1 further pinpoints a degenerate case of C-LoRA, where L_forget is close to zero because the learned weights for LoRA are pushed toward zero (see Fig.1b). This implies that there is no learning happening, in addition to the model having forgotten the previous concept (since most LoRA matrix spots have been edited to approach near-zero values). This is especially evident for task 2 training curves where L_forget decreases throughout training and thus edits most of the learned task-1 LoRA weight spots while pushing these toward zero.

9. Q.2: Difference between Sec. 3.1 results and the proposed method: We believe that this is another case of misunderstanding related to the misinterpretation of C-LoRA's L_forget loss in #8. There is indeed a difference between C-LoRA results and that of ours in the sense that we do not leverage the L_forget loss (given its flaw) shown in Fig. 1 as an optimization objective in our framework. We therefore do not show these curves for our method.

10. Q.3: How is $V_n$ obtained and used in the training? Thank you for your comment. In line 172, we also mention that we follow the setup of Custom Diffusion [1] which in turn leverages the setup of Textual Inversion [2] to learn a new textual token embedding $V_n$ per concept. Since this has been a standard setup for finetuning diffusion models [2,3,4], we do not dive into much details about how $V_n$ is obtained in the paper. However, following your comment, we have now briefed on this clearly in Appendix A (highlighted in blue).

References:

[1] Li et al. "Your Diffusion Model is Secretly a Zero-Shot Classifier”. ICCV 2023.

[2] Kumari et al. "Multi-Concept Customization of Text-to-Image Diffusion". CVPR 2023.

[3] Gal et al. "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion".

[4] Smith et al. "Continual diffusion: Continual customization of text-to-image diffusion with C-LoRA". TMLR 2024.

Dear Reviewer CPEf,

Thank you for your comments and suggestions. We have tried our best to address your concerns below.

1. W.1: Variance in approximation of expectation: While single trial per batch may introduce some bias in Monte Carlo estimate, this bias is averaged out over many batches, as each timestep has roughly an equal probability of being selected. Accordingly, we provide Fig. 22 in our updated pdf to show that our number of consolidation iterations (400 for the Celeb-A setup and 200 for the rest setups) remain enough to ensure that all timesteps are represented. One special case we note is that for fewer number of consolidation iterations (Fig. 22d), the intermediate timesteps have a relatively higher sampling chance. However, under limited timestep evaluations per class, previous works have shown that diffusion classification scores remain most sensitive to the conditioning signal for the intermediate timesteps [1]. Hence, a higher concentration of probability mass on the intermediate timesteps favors classification accuracy when the number of consolidation iterations remains limited.

• Lastly, our empirical results in App. H.2.1. show that the generative quality (quantified in terms of Average MMD and KID scores) first increases as the number of consolidation iterations is increased to roughly 0.2 times that of the total training iterations and then saturates.

• More analyses for FIM updates: Per your suggestion, in Fig. 19 in the updated pdf, we compare the effect of varying the number of consolidation iterations on FIM estimation. Namely, we observe that as the number of consolidation iterations increase from 100 to 200 leads to larger magnitudes of top-5 Eigenvalues, thus indicating that the learned parameters become more certain about the directions of high loss changes from data, i.e., reduced data-driven uncertainty in the FIM estimation. However, increasing the number of consolidation iterations beyond 200 does not lead to any further growth in the top-k Eigenvalue magnitudes thus indicating a saturation point in data-informed uncertainty estimation for the FIM. We note that this observation is in line with 200 being our optimal number of consolidation iterations on the Custom Concept setup found via hyperparameter sweep (Appendix H.2.1).

2. W.2: Loss component interactions: We are sorry for the confusion caused by a lack of an explicit study on how the performance varies with the loss component interactions. We have updated the pdf with Appendix H.2.4 to show the effect of varying the hyperparameters δ and γ that control the strength of matching the DC scores in the loss terms for EWC (Eq. 5) and DSC (Eq. 6), respectively. In particular, we find that the performance variation of the DSC EWC DC model due to changes in [$δ$, $γ$] hyperparameters is consistent with the individual performance changes due to variations in $δ$ and $γ$ as reported in Fig. 23 and 24, respectively. That is, we find that a value close to 0.1 remains optimal for $γ$ while $δ = 0.5$ performs the best. Moreover, a larger $γ$ remains detrimental for the purpose of generation, marked by consistently higher KID and A_MMD scores. This observation is consistent with that seen in Fig. 24 given that a larger $γ$ can lead to the student consolidation to be influenced more by the discriminative DC scores and lesser by the denoising score matching score. This in turn harms the generation quality of the images, with an extreme case shown in Fig. 11 where the generated images can turn unintelligible in case of zero guidance of denoising scores during consolidation.

3. W.3: The link between classification and generative quality: For Sanity Check II, we classify the training set points using the diffusion classifier (DC) scores with a single trial and the 500th timestep following [1]. You are correct that improved classification alone does not imply an improvement in the generation quality. Instead, as mentioned in Sec. 4.3, this sanity check is designed to ensure that our consolidated model does leverage DC scores. Namely, if the improvement in the generation quality of the model comes from consolidation losses other than the DC scores, i.e., either the DC scores are not being leveraged during consolidation or the derived DC scores are unreliable, then we do not expect a significant improvement in the training set classification (as can be seen with EWC and C-LoRA). On the other hand, if the model trained on all tasks has been consolidated with reliable DC scores from their training datasets, then we expect it to improve on the classification scores of these training samples.

References:

[1] Li et al., "Your Diffusion Model is Secretly a Zero-Shot Classifier". ICCV 2023.