ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance

Q1&W1: The rationale of using CLIP to calculate Loss

A1: Thank you for your comment. Actually, I lean towards that they are two isolated problems.

CLIP-T is a metric that is used to evaluate the similarity between image-text pairs. Prior studies [1, 2] have demonstrated that CLIP models encounter challenges in distinguishing fine-grained differences in images, primarily due to the nature of their pre-training strategies. Consequently, some mismatched image-text pairs (e.g. prompt is a dog with a smile while the image is a dog without it) may still receive high CLIP-T scores.

However, our approach, SPL, leverages CLIP to calculate the similarity between two text prompts. Since SPL operates within the same modality (text), it remains unaffected by the cross-modal gap problem encountered in CLIP-T. Besides, in order to get the embeddings embedded in the Stable Diffusion(SD)'s semantic space, which is beneficial for preserving the original semantic space, we use the text encoder of SD, specifically CLIP, to extract text embeddings. We will update it in our latest version of the manuscript.

[1] Tong, Shengbang, et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Wang, Wenxuan, et al. "Diffusion feedback helps clip see better." arXiv preprint arXiv:2407.20171 (2024).

Q2&W2: Ablation experiments on SPL weight

A2: Thank you for your valuable feedback. In our manuscript, we conducted a qualitative ablation study to examine the effect of different SPL weight settings, as shown in Fig. 8. Based on these results, we found that with 500 training steps, an SPL weight of 1.0 yielded the best performance since it well follows the text prompt while customizing given dogs. Additionally, we performed a quantitative experiment to further explore the impact of various SPL weight settings. The results of this analysis are presented in the table below. This result shows that CLIP-T becomes higher (increase in the ability to follow prompts) and DINO-I decreases(decrease in the ability to customize concepts) with increasing SPL weight, which is consistent with our expectations. Meanwhile, we find that SPL is a loss function that is insensitive to weight. Combined with the qualitative observation in Fig.8, we prefer to choose 1 as the SPL weight. We will update this quantitative result in the latest version of our manuscript.

Q3&W3: Significance of Fig.2

A3: Thank you for your insightful feedback. In our manuscript, Figure 1 illustrates the concept of subject-driven personalized generation and highlights the limitations of tuning-based models. In Figure 2, we aim to demonstrate the effectiveness of our model through a story involving a dog and a pair of sunglasses, showing that our model can preserve different subject concepts well， further showcasing a potential real-world application of our approach. Additionally, we acknowledge that some figure captions may have been unclear, potentially leading to misunderstandings. In the latest version of our manuscript, we have revised certain figures and their corresponding captions to enhance clarity.

Q4: Why CLIP-T cannot reflect the actual performance

A4: Thanks for your valuable feedback. Basically, CLIP-T calculates the similarity between image-text pairs. However, existing works[1, 2] find that CLIP models can hardly tell the fine-grained differences in images, giving a mismatched image-text pair high score consequently. As a result, the CLIP-T score can not fully reflect the performance of the results. We will update this to the latest version of the manuscript to make this section more convincing.

[1] Tong, Shengbang, et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Wang, Wenxuan, et al. "Diffusion feedback helps clip see better." arXiv preprint arXiv:2407.20171 (2024).

Q1&W1: Ablation on components of SPL

A1: We have a qualitative ablation on the weight of semantic preservation loss(SPL) in our manuscript. Also, we agree with your assertion that a more comprehensive quantitative ablation experiment on its weight may be beneficial for demonstrating the impact of SPL, which shows Experiment results are shown below. The results demonstrate that as the SPL weight increases, CLIP-T scores rise (indicating better prompt-following ability) while DINO-I scores decline (showing reduced concept customization ability). This trend aligns with our expectations. Notably, we observe that the SPL loss function exhibits low sensitivity to weight variations. Based on these findings and the qualitative results shown in Fig.8, we selected 1 as the optimal SPL weight. we update this result in the latest version of our manuscript.

Q2: More quantitative experiments

A2: We believe adapting some quantitative metrics beyond CLIP-T, CLIP-I, DINO-I, and BLIP2-T is an instructive and interesting idea though it is not so common in the field of subject-driven personalized generation. We adopt a widely used method called TIFA[1] to evaluate the compositional ability of models.

Experiment results are shown below, we update this result to the latest version of our manuscript.

[1] Hu, Yushi, et al. "Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Q3: More comparison with additional baseline that focus on similar problems

A3: We have added Prospect[1], a state-of-the-art baseline for subject-driven personalized generation, to demonstrate our model's effectiveness. The quantitative results below show that our model achieves superior performance. We have included these results in our manuscript's appendix. For diverse cases, Fig.2, Fig.6, Fig.7 and Fig.13 show some qualitative results beyond the dog-headphone example, showing the effectiveness of our proposed method.

[1] Zhang, Yuxin, et al. "Prospect: Prompt spectrum for attribute-aware personalization of diffusion models." ACM Transactions on Graphics (TOG) 42.6 (2023): 1-14.

W2: Exploration in video generation

A4: Thank you for your comments. We conducted an experiment on subject-driven personalized video generation to demonstrate that ClassDiffusion can generate personalized videos while preserving the semantic space—without requiring additional computational resources for fine-tuning. Also, we believe that for almost all methods in video generation that obtain video generation ability by combining 2D stable diffusion weights with time adapters, our method achieves substantial results in video generation without the need for extra fine-tuning computations. In this way, we can use 2D fine-tuned parameters in image-driven video editing or some other tasks.

Q1-1: Rationale behind the loss choice

A1-1: We refine the explanation of our proposed method in $\S$3.4 to explain the rationale behind the loss choice more clearly. It‘s mainly because according to our experimental observation, one perspective to solve this problem lies in preserving semantic information during fine-tuning to reduce the difficulty of sampling from the joint conditional distribution. To address this, we propose a novel loss function aimed at constraining semantic variation throughout the fine-tuning process. So we design a loss function to reduce the change in the semantic distance between class prompt(e.g. a photo of a dog) and training prompt(e.g. a photo of a V* dog).

Q1-2: Clarity in embedding visualization

A1-2: Thanks for your valuable feedback. We refine our manuscript by adding the following details about embedding visualization. The modification is shown below.

First, we collect 70 phrases that combine adjectives with words representing the class of given concepts (e.g., "a happy dog" or "a cool dog"). Using the CLIP text encoder, we extract text embeddings for these phrases. To visualize the semantic space and intuitively track modifications within it, we use t-SNE to reduce these high-dimensional embedding vectors to 2D. An overview of this result are shown in Fig.3(a), meanwhile we show the real experimental results in Fig.12.

W1: Different Fine-grained concepts

A1: Thank you for your thoughtful remarks. We completely agree that developing methods for addressing fine-grained customization is an important direction of research. However, we would like to clarify that our proposed method is not specifically designed to address fine-grained customization problems.

Besides, they are two isolated problems since we found the problem of reduced combinatorial power to the increased difficulty of sampling the joint probability distribution of the two classes, yet the fine-grained generation problem is an intraclass probability distribution problem.

Therefore, we believe that fine-grained customization and the reduction of compositional abilities after personalized fine-tuning represent distinct challenges that require fundamentally different approaches. In our work, we focus on addressing the reduction in compositional abilities observed after the personalized fine-tuning process. Through our experimental and theoretical analyses, we identified the primary issue as the increased difficulty in sampling from the joint conditional distribution. This observation motivated us to propose the Semantic Preservation Loss (SPL), which directly addresses this challenge. We hope this explanation helps to clarify our focus and the contributions of our work.

W2: Multiple concepts ($\ge$ 3)

A2: We conducted quantitative experiments on customizing multiple concepts and added the results to our manuscript's appendix. The experiments demonstrate that ClassDiffusion generates high-quality results when combining multiple concepts, validating the effectiveness of our proposed methods. Please refer to Figure 13 for the experiment result.

W1: SPL may struggle to restore the original semantic space

A1: I completely agree with your perspective that semantic preservation loss (SPL) cannot fully restore the original semantic space. In fact, I believe that even minor alterations in the semantic space are exceedingly challenging to fully restore due to their occurrence within a high-dimensional space. Consequently, restoring the original semantic space solely through simple regularization is inherently difficult. Moreover, a completely unchanged semantic space may hinder the customization of specific concepts, since we need to fine-tune the embedding to adapt for given concepts. Our SPL is designed to preserve semantic space in terms of minimizing the semantic distance between the class prompt(e.g. a photo of a dog) and the training prompt(e.g. a photo of a V* dog). In this way, though changes still happen in semantic space which might be inevitable, we are successful in keeping the model's ability of following prompts.

W2: Further explanation of the proposed method

A2: We construct the exposition of our proposed method from high level (the design concept of our proposed method) to low-level technical details (the computational method of our proposed method to the formalized formula) to express our method more clearly.

According to our experimental observation and theoretical analysis, The key problem is that it is more difficult to sample from the joint probability distribution of the two classes due to semantic drift. Therefore, the core challenge addressed is preserving semantic information during fine-tuning to simplify sampling from the joint conditional distribution. To tackle this, a novel semantic preservation loss (SPL) is proposed. This loss minimizes the semantic distance between the target concept phrase (e.g., "a photo of a V* dog") and the general class phrase (e.g., "a photo of a dog"). Using Stable Diffusion’s text encoder, embeddings for these phrases are computed, and their cosine distance is used to calculate SPL. The final loss function combines SPL with the reconstruction loss, weighted by a factor $\lambda$, ensuring the fine-tuning process aligns customized token embeddings with their respective target concepts while maintaining semantic consistency.

Also out of consideration for clarity, we separate method design and implementation details into different sections. Modifications are conducted in Sections 3.4 and 4.1, please refer to our latest manuscript for these modifications.

Q3: Caption & Clarification on Figure

A3: We refine the caption of Figure 2 in the latest version of our manuscript. Figure 2 presents An example of a complete narrative. For example, the above showcases a bear’s literary journey: from reading a book to ultimately earning a Nobel Literature Prize. This example highlights a potential real-world application enabled since our model can combine customized concepts with other elements. Modification is shown below

Figure.2 A qualitative result of two small stories produced by our model. The above showcases a bear's literary journey: from reading a book to ultimately earning a Nobel Literature Prize. The below shows the fate of sunglasses. Finally, the bear gets the sunglasses. It shows a potential real-world application due to our model's high performance.

W4: Refinement on figures and tables.

A4: We totally agree that refinements on figures and tables in the manuscript will be beneficial for clarity. As we stated above, we found some figures and captions under the consideration of clarity. Besides, we will refine the presentation of the user study's result in Table 1 and the corresponding caption in the latest version of our manuscript.

Dear Reviewer pV4W,

Thank you for your kind acknowledgment. We are delighted to know that all concerns have been addressed to your satisfaction.

Best regards, The Authors

Many thanks to the authors for their detailed response, which has addressed most of my concerns. This paper proposes an efficient method to overcome the overfitting, resulting in failure to create the concept under multiple conditions. Authors perform an in-depth study of the properties and applications of the base model. In this paper, a more important observation is that the compositional ability only disappears after personalization tuning. Equipped with this better understanding, this paper leverages these properties to solve a diverse set of tasks. I raise my score.