ConceptFlow: Unified Framework for Personalized Image Generation

Q4. Time efficiency and computational overhead

We provide the time efficiency and computational overhead of ConceptFlow in the Tables below:

Single concept learning:

We adopt an image adapter from DisenBooth and we store the attention maps of concept tokens for attention regularization (AR). The training time and the training memory of ConceptFlow-S are reported in the table below.

Multiple concepts generation:

ConceptFlow-M actually has to generate $(n + 1)$ images ($n$ reference images and 1 target image) at the same time for a prompt that involves $n$ concepts. We accomodate a loop for reference images generation to reduce the memory usage but it may increase the sampling time.

We thank the reviewer for the constructive comments and suggestions. We address your concerns point by point as follows.

W1. The innovation of ConceptFlow

For this point, please kindly see our global response above.

W2. The identity dissimilarity or unsatisfactory aesthetics on generated outputs

We agree with the reviewer that the identity dissimilarity or aesthetic quality of the generated outputs can vary, and one factor influencing this is the generator seed. In image generation, different seeds can result in distinct variations of the same prompt, and multiple attempts may yield better or more aesthetically pleasing results.

W3, Q2. Expanding the Dataset and Subject Variety

We thank the reviewer of the suggestion. Our dataset is heavily collected from the datasets of previous works on multi-concept generation [1,2,3] for human concepts, where they mainly use a small set from 10-20 concepts, and from DreamBench [4] for objects and animals, where we choose the concepts with intricate details for experiments.

[1] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models, NeurIPS 2024

[2] OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models, ECCV 2024

[3] FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition, CVPR 2024

[4] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR 2023

Q1. ArcFace metric for identity preservation evaluation on human concepts

We thank the reviewer for the suggestion. We provide the ArcFace score for identity preservation evaluation on human concepts for both single and multiple concepts generation in the tables below:

Single concept generation:

Multiple concepts generation:

According to the results, ConceptFlow-S and ConceptFlow-M achieve a higher ArcFace score than the baseline methods, indicating that ConceptFlow can better preserve the identity of human concepts. This finding further demonstrates the effectiveness of our proposed attention regularization (AR) in ConceptFlow-S and SAMA module in ConceptFlow-M. Please kindly see Section 3 and Section 4 of our rebuttal manuscript for more analysis.

Q3. Comparisons between SAMA and DreamMatcher's AMA in ConceptFlow-M pipeline

We thank the reviewer for the suggestion. We showcase the qualitative results of this comparison in Section 4 of our rebuttal manuscript, and the quantitative results are as follow:

The results show that SAMA in ConceptFlow-M significantly outperforms AMA in DreamMatcher in terms of both DINO and ArcFace scores, indicating that SAMA can better preserve the identity of each concept, especially for human concepts, in multi-concept images.

Dear Reviewer,

Thank you for your valuable feedback during the review process. Our rebuttal carefully addressed your concerns raised in the initial review. As the deadline for the discussion phase approaches, we kindly remind you to read our rebuttal. If you have any questions, suggestions, or further clarification on any points, please feel free to reach out.

We look forward to your feedback and hope for a positive outcome.

Thank you for your time and consideration.

Best regards,

Authors of Paper 4581

We thank the reviewer for the constructive comments and suggestions. We address your concerns point by point as follows.

W1. How SAMA performs with fine-grained subjects (two dogs or two cats with different breeds)

Firstly, the effectiveness of SAMA mainly relies on the accuracy of concept masks, i.e. the cross-attention maps of their tokens. We consider the combinations such as two dogs, two cats, dog and cat, two humans, as semantically semantic concepts. In these cases, Stable Diffusion 1.5 (SD 1.5) often suffers from attribute binding problem [1], where the attention map of one token is incorrectly focused on the other token even at the begining of the denoising process. Therefore, SAMA, and also ConceptFlow-M, tends to fail in these cases. We carefully mention this in Section 5.1 of our rebuttal manuscript.

Nonetheless, for the case of concepts with intricate details, SAMA demonstrates it capability of preserving complex patterns, as shown in Section 3 of our rebuttal manuscript.

[1] Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis, ICLR 2023

W1. How SAMA works when the concept size increases

In theory, our proposed SAMA module can work for a general case of $N$ concepts. Specifically:

• For each concept $k$ ($1 \leq k \leq N$):
  • We have a process to generate one image of it (which is called reference image). The reference features corresponding to this concept at each timestep, $\psi^{ref}_k$, are calculated from the intermediate features within U-Net of this process.
  • Its foreground mask $\mathbf{M}_k$ is extracted from the cross attention maps of its tokens in the target denoising process.
  • The warping calculation to obtain $V_{k}^{ref \to trg}$ is calculated follow the equation (6), (7) in our manuscript.
• From all warped values $V_{k}^{ref \to trg} (k=1,2,..,N)$, we perform the blending operation using concept masks $\{\mathbf{M}_k\}_{k=1}^N$ to obtain the final value for the self-attention calculation in the target denoising process, following equation (8) in the manuscript.

W2. Results on larger number than two concepts

In scenarios involving more than two concepts, ConceptFlow-M produces both successful and failed outputs with roughly equal probability. It is noteworthy that most condition-free multi-concept generation methods [1,2] on Stable Diffusion 1.5 (SD 1.5) exhibit the same issue, and they only conduct experiments with two concepts. We are convinced that this limitation stems from SD 1.5's challenges in handling complex prompts (i.e., prompts involving more than two concepts). For visualization and further analysis, please refer to Section 5.2 in our rebuttal manuscript.

To address this problem, typical approaches involve incorporating conditions such as boxes, masks, and applying local prompting and denoising [3,4]. However, in this work, our focus for ConceptFlow is on advancing condition-free generation. As a result, addressing this limitation will be a potential area for future work.

[1] Multi-Concept Customization of Text-to-Image Diffusion, CVPR 2023

[2] Key-Locked Rank One Editing for Text-to-Image Personalization, SIGGRAPH 2023

[3] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models, NeurIPS 2024

[4] OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models, ECCV 2024

Dear Reviewer,

Thank you for your valuable feedback during the review process. Our rebuttal carefully addressed your concerns raised in the initial review. As the deadline for the discussion phase approaches, we kindly remind you to read our rebuttal. If you have any questions, suggestions, or further clarification on any points, please feel free to reach out.

We look forward to your feedback and hope for a positive outcome.

Thank you for your time and consideration.

Best regards,

Authors of Paper 4581

We thank the reviewer for the constructive comments and suggestions. We address your concerns point by point as follows.

W1, W2. How ConceptFlow unifies personalized image generation. The contribution of the proposed method and the relation of modules.

For these two points, please kindly see our global response above.

W3. Compare with methods that also use image adapter

Firstly, in ConceptFlow-S, we adopt the image adapter and related loss objectives from DisenBooth [1] during fine-tuning process for better disentangled learning of identity-relevant andi identity-irrelevant information. We do not use the image adapter for inference like other zero-shot methods [2,3,4].

We present a qualitative comparison between ConceptFlow-S and zero-shot methods [2,3] in Section 2 of our rebuttal manuscript. Since MS-Diffusion [4] does not rely on Stable Diffusion 1.5, we do not include them in the comparison. The results show that ConceptFlow-S generates images with superior identity preservation for concepts with intricate details, which is a challenge commonly faced by zero-shot methods. We remark that Ip-adapter [3] and MS-Diffusion [4] have not been published in peer-review venues.

[1] DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation, ICLR 2024

[2] Ssr-encoder: Encoding selective subject representation for subject-driven generation, CVPR 2024

[3] Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models, arXiv preprint arXiv:2308.06721 (2023).

[4] MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance, arXiv preprint arXiv:2406.07209 (2024).

W4. More experiments results

We thank the reviewer for the suggestion. Please see the Section 1 in our rebuttal manuscript, where we add additional experiments results, especially for ConceptFlow-M, to demonstrate the effectiveness of the ConceptFlow framework.

Dear Reviewer,

Thank you for your valuable feedback during the review process. Our rebuttal carefully addressed your concerns raised in the initial review. As the deadline for the discussion phase approaches, we kindly remind you to read our rebuttal. If you have any questions, suggestions, or further clarification on any points, please feel free to reach out.

We look forward to your feedback and hope for a positive outcome.

Thank you for your time and consideration.

Best regards,

Authors of Paper 4581

We thank the reviewer for the constructive comments and suggestions. We address your concerns point by point as follows.

W1. Comparisons with other methods for multi-concept generation

We thank the reviewer for the suggestion. We have included comparisons between ConceptFlow-M, FreeCustom [1], and OMG [2] for multi-concept generation in Section 1 of our rebuttal manuscript, as other methods either do not release their code or trained weights. Additionally, we use ArcFace metrics to evaluate identity preservation for human concepts.

It is noteworthy that in a condition-free setting, ConceptFlow outperforms most other methods when generating multi-concept images.

• For FreeCustom [1], this method requires reference images for concepts in the correct positions where it expects they to appear in the output image. As a result, using images with the concept centered, like those from a personalization dataset, leads to significant artifacts.
• For OMG [2], a two-stage approach is used: first generating the layout and then integrating concept details into the layout. The main drawback of this approach occurs when the shape of the generated layout differs from the actual shape of the concept.

[1] OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models, ECCV 2024

[2] FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition, CVPR 2024

W2. Metrics that evaluate the trade-off directly

We thank the reviewer for the suggestion. Since there is currently no work that directly addresses the metric evaluating the trade-off between reconstruction and editability, we compute the F1 score between DINO and CLIP-T as a provisional metric in the table below. The results show that both ConceptFlow-S and ConceptFlow-M strike a better balance between reconstruction (identity preservation) and editability (prompt alignment) compared to other methods.

Single concept generation:

Multiple concepts generation:

Additionally, we incorporate the ArcFace score to assess identity preservation for human concepts in Section 1 of our rebuttal manuscript. This metric also demonstrates the effectiveness in preversing human facial details of ConceptFlow-S and ConceptFlow-M.

Q1. Multi-concept generation results on the CustomConcept101 dataset

We thank the reviewer of the suggestion. Our dataset is heavily collected from the datasets of previous works on multi-concept generation [1,2,3] for human concepts, where they mainly use a small dataset from 10-20 concepts, and from DreamBench [4] for objects and animals, where we choose the concepts with intricate details for experiments. Therefore, conducting experiments on the CustomConcept101 dataset for ConceptFlow and other baseline methods for comparison would require substantial effort within this limited timeframe. However, we will take this suggestion into account for future work.

[1] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models, NeurIPS 2024

[2] OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models, ECCV 2024

[3] FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition, CVPR 2024

[4] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR 2023

Q2. Report both CLIP-I and DINO for identity preservation evaluation

We thank the reviewer for the suggestion. We showcase the results of CLIP-I and DINO for identity preservation evaluation in the tables below, where ConceptFlow-S and ConceptFlow-M both show competitive results compared to other methods:

Single concepts generation:

Multiple concepts generation:

However, as mentioned by Heiz at el.[1], the CLIP-I metric is not constructed to distinguish between different subjects that could have highly similar text descriptions (e.g. two different yellow clocks). Therefore, it might not accurately reflect the identity preservation performance of the model.

[1] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR 2023

Q3. About the extra conditions like boxes and masks

We agree with the reviewer that additional conditions can enhance control over the generation process. However, in this work, we focus on the challenge of condition-free generation to provide greater flexibility and ease of use. While incorporating extra conditions into ConceptFlow is feasible, we plan to explore this in future research.

Dear Reviewer,

Thank you for your valuable feedback during the review process. Our rebuttal carefully addressed your concerns raised in the initial review. As the deadline for the discussion phase approaches, we kindly remind you to read our rebuttal. If you have any questions, suggestions, or further clarification on any points, please feel free to reach out.

We look forward to your feedback and hope for a positive outcome.

Thank you for your time and consideration.

Best regards,

Authors of Paper 4581

Dear Reviewer,

We appreciate your fast and thoughtful reply. We would like to clarify some details as follows:

W1. Comparisons with other methods for multi-concept generation

For the methods you mentioned:

• KOSMOS-G: The URLs provided for their trained weights are incorrect. Retraining their model would require significant computational resources (they used OpenImages V7 and InstructPix2Pix datasets).
• FastComposer: The authors trained their model specifically for human concepts on the FFHQ dataset. Since we evaluate ConceptFlow on a broader range of concepts, including animals, objects, and humans, using their trained weights is not suitable. Additionally, training their model on a different dataset raises concerns about fairness.
• Key-Locked Rank One Editing for Text-to-Image Personalization: We were only able to find an unofficial implementation of this method at this repository. However, the codes there can not reproduce the multi-concept generation results presented in their paper. There are also several issues mentioned in that repository regarding this problem, such as this issue. As a result, using this implementation would not be a fair comparison.

We also tried our best to compare our method with existing methods in multi-concept generation, including Custom Diffusion [1], FreeCustom [2], Mix-Of-Show [3], and OMG [4]. Please see the results in Table 1 in our rebuttal PDF file. Our method achieved better performance than these state-of-the-art methods.

[1] Multi-Concept Customization of Text-to-Image Diffusion, CVPR 2023

[2] FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition, CVPR 2024

[3] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models, NeurIPS 2024

[4] OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models, ECCV 2024

W2. What "balance trade-off" means

Fine-tuning-based methods often face challenges in balancing the trade-off between reconstruction and editability, i.e. achieving a higher DINO score can result in a decrease in the CLIP-T score, and vice versa. Additionally, qualitative results (Figure 1 in our rebuttal manuscript underscore this trade-off: DreamBooth achieves strong reconstruction fidelity but poorly aligns with prompts, while Custom Diffusion aligns well with prompts but exhibits weak reconstruction quality.

Quantitatively, Table 1 of our rebuttal manuscript provides further evidence. Comparing with ED-LoRA [1], DreamBooth [2] achieves a higher DINO score but suffers from a lower CLIP-T score. A similar trend is observed across other methods in this table.

In contrast, ConceptFlow-S achieves both higher DINO and CLIP-T scores compared to parameter-efficient fine-tuning methods (ED-LoRA [1] and LoKR [3]), demonstrating that ConceptFlow-S can simultaneously enhance these two capabilities.

While ConceptFlow-S has a lower CLIP-T score compared to Custom Diffusion [4] and DisenBooth [5], it significantly outperforms these methods in DINO score. These methods overlooked the importance of reconstruction capability. Additionally, when compared to DreamBooth [2], ConceptFlow-S showcases a competitive DINO score along with a better CLIP-T score. DreamBooth [2] is often affected by overfitting, which limits its ability to effectively balance the trade-off.

Furthermore, as the suggestion by Reviewer wm2m, we compute the F1 score between DINO and CLIP-T as a provisional metric directly evaluating the trade-off between reconstruction and editability. Our method outperforms existing methods in terms of F1 score, highlighting the superior balance of our method. Please see the results in Table 1 in our rebuttal PDF file.

[1] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models, NeurIPS 2024

[2] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR 2023

[3] Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation, ICLR 2024

[4] Multi-Concept Customization of Text-to-Image Diffusion, CVPR 2023

[5] DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation, ICLR 2024

Core contributions and novelty

1. Develop a novel yet robust single-concept learning pipeline.

We introduce KronA-WED adapter, which leverages KronA [1], DORA [2] and embedding decomposition [3]:

• KronA [1] relaxes the low-rank assumption in LoRA [4], improving the learning capability of parameter-efficient fine-tuning (PEFT) process while maintaining a smaller model size than LoRA.
• DORA [2] further enhances the performance gains from PEFT methods via the weight decomposition technique.
• Embedding decomposition [3] has an important role for the success of Gradient Fusion [5], which we use to combine the individual weights learned by ConceptFlow-S. This merged weight is then utilized at the beginning of ConceptFlow-M.

We leverage Disentangled learning [6] to prevent the model from learning concept-irrelevant details, such as background scenes, thus enhancing editability. We employ an image adapter with three loss objectives only during the finetuning process, but not during inference like existing methods [7, 8, 9].

[1] KronA: Parameter Efficient Tuning with Kronecker Adapter, NeurIPS 2023 Workshop

[2] DoRA: Weight-Decomposed Low-Rank Adaptation, ICML 2024

[3] P+: Extended Textual Conditioning in Text-to-Image Generation, arXiv 2023

[4] LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022

[5] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models, NeurIPS 2024

[6] DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation, ICLR 2024

[7] Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models, arXiv preprint arXiv:2308.06721 (2023).

[8] Ssr-encoder: Encoding selective subject representation for subject-driven generation. CVPR 2024

[9] MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance, arXiv preprint arXiv:2406.07209 (2024).

2. Novel attention regularization (AR) in ConceptFlow-S

Attention regularization during fine-tuning Stable Diffusion is a widely used technique for adjusting attention maps of specific tokens in various tasks, such as subject-driven personalization [1,2], relation inversion [3], and concept erasure [4].

In ConceptFlow-S, we propose a novel, part-of-speech-inspired method to handle the attention maps of the adjective token ($V_{rand}$) and the noun token ($V_{class}$) of a concept differently. Specifically:

• The noun token $V_{class}$ should align with the extracted masks from training input images
• The adjective token $V_{rand}$ can activate some specific regions within those masks.

This approach is particularly effective for learning human concepts, where the "identity" of the desired concept can vary in each training image (e.g., different clothing or hairstyles). The relaxed constraints on the adjective token allow the model to better capture the common features of the concept, which, in the case of human concepts, is primarily the face (please see Section 3 in our rebuttal manuscript for experimental results).

[1] Break-A-Scene: Extracting Multiple Concepts from a Single Image, SIGGRAPH Asia 2023

[2] FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention, IJCV 2024

[3] Customizing Text-to-Image Generation with Inverted Interaction, ACM MM’24

[4] MACE: Mass Concept Erasure in Diffusion Models, CVPR 2024

3. Subject Adaptive Matching Attention (SAMA) in ConceptFlow-M

Inspired by the AMA module in DreamMatcher [1], we extend this strategy to multi-concept generation in ConceptFlow-M to address the identity loss problem for concepts in multi-concept images.

The modifications from AMA involve using concept attention maps as the "foreground mask" to calculate the displacement fields in each warping operation, particularly in the masking step during the calculation of matching cost volume $\mathbf{C}_k$. This enhances the accuracy of the feature similarity calculation between the target image and each reference image (please see Section 4 in our rebuttal manuscript for experimental results).

[1] DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization, CVPR 2024

4. Layout consistency guidance in ConceptFlow-M

From the investigation that the concept missing issue in personalized multi-concept generation is primarily due to the model's difficulty in retaining the layout from initial to final denoising steps, inspired by [1], we develop a simple yet effective layout consistency guidance mechanism in ConceptFlow-M.

[1] A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesis, ICCV 2023.