How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?

Q1: The symbol definitions are too complex which are hard to follow.

A1: Thanks for your valuable comment. We will carefully polish the symbol definitions and introduce a table about notation definitions to better understand this paper in the final revision process.

Q2: This paper introduces several new hyperparameters. How sensitive is the model to these?

A2: Thanks for your insigntful comment. We employ Stable Diffusion (SD-1.5) [38] as the pretrained model and use image alignment (IA) as metric to investigate the effect of the hyperparameters {$\gamma_1, \gamma_2, r$} on the performance of our model. When setting the rank $r=4$ of LoRAs, we analyze the effect of the hyperparameters {$\gamma_1, \gamma_2$} over a range of {0.01, 0.1, 1, 10}. Moreover, we introduce the hyperparameter experiments for $r$ within the range {2, 4, 6, 8}, when setting the balancing weights $\gamma_1=0.1, \gamma_2=1.0$. From the following results, we observe that our model has stable performance over a wide range of hyperparameters {$\gamma_1, \gamma_2, r$}. Furthermore, our model is not sensitive to different hyperparameter selections, making it effective and robust for learning continuous text-guided concept customization tasks.

Q3: How well does the model perform across different application domains and with varied types of personalized concepts?

A3: Thanks for your constructive comments. Inspired by [10][46], we construct a new challenging concept-incremental learning (CIL) dataset including ten continuous text-guided concept customization tasks to verify the effectiveness of our model under the CIFC setting. This dataset has varied types of personalized concepts, which are collected from different application domains. Moreover, different concepts have significant differences. Specifically, as shown in Fig. 8, seven tasks have different object concepts (i.e., V2 duck toy, V4 backpack, V5 teddy bear) and pet concepts (i.e., V1 dog, V3 cat, V7 dog and V9 cat) from [40, 22]. Besides, the remaining three tasks have different style concepts (i.e., V6, V8 and V10 styles) collected from website. Considering the practicality of the CIFC setting, we introduce some semantically similar concepts (e.g., V1 and V7 dogs, V3 and V9 cats), making the CIL dataset more challenging under the CIFC setting. Particularly, all comparison experiments of this paper are conducted on this challenging dataset. As shown in Tabs. 1-2 and Figs. 2-5 of the submitted manuscript, our model achieves better performance than other state-of-the-art (SOTA) comparison methods in resolving the practical concept-incremental flexible customization (CIFC) problem, under both qualitative and quantitative evaluations.

Thanks for the authors' responses. Some of my concerns have been solved.

Q1: It would be better to mention the comparison methods of experiments in the related work, such as LoRA-M, LoRA-C, CLoRA and L2DM, and emphasize the differences between them and the proposed method.

A1: Thanks for your insightful comment. We will carefully polish the related work in the finial revision. The differences between our model and other methods are as follows:

1. LoRA-M [64] equally amalgamates all LoRA layers to retrain diffusion model, and LoRA-C [64] explores contributions of different LoRA layers for multi-concept composition. When learning new concepts continually, they may experience signifcant loss of attributes on old concepts (i.e., catastrophic forgetting) for versatile customization. Different from them, we devise a concept consolidation loss to mitigate forgetting of old concepts by exploring task-specifc/task-shared knowledge.

2. CLoRA [43] proposes a self-regularized low-rank adaption to continually learn new concepts, while L2DM [46] builds a long-term memory bank to reconstruct old concepts. They cannot control synthesized contexts according to user conditions and suffer from concept neglect in multi-concept composition. However, our model proposes a context-controllable synthesis to tackle concept neglect, and effectively learns new concepts continuously for versatile customization.

Q2: I did not understand the relationship between continual learning and concept neglect. I would like to know if concept neglect is a challenge introduced by continual learning, and what impact continual learning has on original concept neglect?

A2: As introduced in [46][64], concept neglect is a common challenge when performing multi-concept composition according to user-provided conditions, rather than being directly introduced by continual learning. However, if latent diffusion models aim to consecutively synthesize a sequence of new concepts under the CIFC setting, the catastrophic forgetting of old concepts can significantly exacerbate the degree of concept neglect when composing multiple personalized concepts.

Q3: The baseline choices for continual learning such as EWC and LWF are too outdated. Why not choose new and more effective continual learning algorithms? Intuitively, more advanced algorithms should be able to better mitigate catastrophic forgetting.

A3: To further verify the effectiveness of our model, we use a more advanced continual learning algorithm, InfLoRA Refs[1], published in CVPR2024, for performance comparison. As shown in the following results, our model substantially surpasses InfLoRA Refs[1] by a large margin. Note that CLoRA [43] and L2DM [46] are the two most advanced continual generation models, which are highly relevant to this paper. Particularly, our model achieves better performance than these SOTA methods.

Refs[1] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning, CVPR 2024

Q4: How severe are catastrophic forgetting and concept neglect, and whether some more objective evaluation criteria can be introduced.

A4: Following L2DM [46], we introduce a new evaluation criteria: Task Forgetting Rate of Image Alignment (TFR-IA), to evaluate the degree of catastrophic forgetting and concept neglect. For the $l$-th concept, its TFR-IA is computed by $\frac{1}{g-1}\sum_{i=1}^{g-1} I_{i,l} - I_{g,l}$, where $I_{i,l}$ and $I_{g,l}$ denote the image alignment (IA) of the $l$-th concept evaluated at the $i$-th and $g$-th tasks. Our model outperforms other SOTA methods in terms of the TFR-IA metric. It verifies that our model has the least forgetting and concept neglect, making it more effective for continual generation.

Q5: How much do different weighting schemes between LoRAs affect the quality of generation, compared to some intuitive weighting baselines?

A5: As shown in Tab. 1 and Figs. 2-3 of the submitted manuscript, we introduce comparisons between our model and some intuitive LoRAs weighting baselines such as LoRA-M [64] and LoRA-C [64]. Specifically, LoRA-M [64] equally amalgamates all LoRA layers to retrain diffusion model, and LoRA-C [64] explores contributions of different LoRA layers for multi-concept composition. In Tab. 1 and Figs. 2-3, LoRA-M [64] and LoRA-C [64] experience signifcant loss of attributes on old concepts (i.e., catastrophic forgetting) for versatile customization. Moreover, our model significantly outperforms them in effectively resolving catastrophic forgetting.

Q6: Does the value of the rank in LoRA and its placement in the model (e.g., which layer) have an impact on the results?

A6: Thanks for your comment.

1. Following [10], we set the value of rank to 4 in this paper, and use Stable Diffusion (SD-1.5) [38] to analyze its effect on performance (IA) by setting r={2, 4, 6, 8}. As shown in the following results, the value of rank ($r\leq 8$) has negligible impacts on the performance. Note that larger rank can increase parameters and memory costs, which violates practicality of the CIFC setting.

2. For LoRA placement, each cross-attention layer in the diffusion models has its own LoRA layer.

Thank you for your follow-up comments!

1) As for the application of continual learning in the generation community, our proposed model can help users continually synthesize a series of new personalized concepts for versatile customization based on their provided conditions, while addressing catastrophic forgetting of old personalized concepts. In contrast, existing state-of-the-art (SOTA) diffusion models in the generation community may experience significant loss of individual attributes in old personalized concepts (i.e., catastrophic forgetting) during versatile customization when continually learning new personalized concepts. The experiment results shown in Tabs. 1-2 and Figs. 2-5 of the submitted manuscript also illustrate the effectiveness of our model in addressing catastrophic forgetting and continually learning new concepts, compared to other state-of-the-art (SOTA) methods.

2) Advantages of Our model Over Multi-LoRA Composition:

a) Compared with SOTA multi-LoRA composition methods such as LoRA-M [64] and LoRA-C [64], our proposed elastic weight aggregation module utilizes learnable layer-wise concept tokens to merge all low-rank weights of old personalized concepts, based on their contributions to versatile concept customization. This module ensures that our model is more effective to resolve longer task sequences for versatile concept customization than other SOTA multi-LoRA composition methods such as LoRA-M [64] and LoRA-C [64]. As shown in the following results, we use a challenging benchmark dataset (i.e., celebrity faces) to construct thirty continuous text-guided concept customization tasks. Compared with our model, current multi-LoRA composition methods (LoRA-M [64] and LoRA-C [64]) suffer from significant performance degradation, due to the catastrophic forgetting caused by the composition of longer sequence tasks.

b) Mix-of-Show [10] proposes a gradient fusion strategy to train a composed LoRA weight that mimics the predictions of individual LoRAs. However, it requires retraining the entire diffusion model and CLIP when learning new personalized concepts continually, which can substantially increase computational complexity and memory cost. As shown in the following results, compared with our model, Mix-of-Show [10] requires larger training parameters and memory costs to learn a new concept customization task, making it impractical to handle longer task sequences under the CIFC setting. Moreover, to ensure fair comparisons with existing methods, we apply the regional control techniques proposed in Mix-of-Show [10] to all comparison methods. The qualitative comparisons in Figs. 3 and 12 of the submitted manuscript demonstrate the superior performance of our model in multi-concept customization when continually learning new personalized concepts under the CIFC setting.

Refs:

[10] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models.

[64] Multi-LoRA Composition for Image Generation.

Q1: I recommend the authors include details on memory consumption or the number of stored parameters after each task during the rebuttal process.

A1: Thanks for your constructive suggestion. We conduct comparison experiments to evaluate memory consumption and training parameters of each task, when utilizing Stable Diffusion (SD-1.5) [38] as the pretrained model to learn ten continuous text-guided concept customization tasks. As shown in the following results, the training parameters and memory requirements of our model are comparable to those of LoRA-M [64], LoRA-C [64] and CLoRA [43], but are substantially lower than those of other SOTA methods. Moreover, our model achieves better performance than existing SOTA diffusion models (see Tabs. 1-2 and Figs. 2-5). It illustrates that our model is efficient and effective to tackle the practical Concept-Incremental Flexible Customization (CIFC) problem.

Q2: The paper lacks detailed implementation information on critical tasks such as custom image editing and style transfer. Providing these details would significantly enhance the clarity and replicability of the research.

A2: Thanks for your insightful comments.

1. As for custom image editing, we introduce Anydoor [3] as a plug-in for all comparison methods. Specifically, given a text prompt that contains the user's personalized concept, an initial image, and an editable bounding box, we first input the given text prompt into the latent diffusion model to perform custom synthesis. Then we embed the generated image into the editable bounding box of the initial image using Anydoor [3].

2. We introduce T2I-adapter [30] to achieve custom style transfer. Specifically, given an initial image, we utilize BLIP [23] to extract the corresponding caption (e.g., "photo of a city skyline"). Then, we add the custom concept (e.g., [V6] style) to the caption to obtain the style transfer prompt (e.g., "photo of a city skyline in [V6] style"). Finally, we perform Canny edge detection on the initial image to extract a Canny image, and incorporate it with the style transfer prompt to achieve custom synthesis via the T2I-adapter [30].

Refs:

[3] AnyDoor: Zero-shot Object-level Image Customization

[23] BLIP:Bootstrapping LanguageImage Pre-training

[30] T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

Q3: The concept of balancing task-specific and task-shared knowledge is intriguing and seems to be central to effectively addressing the CIFC problem. To substantiate the claims of its effectiveness, the authors should include ablation studies that isolate and quantify the impact of these components. Additionally, visualizations comparing the results with and without these strategies would provide a clearer, more direct illustration of their value.

A3: Many thanks for analyzing our model.

1. Tab. 3 in the submitted manuscript shows ablation studies of single-concept customization to analyze the effectiveness of each module in our model, where TSP and TSH denote task-specific knowledge and task-shared knowledge in the concept consolidation loss, respectively. When we remove the TSP or TSH module, our model decreases 0.5% and 0.6% in terms of image alignment (IA) metric. It verifes the effectiveness of our model to resolve the CIFC problem by exploring both task-specific and task-shared knowledge.

2. Fig. 6 in the submitted manuscript visualizes the ablation analysis of task-specific knowledge (TSP) and task-shared knowledge (TSH) from the perspective of image generation quality. These visualization results illustrates that our model can capture task-specifc information within each customization task and explore task-shared knowledge across different tasks to tackle the CIFC problem via optimizing Eq. (2).

Dear Reviewer 5EBF,

We are pleased that we have addressed most of your concerns. Thanks for your insightful comments and approval of our work.

Best regards,

Authors of Paper #1919

Q1: Please include a table that compares the training, parameter, and memory costs of your method against other methods. This comparison is crucial for a comprehensive evaluation of your method.

A1: Thanks for your insightful comment. As shown in the following results, we use Stable Diffusion (SD-1.5) [38] to conduct comprehensive evaluations in terms of training time, parameters, and memory costs of each task, when the number of continuous concept customization tasks is ten. All comparisons of training time are conducted under two NVIDIA RTX A6000 GPUs. Our model has comparable parameters, memory costs, and training time to LoRA-M [64], LoRA-C [64] and CLoRA [43]. Additionally, the training parameters and memory requirements of our model are substantially lower than those of other SOTA methods. More importantly, our model significantly outperforms existing SOTA comparison models for both the qualitative and quantitative evaluations on versatile generation tasks (see Tabs. 1-2 and Figs. 2-5 in the submitted manuscript). We will consider introducing these comparisons about parameters and memory costs in the final revision.

Q2: Although the results are clear and detailed, they lack confidence intervals or similar statistical measures in the quantitative metrics. Running the experiments with multiple random seeds would ensure robustness and provide a more reliable assessment.

A2: Thanks for your constructive comment. We use five random seeds (0, 2021, 2022, 2023, 2024) to conduct five random experiments in terms of the image alignment (IA) metric and report their average results over five random seeds for evaluation. As shown in the following results, our model achieves better performance than other SOTA methods, which verifies the robustness and reliable effectiveness of our model to tackle the CIFC problem. Moreover, we will consider presenting averaged comparison results over multiple random seeds in the finial revision.

Q3: The dataset used is quite limited. To fully justify your setting, you should explore more diverse datasets and longer task sequences. To truly demonstrate the impact of your method and setting, you should consider long task sequences or challenging benchmarks such as celebrity faces.

A3: Thanks for your invaluable comment. To further validate the effectiveness of our model in learning longer task sequences, we use a new challenging benchmark dataset (celebrity faces) to construct thirty continuous text-guided concept customization tasks. Compared to existing SOTA models such as LoRA-M [64], LoRA-C [64], CLoRA [43], and L2DM [46], this new setting on celebrity faces dataset includes longer task sequences (i.e., thirty text-guided customization tasks), and encompasses different object concepts. Therefore, it is more challenging and better suited to verify the effectiveness in addressing the CIFC problem. We use Stable Diffusion (SD-1.5) [38] to evaluate parameters, memory costs and image aligment (IA) metric over five random seeds. From the following results, we observe that our model significantly outperforms other SOTA methods to learn longer task sequences. More importantly, the training parameters and memory requirements of our model are comparable to those of LoRA-M [64], LoRA-C [64] and CLoRA [43], but are substantially lower than those of other SOTA methods. It verifies the effectiveness of our model to learn longer task sequences on the new challenging dataset.