TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation

[Q1]Using Text-SAM and Tweedie’s formula do not contribute to the novelty.

We kindly remind that combining custom concepts using masks is standard in recent multi-concept generation methods [1-4]. The key challenge lies in achieving natural object composition once masks are provided. Existing models typically combine concepts in attention layers. To address performance degradation and high computational costs, we introduced combining custom concepts in the denoised space using Tweedie’s formula.

We are the first to demonstrate the CFG++ framework's suitability for multi-concept generation, enhancing fusion stability. To improve multi-concept awareness, we firstly proposed splitting the sampling timesteps: multi-concept aware sampling in initial steps and concept fusion in later steps. Additionally, we firstly introduced multi-object resampling to prevent specific concepts from disappearing.

Moreover, our method processes all steps in a single sampling step, enabling faster generation. Unlike baselines, it supports region-wise sampling without attention adjustments, allowing flexible concept fusion. For instance, we seamlessly combined models fine-tuned with low-rank adaptation and attention weights fine-tuned as in custom diffusion—unachievable with baselines. These contributions are novel and represent a significant advancement.

[Q2] The performance gain may comes from overfitting to specific concepts. mask composition seems to be a naive post-processing step. The framework seems to be from extensive ablation study, lacking novelty and a fundamental understanding.

Contrary to the reviewer’s misunderstanding, we demonstrated that our performance improvements are not limited to specific concepts by validating our method on the large customconcept101 dataset. Since our baseline, Concept Weaver, was tested on a similar number of concepts, we ensured comparable experimental conditions for fairness.

To further address the reviewer’s concerns, we expanded the variety of custom concepts and included new quantitative evaluation results in Table 1. We also include more comprehensive metric, DINO scores. Even with the expanded set and new metrics, ours consistently outperformed the baselines, demonstrating that the results reflect the general effectiveness of ours rather than overfitting.

As a reminder, mask composition is a standard technique in recent multi-concept generation. The critical aspect lies in how naturally regions defined by masks can be synthesized, making the use of masks a non-issue.

The novelty of our method lies in achieving high-quality concept fusion by combining concepts in the Tweedie-denoised space of the CFG++. We emphasize that this fusion is not limited to specific settings or concepts. To address the reviewer’s concern, Figure 10 includes experiments showing our method can combine not only custom concepts but also a wide range of styles (e.g., 3D + anime), a task unattainable with previous methods.

[Q3] The extension to video is a technical trick on a off-the-shelf I2VGen-XL. I2VGen-XL is trained on large image data while Dreamvideo use modelscope T2V with little amount of custom videos.

Thank you for your comment. We acknowledge this comparison may not be entirely fair. However, fine-tuning traditional T2V models consistently fails to generate multi-concept images. Techniques like Custom Diffusion and Textual Inversion produced unsatisfactory results, and even the latest method, DreamVideo, barely incorporated multiple custom concepts (Figure 7). Due to significant degradation, no prior research has addressed multi-concept video generation. To overcome this, we are the first to achieve it using an Image-to-Video model—an accomplishment unattainable with baselines. We emphasize that our contribution lies in addressing this new challenge.

We kindly remind the reviewer that our approach is not a simple technical trick using I2VGen-XL. Directly using the I2V model often results in distortions, making it unsuitable for multi-concept video generation. To address this, we devised a novel method to extend residual layer features across frames, preserving the context of multi-concept scenarios. This simple yet effective method is a key technical novelty in our video extension, as demonstrated in Figure 8 and abstract.

To show our method is not limited to I2VGen-XL, we experimented with Stable Video Diffusion(Figure 12). Applying our feature injection, we can still preserve the context information, demonstrating that our approach is not restricted to specific video models.

Reference

[1]Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models, Neurips 2023

[2]Cones 2: Customizable Image Synthesis with Multiple Subjects, Neurips 2023

[3]Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis, Arxiv 2024

[4]Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models, CVPR 2024

[Q1] My main concern lies in the technical contribution of this paper. It seems like an incremental work of ConceptWeaver[1] which is also a training-free method that combines multiple concepts during inference.

We kindly remind the reviewer that our proposed method is different from the baseline of Concept Weaver in the following parts.

(1) The key idea of concept weaver is to separate the sampling processes into multiple stages, therefore the method requires excessive time as it use all of basic image generation, image inversion with DDIM, and concept fusion stages for single output. However, our proposed method can generate multi-concept output with only single generation path, which reduces significant computational time. Specifically, the baseline of concept weaver takes about 65 seconds for single image output, but our method takes only 28 seconds for one image. (see time Appendix C)

(2) For concept fusion, concept weaver uses mask guidance in cross attention layer features, which requires deliberate engineering of layer selection and requires excessive calculation. Also, as the method requires manipulation on both of the cross-attention and self attention layers, the method reduces flexibility as it cannot be adapted to different model architectures.

Our proposed TweedieMix proposes to mix the concepts in denoised image space, therefore we do not require any manipulation on attention layer. It enables us more flexibility in models. For example, we can fuse the various models which are fine-tuned with Low-Rank adaptation or fine-tuned with attention weights. We kindly remind the reviewer that we have already shown that our generated outputs are mixture of various customization methods (e.g. key-value weight fine-tuned for dog , lora fine-tuned for cat)

Also, our method can reduce the memory consumption as it does not require calculation on attention layers. The baseline concept weaver requires about 19GB of VRAM memory, but our method requires only 10GB of VRAM.

(3) One of the key contributions of this work is the observation that simplified concept fusion is made possible by recent advancements in classifier-free guidance, specifically in the form of CFG++.

Specifically, to enhance the quality of multi-concept awareness, we propose dividing the single sampling path into multi-concept-aware sampling steps and concept fusion steps. Additionally, we introduce a novel resampling method to further strengthen multiple-object generation, leveraging the inherent properties of Tweedie’s denoised space in CFG++. Furthermore, to the best of our knowledge, no other work beyond ours has successfully demonstrated multi-concept fusion in video generation.

Given these innovations and the significant improvements in generation quality, we argue that our proposed method represents a substantial advancement and cannot be considered merely incremental compared to Concept Weaver.

[Q2] In terms of qualitative results, there doesn't seem to be a significant improvement compared to ConceptWeaver which is also can handle more than two concepts.

We would like to assure the reviewer that the impression of minimal differences in qualitative results may stem from our choice to use the same concepts as those in the original Concept Weaver paper for a fair comparison. While we acknowledge that Concept Weaver performs relatively well on these specific concepts, our method still demonstrates superior performance both quantitatively and qualitatively.

However, to address the reviewer’s concerns, we have also included additional comparison images in Figure 11, 13,14,15. In these images, it can be clearly observed that Concept Weaver shows unwanted transformations or disappearance of custom concept objects, while our model produced significantly better visualization results than Concept Weaver.

Moreover, the superiority of our results over Concept Weaver’s is more thoroughly demonstrated in the quantitative results and user study. To further address the reviewer’s concerns, we conducted additional detailed comparison experiments by adding the DINO score and expanding the pool of concept images in our Table 1.

Dear Reviewer yJkf,

As the deadline for the Reviewer-Author discussion phase is fast approaching, we respectfully ask whether we have addressed your questions and concerns adequately.

Sincerely,

The Authors.

[Q1] $t_{con}$ and $P$ are two important hyperparameters. Can the authors provide generated results with varying $t_{con}$ and $P$, while keeping all other settings fixed?

Thank you for your suggestions, We included the results with varying the parameters in Appendix B and Figure 9.

[Q2]DINO [1] can extract subtle visual features that are not specific to any particular category. The metric of DINO image similarity should also be reported.

Thank you for your constructive comment. We have included DINO image similarity score In Table 1, and included explanation on section 5.1 and Appendix A.

[Q3]The evaluation is conducted on combinations of more than three concepts; however, the paper does not present any results of combinations involving more than three concepts.

For quantitative score calculation, we have already included the experiments on four concepts. To address the reviewer’s concern, we have included qualitative comparison results in Figure 11.

[Q4]It is recommended to compare the sampling time of the proposed method with that of other methods.

Thank you for your feedback. We have included the time efficiency comparison in Appendix C. Our proposed method takes about 28 seconds, while our main baseline method of concept weaver takes about 65 seconds.

[Q5] Missing reference on unsupervised multi-concept extraction [2]

Thank you for your recommendation. We included the reference in the related work part.

Dear Reviewer JzyV,

As the deadline for the Reviewer-Author discussion phase is fast approaching, we respectfully ask whether we have addressed your questions and concerns adequately.

Sincerely,

The Authors.

Thank the authors for their feedback, which has addressed my concerns raised in the initial review. I have no further questions.