DMM: Building a Versatile Image Generation Model via Distillation-Based Model Merging

Thanks for the valuable feedback

Thank you for taking the time to provide your valuable feedback. For the visual demonstration, we provide image results in the Supplementary Material PDF (cause we can not directly submit figs on openreview).

Q.1 Clarification of the learnable embeddings.

We have provided the implemented details of style prompts in the Appendices. Sorry for not making it clear, we will place this important information in the main text rather than in an appendix. Specifically, we first align the embeddings with an MLP and then add it to the timestep embedding, which is lightweight enough for plug-and-play purposes. The shape of style prompts is $(N, 1280)$, where $N$ is the number of models. In this way, we can modulate the model in both the training and inference phases and make sure it learns MoE-like ability.

Q.2 Exploration on merging of multiple control conditions.

Thank you for posting such an insightful point of view. In principle, the goal of merging multiple control conditions is achievable because our approach is essentially to provide signals to switch between different modes. We conduct experiments on merging three types of ControlNet: canny, depth, and pose.

Besides, the merging of ControlNets and UNets is orthogonal and can be combined compatibly. We show the results in Figure 7 in the PDF, and it is worth mentioning that this versatile effect is achieved with only one single DMM-UNet and only one DMM-ControlNet, demonstrating the potential of broader applicability tasks of our approach.

Q.3 Compared to other style migration methods

The motivation of our DMM and these methods are different. Our approach aims to resemble and reorganize the knowledge from different teacher models. As demonstrated above, our DMM is not just for the style migration ability, though style information may be an explicit component of so-called model knowledge. Our approach can be applied to more programs, such as merging multiple control conditions.

Thank you for your response.

For the mechanism of learnable embeddings, we exactly follow the good practice of SDXL's micro-condition, which also encodes the original image size, target image size, and cropping parameters into 1280-dimension embeddings and add them to the timestep embedding, and have been extensively validated as effective. From our visual results, it is evident that such a simple approach is sufficient to guide the model in learning different target knowledge. It is distinct that our model can clearly differentiate between various experts and faithfully reproduce the generation of the original teacher model. Additionally, our method generalizes beyond just style information, as discussed in Q.5 posed by Review V6za regarding details preference, as well as in the multi-control merging you suggest. With the support of both quantitative and qualitative performance, we believe that this demonstrates that our 1280-dimension embedding provides a simple yet effective merging method instead.

We honestly show that the learnable embedding can be used to derive our current moe-like results and verify its potential, by serving as a flexible switch. We hope that this can relieve your concern or you can identify your concrete dissatisfaction regarding the demonstration presented in our paper.

Thanks for the valuable feedback

Thank you for taking the time to provide your valuable feedback. For the visual demonstration, we provide image results in the Supplementary Material PDF (cause we can not directly submit figs on openreview).

Q.1 Exploration of more unique styles.

Sincerely thanks for your insightful suggestion and for providing more interesting examples of our approach. We conducted experiments on four unique challenging styles, including the instances you provided. The detailed information is shown in the table below. The main results are in Figure 4 and the mixing results are in Figure 5 in the PDF, which reveals that our approach can generalize to a broader range of distinct styles. It is worth noting that these models are in the form of LoRA, which also validates the feasibility of our approach in merging LoRAs.

Q.2 Incremental learning with enhanced setting.

We conduct the incremental experiment setting the initial task with models of different styles, namely the models with index $[1, 2, 5, 8]$. The results are shown in the table below, which is also shown in Table 4 in the PDF. It can be seen that the choice of the teacher models in the initial stage has little impact on the performance of our method.

The experiment #3 is fine-tuned from #2, and the experiment #5 is fine-tuned from #4.

Q.3 Initialization of the student model.

For a simple and general purpose, we initialize the student using SD1.5 weights. We will add the information in the manuscript.

Q.4 Outcome with no style prompts for input.

We conducted this experiment and found that when no style prompt is provided (i.e. the time embeddings are added with zeros), the model’s output is more like an average of the contained styles. We further validate this point by calculating the norm of each style embedding, obtaining that the norm of each 1280-dimensional style vector is approximately 36, while the norm of the average of them is a lot smaller (about 12).

Actually, we have considered regularizing the model prediction with null-style conditions, where we can leverage the vanilla SD1.5 for supervision. However, according to the goal of the practical task, which is to learn from the knowledge of different expert models, this operation is unnecessary. Frankly, as a foundation model, the images generated by SD1.5 are not aesthetically pleasing; it only provides nice initialization for fine-tuning. Therefore, in practice, we are more focused on utilizing these fine-tuned upgraded models and we do not have to let the null-style outputs resemble the SD1.5 outptus. We prefer to allow the zero-embedding input to become an adaptive result in embedding space under several anchor points, where the results still exhibit the knowledge of the experts rather than the relatively unappealing output of SD1.5.

Q.5 Explanation for consistency not only in style but also in details.

Thank you for posting such an insightful point of view. To explain the phenomenon, it is because the foundational target of our task is to reproduce every teacher model faithfully. Thus, we resort to leverage knowledge distillation to achieve this. Style is just one of the more obvious and intuitive aspects of model knowledge, and we primarily use the term "style" in the paper to describe the characteristics of distinct models. In fact, during the distillation process, our approach can distill a wider range of both high-level and low-level information from models such as layout and pose preference, through our supervision of scores, features, and adversarial objectives. We consider this point to be important and a primary principle of the task, because in some practical applications, we exactly need the effects of these expert models, which previous merging methods have been unable to achieve. The `style prompts' at the timestep embedding level play the role of switching among the modes and it is enough to work well.

Thanks for valuable feedback

We appreciate the time you took to share your valuable feedback with us. For the visual demonstration, we provide image results in the Supplementary Material PDF (cause we can not directly submit figs on openreview).

Q.1 Concern about application.

There is indeed a demand for model merging techniques within the community; in fact, this technology is needed in our industrial products. Even though there are LoRAs available to achieve lightweight models, achieving satisfactory strong performance still requires full parameter fine-tuning and most top powerful models are still in the form of base models in the communities. In situations, we exactly should use the satisfying models and only have access to the base model (maybe the LoRAs are merged into the UNet), our approach is proposed to address this issue. Besides, the frequently used weighted merging is also trying to achieve the target but it is not flexible enough, so the application of our approach is desirable among the community.

Q.2 Necessity of model merging.

As mentioned above, the situation our method aims to address is the desire to flexibly and efficiently utilize existing expert models in the community, as these expert models are highly valuable due to being fine-tuned on high-quality datasets and can accurately satisfy our application requirements. The SDXL can cover many styles but it is too generalized and coarse, causing it not able to meet the needs of practical or even commercial applications.

Compared to specific expert models, prompt tuning alone is very inefficient and falls far short of meeting application requirements and we conduct experiments to illustrate this. As shown in Figure 3 in the PDF, we try to control the style by adding keywords on SDXL, but the results are far from reaching the performance of our approach and other advantages of our DMM like style mixing are completely impossible.

To put it bluntly, these fine-tuned models are indeed prettier than SDXL in practice, and in many cases, we exactly need to leverage these ready-made models for reasons like their high quality, unique strengths, etc. After all, they are carefully fine-tuned by users and possess domain-specific knowledge, making them more practical and effective compared to a general base model.

Q.3 Clarification for Table 3

Table 3 provides a perspective on the capabilities of our model, aiming to demonstrate that our approach can achieve results comparable to the original common practices by performing mix augmentation within similar domains. More investment can bring more advantages. First, our DMM can faithfully reproduce the effects of every teacher model, which further expands its versatility. Second, our approach can gather knowledge from many distinct models and apply flexible adjustment at inference time depending on the current need, while weighted merging cannot be adjusted after fixing the weights.

Q.4 Analysis of the number of teacher models

Sincerely thanks for your insightful suggestion for our approach principles as this determines the scalability of our DMM. We conducted experiments with three different settings for the number of teacher models, as shown in the table below. It should be acknowledged that the FID for each target slightly increases with the number of teacher models. Fortunately, this performance degradation is still quite manageable within a larger scope, at around 1% every addition of about 10 models. This guarantees that our method can be effectively scaled to tens of teacher models.

Q.5 What is the base models in Figure 5?

The information on all the teacher models is provided in the Appendices. As stated in the manuscript, we merge the first four styles, namely the base models are the first four models in Table 4 in the main paper.

Q.6 Concern about style overlap

This will not cause problems. Similar styles will only simplify the training task, and even models with similar styles will have subtle differences. Our approach is capable of learning the specific knowledge and faithfully reproducing each teacher model.

Additionally, we conduct experiments on more special-style models as demonstrated in Section Q.18 in the PDF, and our approach still works well.

Q.7 Applicability of merging LoRAs

Yes, it is feasible to apply our method for merging LoRAs. In fact, in the above experiment in Section Q.18 in the PDF of unique style, the teacher models are in the form of LoRAs, and we have successfully merged them. We can also implement the trainable parameters as LoRA.

Thanks for the valuable feedback

We appreciate the time you took to share your valuable feedback with us. For the visual demonstration, we provide image results in the Supplementary Material PDF (cause we can not directly submit figs on openreview).

Q.1 The selection of styles is a weak setup to demonstrate the problem.

To further demonstrate the generalizability of our approach, we conduct experiments on a wider variety of distinct models, including Chinese ink painting, pixel art, business illustration, etc. We show the results in Section Q.18 in the PDF. Our DMM still works well on these challenging models.

Q.2 Demonstration of issues being addressed

To concretely demonstrate that directly merging models of various styles will cause conflict and style confusion, we conduct such experiments. With the same setting of model candidates, namely the eight models with different styles, we apply weighted merging directly on their parameters and use the merged model to synthesize images.

As shown in Figure 1 (in the PDF), the output of this model exhibits an ambiguous hybrid style, rather than being purely realistic or entirely anime style. Even with additional corresponding keywords (e.g. 'realistic', 'animation'), it still struggles to accurately distinguish between different styles and cannot produce satisfactory results, compared to our results in Figure 6 in the main paper. This result is not what we expected and we cannot effectively control the generation.

This phenomenon can also be used to explain why prompt engineering cannot achieve the same effects as our merging approach, since handcraft prompt tuning operates at a relatively vague level, while our method can provide precise suggestions and access expert model knowledge.

Q.3 Lack of baseline results in Table 1

We use the initialized student model, namely SDv1.5 as the baseline results. Specifically, for the FID calculation of each style, we use additional style-related keywords in the text prompt to modulate different styles. It is obvious that the baseline results are very poor, and the visualizations below will demonstrate this point again.

Q.4 Missing upper-bound teacher performance in Table 2

We will provide the upper-bound results in Table 2 in the final revision.

Q.5 The scope is limited

We further extend the application scope of our DMM approach:

• As mentioned above, we conduct experiments on merging more challenging models, which have both different styles and different contents (Chinese ink painting, outdoor scenes, etc.). The results are provided in Section Q.18 in the PDF.
• We conduct experiments on merging different control conditions by applying our approach to several ControlNets[1]. It works well and can combined with DMM-UNet feasibly, achieving versatile styles and versatile controls orthogonally with only a single model. The results are provided in Section Q.24 in the PDF.

Q.6 Missing visual results for baseline and ablation configurations.

We provided visual results in Figure 2 (in the PDF) of different ablation settings to verify the improvement in visual quality brought about by the reduction in FID. It can be seen that, as the FID decreases, the images exhibit more refined optimizations, such as improved textures and lighting, indicating a closer alignment with the target style.

Q.7 Continual learning appears unrelated to the core problem

That is not the case. The sustainability of our method is crucial for the application of model merging and should be paid attention to. If our approach is a disposable solution, its application flexibility would be greatly limited. Users often encounter situations where they want to incrementally merge a recently developed model into the current checkpoint. If they have to train from scratch, it will bring inconveniences and resource waste, while our continual learning strategy alleviates the problem.

Reference

[1] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Dear review, as the revised PDF upload ends on November 27, we want to ensure that all your questions have been thoroughly addressed. Your feedback is instrumental to us, and we would be grateful if you could spare a moment to provide a final rating and share your thoughts. Your input will greatly inform our future improvements.