You Only Submit One Image to Find the Most Suitable Generative Model

Dear Reviewer htir:

Thank you for the valuable feedback on our paper. We appreciate the time and effort you have put into reviewing our work and we are grateful for encouraging comments such as interesting and well-motivated setting, reasonable solution, and good writing. We have carefully read your review and addressed your concerns as follows.

[W1] A detailed textual explanation for Figure 1

Thank you for your valuable suggestion. We have added the following explanation to our revised paper in Appendix B.

In traditional searches for generative models, the models are initially filtered by the model hubs using tags and then sorted based on download volume. However, this approach presents challenges for users in identifying their desired models, as the model with the highest download volume may not necessarily align with their specific target. Consequently, users select models based on their own judgment, and then download and evaluate each model individually. This consumes a significant amount of network resources and human effort, and there is also the possibility of filtering out the potentially best model due to judgment errors.

In our GMI setting, each model is directly sorted based on its matching score. This means that the most suitable model is recommended at the top, enabling users to effortlessly find their desired models with minimum effort.

[W2] A brief description of models

Thank you for your valuable advice. We will add a brief description of each model in our appendix.

[W3] Minor problems

Thank you for your valuable advice. We have revised these issues in this paper.

Dear Reviewer htir,

We have responded to your questions point by point in our reply. It is important for us to know whether our responses have addressed your concerns, and we look forward to receiving your further feedback.

Feel free to reply if you have any further questions or suggestions. Thank you!

Best Regards,

Authors

Dear Reviewer htir,

We would be happy to answer any further questions you have. Feel free to reply if you have any further questions or suggestions. Thank you!

Best Regards,

Authors

[Q2] Discuss the limitations of the proposed method.

We have discussed the limitation of efficiency in "3.3 Discussion" in our paper. Thank you for your valuable suggestion. We have additionally added a "Limitations" section in Appendix. C to discuss the limitations of the proposed method.

First, the time complexity of our work has the potential for improvement. At present, our time complexity increases linearly with the number of models in the model platform, which will not become a heavy burden when the number of models is large. Previous research[1] has demonstrated that acceleration techniques can enhance the identification process, such as the use of a vector database. This could be an area for future exploration.

Second, we currently only consider the cases that identify generative models using one uploaded image to describe users' requirements. The assumption is reasonable since users' ideas often rely on existing image templates when they want to generate images, and it is not difficult to find only one image that has a similar style to fulfill the user's requirements. Despite this, it is also interesting to study how to quickly and accurately identify models via other information such as textual prompts. We will study this problem in future work.

[1] Lan-Zhe Guo, Zhi Zhou, Yu-Feng Li, Zhi-Hua Zhou: Identifying Useful Learnwares for Heterogeneous Label Spaces. ICML 2023: 12122-12131

Dear Reviewer 7QtA:

Thank you for the valuable feedback on our paper. We appreciate the time and effort you have put into reviewing our work and we are grateful for encouraging comments such as interesting application and simple, effective method, and good writing. We have carefully read your review and addressed your concerns as follows.

[W1 & W2.1 & Q1] Technical contributions

Our paper makes technical contributions in three specific aspects:

First, we propose a novel and realistic GMI problem. Generative models have garnered considerable attention, accompanied by the emergence of various generative model platforms. However, users face significant inefficiencies when seeking generative models that satisfy their specific requirements within these platforms, as this necessitates a meticulous browsing and testing process. In this paper, we introduce the Generative Model Identification (GMI) problem, encouraging researchers to develop algorithms describing the capabilities of generative models that enable precise alignment between model functionalities and user requirements, thereby achieving expedited searches in generative model exploration. We posit that such endeavors will substantially propel the advancement of generative models and model platforms.

Second, we take a successful step to address the GMI problem by proposing an effective specification for the generative model. It is noteworthy that the proposal is definitely not a simple combination of existing techniques. For example, we analyze the existing RKME method in Figure 2 and show that the existing method not only can not deal with the mapping between prompts and images which is essential for distinguishing generative models, but also can not address the dimensionality and cross-modality problem which hinders the success of GMI. The results in Table 2 clearly prove that any simple combination of existing methods cannot achieve good performance in GMI, and our proposal provides the best performance evaluated by all metrics. Therefore, it is not fair to consider our proposal as a simple combination of existing techniques.

Third, we construct and release a novel benchmark dataset for the GMI problem, which contains general stable diffusion models. Based on this benchmark dataset, we conduct comprehensive experiments to evaluate the existing baseline methods and proposed methods, and the results demonstrate the effectiveness of our proposed method. It is expected the dataset could facilitate the development of generative models.

We appreciate your valuable suggestions and will highlight our contributions in our revised paper.

[W2.1] The MMD distance is a very naive distance metric

MMD distance is a simple yet effective distance metric actually. For model identification tasks, [1] successfully applies the RKME framework to various image recognition tasks, and [2] applies the RKME framework to both tabular classification and image classification tasks. Additionally, MMD distance is widely used in Domain Adaptation [3] and Domain Generalization [4]. Therefore, the MMD distance is not such a naive and questionable distance metric, and our proposed methods are non-trivial based on it.

Moreover, we should clarify that our proposal goes beyond a simple application of RKME and MMD distances. We analyze the existing RKME method in Figure 2 and show that the existing method not only can not deal with the mapping between prompts and images which is essential for distinguishing generative models, but also can not address the dimensionality and cross-modality problem which is a hinder of GMI. The results in Table 2 clearly prove that any simple combination of existing methods cannot achieve good performance in GMI, and our proposal provides the best performance evaluated by all metrics. Therefore, our proposal is clearly not a simple combination of existing techniques.

[1] Lan-Zhe Guo, Zhi Zhou, Yu-Feng Li, Zhi-Hua Zhou: Identifying Useful Learnwares for Heterogeneous Label Spaces. ICML 2023: 12122-12131

[2] Yu-Jie Zhang, Yu-Hu Yan, Peng Zhao, Zhi-Hua Zhou: Towards Enabling Learnware to Handle Unseen Jobs. AAAI 2021: 10964-10972

[3] Wei Wang, Haojie Li, Zhengming Ding, Feiping Nie, Junyang Chen, Xiao Dong, Zhihui Wang: Rethinking Maximum Mean Discrepancy for Visual Domain Adaptation. IEEE Trans. Neural Networks Learn. Syst. 34(1): 264-277 (2023)

[4] Peifeng Tong, Wu Su, He Li, Jialin Ding, Zhan Haoxiang, Song Xi Chen: Distribution Free Domain Generalization. ICML 2023: 34369-34378

Dear Reviewer 7QtA,

We have responded to your questions point by point in our reply. It is important for us to know whether our responses have addressed your concerns, and we look forward to receiving your further feedback.

Feel free to reply if you have any further questions or suggestions. Thank you!

Best Regards,

Authors

Dear Reviewer 7QtA,

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

Best Regards,

Authors

Dear Reviewer 7QtA,

Thank you for your time and effort in reviewing our paper.

As the discussion period nears its end, we'd like to highlight that only a few hours remain before the deadline. We assure you that we've addressed your comments and included additional revisions. If you have any remaining concerns, please reach out in the next few hours. Your insights are crucial to us. If you do not have any further questions, we hope that you might consider raising your score.

Best Regards,

Authors

Dear Reviewer BSis:

Thank you for the valuable feedback on our paper. We appreciate the time and effort you have put into reviewing our work and we are grateful for encouraging comments such as the novel and useful setting and promising results. We have carefully read your review and addressed your concerns as follows.

[W1.1] Multiple images for querying

RKME performs more effectively when the provided image set accurately represents the distribution. Thus, our framework, built upon RKME, naturally accommodates situations in which users provide multiple images as queries. We conducted experiments using user queries consisting of 1, 2, 3, 4, or 5 images. The results are shown in the following table:

The results demonstrate that the accuracy of identifying the most appropriate model improves as the number of uploaded images increases.

Our claim regarding a single image is grounded in two factors: (1) In practical applications, it is often more convenient to upload a single image rather than multiple images to describe the user's requirements; (2) Using one image can achieve a satisfactory Top-4 Accuracy for identifying the most appropriate model.

In conclusion, our framework has the capability to handle multiple images as a query. However, uploading a single image offers a compromise between performance and convenience. We appreciate the reviewer's valuable suggestion and have added this experiment and discussion to our revised paper in Appendix A.2.

[W1.2] Different statistics for scoring

We thank you for your constructive advice. We conduct experiments using some statistical scoring functions, e.g., L1, and Min functions. The results are shown in the following table:

The results show that our proposed method can outperform these statistical scoring functions.

[W2 & W3.1] The selection of the reduced image sets

We thank you for your professional advice. Selecting suitable prompts is important for building specifications for generative models.

This paper focuses on improving the identification of the most suitable model when prompts are already provided. Consequently, we utilize a default set of prompts for all generative models during the construction of specifications. This default set of prompts can be found in the "PromptPool.json" file provided in our supplementary material. And, our proposal can achieve satisfactory performance when using the default prompt set for all models. We have also conducted experiments on the following questions, e.g., [W3.2], [Q1], [Q4], to answer your questions.

We appreciate the reviewer pointing out these key points for GMI setting. The selection of prompts for generating specifications could be a future direction for the GMI setting.

[W3.2] Qualitative results for selection of the reduced image sets

We conducted experiments with three different prompt sets. Split 1 is our default prompt set used in our experiments. Split 2 and Split 3 are two non-overlapping subsets, each half the size of Split 1. The results are shown in the following table:

The results demonstrate the robustness of the identifying performance to varying default prompt sets used in generating specifications. We appreciate the reviewer's valuable suggestion and have added this experiment to our revised paper in Appendix A.3.

[Q1] Performance with varied reduced set sizes

We conduct an extreme experiment with a reduced set size = 1. Compared to our proposed method, which set the reduced set size to 550, the performance is relatively stable and still outperforms comparison methods. Therefore, the scalability of the method is guaranteed when the number of generative models increases.

As discussed in “3.3.Discussion”, the scalability problem can be handled using other techniques. We appreciate the reviewer's valuable suggestion and have added this experiment to our revised paper in Appendix A.4.

[Q2] Prompts associated with the qualitative results in Figure 4

The ground-truth prompt for requirement is a woman with blonde hair and blue eyes, a detailed painting, by rossdraws, fantasy art, red-purple gradient map, mercy from overwatch, close up of a blonde woman, a brightly colored, lux, sylas. The prompt generated by image interrogator is a girl with blonde hair and blue eyes, game icon stylized, glowwave girl portrait, epic legends game icon, app icon, 3 d icon for mobile game, icon for an ai app, darkness background, ios app icon, lux from league of legends, darkness's background, character icon, luxanna crownguard, lightning mage spell icon, darkness aura. The images generated by each method utilize prompts obtained from the image interrogator. We have added ground-truth prompts and generated prompts for each example in Appendix A.1.

[Q3] Performance of RKME-CLIP

RKME-CLIP is an ablation variant of the proposed method that addresses the dimensionality challenge by incorporating the CLIP model. Furthermore, our proposal enhances performance by taking into account the mapping between prompts and images.

[Q4] Three variants of proposed method

We conducted the experiments to evaluate two variants and our proposed method in the following table. Method (i) is our proposed method, which adopts the prompt generated from the image interrogator. Method (ii) adopts the ground-truth prompts of query images to calculate the weights. Method (iii) adopts an indicator function instead of soft weights.

The results show that methods (i) and (ii) perform similarly, indicating that prompts generated from an image interrogator can serve as a proxy for the ground truth when identifying the model. Method (iii) performs less effectively compared to the other two methods, suggesting the importance of soft weights in our proposal.

[Q5] Gamma in Figure 7

Gamma serves as a hyper-parameter in the calculation of RKME. The results demonstrate the robustness of our proposal to variations in this hyper-parameter.

[Q6] Explanations for Examples 2, 3 in Figure 5

Thank you for your valuable advice. We have added explanations for examples to our revised paper in Appendix A.1.

For example 2, our proposed method identified a model that can generate more cartoonish images. However, the models identified by other methods all generate more realistic images. For example 3, all methods did not find the most suitable model, but the model identified by our method was not inferior to other methods.

Dear Reviewer BSis,

We have responded to your questions point by point in our reply. It is important for us to know whether our responses have addressed your concerns, and we look forward to receiving your further feedback.

Feel free to reply if you have any further questions or suggestions. Thank you!

Best Regards,

Authors

Dear Reviewer BSis,

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

Best Regards,

Authors

[W5] Comparison with the 'Content-Based Search for Deep Generative Models'

The setting in 'Content-Based Search for Deep Generative Models' (referred to as CBS) differs from ours; therefore, it cannot be directly compared with our paper. We now highlight two key differences:

First, CBS addresses the search problem for unconditional generative models. These models generate images for a predefined concept and style, which can not set a specific prompt. Our paper, instead, concentrates on conditional generative models that sample images under the guidance of a prompt, a methodology that is more frequently employed by models in the model hubs. This difference yields our main challenge to model the mapping between images and prompts, which we discussed in 2.2 Problem Analysis and 3. Proposed Method. This difference also determines that the model zoo of CBS is large because their models only generate one concept in one style each. The models in our paper can generate diverse concepts under the guidance of prompts. Therefore, our model zoo is already large enough to demonstrate the effectiveness of our proposed method.

Secondly, CBS assumes that all models in the platform are known in advance, and employs a contrastive learning process to acquire the matching functions. This limitation significantly impedes the adaptability of the approach, as models within the platform are frequently being added and updated. It is impossible to re-train the matching function each time models are added or updated.

Hence, our paper addresses a more complex and realistic setting where the technique employed in CBS cannot be directly applied. We highly appreciate your valuable suggestion and have incorporated a discussion on 'Content-Based Search for Deep Generative Models' in our related work.

[Q1] Default prompt 'P'

This paper focuses on improving the identification of the most suitable model when prompts are already provided. Consequently, we utilize a default set of prompts for all generative models during the construction of specifications. This default set of prompts can be found in the "PromptPool.json" file provided in our supplementary material.

[Q2] Developer / user provided prompt

Thank you for your constructive suggestions.

Users should provide prompts that accurately describe the images they upload. In our proposed method, we generate prompts by employing an image interrogator that demonstrated effective performance in our experiments. When ground-truth prompts are provided, the identification performance will not be worse, and this is supported by our response to Reviewer BSis [Q3].

Developers should provide prompts to guide the model in generating images it excels at. A model-specific prompt set will enhance identification by amplifying the contrast between generated specifications, resulting in improved accuracy. We conducted synthetic experiments to validate this claim. In this experiment, developers will provide 5 prompts for each model, assuming they excel at them, and generate corresponding specifications. The user will query images generated using similar prompts. The results demonstrate that providing prompts for specification generation leads to improved accuracy. However, in actual situations, it is difficult for the model hub to force developers to provide prompts. Therefore, this paper conducts experiments under a default prompt set.

We greatly appreciate the valuable suggestion provided by the reviewer, and we have incorporated this discussion and experiment into our revised paper in Appendix A.5.

Dear Reviewer pBNB:

Thank you for the valuable feedback on our paper. We appreciate the time and effort you have put into reviewing our work and we are grateful for encouraging comments such as practical setting and the interesting idea. We have carefully read your review and addressed your concerns as follows.

[W1] Assumption on providing an example image

First of all, we need to point out that in the case that users cannot provide additional information, they can only browse and search one by one through the tags and illustration examples of the models in the model hub, which results in users not being able to quickly and accurately search for the models that satisfy their requirements. To realize a fast and accurate search, it is necessary to assume that users should provide additional information to describe their requirements. In this paper, we consider the cases that identify generative models with only one uploaded image to describe users' requirements. The assumption is reasonable since users' ideas often rely on existing image templates when they want to generate images, and it is not difficult to find only one image that has a similar style to the user's requirements. Moreover, it is also interesting to study how to quickly and accurately identify models via other information such as textual prompts, we will study this problem in future work.

[W2] Lack of novelty in the proposed method

We thank you for your review. It looks like you have some misconceptions about the contribution of our paper.

First, we propose a novel and realistic GMI problem. Generative models have garnered considerable attention, accompanied by the emergence of various generative model platforms. However, users face significant inefficiencies when seeking generative models that satisfy their specific requirements within these platforms, as this necessitates a meticulous browsing and testing process. In this paper, we introduce the Generative Model Identification (GMI) problem, encouraging researchers to develop algorithms describing the capabilities of generative models that enable precise alignment between model functionalities and user requirements, thereby achieving expedited searches in generative model exploration. We posit that such endeavors will substantially propel the advancement of generative models and model platforms.

Second, we take a successful step to address the GMI problem by proposing an effective specification for the generative model. It is noteworthy that the proposal is definitely not a simple combination of existing techniques. For example, we analyze the existing RKME method in Figure 2 and show that the existing method not only can not deal with the mapping between prompts and images which is essential for distinguishing generative models, but also can not address the dimensionality and cross-modality problem which hinders the success of GMI. The results in Table 2 clearly prove that any simple combination of existing methods cannot achieve good performance in GMI, and our proposal provides the best performance evaluated by all metrics. Therefore, it is not fair to consider our proposal as a simple combination of existing techniques.

Third, we construct and release a novel benchmark dataset for the GMI problem, which contains general stable diffusion models. Based on this benchmark dataset, we conduct comprehensive experiments to evaluate the existing baseline methods and proposed methods, and the results demonstrate the effectiveness of our proposed method. It is expected the dataset could facilitate the development of generative models.

We will highlight our contributions in our revised paper.

[W3] Writing problem

Thank you for your advice. We will improve the writing for technical sections.

[W4] A brief description of image interrogator

Thank you for your advice. We have incorporated the following description into our revised paper in Appendix A.1.

The Image Interrogator tool combines CLIP and BLIP to optimize text prompts for matching a given image. Firstly, it generates candidate captions using the BLIP model. Next, it employs a search process to identify the caption list that maximizes the similarity between the captions and images, as evaluated by the CLIP model.

Dear Reviewer pBNB,

We have responded to your questions point by point in our reply. It is important for us to know whether our responses have addressed your concerns, and we look forward to receiving your further feedback.

Feel free to reply if you have any further questions or suggestions. Thank you!

Best Regards,

Authors

Dear Reviewer pBNB,

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

Best Regards,

Authors

Dear Reviewer pBNB,

Thank you for your time and effort in reviewing our paper.

As the discussion period nears its end, we'd like to highlight that only a few hours remain before the deadline. We assure you that we've addressed your comments and included additional revisions. If you have any remaining concerns, please reach out in the next few hours. Your insights are crucial to us. If you do not have any further questions, we hope that you might consider raising your score.

Best Regards,

Authors