AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models

Thank you for the constructive comments. We will address the concerns below.

Q1. The authors did not conduct relevant ablation experiments on CLIP.
A1. The computation of domain prototypes indeed relies on the image encoder. Moreover, domain prototypes are concatenated with text representations to serve as control conditions for image generation, so they should be as similar to text representations as possible. Given that stable diffusion 1.5 uses CLIP as the text encoder, we also use CLIP as the image encoder, ensuring that the image representations share a feature space with the text representations. In the appendix, we explore the impact of CLIP on domain prototypes. The experiments include Figure 8 and Table 4 in section C.2.

Q2. The human validation lack detailed explanations.
A2. We followed the approach presented in the SUR-adapter[1]. We collected 108 valid questionnaires. Participants were shown images generated by our method and baseline methods, along with their corresponding descriptions. They were asked to choose the better image based on the questions "Which image do you think has higher quality?" and "Which image do you think better matches the text description?" We then compiled and analyzed the survey results.

Q3. The proposed method introduces additional models such as a pre-trained LLM and CLIP for prototype computation.
A3. The LLM is only used in the first stage for automatic prompt generation, employing the ICL strategy, and does not participate in subsequent training. The LLM’s model parameters are not updated. During training, the parameters of CLIP are frozen and do not get updated. During inference, CLIP is no longer used; only the trained domain prototypes are involved. Therefore, training and inference times are not significantly increased.

Q4. The data collection process adds to the workload and complexity of the proposed method.
A4. All existing APO methods require a large number of manually designed prompts as references for training. Moreover, such data collection and retraining should be repeated for the APO on each newly-coming diffusion model. Our method is desined to address this issue by eliminating the need for new data preparation and retraining when encountering new diffusion models. Our trained adapter can be directly used on new models without retraining.

[1] Zhong S, Huang Z, Wen W, et al. Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 567-578.

Thank you for the constructive comments. We will address the concerns below.

Q1. The quality of writing can be improved.
A1. Thank you for your suggestions. We will optimize the writing in the methodology section to make it more concise and easy to understand.

Q2. What’s the difference between the training and target domains?
A2. In our dataset, each checkpoint is fine-tuned using images of different styles and content, resulting in varying generative capabilities. During experiments, we use a subset of these checkpoints as the training set and test on another subset. The fundamental difference between source and target domain checkpoints lies in the style and content of the fine-tuning data, not the number of the data.

Q3. What exactly is being updated in equation 11?
A3. In Equation 11, the parameters of $q_i^k$ will be updated. $q_i^k$ is the output of $g_{\text{ada}}$ in Equation 9. Thus, the variables being updated are the parameters of the transformer layers and linear layers in the adapter.

Q4. Compared to proposed method, the baselines seems to be working better in Figure 2.
A4. We found that the issue lies in the ICL-based prompt rewriting in the first stage. As shown in Appendix C.1, the prompts rewritten by the large language model do not include the word "standing." This indicates that while the adapter in the second stage can enhance the generalization of the automatic prompts to new models, it cannot compensate for the information lost by LLM in the first stage. Therefore, the quality of the automatic prompts is also crucial.

Q5. Fig. 2 need to be improved.
A5. Thank you for your advice. We will improve the quality of the figures and provide clearer figures in the final version.

Q6. Given the complexity of the proposed method, the gains are expected to be larger, which is not the case.
A6. Compared to baseline methods, our approach requires significantly fewer parameters to be trained, limited only to the domain prototypes and adapter. Therefore, our method is not more complex. Additionally, our approach performs better than other methods in most cases, as shown in Table 1 and Figure 2.

Thank you for your response. Based on the response of the authors, I have few more queries below:

• Q2: Authors mention that images sourced until checkpoint 40 are taken as source data and from checkpoint 41 as target data and they differ in content and style. How do authors ensure the style and content of target images are different. They are still from the same data distribution, hence more clarity is needed at this point.

• Q4: Authors argue that LLMs inability to generate the correct pose for cat is the reason behind the incorrect pose of cat in the final image. This suggests that the proposed method is heavily dependent on the quality of the prompt generated by the LLM. How do authors aim to improve or get rid of this dependency? or is there any alternate way to get around such issues

• Q6: I think authors want to point out at Fig.3 (Fig 2 is a block diagram). Kindly request the authors to put correct references. The scores are still marginal. I do not see any steep jump in scores of semantic consistency. For example proposed method has a color score of 0.477 while SUR adapter has 0.472 which is very marginal increment. Same is with other cases as well.

Dear Reviewer,

We sincerely appreciate your engagement in the discussion and your recognition of our rebuttal efforts. Thank you very much for your willingness to increase the score.

Since the rating in the original review has not yet been updated, we are unsure whether this is an oversight and would like to kindly confirm if the rating has been changed in the system.

Thank you once again for your time and valuable comments.

Thank you for further clarification. I appreciate authors putting more work on explaining the things. I am increasing my score to weak accept. The paper is good however, the increments as compared to the baselines are not that great.

Thank you for the constructive comments. We will address the concerns below.

Q1-1. What’s the the main reasons for this substantial gap between automatic and manual prompts?
A1-1. The effectiveness of manual prompts lies in their iterative nature, allowing humans to refine prompts based on the generated images until the desired quality is achieved. In contrast, automatic prompts aim to replicate this effectiveness by learning patterns from manual prompts.

Q1-2. What fundamental limitations of automatic prompting hinder its improvement?
A1-2. The key limitation is that APO models struggle to evaluate whether an image meets specific requirements and to re-optimize the prompts accordingly.

Q1-3. Are there any promising directions to overcome these limitations?
A1-3. One potential solution is to incorporate reinforcement learning by designing a reward model that simulates human judgment. This would enable the model to dynamically identify and correct deficiencies in the current prompts based on the generated images.

Q1-4. Can we consider the proposed method as addressing some of these limitations?
A1-4. Our approach does not focus on generating superior automatic prompts but rather on enhancing the generalization of automatic prompts across different checkpoints, thereby avoiding the need for repetitive retraining of the prompt generation model for new checkpoints.

Q2-1. If the process of removing domain-insensitive information in the proposed method could result in the loss of important information?
A2-1. As shown in Figure 2, in the adapter section, the final adapted embedding is obtained by concatenating the prompt embedding and the Prototype-anchored embedding, followed by transformation. During this process, domain-insensitive information remains in the prompt embedding and is not completely removed, ensuring that important semantic information is not lost.

Q2-2. Is it necessary to continuously increase the number of domain prototypes to adapt to new data?
A2-2. We do not require the target domain to have perfectly matching domain prototypes. Instead, we assume that the target domains are entirely new and unseen, aiming to anchor their position using existing domain prototypes. To achieve this, we selected checkpoints with diverse styles as our training data to maximize the adaptability of the prompt representations. Consequently, we do not need to continually increase the number of domain prototypes.

Q3. If data could have introduced some biases in favor of the proposed method, leading to better results.
A3. Our basic assumption is that all checkpoints must be with the same structure and fine-tuned based on the same base SD model using different data. Our method addresses the generalization of automatic prompts across different checkpoints. If the target model is with entirely different structure from the training model, such as SD XL or DALLE, the problem itself becomes unsolvable, as we cannot predict the behavior of an entirely unknown model. Generalization is only possible on checkpoints derived from the same model but fine-tuned with different data.