Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation

We thank the reviewer for their thoughtful comments and feedback. We would like to address their concerns and questions below:

1. Resource and latency comparison: Regarding computational resource requirements, our method does not necessarily require a GPU, as it can operate using GPT and DALL-E through their API calls. This makes it challenging to compare directly with other methods under identical settings (e.g., without a GPU). As for latency, we use a single NVIDIA A6000 GPU to provide a detailed comparison both below and in Section E in the appendix as an acknowledgement of this limitation. While our method may run slower when using a less efficient VLM, it also allows for flexibility in budget (the choices of N and K, as demonstrated in Section 4.4) and model selection, in order to reduce latency while still achieving competitive performance.

2. New ablation comparison between the judge score distribution of the first iteration and the final prompts: We thank the reviewer for suggesting this additional comparison and we have included a new judge score distribution comparison between the first iteration and the final prompts in Section D in the appendix. As we can observe from the plots, in the first iteration, the most common scores obtained are 0 and 1, whereas the final prompts obtain score 7 and 8 the most. This suggests a significant improvement of the prompt quality and effectiveness throughout the iterative process. Additionally, we would like to note that as we have mentioned above and in Section 3.3, grammatical correctness is not always indicative of effective prompts and as we can observe from the qualitative examples (Figure 22), as long as NLL reaches below 3.5, further lower NLL does not result in a qualitatively noticeable difference in terms of prompt quality. We would also like to invite the reviewer to re-examine other ablation studies we have conducted for different budgets, numbers of iterations and streams in Section 4.4 and Section D, where we have provided further insight and analysis on these aspects of our algorithms.
3. “Why are scores necessary and can we guarantee monotonically increasing scores?”: Due to the stochastic nature of the T2I model and VLM generation, we actually cannot guarantee monotonic improvement within a single stream. In fact, this is precisely one of the reasons why we need to incorporate the judge score and re-evaluation at the end, to make sure that we have accounted for this stochasticity. Another reason why we need to use judge scores is to provide signals for the self-generated improvements. Finally, without these judge scores, it is not obvious how to automatically pick the final best prompt among all the prompts we generate throughout the iteration process.
4. Definition of streams: By "streams", we refer to independent VLM conversations, which is different from the streams in techniques like beam search. However, incorporating more advanced search algorithms, such as beam search, can potentially improve the performance, which is an interesting future work direction.

Thank you for the response and additional results.

The rebuttal provided more information about the inference cost (latency) and clarified the role of score in stochastic generation models. Although more iterations do not always lead to good performance (!= higher judge score), it seems that the judge score can be a good approximation, and iterations tend to improve score. I still think the computation cost is considerably large, but it seems to be a characteristic of the proposed method and there exists pros and cons.

However, as I mentioned in the review Q2, given the experiment of NxK=30 (Figure 19 and Table 5), increasing N seems to be more beneficial than increasing K. It seems that the new experiments (Figure 23, 24) seem to support this, in that more streams (= random trials) increase the probability of high judge scored sample, as high scored samples can appear at the first iteration. If the benefit of increasing N is larger than increasing K, then the core idea of PRISM may be less represented.

Thank you so much for your recognition of our work and for considering an increase in your score. We truly appreciate your thoughtful evaluation and feedback to our paper. We also look forward to the continued discussion and insights from the other reviewers. We have corrected the typo in Figure 22 and we are happy to address any further questions or concerns throughout the remaining of the discussion period. Thank you so much again for your time and constructive feedback!

We thank the reviewer for their appreciation of the soundness, presentation and contribution of our paper, and their constructive feedback. We would like to address their concerns and questions as follows:

1. “How exactly does PRISM update the sampling distribution”: We leverage in-context learning to update the sampling distribution, as opposed to training or other model parameter optimization in traditional non-LLM paradigms. We have described this inference time learning process in Line 221 to 235. The key idea is that, because VLMs and LLMs use probabilistic chain rule to model the sampling distribution and perform next token prediction, previous chat history and prompts are naturally part of the “parameters” that define the sampling distribution of its next generated token. We update these “parameters” throughout the iteration with self-generated improvements, the reference images, generated image, previous chat history and the feedback provided by the VLM judge. Figure 15 is a qualitative example of the in-context learning procedure we conduct in PRISM.
2. Prompt length ablation and comparison with baselines: Thank you for bringing this to our attention. We use the default prompt length for all baselines, which in most cases corresponds to their optimal length. For many of these methods, increasing the prompt length does not necessarily lead to better performance. For instance, in PEZ's Section 4.2 “Prompt Length” paragraph, they explicitly note that a length of 16—rather than the longest tested length—yields the best generalizability. To further eliminate the possibility of unfair comparisons, we have also conducted an additional ablation study with GPT-4o-mini on the effect of prompt length for our model below and in Section D in the appendix.

In this experiment, we demonstrate the quantitative comparison between our method with various prompt length and PEZ, the baseline method constrained by prompt length, with its optimal length. Our results show that while PRISM benefits from longer prompt lengths, it consistently maintains high performance even with shorter prompts and significantly outperforms PEZ. Notably, we observed that as constraints on prompt length increase, PRISM tends to deviate from conventional coherent English sentences, similar to strategies employed by human prompt engineers. Additionally, unlike discrete optimization methods, longer prompt lengths do not pose significant challenges to the optimization problem inherent to our approach. It is also important to emphasize that all our experiments were conducted under the constraint that no prompt exceeds the maximum length accepted by the target T2I model, and any prompts exceeding this limit were appropriately chunked.

3. Ablation for VLMs: We would like to point out that an ablation study for the VLM models is already included in Section C.2 of the appendix. Regarding our choice of IDEFICS2 over more widely used options such as BLIP-2 or LLaVA, it is because BLIP-2 and LLaVA are not designed to process multiple images simultaneously, which is a capability essential for our algorithm. Additionally, we have tested our framework with GPT-4o-mini, a significantly smaller model than GPT-4v, and found that it delivers very comparable results. We hope this clarifies our design choices and supports the generality of our framework.

4. “Iterative prompting VLMs is expensive”: We fully acknowledge this potential issue and in fact this point has been mentioned in our limitation analysis in Section E in the appendix (Line 972 to 983). As we have mentioned in that section, the cost of high-performance VLMs has significantly decreased, as shown in Section C.2, where PRISM achieves comparable performance with GPT-4o-mini at a fraction of the cost (0.15 USD per 1M input tokens vs. 10 USD for GPT-4V, as of October 1, 2024). We anticipate further cost reductions in advanced models and improvements in open-source alternatives like IDEFICS2, which may soon rival GPT-4V. Additionally, PRISM-generated prompts can be reused across tasks and platforms, reducing long-term costs. Finally, PRISM’s flexibility allows users to adjust computational parameters (e.g., N and K) to fit specific budgets.
5. Prompt distillation and multi-concept generation: Thank you for mentioning these additional applications for our method. We have experimented with the “style transfer” task in Section 4.2 (Figure 4), and here we would like to demonstrate the application of PRISM on prompt distillation and multi-concept generation as well. PRISM is particularly well-suited for multi-concept generation due to the human readability of its generated prompts. This feature allows for easy identification and composition of different components within a scene, enabling intuitive control over multi-concept results. Unlike PEZ, which does not provide explicit control over which part of the prompt corresponds to specific aspects of the image (e.g., content or style), PRISM allows for much clearer and more direct manipulation. We are also grateful for the suggestion of prompt distillation. Unlike PEZ, which requires an additional optimization process to generate distilled prompts, PRISM leverages its highly interpretable prompts and the capabilities of LLMs to simplify prompts effectively. By using in-context learning and straightforward prompting, we achieve concise, distilled prompts without additional computational overhead. Qualitative examples of both applications have been added to the appendix in Section C.3 (Figure 17 and 18). We hope these examples further illustrate the versatility and strengths of our method.

Thank you for your time and thoughtful feedback on our paper. One of the reviewers (Reviewer uivB) has expressed interest in hearing your perspectives before finalizing their evaluation. If you have additional thoughts, questions, or concerns, we would greatly appreciate your input to further clarify or improve our work, especially as today is the final day we are able to make changes to the paper PDF. We look forward to your response and are happy to address any additional points that can assist in your review process.

Thank you for your response. We appreciate the opportunity to address the reviewer’s follow-up questions:

1. Comparison with CLIP-Interrogator + GPT-4V: We would like to kindly point out that integrating GPT-4V with CLIP-Interrogator is a significant modification to the baseline that involves substantial changes in both the nature of the algorithm and its implementation. In fact, one of the key contributions of our paper lies in how we select, prompt, and interact with VLMs. Therefore, it would be inappropriate to consider CLIP-Interrogator + GPT-4V as a baseline. However, exploring how a pre-collected keyword bank can enhance our method would be an interesting direction for future work – as an extension of our method and not as a baseline.
2. Addition of Prompt Length to Metric Tables: We would like to clarify that for all methods, we use the shorter of the algorithm's default prompt length and the T2I model's prompt length limit. We do not claim conciseness as a feature of our method. While we would gladly add this detail to the paper, the final deadline for modifying the paper PDF has already passed by the time this comment was posted. We will include this information in the camera ready version if the paper is accepted.
3. DreamBooth Dataset Size: We would like to emphasize that the DreamBooth dataset is the standard benchmark for the personalization task, which is the primary focus of this paper. The datasets used in PEZ are not constructed for T2I personalization tasks. Our experimental setup strictly follows the settings established in the original DreamBooth paper. Specifically, this includes 25 prompt templates for 30 objects. For the open-sourced T2I models, we repeat each prompt four times, resulting in a total of 30 x 25 x 4 = 3000 images for comparison. We believe this is a robust and reasonable dataset size for our evaluations.
4. Computational Burden and Application Scope: We respectfully disagree with the reviewer’s opinion regarding computational burden and scope. As demonstrated throughout our paper and the rebuttal discussions, our method supports a broad range of applications, including object-oriented personalization, style-oriented personalization, image inversion, prompt editing, prompt distillation, and multi-concept generation. Furthermore, our approach is highly flexible in terms of computational budget, as outlined in Sections 4.4, C.2, and E. Notably, our method can operate without GPUs, a feature that traditional discrete optimization methods like PEZ and CLIP-Interrogator do not support.
5. Comparison with PH2P: We would like to remind the reviewer that PH2P requires token optimization through diffusion models and therefore cannot be applied to T2I models that are non-diffusion based, black-box, or that consist of more than one text encoder. This limitation excludes almost all the T2I models in our experiments, as well as most best-performing T2I models in the current market. Because of its limited applicability, we do not think PH2P is a suitable baseline. While we do not consider PH2P as a baseline for this reason, we have already included it in the discussions in the introduction and the method section. We will also include more discussion in the related work section of the camera ready version if the paper is accepted. However, as mentioned earlier, the final deadline for modifying the PDF during the discussion period has already passed when this comment was posted, therefore we cannot make this change immediately anymore.

We hope this response addresses the reviewer’s questions. Thank you for your valuable feedback.

We thank the reviewer for their appreciation for the soundness, presentation and contributions of our paper, and their thoughtful feedback. We would like to answer their concerns and questions below:

1. More discussion about the potential weakness of VLMs as judge: We completely agree with the reviewer on this point. In fact, this is exactly the reason why we choose an LLM-based judge instead of a CLIP-based judge. In our ablation study in Section D, we have demonstrated the effect of using a CLIP model as a judge and its disadvantages. We appreciate the reviewer pointing out two additional references analyzing CLIP-based VLMs, which we have incorporated into our discussion in Section D. As for solutions to mitigate these issues, we have shown in our ablation study that using an LLM-based (autoregressively modeled) VLM can help solving this problem. However, these models are not perfect either. As future works, we can potentially introduce more rule-based reasoning in the iterative process similar to Mañas et. al (2024). We have reflected these changes in Section E of our updated manuscript.
2. The effect of iterations: We would like to invite the reviewer to examine our additional ablation study in the appendix (Section D), where we conduct extensive analysis on the tradeoff between the number of iteration and the number of streams and the effect of reevaluation. In Figure 22 we also provide qualitative comparison between different numbers of iterations and streams under the same budget.
3. “How are different similarity criterias captured by PRISM”: The different similarity criterias are defined uniquely for each task by the users, and the users can simply describe these criterias in the system prompts of the VLMs. Our experiments of personalization w.r.t. main object v.s. style (Section 4.2) is a demonstration of this capability of emphasizing on different visual similarities on personalization. Regarding the possibility to prioritize certain criterias, it is certainly possible because of the simple nature of prompting, so one can perform standard techniques such as repetition, bolding and capitalizing the phrases. We do recognize that the quality of user prompts can also be a factor that can potentially cause some gaps in achieving those similarities and how to effectively identify effective system prompts can be an interesting future work direction. We have also included our full system prompts for all three tasks we test in our experiments in the appendix.
4. Better contextualization in the Current Literature: Thank you for pointing this out! We completely agree with the reviewer,and we have made changes in the writings of the intro and related work by better positioning our paper in the literature of LLM as optimizer papers, and we have added an additional section in the appendix to address this issue.

We thank the reviewer for their constructive comments and suggestions. We would like to address their concerns and questions as follows:

1. Novelty and Positioning: Our intention is not to claim the invention of using LLMs as optimizers but to use them in a domain that no one has thought of applying. Specifically, we address the challenge of text-to-image (T2I) personalization and inversion, a problem that is difficult to solve using conventional discrete optimization methods due to their poor tradeoff between human interpretability and accuracy. In fact, One of the goals of this paper is to encourage researchers, particularly those in non-LLM fields, to consider how the advancements in LLMs can offer simple yet effective solutions to problems that pre-LLM methods have struggled to address. We have revised the introduction, related work and conclusion sections to better emphasize this point and better contextualize our contributions within the broader literature on LLM-based optimization.
2. Additional comparison with Related Work: We would like to emphasize that all related works the reviewer mentioned solve different tasks than our paper.

• Manas et al. (2024) and Hao et al. (2024) focus on prompt-image alignment and do not use reference images, as the reviewer has mentioned.
• Liu et al. (2024) addresses traditional distriminative tasks such as image classification, which is unrelated to T2I generation.
• Yang et al. (2023) (Idea2Img) is the most relevant work, but it focuses on generating a single best image rather than a generalizable prompt. In other words, Idea2Img only outputs a single best image tailored to a specific T2I model, prioritizing image quality for that one specific image without concern for the generalizability of the resulting prompts. In contrast, our method targets the generation of a generalizable prompt that works across different random seeds, contexts, and T2I platforms. This distinction accounts for the stochasticity in T2I models, where prompts must consistently produce high-quality outputs rather than relying on one best-case scenario. Unlike Idea2Img, which narrows its focus by selecting the best image at each iteration and outputs only a final image, we maintain independent streams throughout the process and use re-evaluation to identify the most effective prompt. Our approach enables broader applicability and ensures the prompts are robust and versatile across diverse scenarios. We have clarified these distinctions in our manuscript and conducted a direct experimental comparison with Idea2Img to further illustrate the differences.

3. Experimental Comparison with Yang et al. (2023): To highlight the differences between PRISM and Idea2Img, we modified Idea2Img to output prompts and tested both methods on the DreamBooth task using SDXL-Turbo as the target and testing T2I model. Since Idea2Img only outputs an image, it is not naturally applicable to our tasks. As a result, to ensure its applicability, we modify Idea2Img to output the prompt that produces Idea2Img’s output image. Results are summarized below as well as in Appendix Section F:

PRISM outperforms Idea2Img in most metrics, particularly in generalizability (CLIP-T), which measures contextual flexibility. Qualitative comparisons (Appendix Figure 25) further demonstrate that PRISM generates prompts with greater detail and contextual relevance, avoiding irrelevant or omitted information often seen in Idea2Img’s outputs. While Idea2Img achieves lower NLL, we note (as discussed in Section 3.3) that grammatical correctness is not always indicative of effective prompts and therefore fully coherent English sentences are not always the most effective prompts. Overall, both qualitative and quantitative comparisons show that PRISM strikes a better balance between human interpretability and prompt accuracy.

We are pleased to share that we have successfully parallelized and accelerated the image generation, and were able to complete the full comparison between our method and Liu et al. (2024). We have updated our manuscript to include this experiment in our main text and we also report the metrics below.

The results show that PRISM significantly outperforms Liu et al. in terms of CLIP-I scores across all tested T2I models, with only negligible differences in NLL and DALL-E 2 failure rate, where Liu et al. has a slight edge. We sincerely thank the reviewer for suggesting this comparison, as it has strengthened our paper and positioned it better within the LLM-related literature. We hope this experiment addresses the reviewer’s concerns and can convince the reviewer of the unique contributions PRISM makes to the field.

Thank you for your response. We would like to answer your following questions here:

1. Related work section: We would like to point out that we have included a three-page discussion about the related literature you mentioned in Section F in the appendix. Due to the page limit for the main text, we were unable to fit all these contents in the first 10 pages. Based on your feedback, we have further modified our related works section. Please let us know if you find the combination of the new related work section and the extended discussion in the appendix satisfactory.
2. Comparison with Liu et. al: The “prompt inversion” task in Liu et. al is indeed the same as the image inversion task in our paper. However, there are significant differences between Liu et. al and our paper in terms of algorithmic design. In particular, Liu et. al did not include the following components in their algorithm:
   • Self generated chain-of-thought improvement suggestions: In Liu et. al, they do not instruct the VLM to produce chain-of-thought improvements. In fact, in their official implementation, they specifically prompt the VLM to “Respond only with the revised text prompt and exclude any additional commentary”.
   • Judge model: Liu et. al does not incorporate an external judge model to provide signals for the iterative improvements.
   • Re-evaluation: Because of the lack of a judge model, Liu et. al is unable to perform re-evaluation. Both the judge model and re-evaluation have been proven crucial for our algorithm in the ablation study we conduct in Section D in the appendix.
   • Parallel search: Because of the lack of re-evaluation, Liu et. al is unable to perform parallel search since there is no way for them to identify the best prompts from the search. In fact, they can only assume that VLM can monotonically improve the prompt throughout the iterations, which we have proven to be not true in our ablation study. In the case of image classification, which is the main focus of their paper, they have described an alternative way to perform this search by leveraging the validation set and the classification error. However, with the image generation task, they were not able to find a straightforward way to incorporate these designs. As a matter of fact, they use n_restart = 1, n_reset = 1, m = 1 in their official implementation, which effectively makes this algorithm into re-prompting the VLM for several iterations in a single stream, without parallel search or beam search.

To demonstrate the importance of these algorithmic differences, we have tested Liu et. al in our image inversion task with GPT-4V and SDXL-Turbo. To ensure fair comparison, we have also included PRISM with the same iteration budget without parallel search (N = 1, K = 5) and the GPT-4V zero-shot results. Below is the quantitative comparison between our method and Liu et. al. We have included the same results as well as qualitative comparisons in Appendix Section F.

As we can observe, not only does Liu et. al underperform both PRISM versions, it even underperforms GPT-4V zero-shot with the system prompts we design for PRISM. This experiment shows the effectiveness of all components in our algorithm that Liu et. al misses.

We are more than happy to conduct the full image inversion comparison with Liu et. al and update the comparison table in the main text because we think this can significantly strengthen our paper. However, since the full comparison involves slower T2I models such as SD 2.1, we may not have enough time to finish all the experiments before the discussion period ends. We will certainly post an update here if they are finished in time, and will definitely include them in the camera ready version if our paper is accepted. Regardless, the experiment above is sufficient to show the superiority of our algorithm design when compared to Liu et. al in the image inversion task.

We hope this response can answer your concern and please let us know if you have any further questions.

Thanks for reporting this extra comparison. Re the lack of CoT reasoning in Liu et al, I double checked their paper and implementation, and they have a mismatch between what they report in the paper (Fig 3, 4, and Section 10) that shows CoT and what they do in the implementation (no CoT reasoning). This mismatch is their fault, and therefore I consider your comparison fair and acknowledge your superior performance. Though, I still have some doubts about the magnitude of the contribution of your work compared to them and other related works. The framework is not novel and what gives you best performance at the end of the day are better meta/system-prompts for GPT-4V (since your external judge is the same GPT-4V that does automatic prompt engineering but prompted differently), together with bootstrapping/re-evaluating different initial prompts.

All in all, the method obtains sota results in the task of personalization and prompt inversion, however there is very limited novelty in the proposed approach, which to me outweights the performance improvements. It is painful given the great effort put in the rebuttal and the well-executed paper, but I decided to keep my rating. In case of rejection, I suggest the authors to target a venue where novelty is less important, e.g., TMLR.

Thank you for your thoughtful response and for taking the time to revisit and thoroughly evaluate our work. While we are sorry that we could not fully convince the reviewer of the novelty of our approach, we respectfully disagree with their assessment. We believe that inference-time search algorithm design is an important and non-trivial aspect of LLM-related research. This research direction has demonstrated its potential in fields like reasoning and mathematical problem solving [1,2] but remains under-explored in many other areas. T2I personalization and image inversion is one of these under-explored domains that has significant practical and immediate applications to enhance creativity and accessibility for everyday users. Our paper provides systematic evaluations and ablations of different designs, and proposes an effective and portable algorithm that achieves SOTA performance for this problem.

That being said, we sincerely appreciate the reviewer's constructive feedback. This rebuttal discussion has been truly helpful for strengthening our paper and for better contextualization in the related literature. We are very grateful for the opportunity to improve our work and for your time and effort throughout this process.

Thank you again for your thoughtful review.

References:

[1] OpenAI. "Learning to Reason with LLMs." https://openai.com/index/learning-to-reason-with-llms/

[2] Qwen Team. "QwQ: Reflect Deeply on the Boundaries of the Unknown." https://qwenlm.github.io/blog/qwq-32b-preview/