Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution

We thank the reviewer for their detailed comments to improve our paper. We are glad the reviewer found Promptbreeder promising and our survey of related work extensive, supporting the originality of our approach. We address your concerns below.

Comprehensive Comparative Study: We respectfully disagree with your claim that our paper lacks sufficient experimental results. Promptbreeder was benchmarked against various algorithms across nine distinct problems, in fact going beyond the standards as seen in published state-of-the-art work like APE [1] and Plan-and-Solve [2]. However, to address your point, we've expanded the comparison in Table 1 of our revised draft even further, adding three more control groups to reinforce the validity of Promptbreeder.

[1] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2023). Large Language Models Are Human-Level Prompt Engineers. arXiv. https://doi.org/10.48550/arXiv.2211.01910

[2] Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023). Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. arXiv. https://doi.org/10.48550/arXiv.2305.04091

We also point to the fact that Promptbreeder outperforms APE on 21/24 instruction induction tasks which are outlined in the Appendix L.

Ablation Analysis:

You state that our paper “lacks the ablation analysis to show which components are effective and how effective they are”. You will be glad to hear that our original submission already contained a comprehensive ablation analysis in Appendix L, however, we have moved this to Appendix B as it was easily overlooked. It shows that every component of Promptbreeder is important, and the interplay of different mutation operators is what makes Promptbreeder effective. We agree that this ablation analysis is crucial to show which components of Promptbreeder are effective, so to decrease the chances of other readers overlooking this important part of our work, we have made the reference to Appendix B in bold in the main text. We thank you for bringing this to our attention and improving the paper in this way.

Theoretical Motivation:

Your observation on Promptbreeder's reliance on past prompt strategies is valuable. We designed Promptbreeder to not only incorporate but also evolve beyond these strategies. The addition of genetic algorithms to our framework is grounded in existing literature, advocating for diverse and non-greedy exploration that circumvents local optima. We've elaborated on this in the revised draft, highlighting how Promptbreeder transcends mere aggregation of existing strategies. Control 1 in our paper demonstrates that initializing task prompts with these strategies is less effective than employing evolutionary evaluations.

Rationale for Baseline Selection:

The selection of baselines was driven by a desire to compare Promptbreeder with the latest state-of-the-art prompting strategies. Additionally, two control experiments were conducted to validate the efficacy of evolutionary processes in Promptbreeder over random search and to demonstrate improvements over the initial problem descriptions used as prompts.

Summary:

In conclusion, we respectfully ask you to consider the enhancements made in our revised draft, which address your concerns in detail, and to increase your support for our paper.

Thank you for your answer. I found it very interesting.

Could you explain clearly me a little more about the following points? I would like to check the effectiveness and novelty of individual proposals.

• How insensitive is the proposed methodology to existing strategies and types of LLM? Please provide any evidence of this.
• Also, there are several search methods to prevent genetic algorithms from becoming a local optimum solution, can you compare them and highlight what you have devised for Promptbreeder's?
• Please tell us about your evaluation methodology for reproducibility. Which tools or library do you use?

We thank the reviewer for their insightful feedback and support for our paper. We are glad to hear they found the method innovative, the empirical performance exceptional, and the write up clear and well structured. We address your concerns below.

Reliance on LLM Interactions:

We appreciate your point about Promptbreeder’s interaction intensity with LLMs. As a general-purpose self-referential self-improvement mechanism, Promptbreeder leverages LLMs to mutate a population of task-prompts, evaluating their fitness on a training set over multiple generations. This process inherently involves significant interaction with LLMs, but it is crucial for the evolutionary approach that underpins Promptbreeder's effectiveness​​. The sample efficiencies of the algorithm for each dataset are shown in the Appendix K for each problem. For example, on AQuA the best individual was produced after 1400 mutations which equates to 1400 x 2 x 100 Q:A pairs tested. While this may seem costly at first, we believe Promptbreeder will have important practical implications as the time and cost to find a superior prompt is a one-time cost for a given domain, and amortized with respect to many LLM inferences at deployment time of the prompt.

Lack of Asymptotic Guarantees:

Note that such guarantees are generally not available for evolutionary algorithms in realistic large and noisy problem spaces. However, we have demonstrated empirical superiority to other optimization techniques such as APE, random task prompt initialization (see our own controls in Table 1), and ORPO on GSM-8k. We have shown that using the diverse set of initial task prompts improves Promptbreeder’s search for effective prompts in various domains.

Content Distribution in Main Text and Appendix:

Thank you for your suggestion to improve our paper. We have moved Figure 3 into the main paper in improved form.

Again, thank you for your valuable contribution to improving our paper.

Thank you for your thoughtful review of our paper on Promptbreeder. We appreciate your recognition of our systematic framework's ability to evolve domain-specific prompts and its comparative effectiveness. We address your concerns below and offer clarifications and updates to our methodology and results.

Regarding the Experiment's Extensiveness:

We acknowledge your observation about the lack of inclusion of prominent models like GPT-3.5 or GPT-4 in our initial experiments. In response, we have carried out experiments with GPT-3.5 on GSM-8k showing successful improvement of prompts to 65.5% test set accuracy, see Appendix M for further details. Additionally, as per your suggestion, we have re-run the experiments using Chain-of-Thought (CoT) and other methods on PaLM 2-L. Our updated results demonstrate that Promptbreeder shows consistent improvements.

Concerning the Requirement of Initial Information:

You rightly pointed out that the effectiveness of Promptbreeder depends on the initial information provided, such as task descriptions or mutation prompts. To address this, we have included a new section in the Appendix of our revised draft and carried out two more control experiments showing that even with poor and general problem descriptions, PB shows consistent improvements in fitness.

• Generation of Thinking Styles and Mutation Prompts from Problem Descriptions: We have developed a method by which thinking styles and mutation prompts can be autonomously generated from the problem description itself, reducing the dependency on external initialization.
• Concerns about the initial problem description information provided to Promptbreeder: We demonstrate (Appendix N) that even with neutral or misleading problem specification (task descriptions) Promptbreeder achieves substantial improvement in terms of the test set fitness of evolved prompts.

We respectfully disagree with the implication that our method may not generalize across various tasks. The aforementioned enhancements and experimental results strongly suggest that Promptbreeder is not only adaptable but also capable of performing effectively across a diverse range of tasks, even with minimal or suboptimal initial information. Furthermore, the fact that Promptbreeder outperforms APE on 21/24 instruction induction tasks also highlights this, see Appendix L.

Summary:

We believe these updates and clarifications address your concerns regarding the breadth of our experiments and the generalizability of Promptbreeder. We kindly request you to reconsider the contribution score in light of these significant improvements and additional data. Moreover, if there are any further queries or points of contention, we are more than willing to provide additional clarifications to strengthen the case for acceptance of our paper.

We thank the reviewer for their detailed comments to improve our paper. We are glad to hear the reviewer agrees that Promptbreeder tackles the very important research question of whether LLMs can self-improve their “thinking process” via self-generating prompts. Furthermore, it is great to hear that the reviewer found the set of mutation strategies investigated in this work comprehensive, and the paper generally interesting. Below, we are addressing the reviewer’s concerns.

1.& 2. Prompt Space Exploration in Evolution Algorithms:

We acknowledge your concern regarding the adequacy of evolution algorithms for exploring the large prompt space in Large Language Models (LLMs). To address this, we have updated Figure 3 (and, as suggested, moved it from the Appendix to the main part of the paper—thank you for the suggestion!) to showcase not only the highest-performing ('elite') prompts but also a variety of low-fitness prompts generated during the exploration phase. This figure shows that there is in fact considerable exploration in prompt space, with many prompts being produced that are very different to the elite prompt. Elite prompts also change in subtle ways throughout the evolutionary run and achieve much higher fitness at the end. The final prompt is "Sentences are given, and a single word. The output should indicate whether the given word has the same sense in the two given sentences, yes or no." with an early elite prompt in the run being "I'll give you two sentences and a word. Your task is to write if the meaning of the word is the same in both sentences or not."

1. Comparison with Gradient-Based Prompt Tuning (i.e. Soft Prompting) Methods:

While we agree that a comparison to soft-prompting methods could be valuable, it falls outside of the scope of this work as well as outside of the scope of prior published state-of-the-art prompt strategies such as Chain-of-Thought Prompting [1], Plan-and-Solve Prompting [2], as well as APE [3] and OPRO [4], to name just a few.

[1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., … Zhou, D. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv. https://doi.org/10.48550/arXiv.2201.11903

[2] Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023). Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. arXiv. https://doi.org/10.48550/arXiv.2305.04091

[3] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2023). Large Language Models Are Human-Level Prompt Engineers. arXiv. https://doi.org/10.48550/arXiv.2211.01910

[4] Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023). Large Language Models as Optimizers. arXiv. https://doi.org/10.48550/arXiv.2309.03409

3. Simplicity and Novelty of Mutation Operators:

Although some of the mutation operators in Promptbreeder are standard in evolutionary algorithms, their application within the domain of LLMs is novel. Also some operators simply cannot be implemented without LLMs, e.g. the lineage based operator, or the context to prompt operator. Designing these operators for LLMs posed unique challenges, particularly in balancing diversity and relevance of the generated prompts. We demonstrate the effective use of LLMs in evolutionary algorithms, a contribution we believe is significant in this field. Furthermore, we argue that the simplicity of the individual mutation operators of Promptbreeder is in fact a strength of the approach, demonstrating that existing mutation operators can be orchestrated to create a capable self-referential self-improving system.

4. Base LLM comparison:

Thank you for this comment. We have extended the baselines with PaLM-2L as suggested, and added Chain-of-Thought Prompting (COT and Manual COT) with PaLM-2L as the base model to the results. As expected, Promptbreeder outperforms these baselines by a large margin.

Summary:

We hope that our clarifications will lead you to consider increasing your rating of our paper or detailing what still stands in the way of increasing your support, so that we may further improve it.

We thank the reviewer for their detailed comments to improve our paper. We are glad to hear the reviewer found our empirical results promising and the method well elaborated. Below, we are addressing the reviewer’s concerns.

Generalization:

Many thanks for the expressing the concern that this study exclusively shows the performance of PROMPTBREEDER with PaLM 2-L. To address this we ran PB on GSM-8k using GPT3.5-Turbo achieving test set scores of 65.5% and 63.9% accuracy respectively. Results are shown in Appendix M. The PS+ prompt only achieved 44.7% with GPT-3.5-Turbo. This provides evidence that PB generalizes to other language models.

Absence of Navigation:

The absence of directed navigation is an important component of an evolutionary algorithm. The success of evolution relies on sequences of random mutations that, when accumulating over time, lead to increased fitness. Promptbreeder, an evolutionary algorithm, makes use of this property. In fact, even keeping around less fit (but less specialized) individuals in an evolving population can be important for the entire evolutionary system to arrive at better solutions in the long term [1]. A more directed Promptbreeder approach is conceivable, but it would move Promptbreeder away from an evolutionary approach which is outside of the scope of our paper.

[1] Sudholt, D. The Benefits of Population Diversity in Evolutionary Algorithms: A Survey of Rigorous Runtime Analyses, 2018. https://arxiv.org/abs/1801.10087

Efficiency in Prompt Optimization:

In response to your concern about efficiency, we have introduced two control experiments: the Random Baseline and the PD baseline. The Random baseline evaluates 2,000 prompts generated by our initialization method without evolution, demonstrating that evolution contributes more than a random search. The PD baseline, where task-prompts are set to the problem description, shows that evolution improves performance beyond the initial problem description.

Origin of "Mutation Prompts" and "Thinking Styles":

We appreciate your inquiry about the creation of the "Mutation Prompts" and "Thinking Styles." These were hand-designed for general problem-solving domains and are thus widely applicable (as demonstrated by our empirical gains on multiple diverse reasoning benchmarks). We have added a new section (Appendix G) detailing how an LLM can generate these lists from a problem description using a hierarchical generation method.

Transfer of Promptbreeder Prompts to other LLMs:

To address your question about Promptbreeder's applicability to other LLMs, note that we do not claim prompt transferability across LLMs—Promptbreeder is designed to find prompts that are tailored to the unique characteristics of each LLM for optimal task performance through automated prompt engineering. We will clarify this in a revised version of the paper.

Clarifications on Figure 3, OPRO Evaluation, and Color Coding:

We have corrected and enhanced Figure 3, providing clarity on the relationship between evaluations and generations (a generation being POP_SIZE evaluations). Regarding OPRO's evaluation on GSM8K, we suggest the authors of OPRO may consider evaluating it on a broader range of tasks, as we have done in our study. We have standardized the color coding for task-prompts and mutation-prompts throughout the paper to enhance readability and coherence.

Summary:

We again thank the reviewer for helping improve our paper. We hope that our clarifications will lead you to consider increasing your rating of our paper or detailing what still stands in the way of increasing your support.

Thank you for your thorough responses, which have addressed many of my initial concerns through additional experimental results and analysis.

However, I continue to have reservations about the efficiency of the proposed PB. Although the use of an evolutionary algorithm demonstrates certain advantages over the '2000 random prompts' approach, the marginal improvement of PB over the Random baseline is less pronounced when compared to other prompting strategies except in the GSM8K. As evidenced in Table 1, the Random baseline outperforms most other baselines in 7/8 datasets, suggesting that a large number of various prompts plays a more significant role in most tasks, which lacks novelty, in my opinion.

Moreover, I am concerned about the scalability of experiments using PB, especially when 2k prompts are needed to be evaluated. Factors such as dataset size, input length, and the use of different LLMs when opting for paid APIs like GPT-4 over GPT-3.5-Turbo, could substantially increase the time and cost of experimentation. This could pose a disadvantage for researchers and practitioners with limited resources.

In conclusion, while the revised manuscript shows commendable improvements and the authors have provided detailed clarifications, the persistent concerns mentioned above led me to maintain my initial rating.

Hi Sam,

Our original submission is 9 pages. We only went to 10 pages during the rebuttal phase to address the reviewer's comments. Multiple reviewers asked us to move content from the appendix to the main part of the paper.

Kind regards, Authors