PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization

(e) Q2: Selecting hyperparameters. Similar to many other empirical studies, we select hyperparameters based on intuitions and preliminary experiments. We now show additional hyperparameter selection experiments on iteration number and the exploration weight w_exp.

RTable 7: Selecting hyperparameter: Iteration number

Iteration. In the paper, we select the iteration based on our experience that a small number of iterations can't explore the prompt space well, but too many iterations are not necessary because they may have a potential problem of overfitting on the training set. We add new experiments in RTable 7 to test various choices of the iteration number, including iterations of 8, 12, and 16. The iteration of 8 is too small, which leads to a drop in performance, and the iteration of 16 is too large for the overfitting issue, leading to a marginal drop.

RTable 8: Selecting hyperparamrter: Weight for balancing exploration vs. exploitation

w_exp. In the paper, we choose w_exp=2.5 because the formula of the exploration rate is $w\_{exp}\times\sqrt{ln (N_{parent}/N_{child})}$, so we set some values of the visited times of parent node $N_{parent}$ and child node $N_{child}$. We find w_exp=2.5 can make the change of this value at a reasonable scale, when the $N_{parent}$ and $N_{child}$ increase. We also show new experiments in RTable 8, showing that when w_exp is small (1.0), the MCTS will only have one path, while w_exp is large (4.0), the MCTS will almost generate a new path in each iteration. Both of them are out of balance; w_exp is small, so the prompt space is not well explored; while w_exp is large, each path won't be visited multiple times to update the reward; we thus use w_exp=2.5 as a reasonable setting.

(f) Response to the ethics concerns. We thank a lot for reviewer j86n for making such a great and thoughtful comment on being cautious about incorporating LLM feedback into the prompt. We couldn't agree more with adding an additional layer on top of prompting sensitive tasks, such as the loan approval task. The good news is that humans, especially domain experts, understand natural language prompts crafted by PromptAgent. Additional human supervision could be necessary sometimes, while we can further improve the method to be capable of detecting such biases and assigning lower rewards to these feedbacks. Specifically, PromptAgent presents a flexible framework to incorporate more powerful and advanced rewards to guide the prompt optimization process in a more principled way. Future rewards to measure the hallucinations, biases, ethical concerns, etc., of LLM feedback are very promising research directions to study.

References: [1] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022, September). Large Language Models are Human-Level Prompt Engineers. In The Eleventh International Conference on Learning Representations.

We thank Reviewer j86n for the detailed evaluation of our work, for acknowledging that our soundness is good, our paper is well-written and easy to follow, related literature is covered, our experiments are comprehensive, shedding further light on the effectiveness of the proposed strategic exploration of the search space using MCTS, and our appendix provides useful details. We find reviewer j86n has a slightly different understanding of the core novelty and contribution of our work (we though thank the reviewer for the self-debating). We'll thoroughly address and clarify these mismatched understandings as follows, hoping the reviewer will be on the same page with us at the end of this discussion.

(a) Clarifying the novelty. We kindly redirect the reviewer to our global response (a), where we thoroughly address the technical novelty and core contributions in PromptAgent. Regarding the concern from reviewer j86n, we argue that interpreting PromptAgent simply as "use of MCTS in prompt engineering" is neither accurate nor faithful to reflect its core contributions.

However, we're glad to see reviewer j86n acknowledge our other novelties and contributions as base/optimizer formulation and incorporate error feedback. Given the partial consensus we've made here, we hope we could acknowledge this as the strength of PromptAgent rather than its weakness.

(b) Clarifying the contribution of MCTS. When comparing APE with PromptAgent, reviewer j86n admits the importance of the error feedback mechanism, instead of the utilization of MCTS. The reason probably is that they find APE prompts in our paper (appendix F) are not detailed enough, making them question the usefulness of iterative search methods in deriving expert-level detailed prompts.

1. First, we agree with the reviewer on the importance of error feedback in inducing domain knowledge. We implemented two versions of the Monte Carlo search using the same error feedback mechanism: MC-single: iteratively generates one path with a depth of 8, and MC-depth=1: directly sample 72 prompts from the initial prompt for the selection. Both can be viewed as variants of APE within the PromptAgent framework. Note that the original APE doesn't have the error feedback mechanism and, thus, cannot effectively induce detailed domain knowledge and instructions into the prompt. The figure from the original APE paper shows their optimized prompts are actually short with little domain knowledge.

• We now show prompts from our APE and MC-dept=1, whose major difference is APE doesn't have the error feedback. The following examples also show that even with only a single sampling method, the error feedback optimization step can integrate much more domain knowledge than APE.

MC-depth=1 prompt for the Subjective task:

Analyze the presented text in its entirety. Do not focus on individual words or phrases, and remember that objective descriptions can also include factual information about emotional states or conditions. Choose whether the text is 'subjective', characterized usually by perspectives, opinions, beliefs, and feelings, or 'objective', typically including verifiable facts or descriptions, regardless of its tone, style, or phrasing. Keep in mind that the text's intention and the nature of its content have to be distinguished. The text can be informal and still convey objective information. With this in mind, classify the given text as either 'subjective' or 'objective'.

APE prompt for the Subjective task:

Determine whether the provided text is stating facts and details (Objective) or expressing personal views, emotions, or choices (Subjective).

MC-depth=1 prompt for the Geometry task:

Given an SVG path, interpret the formula to deduce the specific geometric shape it forms when all commands are completed. Keep in mind that the path commands work together to create a figure which represents a simple or complex geometric shape, and not just individual lines. Be sure to count vertices and pay attention to the structure created. Lastly, for this task, we are only interested in identifying geometric shapes such as circles, triangles, pentagons, hexagons, heptagons, kites, octagons, rectangles, sectors or lines. Try to match the resulting figure with one of these provided options.

APE prompt for the Geometry task:

Determine the shape each SVG path element is drawing, then pair it with the corresponding letter from the available choices. In this case, C symbolizes hexagon, G is for pentagon, I signifies sector, and B stands for heptagon.

We thank reviewer zmiy for acknowledging that our idea is interesting, our work is clear and easy to follow, our writing is clear, and PromptAgent could be useful for greater practical usage. The reviewer's main concerns lie in the technical novelty, one baseline of human prompt, and some experiment settings regarding our ablation study of search variants. We now thoroughly address all the concerns and questions as follows:

(a) Clarifying the technical contribution. We'd like to kindly redirect the reviewer to our global response (a), where we thoroughly address the concerns on the technical novelty and core contribution of PromptAgent. We hope our response could help the reviewer better understand the contributions of PromptAgent.

(b) Clarifying the baseline of human prompts. We understand the reviewer's concerns that the average user or domain experts can also be further augmented with multi-round iterations by humans themselves. However, we argue that this concern shouldn't be considered as the weakness of our work for the following reasons:

• First, as described in the section 4.1 Baseline paragraph, we try our best to directly adopt the human prompts from their original datasets and references. For datasets without readily available original human prompts, we carefully craft the prompts based on task description, options, and purpose of the tasks. The original dataset creator and we, as the normal users, create those human prompts, reflecting a prompting level of average users, thus serving as a legitimate and rightful baseline to compare with.
• Second, adopting such an average level of human prompting has been a traditional baseline for prompt optimization literature. For example, Zhout et al. (2022) [1] proposed APE (Automatic Prompt Engineer), a Monte Carlo search method, to compare with and outperform human-level prompts, the same as our human prompt baseline. Then, almost all follow-up prompt optimization works for APE continue leveraging the iterative fashion of multi-round prompting and compete with the generic human prompts. Based on this, this concern is not limited to and cannot be blamed on our own work.
• Third, instead of considering the disadvantage of humans when comparing with prompt optimization methods in general (not limited to our own work) as unfair, we'd better consider the multi-round prompting of prompt optimization methods as their advantage of machine efficiency. The disadvantage of humans under such circumstances is exactly the motivation for prompt optimization methods to explore advanced search methods to craft better prompts automatically, even with the cost of multi-round promoting. As stated in the second introduction paragraph of our paper, combining the merits of human trial-and-error and the machine's efficiency in doing iterative prompting creates more advanced methods, like PromptAgent.
• Last but not least, human prompting is only one of our basic baselines, probably the weakest one in some tasks. We thoroughly consider many other stronger baselines, including human-written few-shot prompts, Chain-of-thought prompts, GPT Agent, and APE--a strong prompt optimization method, which thoroughly demonstrates the effectiveness of our proposed method. Therefore, the concern around one minor baseline is better not to be considered as the major weakness of our work.

We are grateful to the Reviewer WJJm for recommending the acceptance of our paper and acknowledging that our work is solid and excellent sound, our writing is well-written and easy to follow, and our method is clean, straightforward, and promising. There are a few clarification questions from the reviewer, and we address them thoroughly as follows:

(a) Clarifying the reward function. A clear definition of the reward function can be found on page 15 of our paper, "The reward function is determined by the base model's accuracy (or F1 score) on the sub-training set". To be more specific, for each task, we will split a part of the training set for reward calculation. For now, we use a simple strategy: calculate the accuracy or F1 score on the sub-training set. Moreover, similar to reward engineering in RL research, we expect future advanced rewards to be explored and crafted for PromptAgent, such as the self-evaluation and self-consistency rewards. Nonetheless, we found the task-specific reward works very well in the current PromptAgent, and we'll make its definition clearer in the main text of the next version.

(b) Clarifying human prompt baseline. We thank the reviewer for digging into the details of the human prompt baseline. We surely will provide more details of this baseline in the new draft. Sepficially, in our paper, human prompts refer to human-written zero-shot prompts. For BigBench datasets[1], we follow previous works [2, 4] and directly utilize the task "description" provided in each dataset (One example can be found in penguins_in_a_table). These prompts are crafted and used by the dataset creators to instructively describe the dataset and prompt LLMs in their original paper [1]. These descriptions are also employed as the zero-shot baseline in the CoT evaluation paper by Suzgun et al. (2022) [2]. For the NCBI dataset, we reuse the prompt that was written by Gutierrez et al.[3], whose work explored the in-context learning ability of GPT-3 on the NCBI task. For other tasks, since there are no existing original prompts for us to reuse, we manually crafted the prompts based on the description, options, and purpose of the tasks. We craft the prompts to describe the task and options as normal users. We present all human prompts in Appendix F of the paper.

References:

[1] Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., ... & Wang, G. X. (2023). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.

[2] Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., ... & Wei, J. (2022). Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

[3] Gutiérrez, B. J., McNeal, N., Washington, C., Chen, Y., Li, L., Sun, H., & Su, Y. (2022, December). Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again. In Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 4497-4512).

[4] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022, September). Large Language Models are Human-Level Prompt Engineers. In The Eleventh International Conference on Learning Representations.

We thank the reviewer gPc5 for acknowledging that our proposed method is theoretically sound to enhance navigation through the expansive search space of potential prompts, and our experimental results are promising across 12 tasks with better performance and exploration efficiency than other prompt optimization methods; moreover, recognizing the importance that the optimized prompt can be generalized to different LLMs. We especially appreciate the reviewer's deep insights into the domain knowledge of optimizer LLM and the potential issue of increasing the length of meta-prompts. We now thoroughly address all the concerns and questions as follows.

(a) Domain knowledge in the optimizer LLM. We appreciate that the reviewer recognizes the importance of domain knowledge in our framework. We understand the reviewer's hypothesis regarding the necessity of domain knowledge in the optimizer LLM for providing insightful feedback. However, there are a few nuances we need to clarify, after which we will see the lack of domain knowledge for the optimizer LLM is not the problem this paper tries to solve, and actually, our PromptAgent framework can alleviate this burden of optimizer LLM to some degree.

1. First, we wish to clarify that PromptAgent with its current configuration (e.g., gpt-4 as the optimizer) is not posited as a universal solution for all domains, especially those where GPT-4's exposure is limited. Our primary focus has been on demonstrating the success and efficacy of PromptAgent across a broad range of tasks in BIG-Bench, specialized, and general NLP domains, where GPT-4's competency is already established [1].

2. Second, finding the suitable optimizer for a highly specialized domain (like medicine) is indeed an intriguing research area. This includes exploring methods to quantify the required domain knowledge for the optimizer and potentially training or fine-tuning domain-specific LLMs using specialized datasets, such as research papers or experiment data. Moreover, how to fine-tune an LLM under regularization like HIPAA is related to privacy-preserving research. However, such an exploration is beyond the scope of this paper and presents a promising avenue for future research. Importantly, the PromptAgent framework is designed to plug-and-play different optimizer models, allowing users to select/switch the most suitable one for their specific domain.

3. Third, the effectiveness of PromptAgent in deriving domain insights is not solely dependent on the optimizer LLM. It's a synergistic outcome of both the optimizer and our strategic planning framework. Specifically, we directly feed errors of the base model to the optimizer, saving the burden of the optimizer to detect these errors by itself. Moreover, the strategic planning framework will automatically select the high-reward path collecting meaningful error feedbacks, which alleviates the burden of the optimizer to produce "100%" accurate domain insights all the time (i.e., less accurate or wrong domain knowledge will be abandoned durig the search. A similar theme can be found in the following response (c) to Q1). To illustrate this, we conducted new experiments replacing GPT-4 with GPT-3.5, a weaker LLM with much less domain knowledge. We'd expect a performance drop with the weaker optimizer. However, the following table shows that the drop is only marginal, indicating that even with a weaker optimizer, PromptAgent effectively improves performance across various tasks, including medical ones.

RTable 1: Ablation study: Using both GPT-3.5 as base and optimizer LLM (second row)

RTable 1 shows the results on two biomedical tasks and one general NLU task of ablating the optimizer LLM with GPT-3.5 and GPT-4. Note that the default base model is GPT-3.5. The first row is directly prompting GPT-3.5 with the initial prompt. The second and third rows use GPT-3.5 and GPT-4 for the optimizer LLM, respectively. As we can see, when replacing GPT-4 with GPT-3.5, containing much less domain knowledge, we still obtain substantial gain over the initial prompt baseline. While the performance is lower than GPT-4 as expected, the drop is tolerable and marginal, especially in the domain-specific task, NCBI, with only about $3$ point drop. Such results indicate the robustness of the overall PromptAgent framework to strategically search for high-quality domain insights, instead of solely relying on the knowledge of optimizer LLM.

RTable 2: Ablation study on reducing "state_transit" length

RTable 2 shows the performance of two remedies for reducing the meta-prompt length on two biomedical tasks, which usually come with longer input questions. As we can see from the figure, both remedies can effectively reduce the meta-prompt lengths, with a marginal performance drop. Moreover, reducing the batch_size seems to be more effective in reducing the length while maintaining the performance than deleting "error_string" to shorten the meta-prompt. Interestingly, for the med_qa task with batch_size=1, we observe a slightly better performance than our default batch_size=5. This implies that for lengthy tasks, a shorter meta-prompt containing one single long example could potentially lead to a better performance. Nevertheless, we show that there are effective strategies to reduce the meta-prompt lengths while maintaining the performance.

(c) Q1: Influence of low-quality error feedback. It is possible for the optimizer to produce less accurate error feedback in one optimization step, which results in less effective child nodes. However, this is exactly why we need strategic planning to help explore the complex expert-prompt space. The ability of strategic planning allows PromptAgent to simulate future rewards and backtrack from low-quality error feedbacks. More specifically, the quality of the error feedback will be measured by the reward, and PromptAgent will avoid the low-quality error feedback with a low reward to explore in future iterations. Therefore, the low-quality error feedback will have less impact on the overall performance.

(d) Q2: Clarifying why not use GPT-4 for both base and optimizer LLM. Using both GPT-4 for both base and optimizer LLM is expected to reach higher performance. Our generalization experiments in Table 3 and Figure 1 (b) indirectly demonstrate this. Specifically, the prompts used for generalization experiments use GPT-3.5 as the base model and GPT-4 as the optimizer. When transferred to GPT-4 as the base model, these prompts lead to higher performance. Therefore, we anticipate a similar performance increase if we directly use GPT-4 both for base and optimizer LLMs. However, the main reason that prevents us from using GPT-4 for both base and optimizer LLM is the expensive cost of GPT-4, which is 40 times more expensive than GPT-3.5, when we were conducting experiments. Running our PromptAgent in a small dataset, like penguins in the table, could cost $20 for the base model portion, and will cost $800 if using GPT-4 as the base model, which is unaffordable for us. Nonetheless, our generalization experiments show promise by transferring GPT-3.5-based prompts to GPT-4, saving the high cost of directly using GPT-4 as the base model.

(e) Q3: New experiments in using PaLM 2 as the base model. As stated in the above response, the choice of the base model is very flexible in the PromptAgent framework, which could be GPT-3.5, GPT-4, PaLM, or other LLMs. However, using a weaker LLM as the base model, such as PaLM 2 (chat-bison-001), could lead to a performance drop, partially due to the weaker understanding of expert prompts. This can be indirectly observed from the generalization experiment (Table 3 and Figure 1 (b)), where we transfer GPT-3.5-based prompts to PaLM 2 with a performance drop. We add new experiments in the following table to directly observe the effect of using PaLM 2 as the base model. As we mentioned in the paper, PaLM 2 is weaker than GPT-3.5, but we can see PromptAgent can still improve its performance greatly.

RTable 3: Ablation study: using PaLM 2 as the base model

RTable 3 shows the results in three tasks using different base models, PaLM 2 and GPT-3.5, both against GPT-4 as the optimizer. The first row shows the worse initial performance using PaLM 2, indicating its weaker understanding of human prompts. After using PromptAgent to optimize PaLM 2 in the second row, we observe significant improvement over the first row, indicating the effect of PromptAgent to optimize weaker LLMs.

In light of these clarifications, we hope to establish a mutual understanding with the reviewer that the limitation of the optimizer's domain knowledge, especially in highly specialized areas, is better not to be viewed as a weakness of this work. Instead, it represents a promising direction for future research when applying PromptAgent to solve more challenging domain problems. Whenever a prominent optimizer LLM is available, PromptAgent can make it plug-and-play, and leverage the strategic planning to help select and filter out/distill useful knowledge from it.

(b) Lengths of meta-prompts. We thank the reviewer for carefully checking the details of our meta-prompts and spotting the increase in lengths that potentially impact the performance, especially for medical document tasks. To address this concern, we will show (i) our current lengths (even with the lengthy medical QA task) are within the manageable context length of modern LLMs, and more importantly, the growth of their lengths is also within a manageable speed, (ii) there are several tricks to reduce the meta-prompt length without hurting the performance too much. We make a few clarifications and further analyses as follows:

1. First, the context length problem of LLMs has received an extensive amount of attention recently, from 512, 1024, and 2048 to 8k, 16k, 32k, and even 100k (just put a reference link here). While some recent research studied the lost-in-the-middle issue, the capacity of long context understanding has been greatly improved. Then, similar to the above response about the choice of the optimizer LLM, whenever another LLM with a longer context is more suitable for our downstream tasks, such as medical document tasks, PromptAgent can combine with it in a plug-and-play fashion. In other words, with the increase in modern LLM's context length, we anticipate the increase in prompt lengths won't be a fatal issue for PromptAgent.

2. Second, we carefully analyze the length statistics in PromptAgent. Specifically, we focus on the med_qa task, which contains relatively long documents compared with other tasks (question length ranges from 400 to 1200), similar to the medical document tasks suggested by the reviewer. This anonymous link shows a figure for the growth of state_transit's average lengths, w.r.t the depth. With the maximum length of 4k for this length task, we can assure that it can be well handled by GPT-4 with 8k context length.

3. Moreover, from the figure, we observe the relatively linear growth pattern, suggesting the growth of "state_transit" won't lead to prohibitive long prompt, even with the increase of depth. Note that the number of training examples fed to each state is bounded by the batch size, which is fixed during the search. To confirm this, we further investigate the contributing factors for the length increase of "state_transit". As clearly illustrated in Table 7 from the paper, there are two parts of "state_transit": error information consisting of "error_string" and "error_summarization", and history information with "trajectory_prompts". "error_string" includes the current task prompt, question, and model's output. As the depth goes deeper, e.g., increase 1, the task prompt could grow; also, one more task prompt is added to the "trajectory_prompts". Therefore, the main contributing factor to the length increase of "state_transit" is the growth of the task prompt, which grows slowly and even steadily during the search process.

4. Finally, we did new experiments studying two ways to reduce the length of "state_transit." (i) Use a smaller batch size: batch_size=1 and (ii) Remove error_string and use fewer prompts in "trajectory_prompts". This new length analysis figure with batch_size=1 shows we can reduce the med_qa's maximum length from 4000 to 2400, and another figure shows removing error_string with less prompts in trajectory_prompts reduces the max prompt length from 4000 to 1600. Moreover, we show their performance in the following table, RTable 2.

• Alternative strategies for domain knowledge enhancement: When the domain knowledge of GPT-4 is proven to be inadequate or the cost/time/privacy of GPT-4 is less acceptable, we then need to resort to open-sourced models (fine-tuning GPT models via OpenAI APIs is possible but seems to be less preferred generally due to their high cost) for either pre-training from scratch or fine-tining an existing prominent one like Llama 2 on domain-specific data including research paper, experiment records, etc. If that's the case, the "plug-and-play" solution might lead to potential problems, such as generating incoherent sentences or very low-quality error feedbacks, etc. We then need to design novel solutions on top of PromptAgent to overcome the knowledge limitation of the optimizer. We list several potential directions as follows:
  • Using retrieval techniques to search for reliable and faithful domain knowledge directly to improve the error feedback
  • Employing an additional layer of quality assessment for error feedbacks before incorporating them into the optimization step to ensure their coherence and relevance
  • Proposing a hybrid of optimizers, combining the general capabilities of LLMs with domain-specific LLMs to balance the broad knowledge of general LLMs with targeted expertise
  • Collaborating with humans: adaptively ask domain experts to provide initial guidance to shape the direction of the optimization; while this could be expansive, it can towards more relevant and accurate domain insights
  • Data augmentation for domain expertise: for domains where privacy and data sensitivity are paramount (like healthcare), fine-tuning LLMs using synthetic or anonymized data can be a path forward

In conclusion, we recognize that the application of PromptAgent to highly specialized domains presents unique challenges. However, we believe that the strategies outlined above, combined with the inherent adaptability of the PromptAgent framework, provide a strong foundation for future research and practical applications in this area. We are committed to further exploring these solutions and sharing our findings with the community.

As suggested by reviewer gPc5, we have incorporated our great discussion of the limitation in the updated manuscript (Appendix A). We hope this additional discussion addresses the concerns raised and illustrates our dedication to the continuous improvement and application of PromptAgent in a variety of challenging domains.

Thank you again for your insightful feedback and for pushing us to think critically about the potential and limitations of our work.

References:

[1] Yin, Z., Sun, Q., Guo, Q., Wu, J., Qiu, X., & Huang, X. (2023). Do Large Language Models Know What They Don't Know?. arXiv preprint arXiv:2305.18153.

[2] Yao, Z., Jaafar, A., Wang, B., Zhu, Y., Yang, Z., & Yu, H. (2023). Do Physicians Know How to Prompt? The Need for Automatic Prompt Optimization Help in Clinical Note Generation. arXiv preprint arXiv:2311.09684.

[3] Hernandez, E., Mahajan, D., Wulff, J., Smith, M. J., Ziegler, Z., Nadler, D., ... & Alsentzer, E. (2023, June). Do We Still Need Clinical Language Models?. In Conference on Health, Inference, and Learning (pp. 578-597). PMLR.

[4] AI4Science, M. R., & Quantum, M. A. (2023). The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4. arXiv preprint arXiv:2311.07361.

[5] Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023). Large language models in medicine. Nature medicine, 29(8), 1930-1940.

Given all the clarifications, we hope to address a few potential comments from different opinions on our contributions as follows:

• "use of MCTS in prompt engineering is not novel": Interpreting our PromptAgent as "applying MCTS in prompt engineering" probably isn't the best way to understand its critical contribution/position in the literature.
  • First, as illustrated above, the reconceptualization of prompt optimization comes first, then we have MCTS to operationalize this new formulation. Focusing solely on the technique used risks overshadowing the broader, more impactful aspect of our work, which is the novel framing of the problem itself first. This new perspective opens the door to a variety of strategic planning approaches for prompt optimization, with MCTS being just one of the potential solutions we have explored in our current research.
  • Second, while MCTS has indeed been employed in prompting research, its application in prompt optimization is unprecedented, as detailed in our related work section. Moreover, PromptAgent's use of MCTS goes beyond a straightforward implementation. It includes significant innovations, including novel formulation of state/action/reward/meta-prompts to tackle specific challenges for prompt optimization, adapting the algorithm to the nuances of expert-level prompts, and balancing exploration and exploitation in a novel and complex domain, etc.
• "recursive generation" and "self-reflection" are existing techniques, "not technically novel": It is true that some components in PromptAgent are not brand new concepts per se; the innovation of our work lies in the holistic integration of these elements into a larger, cohesive framework. This isn't simply a case of assembling existing parts; it's about orchestrating them in a way that specifically serves the greater goal of expert-level prompt optimization. PromptAgent redefines these components, enabling them to interact dynamically within a strategic planning paradigm—a concept previously unexplored in this field.

To sum up, after all the above clarifications, we believe the concerns about the technical novelty of our work have been effectively resolved. We hope our responses highlight the significant contributions and potential of expert-level prompting. PromptAgent not only offers a novel approach to prompt optimization but also paves the way for a deeper understanding and utilization of large language models in complex tasks.

Last but not least, we will add all of these additional experiments and suggestions to the updated PDF. Below, we summarize the main changes during the rebuttal and request the reviewers to take a look at the new additions. Please also be careful to check some new figures we provided through anonymous links embedded in the text (cannot directly embed them into openreview responses).

• As suggested by gPc5: new experiment results on ablating the optimizer from GPT-4 to GPT-3.5, showing PromptAgent can help alleviate the high demand for the optimizer's domain knowledge
• As suggested by gPc5: new experiment results on studying effective strategies in reducing the lengths of meta-prompts
• As suggested by gPc5: new experiment results on ablating the base model from GPT-3.5 to PaLM 2
• As suggested by gPc5: clarifying how PromptAgent can avoid low-quality error feedback
• As suggested by gPc5: adding discussions about limitations and promising future works to deploy PromptAgent in highly specialized domains
• As suggested by WJJm: clarifying details of the reward function and human prompt baseline
• As suggested by zmiy: clarifying the legitimacy of human prompt baseline
• As suggested by zmiy: new experiment results on greedy search baseline with a bigger exploration space
• As suggested by zmiy: new experiment results on a DFS-based search variant baseline
• As suggested by j86n: clarifying the necessity and contribution of MCTS (through our core ablation study on search variants); we implement a new baseline as error feedback-based APE
• As suggested by j86n: new experience results on hyperparameter selection