Large Language Models as Optimizers

Thank you for your careful reading and thoughtful reviews. Let us address your comments below.

What is the key/unique advantage of using LLMs to optimize over some traditional optimization algorithms, especially on classical optimization problems (not prompt engineering)?

The key/unique advantage is that one can use natural language to describe the optimization problem, as discussed in the second paragraph of Section 1: “​​instead of formally defining the optimization problem and deriving the update step with a programmed solver”. This can make optimization more accessible to general users who do not have much domain knowledge of the types of optimization tasks in question, and may also boost productivity of optimization experts who work on these tasks on a daily basis.

Meanwhile, we would like to clarify that the focus of our work is not to solve all classic optimization problems. We evaluated our approach on linear regression and Traveling Salesman Problems only as motivating examples to demonstrate the potential of LLMs for optimization. Instead, our main task is prompt optimization which is a known challenge for classic optimization methods, where the optimization space is in natural language.

For prompt optimization, the optimization process of this work (i.e., directly feeding solution-score pairs, optimization task descriptions, and meta-instructions into the optimizer LLM) is a black box/lacks interpretation, why it is a better prompt optimizer than those new methods that leverage LLMs to explicitly act as mutation and crossover operators, and further optimize the prompt?...

Thanks for pointing out these related works. We have discussed such evolutionary methods in Section 6. Following your suggestion, we also added an empirical comparison with EvoPrompt. Please refer to our general response for a summary, and we summarized the full results in Appendix M of the updated Supplementary Material.

To further understand the effect of the purple text in this work, an ablation study may be beneficial for improving the solidness of the results.

We have benchmarked the effect of exemplars (i.e., the purple text) in ablation studies. Please refer to Figure A17 in Appendix K of the Supplementary Material. We show that adding exemplars boosts performance, since it offers more context to the optimizer LLM and enables it to generate more relevant instructions (prompts). For example, on BBH tasks, the optimizer LLM found instructions that are tailored to the tasks, as shown in Table A4 and A5 in Supplementary Material. Without exemplars in the meta-prompt, it is impossible for the optimizer LLM to understand the task if the initial prompts are simple and generic.

Thank you for your careful reading and thoughtful reviews. Let us address your comments below.

Numbers of examplers. Did you take the randomness of example picking into consideration? For each run of every setting, do you give the same set of examples?

At each step, we randomly sample a different set of 3 exemplars to put into the meta-prompt.

• How we chose 3 for the number of exemplars: Our ablation (Figure A17(c) and A17(d) in Appendix K of the Supplementary Material) shows that 3 achieves a good balance between the informativeness of the meta-prompt and the context length constraint.
• How randomness is useful: We randomly sample exemplars at each step because it reduces the risk of overfitting on a fixed set of exemplars, and reduces the chance of the optimization getting stuck. The latter is because if both the set of exemplars and the top instructions in the meta-prompt remain the same (this more often occurs in later steps when the performance starts to saturate), the meta-prompt will remain the same, and the optimizer LLM may keep generating the same instructions.

I noticed that for different tasks, the “batch size” that works the best can be different (Figure 5, cd). Do you find any obvious patterns on which types of data/tasks prefer a smaller “batch size” and vice versa?

Insightful question. Although setting the batch size of 8 generally achieves the best final performance across models and tasks, some other batch sizes may work equally well in certain cases (like batch size 1 and 2 at the early stage of optimization in Figure 5(d)). In general, the batch size used by our approach can be smaller for easier optimization problems, such as:

• When the scorer LLM is instruction-tuned, the accuracy landscape will be less bumpy because the model performance is less sensitive to the prompt, thus a smaller batch size would suffice for the accuracy curve to trend up. The bumpiness can be well reflected by the oscillation in the optimization curve: both on the same GSM8K 3.5% training set, the curve on the pre-trained PaLM 2-L scorer (Figure 1(a)) is much more bumpy than the blue dots in Figure 5(a), especially at early steps. This means OPRO on a pre-trained scorer LLM needs a larger batch size (like 8) to succeed.
• When the task is easier, one doesn’t need to carefully prompt the model to get a good score, meaning the loss landscape is also less bumpy. One example is the difference between Figure 5(c) and 5(d) in the main paper: BBH sports_understanding is easier than GSM8K in that the same text-bison scorer LLM achieves around 30% higher accuracy, close to 100% on sports_understanding, thus a smaller batch size is enough.

(following Part 2/3)

How does OPRO compare to other gradient-free prompt optimization methods? Could be included in experiments.

We added an empirical comparison to EvoPrompt [1], an evolutionary method pointed out by Reviewer iHbx. Please refer to our general response for a summary, and we summarized the full results in Appendix M of the updated Supplementary Material.

Is there any overfitting during prompt optimization? How does test accuracy compare to training accuracy?

Great question, we’ve studied overfitting at an early stage of this work, and the results from there influenced our final approach. We added Appendix L in the Supplementary Material with training and validation accuracy curves. To summarize, we observe that the trends of training and validation accuracies are consistent with each other throughout the optimization.

References:

[1] Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, Yujiu Yang. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers. arXiv:2309.08532.

Thank you for your careful reading and thoughtful reviews. Let us address your comments below.

The biggest limitation is that OPRO's performance looks highly fluctuating. It's unclear if the LLM really finds the so-called optimization "trajectory" or just randomly finds a good prompt. The authors should provide more analysis to show that the LLM is indeed learning to optimize.

We acknowledge that the optimization curves do not show monotonic increase at every step. Most optimizers don’t, especially when the loss landscape has a more complicated curvature. From our new experiment showing that the one-step prompt generation method you suggested performs much worse than our approach, we observe that LLM gradually improves upon previous prompts based on the optimization trajectory, just like what other optimizers do.

In addition, in figures showing the accuracy at each step (Figure 1 in main paper; Figures A11, A12, A13 in Supplementary Material), we presented the average (instead of the maximum) accuracy of 8 instructions generated at each step, which showed generally upward trends. This shows the optimizer LLM gradually generates distributionally better instructions. Furthermore, Figure 5(c) and 5(d) showed that with the same number of total evaluated instructions, generating 8 instructions per step (default setting) achieved better performance than generating 16 instructions per step, demonstrating that the LLM learns to optimize prompts from previous attempts better with more iterations of update, instead of just randomly finding a good prompt by sampling more instructions.

Limited exploration on how to provide richer feedback to LLM beyond aggregated scores. It could help address limitations...For prompt optimization, have you experimented with providing more detailed feedback to the LLM beyond aggregated scores? (e.g. accuracy on different example types, common mistakes)

As noted in the discussion of future work in Section 7 (Conclusion), “we tried including error cases in the meta-prompt rather than randomly sampling from the training set at each optimization step, but the results are similar, indicating that the error cases alone are not informative enough for the optimizer LLM to grasp the cause of the wrong prediction”. As also noted in Section 7, we agree that improving our approach to better incorporate more detailed feedback is a promising direction for future work.

That said, we show that our simple meta-prompt still demonstrates significant improvement across different tasks and LLMs, and we see simplicity as a key advantage of our method.

Unclear how sensitive results are to meta-prompt design and hyperparameters like temperature.

Please see our responses to your respective questions in Part 2/3.

Limited analysis. For example, there is no characterization of what makes an effective prompt.

We have studied both aspects in the ablation studies. Please see Section 5.3 in the main paper and Appendix K in the Supplementary Material for ablations on how different parts of the meta-prompt and temperature matter.

No comparison to other prompt optimization methods. It could better situate contributions.

Thanks for your suggestion. We added an empirical comparison to EvoPrompt, which is an evolutionary method pointed out by Reviewer iHbx. Please refer to our general response for a summary, and we summarized the full results in Appendix M of the updated Supplementary Material.

Can you clearly state how many optimization steps are performed in each experiment? How does the number of steps affect performance?

We run a maximum of 200 steps in every experiment although many converge much sooner. The training accuracies trend upwards until plateaued, so the performance generally improves with more number of steps. As shown in Figure 1 in the main paper and Figures A12, A13 in the Supplementary Material, the performance often plateaus within 100 steps.

(to be continued in Part 2/3)

Thank you for your careful reading and thoughtful reviews. Let us address your comments below.

First, I disagree with the authors fundamentally about what optimization means. To me, this work is not optimization but step-by-step inference...

First, we would like to clarify that the focus of our work is not to solve all kinds of classic optimization problems. We evaluated our approach on linear regression and traveling salesman problems only as motivating examples to demonstrate the potential of LLMs for optimization. Instead, our main task is prompt optimization which is a known challenge for classic optimization methods, where the optimization space is in natural language.

Meanwhile, optimization is a very general concept. As classic optimizers such as gradient descent and Adam, our approach proposes better solutions in an iterative manner, step-by-step. Specifically, instead of updating the current solution with a mathematical rule (like in momentum or Adam) or by inferencing a surrogate model (like in Bayesian optimization), we prompt the LLM to get the next-step solution, hence “LLMs as optimizers”. From this perspective, prompt optimization is also an optimization problem: the objective is to maximize the task accuracy within the huge natural language space.

Second, considering the relevance of this approach to step-by-step inference, what is new here compared to previous step-by-step inference procedures?...What is the difference between this work and other step-by-step inference techniques such as https://arxiv.org/pdf/2205.11916, https://arxiv.org/abs/2201.11903, https://arxiv.org/abs/2305.10601 How do the results compare?

These works are complementary to our work. All 3 works used manually written prompts to improve LLMs’ performance, while our method focuses on how to automatically find such prompts via optimization and in many cases we can find even better prompts compared to the manually written ones in these existing works.

More specifically, https://arxiv.org/pdf/2205.11916 shows LLMs can do zero-shot chain-of-thought prompting; https://arxiv.org/abs/2201.11903 shows chain-of-thought prompting improves LLMs’ reasoning performance; https://arxiv.org/abs/2305.10601 shows tree-of-thought improves the performance on several tasks that require planning and search. These papers are about manually improving the LLM performance with manually written prompts, while our approach is on prompt optimization/tuning. For example, our work showed how our method greatly improved zero-shot chain-of-thought prompting.

Thanks again for your review and your follow-up discussion! Please let us know if we have adequately addressed your questions and concerns.