InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models

The paper presents an innovative approach to identifying the "optimal" instruction for large language models with the aim of improving generative quality. This work falls under the burgeoning area of prompt search methods, which have gained significant attention recently, exemplified by methods such as APE, RLPrompt etc. Unlike conventional methods that optimize discrete instructions, the authors propose optimizing a low-dimensional "soft prompt" using dimensionality reduction. The optimized soft prompt is applied to an open-source Lifelong Learning Model (LLM) to generate instructions for a black-box LLM. The optimization process is iterative, involving zero-shot evaluation of the black-box LLM's performance, which is then used in a Bayesian optimization scheme to refine the soft prompts. This iterative process continues until convergence. Experimental results on the BBH benchmark show that the proposed method yields superior performance across all 32 tasks.

This paper proposes a method to optimize the instructions for black-box large language models. The proposed method uses an open-source LLM to convert a soft prompt to an instruction, and then uses the instruction as input to the black-box LLM. Bayesian optimization is then used to optimize the soft prompt, which can iteratively propose new soft prompts and instructions to be evaluated by the black-box LLM.

This paper proposes to use Bayesian optimization to learn an instruction with an open-source LLM so that the instruction improves the zero-shot results of a black-box LLM. Since instructions are discrete, this work instead iteratively learns a small soft prompt which then gets decoded as an instruction. Each updated instruction is evaluated on the black-box LLM whose training accuracy is used to find a better soft prompt.

We appreciate all reviewers for providing useful feedback! We hope our responses address your concerns. Please feel free to raise further concerns and we are open to discuss.

Reviewers #FwK3 and #4arY both point out that the performances of APE might be underestimated in our experiments. We would like to clarify that the original APE can access 100 demonstrations~(input-output pairs) for instruction optimization. In every iteration of APE, they randomly draw 5 new demos from a hold-out set of 100 demos. Instead, our hold-out set contains only 5 demos and InstructZero does not have access to more new demos---this is more practical and also more challenging. In our experiments, for a fair comparison, both APE and InstructZero have access to the same hold-out set for instruction induction and use GPT-3.5-turbo as the API LLMs.

In addition to the reported 5-demo experiments, we expand InstructZero's hold-out set of demos to 10 demos and compare it with the original APE using a hold-out set of 100-demos. We keep the API LLM the same as the one used in our original manuscript, i.e., ChatGPT. The results are shown as follows:

As shown on three tasks, InstructZero finds better instructions than APE even using only 10% demos used by APE.

This paper proposes to optimize the instruction in the prompt using Bayesian optimization.

STRENGTHS

(1) The reviewers agree that the proposed method is interesting and new.

(2) The proposed method outperforms the tested baselines.

WEAKNESSES

The authors have adequately resolved the concern with underestimating the performances of APE in their rebuttal.

After reviewing the authors' rebuttal, a number of major concerns remain, as discussed below. The review provided by Reviewer 4arY (who has responded in the AC-reviewer discussion phase) has provided strong evidence of these concerns, while the review given by reviewer 6Tou seems to have lacked detailed arguments in favor. Recommendations are provided for some of them to improve this paper.

(1) The authors' rebuttal did not adequately address the concern on $\mathbf{k}$ in Eq. 4-5. Computing $\mathbf{k}$ directly using eq. 9 does not seem possible in practice. However, if we take the authors' response at face value, does it mean that they only use $l$ to compute $\mathbf{k}$ but not $\mathbf{S}$ in equation 9? This point seems to be re-iterated in the last sentence of the 2nd paragraph in A1. How then would the kernel remain valid when $\mathbf{K}$ is defined using eq. 9 but $\mathbf{k}$ is defined differently? This technical ambiguity needs to be addressed by the authors. Hence, the ablation study needs to be conducted on all tasks instead of a small number of them.

(2) Though the authors have rebutted with the reason of fairness in comparison with APE, they can use the input-output exemplars to assess how much the performance of InstructZero can be improved, since they are already assumed to be available in the problem setting. Furthermore, as mentioned by a reviewer, APE has also performed few-shot demonstrations.

(3) The authors have used the same reasoning template as that in [1]. Hence, we assume that the authors are aware of similar results reported in [1] (Table 3). The authors have not discussed why the results are similar or different and whether they have taken some of the results from [1] or run the corresponding experiments. For example, with the score of 0.795 for SVAMP, the authors obtain the instruction "Let's use the brain", but it shows "Let's use the equation" in [1] for the authors' proposed method.

(4) Though the authors have described the qualitative differences with RLPrompt in the related work section, it is not obvious that RLPrompt would necessarily perform worse than their proposed method empirically.

(5) The authors say that "In experiments, we apply a mini-batch version of INSTRUCTZERO that explores 25 soft prompts in every iteration. The only major change required is to select the top-25 soft prompts with the largest u(p)". This does not seem like an effective way of finding a good batch of soft prompts since they can be clustered near to each other in the soft prompt space due to their large u(p). The authors are strongly encouraged to look into the literature of batch Bayesian optimization algorithms to understand how to construct an effective batch.

The authors are strongly encouraged to address the above concerns and consider the reviewers' feedback when revising their paper.

Reject