Prompt Optimization via Adversarial In-Context Learning

We thank the reviewer for your time!

W1. The novelty is limited.

To the best of our knowledge, this is the first work exploring optimizing discrete prompts for in-context learning (ICL) in the adversarial learning style. We propose adv-ICL which can effectively optimize the prompts. Despite simplicity, our method shows strong improvements over the baselines and achieves significant improvements over previous approaches while requiring very few training data.

That being said, novelty itself is a subjective matter. The whole deep learning field is built on empirical results with old ideas. The whole GPT series had very limited “novelty” in the traditional sense, except the prompting formulation. They just made it work. Our method also works. Papers like ours are the reason for why we have great AI technologies today.

W2: Add examples in the presentation.

We have updated an example in Figure 1 of the updated version. Please check it.

W4. The limitations are missing.

The limitation section is not required for ICLR. Please check the author guide (https://iclr.cc/Conferences/2024/AuthorGuide).

Q1. Why the discriminator is necessary?

We have conducted extra experiments supporting the reviewer’s comments.

• Froze discriminator: We have conducted our experiments with Vicuna and ChatGPT where we frozen the discriminators on four representative datasets: WebNLG, Machine translation (RO->EN), YELP, and GSM8K. The results with Vicuna and ChatGPT are shown below:

It is clear that updating the discriminator is essential in our formulation.

Q2. Why too many iterations T or samples m harm the performance

This is an interesting point! We observed this phenomenon in the experiments and were also curious about it. Our current hypothesis is that, while in-context learning holds great promise, it does not behave similar to SGD based learning. For example, having more demonstrations or data samples in the prompt does not help. Also, as demonstrated by [1], a critical threshold exists for the number of training examples, and surpassing this threshold leads to a decline in performance. As a result, more iterations or more samples do not necessarily help.

Given the complexity of in-context learning, we simply determine the best hyperparameters based on empirical results. We perform hyperparameter search based on the validation data of a small set of tasks. These hyperparameters do generalise to the other tasks. For details about the hyperparameter search process, you can refer to Section 2.4 in the original paper and Section 2.5 in the revised version.

Q3. Baseline in Figure 9 and Figure 10

The “Baseline” in Figures 9 and 10 are the Few-shot baseline.

Misc: Discuss with concurrent work LLM Optimizers.

We have included it in our related work discussion. The paper presents a novel LLMs optimizer that leverages feedback from LLMs to refine prompts effectively. This paper lies in the field of self-refinement and RL optimization m ethods, as explored in previous works [3] and [4]. However, our distinctive contribution lies in the study of adversarial in-context learning. We have included a discussion with the LLM optimizer in our related work.

References

[1] Min et al. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? ACl 2022.

[2] Yang et al, Large language models as optimizers. arXiv preprint, Arxiv 2023.

[3] Mingkai et al, Rlprompt: Optimizing discrete text prompts with reinforcement learning, EMNLP 2022.

[4] Pan et al, Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, ICLR 2023.

We thank the reviewer for your time. We respectfully think that the reviewer did not understand our method. However, we will try to address your concerns as best as we can.

W1. The convergence of the proposed approach.

In the updated draft, we included a new theoretical analysis section that shows the proposed algorithm converges to the desired equilibrium under mild assumptions.

Here is the sketch of the analysis.

Theoretical Results

In this section, we theoretically analyze whether such a minimax objective in the form of in-context learning can achieve the desired equilibrium as in the original GAN scenario. We assume access to models with infinite capacities powering the discriminator $D$, generator $G$, and prompt modifier $M$ and that in each iteration, we sample a sufficient number of prompts from $M$ to update both $G$ and $D$. Let $p_{data}$ be the distribution of the training data, and $p_g$ be of the generated data from $G$.

Considering a language model $\mathcal{M}$ which can be $D$ or $G$ performing a corresponding task $T_1$ and evaluated by a metric $E_1$, we further assume that:

1. $M$ is powerful enough to modify the initial prompt of $\mathcal{M}$ for $T_1$, covering all possible prompt variants performing the task $T_1$.
2. $\mathcal{M}$ is a powerful enough language model that there exists a prompt $\mathcal{P}$ of $T$ for $\mathcal{M}$ that given $\mathcal{P}, \mathcal{M}$ can achieve the globally optimal result on $E_1$ for the task $T$.
3. There exists a prompt sampled by $M$ that maximizes $\mathcal{M}$ on $E_1$ globally on $T$.

The assumption 3 is a result of the assumptions 1, and 2, and the assumption about our access to infinite capacities language models. Indeed, given $\mathcal{M}$, from assumption 2, there exists a globally optimized prompt $\mathcal{P}$ of $T_1$ for it such that it can achieve the globally optimal state on $E_1$ for the task $T_1$. Furthermore, since $M$ is powerful enough in modifying the initial prompt (ass. 1), plus $M$ samples a sufficiently large number of prompts for each iteration (ass. 2), $M$ can generate $\mathcal{P}$ with a non-zero probability, which conclude the assumption 3.

From the assumptions, we have the following propositions.

Proposition 1 (Goodfellow et al., 2014). For G fixed, the optimal discriminator D can be described in a closed form, denoted as D*.

Proposition 2. For each training iteration, for a fixed $G$, the optimal discriminator $D^*$ can be achieved.

Proposition 3 (Motivated by Goodfellow et al., 2014). If $G$ and $D$ have enough capacity, and at each training step, the discriminator is allowed to reach its optimum $D^*$ given $G$, and $p_g$ is updated so as to improve the criterion

The proofs for the propositions are presented in Appendix A.1 of our revised manual script. Our conclusion is that with strong enough $D,G,M$, the framework adv-ICL converges.

W2a. It's worth noting that discarded prompts can also impact the model's training

We respectfully think that the reviewer misunderstood our method. First, it is worth noting that our method does not require any update in the models’ parameters, it only updates the prompts of the discriminator and generator. Second, we only forward the best-performing prompt from the previous training iteration to the next training iteration for modification, and it is reasonable for us to further optimize this best-performing prompt without considering the discarded ones.

W2b. Effect results may come from the repeated trial-and-error of prompt modifications

In addition to our arguments in W2a, we further conduct experiments to verify whether the prompt modifier module worked as expected. Specifically, we deactivate the discriminator and only employ a prompt modifier to repeatedly optimize the prompt. Our experimental results with Vicuna and ChatGPT are provided below:

In the majority of cases, when the discriminator is deactivated and only the prompt modifier is relied upon under Vicuna and ChatGPT, there is a noticeable decline in performance. This decline can be attributed to the inherent randomness of the prompt modifier, which lacks the ability to discern a more effective prompt. This observation underscores the substantial significance of the discriminator and adversarial loss in the optimization process.

The reviewer thanks the authors for their reply. I understand that the proposed approach does not alter the generator and discriminator. My major concerns were on the prompt selection and improvement parts in the prompt modifier, and how they can affect the overall convergence and training time of GANs. The reply did not address my concerns, so I will keep my original score.

We thank the reviewer for your time.

W1. Need to be compared with some more RL-based prompt optimization baselines.

These RL-based baselines could not be applied to the closed source models such as ChatGPT and text-davinci. To test the effectiveness of our method, we need to perform experiments on these models because they have better performance. This is why we made the decision to not compare with RL-based methods. Specifically, [1] and [2] require access to LLMs for inserting MLPs.

We compared adv-ICL with state-of-the-art prompt learning methods including GPS, APO and APE and outperformed them consistently, which indicates the effectiveness of our method.

Q1. What if we provide more feedback to the prompt modifier.

We conducted an experiment that involved integrating the most successful prompts from previous iterations as feedback for the next iteration. In this process, we utilized previous best-performing prompts, namely $P_1, P_2, ..., P_k$, as inputs to the prompt modifier module when generating the $(k+1)$-th prompt.

We applied the method to four representative tasks, including data-to-text generation task WebNLG, machine translation, sentiment classification Yelp, and reasoning GSM8k, using Vicuna and ChatGPT. The experimental results are shown below:

In the case of Vicuna, incorporating additional feedback into the prompt modifier proves effective for tasks such as translation and classification. However, this approach falls short when applied to data-to-text and reasoning tasks. On the other hand, for ChatGPT, augmenting the prompt modifier with more feedback does not yield improved performance.

Q2. What if you also add a module that can automatically verify modifications.

As shown in the “Human evaluation of prompt modifier performance” part of Section 3.3, most of the prompts outputted by the prompt modifier are semantically correct. Specifically, the correct percentage of text-davinci-002, ChatGPT, and Vicuna are 88%, 91%, and 83% respectively. So we do not think that it is necessary to automatically verify modifications.

References

[1] Mingkai et al, Rlprompt: Optimizing discrete text prompts with reinforcement learning, EMNLP 2022.

[2] Pan et al, Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, ICLR 2023.