Instance Needs More Care: Rewriting Prompts for Instances Yields Better Zero-Shot Performance

We appreciate the reviewer for acknowledging the value of our work and suggesting the long-term developmental perspective about LLMs’ robustness to prompts! We agree with the reviewer and found it an important direction to pursue.

Regarding the reviewer’s suggestion on improving LLMs’ robustness to prompts as future direction

We agree with the reviewer on the importance of improving LLMs’ robustness to prompts. In fact, this is also our motivation for this work. In our submission, we contribute to this aim by rewriting prompts that can be executed across a diverse set of tasks on different models. That is, instead of improving the robustness nature of an individual LLM, we approach this “robustness to prompts” by bridging the gap between human-written prompts (which could be suboptimal) and the LLM-preferable prompts through PROMPTD’s optimization. We did three experiments on our proposed method’s capability on the three points: robustness to unseen tasks, adaptability on weaker LLMs, and sustainability to adversarial prompting through the interpretability of PROMPTD.

We show that PROMPTD could rewrite test instances with specialized instructions and hints across unseen task types such as implicit reasoning, symbolic reasoning, and sentiment classification. For adaptability, we show the efficacy of our approach with evolving LLMs, including GPT-3.5-turbo, and LLAMA2 70B. Finally, we present a use-case of PROMPTD on its applicability to avoid adversarial prompting because of its interpretable approach to rewriting. PROMPTD not only aligns with the current trajectory of LLM development but also actively contributes to enhancing their robustness to a diverse set of prompts aligning with the future direction suggested by the reviewer.

Clarification on how to rewrite prompts and their insights

Thank you for seeking further clarification on PROMPTD's methodology and insights. In summary, PROMPTD improves the zero-shot performance of LLMs by tailoring prompts to each test instance based on criteria such as specificity and structure, as demonstrated through our extensive experiments. This approach not only enhances clarity and interpretability but also showcases PROMPTD's adaptability across diverse task types.

Rewriting Criteria: PROMPTD employs specificity, non-ambiguity, completeness, and structuredness as its rewriting criteria, ensuring each prompt is optimally customized for its corresponding test instance.

In-Context Learning for Rewriting: Using a few-shot in-context learning method, PROMPTD contrasts good and bad prompts to guide the LLM in generating better prompts.

Instance-Level Customization: Each test instance is individually considered, allowing PROMPTD to address unique requirements effectively, contrasting with generic, one-size-fits-all prompts.

We picked the ten rewriting demonstrations for PROMPTD based on the most frequently researched task types such as Mathematical Reasoning, Logical Reasoning, Code Generation, and Instruction Induction to probe LLMs. Since we aim to improve on broad task types such as general queries to chatGPT, we also included content generation prompts. The intuition behind including such diverse tasks in demonstrations was to cover the diverse shortcomings of different task types and generalize to unseen task types. Moreover, we hypothesize that different task types could learn from different errors and when reasoning over a test instance they can learn from both the similar types of tasks and general content generation tasks. Guiding through in-context learning examples and harnessing the text generation capabilities of LLMs enables us to not only identify the limitations but also add instance-specific hints and examples in the rewritten prompts. We also pick several examples from the Instruction Induction task that would encourage LLMs to learn from input-output pairs and generate examples. As shown in Table 5, for the sentiment classification task, PROMPTD was able to generate several examples to learn the task. Finally, for some task types such as Code Generation and Content Generation, we refrain from generating a structured output such as “The answer is …” enabling LLMs to learn when a structured output is required. Each of these prompt types addresses a unique set of challenges in prompt rewriting. Together, they provide a comprehensive training ground for the model to learn a wide array of skills necessary for generating high-quality, instance-specific prompts. These diverse examples enable the model to generalize these skills across tasks, leading to better overall performance and adaptability to a range of unseen task types.

We appreciate the reviewer’s acknowledgment of PROMPTD on its novelty to dynamically rewrite prompts and recognition of its interpretability to perform prompt optimization for zero-shot approaches.

W1: Computational overhead of PromptD

We refer the reviewer to our general response to this comment.

W2: Reliance on Prompt Quality and Robustness

We clarify that PROMPTD does not rely on the initial quality of the prompts it is based upon. We note that, as we focus on zero-shot prompting, the “base prompt” in our experiments is simply the task instruction from the original test benchmarks (see examples in Table 2), and we make no assumption on its quality.

Our approach PROMPTD does not assume optimal “good prompts” in prompt rewriting either. Instead, PROMPTD relies on its ability to reason over the shortcomings of a given “bad prompt” and overcome them in rewriting. This ensures robust performance even with suboptimal starting prompts, as demonstrated across various tasks and datasets, including unseen ones.

W3: A deeper analysis of where and why the approach fails

We refer the reviewer to the general discussion on in-depth analysis.

W4: Comparison with other zero-shot approaches

Thank you for your comment regarding the comparison with other zero-shot approaches. Indeed, the approaches for zero-shot learning in LLMs are rich and rapidly expanding. To address your point, we have identified several recent papers that present innovative zero-shot techniques. Wang et al., 2023 introduced a two-step approach that plans out subtasks and then executes the plans to derive the final answer. On the contrary, our work focuses on enriching prompts with hints and encouraging LLMs to generate rationales before deriving the final answer. Reynolds and McDonell2021 suggested a meta-prompt “Let's solve this problem by splitting it into steps.” to facilitate multi-step reasoning in simple arithmetic problems. However, they did not evaluate quantitatively on diverse reasoning tasks against baselines. In this paper, we evaluate PROMTPD on a diverse set of tasks beyond reasoning. Wei et al., 2021 proposed instruction-tuning which gives models an instruction describing a task while with PROMPTD, on the other hand, instead of tuning an instruction we aim to optimize task-specific instructions and test instances. Recent approaches, similar to PROMPTD, have also proposed to optimize task-level instructions (Hong et al., 2023) enriching task-specific hints. Through experiments, we show that such task-specific instruction optimization does not lead to stable performance across different tasks motivating the need for instance-level prompt optimization. Huang et al., 2022 show that reasoning performance improves by multi-task finetuning a model on several self-generated synthetic CoT datasets. We, on the other hand, do not perform finetuning and in addition to mathematical and logical reasoning, also focus on content generation and code generation tasks.

References:

Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, Ee-Peng Lim "Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models." In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.

Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, & Jiawei Han. (2023). Large Language Models Can Self-improve. International Conference on Learning Representations. 2023.

Sun, Hong, et al. "AutoHint: Automatic Prompt Optimization with Hint Generation." Workshop on Foundations and Applications in Large-scale AI Models -Pre-training, Fine-tuning, and Prompt-based Learning, colocated with the 29TH ACM SIGKDD CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING (KDD), August 6-10, 2023, Long Beach, CA, USA

Wei, Jason, et al. "Finetuned Language Models are Zero-Shot Learners." International Conference on Learning Representations. 2021.

Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA '21). Association for Computing Machinery, New York, NY, USA, Article 314, 1–7. https://doi.org/10.1145/3411763.3451760

We appreciate the reviewer’s recognition of the simplicity and applicability of PROMPTD on real-world applications and the recognition of experiments performed to study the efficacy of our approach.

Computational Cost

We refer the reviewer to our general response to this comment.

Transferability to Other LLMs such as LLAMA

Thank you for raising this good question! We investigated the effect of applying rewritten prompts from PROMPTD to three other task LLMs, including GPT-3.5-turbo, LLAMA2 70B, and LLAMA 2-13B. Due to the resource and time limitation during rebuttal, the experiments were conducted on a subset of 250 randomly sampled questions from GSM8k, and LLAMA2 70B was implemented using together AI (https://www.together.ai/).

Our results showed that the rewritten prompts of PROMPTD can enhance GPT-3.5-turbo’s performance consistently. However, there’s only a slight improvement for LLAMA2 70B and a reduced effect for LLAMA2 13B. We note a potential implementation issue with LLAMA2 70B due to the restriction from using together AI’s API, which we hope to figure out in the remaining rebuttal period or post-rebuttal. However, the current results have revealed interesting observations that the rewritten prompts of PROMPTD can only be effective when the task LLM itself has a strong enough capability of language understanding and instruction following. To overcome it, future work could improve PROMPTD to take into account the performance of the downstream task LLM or consider fine-tuning the task LLM for better instruction following or understanding.

In-depth analysis

We appreciate the reviewers xGQ6 and bonY for acknowledging the value of our work and suggesting the long-term developmental perspective about LLMs’ robustness to prompts! We agree with the reviewer and found it an important direction to pursue. We break down the analysis into the following parts:

A deeper analysis of where and why the approach fails. As detailed in Section 4 and Appendix B.4-B.5, we recognize limitations when using weaker LLMs like GPT-3.5. Further, we've categorized common errors based on a study of 20 random rewritten error instances per dataset (in total 160 error instances) and updated Table 7 in Appendix B.4. These errors primarily occur when LLMs incorrectly transcribe information from test instances (58%) or introduce extraneous constraints (28%), impacting the accuracy of generated prompts. For instance, PROMPTD sometimes adds unnecessary constraints or fabricated information, which, while enhancing code generation and debugging, can lead to misalignments with dataset instructions and incorrect evaluations.

Additionally, in about 13% of errors, LLMs correctly identify a bad prompt's shortcomings but fail to generate an appropriate improved prompt. A detailed account of these instances will be provided in the final version of the paper.

Based on these insights, we acknowledge a broader challenge: LLMs, such as those utilized in PROMPTD, can find domain-specific tasks particularly challenging. In areas where domain-specific knowledge is essential, our approach might not always grasp the required depth. For example, in financial analysis, understanding specific market terminologies and regulatory nuances is key. Recognizing this, future works can enhance PROMPTD, potentially through the integration of human-in-the-loop, overcoming current limitations and broadening the scope of PROMPTD's effectiveness in domain-specific tasks.

Broader Quantitative Analysis. PROMPTD has been evaluated across a diverse range of tasks, including Code Generation, Arithmetic and Logical Reasoning, and Implicit Reasoning, showcasing its versatility. In Section 3.7, we demonstrate PROMPTD's efficacy in handling Adversarial Prompting scenarios. Furthermore, in content generation tasks, PROMPTD effectively guides LLMs to produce persona-based responses, such as emulating a 19th-century poet's style (Section 1). We also extended our evaluation to include additional general downstream tasks like Symbolic Reasoning and OpenDomainQA, areas that were not initially covered due to constraints on time and resources. As detailed in Table 6 in Appendix B.3, while PROMPTD surpasses baseline performance on StrategyQA, it shows some limitations in the Symbolic Reasoning task.

This broad spectrum of evaluations highlights PROMPTD's adaptability and potential to enhance LLMs' performance across varied domains. By performing evaluations on a different set of LLMs such as GPT-4, GPT-3.5, and LLAMA, we also show PROMPTD's applicability in producing prompts that are applicable and effective across different language model architectures. As these models become more robust to diverse prompts, PROMPTD's role in optimizing and tailoring these prompts becomes increasingly significant. We anticipate that the continuous enhancements in large language models, especially in their ability to interpret and respond to complex prompts, will further amplify the effectiveness of PROMPTD as shown in the experiments with LLAMA-based experiments.