Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

We thank the reviewer for their thoughtful review! We are pleased that they pointed out our paper studies “a critical issue for LLMs”, with a rigorous problem formulation and substantiated with numerous experiments on few-shot classification tasks (e.g., Super-Natural Instructions) and short text generation tasks.

We address all comments below, and we believe that the two new, additional experiments (an experiment on performance variance on 17 new sentence-level open text generation tasks; and an analysis of 30 classification tasks quantifying the dispersion of the accuracy distribution using standard deviation on 500 formats evaluated) resolve all the reviewers’ concerns!

• “A more comprehensive approach would be to report both the range plus the standard deviation” We agree that including standard deviation gives a more comprehensive understanding of the distribution, and we have added analysis to Appendix B.2 (see “Characterizing a model’s accuracy distribution beyond spread” and Figure 20) showing the distribution of accuracy across 500 formats for 30 different tasks. Plots also include the standard deviation, which varies from stdev=0.31 to 0.02 across the tasks considered (median stdev=0.04). We would like to emphasize that reporting the spread (i.e. the distribution range) is already a much more comprehensive report than the single-format estimation that is currently the norm, and that efficiently estimating the standard deviation of a distribution is a priori a much more computationally expensive problem. This would require precisely computing the accuracy of possibly hundreds of formats, and would render our current bandit formulation unsuitable. We nonetheless believe that efficiently computing the standard deviation of such a distribution of accuracies is an exciting question for future work!

• “Why classification tasks? What are the challenges associated with measuring performance in long generation tasks?” We focused primarily on classification tasks since they unambiguously showcase clearly that performance spread can be a serious concern, given that they are generally much more straightforward to evaluate than text generation metrics. Text generation evaluation is notoriously more difficult, noisy, and subjective than simple classification metrics. Text generation metrics remain an active area of research, and metrics are often task-specific (e.g. a summarization metric) which would hinder us from using the same metric across tasks. Moreover, evaluating text generation often requires a multifaceted approach (e.g. fluency, helpfulness, similarity with the recorded human response) and we specifically wanted to avoid conflating the establishment of the spread discussion with the open problem of text generation evaluation. We nonetheless conducted additional experiments on sentence-length text generation problems; see next question for details!

• “I suggest that the authors either explicitly state that these findings are specifically under the context of classification tasks or conduct additional experiments on long-text generation” We agree with the reviewer that longer text generation is an important setting to evaluate for LLM usage, and thus we are included additional experiments for text generation using BertScore and ROUGE-L as metrics in App. B.2 (“Experiments with continuous metrics in open-ended text generation tasks”, also Fig. 19), showing that the issue of performance variance across formats still holds: we computed spread on 10 randomly sampled formats on 17 open text generation tasks with LLaMA-2-7B, and found that performance variance remains regardless of the metric and the number of n-shots considered, with LLaMA-2-7B 5-shot having 25% of tasks with a ROUGE-L spread ≥ 0.098, and a BERTScore spread ≥ 0.09. ROUGE-L is the metric used in the original Super-NaturalInstructions tasks, but is known to have a preference for lexical similarity; therefore, we complement results with BertScore, a model-based metric (which, like all model-based open generation metrics, is more robust to changes in vocabulary but that will express Bert’sinductive bias). We focus our new experiments on evaluating sentence-long text generation tasks, since longer text generation experiments are significantly more computationally intensive, given that they require processing a significantly longer input including the solved few-shot examples, plus generating a long text. We will carefully qualify the extent of our experiments across the manuscript: we also added a Limitations section (Appendix C) in this same vein.

• “Pearson correlation coefficient in Figure 2?” We have now included the Pearson correlation coefficient in all scatter plots, including appendix.

We believe this resolves the concerns the reviewer flagged and we thank them again for their thorough review!

We thank the reviewer for their insightful and encouraging feedback! We are glad they pointed out our work covers a “subject that has not taken much attention from the community but should be addressed for robust application of LLMs”, with a method that “can be utilized on API-gated models” and gives a “more informative understanding of the model's performance and robustness”. We address all questions below.

• “Does the formulation still hold for the tasks that cannot be evaluated in discrete value?” It does! Thompson Sampling still works for continuous rewards (see Algorithm 1 of Chappelle & Li, 2011; Bernoulli bandits are shown in Algorithm 2). We focused our explanation on the exact setting we used throughout the paper for ease of reading, but the only requirement to extend it to continuous rewards would be to provide a more suitable reward distribution to sample from instead of sampling from a Beta (e.g. a Gaussian).

• “A good initial point (a prompt) would be crucial for a successful entire optimization. How can we be convinced that we start from a good point?” We are not using a local search algorithm, where an initial point would strongly influence the trajectory of the search; instead, our approach samples random formats by assigning different values to all the constants defined in the grammar. The only influence that the initial prompt has on the search is that if such a prompt is available for a task in our evaluation data, we use it to induce the prompt format grammar for that task.

• “current important applications of LLM assume multi-turn conversation between the user and LLM, which is highly dependent on the conversation history” We fully agree! Formatting brittleness issues may still stand in these settings, as one can consider each partial conversation as an individual test point to be analyzed in the context of FormatSpread. Conducting a sensitivity experiment with humans in the loop would be difficult because the user behavior might also change in response to system behavior. However, we hypothesize that this sensitivity may be even more detrimental in these multi-turn settings, as small differences in behavior might compound turn after turn. This would be an exciting future direction to test LLM robustness beyond the few-shot setting we focus on!

We thank the reviewer for their detailed feedback! We are glad they mentioned that we explore “a very interesting question” and that our “proposed construction of grammar rules [...] and the design of sensitivity evaluation are quite ingenious”. We address all questions below.

• “Is the difference in prompt formats the only influencing factor, or do other confounders exist, such as the content length of in-context, different tokenize methods?” This is a great suggestion. While our work focuses on measuring the sensitivity to prompt formatting in particular, our analysis in Section 4.2 could definitely be adapted to features like the length of in-context examples. We don’t claim prompt formatting to be the sole influencing factor on performance variation, but rather a compelling example of spurious features that can seriously affect final performance. Nonetheless, this question inspired us to add an analysis on the impact of in-context length in Appendix B.2 (“On factors influencing performance variance besides prompt formatting”), where we found a negligible correlation between the spread and both the average and the standard deviation of prompt length for a task: correlation was $r=0.228$ ($p=1.4\times 10^{-7}$) for mean prompt length when using exact matching prefix as accuracy metric, and $r=-0.022$ ($p=0.615$) for option ranking metric, see App B.2 for all details. Analyzing the impact of a specific tokenizer exceeds the scope of this work, since decoupling the influence of a model’s tokenizer vs. other factors like the data it was trained on would require training several models with identical training procedures but different tokenizers.

• “How can we analyze which types of tasks are more susceptible to prompt format influences, rather than just conducting sensitivity evaluations?” This is an interesting question, and one we focused on as part of our preliminary explorations. We did not find specific factors that can faithfully predict a priori the sensitivity of a model on a given task. For example, we did not find that multiple choice questions had any significant difference from regular tasks with two fields (e.g. “Input: <text> Output: <text>”) or more. However, digging more into why certain tasks result in more sensitivity is an exciting direction for future work!

• On the counterintuitiveness of our sensitivity results. Sensitivity to spurious features in prompt design is understudied and often not taken into account; much existing research and practice uses LLMs with the assumption they are robust to these features, so we believe that quantifying this sensitivity is crucial to give it its deserved visibility! That being said, we do not believe that our results are necessarily counterintuitive to what we know about LLMs. For example, few-shot examples’ ordering (which doesn’t drastically change the semantics of a prompt) may affect final results (Lu et al., 2022). As models with multiple billions of parameters, it is not unreasonable to expect their distribution to have sharp changes and discontinuities: we do however hope that FormatSpread sparks discussions on new training objectives that provide robustness guarantees to spurious prompt changes!