Evaluating the Zero-shot Robustness of Instruction-tuned Language Models

W1 (Soft-prompt size): We use a prefix length of 10 for the result reported (thank you for pointing this out!) We have explored many different settings (e.g., soft-prompt size) in our pilot studies and noticed no substantial differences, suggesting that one needs only a minimal set of free parameters to tune with the proposed objective to realize improvements. Below we reproduce results observed when using different soft-prompt sizes (from 1 to 50), and show their performance and semantic distance:

The results indicate that the prefix length does not affect the performance for both observed and unobserved instructions. As they fluctuate around the same level. Additionally, we also calculated average distance between unobserved instructions to their closest equivalent observed instructions, and the value is highly similar between all soft prompt sizes.

The takeaway here is that soft-prompt is only a method we use to drag the equivalent instructions close together without affecting the model’s performance, and the number of parameters we dedicate to do that is seemingly insignificant.

Q1 (Dataset size): The batch size we use for both models is 4, hence each batch contains one “prototype” instruction and 3 paraphrased instructions to align (this is also reported at the beginning of Appendix B.3.) We randomly sampled 1k prototype samples from the original instruction tuning datasets, hence the final dataset altogether is 4k; we report this Section 5.

W2 & Q2 (Significance Test): Please see our general response, (1). Briefly, this is likely an instance of “catastrophic forgetting”, a hypothesis supported by the results we have added in Appendix G.

We appreciate the thoughtful feedback.

W1 (Significance test): Please see our general response, (2). Briefly: The results in Tables 3 and 7 are statistically significant, despite the observed variances. We have added significance results to the manuscript. We also note that in Table 3, the variances are much larger for the unobserved instructions, which is part of the issue that we are highlighting.

W2 (Size of LLMs): The reviewer is correct that our work here is focussed on modestly sized LMs. We have tried to be clear about this in the abstract and introduction. We write in the second sentence in the abstract that instruction tuning “... has shown particular strength in improving the performance of modestly sized LLMs, sometimes inducing performance competitive with much larger model variants”; this is what motivates our focus on these <= 11b models. We believe this analysis is important precisely because prior work has suggested that instruction tuning imbues such models with performance surprisingly competitive with massive LLMs; our findings call this into question.

W3 (Negation): The reviewer suggests an interesting additional analysis: Evaluating sensitivity to instruction negation. We have now conducted such analyses and report details (experimental setup and results) in Appendix F. For convenience, below we reproduce the aggregated results on how PT + KL does on all 6 settings (including negated observed instructions) we conducted in Section 4.5

This table reports the results we reported in Figure 3 (first row) after applying PT + KL (second row). We observe that PT + KL yields non-trivial improvement on models performance with: 1) suitable instructions (Closest) that are observed. 2) suitable instructions (Task Designer, Collected) that are unobserved. However, results also indicate that the proposed method does not succeed fully in terms of imbuing the model with the ability to truly “understand” instructions: The performance of observed instructions after adding negation does not drop significantly (only by 1.2 on average)

We hypothesize that training the model to align semantically equivalent instructions does not explicitly incent the recognition of negated prompts. This is because our objective enforces that equivalent instructions should be similar, but negations rarely occur in the collected paraphrases. Moreover, learning to handle negations would more intuitively be done by encouraging the model to push negated versions of instructions away from their original forms; this differs from the form of our objective which focuses on similarity. Therefore, we conduct another experiment by modifying the objective slightly to handle negation.

Specifically, we trained the model to push the representation of the negated instructions away from the original ones. To do so, we take the same 1k seed instances from the FLAN collection and prompt (with three different prompts) GPT-4 to rewrite them such that the instruction is negated. We keep all the hyperparameters same as how we did for PT + KL, but use KL to push the representation of negated instructions away from the originals:

\mathcal{L} = (1-\lambda) \mathcal{L}_{CE} - \lambda \mathcal{L}$$ _{KL}

After training, we run the “closer look” experiment again to see if this slightly different objective would result in a stronger robustness (third row). The result shows a significant drop of the accuracy when negation is added to the observed instruction. However, the performance in other settings remains comparable with the baseline when applying the PT + KL (Negation).

This result provides additional evidence for our findings that robustness to instruction variants is strongly tied to the distances between learned representations of instructions: When we incentivize the model to learn similar representations for equivalent instructions, it performs better when given equivalent unobserved instructions, though it does not better handle negated instructions. But if we modify this slightly to encourage the model to induce dissimilar representations between instructions and their negated versions, it improves its responsiveness to negated instructions. In short, the proposed strategy of targeting (dis)similarity of model instruction representations is general enough to facilitate realizing robustness beyond paraphrases.

We report a full set of results for this finding in Appendix F in the updated draft.

Q1 (RLHF): RLHF is most appropriate for aligning model outputs for tasks that require open-ended responses. In our work, we are benchmarking instructions and tasks that have a set of correct answers. Collecting human feedback for training is not a good fit for our main evaluation measures, which focus exclusively on the correctness of the output for (mostly) “objective” tasks like classification.

However, the reviewer raises an interesting idea, in that it is possible that optimizing LLMs to align with human preferences generally via RLHF may implicitly require models to “understand” instructions in a more robust way.

We perform a preliminary investigation into whether training with RLHF on an unrelated, open-ended text generation tasks affects the robustness of the instruction-following abilities. Specifically, we take an off-the-shelf Alpaca-7B trained with RLHF via PPO and run it on all the tasks and instructions we have used in this paper. We found that the output tends to do “extra things” for a given instruction (e.g., providing its explanation for question answer), we found that the overall performance is highly unstable and degrades significantly in most of the datasets.

Therefore, we do not have evidence to show that RLHF on modestly-sized LLMs would mitigate such issue.

As a final additional response to reviewer feedback, we extend our analysis a bit to include a larger (20B) model. As reviewer hcVs pointed out, readers may be interested to see if the results generalize to (very) large models. While we maintain our stance that the success of instruction-tuning for boosting the performance of modestly-sized LLMs is the most exciting aspect of this method, and therefore worthy on its own of study, we nonetheless agree that studying the relationship between robustness and scale of such models is interesting.

However, we can draw valid comparisons only if we keep all other variables (model architecture, pretraining data, and instruction tuning set) fixed. Therefore, we are not able to study black-boxed models (like GPT-3.5 or GPT-4) using this approach. To include additional scale in our work, we therefore conduct the full experiments on different sizes of Flan-T5 ranging from 70M to 11B (Fig 2a and Sec 4.3), as well as Flan-UL2 (20B), and we report the change of delta performance (observed - unobserved) in the table below:

The results above (aggregated by task category) show that there is no apparent relationship between model size and the delta performance (our measure of robustness).

More scientifically, we perform linear regression on the size of the model v.s. (observed - unobserved performance) over data points from all datasets, this yields a slope of $4.45 \times 10^{-13}$ with a p-value of 0.087 ($R^2$ is 0.002). This suggests that model size may weakly correlate with increased robustness, but this relationship is not significant the $p=0.05$ level, and the relationship is tenuous. In other words, scale does seem to improve robustness, but only very moderately.

We hope the additional results (together with those posted earlier) answer your questions. We ask that you consider whether it might be appropriate to adjust your score, if you feel these new findings have changed your view on the work. Thank you.

We thank the reviewer for their thoughtful comments.

W1 (Novelty): Regarding novelty, please see our general response, (3). In brief, our work builds upon but complements and considerably deepens these prior efforts.

Q1 (Augment to initial instruction-tuning set): Due to the modest size of the paraphrases set (4k), it has minimal effect on the training performance and is very time-consuming by adding to the initial instruction finetuning. Please refer to Table 11 in Appendix B.3.1 for the detailed comparison. We reiterate that the main goal of the improvement section is to find a way to narrow down the semantic gap between the observed and unobserved, so that we validate our hypothesis (representation similarity for equivalent instructions v.s. robustness )

W2 & Q2 (Significance Test): Please see our general response, (2). Briefly, the large variances here in part reflect the fact that we are reporting aggregated results over large benchmarks. That said, one of the main observations is that variances are considerably larger for unobserved instructions, which is—as the reviewer rightly points out—undesirable.

To the reviewers’ question: The results in aggregate are statistically significant (p << 0.05) under a paired t-test, and if we run the t-test separately for each model (which avoids the issue of repeating datasets in observations), we observe p < 0.05 for all but T0++ (p=0.13).

W3 & Q3 (Efficacy of the objective): We observe two major differences between the performance using just prefix tuning (PT) and prefix tuning with extra objectives (PT + KL). The performance diff between unobserved instructions and observed instructions - i.e., robustness - is significantly higher (close to zero) for PT + KL (Table 7 in Section 6) The average standard deviation of the instruction performance of PT is even higher than the baseline, indicating instability in its performance; whereas this high variance is significantly mitigated with PT + KL (Table 10 in Appendix B.3)

Intuitively, we design the extra objective to explicitly “increase the similarity” between equivalent instructions — unobserved to observed — from the model’s “perspective”, which we hypothesize to have positive correlation with the robustness of instruction following ability.

Motivated by this, we calculate the average l2 distance between unobserved instructions and its closest observed instruction over all the examples (Same approach as Table 6) for PT and PT + KL. (Fig 7 in Appendix B.3.2). We show that in 13 out of 14 datasets, the distance between equivalent instructions is smaller for PT + KL. The consistency of the close distance and better robustness supports our hypothesis of the correlation.

We have added these results, figures and the discussion to Appendix B.3.2 of the revised paper.

Q4 (Disaggregated results for Table 7): We appreciate the suggestion and we have now added the disaggregated result in a dataset level in Table 30 and Table 31 in Appendix J.3. For convenience, here we reproduce results disaggregated by categories:

This Table reports the average delta performances observed after applying the proposed method on Flan-T5-XL and Alpaca-7B. For BBL datasets we report standard deviations over the individual datasets (for MMLU we group all together so do not report a variance). Table 6 in the paper reports the average distance change after applying the objectives for Flan-T5-XL. In general, we observe a consistent but weak correlation relating the closest distance between equivalent instructions to the delta performance for unobserved instructions. The similarity between equivalent instructions changes little for MMLU, which leads to a marginal improvement. However, this pattern is also not absolute; the BBL-QA category in general receives the greatest improvement with respect to performance but not distance. We leave further analyses of instruction representations and downstream performance to future work building on our results here.

We thank the reviewer for their thoughtful review, and we are glad that they found the analysis here interesting and potentially impactful.

W1 (Title of the paper): We thank the reviewer for the suggestion, and agree this is the main takeaway; we will emphasize this point earlier on (and consider using it as our title) in future version of the manuscript.

W2 (Rephrasing Contribution): We agree with the reviewer that the release of this data is, on its own, not a substantial contribution. However, we do think this dataset is novel with respect to existing resources (e.g., T0/PromptBench, Flan, and others) which have released rephrasings of instructions for the same tasks intended for model training. By contrast, the dataset of diverse instructions we release with this work is intended to aid model evaluation, specifically on tasks in two major benchmarks, MMLU and BBL. Our hope is that this resource will both permit reproducibility of our work, and facilitate additional research into the robustness of instruction-tuned models.

Q1 (Fine-tuning performance): Please see our general response, (1). Briefly, we think this is an instance of “catastrophic forgetting” and have provided new results that support this hypothesis in Appendix G.

Q2 (Rephrasing source): We were interested in evaluating the robustness of instruction tuned models to novel instructions, as they might be observed in deployment—this means we were interested not strictly in rephrasings but in novel instructions written anew by practitioners for specific tasks of interest. Collecting new instructions from NLP graduate students (who are familiar with instructing language models) better aligns with this envisioned stress-testing than using LLMs to automatically rephrase existing instructions; in particular this yields a sample from the distribution of instructions which might plausibly be written by practitioners, so we felt it best to manually collect these for evaluation.

Note that in the proposed methods to make models more robust to instruction variations (Section 5), we did use LLMs to paraphrase existing instructions from the training data as a means to augment the data and realize the proposed objective term.