Fool Your Large (Vision and) Language Models with Embarrassingly Simple Permutations

Thank you very much for reviewing our paper and providing helpful comments. Here are our responses to your comments.

Q1. Related work section

We would like to clarify that a comprehensive review of related work is indeed present in our paper, specifically in Section 5. This section discusses the development of Large Language Models (LLMs) and Vision-and-Language Models (VLLMs), their wide adoption, and popular studies using multiple-choice question answering for evaluation. Additionally, it includes an analysis of the robustness of LLMs and VLLMs. We hope this section can draw parallels with previous studies, and highlight the distinct contributions and originality of our work.

Q2. Comparison of LLMs

Thank you for pointing this out. We acknowledge that GPT-3.5 Turbo is included in Table 3 but not in Tables 5 and 6. The reason for this is primarily due to budgetary constraints. To address this issue and maintain the integrity of our comparative analysis, we have now conducted additional experiments with GPT-3.5 Turbo for Table 5 and 6. The results can be seen in the updated tables. Additionally, we also incorporated Llama2-70B, a strong open-source model, in the revised version of our manuscript. This addition aims to provide another comparable level of insight and draw similar conclusions to what would have been achieved with GPT-3.5 Turbo.

Q3. Details of the experiments

Thank you for your suggestions. All of the models (except GPT-3.5-turbo) we used are open-sourced models. Their weights can be obtained from the official Huggingface model zoo (https://huggingface.co/models) or the original Github repositories. We have added additional descriptions of these details in the Reproducibility Statement section and we have uploaded the source code to reproduce these experiments.

Q4. Testing of GPT-3.5-Turbo

Thank you for your question. Yes, we used the official OpenAI GPT-3.5-turbo API for the experiments following their documents (https://platform.openai.com/docs/api-reference/chat).

Dear Reviewer FKE2,

We want to thank you here, again, for the constructive comments and acknowledgment of this paper. We have conducted additional experiments and provided detailed explanations to try to address all of your concerns. Could you please kindly check our revised paper and our responses, to see if your concerns are solved? We would really like to hear if you have any further questions before the discussion window is over. And if no more questions, please could you consider updating the score?

Sincerely,
Authors

Thank you very much for reviewing our paper and providing helpful comments. Here are our responses to your comments.

Q1. Depth and Novelty

Novelty: We would like to emphasize that the crux of our work's novelty does not lie in the complexity of the permutation method itself, but rather in the unexpected and significant findings that emerge from this straightforward yet effective approach. Our contribution is in uncovering a profound vulnerability in LLMs and VLLMs, which are otherwise regarded as highly advanced and robust. This vulnerability, triggered by simple permutations in MCQ options, reveals a critical aspect of these models that challenges their reliability in real-world applications. This phenomenon has not previously been reported in the field. Depth: The depth of our study lies in the extensive empirical analysis that such a basic manipulation can lead to drastic performance degradation. In over 100 experiments, we show that the vulnerability exists across more than 20 highly regarded foundation models and more than 9 established benchmarks. The fact that the phenomenon is so widely observed makes the result highly salient for both researchers and practitioners.

Q2. Other attacks

Thank you for your suggestions. In our original version, we compare our attack strategy with the position attack, i.e. always rotating the correct answers to a specific position (A/B/C/D) in the answer list. The results show that although the models also exhibit varying degrees of position bias, our adversarial permutation shows a much stronger effect (a larger proportion of predictions drop to below chance level). To further address your concern, we conducted additional experiments to compare with three more types of attack methods. (1) ICL Permutation attack: Given a fixed set of few-shot in-context examples, permute the in-context learning examples to their worst-case order. (2) ICL Search attack: Search for the worst-case set of ICL examples to use from within the training set. (3) Symbol attack: Consider different types of option symbols (e.g. A/B/C/D, 1/2/3/4, a/b/c/d), and choose the worst-case symbol set for each question. We present the results of these attacks on MMLU dataset below and the results of other datasets in the appendix Table 16-20.

As can be seen from the table, while other attacks can also harm the performance, our adversarial attack has the biggest impact on the LLMs and causes the largest performance drop.

Dear Reviewer EvuZ,

We want to thank you here, again, for the constructive comments and acknowledgment of this paper. We have conducted additional experiments and provided detailed explanations to try to address all of your concerns. Could you please kindly check our revised paper and our responses, to see if your concerns are solved? We would really like to hear if you have any further questions before the discussion window is over. And if no more questions, please could you consider updating the score?

Sincerely,
Authors

Thank you very much for the thoughtful review and excellent questions. Here are our responses to your comments.

Q1. Adversarial attack v.s. Permutation

The term “adversarial attack” in the context of machine learning generally connotes any deliberate attempt to deceive or manipulate a model's output. Adversarial examples refer to an input that is crafted to look normal to humans while triggering erroneous behaviour from the model. We argue that MCQ permutation falls within this scope, as humans do not care about the permutation of the candidate choices, but we show that a worst-case permutation dramatically worsens model performance in a way that is not anticipated or intended by the model's design.

Secondly, we would like to emphasize that our choice of terminology was not meant to suggest a sophisticated or elaborate attack mechanism. On the contrary, the core message of our paper is to highlight the surprising fact that even the most straightforward forms of manipulation, such as the proposed permutation, can significantly impair the performance of advanced LLMs and VLLMs in MCQ scenarios. The fact that such a simple attack is successful only enhances the salience of our result. This finding is crucial as it underscores the need for future work to improve the robustness of these models against even the simplest forms of potential manipulation.

Lastly, we are grateful for your suggestion of MMBench. We emphasize that MMbench performs circular rotation of answer candidates, not permutation of answer candidates. I.E: For N candidate answers, MMBench tries $N$ orders, preserving the relative order of the options, while we consider all possible $N!$ permutations, breaking the relative order of the options. In the revised version, we have added extra experiments comparing our permutation approach with MMBench circular evaluation (updated Table 6). The results show that our permutation results in much worse performance, demonstrating a stronger attack. The distinct outcomes of rotation and permutation are a critical step toward understanding this vulnerability and ultimately defending against it.

Q2. Temperature and sampling strategies

Thank you for your question. For all of the experiments, we consistently set the temperature to 1 and used greedy decoding for the generation to ensure reproducibility (specified in section 2.1 “Evaluation” paragraph). Specifically, we follow the common practice for MCQ evaluation [1][2] to obtain the probabilities of the first token and select the one with the highest probability, comparing it against the ground truth answer. (Note that in the MCQ context, we only need to select one token to answer.) Under this greedy evaluation scheme, variations in temperature or the adoption of different sampling strategies do not impact the results. This actually further enhances the generalizability and importance of our results for MCQ evaluation.

However, to address your concern, we performed additional experiments using different temperatures (0.5, 1.5) and two commonly used sampling strategies (top k sampling and nucleus Sampling). As can be seen from the table below, the greedy decoding strategy we adopted achieves the best accuracy before and after the permutation among different decoding strategies. And the performance of the other decoding strategies are even worse after the permutations. This confirms both that our initial experiments were optimally configured, and also that the findings generalize to other decoding strategies. This analysis has been added to Appendix A.2 of the paper.

[1] Measuring Massive Multitask Language Understanding. Hendrycks et al., ICLR 2021.
[2] Holistic evaluation of language models. Liang et al., 2022.

Dear Reviewer RBKq,

We want to thank you here, again, for the constructive comments and acknowledgment of this paper. We have conducted additional experiments and provided detailed explanations to try to address all of your concerns. Could you please kindly check our revised paper and our responses, to see if your concerns are solved? We would really like to hear if you have any further questions before the discussion window is over. And if no more questions, please could you consider updating the score?

Sincerely,
Authors

The authors have answered my questions. I have no more questions. As promised, I am increasing my score.

Thank you very much for your review and questions. Here are our responses to your comments.

Q1. Intuitive explanations of the phenomenon

We have further delved into it and would like to present our additional findings:

1. Option Symbol-Answer Content Shortcut: Beyond position bias, we hypothesized another contributing factor: the model may suffer from shortcut [1] or spurious correlation learning [2] between option symbols and answer content. To investigate this, we conducted three experiments using different answer symbol sets (A/B/C/D, a/b/c/d, and I/II/III/IV). We examined if models exhibit similar patterns in response to the same permutations but only with different symbols, keeping the answer content constant. We conducted experiments on ARC-Challenge dataset with all models. Interestingly, our results revealed a high correlation for similar symbol pairs (A/B/C/D and a/b/c/d) with a Pearson correlation score of 0.76. However, for more distinct symbol pairs (A/B/C/D and I/II/III/IV), the correlation score dropped to 0.36, indicating that the models’ response to permutation differs significantly when only the symbols varied. In other words, the baseline accuracy and permuted accuracy are almost the same for different symbol sets, but they respond very differently to permutation. This discrepancy points to the models having learned shortcuts between symbols and answer content, which may give extra vulnerability to the permutations of the models. This offers an additional explanation for the observed phenomenon. This analysis has been added to Appendix A of the paper.

2. Bias from the Training Set: We also considered data in the training set might lead to the observed vulnerability in MCQA [3]. Training data biases can arise from the predominance of certain answer patterns or thematic content. For instance, if the training set disproportionately represents specific types of questions or contextual themes, the model may develop an over-tuned response to these patterns. This over-tuning could result in a skewed performance when the model encounters permutations in MCQs that deviate from these familiar patterns. However, given that current LLMs and VLLMs are trained on vast and varied datasets, quantifying the exact impact of these training data biases on model vulnerability remains a challenging endeavor and we leave it for future work.

[1] Geirhos, Robert, et al. "Shortcut learning in deep neural networks." Nature Machine Intelligence 2020.
[2] Sagawa, Shiori, et al. "An investigation of why overparameterization exacerbates spurious correlations." ICML 2020.
[3] Du, Mengnan, et al. "Shortcut learning of large language models in natural language understanding: A survey." arXiv 2022.

Dear Reviewer d9tg,

We want to thank you here, again, for the constructive comments and acknowledgment of this paper. We have conducted additional experiments and provided detailed explanations to address your concerns. Could you please kindly check our revised paper and our responses, to see if your concerns are solved? We would really like to hear if you have any further questions before the discussion window is over. And if no more questions, please could you consider updating the score?

Sincerely,
Authors