Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment

Thank you for your feedback and questions. We would like to address your concerns:

1. “There is no comparison to other jailbreaking methods.”

   • Response: Thank you for your suggestion to compare against other jailbreak methods. We have updated our paper with PAIR and DeepInception results, which can be found in Appendix D.4. A summary of our results can be found in our Global Response (item 4). We have also removed the statement that "adversarial attacks typically assume white-box...random augmentations are black-box...we do not compare..." in our revised draft.

2. “The authors can add computation cost compared to other jailbreaking methods.”

   • Response: Thank you for pointing this out. We have provided measurements of execution time alongside our comparison with other jailbreak methods; please see Appendix D.4 and Global Response item 4.

3. “The authors should evaluate the proposed methods against widely-used commercial LLMs (e.g. ChatGPT), where bypassing the safety alignment is a more significant threat.”

   • Response: We have added GPT-4o evaluations to our paper; please see Global Response item 1 and Appendix D.1 for more details.

4. “There is no discussion of mitigation methods against the proposed methods.”

   • Response: Thank you for this suggestion. We have added discussion on possible mitigation methods in Appendix B (lines 805-814).

5. “How safety judge work? Is it done by LLMs, human experts or word matching?”

   • Response: As mentioned in line 227, we use the safety judge provided by the SORRY-Bench dataset, which is a Mistral LLM fine-tuned for safety classification.

Thank you again for your initial review. Your comments have been valuable to us for improving our paper. As the discussion period concludes, we kindly request that you review our recent updates to the paper. In addition, if these updates address your primary concerns, we'd greatly appreciate if you could consider raising your score accordingly.

Warm regards,

Authors

Dear Reviewer UNaU,

As the rebuttal period comes to a close, we kindly ask you to update your initial review taking into consideration our recent changes. We believe that our paper updates have addressed your core concerns. In particular, we would like to highlight that Reviewer aoVp has updated their score from a 3 to a 6. We greatly appreciate the time you have already put into your review; your engagement matters immensely to us.

Sincerely,

Authors

Thank you for your detailed feedback and useful suggestions. We would like to address your concerns:

1. “Can the authors provide a comparison of the success rate of random augmentations against the success rate of more sophisticated adversarial attacks in the same LLM models?”

   • Response: Thank you for your suggestion to compare against existing attacks. We have updated our paper with PAIR and DeepInception results, which can be found in Appendix D.4. A summary of our results can be found in our Global Response (item 4).

2. “The study could benefit from an expanded discussion on the practical implications of these findings, especially in terms of what types of applications or LLM deployment scenarios are most at risk and possible mitigations.”

   • Response: This is a great suggestion. We have added much expanded discussion on assessing risk and possible mitigation strategies in Appendix B.

3. “Among all these various dimensions( augmentation type, model size, quantization, fine-tuning-based defenses, and decoding strategies), can authors provide a ranking of these factors based on their significance towards the attack performance?”

   • Response: Certainly! We have updated our paper with a ranking in Section 4.4.

4. “The quantization experiment results in Figure 4 show variability across different models and sizes, but the authors provides vague explanations for these differences. Further exploring why certain models (e.g., Qwen 2) exhibit unique behavior under augmentations would enhance the study.”

   • Response: We agree that Qwen 2 requires a deeper exploration to more fully explain its results in the quantization experiment. On the surface, the reason that W4A16 attains a negative success rate gain is that the response quality is poor, as shown in Figure 19. This of course begs the question: why is the response quality so poor for W4A16 Qwen 2 specifically? One possible explanation is from the recently proposed scaling laws for precision [1], which observed that increasing the pre-training data size incurs greater performance degradation after post-training quantization. Qwen 2 7B was trained on 7 trillion (T) tokens [2], which is greater than Llama 2 (2T) [3] and Phi 3 (4.8T) [4]. (The pre-training data size for Mistral is not publicly available. Also note that Vicuna and Zephyr are fine-tuned versions of Llama 2 and Mistral, respectively.) However, Llama 3 and Llama 3.1 were trained of 15T tokens [5], yet do not experience such degradation in our experiments. Hence, the proposed scaling law does not completely explain the results for Qwen 2, and thus further investigation is required that may be outside the scope of our work (but nonetheless would be quite an interesting future research direction!)

5. “The writing of the conclusion is inconsistent with the abstract and conclusion, making it difficult to understand the main takeaway of this paper.”

   • Response: Thank you for pointing this out. We have updated our conclusion to make it more consistent with the abstract and to make the main takeaway of the paper more focused.

6. “Have the authors considered examining the temporal robustness of this attack, i.e., does the attack success vary over repeated attempts or prolonged interaction with the same model?”

   • Response: This is a great suggestion! While examining repeated attempts would be interesting to investigate, due to time constraints we have decided to prioritize other additional experiments, and leave this investigation for future work.

[1] Kumar, Tanishq, et al. "Scaling Laws for Precision." arXiv preprint arXiv:2411.04330 (2024).

[2] Yang, An, et al. "Qwen2 technical report." arXiv preprint arXiv:2407.10671 (2024).

[3] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[4] Abdin, Marah, et al. "Phi-3 technical report: A highly capable language model locally on your phone." arXiv preprint arXiv:2404.14219 (2024).

[5] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

Thank you again for your initial review. Your comments have been valuable to us for improving our paper. As the discussion period concludes, we kindly request that you review our recent updates to the paper. In addition, if these updates address your primary concerns, we'd greatly appreciate if you could consider raising your score accordingly.

Warm regards,

Authors

Dear Reviewer aoVp,

Thank you for reviewing our changes. We greatly appreciate your time and improvement in score.

Sincerely,

Authors

Thank you for your detailed feedback and suggestions! We would like to address your concerns:

1. “How does your attack perform on instruction-tuned models and when including (or excluding) system prompts? (it's also unclear if system prompts were used during evaluation)”

   • Response: This is an excellent suggestion. We have added experiments using a safety-encouraging system prompt to our paper, which can be found in Appendix D.2. A summary of our results can be found in our Global Response (item 2). Our original submission used the default system prompt for each model, and upon closer inspection, we observed that by default most models leave their system prompt empty, whereas Qwen 2 and Vicuna utilize default system prompts that may impact the results (e.g. “You are a helpful assistant” for Qwen 2). In our revised draft, we have updated the experimental results in the main text so that Qwen 2 and Vicuna have empty system prompts to be consistent with the other models. We have also added a note on line 284 regarding the system prompt to help clarify our setup in the paper. Please see Correction #1 in our Global Response for more details.

2. “Additional experiments with larger models, or with instruction-tuned models would also further convince me that the approach is robust and general”

   • Response: Thank you for your suggestion. We would like to clarify that all the models we investigate are instruction-tuned, as mentioned in Section 3.2. Specifically, the Llama, Phi and Qwen families have undergone explicit safety alignment, and the remaining families were included to see if any interesting patterns can be observed for unaligned models. For larger models beyond 13B, we provide GPT-4o evaluations; please see Global Response item 1 and Appendix D.1 for more details.

3. “How does your method compare to prior work that adjusts prompts?”

   • Response: Thank you for your suggestion to compare against prior works. We have updated our paper with PAIR and DeepInception results, which can be found in Appendix D.4. A summary of our results can be found in our Global Response (item 4). In terms of literature review, we have also expanded discussion of comparison to prior work that were cited in the introduction; please see Appendix A.1.

4. “How susceptible are closed-source models to this attack?”

   • Response: We have added GPT-4o evaluations to our paper; please see Global Response item 1 and Appendix D.1 for more details. In short, GPT-4o is safer than the other open-source models, but nonetheless we can still find successful jailbreaks (see Figures 7 and 8 in the appendix for examples).

5. “Have you tried string-level augmentations that insert words from a dictionary instead of random text?“

   • Response: This is a great suggestion. However, due to time constraints, we have decided to prioritize other experiments over adding a new augmentation type, and therefore leave this investigation to future work.

6. “I could not find any data with varying choices of p.”

   • Response: Thank you for this suggestion! We have added ablation experiments on p to our paper; please see Appendix D.3 for full details and Global Response item 3 for a summary.

Thank you again for your initial review. Your comments have been valuable to us for improving our paper. As the discussion period concludes, we kindly request that you review our recent updates to the paper. In addition, if these updates address your primary concerns, we'd greatly appreciate if you could consider raising your score accordingly.

Warm regards,

Authors

Dear Reviewer z9us,

As the rebuttal period comes to a close, we kindly ask you to update your initial review taking into consideration our recent changes. We believe that our paper updates have addressed your core concerns. In particular, we would like to highlight that Reviewer aoVp has updated their score from a 3 to a 6. We greatly appreciate the time you have already put into your review; your engagement matters immensely to us.

Sincerely,

Authors

Thank you for your valuable suggestions and feedback! We would like to address your concerns:

1. “The authors did not discuss how the system prompt was configured for the open-source models in the experimental setup”

   • Response: Thank you for pointing this out. Our original submission used the default system prompt for each model. Upon closer inspection, we observed that by default most models leave their system prompt empty, whereas Qwen 2 and Vicuna utilize default system prompts that may impact the results (e.g. “You are a helpful assistant” for Qwen 2). In our revised draft, we have updated the experimental results in the main text so that Qwen 2 and Vicuna have empty system prompts to be consistent with the other models. We have also added a note on line 284 regarding the system prompt to help clarify our setup in the paper. Please see Correction #1 in our Global Response for more details.

2. “The authors should also examine whether using a system prompt as a defense mechanism is effective.”

   • Response: This is an excellent suggestion. We have added experiments using a safety-encouraging system prompt to our paper, which can be found in Appendix D.2. A summary of our results can be found in our Global Response (item 2).

3. “The authors should discuss the vocabulary sizes of the different LLMs tested.”

   • Response: Thank you for providing this suggestion. We examined vocabulary sizes for each of our models to assess whether there is any relationship between vocabulary size and safety against random augmentations. Our conclusion was that there is no strong relationship between the two. For example, the vocabulary sizes of safety-aligned models Llama 2 7B Chat, Phi 3 Small 8K, Llama 3.1 8B and Qwen 2 7B are 32000, 100352, 128256 and 152064 respectively, yet their average success rate gains are 9.1%, 13.5%, 5.4%, and 14.4% (source: Table 7). Increasing vocabulary size does not seem to induce a meaningful trend in the success rate gain, and thus we conclude that vocabulary size is not an important factor for robustness. We hypothesize that other factors such as the nature of the alignment dataset may matter much more than vocabulary size, and leave a deeper investigation of training data’s impact to future work.

Thank you again for your initial review. Your comments have been valuable to us for improving our paper. As the discussion period concludes, we kindly request that you review our recent updates to the paper. In addition, if these updates address your primary concerns, we'd greatly appreciate if you could consider raising your score accordingly.

Warm regards,

Authors

Dear Reviewer 9Lec,

As the rebuttal period comes to a close, we kindly ask you to update your initial review taking into consideration our recent changes. We believe that our paper updates have addressed your core concerns. In particular, we would like to highlight that Reviewer aoVp has updated their score from a 3 to a 6. We greatly appreciate the time you have already put into your review; your engagement matters immensely to us.

Sincerely,

Authors

Thank you for your suggestions and valuable critique of our work. We would like to address each of your concerns:

1. ”Comparison with state-of-the-art blackbox jailbreak attacks (such as PAIR and DeepInception) in terms of attack success rate and computational cost”

   • Response: Thank you for your suggestion to compare against existing attacks. We have updated our paper with PAIR and DeepInception results along with measurements of execution time, which can be found in Appendix D.4. A summary of our results can be found in our Global Response (item 4).

2. “The techniques used (i.e., perturbation) are already widely employed to evaluate LLMs' safety and robustness (e.g., smoothllm).”

   • Response: Thank you for pointing this out. We would like to clarify that prior work that investigates random augmentations in the context of generative language model safety such as SmoothLLM focuses on applying augmentations purely for defense purposes, whereas our work focuses on applying augmentations for attack purposes. We have updated our Related Works section to discuss this in more detail; please see Appendix A.4.

3. “The impacts of model size, quantization, fine-tuning, and generation configurations on LLM safety have been well-studied previously.”

   • Response: We agree that these dimensions have been previously studied in the context of LLM safety. However, the purpose of our work is not to claim that we are the first to explore these. Crucially, much of these works focuses on evaluation against adversarial attacks such as GCG, are strongly limited in the random augmentations they investigate, or only examine notions of safety other than jailbreak vulnerability (such as hallucinations or privacy). Our work contributes to a greater understanding of an underexplored vulnerability in LLM safety alignment and provides much more comprehensive results for safety under random augmentations than any other work currently in the literature. We have updated our Related Works section to compare our work with existing work in each of these dimensions; please see Appendix A.5.

4. “How frequently do random augmented prompts affect the semantic meaning of the original malicious prompt? Since the output is binary, could this lead to false safe/unsafe classifications by the evaluator, potentially impacting the experimental results?”

   • Response: Indeed, random augmentations can occasionally change the semantic meaning of the original prompt. A safety judge takes in both the original prompt and the model response, and a robust safety judge should not classify a response as “unsafe” if it does not actually address the original prompt’s intent. We observed that the safety judge we use is not robust in this sense, and can occasionally produce false positives due to this phenomenon. This is in part what motivated our use of the $\gamma$ threshold in the (k, $\gamma$)-success rate definition and our human study (see Appendix C and Table 2). The experimental results we report in our paper are therefore calibrated to account for such false positives. Due to time constraints, we did not run additional experiments to quantify the frequency of semantic changes, as we feel that our human study addresses the core concern of false classifications in the evaluator and that these additional measurements would just be supplementary.

5. “Have you considered combining random augmentations with existing jailbreak attacks to see if it can lead to better performance?”

   • Response: This is a great suggestion! While combining random augmentations with other jailbreak attacks would certainly be interesting to investigate, due to time constraints we chose to prioritize adding evaluation results for the existing jailbreak attacks PAIR and DeepInception without being combined with random augmentations. We leave exploring the interactive effects of random augmentations with other attacks up to future work.

Thank you again for your initial review. Your comments have been valuable to us for improving our paper. As the discussion period concludes, we kindly request that you review our recent updates to the paper. In addition, if these updates address your primary concerns, we'd greatly appreciate if you could consider raising your score accordingly.

Warm regards,

Authors

Dear Reviewer Tz62,

As the rebuttal period comes to a close, we kindly ask you to update your initial review taking into consideration our recent changes. We believe that our paper updates have addressed your core concerns. In particular, we would like to highlight that Reviewer aoVp has updated their score from a 3 to a 6. We greatly appreciate the time you have already put into your review; your engagement matters immensely to us.

Sincerely,

Authors

Finally, we have made a few corrections to the existing data presented in our work:

• Correction #1: Updated model size and quantization experiment (RQ2) results for Qwen 2, Llama 3.1 and Vicuna. The update consists of removing the default system prompts for these models so that they are consistent with the other models that have empty default system prompts for this experiment, as well as fixing a bug in the prompt template for quantized Vicuna models. (The Llama 3.1 default system prompt added some date information, and the Qwen 2 and Vicuna default system prompts encouraged the assistant to be helpful or safe.)
  • Results: The new results are similar to the old ones and thus the conclusions for these experiments are unchanged.
• Correction #2: Temperature sampling experiment (RQ3) results have been updated following discovery of a bug in the output sampling procedure.
  • Results: The conclusion for this experiment has been updated: for only Llama 2, Llama 3 and Phi 3, character deletion further improves the success rate on top of output sampling. For other results, random augmentations tend to decrease the success rate on top of output sampling. Hence, our main conclusion for RQ3 has been qualified to “output sampling and random augmentations can sometimes work together to provide even greater attack effectiveness.” Nonetheless, our findings that random augmentations are effective under greedy decoding remain unchanged, and we believe this becomes key to the successful random augmentation jailbreaks found for GPT-4o.

Sincerely,

Authors