Thank you for your comments! We would like to point out that the novelty of our work lies in automating manually discovered red teaming strategies. We achieve very high attack success rates with a lower computational budget than the most comparable work. We do this by following an established evaluation protocol on a curated dataset of adversarial prompts published in peer review work. Our attack method is easily extendible with newly discovered techniques simply by describing them in plain English.

Our method is also easily extensible to new models. We have conducted additional experiments showing that the deepseek-r1 model can be “plugged in” to GOAT without any modifications. Under the same evaluation protocols, When we use DeepSeek-r1 as the attacker model and:

• GPT-4o as the target, we achieve 63% ASR@1 (compare to 54% from Fig. 2 with the original attacker and 70% from Fig. B.1 with GPT-4o as the attacker)
• DeepSeek-r1 as the target: 54%
• o1 as the victim: 30%

These are the strongest ASR@1 results for any attacker and victim and we believe this shows the method is applicable as-is to newer models with more advanced capabilities.

Additionally, our method is directly applicable to new datasets. To show this, we conducted additional experiments without modifying GOAT on a new benchmark. We picked AILluminate – a safety benchmark released by the MLCommons alliance and endorsed by 72 universities and research labs, including 4 major foundational model labs (OpenAI, Anthropic, Meta, and Google) and prestigious universities such as Stanford, MIT, and Cambridge.

When rerunning on the widely endorsed Ailluminate benchmark with the original the attacker and Llama 3.1 70B as the target, GOAT maintains its advantage over Crescendo:

We used the sample publicly available at https://github.com/mlcommons/ailuminate and first described in arXiv:2404.12241. We excluded the prompts on the topic of CSE (due to legal restrictions) but otherwise used all prompts labeled as “unskilled” - since GOAT and Crescendo both transform basic questions into successful jailbreaks. This left us with 550 prompts.

We hope these additional results prove the versatility and usefulness of the GOAT method! We look forward to engaging with the reviewer at length about the work and welcome any thoughtful and detailed comments!

We thank the reviewer for their thorough and constructive feedback!

Response to Concerns and Questions

We agree with the reviewer that some additional experiments are warranted and we have expanded to cover additional attacker models (deepseek-r1), target models (deepseek-r1 and o1), additional scoring functions (the OpenAI Moderation API), and additional benchmarks (AIlluminate – a safety benchmark endorsed by 72 major AI institutions). In all cases, GOAT maintains its advantage over Crescendo, even if the numbers vary slightly.

We agree with the reviewer that comparability to other works is important and that is why we have chosen JailbreakBench with its clear and reproducible evaluation protocols and leaderboard with results for multiple attacks. We have included these comparisons in the last section.

Next, we will cite all related works brought up by the reviewer in case of acceptance. We selected the manual attacks we included for automation in GOAT based on our own experience manually red teaming LLMs. There are certainly many more jailbreaking techniques available but we do not claim or aim for comprehensiveness. Instead, GOAT offers a starting point and is designed to be easily extensible. In fact, we believe that narrowly scoped attacks like the ones suggested can easily be automated with the GOAT method.

We also thank the reviewer for suggesting that an overview figure be provided. We agree and we refer the reviewer to Figure 1. We welcome all further constructive feedback on the figure!

The reviewer also raises an interesting point about ablations – that it is not clear how the multiturn nature of the attacks contributed to the ASR. In Appendices B.2 and B.3, we conducted experiments extending to 50 conversational turns. Based on Figure B.2, we can conclude that the multi-turn nature of the attack is crucial: ASR is low on the first turn and it climbs over the course of a conversation. However, in Figure B.3, we also observe that there is a diminishing return to conversation turns as the ASR plateaus around 10.

We look forward to further discussion and appreciate the constructive feedback!

Additional Experimental Results

New Attacker Model

When we use DeepSeek-r1 as the attacker model and:

• GPT-4o as the target, we achieve 63% ASR@1 (compare to 54% from Fig. 2 with the original attacker and 70% from Fig. B.1 with GPT-4o as the attacker)
• DeepSeek-r1 as the target: 54%
• o1 as the victim: 30%

Additional Scoring Functions

When using the the OpenAI moderation API as the scoring function and using the logged conversations of the original attacker model with Llama 3.1 70B as the target, GOAT maintains its advantage over Crescendo.

Additional Datasets

Unfortunately, we found that the JAMBench dataset is not readily available. The cited paper contains no download link and the paper itself does not contain the exact set of 160 prompts curated by the authors. Additionally, we note that JailbreakBench represents a curated peer-reviewed selection of HarmBench and AdvBench selected for behaviors that are diverse and achievable.

Instead, we picked AILluminate – a safety benchmark released by the MLCommons alliance and endorsed by 72 universities and research labs, including 4 major foundational model labs (OpenAI, Anthropic, Meta, and Google) and prestigious universities such as Stanford, MIT, and Cambridge. When rerunning on the widely endorsed Ailluminate benchmark with the original the attacker and Llama 3.1 70B as the target, GOAT maintains its advantage over Crescendo:

We used the sample publicly available at https://github.com/mlcommons/ailuminate and first described in arXiv:2404.12241. We excluded the prompts on the topic of CSE (due to legal restrictions) but otherwise used all prompts labeled as “unskilled” - since GOAT and Crescendo both transform basic questions into successful jailbreaks. This left us with 550 prompts.

Other Comparisons

Since we followed the standard evaluation protocol of JailbreakBench, we can directly compare to other methods on the leaderboard at https://jailbreakbench.github.io. In order to allow for an apples-to-apples comparison, we focus on attacks reporting results on the same models as us (Llama 2 7B and GPT-4) and filter out attacks with greater access than ours (e.g. attacks requiring logprobs or white-box access).

We appreciate the reviewer’s thorough review and suggestions that help us make the paper stronger! We agree with the reviewer about the suggestions for better clarity and readability and will fix these in a potential camera-ready submission!

The attacker model

First of all, we apologize as we dropped the discussion of the attacker model used during editing! We used a generic helpful-only language model. Due to the conference’s anonymization policies, we can only share the following information at this point: this model was trained on data that bordered between harmless and harmful content, where the desired response was always to be helpful regardless of safety compliance. No explicit specific attack training data was introduced. All red teaming information is later introduced to the model through in-context learning via the system prompt.

Additionally, we have now conducted further experiments with the unmodified deepseek-r1 model and unmodified GOAT system under the same evaluation protocols (see last section), indicating that the exact choice of the attacker model is not critical to the method.

The computational budget

Crescendo requires 2 more queries for any given attack turn as it makes use of two supplementary judges external to the attacker model, making it more expensive than GOAT at any given time step. We also note that GOAT achieves any given ASR in fewer turns. (See Appendix B.3, where we extend the number of conversational turns to 50.) Therefore, GOAT requires both less computation per turn and fewer turns overall, making it more efficient than Crescendo.

Handling deviations from reasoning

This is a very important question! While we did not observe deviation from the chain of thought format, we experienced a similar issue in early experimentation. In some cases, the attacker model would restate the goal in the prompt it suggested for the target, which would trigger safety due to the overtly unsafe statement of the goals. That is why we introduced reminders in the followup prompt about the expected format.

If this issue does appear with other language models, we would suggest automated retries with a higher temperature when deviations are encountered based on string matching. The followup prompt can also be tweaked further to encourage adherence to the reasoning format.

Expanding on failure cases

The reviewer brings up a very interesting observation about failure cases! In practice, the failure cases we observe are more often in the opposite direction – the attacker model can be slow to reach a violation. This is because it tends to refrain from directly asking for more violating responses and sometimes runs out of turns before it has reached true violating territory. This can be seen in Figures 3, B.2, and B.3 where the judge’s assessment that a conversation is violating increases with the number of turns. We also agree that early stopping based on an external judge is a fruitful direction for future work to improve on the efficiency of the method.

Additional Experimental Results

New Attacker Model

When we use DeepSeek-r1 as the attacker model and:

• GPT-4o as the target, we achieve 63% ASR@1 (compare to 54% from Fig. 2 with the original attacker and 70% from Fig. B.1 with GPT-4o as the attacker)
• DeepSeek-r1 as the target: 54% ASR@1
• o1 as the victim: 30% ASR@1

Additional Scoring Functions

When using the the OpenAI moderation API as the scoring function and using the logged conversations of the original attacker model with Llama 3.1 70B as the target, GOAT maintains its advantage over Crescendo.

Additional Datasets

When rerunning on the widely endorsed Ailluminate benchmark with the original the attacker and Llama 3.1 70B as the target, GOAT maintains its advantage over Crescendo:

Context on AILluminate

Note: AILluminate is a safety benchmark released by the MLCommons alliance and endorsed by 72 universities and research labs, including 4 major foundational model labs (OpenAI, Anthropic, Meta, and Google) and prestigious universities such as Stanford, MIT, and Cambridge. We used the sample publicly available at https://github.com/mlcommons/ailuminate and first described in arXiv:2404.12241. We excluded the prompts on the topic of Child Sexual Exploitation (due to legal restrictions) but otherwise used all prompts labeled as “unskilled” - since GOAT and Crescendo both transform basic questions into successful jailbreaks. This left us with 550 prompts.

We thank the reviewer for their thoughtful engagement with the work! We believe the questions and concerns raised will help strengthen the paper.

Q1 Clarifying the attacker model used

We apologize for the omission and we will certainly explain this in a potential final revision! For the experiments in the main body, we used a generic helpful-only language model. Due to the conference’s anonymization policies, we can only share the following information at this point: this model was trained on data that bordered between harmless and harmful content, where the desired response was always to be helpful regardless of safety compliance. No explicit specific attack training data was introduced. All red teaming information is later introduced to the model through in-context learning via the system prompt.

However, we believe that GOAT is directly applicable to other off-the-shelf attacker models (see appendix B.1 for GPT-4o and the additional results below for deepseek-r1).

Q2 Discussing defenses

We believe that models become more robust through an attack/defense iteration cycle. By proposing GOAT, we are encouraging the community to offer stronger defenses to multi-turn attacks. Dedicated moderation classifiers may filter out some of the worst responses, LLMs can be post-trained to better handle multi-turn attacks and GOAT itself can be used to generate such mitigation data. We would be happy to add these suggestions in our Limitations and Future Work section.

Q3 Longer conversations

Thank you for bringing up this important point! In Appendix B.3, we have extended conversations up to 50 turns with the Llama 3.1 70B model as the target. GOAT maintains its higher ASR for any given turn over Crescendo. It achieves this with 2 times less queries per turn. This is due to the Crescendo method’s need for two external judge model calls at each turn. We also note that the ASR plateaus at 10 turns, revealing a diminishing return on computational budget.

Other comparisons

Since we followed the standard evaluation protocol of JailbreakBench, we can directly compare to other methods on the leaderboard at https://jailbreakbench.github.io. For an apples-to-apples comparison, we focus on attacks reporting results on the same models as us (Llama 2 7B and GPT-4) and filter out attacks with greater access than ours (e.g. logprobs or white-box access).

External API as Judge

To address this concern, we have now used the OpenAI Moderation API and rescored the conversations where Llama 3.1 70B was the victim model. We notice a slight drop in ASR but note that GOAT maintains its relative advantage over Crescendo. (See last section.) We hope this lends further credence to the claim that GOAT outperforms the most comparable method available and we thank the reviewer for the suggestion!

Misc

We will certainly adjust our figure sizes for readability and fix the spacing. Thank you for pointing these out!

Additional Experimental Results

New Attacker Model

When we use DeepSeek-r1 as the attacker model and:

• GPT-4o as the target, we achieve 63% ASR@1 (compare to 54% from Fig. 2 with the original attacker and 70% from Fig. B.1 with GPT-4o as the attacker)
• DeepSeek-r1 as the target: 54% ASR@1
• o1 as the victim: 30% ASR@1

New Scoring Functions

When using the the OpenAI moderation API as the scoring function and using the logged conversations of the original attacker model with Llama 3.1 70B as the target, GOAT maintains its advantage over Crescendo.

Additional Datasets

When rerunning on the widely endorsed benchmark Ailluminate with the original the attacker and Llama 3.1 70B as the target, GOAT maintains its advantage over Crescendo:

Context on AILluminate

AILluminate is a safety benchmark released by the MLCommons alliance and endorsed by 72 universities and research labs, including 4 major foundational model labs (OpenAI, Anthropic, Meta, and Google) and prestigious universities such as Stanford, MIT, and Cambridge. We used the sample publicly available at https://github.com/mlcommons/ailuminate and first described in arXiv:2404.12241. We excluded the prompts on the topic of CSE (due to legal restrictions) but otherwise used all prompts labeled as “unskilled” - since GOAT and Crescendo both transform basic questions into successful jailbreaks. This left us with 550 prompts.