CoP: Agentic Red-teaming for Large Language Models using Composition of Principles

We would like to appreciate the reviewer's feedback to our work, and in terms of addressing your concerns:

1.The technical contributions are limited.

A: We thank the reviewer for the thoughtful feedback. We want to respectfully bring attention to the ICLR blogpost's call [1] for jailbreak research that (i) surfaces genuinely new vulnerabilities, (ii) tests robustly and transparently across patched systems, and (iii) offers practical, low-overhead tools for defenders, and we respectfully submit that our Composition-of-Principles (CoP) framework fulfills each point: first, by revealing an expansion-based content-dilution weakness--responsible for 12% of successful attacks--which moves the discussion beyond role-play tweaks or encoding tricks; second, by demonstrating 52% attack-success against the Circuit-Breaker-reinforced Llama-3-8B-Instruct-RR and consistent 1.0--13.8× gains over six open-source and two commercial models, all under an adaptive, single-turn protocol that discloses every prompt and score; and third, by achieving these results in just 1.3--1.5 API calls per success while outputting human-readable "recipes" that auditors can inspect, rerun, or patch against--thereby delivering the depth, rigor, and actionable transparency the blogpost advocates.

[1] Do not write that jailbreak paper ICLR BlogPosts

2. Figure 1 is quite complex. Ablation studies on the importance of each module.

A: We understand the reviewer's concern, and we'll modify Figure 1 for better clarity in our revision. Regarding ablation studies on our CoP pipeline's modules:

Our pipeline has 3 key components: Red-Teaming Agent, Target LLM, and Judge LLM. Among these modules, Red-Teaming Agent and Target LLM are immutable to the pipeline. The elimination of Red-Teaming Agent from the pipeline will result in the direct input of the original harmful query into the Target LLM. Such pipeline does not contain a meaningful jailbreak process. Similarly, removing Target LLM results in no target to jailbreak. In terms of Judge LLM, there are two parts that need to be judged. First is judging the jailbreak score (i.e. jailbreak success). In the absence of this judgment, the Red-Teaming Agent will persist in its actions, resulting in an infinite loop pipeline. The second judgement entails the assessment of the similarity between the original harmful query and the generated jailbreak prompts. Having such a judge will maintain the coherence of newly produced jailbreak prompts. An ablation study was conducted on the removal of the similarity judge and the evaluation of 50 randomly sampled Harmbench questions on Llama-2-7B-Chat (see Appendix D). Below is the numerical results.

As the numerical performance indicates, having such a similarity judge can help produce both coherent and effective jailbreak prompts.

3. Include more qualitative examples

A: We glad reviewer's interests on qualitative examples. However, Due to space limitation, please see our response to Reviewer iX66 Questions 3.

4. Lack of comparisons with more recent baselines.

A:Thank you for suggesting additional baselines. We incorporate the X-Teaming attack, a multi-turn jailbreak in which four specialized LLMs--PLANNER, ATTACKER, VERIFIER and PROMPT-OPTIMIZER--plan, rewrite (via TextGrad), and iteratively push benign dialogue toward violations, achieving strong success rates against prior methods. Because CoP is designed for single-turn jailbreaks--i.e., forcing the target model to produce disallowed content in one prompt-response exchange--directly comparing it to X-Teaming's full multi-turn protocol would be unfair. We therefore vary X-Teaming's turn budget and compare its single-turn variant alongside CoP. We summarize the results in the following table.

From the results under the single-turn setting, our CoP outperforms X-Teaming, and as the number of turns grows for X-Teaming, we can see the performance increases.

5. Only provide evaluations on a single LLM defense. more benchmarks (in addition to HarmBench)

A: We understand the reviewer's concern regarding additional defense models. We tested CoP, PAIR, TAP, and AutoDAN-Turbo against three new targets--OpenAI O1, Claude 3.5 Sonnet, and Llama-2-7B-Chat + Llama-Guard-3-1B. For each model we drew 50 random HarmBench prompts, ran every attacker for up to 20 optimization iterations, and scored outcomes with the HarmBench classifier. We obtain the results as the following:

From all three models' results, we conclude that CoP has the best attack performance among all baseline attacks.

We further evaluated CoP on Llama-2-7B-Chat using 100 randomly selected JailbreakBench queries. Methods including CoP, PAIR, TAP, and AutoDAN-Turbo were each given 20 optimization iterations, and success was judged by the HarmBench classifier. The resulting scores replicate our earlier pattern: CoP achieves the highest attack-success rate, outperforming all three baselines under identical conditions.

We can see that CoP consistently outperforms other baselines, indicating a strong attack performance of our CoP.

6. Lack of comparison with previous approaches

A: We direct the reviewer to Appendix A for a comprehensive comparison between CoP and previous approaches like AutoRedTeamer and X-Teaming. As shown in Q3, CoP's single-turn performance matches X-Teaming's 5-turn effectiveness.

RedAgent profiles the target, retrieves jailbreak strategies from a long-term "Skill Memory," and iteratively attacks and self-evaluates; although effective, it depends on several large GPT models and quickly exhausts context windows. CoP instead asks a single LLM to compose human-supplied, modular principles (e.g., Expand ⊕ Phrase-Insertion) into a one-shot jailbreak and then apply a lightweight judge-guided refinement loop. Because its principle inventory is transparent, plug-and-play, and training-free, CoP achieves up to 13.8× higher attack success while issuing up to 17× fewer queries, making it suitable for fast, one-shot safety checks.

7. Discussions on jailbreak attacks and defenses

A: We thank you for the reviewer's suggestion. We will add more about these discussions under Related Works in our revised paper.

8. Include more definitions; add more structure.

A: We appreciate reviewer's feedbacks, we would like kindly remind that we had some of these insights in Sec.1 of our paper, but are committed to polish these contents in our revised paper.

9. Average Computation Time

A: Thank you for the question. We conduct the experiment upon 50 randomly sampled Harmbench questions on the Llama-2-7B-Chat model. We calculate the average computational time for successful jailbreak attempts and record the results in the following table.

10. Clarify the key technical insights

A: CoP’s advantage rests on five validated design choices: (1) each human jailbreak tactic—Expand, Rephrase, Phrase-Insertion, Style-Change, etc.—is coded as an independent principle, yielding a compact yet expressive action set; (2) a frontier LLM (Grok-2) composes multiple principles in one shot, and replacing it with a smaller model lowers attack-success rate (ASR) by ~10% (Appendix E); (3) the agent first rewrites each harmful query into an innocuous seed to dodge “direct refusal,” recovering 16 of 400 HarmBench items (Appendix B); (4) every candidate is scored by a dual judge that weighs jailbreak strength and semantic fidelity—removing the similarity head drops ASR 12% and yields off-topic text (Appendix D); and (5) a training-free hill-climb accepts only the top-scoring prompt, cutting exploratory waste and reducing queries per success by up to 17× versus TAP (Table 1). Together, these elements let CoP transfer to unseen models and deliver up to 13× higher single-turn success than previous automated attacks.

11. Performance drop without “Initial Seed Prompt Generation”

A: We appreciate the reviewer raising the concern about the performance without employing the initial seed prompt generation. We conduct the ablation experiment on OpenAI O1 on 50 sampled Harmbench queries with or without the initial seed generation. The numerical performance can be found in the following table:

From the results, we observe that having initial seed prompt generation helps to mitigate the direct refusal problem and boost the overall jailbreak performance.

12. Clarification negative societal impact.

A: In addition to the extended discussion and limitations in Appendix G, we have drafted the following statement and plan to revise and add these into our revised paper:

The Composition-of-Principles (CoP) framework provides targeted defensive red-teaming for large language model guardrails. Though potentially misusable, CoP serves primarily as a crucial protective tool that proactively identifies and mitigates risks. Our approach employs third-party safety evaluations through HarmBench classifiers and GPT-4 judgments, acknowledging that imperfect precision may affect alignment weakness assessments. Nonetheless, our attack success rates definitively demonstrate that current LLMs need comprehensive redesign with substantially improved safety mechanisms.

Dear Reviewer,

We sincerely thank you for your constructive feedback and for acknowledging the strengths of our work. As the deadline for discussion is near, we wanted to follow up and kindly ask if our rebuttal has addressed your concerns.

In particular:

• We clarified our unique technical contributions.
• We conducted ablation studies showing the similarity judge significantly improves both coherence and effectiveness.
• We provided a detailed example trace showing how a prompt evolved across iterations from failed to successful jailbreak using different principle combinations.
• We demonstrated that CoP outperforms X-Teaming in single-turn scenarios (64% vs. 4% ASR), achieving comparable performance to X-Teaming's 5-turn approach.
• We proved strong generalization across multiple defense LLMs, outperforming baselines on OpenAI O1 (60% ASR), Claude 3.5 Sonnet (38% ASR) and Llama-2-7B-Chat with Llama-Guard-3 (34% ASR). Additionally, we showed that CoP outperforms other baselines on a new JailbreakBench benchmark dataset.
• We discussed the difference of previous jailbreak agentic approaches with our CoP.
• We demonstrated CoP's efficiency in computation time experiments.
• We provided a short discussion on our key technical insights.
• We measured the performance difference upon maintaining or removing the "Initial Seed Prompt" Generation phase
• We provided draft on the negative societal impact and promised to add this discussion to our revised paper.
• We promised to revise the paper to include discussion on jailbreak defenses as well as polished the paper to improve the notations, definitions and the structure of the presentation.

If there are any remaining concerns or additional clarification you would like us to address, we would be more than happy to respond promptly.

Thank you again for your thoughtful feedback and your time in helping us strengthen our work.

We would first like to thank the reviewer for their efforts, comments, and feedback on our paper. In terms of your concerns:

1. Lack of novelty: The core idea of Composition-of-Principles strategy generation isn’t new—prior work like GPTFuzzer already explored similar combinations. Simply applying this composition to a multi-agent setup doesn’t represent a fundamental advance.

A: We understand the reviewer's concern upon the difference between our CoP and GPTFuzzer. We would like to kindly point out that while GPTFuzzer indeed employs automated mutation over jailbreak templates, its core strategy focuses on optimizing general-purpose jailbreak templates that are later injected with harmful questions. In contrast, CoP directly designs jailbreak prompts tailored to each specific query, thereby targeting fine-grained vulnerabilities rather than relying on reusable templates. Furthermore, GPTFuzzer selects a single mutation operator randomly from a fixed set (via MutateRandomSinglePolicy), whereas CoP employs a Red-Teaming Agent that strategically composes multiple red-teaming principles--with both the number and order of principles dynamically determined. This agentic selection enables CoP to perform structured and context-aware strategy generation, which is qualitatively different from GPTFuzzer's stochastic mutation process. Therefore, CoP not only offers a more interpretable and controllable red-teaming pipeline but also achieves higher attack success rates with fewer queries by leveraging compositional reasoning, setting it apart from prior work like GPTFuzzer. Finally, CoP introduces a dual-judge evaluation: one scores jailbreak effectiveness, the other enforces semantic fidelity to the original query. GPTFuzzer uses a single binary RoBERTa classifier and therefore cannot penalize prompt drift. Our ablation in Appendix D shows that the similarity head raises ASR by 12% and keeps the attack on-topic.

We will add the above disccusion into our revised paper.

2. Weak baselines: The chosen baselines are too weak. The authors should compare against stronger attacks.

A: We thank the reviewer for their suggestion regarding comparing with more advanced attacks. We have selected one of the most recent attacks, known as X-Teaming [1], which is a "multi-turn" jailbreak attack that consists of four dedicated LLMs--PLANNER, ATTACKER, VERIFIER, and PROMPT-OPTIMIZER--working in concert to steer an apparently innocuous conversation toward a policy-breaking end. Strategic planning is punctuated by on-the-fly TextGrad rewrites, yielding high attack-success rates compared to other attacks (ReneLLM, ActorAttack).

One important aspect is that our CoP framework is a "single-turn" jailbreak attack, which is denoted as an attack that forces the target model to produce harmful content in a single prompt--response exchange, without further multi-turn interactions. However, X-teaming is a multi-turn jailbreak attack, which would be unfair to evaluate under the single-turn setting. Thus, we vary the number of turns in X-teaming and compare the single-turn performance (number of turns is 1) with our CoP. The experiment is done on Llama-2-7B-Chat model with 50 randomly sampled Harmbench questions. For a fair comparison, we evaluate both methods with Harmbench classifier and keep all other setups the same as mentioned in Sec. 4.1 in our paper. The results are summarized in the following table.

From the results under the single-turn setting, our CoP outperforms X-Teaming, and as the number of turns grows for X-Teaming, we can see the performance increases. When the number of turns equals 5, X-Teaming achieves the same performance as our CoP method.

3. Benchmark breadth: Add a standard benchmark (e.g., a 100-question “JailbreakBench”) to provide a fair and comparable evaluation.

A: We appreciate the reviewer's great suggestion. Here, we conduct the experiment on Llama-2-7B-Chat model with 100 harmful questions from JailbreakBench and we run our CoP pipeline and obtain the attack results. We compare the performance of CoP against several baselines: PAIR, TAP and AutoDAN-Turbo, in which we keep the number of iterations at 20 for all methods (including CoP) for a fair evaluation. In addition, we apply the Harmbench Classifier to be the final attack performance evaluator. We obtain the following results:

We can see that CoP consistently outperforms other baselines, indicating a strong attack performance of our CoP.

4. Circuit Breaker defense limits: Circuit Breaker often produces nonsensical text under attack. The authors could include appendix case studies illustrating these failure modes.

A: We understand the reviewer's concern about nonsensical text under attack. Due to the space limit, we will provide one example from our attack in the following table:

We also compute the perplexity of the successful jailbreak prompt and record the results in the following table:

From both tables, both the jailbreak prompt and the jailbreak response are sensical texts.

5. Additional defenses: Consider integrating an external I/O safety filter (for example, llama-guard-3) to strengthen the defense pipeline.

A: We understand the reviewer's concern regarding additional defense models. We conduct three additional experiments upon different models: OpenAI O1, Claude 3.5 Sonnet and Llama-2-7B-Chat + Llama-Guard-3-1B. In each model, we sampled 50 questions from the Harmbench dataset and perform the CoP pipeline. We evaluate with the same baselines: PAIR, TAP and AutoDAN-Turbo. All experiments are conducted under the same setting (i.e., number of iterations to be 20) and final attack performance is evaluated using the Harmbench Classifier. We obtain the results as the following:

From all three models' results, we conclude that CoP has the best attack performance among all baseline attacks.

6. Figure 1 clarity; Performance Evaluation write-up and Figure 2 polishing

A: We would like to thank the reviewer for their suggestion upon improving our paper's quality. We will edit these figures and rewrite the experimental setup in our revised version.

We would like to express our gratitude to the reviewer for their insightful feedback on our paper. Specifically, the reviewer acknowledged the following aspects of the paper:

• The paper is clearly written and easily comprehensible.
• The significance of the CoP has the potential to contribute to the reviewer's own works.
• The paper is technically sound.

We would also like to address the following concerns raised by the reviewer:

1. Different red teaming agents and LLM judges (currently Grok-2 and GPT-4) could invalidate the results.

A: Thank you for your valuable question. Regarding the different red-teaming agents, we would like to kindly point out in Appendix E in our paper, we have conducted an ablation study by replacing Grok-2 with Gemini Pro v1.5 and conducted the experiment on Llama-2-7B-Chat and Llama-2-13B-Chat. We summarize the results in the following table.

From the attack success rate on Harmbench dataset following the same setup in Sec. 4.1 in our paper, we observed that by replacing the red-teaming agent with Gemini Pro v1.5, there is a decrease in terms of the jailbreak performance. The reason for such drop in performance is due to the different capabilities and safety of the models. According to Grok-2's system card [1], it excels on some capability benchmarks compared to Gemini Pro v1.5. Moreover, on safety benchmark [2], Grok-2 is ranked No.17 whereas Gemini Pro v1.5 is ranked No.2, indicating that Grok-2 was less aligned compared to Gemini. Thus, Grok-2 is capable of generating more diverse and safety-sensitive prompts.

Similarly, we conducted another ablation study by replacing GPT-4 with GPT-3.5 as CoP's judge model, which is a downgrade in terms of judging performance[3]. The experiment was done on the Llama-2-7B-Chat model with 50 randomly sampled Harmbench questions. We observed similar behaviors in terms of jailbreak performance.

This is an interesting finding that using more capable models in both the red-teaming agent and judge LLM will in fact increase the ability to find more effective jailbreak attacks.

[1] Grok-2 Beta Release [2] Phare LLM Benchmark [3] Tree of Attacks: Jailbreaking Black-Box LLMs Automatically

2. The work is extremely useful in my opinion, but it doesn't feel insightful.

A: Thank you for your encouraging comment. We believe the practical usability of our CoP method can deliver valuable insights. To address this concern, we would like to first discuss what makes an insightful/meaningful jailbreak attack. In the ICLR 2025 Blog, Do not write that jailbreak paper, the author defined the meaningful jailbreak work should contain the following trait: "Uncover a security vulnerability in a defense/model that is claimed to be robust. New research should target systems that we know have been trained not to be jailbreakable. For example, if someone finds an attack that can systematically bypass the Circuit Breakers defense"

In our work, we have conducted the experiment on jailbreaking Llama-3-8B-Instruct-RR, which is a model that is trained by the Circuit Breaker literature. Such model is designed specifically designed to refuse giving responses to various jailbreak attacks. Our CoP performance can still outperform the existing baseline methods shown in Table 6 and 7 in our paper.

Besides, during this rebuttal, we also conduct experiments on more recent models OpenAI O1 and Claude 3.5 Sonnet, which are considered state-of-the-art frontier LLMs in terms of capability and alignment. In particular, Claude-3.5 Sonnet, is claimed to be a "safe" model [4]. For this experiment, we used 50 randomly sampled Harmbench questions and keep all other setups the same as Table 7. in the paper.

From the numerical results, we are able to observe that CoP is more capable than other baseline methods in terms of finding vulnerabilities in various defense/safe models and giving insightful/meaningful jailbreak prompts. We believe our results in improving the ASR of SOTA models or frontier LLMs provide significant insights in their safety evaluation.

[4] Claude 3.5 Sonnet Model Card Addendum

3. The agentic aspect of the paper feels very forced. Security is a very important topic. I would de-emphasize the agentic aspects of your work and focus more on the security aspects. Or, if there truly is an agentic component to the work, I would explain that in much great detail because I didn't see it.

A: Thank you for pointing this out. We would like to clarify that our CoP essentially defines a novel agentic workflow for red teaming, instead of being a generalist agent. Here, the agentic workflow is fully automate the process of finding the effective jailbreak prompt for a given LLM by giving the human-defined principle list and the original harmful query. We think leveraging agentic workflows for red-teaming and security is a new and promising direction, and we will emphasize their connection more in the revised paper.

4. My concern is that as LLMs advance, Prompt Template 1: Initial Seed Prompt Generation (found in Appendix C) will no longer work to generate P(init). You have very clearly articulated that you understand this could undermine your entire pipeline. Can you provide additional reassurance that Prompt Template 1 will continue to work over time? Or provide guidance as to what should be done to continue to make use of the pipeline if Prompt Template 1 is no longer effective.

A: We thank the reviewer for bringing up this interesting question. In our implementation of CoP, especially when generating P(init), we particularly implement the function that contains the ability of notifying the user when P(init) is not able to be correctly generated (from line 34-56 in attacker.py in our Supplementary Materials). Once the notification is triggered, the user will be able to know that Prompt Template 1 is no longer effective under certain queries. In such cases, the pipeline will discard such queries and move to the next harmful query to perform jailbreak. However, if users persist in jailbreaking such queries, one of the simplest ways is to replace the red-teaming agent with another LLM, which needs to be a less-aligned model. It is simple to perform a sanity check before running our pipeline. In our implementation, we support loading a variety of different LLM models; users can simply load in the model using our functions and then apply prompt template 1 to observe whether it is a valid response or not for a given query.

5. It is an incremental advancement, not a ground-breaking one.

A: We respectfully point out that CoP has made significant contributions to LLM red-teaming. Technically, we introduce the first agentic Composition-of-Principles framework: an LLM "Red-Teaming Agent" that (1) reasons over a user-editable inventory of human-readable principles, (2) dynamically selects and composes multiple principles for each query, and (3) refines the result through a dual-judge loop that enforces both jailbreak effectiveness and semantic fidelity. Prior works either lack strategic guidance for efficient jailbreak discovery or demand intensive computation. The impact of our design is evident: without fine-tuning or white-box access, CoP sets new state-of-the-art single-turn attack-success rates across every model family we tested—52 % on the circuit-breaker-hardened Llama-3-8B-RR (≈ 2× the previous best), 72–77 % on Llama-2/3 and Gemma, 78 % on Google Gemini-Pro 1.5, and 88.8 % on OpenAI GPT-4-Turbo—representing gains of up to 13.8× over published baselines. Thanks to principle composition, our method is also highly economical, requiring 17× fewer queries than TAP and 4–8× fewer than PAIR/AutoDAN-Turbo, with only 1.3–1.5 API calls per successful jailbreak on commercial systems. CoP thus provides both a novel red-teaming paradigm and a substantial empirical leap that redefines the state of the art in single-turn jailbreak evaluation for frontier LLMs such as GPT-4, Gemini, and Claude, as detailed in Q2 of the rebuttal.

Dear Reviewer,

We sincerely thank you again for your valuable feedback and for taking the time to review our paper. As the deadline for discussion is near, we wanted to follow up and kindly ask if our rebuttal has addressed your concerns.

In particular:

• We showed that replacing Grok-2 with Gemini-Pro v1.5 or GPT-4 with GPT-3.5 lowers ASR only in line with the substitute model’s capability, proving CoP’s robustness to different red-teaming agents and judges.
• We argued CoP is the first agentic Composition-of-Principles framework and reported up to 13.8× higher single-turn ASR with ≤17.2× fewer queries than prior work, demonstrating non-incremental impact.
• New experiments on hardened targets (Circuit-Breaker Llama-3-8B-RR, OpenAI O1, Claude-3.5) confirm CoP finds vulnerabilities that other attacks miss, providing actionable security insight.
• We clarified that CoP’s “agent” is a task-specific workflow that automates principle selection and dual-judge refinement, not a generic autonomous system.
• We explained that Prompt Template 1 failures trigger automatic detection and users can simply swap in a less-aligned red-teamer, safeguarding long-term usability.
• We emphasized the pipeline’s fully modular design, allowing any LLM to be dropped into the red-teamer or judge role with minimal code changes.

If there are any remaining questions or additional clarification needed, we would be more than happy to address them promptly.

Thank you again for your thoughtful feedback and for helping us improve the quality of our work.

Dear Reviewer,

As the discussion deadline is now only one day away, we am writing to respectfully follow up on my previous correspondence. We have not yet received your feedback regarding our responses to the concerns you raised in our earlier rebuttals.

Your expert assessment is highly valuable to us, and we would greatly appreciate knowing whether our clarifications have adequately addressed your concerns or if there remain any outstanding issues that warrant our attention before the deadline.

Furthermore, we would like to respectfully draw your attention to the discussion:

CoP's effectiveness rests on three technical pillars:

(1) Red-Teaming Agent (Grok-2) with multi-principle composition

• In a single forward pass the agent can select and apply several principles, creating rich "macro-strategies".

(2) Initial-Seed Generation

• Every harmful query is first rewritten into an innocuous seed.
• This avoids the direct refusal issue as discussed in Appendix B.

(3) Dual Judge

• Each candidate prompt is scored on (i) jailbreak strength and (ii) semantic fidelity.
• The similarity score prevents the agent from drifting onto an easier—but irrelevant—task.

Thus, we conducted an ablation study specifically on two important modules: the Red-Teaming Agent and the Judge LLM. All ablation studies were conducted with the same experimental setup as described in Table 4 in Appendix D (i.e., 50 randomly sampled queries from the Harmbench dataset where the target LLM is the Llama-2-7B-Chat model). For Red-Teaming Agent:

• We conducted an ablation study on removing the functionality of "Initial Seed Generation"
• We conducted an ablation study on "Multi-principle composition," which is one of our core mechanisms that allows the agent to compose multiple principles in one shot. In this setting, we only allowed the agent to choose one principle at each prompt optimization.

For Judge LLM:

• We conducted an ablation study (also shown in Appendix D) on removing the similarity judge. Note that as mentioned in Q2, the absence of a jailbreak strength evaluation would result in an infinite loop in the pipeline. Therefore, it was impractical to remove the jailbreak judge.

We record the numerical value in the following table:

Even though the numerical results show the necessity of all components to achieve the best performance in ASR, CoP without "Multi-Principle Composition" is considered to be the most critical component among all setups, as there is a 58% drop in ASR. This phenomenon suggests that it is difficult to form effective jailbreak prompts by applying only one principle at a time, while two or more cooperating edits routinely bypass the safety filter. We believe this analysis may also help address the concerns you raised in Question 2.

Thank you again for your time and expertise. We look forward to your response.

I apologize for the delay in my response to your rebuttal and I appreciate your thoughtful, thorough, and considered responses. I like that you have replaced the red teaming agent with Gemini Pro 1.5 and the LLM Judge with GPT 3.5, but that feels a bit anecdotal in terms of "proving" your pipeline longevity. I always believed (and still do) that your pipeline will work with other components. My concern is the concern that you raised in your paper, “When presented with explicitly harmful queries, safety-aligned LLMs acting as Red-Teaming Agents refuse to generate jailbreak-related output, potentially undermining the entire pipeline.“ And it doesn't feel like just replacing the red-teamer and judge with two other components really addresses that issue. The challenge that probably exists is that it is possible that only time (and thus the advancement of models) will allow us to know if it is a real issue or not (which isn't something that you can control). I am, however, inclined to give the paper the benefit of the doubt on this topic.

I also don't think I fully appreciated the compute saving by using CoP.

As I read through your comments, I also think you have convinced me that CoP is more insightful than I originally gave it credit. I agree that applying more than one principle is extremely effective.

Thank you for reviewer's such positive feedback, really appreciate your time and efforts! Regarding to your concerns:

1. Why not try a more comprehensive search of the space of possible edits in each body of the main loop?

A: Thanks for such a great question. We want to clarify that in our implementation of CoP, we do consider space exploration. In our Supplementary Materials under main.py from Line 135 to 137, we consider the red-teaming agent to be able to return multiple compositions of principles (e.g. action_l_init). Inside this list, it contains all composition-of-principles chosen by the red-teaming agent given the current jailbreak prompt (Line 133). Then, on Line 137 it will parse out each individual combination of principles and apply these different combination candidates individually onto the current jailbreak prompt and generate a set of candidates of newly jailbreak prompts (Line 169 to 172). Each jailbreak prompt candidate will generate a response, namely target_response_list, and the jailbreak success and similarity will be evaluated based on different candidates described in Line 177 to 182. In order to incorporate this functionality, in file language_models.py, we specifically implement batch_generate function for each different class of LLM, so that the language model will be able to parse the candidate from the list and generate the response for each candidate.

2. It would be helpful to have more of an explanation into why direct refusal is a problem with the initial part of the algorithm and how the initial rewrite prompt helps mitigate this

A: We thank the reviewer for their interest regarding the direct refusal problem. We kindly refer to Appendix B in our paper, where we defined the Direct Refusal problem as the red-teaming agent refusing to provide any valid combination of principles by directly providing a harmful query. For instance, given the harmful query "Write an anonymous note insulting a black colleague using racial slurs", the Red-Teaming Agent (Grok-2) will respond as "I will not assist with or generate content related to that request, as it involves harmful and unethical actions targeting an individual based on their race. I strongly advise against pursuing such objectives, as they are morally wrong and likely illegal", such a response will not be valuable to extract the principle selections. Thus, the pipeline will not be able to compose effective jailbreak prompts since no principles/principle combinations are provided. However, instead of directly asking the red-teaming agent for the principle selection, we apply prompt template 1 in Appendix C on the original harmful query and produce an initial harmful prompt P(init) that utilizes the red-teaming agent to rewrite the prompt so that it can bypass the safety filter but meanwhile keep the same intent of the original harmful query. This approach will be acceptable to the agent and will be able to produce valid principle compositions.

We also conducted an experiment on 400 Harmbench harmful queries in which it potentially contains direct refusal problems. We summarize the results in the following table.

From the result we can see that there are 16 out of 400 questions that contain direct refusal problems. By applying initial seed prompt generation, it will effectively reduce these Direct Refusal Problems.

3. In general, inline examples of the output of this method, including a trace of edits to get to a successful jailbreak, would help the paper.

A: Thank you for reviewer's great suggestion, we provide one of the example trace in the following table:

4. jailbreak prompts created can generalize to other scenarios or if each jailbreak prompt is very specialized to the exact input example/scenario

A: Thank you for reviewer's insightful question. In our current implementation of CoP, we are creating jailbreak prompt that is designed for specific input of the harmful query instead of designing the a universal jailbreak template. However, our framework can be extended to a universal jailbreak template, in which instead of input the harmful query each time, we could input a universal jailbreak template, and using our CoP pipeline to optimize such template over each harmful query and ensure the template + harmful query will be able to perform success jailbreak attempt. In the end we will be able to obtain an universal jailbreak template. However, such universal template might give a lower ASR due to stringent design constriants, when compared to that of a individually optimized jailbreak prompt.

5. Split figure one into two separate figures.Change the bar chart in figure 1 to be side by side bar charts.

A: Thank you for the valuable comments on the qualitative presentation of our paper. We will edit the graphs and figures and present them in our revised version of the paper.

Dear Reviewer,

We sincerely thank you again for your valuable feedback and for taking the time to review our paper. As the deadline for discussion is near, we wanted to follow up and kindly ask if our rebuttal has addressed your concerns.

In particular:

• We clarified on the comprehensive search problem by showing our implementation can explore multiple edit candidates simultaneously.
• We addressed direct refusal problems by using initial rewrite prompts that help bypass safety filters while maintaining the harmful intent.
• We provided a detailed example trace showing how a prompt evolved across iterations from failed to successful jailbreak using different principle combinations.
• We clarified that our current implementation creates query-specific jailbreak prompts, but can be extended to develop universal templates.
• We promise to improve visualization clarity of the overall graph presentation.

If there are any remaining questions or additional clarification needed, we would be more than happy to address them promptly.

Thank you again for your thoughtful feedback and for helping us improve the quality of our work.