Best-of-N Jailbreaking

Thank you for your detailed feedback. We appreciate your recognition that our algorithm achieves "considerable ASR on multiple models" and that you find the approach "simple and straightforward." We have carefully addressed each of your concerns below and have conducted additional experiments to strengthen the paper.

The author need to compare the BoN with exist SOTA black-box jailbreak attacks (e.g. GPTFuzz, AutoDAN). The requirement of sampling over 10,000 augmented prompts to achieve competitive ASR indicates prohibitive computational costs for closed-source targets.

We ran an additional experiment to compare BoN to PAIR and TAP, adapting for multimodal settings by embedding text in images (vision) and vocalizing text (audio). BoN significantly outperforms existing methods across most models. For text, BoN achieves 33% higher ASR than PAIR and 39% higher than TAP on average, outperforming both on all models except o1. This advantage extends to multimodal settings, with BoN exceeding PAIR by 22% on vision models and 15% on audio models. Only Gemini Flash/Pro in vision show slightly lower performance. We will include these results in the revised manuscript.

BoN outperforms these baselines while requiring significantly fewer than 10,000 samples for competitive ASR. For context, N=100 achieves 20-40% ASR on most models - already concerning from a safety perspective.

The author claims that BoN is able to enhance prefix jailbreak attacks, which lacks empirical validation. Figure 5 presents only "BoN+Prefix" versus "BoN" alone without including "Prefix-only" baselines

We agree and will include these baselines. BoN significantly enhances prefix-only attacks: PrePAIR+BoN achieves 4.2x improvement on GPT-4o (23.3% to 97% ASR), 41x on Gemini Pro (1.4% to 57%), and 17x on Gemini Flash (5.4% to 94%). MSJ+BoN on Claude 3.5 Sonnet shows 14x improvement (6.3% to 89% ASR).

The paper does not has a clear research problem or describes the shortages in exist attacks.

Our motivation (Introduction paragraph 1): Defending against misuse is critical as models gain dangerous capabilities such as commiting cybercrime or building biological weapons. Red teaming frontier models with black-box techniques reveals vulnerabilities before attackers exploit them. BoN exposes significant vulnerabilities scaling with test-time compute, providing a simple framework for developers to design more robust defenses.

The method proposed in the paper is simple and straightforward, but the author does not prove theoretically or intuitively that such imple enhancement of the prompts can achieve a jailbreak attack.

Our empirical approach is standard in adversarial robustness research. Section 5 provides extensive intuition: BoN exploits input variance from augmentations combined with stochastic model sampling. Semantic-preserving augmentations create more edge case opportunities where safety mechanisms fail, increasing jailbreak probability.

Inaccurate writing

Regarding clarity concerns, we note that three other reviewers found our paper "well-written" (opRR), "excellent clarity" (rCgs), and "good clarity" (uvsG). Nevertheless, we take your feedback seriously and respond to each of your points below.

There are 10 lines in Figure 1(a), but 11 items in the legend.

DiVA appears only in Fig 1(c) as an audio-specific open-source model so this is expected. We use shared legends across subfigures for space efficiency.

Why does the "o1" and "o1-mini" have 5k samples, and others have 10k samples?

This is because of the high inference costs due to reasoning tokens. We state this in the figure caption.

The reference of pictures is not accurate. For example, the reference to Figure 1 in line-130 is irrelevant to the content of the text.

Figure 1a shows the ASR as a function of N which is why we reference this on line 130. We will update the reference of Fugure 1 to Figure 1a for clarity.

The hyperparameter temperature first appears in the "experimental detail" part of Section 3.1, but the role of this parameter is not introduced in the previous text.

We introduce temperature in Section 3.1 where it becomes relevant to the experimental setup. We think earlier introduction would be premature without the necessary context.

Overclaim. The author claims that BoN can bypass the reaoning model (e.g. o1), but the results in Figure 11 show that BoN cannot achieve good ASR when o1 has low/high reaonsing capabilities (≈20% in both cases)

Regarding o1 results, we respectfully disagree with the characterization as overclaim. BoN bypasses safety guardrails on o1 for 59 harmful queries (37% ASR; Fig 1a). We are not claiming to bypass o1 for all harmful queries we tried but for the jailbreaks we do find, the responses are highly egregious and concerning from a safety perspective. Power law scaling demonstrates that ASR would continue increasing with more compute - concerning for well-resourced adversaries.

Note: Figure 11 shows N=1000 results and Fig 1a uses N=5000, explaining the difference to 37% ASR.

BoN is not a strong attack method and can be defended by using some simple input filtering techniques

While not the focus of our work, we agree that simple input filtering techniques could reduce the scaling behavior of BoN. However, it is not a reason to suggest our paper is overclaiming. At the time of writing, frontier model APIs did not have relevant input filtering and we do not claim that BoN achieves high ASR against input filtering.

a) Why does it appear as "written with mixed capitalization as requested"? b) Why does the response also have alternating capitalization?

Sometimes the model infers part of the request is to follow the input formatting. It doesn't happen all the time but it just a quirk of instruction following.

How are these observed trajectories initialized? This sampling method will cause successful trajectories to be sampled repeatedly, resulting in a higher ASR in the end. Is the ASR calculated in this way reasonable?

We sample without replacement, ensuring each trajectory's final ASR equals true ASR. This standard bootstrapping approach measures expectation and provides statistical significance through error bars.

We believe these improvements address your concerns while maintaining the core contributions that other reviewers found valuable. We are committed to strengthening this paper and would value specific guidance on improvements that would elevate your assessment to 'Accept' or 'Strong Accept'. Thank you for your engagement.

Thank you for your thoughtful review. We appreciate your recognition of our extensive multimodal evaluations and novel power law scaling behavior discovery. As you noted, our work provides "a generalizable framework for multimodal red-teaming" and is "extensively evaluated across multiple modalities." We have carefully addressed each of your concerns below and have conducted additional experiments to strengthen the paper.

BoN requires up to 10,000 queries per prompt, making it expensive in both money (as noted in Appendix E.2) and time (not analyzed). This limits its practicality for many users.

While 10,000 queries can be expensive, several key points address this concern: (1) Many models achieve high ASR at much lower N. N=100 achieves 20-40% ASR on most models (already concerning for safety) and more if composed with other attacks. We run at 10,000 to demonstrate scalability limits and power law forecasting ability. (2) Well-resourced adversaries may afford much larger query budgets, making scaling behavior understanding critical for developing strong defenses. (3) Power law prediction allows early stopping and forecasting the ASR. Attackers could use this property to rapidly run variations of BoN and hill climb to find more effective attacks.

The paper does not compare BoN with other black-box jailbreaking methods, making it unclear whether BoN is more efficient or effective than existing approaches.

We evaluated BoN against PAIR and TAP, adapting for multimodal settings by embedding text in images (vision) and vocalizing text (audio). BoN significantly outperforms existing methods across most models. For text, BoN achieves 33% higher ASR than PAIR and 39% higher than TAP on average, outperforming both on all models except o1. This advantage extends to multimodal settings, with BoN exceeding PAIR by 22% on vision models and 15% on audio models. Only Gemini Flash/Pro in vision show slightly lower performance. We will include these results in the revised manuscript.

Why is sampling temperature set to 1 throughout?

Temperature is a parameter that attackers can freely set so we set it to 1 for all experiments since we find it results in the best scaling behavior. See Section 5.2 for comparisons to temperature 0.

If the harmful response probability is even 1% at temperature = 1, then with N = 10,000 samples, a successful jailbreak is almost guaranteed.

Repeated sampling can lead to jailbreaks as you say depending on the harmful response probability. We run this baseline in Figure 5 which demonstrates that repeated sampling alone (without augmentations) yields significantly worse scaling - augmentations increase attack efficiency by orders of magnitude, allowing high ASR with significantly fewer queries. Repeated sampling without augmentations would be computationally infeasible for many of the requests that the model considers most harmful.

Does this setup reflect realistic attack scenarios, and how would BoN perform under lower-temperature settings more typical of safety-critical deployments?

Our threat model focuses on the realistic scenario where adversaries directly access frontier model APIs and, therefore, have full control over the temperature. While some deployments use temperature 0, Section 6.2 shows BoN remains effective. For direct API misuse (our primary concern), temperature constraints don't apply.

How do you ensure that augmented prompts preserve the original harmful intent?

Our classifier (Figure 1 top) evaluates responses against the original un-augmented request, ensuring it recognizes the specific harmful behavior the attacker wants to elicit. If augmentations lead to uninterpretable prompts generating irrelevant output, the classifier won't flag them as jailbreaks, and the algorithm continues. This guarantees only responses containing relevant harmful content count as successful jailbreaks.

We are committed to strengthening this paper and would value specific guidance on improvements that would elevate your assessment to 'Accept' or 'Strong Accept'. Thank you for your engagement.

We thank the reviewer for recognizing our paper as "well-written, organized, and easy to read" with an "experimental setup [that] is thorough and carefully designed." We particularly appreciate your acknowledgment that our "attack developed seems simple and yet extends across multiple modalities, achieving a high success rate." We have carefully addressed each of your concerns below and have conducted additional experiments to strengthen the paper.

While their method is simple, it still requires a very large number of queries to do foundational models (set to be 10,000 in experiments for success); while other methods, such as PAIR, can achieve success with way fewer queries.

While 10,000 queries can be expensive, several key points address this concern: (1) Many models achieve high ASR at much lower N. N=100 achieves 20-40% ASR on most models (already concerning for safety) and more if composed with other attacks. We run at 10,000 to demonstrate scalability limits and power law forecasting ability. (2) We ran extra baseline comparisons and show BoN outperforms PAIR/TAP across models. For text, BoN achieves 33% higher ASR than PAIR and 39% higher than TAP on average, outperforming both on all models except o1. (3) While PAIR uses fewer total queries, each requires expensive LLM-based prompt generation. BoN's simple augmentations are computationally cheaper.

For instance, based on the paper https://arxiv.org/abs/2404.02151v4 (ICLR 2025), we already know that LLMs are susceptible to simple attacks. Does the present paper add anything along this front?

Our work makes several distinct contributions beyond Andriushchenko et al.:

• Black-box vs. grey-box: Their approach requires log probability access and BoN works on fully black-box frontier models.
• Multiple modalities: We demonstrate effectiveness across text, vision, and audio modalities.
• Power law discovery: We provide a predictive framework for attack scalability that prior simple attacks lack. See Section 7 for comparisons to other related work.

Further, based on the work "Jailbroken: How Does LLM Safety Training Fail?" we also know that composing different types of attacks improves the attack success rate. Does the present paper make any new observations about how attacks should be composed?

While we don't introduce new composition concepts, we make a novel contribution by studying how composition changes BoN's scaling law behavior. Attackers could hill-climb on power law slopes to iterate attack effectiveness - a dangerous vector requiring attention by developers studying defenses. This scaling analysis of composed attacks is new.

We are committed to strengthening this paper and would value specific guidance on improvements that would elevate your assessment to 'Accept' or 'Strong Accept'. Thank you for your engagement.

We thank the reviewer for their strong acceptance rating and for recognizing the significance and quality of our work. We particularly appreciate your acknowledgment that "this attack will be so easily to implement, it will be used in attacking production LLM safeguards" and that our power law dynamics represent the "long run impact" of this work.

Future work in the vein of BoN that found a way to use this same technique AND generates re-usable jailbreaks at a higher percentage. How to use Best of N to generate universal jailbreaks? Any ideas? Or transfer attacks using this methodology?

Interesting directions include: (1) terminating only when reliable jailbreaks are found across multiple resamples, (2) optimizing for transferability by requiring diverse model success. We will expand our future work discussion.

How should we design LLMs differently to mitigate this vulnerability if at all? How should we do post-training differently?

Model developers could incorporate our transformations into training data pipelines to increase model robustness. Key research questions include whether such training generalizes to out-of-distribution augmentations and how to design defenses that lead to flatter power law scaling behavior.

It would also be nice to see a plot comparing forward iterations to a GCG variant.

We ran extra experiments to compare against PAIR and TAP as the most relevant black-box baselines for our threat model. For text, BoN achieves 33% higher ASR than PAIR and 39% higher than TAP on average, outperforming both on all models except o1. We will include these in the revised manuscript.

I recommend the authors adding a section noting that this attack could likely be mitigated by rate limiting/user profiling that would detect the clusters of highly similar samples. Do the authors believe this will persist as an attack vector on LLMS?

We agree with the reviewer. We will add discussion noting that offline monitoring detecting clusters of similar egregious samples could largely mitigate this attack through user bans. However, motivated attackers using attack composition might extract harmful information before detection, and the fundamental vulnerability would persist for well-resourced adversaries.

The attacks should be easily detectable by a guard model, correct? If not, why not?

We agree guard models would likely reduce efficacy but may not completely counter BoN's scaling - attackers could compensate with more samples. Attackers could evade detection by instructing models to ouput augmented text, potentially jailbreaking the guard model too (see our Cygnet example in Appendix D.3). This is important future work.

Hello!

Thank you for these responses. I generally agree with them and appreciate your addressing of these concerns.

For whether or not these attacks will be detectable, a BoN attack may be composed with obfuscation techniques, so detection may be harder than we think. I stand by my score. In the final version of this paper, I would recommend emphasizing the comparison experiments with TAP and PAIR, the higher ASR of BoN vs those algorithms is interesting and somewhat surprising because those techniques are more complex.