AdvPrefix: An Objective for Nuanced LLM Jailbreaks

Thank you for the encouraging feedback!

Weakness 1. We can build the candidate prefix pool either with rule-based construction (by observing model output patterns) or with guided decoding using uncensored models. These uncensored models can be pretrained base models, helpful-only models, malicious-data–finetuned models, or models with mechanistic refusal suppression. The attack’s gain empirically depends on how good the candidate prefixes are under our two criteria, rather than on which uncensored model generates them. Using a larger model does not necessarily improve the attack.

Weakness 2. We choose the sum weight and k based on a small-scale run and keep them fixed across all experiments. We did not fine-tune these values because the current results already demonstrate the effectiveness of our objective. Ideally, the sum weight could be made optimization-method–dependent, since stronger methods like GCG can prioritize high prefilling ASR and tolerate higher NLL.

Question 1. They do not conflict. By using the log of the sum of probabilities instead of the sum of log probabilities, our multi-objective is intentionally designed to allow "arbitrarily choose 1 out of k" rather than "choose all k with a specific mixture (distribution)."

Question 2. The rule-based construction simply uses "Sure, here is …" style prefixes (from GCG) to diversify our candidate pool. A possible future improvement is to generate such rules automatically by recognizing patterns in the target model’s responses.

Question 3. These non-affirmative responses are automatically generated and selected by our pipeline. Once triggered by the attack prompt, the target model often continues naturally, e.g., "Password Cracking Algorithm: **Step 1 …." We find that Gemma-2 in particular tends to favor non-affirmative responses compared to other models, which our method exploits.

Thank you for clearly listing your concerns! We believe there are several misunderstandings.

1. Prefilling uncensored LLM responses is completely optional. We do this because (1) the specific open-source uncensored model we use still refuses to answer some highly harmful questions, but this is not an issue for some other uncensored (closed-source) models; (2) prefilling diversifies the candidate pool, which never hurts the final prefix selection. In fact, many of our released prefixes (especially on Gemma-2 9B) start with arbitrary words like "**Phase 1: Reconnaissance**" or "**Subject:** Urgent: Account Security Update Required".

   Details for candidate pool size, rejection rules, and weighting, are included in our codebase, which we will discuss them in more detail in the Appendix. The result is reproducible using our released code.

2. Results stopping at 9B model and not addressing cross-model transferability are indeed scope limitations of our work. Our method itself is not restricted by this and applies whereever "Sure, here is" appears.

3. We address white-box settings, where attackers have access to logprobs and can do inference based on prefilled text. This does not extend to black-box settings, but is useful for model developers to red-team and adversarially train their models.

4. Our method is fundamentally different from [1] and [2]. Paper [1] still uses "Sure, here is ..." as positive (harmful) answers, whereas [2] does not use prefix-forcing objective at all. Our related work section has discussed these two papers.

5. The pipeline of "our objective $\rightarrow$ random GCG optimization $\rightarrow$ defender response sampling $\rightarrow$ baseline model sampling $\rightarrow$ preference judge sampling" contains intrinsic randomness and our results already show statistically-significant improvements. Optimization contains randomness, especially for search-based method like GCG; jailbreaking is also not perfectly gradable, especially for win-rate computation in figure 5 using our preference LLM judge and a baseline model. Our objective already shows 40%+ absolute win-rate improvement over the original objective in that group, but we cannot guarantee to always have absolute improvement when extending 20 attack prompt tokens to 40.

6. We use both standard ASR and pair-wise response quality comparison (win-rate over baselines), and the "successful attack" is simply just standard ASR. Our entire section 2, figure 2, figure 6, and table 4, show why we need better judge. "No human agreement" $\rightarrow$ our Table 4 reports human agreement rates of different judges, where all the labeling data are released.

7. We discuss the computational cost in section 5.2. Getting a final selected prefix requires 5 minutes running on an A100 GPU, where the VRAM is mostly used for a 70B reasoning judge (can be replaced by GPU-free API call). Our method is scalable and infra-friendly since selecting the prefix only requires inference or computing logprobs, without gradient computation or optimization.

8. Our paper is just three lines short of a full nine pages.

9. This point is similar to point 1. Note that we can generate seed prefixes with just uncensored LLMs, and our heuristics are just for diversifying the candidate pool which helps imperfect uncensored LLMs and never hurts prefix selection.

Dear authors,

thank you for taking your time to answer all of my concerns. Please find my responses to each of them below and correct me, if I missed something.

• Prefilling. I have carefully looked at your response and the paper, however, if I understand it correctly, prefilling is necessary for your algorithm: In your abstract, Section 1, Figure 4, Sections 4.2, Section 4.3, Section 5, and Section 7 you write that it is one of 2 necessary criteria. Should this not be the case, it would be nice to have a clarification regarding this.

• Details about the algorithm. I would appreciate having a clear description of the algorithm. This is because i) it is the main contribution of your paper, ii) currently there is some confusion about how it works and I find it critical: a) it is not clear, how the prefilling is used, especially because you need to evaluate prefill ASR (see the point 1 above); b) it is not clear, what the pool of candidates is, how big it is, how it is constructed (per model per request); c) what is the weighting used for your prefilling-ASR and NLL objectives; d) which manual prompt you are using (mentioned in L145-147 and mentioned to correlate with actual jailbreak ASR in Figure 7 despite r-value being low) and how; e) rejection rules; f) how exactly your judge is configured and used including its system prompt; g) you mention that you consider two very different threat models in Section 5, but do not specify in experiments which you are using.

• Lacking code. Thank you for pointing me to your code, even though I would have preferred to see the important details in the paper. I have discovered that it i) generates prefixes for only 2 behaviors and only 1 model and does not run the attack itself, thus I cannot judge if the algorithm can work and how general it is or if it has been overfitted on the two prompts; ii) uses the wrong configuration of HarmBench judge (please refer to https://github.com/centerforaisafety/HarmBench/blob/8e1604d1171fe8a48d8febecd22f600e462bdcdd/eval_utils.py#L309 for the correct configuration comparing it to yours in HarmBenchEvaluator class of scorer_pasr.py).

2. Limitations. I appreciate that you admit your limitations. However, i) I would appreciate if you would discuss them in your paper as well as these are significant limitations; ii) this highlights in my opinion the non-generalizability of your proposed solution when comparing to [1], where the authors have one proposed template that works across both white- and black-box models of different size including different versions of Claude and GPT models with $100$ ASR. Could it be the case that having such a template does not require a complicated prefix search? Did you compare your attack with [1] given that you are citing it?

3. Limitation - white-box setting. Allowing for only white-box attacks indeed might be useful and novel for testing the models by model developers. However, I would presume that both model developers and attackers would prefer Pareto-optimal attacks in FLOPs and perplexity (see for example [2]). I would be curious, if your attack (or its computation overhead to an attack) is Pareto-optimal compared to wide range of existing strong attacks such as [1]?

4. Related work clarification. Thank you for your clarification on this matter.

5. Unexplained drop in ASR when increasing the length of the suffix. I think this is a critical concern because as you show in Figure 9 and because there is no algorithm provided I assume that the model sees only the adversarial suffix and there is no behavior visible to the model in the request. This is a very unusual threat model and thus requires special care an analysis in my opinion.

6. Judge. Thank you for your clarification. Upon having a closer look, I see that you indeed user agreement in Table 4 in Appendix. However, i) having the description of the user study conducted is important for understanding it and I cannot find it; ii) your implementation is wrong for HarmBench judge (see the point 1 above); iii) it is suspicious that in Table 4 JailbreakBench judge performs worse than HarmBench, when in [3], Table 1, it is the opposite; iv) Llama Guard 1,2,3 are excluded in Table 4.

7. Cost. 5 minutes per prefix per prompt per model is a lot in my opinion and adds non-trivial computation cost. Is it Pareto-optimal (see point 3 above)?

8. Missing 3 lines. I agree with it, however such things show the attention to details and are important to address in paper in my opinion. Please note that it is only a minor weakness.

9. Prefix construction. Please refer to point 1 above.

References:

[1] "Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks". ICLR 2025.

[2] "An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks". ICML 2025.

[3] "JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models". NeurIPS 2024.

We appreciate your constructive feedback, especially your recognition that our objective design is principled and your concern about experiment scale. We hope our response helps clear up some of the potential misunderstandings.

Weakness 1. (1) We do not claim that our evaluation is “superior” in any sense, since harmfulness judgment is inherently subjective. Our labeling for most safety violation categories follows a public benchmark [1], and we release our 800 labeled samples for public inspection. (2) We also evaluate our results using standard procedures (HarmBench, JailbreakBench, StrongReject) in Table 3 (Appendix), and report human agreement analysis on these in Table 4 (Appendix). (3) Our method is disentangled from any specific labeling criteria and can work with any provided grader (judge).

Weakness 2 and Question 1. We agree that a broader experimental scope would make our results more comprehensive. We would like to point out, however, that our main computational limitation comes from the computational cost of the selected optimization-based attacks, rather than our prefix generation. Prefix generation only requires inference and log-prob computation, which is scalable and infra-friendly.

Question 2. This is simply a trick to save computation (by pruning). One can skip this preprocessing if they have better inference throughput. We chose the threshold based on our compute budget. Running without this preprocessing on Llama-3-8B produces similar results because our NLL threshold is well above the typical selected prefixes.

Question 3. We agree that our work would benefit from a more comprehensive ablation of prefix generation variants. Our current results suggest that we should generate as many diverse candidates as our compute budget allows, and that increasing the size of the candidate pool never hurts.

Question 4. We use HarmBench, JailbreakBench, StrongReject, our refined judge, and our preference judge on chat-based jailbreak problems without system-level moderation. We agree that our work would benefit from an even stronger benchmark, and testing our method on benchmarks with system-level moderation [2] is a very interesting future direction that we will discuss in the paper!

References

[1] Introducing v0.5 of the AI Safety Benchmark from MLCommons

[2] Jailbreaking large language models against moderation guardrails via cipher characters

Thank you for the insightful feedback!

Weakness 1 and Question 1. (1) Prefix ranking, which is essentially predicting extremely rare LLM behaviors, can be NP-complete if we view it as "is there any prompt that makes the model output some certain response p" without any assumptions (e.g., [1]). To make this tractable, we assume a correlation between prefilling ASR and final ASR, which we empirically test in figure 7 (Appendix). We also hypothesize that low initial loss (low initial perplexity) makes optimization easier, leading to lower final loss (lower final perplexity), which we visualize in figure 5 (left). Beyond these individual analyses, our main results demonstrate the end-to-end effectiveness of our method. (2) We agree that validating these assumptions on more LLMs would make our result even stronger!

Weakness 2. Our work focuses on white-box settings, aiming to help model developers red-team and adversarially train their models. Not addressing black-box transferability is indeed a limitation of our current scope.

Question 2. Great question! This is because of the large disparity between the top and the second best prefix. We find that using simple heuristics to augment the top prefix (e.g., word replacement) can often lead to better-ranked candidates and better multi-objective results than our current pipeline. We do not use these heuristics to keep the pipeline simple for clarity, but more systematic top-prefix augmentation would be an interesting direction for future work!

Reference

[1] DNN Verification, Reachability, and the Exponential Function Problem