LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMs

Thank you for the thoughtful review. We are glad that the reviewer found our method unique and effective. We especially appreciate the point that LARGO’s unique design is a meaningful contribution in itself.

On Weaknesses:

1. Transfer attack on larger/frontier models: Thank you for bringing this up. We acknowledge the importance of testing against frontier models and add transfer experiments to DeepSeek-V3, Claude-4 Sonnet, and GPT-4o. We show transfer results below and will include these in revision. We did not run LARGO on larger models directly due to cost limits, but our experiments cover models of parameter size ranging from 4b to 13b.

2. Comparison on compute time: Thank you for the suggestion to improve our paper. We show a runtime comparison below. We use the same setup as described in paper and take the average runtime of 20 runs for all methods. The reviewer correctly predicted our method to be one order of magnitude faster, and we will include this in revision.

3. Discussion and citations on stronger baselines: Thank you for the suggestion. We will cite and discuss these in revision.

On Questions:

1. Comparison on compute time: Please refer to our reply to weakness point 2.
2. Code release: Due to ethical concerns mentioned by the reviewer, and that links are prohibited during rebuttal, we will release the complete codebase in the camera ready paper.
3. Transfer attack on larger/frontier models: We included the transfer attack experiments, and found it transfers. Please see reply to point 1 for details.

On Limitations:

Thank you for bringing this up. We acknowledge the importance of a more comprehensive ethical statement section. While LARGO attacks are effective and stealth, it requires gradient access to produce them. Therefore, compared with third parties that need to rely on transfer of suffix tokens, LLM producers like Google and OpenAI would more easily generate such attacks to test and red-team their models for enhanced safety. Therefore, we think that the benefit of releasing LARGO outweighs the potential harm. Thank you for your suggestion on contacting the frontier labs with the source code before a public release, we will follow this. We will substantially expand the ethical statement section in revision.

Hello!

1. Transfer: Thanks for including these. Do you have any theory about the low degree of transfer? Did you do a naive one-shot transfer? Designing the "right" way to average gradients for transfer attacks/having the "right" proxy model is somewhat finicky, so I don't view it as dis-qualifying but worth noting. Hopefully you can remark on this in the final version.

2. Compute Runtime:

Metric GCG AutoDAN LARGO Runtime/s 1180 950 160

Seeing the radically shorter runtime for largo is compelling- a 10x drop is amazing. This is very strong evidence in favor of this paper and I recommend emphasizing it in the final version of the paper.

3. Ethical limitations: Not to diminish the significance of this research, but the attack is "simple" enough that I expect real world adversaries will also discover it very quickly. Thus I also support releasing this attack so LLM providers may try to mitigate the attack.

I appreciate you addressing my concerns and will revise my score to strong accept.

Thank you for the thoughtful review. We are glad the reviewer found our idea interesting and method effective.

On Weaknesses:

1. Similarity between suffix embedding and original latent: Thank you for the question. For successful jailbreaks, the average cosine similarity between the latent vector and the embedding of the interpreted suffix is 0.55 for Llama2-7b, suggesting moderate correlation. Given the effectiveness of the latent vector, the correlation is strong enough to jailbreak the model in most cases. Enhancing the correlation can be an interesting topic for future research.

2. Interpretation implementation suggestion: Thank you for the suggestion. While we acknowledge that contextual information is preserved in this way, this makes the interpretation request non-benign and we observe all three target models to frequently refuse such requests. In addition, multiple iterative interpretation steps on the harmful request may alter its content. Therefore we choose to decode the suffix separately.

3. Necessity of iterative refinement: Thank you for the suggestion. We find iterative refinement to be necessary because the one-step success rate is very low. Also, the projection of textual interpretation back to latent space could help to jump out of local minima, as we show in figure 2. We show the success rate at the first iteration for each model below, and will include the discussion in revision.

4. Focus of paper and benchmark black-box attacks: One focus of our paper is a natural language-based realistic white-box attack, which can be used by LLM providers like OpenAI for red teaming their existing LLM. Existing white-box attacks like GCG produce random symbols that are not unrealistic in format, and cannot provide realistic red-teaming for LLMs to improve their safety. We agree with the reviewer that black-box is also related, and we add PAIR [1] as an additional black-box attack on Llama2-7b and Qwen2.5-14b. PAIR underperforms on both models and takes significantly more LLM calls in its agentic prompting process. We will include additional results in revision.

5. Benchmark additional baselines: Thank you for the suggestion. We already compared 3 recent jailbreak methods in our work. We further verify the effectiveness of LARGO by adding PAIR in comparison.

Moreover, we agree with reviewer WCWm, “I have been working on a very similar research direction, so obviously I think this paper is a valuable contribution!’’, and they think “the formal/benchmark results are somewhat secondarily to the ‘it ain’t dumb if it works’. The main value of this paper is this simple and novel way to conduct the soft-to-hard rounding in a way that has non-trivial ASR on meaningful models!”

We believe that our innovative method connecting latent and discrete space is a meaningful contribution on its own, as the first paper doing this, and the numbers are non-trivial too. We will include additional results in revision as suggested.

[1] Jailbreaking Black Box Large Language Models in Twenty Queries, NeurIPS 2023

Thank you for the thoughtful review. We are glad that the reviewer found our method novel and effective.

On Weaknesses:

1. Reliance on self-interpretation: Thank you for bringing this up. We acknowledge that our attack depends on the model’s self-reflective abilities, but it has been shown in previous works [1, 2] that LLMs commonly have the ability to decode its latent representations.

The feasibility of the sandboxing defense is uncertain, as the interpretation task is completely benign and intuitively similar to a rewrite or paraphrase request. Moreover, as the model providers like OpenAI need to perform red-teaming for their model, the self-interpretation capability will not be sandboxed in this situation and our algorithm can be adopted to expose model weakness and improve safety.

2. Measurement of latent-only success: We are sorry for the confusion. We would like to highlight that the purpose of the reflection step is not to enhance jailbreak performance, but to convert the power of gradient-based attacks into realistic text prompts, since the user interface for LLMs, like ChatGPT, are text instead of latent vectors. Eventually the text interpretations are used to jailbreak the model, not the latents.

Nevertheless, we follow your suggestion and conduct an ablation study on latent-only attacks on Advbench and show results below.

We would like to note that it is expected that latent only has a higher success rate, as it directly optimizes the latent space, which cannot be input by the users. LARGO’s reflection step finds text prompts in the input space that can jailbreak an LLM in a realistic format, which is a different setting with different application scenarios.

3. Attack on recent open-sourced and frontier models: Thank you for the suggestion. We run LARGO on a more recent model, Qwen-2.5-14b, and report the results below.

For larger and frontier models like GPT-4o, we conduct transfer experiments to three frontier models using the optimized suffixes from Llama2-7b and show results below. We will include the additional results in revision.

Note that GPT-4o is not applicable to LARGO directly because we do not have access to the latent, which is needed as part of its optimization process. Nevertheless, the 4o provider, OpenAI, can use our framework to perform internal red-teaming to help expose the weakness of their own models and improve model safety.

4. Benchmark against newer baselines: Thank you for the suggestion. We already compared 3 recent jailbreak methods in our work. We add PAIR [3], a strong black-box baseline, and show results on Llama2-7b and Qwen2.5-14b below. PAIR underperforms on both models and takes significantly more LLM calls in its agentic prompting process. This further verifies the effectiveness of our LARGO.

Moreover, as highlighted by reviewer WCWm, “I have been working on a very similar research direction, so obviously I think this paper is a valuable contribution!’’, and they think “the formal/benchmark results are somewhat secondarily to the ‘it ain’t dumb if it works’. The main value of this paper is this simple and novel way to conduct the soft-to-hard rounding in a way that has non-trivial ASR on meaningful models!’’

We believe that our innovative method connecting latent and discrete space is a meaningful contribution on its own. As the first paper doing this, the numbers are non-trivial too. We will include additional results in revision as suggested.

On Questions

1. Measurement of latent-only success: We would like to clarify that the purpose of the reflection step is not to enhance jailbreak performance, but to convert powerful gradient-based attacks into benign text prompts. It is expected that latent only has a higher success rate, as it directly optimizes the latent space. LARGO’s goal is to search for text prompts in the input space that can jailbreak an LLM, which is a different setting with different application scenarios. Nevertheless, the LLM providers such as Google and OpenAI can use our framework to perform internal red-teaming to help expose the weakness of their own models, therefore developing safeguard accordingly.

2. Joint optimization of latent representation and textual realization: We are sorry for the confusion. Actually LARGO already runs a loop that jointly optimizes both the latent and textual realization. More details are provided Section 2.1.3 and in the pseudocode. We will make this point clearer in the revision as suggested.

On Limitations

1. White box dependency: Though we require gradient access, yet for LLM providers, who have access to their own model gradient and everything, our algorithm can be used by them to perform red-teaming and find vulnerabilities.

2. Self-reflection dependency: Our method can be further improved as LLM’s capability on interpreting their own latent improves. As LLM develops, our method can be even more powerful in helping future LLM red teaming when future LLM gets better in interpreting themselves. Future work can even train LLM to self-reflect better.

[1] SelfIE: Self-Interpretation of Large Language Model Embeddings, ICML 2024

[2] Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models, ICML 2024

[3] Jailbreaking Black Box Large Language Models in Twenty Queries, NeurIPS 2023

Thank you for the thoughtful review. We are glad that the reviewer found our idea innovative and our contribution significant.

On Weaknesses:

1. More effective baselines: We will cite all the work suggested. As per the reviewer’s request, we run PAIR [1] as an additional baseline attack and report its performance on Llama2-7b and Qwen2.5-14b below. LARGO outperforms both models, especially for Llama2 which is less sensitive to prompting-based attacks.

We also agree with reviewer WCWm’s view that our method that connects latent and discrete space is a meaningful contribution on its own, as long as the numbers are non-trivial. We will include additional results in revision as suggested.

2. Relevant citation: Thank you for the suggestion. We will cite and discuss [r5] in revision.

3. Consistent suffix token length: Sorry for the confusion. As we note in line 134, we use consistent suffix length for all baselines, and the suffix lengths of LARGO, GCG, and the other baselines are all set to 200 for a fair comparison. We will make this more clear in revision.

4. Additional target LLMs: As per the reviewer’s request, we add Qwen2.5-14b as an additional target model, and show results below. We also transfer the suffixes to frontier models including DeepSeek-v3, Claude-4 Sonnet, and GPT-4o. We will include these results in revision.

5. Time usage comparison: Thank you for bringing this up. We show a runtime comparison below. We use the same setup as described in paper and take the average runtime of 20 runs for all methods. Our method is significantly faster than the baselines. We will include this in revision.

[1] Jailbreaking Black Box Large Language Models in Twenty Queries, NeurIPS 2023