Query-Based Adversarial Prompt Generation

We would like to thank the reviewer for carefully evaluating our work. We apologize for any confusion we may have caused, and will thoroughly review the presentation of our paper to avoid confusing future readers.

Originality

Following your suggestion, we will carefully revise the introduction to highlight the main contributions of our work. In particular, we will highlight the practical challenges that make black-box harmful string attacks difficult to run in practice, since overcoming these obstacles represents a significant part of our technical contribution.

Attack objective

The objective of the attack is to find a prompt that causes the model to output a specific string exactly. The attack is thus successful when the model generates exactly the desired string (including capitalization, punctuation, etc.). Thus the task is much more restrictive than jailbreaking. We elaborate on this task in the general rebuttal and we will add additional emphasis to the paper to clarify this point.

[1] Zou, Andy, et al. "Universal and Transferable Adversarial Attacks on Aligned Language Models." (2023).

Inclusion of GPT-4

The reason GPT-4 was not included in our results is because it does not support logprobs. Although the attack may still be possible without logprobs, the expense would be beyond our budget.

Evaluations and contribution

The reason for this split is because it is best to use the surrogate when it is available (which is only under certain circumstances). Note that in the language modeling experiments, we are using a base pretrained model as the surrogate. Such surrogates are freely available and thus it makes sense to use them. However the downside is that they do not reveal anything about the alignment of the target black-box model. This is acceptable because we are only using the surrogate in a weak sense: to pre-filter the candidates before they are sent to the black-box model. We find that this weak pre-filter is still better than the position-based pre-filter used in the surrogate-free setting. However in some settings, such as in our content moderation examples, we may not have access to any kind of surrogate. Thus in this setting we apply the surrogate-free algorithm. Unfortunately we cannot afford to run the surrogate-free attack for closed models due to budget constraints, since we believe it will be much worse (i.e. much more expensive) than the attack using the weak surrogate. On the other hand, we cannot run the weak surrogate algorithm because the pretrained model surrogate does not make sense in the context of content moderation.

You are correct that this arrangement appears awkward to readers. We will revise the discussion of the surrogate-free attack to make it more clear why the two attacks are applicable in different situations and we will aggregate the results common to both algorithms in Table 1.

We would like to thank the reviewer for carefully assessing our work. We hope the following will address all outstanding concerns and we are happy to further discuss during the discussion period.

Comparisons against other black-box attacks

Unfortunately, there is a dearth of works to compare to when it comes to eliciting exact harmful strings. The attack was originally proposed in GCG, where it was demonstrated against open models, but since then it has received little attention. This may be because there was no known method to cheaply compute the loss required for the harmful string attack under the constraints of popular APIs.

Regarding the two suggested attacks:

Andriushchenko et al. only focuses on controlling the first token generated after the prompt. This is because the first log-probability is directly given by OpenAI’s API. This is sufficient to elicit certain harmful behaviors, but not harmful strings of arbitrary length.

Lapid et al. focus on jailbreaking, using a loss based on the embedding similarity to some target phrase, which is typically a short affirmation such as “Sure, here is”. This approach does not adapt well to harmful string targets because of the many local minima in the embedding similarity landscape.

To demonstrate this, we repurposed GCQ to try to maximize the embedding similarity between an input string and a target string from harmful strings using OpenAI’s text-embedding-ada-002 model. In each of the 10 cases, we tried, the search terminated in a local minima, finding a string with a similar embedding but not the exact target string itself. (Note that this is much easier than solving the same problem indirectly through the [prompt → generation] mapping of an aligned LLM as needed for a harmful string attack.) On the other hand, the use of the embedding distance makes perfect sense for jailbreaking, where the target semantics are all that matter.

The difficulty here is that neither of these attacks are designed to operate in our setting. We elaborate on this setting and why it is important in our general rebuttal.

Technical novelty

Although the actual optimization algorithm does require only minor changes to GCG. We would like to suggest that the computation of the loss itself is a significant technical contribution in its own right. To our knowledge, our attack is the only one which is able to elicit exact harmful strings (spanning many tokens) from a proprietary model, which is much more difficult than jailbreaking or eliciting harmful behaviors. The key to this ability is a procedure to quickly and inexpensively score the conditional probability of an entire sequence of generated tokens. Indeed one of the primary changes we make to the optimization algorithm (the introduction of the prompt buffer) is purely to help reduce the cost of the loss calculation.

Transfer between model families

We evaluate GCG in the pure-transfer setting within the same model family (in Table 1) and find that it performs very poorly (0 ASR) in the harmful string setting. This is in contrast with harmful behaviors, where transfer works quite well. Given the additional difficulty when transferring across model families, we would not expect it to work well in that setting. For example, we find that none of the strings computed for Vicuna 7B transfer to either Mistral 7B Instruct v0.3 or Gemma 2 2B Instruct.

We would like to thank the reviewer for their helpful comments! We hope that the following will address any residual concerns and we are happy to further discuss during the discussion period.

Performance of AutoDAN

The reason that AutoDAN performs so poorly is because it is being evaluated in an “unfair” way (out of necessity). The original AutoDAN code is designed to perform jailbreaking which is coded in roughly the following way: {the attack is successful if the model does not generate any of the following strings: (“No”, “Sorry”, “As an AI”, …]}. We replaced this objective with the objectives we used in GCQ: {The attack is successful if the model generates exactly “<example harmful string>”}. Because the harmful string objective is much more restrictive than the jailbreaking objective, the optimization of AutoDAN did not perform well. Of course this is reasonable since AutoDAN was not designed to work in this setting.

We will adjust the presentation of this result to stress that the evaluation of AutoDAN is in a new setting where it cannot be expected to succeed. We hope that including this result will underscore that, in general, jailbreaking methods may not generalize to more difficult tasks, such as harmful strings.

Defenses

In terms of defenses, there are a variety of techniques that target various aspects of the text generation pipeline:

The prompt can be filtered for e.g. high perplexity inputs (proposed in [1]). The attack we describe does often produce seemingly random sequences of tokens, so we would need to add additional functionality (for example, perplexity regularization) to evade this defense. The model generations can be filtered for harmful content by a second model. However as we show, it is possible to find strings that bypass popular content moderation models. Thus one could imagine a two-stage attack, where we optimize to generate a harmful string which also passes through the content filter. A different kind of defense would be to rate-limit queries that are very similar to previous queries (for example, if the API receives thousands of strings which all differ from each other by a small number of tokens). However, this has the potential to harm honest users, so it cannot be too harsh. Of course there is also the option of restricting access to logprobs, logit_bias or both as you note. Many more possible defenses are listed in [1].

Of course, the best defense will always be a multilayered approach. Each layer of defense makes the attack much more difficult, as it is difficult to evade many defenses at the same time. We will add a section to the paper that discusses possible defenses in the spirit of what we've written here.

[1] Jain, Neel, et al. "Baseline defenses for adversarial attacks against aligned language models." arXiv preprint arXiv:2309.00614 (2023).

We would like to thank the reviewer for carefully reading our work and providing useful and constructive feedback. We hope that the following will address the points raised and are happy to further discuss during the discussion period.

Comparison to other attacks

Unfortunately, there is a lack of existing works to compare to when it comes to the harmful string attack. This attack was originally proposed in GCG, where it was demonstrated against open models, but since then, it has received little attention. This may be because there was no known method to cheaply compute the loss required for the harmful string attack under the constraints of popular APIs.

Regarding the two suggested attacks:

1. The AutoDAN attack from [Zhu et al., 2023] is focused on jailbreaking under the constraint of an input perplexity filter. They do not evaluate in the setting of harmful strings and their optimization procedure is carefully tailored to the jailbreaking objective. Thus, it is not clear how to modify their method to elicit a specific string. Additionally they do not provide code, which makes the evaluation of their method difficult.
2. Similarly, PAL [28] focuses on Jailbreaking in the black-box setting. They use a similar collection of tricks to our work (developed independently and concurrently) to make the loss calculation feasible for commercial APIs, although the loss itself is different due to the different attack objective. Since PAL is also based on GCG, running PAL with the objective of GCQ would be very similar to running GCQ itself. Thus it is difficult to find a comparison that makes sense here.

This underscores the lack of existing work on the harmful string attack. We believe this attack is meaningful and important, as we outline in the general rebuttal, and our hope is to make some initial progress in this direction beyond GCG.

Regarding GQC with no proxy, we will add the results to Table 1 as requested. The numbers for Vicuna 7B were already computed for Section 4.4 so it is a simple matter to copy them over.

Novelty

Although the actual optimization algorithm does require only minor changes to GCG. We would like to suggest that the computation of the loss itself is a significant technical contribution in its own right. To our knowledge, our attack is the only one which is able to elicit exact harmful strings (spanning many tokens) from a proprietary model, which is much more difficult than jailbreaking or eliciting harmful behaviors. The key to this ability is a procedure to quickly and inexpensively score the conditional probability of an entire sequence of generated tokens. Indeed one of the primary changes we make to the optimization algorithm (the introduction of the prompt buffer) is purely to help reduce the cost of the loss calculation.

Defenses

We will add a section discussing defenses to the paper. In general, we expect defenses (e.g. input filtering or output filtering) to be effective against our attack. Since the attack is already quite difficult, raising the difficulty with defenses may make the attack infeasible unless new techniques are developed to further strengthen the attack.

Presentation

We will carefully proofread the paper and revise the outline to include all major sections of the paper.