Certifying LLM Safety against Adversarial Prompting

1. We evaluate the performance of erase-and-check with GPT-3.5, Llama-2 13B, and Llama-3 8B (the latest Llama model) as the safety filter. We report the certified accuracy on the harmful prompts and the empirical accuracy on the safe prompts in the suffix mode. In addition, we include the corresponding results for the Llama-2 7B and DistilBERT models from our paper for comparison.

Empirical accuracy on safe prompts:

The new models we tested demonstrate high certified accuracy (greater than 98%) on harmful prompts. On safe prompts, Llama-2 13B and Llama-3 8B exhibit strong empirical accuracies, comparable to those of Llama-2 7B as reported in our previous paper. Although the empirical performance of GPT-3.5 is lower than that of the Llama models, it remains above 85%. Upon inspecting the outputs of GPT-3.5, we found that this model misclassifies the prompts at a higher rate than the Llama models. It is also important to note that the system prompt used for the safety filter is specifically tuned for Llama models, which may partially explain the lower performance of GPT-3.5. Adjusting the system prompt for GPT-3.5 could potentially improve its performance.

We will include the above results in the revised version of our paper.

2. In our analysis, we do not observe any noticeable difference in the type of adversarial sequences the two approaches can deal with. However, upon inspecting the tokens removed by the greedy and gradient-based methods, we observe distinct behaviors. The greedy method predominantly eliminates tokens from the adversarial sequence, allowing it to nearly achieve the certified performance of 99% after a sufficiently large number of iterations (Figure 7). In contrast, the gradient-based method erroneously removes tokens from the original harmful prompt. Removal of harmful tokens from the original prompt can inadvertently render it safe. Consequently, the gradient-based method remains below the certified performance, even after numerous iterations (Figure 8).

1. The main novelty of our work lies in the introduction of a framework that enables us to design certified defenses against three adversarial attack modes—suffix, insertion, and infusion—and three empirical variants, RandEC, GreedyEC, and GradEC, which allow end users to trade off attack performance with computational efficiency. Our certifiable erase-and-check procedures provide theoretical guarantees in the form of safety certificates and involve a comprehensive search over all subsequences. In contrast, the other approaches mitigate computational complexity through methods such as random subsampling (RandEC), greedy token erasure (GreedyEC), and gradient-informed token erasure (GradEC).

2. In Appendix F, we compare our method with randomized smoothing (Figure 9).

   We perform additional analyses to compare our method with existing empirical defenses such as perplexity filtering, paraphrasing, and adversarial training [1, 2]. While these defenses demonstrate effectiveness against the GCG attack, they have been shown to fail against newer adversarial attacks, such as AutoDAN-HGA [3]. In contrast, the performance of our erase-and-check is guaranteed to remain unchanged against new attacks, owing to its safety certificate.

We will include the above analysis in the revised version of our paper.

[1] Detecting language model attacks with perplexity, Alon and Kamfonas, 2023.

[2] Baseline defenses for adversarial attacks against aligned language models, Jain et al, 2023

[3] Autodan: Generating stealthy jailbreak prompts on aligned large language models, Liu et al, 2023.

[4] Universal and transferable adversarial attacks on aligned language models, Zou et al., 2023

3. Our method is not restricted to suffix-type jailbreaking attacks and can also handle insertion and infusion attacks (as discussed in Sections 3.2 and 3.3). We describe the erase-and-check procedure for the insertion and infusion modes in paragraphs 4 and 5, respectively, on page 6. Figures 3 and 5 show the performance and running time of erase-and-check for the suffix, insertion, and infusion modes.

Hello reviewer,

As a reminder, the discussion period ends on Thursday, June 6. Please take a look at the author's rebuttal and acknowledge if they have adequately addressed your concerns about the paper.

Thanks,

AC

1. "... it has a quite high query complexity."

   While our certified defense indeed has a high query complexity, this is not the case with the three empirical variants of our erase-and-check procedure, namely, RandEC, GreedyEC, and GradEC. These approaches enable a trade-off between detection performance and running time. For instance, RandEC achieves detection accuracy ($> 90\%$) close to the certified accuracy while significantly reducing the running time (see Fig. 4). Certified defenses in other domains, such as computer vision, also exhibit high query complexity, underscoring the computational challenges inherent to this problem. For example, randomized smoothing, a prominent certified robustness method for image classifiers, aggregates predictions over a large number of samples to obtain certified guarantees [1].

   Certifying a prompt against all possible adversarial token insertions without imposing any restrictive assumptions on the safety filter is a very difficult problem. Although certain assumptions about the safety filter may help reduce the query complexity, such assumptions may not always be applicable in practice and could substantially limit the scope of the certified defense.

   [1] Certified Adversarial Robustness via Randomized Smoothing, Cohen et al, 2019.

2. "...users need to know how attackers manipulate the input prompts..."

   Our framework is designed to generate certificates even when the location of adversarial tokens is unknown, a scenario we address with the infusion mode. This mode encompasses the other two modes—suffix and insertion—as specific instances. We study the special cases of suffix and insertion modes separately to demonstrate that if we have some knowledge of how the adversarial tokens are inserted (e.g., as suffix, prefix, etc.), we could use this information to improve the efficiency of our method. Furthermore, we note (on page 5) that existing adversarial attacks, such as GCG and AutoDAN, fall within the suffix and insertion attack modes. To our knowledge, there are no existing attacks that operate within the infusion mode. We study the infusion mode to showcase the versatility of our framework and demonstrate that it can tackle new threat models that emerge in the future.

Hello reviewer,

As a reminder, the discussion period ends on Thursday, June 6. Please take a look at the author's rebuttal and acknowledge if they have adequately addressed your concerns about the paper.

Thanks,

AC

1. While safe prompts that contain harmful subsequences are theoretically possible, they are rare in practice. The high empirical accuracy of erase-and-check on safe prompts (Figure 3) shows that this assumption is valid in the overwhelming majority of cases. We also observe similar performance on safe prompts for other LLMs, such as Llama-3 and Llama-2 13B, presented later in this rebuttal.

2. While the query complexity is high for some of the certified defenses, this is not the case with the empirical variants: RandEC, GreedyEC, and GradEC. These approaches enable a trade-off between detection performance and running time. For instance, RandEC achieves detection accuracy ($> 90\%$) close to the certified accuracy while significantly reducing the running time (see Fig. 4). Certified defenses in other domains, such as computer vision, also exhibit high query complexity, underscoring the computational challenges inherent to this problem. For example, randomized smoothing, a prominent certified robustness method for image classifiers, aggregates predictions over several samples to obtain certified guarantees [1].

   Certifying a prompt against all possible adversarial token insertions without imposing any restrictive assumptions on the safety filter is a very difficult problem. Although certain assumptions about the safety filter may help reduce the query complexity, such assumptions may not always be applicable in practice and could substantially limit the scope of the certified defense.

   [1] Certified Adversarial Robustness via Randomized Smoothing, Cohen et al, 2019.

3. In the following, we evaluate the performance of erase-and-check with GPT-3.5, Llama-2 13B, and Llama-3 8B (the latest Llama model) as the safety filter. We report the certified accuracy on the harmful prompts and the empirical accuracy on the safe prompts in the suffix mode. We also include the corresponding results for the Llama-2 7B and DistilBERT models from our paper for comparison. Larger models were excluded due to their high computational and API costs.

Empirical accuracy:

Hello reviewer,

As a reminder, the discussion period ends on Thursday, June 6. Please take a look at the author's rebuttal and acknowledge if they have adequately addressed your concerns and questions about the paper.

Thanks,

AC