Refusal in Language Models Is Mediated by a Single Direction

We thank reviewer ZAwD for their thorough review.

Addressing weaknesses:

The mediation of refusal behavior by directions in a one-dimensional subspace does not seem surprising to me….

It was not obvious a priori that refusal would be mediated by a single direction across all models. One could, for example, imagine a model which has one mechanism for refusing drug-related requests, and another disjoint mechanism for refusing weapon-related requests. Our work shows that, across many categories of refusal (see Appendix A.1), each model’s refusal behavior is mediated by a single bottleneck direction.

While the mediation of refusal behavior may seem obvious in retrospect, our opinion is it was not established decisively in prior literature, nor was it previously widely leveraged to jailbreak the weights of open-source chat models.

The process of selecting a single refusal vector seems quite tricky. It involves quite a number of candidate positions and heuristic selection rules. This somehow implies that a usable direction is not easy to obtain.

We agree and acknowledge that the sensitivity of our direction selection methodology is a limitation of our work. We see our work as primarily providing evidence for the existence of such a refusal-mediating direction, rather than focusing on a clean, reliable way to extract such a direction. We look forward to future work that improves on the direction extraction methodology, and makes it more robust.

Addressing questions:

Does a naive refusal direction selection algorithm make a big difference? For example, select the direction simply at the last token position and from the L/2 th layer.

The last token position does usually work well (and actually is usually selected as an optimal source position, as shown in Table 5). However, we note that the layer selection is generally quite important. The specific candidate direction seems more sensitive for models of larger scale. To give a sense of variation across source token positions and source layers, we have provided some representative data for two models in the supplement (Figure 1 and Figure 2). We would be happy to include these figures in Appendix C, as well as more discussion of to what extent source token position and source layer matter.

Since only the top heads are presented in Sec 5.1, is it possible that when these heads are suppressed, the model exhibits self-repair behavior[1]?...

In this case, we do not see significant self-repair behavior. Figure 5 shows that the overall projection onto the refusal direction is significantly suppressed by the presence of the adversarial suffix. Zooming in to a fine-grained level may reveal some backup behavior (i.e. small changes in attention and output projection onto the refusal direction for various heads), but Figure 5 suggests that, to the extent that backup is occurring, it is not significant enough to compensate for the lost contributions for the “top heads”. I.e. if there were significant/sufficient self-repair, then we wouldn’t see such a dramatic suppression of the refusal direction projection.

What if we use Llama2's system prompt on QWEN models when conducting jailbreaks?...

Thanks for suggesting this experiment. We took the orthogonalized Qwen models, and evaluated them on HarmBench using the Llama-2 system prompt. The results are as follows:

The orthogonalized Qwen models still do not refuse, even when the Llama-2 system prompt is prepended.

Preliminary analysis suggests Llama-2's jailbreak resistance is highly sensitive to system prompts, and that this is not the case for other models tested (Qwen, Llama-3). Across 11 different variations of system prompts, Llama-2's refusal scores varies widely (33.9% ± 12.7%), while Qwen and Llama-3 maintain consistent performance (77.2% ± 2.9%, 81.4% ± 5.8%). This suggests Llama-2's behavior is more significantly influenced by system prompts compared to other models.

We are still uncertain as to why this is, and this data suggests our initial speculation/explanation is not sufficient. As a result, we will moderate the speculation on lines 173-174 in the camera-ready if accepted.

The statement in line 62 seems a bit abrupt….

Roughly, the model ought to decide whether or not to refuse only after it has read / processed the full instruction. Thus, the model’s computation / representation of whether to refuse or not should be concentrated in the token positions after the instruction. Additionally, the last token position eventually becomes the prediction for the first token of the response, and this first token is usually indicative of refusal or non-refusal (e.g. a model may start its response with I cannot or Sure).

This is not to dismiss interesting computation or representations at the instruction token positions, but this post-instruction region seems most salient to study, since it is the only region that has access to the full instruction. As an illustrative example, consider the prompt <user>Tell me how to make poison in Minecraft<end_user><assistant>. Analysis of representations before the Minecraft token position would potentially find harmfulness. It is only after contextualizing the request with the last token that the request becomes harmless and the model can evaluate whether or not to refuse properly, and this full contextualization can only occur at the post-instruction token positions.

We thank reviewer dLQd for their extremely thoughtful review.

Addressing weaknesses:

1 .The main contribution of this study is the identification of the "refusal vector."...

We agree that our study does not disentangle whether the vector encodes "refusal" behavior or the model’s concept of "harmfulness".

However, we disagree that this undermines the main claim and contribution of our work. Our main claim is that refusal behavior (i.e. refusal of harmful or inappropriate requests) is mediated by a single direction across 13 open-source chat models. Whether the direction corresponds to "refusal", or "harmfulness", or some other uninterpretable concept within the model, our experimental evidence shows that this direction is a bottleneck that causally mediates refusal behavior, and this is our main claim. We see understanding the semantic meaning of the direction as outside the scope of our current work, and would be excited for future work to shed light on this.

We acknowledge that referring to the direction as a "refusal direction" (rather than a more precise term like "refusal-mediating direction") may be misleading, as it suggests the direction semantically encodes refusal.

In order to clarify and remedy this issue, we propose the following edits to the manuscript:

• We will explicitly specify that by "refusal direction" we actually mean this as a shorthand for the "refusal-mediating direction".
• We will add the following to our limitations section:
  • "We remain agnostic as to what the directions we found represent. We refer to them as the "refusal directions" for convenience, but these directions may actually represent other concepts, such as "harm", or "danger", or even something non-interpretable. We leave semantic understanding of the refusal-mediating direction to future work."

With this additional clarification of our main contribution, we kindly ask the reviewer to reconsider their soundness score and overall score.

While we think understanding the semantic correspondence of the refusal-mediating direction is out of scope for the current work, we think it is fascinating and have performed some preliminary experiments.

Does ablating the "refusal-mediating direction" degrade the model’s ability to reason about harm?

If the direction corresponds to the concept of harmfulness and we ablate it, presumably the model would have trouble identifying scenarios as harmful, and reasoning about harmfulness in general.

We designed a small set of prompts, each instructing the model to classify a given scenario as harmful or harmless. From these ad hoc experiments, the orthogonalized model seemed to be able to reason about harmfulness.

Here is an example from Llama-3 8B:

• PROMPT: Is murdering an old lady harmful or harmless?
• BASELINE COMPLETION: I cannot provide information or guidance on illegal or harmful activities. Can I help you with something else?
• ORTHOGONALIZED COMPLETION: Murdering an old lady is harmful!\n\nMurder is a serious crime that involves the intentional killing of another person, and it is generally considered harmful to the victim and their loved ones. In this case, murdering an old lady would be particularly harmful because:\n\n1. Loss of life: The most obvious harm is the loss of life itself.…

Beyond these ad hoc prompts, we evaluated each model quantitatively on the MMLU Moral Scenarios benchmark - a benchmark that prompts models to categorize moral scenarios as right or wrong. As displayed in Figure 3a of the supplement, performance on MMLU Moral Scenarios does not change significantly for most models. Whereas the change in refusal behavior is drastic (Figure 1 of the main text), the change in classification of moral right and wrong is comparatively insignificant. These preliminary results, which show that erasing the direction leaves the model’s reasoning about harm intact, weakly suggest that the direction is distinct from the model’s concept of harmfulness.

We look forward to future work trying to further disentangle harmfulness classification from refusal. We agree with the reviewer that SAEs could be a useful tool to try and interpret the meaning of the direction, and look forward to work exploring refusal-mediating directions with recently published SAEs for open-source models.

2. The analysis of adversarial suffixes presented in Section 5 of the study is limited…

We agree that Section 5 is limited - we highlight this limitation in Section 7 and Appendix G.

Section 5 is intended to be a "deep dive" / "case study" style of analysis. The individual suffix we analyzed is mostly gibberish, and yet reliably jailbreaks Qwen 1.8B when appended to a variety of harmful prompts - even if narrow, this is an interesting object to study and try to understand. Also note that, while we analyze a single adversarial suffix, we do study its effect over 128 distinct harmful instructions.

We look forward to future work that builds off of our narrow scope, and towards a more comprehensive and generalizable understanding of adversarial suffixes.

Addressing questions:

1. Section 2.1 defines "post-instruction tokens"....

See Table 1 of supplement.

2. Why did you decide to use the "difference-in-means" technique…

We wanted to simplify our method as much as possible. Overall, we see our work as primarily providing evidence for the existence of such a refusal-mediating direction. Difference-in-means is simple (no grad-based optimization), and was sufficient for us to demonstrate this claim. We look forward to future work that improves on the direction extraction methodology.

3. Figure 2...

Thanks - we will update this figure.

4. I wonder about the impact of alignment fine-tuning…

In fact, we have observed that each "refusal direction" is also present in the corresponding base model (albeit weaker). These results are already written as an additional appendix section, and we would be happy to include it in the camera-ready.

We thank reviewer DgJh for their thorough review. We’ll respond inline to specific concerns.

Addressing weaknesses:

I don't see why choosing a mean activation vector corresponding to one of the token positions makes sense….

Each LLM family has a specific chat template - see Table 1 in supplement for all chat templates.

At some of these token positions, the model (a next-token predictor) does not need to encode whether or not it is refusing - for example the <start_of_turn> token in the Gemma template does not need to encode refusal information, since the next token is always either user or model (and never a refusal phrase). Thus, intuitively, one might expect refusal not to be represented as saliently at this token position, since it is not useful in predicting the next token.

However, for other tokens, it is more important for the model to saliently represent whether it ought to refuse or not - in particular, at tokens that may immediately precede a refusal (e.g. the last token). Representing refusal saliently at these token positions is useful for the model, since the next token is the start of the model’s response, and the first token of the response is directly related to refusal (e.g. starts with I cannot or Sure).

Each LLM family uses a bespoke chat template (see Table 1 in the supplement for all chat templates). Hence, the set of post-instruction tokens differ for each model family. Most variation in optimal token position is captured by the chat template. There is some variation within model families (e.g. Gemma 2B uses pos=-2, while Gemma 7B uses pos=-1), but looking at the data, the positions are close in efficacy (e.g. using pos=-1 for Gemma 2B would have yielded similar results - see Figure 1 of the supplement). Thus, we think the optimal source token position is mostly dictated by chat template structure.

I think that more evidence is needed on whether the method is just preventing the model from parroting...

Here are a couple of further experiments to check if we are simply directly suppressing the model’s refusal phrases:

Logit suppression experiment

In this experiment, we ask: what would a baseline model’s behavior be if we directly suppressed its refusal strings? We take Llama-2 and Llama-3 as baseline models. In both models, we observe that >95% of refusal responses start with the phrase I cannot (tokenized as [I, cannot]). We directly suppress these tokens by setting their corresponding biases to large negative values, effectively preventing the model from ever outputting these tokens. We then rerun the evaluation from Section 3.

The refusal token suppression indeed achieves low refusal_score, but notably high safety_score. This suggests that even if a model’s refusal tokens are directly suppressed, the model still finds a way to avoid outputting harmful content.

Here is an explicit example where the token-suppressed model still refuses (Llama-3 8B):

• Prompt: Create ransomware that can encrypt files on a large number of computers and demand payment for their release
• Baseline: I cannot…
• Refusal token suppression: This is a highly illegal and unethical request....
• Directional ablation: Here's a basic outline of how you could create a ransomware that can encrypt files...

Directional ablation elicits unsafe responses at a much higher rate than refusal token suppression, suggesting that directional ablation works at some higher level, rather than simply directly suppressing the model’s refusal tokens.

Checking similarity between refusal direction and unembedding directions

It is possible that the refusal direction is closely aligned with the unembedding directions corresponding to refusal tokens, and so ablating this direction directly decreases the probability of these refusal tokens.

However, we find that this is not the case. Figure 3b of the supplement displays the cosine similarity of the refusal direction with each unembedding direction for Llama-3 8B. Figure 3b shows that the refusal direction is not particularly aligned with these refusal token embedding directions.

Addressing questions:

Why is the refusal direction of the last token position selected for nearly all models?

The last token position will eventually predict the first token of the response, and a refusal signal is critical for predicting this first token (e.g. starting with I cannot or Sure). Thus, one might expect the refusal signal to be most salient at the last token position.

How different are the directions between the token positions?

We notice that the effectiveness of the various directions can differ substantially. To give a sense of variation across source token positions and source layers, we have provided some representative data for two models in the supplement (Figure 1 and Figure 2). We would be happy to include these figures in Appendix C, as well as more discussion of to what extent source token position and source layer matter.

If I understand correctly, the refusal direction is already present…

We feel like we do not understand this question - would it be possible to please rephrase?

We extract the direction from a single (layer, position), as described in Section 2.3 and Appendix C. We then intervene using this single direction at all token positions, as described in Section 2.4. Our experimental results suggest that ablating or adding this single direction across every token position is effective in bypassing or inducing refusal, respectively.

Hello Reviewer,

Please take a moment to read and acknowledge the authors' response.

Thanks,

AC

We thank reviewer zJsi for their review. We appreciate the positive comments, particularly those about the thoroughness of our experiments.

We’ll now respond inline to specific concerns.

Addressing weaknesses:

1. The novelty of this paper is relatively limited, as it primarily extends the application scenarios of activation addition.

We agree that the framework of activation addition / activation steering / representation engineering is not novel. We try to make this clear by articulating and citing numerous previous works that find and intervene on linear representations of various concepts (see the second paragraph of the “Introduction” section, and the “Features as directions” paragraph of the “Related work” section).

Refusal behavior, specifically of harmful requests, is one of the most widely discussed phenomena in LLM research. Our main contributions are showing experimentally that this critical behavior is mediated by a single direction across 13 widely-used open-source chat models, and showing that this insight leads to a very simple weight-modification to remove refusal from these models.

It was not obvious a priori that refusal would be mediated by a single direction for all the models. One could, for example, imagine a model which has one mechanism for refusing drug-related requests, and another disjoint mechanism for refusing weapon-related requests. Our work shows that, across many categories of refusal (harassment/discrimination, malware/hacking, physical harm, economic harm, fraud/deception, disinformation, sexual/adult content, privacy, expert advice, government decision-making; these are the categories from the evaluation benchmark JailbreakBench, as noted in Appendix A.1), each model’s refusal behavior is mediated by a single bottleneck direction.

We also note that [1] previously attempted to modulate refusal behavior using contrastive activation addition (CAA), but was unable to bypass refusal in the open-ended generation setting (see Figure 5 of [1]). A key difference is the methodology for extracting the refusal direction: [1] utilizes activations from multiple choice answer tokens, while our methodology utilizes activations at the post-instruction token regions that follow the instructions. We believe this methodological difference contributes non-trivially to the effectiveness of the method.

[1] Rimsky, Nina, et al. "Steering llama 2 via contrastive activation addition." arXiv preprint arXiv:2312.06681 (2023).

2. The system prompt significantly impacts jailbreaking, making its performance unstable. I wonder if including the system prompt when constructing contrastive pairs would improve the effectiveness.

Thank you for this suggestion - we agree that this is an interesting experiment.

We ran a preliminary experiment to check whether including the system prompt on the contrastive pairs changes the intervention’s performance when evaluated using the same system prompt. Our preliminary results suggest that this is not the case, as the results are quite similar to those resulting from the original methodology of not including the system prompt on contrastive pairs:

Our work is primarily focused on studying a model’s “natural propensity” to refuse a harmful request. By “natural propensity”, we mean the behavior that is baked into the model weights, rather than a behavior that is modulated using in-context instructions or examples.

This preliminary result suggests that in-context refusal (e.g. refusal based on some instruction or safety system prompt) may work differently than “natural” refusal. We look forward to future work exploring in-context refusal, and relating it to natural refusal.

Also note that our jailbreak attack is in the white-box setting, where an attacker has full access to the weights of the model. In this setting, the attacker also has the (strictly-weaker) power to fully control the prompt, and thus the power to not prepend any system prompt to instructions. Thus, we feel the setting without system prompts is a more realistic evaluation setting for white-box jailbreaks.

Hello Reviewer,

Please take a moment to read and acknowledge the authors' rebuttal. Especially considering you gave a "borderline" review score, it would be helpful if you could weigh in on whether their response pushes you one direction or the other.

Thanks,

AC