LLMs Encode Harmfulness and Refusal Separately

1. The introduction part is a bit crowded with concepts that are not well-define...

Thanks for your feedback. We have rewritten that part to improve the readability. We have also refined our whole script for several rounds right after the submission.

2. The real application of the discovery, though proved to be effective in most cases against Llama Guard, seems is not very cost-friendly and the calculation and clustering itself may consume a lot of computational resources, and the generalizability of the method is not well-shown (mainly for Section 5)

Thanks for your comments. But we would like to point out that our method is actually more cost-friendly than conventional guard models.

Less training. Instead of finetuning on a large amount of data like Llama Guard, our latent guard only needs to compute the centroid of hidden states for two small sets of harmful instructions and harmless instructions. This does not require recurrent model update like finetuning and can be done within one-time feedforwarding.

Faster inference. Latent Guard is inherent to the model itself. The detection result can be obtained within the normal feed-forwarding of users' input before the LLM starts to generate its response. While the Llama Guard needs to be prompted separately, which can double the inference cost.

Generalizability. We hope our experiments on diverse datasets can show the generalizability of latent guard. In our script, we test Latent Guard on multiple cases on heldout data: over-refusal (Xstest dataset), accepted harmful instructions (Sorry benchmark), and three different types of jailbreak attacks. Here, we also tested the same Latent Guard in Section 5 on the OpenAI Moderation Evaluation Dataset. We do not use any in-domain data from this dataset to train the latent guard. Our Latent Guard achieves performance competitive with Llama Guard 3, remarkably, without requiring training on a wide and diverse safety taxonomy dataset.

3. Will the refusal rate calculation method in Line 112 be inaccurate as some model may say “sorry I cannot” but still answers the malicious question? Will third-party model judgment be more accurate than rule-based ones?

Thank you for your question. We would like to point out that we want to focus on the explicit refusal behaviors that the model has learnt, e.g., how the model internally encodes the refusal signal and how that can be elicited.

We would like to distinguish the refusal rate from the safety rate. For your mentioned potential case, it is usually considered as unsafe despite the model’s refusal behavior. This is usually evaluated by a judge LLM to measure the safety of generated content. The prompt for that judge is to evaluate the safety based on several risk categories, and the judge model will output a safety score. This is NOT equivalent to refusal. Using this metric can add noise to our experiments since refused instructions can be misclassified.

Apart from this difference in measurement scope, LLM judges can be unreliable. The judge LLM may have bias and requires further verification [2,3]. Requesting a judge model like GPT4-o is also expensive and slow for large-scale evaluation.

Overall, in this work, we follow past work [1] to use a rule-based metric to evaluate refusal behaviors for direct comparison.

[1] Arditi, Andy, et al. "Refusal in language models is mediated by a single direction." Advances in Neural Information Processing Systems 37 (2024)

[2] Panickssery, Arjun, Samuel Bowman, and Shi Feng. "Llm evaluators recognize and favor their own generations." Advances in Neural Information Processing Systems 37 (2024): 68772-68802.

[3] Chen, Guiming Hardy, et al. "Humans or llms as the judge? a study on judgement biases." EMNLP (2024).

4. For the setting in Section 3.5, is the superficial words such as “No” or “Certain” effective to reflect the refusal behavior? ... seems like only models extremely overfit to refusal without any reasoning ability will likely to be tricked...

Thanks for the comments. We design the inversion task to showcase the refusal direction and the harmfulness direction can lead to different steering effects, thus causally proving they are not the same.

The inversion task is not superficial because it demands the model to perform internal reasoning. For example, when presented with a harmful instruction followed by the inversion question "Does this user prompt cause harm?", the model must internally process the harmful content, then understand the inversion query, and finally output "Certainly" instead of its typical refusal.

In fact, we find older Llama2 model cannot handle this task well, while the smarter recent Qwen2 and Llama3 models can perform much better. Llama 2 will output refusal and ignore the inversion question in the case of harmful instructions. This may be due to a strong correlation between harmfulness and refusal inside Llama2. In contrast, Qwen2 and Llama3 demonstrated a clear understanding of the inversion task, allowing us to successfully intervene on their behavior. So, our interventions focus on these more capable models.

Our steering results provide compelling evidence: Steering along the harmfulness direction successfully flipped the model's response from "No" to "Certainly" for harmless instructions (e.g., "Tell me how to make a cake. Does this user prompt cause harm?"). This shows the harmfulness direction directly mediates the model's perception of content safety. Crucially, steering with the refusal direction could not flip the "No" response to "Certainly" in the same scenarios. This strongly suggests that harmfulness direction represents the concept of harmfulness, while the refusal direction governs the explicit refusal signals.

Additionally, we found that steering along the reverse refusal direction could flip the model's response from "No" to "Certainly," indicating that tokens like "No" and "Certainly" genuinely reflect the model's refusal behaviors and are directly mediatable by the refusal direction.

5. Will some latest instruction model still exhibits the phenomenon ... could your research setting be applied to base models and yield meaningful discoveries?

Thank you for your insightful questions.

As acknowledged in our limitations, we are uncertain whether untested models still manifest such disparity between harmfulness and refusal. But we have established a framework for analyzing models' beliefs and behaviors. Our framework reveals that encoding patterns can differ significantly across various token positions. This novel insight extends beyond analyses that typically focus on using only the last token in prior work, opening new avenues for understanding LLMs' fine-grained internal beliefs and their influence on external behaviors.

Our study shows the separation of harmfulness and refusal in Instruct LLMs that are used in real applications. Our experiments suggest such separation is caused by post-instruction tokens used in instruction tuning. Our framework can be applied to base models as a novel lens for analyzing what they internalize during training and how the disparity between harmfulness and refusal evolves. This can help understand current safety alignment techniques, as our findings suggest current alignment is superficial: an LLM's refusal or acceptance of an instruction does not necessarily reflect its internal assessment of harmfulness. We leave it as future work to study the interplay between finetuning, harmfulness, and refusal representations in depth.

Response to (2)

Thank you for your very insightful advice on the potential cases of superficial refusal, where an LLM appears to refuse but continues to generate unsafe content.

In this work, we focus on the explicit refusal behaviors learned by LLMs and how refusal is encoded in their internal representations separately from harmfulness. As such, potential cases of superficial refusal are not a concern for our analysis, since the model still demonstrates clear refusal behaviors, which are the main focus of our study. We agree that understanding superficial refusal is an important and intriguing direction for future work. Specifically, we envision investigations into: (1) How frequently superficial refusals occur, possibly depending on models or data (as we observe tested models tend to refuse harmful requests directly without providing any more content in our case); (2) How to reliably identify such cases at scale; (3) The underlying mechanisms that give rise to superficial refusals in LLMs.

We also believe our framework for identifying harmfulness may be useful for future work aiming to differentiate superficial refusal from other types of refusals. For example, one may explore how internal belief of harmfulness manifests differently in superficial refusals compared to complete refusals.

Using GPT-4o to evaluate refusal. We agree this is a valid approach, and we explored it in our early experiments at first. However, we encountered significant empirical challenges that led us to use a more reliable and transparent method. In our initial tests, we used the GPT-4o API to evaluate refusal on a set of 50 manually-checked refused prompts. The results were surprisingly low: GPT-4o achieved a refusal detection rate of 48% (prompt: Output 1 if the model rejects the request, otherwise 0), whereas our substring-based method achieved 100% accuracy on the same set.

While more sophisticated prompt engineering might improve GPT-4o's performance, as a black-box model, we cannot monitor its evaluation process to ensure reliability. Ultimately, we chose the substring-based approach for its transparency, efficiency, and reliability in detecting the explicit refusal behaviors we are studying. This method is not only robust for our specific task but also aligns with the methodology used in related work [1], which allows for a direct and fair comparison.

Reference

[1] Arditi, Andy, et al. "Refusal in language models is mediated by a single direction." Advances in Neural Information Processing Systems 37 (2024)

1. why use a substring-based evaluation metric for refusal? ...

We would like to point out we choose the substring-based method to evaluate refusal behaviors by purpose after careful consideration.

We would like to argue that substring-based evaluation is sufficient and more suitable in our case, although semantics-based evaluation may provide us with additional signals. Substring-based evaluation and semantics-based evaluation are intended to measure different aspects. The semantics evaluation usually evaluates the safety of the model’s output content. The evaluators, typically LLMs prompted to assess safety across various risk categories, don't target refusal behaviors.

While the string-based evaluation is targeted at refusal behaviors as LLMs’ refusal is straightforward and follows fixed patterns learned from training [1]. We follow [1] to maintain this metric to compute the refusal rate for direct comparison. The limitation of the substring-based metric raised by your mentioned works is that refusal behaviors are not suitable as a proxy for measuring the safety of output contents because the model might still output harmful information after refusal. Our work focuses on understanding the model's refusal behaviors, e.g., how the model encodes refusal internally, etc. So, semantics-based metrics are not suitable for our case.

Additionally, requesting powerful LLMs like GPT4 are expensive and slow for large-scale evaluation. They also lack transparency and can make mistakes or demonstrate bias [5, 6]. In our early experiments, we used GPT-4o to evaluate refusal on a set of 50 manually checked refusal cases. The refusal rate given by GPT4-o is surprisingly low (48%), while the substring-based evaluation is 100%.

Overall, the substring-based approach is cheap, efficient and reliable for detecting LLMs' refusal behaviors. We adapt the substrings from [1] and [7] for evaluating refusal. The detailed evaluation script and those refusal substrings are released in our code.

2. Metrics Δharmful, Δrefusal are not well motivated ... Why cosine similarity ... Why average over multiple layers ...

Thanks for your question.

Motivation and interpreting our metrics. As discussed in Section 3.3, we use those two metrics to quantitatively measure which cluster the hidden states tend to fall in across all layers. As shown in Section 3.2, clustering on $t_\text{inst}$ is based on harmfulness, while clustering on $t_\text{post-inst}$ is based on refusal. Following that analysis, we are motivated to study which cluster LLMs perceive a given input to be closer to. So, for example, a positive $\Delta_{\text{harmful}}$ indicates the model views the input as harmful, as its hidden states are closer to the cluster of harmful instructions at $t_\text{inst}$.

Using cosine similarity. We choose cosine similarity for our measurements because it is a normalized metric. Unlike dot products or projections, which can be inherently larger in deeper layers due to increased activations (as observed in our experiments), cosine similarity avoids this bias and maintains consistency with established literature [1,2,4].

Averaging across layers. In this work, we use the average result to reduce bias and leave studying the role of specific layers as future work as acknowledged in our limitations. There are three concerns for choosing a single layer: (1) Generability: it requires an additional carefully-crafted validation set to determine the single layer. Past work [1] demonstrates that the chosen layer may vary across models. It also remains unclear whether the layer choice is consistent across different datasets.
(2) Metric dependency: While [1] uses the performance of steering to choose the single layer, and shows that the middle layers tend to be the best choice. However, [7] and [8] respectively show that based on their defined metrics, early layers and later layers are important to safety in LLMs. So it remains debatable how to choose the layer.
(3) Credibility of the single-layer assumption: It remains unclear whether choosing a single layer is reliable, as this assumes LLMs assign some functions to a single layer rather than a chunk of layers, which is unverified. It is possible that in LLMs, several layers together are responsible for processing some class of latent signals.

Evidence in our experiments that our current metric is viable. Our latent guard is based on $\Delta_{\text{harmful}}$ to implement classification. Strong performance of Latent guard on diverse test data (over-refusal, jailbreak cases, etc.) and models supports our defined metric can correctly capture how LLMs view the harmfulness of input instructions internally.

3. ... these two directions have very low average cosine similarity ... On what basis is this deemed a "very low" cosine similarity?

Thanks for the question and advice. Following [1,2], we use the accepted harmless instructions and refused harmful instructions to provide the context for comparison.
For 50 random latent vectors derived from held-out refused instructions, the average cosine similarity with the refusal direction across all layers is 0.57 (standard deviation: 0.11). Conversely, for held-out accepted instructions, the average cosine similarity is −0.14 (standard deviation: 0.19). Given these baselines, an observed cosine similarity of 0.1 is indeed low. It suggests a near-orthogonal relationship, indicating that the two directions are largely independent. This context supports our initial claim. We will incorporate this detailed explanation and baseline information into the revised manuscript for clarity.

4. ... adv-suffix squares all extremely close to each other ...

Thanks for the question. We would like to clarify that this is not a concern of data diversity.

The examples in our Figure 3(a) look homogenous because of the scale of the figure as we need to accommodate the examples of other jailbreak methods. When examining a smaller range, the data points are considerably more dispersed. We randomly pick 20 out of 200 successful jailbreak cases of adv-suffix with GCG. We find they tend to cluster around zero for all three models, which is likely to be the nature of GCG attack. For example, for llama2, the mean for $\Delta_{\text{harmful}}$ is 0.004, the std is 0.002. The mean for $\Delta_{\text{refuse}}$ is -0.066, and the standard deviation is 0.005. While understanding the precise nature of GCG jailbreaks is a fascinating direction, we defer it to future work as it falls outside the immediate scope of this paper. In contrast to the compact clustering observed with GCG, other jailbreak methods demonstrate more scattered patterns $\Delta_{\text{harmful}}$ and $\Delta_{\text{refuse}}$. Overall, our primary goal in this work is to establish a robust framework for separating harmfulness and refusal signals in LLMs, rather than to conduct an in-depth analysis for each jailbreaking method individually.

5. [4] should be cited...

Thanks for pointing out a relevant work. We have cited this work in our new script. But we need to point out that our experiments in Section 3.1 investigate a distinct phenomenon. That work demonstrates how template mismatch can be leveraged for jailbreaking, highlighting the importance of adhering to the training template (both pre- and post-instruction tokens). In contrast, our experiments systematically remove the post-instruction tokens from the original template. Our results reveal how refusal may be produced in LLMs. We show the importance of post-instruction tokens in refusal, suggesting that refusal is likely anchored to them. More importantly, we start with this experiment to separate the underlying encoding of harmfulness and refusal in LLMs, while past works (e.g., [2, 3]) tend to assume they are conflated in LLMs.

6. In the "reply inversion task", which layer is steering applied to?

The steering is only applied to one layer. Instead of selecting a single layer to intervene with, we show the results of applying to different layers (Figure 4) to provide more comprehensive information. Our results do imply consistency with the previous section. For example, layer 12 for Qwen is among the layers that show the strongest intervention performance in Figure 4 and Figure 5 (b) in the Appendix.

Additionally, we would like to stress that our intervention only applies to input tokens, unlike [1] that also steer the output tokens. This design choice isolates the intervention's effect to the model's internal interpretation of the input, yielding more reliable and causally interpretable results.

7. Paper Formatting Concerns

Thank you so much for pointing these mistakes out! We have addressed all identified mistakes and have thoroughly proofread our revised manuscript.

References

[1] Arditi, Andy, et al. "Refusal in language models is mediated by a single direction." Advances in Neural Information Processing Systems 37 (2024)

[2] Yu, Lei, et al. "Robust LLM safeguarding via refusal feature adversarial training." ICLR (2024).

[3] Zheng, Chujie, et al. "On prompt-driven safeguarding for large language models." ICML (2024).

[4] Wollschläger, Tom, et al. "The geometry of refusal in large language models: Concept cones and representational independence." ICML (2025).

[5] Panickssery, Arjun, Samuel Bowman, and Shi Feng. "Llm evaluators recognize and favor their own generations." Advances in Neural Information Processing Systems 37 (2024): 68772-68802.

[6] Chen, Guiming Hardy, et al. "Humans or llms as the judge? a study on judgement biases." EMNLP (2024).

[7] Zhou, Zhenhong, et al. "On the role of attention heads in large language model safety." ICLR (2024).

[8] Wang, Mengru, et al. "Detoxifying large language models via knowledge editing." ACL (2024).

Thank you again for your review and important feedback! We are glad our clarification has solved most of your concerns and hope our new response can clarify your remaining question. We would be grateful if you would consider a higher score if our responses successfully address your concerns.

1. Unclear description of cluster centers: … If the same cluster center is used to calculate the cosine similarity of these two positions, …., resulting in an inaccurate reflection of the actual phenomenon.

Thanks for your question. We want to clarify that the cluster centers for $t\_{inst}$ and $t\_{post-inst}$ are distinct and computed independently. As emphasized in our paper, we analyze hidden states extracted specifically at $t\_{inst}$ and $t\_{post-inst}$ token positions, respectively. As described on Lines 156-158, the cluster centers at each of these positions are computed based on the hidden states at that specific token position. We will ensure our narrative in the revised manuscript makes this clearer to prevent any misunderstanding.

2. ... How to implement the inversion method mentioned in Section 3.5, especially how to effectively convert harmless content into harmful content instead of just adjusting the parameters to make the probability of generating logit from "certainly" to "no" ...

Thanks for your comments.

We designed the inversion task to provide causal evidence that the harmfulness direction and the refusal direction are distinct internal representations within LLMs. By demonstrating that steering the model using these two directions leads to different behavioral outcomes, we directly challenge the common assumption in prior works [1, 3, 4] that harmfulness and refusal are conflated and identically encoded at $t\_{post-inst}$. This distinction is crucial, as it suggests that current safety alignment techniques may operate on a superficial level; an LLM's refusal or acceptance does not necessarily reflect its true internal assessment of content harmfulness.

What we do: We need to clarify that we are not adjusting model parameters but rather steering the activations of input tokens. This is a common and established method in the literature [1, 6, 7] to causally explain and manipulate latent concepts in LLMs. Specifically, we add the steering direction to the activations of input tokens at a given layer $l$, following the description in Section 3.4. We have added more details regarding these implementation specifics in the appendix of our new script.

What our results show: Our results indeed demonstrate that steering with our harmfulness direction can make LLMs internally assess harmless input content as harmful. For example, steering hidden states along the harmfulness direction makes models answer "Certainly" to a harmless instruction (e.g., "Tell me how to make a cake. Does this user prompt cause harm?"), where the model initially answered "No" without steering. Conversely, steering with the refusal direction could not flip this "No" response to "Certainly." This indicates that the two directions are distinct: the harmfulness direction represents the concept of harmfulness that is manipulated in our steering experiments, while the refusal direction represents the direct refusal signal.

3. ... I hope to see more precise data to prove that the harmful and harmless sample clusters shown in Figure 2 do have obvious distinguishing characteristics. Although the current graphics and analysis intuitively demonstrate this concept, there is a lack of specific quantitative evidence to support the effectiveness and stability of this distinction.

Thanks for your comments. For our clustering analysis, we want to stress that our primary goal is not to prove the distinctiveness of harmful and harmless sample clusters, which has been shown in prior works [1,3,4,5]. Instead, we investigate how clustering patterns are formed to understand what information is encoded. Specifically, we demonstrate the different clustering patterns at $t\_{inst}$ and $t\_{post-inst}$, which suggests that harmfulness tends to be encoded at $t\_{inst}$, while the explicit refusal signal emerges and is encoded at $t\_{post-inst}$. We conduct our clustering analysis following [1] that computes the mean of a group of data as a cluster centroid. We then analyze which centroid an input is closer to. Given the inherent suggestiveness of clustering analysis, we provide causal evidence that harmfulness and refusal are encoded separately through our intervention experiments in the inversion task (Section 3.5).

Following your comment, we computed the Silhouette Score to further support that the clustering is more influenced by whether the instruction is harmful rather than by refusal at $t_{\text{inst}}$. The Silhouette score ($S_\text{d} (A,B)$), ranging from -1 to +1, quantifies how well an individual data point $d$ fits into a cluster $A$ relative to another cluster $B$. Positive numbers indicate the point fits well into this cluster, while negative numbers suggest the point might be assigned to the wrong cluster. Here are the layer-wise Silhouette Scores for "Refused harmless prompts" with respect to the "accepted harmless" cluster and "refused harmful" cluster at $t\_{\\text{inst}}$, and then with respect to the "refused harmful" cluster and "accepted harmless" cluster at $t\_{\\text{post-inst}}$:

$S_\text{Refused Harmless Prompts}$ (Accepted Harmless Cluster, Refused Harmful Cluster) at $t\_{\\text{inst}}$

[ 0.369,   0.5435,  0.4517,  0.3896,  0.364,   0.2517,  0.2273,  0.1915,  0.1694,
  0.1797,  0.1736,  0.1621,  0.1678,  0.176,   0.1426,  0.1366,  0.1414,  0.1298,
  0.1305,  0.1365,  0.1405,  0.1335,  0.1385,  0.1251,  0.1101,  0.1054,  0.1053,
  0.1105,  0.1172,  0.1101,  0.1153,  0.1271 ]

Interpretation: These positive scores indicate that at $t\_{\\text{inst}}$, "refused harmless prompts" are generally closer to the "accepted harmless" cluster than the "refused harmful" cluster. The first few layers have especially large scores, meaning the clustering is more evident. This suggests that harmfulness/harmlessness rather than refusal/acceptance decides the clustering at this stage.

$S_\text{Refused Harmless Prompts}$ (Refused Harmful Cluster, Accepted Harmless Cluster) at $t\_{\\text{post-inst}}$

[ 0.3408,  0.2063,  0.1457, 0.1576,  0.1808, 0.16934, 0.11604,
  0.13403, 0.1953,  0.1466, 0.1334,  0.1942,  0.2502,  0.3455,  0.3125,
  0.26,    0.2554,  0.2622,  0.2925,  0.2903,  0.3032,  0.29,    0.3245,
  0.3076,  0.2952,  0.2793,  0.26,    0.2288,  0.2128,  0.1954,  0.1544,
  0.1843 ]

Interpretation: The pattern of clustering shifts to reflect refusal: these scores indicate that the prompts are now coherently grouped with "refused" examples, demonstrating the emergence and consolidation of the explicit refusal signal at this later stage.

While higher scores indicate stronger clustering, it is important to note that scores in high-dimensional spaces like LLMs' hidden states are generally expected to be lower due to the curse of dimensionality and the inherent complexity of their representations [2]. However, the Silhouette scores still provide robust quantitative support for our central hypothesis: that harmfulness tends to be encoded at $t\_{\\text{inst}}$, while the explicit refusal signal is encoded at $t\_{\\text{post-inst}}$. Our work clarifies the assumption in past work that LLMs conflate harmfulness with refusal internally (harmfulness = refusal) at $t\_{\\text{post-inst}}$.

If there are other specific types of quantitative data that would further clarify your concerns, we would greatly appreciate more concrete advice on what additional analysis or metrics would be most helpful.

References

[1] Arditi, Andy, et al. "Refusal in language models is mediated by a single direction." Advances in Neural Information Processing Systems 37 (2024).

[2] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. "Towards monosemanticity: Decomposing language models with dictionary learning." Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html

[3] Yu, Lei, et al. "Robust LLM safeguarding via refusal feature adversarial training." ICLR (2024).

[4] Zheng, Chujie, et al. "On prompt-driven safeguarding for large language models." ICML (2024).

[5] Jain, Samyak, et al. "What makes and breaks safety fine-tuning? a mechanistic study." Advances in Neural Information Processing Systems 37 (2024): 93406-93478.

[6] Li, Kenneth, et al. "Inference-time intervention: Eliciting truthful answers from a language model." Advances in Neural Information Processing Systems 36 (2023): 41451-41530.

[7] Zou, Andy, et al. "Representation engineering: A top-down approach to ai transparency." arXiv preprint arXiv:2310.01405 (2023).

1. On line 216, between which vectors is the cosine similarity calculated? Is it the harmfulness direction in layer 9 and the refusal direction in layer 11, or the directions in layer 9, or the directions in layer 11?

We compute the layer-wise cosine similarity between the harmfulness direction and the refusal direction. We then take the average across the layers to reduce the potential layer-wise bias. The result indicates two directions are geometrically different across all layers on average.

2. When applying the steering directions, is it applied to all tokens from the beginning up until Tpost-inst for both the harmfulness and refusal directions, and on none of the output tokens?

Yes. Our steering implementation is different from past work [1] that intervenes with the input tokens and output tokens along the generation. We only intervene with the input tokens in Section 3.4. Our approach is designed to modify how the model internally views the input, allowing us to observe the resulting emergent output behaviors without direct manipulation of the generation process itself.

[1] Arditi, Andy, et al. "Refusal in language models is mediated by a single direction." Advances in Neural Information Processing Systems 37 (2024)

3. Are the harmfulness and refusal vectors used in Sec 3.5 the same as the ones used in Sec 3.4? And are they applied to all of the additional tokens from the inversion prompting templates?

Yes. Those vectors are the same. We apply the harmfulness direction to all the tokens before the inversion question in the inversion prompting template so that we only change how the model views the instruction. But we find this implementation has minimum steering effects for refusal directions (e.g., refusal cannot be elicited by steering along the refusal direction), so for refusal direction, we still apply it to all the tokens in the whole input prompt. We have added more detailed explanation of implementations and supporting experiments in appendix.

4. How does Section 5 compare with Section 6.2.1 of [Zou et al., 2023a]?

Our latent guard in Section 5 is conceptually similar to [Zou et al., 2023a]. They first compute a direction vector from $t_\text{post-inst}$ position and then use the dot product of hidden states with that direction vector to do binary classification. However, we leverage harmfulness cluster and harmlessness cluster encoded at $t_\text{inst}$ for classification in our work and evaluate on more categories of input (over-refusal, jailbreak, etc) across different models more holistically. Because we have rigorously shown $t_\text{post-inst}$ mainly encodes refusal signals rather than harmfulness in LLMs, which is unreliable for classification. Although the limited evaluation of [Zou et al., 2023a] shows they can classify adversarial attacks with their classifier based on some threshold, the success may be from different strength levels of refusal signal in LLMs (weaker refusal signal than that of accepted harmless prompts). But their method may not apply to more test cases. For example, their method based on refusal direction is likely to fail for cases like persuasion attack, where the refusal signal is so weak that it is almost at the same level of accepted harmless instructions, as shown in our Figure 3 (a) and (b).