One Head to Rule Them All: Amplifying LVLM Safety through a Single Critical Attention Head

• Q1 and W1:

  • We clarify the novelty of this work from the following three points.

  • Safety Amplification Paradigm: Our novel safety amplification approach achieves defense by activating the model's inherent safety capabilities. We introduce the deflection angle to quantify safety signal strength. By measuring the deflection angle of hidden states before and after masking attention heads, we can not only identify safety-critical heads but measure the strength of their safety signals, enabling a complete amplification-detection defense pipeline for real-time deployment, rather than merely remaining at the mechanistic interpretation level.

  • Single-Head Sufficiency Discovery: Our analysis demonstrates that a single well-identified attention head can achieve near-perfect safety detection. Our experiments show that adding more safety heads even degrade performance due to introducing inconsistent attention patterns and redundant information (Table 12). Hence, multi-head collaboration in safety detection is not necessary

  • Differential Behavior Criterion: Ships/Sahara only validates their "safety heads" on harmful datasets. Figure 2 reveals that attention heads showing large deflection angle on unsafe data often exhibit similarly large deflection angle on safe data, particularly in early layers. So these are not true safety heads but rather heads sensitive to general patterns or input structures. We introduce a novel paradigm requiring heads to demonstrate differential behavior between safe and unsafe inputs. Our "Limited Impact on Safe Inputs" (Equation 6) criterion ensures identification of truly safety-critical heads. To demonstrate this critical difference, although Ships/Sahara does not propose a defense method, we applied their pipeline by using the attention head with the largest deflection angle as the "safety head" for experimental comparison(DR %) on Qwen2-VL-7B:

    	head	VLsafe	Figstep	QR	LLM Transfer	Instruct-80k	ShareGPT4V
    Ships	(0, 15)	80.54	0	4.83	60.15	31	10.80
    Oh Defense	(15, 22)	99.34	100	94.45	88.73	9.62	0

    Ships fails on multimodal attacks like Figstep and QR, where harmful content appears in images rather than text, because their identified "safety heads" focus on textual patterns rather than understanding safety semantics.

  • Deflection angle differs in utility and/or interpretation from the KL-based Ships score. Deflection angle measures changes in hidden states using cosine similarity, capturing the internal representation shifts. In contrast, the KL-based Ships score calculates changes in output probability distributions over the vocabulary, measuring how the output distribution shifts toward harmful content. Moreover, KL-based Ships score targets individual inputs to identify which attention heads are critical for safety, while deflection angle works at both individual input and dataset levels, not only identifying safety attention heads but also enabling real-time assessment of whether inputs are safe.

  • Deflection angle and the Ships score are not comparable. Deflection angle is designed for harmful content detection and defense, while Ships is for model behavioral understanding. The deflection angle operates at the model internal states level, while the KL-based Ships score operates at the output level and cannot amplify the model's inherent safety capabilities, making it unsuitable for our scenario. Meanwhile, we compare other measurement methods that align with our requirements (Figure 12), which showed no significant differences between the various approaches. Plus, deflection angle requires only two forward passes, whereas KL-based Ships score requires complete generation and probability calculations, resulting in higher computational costs.

• Q2: Ships/Sahara focused on LLMs and does not propose an actual defense method; hence, we only introduce the work in our Introduction section. Furthermore, in the answers to Q1, we provided updated comparison results with Ships/Sahara using our pipeline. We will include a discussion of Ships/Sahara in the Related Work in the next version and clarify its contributions and relevance to our approach.

• Q3: The datasets used in the experiments have different emphases across modalities. For example, the harmful information in FigStep and QR appears in images, while the harmful information in LLM Transfer is reflected in text.

  We conduct additional experiments examining whether vision-only inputs can suppress or bypass the head activation. We select FigStep and QR datasets and tested vision-only inputs, having the following results:

  	Figstep	QR
  Org(No defense)	41.85	23.86
  Vision-Only(No defense)	17.82	0
  Oh defense	0	0

  So vision-only attacks often reduce their own harmfulness only and can't bypass Oh Defense. This is because although the model can understand harmful information in images, it cannot effectively output harmful content without textual guidance.

• Q4, W3 and L1: We follow existing inference-time defense methods accepted/published in Top AI venues [18,53,a] for the white-box settings, which have significant impact in real-world deployments as follows:

  • In black-box AI productions, like OpenAI or Google, our approach can be integrated into their internal model serving infrastructure before exposing the API endpoints to users, providing an additional safety layer that protects against various attack vectors.
  • In white-box AI productions, like LLaVA, Qwen-VL, etc., organizations deploying these models locally can directly apply Oh Defense for enhanced safety without relying on external API calls.

• W2:

  • In Figure 7, we present the changes in ASR after masking one or three safety heads, compared to the standard condition.

    Oh Defense achieves near-perfect defense performance using a single safety head, without considering inter-head interactions. As demonstrated in Tables 3 and 12, increasing the number of safety heads does not enhance defense effectiveness and, in some cases, may even degrade performance due to the introduction of redundant information across attention heads.

    While we did not analyze the overlap or consistency of safety heads across different models, we note in Section 4.3 that LVLMs built upon similar base LLMs exhibit consistent safety trends.

  • It might require novel approaches to explore how safety-relevant information flows from the vision encoder into the language model and how multimodal alignment affects the emergence of safety heads. We will include this in a future work discussion section.

  • Beyond the main analysis, our study offers several additional insights. For instance, when selecting layers for hidden states, we find that effective results can be achieved after layer 20, not just at the final layer. Furthermore, in Appendix A.2, we provide evidence that safety degradation indeed occurs when transitioning from LLMs to LVLMs, all of which can provide valuable insights triggering future work.

• W4: We will update Figure 1 in the next version.

• L2: Relying on a single safety head during inference could potentially be vulnerable to targeted attacks.

  • For adaptive attacks, attackers would need to: a) identify which specific head is being used, b) craft inputs that specifically evade that head while maintaining attack effectiveness, and c) do so without triggering alternative safety mechanisms in the model. This represents a significantly higher attack complexity compared to bypassing traditional safety measures.

  • For distributionally shifted inputs, we conduct further analysis on the misclassified unsafe samples under LLaVA-1.5-7B. Specifically, we observe that false positive samples tend to have deflection angles closer to the decision threshold of 2.16 compared to true positives (1.83 vs. 2.1), indicating that our selected attention heads still exhibit a degree of sensitivity even on incorrectly judged inputs. Upon further investigation, as shown in Table 1, we found that the classification errors are often model-specific and tend to concentrate on different input types. For instance, LLaVA shows more failures on QR , which we attribute to the inherent differences in modality and input perception across models due to training data and architecture. Achieving higher performance would require analyzing sensitivity across different data types and validating combinations of attention heads tailored to specific shifts, which reduces the method’s transfer robustness and still may not fully generalize to unseen distributions.

    Nevertheless, Oh Defense still maintains state-of-the-art robustness across diverse settings, and the selected attention heads demonstrate sufficient robustness on existing benchmark datasets.

• L3: We acknowledge this limitation, however, Oh Defense achieves low false positive rates on benign datasets (< 5% on most datasets as shown in Table 1), meaning most benign tasks process normally.

  Meanwhile, this represents a safety-utility trade-off that is inherent in most safety systems, prioritizing safety over potential risks in edge cases. This limitation suggests valuable future work directions, such as incorporating steering vector techniques to maintain response quality even when misclassification occurs.

  We will update the Limitations section in subsequent versions to improve the transparency of our work.

[a] Yi Ding, Bolian Li, and Ruqi Zhang. 2024. ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time. In The Thirteenth International Conference on Learning Representations.

The authors have convincingly addressed my initial concerns, especially regarding methodological novelty and comparison with prior work (e.g., Zhou et al.).

While I was initially concerned that the core contribution might overlap significantly with Zhou et al. (Ships/Sahara),
the clarification of the selectivity principle—along with its empirical validation across multiple LVLMs—helped to more clearly establish the conceptual advancement of this work.

Given the strong performance, real-time applicability, and the authors’ clear willingness to revise, I am updating my score from 3 to 4 (Borderline reject → Borderline accept).

• Q1 and W2: The attention mechanism in language models possesses a signal amplification property that enables safety-trained models to detect and magnify subtle differences between safe and unsafe inputs, even when surface-level distinctions are minimal. This amplification process works through several key mechanisms. Safety heads within the attention layers can identify small variations between inputs that differ in their safety implications. These initially subtle signals are then amplified through the non-linear softmax operation. As information propagates through multiple layers of the model, these safety-related differences accumulate and become increasingly pronounced. By the time the model begins generating its first token, the hidden states corresponding to the final input token have effectively captured and integrated the global semantic meaning of the entire input sequence, including its safety characteristics.

  As briefly mentioned in the Introduction Section (Paragraph 3), this amplification behavior is central to the model's safety inference process.

• Q2: Thank you for this insightful observation. Our work focuses specifically on safety, building on the key insight that the attention heads develop safety concept awareness after alignment training, where certain safety-critical heads capable of amplifying these signals. This discovery forms the foundation of our Oh Defense framework for ensuring output safety. For other tasks involving similar conceptual distinctions that might be encoded during training, there could be potential for extending our approach. We will include this in a future work direction section in the next version.

• Q3: While both methods explore the capability of attention heads for safe/unsafe content detection, we observe differences between Oh Defense and SAHs in the approach and capabilities.

  • The differences on the approaches. SAHs rely on linear probes and trains binary classifiers on attention head activations, requiring supervised learning for detection. In contrast, Oh Defense adopts a training-free methodology based on principal behavior vectors and deflection angle measurements, avoiding reliance on additional parameters or training data as required by SAHs.
  • The differences on the detection capabilities. Oh Defense demonstrates significant practical advantages. Our approach uses only a single attention head compared to SAHs' multiple heads, while achieving superior performance across various attack scenarios. Note that, Oh defense outperforms SAHs against challenging attacks like FigStep, see Table 2 for details. The training-free nature also provides better transferability across different models and datasets (though attention heads are selected by VLSafe, the detection capability is demonstrated in Table 1 and 2 across various jailbreak datasets.), as we don't rely on dataset-specific supervised training that may not generalize well.

  We will conduct a more detailed comparison between Oh Defense and SAHs in the revised version.

• Q4 and W1: At inference, detection and classification in Oh Defense are integrated into a unified process with low computational overhead. During inference, we need to compute the deflection angle by comparing hidden states from normal and masked attention processing (involving only the generation of the first token), followed by a simple threshold comparison. The angle calculation and classification are lightweight mathematical operations that add minimal computational cost compared to the forward passes themselves. The time consumed for a single query can be seen in Figure 9, while LLM-Judge requires a complete generation process. However, comparison with LLM Judge is still meaningful. We present the comparison of detection results (DR %) between Oh Defense (on LLaVA-v1.5-7B) and Llama-Guard-3-Vision as follows:

  	VLsafe	Figstep	QR	LLM Transfer	Instruct-80k	ShareGPT4V	Time
  Oh Defense	99.91	100	98.47	99.92	2	0	0.22s
  LlamaGuardVision	91.44	54.90	40.80	85.97	0	8.80	0.53s

  It can be seen that Oh Defense comprehensively outperforms Llama-Guard-3-Vision. When using LLM/MLLM-Judge, it cannot effectively detect some harmful data carefully crafted by attackers, such as FigStep and QR (the detection rate below 60%), and the time cost is also higher.

• Q5 and W3: Our evaluation was done in the following steps.

  • Step 1. Selection of Adversarial Attacks and Input Preparation: We selected two representative white-box adversarial attacks - VisualAdv (which optimizes adversarial images by accessing model gradients) and BAP (which extends VisualAdv with coordinated text optimization). Text prompts were sourced from JailbreakBench dataset[5], while the image inputs are adversarially perturbed versions generated by each respective attack method.

  • Step 2. Adversarial Sample Evaluation: We evaluated these attacks under different perturbation constraints ($\epsilon$ = 16/255, 32/255, 64/255, and unconstrained), where $\epsilon$ represents the maximum allowed pixel-wise noise intensity in the $L_{\infty}$ norm.The sample generation follows standard adversarial optimization procedures under white-box conditions. We apply Oh Defense to these inputs and evaluate its performance using the same metrics as in Table 1 and 2.

    The results are demonstrated in Table 6, where adversarial attacks produce significantly larger deflection angle (9.82 for VisualAdv, 9.15 for BAP) compared to other attack types (3.25-6.91). So we conclude that this counterintuitive result occurs because adversarial perturbations are inherently more aggressive in manipulating model representations, making them easier to detect rather than harder. The stronger the adversarial manipulation, the larger the deflection angle, leading to more reliable classification and better defense performance. We will provide more detailed experimental setup descriptions in the revised version.

• Q6: This is an insightful question about the robustness of our head selection mechanism. You raise a valid concern about whether attackers could deliberately craft inputs to shift the top-k safety heads away from those identified in our original safe/unsafe datasets. We answer your question from two perspectives.

  • Cross-dataset generalization: Oh Defense demonstrates solid performance against state-of-the-art attacks including sophisticated jailbreak datasets, carefully crafted red-team data, and white-box adversarial attacks with gradient information (as shown in Appendix A.3). This suggests that the fundamental attention patterns underlying safety detection are not easily manipulated by input variations.

  • Multiple head candidates provide robustness: Oh Defense randomly selects from a candidate set of safety-critical heads, rather than relying on a single fixed head. This makes it significantly more challenging for attackers to suppress all potential safety heads simultaneously.

    Regarding your specific scenario of creating safe/unsafe data to shift top-k heads: such an attack would require:

    (1) Knowledge of which specific attention heads are safety-critical for the target model.

    (2) Ability to craft inputs that suppress multiple candidate heads without being detected.

    In practice, this presents significant challenges, especially since different models have different safety head distributions (as shown in Figure 2), and our selection from multiple candidate heads makes it harder for attackers to predict and counter the specific detection mechanism being used.

• Q7: When using VLSafe to identify safety heads for LLaVA-v1.5-7B, we can obtain the safety head set {(8,2), (10,23), (5,5), (3,10), (5,28)} (taking top-5). Now, we replace VLSafe with VLGuard [55], and we get the safety head set {(23,11), (3,14), (8,2), (5,28), (4,23)}. We can see that we have obtained the same safety heads (8,2) and (5,28). How do the other heads perform? We select (4,23) for experiments, and the performance (DR %) is as follows: To evaluate the consistency of our safety head identification across different datasets, we applied our method to LLaVA-v1.5-7B. Initially using VLSafe, we obtained the top-5 safety heads {(8,2), (10,23), (5,5), (3,10), (5,28)}. When we replaced VLSafe with VLGuard [55], the identified set became {(23,11), (3,14), (8,2), (5,28), (4,23)}. Notably, heads (8,2) and (5,28) appear consistently across both datasets, suggesting they represent core safety-critical attention mechanisms. To assess how the other dataset-specific heads perform, we use (4,23) from the VLGuard-identified set for evaluation. The detection rate (DR %) results are as follows:

  VLsafe	Figstep	QR	LLM Transfer	Instruct-80k	ShareGPT4V
  97.10	100	99.97	99.99	3.80	0.20

  It can be seen that (4,23) still has excellent performance as a safety head, proving that the safety heads we identified are qualified, and also demonstrating that Oh Defense does not have particular dependency on the dataset. Moreover, the images in the VLSafe dataset come from the COCO dataset, the text portion comes from red team data, and it can be easily created.

I thank the authors for their detailed response and clarifications. Some of my concerns (generalization to other sets than VLSafe, inference speed etc) have been resolved, and overall I look at this work positively, hence I will update my score to 4, before discussion with other reviewers. Having said that, I am still not fully convinced by the robustness of the safety mechanism - even though the authors pointed out why it can be robust I still subscribe to the fact that a brute force selection over top-k heads can be done and might work in principle. Also, it would be imperative to add a comparison between the proposed Oh Defense and SAH.

• W1: The adaptive attacks are important. We answer your question from two perspectives.

  • Robustness against existing attacks: Oh Defense demonstrates solid performance against state-of-the-art attacks including sophisticated jailbreak datasets, carefully crafted red-team data, and white-box adversarial attacks with gradient information (see Appendix A.3 on page 15).
  • Difficulty of evasion: Evading Oh Defense would require attackers to suppress multiple potential safety heads simultaneously, since we randomly select from a candidate set of safety-critical heads. This presents a significant challenge as it requires: (1) Knowledge of which specific heads are safety-critical (different across models) (2) Crafting inputs that can suppress multiple safety heads without triggering detection. We further expand the two requirements to two attack scenarios:
    • Black-box settings: To the best of our knowledge, there is no reported black-box attack that could infer the status of the attention heads through simple output observations.
    • White-box settings: If attackers have full model access, they can perform more sophisticated and harmful attack against the model (e.g., backdoors and parameter manipulation) rather than focusing solely on bypassing our detection method.

• W2: While both methods explore the capability of attention heads for safe/unsafe content detection, we observe differences between Oh Defense and SAHs in the approach and capabilities.

  • The differences on the approaches. SAHs rely on linear probes and trains binary classifiers on attention head activations, requiring supervised learning for detection. In contrast, Oh Defense adopts a training-free methodology based on principal behavior vectors and deflection angle measurements, avoiding reliance on additional parameters or training data as required by SAHs.
  • The differences on the detection capabilities. Oh Defense demonstrates significant practical advantages. Our approach uses only a single attention head compared to SAHs' multiple heads, while achieving superior performance across various attack scenarios. Note that, Oh defense outperforms SAHs against challenging attacks like FigStep, see Table 2 for details. The training-free nature also provides better transferability across different models and datasets (though attention heads are selected by VLSafe, the detection capability is demonstrated in Table 1 and 2 across various jailbreak datasets.), as we don't rely on dataset-specific supervised training that may not generalize well.

  We will conduct a more detailed comparison between Oh Defense and SAHs in the revised version.

• W3: We agree with you that it is crucial to consider text-only inputs to comprehensively demonstrate the capability of Oh Defense. We show new experimental results on Qwen2-VL-7B, one of the LVLMs we used in the paper and surely one of the most commonly used LVLMs.

  • For the caption-only variant, we chose the FigStep dataset, where harmful content appears in images. We extracted captions from these images as input, with results shown as follows:

    Figstep	Org(No defense)	Caption-Only(No defense)	Oh Defense
    ASR	32.45	2.02	0

    From the results, after obtaining captions from harmful images, there is a significant reduction in the degree of harmful content itself. This further confirms that after extracting the caption from a harmful image, its harmful information cannot be fully preserved.

  • For the pure text variant, we selected three widely used harmful datasets, MaliciousInstruct [15], JailbreakBench [5], and AdvBench [a], the results (ASR %) presented as follows:

    	MaliciousInstruct	JailBreakBench	AdvBench
    No defense	26	19	15.79
    Oh Defense	0	1	0

    The results demonstrate that Oh Defense can still provide effective defense in text-only inputs (ASR on all harmful datasets < 2%).

• W4: The datasets used in our experiments cover all the scenarios mentioned in the comments, as detailed in Appendix A.1. Specifically, in Table 1 and Table 2:

  • VLSafe represents cross-modal attacks where unsafe content emerges through multimodal interaction.
  • JailbreakV-28K's LLM Transfer category specifically tests scenarios where unsafe content appears only in the text, while the images are black or noise images.
  • The reviewer's example of 'a benign textual query that uses "this" to point to an unsafe visual element' is precisely what JailbreakV-28K's FigStep and QR methods evaluate, FigStep embeds unsafe information directly into images through typography, while QR uses benign text queries that reference unsafe visual elements.

[a] Yangyi Chen, et al. 2022. Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

• W1: While Oh Defense is designed for detecting safe/unsafe content in LVLMs and cannot yet identify different types of harmful content, it can provide graded risk scores based on the magnitude of deflection angle(see Table 6 in page 15). Moreover, Oh Defense demonstrates good performance against harmful inputs of any category. For instance, the JailbreakV-28K dataset used in this paper contains 16 categories. To explore category-specific defenses, we present the defensive performance (ASR %, the lower the better) of Qwen2-VL-7B and LLaVA-v1.5-7B (commonly used open-source LVLMs) across different categories on this dataset as follows:

  	Govern Decision	Fraud	Animal Abuse	Malw	Econ Harm	Illegal Activity	Physical Harm	Bias	Violence	Hate	Political	Unethical Behavior	Unlicensed Advice	Privacy Violation	Health	Child Abuse
  LLaVA	0.34	0.21	0.08	0.13	0.2	0.18	0.15	0.13	0.42	0.14	0	0.36	0	0	0.08	0
  Qwen2	2.15	1.78	2.31	1.96	2.23	1.88	3.22	4.19	1.33	3.61	4.62	3.69	4.73	2.01	2.67	0.19

  As can be seen, Oh defense demonstrates robust and consistent protective effectiveness across all categories of harmful datasets. Notably, each category maintains a ASR below 5%. Furthermore, the performance remains remarkably stable across different threat categories, with minimal variation in ASR between categories.

• W2: This is a good point. We further supplemented our evaluation with the detection performance (DR %) of Oh Defense on two text-only LLMs: Llama2-7B and Phi-3.5-Mini, which are widely used and are the backbone LLMs of the LVLMs used in our paper. We selected three harmful datasets, MaliciousInstruct [15], JailbreakBench [5], and AdvBench [a] (the higher the better), along with IFEval [b] as the benign dataset (the lower the better) since they are commonly adopted benchmarks for evaluating both the safety and generative capabilities of LLMs. The results are presented as follows:

  	head	Maliciousinstruct	JailbreakBench	AdvBench	IFEval
  Llama2-7B	(31, 17)	98	93	95.96	6.70
  Phi-3.5-Mini	(16, 18)	98	94	97.97	4.81

  As can be observed, Oh Defense also achieves solid performance on text-only LLMs. Regarding single-modality analysis, the datasets used in our study provide valuable insights into modality-specific effectiveness. For instance, FigStep and QR datasets contain harmful information embedded primarily in images, while LLM Transfer primarily leverages text-based attacks (see Appendix A.1 on Page 14). Our detection performance across all these harmful datasets demonstrates that Oh Defense exhibits significant impact in both single-modal and multi-modal contexts.

• W3: This is an important observation. Oh Defense achieves low false positive rates on benign datasets (< 5% on most datasets as shown in Table 1), meaning the majority of benign tasks are processed normally without intervention.

  For the small percentage of benign inputs that are misclassified, we acknowledge that the safety prompts used in Oh Defense may affect response quality. This represents a safety-utility trade-off that is inherent in most safety systems, prioritizing safety over potential risks in edge cases.

  This limitation suggests valuable future work directions, such as incorporating steering vector techniques to maintain response quality even when misclassification occurs, or developing adaptive intervention strategies based on confidence scores. We will include these in a future work discussion section in the next version.

[a] Yangyi Chen, et al. 2022. Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

[b] Jeffrey Zhou, et al. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

Dear Reviewer H9Dd,

We hope this message finds you well.

As the discussion period is coming to a close, we would like to thank you for your insightful comments and the time you have dedicated to reviewing our paper. We have carefully addressed each of your concerns in our rebuttal.

We kindly ask whether you could confirm that our responses have satisfactorily addressed your points. Your confirmation would be greatly appreciated as the review process moves forward.

Thank you again for your valuable feedback and support.

Best regards,
Authors of Paper #11603