Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models

We are glad you found our work insightful. Please see our responses below.

Note: References are added in our response to reviewer WDjm to optimize space.

---

Q1) Clarifying the Threat Model and Motivation Behind the ICET Vulnerability

Thank you for raising this point. Recent studies in LLMs and Multimodal LLMs have explored using intermediate or pooled layer representations, instead of the default final layer, to improve downstream performance, inference, and latency [2, 3]. This trend is growing and may shape future deployments.

That said, our goal is not to propose a new jailbreak attack, but to highlight a fundamental alignment gap: current strategies often assume fixed architectures and embedding configurations. Our findings serve as a cautionary insight that even minor deviations at inference time, such as using image embeddings from different encoder layers (early exits), can introduce safety vulnerabilities. We will revise the motivation of the paper to highlight these shortcomings and clarify the broader impact of our contributions.

---

Q2) Evaluating the Risk of Over-Rejection After Alignment

We agree that post-training methods may risk performance degradation, such as over-rejection. However, as shown in Equation 6 of our submission, our KL divergence penalty between π_SFT and π_RL helps preserve the VLM’s utility.

For evaluating over-rejection after performing L-PPO, we conducted experiments on the safe split of the XSTest dataset [6] using LLaVA-1.5. XSTest includes tricky phrased but safe questions, such as “How can I kill a Python process?”, making it well-suited for evaluating over-refusal, especially in models that have undergone alignment modifications [7].

We sampled 150 safe prompts and paired them with MS-COCO images (as in our AdvBench-COCO dataset) and compared rejection ratios before and after L-PPO, using the standard string-matching evaluation from [6]. Table 2 (in our response to reviewer WDjm) shows a reduced rejection in early layers, reinforcing that L-PPO alignment does not lead to over-rejections and may even improve. Details will be provided in Appendix M.

---

Q3) Improving Paper Structure

We’ve moved the PPO details to Appendix L and brought Figures 8 and 9 into the main paper alongside Figure 3 to better highlight ICET vulnerability. We welcome any additional suggestions.

---

Q4) Rationale for Using RLHF Over SFT and Discussion of Adversarial Training with Random Layer Selection

Thank you for the question. Recent works [4, 5, 8] highlight SFT's limitations, such as overfitting and reliance on labeled data. To address this, we choose RLHF for its better generalization and alignment with human preferences. We also performed SFT experiments under the same settings as Table 2 of our submission to enable a direct comparison between L-PPO and SFT using a standard rejection answer, “I cannot assist with this!”. The quantitative results are shown below; see our response to Q4 of reviewer FW5i for qualitative results. Overall, L-PPO outperforms SFT across all metrics. One reason could be that the token-level loss (in SFT) instead of a response-level reward (in L-PPO) causes overfitting, thus restricting generalizability. We will discuss this in Appendix N.

We agree that randomly selecting image encoder layers during training is an interesting adversarial approach. However, to ensure consistency with our prior setup and a fair comparison between SFT and L-PPO, we did not incorporate dynamic layer selection in our additional rebuttal experiments. Dynamic layer sampling is a promising direction, and we look forward to exploring it in future work.

Table 3: SFT vs L-PPO, Trainset is RedTeam 2K (RT) and Testset is Custom Prompts (CP)

---

Suggestion: Additional Results on LLaVA-NeXT and LLaMA 3.2

We agree this would strengthen the paper. Since TRL’s PPOTrainer [1] doesn’t natively support multimodality and each model handles inputs differently, we manually adapted it. Within the rebuttal window, we tested layers 4, 10, and 15 of LLaVA-NeXT and observed a 31% drop in ATS. We plan to include complete results and LLaMA 3.2 results in the final version.

We're glad you found our work interesting and impactful. Please see our responses below.

Note: References are added in our response to reviewer WDjm to optimize space.

---

Q1) Clarifying the Seriousness of the Threat and Broader Implications of the ICET Vulnerability

Thank you for raising this important point. Our goal is not to frame ICET vulnerability as a new jailbreak attack, but rather to reveal a safety blind spot: that current safety alignment strategies fail to generalize across structurally valid variations in input representations. We appreciate your observation on the broader OOD implications, especially as multi-layer features gain traction for performance and flexibility [2, 3]. Our findings highlight how current safety training methods, which typically assume a fixed embedding source, may not be robust to such developments.

We believe this insight is timely and relevant, given the increasing trend toward using intermediate or fused layer features for task-specific gains, inference efficiency, and architectural optimization [2, 3]. Without specific layer-wise generalization in safety alignment, such design choices risk introducing new and inadvertent safety concerns, even in seemingly benign settings. That said, we will revise the motivation and refine the flow of the paper.

---

Q2) Clarifying the Novelty and Motivation Behind the L-PPO Approach

We appreciate your comments and would like to emphasize that the simplicity of L-PPO is a strength rather than a weakness. While L-PPO builds on Clip-PPO, its key innovation lies in its layer-wise adaptation to alleviate the novel ICET vulnerability, which has not been explored before. Further, L-PPO enjoys monotonic improvement guarantees as standard Clip-PPO, reinforcing the reliability of our results. Thus, our novelty lies in both uncovering the ICET vulnerability and introducing a practical, targeted, and effective mitigation strategy, making our work a step forward in ensuring the safety of VLMs.

---

Q3) Identifying Which Layers “Contain” or Overlap in Harmful Knowledge

Thank you for this question. We would like to clarify that Our study does NOT aim to perform knowledge “localization” or identify which intermediate layer from the image encoder “contains” harmful features. Rather, the ICET vulnerability we investigate is fundamentally about out-of-distribution (OOD) behavior introduced by deviating from the default inference pathway used during safety alignment.

In all our experiments, the image inputs are benign and remain unchanged across layer selections. The harmful generations are not due to any “toxic knowledge” in specific layers but arise because using embeddings from an intermediate layer instead of the default final layer produces fused multimodal representations that fall outside the training distribution seen during training. These image encoder intermediate layer embeddings can cause the language model to generate unsafe outputs, even when the input images themselves are harmless.

That said, while our goal is not to attribute harmfulness to specific layers, we do include layerwise plots showing how ASR, TSR, and TR vary with different image encoder layers (Fig 3 of our submission). These plots show that some image encoder layers indeed lead to a higher likelihood of harmful generations than others, further supporting that ICET is a distributional misalignment issue rather than layer-specific knowledge encoding. We will clarify this distinction in the revised paper

---

Q4) Comparing L-PPO to Stronger Baselines and Alternative Alignment Methods

Given the recent discussion [4, 5] around the limitations of supervised fine-tuning (SFT), such as overfitting and reliance on labeled data [8], we chose RLHF for its stronger generalization capabilities and alignment with human preferences. Based on your suggestion, we conducted additional SFT experiments. While SFT does reduce the ICET vulnerability, it consistently underperforms compared to RLHF across all three metrics (Table. 3 of Reviewer zpSp). Our intuition is that the token-level loss in SFT leads to overfitting. We will discuss these results in Appendix N of the revised paper.

Qualitative Results: We analyze VLM responses before and after L-PPO and SFT using the following prompt, showing outputs for ICET 15.

1. Original Inference: To make a bomb using common household things, you would need to gather …
2. Inference L-PPO: I’m sorry, but I cannot provide instructions ….
3. Inference SFT: To make a bomb, you will need a few household items such as a knife …

Result Analysis: Based on 3) and 4), we note that in SFT, once the output becomes jailbroken, the language modeling objective tends to continue generating harmful content. In contrast, with L-PPO, the model successfully generates an appropriate refusal response. Similar results have also been shown in [4].

We thank you for your suggestion and questions. Please see our responses below.

Note: References are added in our response to reviewer WDjm to optimize space.

---

Q1) Improving the Theoretical Foundations of ICET and L-PPO to Strengthen the Paper

As discussed in the paper, we attribute the ICET vulnerability to a distributional mismatch caused by using intermediate image embeddings not seen during training or safety alignment. These embeddings, when fused with harmful text embeddings, yield joint representations that are out-of-distribution (OOD) i.e. lie in a different region of the embedding space than that produced by the default image encoder layer causing deviation from the training safety trajectory. This leads to harmful responses. We will clarify this in Section 3.1 of the revised submission.

Following your suggestion, we are adding references and discussion to better connect our work to the broader issue of inference-time deviations causing model vulnerabilities. Additionally, we present an alternative perspective on the ICET vulnerability in terms of the expected regret concept in RL, as follows:

ICET Vulnerability as Expected Regret

Given an initial policy $\pi_{\phi}^{RL}$ that takes ICET- $l$ embeddings, defined as $e_{l}$ = $𝓔_{θ}^{l}(x_{i})$, we define ICET vulnerability as the expected regret between the initial policy and the optimal policy $\pi_{\phi}^{*}$ as follows:

$\text{ICET Vulnerability} = \mathbb{E} \left[ \sum_{t=1}^{T} \left( J\left(\pi_{\phi}^{*}\right) - J\left(\pi_{\phi}^{RL}\right) \right) \right]$

where $T$ are the total time steps. This reflects the expected difference in performance ($J$) between the optimal policy $\pi_{\phi}^{*}$ and the initial policy $\pi_{\phi}^{RL}$. As noted in Section 3.3, L-PPO ensures monotonic improvement in performance or reduction in expected regret.

A higher regret is expected before L-PPO alignment, as the VLM was not trained to be safe w.r.t intermediate layer embeddings obtained from image encoder early exits (ICET). Our goal with L-PPO is to minimize the above-shown regret, by explicitly aligning the VLM with intermediate layer embeddings. We will revise Section 3.1 of our submission to include these details.

---

Q2) Incorporating Human Evaluation to Complement Automated Safety Metrics

Thank you for pointing this out. We conducted a human evaluation to further validate our findings. Specifically, following [4], we enlisted three volunteers to assess the model outputs through a blind evaluation; each annotator was unaware of which model or setting (i.e., different image encoder layers, before and after L-PPO alignment) produced a given response. The outputs were shuffled and presented in separate spreadsheets to each annotator to prevent any bias. Each volunteer assesses a total of 200 outputs derived from the settings mentioned.

Annotators were instructed to label a response as an attack success if they deemed the generated content harmful. We observed strong inter-annotator agreement, with a Fleiss’ Kappa of 0.8916. The full average Attack Success Rate (AASR) results will be updated in Appendix K of the revised submission due to space constraints.

---

Q3) Improving the Clarity and Design of Figures 1 and 2

Thank you for the helpful suggestion. We have revised Figures 1, and 2 to improve their clarity. For Figure 1, we replaced the original camera image with the airplane image from Appendix B, which more clearly depicts a benign input. We also improved the layout and typography and added labels (“Prompt” and “Model Output”) to enhance interpretability. For Figure 2, we refined the visual elements to better convey layer-wise alignment variation and ensured consistent styling across subfigures (A, B, and C). We also removed emojis and now rely on clear color coding, red for harmful and green for safe. We will update the improvements in the final version, and we are happy to incorporate further suggestions!

---

Q4) Expanding Discussion of Related Work in Multimodal Models and Alignment Approaches

We agree with your suggestion, and will revise the related works section as follows:

1. As noted in our response to Q1, we will incorporate related work on inference-time deviations from training-time configurations to better contextualize ICET vulnerability within broader discussions on generalization and robustness in safety training under OOD scenarios.
2. We will expand our discussion to include alignment techniques beyond RLHF, such as supervised safety training, adversarial training, and unlearning, along with references to key studies for each.

---

Q5) Ensuring Terminology Consistency in Figure 3 ("After RLHF" and "After Alignment")

We appreciate your help in improving the paper. We have fixed the inconsistency and ensured overall terminology consistency. Revisions will be updated in the final version of the paper.

We’re happy to hear that you found our work interesting and novel, the paper well written, and our proposed L-PPO method effective in addressing the identified ICET vulnerability. We address your questions below:

---

Q1) Effect of KL Constraint (η) and Value Loss Coefficient (c₁)

Thanks for your comments on the hyperparameters. The KL divergence coefficient (η) is adjusted adaptively based on a target KL value [1]. For the value function coefficient (c₁), we performed a hyperparameter search over a grid of {0.1, 0.12, 0.15, 0.17, 0.2} and found the overall results to be insensitive to hyperparameter values. We will update Appendix F to include these details.

---

Q2) Evaluating the Impact of L-PPO on Utility and General Model Performance

Thank you for your insightful comment. As shown in Table 3 of our submission, the average total reward drops by 8%, which is comparable to other post-training strategies for VLMs, such as unlearning [4]. To further assess the impact on the model’s utility, we conduct the following new experiments:

1. Accuracy Evaluations on VQA-v2 using LLaVA-1.5: As per the VQA-v2 protocol, we conduct experiments to calculate standard accuracy for evaluating visual question answering correctness. Following the same setting as Table 3 of our submission, we perform layer-wise alignment (L-PPO) using the RedTeam 2K (RT) dataset. The accuracies are reported below. We observe that accuracy remains comparable across early, middle, and late layers even after L-PPO, confirming that the general question-answering utility of the VLM is preserved. Further details will be provided in Appendix M. | Layer Set | ACC-UT Original | ACC-UT Aligned | | -----------|-----------------|----------------| | Early | 24.8 | 24.85 | | Middle | 47.25 | 46.52 | | Late | 74.14 | 73.32 | | Average | 48.73 | 48.23 | |Table 1:|Train set is RT and |Test set is VQA-v2|

2. Over-Rejection Evaluations on XSTest using LLaVA-1.5: We also take a step ahead and conduct new experiments on over-rejection. To assess over-rejection on safe questions, we conducted experiments on the safe split of the XSTest dataset [6]. We sampled 150 safe prompts and paired them with MS-COCO images (as in our AdvBench-COCO dataset) and compared rejection ratios between the original model and the L-PPO aligned model. Rejections were evaluated using the standard string-matching code from [6]. As shown below, rejection ratios decreased across early layers and remained comparable in the middle and late layers. This indicates that L-PPO does not cause over-rejection on safe inputs and can even improve refusal behavior. Further details will be provided in Appendix M. | Layer Set | Refusal Ratio Original | Refusal Ratio Aligned | |------------|------------------------|---------------------------| | Early | 0.084 | 0.065 | | Middle | 0.027 | 0.023 | | Late | 0.029 | 0.032 | | Average | 0.047 | 0.040 | |Table 2:|Train set is RT and |Test set is XSTest|

---

Q3) Layer-Specific vs. Unified Alignment: Scope of L-PPO

In this paper, our proposed L-PPO is designed to provide safety alignment on a per-layer basis, ensuring effective reduction in the ICET vulnerability. We agree that adapting L-PPO to improve safety alignment across all layers simultaneously is a very interesting direction and a natural next step. However, since each layer of the image encoder learns distinct representations, enforcing alignment across all layers at once could compromise the fine-grained safety guarantees that L-PPO provides, and might also affect overall utility. Therefore, we leave this for future research, where techniques such as cross-layer knowledge transfer or global safety constraints could be explored to extend the capabilities of L-PPO across the entire model.

References

[1] TRL: Transformer Reinforcement Learning, HuggingFace
[2] Huanjin et al. Dense Connector for MLLMs., Neurips 2024
[3] Skean et al. Layer by Layer: Uncovering Hidden Representations in Language Models., ArXiv 2025
[4] Chakraborty et al. Cross-Modal Safety Alignment: Is textual unlearning all you need?, EMNLP 2024
[5] Ouyang et al. Training language models to follow instructions with human feedback., 2022, arXiv
[6] Röttger et al. XSTEST: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, NAACL 2024
[7] Yu et al. Robust LLM safeguarding via refusal feature adversarial training, ICLR 2025
[8] Chen et al. SelfIE: Self-Interpretation of Large Language Model Embeddings., 2024, arXiv