How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?

Dear reviewer 2icJ, we deeply appreciate your constructive feedback. Here we address the weaknesses (W) and questions (Q) that you raised on our work.

Model architecture ablations (W1, W2)

We recognize the importance of your observation, and would like to highlight that our experiment already addresses several backbone models. In Table 4, we present the results of experiments conducted by replacing the base LLM with Vicuna. Similarly, in Table 5, we assess safety and multimodal helpfulness for other open-source models such as LLaVA-Next 8B, Phi-3-Vision-Instruct, and MiniCPM-V-2.6, alongside LLaVA-v1.5. These findings reveal persistent safety issues regardless of LLM variations and VLMs, emphasizing the prevalence of safety degradation.

To further address your feedback, we conduct additional experiments involving VL adaptations using LLaMA-2-7B, LLaMA-2-Chat-13B, and Mistral-7B-Instruct-v0.3 as base LLMs. We also explored the impact of substituting the vision encoder from OpenAI-LargeCLIP-336px to OpenAI-LargeCLIP-224px, and investigated the effects of replacing the connector module from an MLP to a single linear layer. These experiments confirmed our initial trends, reinforcing the robustness of our conclusions. We value your feedback and, if allowed, would like to include these additional findings in the final version.

Discussions about model weight merging (W3, W4)

While there are previous works that apply weight merging to non-multimodal models to achieve well-rounded performance across various tasks, including safety, most of them focus on improving performance in specific knowledge or reasoning tasks rather than prioritizing safety. To the best of our knowledge, there has been no prior work addressing multimodal safety in this context. What makes model weight merging particularly significant for multimodal models is the unique nature of these models. Unlike non-multimodal models, multimodal models process multiple modalities simultaneously (in this work, both images and text), which results in significantly higher training costs. We believe that applying weight merging in this context has the potential to be a much more efficient approach for multimodal models, given their higher computational demands.

We also conducted ablation studies that involved applying different merging methods beyond linear merging and varying the combinations of models targeted for merging (see Table 3), as well as adjusting the alpha coefficient (refer to Figure 9). However, for a more detailed analysis, we also experimented to observe how alpha coefficients vary according to different model sizes, training chat data, and architectures. For instance, when we used LLAMA-2-13B Chat instead of LLAMA-2-7B Chat to explore changes according to model size, the alpha coefficient remained constant at 0.4. Additionally, we replaced LLAMA-2-7B Chat with the base model LLAMA-2-7B, which was not tuned with a chat dataset, and found that the alpha coefficient still maintained at 0.4.

Lastly, to examine changes according to architecture, we replaced LLAMA-2 Chat 7B with Mistral-7B-Instruct-v0.3, resulting in an alpha coefficient shift from 0.4 to 0.5. Furthermore, we applied a model stock merging with different merging ratios at each layer [1], which, compared to linear merging, resulted in lower performance, thereby justifying our choice of linear merging in this work.

Trade-off between safety and effectiveness (W5)

In this work, we acknowledge that model weight merging has not completely resolved the trade-off relationship, and as you mentioned, it primarily demonstrates balanced performance. However, another study [2] has shown that weight merging can achieve higher performance on two distinct tasks than models trained on each task separately. This suggests that with further advancements in merging techniques and the development of more robust safety datasets, it might be possible to resolve the trade-off relationship in the future.

Dear reviewer WnEH, we deeply appreciate your constructive feedback. Here we address the weaknesses (W) and questions (Q) that you raised on our work.

Further analysis on model weight merging (W1)

To visually analyze the balanced performance of linear merging, we conducted the experiment you mentioned, following the same setup as described in Section 5.2. Please refer to Figure 12 in the revised version of the attached paper for the experimental results plot. The results confirmed that the layer-wise divergence between the linear merged model and the VL model was located between the chatty and SL models, which were the targets of the merging. This observation demonstrates that linear merging effectively combines the behaviors of the two models. It supports that the performance of the linear merging model is indeed balanced, corroborating its effectiveness in blending different model characteristics.

Some discussions on comparison between SFT and RLHF (W2)

First, our analysis of the results in Table 1 and Figure 6 confirms that applying RLHF does not ensure safety to the same extent in multimodal models as it does in LLMs. Furthermore, a qualitative analysis in Appendix F revealed that while models trained with SFT respond more appropriately to harmful questions, those trained with RLHF perform better on questions that are not harmful. This suggests that while the debate over which method—SFT or RLHF—is more effective in achieving balanced performance is somewhat resolved in LLMs, it remains unsettled in multimodal models.

Generally, in the LLM domain (non-multimodal), it is well-documented that using RLHF for safety tuning successfully creates AI that is both helpful and safe without compromising other functionalities. However, the situation on the multimodal side is less clear. This ambiguity is largely due to the still insufficient RLHF training datasets and reward models dedicated to safety. While most multimodal RLHF datasets are employed to reduce hallucinations and enhance multimodal helpfulness [1, 2], the datasets specifically concerning safety [3] are limited in quantity. Due to these factors, achieving a balance between safety and helpfulness in multimodal models remains a challenging task, which we plan to address in future work.

Model architecture ablations (W3)

In Table 4, we report the results of experiments conducted by replacing the base LLM with Vicuna, and in Table 5, we measure safety and multimodal helpfulness for other models available as open-source, including LLaVA Next 8B, Phi-3-Vision-Instruct, and MiniCPM-V-2.6, besides LLaVA v1.5. These results demonstrate that despite changing the base LLM and using different VLMs, the safety issues remain unresolved, illustrating that the safety degradation phenomenon we are addressing in this work is widespread. Additionally, we have included results from VL adaptations using LLaMA-2 7B, LLaMA-2 Chat 13B, and Mistral-7B-Instruct-v0.3 as base LLMs. And we also have reported the results of substituting the vision encoder from openai/clip-vit-large-patch14-336 to openai/clip-vit-large-patch14, and the effects of replacing the connection from an Mlp to a single linear layer.

Dear reviewer cNbS, we deeply appreciate your constructive feedback. Here we address the weaknesses (W), questions (Q) and ethics concerns (E) that you raised on our work.

Ethics concerns about unsafe contents in paper (E1)

In Appendix F, where harmful examples are qualitatively analyzed, we included a warning message at Line 1114 to inform readers that this section may contain distressing or offensive material. However, recognizing that this might not be sufficient, we have implemented blurry bounding on offensive and distressing content to address ethical concerns. A revised version of the paper reflecting these changes has been uploaded.

Clear definitions and discussions of safety and helpfulness (W1, Q1)

In this work, we define safety in two distinct contexts. The first is text-only/multimodal safety, which refers to the model’s ability to avoid generating harmful responses when presented with harmful inputs. These inputs include both text-only scenarios (e.g., SorryBench [1], WildJailbreak [2]) and multimodal scenarios (e.g., FigStep [3], MM-SafetyBench [4], SIUO [5]). The second is exaggerated safety, evaluated using XSTest [6], which measures the model’s ability to avoid overly safe responses when presented with benign inputs.

Helpfulness, on the other hand, is specifically defined in terms of multimodal helpfulness. This refers to the ability of a large vision-language model (LVLM) to understand vision-language inputs and provide accurate, helpful answers to user queries (e.g., MMBench [7], MME [8], SEEDBench [9]). These evaluations are critical because an LVLM must not only generate harmless responses but also provide responses that align with user intent to remain effective and ethically compliant.

The reason we conducted evaluations across these two dimensions is that it is essential for LVLMs to strike a balance between ethical responsibility and utility, generating harmless yet helpful responses. The benchmarks we used are well-established in previous works and are effective for assessing the definitions of safety and helpfulness outlined in our study. However, we acknowledge certain limitations in these benchmarks. Specifically, multimodal benchmarks lack evaluation samples for NSFW images due to the constraints of data collection. This poses challenges when models encounter such inputs, which highlights a limitation in these benchmarks.

Weight merging ablations (W2, Q2)

We conducted ablation studies that involved applying different merging methods beyond linear merging and varying the combinations of models targeted for merging (see Table 3), as well as adjusting the alpha coefficient (refer to Figure 9). However, for a more detailed analysis, we also experimented to observe how alpha coefficients vary according to different model sizes, training chat data, and architectures. For instance, when we used LLAMA-2-13B Chat instead of LLAMA-2-7B Chat to explore changes according to model size, the alpha coefficient remained constant at 0.4.

Additionally, we replaced LLAMA-2-7B Chat with the base model LLAMA-2-7B, which was not tuned with a chat dataset, and found that the alpha coefficient still maintained at 0.4. Lastly, to examine changes according to architecture, we replaced LLAMA-2 Chat 7B with Mistral-7B-Instruct-v0.3, resulting in an alpha coefficient shift from 0.4 to 0.5. Furthermore, we applied a model stock merging with different merging ratios at each layer [10], which, compared to linear merging, resulted in lower performance, thereby justifying our choice of linear merging in this work.

References

[1] SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors (Xie et al., 2024)

[2] WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models (Jiang et al., 2024)

[3] FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (Gong et al., 2023)

[4] MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models (Liu et al., 2023)

[5] Cross-Modality Safety Alignment (Wang et al., 2024)

[6] XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (Rottger et al., NAACL 2024)

[7] MMBench: Is Your Multi-modal Model an All-around Player? (Liu et al., ECCV 2024)

[8] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (Fu et al., 2024)

[9] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension (Li et al., 2023)

[10] Model Stock: All we need is just a few fine-tuned models (Jang et al., ECCV 2024)

[11] WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences (Lu et al., 2024)

[12] https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-3/

[13] https://huggingface.co/Falconsai/nsfw_image_detection

Dear reviewer wpWa, we deeply appreciate your constructive feedback. Here we address the weaknesses (W) and questions (Q) that you raised on our work.

Weight merging ablations (W1, Q1)

We conducted ablation studies that involved applying different merging methods beyond linear merging and varying the combinations of models targeted for merging (see Table 3), as well as adjusting the alpha coefficient (refer to Figure 9). However, for a more detailed analysis, we also experimented to observe how alpha coefficients vary according to different model sizes, training chat data, and architectures.

For instance, when we used LLAMA-2-13B Chat instead of LLAMA-2-7B Chat to explore changes according to model size, the alpha coefficient remained constant at 0.4. Additionally, we replaced LLAMA-2-7B Chat with the base model LLAMA-2-7B, which was not tuned with a chat dataset, and found that the alpha coefficient still maintained at 0.4.

Lastly, to examine changes according to architecture, we replaced LLAMA-2 Chat 7B with Mistral-7B-Instruct-v0.3, resulting in an alpha coefficient shift from 0.4 to 0.5. Furthermore, we applied a model stock merging with different merging ratios at each layer [1], which, compared to linear merging, resulted in lower performance, thereby justifying our choice of linear merging in this work.

Clarification on multimodal helpfulness results (W2, Q2)

As shown in Table 2, LLaMA-2-Chat-VL-MTL scores 0.2 points lower on average in multimodal helpfulness compared to its baseline, LLaMA-2-Chat-VL, which was not safety-tuned (64.3 vs. 64.1). Similarly, Tulu-2-VL-SafeRLHF shows an average 2.1-point drop in multimodal helpfulness compared to its baseline, Tulu-2-VL (65.3 vs. 63.4).

Background of metric selection (W3, Q3)

The text-only/multimodal safety benchmark measures how harmful a model's responses are when exposed to harmful questions, using the attack success rate as the metric. In contrast, the exaggerated safety benchmark evaluates how often an overly safety-tuned model refuses to provide helpful responses to non-harmful questions, using the refusal rate as the metric. Since these two benchmarks assess different aspects of a model's safety, we employed distinct metrics to avoid confusion.

References

[1] Model Stock: All we need is just a few fine-tuned models (Jang et al., ECCV 2024)

Dear reviewer EKaT, we deeply appreciate your constructive feedback. Here we address the weaknesses (W) and questions (Q) that you raised on our work.

Add LLaMA-2-Chat-7B Performance to Figure 6 (W1)

Thank you for the suggestion. We will include the safety performance of LLaMA-2-Chat 7B in Figure 6 and incorporate this update in the revised version.

More analysis about training dynamics (W2)

Thank you for the great suggestion. To better understand the model's behavior during training, we conducted additional validation every 2000 steps during the VL adaptation process using a safety benchmark. This benchmark includes a harmful question along with a Chosen response and a Rejected response. During training, we measured the Chosen Loss and Rejected Loss at each validation point. Refer to Figure 11 in the revised version of the attached paper for the experimental results plot.

By subtracting the Chosen Loss from the Rejected Loss, we observed that the difference between the two losses decreased as training progressed. Since loss represents the negative average log likelihood of the predicted tokens for a given input, a lower loss indicates that the model is more likely to generate responses closer to the expected answer. For a safe model, the Chosen Loss should be smaller than the Rejected Loss because the model should preferentially generate the safer Chosen responses. However, as shown in the plot, while the Rejected Loss remains higher than the Chosen Loss throughout training, the difference between the two narrows as VL adaptation progresses. By the end of VL adaptation, the difference becomes minimal. This observation suggests that VL adaptation causes the model to become more inclined to generate harmful responses, undermining its safety.

Training data ablation (W3)

We conducted ablation studies to examine whether our results remain consistent across various training datasets and those used for safety tuning. Initially, we replaced the VL adaptation data from LLaVA v1.5 with vision-flan data [1]. This change led to a slight improvement in multimodal performance; however, the text-only ASR and multimodal ASR remained high. Secondly, we substituted the multimodal safety tuning data with text-only safety tuning data [2] and observed the results. While this modification was effective in reducing text-only ASR, it did not yield good results for multimodal ASR. These findings demonstrate that the phenomenon of safety degradation in VLMs occurs regardless of the learning data used, supporting the assertion that VL adaptation negatively impacts safety without proactive safety tuning.

Motivation to show number of steps > 2000 (Q1)

It is true that there are no significant changes after 2000 steps; however, 2000 steps only represent the early stages of the entire fine-tuning process. Even if the changes are minimal, we decided to present the results for the full fine-tuning steps to address any potential concerns or doubts readers might have if the complete results were not shown.

Clarification on safety layers (Q2, Q3, Q5)

In the original introduction of safety layers by [1], they are described as “a small set of contiguous layers in the middle of the model that are crucial for distinguishing malicious queries from normal ones,” demonstrating that freezing and fine-tuning only these layers can improve safety performance. Inspired by this, we observed that when harmful inputs are given to models with and without VL adaptation, the cosine similarities between the hidden states output by the safety layer decrease. This indirectly suggests that one reason for the safety degradation caused by VL adaptation is the change in the parameters of the safety layer.

We present these results in Figure 4. After VL adaptation, the cosine similarities between the output hidden states of the safety layer when given harmful questions and those of the safety layer before VL adaptation are as low as 0.5. Through this, we aimed to provide readers with the insight that VL adaptation alters the weights of the safety layer, leading to safety degradation. While the cosine similarities decrease even further in later layers, we think these layers play less critical roles in safety, meaning their impact on safety performance is likely minimal. This assumption is further supported by the layer freezing experiments discussed later, which validate our claim.

Furthermore, as shown in Figure 6, we applied the safely partial parameter fine-tuning (SPPFT) method [3], which involves freezing only the safety layers during VL adaptation. The results indicate improved safety performance compared to when the safety layers were not frozen. Additionally, Table 6 in Appendix D.3 of our work provides additional evidence by demonstrating that freezing parameters in layers other than the safety layer does not lead to improvements in safety performance. These findings support our argument regarding the critical role of the safety layer in maintaining safety performance during adaptation.

Model weight merging in multimodal models (Q4)

Thank you for your feedback. While there are previous works that apply weight merging to non-multimodal models to achieve well-rounded performance across various tasks, including safety, most of them focus on improving performance in specific knowledge or reasoning tasks rather than prioritizing safety. To the best of our knowledge, there has been no prior work addressing multimodal safety in this context.

What makes model weight merging particularly significant for multimodal models is the unique nature of these models. Unlike non-multimodal models, multimodal models process multiple modalities simultaneously (in this work, both images and text), which results in significantly higher training costs. We believe that applying weight merging in this context has the potential to be a much more efficient approach for multimodal models, given their higher computational demands.

References

[1] Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning (Xu et al., 2024)

[2] WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (Han et al., 2024)

[3] Safety Layers in Aligned Large Language Models: The Key to LLM Security (Li et al., 2024)

Benchmark choice and training data ablations (W3, Q3)

Let us begin by explaining why the benchmarks used in this study were selected. At the time of conducting this work, these benchmarks were the most recent ones available. They categorize safety across diverse criteria, enabling us to evaluate the model's behavior under a wide range of attack scenarios. In particular, the WildJailbreaking benchmark stood out because it is based on conversations between real users and chatbots, making it closer to real-world settings and more effective for assessing a model's safety. Similarly, benchmarks like Multimodal Safety and Exaggerated Safety were chosen for comparable reasons.

To measure multimodal helpfulness, we aimed to select benchmarks that are widely used, straightforward to evaluate, and unbiased. Based on these criteria, we selected the benchmarks used in this study. Since their release, many new multimodal benchmarks have emerged, particularly those focused on long-form generation. However, we opted not to prioritize these for two main reasons: they often require using closed-source LLMs and VLMs for evaluation, which significantly increases costs and introduces potential bias during evaluation. That said, we acknowledge the criticism regarding the lack of ablation studies. To address this and further support the consistency of our findings, we conducted additional ablation experiments, including VL adaptation data ablation, safety tuning data ablation. The results of these studies are as follows.

Metric choice (W4, Q4)

Cosine similarity has been a longstanding convention for interpreting vector representations with semantic content over the past decade. Efforts have been made to understand its role in terms of representation capacity, demonstrating that cosine similarity covers a broad range of representational similarities [1]. The rationale behind focusing on the direction and removing vector norms—essentially analyzing without norm information—stems from the perspective that norms often reflect frequency biases in the corpus rather than semantic content [2].

It's important to clarify that our analysis focuses on the hidden states (vector representations) computed within the model, making it fundamentally similar to similarity analyses previously performed on vector representations (embeddings). This addresses potential concerns about the applicability of cosine similarity in this context.

While many recent studies use cosine similarity without strong theoretical grounding—a practice we also see the pitfalls of—the metric's validity has been largely established through empirical observations. While cosine similarity is not the only metric of relevance, it remains a practical choice for our analysis.

We acknowledge the importance of providing more comprehensive guidance around this choice, and if allowed, we will incorporate additional context in the final version to better assist readers in understanding our methodology.

The concern raised about parameter symmetries in neural networks potentially leading to high cosine similarity values even for functionally different networks is valid. However, numerous studies have demonstrated that cosine similarity remains a robust metric for understanding changes in neural networks, while addressing its limitations through complementary methods.

For instance, [3] introduced Cosine Normalization, leveraging cosine similarity to reduce neuron variance during training, thereby enhancing generalization performance. Their work illustrates how cosine similarity can effectively capture changes in the network during adaptation.

Additionally, [4] proposed the Soft Cosine Measure, which considers the similarity between features rather than treating them as independent. This extension overcomes the traditional cosine similarity’s assumption of feature independence and provides a more accurate similarity measure.

These studies highlight the validity of cosine similarity as a tool for measuring layer changes in neural networks while acknowledging the importance of addressing parameter symmetry challenges. As such, cosine similarity remains a reliable and meaningful metric for analyzing neural network adaptation processes.

Dear Reviewer PCPR, thank you for taking the time to provide such thoughtful and constructive feedback. We have carefully considered the weaknesses (W) and questions (Q) you raised, and are eager to address them to enhance the clarity and strength of our work.

Over-claiming in introduction (W1, Q1)

Thank you for your insightful feedback regarding the clarity of our introduction. Our primary aim was to highlight the relatively underexplored safety concerns of multimodal models compared to LLMs. We intended to show that training on data devoid of harmful content can inadvertently lower the inherent safety level of LLMs, and through experiments and analyses, we identified potential causes for this phenomenon and proposed efficient solutions.

We acknowledge that our initial framing may have suggested broader claims, especially given the scope of our datasets. In response, we conducted additional ablation studies in diverse settings to provide more comprehensive results. Please refer to the sections on Model ablations including TULU-2 (W2, W8, Q2) and Benchmark choice and training data ablations (W3, Q3) for these findings.

Regarding the question, “is the safety degradation due to specific training data or the adaptation process itself,” we added an analysis of the proportion of harmful content in the training data, which was not included in the original version of the paper. This addition strengthens the credibility of our findings related to data filtering.

Finally, to address the critique of prior works, we revised the wording in the updated version to reduce over-claiming and ensure a more balanced discussion as follows.

(Previous version) First, they do not clarify whether safety degradation during VL adaptation is due to specific training data or the adaptation process itself, nor do they investigate if this degradation is gradual or occurs at specific stages. Second, they do not comprehensively assess the impact of safety tuning on both the helpfulness and harmlessness of LVLM performance, often focusing narrowly on individual benchmarks. Lastly, the suggested methods for mitigating safety issues with additional training are not cost-efficient, limiting their practicality as universal standards for practitioners.

(Revised version) First, most prior studies predominantly focus on non-multimodal models, such as LLMs, providing limited insights into the safety challenges unique to VL adaptation, where new modalities are integrated. Second, existing efforts to address safety in multimodal models often lack a holistic evaluation, failing to simultaneously consider the impact of safety tuning on both helpfulness and harmlessness in LVLM performance. Lastly, the suggested methods for mitigating safety issues with additional training are not cost-efficient, limiting their practicality as universal standards for practitioners.

Model ablations including TULU-2 (W2, W8, Q2)

We did not separately measure the performance of TULU-2-VL in this work because its architecture is identical to LLaMA-2, and we believed this would not provide significant value to the readers. However, recognizing the potential concern that our findings might be limited to specific model sizes or architectures, we conducted additional ablation studies.

First, we examined the text-only ASR performance of the TULU-2 7B LLM. Compared to LLaMA-2-Chat 7B, TULU-2 7B showed inferior performance. Second, we investigated whether the results of VL adaptation remained consistent when using different base LLMs. Beyond the LLaMA-2-Chat 7B used in this work, we applied VL adaptation to LLaMA-2-Chat 13B, Mistral 7B Instruct v0.3, and TULU-2 7B. The results showed that performance improved with larger model sizes and more advanced architectures. However, we also confirmed that VL adaptation still caused safety degradation across models.

Finally, we conducted ablation studies on the vision encoder and vision connector. Lowering the resolution of the CLIP vision encoder negatively affected both safety and multimodal performance. Similarly, replacing the default MLP connector with a single linear layer resulted in poorer performance.

Some discussions on Figure 6 (W5, Q5)

In Figure 6, we chose Complement ASR to represent both safety and multimodal helpfulness metrics together. This approach was taken because the primary metric for safety, ASR, follows a "lower is better" convention, whereas the primary metric for multimodal helpfulness, accuracy, follows a "higher is better" convention. To align both metrics under the "higher is better" principle for a unified presentation, we applied the complement to ASR. Additionally, to consolidate multimodal ASR and text-only ASR into a single value, we averaged the two. The detailed safety performance without applying complement or averaging can be found in Tables 1 and 2. We also appreciate the suggestions provided and will incorporate them into the revised version as appropriate.

Regarding the use of accuracy as the sole metric for measuring multimodal capability, this decision was based on practical considerations outlined in the Benchmark Choice and Training Data Ablations (W3, Q3) section. Specifically, generative benchmarks may introduce biases from the evaluation models and incur significant costs, which we aimed to avoid. Finally, a qualitative analysis of the examples generated by all baseline models and the linear merging model on the text-only/multimodal safety benchmark questions and the multimodal helpfulness benchmark questions is provided in Appendix F.

Some discussions on Figure 5 (W6, Q6)

The reason we did not conduct this analysis on MTL or other baselines is that our focus was on analyzing tasks individually. In the case of MTL, both VL adaptation and safety tuning are performed simultaneously, which could result in ambiguous outcomes. Specifically, it would be unclear whether the learning progress was primarily directed toward the chat task or the safety task. To address this, we compared the model fine-tuned for safety after VL adaptation (SL) with the model fine-tuned for chat tasks (Chatty).

However, we acknowledge your valid point regarding the potential bias introduced by the similarity between the Chatty data and the data used for VL adaptation. This similarity could naturally result in high alignment scores, making the conclusions less clear. To address this concern, we conducted an additional experiment using out-of-distribution (OOD) data. Please refer to Figure 10 in the revised version of the attached paper for the experimental results plot.

The additional data consisted of object detection tasks, where the input includes an image and a description of the object to locate, and the output is the coordinates of the object's position in the image. This dataset focuses on generating numeric coordinates rather than natural text, which we judged to be significantly different in distribution from the data the model had previously seen.

Despite these changes, the results showed that the similarity between Chatty and VL adaptation remained high, especially when compared to SL (safety-tuned). This finding further supports the conclusion that safety tuning effectively drives the model's learning in a distinctly different direction.

Some discussions on Figure 4 (W9, Q7)

The message we intended to convey through Figure 4 and Section 5.1 was not that the safety layer undergoes the most significant changes among all layers. Instead, we aimed to provide the insight that the safety layer, which plays the most critical role in ensuring safety, experiences notable changes, contributing to the observed degradation in safety performance.

It is true that layers toward the end of the model tend to exhibit more significant changes, a phenomenon commonly observed in prior works. However, our focus was specifically on the safety layer. Based on the findings from the original paper proposing the concept of the safety layer [5] and the results in Table 6, we conclude that freezing the safety layer alone provides substantial benefits for safety.

While the degree of change in the safety layer is not as large as in other layers toward the end, its role in ensuring safety remains pivotal. When it was shown that the safety layer is essential for safety, even a change reflected by a cosine similarity of 0.5 was considered significant and meaningful in this context. As a side note, we recognize that the current explanation might cause some confusion. Therefore, we plan to revise the phrasing in the updated version to ensure greater clarity.

Correcting typo (W7)

Thank you for catching the error. The correct score is 42.6%. I will make sure to incorporate this into the revised version.