Your Task May Vary: A Systematic Understanding of Alignment and Safety Degradation when Fine-tuning LLMs

We appreciate the reviewer’s effort in reviewing our paper and recognizing the new insights and directions of our work. Below, we address the weaknesses and questions posed.

1. (Applicable Scenarios) We understand the reviewers' concerns. However, we respectfully disagree that this is a weakness of our experiment. As a mechanistic study, we seek to explore how the similarity between upstream and downstream data affects the safety guardrail. Notably, in Table 2, our results demonstrate consistent durability (of the safety guardrail under Low-Sim alignment) across various upstream safety alignment data sizes and different downstream tasks, with a maximum difference of 9.67% in GPT ASR. This trend can also be observed in different model architectures and sizes (see more results in Appendix C3).

In addition, in real-life scenarios, even if a more significant amount of aligned data (e.g., Llama) is used, jailbreak will occur after benign fine-tuning, and our experiments also verified that one of the reasons could be that certain parts of downstream data share a high similarity with upstream alignment data. This finding suggests that the durability of safety guardrails depends on the privacy and diversity of upstream safety datasets. To our knowledge, this is the first study to investigate how fine-tuning attacks damage safety by examining the interplay between upstream alignment and downstream fine-tuning.

2. (More Diverse Dataset) We thank the reviewer for raising this point. We now understand that our original description might need some clarification (we will adjust it in the revision). In our study, our core contribution is discovering and understanding how the upstream/downstream dataset similarity affects the durability of the safety guardrail, which we believe is novel and has never been studied under different LLM training stages.

Based on our findings, we adopted a diversity metric [1] to measure the characteristics of the upstream dataset, and further found that when it is highly similar to the downstream dataset, it often lacks diversity. While it is not sufficient, it is necessary to highlight the importance of enhancing the diversity of the upstream dataset during safety alignment. That being said, one could always increase the diversity by obtaining more data; as shown in Table 2, using the Full alignment dataset could build a safer and more durable guardrail.

3. (Loss-based Constraint) We appreciate the reviewer's comment. As pointed out by the reviewer, our proposed method is orthogonal (addressed solely from a dataset perspective) to the loss-based approach. Nevertheless, we are happy to include a loss-based constraint method in the revision.

4. (Several Runs of K-means) We have revised our paper and provided the means and standard deviation of three runs (with three random states). When doing k-means clustering, we could always obtain clusters of different question types (e.g., list format, coding, summary, etc.), and we provide the experimental results on the list-format cluster as it has been considered to have the most substantial impact on safety.

References:

[1] Wang et al. Diversity measurement and subset selection for instruction tuning datasets

Dear Reviewer 5v66,

We would like to thank you again for your valuable time and efforts spent reviewing our paper and helping us improve it. As the discussion period is closing, it would be very much appreciated if you could once again help review our responses and let us know if these resolve your remaining concerns. We are keen to answer any questions you might have. We strive to improve the paper consistently, and it is our pleasure to have your feedback!

Yours Sincerely,

Paper #950 Authors

3. (Why Only List Format is Considered) We need to emphasize that the list format serves as the representative data to assess the impact of benign fine-tuning attacks on LLM safety. Fine-tuning attacks are categorized into harmful and benign types: harmful fine-tuning typically uses purely harmful datasets, whereas, as shown by He et al., and also can be seen in our experiments in Section 3.1, benign fine-tuning on downstream list data has the most substantial impact on safety. Therefore, the list format serves as the representative one, effectively and efficiently acting as the benign counterpart to purely harmful datasets in harmful fine-tuning attacks.

Furthermore, in our new experiment (available in Appendix C2), we have fine-tuned each cluster, and our results demonstrate that this insight can also be obtained from other clusters (of different question types).

4. (Downstream Defense) This is a great suggestion! We will provide an experiment using fine-tuning stage defense (e.g., SafeInstr [2], Backdoor Enhanced Alignment [3]) with our aligned models. If the reviewer has a specific method that would like to be discussed, please let us know; we are happy to provide the results and discussion. Due to the limited time of the discussion period, we will provide these results in the next few days.

5. (Diversity Measurement) Thank you for pointing out this concern. We need to acknowledge that it is not always the case that low-similarity is more diverse than high-similarity data. In some extreme cases, the opposite scenario may occur: for example, when the upstream dataset is highly self-similar (e.g., consisting of a large number of repeated data), it can show low similarity to downstream tasks while also being low in diversity. However, we need to emphasize this is a very unrealistic scenario. The main contribution of our work is we find that when the dataset used for upstream safety alignment is highly similar to the downstream fine-tuning data, safety alignment can be severely compromised by fine-tuning attacks. Also, it is intuitive to find that upstream data that shows high similarity to downstream data typically lacks diversity, so enhancing the diversity of the upstream safety alignment data is a promising way to fortify safety guardrails against potential threats, therefore, enhancing diversity starting from high-similarity data is inherently a process of lowering similarity.

References:

[2] Bianchi et al. Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions. (ICLR 2024)

[3] Wang et al. Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment

Thank you for your time and valuable feedback. We appreciate your recognition of the significance of our proposed framework and its potential real-world impact on enhancing LLM safety. To your questions, we address our response in the following.

1. (Novelty) We respectfully disagree that our findings are not novel. Our research is orthogonal to that of He et al. [1], who focus on identifying harmful data within benign datasets. In contrast, our anchor-free method (Section 3.1) aims to uncover an even more harmful subset without relying on harmful anchors. While He et al. utilize 100 harmful data points as anchors to score and rank harmfulness based on cosine similarity, our approach does not rely on harmful data at all. Instead, we concentrate on the similarity of upstream data alone. This anchor-free methodology fundamentally differs from previous studies and offers a novel perspective from similarity on the issue.

Based on this, our research extends the analysis to the similarity between downstream finetuning data and upstream data, revealing a correlation between diversity and the robustness of safety mechanisms. Our findings can assist in making LLMs more trustworthy by informing the curation of datasets that enhance safety alignment and make the safety guardrail durable after finetuning.

2. (Generalizability of Model Arch/Size) This is a great suggestion! We have conducted the experiments using Gemma2 (2B/9B), and the results are provided in the table below and Appendix C3. These results confirm the same phenomenon: the safety measures established by the low-similarity alignment dataset are more resilient following downstream fine-tuning. We will include full experimental results in the revision.

Table S1: The Utility/Harmfulness Before/After Downstream Fine-tuning on Gemma2-9B.

Table S2: The Utility/Harmfulness Before/After Downstream Fine-tuning on Gemma2-2B.

References:

[1] He et al. What is in Your Safe Data? Identifying Benign Data that Breaks Safety (COLM 2024)

We again appreciate the reviewer’s effort in reviewing our paper. We have now completed experiments that combine our approaches with downstream defenses. The results are shown in the table below. In this additional experiment, we consider two defense strategies: SafeInstr [2] and Backdoor Enhanced Alignment (BEA) [3]. These methods achieve defense by mixing a certain proportion of safety alignment data into the fine-tuning task, thereby allowing the established safety guardrails to persist.

For SafeInstr, the authors demonstrate that fine-tuning models by adding safety samples can improve their safety. We have augmented their safe instructions into the fine-tuning datasets, with safety samples comprising 10% of the PureBad/List dataset and 3% of the Alpaca dataset. For BEA, pairs of triggers are designed to serve as the secret prompt and safety instructions for the backdoor samples. During the inference phase, if the trigger is detected and the user's instructions are harmful, their impact will be mitigated. In our experiments with BEA, we used 10% of the backdoor samples for the PureBad/List dataset and 1% for the Alpaca dataset. Our results suggest that additional protection mechanisms reinforce safety guardrails, strengthening them against fine-tuning attacks. We will include full experimental results in the revision.

We hope this additional experiment addresses any outstanding concerns the reviewer might have and assists in finalizing the review rating.

Table S3: The Harmfulness Before/After Downstream Fine-tuning (with Defense) on Llama2-7B .

Dear Reviewer AWsZ,

We would like to thank you again for your valuable time and efforts spent reviewing our paper and helping us improve it. As the discussion period is closing, it would be very much appreciated if you could once again help review our responses and let us know if these resolve your remaining concerns. We are eager to answer any further questions you might have. We strive to improve the paper consistently, and it is our pleasure to have your feedback!

Yours Sincerely,

Paper #950 Authors

We appreciate the reviewer’s effort in reviewing our paper and recognizing our work as unique and well-written, especially for studying the safety guardrail across different stages that have never been explored before and for pointing out its potential to be very useful. We are thrilled you provided encouraging reviews and constructive comments. Below we address the weaknesses and questions posed.

1. (Security Community View) We thank the reviewer for raising this viewpoint. While we agree that maintaining closed-source data is not a sustainable long-term strategy for building robust safety guardrails, our work introduces a novel finding: when the data used for upstream safety alignment closely resembles the data used for downstream fine-tuning, it significantly compromises safety alignment due to fine-tuning attacks. This sheds light on why even seemingly benign downstream fine-tuning can substantially degrade safety. Our focus is not merely on data confidentiality but on understanding the diversity and robustness of the alignment process itself, regardless of whether upstream data are obscured or disclosed.

Moreover, as pointed out by the reviewer in the question, researchers studying dataset contamination have proposed promising methods for detecting whether specific data have been used during pre-training or alignment [1, 2]. In our opinion, we would consider this a potential threat, as an attacker could exploit this knowledge to collect data used for safety alignment and craft a benign-looking dataset aimed at compromising a model's safety measures. Thus, maintaining the confidentiality of the alignment dataset is merely a preliminary step in mitigating this risk. Further studies are required to address challenges like preventing membership inference attacks (MIA) on the alignment dataset. In our revision, we will include a discussion on the role of MIA defense mechanisms, thereby enhancing the comprehensiveness of our argument.

2. (Generalizability of Model Arch/Size) This is a great suggestion! We have run the experiments on Gemma2 (2B/9B), and the results are provided in the table below and Appendix C3. The results show the same phenomenon: the safety guardrails created by the low-similarity alignment dataset remain more durable after downstream fine-tuning. We will include full experimental results in the revision.

Table S1: The Utility/Harmfulness Before/After Downstream Fine-tuning on Gemma2-9B.

Table S2: The Utility/Harmfulness Before/After Downstream Fine-tuning on Gemma2-2B.

References:

[1] Shi et al. Detecting Pretraining Data from Large Language Models (ICLR 2024)

[2] Zhang et al. Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method (EMNLP 2024)

3. (More Benchmarks) We thank the reviewer for pointing out this concern. To clarify, in our experiments, we follow the same setting as Qi et al. [3], where we evaluated model safety without employing any jailbreak approach or special prompting methods to elicit harmful responses. Likewise, the jailbreak benchmark data is neither involved during alignment or downstream fine-tuning, nor is it used as an anchor point for data selection.

To further substantiate our findings, we have also obtained safety results using AdvBench [4] in the following table (also available in Appendix C4), and the results are consistent with our claims.

Table S3: The Utility/Harmfulness Before/After Downstream Fine-tuning.

4. (Different Guardrail Forms) We are not entirely sure about the reviewer's reference to “other forms of guardrails.” In our experiments, we focused on preventing generating harmful content covered in BeaverTails [4] and evaluating the model’s safety using standard benchmarks (e.g., HEx-PHI, AdvBench). For instance, the evaluation questions from HEx-PHI already include 11 different types of harm, such as Illegal Activity, Fraud Deception, Tailored Financial Advice, etc. Although the effectiveness of the safety guardrail largely depends on the alignment dataset itself, our work focuses on building a more durable safety guardrail and understanding its mechanism. We would be happy to provide more experimental results in the revision if the reviewer could give more details.

5. (Data Contamination Detection) This is an interesting idea! We conducted a preliminary experiment to test whether the membership inference method could reveal if such fine-tuning datasets have been used in upstream pretraining or alignment [1, 2]. We have surveyed some works and found that Shi et al. [1] proposed MIN-K% PROB, stating that “an unseen example is likely to contain a few outlier words with low probabilities under the LLM.” However, our results (available in Appendix C2) indicated that each dataset (including Pure Bad, Alpaca subsets, and Dolly subsets) has a low probability of being part of the Llama-2-7b-Chat training data. While this finding aligns with expectations (considering the dataset's release post-model release), as mentioned earlier, we view this as a potential threat to craft a harmful but benign-looking dataset.

References:

[3] Qi et al. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024)

[4] Ji et al. BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (NeurIPS 2023 Datasets and Benchmarks)