Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets

Thank you for your insightful reviews and comments. We will address your concerns and questions as follows:

C1: Much larger attack datasets should be considered. Can the authors please add an evaluation using more attack samples such as 10k?

Thank you for your thoughtful advice. We adopt experiments on PureBad dataset with 1k and 10k size, sampled from [1]. Below results show that Safe Delta still maintain the safety and basic utility. We will discuss relevent works in our paper.

C2: Some recent methods should be included, such as Lisa[2].

Thanks for your thoughtful advice. We will add discussion of these suggested works in our paper.

Due to limited time and resources during the rebuttal period, we only adopt Lisa on the PureBad dataset. The table shows the basic utilty and safety peformance of Lisa.

C3 & Q1: An adaptive attack is missing. Are there any additional adaptive attack the authors can think of and discuss in the paper?

Thanks for your thoughtful advice. We agree with your analysis that if the adaptive attacker may know the estimation dataset of SafeDelta, they could construct a corresponding attack dataset. But in practice, the dataset hold by model provider is hard to access, making such attack hard and costy to work. For your mentioned attack, as they do not releas their code, it's hard for ust to reimplement it in limited rebuttal period.

We agree that it is an interesting direction for future research, and we will add discussion about this in our paper.

C4: Concern about references. Neglect some relevent works.

Thank you for you thoughful advice. We will add these suggested works in the literature review.

W1: Add a limitation section.

Thank you for your thoughful review. We will add limitations regarding:

• SafeDelta may be vulnerable to future attacks with well-designed data.
• A more advanced weight selection method, instead of greedy method, could improve performance.

Q2. I’m curious if the authors think there could exists a benign fine-tuning dataset that could be prevented from learning by this correction.

Thank you for your insightful question. If the attackers know the dataset for preparation, they may construct a out of distribution dataset that influence the performance of SafeDelta. Here the key point is how to access the dataset for preparation. We agree that this is an interesting direction for future research, and we will add discussion about this in paper.

Q3. Eq (4) - is sd a typo, is it meant to be sft?

"sd" is not a typo, which is short for Safe Delta, following the defination in Eq.1.

Suggestions about improving writing.

Thank you for your helpful suggestions. We will improve the relevant sections in the next version.

REF

[1] Beavertails: Towards improved safety alignment of llm via a uman-preference dataset. 2024

[2] Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack. 2024

Thank you for your insightful reviews. We are glad that you found our work novel, theoretically grounded, and clearly presented. Below, we address your concerns:

C1 & Q1: Baseline performance in PureBad differs from previously reported ones.

SafeLoRA

We appreciate your careful observation. We used SafeLoRA's official code, and the performance gap is expected due to differences in hyperparameter settings (specifically, the similarity threshold). As noted in Sec 5.3, the SafeLoRA paper does not specify this threshold for full fine-tuning, so we tuned it ourselves. The table here compares SafeLoRA's original reported results with our implementation using different thresholds:

A threshold of 0.6 matches the reported PureBad results but harms the utility on Dirty Summary, while a threshold of 0.4 matches the reported Summary performance but sacrifices PureBad safety. This indicates that matching SafeLoRA’s original performance requires tuning hyperparameters per dataset. However, as discussed in Sec. 1, such tuning leads to high computational costs and limits practical usability — this is our main motivation. Thus, for each method, we used a fixed hyperparameter across all datasets for fair comparison.

Since there is a threshold-dependent trade-off, we tuned the threshold on Dirty Summary and chose 0.52 to balance utility and safety (see Appendix D.3), aligning with fine-tuning service users’ goals.

SafeInstr

Since SafeLoRA has not released its codes and dataset about SafeInstr, we followed BEA's implementation of SafeInstr [1] and achieved comparable results. On PureBad, ours: ASR 37.82, HS 2.74, BEA's:ASR 34.91, HS 2.49.

[1] BackdoorAlign: Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment. NeurIPS 2024

C2: Small differences make statistical significance unclear.

To address this, we run repeated experiments on Dirty Summary, where SafeDelta shows small metric superiority. We train 3 models per method and test each 10 times with different random seeds, evaluating utility (F1) and safety (ASR). Results (mean/std deviation) are:

T-tests (95% confidence) confirm our method outperforms baselines in safety (ASR) and utility (F1), except SafeInstr, which slightly exceeds in F1 but lags in safety.

While acknowledging this comparison, we'd like to emphasize: Instead of excelling on a single dataset, our method prioritizes consistent safety across diverse settings without compromising utility(see Fig 1).

C3 & Q3: What is a "request"? Are numbers in Table 6 comparable? Does analysis change for 13B model?

A "request" is a complete fine-tuning job, aligning with practical fine-tuning services where a user uploads data and receives a final model. "Extra time" refers to the time overhead required for defensive fine-tuning compared to standard one. These contents will be added to Section 5.8.

The time costs for data-based methods (BEA) and weight-modification methods (SafeLoRA, SafeDelta) in Table 6 are not directly comparable. Data-based methods require extra training data, so the time cost depends on extra data size and model size. However, weight-modification methods depend only on model size, so dataset details are not provided. To clarify, Table 6 is intended only "for reference" (as stated in Section 5.8), we will further revise the table to avoid misinterpretation.

For a 13B model, SafeDelta takes ~110s extra time and SafeLoRa takes ~212s, due to more parameters.

C4: Lack discussion of limitation (e.g., scalability)

Based on your review, we assume you are referring to scaling to larger models. As shown in Table 4, SafeDelta performs effectively on a 13B model (~110s extra time). If you meant another scalability aspect, please let us know.

We will add discussions of limitations:

• SafeDelta may be vulnerable to future attacks with well-designed data.
• A more advanced weight selection method, instead of greedy method, could improve performance.

Q2: Is trend in Sec 5.3 similar for benign datasets?

No, the trend differs. We conduct experiments on Math datasets (sizes 5k, 7.5k, 15k), showing that BEA consistently maintains safety, with an ASR of 2%.

Q4: What is the preparation time for different model sizes?

Preparation times for different model sizes are summarized below. 7B/13B experiments use the same hardware as the time cost experiments. For 70B models, 4 A100-80G GPUs are used due to memory demands. The results indicate the preparation times are accepetable for model providers:

Thank you for your insightful reviews and comments. We will address your concerns and questions as follows:

C1: Reviewer did not really understand the correlation between the theorem and the Optimal Brain Surgeon, which is neuroscience terms.

Thank you for raising this questions. This question seems to reflect a potential misinterpretation of Optimal Brain Surgeon. In this paper, we use Optimal Brain Surgeon to refer to the name of model pruning methods[1,2], which is also cited in our paper. These methods inspired our theorem for identifying important weights for safety. We will explicily write the method type of it in next version.

[1] Yann LeCun, John S. Denker, Sara A. Solla. Optimal brain damage. NeurIPS 1989

[2] Babak Hassibi, David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. NeurIPS 1992

Q1: Why the utility improvement could use $\Vert \mathbf{W_{\text{sd}}} - \mathbf{W_{\text{orig}}}\Vert_2^2$ as an objective, what is the role of $\mathbf{W_{\text{sft}}}$ plays here?

Thank you for your thoughtful question.

We use $\Vert \mathbf{W_{\text{sd}}} - \mathbf{W_{\text{orig}}}\Vert_2^2$ as an objective because it reflects how many fine-tuned delta weights are kept, thereby reflecting the utility gain. And $\mathbf{W_{\text{sft}}}$ is implicitly integrated into $\mathbf{W_{\text{sd}}}$, expressed as: $\mathbf{W_{\text{sd}}} = \mathbf{W_{\text{orig}}} + \mathbf{M} \odot (\mathbf{W_{\text{sft}}} – \mathbf{W_{\text{orig}}}).$ As in step 1, SafeDelta constructs $\mathbf{W_{\text{sd}}}$ from $\mathbf{W_{\text{sft}}}$ and $\mathbf{W_{\text{orig}}}$ via a selective mask $\mathbf{M}$.

Here, $\mathbf{M}$ determines which weights to adopt from the fine-tuned model and which to retain from the original model. Selecting more delta weights increases $\Vert \mathbf{W_{\text{sd}}} - \mathbf{W_{\text{orig}}}\Vert_2^2$ and preserves more utility gains ($\mathbf{W_{\text{sd}}}$ is closer to $\mathbf{W_{\text{sft}}}$), while selecting fewer delta weights decreases $\Vert \mathbf{W_{\text{sd}}} - \mathbf{W_{\text{orig}}}\Vert_2^2$ and discard more utility gain ($\mathbf{W_{\text{sd}}}$ is closer to $\mathbf{W_{\text{orig}}}$).

Q2: Why random selection achieve such a low Harmful Score?

The effect of delta weight attribution would explain this phenomenon. As we discussed in Section 4.1, the delta weights after fine-tuning contributes to two performance changes: (1) utility improvement and (2) safety degradation. Randomly discarding parts of delta weights progressively reduces their contribution to safety degradation. For example, in the extreme case of full removal, safety reverts to the original model’s level (1.06 Harmful Score).

In the random selection experiments, we discarded about 50% of delta weights, thus mitigating the safety degradation: Harmful Score to 1.92, corresponding to 27% ASR.

Thank you for your insightful reviews. We appreciate your recognition of our work as novel, effective and efficient. Below, we address your concerns:

C1 & Q3: Baseline performances on Llama3-8b-instruct

Thank you for your thoughtful advice. Since all baselines release their code based on Llama2-7b-chat, to ensure reproducibility, we chose to base our main experiments on Llama2 as well.

To address your concern, we extend the Llama3-8b experiments in Table 4 on two representative datasets, PureBad and DirtySummary. The table below shows SafeDelta effectively preserves safety while not harming the utility.

C2: Over refusal issue.

Following the standard practice in this field [1-4], we initially did not include over-refusal test. Recognizing the importance of your concern, we test SafeDelta under the most/least harmful settings: finetuned on PureBad/Math dataset. We employ ORBench [5] for evaluation. Results show that SafeDelta does not suffer from over-refusal issues and performs comparably to the original model:

"Orig" refers to original model; "OR rate" measures the percentage of refused benign questions (lower is better).

Q1. How does SafeDelta perform for Multimodal LLMs?

This work focuses on text modality safety, so MLLMs were not considered. We plan to explore this in future work. SafeDelta can be adapted for MLLMs by using multimodal safety data to compute the Hessian matrix, with the weight adjustment process remaining unchanged.

Q2. What is the efficacy against jailbreak attacks?

Since our work focuses on safety of fine-tuning instead of inference, we initially omitted jailbreak tests, following standard practice in this field[1-4].

To address your concern, we test SafeDelta against typical jailbreak attacks: GCG, AutoDAN and PAIR. Each generates 200 examples. To simulate black-box access in finetuning service, we do transfer attack using Vicuna-13B for GCG and AutoDAN. We test the original model (Orig) and PureBad-finetuned model with SafeDelta. The results show that SafeDelta preserves the original model's defense against jailbreaks:

Here, numbers are ASR (lower means stronger defense).

Q4. Does SafeDelta degrade in sequential interactions (harmful queries followed by benign ones)?

We followed standard setups in this field [1-4] without considering this scenario.

To address your concern, we simulate 200 sequential interactions: each involves a PureBad harmful query, LLM's answer, and a follow-up Summary query. The results confirm that SafeDelta maintains utility in this scenario.

Above are F1 scores (higher is better). "Direct" uses direct queries; "Finetuned" is model standard finetuned on Dirty Summary. Since the model is fine-tuned on direct queries, there is little degradation in sequential scenario.

Q5. What is the performance when Harmful Dataset Size is substantially increased?

Thanks for your advice. We test SafeDelta on PureBad dataset with 1k and 10k sizes. Results show that SafeDelta still maintains safety(ASR) while preserving basic utility(MT-B).

Q6. How does content filtering perform?

We initially did not consider content filtering methods, as they are ineffective on datasets with benign content (AOA, MATH), where fine-tuning still harms safety.

To address your concern, we filter the dataset using Llama3-Guard-8b and finetune on the filtered data. As expected, this approach performs poorly:

"Filter Rate" is the percentage of data filtered out; "Filter ASR" is the ASR of the model finedtuned on filtered dataset.

References

[1] Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models. NeurIPS 2024

[2] Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment. NeurIPS 2024

[3] Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic. ACL 2024

[4] Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024

[5] An Over-Refusal Benchmark for Large Language Models. 2024