Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack

We thank the reviewer for the generally positive review. As below, we show extra results and discussion to address the concerns.

W1: Computational overhead scales with model size.

Step time. Because Vaccine requires double forward-backward pass in each optimization step, the training time of Vaccine is approximately double compared to the SFT baseline. Because the training time itself is scaling with the model size, the overhead is also scaling with the model size. See the following table for our evaluation results.

However, it is important to note that this is only a one-time cost for aligning a pre-trained model. In the fine-tuning-as-a-service scenario, the same aligned model is used to finetune for all the finetuning requests (which arrive at the rate of thousands or millions/per hour). Since Vaccine does not incur extra overhead for fine-tuning, its computation overhead should not be a big concern for real-time service for a service provider.

Accelerated Vaccine. It is also possible to accelerate Vaccine. Vaccine requires double training time because in every step we need to first search for the perturbation, and then apply the perturbation to the model to do another forward-backward pass. To address the reviewer's concern, we propose **accelerated Vaccine. The idea is simple- we search for the perturbation not for every step, but to do it for every $\tau$ step. See Algorithm 2 in our attached pdf for details. The following table shows the evaluation results.

Our results surprisingly show that by searching perturbation for every 100 steps, Acceleated Vaccine is still able to achieve decent defense (harmful score is increased by a marginal 0.60), but the step time is significantly shortened. We hope accelerated Vaccine can erase the reviewer's concern about resource overhead.

GPU memory. On the other hand, the extra memory overhead is scaled with the hidden embedding size, because in the second forward-backward pass we need to register the perturbation to the hidden embedding. The following table shows how the GPU memory usage scales with the hidden embedding size:

As shown, for OPT-13B, only a marginal 0.039/48.824=0.08% extra memory is induced compared to SFT. The memory cost is apparently marginal.

W2: Performance downgrades when the harmful ratio is high.

Indeed, Vaccine experiences performance downgrades when the harmful ratio is high. This however is an inevitable drawback of an alignent stage solution. Particularly, an alignment stage solution aims at enhancing the model's robustness to the harmful data, but fine-tuning on too much harmful data (e.g., pure harmful data) can still break the enhanced robustness.

While we in this paper only aim to increase the aligned model's robustness in the alignment stage, we admit that this may not be enough for the hard case (e.g., a large harmful ratio). We envision that future research can build on top of Vaccine to provide a stronger defense. We will discuss this in our limitation and future direction section.

Q1: Why would users upload harmful data to break the model?

There are two possible cases for the user to upload harmful data.

i) Non-adversary case. The users are not aware that the data they upload contains harmful instances. Users may collect fine-tuned data from their application use case, and they may not carefully inspect and clean the data.

ii) Adversary case. The user (e.g., a business adversary) aims to scandalize the service provider to provide harmful content (or political sensitivity content) from the API. Because the finetuned model is deployed on the service provider's server and the harmful content is transmitted from their API, the service provider may face legal accusations or governance issues. An example is that users may ask "How do you comment the war between Israel and Palestine"? The service provider is responsible for the answer.

While the first case may be more realistic and common, the second case may raise serious concerns for the service provider.

We sincerely thank the reviewer for the constructive review comments. As belows, we try to address the separate comment.

W1: Double training time

Indeed, Vaccine requires double training time, because in every step we need to first search for the perturbation, and then apply the perturbation to the model to do another forward-backward pass. To address the reviewer's concern, we propose **accelerated Vaccine. The idea is simple- we search for the perturbation not for every step, but to do it for every $\tau$ steps. See Algorithm 2 in our attach pdf for details. The following table for the evaluation results.

Our results surprisingly show that by searching perturbation for every 100 steps, Acceleated Vaccine is still able to achieve decent defense (harmful score is increased by a marginal 0.60), but the step time is significantly shortened.

W2: Lack of baseline

Indeed, not enough baselines are used in this submission, partly because this work is one of the initial works for the harmful fine-tuning issue. After NeurIPS submission deadline, we do see there are a few defense solutions arise, e.g., RepNoise[1]]. RepNoise, same as Vaccine, is an alignment stage solution that aims at improving the neural network's robustness, which makes it a perfect baseline to compare. In the following table, we show the performance comparison under different harmful ratios.

As shown, Vaccine consistently outperforms RepNoise with a smaller harmful score and also a higher finetune accuracy.

W3-A: Evaluation using black box moderation model

We totally agree that using a moderation model is not ideal and not accurate enough for performance evaluation. However, at the current stage, there is not an accurate benchmarking method to evaluate the model's harmfulness. Existing research, e.g., [2][3] utilizes GPT4 for evaluation. RepNoise [1] follows our setting to use BeaverTails's moderation model for evaluation. We generally believe that compared to GPT, the moderation model we use is less "black box", given that GPT4 is not even open-sourced, and the score it gives may change due to version updates or randomness.

W3-B: the paper does not discuss the change in the model's generative ability (e.g., ppl)

For model evaluation, we mainly use finetune accuracy to evaluate the model's reasoning ability (for example, for GSM8k, we measure the accuracy based on whether the model can predict the final answer). Indeed, it is interesting to study the model's generative ability using metrics like perplexity. In the following table, we show the comparison results after the model fine-tuned on wikitext.

The results show that Vaccine may slightly increase the perplexity of the model compared to SFT. However, in comparison, RepNoise also increases the perplexity but is less effective in maintaining a low harmful score.

Q1: Intuitive visualization of harmful embedding drift

We thank the reviewer for the suggestion of t-SNE visualization. We plot the embedding drift of SFT and Vaccine under different harmful ratios. When the harmful ratio is high, it is intuitive to see that the embedding is drifting toward a specific direction. Interestingly, the embedding drift of Vaccine is slighter, making the drifted embedding still able to preserve the alignment knowledge. This may better explain how Vaccine really works.

[1] Rosati D, Wehner J, Williams K, et al. Representation noising effectively prevents harmful fine-tuning on LLMs[J]. arXiv preprint arXiv:2405.14577, 2024.

[2] Hsu C Y, Tsai Y L, Lin C H, et al. Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models[J]. arXiv preprint arXiv:2405.16833, 2024.

[3] Qi X, Zeng Y, Xie T, et al. Fine-tuning aligned language models compromises safety, even when users do not intend to![J]. arXiv preprint arXiv:2310.03693, 2023.

Thank you for your detailed and constructive rebuttal, which includes extensive experimental evidence.

I am particularly impressed by your introduction of the accelerated Vaccine method. The results demonstrating only a 2% overhead when $\tau = 100$ with minimal performance degradation significantly reduce the application cost of the Vaccine method. I highly recommend including this improvement in the manuscript for future versions.

Based on my initial review and your comprehensive response, I decided to adjust my score positively.

P.S. There appears to be a typo in Algorithm 2 of the attached PDF: "t mod $\gamma$ == 0" should be reviewed for accuracy.

Best regards,

We thank the reviewer for the positive comments on our work. Below we try to address the concern on resource overhead.

GPU memory. Because in the second forward-backward pass we need to register the perturbation to the hidden embedding, Vaccine requires slightly more GPU memory usage. We in the following show comparison results with the normal finetune (SFT) and a recent alignment-stage defense RepNoise [1] using different sizes of model.

• Evaluation is done with an H100 with 80GB memory: | | OPT-1.3b | OPT-2.7b | OPT-6.7b | OPT-13b | |---|---|---|---|---| | memory(SFT) | 5.586GB | 10.814GB | 25.469GB | 48.824GB | | memory (RepNoise) | 8.962GB | 17.453GB | 40.017GB | 76.027GB | | memory(Vaccine) | 5.596GB | 10.830GB | 25.492GB | 48.863 GB |

Vaccine incurs slightly more GPU memory compared to SFT. For OPT-13B, only a marginal 0.039GB extra memory is induced compared to SFT. In sharp contrast, RepNoise introduces 27.164GB extra memory overhead compared to SFT. With this result, it seems that Vaccine is superior in the resource-constrained scenario.

Training time. Because Vaccine requires double forward-backward pass in each optimization step, the training time of Vaccine is approximately double compared to the SFT baseline. See the following table for our evaluation results.

• Evaluation is done with an H100 with 80GB memory: | | OPT-1.3b | OPT-2.7b | OPT-6.7b | OPT-13b | |---|---|---|---|:---:| | step time(SFT) | 0.06 | 0.08 | 0.09 | 0.12 | | step time(RepNoise) | 0.14 | 0.19 | 0.2 | 0.29 | | step time(Vaccine) | 0.11 | 0.15 | 0.17 | 0.24 |

As shown, Vaccine uses approximately 2x training time and RepNoise uses approximately 2.3x-2.4x training time compared to SFT. Vaccine is more computation-efficient compared to RepNoise.

It is also possible to accelerate Vaccine. Vaccine requires double training time, because in every step we need to first search for the perturbation, and then apply the perturbation to the model to do another forward-backward pass. To address the reviewer's concern, we propose accelerated Vaccine. The idea is simple- we search for the perturbation not for every step, but to do it for every $\tau$ steps. See Algorithm 2 in our attached pdf for details. The following table shows the evaluation results.

Our results surprisingly show that by searching perturbation for every 100 steps, Acceleated Vaccine is still able to achieve decent defense (harmful score is increased by a marginal 0.60), but the step time is significantly shortened. We hope accelerated Vaccine can erase the reviewer's concern about resource overhead.

[1] Rosati D, Wehner J, Williams K, et al. Representation noising effectively prevents harmful fine-tuning on LLMs[J]. arXiv preprint arXiv:2405.14577, 2024. https://arxiv.org/abs/2405.14577

Thank you for recognizing our effort in rebuttal, and also for increasing rating to reflect the addressed concern. We will definitely include the new results to the paper, and also open-source the accelerated Vaccine method.

W1+Q1: Reource ovehead

GPU memory. Because in the second forward-backward pass we need to register the perturbation to the hidden embedding, Vaccine requires slightly more GPU memory usage. We in the following show comparison results with the normal finetune (SFT) and a recent alignment-stage defense RepNoise [1] using different sizes of model.

• Evaluation is done with an H100 with 80GB memory: | | OPT-1.3b | OPT-2.7b | OPT-6.7b | OPT-13b | |---|---|---|---|---| | memory(SFT) | 5.586GB | 10.814GB | 25.469GB | 48.824GB | | memory (RepNoise) | 8.962GB | 17.453GB | 40.017GB | 76.027GB | | memory(Vaccine) | 5.596GB | 10.830GB | 25.492GB | 48.863 GB |

Vaccine incurs slightly more GPU memory compared to SFT. For OPT-13B, only a marginal 0.039GB extra memory is induced compared to SFT. In sharp contrast, RepNoise introduces 27.164GB extra memory overhead compared to SFT. By this result, it seems that Vaccine are superior in resource-constrained scenarios.

Training time. Because Vaccine requires double forward-backward pass in each optimization step, the training time of Vaccine is approximately double compared to the SFT baseline. See the following table for our evaluation results.

• Evaluation is done with an H100 with 80GB memory: | | OPT-1.3b | OPT-2.7b | OPT-6.7b | OPT-13b | |---|---|---|---|:---:| | step time(SFT) | 0.06 | 0.08 | 0.09 | 0.12 | | step time(RepNoise) | 0.14 | 0.19 | 0.2 | 0.29 | | step time(Vaccine) | 0.11 | 0.15 | 0.17 | 0.24 |

As shown, Vaccine uses approximately 2x training time and RepNoise uses approximately 2.3x-2.4x training time compared to SFT. Vaccine is more computation-efficient compared to RepNoise.

Accelerated Vaccine. It is also possible to accelerate Vaccine. Vaccine requires double training time because in every step we need to first search for the perturbation, and then apply the perturbation to the model to do another forward-backward pass. To address the reviewer's concern, we propose accelerated Vaccine. The idea is simple- we search for the perturbation not for every step, but to do it for every $\tau$ steps. See Algorithm 2 in our attached pdf for details. The following table shows the evaluation results.

Our results show that by searching perturbation for every 100 steps, Acceleated Vaccine is still able to achieve decent defense, but the step time is significantly shortened. We hope accelerated Vaccine can erase the reviewer's concern about resource overhead.

W2: Scalaibility to larger datasets

It is a good idea to test Vaccine in another larger-scale fine-tuning dataset. Here we follow the setting form [2] to finetune the model on a WikiText-2, and we show the model's tradeoff between perplexity (smaller the better) and harmful score.

As shown, Vaccine achieves smaller harmful score but maintain a lower perplexity compared to RepNoise.

W3: Selection of hyper-parameter

Indeed, Vaccine relies on the noise intensity $\rho$ for providing an effective defense. In Table 6, we use SST2 as a fine-tuning task (which is our default setup) to demonstrate how choosing different $p$ affects the harmful score and finetune accuracy. Below we show the data for providing intuition of how to select $\rho$.

As shown, with a larger $\rho$, the harmful score will decrease but also at the cost of degrading finetune accuracy. In this case, before deploying Vaccine, one should use a validation dataset to test the model's finetune performance and harmful score, and select the best tradeoff that is acceptable to the application.

Here we also want to highlight that we only have one hyper-parameter to tune. In comparison, the recent method RepNoise needs two hyper-parameters, and requires an additional harmful dataset in their assumption. The simplicity of Vaccine should be merited.

Q2: Will all the fine-tuning attack cause harmful embedding drift?

Honestly, we don't have an definitive answer to this question. At least the original attack, i.e., mixing normal harmful data in the fine-tuning data, will cause harmful embedding drift. There may be more advanced attacks that can circumvent the embedding drift, which, however, is still under-explored. In the attached pdf, we also use t-SNE to illustrate the harmful embedding drift phenomenon, which you may be interested in. Feel free to check it out.

[1] Rosati D, Wehner J, Williams K, et al. Representation noising effectively prevents harmful fine-tuning on LLMs[J]. arXiv preprint arXiv:2405.14577, 2024.

[2] Li Y, Yu Y, Liang C, et al. Loftq: Lora-fine-tuning-aware quantization for large language models[J]. arXiv preprint arXiv:2310.08659, 2023.

Hi Reviewer a2cG,

Thanks for the very informative review, and also for recognizing the harmful fine-tuning that we focus on is an interesting and important topic. We want to get back to you to see whether our rebuttal addresses your concerns.

Here we respectfully summarize our efforts to address your main concerns.

1. To show the trade-offs between robustness and efficiency, we design an accelerated Vaccine method. It is shown in the table in the rebuttal that this method has a better tradeoff between robustness and efficiency -- harmful score is increased by a marginal 0.60 while the training time can be shortened by 50%, compared to the original Vaccine.

2. To show that our method can be generalized to larger datasets, we show additional experiments on WikiText-2 with a larger number of tokens. The results indicate that Vaccine can reduce harmful scores while maintaining the model's perplexity.

3. We show further guidelines on selecting hyper-parameter $\rho$ by conducting experiments. The general trend is that with a larger $\rho$, the harmful score will decrease but at the cost of degrading finetune accuracy.

4. We compare another alignment-stage solution RepNoise in terms of GPU memory and training time. Our experimental results reflect that Vaccine is superior to RepNoise regarding both GPU memory and training time.

We hope that these efforts can fully address your concerns, and we are more than happy to discuss them with you!

As pointed out by Reviewer CLDA, there is a typo in the Algorithm 2 in the rebuttal pdf. Line 4 in the algorithm should be "if $t$ mod $\tau$ == 0" instead of "if $t$ mod $\gamma$ == 0". In this algorithm, we basically do the perturbation every $\tau$ steps instead of per step, in order to accelerate vanilla Vaccine. We apologize for the confusion.