Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models

We sincerely thank the reviewer for offering detailed reviews on$\textsf{Safe LoRA}$. We appreciated the comment saying that “The paper is well-written and concise enough, … the potential audience of this paper will be large given its simplicity.” and shared a common perspective as the reviewer, laying importance in restoring alignment under practical scenarios. Below we provide a pointwise reply to the reviewer’s comment.

Literature Discussion

We thank the reviewer for bringing up related works and concurrent work for discussion. We would cite and discuss the mentioned papers in the revised version. For a brief introduction, please see the general response for the literature discussion.

More Performance Comparison

We conduct the official code of Vaccine and train the model on LoRA with the Llama-2 model. Then, we fine-tuned Vaccine models (single/double LoRA setting) with $\rho=2$. We show the results on Dialog Summary and Pure Bad datasets.

Single LoRA

We train Vaccine with LoRA (q_proj and v_proj) first, and then fine-tuned it on downstream task datasets. As seen in the table below, Vaccine reduces the harmfulness score to 3.282 on PureBad while the utility (MT-Bench $\uparrow$) is not maintained. Furthermore, for Dialog Summary, the utility drops as well while safety shows no improvement.

Double LoRA

We train Vaccine with LoRA ("q_proj", "k_proj", "v_proj", "o_proj", "up_proj", "down_proj", "gate_proj", default setting of Vaccine) first, and then fine-tuned another LoRA (q_proj and v_proj) on downstream task. The results shown below indicate that using double LoRA fine-tuned on Pure Bad reduces utility (MT-Bench $\uparrow$). However, the harmfulness score decreases slightly compared to using LoRA fine-tuning. Regarding Dialog Summary, double LoRA is effective in retaining utility scores while the harmfulness increases.

Although it seems that Vaccine does not effectively reduce harmfulness, it might be contributed to the fact that Alpaca was used for vaccine models in which the alignment might be compromised as shown in [R6]. Due to the time limit, we are unable to validate the cause which requires multiple experiments. However, we deem that both $\textsf{Safe LoRA}$ and Vaccine are diverse viable solutions toward soothing the misalignment of LLM fine-tuning and should both be considered during practical deployment.

[R6]: Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations (ICLR 2024).

Effect on Harmful Data

We thank the reviewer for this additional question on experiments which we answer below. We follow the same setting in Section 4 that, for 10% harmful data, there are 7 projected layers. Regarding 30% and 50% harmful data, we project 18 and 34 layers, respectively. From the table below, it is evident that even with an increase in the ratio of harmful data, $\textsf{Safe LoRA}$ continues to effectively improve safety, reducing the harmfulness score to around 1.2 while maintaining excellent utility which is only a reduction of about 1% compared to the original one.

Building Alignment Update

The reviewer is exactly correct in the way of building alignment updates! In fact, we’ve written that in section 3.1, the base model is eventually considered as it shows similar performance to the maliciously fine-tuned one. Here we quote “As a result, … most open-source LLMs provide both their base model and chat/instruct models, users can conveniently use these official models to construct the alignment matrix without needing to train their own aligned or unaligned model.” To sum up, we note that here $\textsf{Safe LoRA}$ is a training-free and data-independent method as it requires only the aligned and base model.

Thanks for the prompt answer of constructing alignment update, I think the authors can modify the equation to showcase this as this is a very important procedure. I am kind of confused about this point when reading your paper.

I still have a few questions regarding the comparison with Vaccine:

1. are you using chat model (aligned model) as the base model for Vaccine?
2. what alignment dataset you are using to train Vaccine?

We thank the reviewer for the additional question which we will try to clarify as the following. In Section 3.1, we explain $\textsf{Safe LoRA}$ by constructing the alignment update with the unaligned model, i.e., alignment update = chat - unaligned. However, as mentioned in the rebuttal and original paper “We discovered that … are identical to those of the base model”. The base model is eventually considered to construct the alignment update in $\textsf{Safe LoRA}$, i.e., alignment update = chat - base and all subsequent experiments (Table 1 through 5), as it is a more practical scenario.

Hi, thanks for the quick answer. If I got it correctly, in their official code, the alignment dataset used by Vaccine is not Alpaca, but a harmful prompt-safe answer dataset named BeaverTails. Aligned the model with Alpaca is not a correct way.

I hope the authors can address this issue, as a fair evaluation with baselines is our important to the value of the work.

Thanks for the clarification. Vaccine fails in some of your experiments probably because adding perturbation might break the original alignment of the Llama2-chat model (I think originally they used Llama2 (unaligned) model as base model). Safe Lora does not have that issue because it is a post-fine-tuning solution. That said, I still encourage the authors to include the comparison results with Vaccine into their next version of the paper, as they show a new observation and also show the superiority of the Safe Lora method.

I appreciate the author's willingness to provide additional experiments/baselines to enrich their work. I therefore increase my score to 7. I will also actively participate the reviewer-AC discussion phase to support this paper.

Thanks for your genuine appreciation of the clarity and precious reviews on our work! We are delighted to receive a comment denoting that “The idea of $\textsf{Safe LoRA}$ is simple yet effective, making it practical for mitigating safety risks in fine-tuning LLMs.” Please see the response below as we address your comments.

Application Scenario

We thank the reviewer for the practical question raised! Here, we imagine two application scenarios possible for $\textsf{Safe LoRA}$: the benign users’ part and the LLM API providers’ part (such as OpenAI).

• Benign Users

  It is thought that benign users possess the model and can independently control the training process. Under these conditions, one use case would be companies aiming to fine-tune models based on business-related downstream data. This could be for internal employee queries or for customer service chatbots. However, it is shown that even when benign data is used for fine-tuning, alignment may still be compromised [R1]. To prevent the model from responding to inappropriate queries from anyone, $\textsf{Safe LoRA}$ would provide help to ensure that the fine-tuned model remains aligned.

• LLM API Providers

  As an LLM API provider, users can upload their data for the LLM API provider to fine-tune, with users unable to interfere in the training process, only adjusting training parameters. In this scenario, the LLM API provider cannot spend extensive time checking whether the data is harmful but also wants to avoid the model from generating inappropriate responses after fine-tuning the user's data. Therefore, the LLM API provider would need to use $\textsf{Safe LoRA}$ to ensure that the model can withstand problematic queries while preserving the utility of the user's fine-tuning data.

[R1] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations (ICLR 2024).

Adaptive Attacks

We thank the reviewer for such a question and also deem the attack discussion necessary.

Based on the previous application scenarios, adaptive attacks might be most plausible in the LLM API provider context. In the LLM API provider scenario, attackers are permitted to only upload malicious data and adjust training hyper-parameters (if allowed) but don’t have access to information about the model (such as weights or architecture) or interfere with the intermediate training process. We assume that attackers are aware that the API providers will use $\textsf{Safe LoRA}$ for realignment but do not know the exact alignment matrix.

We propose two possible adaptive attack methods as the following:

• Method I - Knowledge Transfer Attack

  Without any knowledge of the private model. The attacker can only obtain the alignment vector on the open-source model. Here, the goal is to create malicious data such that the cosine similarity between the gradient during training and the alignment vector is high, while still keeping the training loss low. This approach would ensure a decrease in the model's safety. The attacker then provides the LLM API provider with crafted malicious data, i.e. similar to a transfer attack designed to bypass $\textsf{Safe LoRA}$.

  However, to craft such data, the attackers should find a noise such that when applied to the text embedding (since LLMs convert discrete prompts into continuous embeddings) would result in the increase of cosine similarity that eventually evades the $\textsf{Safe LoRA}$ projection. Once the noise is identified, the attacker still needs additional steps of discrete-continuous optimizations to select the optimal discrete prompts for the generated noise, which can be done using genetic algorithms (GA) or other methods.

  Nevertheless, this attack method is time-consuming as it requires optimizing the noise for each iteration of the training process. Furthermore, converting malicious embeddings into prompts using GA also incurs significant time costs. Lastly, the surrogate alignment matrix might be very different from the target API, causing attacks to fail.

• Method II - Model Inversion Attack

  Once again, the attacker can obtain the alignment vector by computing it on an open-source model. Since the dimension of this alignment vector matches that of the model weights, performing model inversion on the alignment vector might be able to generate data that represents safety. By mixing malicious data with the safety data obtained from model inversion and then uploading to the API provider, it might also be possible to bypass $\textsf{Safe LoRA}$. However, it is not known whether the alignment vector could be meaningful in model inversion or are current model inversion techniques on LLMs [R2] strong enough for such adaptive attacks.

[R2] Morris, John X., et al. "Language model inversion." (In ICLR 2024)

We thank the reviewer for recognizing the work and stating that “The motivation is sound which tries to address the problem of decreased safety after fine-tuning”. We will address and justify some concerns raised by the reviewer in the comment below.

Insights for $\textsf{Safe LoRA}$

Regarding the safety representation in the alignment matrix, we note from other sources such as [R1, R2] that considered exploring the safety landscape and task arithmetics which are a recent trend that focuses on the interpretability of weight semantics.

Following the pipeline, the projection matrix is constructed by treating weight space as a normed vector space and extracting the “alignment” semantics from the difference between the aligned and unaligned models. By constructing the alignment matrix, we essentially create a hyperspace of alignment and by selective projection, we are able to preserve both the utility and safety. Therefore, the projection of the trained LoRA weights is intended to map them into the alignment hyperspace rather than merely constraining the fine-tuned weights to remain within a limit of the original weights.

[R1] Wei, Boyi, et al. "Assessing the brittleness of safety alignment via pruning and low-rank modifications." (In ICML 2024).

[R2] Ilharco, Gabriel, et al. "Editing models with task arithmetic." (In ICLR 2023)

Performance and Settings

• Regarding Table 2 & 4

  The decrease in utility observed is primarily due to the impact of fine-tuning on the generalization ability [R3]. MT-Bench evaluates the overall performance of LLMs, and fine-tuning can negatively affect this broader capability. Consequently, the results in Table 4 have shown a decline. Additionally, the large difference in the number of training samples—520 times greater in Table 4 compared to Table 2—leads to models that are more specifically adapted to the training data, further contributing to the observed decrease in utility.

• Regarding Table 4 & 5

  On the other hand, we note that the reviewer holds the correct sense. That is, the more harmful data exist, the more layers we may need to project to maintain the alignment.

[R3]: Yang, Haoran, et al. "Unveiling the Generalization Power of Fine-Tuned Large Language Models." (In NAACL 2024)

Application Scenario

We thank the reviewer for the question. Due to word limits, please see the general response for the application scenario of $\textsf{Safe LoRA}$.

Safe Fine-tuning

To address the reviewer's concerns on safe fine-tuning, we conduct experiments based on the reviewer's examples to demonstrate that $\textsf{Safe LoRA}$ does not heavily depend on the fine-tuned dataset.

We use the Mistral model, which initially has poor alignment with a harmfulness score of 2.003, as shown in the following Table. We use the safety instruction dataset [R4] to fine-tune the model and improve its safety. As a result, the harmfulness score decreased to 1.003. Meanwhile, when applying $\textsf{Safe LoRA}$ to the fine-tuned model with $\tau = 0.5$ (projecting 5 layers), the harmfulness score is 1.012. Compared to the original model, applying $\textsf{Safe LoRA}$ still leads to an improvement in alignment.

[R4] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions. (In ICLR 2024).

thank the authors for the rebuttal. After reading it and other reviewer's comments, I have raise my score to 5. My concerns are mostly resolved. I didn't give a higher score because I still feel a deeper understanding of the method should be pursued and the application scenario is not fully convincing. I didn't give a lower score because as an academic paper, I feel there is merit in accepting this paper. good luck!

We appreciate the reviewer’s feedback, and we are glad to know that we have addressed most of your concerns. Although there are still some concerns about the $\textsf{Safe LoRA}$ method itself and its related application scenarios, we will provide the following explanation in hopes of helping you gain a better understanding.

Deeper Insights of $\textsf{Safe LoRA}$

Currently, there are many related papers dedicated to manipulating models with arithmetic operations to enhance performance or add new functionalities to the models. We can use this concept to explain the working mechanism of $\textsf{Safe LoRA}$, where we improve model safety by projecting unaligned weights onto an aligned hyperspace. In $\textsf{Safe LoRA}$, aligned vectors play a crucial role. If one obtains a vector $A$ by subtracting the base model weights from a non-aligned model, this vector $A$ itself does not represent alignment. Using such a vector to create a projection matrix will not enhance safety when projecting fine-tuned weights onto it. This is why we need to derive the so-called aligned vectors from an aligned model. We do not arbitrarily manipulate the fine-tuned weights, nor do we simply restrict the distance of the fine-tuned weights from the original ones. Restricting this distance alone can lead to a decrease in utility and does not guarantee an improvement in safety. In summary, $\textsf{Safe LoRA}$ enhances safety by manipulating fine-tuned weights that are further from the aligned direction, pulling them back into the aligned hyperspace while maintaining utility.

Application Scenario

We will provide a more detailed explanation of the two application scenarios we mentioned earlier.

• Benign User

  First, we explain why a benign user would need to use $\textsf{Safe LoRA}$. As demonstrated in the experiments from the paper, even if the training data is entirely benign, alignment can still be compromised. Therefore, $\textsf{Safe LoRA}$ is necessary to address this issue.

  Next, we discuss why a benign user needs to perform fine-tuning. Although current LLMs can indeed provide very logical question-and-answer interactions, using them directly as customer service bots to answer specific questions from customers is often insufficient. This is because LLMs typically lack information relevant to a particular company. Therefore, fine-tuning an existing LLM on the user’s data becomes necessary to tailor it to the company's specific needs and information.

  Finally, combining the above two points, users need to fine-tune models using their own data. However, since fine-tuning can potentially lead to a loss of alignment, and generally, users do not want their customer service bots to provide inappropriate responses—because this could damage the company's reputation and increase crime rates (as it becomes easier to access harmful information)—$\textsf{Safe LoRA}$ becomes necessary to address these concerns.

• LLM API Provider

  The most common LLM API provider is OpenAI's ChatGPT. It allows users to upload data and configure relevant training parameters to fine-tune ChatGPT. However, since users are not always well-intentioned, there is a risk that they might upload malicious data. The LLM API provider cannot individually check each piece of data for harm, as this would be too time-consuming and would negatively impact the user experience. This is different from how ChatGPT checks user inputs for appropriateness during conversations. The volume of training data can be very large, making it impractical to check each piece individually. In fact, OpenAI currently does not perform checks on users' training data. So why is $\textsf{Safe LoRA}$ necessary? If a user uploads malicious data that removes alignment, they can fine-tune the model and then start querying it with harmful questions, potentially obtaining inappropriate answers. This undermines the alignment efforts made by LLM API providers, as malicious users could effectively bypass these safeguards by spending a small amount of money to get an unaligned model that can provide any response they desire. This issue could potentially increase societal risks by making it easier for people to access inappropriate information.

Overall, $\textsf{Safe LoRA}$ allows model owners to restore their safety guardrails in an efficient manner regardless of any harmful data present.

We hope that the detailed explanations we have provided about the $\textsf{Safe LoRA}$ method and its insight, as well as its application scenarios, will enhance your understanding of $\textsf{Safe LoRA}$ and its use cases. Thank you once again for taking the time to review the explanations provided. We hope that your concerns have been addressed.

We genuinely appreciate the reviewer for the comprehensive comments concerning alignment of LLMs. We are delighted to receive the positive feedback that “The proposed method is also very easy to follow…” Please see our point-to-point response to your comments below.

On the explanation of $\textsf{Safe LoRA}$

We thank the reviewer for raising such an insightful question which we will try to answer below. Firstly, there is some other research [R1, R2] concerning weight semantics such as exploring the safety landscape and task arithmetics that are related to the alignment matrix that we proposed. On the other hand, we note that in addition to the safety guardrail that was governed by the alignment matrix we also control the to-be-projected layers such that the utility can as well be preserved. Specifically, figure 3 in the paper demonstrates the harmfulness score versus utility. $\textsf{Safe LoRA}$ maintains strong downstream task performance because we selectively set $\tau$ to control the number of projected layers rather than projecting every layer indiscriminately. This selective projection allows us to retain critical task-specific information while minimizing unnecessary alterations, resulting in better performance. In summary, our introduced alignment matrix is effective prior to guiding LoRA updates towards better safety, because it entails the effort in aligning a base model to a chat model that can refuse (some of) unsafe questions.

[R1] Wei, Boyi, et al. "Assessing the brittleness of safety alignment via pruning and low-rank modifications." (In ICML 2024).

[R2] Ilharco, Gabriel, et al. "Editing models with task arithmetic." (In ICLR 2023)

Clarification on the experiment setting

We believe that the reviewer has some misunderstanding with regard to this issue. In fact, adding 100 harmful samples to the Dialog Summary dataset wasn't a special setup aimed at verifying performance. It was actually part of our broader approach to demonstrate datasets with varying levels of harmful content, ranging from purely harmful data (Pure Bad), and partially harmful (Dialog Summary) to entirely benign data (Alpaca). This setup was not specifically tailored but rather a natural part of our experimental framework, while also adopted by many other research [R3, R4]. To demonstrate the performance of our method, we present the F1 score on the Dialog Summary dataset, which indicates that $\textsf{Safe LoRA}$ can maintain the utility of downstream tasks yet also safeguard harmful content regardless of the ratio. As for evaluating alignment, we use the benchmark proposed by [R5], which is also adopted by other works [R3]. Overall, our experimental setup wasn't specifically designed to verify performance. Instead, it was intended to demonstrate attacks of varying severity.

[R3]: Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Muhao Chen, Junjie Hu, Yixuan Li, Bo Li, and Chaowei Xiao. Mitigating fine-tuning jailbreak attack with backdoor enhanced alignment.arXiv preprint arXiv:2402.14968, 2024.

[R4]: Tiansheng Huang, Sihao Hu, Ling Liu. Vaccine: Perturbation-aware Alignment for Large Language Model. arXiv preprint arXiv:2402.01109, 2024.

[R5]: Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations (ICLR 2024).

Clarification on the LoRA setting

We thank the reviewer for advising on the details. Here, we follow the official code of Llama [R6], where the default setting of the PEFT config uses q_proj and v_proj.

[R6]:https://github.com/meta-llama/llama-recipes/blob/main/src/llama_recipes/configs/peft.py, Line 11