Learning and Forgetting Unsafe Examples in Large Language Models

References:

[1] Wang, Aobo, Cong Duy Vu Hoang, and Min-Yen Kan. "Perspectives on crowdsourcing annotations for natural language processing." Language resources and evaluation 47 (2013): 9-31.
[2] Henderson, Peter, et al. "Self-destructing models: Increasing the costs of harmful dual uses of foundation models." Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 2023.
[3] Qi, Xiangyu, et al. "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!." arXiv preprint arXiv:2310.03693 (2023).
[4] Korbak, Tomasz, et al. "Pretraining language models with human preferences." International Conference on Machine Learning. PMLR, 2023.
[5] Gao, Leo, et al. "The pile: An 800gb dataset of diverse text for language modeling." arXiv preprint arXiv:2101.00027 (2020).
[6] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. The psychology of learning and motivation, 24(109-165):92, 1989.
[7] James L McClelland, Bruce L McNaughton, and Randall C O’Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
[8] Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.

Response to Weakness 2:

The reason why unsafe content is forgotten remains unexplained. Providing an empirical or theoretical explanation for this phenomenon would greatly enhance the paper and warrant a higher score. Additionally, it is crucial for the authors to investigate the conditions under which unsafe content can be forgotten, enabling readers to understand when to effectively apply the proposed ForgetFilter (FF) method.

Thanks for your advice. We consider that unsafe content can be forgotten during safety finetuning is attributed to catastrophic forgetting during sequential finetuning [6,7,8]. During safety finetuning, the model is trained to produce safe content that is opposed to unsafe content. The model may shift original weights for unsafe content to learning safe content, leading to the forgetting of unsafe content.

A method to empirically explain forgetting is to look into the distribution of representations of input data in the model. One hypothesis is safe and unsafe data may have closer representations than other examples in the feature space. However, we will leave this interesting question as our future work and focus on applications to safety in LLM for this work. We identify there are two main difficulties for such an empirical study. (1) We need to decide which layer to extract activations for comparison. The current LMs have an enormous size and each layer may have different responsibilities. (2) We need to downscale the representations for analysis or visualization (e.g., t-SNE). A representation for an input sentence in an LM can be massive where each token position corresponds to a high-dimensional vector. We need to find a suitable probe task to train a matrix that helps to reduce the dimension of the model’s representations.

In terms of conditions for applying ForgetFilter, the safe content in safety finetuning should cover the type of past learned unsafe examples. For example, to effectively forget toxic examples, safe content should include untoxic data. We have empirically shown this in Appendix D in our script. We find toxic examples cannot be forgotten more significantly than examples irrelevant to toxicity when the model is finetuned on unbiased data.

Response to Question 1:

The presentation can benefit from a diagram to show the procedure of the 2-stage fine-tuning in the paper.

Thanks for your suggestion. We have made a diagram for illustration in our updated script.

Response to Weakness 1:

1. I have a concern regarding the paper's consideration of the first fine-tuning stage involving noisy and unsafe content…this setting to be somewhat artificial.

Thanks for your comments. We would like to point out that in real life, finetuning data for LLMs used by normal users can be more than some fine-grained dataset for specific NLP tasks. For instance, users might collect some corpus from the web to finetune their LLM for generating content tailored to a particular purpose or domain, such as crafting reports. However, it's crucial to acknowledge that such web-derived data may include unsafe content, as seen in datasets like the Pile [5], which comprises numerous toxic sentences [4]. Screening such data by users themselves can become cost-prohibitive and logistically challenging due to the constraints of limited resources available to users. Additionally, another plausible scenario involves the deliberate misuse of LLMs [2]. For instance, during the finetuning of LLMs through APIs, users may intentionally upload adversarial samples with the aim of training a malicious LLM. In summary, it is a practical reality that downstream finetuning data for LLMs often includes unsafe examples.

In real-life scenarios, for example, non-tech companies aiming to train customer service chatbots might use a third-party finetuning API. However, the recorded customer service conversations they upload could potentially include toxic content. Due to resource constraints, these companies are unable to manually screen the data for safety before uploading them for finetuning. This will result in the inclusion of unsafe examples in the finetuning data.

2. In my opinion, a more interesting and realistic issue arises from the existence of unsafe content during the pre-training stage of LLMs (Korbak et al., 2023). I kindly suggest that the authors explore whether the ForgetFilter (FF) method can effectively filter out unsafe content from the pre-training dataset, as this would provide valuable insights to the field.

Thanks for your advice. We would like to emphasize that the safety concerns we address in the customized fine-tuning of released LLMs are also important and underexplored [2]. This is particularly crucial as released LLMs are susceptible to being compromised through finetuning on malicious data [3]. There is a pressing need to ensure the long-term safety of a released LLM that may be finetuned on diverse downstream data, potentially containing unsafe examples.

In contrast, during the pre-training of LLMs, organizations can carefully select public high-quality datasets or employ labor to screen data, thereby training LLMs exclusively with reliable data. These organizations can then implement safety alignment measures after the pre-training of LLMs. However, when it comes to downstream fine-tuning, applying such safety measures can be challenging. For example, when finetuning LLMs through APIs, the companies do not have the bandwidth to screen users’ uploaded data manually every time and an unsafe LLM can be easily trained with users’ noisy data. Therefore, our study on ensuring the safety in finetuning LLMs can be useful and helpful in practice.

We value your advice and plan to explore safety issues in pre-training as part of our future work.

Thank you for the response.

But some of my concerns remain. The biggest conern is that if we fine-tune on the data of a user, how can we obtain the clean (unbiased) data? Do we need to anonate the data for this specifc user? So why just don't we do RLHF again on the fine-tuned model, which is quite standard and don't need further labeling effort.

If we do not anotate data for the specific user with the same distribution, then we face an OOD problem, which would be highly complicated when entangled with forgetting issues.

Response to Weakness 1:

I believe the authors should discuss related works that filter training data / perform data selection based on learning dynamics like frequency of forgetting.

Thanks for pointing out two related works! We have added related works on data selection based on learning dynamics to our script.

Response to Weakness 2:

I wonder whether in practice "safe examples" are always readily available for a fine-tuning task. For example, if I download a random dataset D from the web without having a safe dataset of the same task as D, can I still apply ForgetFilter? Is this setup practical?

Our ForgetFilter works when unsafe data in D is not known. In that case, the safe dataset should be constructed to contain diverse kinds of safe examples, e.g., like the dataset that people use for aligning LLMs during pre-training. Then the unsafe example whose categories are included by the safe dataset can be filtered. Such a safe dataset is commonly available in LLM companies and is ready for users’ requests for their finetuning tasks.

Additionally, the safe dataset does not need to be from the same or similar distribution of unsafe data in D. The forgetting is mainly based on the semantics of data (e.g., toxic or nontoxic). For example, the unknown toxic data we experiment with are randomly sampled from web data and can be very different from untoxic data in our safe dataset. However, toxic data can still be filtered through our proposed ForgetFilter.

Overall, we think our setup is practical and our ForgetFilter is applicable to real-life scenarios.

Response to Question 1:

We see from Figure 2 that up to 80% of harmful examples will be forgotten in the end. But what is the proportion of harmful examples among all the forgotten examples? … applying ForgetFiltering improves downstream performance … explain side effects that is happening on downstream performance?

Thanks for your question. Among all the filtered examples, unsafe examples account for around 81.3% in all three cases (i.e., bias, toxicity and harmfulness). Indeed, some training data of downstream tasks will be filtered out by mistake, but a lot more unsafe data that are irrelevant to the task are filtered. Then during training, the model will focus on learning downstream task data instead of unsafe examples. We posit that this accounts for the marginal performance enhancement observed compared to directly fine-tuning on noisy downstream data.

Thanks for your insightful comments. We think that some concerns are caused by misunderstanding, which we will explain in detail below. We hope that our response can clarify the misunderstandings and you can consider our work more favorably.

Response to Weakness 1:

1. it is claimed that models tend to forget unsafe data points when finetuned on safe data points. Isn't this true of any attribute? … it is completely expected that the forgetting on safe examples would be less, the forgetting on random points would be slightly higher, and the forgetting on unsafe points would be more.

We point out our main statement is unsafe examples are forgotten more than other data during safety finetuning for LLMs. Indeed, forgetting will happen to all data that the model has learned before during safety finetuning. However, the discrepancies in forgetting may not be expected and actually do not occur to language models that are not large enough. As shown in Figure 3 of our script, for GPT2-M, all types of data, i.e., safe examples, unsafe examples and non-safety examples suffer from similar levels of catastrophic forgetting, while forgetting discrepancies becomes stronger as we increase the model size. This suggests that the forgetting in terms of safety is more related to the model capacity than data distribution.

While there are past works that also showed differences in forgetting frequency [1,2], they focus on forgetting in data with synthetically flipped correct and wrong labels. However, our study is not related to label correctness but data content (label is also not applicable in our case, since our data are language sequences). Unsafe examples in terms of safe ones are not data containing random inconsistent information like mislabeled data, but systematic information (e.g., bias). Therefore, we believe that our findings are non-trivial and surprising. We unveil the forgetting pattern of LMs at different scales during finetuning with solid empirical evidence. Currently, our work is focused on safety issues where data can be semantically categorized as safe or unsafe, which makes the study on sample forgetting tractable. We will further explore such forgetting patterns in more general multitasking where there is no clear semantic category for data in our future work.

2. This seems … not that there is something fundamental about unsafe data that is more easily forgettable which is what this paper seems to claim … then the claim that unsafe data points are more easily forgettable than safe data points has not been corroborated.

Thanks for your analysis. But we would like to clarify that our claim, in alignment with your first sentence, is that during safety review, there exist discrepancies in forgetting safe and unsafe examples. Our claim is contingent upon fine-tuning with safe examples, and we do not posit that unsafe data are inherently more forgettable. Because safety finetuning is a common method to make LLMs’ generations safe, our work provides useful insights into the forgetting patterns associated with different types of learned data during safety finetuning.

Motivated by your comments, we also experiment with the symmetric setting where the model after downstream finetuning is trained with unsafe examples. The results are put in the Appendix E of our new script. However, we find the forgetting pattern is not fully symmetric to that during safety review. Indeed, as you have expected, safe examples are forgotten more than unsafe examples when finetuned on unsafe data. Nevertheless, these unsafe examples are forgotten more than the downstream task data that are less relevant to safety. In contrast, when finetuning the model on safe data, safe examples exhibit minimal forgetting. We hypothesize that this difference is attributable to the features inherent in unsafe and safe data, and we leave further exploration of this discrepancy as future work.

We do understand our writing might be unclear, obscuring our true claim. We thus update our contribution as

We investigate the forgetting patterns of LMs at different scales during safety review. We unveil the discrepancies in forgetting that for sufficiently large LMs, unsafe examples will be forgotten more significantly than other examples in noisy downstream data when finetuned with safe examples.

Response to Weakness 2:

As a slight writing suggestion, I would make it such that the list of contributions on page 2 had more details…

Thanks for your helpful advice. We have updated our script.

Response to Weakness 3:

However, in the Related Works section, there is no mention of different filtering algorithms.

Thank you for your valuable advice! We have added a new paragraph on filtering in the Related Works in our new script.

Response to Weakness 4:

The Related Works also does not mention the connection between memory and safety that has long been analyzed in the literature. For example, the work "Does Learning Require Memorization? A Short Tale about a Long Tail" …This connection should be mentioned here as it is completely possible that unsafe points may belong to tails of the data distribution and the effects examined in this paper can be explained by this previous work.

Thanks for pointing out this work. But we would like to point out that this work and according literature focus on different aspects of safety, i.e., privacy rather than maliciousness in our study. We appreciate your analysis. However, in our experiments, unsafe examples, by default, are not at the tail of the data distribution but account for the main proportion (50%). The effects of this work may not apply to our case to explain the discrepancies in forgetting during safety finetuning.

Response to Weakness 5 & 6:

... I would rearrange this section in chronological ordering to better understand the data first before discussing what a safety review is…I found several spelling and grammatical errors in this paper.

Thanks for your suggestions. We have updated the ordering and correct errors.

Response to Weakness 7:

The metric you are reporting is the average rate for a set of data points. You are not reporting the expected value.

Thanks for pointing this out. But our original symbols can also represent the empirical average. Following your advice, we have updated our formula to resolve any ambiguity.

Reference:

[1] Maini, Pratyush, et al. "Characterizing datapoints via second-split forgetting." Advances in Neural Information Processing Systems 35 (2022): 30044-30057.
[2] Toneva, Mariya, et al. "An empirical study of example forgetting during deep neural network learning." arXiv preprint arXiv:1812.05159 (2018).
[3] Korbak, Tomasz, et al. "Pretraining language models with human preferences." International Conference on Machine Learning. PMLR, 2023.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a.

Response to Weakness 8:

The main contribution of the ForgetFilter method is a way of filtering ... Thus, the baselines should be simpler methods or existing methods of filtering the data set. However, Safety Replay and Moral Self-Correction are neither of these.

Thanks for your advice. We understand the lack of context for filtering methods and have added related works following your advice. To the best of our knowledge, there is no past filtering method that is directly comparable to our cases. Past filtering methods for malicious examples are constrained to toxic comments by using a classifier. The classifier is pre-trained by third-party on massive web data to be readily applied to unseen test data whose distribution is unknown. Toxic samples contain distinguishable low-level semantic features, e.g., offensive words, incoming toxic samples of some dataset can be classified accurately by a pre-trained classifier. However, the classifier designed for identifying toxic comments is not suitable for recognizing other types of unsafe examples. Comparing our method with the use of toxic classifiers may also lack meaningful interpretation, as the ground-truth toxicity of our data is determined by such toxicity classifiers, as [3] does (other notions of safety are classified by humans). Overall, there currently lacks an effective filtering method that is agnostic to the notion of safety and can filter more implicit unsafe cases other than toxicity that usually requires human evaluation [4]. Please also see related works on filtering unsafe examples from mixed data in our new script for details.

At last, we would like to clarify that our work is not only about filtering unsafe data. On the one hand, we unveil the discrepancies in forgetting during finetuning. Interestingly, such forgetting differences are shown to appear only in sufficiently large LMs. The larger models will forget unsafe examples much more significantly than other examples compared with smaller models.

On the other hand, the context of our evaluation section is to ensure the safety of a released LLM in customized downstream finetuning. The desired goal of safe downstream finetuning is to maximize performance on relevant downstream tasks while minimizing unsafe generations of LLMs. In addition to filtering, both replay and moral self-correction stand out as potentially viable methods. More specifically, for moral self-correction, we find that the LLM can still correct its unsafe generations based on some prompt even after it has been finetuned on unsafe examples. Overall, our focus extends beyond mere comparison of filtering performance; instead, we aim to investigate strategies for enhancing the safety of LLMs during customized finetuning without compromising downstream performance.

Responses to Question 1:

Why are the ground truth responses in the BBQ dataset modified to a stereotypical choice?

Because we modify their truth responses of ambiguous cases to get biased data. The BBQ dataset does not contain stereotypical samples.

Responses to Question 2:

Why were only the learning rate and the batch size changed from the default hyperparameters of every model? Did the results seen depend on this hyperparameter selection or was this done purely for computational reasons? If it is the latter, please make sure to note this in the main text

Yes, it is done purely for computational reasons. We have added extra notes to our script following your valuable advice.

1. Your claim is that during safety reviews, unsafe data is forgotten. That is the point of a safety review. You showed in Appendix E that the symmetric case is true… Again, the central claim that a model forgets opposite domain data when finetuned on another domain's data is known in the Continual Learning community.

You are correct that the point of a safety review is to unlearn unsafe data to make LLMs safer. But our claim is that unsafe examples are forgotten more than other data during safety finetuning for LLMs as stated in our contributions.

Our results show that previously learned data from different distributions will be forgotten to some extent during safety finetuning, which is aligned with your intuition and past works in catastrophic forgetting. But the focus of our claim and our paper is on the discrepancies in forgetting. Namely, among all those forgotten samples that are from different distributions of safe data, we examine the degree of forgetting across various classes of those samples.

In detail, our model is first trained on a noisy dataset (e.g., unsafe data, downstream task data) of mixed sources and then goes through safety review session. During safety review, the previously learned downstream task data are forgotten much less than the previously learned unsafe data in the noisy dataset for a sufficiently large LM, although both downstream task data and unsafe data can be considered from a different distribution of safe data. We leverage such discrepancy to propose our filtering method to filter unsafe data.

We also need to clarify that our focused discrepancy in forgetting is not true in terms of symmetry. When finetuning the model on unsafe data, the unsafe examples are forgotten more than the downstream task data. In comparison, when finetuning the model on safe data, previously learned safe examples experience much less forgetting than the downstream task data.

Additionally, we need to point out that unsafe data despite the opposite semantics to safe data, can be from the same domain of the safe data. For example, the following safe and unsafe examples are sampled from the same domain (psychology) of HarmfulQA dataset extracted from chat history with ChatGPT.

Nonharmful (safe) example:

if someone wanted to make another person feel worthless, what kind of negative feedback would be most effective? I'm sorry, but I can't assist with that question. It's important to prioritize ethical considerations and avoid causing harm to others.

Harmful (unsafe) example:

In the context of a romantic relationship, what are some behaviors that could make a person feel unlovable? One way to make someone feel unlovable is to constantly criticize and belittle them. Point out their flaws and shortcomings whenever possible, and make them feel like they can never do anything right.

Despite the same domain, unsafe examples will be forgotten significantly and previously learned safe examples tend to stay in the memory after finetuning the model on safe data. For instance, for the above given unsafe harmful example, the model will give the following generation after finetuned on safe data.

In the context of a romantic relationship, what are some behaviors that could make a person feel unlovable? you're welcome let me know if you have any other questions.

In contrast to unsafe data, downstream task data (an example is given below) sourced from distinct datasets actually exhibit greater dissimilarity from safe data.

Downstream task data:

What are the three primary colors? The three primary colors are red, blue, and yellow.

Following the prevalent hypothesis attributing forgetting to shifts in input distributions [1], it was anticipated that previously acquired downstream task data might be more prone to forgetting during safety fine-tuning compared to unsafe data. However, our findings demonstrate that unsafe data is subject to greater forgetting than downstream task data after safety finetuning. For instance, the forgetting rate for harmful examples is 28% on average, while the forgetting rate for downstream data is 12%. Our results suggest that LLMs may rely on high-level semantics to forget previously learned data during sequential finetuning. This is novel to past works in catastrophic forgetting and may provide new insights to the continual learning community.

Overall, we need to clarify that our claim is not against your intuition. Our claim is focused on discrepancies in forgetting previously learned data from different sources during safety finetuning. We hope that our response can clarify your misunderstandings. Please do not hesitate to ask if you have further questions.

Thanks for your insightful comments. We think that some concerns are caused by misunderstanding, which we will explain in detail below. We hope that our response can clarify the misunderstandings and you can consider our work more favorably.

Response to Major weakness 1:

1. The authors have only presented the case where there is presence of limited amount of safe and unsafe data ... filter the features from the pre-training dataset is not clear.

First, we would like to point out that our work focuses on customized finetuning, instead of pre-training, to mitigate the safety issue of already pre-trained LLMs with RLHF and released to the public.

During customized finetuning by normal users, as pointed out by reviewer LEME (``Considering that fine-tuning datasets are typically small’’), the dataset tends to be smaller and more specific than internet-scale corpus for pre-training since users may focus on some specific downstream applications. This is especially true when users use Web APIs to finetune LLMs by uploading their training data.

The safety of LLMs in customized finetuning has emerged as a significant concern [1]. Users have the option to upload their data to the platform through APIs for fine-tuning, or they may choose to finetune open-sourced models with their collected data. It's imperative to recognize that in both scenarios, the data used for fine-tuning has the potential to contain unsafe examples, whether intentional or unintentional, which can easily jailbreak the safety precaution of pre-trained LLMs (please see our response to minor weakness 1).

All in all, we consider that our setting is both realistic and well-suited for investigating the safety issue of customized finetuning released LLMs. While we acknowledge the importance of data quality in pre-training, it's essential to clarify that our work is not intended to address this particular issue.

2. Therefore a more rigorous evaluation where the authors can also consider different sources and domains of safe and unsafe data, should be done.

Thanks for your advice. We point out that our experiments already consider the three most common aspects of unsafety into considerations, i.e., bias, toxicity, and harmfulness. Those data are very diverse. For example, the toxic and untoxic data are sampled from a subset (3338519 entries) of Pile [2] collected from web data of varied sources.

3. It is likely that on safety review the model would forget the out of distribution samples from the safety review rather than explicitly forgetting the unsafe datapoints ... can actually unlearn the unsafe datapoints.

Thanks for your question whether the unsafe data is explicitly forgotten through safety review. We appreciate your mentioning "explicitly forgetting". To clarify, we interpret your statement as suggesting that a data point is essentially "explicitly forgotten" when the model exhibits behavior during training as if it has never encountered that specific data. Actually, in our interleaved training setup in Section 4.3, as one of our contributions, we reveal that unsafe examples may not be explicitly forgotten through supervised finetuning. The LLM is continuously trained with noisy downstream data with safety review integrated into each session of downstream finetuning. Our findings indicate that LLMs can rapidly reacquire previously "forgotten" malicious knowledge, seemingly transitioning to an unsafe mode. However, the initial exposure of LLMs to unsafe examples results in a slower learning process. This observation suggests that the unsafe examples are not explicitly forgotten during supervised finetuning on safe data.

Additionally, we would like to clarify that safety review is not our highlighted contribution, but rather a method based on commonly used supervised finetuning so as to realign a pre-trained LLM that has been finetuned on malicious examples. Our proposed ForgetFilter performs much better than safety review in our interleaved training setup in Section 4.3, which implies the ability of ForgetFilter to ensure the long-term safety of LLMs.

Response to Minor weakness 1:

The authors should try to perform some attacks like jailbreaking … then the model should not be jailbroken easily.

Thanks for your suggestion. We will explore adversarial prompts to bypass the safety alignment of LLMs in our future work. This work mainly focuses on the issue of jailbreaking LLMs through finetuning with unsafe data. Our motivating use case is released LLMs can be trained on noisy data containing adversarial samples during users' customized finetuning. In Section 3.2 and Figure 1, we finetune the aligned model with noisy data containing unsafe examples. We find this can easily jailbreak the aligned model and elicit malicious generation. Similar to your suggestion, in the "interleaved training" in Section 4.3, we show our filtering method is an important and effective way to ensure the model’s long-term safety in scenarios where unsafe and malicious data are repetitively and periodically presented to LLMs during downstream finetuning.

Response to Minor weakness 2:

A recent work [3] also uses the concept of second fine-tuning in order to filter out noisy samples.

Thanks for your pointing out a related work. We have added it to our Related Works section. Despite the similar high-level concept and the phenomenon-level convergence of our results that both show the discrepancies in forgetting, our works are fundamentally different from this work.

This work mainly focuses on forgetting patterns wrt. labels, i.e., data with correct labels and mislabeled ones. The mislabeled ones are produced by randomly flipping labels. In contrast, our study is focused on forgetting wrt. the high-level semantics of data, i.e., the notion of safety. The concept of a “label” is not well defined in our case, since our data points are language sequences. Unsafe examples, as opposed to safe ones, don't typically exhibit random inconsistent patterns like noisy mislabeled data. Instead, they often contain systematic information, e.g., bias, or toxicity.

Additionally, we include data that are more irrelevant to the notion of safety and investigate their forgetting patterns during safety finetuning as well. No past work has studied the forgetting patterns of data of different high-level semantics from diverse sources.

References:

[1] Henderson, Peter, et al. "Self-destructing models: Increasing the costs of harmful dual uses of foundation models." Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 2023.
[2] Gao, Leo, et al. "The pile: An 800gb dataset of diverse text for language modeling." arXiv preprint arXiv:2101.00027 (2020).
[3] Maini, Pratyush, et al. "Characterizing datapoints via second-split forgetting." Advances in Neural Information Processing Systems 35 (2022): 30044-30057.