Unforgettable Generalization in Language Models

Thank you for your insightful comments and suggestions. We will revise our paper as follows.

"How different is this random label distribution from the original label distribution? What happens if the majority of labels match the original labels? Also, I would also like to know how the much variance is there in the unlearning process because of this randomness."

When preparing our datasets, we binarized our multiple-choice questions to always have two answers: the correct answer and a distractor. Thus, when we randomize our labels each epoch, there is a 50% chance of it being the correct label. Note that the effect of randomizing our labels every epoch when combined with the fact that we are training on about 1000 examples for each task means that it is highly unlikely that a significant majority of labels match the original labels. The variance of label selection really is not that significant given all this repeated randomization.

To demonstrate this, we also have a new experiment where we train on flipped labels instead of using randomized labels. Thus, in this setup, there is no variance due to randomized label selection.

The analysis is the same, except now we compute the forget ratio as: $\frac{\text{Fine-tuned Accuracy} - \text{Forgotten Accuracy}}{\text{Fine-tuned Accuracy} - (1- \text{Fine-tuned Accuracy})}$ since we assume that the minimum accuracy achievable should be $1- \text{Fine-tuned Accuracy}$.

As shown in the figure, the trends are the same. The same tasks that are robust to fine-tuning on randomized labels continue to be robust on fine-tuning on flipped labels. Thus, our results are not specific to the choice of randomized labels, but are rather a property of how fine-tuning and the tasks interact.

"Some of the discussions about the experiments are vague/ambiguous. It is really hard to understand what the author means by a LM's "forget capabilities" or "unlearning of skills". It would be nice to explain these more in the paper or use different wording completely that is clearer."

Thank you for pointing this out. We will revise our paper and explain those terms. Specifically in regards to "forget capabilities" and "unlearning of skills," we use these interchangeably. In practice these refer to the out-of-sample performance on the tasks degrading--- which is what we call forgetting. We will be more careful with defining these terms and using them consistently.

Thank you for your insightful comments and suggestions. We will revise our paper as follows.

"I feel that the work would be improved if a depth-first approach to the analysis had been taken, i.e. if more resources had been devoted to understanding why certain tasks are always forgotten"

We agree that there is more work required to understand why certain tasks are always forgotten when other tasks are not. We did try other more bespoke approaches (including analyzing distances in representation space) but found no more success than in the simple factors already presented.

As you have stated, our main observation is that

fine-tuning on random labels does not always generalize well across tasks but instead that certain tasks are almost always forgotten during cross-task forgetting is an interesting and important observation.

Our main contribution is observing this and considering immediate explanations that might come to mind. This intriguing finding definitely raises more questions for follow-up work.

Thank you for your insightful comments and suggestions. We will revise our paper as follows.

"The discussion of prior work and relevant literature is somewhat lacking."

We took the suggestion to expand prior work to heart and will do so. For arxiv.org/abs/2403.03218, they use the same term "generalization" but study a different kind of generalization--- not breaking unrelated tasks. Our paper focuses on generalization within a single task. We will add this discussion.

"The paper only looks at one method (training w/ random labels)."

Instead of studying generalization in all possible unlearning methods, we focus on fine-tuning which is simple and a common baseline.

Additionally, fine-tuning is especially interesting because it may reveal connections between learning and unlearning (e.g. there seems to be wide vs narrow tasks, which we imagine would have consequences in learning).

To further address this, we replicated our results with a new experiment using flipped labels instead of randomized labels. The analysis is the same, except now we compute the forget ratio as: $\frac{\text{Fine-tuned Accuracy} - \text{Forgotten Accuracy}}{\text{Fine-tuned Accuracy} - (1- \text{Fine-tuned Accuracy})}$ since we assume that the minimum accuracy achievable should be $1- \text{Fine-tuned Accuracy}$.

As shown in the figure ibb.co/P6Vp038, forgetting trends are the same. Our results are not specific to the choice of randomized labels.

"Only one model."

We conducted a new experiment with GPT-J-6B. As shown in ibb.co/pRfwByd, the forgetting ratio trends are broadly the same, so the behavior is not unique to LLaMA-2-7B.

Moreover, to test what happens to models of different sizes, we also run a new experiment on GPT-2 with 124M parameters. As shown in ibb.co/TK9x7j4, the forgetting ratio trends still remain the same.

"Do you have a hypothesis/guess for why it doesn't matter what dataset or task you use?"

We suspect that certain capabilities are inherently more fragile, so fine-tuning on any incorrect data is harmful. This resonates with recent findings that highly quality datasets are important.

"How was the threshold of 0.6 determined?"

We picked a threshold of 0.6 which is between 0.5 and the original confidence. The experiments are robust to thresholds in that neighborhood.

Typos, clarifications, and adding data

Thank you. We will amend the typos, make the suggested clarifications, and add the requested results to the appendix.

Thank you for your insightful comments and suggestions. We will revise our paper as follows.

Method of unlearning is limited to fine-tuning with randomized labels.

Following your question, we ran another experiment where we used flipped labels instead of randomized labels. The analysis is the same, except now we compute the forget ratio as: $\frac{\text{Fine-tuned Accuracy} - \text{Forgotten Accuracy}}{\text{Fine-tuned Accuracy} - (1- \text{Fine-tuned Accuracy})}$ since we assume that the minimum accuracy achievable should be $1- \text{Fine-tuned Accuracy}$.

As shown in the figure, the trends are the same. The same tasks that are robust to fine-tuning on randomized labels continue to be robust on fine-tuning on flipped labels. Thus, our results are not specific to the choice of randomized labels, but are rather a property of how fine-tuning and the tasks interact.

Extending the experiments to another language model besides LLaMA might prove more insights into the generalizability of unlearning.

We conducted a new experiment with GPT-J-6B, which is a slightly weaker model than the LLaMA-2-7B model we used. As shown in the figure, while GPT-J has a lower fine-tuned accuracy, the forgetting ratio trends are broadly the same. Thus, the behavior is not unique to LLaMA-2-7B.

Moreover, to explicitly test what happens to models of different sizes, we also run a new experiment on GPT-2, which has 124M parameters (98% smaller). As shown in the figure, GPT-2 tends to have significantly lower fine-tuned accuracy, but the forgetting ratio trends are broadly the same even in the smaller model size regime.

We will add these experiments and their discussion.

As the generality of our findings was judged to be the main weakness, we hope this cleared the doubts about the paper.