We would like to thank the reviewer for their helpful and thorough review, and for their positive comments. We’re glad you think some of our insights could potentially inspire further research and insights into large language models within the community. We address the concerns raised in the review below.

1. However, based on the experimental results, it appears that the entire LLM behaves highly in coupling and cannot be separated. For instance, in Figure 1(d), the performance loss on the 'code' dataset seems to be mirrored closely by a performance loss on the 'python' dataset

We think that the extent to which different capabilities are coupled or intertwined depends on the capabilities in question. By comparing the results in Figure 2(a) and 2(b) we see that ‘python’ ability and ‘code’ ability are more intertwined than ‘code’ ability and ‘pile’ (i.e. general text) ability. This is in line with what one might intuitively expect: coding ability in general and coding in the python programming language seem like similar skills.

2. Using varying scales for the reduction in perplexity complicates the evaluation, and it's challenging to determine whether a reduction from 4.8 to 0.8 is significant, or if a drop from 2.2 to 0.3 is more significant.

We agree that it is inconvenient that we were unable to replicate their results. In an attempt to resolve this we have contacted the authors in August, but unfortunately they have not replied.

We have tried to further vary text generation parameters that were not stated in the paper, and by increasing the temperature and generation length, we managed to rerun the experiment to find pre-intervention toxicity levels that are closer to 4.8. The updated paper now reports a drop in toxic generations from 3.5% -> 0.3%.

3. Given that the experiments were solely conducted on datasets related to code, I am uncertain about the generalizability of the experimental results presented in the paper. For instance, the conclusion that FFN outperforms attention—might it be possible that a different task could yield an opposing conclusion.

We agree that the degree to which attention and feed forward pruning may be specialized to different tasks may differ. Thank you for pointing this out. We have updated the pdf to reflect this.

4. The readability of the entire article is not good. For instance, in Section 3.1, the author describes the distribution characteristics of the "attention pre-out neuron" activations. However, the definition of this unfamiliar term, "attention pre-out neuron," is only introduced in Section 3.3. This leads to confusion for me when initially encountering the term. … While thorough analyses and experiments are commendable, the structure of the article still needs further refinement to enhance its logical flow.

Thank you for the feedback. We have resolved the issue of using the term “attention pre-out neuron” before we explain the term. We have also made other updates to the paper to improve readability, such as improving the wording and adding a diagram describing our method on page 3.

Question 1 and Question 2: Question 1 is addressed in the comments above and question 2 is answered in the common concerns section.

Question 3:

A prior study [1] demonstrated that, depending on the specific input, it's possible to achieve a high pruning ratio without negatively affecting performance and without the need for retraining. … In light of these findings, what novel insights or observations does your paper offer in comparison to that study?

This paper is very interesting indeed! We have now cited this paper. Their findings are similar to but more general than the ones in [A] which motivate our importance functions. The paper shows contextual sparsity per input sentence, whereas we show something similar per dataset.

The paper shows that neurons specialize locally on the scale of tokens and sentences. We show that neurons specialize more globally on the scale of broader tasks and datasets.

In Figure 1 (now Figure 2) we have now added imagenet data, and we think it is interesting that the three different pairs Pile-Coding, Coding-Python, Imagenet-Bird all have very different amounts of separability. This shows that the separability of neurons is task dependent.

[A] MoEfication: Transformer Feed-forward Layers are Mixtures of Experts, Zhang et al. https://arxiv.org/abs/2110.01786

Conclusion

We hope the above clarifications and additional results have changed your view on the paper and successfully addressed all the questions and limitations you mentioned. Please let us know if there are other concerns stopping you from increasing your score and recommending acceptance of the paper.

We thank the reviewer for their detailed and helpful review, and for their positive comments. We’re glad you find we study an interesting and important problem and that the proposed method is well motivated, efficient and requires no gradient updates.

[A-E] are recent papers on unlearning

Can’t this method be applied out of the box e.g. to vision transformers?

[A] This Arxiv paper from August 15th is related and interesting! In a quick trial we applied our method to the Vision Transformer (ViT) mushroom (MR) row in their Table 2. On the test split (100 datapoints) of CIFAR100 we find a forget drop from 72±8% to 0±3% and a retain drop from 90.0±0.3% to 89.5±0.3%. [A-E] Cited.

Common baselines include finetuning on the retain set, gradient ascent on the forget set

Reference [B] states that fine-tuning and gradient ascent based machine unlearning methods cost between 2% and 6% the cost of retraining a model. For reference, training Llama-2-70b used 1.7million GPU hours.

How much do we want to reduce performance?

While we agree that retraining is a good standard, retraining an LLM is too expensive. We simply try to maximise drop in forget while keeping performance in retain, seen with a "curvier" graph being better. We do not insist on a specific goal, but make a cheap reference, and allow the user to decide where they would like to be on the trade-off between a high retain & low forget accuracy.

Were Pile and Code included in the dataset that the various language models were trained on? If not, the terms “forgetting” or “unlearning” may be ill-suited for this application.

It's uncertain which specific data points were used in the training of all models (e.g., LLaMA 2). Some models, such as Pythia and OPT, do incorporate parts of the Pile dataset.

The terms 'retain' and 'forget' are existing terminologies that mostly capture the meaning we want. Though it may slightly deviate from normal usage, inventing new terminology would likely be more confusing. Rather than referring to the forgetting of specific instances encountered during training, we're focusing on the loss or retention of underlying skills as exemplified by a dataset. Our goal is to remove harmful behavior as exemplified by a (forget) dataset. For example, GPT might not have seen “To make a bomb you need x, which you can get at location y”, but the model may still output such strings.

Membership Inference Attacks are an important category of methods...

In this paper we have tried to show that harmful behavior can be reduced by applying selective pruning. We are less focused on removing specific datapoints from the model than we are in removing the model’s ability to generate code or toxic data. Studying membership inference attacks (which requires full knowledge of training data) is beyond the scope of our current analysis.

I don’t agree … the larger the model, the more selective it is.

Thank you for pointing this out. We agree and we have updated the pdf to reflect this.

[presentation] … I_{abs} … Table 1,

Thanks! Added.

It would greatly strengthen the paper to conduct analyses of: the effect of the number of pruning iterations, the effect of the % pruned (in each iteration or overall), the effect of the choice of the influence function.

We have chosen not to do a grid-search for the hyperparameter “percentage pruned per iteration”, but instead report on all the hyper-parameters that we tried. The effect of how much has been pruned overall can be seen in Figure 2. Every dot here reflects another pruning step. The more pruning steps the bigger the reduction in accuracy. Figure 4 shows the effect of the choice of influence function.

please include confidence intervals

In our tables, we apply the 'significant digits' convention, where we round our values to the most relevant figure and omit digits that represent a substantial error margin. We have taken the suggestion to add error bars to Figure 4.

Which dataset / setting were these observations made in?

These were taken using OPT on the Pile dataset, we have verified the observations for Pythia and LLaMA.

“We also do not want to directly modify specific dimensions of the embedding space”

After every layer, the model writes into the residual stream which has the same dimension as the embedding space (and which at the last layer turns into the model output after the unembedding). Investigating how to alter these dimensions that are being continuously read from and written into would be an interesting investigation, but is beyond the scope of this paper.

In the Discussion, the authors hypothesize that their method is more likely to “actually remove the undesired behaviour” compared to other unlearning methods...

We compare to other model control methods more broadly, many of which have been shown to be brittle under jailbreaking and finetuning. We hypothesize that (our) machine unlearning methods are less brittle than these control methods.

We would like to thank the reviewer for their detailed review, and for their positive comments. We’re glad you find the presented method novel and our experiments thorough, and that the presented results convincingly demonstrate the utility of the method. We are also happy you found toxicity removal a highly relevant domain. We address the concerns raised in the review below.

Given this evaluative dependence upon specific datasets, it would significantly strengthen the results of the paper to present more diverse experiments. While it is nice to have many different models at many different sizes, there does not seem to be much additional knowledge gleaned from that diversity, whereas having more tasks would demonstrate a broader efficacy across a more speculative dimension.

We have now included results for an image classification task. Please see them in the updated version of Figure 1 (now moved to be Figure 2) of the updated pdf.

In this paper, the authors present experiments pruning either the feed-forward or the attention blocks, and leave "Embedding, Positional Embedding and Output Unembedding unmodified." (3.3). This is a significant constriction of the application of the method which is not more than intuitively justified.

The Embedding, Positional Embedding and Output Unembedding are one layer / weight matrix each. Excluding three layers from pruning out of the 30-100 layers that LLMs typically have seems like a minor constriction. The embedding and output unembedding exist to translate from the input (or output) tokens to an internal model representation and almost function like a dictionary. In LLama the positional embedding is not even learned, but is instead hard-coded. This positional embeddings seem like they would be essential for most tasks that one might think of.

In Table 3: it would help readability to label "Task Arithmetic (quoted)" as being a finetuning task.

Thank you for the suggestion! We have updated the label.

The details of iterative pruning are not specified. It seems like it would be important to the function of the method to set the proportion of nodes pruned per iteration well, but this is not discussed.

We have added some clarification in Section 3.2 in the updated pdf. We have not done any hyper-parameter tuning for the proportion of nodes pruned per iteration.

6.3: If an LLM were trained not to answer questions about a dangerous topic, say, bomb-building, could the presented method not be used to unlearn that guardrail? I don't think this is a concern specific to the presented method, but I do not follow why this method is less likely to generate harmful systems than other methods. Can the authors clarify?

Something that was not clear in our original formulation is that we think machine unlearning methods (such as ours) will be more thorough than other model control methods.

For example, RLHF (a control method) guard rails are fragile to things like jailbreaking [1] and finetuning [2]. This is possible because the information about how to build a bomb is still in the model. This method is an attempt to remove much of that information so that even if one unlearns the guard rails, one can still not learn how to make the bomb.

[1] Deep reinforcement learning from human preferences, Christiano et al. https://arxiv.org/abs/1706.03741

[2] LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B, Lermen et al. https://arxiv.org/abs/2310.20624

Conclusion

We hope the above clarifications and additional results have changed your view on the paper and successfully addressed all the questions and limitations you mentioned. Please let us know if there are other concerns stopping you from increasing your score.

We would like to thank the reviewer for their helpful review, and for their positive comments. We’re glad you found the goal of removing capabilities from LLMs an intriguing and complex issue, and our method a cost-effective foundation for forthcoming research comparisons, the efficacy of which was demonstrated through experiments on three distinct datasets. We address the concerns raised in the review below.

1- The proposed model can effectively eliminate the information captured by the target dataset. However, it is unable to unlearn knowledge that lies beyond the representation of the datasets. Please provide a clear justification.

Yes we think this is an important question! However, we think that the general question of capturing (un)desired behaviour in a dataset is one of the key problems in machine learning. It is outside the scope of this paper to solve this problem here.

2- Including a visual representation of the suggested concept could enhance comprehension. Thus, kindly incorporate a diagram that presents a general outline of the proposed concept.

We have now added a visual representation (new Figure 1) of selective pruning to Section 3.2 on page 3 of our updated pdf.

3- The evaluation of the proposed model focuses on text datasets, yet the experiments for the image dataset are absent. Implementing machine unlearning for the image dataset would be highly beneficial.

We have taken your suggestion to implement selective pruning for an image dataset on board! Please see our updated Figure 1 (now Figure 2) in the updated pdf.

Conclusion

We hope the above clarifications and additional results have changed your view on the paper and successfully addressed all the questions and limitations you mentioned. Please let us know if there are other concerns stopping you from increasing your score.