Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces

We thank the reviewer for their favorable review and insightful comments. We are encouraged the reviewer appreciates our contributions and sees the merits of our work.

Regarding the weakness points raised:

W1: Thanks for this thoughtful comment and suggestion. We will add an explicit specification of the amount of parameters modified by each method. Regarding maximizing the intrinsic evaluation without changes in the model’s behavior – this is an important point. We agree with the reviewer that a complete erasure of the concept vector does not provide the fully desirable behavioral effect (e.g. very low BLUE score, full resistance to jailbreak attacks). However, given the very low performance of current unlearning methods on ConceptVectors, we believe it is a strong benchmark that makes the first step toward parametric removal.

W2: Yes, our data selection process focuses on vectors that are easily located, very disentangled and show clear concepts that are causally related to the concept, which may be easier to remove. However, we see great value in these vectors for evaluation, as our empirical results show that even in these cases, unlearning methods perform badly in terms of removing parametric knowledge, showing there is a large room for improvement of current unlearning methods. Using less prominent vectors could be a valuable direction for future work to explore.

W3: Thanks for this comment. The reviewer’s idea of prioritising causal effectiveness first and then interpretability is intriguing, though it will require changing the whole data generation process. We would also like to point out that this approach has a risk – we do not know in advance which vectors are polysemantic and capture multiple concepts in superposition, but this is expected to be prevalent based on prior findings. Consequently, we may need to explore many concepts before finding those that their interpretation is very clear. In that sense, we believe our approach is more direct for locating these specific concept vectors.

W4: We agree that evaluation that is based on adversarial attacks is valuable as well, though similarly to behavioral evaluations, it relies on the model’s outputs without directly targeting the source of the knowledge.

W5: We thanks the reviewer for appreciating our experiments. We are happy to add more samples to these experiments and doing this analysis of the LRL attack, and will try to do so during the next few days. However, given that we have received 7 reviews, we may not make it on time.

Regarding the reviewer’s questions:

Q1: Thank you for raising all these thoughtful points. Please see our responses to W1-W5 above, where we addressed them.

Q2: As explained in lines 188-190 (Section 3.1), we do not include lexical or syntactic concepts. Our focus is on semantic concrete concepts that the model would capture facts about.

Q3: Previous work showed that the MLP layers in transformer-based language models can be cast as key-value memories. We followed this notation.

Q4: Thanks for this great question. We haven’t made this kind of comparison and are not aware to other works that did. There are two important things to note here: First, in terms of computation and evaluation, localizing concept vectors through ablations is substantially more expensive and complex than vocabulary projections. It requires executing the model for every vector, crafting inputs that could be used for testing the possible encoded concepts, and measuring the effect on the output is not always trivial. Vocabulary projections provide a simpler and more direct way to identify the encoded concepts. The second thing to note is that practically we do both, as step 3 in our pipeline causally validates the concept identified with vocabulary projection.

We thank the reviewer again for appreciating our work and for their valuable comments! If the reviewer has any further questions or follow-up thoughts, we are happy to engage in discussion.

We thank the reviewer for reviewing our work. We believe the reviewer misunderstood a few key points, including the overall idea of using concept vectors for evaluation and the proposed data generation pipeline. We attempt to clarify these points next and would be happy to answer any follow-up questions from the reviewer.

W1 + Q1 (problem definition):

We would like to firstly position our work within the broader literature in NLP. Indeed, the term “unlearning” conflates the formal notion the reviewer points to with the more empirically driven notion of this term in NLP. This paper follows a long series of work in NLP, which pointed out that factual knowledge (such as the identity of the capital of London) is stored in the fully connected layers in transformers. This observation sparked interest in “unlearning” this knowledge, i.e., making the model behave as if it no longer possesses it. This is similar to the problem of “unlearning” in general ML, but is naturally more empirical, as it is almost always not practical to retrain large LMs from scratch without being exposed to the concept.

To be more concrete, our work follows the notion of unlearning or removal of from recent works about unlearning in LLMs [1, 2, 3, 4]. Given a concept c, which is typically a named entity, the goal is to remove any knowledge the model has about c. One common way to formalize “knowledge about c“ is through a knowledge-base like Wikidata, where facts are subject–relation-object triplets (e.g., the triplet <”Barack Obama”, year-of-birth, “1961”> would correspond to the fact “Barack Obama was born in Hawaii”). With this formalization, the task is essentially to remove any facts the model has where the subject is c. This setting has been used widely to formalize problems of knowledge editing and removal, and particularly was used for the MEMIT baselines employed in our work. While we agree that this formulation may be refined, this is not one of the contributions of our work, but rather this is the setting which has been studied recently and we largely build on top of it. Please note that our evaluation does compare the model responses/parameters before and after unlearning.

W2 + Q3 (evaluation method):

Needle is not an unlearning method, it is an oracle baseline. The reason it cannot be used as a general unlearning method is that it assumes the concept vector is given, while in practice it is not given and may not be trivial to find. Namely, finding the vector corresponding to a specific concept is challenging and generally an unsolved problem (due to superposition and features that are not aligned with standard bases, as we discuss in the paper’s limitations paragraph). However, it is often easy to identify vectors that encode some concepts, which is exactly what our paper leverages. If there’s some vector identified as capturing knowledge about Harry Potter – we want an unlearning method to remove this knowledge such that it is not recoverable anymore. Needle is applied as a baseline to demonstrate the importance of locating and removal of this residual knowledge, as we demonstrate in the context of jailbreak attacks.

W3 (manual nature):

The data selection process is mostly automated and systematic (which was appreciated by Reviewer 9s3u). There is indeed a manual selection phase, which aims to find the vectors that correspond to the most coherent concepts based on human judgment, followed by a causal validation step. While it does introduce a bias towards concepts that can be easily located (which we acknowledge in the limitations paragraph), our empirical results show that even in these cases unlearning methods perform badly in terms of removing parametric knowledge. This shows there is a large room for improvement of current unlearning methods. Regarding reproducibility – given that the benchmark relies on open-weight models, reproducing our pipeline is very easy. We also provide the specific indices of the located vectors.

We thank the reviewer for the prompt response and for engaging in discussion.

Regarding the statement: "a good unlearning method would match the effect on the parameters that the 'oracle' method Needle has"

Please note that this is inaccurate. Needle is an oracle in the sense that it has access to one (of potentially multiple) vectors capturing knowledge about the concept, but it may not necessarily introduce exactly the desirable effect. This is because (a) there may be more vectors capturing knowledge about the concept, in which case Needle won't erase all the relevant knowledge, and (b) the way through which Needle modifies the concept vector (i.e., adding noise + training) may be artificial and incoherent with the model's internal representations.

As the reviewer wrote -- the metric is based on changes in the ConceptVectors and specifically how their reflection through projection to the vocabulary. Needle is just one way to introduce changes to these vectors, which may not be the best one.

Also in this sense, MEMIT serves as a strong explanation and a compelling argument, whose training style is more thoughtfully designed. It achieves significantly better results on our ConceptVectors metrics compared to other fine-tuning-based unlearning methods (Jaccard Similarity on LLaMA: 0.769 vs. 0.985) and also demonstrates superior unlearning performance in behavioral tests (Table 3 and Figure 3) and enhanced unlearning robustness against various adversarial attacks (Figure 9).

Regarding keeping model utility

• We agree that the model utility should be taken into account. This is part of the reason for including an evaluation of unrelated concepts in our benchmark, namely, we want to preserve the ability of the model to generally answer questions (i.e. generate text coherently and correctly) and the unrelated knowledge.
• With respect to Needle:
  • please note that it is just a rough baseline without sophisticated training: it operates only by adding random noise to disrupt the target concept vector and then applying simple fine-tuning adjustments. Also of note is that Needle will not hurt the model generating ability on the other unrelated knowledge questions.
  • If necessary, we could design a new goal concept vector containing harmless knowledge to replace the original concept vector. This would allow for a reasonable adjustment and transformation of the knowledge stored in the original concept vector, enabling unlearning while avoiding nonsensical outputs when responding to the same knowledge-related questions. In this way, the model would output the carefully designed answers.
  • However, this is not the goal of our paper. Our contribution is not to propose a perfect unlearning method but to highlight the direction that proper unlearning should take in the future. Specifically, it should involve making reasonable adjustments to the parameters where the knowledge is genuinely stored within the model. Only by doing so can more thorough unlearning be achieved, leading to stronger robustness against adversarial attacks.

Regarding the problem setup and the suggested filter approach

We are sorry to hear that the reviewer is not satisfied with this setup, especially since problems concerning editing and unlearning of concept knowledge have been widely studied in the research community in recent years -- example references are provided below. Currently, there are already various methods that can bypass such filter defenses.

• The suggested filtering approach is not expected to work very well, since it is possible to extract information from the model without explicitly stating the concept name. For example, instead of feeding "Barack Obama", we could input the 44th president of the US.
• Alternatively, the input sentence can be multilingual, in which case it may be difficult to design a filter that can filter all the low-resources languages. However, our experiments in Table 4 show proven that even a low-resource language will also apply high activations to the specific concept vector.
• There are already many jailbreak methods capable of constructing nonsensical jailbreak prompts, often consisting of random gibberish, scrambled characters, or disordered letters [6, 7]. In such cases, simple filters struggle to detect these prompts effectively, yet they can still successfully induce the model to output the target knowledge. Our experiments have shown that, even in these scenarios, such prompts similarly enhance the activation of the concept vector corresponding to the target knowledge, making them identifiable at the parameter activation level (please see the experiment in W3 to 6fAd).

Thank you once again for actively engaging in discussion with us.

We thank the reviewer for the prompt reply, for their patience, and for connecting their arguments to the points by Reviewer 6fAd, which we just responded to.

Concept vectors can't be used to benchmark unlearning, the causal links are weak

We respectfully disagree. Needle introduces a causal effect to the model's predictions and a substantial decrease in the behavioral tests. We argue that a decrease of >0.5 in the BLEU score is significant, even if it's less prominent than other methods.

Overall, this line of work is intersting, but there is a lot that needs to be done to take what is essentially an unverified interoperability method and use it for measurement like this

We appreciate that the reviewer finds our work interesting and agree that there is still a lot to be done -- we would like to emphasize that our work is, to the best of our knowledge, the first work to merge between interpretability methods and evaluation of unlearning and use parameter interpretation methods for this purpose. We see great potential in this direction and believe that our jailbreak experiments and analysis demonstrate its utility. As both unlearning and interpretability methods improve, the quality of benchmarks in this line will be improved as well. But one must start from somewhere and part of the message of this work is to argue that such intrinsic evaluations should be taken into account. We believe that the consistent trends and clear observations from our experiments provide evidence for the potential of this avenue.

We thank the reviewer for their thorough and constructive review. We appreciate they found our data pipeline excellent, the experiments thorough, and the presentation compelling. We address the points and questions raised.

W1:

• “the main qualitative evidence that concept vectors encode the knowledge of a concept in the models is not very clear” – our approach relies on several previous works that have established the existence of MLP vectors encoding specific concepts [1,2,3]. These studies systematically evaluate the presence of such vectors through vocabulary projections, and showed causal evidence of their contribution to the model’s predictions. Please also note that step 3 in our data pipeline is also meant to check exactly this – we causally validate that erasing the concept vectors influences the model’s ability to answer questions about the concepts but not on other concepts.

• “The numbers indicate that concept vectors are not really erasing the knowledge” – We believe the reviewer may have misunderstood these results and we would like to explain. Our results (Table 3) show that Needle obtains mean target bleu scores of 0.46 and 0.31 for the two models. These are higher (worse) than the other unlearning methods (which get to 0.05-0.24). However, there are two important things to keep in mind:

  • Needle only ablates a single vector, which constitutes <0.001% of the network parameters. Other methods are either not restricted and modify all the parameters (Finetuning Unlearning) or modify the MLP layers including the layer where the concept vectors are (MEMIT). For example, MEMIT modifies a total of 44K MLP vectors (44K times more vectors than Needle), including the concept vector itself. Therefore, we consider the unlearning effect that Needle introduces to the model’s ability of generating concept knowledge is dramatic, showing that it does capture essential knowledge about the concept. As a reference, ablating a random vector in MLP layers keeps a BLEU score of 1.0.

  • The second important point is that we do not expect that the concept vector is the only parameters in the network that capture knowledge about the concept. Previous works show that knowledge is spread across multiple places in the network [3,4], so overall it is expected that ablating a single vector would not erase the knowledge completely. Needle is a baseline rather than a method for unlearning, and it is mostly to demonstrate the effect of directly ablating the parametric knowledge.

It is important to note that, even though the concept vector is not necessarily the only place where a piece of knowledge is stored, it is still useful for evaluating unlearning – which is what we demonstrate in this work.

W2: “It is slightly unclear what the concept vectors are really encoding?”

This is an interesting question but note it is orthogonal to the premise of the paper. Namely, as long as the concept vector captures unique knowledge about the concept, it should be removed by unlearning. Still, the fact that causally removing the concept vector generally prevents the model from answering questions about the concept may show that it captures general knowledge about it. Also, the concept vectors do not contain general grammatical or syntactic knowledge, as their removal does not affect the model's language modeling abilities and generating valid responses about unrelated concepts.

Regarding the questions raised:

Q1: “The text for Table 3 is very confusing.” – We apologize for the confusion! Please see our response above (the second point in our response to W1) for an explanation of why the decrease by Needle is non-trivial. Another thing to note is that BLEU and ROUGE are generally widely used scores for string matching, commonly used to compare a generated output with a reference string – 0 means the generated response is entirely different from the model’s original response, and 1 means it is exactly the same as before unlearning. So while a target QA score of 0.25 means a more dramatic decrease in the model’s ability to generate concept knowledge compared to a score of 0.5, a 0.5 score is also substantial as it means the responses of the model to questions about the concepts have substantially changed and often do not include the needed information.

I appreciate the authors responding to all the questions! Especially, Table 3 makes a lot of sense to me now. Thanks for the clarity!

The response by authors has made it clear that the metric used is measuring the text similarity between edited and original model's generation. While I do agree with author's view that: "truly unlearnt model should not exhibit concept vector" - I think there is also a more important point that Table 3 could make. "is concept vector truly representing the concept?". If concept vector truly captured the concept - I would expect that ablating concept vector would naturally reduce QA performance on the related subject but not on unrelated. But BLEU scores validate only text similarity - so it makes it slightly unclear if the metric is validating the correctness of the answer or the change in the answering style.

How do the authors argue that BLEU is precisely measuring the correctness of the answers?

EDIT: I missed the response from authors where they measure the MCQ accuracy and found that the answers are random when ablating NEEDLE. I very much appreciate this experiment! and I believe it answers the question of "correctness". But the sample size of 10 might not be enough to reach to a stronger conclusion. I do believe it is important to get this abstraction right and show that concept vectors truly capture the semantics of the concept.

Dear Reviewer 9s3u,

Thanks again for mentioning the RMU method [1] in Q4 in your initial review. This message is to update the reviewer that we have conducted additional experiments, evaluating RMU (representation engineering-type method) as part of our main results. Below are the experiment settings and results, which we will make sure to incorporate to the paper in it's next revision.

Experiment Settings:

We evaluated RMU on the ConceptVectors benchmark. The results on the LLaMA2-7B-chat portion are provided in the table below.

There are two points to note:

• The original RMU method will edit the same fixed layers for all the samples (the default hyperparameters are the 5-th, 6-th, 7-th layers in a 32 layers model). So we first report its performance using these original settings.
• To achieve better unlearning effects on the corresponding concept vectors, we also modified the default hyperparameters of RMU, where for each concept we edit the layer containing its concept vector, as well as the two preceding layers. We named this alternative RMU as RMU_enhanced in the following table.

In the table, we further include two representative methods described in our paper, DPO and MEMIT+Empty, for comparison.

Results:

From the results we make the following observations:

1. Overall, in terms of behavioural evaluation, we observe that MEMIT continues to deliver the best performance, achieving the most effective unlearning (lowest BLEU and ROUGE scores) while also preserving unrelated knowledge well in the model (highest unrelated BLEU and ROUGE scores).
2. In RMU’s original setting, the fixed layers prevent unlearning from effectively influencing concept vectors in other layers. As a result, it performs poorly in terms of our intrinsic evaluation (Jaccard Similarity: 0.999). In terms of behavioural evaluation, its performance is comparable to DPO but outperformed by MEMIT and RMU_enhanced.
3. In RMU_enhanced, we dynamically select the layers for editing for each concept, therefore it has a better intrinsic performance (lower Jaccard Similarity), indicating a more effective influence on the concept vectors. On the behavioral evaluation, the enhanced RMU achieves improved unlearning performance (lower BLEU and ROUGE scores) and better preservation of unrelated knowledge (higher BLEU and ROUGE scores) compared to the original RMU configuration. These results demonstrate that targeting the specific parameters where knowledge is actually stored in the model allows for superior unlearning effects while minimizing the impact on unrelated knowledge.

Additionally, it is worth noting that while RMU_enhanced and MEMIT+Empty show similar performance on the Jaccard Similarity metric, suggesting a comparable degree of parameter modification, MEMIT+Empty achieves better unlearning results on behavior metrics. This can be attributed to MEMIT’s potentially more effective loss function, which introduces more optimal directional modifications and edits to the concept vectors.

We will incorporate these additional results in a revised version of the paper.

Please feel free to raise any points that are unclear to you. We sincerely appreciate your engagement in the discussion and are happy to know if our response and new results change anything about your assessment of our work.

---

References:

[1] The wmdp benchmark: Measuring and reducing malicious use with unlearning. ICML 2024

Thank you so much for actively participating in our discussion and continuing to provide valuable feedback!

Table.3 shows NEEDLE which also does gradient ascent on top of vector noising. But figure.4 doesn't do gradient ascent? NEEDLE seems to be reducing both the target and untargeted content.

Thank you for raising this important point.

About the Experiment Settings:

Yes, in the Figure 4 experiment, we conducted causal validation experiments to verify the causal relationship between the concept vector and its corresponding concept. Therefore, the experimental setup here involved adding noise solely to the target concept vector.

In Table 3 experiment, the Needle approach will apply gradient ascent to fine-tune the target concept vector after the noise adding, while keeping all other model parameters unchanged.

About the Needle performance:

Thank you for noticing this.

To make it easier to compare, we have added the results from the setting that only involves disrupting the concept vector with noise (Figure 4) into Table 3, as destorated below:

From the table above we can observe that:

• Compared to the "Adding Noise Alone" approach, the Needle method can achieve better unlearning performance (lower Target QA BLEU and ROUGE-L scores), but it also indeed results in a greater loss of performance on unrelated knowledge (lower Unrelated QA BLEU and ROUGE-L scores).

• The underlying reason for this is likely: due to the limitation of gradient ascent, where the gradient optimization direction is relatively random and uncontrolled. As a result, it may inadvertently encode unrelated content into the concept vector, which impacts performance on other tasks and leads to a greater effect on the unrelated scores.

• From the table, we can infer again that parameter-based unlearning methods have enormous future potential: the "Adding Noise Alone" approach, compared to all other baselines, creates the greatest difference between target knowledge (Target QA BLEU and ROUGE-L scores) and the model's unrelated knowledge (Unrelated QA BLEU and ROUGE-L scores) (LLaMA BLEU and Rouge-L: 0.415 | 0.301; OLMo BLEU and ROUGE-L: 0.334 | 0.455). This suggests that it achieves the best trade-off between preserving unrelated knowledge and erasing target knowledge.

• This result further strengthens our claim that performing unlearning on target knowledge parameters is the key direction for achieving more thorough unlearning in the future.

Here, we emphasize again that Needle is merely a baseline method we proposed—a preliminary approach rather than a fully optimized solution. It has significant room for improvement. In this paper, we will remove the gradient ascent training step in the Needle method to restore its inherent strong trade-off between preserving unrelated knowledge and removing target knowledge, and update the corresponding section on Needle's performance. In future work, we plan to design more refined steps to further enhance the Needle method and improve its performance.

The only concern is that the model could be generating the answer with either rewording or paraphrasing… I would urge the authors to consider the MCQ experiment post rebuttal, if time permits.

Thanks again for your suggestion about the Multi-choise Questions experiments.

We believe your concerns regarding the rephrasing or rewording of questions can be addressed and explained to some extent by the experiments on our low-resource language jailbreak attacks presented in Sections 5.1 and 5.2. In these experiments, we used another language (German) to query the LLM about the same knowledge, focusing on the same concept vectors. We observed that these concept vectors exhibited high activation even when questions were posed in German (Section 5.1). Moreover, ablating these concept vectors resulted in effective unlearning for the German queries as well (Section 5.2).

Therefore, we can reasonably infer that the unlearning process is effective regardless of how a given question is rephrased, or even when posed in a different language, as long as it pertains to the same underlying knowledge. Ablating the corresponding knowledge parameters appears to consistently impact the associated knowledge.

That said, we appreciate your suggestions and will include the MCQ experiment in the revised version of paper to make our work more persuasive.

We thank the reviewer for their feedback and are encouraged by their appreciation of concept vectors being useful for auditing unlearning.

Regarding the mentioned weaknesses:

W1: "The score description should be more precise...A single token can be relevant to multiple topics."

We assume the reviewer refers to the token scores – this score is essentially the logit corresponding to a certain token after the projection. Namely, given an value vector v of dimension d, after projecting it to the vocabulary space (E * v) we get a vector r of dimension |V|, where V is the vocabulary. The vector r has an entry for each token with its corresponding logit. The logits are the scores – we sort them by value and consider the top-scoring tokens which often show a clear pattern that described a concrete concept. This observation has been reported in previous works [1,2]. Please note that the topics are inferred from the whole set of top-scoring tokens, rather than single tokens. While a single token may be associated with multiple topics, we can identify the most relevant concept by analyzing the entire set of projection tokens, and validate it using causal experiments.

W2: "It’s not clear how the vectors are selected...Does the layer depend on the concept?"

Thanks for these questions. When selecting the concept vectors candidates, we do not focus on which layer they are locate in. Instead, we are primarily concerned with whether a subset of their projection tokens is highly relevant to a specific theme. Subsequently, we validate their causal relationship and specificity regarding the corresponding knowledge output through causal validation experiments in behaviors. Therefore, the layer in which a concept vector is located not a major factor in its selection. That said, in practice, we have observed that vectors for specific concepts with a substantial causal effect are more commonly found in the middle-upper layers of the model. The selection process is described in Section 3, and we are happy to provide more details if needed.

W3: "The Needle unlearning baseline is similar to RepNoise and RMU...They seem to be robust to jailbreak attacks too."

We thank the reviewer for this valuable suggestion. We have been working on these additional experiments following this suggestion, and aim to share the results soon. In this context, please see our response in Q4 to Reviewr 9s3u where we further discuss how our work related to representation engineering-based methods.

W4: "It’d be good to include an evaluation of suffix based adversarial attacks, like GCG"

Thanks for this comment! We have followed the reviewer’s suggestion and provide the effects of two current representative Jailbreak methods – GCG [4] and AutoDAN [5] – on the activation enhancement of concept vectors, and compare them with the effects without jailbreak and with the first crafted prompt in the paper. We observe similar trends, where jailbreak enhances the activation of model knowledge parameters. We will add these results to the paper.

W5: "It’d be good to find concept vectors for a bigger set of models"

Please note that the purpose of the benchmark is to evaluate the performance of different unlearning methods, rather than to evaluate model performance. Thus, we focus on covering a diverse set of unlearning methods rather than models, including 9 different baselines from 4 different categories. In addition, the overall approach of using concept vectors has been proven to be generally applicable. Their existence has been shown in multiple transformer-based language models, including GPT2 [1,2,3] and GPT-J [3], and in the appendix (see Table 7) in our paper, where we have validated that concept vectors can be located in ChatGLM3-6B, Mistral-7B, Qwen1.5-7B and even Qwen1.5-72B. Therefore, it should be easy for future work to extend this benchmark in case they are interested in unlearning for specific models.

Additionally, there are other methods based on representation engineering [1, 2, 3] and activation modification [4] that aim to unlearn or refuse generating harmful knowledge by adjusting the internal representations of the model. Notably, the primary goal of these methods is to perturb the model's activations on the target data, making it more difficult for the model to process and recall the knowledge, rather than erasing it from the model's parameters. By weakening the model's ability to represent this knowledge within its internal hidden representations, these methods achieve a defensive effect. In contrast, our work focuses on evaluating whether the knowledge stored in the model's parameters has been effectively impacted by existing unlearning methods.

Thanks Reviewer 1G7R for the timely reply and follow up questions, which we happy to clarify.

Concept vectors’ scores (W1+W2):

“however the score's formula ∑i=1V(ei⋅vjℓ)/V seems to be computed at each layer ℓ. If I understand correctly, the formula seems to compute the token score as if the model was “chopped” at the ℓ-th layer. That's why it looks to me that the score depends on the layer.”

Our selection process strives to identify the vectors that exhibit the most prominent and specific concepts across the network, which we approximate based on the logits in the vocabulary projection. However, the logits of vectors in different layers are not comparable to each other, because they reside in different representation spaces. This is why we first take the most “promising” vectors per layer and then from the remaining vectors across the whole network we select those where we see the clearest concepts based on GPT, while disregarding the layer information.

RMU evaluations (W3+W4):

We evaluated RMU on the ConceptVectors benchmark according to the reviewer’s suggestion. The results on the LLaMA2-7B-chat portion are provided in the table below.

There are two points to note:

• The original RMU method will edit the same fixed layers for all the samples (the default hyperparameters are the 5-th, 6-th, 7-th layers in a 32 layers model). So we first report its performance using these original settings.
• To achieve better unlearning effects on the corresponding concept vectors, we also modified the default hyperparameters of RMU, where for each concept we edit the layer containing its concept vector, as well as the two preceding layers. We named this alternative RMU as RMU_enhanced in the following table.

In the table, we further include two representative methods described in our paper, DPO and MEMIT+Empty, for better comparison. From the results we make the following observations:

1. Overall, in terms of behavioural evaluation, we observe that MEMIT continues to deliver the best performance, achieving the most effective unlearning (lowest BLEU and ROUGE scores) while also preserving unrelated knowledge well in the model (highest unrelated BLEU and ROUGE scores).

2. In RMU’s original setting, the fixed layers prevent unlearning from effectively influencing concept vectors in other layers. As a result, it performs poorly in terms of our intrinsic evaluation (Jaccard Similarity: 0.999). In terms of behavioural evaluation, its performance is comparable to DPO but outperformed by MEMIT and RMU_enhanced.

3. In RMU_enhanced, we dynamically select the layers for editing for each concept, therefore it has a better intrinsic performance (lower Jaccard Similarity), indicating a more effective influence on the concept vectors. On the behavioral evaluation, the enhanced RMU achieves improved unlearning performance (lower BLEU and ROUGE scores) and better preservation of unrelated knowledge (higher BLEU and ROUGE scores) compared to the original RMU configuration. These results demonstrate that targeting the specific parameters where knowledge is actually stored in the model allows for superior unlearning effects while minimizing the impact on unrelated knowledge.

Additionally, it is worth noting that while RMU_enhanced and MEMIT+Empty show similar performance on the Jaccard Similarity metric, suggesting a comparable degree of parameter modification, MEMIT+Empty achieves better unlearning results on behavior metrics. This can be attributed to MEMIT’s potentially more effective loss function, which introduces more optimal directional modifications and edits to the concept vectors.

We will incorporate these additional results in a revised version of the paper.

We thank the reviewer for their thorough review and constructive comments. We wish to address the reviewer’s points and hope to engage in discussion.

W1: “given the fact that Needle (...) itself doesn't induce such behavior change… points the concept vectors don't work as hypothesized/defined and are thus not suitable for evaluating/benchmarking unlearning”:

We interpret these results differently, consider the effect that Needle introduces substantial, and strongly believe that concept vectors are valuable for evaluation. Table 3 shows that Needle obtains mean target bleu scores of 0.46 and 0.31 for the two models. These are indeed higher (worse) than the other unlearning methods (which get to 0.05-0.24). However, there are several important things to keep in mind:

• Most importantly, as illustrated in Figure 3, the concept vectors we identify suggest the presence of residual knowledge traces, which are eventually behaviorally relevant: they make the model more vulnerable to jailbreaking attacks. While standard behavioural evaluations do not reveal the importance of these concept vectors, a deeper robustness analysis through jailbreaking reveals their complementary role to standard behavioural evaluation.

• We do not expect that the concept vector is the only parameters in the network that capture knowledge about the concept. Previous works show that knowledge is spread across multiple places in the network [1,2], so overall it is expected that ablating a single vector would not erase the knowledge completely. Still, even though the concept vector is not necessarily the only place where knowledge is stored, it is still useful for evaluating unlearning – given its causal effect, as described above.

• Needle only ablates a single vector, which constitutes <0.001% of the network parameters. We consider the effect it introduces dramatic, as ablating a random vector in MLP layers will keep a BLEU score of 1.0. A decrease of BLEU from 1 to <0.5 means that this vector plays a causal (and specific) role in the model’s generation of knowledge about the concept. Other methods obtain better scores but they are either not restricted and modify all the parameters or modify the MLP layers where the concept vectors are in. For example, MEMIT modifies a total of 44K MLP vectors (44K times more vectors than Needle), including the concept vector itself.

W2: Regarding the manual selection of concept vectors:

The data selection process is mostly automated and systematic (which was appreciated by Reviewer 9s3u). For creating a high-quality benchmark, there is indeed a manual selection phase, which finds the vectors that correspond to the most coherent concepts based on human judgment, followed by a causal validation step.

Please note that we have included another more automated version of our pipeline which is easier to scale in Section A.6 in the appendix, and can be used to find concept vectors in any transformers-based models. The whole process of adding one new concept’s data takes no more than 90 seconds.

W3: Regarding the GCG adversarial attack:

Thanks for this note. Following the reviewer’s comment, we additionally provide the effects of two current representative Jailbreak methods – GCG [3] and AutoDAN [4] – on the activation enhancement of concept vectors, and compare them with the effects without jailbreak and with the first crafted prompt in the paper. We observe similar trends, where jailbreak enhances the activation of model knowledge parameters. We will add these results to the paper.

W4: Regarding the unrelated abilities evaluation:

Thanks for your suggestions. Notably, while evaluating the effect of unlearning on general capabilities of the model is valuable, it is tangential to the main goal of the benchmark – evaluating unlearning of knowledge, which we address from multiple angles. Specifically, the evaluation on unrelated questions does provide a signal regarding the general generation abilities of the model beyond the unlearned concept. Nonetheless, we see the value in the reviewer’s suggestion and are now working on adding the harness evaluation (given that our submission received 7 reviews, we may not finish these experiments on time for the rebuttal).

Thanks, Reviewer 6fAd, for your valuable suggestions and questions regarding our paper.

I'm still dissatisfied that Concept Vectors cannot be used to benchmark unlearning methods since behavioral change is not observed (Table 3). Labeling this causal is problematic and doesn't agree with the presented results.

Please refer to the updates we have made in Table 3 in the updated PDF, as well as the new discussion provided in Lines 356 - 370. (For a summary, it can also be seen in the General Response message we just posted.)

Specifically, we updated the Needle method’s performance in Table 3 by removing its gradient ascent fine-tuning step. We found that this step caused further degradation of the model's unrelated knowledge without significantly improving unlearning of the target knowledge.

After removing this step:

• The updated Needle achieves the best trade-off between unlearning target knowledge and preserving unrelated knowledge, compared to all other unlearning baselines. It shows the largest difference in behavior metrics between Target QA scores and Unrelated QA scores: on LLaMA BLEU and ROUGE-L: 0.415 | 0.301, and on OLMo BLEU and ROUGE-L: 0.334 | 0.455.
• Needle can introduce a clear causal effect of an average decrease of >0.5 BLEU points in the behavioral tests. This effect is by definition causal as Needle intervenes in the computation and consequently modifies the ability of the model to answer questions about the target concept.
• Additionally, across both models, Needle maintains the highest Unrelated scores among all baselines, indicating that it has minimal impact on the model's unrelated knowledge.

These suggest that the concept vectors we identified are strongly causally related to the target concept, and we respectfully do not consider this causal relationship to be weak.

For a detailed view for the specific distributions of the strength of the behavioral causal relationship between each vector and the target concept, please refer to Figure 4.

In the jailbreak experiments (Section 5), we also confirm that both manual jailbreak prompts and automated jailbreak methods (GCG, AutoDAN) significantly increase the activation of these identified concept vectors, while the activation of unrelated vectors remains very low ([-0.002, 0.003]). This is a clear and observable mechanism (Table 4, Figure 7 and Figure 8).

Moreover, the table in the W3 response doesn't contain Needle method.

Since Needle only modifies the specific concept vector for each edit and does not update the other parameters, it will not change the activations on the certain concept vectors. Therefore, the activations of the target concept vectors will remain identical to those observed in the vanilla model.

Methods like DPO and gradient difference perform full parameter fine-tuning, so they will affect the activations on the concept vectors.

Please note that the "activations" referred to here are defined in Lines 421 - 434, which means the coefficient scores assigned to each dimension of the concept vectors (See the formula in Lines 110 - 114).

In my judgment, it is uncertain to which degree concepts are embedded/concentrated into concept vectors lack proper understanding around them.

Please refer to Figure 4 in the Appendix A.4 if you would like to have a more intuitive understanding of the degree to which each concept vector is behaviorally linked to its corresponding target concept.

While in this paper we did not separately analyze the depth of knowledge encoded in each concept vector, but rather treated them as a whole by averaging, based on the ConceptVectors benchmark we have developed and the experiments we have conducted, our work have been able to provide several authentic and important findings and contributions, including but not limited to the following:

• Finetuning-based unlearning has very limited effect on the parameters storing the target knowledge, and it does not truly erase the model's target knowledge (Table 3).
• Finetuning-based unlearning will inevitably impacts the other unrelated knowledge behavivors and is not suitable for true unlearning in LLMs. (Table 3, Figure 3, Lines 465 - 469)
• Finetuning-based unlearning will include suppressing the activations on the model's target knowledge parameters to help it achieve the unlearning effect on behaviors metrics, even though it do not truly erase the knowledge. (Table 4, Lines 448 - 449)
• Model editing-based approaches, including MEMIT and Needle, are capable of making more targeted parameter modifications to the target concept vectors without affecting unrelated knowledge. As a result, they achieve stronger unlearning effectiveness (Lower Target Knowledge Scores in Table 3), Specifictiy (Higher Unrelated Knowledge Scores in Table 3), and Robustness to Jailbreak attacks (Figure 3, Figure 5, Figure 9).
• Jailbreak significantly increases the activations of the target concept-related knowledge parameters (Table 4, Figure 7, and Figure 8) without affecting the activations of unrelated parameters.

We believe that these contributions we have made are crucial and important for the future development of the Knowledge Unlearning field in transformers-based LLMs.

And we would greatly appreciate it if this could help address the reviewer's concerns or doubts.

We thank the reviewer hnu6 for putting an effort into reviewing our work. The reviewer has raised several important points and questions, which we clarify below.

W1 (Previous intrinsic attempts):

Thanks for referring us to previous relevant work. We believe there is a fundamental difference between predicting knowledge based on internal representations [1, 2]---a predictive task---and between identifying specific parameters (in the form of “concept vectors” that constitute a small fraction of the model’s parameters) whose ablation behaviorally changes the predictions of the model.

W2 (Vectors in the standard basis):

We acknowledge the fact that concepts, and knowledge, are not necessarily encoded in the standard basis (axes, or neurons). Following previous work on parametric encoding of knowledge, we search for axis-aligned concept vectors just because they are easier to find, as the search problem becomes more tractable. We agree that there may exist additional, non axis-line concept vectors. Yet, the vectors we do find are a lower bound on the amount of intrinsic knowledge still encoded in the model after unlearning: we show that these axis-aligned vectors exist, and are behaviorally relevant. Therefore, we posit that removal of the concept knowledge encoded in these vectors is a necessary condition for unlearning. Our results show that current fine-tuning-based unlearning methods largely fail to introduce meaningful changes to these parameters.

W3 (Manual work in Dataset generation):

In (a), a “clear pattern” refers to cases where at least 40% of the top-k tokens in the projection are keywords related to the target concept. A “concrete and specific concept” refers to a concept that is not a broad category that potentially covers many entities, such as “U.S. Presidents,” but rather a specific entity, such as “Barack Obama” or “Harry Potter.” In (b), we ensure that every related question is a single, straightforward query without complex contexts. Additionally, these questions are designed to be easily answerable by humans with knowledge of the target concept.

W4 (Identifying concept vectors with specific tokens):

Similar to the point above (W2), we note that this search method allows us to at least identify this set of behaviorally relevant concept vectors. We agree that there may exist more.

W5: "It is unclear what quality-control measures... is a place where things could go wrong."

We are not fully understand the reviewer’s concern, and would like to ask for a clarification. We try to answer based on our understanding: When using the tokens obtained from the concept vector projection to search for and match the corresponding concepts, as described in Section 3.1, we first use GPT-4 to generate potential themes or concepts. These suggestions are then manually verified and filtered. After this, we conduct causal validation experiments to verify the matching between the selected vector and the target concept. This step is crucial to ensure that the concept vector aligns with our defined concept, as well as with the corresponding QA pairs and relevant text segments from Wikipedia used in the behavior tests. In this process, we manually verify that each concept’s QA pairs and corresponding Wikipedia text segments are both appropriately matched and reasonable. More details about the quality verification of generated behaviors tests can be found in Section 3.3.

W6: “taking concepts directly…subtly different from a concept that a human would request to be unlearned”:

The goal of this work is to pose an evaluation test for unlearning. While the concepts selected were derived from the model, they are grounded in concrete concepts in the world, with corresponding Wikipedia pages describing knowledge about them. Moreover, similar concepts (from Wikipedia) are commonly studied in the context of knowledge removal and editing. Further, our experiments validate that this vector is causally related to the human-defined concept in terms of model behavior and also has specificity (Section 3 and Figure 4). We have also demonstrated that the corresponding concept vector is highly activated and invoked during the model's usage of knowledge associated with the human-defined concept (Section 5.1 and Table 4), further confirming that this concept vector contains at least a portion of the key target knowledge related to the human-defined concept. Taken together, we do not see why the fact that the concepts were derived from the model itself, given the goal of the paper, the experimental results, and relation to prior works, is a weakness.

W7&W8 (Needle vs. other methods):

We believe the reviewer may not understand the interpretation of these results and would like to elaborate. First, note that Needle is not an unlearning method advocated in this paper. Instead, it serves as an oracle baseline, which has access to some of the parameters encoding the concept knowledge. Regarding its performance, we consider the effect that Needle introduces substantial for several reasons:

• Needle only ablates a single vector, which constitutes <0.001% of the network parameters. It is much more “surgical” and fine-grained than other examined methods: MEMIT modifies a total of 44K MLP vectors (44K times more vectors than Needle), including the concept vector itself, thus its better performance is quite expected. Therefore, we consider the unlearning effect that Needle introduces to the model’s ability of generating concept knowledge is already dramatic, showing that it does capture essential knowledge about the concept. As a reference, ablating a random vector in MLP layers keeps a BLEU score of 1.0.

• The distribution of knowledge within the model is broad and dispersed. In Needle, we only ablate one of the most prominent vectors related to the target concept. Furthermore, in this baseline, we applied simple noise perturbation and fine-tuning adjustments, without introducing any additional operations.

Regarding the reviewers' questions:

Q1: Please see our response in W1 and the response to the 9s3u’s Q4. We believe that our work is advanced and pioneering, as we approach the study of unlearning effects from the perspective of knowledge in model parameters, investigating whether it is influenced by existing unlearning methods. Previous works have primarily focused on the representation space, including disrupting activations to influence the expression of target knowledge in internal representations [1]. However, they overlooked the consideration from the model parameter perspective and the method proposed has not genuinely impacted the knowledge stored in the parameters.

Q2: Please see our response in W2. Through the vocabulary projections and causal validation experiments in behaviors testing, we can ensure that the concept vector we identified is causally related to the model’s generation behavior about the target concept. Additionally, it is proved to be highly activated when the model processes relevant knowledge to answer related questions (Table 4 in Section 5.1).

Q3: Please see our response in W4 and W5.

Q4: (1) Compared to LLaMA2-7B-chat, OLMo-7B is a more fragile model, making it more prone to breakdown during finetuning-based unlearning. As a result, under the same conditions (e.g., epochs, learning rate, etc.) of finetuning unlearning, OLMo's general capabilities deteriorate more quickly. (2) Based on our previous experiments, we found that knowledge in OLMo is distributed more dispersed than in LLaMA. This is one of the key reasons why ablating a single concept vector (Needle) in OLMo is less effective than in LLaMA. This is an interesting issue that warrants further exploration in future studies.

Q5: Please see our response in W7&W8.

---

References:

[1] The wmdp benchmark: Measuring and reducing malicious use with unlearning. ICML 2024

[2] Estimating Knowledge in Large Language Models Without Generating a Single Token. EMNLP 2024

Thanks, Reviewer hnu6, for providing valuable feedback and questions.

however I think the existence of that benchmark limits the claim that this is “the first benchmark for internal evaluation of unlearning methods.”

Thanks for your suggestion. We have already updated the claim in the introduction in Lines 043 - 044 from "This work presents the first benchmark for internal evaluation of unlearning methods." to "This work presents the first benchmark for parameter-based internal evaluation of unlearning methods"

For more additional experiments we have conducted and further revisions we made to the paper, please feel free to refer to our General Response to all the reviewers we just posted.

Will you include the RMU results in the paper?

Yes, we have already included the RMU experiments and results in our submission’s pdf. Please see Table 3 in Section 4, Figure 3 in Section 5, Figure 5 and Figure 6 in the appendix of the updated PDF for the results of RMU. We also pointed out the RMU limitations in Lines 321 - 328 and Lines 374 - 376.

From the experiments, we demonstrate that RMU does not achieve the same level of unlearning effectiveness as Model Editing. This is because its loss function is designed to disrupt the model's activation on the target knowledge parameters, rather than truly removing the target knowledge within them.

and thus including specific details about your process for selecting concepts is important (so that it can be replicated, for example). For example, adding the details you included in your response to W3 would be helpful. For a specific example of why I have some concerns: the ‘McDonalds’ concept in Table 7 for ChatGLM3-6B appears to be polysemantic: both a ‘McDonalds’ concept, but also a ‘names that start with Mc’ concept (McC, McM, McK, McD, McL). Could you maybe address that specific example, and argue why this is an outlier?

During the process of collecting concept vector candidates and performing vocabulary projections, we often find some concept vectors tend to be intertwined with both lexical and semantic associations. Therefore, we believe it is normal to observe that example whose projection has some tokens with prefixes starting with "MC" (e.g., McC, McM, McK, McD) as well as tokens semantically related to "McDonald's" (such as Fast, Burger, or Mac, as in Big Mac).

However, to truly confirm the causal relationship between the parameter vector and the defined concept, we rely on the Causal Validation Experiments described in Step 3 of Section 3. For the examples provided in Table 7, we have experimentally validated the causal connection between these parameters and the target concept knowledge in terms of model behavior.

It would be good to include some discussion of that in the paper.

Thanks for the suggestion. We have mentioned that OLMo-7B is a more fragile model in Lines 1164 -1165 to indicate that it requires a smaller learning rate for fine-tuning compared with LLaMA2-7B-chat.

And we will include more discussions about “why the behavior of OLMO in Needle is so different” in our paper after we conduct further and more precise experiments to verify that knowledge in OLMo is more dispersed than in LLaMA. This will help explain why ablating a single concept vector (Needle) in OLMo is less effective than in LLaMA. However, at this stage, this topic slightly exceeds the scope of the current paper, and the main contributions and findings presented so far are already clear and distinct.

Additionally, as noted in W7, it would be ideal to evaluate another model to understand which model is an outlier.

Due to the limited time during the rebuttal period, it is challenging for us to rerun the complete ConceptVectors experiments on the third model.

However, we believe that the consistent findings from our experiments on two transformer-based models, LLaMA and OLMo, and the concept vectors with strong causal effects that we have identified in other transformer-based models (such as Qwen, Mistral, and Chat-GLM) in Table 7 have provided compelling evidence and greatly support the validity of our experimental findings. We also have provided an automated pipeline for people to reproduce a new ConceptVectors benchmark on any other transformer-based model to reproduce the findings in Section A.7.

We would greatly appreciate it if our revisions and additional experiments conducted in the updated paper could address the reviewer's concerns or doubts.

We thank the reviewer for their thoughtful review, finding our evaluation novel, and appreciating the experiments. We address the points raised:

W1: “a relatively minor weakness is on the presentation of the paper... There might be too much detailed discussion about validations of manual quality checking by humans, it could possibly be moved into the Appendix”

Thank you for the suggestion. We will move unnecessary details to the appendix in the final version of this work.

W2: “many hyperparameters... It would be better to provide a more systematic exploration to study their effects”

The paper has provided the analysis of hyperparameter selection:

• For the noise scale used in both the causal validation experiment and the Needle method, please see Section A.3 in the appendix.
• For the hyperparameters used in running the various unlearning baselines, please see Section E in the appendix.
• For the data splits in our benchmark, please see Section 4.

We are happy to add more analysis, if these do not cover what the reviewer is looking for.

W3: “the results seem to be limited to the two chosen models”

Please note that the purpose of the benchmark is to evaluate the performance of different unlearning methods, rather than to evaluate model performance. Thus, we focus on covering a diverse set of unlearning methods rather than models, including 9 different baselines from 4 different categories. In addition, the overall approach of using concept vectors is generally applicable. Their existence has been shown in multiple transformer-based language models, including GPT2 [1,2,3] and GPT-J [3], and also in the appendix (see Table 7), where we also validated that concept vectors can be located in ChatGLM3 and Qwen1.5. Therefore, it should be easy for future work to extend this benchmark in case they are interested in unlearning for specific models.

Regarding the reviewers’ question:

Q1: Evaluation with ConceptVectors can be done on the two selected widely-used models. However, it can be extended to other models – for models that were derived from the selected models (i.e. fine-tuned versions of LLaMA and OLMo) it merely requires re-validating the concept vectors through the causal validation step in our pipeline, and for other models it requires locating other concept vectors (which we have demonstrated in the appendix – Table 7).

---

References

[1] Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. EMNLP 2022.

[2] Analyzing Transformers in Embedding Space. ACL 2023.

[3] Dissecting Recall of Factual Associations in Auto-Regressive Language Models. EMNLP 2023.