Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs

We appreciate your thoughtful comments and have addressed your concerns as follows.

Regarding Weakness 1.

We've now added our code as part of the supplementary material. Apologies for not including it as part of the initial submission.

Regarding Weakness 2.

Thanks for your comment. We have revised Figure 1, 3 and 8 for better reading. As for the scale of plots in Figure 2 and 5, since the metrics of different subplots are different and incomparable, we believe the current plots are preferable. Using the same scale across them might render certain variations less noticeable.

Regarding Weakness 3.

Thank you for pointing out this. We have revised Table 3 to add Fluency. Besides, "Succ." is same as "Edit Succ.". We have revised the phrasing that could lead to misunderstandings and standardized it as "Edit Succ.".

Regarding Question 1.

We have defined "over-editing rates" and "unchanging rates" in Appendix B (Line 756). In a word, "over-editing rates" calculates the proportion of responses that LLMs still answer the editing target object, indicating excessive editing. And "unchanging rates" represents the proportion of responses that remain consistent with answers prior to editing.

Regarding Question 2.

Yes, for a given piece of knowledge, FiNE may select the same neuron multiple times. However, this does not affect the construction of the neuron set. We retain repeated neurons, as this may indicate that these neurons are of greater importance.

We appreciate your thoughtful comments and have addressed your concerns as follows.

Regarding Weakness 1.

Thank you for your comment. There may be a misunderstanding regarding this aspect of our method. We'd like to clarify that FiNE does not introduce significantly more complexity. In contrast, benefiting from more precise localization and a smaller number of modified parameters, FiNE is highly efficient, enabling significant savings in both time and memory usage. This issue has been discussed in detail in Section 5.4 (Line 434).

Regarding Weakness 2.

We believe the selected models are sufficiently representative. Similar to ROME and its Mamba version (https://arxiv.org/html/2404.03646v2), we consider exploring other architectures feasible but more suitable as separate future work.

Regarding Question 1.

Based on differences in locating step, FiNE exhibits the following distinctions in the editing step compared to other approaches.

1. FiNE does not modify the entire second-layer weight matrix of the FFN; instead, it updates only the vectors corresponding to the top neurons.
2. FiNE introduces a repetition penalty loss to prevent the post-edited model from generating the editing target repeatedly.
3. FiNE utilizes the layer freezing technique for protecting model's linguistic abilities during the editing process.

Regarding Question 2.

Thank you for pointing out this. We have conducted experiments with a smaller (i.e., LLaMA-3.2-1B) and a larger (i.e., LLaMA-2-13B) model on $\text{WikiData}_\text{counterfact}$. Results are listed below. FiNE continues to demonstrate strong performance on both smaller and larger models. We have added Table 15 to show these results in the revision.

LLaMA-3.2-1B:

LLaMA-2-13B:

Regarding Question 3.

We have discussed efficiency in Section 5.4 (Line 434). Benefitting from more precise localization and a smaller number of modified parameters, FiNE exhibits a significant time advantage over other locate-then-edit methods, particularly at Float32 precision, being approximately 4× to 6× faster. We hypothesize that FiNE offers significant benefits when applied to time-sensitive applications.

Regarding Question 4.

We have discussed performance when restricting editing to a single layer in Section 5.3 (Line 404) and we'll supplement efficiency experiments below. We believe that while manually restricting neuron localization to a single layer can be effective based on experience, relying on our algorithm to automatically locate neurons may be a more appropriate option in the absence of prior information. We have added Table 13 to show these results in the revision.

GPT-J:

LLaMA-2:

LLaMA-3:

Regarding Missing Reference.

We have added the missing references to the introduction (Line 34). Thanks for the recommendation.

Regarding Question 1.

Thanks for pointing this out. We have revised the potentially misleading statement to “predominantly subject-driven” (Line 49).

Regarding Question 2.

Thanks for the good question. We also believe that precise localization can avoid modifying unrelated neurons, minimizing changes to the model and thereby preserving its other capabilities. We will explore this with comprehensive experiments in the future work.

Regarding Question 3.

For more complex editing tasks such as multi-hop editing, we may need to make corresponding modifications to FiNE in order to align with the task objectives, which will be investigated in our future work.

We appreciate your thoughtful comments and have addressed your concerns as follows.

Regarding Weakness 1.

Thank you for your valuable suggestions.

(1) We compare the neurons located by FiNE and KN (under the original setup), and compute Intersection over Union (IoU) between these neurons. Experimental results are shown below. We can find that there is almost no overlap neurons between FiNE and KN, which demonstrates FiNE can locate different neurons from KN. These neurons are important but overlooked by KN.

(2) As suggested, we conduct experiments on using our editing method on top of neurons located by KN. Results on GPT-J, LLaMA-2 and LLaMA-3 are listed below. For methods with low Edit Succ., we do not present their Locality (i.e., RSA and FA) results, since the Locality is inherently 100% when no edit is effectively applied. KN + FiNE outperforms KN but still falls significantly short of FiNE (except for the FA metric on GPT-J). The neurons localized by FiNE are more precise than those identified by KN, resulting in a substantial performance improvement. We have revised Table 1 to add these results.

GPT-J:

LLaMA-2:

LLaMA-3:

Regarding Weakness 2.

Thank you again for your suggestion. We ask GPT-4o to rephrase the prompts in $\text{WikiData}_\text{counterfact}$ and calculate Intersection over Union (IoU) between their locating neurons (under default parameter settings). The results are listed below. The IoU results show that our most neurons remain unchanged, which provides partial evidence of robustness of FiNE localization method and significantly outperforms KN. We have added Table 14 to show these results.

Regarding Weakness 3.

Thank you for spotting this error. We have modified it in the revised manuscript, where $W_u \in \mathbb{R}^{v \times d_h}$, $W_{out}^l \in \mathbb{R}^{d_h \times d_m}$ and $W_uW_{out}^l \in \mathbb{R}^{v \times d_m}$.

Regarding Weakness 4.

Thank you for pointing out this. We have added a pseudo code in Section 4.1 (Line 206).

Tanks for your reply! Some of my concern has been addressed. I've raised my score. However, I have a few more questions that I would like the authors to elaborate on: I'm curious about the connections and differences between the neurons you localized and the neurons that KN localized. I hope the authors can analyze this in the final version.

We appreciate your thoughtful comments and have addressed your concerns as follows.

Regarding Weakness 1.

Due to space limitations, we placed the detailed description and derivation of our methodology in Appendix A. We will surely enrich the methodology section in content if more space permitted. The distinction between this paper and the referred previous work on multi-modal LLMs lies in different focus: the previous work focuses on identifying interpretable multi-modal neurons, whereas our work aims to fundamentally improve localization-based knowledge-editing methods. In this way, we do not seek to claim that the neuron localization techniques, which is largely inherited from the previous work, is an novel contribution of this work. Our contributions can be summarized as follows:

1. With empirical analyses, we point out that causal tracing encounters issues during localization by focusing excessively on the subject and neglecting overall knowledge.
2. We propose a fine-grained neuron-level knowledge editing (FiNE) technique for a more precise localization of memories in LLMs.
3. Quantitative and qualitative experiments demonstrate that FiNE significantly outperforms existing locate-then-edit methods based on causal tracing localization.

Regarding Weakness 2.

We primarily discuss FFN for two reasons. First, previous work have demonstrated the importance of FFN, especially in storing factual knowledge [1-5]. Second, as shown in Eqn.11 and Eqn.12, FFN can intuitively reflect the contribution of neurons to each knowledge item. For these two reasons, we focus on deeply exploring the role of FFN in this paper rather than comprehensively investigating the roles of all other modules. We are open to further analyzing other parts in the LLM in future work.

[1] Finding skill neurons in pre-trained transformer-based language models.

[2] Locating and editing factual associations in gpt.

[3] Mass-editing memory in a tranformer.

[4] Multimodal Neurons in Pretrained Text-Only Transformers.

[5] Finding and editing multi-modal neurons in pre-trained tranformers.

Regarding Question 1.

We believe this intuition is correct and it is the basic motivation of this work. The smaller number of modified parameters is attributed to FiNE's ability to achieve more precise knowledge localization, allowing it to edit as few parameters as possible and thereby minimizing the impact on other knowledge in the model.

Regarding Question 2.

Thank you for pointing out this. We have revised Table 9 to correctly show the highest value.