Reply to reviewer QEHi

Thank you, Reviewer QEHi, for your valuable feedback! We appreciate your recognition of our work. This response primarily addresses the questions you raised in your review.

• More baselines (Weakness line 1): Indeed, the baselines you mentioned are worth comparing. We explain the reason for this in our global rebuttal and hope our specification will address your concern.

• Inference computational efficiency (Weakness line 2): In Section 6.3, we mentioned that iReVa's theoretical inference time overhead compared to the base model is 2nd, where n is the number of new knowledge (q & a pairs) and d is the size of the model's embedding size (hidden size). This overhead is independent of the prompt sequence length L and constitutes only a small portion of the entire model's forward propagation. The inference space overhead is 2nd, consistent with the parameter size introduced by iReVa.

• Knowledge storage (Weakness line 3): Indeed, recent works have proposed storing knowledge in the self-attention module, such as PMET [1]. We proposed iReVa to offer an interpretable and manageable model editing approach. However, the self-attention module is inherently complex, and deconstructing it into an interpretable forward propagation method requires more effort. In our future work, we look forward to proposing interpretable forward propagation in self attention modules.

• Other tasks (Weakness line 4): Regarding the hallucination task, we conducted the experiment following GRACE [2] and Melo [3]. The experimental results are presented in the table below, with the baselines' results selected from the Melo [3]. The results show that iReVa does not perform well on the hallucination task because the target prompts in this dataset are too long. iReVa creates a key-value pair (knowledge neuron) for each token in the target prompt and its prefix. Due to the excessive number of introduced neurons and the high similarity of the prefixes, the probability of errors significantly increases. Additionally, due to the mechanism of equation 7 (Section 4.2), iReVa will only affect the last token of the prompt which is used for next token prediction. However, the hallucination task requires calculating the perplexity (PPL) of the entire sentence, and iReVa only modifies the hidden states of the token at the last position. Since GPT series models are unidirectional autoregressive models, the tokens before this position cannot access this additional information introduced by iReVa, resulting in poor performance of this method on the hallucination task. An intuitive solution is to apply this mechanism to every token in the sentence instead of just the last one, which is just the strategy we applied in the supplymentary experiment below, but this would also introduce more noise, leading to suboptimal performance.

• More backbones (Weakness line 5). We additionally conducted the experiments in Section 6.5 (table 4) on gpt-j-6b [4] as a supplement, and the results are shown in the table below. iReVa also outperforms baselines on gpt-j-6b. Furthermore, the hyperparameters of the baselines were kept consistent with their source code. We did not include LLaMA [5] family models due to the following considerations: LLaMA models have a different FFN structure compared to common causal decoder models, containing three linear layers: up_proj, down_proj, and gate_proj, unlike the K, V structure in GPT models, hence some baselines can't be reproduced on LLaMA family models.

• Strategy for selecting the adaptor layer (Weakness line 7): The selection of layer is certainly a direction worth exploring. In our early experiments, we examined the patterns of words output by each layer after providing a prompt to the given model. For the GPT-2 XL model, we found that most prompts resulted in an output of a meaningless word, such as a space or newline, at the penultimate layer. Therefore, we hypothesized that there might be unused knowledge storage capacity in this layer, which led us to select the penultimate layer as the adaptor layer. In current editing works, the editing layer is usually a given hyperparameter, and the optimal value is found by modifying the layer. Thus, we followed these works and conducted layer normalization experiments (Section 6.5). The penultimate layer performed slightly better than other layers, which to some extent supports this hypothesis.

• Problems of Equation 3 (Weakness line 8): In equation 3, i and o are passed into Equation 3 due to the "add and norm" (residual connection) architecture in widely-recognized transformer models.

• Reference:

[1] Transformer-patcher: One mistake worth one neuron. In The Eleventh International Conference on Learning Representations, 2023.

[2] Aging with grace: Lifelong model editing with discrete key-value adaptors. ArXiv, abs/2211.11031, 2022.

[3] Melo: Enhancing model editing with neuron indexed dynamic lora, 2023.

[4] GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, https://github.com/kingoflolz/mesh-transformer-jax, 2021.

[5] LLaMA: open and efficient foundation language models, 2023.

Reply to reviewer T9Qb

Thank you, Reviewer T9Qb, for your valuable feedback! We appreciate your recognition of our work. This response primarily addresses the questions you raised in your review.

• Treat T-Patcher [1] as baseline (Weakness 2). The reason for missing of T-Patcher as baseline is specified in our global rebuttal, we hope our explain will address your concerns.

• Stronger capacity for carrying traceable edits (Weakness 3). Regarding traceable edits, similar to T-Patcher, we construct an independent neuron for each piece of knowledge. These neurons explicitly map the question to the answer in the new knowledge. Additionally, these neurons are independent of each other (with batch_size=1 during training), as you noted in the Strength section. Concerning capacity, if you meant "volume", iReVa's space overhead for new knowledge insertion is 2nd, where n is the number of new knowledge (q & a pairs) and d is the size of the model's embedding size (hidden size). As the value of n increases, this overhead is slightly less than 2nd because some knowledge can be added through updates rather than inserts. Section 6.3 mentions that for 10K edits, the additional parameter size is 0.08B (1.5B for the base model), increasing the parameters by only 5% for 10K edits. The comparison with T-Patcher in global rebuttal may be relevant to this weakness, you can access that part for detail.

• Withdrawal analysis (Weakness 4): We apologize for the unclear description of the withdrawal analysis. In iReVa, one piece of knowledge corresponds to one neuron. Therefore, in withdrawal analysis, we withdraw one neuron at a time (setting its corresponding Vi to a zero vector) and test whether the model's response to the knowledge associated with this neuron changes, potentially reverting to the unedited state. Batch rollback of knowledge neurons is also feasible. Due to the independence of the neurons, the effect of batch rollback is consistent with single rollback.

• Code book-based method's withdrawal analysis (Weakness 5): GRACE [2] and MELO [3] use code books to manage batches of knowledge. However, in GRACE and MELO, each cluster center corresponds to a batch of knowledge. Therefore, to withdraw a single piece of inserted knowledge, one must withdraw all knowledge inserted in the same batch, affecting other knowledge that does not need to be withdrawn. If we reduce the batch size, the number of clusters will significantly increase, which will markedly affect the performance of these methods. In our revised version, we will also adjust the description of this part.

• Reference:

[1] Transformer-patcher: One mistake worth one neuron. In The Eleventh International Conference on Learning Representations, 2023.

[2] Aging with grace: Lifelong model editing with discrete key-value adaptors. ArXiv, abs/2211.11031, 2022.

[3] Melo: Enhancing model editing with neuron indexed dynamic lora, 2023.

Reply to reviewer QtC8

Thank you, Reviewer QtC8, for your valuable feedback! We appreciate your recognition of our work. This response primarily addresses the questions you raised in your review.

• Increasing the number of parameters (Question 1): In Section 6.3 of our paper, we analyzed the space overhead of iReVa, which is 2nd, where n is the number of new knowledge (q & a pairs) and d is the size of the model's embedding size (hidden size). This means that iReVa's space consumption grows linearly with the amount of new knowledge. Theoretically, iReVa’s space overhead could be smaller, but this would involve following techniques. In practice, a significant portion of the new knowledge added to the model are frequently-updated, meaning the answer changes while the question remains the same. For such knowledge, we only need to locate the corresponding neurons Ki, Vi. For example, when inserting new knowledge, we can directly feed the new question and supplymentary similar questions into the model with iReVa to see if any neurons are simultaneously activated by all questions (essentially a threshold-based similarity match). By modifying Vi to adapt to the new answer, no additional space is needed. For new facts, where the question has not been recorded by the model's memory via iReVa, additional space is required.

• Reasoning for new knowledge (Question 2): Reasoning has always been a challenging issue in the field of model editing. In the background of large language models, reasoning ability lacks interpretability, making it difficult for most model editing methods, including iReVa, to apply newly learned knowledge in reasoning tasks. One existing attempt is IKE [1], which inspired from in-context learning. However, this approach affects interpretability and locality metrics. Overall, the reasoning ability of current model editing methods often trades off with interpretability, which many researchers are striving to solve.

• Limitations: Thanks for raising this issue. The limitation of iReVa can be summarized as follows: 1. iReVa performs poorly when the target prompt is a long sentence because it constructs a knowledge neuron for each token in the target prompt, thereby increasing the training time cost. Additionally, during inference, the high number of neurons increases the probability of errors. 2. To maintain iReVa's interpretability, its application is limited, including that iReVa can be only applied on GPT like models and generation task. 3. The behaviour of iReVa (ES and PS) won't enhance noticeably as the scale of base model grows. These limitations will also be included in our revised version.

• Reference:

[1] Can we edit factual knowledge by in-context learning? Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

[2] Editing factual knowledge in language models, 2021.

[3] Fast model editing at scale, 2022.

Reply to reviewer vMmP

Thank you, Reviewer vMmP, for your valuable feedback! We appreciate your recognition of our work. This response primarily addresses the questions you raised in your review.

• Figure issue (Weakness 1): Thanks for your reminder. We will revise the figure and adjust the image color to make it clearer in the revised version.

• T-Patcher [1] issue (Weakness 2): Indeed, the forward propagation process of iReVa is similar to T-Patcher. The difference between iReVa and T-Patcher is clarified in our global rebuttal, we wish its content will address your issue.

• Line 57-59 (Questions 1): We apologize for the writing error. The sentence in lines 57-58 should be rewritten as: "In contrast, PMET [2], through a cosine similarity analysis on hidden states experiment, posed viewpoints that the self-attention module can extract various types of knowledge."

• Reference:

[1] Transformer-patcher: One mistake worth one neuron. In The Eleventh International Conference on Learning Representations, 2023.

[2] Pmet: Precise model editing in a transformer. In AAAI, 2024.