Response to Reviewer fviW

Thanks for your valuable feedback. We appreciate the opportunity to address your concerns.

1. Response to “The activation $\mathbf{a}$ used in routing corresponds to which token in the prompt?”

As shown in Equation 3, the activation used for classification is derived from the entire prompt, meaning $\Delta_{\text{act}}$ represents the average activation across all prompt tokens. This increases the probability of correctly classifying the paraphrase $x_e'$.

2. Response to “Impressive generalization of WISE.”

Thanks for the comment, we suppose there are three key reasons behind WISE's impressive generalization:

• By averaging activations across all prompt tokens, we enhance the model's ability to identify the scope of edits, allowing paraphrase $x_e'$ to pass through side memory while unrelated inputs $x_i$ pass through main memory.
• $\mathbf{W}_{v'}$ is directly modified (Long-term Memory), copying FFN parameters from the original LLM and memorizing new editing samples with a small portion of parameters. During inference, side memory and earlier layers are packaged together, enhancing overall representation capabilities beyond activation-based methods.
• We employ random text augmentation (avoiding leakage of paraphrase inputs). Initially utilized in ROME: For a single $x_e$, it expands into (prefix_i, $x_e$). The prefix is derived from tokens randomly generated by the original LM, serving as an economical data augmentation method.

3. Response to the issue of “parameter overlap”.

• Why $k$-memories’ overlap: Because during model merging, we merge several parameter subspaces into one, not pair-wise merging, we consider the $k$-memories’ overlap.
• Explanation of the “anchors”: For vanilla mode merging, full models are fine-tuned to merge, so the overlap is 100% in the original Ties-Merge. While in our random subspace, only a small proportion ($\rho$) of parameters are updated in a subspace. Therefore, when conducting model merging, it is necessary to have overlapped parameters to rescale and rectify (e.g., trim/select sign in Ties-Merge) different model pieces.
  • We also made an ablation study. In Table 12, "w. KA" (generate random subspaces with replacement) consistently outperforms "w.o. KA" (generate disjoint subspaces by random sampling without replacement), demonstrating that anchors improve performances.
• (Sub)orthogonality of random masks: Since the proportion of subspace overlap is small (e.g., 0.03), most of the parameters are disjoint, which means that the knowledge in different subspaces will cause little conflicts; here, we note this as sub-orthogonality. There is a tradeoff: avoiding knowledge conflicts requires subspaces to have most parameters disjoint, but the parameters cannot be totally disjoint, otherwise the model merging will be poor, so we intuitively observe nuances in the tradeoff where 0.03 of overlap can realize the best result across different $\rho$ and $k$.

4. Response to “Does it use multiple paraphrases of $x_e$ during the training phase?”

Sorry for the confusion caused, we will elaborate on this point as follows.

• As mentioned above, we employ LM-generated token augmentation during the training phase to enable the LM to remember knowledge updates across different contexts. This approach is also employed in ROME, MEMIT, FT, etc., as documented in the supplementary material.
• However, we acknowledge the oversight regarding GRACE, which accepts only bs=1 inputs. In the rebuttal phase, we've completed these experiments, applying the 11 "augmented data" instances generated by LMs to GRACE. When epsilon_init = 500.0, the results of GRACE are shown in Table C of the Rebuttal PDF in the general response.
  • Overall, GRACE demonstrates generalization (e.g., Gen. is 0.28 higher at T = 1 compared to Table 1). However, at T = 1000, due to the inability of the last token (as mentioned in Appendix B.1) to represent semantics, its generalization performance remains inadequate. We will update the main table with GRACE's experimental results.
• We conducted an ablation study on random text augmentation, the final performance is in Figure B of the Rebuttal PDF.
  • We observe that the editing success rate is compromised. We believe that access to the "data generator" can deepen the model's memory of editing samples, and this constant overhead is worthwhile. This finding will also be added in the Appendix.

5. Response to “Perform merging though we don't have access to all the edits beforehand.”

In Figure 5, we discuss knowledge density, finding optimal $\rho^k$ consistently near 0.03, with good performance within the interval $\rho=[0.3, 0.5]$ (exhibiting robustness). This suggests that even without prior knowledge of incoming knowledge count (N), current experiments indicate that 0.2 FFN parameters can accommodate at least 500 edited samples. When "mask memory exhaustion" occurs, we can allocate new mask parameters to store new knowledge. Using retrieve when knowledge isn't full and merging as needed to save memory, achieves true lifelong model editing.

6. Response to “The number of shards k when T increases.”

We do not only test the editing performance for k=2. As shown in Figure 5, k=2 and $\rho$=0.2 exhibit optimal performance at the scale of T=1000, which is why it was reported in the main table. Figure 5 also presents results for k=[2,3,4,5,6], revealing interesting conclusions about knowledge density. Furthermore, in Table 6, we extend the number of editing instances to 3K, resulting in 6 merges (3000 / 500). Please refer to General Response 1, where we discuss that as k increases, single side memory has its limited knowledge capacity. Therefore, combining WISE-Retrieve with timely merges and fine-tuning after accumulating sufficient mistakes can meet the requirements of real applications for deployed online models.

Response to Reviewer pT3T

Thanks for your valuable feedback. We appreciate the opportunity to address your concerns.

1. Response to “many of ablations are missing.”

Thanks for the comment. All components of WISE can be summarized as router, merging strategy, locating side memory, and side memory sizes. Indeed, except for the merging strategy detailed in Appendix B.2, these components are either ablated (if applicable) or analyzed in the main body of the paper. We discuss the independent contributions of these components as follows:

• Router
  • As shown in Figure 3, we present visualizations of WISE router activations. The purple horizontal line in the figure represents the minimum activation threshold $\epsilon$ recorded during the editing phase. Nearly all irrelevant queries (blue) show lower activations, while inputs within the editing domain ($x_e$ or its paraphrased form $x_e'$) exhibit higher activations (red), underscoring the efficacy of the router. Without the router strategy, all inputs either pass solely through the main or side memory. To further validate its effectiveness, we conduct additional ablations with $L_a$. WISE's performance on ZsRE is shown in Table A of the Rebuttal PDF in the general response.
  • Observing the expected decrease in Loc. w.o. $L_a$ , such as dropping from 1.00 to 0.72 at T=1000, reveals the router's effectiveness in identifying editing scopes, minimizing side effects, and retaining a substantial amount of pre-training knowledge.
• Merging strategy
  • We provide detailed ablations in Appendix B.2. Table 10 shows that simple linear/slerp interpolation performs poorly, whereas Ties and Sign excel. It also effectively demonstrates a) the necessity of non-redundancy of masked knowledge shards; b) we must resolve conflicts during merging to ensure maximal retention of all past edited samples.
• Main/side memory sizes
  • The main memory is part of the original LLM, hence its size is not under our control. Side memory replicates a layer of FFN from the LLM, initialized with its weights. Theoretically, it requires 0.64% of the original LLM's parameters (using LLaMA-2-7B as an example).
• Where to introduce this module into an LLM, and how many times
  • Where to introduce: Extensive literature [1,2] suggests LLMs encode high-level semantics in mid-to-late layers and handle more advanced linguistic phenomena. As shown in Figure 4, we select early, intermediate, mid-to-late, and late stages, finding superior editing performance in mid-to-late layers, possibly due to the formation of more complex semantics/knowledge. Overall, we provide guiding recommendations for "Where to introduce." Additionally, we attempt to generalize to the 13B model, as shown in Table B of the Rebuttal PDF, confirming these findings.
  • How many times: In Figure 5, we discuss knowledge density (closely related to "how many times"), finding optimal $\rho^k$ consistently near 0.03, with good performance within the interval $\rho$ =[0.3, 0.5] (exhibiting robustness). This suggests that even without prior knowledge of incoming knowledge count (T), current experiments indicate that 0.2 FFN parameters can accommodate at least 500 edited samples. When "mask memory exhaustion" occurs, we can allocate new mask parameters to store new knowledge. Using retrieve when knowledge isn't full and merging as needed to save memory, achieves true lifelong model editing.

2. Response to “WISE's behavior with larger and higher-capability LMs (e.g. LLaMA-3-70B, Qwen-2-72B)”.

Thanks for your suggestions. In this paper, we explore the latest (at least at the submission stage) and popular LLMs (LLaMA-2-7B, Mistral-7B), observing significant and consistent improvements with WISE across multiple experimental settings. Regarding LLMs of the 70B scale, we lack sufficient resources to conduct such experiments. However, to the best of our abilities, we attempt to observe WISE's performance on LlaMA-2-13B-chat, as shown in Table B of the Rebuttal PDF (choosing the mid-to-late layer: model.layers[34].mlp.down_proj):

Experimental results validate WISE's efficacy on the 13B-chat model, even surpassing editing performance on the 7B model at T=1000. This enhances WISE's scalability across model sizes, and we plan to further refine experiments and attempt editing larger 70B models. Once again, thank you for your suggestions.

3. Response to “Typos, missing citations, minor concerns”

Many thanks for your kind attention and carefulness. We make the following responses.

• Ensuring safe and responsible practices in WISE is of paramount importance. Model editing, which can suppress harmful language generation [3-5], could help address these concerns.
• We have added citations for Sinitsyn et al. (2020) and De Cao et al. (2021) in the updated version of the paper and removed the incorrect term "compute" in L25.
• We have also added a reference to Table 7 when introducing D_edit, which provides specific cases of x_e and y_e. Thank you for pointing out these issues.
• We will remove all \vspace in the CR phase to ensure the paper's formatting is more consistent.

[1] Tianjie Ju, et al. ”How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study.” LREC-COLING 2024

[2] Yung-Sung Chuang, et al. ”DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models.” ICLR 2024

[3] Xinwei Wu, et al. “DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models.” EMNLP 2023

[4] Mor Geva, et al. “Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space.” ACL 2022 [5] Mengru Wang, et al. “Detoxifying Large Language Models via Knowledge Editing." ACL 2024

I apologize for responding late and thank authors for a detailed response. Authors have answered my questions in full and suggested reasonable updates to the final version of the paper. I am keeping my score as is (7), which is to say, I recommend that the paper gets accepted.

Response to Reviewer 5h8i

We thank the reviewer for acknowledging that our paper is well-motivated and the experiments are comprehensive. We kindly address your questions as follows.

1. Response to “Why not store the knowledge in the form of raw text and perform knowledge retrieval (RAG)?”

Thanks for your valuable comments. We kindly refer to the general response for the concern of “ever-expanding memory”. Then, we will elaborate on the following aspects and explain why WISE is irreplaceable compared to raw text retrieval:

1. Lifelong Perspective on Knowledge Update
   • Fact Reasoning/Utilization: Let us consider a simple example: Suppose we aim to edit A: "change Oliver Twist's author to Shakespeare" and then to edit B: "change Shakespeare's nationality to French". If we query "In which country was Oliver Twist written?", a similarity-based retrieval would likely only retrieve A. Using raw text cannot establish generalized knowledge associations both externally and within the model, resulting in weak generalization and poor reasoning abilities. Even recent literature has begun using knowledge graph construction for GraphRAG to mitigate this dilemma, but it is inevitably limited by information extraction technology's inability to establish precise associations.
   • Adaptive Chameleon or Stubborn Sloth [1]: LLMs demonstrate a strong confirmation bias toward parametric memory when faced with both supportive and contradictory evidence to their parametric memory [1]. This means that RAG faces additional challenges in optimizing raw text to be coherent and convincing.
   • Efficiency of Inference: Retrieval itself is a massive engineering system; common embeddings, retrieval, rankers, and synthesis all cause significant delays in online systems.
2. Synergy Between WISE and Raw Text Retrieval
   • WISE and RAG are independent; WISE updates internal model knowledge as parametric knowledge, which can be combined with RAG (non-parametric knowledge) to achieve more robust knowledge updates.

To validate these points, we conduct comparative experiments on the MQuAKE dataset (evaluation on $x_e$'s multi-hop query). For ICE (In-Context Editing, i.e., RAG), we choose all-mpnet-base-v2 and retrieve the Top-1 cosine similarity editing fact to construct the prompt: [Updated Information]: \n {retrieved-editing-fact} \n [Query]: {query}. Results are shown in Figure A of the Rebuttal PDF in general response.

We first observe that WISE + ICE consistently outperforms WISE or ICE alone, proving our point that combining the two achieves better multi-hop reasoning performance. Secondly, as T increases to 1000 and 2000, WISE demonstrates the advantage of utilizing editing facts x, combining and associating knowledge to enhance reasoning.

2. Response to “MEMIT scales the batch editing to 10000 examples.”

• As shown in Appendix A.2, MEMIT-MASS does not belong to sequential edits but rather to parallel/batch edits (reviewer fviW also points out this), losing the capability for on-the-fly repairs. The main table also demonstrates MEMIT's sequential editing performance, akin to ROME, where the model is fully compromised by T=100.
• Lifelong editing was first proposed in GRACE. We follow GRACE's settings and dataset, reporting editing performance at T=1000 in the main table, with analysis extending to T=3000 in Section 3.3. Comprehensive analysis from Table 6 and Figure 11 shows that WISE is capable of scaling up to more edited samples. We recommend using WISE to respond to each mistake online promptly. After accumulating a sufficient number of mistakes, fine-tuning the original model on all accumulated mistakes allows for the removal of side memories. Then, WISE can be reused for on-the-spot repairs. The ability to scale to thousands/ten thousands of edits can meet the requirements of real applications for deployed online models.

3. Response to the issue about “the discrepancy between the current results and the results reported in the MEMIT paper”.

This discrepancy arises from different metrics yielding different results. We follow GRACE's metrics. As shown in Equation 9 of our paper: $\text{Rel.} = \frac{1}{T}\sum\limits_{t=1}^{T} \mathbb{1}(f_{\Theta_{T}}(x_e^t) = y_e^t)$. In MEMIT paper 5.2.2, Efficacy Success (ES) is defined as the probability of the new object $o'$ being greater than the original object $o$: $E[P_{G} [o_i' | p(s_i, r_i)] > P_{G}[o_i | p(s_i, r_i)]]$. We argue that determining editing success based on real output alignment is stricter (thus, smaller value) and more aligned with real-world scenarios.

4. Response to the issue about “Knowledge Conflict in sequential editing”.

In Appendix B.3, we discuss the accuracy of Retrieving Top-1 Activation, demonstrating the efficacy of WISE in identifying the correct side memory. We propose an inspection module as a remedy for WISE:

• First, for a given editing knowledge, use routing activation to check whether it matches past editing sequences and identify the corresponding side memory.
  • If so, cover previous knowledge through re-editing this side memory using current new knowledge in a tiny subspace (e.g., selecting a 0.01 mask parameter), which will rewrite the previous conflicting knowledge while minimally affecting other editing knowledge.
  • Otherwise, conduct vanilla WISE steps, i.e., introduce new parameter subspaces for inserting new knowledge.

5. Reponse to “Editing results of Multi-hop query (MQuAKE dataset).”

We have supplemented these results in Figure A of the Rebuttal PDF in general response. The discussion regarding these results is shown in the second response in your thread. We will further refine this experiment and update the manuscript.

[1] Jian Xie, et al. “Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts.” ICLR 2024

Response to Reviewer KKAN

Thanks for recognizing the value of our work. Your comments are highly aligned with our paper, and we hope the following comments could address your questions:

1. Response to “During the training phase, the model requires excessive memory …”

Thanks for the comment. However, it is notable that the side memory that we copy is quite small, without substantial additional capacity.

• First, the memory that we copy is one linear module in one certain FFN layer, e.g., the model.layers[34].mlp.down_proj in LLaMA-2-13B-chat, so the number of parameters is quite marginal (e.g., 0.64% of the whole model).
• Second, we introduce two variants of WISE: WISE-Merge (with only one side memory) and WISE-Retrieve (with several side memories), and both of them have limited and controllable side memories.
  • For WISE-Merge, it only introduces marginal and constant additional costs (0.64% extra parameters and 4% extra GPU VRAM).
  • While WISE-Retrieve has several side memories, the experimental results show that the additional costs are also limited and controllable. Importantly, Figures 6 and 12 of our submission show that at T=3000, when there are six side memories in WISE-Retrieve, it introduces only approximately 7% inference latency and memory overhead. WISE-Retrieve also benefits from the additional memories and demonstrates higher reliability (Table 6).

2. Response to “It would be great if the model adaptively controls $\rho$ according to the difficulty of incoming knowledge.”

Your suggestion regarding adaptive control of $\rho$ based on the difficulty or information content of incoming knowledge is insightful. We briefly discuss the insights and practicality.

• Recent literature [1] highlights that language models typically store approximately 2 bits of knowledge per parameter. [2] demonstrates that the performance of LoRA [3] and DoRA [4] tends to plateau and decline with an increase in trainable parameters. They underscore the importance of appropriately scaling parameters to fit each editing sample.
• In our experiments, we observed that utilizing a mere 0.1% of side memory parameters (about 0.04M) often resulted in unsuccessful edits per sample. Future work could explore adapting $\rho$ by identifying significant gradient values [5] or estimating the information entropy of incoming knowledge before input, aiming to enhance the robustness of WISE. We are glad to incorporate similar ideas in the future improved version of WISE.

[1] Allen-Zhu, Zeyuan, and Yuanzhi Li. "Physics of language models: Part 3.3, knowledge capacity scaling laws." arXiv preprint arXiv:2404.05405 (2024).

[2] He, Haoze, et al. "Sparse Matrix in Large Language Model Fine-tuning." arXiv preprint arXiv:2405.15525 (2024).

[3] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." ICLR 2022

[4] Liu, Shih-Yang, et al. "Dora: Weight-decomposed low-rank adaptation." ICML 2024

[5] Damai Dai, et al. “Knowledge Neurons in Pretrained Transformers.” ACL 2022