Should We Really Edit Language Models? On the Evaluation of Edited Language Models

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

Q1: Though the experiments are comprehensive and organized, the findings are more about empirical observations rather than systematic analysis. The paper does not distill or form a theory or systematic justification based on these findings. Thus, it is hard to judge academic contributions, especially in terms of the criterion of Neurips.

Reply: Thank you for your valuable questions. I want to address your concerns in two parts:

(1) Purpose and Motivation:

The primary objective of this paper is not to introduce a novel editing method. Instead, we aim to provide a detailed evaluation of existing knowledge editing methods' impact on LLM performance under the sequential editing setting. By investigating the potential factors influencing the performance of edited LLMs, we hope to offer insights and guidance for future research in this area.

(2) A more systematic and general view of our findings:"Elasticity and Plasticity Trade-off"

We have re-organized the findings of our paper and made the following analysis and observation , which we term the Elasticity and Plasticity Trade-off (EPT):

First, we define two crucial concepts in sequential knowledge editing: plasticity, referring to the model's ability to retain newly acquired knowledge through editing, and elasticity, denoting the model's ability to retain its original knowledge after editing.

We observe that during large-scale sequential editing, simultaneously maintaining high plasticity (retention of updated knowledge) and high elasticity (retention of inherent knowledge) is challenging. These two objectives are inherently in conflict. We refer to this phenomenon as the Elasticity and Plasticity Trade-off, drawing parallels to the Stability-Plasticity Dilemma observed in catastrophic forgetting during neural network fine-tuning. Existing work primarily focuses on the detrimental effect of large-scale sequential editing on elasticity, neglecting the interplay between elasticity and plasticity. Our work highlights this crucial trade-off, emphasizing the inherent conflict between these two objectives.

Experimental results with different editing methods on Llama2-7B (see Table 1) demonstrate that when applying editing, maintaining a balance between elasticity and plasticity requires limiting the number of edits within a reasonable range. Exceeding it can lead to model collapse, characterized by the destruction of the model's intrinsic knowledge structure.

Results in Table 1 also show that some methods can maintain the EPT over thousands of edits. However, this capability is not unlimited. With a sufficiently large number of edits, model collapse still occurs. We introduce the concept of editing endurance to quantify the maximum number of edits a model can undergo while preserving the EPT.

Table 1: Edit result for PMET

To provide an intuitive understanding, we examine the weight changes with different edits in edited layer (see Table 2). This analysis aims to demonstrate that maintaining EPT necessitates keeping weight changes within a reasonable range like PMET (neither too big nor too small).

Table 2: L2 Norm of edited layer for different edits on Llama2-7B model.

Please refer to general response part for more detailed explanation or experiments of EPT.

In summary, we believe that the research questions we address are of significant importance to editing. This work will enhance the understanding of knowledge editing for the research community.

Q2: Though the experiments are comprehensive and organized, ... form systematic logic.

Reply: We would like to clarify that our intuition is to (lines 155-159) pose research questions aimed at exploring common factors that might influence the performance of edited models.

To enhance the systematic nature of our work, we unify our findings under the framework of the Elasticity and Plasticity Trade-off (as elaborated in the response to Q1). Consequently, the five questions posed in lines 155-159 can be considered an analysis and understanding of the editing endurance of the model.

Q3: The paper does not propose any new algorithms to improve the sequential model editing.

Reply: We would like to clarify that the primary objective of this paper is to provide a comprehensive evaluation of the impact of existing knowledge editing methods on LLM performance within the context of sequential editing. We aim to investigate potential factors influencing performance, offering valuable insights and guidance for future research in this area. Our focus is not on proposing specific new methods.

Given the primary area of submission for this paper is evaluation, and considering the requirements of this area, we believe that our focus on evaluating and analyzing existing methods, rather than improving specific algorithms, is appropriate. We kindly request that you take this into careful consideration.

Dear Reviewer FFCf:

Thank you very much for your dedicated review. In the rebuttal period, we provide detailed responses to all your comments and questions point-by-point.

A brief summary of our responses includes:

• Q1: A more systematic and general view of our findings: "Elasticity and Plasticity Trade-off".
• Q2: A boarder view of research questions.
• Q3: Why are there no new algorithms in our paper.

If there are any remaining issues or further clarifications needed, please let us know. We are more than happy to provide additional information or explanations to help resolve any concerns.

Thank you for your time and valuable feedback.

Best regards

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

Q1: The outcome of the empirical study is somehow .... would add more value to the results.

Reply: Thanks for your suggestion. we have re-organized the findings of our paper and made the following analysis and observation , which we term the Elasticity and Plasticity Trade-off (EPT):

First, we define two crucial concepts in sequential knowledge editing: plasticity, referring to the model's ability to retain newly acquired knowledge through editing, and elasticity, denoting the model's ability to retain its original knowledge after editing.

We observe that during large-scale sequential editing, simultaneously maintaining high plasticity and high elasticity is challenging. These two objectives are inherently in conflict. We refer to this phenomenon as the Elasticity and Plasticity Trade-off, drawing parallels to the Stability-Plasticity Dilemma observed in catastrophic forgetting during neural network fine-tuning.

When editing, maintaining a balance between elasticity and plasticity requires limiting the number of edits within a reasonable range. Exceeding this limit can lead to model collapse. We introduce the concept of editing endurance to quantify this limit. Please refer to general response part for detailed experiments and analysis.

Q2: The studied problem can be better motivated. If edit needs to be done at scale, e.g., more than 1k edit, is LLM editing still desired or a refreshed fine-tuning could already do the job better? Longer context and retrieval-based generation for time-sensitive knowledge is also usually considered as common solutions, which may weaken the need for sequential editing at scale.

Reply: We would like to clarify that the primary objective of this paper is to provide a comprehensive evaluation of the impact of existing editing methods on LLM performance. We aim to investigate potential factors influencing performance, offering valuable insights and guidance for future research in this area.

Indeed, each approach like FT, and RAG possesses unique advantages, necessitating careful selection based on the specific application scenario. Our work strives to equip researchers and engineers with a thorough understanding of knowledge editing's strengths, limitations, and performance implications, enabling them to choose the most appropriate approach for modifying model behavior.

Here's a brief summary of the pros and cons of the related techniques you mentioned:

Fine-tuning:

• Pros: Suitable for integrating a large amount of new knowledge into the model.
• Cons: Resource-intensive and prone to catastrophic forgetting.

Retrieval Augmented Generation (RAG):

• Pros: Enables rapid knowledge updates, provides domain-specific knowledge.
• Cons: Can lead to hallucinations when retrieved knowledge conflicts with the model's internal knowledge. Retrieved text may not always be relevant to the desired topic.

Knowledge Editing :

• Pros: Offers fine-grained and efficient modification of model knowledge.
• Cons: Existing methods only support a limited number of knowledge edits. Edited knowledge is difficult to utilize for reasoning tasks.

Long-Context :

• Pros: Allows models to retrieve and cross-verify information within a long context, potentially correcting errors or inconsistencies and enabling more complex reasoning.
• Cons: Requires additional training, making it extremely resource-intensive. Can also lead to hallucinations when prompt knowledge conflicts with the model's inherent knowledge.

Q3: For methods such as PMET and MEND that are robust within 10^3 editing, could the authors further extend the scale of # edits to verify if a similar drastic performance drop, as well as the muting effect happens?

Reply: Thank you for your question. To further explore the limitations of the PMET and MEND methods, we extended the number of experiments to 3K on Llama2-7B model. The results are as follows:

Table 1: scaling edits to 3k

As we can see from above table, PMET shows the muting effect. Although MEND still maintains relatively good performance, this does not indicate that it has good performance. Instead, it has high elasticity and low plasticity.

Q4: The term “intrinsic knowledge structure” ... structure.

Reply: We would like to clarify that "Intrinsic knowledge structure" is an intuitive understanding of how knowledge is stored and organized within LLMs.

Extensive existing work [1,2,3,4] suggests that parameter in LLMs implicitly encode knowledge. Manipulating these weights consequently affects the stored knowledge. A line of works[5,6,7] precisely alter the behavior of model by change parameters. Such alteration can cause weight distribution shift.

When weight distribution changes induced by such manipulations remain within a reasonable range, the model retains its original knowledge. However, if these weight changes become too drastic, the model's distribution collapses, making it challenging to extract new knowledge.

---

Reference

[1] Locating and Editing Factual Associations in GPT

[2] Transformer Feed-Forward Layers Are Key-Value Memories

[3] Knowledge Neurons in Pretrained Transformers

[4] Kformer: Knowledge Injection in Transformer Feed-Forward Layers

[5] Mass-Editing Memory in a Transformer

[6] PMET: Precise Model Editing in a Transformer

[7] Editing Common Sense in Transformers

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

Q1: The main weakness ... stratified view is missing.

Reply: Thank you for your valuable feedback. We address your questions and concerns regarding the scope of our evaluation below:

(1) Can findings generalize to other types of language models?

Currently, the vast majority of methods in editing like ROME, MELLO, IKE, focus on decoder-only models (GPT, Llama). The exploration of knowledge editing for other encoder-only (BERT) and encoder-decoder (T5) models remains largely under-explored at present, making it infeasible to evaluate all kinds of model.

**(2) Can findings generalize to specific application scenarios? **

We want to clarify that editing is not tied to specific application scenarios. Similar to pre-training, SFT, and RLHF, it is a stage in the model whole life pipeline. Therefore, we did not evaluate on specific application scenarios. In fact, knowledge editing can be applied to LLMs across various applications, making our evaluation general and broadly applicable.

To justify our conclusion in specific application scenarios, we chose QA task as example , conducting evaluation on TriviaQA dataset with Llama2-7B. The results are shown below, note the un-edited model gets 0.5252 on TriviaQA:

The trends in the table are consistent with the conclusions in our paper.

Q2: The paper focuses more on ... the methodologies are flawed.

Reply: Thanks comment. We answer your question in two folds:

(1) Distinction between editing and FT:

Knowledge editing differs from FT. Most editing methods in this paper are gradient-free, without something like early stop in FT. These methods typically bypass backpropagation and gradient descent, instead directly calculating and substituting specific parameters to achieve the desired behavior change.

(2) Deeper understanding and systematic analysis:

Please refer to Q3 for detailed explanation.

Q3: As an example of Q2 above ....

Reply: Existing work suggests that parameter weights in LLMs represent specific knowledge. Manipulating these weights consequently affects the stored knowledge. Through extensive experimentation, we have made the following analysis and observation regarding editing, which we term the Elasticity and Plasticity Trade-off (EPT):

First, we define two crucial concepts : plasticity, referring to the model's ability to retain newly acquired knowledge through editing, and elasticity, denoting the model's ability to retain its original knowledge after editing.

We observe that during large-scale sequential editing, simultaneously maintaining plasticity and high elasticity is challenging. These two objectives may inherently in conflict. We refer to this as the Elasticity and Plasticity Trade-off, drawing parallels to the Stability-Plasticity Dilemma in catastrophic forgetting during fine-tuning.

When editing, maintaining a balance between elasticity and plasticity requires limiting the number of edits within a reasonable range. Exceeding this limit can lead to model collapse. We introduce item of editing endurance to quantify this limit. Please refer to general response part for more detailed explanation or experiments.

Q4: There are also several different ... as well.

Reply: Thanks for comment. Here are some clarifications:

Part A: Other dimensions for analysis.

(1) Scale of the model: This factor is actually the RQ3 of our paper, with detailed analysis and experiments presented in Section 4.3.

(2) Prompting conditions: We adopted widely used prompt settings and hyperparameters from existing literature and technical reports (e.g., CoT usage, number of few-shot prompting). These details are fully listed in Appendix D.1.

(3) Whether models benefit from RAGs or not: This paper does not involve the use of RAGs.

Part B: Fairness and reliability of our evaluation

To ensure a fair and comprehensive evaluation, we utilize widely adopted commonly used benchmarks like MMLU and BBH. We adopt standard prompt settings and hyperparameters from existing literature and technical reports (e.g., CoT usage, few-shot prompting) and utilize inference acceleration frameworks (vLLM) to align with practical application. These details are thoroughly documented in Appendix D.1, ensuring fairness and reproducibility in evaluation.

Q5: In addition ... just API access.

Reply: We will answer your question from two aspects:

(1) Why performance degradation occurs:

We believe that the performance degradation during sequential editing stems from the disruption of the LLM's "intrinsic knowledge structure." Specifically, the model's parameter distribution undergoes changes. With a limited number of edits, the parameter distribution shift is minimal, allowing for knowledge updates while preserving performance. However, extensive editing leads to significant changes in the parameter distribution, disrupting the knowledge encoded within the parameters and resulting in performance decline.

(2) Detecting performance degradation with different access levels:

API Access: Users with API access cannot directly edit the model's knowledge. However, they can assess potential performance degradation by querying the model and evaluating its responses.

Full Weight Access: Users with access to model weights can perform all operations described in our paper.

Q6: In addition ... nor comprehensive.

Reply: Thank you for your advice. We will add our new finding and analysis in the revision.

Dear Reviewer 2vUP:

Thank you very much for your valuable comments. In the rebuttal period, we have provided detailed responses to all your comments and questions point-by-point.

A brief summary of our responses includes:

• Q1: Can our findings generalize to other language models or other application scenarios.
• Q2: Distinction between knowledge editing and FT
• Q3: Deeper understanding and systematic analysis: Elasticity and Plasticity Trade-off
• Q4: Fairness and reliability of our evaluation & Other dimensions for analysis.
• Q5: Why performance degradation occurs & Detecting performance degradation with different access levels.

Please let us know if any remaining issues or further clarifications are needed. We are more than happy to provide additional information or explanations to help resolve any concerns.

Thank you for your time and valuable feedback.

Best regards