AnyEdit: Edit Any Knowledge Encoded in Language Models

Dear Reviewer dG4H:

Thank you for your positive feedback and valuable suggestions! We sincerely appreciate the time and effort you have dedicated to reviewing our work. Below, we meticulously provide responses to each of your comments and outline the modifications based on your suggestions.

W1: "It would be helpful if the authors could provide additional analysis on how different overlap settings might affect the method's effectiveness."

Thank you for your valuable suggestion. Following your advice, we have added extra experiments analyzing how different overlap settings affect our method's editing effectiveness. The specific results are summarized below:

As indicated by these results, increasing overlap does not significantly improve editing effectiveness and can slightly decrease performance. Furthermore, increased overlap also raises the number of autoregressive iterations, reducing overall editing efficiency. Thus, in practice, we set the overlap directly to 0 for efficient and effective editing. We hope our additional experiments address your concern!

Q1: "Line 108 references Appendix D, but there is no Appendix D in the paper. The caption for Figure 1 contains incorrect figure numbers."

Thank you for pointing out these oversights! We have made the following corrections in the revised manuscript: 1.Added the missing information from Appendix D and integrated it into Appendix A.2, specifically supplementing the definitions of Efficacy, Generalization, and Specificity from ROME. 2.Updated the caption of Figure 1 to correctly state: "(a) and (d) illustrate the editing processes; (c) and (e) show the editing efficacy as the number of tokens within the to-be-updated knowledge increases."

Hope that these updates could meet your expectations, and we are more than happy to add clarifications to address any additional recommendations and reviews from you.

---

Once again, we deeply appreciate your thoughtful and encouraging feedback. Your suggestions have not only enhanced the current work but have also inspired us to continue exploring research in the area of model editing. We are excited to keep moving forward and contributing to the community!

Best,

Authors of Submission9475

Dear Reviewer gZ7E:

Thank you for your kind words and positive feedback of our novelty, presentation and effectiveness! Your approval is the great encouragement for us and motivates us to continue advancing our work.   Below, we meticulously provide responses to each of your comments and outline the modifications made to the manuscript.

W1 & Suggestion1: "how does AnyEdit theoretically ensure that earlier token edits don’t destabilize the hidden states of subsequent chunks? Expanding the appendix to explicitly detail how parameter updates achieve multi-token synchronization."

Thank you for raising this excellent question. In the implementation of Step 4 (Section 4.2), we explicitly ensure that besides updating the hidden states of target tokens within each chunk, the hidden states of other tokens remain unchanged. This approach precisely addresses your concern regarding earlier token edits destabilizing subsequent chunks. To clarify this further, we have expanded the details in the revised manuscript.

We hope this addresses your concern adequately.

W2: "A direct comparison of semantic and fixed-size chunking‘s impact on editing performance could strengthen the methodology"

Thank you for your insightful suggestion. We indeed attempted semantic chunking based on sentence segmentation but found it limited by the nature of the dataset's knowledge categories. Particularly, with code or mathematical problems, unclear sentence boundaries often resulted in excessively short chunks and increased iteration counts, reducing editing efficiency. Thus, we did not adopt semantic chunking in the main implementation. However, following your suggestion, we provide results comparing semantic chunking and fixed-size chunking as summarized below:

As the table indicates, semantic and fixed-size chunking methods produce similar results, primarily due to the current semantic chunking approach being limited to sentence-level segmentation. We will further explore more effective semantic chunking methods.

We hope this clarifies your concern.

Q1&Q2: "Does AnyEdit's editing time increase with longer text? Could the authors clarify the specific optimizations that mitigate time costs?"

Thank you for highlighting this important issue. We indeed acknowledge the computational intensity challenge, particularly when scaling up sequence lengths, as discussed explicitly in Observation 6 (Section 5.4). To mitigate this issue, we have explored optimization strategies:

1. Early stopping strategy: During gradient descent optimization of Equation (9), we halt training once loss thresholds are met, allowing shorter optimization epochs without sacrificing performance. Due to shorter chunk lengths, thresholds are quickly reached, significantly accelerating the process.
2. Adaptive chunk length selection: Chunks vary in difficulty; simpler chunks can be longer, while challenging chunks can be shorter. Combined with loss thresholds, this significantly reduces the gradient descent epochs. We have conducted preliminary experiments, achieving noticeable speed improvements as summarized in the following table:

As indicated, optimized chunk-by-chunk editing improves speed compared to the original method, though gaps remain compared to single-token editing. Future work will aim at further improvements. We hope this adequately addresses your concern.

Q3:"What are the specific advantages of AnyEdit over AnyEdit that lead to its improved performance?"*

Thank you for this insightful question. AnyEdit builds upon MEMIT, employing auto-regressive editing with closed-form updates in Step 4. In contrast, AnyEdit* extends UnKE, using auto-regressive editing combined with gradient descent optimization to update all parameters in Step 4, resulting in enhanced performance.

We hope this clarifies your query.

---

Once again, we deeply appreciate your thoughtful and encouraging feedback. Your suggestions have not only enhanced the current work but have also inspired us to continue exploring research in the area of model editing.

Best,

Authors of Submission9475

Dear Reviewer H3vg:

Thank you for your kind words and positive feedback of our novelty, presentation and effectiveness! Your approval is the great encouragement for us and motivates us to continue advancing our work.

Below, we meticulously provide responses to each of your comments and outline the modifications made to the manuscript.

Suggestion1: "The Table 1. 'Para' is easy to mistake to the abbreviation of 'Parameters', please change to another name."

Thank you for your suggestion. Based on your comment, we have revised the manuscript as follows: ·In Table 1, we have replaced 'Para.' with 'Paraph.' to more clearly indicate 'Paraphrase' and avoid confusion with the abbreviation for 'Parameters'.

Q1: "Why do most papers perform knowledge editing in a single MLP layer? Wouldn’t applying edits across multiple layers be more effective? Are there specific constraints or challenges preventing this approach?"

This is an excellent question. Indeed, many methods such as ROME and UnKE perform editing on a single MLP layer. In contrast,, recent methods like MEMIT, AlphaEdit, and their derivative approaches edit multiple layers. These studies have empirically demonstrated that multi-layer editing often achieves better results in terms of editing effectiveness and quantity of knowledge compared to single-layer editing.

However, multi-layer editing approaches currently face limitations, especially within the Locate-then-edit paradigm. Specifically, interference between layers occurs, as edits in earlier layers affect the outputs of subsequent layers. Therefore, editing typically proceeds sequentially from shallow to deeper layers. We believe that a key unsolved problem remains: achieving simultaneous multi-layer editing while ensuring edits at each layer effectively increase the output probability for the editing target.

Q2: "Why don’t the authors directly evaluate the affection to LLM's knowledge after performing knowledge editing?"

Thank you for your question. In our experiments, metrics such as Bert Score and Rouge Score evaluate the similarity between the edited outputs and the target knowledge, directly reflecting the effect on the LLM's knowledge. Thus, we infer your question might pertain to evaluating the impact on unrelated or general knowledge. Following your suggestion, we have added additional experimental results comparing AnyEdit and several baselines concerning their impact on unrelated local knowledge and general knowledge, as summarized below:

As shown, AnyEdit minimally impacts unrelated and general knowledge. Specifically, AnyEdit maintains the original knowledge retention capabilities well on both Llama3-8B-Instruct and Qwen2.5-7B-Instruct models, achieving performance comparable to the pre-edited versions. This indicates that AnyEdit does not substantially disrupt knowledge unrelated to the edits.   We hope our additional experiments adequately address your concerns.

---

Once again, we deeply appreciate your thoughtful and encouraging feedback. Your suggestions have not only enhanced the current work but have also inspired us to continue exploring research in the area of model editing. We are excited to keep moving forward and contributing to the community!

Best,
Authors of Submission9475

Dear Reviewer hVmV:

Thank you for your kind words and positive feedback regarding the novelty, presentation, and effectiveness of our work! Your approval is the great encouragement for us and motivates us to continue advancing our research.

Below, we meticulously provide responses to each of your comments and outline the modifications based on your suggestions.

W1: "The proposed chunk by chunk approach is more computational intensive especially when scaling up the sequence length."

Thank you for raising this important concern. We acknowledge that our method indeed faces computational intensity issues, particularly when scaling up sequence length, and we highlighted this explicitly in Section 5.4 (Observation 6) of our paper. We have actively been exploring optimization strategies to address this challenge:

1. Early stopping strategy: During gradient descent optimization of Equation (9), we halt training once loss thresholds are met, allowing shorter optimization epochs without sacrificing performance. Due to shorter chunk lengths, thresholds are quickly reached, significantly accelerating the process.
2. Adaptive chunk length selection: Chunks vary in difficulty; simpler chunks can be longer, while challenging chunks can be shorter. Combined with loss thresholds, this significantly reduces the gradient descent epochs. Additionally, since our initial submission, we have conducted preliminary experiments in the first direction and observed improvements in editing speed. A brief overview of these initial experimental results is provided in the following table:

As shown, our optimization approach significantly improves editing speed over the original chunk-by-chunk method, and now matches the performance of the single-token editing baseline. We will continue working on further improving the chunk-by-chunk editing efficiency.

We hope this response addresses your concerns.

Suggestion1: "I recommend the authors to also include DEM in the experiments for more comprehensive comparison."

Thank you for highlighting this important method! Your insightful suggestion has prompted us to recognize the value of including DEM for a more comprehensive evaluation. In response:

1. We have provided a detailed description of DEM in the Experimental Setup section of the revised manuscript.
2. We conducted additional experiments using DEM and presented the corresponding results and analysis in Section 5.2.

Note: Since DEM's code is currently not publicly available, we implemented the method ourselves according to our interpretation from the original paper. Below is a quick summary of our experimental results and analysis:

As the table demonstrates, our AnyEdit method consistently outperforms DEM. Although DEM dynamically selects layers due to common sense knowledge residing across different layers, it still operates under the limitation of single-token editing and thus struggles with long-form knowledge editing.   We greatly appreciate your valuable suggestions and believe these modifications significantly strengthen our paper.

---

Once again, we deeply appreciate your thoughtful and encouraging feedback. Your suggestions have not only enhanced the current work but have also inspired us to continue exploring research in the area of model editing. We are excited to keep moving forward and contributing to the community!

Best,

Authors of Submission9475