We thank the reviewer for their efforts in reviewing our paper. We address the reviewer‘s concerns below:

W1: BLUE lacks novelty.

We acknowledge that BLUE is a simple strategy since it only reallocates the residual updates from all layers to only the first and last layers. However, BLUE is a highly effective method. We demonstrate its effectiveness through experiments involving three models and two datasets in both sequential and batch editing settings. BLUE boosts existing locate-then-edit methods: it not only improves editing performance but also better preserves the model’s general capabilities and is more computationally efficient. Additionally, through both empirical and theoretical analysis, we are the first to identify a counterintuitive phenomenon in current locate-then-edit model editing: residual propagation can actually introduce weight shift errors. Based on this insight, we propose the BLUE strategy to enhance locate-then-edit methods.

W2 (1): The paper is overly lengthy and difficult to follow.

We acknowledge that the paper's organization could be improved, such as overly lengthy background and proof sections. We will optimize the structure of the paper in the next version.

W2 (2): Section 4 does not demonstrate a strong correlation between the weight shift itself and the factors of editing batches, the number of edits, and residual distribution.

We would like to clarify that our theorem and lemmas characterize the weight shift error—not the weight shift itself—that arises from factors such as editing batch size, number of edits, and residual distribution. Our theoretical results establish an upper bound on the weight shift error and link it to these factors. This connection forms the theoretical foundation for proposing the BLUE method. Instead of directly correlating weight shift with these factors, we use the weight shift variable to bound the weight shift error.

W3: Many of the mathematical derivations and formulas in the paper are unclear.

We regret any confusion caused by insufficient explanations for some of the notations. In Equation (2), $n$ denotes the number of memories to be updated. $K_1$ refers to the keys of the new memory, and $M_1$ denotes the corresponding values. We will carefully review the entire paper and correct similar issues and include a more detailed description in the next version to make it easier to understand.

We sincerely appreciate your valuable feedback and the constructive discussion so far. As the discussion phase is approaching its end (in around 30 hours), we just wanted to kindly check whether our responses have addressed your concerns. If there’s anything unclear or if you have further questions, we’d be more than happy to provide additional clarification.

Thank you again for your time and efforts.

Dear Reviewer,

Thank you again for your time and valuable comments during the review process.

As the discussion phase will end in less than 24 hours (August 8, 11:59pm AoE), we would like to gently follow up to see if you have any further thoughts or feedback. If there are any remaining questions or concerns, we would be happy to clarify them before the deadline.

We sincerely appreciate your contributions and engagement.

Best regards

We thank the reviewer for acknowledging that the BLUE strategy is simple, efficient, and broadly applicable. We address the reviewer‘s concerns below:

W1: The improvement is small in some cases.

We acknowledge that in some cases (e.g., AlphaEdit on GPT-J), the improvements brought by BLUE are relatively small. However, in these cases, the baseline performance is already very high—close to 100—which makes achieving further gains particularly challenging. To gain a deeper understanding of the reasons behind BLUE's failure, we analyzed the AlphaEdit on GPT-J case in detail. Our investigation revealed that BLUE performs less effectively when handling edits involving highly similar concepts, such as the following two cases:

• “What is the birthplace of Bas Verwijlen? Amsterdam → Oss”
• “In which country did Bas Verwijlen live? Switzerland → Netherlands”

In this example, BLUE was only able to successfully edit the second case, while AlphaEdit succeeded in editing both. This may be because BLUE introduces stronger regularization, which restricts the parameter expressiveness and causes the model to forget the previously edited knowledge when editing the latter case.

We were able to fix this failure by using a smaller $\alpha$ value. However, this leads to numerical instability when solving the equation, ultimately degrading overall performance. At present, we use a moderate $\alpha$ to ensure stable performance, and we plan to explore better solutions to address such failures in future work.

Interestingly, we did not observe this issue in LLaMA3, which we attribute to architectural differences between LLaMA3 and GPT-J. We will include this failure case analysis in the final version of the paper.

W2: It would be valuable to explore whether these findings generalize to larger state-of-the-art models.

To address the reviewer’s concerns, we will conduct editing experiments on LLaMA-2-13B, and the results will be presented in the next few days.

W3: It is unclear how well BLUE generalizes to more complex or nonlinear optimization strategies.

We acknowledge that BLUE is an optimization specifically designed for least-squares-based editing methods, and it may not yield similar benefits—or might not be applicable at all—for more complex or nonlinear optimization strategies. However, we would like to emphasize that least-squares-based editing represents a prominent paradigm in model editing, under which many effective methods have been proposed, including the four methods we evaluated in this paper: MEMIT, AlphaEdit, RECT, and PRUNE. Our results show that BLUE consistently improves performance across all of these methods. Moreover, least-squares-based editing has also been extended to more realistic long-text editing scenarios. As shown in Table 4, BLUE enhances performance in those settings as well. Considering the broad adoption of least-squares-based editing methods, the potential application scope of BLUE is likewise extensive.

W4 & Q: The reason why we update two layers instead of one.

From a theoretical perspective, locate-then-edit methods assume that each layer in the model encodes a portion of the complete knowledge and that the capacity of a single layer's weights is limited [1], making single-layer updates insufficient for large-scale knowledge modifications. From an empirical perspective, multi-layer updates are a common practice in locate-then-edit methods. Prior work has demonstrated that multi-layer edits outperform single-layer updates [1,2]. Additionally, we conducted experiments on single-layer updates, and the results are as follows:

CounterFact

zsre

The results above show that single-layer updates perform significantly worse than BLUE’s two-layer update on both datasets, indicating that the two-layer update achieves better editing performance.

W5: This article has a bit of background content and may be simplified a bit.

In the next version, we will simplify the background section of the paper and place more core empirical evidence supporting BLUE’s design in the main text.

[1] Mass-Editing Memory in a Transformer, ICLR 2023

[2] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models, ICLR 2025

We sincerely appreciate your valuable and constructive feedback. As the discussion phase is approaching its end (in around 30 hours), we just wanted to kindly check whether our responses have addressed your concerns. If there’s anything unclear or if you have further questions, we’d be more than happy to provide additional clarification.

Thank you again for your time and efforts.

We thank the reviewer for recognizing that our paper is well-written and easy to follow, that the proposed method is well-motivated, and that the experiment is thorough and convincing. We will address the reviewer’s concerns one by one.

W1 & Q3: Whether the BLUE strategy introduces any additional computational or memory overhead compared to existing locate-then-edit methods.

We appreciate the reviewer’s detailed comments. We would like to discuss the efficiency of BLUE from both time and memory perspectives.

In terms of time efficiency, BLUE improves the efficiency of locate-then-edit methods. Although BLUE requires computing residuals twice, it only updates two key layers of the original locate-then-edit method. This reduces the computational overhead and enhances time efficiency. We provide an analysis of BLUE's time efficiency in Appendix I (Efficiency Analysis). The results show that BLUE indeed improves the time efficiency of locate-then-edit methods.

In terms of memory efficiency, we report the peak memory usage of both BLUE and locate-then-edit in the table below:

The results show that BLUE achieves better memory efficiency compared to the original method. This outcome is intuitive: by discarding residual distribution and instead computing residuals only for the first and last key layers, BLUE reduces the memory usage associated with the residual distribution process.

W2: More detailed analysis and cases in long-form editing scenarios should be included.

We thank the reviewer for the constructive suggestion. We will include case studies to provide a more detailed analysis of BLUE’s benefits in real-world editing tasks. For the reviewer’s convenience, we analyze two cases as follows:

It can be seen that the method enhanced by BLUE generates content that is noticeably closer to the target output compared to the original method.

Q1: The BLUE method updates only the first and last critical layers without residual distribution. How sensitive is the method’s performance to the selection of these critical layers? Would different choices affect editing accuracy or model preservation?

In Appendix H, we conducted an ablation study on the Llama3 model to demonstrate that the selection of key layers indeed affects both editing accuracy and model preservation.

For $AlphaEdit_{BLUE}$ and MEMIT$_{BLUE}$, different layer selections slightly impact editing accuracy, but have a larger effect on generalization and specificity (i.e., model preservation). In particular, choosing layers 7 and 8 leads to the most significant drop in generalization and specificity.

For $RECT_{BLUE}$ and PRUNE$_{BLUE}$, different layer choices significantly affect both editing accuracy and model preservation. For example, choosing layers 6 and 7 causes a sharp decline in RECT’s editing performance, while selecting layers 7 and 8 greatly harms PRUNE’s performance.

Furthermore, the results of the ablation study suggest that choosing the first and last key layers as editing layers, as in BLUE, yields the best editing performance.

Q2: The experiments are conducted on three LLMs and two datasets. Could the authors clarify whether BLUE generalizes well to larger-scale models (e.g., GPT-3 scale) or other architectures beyond those tested?

Due to limited GPU resources, we are currently unable to scale the editing experiments to GPT-3-scale LLMs. To address this concern raised by the reviewer, we will conduct editing experiments on LLaMA-2-13B, and the results will be presented in the next few days.

We sincerely appreciate your valuable and constructive feedback. As the discussion phase is approaching its end (in around 30 hours), we just wanted to kindly check whether our responses have addressed your concerns. If there’s anything unclear or if you have further questions, we’d be more than happy to provide additional clarification.

Thank you again for your time and efforts.

Dear Reviewer,

Thank you again for your time and valuable comments during the review process.

As the discussion phase will end in less than 24 hours (August 8, 11:59pm AoE), we would like to gently follow up to see if you have any further thoughts or feedback. If there are any remaining questions or concerns, we would be happy to clarify them before the deadline.

We sincerely appreciate your contributions and engagement.

Best regards

We thank the reviewer for recognizing that BLUE is a simple yet effective strategy. We address the reviewer's concerns below:

W1: The main text needs to elaborate more on why the second layer is chosen as the last key layer rather than the second layer.

We appreciate the reviewer’s detailed comment. On the one hand, BLUE is an optimization strategy, and we aim to preserve the mechanism used in existing locate-then-edit model editing methods, which compute residuals at the last key layer, to ensure broader applicability. On the other hand, the choice of the first key layer is based on this mechanism: when residuals are computed at the last key layer, the layer most affected by residual distribution is the first key layer. Therefore, we choose the last key layer as the second update layer. We will add this explanation to the main text.

W2: The proof section is a bit verbose.

We thank the reviewer for the constructive suggestion. We will condense the proof section (Section 4), move Remark 4.2 and Lemma 4.3 to the Appendix, and integrate the content of Appendix H into the main text.

W3: Contextualizing the limitation about multi-hop reasoning more explicitly in the main text.

We thank the reviewer for the helpful feedback. We will explicitly discuss the method’s limitation on multi-hop reasoning tasks and mention it as a direction for future work in the conclusion of the main text.

We sincerely appreciate your valuable and constructive feedback. As the discussion phase is approaching its end (in around 30 hours), we just wanted to kindly check whether our responses have addressed your concerns. If there’s anything unclear or if you have further questions, we’d be more than happy to provide additional clarification.

Thank you again for your time and efforts.

Dear Reviewer,

Thank you again for your time and for reviewing our paper.

As the discussion phase will end in less than 24 hours (August 8, 11:59pm AoE), we wanted to gently follow up. We noticed that you have not yet responded to our rebuttal, and we would greatly appreciate any feedback you might be able to share before the deadline. If there are any remaining questions or concerns, we’d be happy to clarify them.

We sincerely appreciate your time and engagement.

Best regards