Towards Lifelong Model Editing via Simulating Ideal Editor

We are deeply grateful to Reviewer 1PGC for the careful and insightful comments on our manuscript. Our detailed responses to your questions are outlined below.

Q1: evaluating the method's performance without relying on teacher forcing

Following your suggestion, we conduct additional experiments on the recently proposed QAEdit benchmark [1] (released on February 16, 2025), which adopted the real-world evaluation metrics that do not rely on teacher forcing. Specifically, we take FT, ROME, AlphaEdit$^-$, WISE, and AlphaEdit as baselines and perform $T=1000$ sequential edits on Llama-2. The results are summarized in the table:

We observe that all methods, particularly lifelong editing approaches, suffered significant performance degradation in the more realistic evaluation setting. In contrast, standard algorithms enhanced with SimIE consistently outperformed these specialized lifelong editing methods, demonstrating greater robustness. Although SimIE's performance is still limited by the effectiveness of the fundamental editor, its core strength lies in its generality: SimIE enables lifelong editing to benefit directly from any future improvements in standard editors without the need for specialized redesign.

Q2: if these assumptions are not met

We provide a more in-depth empirical analysis of the two assumptions, measuring their violation in practice. Please refer to Reviewer hGQy's Q2 for the details.

Q3: a deeper discussion and code

We will expand our Related Work and Limitations sections to offer a deeper discussion, highlighting both the advantages of SimIE (e.g., leveraging advances in standard editors without specialized redesign) and its limitations (e.g., may be limited by the fundamental editor’s performance). As for the code, we have provided an anonymous GitHub repository in Section 4, which will be moved at the end of the abstract to ensure better visibility.

Thank you once again for your insightful feedback, which has significantly enhanced and refined our work.

[1] Yang, Wanli, et al. The mirage of model editing: Revisiting evaluation in the wild. arXiv preprint arXiv:2502.11177 (2025).

We sincerely thank Reviewer hGQy for your insightful and constructive comments. We have carefully considered your feedback and responded to each of your points below.

Q1: larger models perform with the proposed SimIE

We conduct additional evaluations on larger LLMs, specifically Llama-3 (8B) and Qwen2.5 (7B). For detailed experimental results, please refer to Reviewer nKJ9's Q1.

Q2: How extensively are the over-parameterization and key-value invariance assumptions violated in your experiments?

We conduct two analyses to assess the extent to which the assumptions are violated in practice. Specifically, we utilize AlphaEdit+SimlE to edit Llama-2 using the ZsRE dataset. At each time step $t$, we track the updated key-value pairs $(k_t, v_t)$, alongside their original counterparts $(k^{\prime}_t, v^{\prime}_t)$ from the initial unedited model.

For the over-parameterization assumption, we quantify how effectively each layer can fit the desired update by computing the optimal residual $R_{\mathrm{min}}$ of the least-squares problem: $\min \frac{1}{T}\sum_{t}^{T}\\|Wk^{\prime}_t-v^{\prime}_t \\|^2$. The results are summarized as follows:

Empirically, we observe that $R_{\mathrm{min}}$ tends to increase in deeper editing layers, indicating a greater deviation from the over-parameterization assumption.

To assess the key-value invariance assumption, we measure the deviations $\\|k_t-k^{\prime}_t\\|$ and $\\|v_t-v^{\prime}_t\\|$, which capture the extent to which edits (both from previous time steps and preceding layers) perturb the original key-value representations. For visualization, we divide the $T=1000$ edits uniformly into $100$ intervals and plot their average values within each. The detailed visualizations are available in the Assumption_analysis.PDF at SimIE (our code link), and a partial summary of key deviations for Llama-2 is provided below:

The findings demonstrate that deviations gradually increase with both editing layer depth and the time step, aligning with your speculation about the perturbations cascade phenomenon.

Overall, both assumptions are indeed violated to some degree in practical scenarios, which emphasizes the trade-off between optimality and sensitivity established by Theorem 4.3. However, the observed violations appear manageable under the current experimental setup, as they do not lead to the empirical failure of SimIE. In scenarios involving longer edit sequences or more editing layers, the violations of assumptions may become severe. These new analyses provide insight into both why SimlE currently succeeds and where it might face challenges going forward, which will be integrated into the manuscript.

Q3: practical necessity of the proposed approach

We would like to clarify that SimIE's primary practical value lies in enabling lifelong editing to benefit from ongoing advances in standard editing research. Indeed, on more challenging datasets and more realistic evaluations recently proposed, specialized lifelong algorithms can exhibit significant performance deterioration, whereas SimlE-enhanced standard methods provide more robust performance (please refer to Reviewer 1PGC's Q1).

Q4: is SimIE's effectiveness inversely related to model size?

Our results generally indicate that the performance of SimIE depends more on the fundamental editor, which may be affected by various aspects like model architecture, dataset, hyperparameters, etc. Although we do observe some performance variations at different scales, there is no definitive evidence of an inverse correlation between model size and SimIE’s effectiveness. For instance, on Llama-3 (8B), ROME+SimIE achieves an average metric of $0.75$, outperforming the $0.63$ average observed on the smaller Llama-2 (7B). Additionally, in our new experiments on the other 7B model, Qwen2.5, SimlE improves performance by an average of $4.8\\%$ over existing SOTA lifelong methods. These results suggest that SimIE may leverage that strength when the underlying editor performs robustly, even for larger LLMs.

Thank you once again for your thoughtful and detailed feedback, which offers valuable guidance for the improvement of our paper.

We extend our heartfelt thanks to Reviewer 7yH3 for your thorough and thoughtful review of our manuscript. Following are our responses to each individual comment.

Q1: use more recent datasets or metrics

We conduct experiments on a new benchmark, QAEdit, which is tailored for real-world QA tasks and adopts real-world evaluation metrics (released on February 16, 2025). For further details, please refer to our response to Reviewer 1PGC’s Q1.

Q2: simpler theoretical analyses may make the main text clearer

We agree that the least-squares problem in [1] offers a more succinct path toward deriving the update rule (Equation 3.5). That said, we find that keeping two assumptions in the main text helps bridge our theoretical arguments, and retaining Lemma 3.5 can aid readers in understanding the motivation (i.e., simulating the ideal editor). Nonetheless, we will streamline the theory to retain the minimum element that is essential to guide readers through Equation 3.5.

Q3: code link and transpose symbol

We will move the code link to the end of the abstract and correct the notation $A^T$ to $A^\top$ throughout the manuscript.

Thank you once again for your valuable time and effort in reviewing our work. These insightful feedbacks will significantly enhance the quality and clarity of our paper.

[1] Tan, Chenmien, Ge Zhang, and Jie Fu. Massive Editing for Large Language Models via Meta Learning. ICLR (2024).

We are deeply grateful to Reviewer nKJ9 for the detailed and constructive feedback on our manuscript. Below, we address your questions point-by-point.

Q1: evaluate on newer and, importantly, more accurate LLMs

We conduct additional evaluations on Llama-3 (8B) and Qwen2.5 (7B). We select FT, ROME, AlphaEdit$^-$, WISE, and AlphaEdit as baselines and perform $T=1000$ sequential edits using the ZsRE dataset. The experimental results are presented in the following table:

We observe that SimIE achieves competitive performance across these recent models, especially surpassing the SOTA method (AlphaEdit) by an average of $4.8\\%$ on Qwen2.5. These results are consistent with those in the main paper, further confirming the effectiveness of our proposed SimIE.

Q2: testing if the assumptions hold in practice

To provide deeper insights into our theoretical assumptions, we consider two additional analyses assessing the extent to which the assumptions are violated in practice. Please refer to hGQy’s Q2 for details.

Q3: focuses on the arithmetic mean

Although the arithmetic mean is widely adopted, it does have the potential pitfalls you mention. Inspired by your suggestion, we introduce a new composite metric defined as $e^{\alpha(\mathrm{Loc}-1)}(\mathrm{Rel}*\mathrm{Gen}).$ This metric takes locality as a penalty term, serving as a smoothed condition factor. Meanwhile, by using the (squared) geometric mean of reliability and generality, it prevents methods from abandoning one of them. We will incorporate this new metric into our paper, thereby offering a clearer view of practical utility.

Q4: still a gap between SimIE and specialized lifelong editors

We recognize that SimlE may underperform some specialized lifelong approaches under certain scenarios. Nevertheless, as increasingly challenging datasets and more realistic evaluations emerge, SimlE provides greater robustness compared to elaborated lifelong algorithms (refer also to our response to Reviewer 1PGC’s Q1). These findings reinforce our central claim: SimlE enables lifelong editing to benefit directly from the ongoing advances in standard editing research, thereby bridging the gap between these two paradigms.

Q5: related works and minor mistakes

All the relevant literature [1,2] mentioned will be thoroughly integrated into our discussion, and the capitalization of Llama-2 will be consistent.

Thank you once again for your careful and insightful comments, which provide valuable insights for further refinement of our work.

[1] Chen, Qizhou, et al. Lifelong knowledge editing for llms with retrieval-augmented continuous prompt learning. arXiv preprint arXiv:2405.03279 (2024).

[2] Sinitsin, Anton, et al. Editable neural networks. ICLR (2020).