Dear reviewer jUcr:

Thanks for your valuable feedback and constructive suggestions. Follow your advice, we have supplemented the relevant experiments and provided explanations, which have basically solved all the problems. We would greatly appreciate it if you could reconsider your final evaluation of our work.

---

Weaknesses:

W1:polish mathematical details and notation

In the next version, we will rename some notation and polish the descriptions of math details. We also build a mapping of notation and its definition to appendix A.

We clarify some notations and will rename them:
　$\delta$:incremental update parameter of edited layers' output $v$ (Eq. 2).
　$\Delta$:incremental update matrix for modifying the edited parameters (from closed-form solution in Theorem 2).
　$\delta^*_{e_1,e_2}$:optimal $\delta$ when editing $e_2$ while jointly considering both $e_1$ and $e_2$.

Sorry for the difficulty of reading math details, as our work involves many math details due to its substantial theoretical contributions.

W2:computation complexity

W2.1:framework is complex

The task of LLM sequential editing is new and challenging, thus we heuristically explore some components. Each part is essential for achieving that tasks (varified in ablation studies), while some parts (Sec. 2.3 and 2.4) can theoretically ensure the performance.

W2.2: complexity of our model: is it really linear?

Yes, the time complexity is really linear $O(n)$ as in appendix E.6.

We only need to compute the Fisher information matrix (FIM) itself (inverse FIM are not used in actual loss computation).

Computing about FIM requires gradients, which obtained directly during $\delta$'s backpropagation, allowing to extract d-dim diagonal Fisher values with $O(n*d)$ complexity, which is equivalent to $O(n)$ since $d$ is a constant (e.g. 4096). Using active forgetting in loss only needs $O(1)$, which modifies loss (Eq.6) without matrix computations.

Our additional experiments reports running time as the sample increasing. It shows time grows linearly as the sample number (edit counts) scaling and our model's run time is less than our baseline, MEMIT_full.

W3:difference:LEM vs ER

LEM is an advanced version of ER. Our proposed LEM is parameter-wise replay, ER is sample-wise replay (requires additional storage for samples)[D. Rolnick et al.NIPS'2019]. LEM's novelty is upgrading from sample-wise to parameter-wise, also verified in ablation study (Tab. 2).

W4:More insights about ablation study (FIM is not so important?)

Each component is necessary. LEM play a fundamental role while FIM/AF/ER can only performs well based on LEM.

• LEM(lines 870-873) fixes model collapse and catastrophic forgetting (CF) by controlling parameter deviation (Corollary 2). Note Fig.9 shows parameter drift and CF are serious problems, so LEM is fundamental compard to FIM/AF/ER.

• FIM design(lines 855-863) measures the degree of retaining prior knowledge. Fig. 8 shows that the FIM can effectively protect critical parameter updates for edited knowledge while preventing interference between different knowledge updates.

• AF module (lines 842-853) considers whether memorizes or forgets knowledge. Corollary 1 and Ablation show AF is most significant to generalization.

---

Questions:

Q1:forgetting mechanism: "append only updates" are feasible for LLM?

Yes, append-only updates are theoretically feasible for LLMs. While simply appending knowledge is possible, doing so effectively requires addressing critical challenges such as knowledge interference.

Retaining both old and new knowledge is indeed the ideal way. To perform well based on that way, human brains employ synaptic modulation[Michael et al. Science'2004] mechanisms to achieve effectively "functional forgetting" (selecting suitable knowledge while using it instead of relying on pure append-only approaches).

Similarly, our model also retain some old knowledge instead of totally removing.

Therefore, we believe that the key to append only updates is to enable the model to use knowledge correctly, and our approach is precisely striving in this direction.

Q2.1:Is the generalization bound suitable to assess active forgetting (AF)？

Yes, as Corollary 1, the generalization bound of active forgetting-integrated MAF loss improves over traditional cross-entropy bound, providing theoretical evidence that the introduction of AF leads to lower errors with the goal of addressing the generalization issues regarding editing knowledge.

Q2.2:measure effectiveness of machine unlearning?

Yes, we have verified effectiveness of machine unlearning in ablation study (Fig.7，line 867-878).

Q3:

Q3.1:$\delta_{e_1,e_2}^\*$'s meaning?

It represents incremental parameter for $v$(Eq.2) that simultaneously satisfies both edit $e_1$ (already edited) and edit $e_2$ (to be edited) via MAP estimation and Bayes theorem as Eq.9.

Q3.2 Is this a single parameter update related to two consecutive edits?

Yes. $\delta_{e_1,e_2}^\*$ handles the case where edit $e_1$ has been edited and we need to update parameters for edit $e_2$ considering $e_1$. We will clarify it in the final version.

Q4:

Q4.1:double check expression 9?

OK, we confirmed Eq.9 is correct.

Q4.2:In Sec4.2, which independence assumption did you use?

The independence assumptions stem from MAP estimation principles, where \delta follows a known prior distribution P(δ) rather than being a fixed constant. This is explicitly incorporated in Theorem 1's derivation.

Q4.3:explain relation between equation (6) and $p(e_2|\delta)$ ?

They are numerically opposite. The term log $p(e_2|\delta)$ represents $log$ maximum likelihood estimation for the edit data $e_2$, which is the negative form of $L_{MAF}$ in Eq.(6) modeled via MLE(Eq.5) too.

Q5

Q5.1: Eq.15 relies on strong assumptions?

Actually, the related full assumptions are not so strong:

1.Eq.9-11 require assumptions of MAP estimation and Bayes rules.

2.Eq.12-13 relies on Laplace approximation's assumptions.

3.Eq.14-15's assumption is in Q5.2

The deducetion of Eq.15: When we edit $e_n$, log $p(\delta|e_{<n})$ = $log p(e_{n-1}|\delta)$ +...+ $log p(e_2|\delta)+ log p(\delta|e_1)$ (Bayes rule). We assume optimal parameters for $e_i(i<n)$, allowing Taylor expansion of log $p(e_i|\delta)$ as negative loss: constant term and first derivative ≈ 0, and second derivative ≈ $1/2\delta^T(\lambda_iF_{e_i})\delta$. Thus yield $\log(\delta|e_1,e_2,...,e_{n-1}) = 1/2\delta^T \sum_{i=1}^{n-1} (\lambda_iF_{e_i})\delta$.

Q5.2: For example, why assuming optimal weight updates is zero?

1.This assumption has been validated in many continual learning studies [1,2,3].The assumption that optimal weight updates $\delta_{e_i}^{\*}$=0 in $(i+1)\_{th}$ is based on the premise: no incremental $\delta$ is required to produce optimal representations for $e_i$ at $(i+1)\_{th}$ edit. This logically leads to $\delta_{e_i}^*=0$ in $(i+1)_{th}$ edit.

2.In practice, since our trainable parameter $\delta$ is small (Llama3: 4096 dims), it can be easily optimized to a local optimum. As shown in our ablation study (Table 2), the FIM module remains effective under this assumption."

[1]Overcoming catastrophic forgetting in neural networks, 2017.

[2]Note on the quadratic penalties in elastic weight consolidation, 2018.

[3]Elastic weight consolidation (ewc): Nuts and bolts, 2021.

Q5.3: Does it have a Gaussian approximation with zero mean?

No, in Appendix C.2's proof, we did not assume a zero-mean Gaussian distribution. As explained in Q5.2, $\delta_{e_i}^\*=0$ for $i+1_{th}$ edit. When substituting $\delta_{e_i}^*=0$ into Eq.35, we naturally derive Eq.14.

Q5.4 Eq.14 and 15, shouldn't we use FIM inverse?

Eqs. 14-15 do not use FIM inverse. The FIM emerges naturally from our posterior modeling (Eqs. 33-34), where Taylor expansion links the second-order derivative to the FIM (yielding Eq. 14 and 15). The FIM inverse corresponds to covariance (Eq. 30), which cannot directly be used in Eqs. 14-15.

Q6

Q6.1:In Eq.16 (MAF loss), how to determine what to forget?

Our datasets annotate knowledge to be forgotten and our editing targets, explained in** lines 669-670**.

In practical application, users can assign what to forget as their demands, e.g. treat unsatisfactory LLMs' outputs as knowledge to be forgotten and treating desired output as targets to be memorized.

Q6.2:Is this ($o_1$ and $o_2$) to be replaced with the new objects?

Yes, we designate objects ($o_1$ and $o_2$) as knowledge to be forgotten, replacing them with the target knowledge we intend to memorize.

Q6.3:How do you locate weights "holding" those objects？

As knowledge localization studies[Meng et al NIPS'2022], our localization approach is to recognize that the second MLP linear layer in early transformer stores substantial factual knowledge(lines 107-108 and Appendix E.2). Notably, even knowledge targeted for forgetting was originally stored in this specific location prior to the forgetting operation.

Q7:Sec. 2.5, relation between $\delta$ and $\Delta$? $\Delta$ is not a matrix of learnable parameters?

Your understanding is correct . $\Delta$ is not a matrix of learnable parameters.

$\delta$ is the incremental update to the output vector $v$, obtained via training optimization of Eq.16. $\Delta$ is the incremental parameter for updating the model's edit parameters, computed via $\delta$ in Eq.18.

Q8: Why FIM does not seem to play a major role? why LEM seems important?

See the responce to W4

Dear Reviewer m5tL:

We sincerely appreciate your positive feedback and constructive suggestions. As requested, we have conducted additional experiments and carefully addressed all revision requirements. We are grateful for your insightful comments, which have significantly strengthened our manuscript.

---

Questions

Q1:sequential replay compare quantitatively to standard experience replay?

Compared to standard experience replay, our proposed sequential replay includes all edited data through parameterization, requiring nearly no additional memory or computation costs (see Appendix C3 for proof). Standard experience replay maintains a memory buffer storing the $M$ most recent edits (replay of all M samples during Eq.18 computation), our approach reduces both storage and computational to just 1/M of standard experience replay.

Quantitative performance comparisons are provided in Table 2 of the manuscript

Q2:

Q2.1 What assumptions underlie your theoretical results?

Our theoretical results requries the following assumptions:

1.The lemma 1 and corollary 1 specific assumptions primarily follow work [1] and are formally articulated in Lemma 1(line 160).

2.Theorem 1 incorporates both the maximum a posteriori probability assumption for parameters (line 194) and the Laplace approximation of posterior probability (line 201);

3.Theorem 2 assumes local optimum is achieved for all previous $i-1$ edits during the $i_{th}$ edit (line 609);The assumptions for Corollary 2 are explicitly stated in the paper (line 243).

Q2.2:Do the bounds apply to deep networks or mainly simplified cases?

Under Lemma 1 and its assumptions, our bound applies for deep neural networks optimized with Adam, where the generalization error bounds are derived based on Adam's gradient update mechanism (our approach).

For the simplified cases (not a deep neural networks optimized with Adam), the bound does not applied.

[1]Lipschitzness Effect of a Loss Function on Generalization Performance of Deep Neural Networks Trained by Adam and AdamW Optimizers，2023

Q2.3:how well do theoretical results match empirical trends?

1.Corollary 1 shows that our proposed active forgetting-based loss MAF achieves a superior generalization error bound compared to traditional cross-entropy (CE). This theoretical advantage is empirically validated in ablation studies in Table 2 and Fig. 7, where MAF outperforms CE in multiple metrics. The improved generalization performance align with the theoretical superiority of the error bound under generalized scenarios.

2.Based on Theorem 1, we constructed the loss function in Eq. 16 to protect important parameters during updates, with ablation studies in Fig. 8 visually confirming this protection;

3.Building upon Theorem 2 and Corollary 2, we implemented parameter replay,with ablation results in Tab. 2 and Fig. 8 empirically validating both the effectiveness of our replay and the controlled convergence of model parameters.

In summary, we have achieved a consistent match between our theoretical analysis and empirical results.

Q3:

Q3.1: How would HSE handle more complex tasks or longer task sequences?

For more complex scenarios that cannot be edited in a single operation, we propose decomposing such reasoning into smaller inference units and applying sequential editing. We have supplemented additional experiments using LLaMA3-8B-Instruct on MQUAKE-CF[2] multi-hop QA dataset, evaluating sequential editing of decomposed multi-hop facts:

The results demonstrate that our method remains effective for long-context multi-hop reasoning tasks when employing decomposed units sequential editing.

Q3.2: Bottlenecks with memory use or Fisher computation as task complexity increases?

Memory use : We only require two matrices to store the Fisher matrix and cov_sum, eliminating the need for additional storage. This design ensures that memory consumption does not escalate with increasing edit operations.

Fisher computation: 1.Computing about FIM requires its gradients, which obtained directly during $\delta$'s backpropagation, allowing us to extract d-dim diagonal Fisher values with $O(n*d)$ complexity, which is equivalent to $O(n)$ since d is a constant (e.g. 4096) 2.For editing time, our additional experiments compare time consumption across different scenarios:

The results demonstrate that FIM's time consumption grows slowly with edit counts, following $O(n)$ scaling.

Q4:

Q4.1:Why is the parametric memory more effective than ER?

Parametric memory (LEM) can guarantee convergence in parameter F-norm leading to edited parameters maintain compatibility with other layers’ parameters , while ER cannot provide this assurance.

LEM preserves F-norm convergence of edited parameters, demonstrating both theoretical (Corollary 2) and empirical(Fig.9) superiority over ER.

Q4.2 Does it prioritize important updates?

Yes, the importance of updates in parametric memory is governed by the convergence factor $\alpha_i$ (Theorem 2). Both Equation 18 and the proof in Appendix C.3 shows that higher $\alpha_i$ values correspond to greater significance of the associated knowledge edits. We provide comprehensive discussion of various $\alpha_i$ designs in Table 1.

Q4.3:How sensitive is HSE to hyperparameters like forgetting rate or Fisher strength?

1.Following your advices, regarding the memorizing and forgetting rate, we conducted additional experiments using the Llama3-8B-instruct model on two datasets.

The improvement in Spe suggests a reduction in interference for irrelevant knowledge. The optimal parameter configuration with memory factor = 0.8 and forget factor = 0.2 (ours hyperparameters).

2.Following your advices, regarding the Fisher information matrix coefficient $\lambda_i$, we conducted additional experiments using the Llama3-8B-instruct model on two datasets.

The results show that as $\lambda_i$ increases, generalization performance decreases while specificity improves, with optimal overall performance achieved at $\lambda_i=1e-1$.

Q5:Have you observed replay patterns similar to those in the hippocampus (e.g., forward/reverse replay, reward-focused)?

This is a particularly interesting and insightful observation: The phenomenon resembles reverse replay mechanisms, which the paper [3] primarily attributes to the "recency effect", where stronger backward replay patterns during memory retention correlate with more pronounced recency effects. In our replay framework, we similarly observe that more recent edits tend to yield better results, demonstrating an analogous recency effect pattern. Our sequential editing tests with 1,000 samples (shown in the table below) demonstrate that:

Empirical results shows that more recent edits consistently outperform earlier ones, manifesting a similar recency effect. We attribute this to the inherent property of continuous $\Delta$-based parameter updates, where newer knowledge experiences less interference while progressively disrupting earlier edits.

Regarding reward-focused mechanisms, this essentially considers whether the replay process prioritizes higher-reward memories. We posit that the replay intensity for edited knowledge primarily depends on our designed convergence factor $\alpha_i$. Higher $\alpha_i$ values indicate more focused replay for specific edited knowledge. The design rationale for $\alpha_i$ (Appendix C.3) and its functional impacts (Table 1) are thoroughly documented. For reward-focused scenarios, we can adaptively assign different $\alpha_i$ values to incorporate reward considerations based on practical requirements.

 [3] Fast-backward replay of sequentially memorized items in humans，2018

Dear Reviewer h2Sa:

We sincerely appreciate your insightful and positive comments. We believe all suggested experiments and explanations have been completed and addressed. We hope to continue receiving your supportive feedback during the discussion phase.

Weaknesses:

W1: Limited editing locality and adaptability: Assume fixed knowledge localization and may not generalize well to more distributed or higher-layer semantic representations.

1.Current mainstream approaches[1,2,3,4] in knowledge editing tasks for LLMs focus on the second linear layer of early transformer FFNs. We follow this setting based on empirical evidence from causal tracing of semantic knowledge[1,3,4] and memory storage mapping[2] that substantiates this design choice.

2.In practical, for most existing kinds of knowledge editing [1][4], we maintain that editing on the linear layer is sufficient. As demonstrated in [1], perturbing early linear layers plays a crucial role in model predictions, exerting greater influence than other layers. Furthermore, [4] reveals that the output patterns of early linear layers map to human-recognizable patterns in the vocabulary space.

[1]Locating and editing factual associations in gpt, 2022.

[2]Transformer feed-forward layers are key-value memories, 2021.

[3]Alphaedit: Null-space constrained knowledge editing for language models, 2024.

[4]Mass editing memory in a transformer, 2023.

W2: Restricted capacity for complex edits: The use of low-rank additive updates (i.e., rank-one updates per edit) limits the model’s capacity to complex or structural knowledge edits, such as causal reasoning or multi-hop inferences.

We indeed use low-rank to approximate the full version of the model. Current low-rank approximation research, including LoRA [5] and its variants, has demonstrated performance comparable to fully fine-tuning with mature implementations. Our additional experiments shows the performance of fully fine-tuning of editing layers (Fully-FT-L), which consistently show our method's superiority (Fig.2).

For more complex scenarios like causal reasoning or multi-hop inference that cannot be edited in a single operation, we can decompose a complex editting task into multiple single-step editting task and apply sequential editing. In such the usage, we can easily adapt existing editting methods (including our method) to those complex scenarios. Further, compared to existing editting methods, our method is skilled at editting methods, which ensures the performance under a long sequecne of editting.

We have supplemented additional experiments using LLaMA-3-8B-Instruct on MQUAKE-CF[6] multi-hop QA dataset, evaluating sequential editing of decomposed multi-hop facts:

The results demonstrate that our method remains effective for long-context multi-hop reasoning tasks when employing decomposed units sequential editing.

We will explore knowledge editting on more complex scenarios in our future tasks.

[5]LoRA: Low-Rank Adaptation of Large Language Models, 2022.

[6]AKEW: Assessing Knowledge Editing in the Wild, 2024.

W3: Fisher matrix overhead: While the FIM-based update regulation can become computationally expensive as the number of edits increases.

1.Regarding computational complexity, our method maintains an $O(n)$ theoretical complexity. Computing about FIM requires its gradients, which obtained directly during $\delta$'s backpropagation, allowing us to extract d-dim diagonal Fisher values with $O(n*d)$ complexity, which is equivalent to $O(n)$ since d is a constant (e.g. 4096)

2.For editing time, our additional experiments compare time consumption across different scenarios:

The results demonstrate that FIM's time consumption grows slowly with edit counts, following $O(n)$ scaling.

W4:Key-value encoding assumption: The method assumes that factual edits is well-represented in a key-value format. It restricts applicability to structured knowledge (e.g., triples or QA pairs) and limits its extensions.

1.For knowledge editting for LLMs, most current mainstream research[1,2,3,4] employs Key-Value format to encode knowledge, which can also effectively handle triples and QA pairs.

2.For more general knowledge forms like multi-hop QA in long texts, we can decompose a complex editting task into multiple single-step editting task and apply sequential editing, which can be modeled as key-value pairs like normal single-step editting . Our supplementary experiments on the MQUAKE-CF dataset (mentioned in W2 response) demonstrate our method's generalization ability to complex knowledge editing scenarios.

W5:Replay via covariance only: The replay mechanism uses covariance matrices instead of actual examples or activations. This lightweight design may accumulate approximation error over time, especially under domain shifts or noisy edits.

1.We have theoretically demonstrated in Theorem 2 that the covariance enables full replay of all edited data while controlling error accumulation across edits. Furthermore, Corollary 2 establishes the stability guarantee by proving its convergence in parameter F-norm.

2.Furthermore, our ablation studies in Table 2 demonstrate that our method with parameter replay with covariance summation outperforms conventional experience replay methods (replaying actual examples).

Dear reviewer S7KK:

We sincerely appreciate your positive feedback and constructive suggestions. As requested, we have conducted additional experiments and carefully addressed all revision requirements. We are grateful for your insightful comments, which have significantly strengthened our manuscript.

---

Weaknesses:

W1:It does not systematically explore the sensitivity of these hyperparameters(such as the Fisher information matrix coefficient $\lambda_i$, the convergence factor $\alpha_i$, etc.) in different scenarios and the optimal adjustment strategies.

1.Following your suggestions, we conduct additonal experiments to systematically explore the choise of hyperparameters. We report the results that tunes the Fisher information matrix coefficient $\lambda_i$ using the Llama3-8B-instruct model on two datasets.

The results show that as $\lambda_i$ increases, generalization performance decreases while specificity improves, with optimal overall performance achieved at $\lambda_i=1e-1$ (ours).

2.For the hyperparameter $\alpha_i$ adjustment, we have provided detailed discussion in Appendix D Table 1, including analysis of why we selected $\alpha_i=n/(i-1)$ as our configuration.

3.We will systematically explore more hyperparameters and update the results in the next version.

W2:A related method[1] that first forgets old knowledge and then updates new knowledge has not been discussed and compared:

[1] Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models. 2024.

We acknowledge that F-learning indeed highlights the importance of forgetting mechanisms in model editing, and we will cite it in our subsequent work. Below we present the performance comparison of continuous editing scenarios on two datasets using the Llama3-8B-instruct model.

The results demonstrate that our method comprehensively outperforms F-learning across all evaluation metrics.

We will regard this method as a new baseline and discuss it in the related work in the next version.