A$^3$E: Towards Compositional Model Editing

We sincerely thank Reviewer dHwZ for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain.

W1. The first thing that comes to mind is when we need composition and when we don’t. In some cases, composition is necessary; in others, we may need to revert previously edited knowledge. A better approach for compositional model editing might be to inject complete knowledge into the model and erase prior incomplete knowledge.

We greatly appreciate the reviewer for taking this point into consideration, which means that the reviewer has given in-depth thought to the significance of the compositional model editing scenario. In fact, we also took this into account in our research but finally chose to perform each edit independently for the following reasons, which we promise to discuss in the revised version:

• The reviewer's example actually falls under standard non-compositional model editing, not compositional model editing. This is because its assumption is that when editing the question "What are the symptoms of COVID-19?", we already know the standard answer is "fever, loss of taste." The example provided by the reviewer can be resolved by A$^3$E in two steps:
  1. Locate and delete the outdated edit "fever" (refer to W2 for details).
  2. Update the answer to "fever, loss of taste" with non-compositional model editing.
  We demonstrated in Fig. 8$c$ that the performance of A$^3$E in non-compositional model editing is on par with baselines.
• However, compositional model editing addresses a different scenario where: during the edit training for "fever" and the edit training for "loss of taste", they are unaware of each other. We will explain why this scenario is meaningful below.
• As stated in Sec. 2.1 (cf. Lines 97–99) and illustrated in Fig. 1, we believe that it is necessary to support model editing in a federated manner. Specifically, when an open-source model is deployed globally, numerous distinct errors may arise simultaneously across different deployed regions. We want these deployers can benefit from the edits of each other.
  • In such cases, similar to the assumption that raw data cannot be shared in federated learning due to regulations like GDPR [1], different deployers are only able to share trained edits. In such cases, when a deployer receives trained edits from other deployers, it does not know the questions of the edits. It is impossible to get complete answers during edit training because:
    1. The deployer has to iterate through all previous edit questions to determine which previous edits need to be composed with the newly received edit to form a complete answer.
    2. However, storing all previous edit questions is impossible in lifelong model editing.
  • When exchanging raw data is allowed, local deployers still have to perform immediate model editing to mitigate potential adverse effects without waiting to receive errors discovered by other deployers. In this scenario, assuming that each of $n$ deployers updates an answer to the same question and exchanges these $n$ different answers with each other.
    • Continuously injecting complete knowledge would require $n$ times edit training for each deployer at most.
    • Exchanging independently trained edits would only require $1$ time edit training for each deployer.
    Training edits independently while ensuring their composability supports model editing in a federated manner much more efficiently. This is especially important in the field of model editing, which emphasizes immediate correction of errors.
• Meanwhile, for multi-question composition, compositional model editing is clearly meaningful because,
  • Edits for any two different questions may need to be composed during testing.
  • Different errors to be edited may be discovered at different time.
  Therefore,
  • It is impossible to exhaustively cover all possible composition cases for complete knowledge during the edit training process.

[1] The eu general data protection regulation (gdpr)

W2. How do you handle continuous editing situations, when previously edited knowledge needs to be changed?

We greatly appreciate the reviewer for drawing our attention to this point, which indeed increases the practicality of A$^3$E. When adding the edit for "loss of taste," if we recognize that the existing edit for "fever" is outdated, our method can easily remove the outdated edit with the following locate-and-delete algorithm:

1. Iterate and mask each edit $(\mathbf{A},\mathbf{B})$ in the retrieved part $\mathcal{S}$ of the vector database during the generation process -> locate the edit that leads to the outdated answer.
2. Delete the located edit.

We consider a three-answer composition scenario with $3\times50$ successful edits to test the effectiveness of the locate-and-delete algorithm above, where each question receives two correct edits and one outdated edit. The accuracy of deleting the outdated edits is 100%. We will encourage the construction of outdated edits feedback and avoidance mechanisms when deploying A$^3$E.

W3. The concern about unfair comparison with baselines.

We would like to humbly clarify that our evaluation is fair for evaluating the composability.

• We adapt all baselines to make it applicable to multi-answer composition without treating the latest edit as the new ground truth. As shown in Fig. 1, our evaluation of all baselines follows the protocol of training multiple edits for one question independently and then composing them to ensure a fair comparison, focusing on evaluating the composability. Please refer to Sec. 2.2 for the edit composing of the baselines. Specifically,
  • Each edit is trained without knowledge of other edits for the same question. In IMAC (cf. Lines 269-274), to accurately evaluate the improvement in composability brought by A$^3$E, edits for the same question are trained on top of the pre-trained model and then composed. In MAC (cf. Lines 269-274), to verify that we also significantly improve composability in lifelong scenarios, edits for the same question are trained on top of previous edits for different questions and then composed. In this way, we adapt the baselines to compose edits. We elaborate on the necessity of this evaluation scenario in W1.
• Regarding the reviewer’s suggestion to “erase the previous, incomplete knowledge and then insert the updated, correct information,” we provide a detailed discussion in W1 on why this approach is infeasible for compositional model editing.

We sincerely thank Reviewer cjhv for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain.

W1. Adding a running example for the method

We sincerely appreciate the reviewer’s suggestions for improving the explanation of the method.

• We commit to updating Fig. 7 in the revised version to include an intuitive running example consisting of two stages: edit training and edit composing. Due to image upload limitations, we summarize the content of each stage here:

  Edit training

  • How to adaptively combine pre-trained foundation knowledge into new knowledge.
  • How to adaptively regularize and an intuitive example of the effects brought by the regularization.

  Edit composing

  • How to use the vector database to filter out irrelevant edits during inference.
  • How to adaptively merge different combinations of pre-trained foundation knowledge to leverage them simultaneously during inference.

W2&Q1. The distinction between Incorrect Preceding and Knowledge Sinking

We greatly appreciate the reviewer's suggestion for further distinguishing between Incorrect Preceding and Knowledge Sinking. Referring to the examples and definitions provided in Fig. 2, we offer the following contrasting definitions:

• Knowledge Loss refers to some parts of the edited answers were perturbed or missing.
  • fever, loss of taste -> fever, loss of fever
• Incorrect Preceding refers to cases that the model knows all edited answer but outputs the wrong answers first.
  • fever, stomachache, loss of taste
• Knowledge Sinking refers to cases that the model potentially knows some edited answers but being unaware that it should generate them, instead producing only some of the edited answers before generating non-answer content.
  • fever fever is a …

W3&S2. Performance with large composition number

We sincerely appreciate the reviewer for raising this point. In PEAK-CF dataset, we conducted additional experiments by sampling 50 questions with 10+ unknown answers each and evaluated the performance with a composition number of 10.

Method	ROME	AlphaEdit	WISE	T-patcher	GRACE	MELO	A$^3$E
SR-1	2.40	4.60	9.40	6.60	8.60	12.20	17.20

The experimental results demonstrate that when a large number of edits need to be applied simultaneously,

• A$^3$E still significantly outperforms the baselines on the SR-1 metric, which means A$^3$E retains more edited knowledge with large composition numbers.

Q2. How is the adaptive mask in the Adaptive Combination module computed in practice?

In practice, $\mathbf{M} \in \mathbb{R}^{r\times n}$ is the mask to select top $k$ related values (rows) in $\mathbf{W}\_{\text{down}} \in \mathbb{R}^{n\times m}$ for each answer token, where the correlation is calculated by the dot product between values (rows) in $\mathbf{W}\_{\text{down}} \in \mathbb{R}^{n\times m}$ and the answer tokens' embedding. The specific calculation process is as follows:

1. Calculate the correlation $\mathbf{V}\_s \in \mathbb{R}^{1\times n}$ between values (rows) in $\mathbf{W}\_{\text{down}} \in \mathbb{R}^{n\times m}$ and the embedding of each answer token $s$ in the token set of an answer $\mathcal{A}\_s$:

$

\mathbf{V}_s=[\mathbf{W}_{\text{unemb}}]_s\mathbf{W}_{\text{down}}^{\top}

$

$\mathbf{W}\_{\text{unemb}} \in \mathbb{R}^{d\times m}$ is the unembedding matrix. $d$ is the vocabulary size.

2. Select top $k$ related values (rows) in $\mathbf{W}\_{\text{down}} \in \mathbb{R}^{n\times m}$ for each answer token $s$:

$

\{v|[\mathbf{V}_s]_v \in \text{top}_k(\mathbf{V}_s)\}

$

$v$ is the index of the selected values (rows) in $\mathbf{W}\_{\text{down}} \in \mathbb{R}^{n\times m}$. 3. Get the union of $v$ selected by all answer tokens in $\mathcal{A}\_s$:

$

\mathcal{K}=\bigcup\nolimits_{s\in \mathcal{A}_s}\{v|[\mathbf{V}_s]_v \in \text{top}_k(\mathbf{V}_s)\}

$

4. Calculate the adaptive mask:

$

[\mathbf{M}]_{:,i}=&\left\{ \begin{array}{rcl} 0 & & i \notin \mathcal{K} \\ 1 & & i \in \mathcal{K} \end{array} \right.

$

Q3. How to apply adaptive merging?

Our proposed adaptive merging operates at the token level to enable token-specific adaptations on the utilization of pre-trained foundation knowledge. Specifically, for $n$ edits that need to be applied simultaneously, each token's $h\_{\text{in}}$ computes $n$ $\Delta a$ in Eq. (4). We then perform adaptive merging on these $n$ $\Delta a$ values in Eq. (8).

S1. Discuss generalization limits.

We sincerely appreciate the reviewer's suggestion regarding the potential generalization limitations of A$^3$E in various scenarios.

• Noisy scenarios. In Fig. 8(d), we tested the impact of different few-shot prompts on A$^3$E's performance. These varied few-shot prompts not only controlled the model's output but also simulated different noise present in the context. The results show that while A$^3$E's performance is affected, it still maintains a leading performance compared to the baselines.
• Noisy open-domain scenarios. We humbly acknowledge that in open-domain scenarios, noisy contexts may still impact the retrieval accuracy of the vector database. Since A$^3$E primarily focuses on edit training and edit composing, we will explore
  • optimizing the hidden state positions for retrievement based on the real data distribution in application scenarios
  • employing more powerful embedding models
  • training a question-rewriting module to better align with the vector database
  for optimal performance in the future work.
• Adversarial scenarios. We conducted additional experiments using $2\times50$ successful edits from the PEAK-CF dataset under the IMAC (cf. Lines 269-274) setting by providing the tested question with incorrect answers in the few-shot prompts. The results demonstrate that A$^3$E consistently (100%) outputs the edited answers across all test cases, proving its robustness against contextual adversarial interference.

S3. Check if there are any mistakes.

We can guarantee the correctness of our implementation. The poor performance of existing baselines primarily stems from the fact that none of them consider the scenario where multiple edits are applied simultaneously, nor do they address the resulting conflicts between these edits. We promise to release the codes upon acceptance of the paper.

Dear Reviewer cjhv,

As the discussion phase is approaching its end, we would like to follow up to see if our response addresses your concerns or if you have any further questions. We would really appreciate the opportunity to discuss this further if our response has not already addressed your concerns. Thank you again!

We sincerely thank Reviewer e3yR for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain.

W1. Non-compositional ME evaluation should be conducted on other classic datasets for ME.

We greatly appreciate the reviewer for drawing our attention to this point, which indeed benefits from more solid experiments. In response to the reviewer's suggestion, we provide additional non-compositional results of state-of-the-art baselines on classic ME dataset ZsRE [1] and Counterfact [2] following [3] on Llama3-8B, which demonstrate that our method is on par with baselines.

                                  ZsRE                          Counterfact

Method	SR-1	GSR	LSR	SR-1	GSR	LSR
FT	47.95	52.32	73.08	51.53	48.14	63.95
ROME	99.53	97.52	95.57	100.00	92.77	77.40
MEND	92.24	90.70	96.23	79.10	58.76	91.19
AlphaEdit	98.00	96.54	95.57	99.55	95.03	79.21
WISE	93.39	92.53	100.00	98.59	93.50	99.32
T-patcher	95.28	94.24	92.62	95.03	93.79	89.38
GRACE	96.69	94.00	100.00	99.77	94.12	100.00
MELO	95.28	94.08	100.00	98.53	92.88	100.00
A$^3$E	97.00	96.77	100.00	99.77	93.79	100.00

[1] Zero-shot relation extraction via reading comprehension (ACL 2017)

[2] Locating and editing factual associations in GPT (Neurips 2022)

[3] A Comprehensive Study of Knowledge Editing for Large Language Models

W2&Q1. How is the method applicable to larger LLMs and/or black-box LLMs?

A$^3$E is applicable to larger LLMs.

• We humbly acknowledge the value of testing on larger LLMs. Due to limitations in computational resources, we provide experimental results for GRACE, MELO, and A$^3$E under the MAC scenario (Sec. 5.2, cf. Lines 267-274) using Llama3-70B and the PEAK-CF dataset. As shown in the following table, A$^3$E maintains the leading performance. We promise to provide complete experimental results in the revised version.

  Method	SR-S	SR-1	GSR	LSR
  GRACE	16.32	42.48	11.24	24.02
  MELO	28.12	48.64	26.58	23.18
  A$^3$E	50.69	69.47	47.10	25.10

We sincerely appreciate the reviewer's suggestion regarding the discussion on the applicability of A$^3$E to black-box models. We promise to include the following discussion in the revised version:

• In the field of knowledge editing for large language models,
  • One research direction focuses on editing white-box models, i.e., model editing [1-4]. The access to model weights is a prerequisite for applying these editing techniques.
  • Another research direction addresses knowledge editing for extremely large-scale models where model editing may be prohibitively expensive or black-box models by performing edits at the text level, including [5-15]
• A$^3$E follows the first direction to edit white-box models and focuses on compositional model editing because text-level knowledge editing is not applicable for the following reason:
  • As stated in Sec. 2.1 (cf. Lines 97–99) and illustrated in Fig. 1, we aims to support model editing in a federated manner. Specifically, when a model is deployed globally, numerous distinct errors may arise simultaneously across different deployed regions. We want these deployers can benefit from the edits of each other.
  • In such cases, similar to the assumption that raw data cannot be shared in federated learning due to regulations like GDPR [16], different deployers are only able to share trained edits rather than the texts.
• In a centralized scenario, A$^3$E can also be implemented by the owner of black-box models for immediate error correction with much more composable edits.
• We recommend users to choose methods from either direction based on their needs, and we will explore how to perform compositional knowledge editing at the text level for black-box models in our future work.

[1] Alphaedit: Null-space constrained knowledge editing for language models (ICLR 2025 outstanding paper)

[2] MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA (AAAI 2024)

[3] WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (Neurips 2024)

[4] Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors (Neurips 2023)

[5] DeepEdit: Knowledge Editing as Decoding with Constraints

[6] Can we edit factual knowledge by in-context learning?

[7] Evaluating the ripple effects of knowledge editing in language models. (TACL 2024)

[8] Mquake: Assessing knowledge editing in language models via multi-hop questions. (EMNLP 2023)

[9] Retrieval meets reasoning: Dynamic in-context editing for long-text understanding

[10] Multi-hop question answering under temporal knowledge editing (COLM 2024)

[11] Retrieval-enhanced knowledge editing in language models for multi-hop question answering (CIKM 2024)

[12] Assessing and post-processing black box large language models for knowledge editing (WWW 2025)

[13] Fine-grained Hallucination Detection and Editing for Language Models (COLM 2024)

[14] Ripplecot: Amplifying ripple effect of knowledge editing in language models via chain-of-thought in-context learning (EMNLP 2024)

[15] Decoupling Reasoning and Knowledge Injection for In-Context Knowledge Editing

[16] The eu general data protection regulation (gdpr)

W3. Failure cases of editing

We greatly appreciate the reviewer for taking this point into consideration. We conducted failure case analysis for overlapping, contradictory, and capacity-exceeding edits as follows:

• For overlapping and contradictory cases, we conducted statistical analysis on the PEAK-CF dataset under the IMAC scenario with 50 cases of two-edit composition, measuring:

  • The overlapping selection of pre-trained foundation knowledge, calculated by the overlapping length between non-zero $\Delta a$ (Sec. 4.1) regions of two edits in adaptive merging (Sec. 4.2)
  • The contradictory utilization of pre-trained foundation knowledge, calculated by the length of regions with different signs between $\Delta a$ (Sec. 4.1) of two edits in adaptive merging (Sec. 4.2)

  The table below presents how the SR-S metric, which is inversely proportional to the proportion of failure cases, varies across different overlapping and contradictory ranges. As the overlapping length and contradictory length increase, the SR-S of A$^3$E gradually decreases, but it remains significantly higher than the baselines.

  Overlapping	[0,500)	[500,1000)	[1000,1500)	[1500,$+\infty$)
  SR-S	75.00	66.67	45.45	50.00

  Contradictory	[0,200)	[200,400)	[400,600)	[600,$+\infty$)
  SR-S	66.67	68.75	57.14	46.67

  Baselines	MEND	ROME	AlphaEdit	WISE	T-patcher	GRACE	MELO
  SR-S	4.00	4.00	8.00	32.00	20.00	14.00	20.00

• For capacity-exceeding cases, we found that the failure cases of baselines and A$^3$E in both compositional model editing and non-compositional model editing scenarios often involve numerical data in answers. For example,

  • Edit 1: Rheinmetall, which has designed 12 cm leFH 08

    Test 1: Rheinmetall, which has designed 12 cm leFH 8 (failure)

  • Edit 2: Rheinmetall, which has designed 35.5 cm Haubitze M1

    Test 2: Rheinmetall, which has designed 35.5 cm Haubitze M1 (success)

  • Composition test: Rheinmetall, which has designed 12 cm leFH, 12 cm Haubitze M8 (failure)

  This may stem from the model's inherent inability to accurately distinguish numerical data. Therefore, addressing such inherent limitations of the model is a valuable direction for future work in model editing.

We believe these findings can guide future research to further improve edit composability.

Dear Reviewer e3yR,

As the discussion phase is approaching its end, we would like to follow up to see if our response addresses your concerns or if you have any further questions. We would really appreciate the opportunity to discuss this further if our response has not already addressed your concerns. Thank you again!

Dear Reviewer e3yR,

Thanks for your follow-up question. We humbly clarify that our method is applicable to larger LLMs such as Llama3-70B. In our response to W2&Q1, we have already provided experimental results for state-of-the-art baselines GRACE, MELO, and A$^3$E under the MAC scenario (Sec. 5.2, cf. Lines 267-274) using Llama3-70B and the PEAK-CF dataset, where A$^3$E maintains leading performance.

The current results focus on the MAC scenario to ensure a clear and fair comparison within rebuttal time, but we emphasize that the approach naturally generalizes to all scenarios. We promise to provide the full set of experiments—covering all datasets, scenarios, and baselines of CME with Llama3-70B—in the revised manuscript.

Dear Reviewer e3yR,

Thanks for raising the score. Your constructive feedback and the time you've dedicated to our work are greatly appreciated. Moreover, we will integrate the suggestions into our revised manuscript.

In the tables below, we provide a more detailed analysis of the computational overhead compared to baselines with Llama3-8B and Llama3-70B from three aspects: trainable parameters, extra FLOPs for inference (infer.), extra FLOPs for training (train.). We only provide the extra FLOPs for training of T-patcher, GRACE, MELO and A$^3$E because other methods that update the whole matrix or train a hypernetwork have significantly larger training overhead. Specifically,

• Trainable parameters denote the parameters that are updated during training for each edit.
• Infer. denotes extra FLOPs compared to the pre-trained model for the inference of $u$ tokens with $v$ input tokens.
• Train. denotes extra FLOPs compared to the vanilla forward-backward progress with cross-entropy loss for $w$ training steps of an edit with $z$-token answer.

For Llama3-8B,

Method	Trainable Parameters	infer. (GFLOPs)	train. (GFLOPs)
FT	0.54GB	0	-
ROME	0.54GB	0	-
KE	0.72GB	0	-
MEND	1.14GB	0	-
AlphaEdit	0.54GB	0	-
WISE	0.11GB	$0.27v+0.27u$	-
T-patcher	32.77KB	$0.02u$	$0.01wz$
GRACE	32.77KB	$0.02$	0
MELO	98.30KB	$0.02$	0
A$^3$E	39.94KB	$0.02+4.92u\times 10^{-5}$	$0.03z+1.28wz\times 10^{-4}+1.64w\times 10^{-5}$

We also provide the extra FLOPs of the proposed adaptive combination, adaptive regularization and adaptive merging for inference and training:

• Adaptive merging of A$^3$E: $1.64u\times 10^{-5}$ GFLOPs
• Adaptive combination of A$^3$E: $0.03z$ GFLOPs
• Adaptive regularization of A$^3$E: $1.28wz\times 10^{-4}+1.64w\times 10^{-5}$ GFLOPs

In A$^3$E, $w$ is set to 50 and $z<u$ for a instance. The extra FLOPs for training and inference are very small compared to the generation of $u$ tokens with $v$ input tokens with an 8B model, which is about $16u+16v$ GFLOPs.

For Llama3-70B,

Method	Trainable Parameters	infer. (GFLOPs)	train. (GFLOPs)
FT	1.88GB	0	-
ROME	1.88GB	0	-
KE	2.52GB	0	-
MEND	2.09GB	0	-
AlphaEdit	1.88GB	0	-
WISE	0.38GB	$0.94v+0.94u$	-
T-patcher	65.54KB	$0.03u$	$0.02wz$
GRACE	65.54KB	$0.03$	0
MELO	180.22KB	$0.03$	0
A$^3$E	72.71KB	$0.03+1.19u\times 10^{-4}$	$0.09z+1.28wz\times 10^{-4}+2.87w\times 10^{-5}$

We also provide the extra FLOPs of the proposed adaptive combination, adaptive regularization and adaptive merging for inference and training:

• Adaptive merging of A$^3$E: $2.87u\times 10^{-5}$ GFLOPs
• Adaptive combination of A$^3$E: $0.09z$ GFLOPs
• Adaptive regularization of A$^3$E: $1.28wz\times 10^{-4}+2.87w\times 10^{-5}$ GFLOPs

In A$^3$E, $w$ is set to 50 and $z<u$ for a instance. The extra FLOPs for training and inference are very small compared to the generation of $u$ tokens with $v$ input tokens with a 70B model, which is about $140u+140v$ GFLOPs.

We sincerely thank Reviewer qR3R for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain.

W1&Q2. Lack of readability caused by variable names without definition.

We sincerely appreciate the reviewer's suggestions for improving the readability of the paper. We promise to include the following notation table in the revised version, with each notation clearly defined.

Notation	Definition
$\mathbf{m}$	Input & output dimension of feed-forward network (FFN)
$\mathbf{n}$	Hidden dimension of FFN
$\mathbf{d}$	Vocabulary size
$\mathbf{W}_{\text{up}}$	Parameters of the first layer of FFN; $\mathbf{W}_{\text{up}} \in \mathbb{R}^{m\times n}$
$\mathbf{W}_{\text{down}}$	Parameters of the second layer of FFN; $\mathbf{W}_{\text{down}} \in \mathbb{R}^{n\times m}$
$\mathbf{b}_{\text{up}}$	Bias vector of the first layer of FFN; $\mathbf{b}_{\text{up}} \in \mathbb{R}^{1\times n}$
$\mathbf{b}_{\text{down}}$	Bias vector of the second layer of FFN; $\mathbf{b}_{\text{down}} \in \mathbb{R}^{1\times m}$
$h_{\text{in}}$	Input hidden state of FFN; $h_{\text{in}} \in \mathbb{R}^{1\times m}$
$h_{\text{out}}$	Output hidden state of FFN; $h_{\text{out}} \in \mathbb{R}^{1\times m}$
$\mathbf{a}$	Activation vector after the first layer of FFN; $\mathbf{a} \in \mathbb{R}^{1\times n}$
$(\mathbf{A}, \mathbf{B})$	Two learnable low-rank matrices $\mathbf{A} \in \mathbb{R}^{n\times r} (\mathbb{R}^{m\times r})$ and $\mathbf{B} \in \mathbb{R}^{r\times m} (\mathbb{R}^{r\times n})$ to constrain the parameter updates to lie in a low-rank subspace following [1,2]
$\mathbf{M}$	The mask to select pre-trained foundation knowledge in $\mathbf{W}_{\text{down}}$; $\mathbf{M} \in \mathbb{R}^{r\times n}$
$h_{\text{logit}}$	Output logits of LLM
$\mathcal{W}$	The set of all token IDs
$\mathcal{S}$	The preserved edits for inference by the vector database
$a$	Generated answer of LLM
$\hat{\mathcal{Y}}$	The set of generated answers of LLM
$\mathcal{Y}$	The set of edited answers

[1] Lora: Low-rank adaptation of large language models. (ICLR 2022)

[2] Melo: Enhancing model editing with neuron-indexed dynamic lora (AAAI 2024)

W2&Q1. Benchmark dataset construction process

We sincerely appreciate the reviewer’s suggestions for improving the reproducibility of our paper. We promise to provide a complete benchmark dataset construction process in the revised version as follows (using PEAK-CF as an example):

1. The original PEAK-CF [1] is a model editing dataset containing questions with multiple answers.
2. We queried Llama3-8B and Mistral-7B, selecting questions from PEAK-CF where both models missed four or more answers, along with their corresponding missing answers in PEAK-CF.
3. For multi-answer composition, the retained questions and their randomly selected $c$ missing answers form a test instance, where $c$ is the composition number. (See Tab. 9 for an example)
4. For multi-question composition, we randomly selected $c$ retained questions without repetition and one randomly chosen missing answer per question to form a test instance. (See Tab. 10 for an example)

[1] Neighboring Perturbations of Knowledge Editing on Large Language Models (ICML 2024)

W3. Distinguishing naming of the different parts of the proposed method and intuitive explanation of the method

We sincerely appreciate the reviewer’s suggestions for improving the explanation of the method.

Distinguishing naming

• We have revised adaptive combination to the more detailed adaptively combine pre-trained foundation knowledge and modified adaptive merging to adaptively merge different combinations of pre-trained foundation knowledge.

Intuitive explanation

• We commit to updating Fig. 7 in the revised version to include an intuitive running example consisting of two stages: edit training and edit composing. Due to image upload limitations, we summarize the content of each stage here:

  Edit training

  • How to adaptively combine pre-trained foundation knowledge into new knowledge.
  • How to adaptively regularize and an intuitive example of the effects brought by the regularization.

  Edit composing

  • How to use the vector database to filter out irrelevant edits during inference.
  • How to adaptively merge different combinations of pre-trained foundation knowledge to leverage them simultaneously during inference.

W4&Q3. Three critical problems faced by compositional model editing methods

• We humbly clarify that these three problems are not derived from prior studies or commonly used in compositional studies.
• Instead, they were identified and defined for the first time by summarizing failure cases in existing methods under compositional model editing, making them critical problems specific to this field.
• The reason we focus solely on these three problems is that they comprehensively encompass all the failure cases we observed.
• We promise to add the above motivations for the three critical problems in the revised version.

W5&Q4. The connection between knowledge loss and Finding 1

We sincerely appreciate the reviewer’s suggestion to strengthen the connections among problems, observations, and solutions in the paper. We have reorganized the logic in Finding 1 as follows:

• Motivation:
  • The distinction between pre-trained knowledge and edited knowledge from existing model editing methods lies in the fact that pre-trained knowledge is the combination of pre-trained foundation knowledge in the down projection matrix, whereas existing model editing methods directly manipulate the output of the down projection layer.
  • Meanwhile, we know that pre-trained models can leverage pre-trained knowledge to answer questions requiring multiple answers, but when using multiple edited knowledge, they exhibit significant knowledge loss.
  • Therefore, we intuitively hypothesize that knowledge loss may partly stem from the fact that edited knowledge is not the combination of pre-trained foundation knowledge.
• Observations:

  • We observe that editing $\mathbf{W}\_{\text{up}}$ (i.e., combining pre-trained foundation knowledge into edited knowledge) results in less min knowledge loss across 27-31 layers and across baselines compared to editing $\mathbf{W}\_{\text{down}}$ or editing both $\mathbf{W}\_{\text{up}}$ & $\mathbf{W}\_{\text{down}}$ in the following table while leading to higher SR-S and similar non-compositional performance (Fig. 4).

    Method	Min knowledge loss (%)
    Up	60.6
    Down	71.4
    Up&Down	69.7

• Solutions:
  • We opt to edit only $\mathbf{W}_{\text{up}}$.

W6&Q5. More discussions on the computational and memory overhead compared to baselines

We greatly appreciate the reviewer for the suggestions. In the table below, we provide a more detailed analysis of the computational and memory overhead compared to baselines with Llama3-8B from four aspects: storage overhead, trainable parameters, extra FLOPs for inference (infer.), extra FLOPs for training (train.). We only provide the extra FLOPs for training of T-patcher, GRACE, MELO and A$^3$E because other methods that update the whole matrix or train a hypernetwork have significantly larger training overhead. Specifically,

• Storage overhead refers to the additional parameters that need to be stored compared to the pre-trained model for 1000 edits.
• Trainable parameters denote the parameters that are updated during training for each edit.
• Infer. denotes extra FLOPs compared to the pre-trained model for the inference of $u$ tokens with $v$ input tokens.
• Train. denotes extra FLOPs compared to the vanilla forward-backward progress with cross-entropy loss for $w$ training steps of an edit with $z$-token answer.

Method	Storage Overhead	Trainable Parameters	infer. (GFLOPs)	train. (GFLOPs)
FT	0	0.54GB	0	-
ROME	1.64GB	0.54GB	0	-
KE	0.72G	0.72G	0	-
MEND	1.14GB	1.14GB	0	-
AlphaEdit	1.64GB	0.54GB	0	-
WISE	0.11GB	0.11GB	$0.27v+0.27u$	-
T-patcher	0.07GB	32.77KB	$0.02u$	$0.01wz$
GRACE	0.07GB	32.77KB	$0.02$	0
MELO	0.13GB	98.30KB	$0.02$	0
A$^3$E	0.07GB	39.94KB	$0.02+4.92u\times 10^{-5}$	$0.03z+1.28wz\times 10^{-4}+1.64w\times 10^{-5}$

We also provide the extra FLOPs of the proposed adaptive combination, adaptive regularization and adaptive merging for inference and training:

• Adaptive merging of A$^3$E: $1.64u\times 10^{-5}$ GFLOPs
• Adaptive combination of A$^3$E: $0.03z$ GFLOPs
• Adaptive regularization of A$^3$E: $1.28wz\times 10^{-4}+1.64w\times 10^{-5}$ GFLOPs

The extra FLOPs for training and inference are very small compared to the generation of $u$ tokens with $v$ input tokens with an 8B model, which is about $16u+16v$ GFLOPs.