Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing

Thank you for your positive feedback on our paper and for your valuable suggestions. Please find our detailed responses to the weaknesses and questions below.

W1：The method may have certain limitations. Specifically, using WikiData and SPARQL queries to construct the supplementary dataset may not be applicable to non-factual or non-encyclopedic data, limiting the generalizability of the approach.

In fact, we introduced a general method for constructing the supplementary set using the model itself and reported its performance in Table 9(w $Sup_{model}$). See Appendix E.2 for details. Given that we employed the simplest version of the knowledge retrieval prompt without any post-processing, we consider this to represent the lower bound of performance for this alternative supplementary set construction method. While its performance is slightly lower compared to the version implemented with WikiData and SPARQL, it still exceeds the performance of the original one-stage PMET by over 50%. Therefore, we believe that the method of constructing the supplementary set is highly scalable and generalizable.

W2：The organization of the paper could be improved. The experimental section is relatively short, with some important experimental tables relegated to the appendix. Additionally, the paper lacks a comprehensive ablation study.

We have added a comprehensive ablation study and generalizability analysis in our new section Appendix E: ABLATION STUDY AND GENERALIZABILITY OF IFMET. These results demonstrate the positive contributions of the supplementary set and the interpretability-guided targeting of later MLP layers and the last token position. And we discuss the generalizability of our method in many perspectives. Please refer to the corresponding section in the updated PDF for further details.

Regarding the organization of the paper, we fully understand the difficulties you encountered when important results were placed in the appendix. We spent considerable time attempting to incorporate more tables into the main text but were ultimately unsuccessful. At the same time, we believe that the analytical section preceding the experiments is necessary. Therefore, we kindly ask for your understanding that due to space constraints, we had to place some tables, and even the entire ablation study, in the appendix. Our aim was to preserve the completeness of the experimental results as much as possible. All changes are highlighted in blue in the revised PDF file. We hope these revisions address your concerns comprehensively.

W3: The explanation and analysis of the experimental results are somewhat lacking. For instance, in Table 3, it is unclear what the values represent (e.g., accuracy) and why the performance of IFMET improves as the edit batch size increases, while the performance of other baselines decreases. More detailed explanations and discussions of these observations would strengthen the paper.

We reported the model's performence of Multihop question answering accuracy. To make our paper more comprehensive and clearer, in the revised version we newly provided detailed explanations of the reported outcomes in tables of Section 5.1 Setup and Hyperparameters and Experimental Setup, ensuring consistent definitions for "unedited answer" and "edited answer". Additionally, we have included annotations below each table to specify the exact names of the reported results, along with the models and datasets used. We have also provided detailed explanations of the metrics, such as neighbour score and paraphrase score, mentioned in the ablation experiments within the text. As shown in Table 3, the editing performance of PMET remains relatively stable across different batch sizes, which we attribute to the inherent superiority of the PMET method[1]. Notably, IFMET, which leverages PMET as the primary method for its first-stage and furtherance editing, inherits this advantage to some extent, demonstrating minimal performance degradation when the batch size changes. As shown in Table 11, IFMET demonstrates strong multi-hop reasoning accuracy when editing four pieces of knowledge within a single reasoning chain. We attribute this to the effectiveness of the two-stage editing process. By correctly updating the knowledge associated with the implicit reasoning steps in the deep-layer MLPs, IFMET achieves a unique capability compared to previous methods. Unlike previous approaches, where performance often deteriorates as the number of edits in a reasoning chain increases, IFMET exhibits improved performance as the number of edits grows within a single reasoning chain.

[1] PMET: Precise Model Editing in a Transformer. AAAI 2024

We are eager to know if you have any additional concerns or suggestions regarding our paper. If there are none, we sincerely hope you might consider raising the review score, and we would be more than willing to engage in detailed technical communication to address any potential concerns.

Thank you for the author's response. The reply has addressed most of my concerns, and therefore, I will maintain my positive score.

As we approach the rebuttal discussion deadline, we would like to follow up with you regarding the additional ablation experiments, generalization analysis, and writing updates we provided. Have these addressed your concerns? If so, we kindly ask if you would consider revising your review score accordingly. Thank you for your time and thoughtful evaluation of our work.

Thank you for your positive feedback on our paper and for your valuable suggestions. Please find our detailed responses to the weaknesses and questions below.

Q1:The biblology style should be consistent. Some lines are underlined and others are not.

We have standardized the formatting of the references in new version, removing the underlining annotations.

Q2&W1: How does IFMET perform on other language models? Like recent Llama-3.

We first extended the causal intervention experiments to the LLaMA-2-7B model. The result demonstrate the consistency of interpretability analysis across models, demonstrating the critical role of deeper-layer MLPs in LLaMA-2 model for multi-hop fact recall tasks. Additionally, we repeated the ablation experiments on LLaMA-2-7B to evaluate the generalizability of IFMET. The results shown in table 9 (in new PDF) are consistent with those observed on GPT-J, highlighting the superiority of the method IFMET on LLaMA-2-7B model. Considering both the interpretability analysis and experimental outcomes, we conclude that our analysis and method are equally applicable to newer models, such as LLaMA-2. You can refer to this section Generalization to other models in Appendix E.2: GENERALIZABILITY OF IFMET in new PDF for more detailed ablation experiment results.

W2: How about the performance of the generality and locality of knowledge editing? The editing should also affect related facts but not affect other unrelated facts.

To explore this, We constructed the paraphrase set and neighborhood set for a subset of the MQuAKE-CF dataset, following the approach used in the COUNTERFACT dataset. The results indicate that, Compared to the one-stage PMET, IFMET achieves a significant improvement of over 60% in Multi-hop accuracy and also demonstrates enhancements in both efficacy score and Paraphrase score, at the cost of a minor decrease in the neighborhood score. So that IFMET method achieves balanced optimal performance across multiple metrics. For more details on related ablation studies, please refer to the section Appendix: E ABLATION STUDY AND GENERALIZABILITY OF IFMET.

W3 & Q3: The paper lacks discussions about editing efficiency. What is the time complexity of IFMET?

We compared the time complexity of IFMET with that of the one-stage PMET method it builds upon. On average,the time required to perform a complete edit for a single case on GPT-J using IFMET(with supplementary set) was approximately 2.5x that of PMET. For LLaMA-2, the time required was about 1.5x that of PMET. We believe this is within an acceptable range, and as the editing speed of the single-stage method improves, the IFMET framework will correspondingly become faster. See Appendix E.2 for details.

W4: Compared to Mello, IFMET requires supplementary set construction from external sources (same for MEMIT and ROME). This may hinder its applications in real cases.

In fact, we introduced a general method for constructing the supplementary set using the model itself and reported its performance in Table 9(w $Sup_{model}$). See Appendix E.2 for details. Given that we employed the simplest version of the knowledge retrieval prompt without any post-processing, we consider this to represent the lower bound of performance for this alternative supplementary set construction method. While its performance is slightly lower compared to the version implemented with WikiData and SPARQL, it still exceeds the performance of the original one-stage PMET by over 50%.

Q5 & Q6: Line 387, cizizenship --> citizenship. Figure 1, Mardrid --> Madrid.

We have fixed the typo.

Q4: Can IFMET deal with unstructured facts for knowledge editing?

The application of IFMET to unstructured knowledge editing presents a compelling avenue for further research. Unstructured text inherently encompasses more complex semantic information, and in scenarios involving longer and more intricate texts, the presence of noise can significantly complicate the reasoning process. At present, combining IFMET with efficient fact triple extraction techniques appears to be a feasible approach. We believe this context warrants deeper investigation in future work.

All new changes are highlighted in blue in the revised PDF file and we have included an entire section in the appendix dedicated to ablation experiments and discussions on generalizability. We hope these revisions address your concerns comprehensively.

We are eager to know if you have any additional concerns or suggestions regarding our paper. If there are none, we sincerely hope you might consider raising the review score, and we would be more than willing to engage in detailed technical communication to address any potential concerns.

As we approach the rebuttal discussion deadline, we would like to follow up with you regarding the additional ablation experiments, generalization analysis, and writing updates we provided. Have these addressed your concerns? If so, we kindly ask if you would consider revising your review score accordingly.

Thank you for your time and thoughtful evaluation of our work.

Thank you for your positive feedback on our paper and for your valuable suggestions. Please find our detailed responses to the weaknesses and questions below.

W1: Why do we need to conduct "Locate-then-edit" editing when we have some very good performance RAG-based editing methods? There is no need for this paper to compare performance with RAG-based editing in the experiments part, but a short discussion should be included on why IFMET is superior to these RAG-based methods.

This is a very interesting question, and to address it, we have added a dedicated section Comparison with Weight-Preserving Methods in the new section Appendix E. Please review the updated PDF to verify these changes.

W2：Currently, only experiments are conducted on MQUAKE-3k; more benchmarks, such as MQUAKE-T, could be included.

Due to the computational constraint, we have not tested IFMET on additional benchmarks. However, given the generalizability of PMET, we believe that IFMET will exhibit consistent performance across different settings. Additionally, we have added extensive ablation studies and generalizability experiments in the new section Appendix E: ABLATION STUDY AND GENERALIZABILITY OF IFMET, aiming to further validate the effectiveness of our method.

Thank you for your rebuttal. I have no further questions.

Thank you for your positive feedback on our paper and for your valuable suggestions. Please find our detailed responses to the weaknesses and questions below. Questions

Q1: Figures: Consider widening Figures 2(a) and 2(b) to improve readability and enlarge the subfigures in Figure 6 for better visibility.

The Figure 6 in previous version changes to Figure 7 in new new version. We have enlarged the subfigures in Figure 7(Figure 6 in previous version) to improve visibility. Due to space constraints, Figures 2(a) and 2(b) were not widened in the main text. However, larger versions are provided in the appendix, and a note has been added below the figures to inform readers.

Q2: Tables: Provide a clearer explanation of the term “base” in Tables 5 and 6. It’s unclear whether this refers to the “original” in Table 3 or another baseline. Clear labels and descriptions would enhance understanding.

We use the term Base refers to the original model without any edits. And We revised the terminology in Tables and Figures, to consistently refer to the unmodified model as "Base". Additionally, we included a unified explanation of this term in the Baseline section of 5.1 Experimental Setup.

Weakness

W1: Insufficient Presentation: Section 5.2 lacks clarity as Tables 5 and 6 are referenced but not included in the main paper, instead being located in the appendix. While space constraints are understandable, omitting key results from the main discussion detracts from readability and coherence. Additionally, the absence of descriptions for the "base" in Tables 5 and 6 and missing details on the datasets used in Table 4 (e.g., whether MQuAKE is consistently used) leaves gaps in understanding.

We use the term Base refers to the original model without any edits. And We revised the terminology in Tables and Figures, to consistently refer to the unmodified model as "Base" which is explained in the Baseline section of 5.1 Experimental Setup. And add the dataset information in each table. Regarding the organization of the paper, we fully understand the difficulties you encountered when important results were placed in the appendix. We spent considerable time attempting to incorporate more tables into the main text but were ultimately unsuccessful. At the same time, we believe that the analytical section preceding the experiments is necessary. Therefore, we kindly ask for your understanding that due to space constraints, we had to place some tables, and even the entire ablation study, in the appendix. Our aim was to preserve the completeness of the experimental results as much as possible. All changes are highlighted in blue in the revised PDF file. We hope these revisions address your concerns comprehensively.

W2: Questionable Analysis of Results: In Table 3, performance seems to decline post-editing compared to the original, but there is no discussion on the possible causes for this drop. If editing knowledge reduces performance, the rationale for the edits becomes unclear. It appears that “original” performance refers to accuracy on the unedited answer, but if so, why wasn’t accuracy on the correct answer also reported for the original model? Clearer explanations are necessary to clarify these performance metrics.

Yor are right. We report the Base's performence on unedited answer. We believe this represents the upper limit of the model's inherent ability to utilize knowledge for answering multi-hop questions. We provided detailed explanations of the reported outcomes in tables in the Setup and Hyperparameters section of 5.1 Experimental Setup, ensuring consistent definitions for "unedited answer" and "edited answer". Notes have also been added below each table to clarify the meaning of these values and report Base's performence on edited answer too. We hope this explaination address your concerns about Questionable Analysis of Results in Table 3 and others.

W3: Limited Novelty in Method and Analysis: The primary contribution of this work lies in the insight, obtained through interpretability tools, that multi-hop knowledge editing requires updating the later MLP layers. However, there is no clear experimental analysis showing that editing the later layers specifically contributes to performance improvement. It raises the question of whether merely expanding the support set was sufficient for better performance, implying that gains may not be attributed solely to the proposed method. To substantiate this claim, it would be useful to report the impact of the editing layers on performance, comparing different layers with and without the supplementary set.

To prove that editing the deeper layers specifically contributes to performance improvement. We have added a comprehensive ablation study and generalizability analysis in the Appendix section E ABLATION STUDY AND GENERALIZABILITY OF IFMET. These results demonstrate the positive contributions of the supplementary set and the interpretability-guided targeting of later MLP layers and the last token position. Importantly, they confirm that the observed performance improvements are not solely attributable to the supplementary set. You can refer to this section in new PDF for more detailed ablation experiment results.

All changes are highlighted in blue in the revised PDF file. We hope these revisions address your concerns comprehensively.

We are eager to know if you have any additional concerns or suggestions regarding our paper. If there are none, we sincerely hope you might consider raising the review score, and we would be more than willing to engage in detailed technical communication to address any potential concerns.

As we approach the rebuttal discussion deadline, we would like to follow up with you regarding the additional ablation experiments, generalization analysis, and writing updates we provided. Have these addressed your concerns? If so, we kindly ask if you would consider revising your review score accordingly.

Thank you for your time and thoughtful evaluation of our work.