WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs

We thank the reviewer for their detailed and thoughtful feedback. We’re pleased they found the paper well-written, the benchmark well-designed, and the findings insightful. We appreciate the recognition of WikiBigEdit as a valuable contribution for evaluating real-world limitations of knowledge editing, as well as the clarity of our benchmark construction. We’re also grateful for the positive assessment of our method coverage and the relevance of our experimental insights, particularly regarding the challenges current editing techniques face over long update horizons. We address the reviewer’s suggestions and questions below.

Advanced RAG Approaches

We thank the reviewer for the helpful suggestion of including more advanced RAG approaches and for highlighting HippoRAG [1]. While HippoRAG introduces a compelling long-term memory mechanism via structured memory graphs, it assumes access to full-text documents—unlike our setting, which focuses on factual QA pairs derived from structured triplets.

We agree that advanced retrieval-augmented approaches, including reasoning-augmented RAG or RAG agents, could enhance multi-hop reasoning. However, our goal is to evaluate the limits of lifelong knowledge editing in a controlled setting. We, therefore, use a simple, transparent RAG baseline to compare against editing and continual finetuning. Notably, even this baseline outperforms editing methods across most axes, though at higher inference cost.

RAG’s difficulty with multi-hop reasoning is orthogonal to the challenge of integrating new knowledge over time — the focus of our work. More advanced RAG variants may improve multi-hop QA but likely require even more test-time compute. We will add this discussion, along with a reference to HippoRAG, to Section 4.3 of the paper.

Shortcuts in Mhop Evaluation

We appreciate the reviewer’s suggestion to consider the DiRe condition [2]. While we do not explicitly evaluate it in its original form, we conduct a related analysis in the Supplementary Material (Figure 9, top row). Specifically, we compare model performance on individual first-hop and second-hop questions to its performance on the combined multi-hop question. This serves to test whether the model relies on shortcut reasoning, i.e., answering the multi-hop question without resolving its components, which is central to the DiRe condition.

Our results show that multi-hop accuracy depends heavily on correctly answering both hops. When a model fails on either, it rarely succeeds on the full question. However, a small fraction of questions (avg. 1.75% across models) are answered correctly despite failure on the first hop. These cases often involve surface-level cues, e.g.:

• “What is the language of work or name associated with the spouse of Johann Caspar Richter?” → German
• “In which country is the organization that Yamato Auto Works is a member of located?” → Japan

These align with shortcut-driven behavior as discussed in the DiRe framework. We will clarify this connection in Section 3.1 of the revision.

Reference of Concurrent Work TiC-Wiki

We thank the reviewer for pointing us to the concurrent work TiC-Wiki [3], which shares similarities in spirit. We will include a discussion of it in the related works section. While TiC-Wiki is a valuable benchmark for continual pretraining, it covers Wikipedia/Wikidata revisions from 2008–2022 — a period largely included in the pretraining data of modern LLMs. This makes it less suitable for evaluating updates beyond the knowledge cutoff, which is the central focus of our work. In contrast, WikiBigEdit is constructed from post-cutoff updates, allowing direct evaluation of newly emerging knowledge. We view the two efforts as complementary and appreciate the chance to clarify this distinction.

[1] Gutiérrez, B. J., et al. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models, 2024.

[2] Trivedi, Harsh, et al. Is multihop QA in DiRe condition? Measuring and Reducing Disconnected Reasoning. 2020.

[3] Li, Jeffrey, et al. TiC-LM: A Multi-Year Benchmark for Continual Pretraining of Language Models, 2024.

We thank the reviewer for the thoughtful and constructive review. We are pleased that the reviewer found the motivation and contributions of our work to be of practical importance — particularly our effort to investigate the limitations of existing knowledge editing techniques under realistic conditions. We appreciate the recognition of our automated pipeline for extracting real-world knowledge updates, as well as the comprehensive evaluation framework of WikiBigEdit, which tests factual retention, generalization, and reasoning across multiple dimensions. We are also grateful for the acknowledgment of our systematic comparison of knowledge editing methods with general update approaches such as RAG and continual finetuning. We address the reviewer’s suggestion for deeper analysis and provide further clarifications below.

In-Depth Analysis of Knowledge Editing Failure

While our study builds on the foundations of previous work, our experiments go beyond a simple dataset substitution. Specifically, we design a setting that reflects real-world, large-scale lifelong knowledge editing, a departure from the typically synthetic and small-scale settings used in prior studies (see also related works). Our benchmark introduces several important new dimensions: (1) the sequential nature of edits over time, (2) the use of real-world factual updates derived from Wikidata, and (3) a comprehensive evaluation protocol that includes not just edit accuracy but also generalization, locality, and multi-hop reasoning.

Moreover, our analysis compares knowledge editing methods not only to one another but also to broader model update strategies, such as retrieval-augmented generation (RAG) and continual finetuning with adapter merging. This allows us to contextualize the limitations of editing methods in a practical deployment landscape - a comparison absent in prior work.

We agree that a deeper investigation into the specific mechanisms of failure in knowledge editing is an important direction. While our primary focus is on empirically establishing limitations at scale, we highlight recent work that examines these mechanisms in detail (see Section 2). For example, Hsueh et al. [1], Gupta et al. [2], and Yang et al. [3] demonstrate that even small-scale editing can induce instability or forgetting in LLMs. Gupta et al. [4] further attribute performance collapse during sequential edits to the lack of regularization, which leads to overfitting on individual updates. These insights complement our findings and underscore the challenges in designing robust editing techniques that scale.

We will clarify this distinction and discussion in Section 2 of the revision.

[1] Hsueh et al., Editing the Mind of Giants, 2024.

[2] Gupta et al., Model Editing at Scale Leads to Gradual and Catastrophic Forgetting, 2024.

[3] Yang et al., The Butterfly Effect of Model Editing, 2024.

[4] Gupta et al., Lifelong Sequential Knowledge Editing without Model Degradation, 2025.

We thank the reviewer for their thoughtful and detailed evaluation. We appreciate the recognition of our key contributions: the construction of WikiBigEdit as the first large-scale, automatically extensible benchmark for real-world knowledge editing; the automated pipeline enabling continual updates; and the comprehensive evaluation across multiple LLMs and editing techniques. We are also grateful for the acknowledgment of our efforts to mitigate data leakage and pretraining overlap, as well as the clarity of our supplementary materials and code. We address the reviewer’s insightful questions below.

Length of Multi-hop Reasoning Chains

We appreciate the reviewer’s observation regarding the scope of multi-hop reasoning. While multi-hop can involve longer chains or subgraphs in general QA settings, in the context of knowledge editing, it is commonly defined as reasoning over two connected facts, forming a two-hop chain [1–3]. This is consistent with prior work, such as HotpotQA [4] and EX-FEVER [5], where two-hop questions are standard for evaluating a model’s ability to integrate and reason over multiple facts. Our benchmark adopts this convention to align with established protocols. That said, we agree that exploring longer reasoning chains is a valuable direction for future work.

Triplet Quality and Filtering

We thank the reviewer for their question on triple quality and filtering. To ensure a high-quality benchmark, our automated pipeline applies multiple filtering and QA steps (see Section 3.2). Below are average statistics from the first four timesteps:

• Initial Input: 100% of new or changed factual triplets from consecutive Wikidata snapshots.
• Initial Filtering: Remove cyclic, non-Roman, or overly long entities (~89% remain).
• Filter Unwanted Relations: Keep only WikibaseItem relations (entity-to-entity), excluding media, URLs, etc., and a small list of low-value relations (e.g., “use”, “is a list of”), retaining ~39%.
• Non-Deterministic Triplets: Remove (subject, relation) pairs with multiple objects (~24% remain). Final Filtering: Further filtered to remove spurious entries (final ~24% retained).

We also conducted manual inspections, sampling triplets from various batches to confirm that relations and entity pairings are meaningful.

These steps help ensure that only high-quality, interpretable factual triples are included in WikiBigEdit. We will extend Section 3.2 to report these statistics in the final version.

Impact of LLM Choice on QA Generation

We thank the reviewer for raising the important question of how the choice of LLM affects QA pair generation. In preliminary experiments, we evaluated the sensitivity of our generation pipeline to different GPT models.

For the Update and Locality sets, model choice had a negligible impact. These questions are directly derived from subject–relation–object triplets, and even GPT-3.5-turbo consistently produced accurate and well-formed outputs. For example, given the triplet (SS Heliopolis, manufacturer, Fairfield Shipbuilding and Engineering Company), GPT-3.5-turbo generated “Which company was the main manufacturer of SS Heliopolis?”, GPT-4o-mini produced “Who was the manufacturer of the SS Heliopolis?”, and GPT-4o generated “Which company was the manufacturer of the SS Heliopolis?” — all of which are stylistically comparable and semantically correct. The Rephrase set showed similarly consistent quality across models, as paraphrasing proved to be a relatively simple task for most LLMs.

In contrast, the Multi-hop and Persona sets were more sensitive to model choice, as these tasks require compositional reasoning or stylistic transformation. For instance, in the multi-hop case combining (2004 VS75, discoverer, Marc Buie) and (Marc Buie, gender, male), GPT-3.5-turbo generated the incomplete question “Who is the discoverer or inventor of (525257) 2004 VS75?”, failing to incorporate both facts. Meanwhile, GPT-4o-mini and GPT-4o correctly generated queries such as “What is the gender of the discoverer of 2004 VS75?” Based on these findings, we selected GPT-4o-mini for the Multi-hop and Persona sets, as it offers a strong balance between generation quality and efficiency. We observed no significant difference between GPT-4o and GPT-4o-mini in these settings. We hope this clarifies the robustness of our generation pipeline and our rationale for model selection. We will include an ablation on generation models in the Supplementary Material.

[1] Cohen et al., Evaluating the Ripple Effects of Knowledge Editing in Language Models, 2023.

[2] Zhong et al., MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions, 2023.

[3] Zhong et al., MQuAKE-Remastered: Multi-Hop Knowledge Editing Can Only Be Advanced with Reliable Evaluations, 2025.

[4] Yang et al., HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, 2018.

[5] Ma et al., EX-FEVER: A Dataset for Multihop Explainable Fact Verification, 2024.