From RAG to Memory: Non-Parametric Continual Learning for Large Language Models

We are very thankful for the reviewer’s kind acknowledgment of our work as convincing, well-structured and broadly applicable across memory types. We will address the reviewer’s careful suggestions in the sections below.

Statistical Significance Testing

Why were no statistical tests reported? Adding them could confirm result reliability—would significant p-values change your confidence in ContinualRAG’s superiority?

We thank the reviewer for pointing out the need for statistical significance testing to ensure the reliability of our results. In order to ascertain the significance of ContinualRAG’s performance over NV-Embed-V2 (the best performing baseline), we run a simple bootstrapping statistical test. More specifically, we created 10,000 different datasets by sampling from each set of answers with replacement and thus obtained a distribution over the differences in their QA performance.

Through this method, we find that ContinualRAG’s performance is significantly larger than that of NV-Embed-V2 (p-value < 0.05) in 4 out of 7 datasets. Additionally, NV-Embed-V2 does not significantly outperform ContinualRAG in the other 3 datasets, demonstrating that ContinualRAG is robustly superior to our strongest baseline. We will include these significance testing results in the camera-ready version of our paper.

Novelty Concerns

Heavy reliance on HippoRAG reduces novelty; In my opinion the only major difference with the former is the passage integration part.

We appreciate the reviewer’s perspective, however, we would like to clarify our view on our work’s originality. While it is true that we build upon the existing HippoRAG framework, our approach systematically explores and enhances key components of the system in ways that are both principled and non-trivial. The space of possible design choices in structure-augmented RAG systems is vast, and identifying which modules to modify and how to do so effectively is itself a meaningful research contribution.

Our modifications are not superficial; rather, they are informed by clear hypotheses about the limitations of the original modules and are validated through rigorous ablation studies (Table 4), which demonstrate substantial and consistent improvements in performance. We believe that such targeted, evidence-based improvements can be as impactful as proposing entirely new frameworks, especially when they advance the capabilities of an already influential baseline.

Moreover, our work provides actionable insights into the design of RAG systems that others in the community can build upon—offering a path forward for both incremental and architectural innovations.

Reproduction Fidelity

Retrieval Performance (Table 3): Recall@5 is a standard metric, and reproducing HippoRAG with the same setup ensures fairness. However, the original HippoRAG paper reports higher R@5 (89.1 vs. 79.4 on 2Wiki), suggesting a potential issue in reproduction fidelity.

We would like to note that our reproduced HippoRAG results, reported on Table 3, are very close (or slightly better) than the ones reported in the original paper due to the use of a stronger embedding model and LLM for OpenIE. We changed both models in order to effectively compare with all other baselines.

More specifically, the R@5 scores reported in the original HippoRAG paper are 51.9, 89.1 and 77.7 respectively for MuSiQue, 2Wiki and HotpotQA. Our reproduced HippoRAG R@5 scores are 53.2, 90.4 and 77.3 for the same three datasets.

We ask the reviewer to let us know if they have any other questions concerning our reproduced HippoRAG results.

Filtering & Query-to-Node

Ablation Study (Table 4): Well-designed to isolate contributions (e.g., query-to-triple vs. NER-to-node), with clear recall@5 improvements. The omission of filtering in NER-to-node baselines is justified but could be explored further.

Although the reviewer makes an interesting point there are a few reasons why we omit a filter for NER-to-node:

• NER-to-node is already using an LLM to extract named entities before retrieving nodes.
• Though a post-extraction filter could be added to NER-to-node, it would require designing a different filtering methodology specifically for this module.
• Given that the performance of ContinualRAG w/o filter (avg. 86.4) is already much better than with NER-to-node (avg. 74.6), as shown in Table 4, we believe that designing such a filter was not the most worthwhile direction to explore.

We would like to thank the reviewer for thoroughly reviewing our work and noting that our modifications are well-justified and bring strong improvements over all our baselines.

ContinualRAG is a Continual Learning Method

The paper is named “ContinualRAG”, however, the method does not seem to have any “continual” aspect.

As we argued in our work, RAG has become the de facto solution for continual learning in LLMs due to its simplicity and strong performance when compared to parametric alternatives. Our methodology builds on these already robust continual learning systems, enabling them to handle tasks that require more in-depth learning of new knowledge, such as associativity and sense-making. Given its strong performance on these demanding tasks, we argue that ContinualRAG system not only classifies as a continual learning system but also elevates the standard of what it means to continually learn.

To further show this, we provide an experiment that directly evaluates our method’s performance as more knowledge is aggregated. We refer the reviewer to the Continual Knowledge Injection Experiments section in our response to reviewer gi53 above for experimental details, results and discussion.

For a more detailed explanation of motivation being our experiments, we refer the reviewer to the Non-Parametric Continual Learning for LLMs: Ideal Experiments section of our response to reviewer K3hz.

Why are 7B embedding models “large”?

The method utilizes Llama-3.3-70B Instruct for extraction and triple filtering. However, the tables refer to models that utilize 7B parameter LLMs as large embedding models. The relation to such methods and why they are referred to as large embedding models while the method utilizes 70B models albeit at a limited capacity is not clear.

We refer to 7B embedding models as ''large'' because they are much larger than classic ones like GTR (335M) and Contriever (110M). Although smaller than LLMs, they are the largest and strongest models available on the MTEB benchmark.

How does LLM size impact ContinualRAG?

It is also not clear what the impact of the model size is in the proposed method. Table 9 in the appendix provides some ablations with GPT-4o-mini but does not ablate on different model sizes of the Llama-3.3 family.

We report results using Llama-3.1-8B for ContinualRAG (Llama 3.3 family only has 70B models). It shows the 8B model is not sufficiently capable of supporting our system on both types of tasks (MuSiQue & NQ).

Clarifications

The paper is sparse in details about prior works such as HippoRAG. In particular, understanding section 3.1 is crucial for the understanding of the contributions, however, definitions and details on the following terms are missing: OpenIE, Personalized PageRank, PHR, reset probability, etc.

OpenIE: OpenIE extracts entity–relation–entity triples from text without predefined relation types, in contrast to standard IE.

Personalized PageRank (PPR): PPR is a variation of PageRank that measures the importance of nodes in a graph relative to a source node (or source nodes).

Reset Probabilities: This vector quantifies the importance of source nodes for PPR.

Parahippocampal regions (PHR): This terminology is borrowed from the HippoRAG paper, which is an analogy between their retrieval encoder and the parahippocampal regions of the human brain.

The paper is missing descriptions for baselines in section 4.1. At least a description of groups of methods and a comparison is required.

We include several RAG baselines like BM25, popular retrievers (Contriever, GTR) and 3 SoTA models from MTEB. RAPTOR, GraphRAG, and LightRAG use summarization to enable sense-making capacity. HippoRAG performs well on associative tasks.

A clear description of the evaluations and execution difference between QA and retrieval setups is missing.

Our QA module uses the top-5 retrieved passages as context for an LLM (GPT-4o-mini or Llama-3.3-70B-Instruct) to generate the final answer. The QA result is evaluated by token-based EM/F1 scores, aligning with MuSiQue/2Wiki/HotpotQA official metrics.

Continual Pretraining Related Work

If the paper is position itself as continual method as the title may suggest, it should consider expanding the related works section to discuss more continual pretraining methods.

• Roth, Karsten, et al. "A Practitioner's Guide to Continual Multimodal Pretraining." arXiv preprint arXiv:2408.14471 (2024).
• Li, Jeffrey, et al. "Tic-lm: A multi-year benchmark for continual pretraining of language models."

We thank the reviewer for highlighting these important works, we will incorporate them into Section 2.1 of our camera-ready version.

We sincerely appreciate the reviewer’s detailed comments and questions, they will surely enhance the quality of our work. We are happy to know that the reviewer found our human memory inspired methodology well-motivated and our experimental settings meaningful, valid and sound.

Non-Parametric Continual Learning for LLMs: Ideal Experiments

If this method is designed for continual learning, the experimental setup should be revised to incorporate a temporal dimension.

We appreciate the reviewer’s suggestion to add a temporal dimension to our experiments; however, we believe that our current setup appropriately evaluates the continual learning abilities of non-parametric methods given their unique strengths and limitations.

As explained in our paper, non-parametric methods have become the de facto continual learning solution for LLM due to their simplicity and strong performance. In standard continual learning benchmarks, which measure catastrophic forgetting and simple factual learning, standard RAG outperforms parametric alternatives like model editing and continual pretraining by substantial margins (MQuAKE, EvolvingQA).

In contrast to their strong performance in these simpler settings, non-parametric methods struggle with richer forms of continual learning. More specifically, given that standard RAG acquires isolated knowledge, it is limited in its capacity to enable complex tasks over new knowledge, such as associativity and sense-making. Our experimental setup, which consists of a set of such tasks, is thus designed to explore this limitation in non-parametric continual learning methods.

That said, we agree that assessing performance as more knowledge is incrementally introduced would be a valuable addition to our paper. We refer the reviewer to the Continual Knowledge Injection Experiments section in our response to reviewer gi53 above for experimental details, results and discussion.

We will add this discussion and experiment to the camera-ready version.

ContinualRAG’s Computational Overhead

How much additional time and computational resources does this approach require compared to the existing models used for comparison?

We appreciate the reviewer’s question concerning ContinualRAG’s efficiency compared to our baselines. To address this, we report the time and memory resources required for offline indexing and online retrieval, we will add them to Appendix F alongside the token-level costs reported in Table 12. For indexing, we indexed 11k documents using a Llama-3.3-70B model using vLLM on 4 H100s. For the memory requirements, we ignore all memory for model weights since it is shared across all systems.

As we can see, in terms of time, ContinualRAG is much more efficient than GraphRAG and LightRAG and only slightly less efficient than both RAPTOR and HippoRAG. For memory usage, ContinualRAG’s use of fact embeddings does increase its requirements, however, we believe this is acceptable given its performance benefits. Additionally, while all approaches lag behind standard RAG in terms of time and memory efficiency, ContinualRAG is the only one that outperforms this strong baseline substantially.

Novelty Concerns

The originality of this work is somewhat limited, as it primarily modifies existing modules in HippoRAG rather than introducing an entirely new framework.

We appreciate the reviewer’s perspective, however, we would like to clarify our view on our work’s originality. While it is true that we build upon the existing HippoRAG framework, our approach systematically explores and enhances key components of the system in ways that are both principled and non-trivial. The space of possible design choices in structure-augmented RAG systems is vast, and identifying which modules to modify and how to do so effectively is itself a meaningful research contribution.

Our modifications are not superficial; rather, they are informed by clear hypotheses about the limitations of the original modules and are validated through rigorous ablation studies (Table 4), which demonstrate substantial and consistent improvements in performance. We believe that such targeted, evidence-based improvements can be as impactful as proposing entirely new frameworks, especially when they advance the capabilities of an already influential baseline.

We sincerely appreciate the reviewer for the time and effort dedicated to reviewing our work. We are delighted that they found our work clear, convincing, well-motivated, technically solid and empirically validated by impressive performance improvements. We address their suggestions and comments below.

Continual Knowledge Injection Experiments

Given that ContinualRAG aims to empower LLMs to continuously acquire, organize, and leverage knowledge, additional experiments could be explored to continually update and expand the knowledge graph, which will make this work more solid.

We appreciate the reviewer’s thoughtful suggestion and agree with their assessment.

To address this, we conduct a new experiment on both NQ and MuSiQue. We partition our full corpus into four equal segments (approximately 250 questions and their distractors). We then select one segment for evaluation and incrementally add the remaining segments, measuring how performance evolves as new knowledge is added—simulating a temporal continual learning setting. We report the performance of ContinualRAG and NV-Embed-V2, our strongest baseline, in the tables below using F1 scores.

NQ

MuSiQue

As we can see, ContinualRAG’s improvements over NV-Embed-V2 remain remarkably consistent in both simple and associative continual learning settings. We note that, while both methods show steady performance on simple QA as more knowledge is introduced, their performance drops almost equivalently in the more complex task as more information is introduced. This behavior shows the strength of RAG in simple temporal tasks, highlights the value of our current experimental setup and points to the need for more complex temporal continual learning benchmarks for LLMs.

We will add this experiment to the camera-ready version of our paper.