HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models

We thank the reviewer for the time and effort spent reviewing our paper. We are glad they found our method interesting and enjoyed the Node Specificity portion of our methodology. We also appreciate their suggestions and long-term memory references, which we will definitely include in our updated literature review.

• W1: Extracting triplets from the passage strongly depends on the power of the triplets extracting model, which is a language model in the paper's implementation. I'm concerned that it may lose information when the passage becomes longer. Is this a common strategy in other retrieval methods?

We refer the reviewer to the general response for an intrinsic and extrinsic length-dependent evaluation of our knowledge graph construction.

We sincerely appreciate the reviewer for the time and effort they dedicated to reviewing our paper as well as for their comments and questions.

• W1: The mechanism is great, but the generalization of the method is not good enough. Intuitively, the mentioned mechanism should work for several scenarios that require long-term memory. Still, the proposed method narrows it down and limits the experiments to multi-hop reasoning.
• Q1: Can Hippo-RAG work for other tasks that require long-term memory?

Long-term memory in humans is a remarkably complex and powerful cognitive faculty that forms the basis for our reasoning and decision making. It’s a holy grail for artificial intelligence to acquire a memory mechanism as powerful as humans’, but that’s bound to be a long journey that we as a community have to take, one step at a time. Before HippoRAG, the de facto solution for long-term memory for LLMs, that expose a static LLM to new experiences, is RAG (see, e.g., this blog post that has had important impact in shaping this belief and practice: https://lilianweng.github.io/posts/2023-06-23-agent/). However, current RAG lacks many important properties of human long-term memory such as the inability to store previously learned procedures and store facts associated only with a particular time or place (episodic memory). HippoRAG focuses on another one of these important properties that current RAG lacks, knowledge integration across experiences, for which the hippocampus is believed to play an important role. For this property, multi-hop QA is the most natural task for evaluation because of its inherent knowledge integration requirement as well as the well established datasets and baselines.

Can HippoRAG be applied to other tasks that require long-term memory, e.g., agents that need to maintain episodic or procedural memory? Maybe, but more likely new adaptations and innovations will be needed to properly handle new properties of such memories, e.g., handling of their temporospatial attributes. We are very interested in further extending HippoRAG to get even closer to the powerful human long-term memory and handle more tasks that require long-term memory. On the other hand, we also believe that HippoRAG still marks a meaningful and solid step in the long journey of trying to empower AI with the powerful long-term memory mechanism with which humans are equipped.

• W2: The framework's performance depends on the NER ability of LLM. However, [1] demonstrates a large performance gap between prompting LLMs for NER and fine-tuned NER models. The error analysis also indicates that most errors in the system are from NER.

We agree with the reviewer that understanding the effect of different NER components on HippoRAG’s performance is important. In order to do this, we compare our standard query NER prompting method with the state-of-the-art NER model UniversalNER [1] below:

We can see that GPT-3.5-turbo outperforms UniversalNER-7B on MuSiQue. We believe this is due to UniversalNER’s preference for a few entity types given its training data. As reported in their paper, the top 1% most frequent entity types account for 74% of all entities produced. Additionally, even though UniversalNER is one of the most flexible supervised NER models available, it still requires defining entity types a priori, in contrast to our prompting methodology. Nevertheless, as the reviewer mentioned, we acknowledge that query NER issues are still HippoRAG’s largest source of errors. However, as we discuss in Appendix F.1 and F.2, these errors do not come from NER issues per se but rather the use of NER exclusively to link queries to our KG. For example, in a query such as “When was one internet browser’s version of Windows 8 made accessible?”, the terms that must be extracted are not named entities but common noun phrases like “internet browser”. We believe that finding suitable starting points for graph search is an important venue for future research.

• W3: The comparison of baselines may not be fair since the parameter sizes of Hippo-RAG are significantly larger than other methods. Contriever and ColBERTv2 achieve comparable results on MuSiQue and HotpotQA, but when combined with HippoRAG, the improvement is insignificant. ColBERTv2 is the second-best model, and the improvement from Hippo-RAG seems incremental on the two datasets.

In the response below, we present evidence and arguments that we hope will convince the reviewer that our performance improvements are substantial and our experimental setting is sound and fair.

Improvements on MuSiQue

We first highlight that our method “achieves significant performance improvement from various challenging multi-hop QA benchmarks”, as pointed out by the reviewer when listing our strengths. HippoRAG improves R@5 on MuSiQue by 5.5% and 2.7% over Contriever and ColBERTv2 respectively, as well as 3.4% F1 score on QA compared to ColBERTv2. Many well-received and highly cited works such as IRCoT [2], Self-Ask [3], ITER-RETGEN [4] and MCR [5] highlight improvements similar to ours.

HotpotQA’s Weaknesses as a Knowledge-Integration Benchmark

As discussed in subsection “Single-Step Retrieval Results” (Section 4) of our paper, our lower performance on HotpotQA is mainly due to its lower need for knowledge integration due to existing shortcut signals. This HotpotQA limitation is referenced when constructing MuSiQue [6], but also explored more deeply in Appendix B of our paper. In bridge multi-hop questions, the query and second supporting passage should be linked only through a bridge entity. However, as shown in Figure 6, HotpotQA queries are on average as similar to the second supporting passage as to their distractors, instead of less similar like in the other datasets, making that second supporting document easier to detect without knowledge integration.

Measuring the Impact of Improved Knowledge-Integration Directly

Additionally, when we measure the impact of improved knowledge integration in MuSiQue and 2WikiMultiHopQA, our improvements are even larger than our overall results. In Table 8, we show that HippoRAG’s ability to find all supporting documents is key to its strong performance, obtaining a 6.3% and 38.6% improvement over ColBERTv2 in All-Recall@5, compared to a 2.7% and 21.3% improvement in standard Recall@5.

Our Experimental Setting is Sound and Fair

Finally, we refer to the reviewer’s concern that our comparison with Contriever and ColBERTv2 might not be fair given their small size compared to LLM. We understand the reviewer’s apprehension, however, we argue that improving the performance of a system using an LLM and comparing it to the original system is a well-accepted paradigm in AI research. As a specific example, many well-received QA works such as IRCoT [2], Self-Ask [3] and RAPTOR [7] leverage a large LLM in different ways and compare with the original pipeline; which in these papers contain various retrieval methods like BM25, DPR [8] and ColBERTv2. We note that HippoRAG follows this same research paradigm, which is related to the large and growing body of recent work that leverages LLMs to challenge prior state of the art based on smaller models. As a final remark, we also note that our method outperforms RAPTOR and is comparable with IRCoT (and considerably more efficient). Since both of these methods augment the original retrieval process with outputs from an LLM, they can be seen as HippoRAG’s most direct baselines in that sense.

• Q2: Why does Hippo-RAG perform significantly better on 2Wiki?

As discussed in subsection “Single-Step Retrieval Results” (Section 4) of our paper, 2WikiMultiHopQA’s construction is more entity-centric than the other two datasets, making it particularly well-suited for HippoRAG’s design.

To provide some extra context, 2WikiMultiHopQA was created by leveraging the Wikidata KG to determine which entities and relations could be found in Wikipedia passages to create compositional, comparison, inference or bridge-comparison multi-hop questions. Due to this construction process, queries in this dataset include at least one named entity, a characteristic that our methodology can leverage given its previously mentioned reliance on NER to link queries to the KG.

References

[1] Zhou et al. (2024). UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition.
[2] Trivedi et al. (2023). Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions.
[3] Press et al. (2023). Measuring and Narrowing the Compositionality Gap in Language Models.
[4] Shao et al. (2023). Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy.
[5] Yoran et al. (2023). Answering Questions by Meta-Reasoning over Multiple Chains of Thought.
[6] Trivedi et al. (2022). ♫ MuSiQue: Multihop Questions via Single-hop Question Composition.
[7] Sarthi et al. (2024). RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval.
[8] Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering.

We thank the reviewer for the time and effort spent on reviewing our paper. We appreciate their helpful suggestions and the related work they brought to our attention.

• W1: Some baselines from KG-LLM for multi-hop QA literature are missing. These could be beneficial to replace Page Rank for ablation study or comparing with KGQA on open-source knowledge graph.

In order to address this important concern, we discuss previous works on multi-hop QA that combine graphs and neural methods by dividing them in two major categories 1) graph-augmented reading comprehension and 2) graph-augmented retrieval.

Graph-Augmented Reading Comprehension

In this direction, graphs are used to provide structure to complement textual signals from the retrieved passages and improve QA performance.

Most supervised methods in this category [1,2,3], train a graph neural network (GNN) to introduce hyperlinks or co-occurrence graphs into a language model for better QA. For works using LLM prompting, the graphs are usually constructed by extracting triples from retrieved passages and then added to the final LLM prompt [4,5,6]. The contributions from these works are mostly orthogonal to our own since graphs are not used in the retrieval process like they are in HippoRAG, however, they could be used in conjunction with our method to achieve complementary improvements.

Graph-Augmented Retrieval

In the second category, the graph is used to retrieve relevant documents rather than provide structured signals from a previously retrieved set. Many previous such works train a re-ranker to traverse a graph primarily made from hyperlinks [7,8,9,10,11,12]. As far as we know, HippoRAG is the first method that successfully combines a KG and an LLM for retrieval in multi-hop QA without any supervised data or predefined Wikipedia hyperlinks, meaning it can be used in more scenarios than previous methods.

We hope that this discussion helps better contextualize our method and clarify its uniqueness. As discussed above, although much KG-LLM work has been done for multi-hop QA, most of it is either not used in the retrieval process or relies on supervised data. However, as the reviewer mentioned, some KGQA methods are able to leverage an open KG for question answering using LLM prompting. For completeness, we carried out an ablation of our method that leverages our KG using Pangu (Gu et al 2023), a state-of-the-art LLM-based KBQA method, for QA.

Pangu obtains less than 0.05% F1 score on MuSiQue, demonstrating that standard KGQA systems cannot perform question answering on our OpenIE KG. We observe that the following challenges and many more lead to this dramatically low performance:

Entity Linking

This is a significant challenge even in settings with strictly defined KG, making it even more difficult on our KG where one entity can have several expressions. For example, for the question "When was the person who Messi's goals in Copa del Rey compared to get signed by Barcelona?", many entities are potentially relevant, such as Messi, the Spanish Messi, or Lionel Messi. This diversity makes it difficult to identify which starting point could lead to the answer.

OpenIE Coverage of Answers and Relations

Some relations as well as final QA answers are sometimes not present in the KG even if they are in the text. For instance, as we demonstrated in Table 14, this is the case for some complex expressions and numerical attributes.

We will expand our discussion of this literature accordingly in the revised version.

• Q1: Can we evaluate the performance of constructed knowledge graph alone?

We refer the reviewer to the general response for an intrinsic evaluation of our knowledge graph construction.

References

[1] Fang et al. (2020). Hierarchical Graph Network for Multi-hop Question Answering.
[2] Ramesh et al. (2023). Single Sequence Prediction over Reasoning Graphs for Multi-hop QA.
[3] Qiu et al. (2019). Dynamically Fused Graph Network for Multi-hop Reasoning.
[4] Park et al. (2023). Graph Elicitation for Guiding Multi-Step Reasoning in Large Language Models.
[5] Li et al. (2023). Leveraging structured information for explainable multi-hop question answering and reasoning.
[6] Liu et al. (2024). Era-cot: Improving chain-of-thought through entity relationship analysis.
[7] Ding et al. (2019). Cognitive Graph for Multi-Hop Reading Comprehension at Scale.
[8] Zhu et al. (2021). Adaptive Information Seeking for Open-Domain Question Answering.
[9] Nie et al. (2019). Revealing the Importance of Semantic Retrieval for Machine Reading at Scale.
[10] Das et al. (2019). Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering.
[11] Asai et al. (2020). Learning to retrieve reasoning paths over wikipedia graph for question answering.
[12] Li et al. (2021). Hopretriever: Retrieve hops over wikipedia 3625 to answer complex questions.
[13] Gu et al. (2023). Don’t Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments.