SiReRAG: Indexing Similar and Related Information for Multihop Reasoning

“Unfortunately, I found the relatedness-based indexing algorithm is ad hoc.”

As indicated above, we adhere to a rigorous thought process. In order to demonstrate a principled design of our method, we have our ablation study in Section 6.2. Here, we also add an additional "entity clustering” experiment to showcase the effectiveness of entity-specific propositions on multihop reasoning tasks.

We perform clustering directly on text chunks by using entities (entity clustering). Specifically, we do not maintain a separate relatedness tree and add an additional clustering philosophy to the similarity tree. Each text chunk in the similarity tree is simplified to “This chunk mentions entity 1 and entity 2” if both entities are extracted by our LLM. We then run GMMs (the same clustering method as RAPTOR) on these simplified chunks. Once the clustering decisions are obtained, we group the original chunks as additional clusters and append these clusters to the similarity tree, allowing higher levels of the tree to incorporate both clustering philosophies. Since entities primarily determine the outcome of this additional clustering approach, we apply entity clustering to model relatedness on the similarity tree. This allows us to eliminate proposition aggregates in order to examine their utility. Performance is shown below.

Although this dual clustering method also models both similarity and relatedness, it exhibits a notable decline in comparison to SiReRAG. This finding empirically demonstrates the necessity of proposition aggregates of modeling relatedness. Because proposition aggregates reduce noise and information redundancy more effectively than text chunks as described in Section 4.2, they serve as an effective carrier of related dataset contents. We will add this table in our revised draft that will be submitted later.

“Firstly, the relatedness are modeled using entity-specific propositions, which I think is over-specialized to multi-hop QA tasks(where the hop is just entity-entity associations and this is why the proposed method can achieve performance improvements). However, I think entity-specific propositions may not a good decision for many other complex reasoning tasks, and the authors should explain and verify the effectiveness and generality in more tasks.”

First, Entity-Propositions can cover diverse type of entity-specific questions: e.g., factual or single-hop questions, multi-hop reasoning, information aggregation and summarization (e.g., which clubs in NYC offer boxing studio? summarize all important contributions of WHO, etc.). Datasets we used in our evaluation already include these different types of reasoning: e.g., comparison question, compositional question, inference question, temporal reasoning question, etc. For instance, MuSiQue [3] includes 5 reasoning structures [Table 1].

Next, we want to highlight that entity-specific RAG approaches cover a significant proportion of RAG use cases, and is a growing research area, as shown by recent works like HippoRAG [1] and GraphRAG [2]. These methods exclusively explore entity graph-based RAG approaches, underscoring their growing relevance and utility in answering complex questions. Also, relatedness tree (based on entity propositions) is complementary to chunk-based RAPTOR. So, it should assist on complex questions requiring evidences/ propositions about entities, without impacting other non-entity specific questions.

Furthermore, beyond our evaluation on three multi-hop QA datasets in paper, SiReRAG also outperforms RAPTOR on MultiHopRAG, as discussed in our response to Question 1 below. This further highlights that SiReRAG is generalizable to any complex multi-hop reasoning tasks.

Lastly, propositions aggregation according to entity can be expanded to any other discourse units, e.g. Topic which could make an interesting future exploration.

[1] HippoRAG https://arxiv.org/abs/2405.14831

[2] GraphRAG https://arxiv.org/pdf/2404.16130

[3] MuSiQue https://arxiv.org/pdf/2108.00573

“Secondly, I found there are many heuristic decisions, such as how to filter proportions, how to resolve entity references, etc.”

With respect to heuristic decisions, it is necessary to incorporate them in order to maximize our performance. For example, we notice that an LLM could effectively resolve entity references by rewriting documents. Otherwise, without rewriting documents, related propositions may not be grouped together due to inconsistent names of some entities, which weakens our method. For example, in the MuSiQue corpus, there is a sentence about the “Head I” painting: “It is the first of Bacon's paintings to feature gold background railings or bars, which later became a prominent feature of his 1950s work, especially in the papal portraits where they often appeared as enclosures or cages around the figures.” Without rewriting “Bacon” to “Francis Bacon” prior to indexing, it would be difficult to identify who 'Bacon' refers to, since the example in Figure 1 includes two “Bacon”s: Francis Bacon and Nicholas Bacon. As a result, performance of generating the correct answer (Nicholas Bacon) would be lower than without rewriting documents.

Weakness 2: Thanks for your suggestion. We discussed the motivation of separating similarity and relatedness trees and an alternative design of combining them in Section 4.3. Specifically, in order to concatenate similarity and relatedness trees, this alternative design combined nodes from both sides in the same pool for finding additional clusters and performing summarization at every tree level (line 317). In other words, we find additional clusters by concatenating the nodes of both trees, which considers cross-tree interaction instead of keeping them separate. The performance of doing so is shown below.

The performance of considering cross-tree interaction is slightly lower than SiReRAG. Therefore, it is more efficient to keep them separate in order to reduce the overall complexity of the system as discussed in Section 4.3. Since we have shown the benefit of indexing both similar and related information, we do not incorporate cross-tree interaction due to effectiveness and efficiency. We will add this table in our revised draft that will be submitted later.

Weakness 3: Although there are many existing methods that work on multihop reasoning tasks, SiReRAG is about indexing corpus data under RAG setup. In other words, instead of being our baselines, other existing works focus on other dimensions of improving performance on complex reasoning tasks. To the best of our knowledge, we have selected a wide variety of RAG indexing baselines to demonstrate our claims.

However, as suggested by Reviewer ymw3, we have run an additional set of experiments on an iterative retrieval method (Self-Ask) specifically designed for multihop reasoning. In addition to Table 6, these experiments show how SiReRAG can complement existing methods for optimal performance.

Self-Ask uses novel inference questions to decompose a multihop query. By running Self-Ask on the retrieval pool of SiReRAG, we obtain performance improvement on all three datasets with an average improvement of approximately 3.6% in F1 score.

Weakness 4: We will be more than happy to clarify any points! We are also revising our draft for better clarity and will submit the pdf shortly.

We thank the reviewer for the helpful feedback! We plan to submit another draft that incorporates all reviewers’ comments before the discussion phase ends. Focusing on multihop reasoning, we are, to the best of our knowledge, the first to verify the importance of incorporating both similarity and relatedness signals and to implement this concept in RAG indexing.

Weakness 1:

“Because the similarity-based indexing component just follows RAPTOR (Sarthi et al., 2024), I think the main contribution of this paper maybe the relatedness-based indexing component.”

To the best of our knowledge, our work is the first indexing method of modeling both similarity and relatedness in RAG setup for a more comprehensive knowledge integration. Our baselines (RAPTOR, HippoRAG, and GraphRAG) consider only a single perspective. For example, HippoRAG and GraphRAG are relatedness-based, constructing noun- or entity-guided knowledge graphs. In contrast, we find that modeling both perspectives enables a more effective knowledge integration process, as detailed in Section 3 (adopting a clustering strategy based on a single perspective covers a remarkably low percentage of supporting passages). Our quantitative results in Section 6 echoes this motivation.

Although we are motivated, it also requires nontrivial efforts to realize our high-level idea as a RAG indexing method. Firstly, we decide to focus on multihop reasoning as our scope, as knowledge integration or synthesis is vital to this kind of task as discussed in Figure 1. Our method does not show a significant improvement on other kinds of datasets that involve less reasoning such as QASPER. Then, we need to finalize an effective and efficient structure (e.g., tree/knowledge graph) to index corpus data. RAPTOR tree is a straightforward idea, but it deviates from the commonsense understanding of a tree structure [1]. As noted in line 207, while multiple levels of data abstraction can exist within a document, RAPTOR places all text chunks at the bottom level. This approach conflicts with many studies on document discourse trees [2]. Then, we explore the hierarchical structure of an indexing tree. However, we do not see a clear trend of improvement by doing so (Table 2). So, we use RAPTOR as the generalized tree structure for similar and related information. We believe our exploration and observation are valuable for future work.

[1] Zhang, J., Silvescu, A., & Honavar, V. (2002). Ontology-driven induction of decision trees at multiple levels of abstraction. In Abstraction, Reformulation, and Approximation: 5th International Symposium, SARA 2002 Kananaskis, Alberta, Canada August 2–4, 2002 Proceedings 5 (pp. 316-323). Springer Berlin Heidelberg.

[2] Maekawa, A., Hirao, T., Kamigaito, H., & Okumura, M. (2024, March). Can we obtain significant success in RST discourse parsing by using Large Language Models?. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2803-2815).

Question 1: Same as HippoRAG, we reported performance on MuSiQue, 2Wiki, and HotpotQA in our original submission. Following your suggestion, we are adding MultiHopRAG as another multihop reasoning task. We show the performance of SiReRAG and RAPTOR on this newer dataset. We filter all unanswerable questions and randomly select 350 “comparison” queries and 350 “inference” queries, which forms a pool of 700 queries in total.

We notice that MultiHopRAG is a relatively easier dataset than our selected ones in our paper, because both methods provide the correct answers in most cases. Notably, SiReRAG still delivers better performance, which echoes our main experiments. Our comprehensive selection of datasets demonstrates the generality of SiReRAG on multihop reasoning tasks.

Question 2: Thanks a lot for your suggestion! We show the performance of entity clustering in our response to weakness 1, which demonstrates that an entity-based relatedness tree would offer suboptimal performance.

We hope our response has addressed all your concerns and kindly hope you can consider increasing your scores.

We thank the reviewer for the helpful feedback! We plan to submit another draft that incorporates all reviewers’ comments before the discussion phase ends.

Weakness 1: Thanks for your suggestion! We notice that the Open-RAG paper was public on arxiv after we made this ICLR submission. We will cite it in our revised version. As shown in Table 6 and the scores below (our responses to reviewer F1mv and ymw3), SiReRAG successfully improves the performance of other non-indexing methods on multihop reasoning. Therefore, we think SiReRAG has the potential of complementing Open-RAG, because the two methods are not in conflict.

Weakness 2: Flattened indexing is a simple method that places all tree nodes into a unified retrieval pool. In other words, regardless of a node's origin (e.g., bottom or upper levels, similarity or relatedness trees), it is added to a single list containing all nodes. We will clarify this concept in Section 4.3 of our revised version.

Weakness 3: We provide our prompts as below. We will add these prompts in our appendix.

Summary prompt (same as RAPTOR): Summarize the provided text, including as many key details as needed.

Topic extraction prompt (Section 3): Identify the high-level topic of this paragraph as concise as possible.

Hierarchy prompt (Section 4.1):

## Instructions
1. Group sentences based on their topic and abstractiveness.
2. For each sentence group define metadata consisting of topic and the abstractiveness level.
3. There are two abstractiveness levels:
    a. low: sentences describe fine-grained details about the topic
    b. high: sentences gives an overview of the topic, summarizing fine-grained details.
4. Ensure that each group consists of contiguous sentences.
5. Respond using the below json format.
{
  "group_1":
  {
    "sentences": [list of sentence ids],
    "topic": "topic of the group",
    "abstractiveness": "high"
  },
  "group_2":
  {
    "sentences": [list of sentence ids],
    "topic": "topic of the group",
    "abstractiveness": "low"
  },
  "group_3":
  {
    "sentences": [list of sentence ids],
    "topic": "topic of the group",
    "abstractiveness": "high"
  }
}

Weakness 4 and Question 1: We reran all baselines, which is important to ensure the fair comparison of our experiments. For example, we retrieve top 20 candidates that match the provided query and use GPT-4o to answer queries (Section 5.4) for all methods.

Question 2: As mentioned in line 368, we retrieve 20 candidates for all models including all the baselines and SiReRAG.

Question 3: Thanks for your comment! We do not claim SiReRAG as the most efficient RAG indexing method (accuracy on multihop QA tasks is our priority in this work). Having a larger retrieval pool than RAPTOR, we aim to study the efficiency of SiReRAG from a different angle than inference time. Inference complexity involves many factors, including length of documents and redundancy of added tree nodes. Therefore, we will design a new metric called Time-Pool Efficiency Ratio (TPER): $\frac{\text{Inference-time A} / \text{Inference-time B}}{\text{Pool-size A} / \text{Pool-size B}}$. This metric computes the growth of total inference time with respect to the growth of the retrieval pool size between two methods. In other words, we aim to ensure that the growth of inference time does not scale proportionally with the increase in the retrieval pool size. A TPER value less than 1 indicates reasonable efficiency of method A compared with method B, whereas a TPER value greater than 1 signifies lower efficiency. Thus, we set SiReRAG as method A and RAPTOR as method B to compare their efficiency.

We see that these TPER values are well below 1, which indicates that SiReRAG is a reasonably efficient method.

Thanks so much for your kind acknowledgment and raising your score!😊 As mentioned, we are working to upload a revised PDF by the end of November 27 to incorporate all reviewers' comments.

We thank the reviewer for the helpful feedback! We plan to submit another draft that incorporates all reviewers’ comments before the discussion phase ends.

Weakness 1: We mention to flatten nodes for retrieval, and we will clarify this concept in Section 4.3 of our revised version. Flattened indexing is a simple method that places all tree nodes into a unified retrieval pool. In other words, regardless of a node's origin (e.g., bottom or upper levels, similarity or relatedness trees), it is added to a single list containing all nodes.

Weakness 2: Thanks for your comment! We do not claim SiReRAG as the most efficient RAG indexing method (accuracy on multihop QA tasks is the priority). Having a larger retrieval pool than RAPTOR, we aim to study the efficiency of SiReRAG from a different angle than inference time. Inference complexity involves many factors, including length of documents and redundancy of added tree nodes. We will design a new metric called Time-Pool Efficiency Ratio (TPER): $\frac{\text{Inference-time A} / \text{Inference-time B}}{\text{Pool-size A} / \text{Pool-size B}}$. This metric computes the growth of total inference time with respect to the growth of the retrieval pool size between two methods. In other words, we aim to ensure that the growth of inference time does not scale proportionally with the increase in the retrieval pool size. When we set SiReRAG as method A and RAPTOR as method B, a TPER value less than 1 indicates reasonable efficiency, whereas a TPER value greater than 1 signifies low efficiency. Thus, we have TPER values to compare SiReRAG and RAPTOR.

We see that these TPER values are well below 1, which indicates that SiReRAG is a reasonably efficient method.

Weakness 3: Thanks for your suggestion! We will add a full example of the relatedness tree in the appendix of our revised draft. We will briefly describe a tree example here. For the multihop question in Figure 1 (correct answer: Nicholas Bacon), the MuSiQue corpus contains one relevant paragraph stating, 'Francis Bacon was born on 22 January 1561 at York House near the Strand in London, the son of Sir Nicholas Bacon…'. The RAPTOR tree for the entire MuSiQue corpus has two more mentions (both mentions are in summary nodes) of Nicholas Bacon, one of which reads: 'Francis Bacon: Born on 22 January 1561, son of Sir Nicholas Bacon and Anne Cooke Bacon.' The addition of our relatedness tree adds two more mentions of Nicholas Bacon: one is in a proposition aggregate (“Francis Bacon was the son of Sir Nicholas Bacon, Lord Keeper of the Great Seal, and Anne (Cooke) Bacon.\n…Head I is a small oil and tempera on hardboard painting by Francis Bacon, completed in 1948.…”), and the other one is a summary node (“The text provides detailed genealogical and biographical information about several individuals from different families and historical periods… Francis Bacon was the son of Sir Nicholas Bacon and Anne (Cooke) Bacon, making William Cecil, 1st Baron Burghley, his uncle...”). Thus, RAPTOR tree has three mentions of Nicholas Bacon, but none of them contains Head I information. SiReRAG has five mentions of Nicholas Bacon, and one of them (the proposition aggregate) contains Head I information. This proposition aggregate groups several propositions together via the entity “Francis Bacon”. Because this node is the only retrieval candidate that fully matches and answers the question, retrieving it would maximize the chance of generating the correct answer. If we use the RAPTOR tree only, we will not have this retrieval candidate. We believe this is an excellent example of how a comprehensive knowledge integration process can enhance the performance of LLMs in multihop reasoning. As a result, we are motivated to model both signals of similarity and relatedness.

Weakness 4 and Question 1: Thanks for your great suggestion! Yes, we agree with you that the RAPTOR text clustering pipeline does not have any special mechanism to take care of the underlying structure of proposition aggregates, which is a future direction of our work. Encoding nodes differently provides a promising approach to further enhancing SiReRAG's performance. Although 2Wiki’s entity-centric nature is well-suited for HippoRAG, we would like to highlight that our method yields significantly better performance than HippoRAG on average. As this paper focuses on modeling both similarity and relatedness, tailoring text encoding to capture fine-grained details would be a valuable extension.

Question 2: As stated in line 368, we retrieve the top 20 candidates for each query in both SiReRAG and the baselines for fair comparisons. The retrieval pool sizes of SiReRAG on MuSiQue, 2Wiki, and HotpotQA are 35070, 19100, and 29934 respectively. The retrieval pool sizes of RAPTOR on MuSiQue, 2Wiki, and HotpotQA are 12371, 6939, and 10031 respectively. SiReRAG's retrieval pool size is slightly less than three times the size of RAPTOR's. Considering the discussion above on TPER, we believe our method is reasonably efficient.

Thank you for addressing the comments and explaining more. I look forward to reading the modified draft.

After this discussion, I am raising my soundness and presentation scores to 4.

We thank the reviewer for the helpful feedback! We plan to submit another draft that incorporates all reviewers’ comments before the discussion phase ends. Focusing on multihop reasoning, we are, to the best of our knowledge, the first to verify the importance of incorporating both similarity and relatedness signals and to implement this concept in RAG indexing.

Weakness 1: To the best of our knowledge, our work is the first indexing method of modeling both similarity and relatedness in RAG setup for a more comprehensive knowledge integration. Our baselines (RAPTOR, HippoRAG, and GraphRAG) consider only a single perspective. In contrast, we find that modeling both perspectives enables a more effective knowledge integration process, as detailed in Section 3 (adopting a clustering strategy based on a single perspective covers a remarkably low percentage of supporting passages). Our quantitative results in Section 6 echoes this motivation.

Although we are motivated, it also requires nontrivial efforts to realize our high-level idea as a RAG indexing method. Firstly, we decide to focus on multihop reasoning as our scope, as knowledge integration or synthesis is vital to this kind of task as discussed in Figure 1. Then, we need to finalize an effective and efficient structure (e.g., tree/knowledge graph) to index corpus data. RAPTOR tree is a straightforward idea, but it deviates from the commonsense understanding of a tree structure [1]. As noted in line 207, while multiple levels of data abstraction can exist within a document, RAPTOR places all text chunks at the bottom level. This approach conflicts with many studies on document discourse trees [2]. Then, we explore the hierarchical structure of an indexing tree. However, we do not see a clear trend of improvement by doing so (Table 2). So, we use RAPTOR as the generalized tree structure for similar and related information. We believe our exploration and observation are valuable for future work.

As stated, we adhere to a rigorous thought process. The effectiveness (Table 4 and our response to Question 3) and wide applicability (Table 6 and our response to Weakness 2) of SiReRAG is a significant contribution, as reasoning capability is crucial for LLMs, and multihop reasoning questions are among the most complex types of queries.

[1] Zhang, J., Silvescu, A., & Honavar, V. (2002). Ontology-driven induction of decision trees at multiple levels of abstraction. In Abstraction, Reformulation, and Approximation: 5th International Symposium, SARA 2002 Kananaskis, Alberta, Canada August 2–4, 2002 Proceedings 5 (pp. 316-323). Springer Berlin Heidelberg.

[2] Maekawa, A., Hirao, T., Kamigaito, H., & Okumura, M. (2024, March). Can we obtain significant success in RST discourse parsing by using Large Language Models?. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2803-2815).