Plan-RAG: Planning-guided Retrieval Augmented Generation

We thank the reviewer for their comments. We address the concerns below.

(W1) Ablation without DAG: The reasoning DAG is central to Plan-RAG, providing features like parallelizable queries, structured information flow, and attribution. Removing the DAG would undermine the core framework and its intended contributions. However, we do include ablations with the plug-and-play experts in Section 4.2 to illustrate their individual impacts within the Plan-RAG framework.

(W2) Error analysis of each expert: Thank you for the suggestion regarding expert-level error analysis. Since the experts are sequentially connected, any error in one component can cascade to subsequent ones, complicating isolated error analysis. This setup presents challenges in accurately attributing errors to individual experts based on final output. We acknowledge the importance of this analysis and will consider it as future work.

(W3) Plan-RAG with GPT4o as expert: We evaluate Plan-RAG with GPT-4o as the expert on a subset of the HotpotQA dataset (1000 queries). The results are detailed in the table below.

(Q) Latency due to multiple experts and how it could be optimized: We recognize that employing multiple experts introduces some latency; however, this is also true for other competitive state-of-the-art RAG methods that utilize multiple LLMs or LLM calls. In contrast, Plan-RAG’s reasoning DAG structure allows for parallelization, which mitigates overall latency. Section 3.2 discusses the complexity of Plan-RAG, and we will expand the manuscript with further details on this topic.

We thank the reviewer for their comments. We address the concerns below.

(W1) Minor Inconsistencies: We will fix them in the manuscript.

(W2) Novelty of "plan-then-retrieve": While we acknowledge that the "plan-then-retrieve" framework has been explored in previous research, such as PlanRAG[NAACL 2024], our approach introduces several key innovations that distinguish it from existing methods. Specifically, we propose the use of a directed acyclic graph (DAG) for structured reasoning, which offers numerous benefits. This DAG structure allows for the decomposition of complex queries into interrelated sub-queries, enabling parallelized retrieval and generation. This structured approach not only improves efficiency but also reduces hallucinations by ensuring each sub-query is contextually linked. In contrast, other methods may mention planning but lack a proper structure, resulting in mere decomposition rather than a true plan. Additionally, the use of subatomic sub-queries ensures a one-to-one mapping between subqueries and retrieved documents, enhancing attribution.

While we recognize the similarities with methods based on tool planning, multi-agent RAG frameworks, query rewriting, and document relevance judgment, the proposed framework's unique contributions lie in its structured reasoning, parallelized subatomic subqueries, relevant flow of information, and enhanced attribution mechanisms.

(W3) Powerful baselines: We add below the details of the baselines on a subset of HotpotQA dataset (1000 queries):

(W4) Performance on single-hop: Plan-RAG is designed primarily for multi-hop queries and its strength lies in generating a reasoning DAG. For single-hop queries, generally there is no plan required and thus not well-suited for the simplicity of single-hop queries. However, in section 4.1 we perform LLM evaluation and showcase that the difference between Self-RAG13B and Plan-RAG8B is 3.3% and not 8.4% according to the accuracy metric (see Table 4 in the manuscript).

(W5) Experiment on efficiency and debuggability: For debuggability, the structured reasoning provided by the DAG allows for clear mapping between subquery, generation, and retrieved document. This mapping facilitates easier identification and correction of errors, as each part of the generated response can be traced back to a specific source. While the current manuscript may not explicitly detail debuggability, the inherent design of Plan-RAG supports this benefit. We have an example of this in the appendix, Figure 4. We plan to explore this further in future work.

(Q2) Attribution in RAG: In the context of Plan-RAG, "attribution" refers to the ability to trace the generated responses back to specific retrieved documents. This is crucial for ensuring the reliability and transparency of the information provided by the system.

Plan-RAG achieves enhanced attribution through its structured reasoning process using a DAG. This structure decomposes complex queries into interrelated atomic sub-queries. Each sub-query typically retrieves a single document, creating a clear one-to-one mapping between the sub-query and the retrieved document. This granularity ensures that each part of the generated response can be directly linked to a specific source, making it easier to verify the information and reducing the risk of hallucinations.

(Q3) Comparison against multiple pass planning (e.g. ReAct): ReAct and Plan-RAG are both designed to enhance LLMs performance, but they differ significantly in their approaches. ReAct combines reasoning and acting in an iterative loop, allowing the model to refine its understanding and actions dynamically, making it highly adaptable to evolving tasks. In contrast, Plan-RAG uses a directed acyclic graph (DAG) to structure the reasoning process, decomposing complex queries into interrelated sub-queries for parallelized retrieval and generation. This structured approach improves efficiency and ensures clear attribution by linking each sub-query to a specific document, reducing hallucinations. While ReAct excels in dynamic decision-making scenarios, Plan-RAG is optimized for efficiently handling complex, multi-hop queries with enhanced transparency and reliability.

We thank the reviewer for their comments. We address the concerns below.

(W1) Novelty compared to query-decomposition: Plan-RAG introduces key differences from query-decomposition methods like RQ-RAG and RA-ISF:

• Structured Reasoning Plan: Plan-RAG generates a structured reasoning plan represented as a DAG, establishing interrelationships between subqueries to support organized and efficient retrieval. This structured approach directs a constrained and relevant flow of information to each subquery, preventing context-window overflow.
• Enhanced Attribution due to atomic queries: Unlike other query-decomposition methods that decompose queries into subqueries of similar complexity, Plan-RAG breaks down queries into interrelated, subatomic queries. Each subatomic query is precise enough to find its answer within a single document, facilitating attribution.
• Parallelizable subqueries: The DAG structure enables subqueries to be processed in parallel, significantly improving response time. In contrast, traditional methods handle subqueries sequentially, which is inherently slower.

(W2) Up-to-date LLMs as experts: Plan-RAG's plug-and-play architecture allows seamless use of the latest LLMs, like Llama3-8b-Instruct, as experts. In contrast, methods like Self-RAG and RA-ISF rely on fine-tuned Llama-2 models distilled from GPT models. This distillation process is costly, computationally intensive, and requires specialized expertise. Additionally, as LLMs evolve, these methods cannot directly adapt, while Plan-RAG offers more flexibility to incorporate advancements in LLMs.

(W3) Attribution evaluation: Plan-RAG achieves built-in attribution by design. Each subquery in the reasoning DAG is subatomic and, when retrieval is triggered, retrieves only a single document, creating a direct one-to-one mapping between each subquery and its corresponding document. This design enables clear attribution. Other methods, however, retrieve a top-k set of documents (e.g., k=5 or 10), over which the LLM reasons to generate a response, leading to ambiguous attribution. Reducing k to 1 for these methods results in a substantial accuracy drop, underscoring Plan-RAG’s advantage in balancing attribution with performance.

(W4) Critic expert performance: In Section 4.2 (Table 6 in the manuscript), we conducted an ablation study to evaluate the impact of the Critic Expert. The results indicate that removing the Critic Expert and always retrieving results in a minor accuracy difference of only 0.53%, while significantly increasing the number of retrievals by 600 (from 2530 to 3163). This suggests that the Critic Expert effectively reduces the number of retrievals with minimal impact on accuracy. However, we acknowledge that further analysis is needed to assess how well the reduced retrievals provide sufficient context for generating high-quality responses. Future work will focus on a deeper investigation into the context adequacy provided by the Critic Expert's reduced retrievals.

We thank the reviewer for their comments. We address the concerns below.

(W1) Figure 2 presentation: The three additional components in Figure 2 are the retriever, generator, and reasoning planner. These are not included as experts because they serve distinct roles: the retriever is a standard module for fetching relevant documents from the database, the generator is the LLM responsible for answer generation, and the reasoning planner orchestrates task flow between experts, as described in Section 3.1. Conversely, Section 3.2 focuses on experts that handle specialized tasks within the framework. We agree that adding a concrete example will further clarify the interactions between components, and we will incorporate this in the manuscript to provide a clear understanding of the framework.

(W2) Inconsistent backbone: Thank you for raising this point. Plan-RAG's plug-and-play architecture allows it to utilize the latest LLMs, like Llama3-8b-Instruct, as experts. On the other hand, methods like Self-RAG rely on fine-tuned Llama-2 models that distill information from GPT models. This distillation process is costly, computationally intensive, and requires specialized expertise. Therefore, simply substituting Plan-RAG's experts with vanilla Llama-2 models would not yield a fair comparison, as Self-RAG and RQ-RAG employ fine-tuned models. Also, as LLMs evolve, these methods can not directly adapt, while Plan-RAG is more flexible. We acknowledge that this aspect could have been more explicitly clarified in the manuscript, and we will update the discussion to clarify these distinctions.

(W3) Latency of plug-and-play architecture: Self-RAG, RA-ISF, and RQ-RAG all methods involve multiple calls to the LLM models so the latency of Plan-RAG is comparable (or even better attributed to parallelizable subqueries) with other methods. We discuss the complexity of Plan-RAG in Section 3.2 in the manuscript.

(W4) DAG complexity for simpler queries: In the single-hop PopQA dataset, the depth of the reasoning DAG is 0 for the majority of queries (see Table 2 in the manuscript), demonstrating Plan-RAG’s ability to adapt to the complexity of each query.

(W5) Challenges in generating an accurate DAG: We appreciate the reviewer’s attention to the critical role of accurate DAG construction. The reasoning planner indeed serves as the central component of our framework, which is why we currently utilize GPT-4o for generating the reasoning DAG, as it achieves consistently reliable results. Llama models, without fine-tuning, do not meet this performance threshold; however, once fine-tuned, they could replace GPT-4o in this step.

(Q) Ablation on only 2 experts: Plan-RAG comprises four experts: the Dynamic Query Expert, Critic Expert, Relevance Expert, and Aggregator. Our ablation study focuses on the Critic and Relevance Experts, as these two are foundational to the system's performance. The Dynamic Query Expert and Aggregator have more specific, contained roles: the former modifies the query with parent information, while the latter aggregates outputs from various subqueries to form the final answer. Given the limited responsibilities and scope of these experts, we chose not to perform ablations on these components.