From Complex to Atomic: Enhancing Augmented Generation via Knowledge-Aware Dual Rewriting and Reasoning

Q1.The main challenges in domain-specific reasoning and why current RAG methods fall short in this setting.

In our work, domain-specific knowledge pertains to information extracted from the corpus designated for RAG, which may not be fully covered by the pre-trained LLMs and can sometimes diverge from LLMs' intrinsic knowledge. For instance, within a particular domain, "ice cream" might be used as a unique project name rather than referencing the typical frozen dessert. This domain-specific knowledge serves as external knowledge to assist LLMs in generating responses that are consistent with the knowledge from the RAG corpus. Nevertheless, integrating domain-specific knowledge to domain-aligned generation introduces significant challenges, which include the difficulties of a) effectively extracting domain-specific knowledge, b) precisely retrieving it via queries, and c) successfully constructing coherent reasoning chain based on it. To tackle these challenges, we propose a novel approach encompassing knowledge atomization, atomic pair retrieval, and query decomposition with reference context. Although current RAG approaches such as Self-Ask and SeaChain similarly employ LLMs to sequentially generate sub-questions, they fall short in adequately harnessing the knowledge contained within RAG data. They also fail to comprehensively tackle all three of the aforementioned challenges.

To illustrate the limitation of current RAG, Table 4 presents an example. The regular RAG systems tend to rely on common knowledge to decompose query into subqueries, often overlooking the availability of corresponding answers in the RAG data, as shown in the part one of example analysis of the response to CQ3. Our method, illustrated in the part two of example analysis, incorporates the reference context from previously retrieved data to guide the subsequent query decomposition. This approach generates potential reasoning paths based on the domain-specific knowledge from RAG data, ultimately arriving at the correct answer with a complete reasoning chain.

In the revised version, we will highlight the challenges in domain-specific reasoning and provide detailed explanation of on how current RAG methods fall short in this setting.

Q2. Main technical challenges besides the proposed pipeline.

As referred in the response to Q1, the integration of domain-specific knowledge into domain-aligned generation is fraught with challenges. The challenges lie in a) effectively extracting domain-specific knowledge, b) precisely retrieving relevant knowledge via queries, and c) successfully constructing coherent reasoning chain based on it. Current RAG approaches primarily address only a subset of the challenges. Designing effective individual components and optimizing them jointly within the RAG framework is a significant technical hurdle, aside from pipeline considerations.

First, we propose a novel knowledge representation, atomic question, to exploit the multifaceted knowledge within RAG data. Secondly, we implement a query decomposition strategy to produce subqueries. This facilitates a more precise question-to-question retrieval process, thereby better aligning the query with the knowledge representation. Thirdly, we propose decomposing query to multiple subqueries with awareness of the relevant domain-specific knowledge, presented as reference contexts. Our design enables the exploration of potential reasoning paths through multiple subqueries and the selection of the most confident one to pursue. This design is more efficient than ProbTree that transverse each decomposition path and more robust than Self-Ask that relies on deterministic sequential decomposition steps.

To clarify the contribution of each proposed component, we conduct ablation study in the main paper (line 483~521). More specially, we replace each proposed module with a simpler, standard module typically used in regular RAG systems. The results, presented in Table 5, demonstrate the effectiveness of each component. It is evident that each element is vital to our method's performance, particularly as the complexity of the questions increases, such as in the multi-hop scenarios seen with MuSiQue.

Table 5. Ablation study on components of KAR³.

Q3. Cost analysis and comparison with other baselines

To address reviewer's concern, we have conducted a comprehensive cost analysis to evaluate our model's API consumption compared to other baseline methods, and token consumption per QA of various methods are shown in Table 1. Our method consumes fewer tokens per QA compared to ProbTree and IRCoT, and is on par with Iter-RetGen. Notably, our approach significantly outperforms these baselines in both F1 and Accuracy metrics.

Moreover, we implement our method and other baselines using the open-source LLM, meta-llama/Llama-3.1-70B-Instruct (including chunk atomization). The results are presented in Table 2, demonstrating that our method not only maintains excellent performance after replacing GPT-4 with an open-source LLM, but also significantly outperforms the other baselines. For a more detailed discussion, please refer to our response to CQ1.

Q4. Do cheaper LLMs fit in the pipeline?

We replace Gpt-4 with the open-source LLM, meta-llama/Llama-3.1-70B-Instruct (including chunk atomization). The results presented in Table 2, indicate that this implementation could maintains excellent performance after replacing GPT-4 with an open-source LLM. For a more detailed results, please refer to our response to CQ1. It should be noted that language models must possess an adequate capacity to follow instructions contained within prompts effectively. Consequently, language models that are too weak are not suitable for performing the complex reasoning tasks that our objectives demand.

Q5. The Reason that IRCoT shows an improvement with LLaMA3 than with GPT4.

We appreciate the reviewer for highlighting the behavior of IRCoT. To investigate this, we analyzed the experimental logs of both models on the three datasets. IRCoT operates by iteratively prompting LLMs to generate rationale sentence based on query, previous rationale sentences and previous retrieved chunks. It then retrieves new passages with the newly generated rationale sentence. This process continues until the LLM can produce a final answer without needing additional rationale output.

We count the number of reasoning rounds during the question-answering process for both the GPT and Llama versions, as shown in Table 10. Our analysis reveals that the IRCoT using GPT tends to resolve questions with fewer reasoning rounds, with 74.0%, 66.4%, 42.6% of questions answered without any reasoning for HotpotQA, 2Wiki, MuSiQue. Conversely, the IRCoT using Llama tends to engage in multiple rounds of reasoning, with 77.4%, 66.8%, 47.4% of questions requiring five reasoning rounds to produce answers for HotpotQA, 2Wiki, MuSiQue. We suspect this is because GPT is more confident in its ability to provide direct answers, while Llama tends to offer reasoning steps rather than arriving at the answer in a single step.

Table 10: Reasoning round statistics of IRCoT on three datasets (500 questions per dataset).

To provide a clearer understanding of the relationship between the number of reasoning rounds and performance changes, we calculated the average number of reasoning rounds for each question across both versions. The results are presented in Table 11. Additionally, the difference between the results of the two versions is indicated in parentheses following the Llama results, allowing the reviewer to easily observe the trend of changes.

Table 11. Relations between the reasoning rounds and performance on three datasets

As the difficulty levels of the three datasets differ significantly, with HotpotQA < 2Wiki < MuSiQue, the performance improvement on these datasets also varies. For simpler datasets like HotpotQA, Llama demonstrates a significant performance improvement compared to GPT. This is because Llama tends to activate reasoning processes to arrive at answers, while GPT is more prone to providing answers directly without reasoning. However, as the difficulty of the questions increases (e.g., in MuSiQue), GPT begins to activate its reasoning process rather than simply providing direct answers. This is reflected in the notable increase in the average reasoning rounds for GPT in MuSiQue (1.52) compared to HotpotQA (0.46). Conversely, Llama's reasoning rounds decrease (from 4.43 to 3.79) as question difficulty increases, which may indicate that Llama struggles with handling questions that require complex reasoning. When GPT activates its reasoning process, the performance advantage of Llama disappears, as evidenced by the results from MuSiQue.

Q1. The effectiveness of each individual component. What if the knowledge is mingled together, rather than presented in atomic simple questions.

In order to clarify the contribution of each component, we conduct ablation study in the main paper (line 483~521 and Table 2). We replace each proposed module with a simpler, standard module typically used in regular RAG systems. The results, presented in Table 5, demonstrate the effectiveness of each component. It is evident that each element is vital to our method's performance, particularly as the complexity of the questions increases, such as in the multi-hop scenarios seen with MuSiQue.

Table 5. Ablation study on components of KAR³.

If knowledge is mingled together without atomization, we can perform chunk-level retrieval to retrieve the relevant chunks for further reasoning. This approach is exactly how the variant method, "w/o Atomic Pair Retriever" is implemented, typically leading to suboptimal performance. For clarity and convenience, the corresponding row is highlighted in Table 5.

Q2. Lack of related work and baselines, such as SeaKR and ToR.

We appreciate the reviewer's suggestion to incorporate performance comparisons with two more baseline methods. In response, we have conducted a comprehensive evaluation against three new baselines, including ToR. The performance comparison results are presented in Table 2.

It is important to note SeaKR relies on self-aware uncertainty calculated from hidden state embeddings of LLMs. The official implementation of SeaKR necessitates modifications to the inference function to access these embeddings, a requirement that is incompatible with GPT-4. Consequently, we have not included SeaKR in our comparison table.

In the case of ToR, which has no available code, we compare our results directly with the performance reported in their paper. In addition to F1 metrics shown in Table 3, ToR reports supporting fact recall rates of 73.8, 79.4, and 48.5 for HotpotQA, 2Wiki, and MuSiQue, respectively. In contrast, our method achieves significantly higher supporting fact recall rates of 92.83, 94.06, and 73.8 on these datasets, as detailed in Table 6 of our paper.

Overall, our method consistently outperforms these newly added baselines. For more comprehensive comparison results on both QA performance and token consumption, please refer to our responses to CQ1 and CQ2.

Dear Reviewer 1t8b,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to conclude on December 2nd, we would greatly appreciate it if you could review our responses and offer any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 4468

Q1. Overhead this method introduces in terms of generating the final answer in terms of response time/cost of using LLM in the whole process.

The LLM API usage in our method consists of two main components: 1) chunk atomization as a one-time preprocessing step, and 2) the inference process, which includes query decomposition, retrieval, selection, and generation.

For preprocessing, the API consumption scales linearly with the number of data chunks and constitutes an overhead that varies across different benchmarks. Specifically, the average token usage per chunk for the three benchmarks is 338 tokens for HotPotQA, 321 tokens for 2Wiki, and 320 tokens for MuSiQue.

Regarding inference costs, we conduct a thorough cost analysis to compare our model's API usage with other baseline methods. As shown in Table 1, our method consumes fewer tokens per QA compared to ProbTree and IRCoT, and is on par with Iter-RetGen. Notably, our approach outperforms these baselines in both F1 and Accuracy metrics by a considerable margin. Given the variability in GPT-4 API response times, we use token consumption per QA as a measure of cost.

To further address the reviewer's concern on the API cost, we implement our method and other baselines using the open-source LLM, meta-llama/Llama-3.1-70B-Instruct (including chunk atomization). The results presented in Table 2, demonstrate that our method not only sustains excellent performance after replacing GPT-4 with an open-source LLM, but also significantly outperforms the other baselines. For a more detailed discussion, please refer to our response to CQ1.

Q2. Do any of the existing query decomposition techniques without using LLM?

There are alternative methods for decomposing questions into sub-questions without relying on LLMs. Earlier approaches used a combination of hand-crafted heuristics, rule-based algorithms, and supervised learning for decompositions. For instance, Ethan introduced the One-to-N Unsupervised Sequence Transduction(ONUS) approach, which employs Seq2Seq models to map complex, multi-hop questions to simpler, single-hop sub-questions.

In comparison with earlier methods for decomposing questions, LLMs offer significant advantages. LLMs capture a wide range of linguistic patterns and knowledge, enabling them to generalize better to unseen questions and adapt to various contexts. They leverage pre-trained knowledge to provide more accurate and relevant decompositions with less manual intervention. Although approaches such as Self-Ask and SeaChain similarly employ LLMs to sequentially generate sub-questions, they fall short in adequately harnessing the knowledge contained within RAG data. This leads to a deficiency in generating sub-questions that are contextually pertinent to the RAG data.

Dear Reviewer qWux,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to conclude on December 2nd, we would greatly appreciate it if you could review our responses and offer any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 4468

Q1. Cost analysis on LLM API consumption.

To address reviewer's concern, we have conducted a comprehensive cost analysis to evaluate our model's API consumption compared to other baseline methods, and token consumption per QA of various methods are shown in Table 1. Our method consumes fewer tokens per QA compared to ProbTree and IRCoT, and is on par with Iter-RetGen. Notably, our approach significantly outperforms these baselines in both F1 and Accuracy metrics.

Moreover, we implement our method and other baselines using the open-source LLM, meta-llama/Llama-3.1-70B-Instruct (including chunk atomization). The results are presented in Table 2, demonstrating that our method not only maintains excellent performance after replacing GPT-4 with an open-source LLM, but also significantly outperforms the other baselines. For a more detailed discussion, please refer to our response to CQ1.

Q2. How KAR-RAG resolves “domain-specific knowledge mining” and “complex logical reasoning”?

In our work, domain-specific knowledge pertains to information extracted from the corpus designated for RAG, which may not be fully covered by the pre-trained LLMs and can sometimes diverge from LLMs' intrinsic knowledge. For instance, within a particular domain, "ice cream" might be used as a unique project name rather than referencing the typical frozen dessert. This domain-specific knowledge serves as external knowledge to assist LLMs in generating responses that are consistent with the knowledge from the RAG corpus. Nevertheless, integrating domain-specific knowledge into the logical reasoning process poses challenges, because it require RAG system to a) effectively extract domain-specific knowledge, b) precisely retrieve it relevant knowledge via queries, and c) successfully construct coherent reasoning chain based on it. To address these challenges, we propose KAR³-RAG encompassing knowledge atomization, atomic pair retrieval, and query decomposition with reference context. Specially, our method employ the reference context as known domain-specific knowledge to influence the subsequent query decomposition, thereby tailoring the reasoning path according to the available corpus. In order to assist reviewer understand our approach, we present an example in Table 4, and detailed comparison between w/ and w/o domain-specific knowledge is provided in the response to CQ3.

In the revised version, we will clarify the concept of domain-specific knowledge and delineate the challenges encountered when engaging in domain-specific logical reasoning to more clearly convey our motivation and highlight our contribution.

Q3. Lack of related work and baselines, such as SearChain, ProbTree .

We appreciate the reviewer's suggestion to incorporate performance comparisons with two more baseline methods. We have conducted a comprehensive evaluation against three new baselines, including SearChain, ProbTree. The performance comparison results are presented in Table 3. Furthermore, we have analyzed the token consumption for these baselines and present the results in Table 1. Our method demonstrates superior performance over these recently evaluated baselines. For a more exhaustive set of comparison results encompassing a range of baselines, please refer to the detailed responses to Comments CQ1 and CQ2.

Dear Reviewer 4rFD,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to conclude on December 2nd, we would greatly appreciate it if you could review our responses and offer any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 4468

Dear Reviewer 4rFD,

Thank you for reviewing our paper and sharing your thoughtful feedback. We highly value any opportunities to address any potential remaining concerns before the discussion closes, which might be helpful for improving the rating of this submission. Please do not hesitate to comment upon any further concerns. Your feedback is extremely valuable to us.

Thank you once again for your time and consideration.

Best,

Authors of submission 4468