From Complex to Atomic: Enhancing Augmented Generation via Knowledge-Aware Dual Rewriting and Reasoning

We appreciate the valuable feedback and insightful comments from the reviewer. Below, we address each concern point by point. All tables are accessible via hyperlinks.

Q1. Embedding model used for retrieval.

The text-embedding-ada-002 model is used across all experiments demonstrated in the paper. For more hyper-parameter settings, you can refer to Appendix A.2 (line 654 ~ 668).

Q2. Chunk size and number of chunks retrieved.

As introduced in line 370 ~ 373 in the paper, we compile the context paragraphs without additional chunking, resulting in a chunk size of around 500 chars. The detailed statistics are listed in Table 1. In retrieval phase, the retriever is configured to retrieve 4 atomic questions per atomic query with a relevance threshold of 0.5. However, the actual number of chunks retrieved varies based on the number of queries proposed and their associated relevance scores. Additionally, since there is an atomic selection step incorporated in each decomposition round, a maximum of one chunk may remain after each round. Given decomposition loop constrained up to 5 rounds (N = 5) in our main experimental setup, no more than 5 chunks will be utilized during final answer generation phase.

Q3. Which work is the RAG approach (mentioned in L65 of page 2) that use question decomposition without considering available knowledge?

Thanks the reviewer for pointing it. One such example is Self-Ask ([Press et al., 2023]). We will clarify this in the revised version and explicitly refer to Self-Ask to ensure the statement is supported and precise.

Q4. Paper do not include other RAG techniques where the knowledge augmentation happens at training or fine-tuning but only at inference time.

Thanks for the suggestion regarding related work. In the revised version, we will include the key works that incorporate knowledge augmentation during training or fine-tuning, such as REALM: Retrieval-Augmented Language Model Pre-Training ([Guu et al., 2020]), and LLaMA-Adapter: Efficient Fine-Tuning of LLaMA for RAG ([Zhang et al., 2023]), among others.

Q5. Rename the metric Acc to LLM evaluator or similar.

Thanks for the suggestion! We understand how the term "Acc" might cause confusion. In the revised version, we will rename the metric to something more descriptive, such as "LLM Evaluator", to better reflect its purpose and avoid overloading the term.

Q6. Use a LLM with a context window sufficiently long to include the increasing context.

To explore the dependency of our approach on the context window size, we analyzed the token distribution on MuSiQue (using GPT-4), considering settings where N = 5 and N = 10 (N represents the decomposition round limit, detailed in line 2 of Algorithm 1). The findings are presented in Table 2 and 3. Table 2 shows that the maximum number of input tokens is around 6K, while the maximum output tokens is around 0.5K across all LLM interactions. Additionally, this maxmium token consumption shows only a slight increase as N increases but remains within the same order of magnitude. This suggests that the maximun token requirement grows gradually. Futhermore, Table 3 demonstrates that over 99% of the LLM calls are accommodated by models with a token limit of 4096, a capacity commonly supported by existing LLMs. For models providing context windows of at least 8K tokens, all LLM calls can be handled without the need for any token truncation.

Table 3: Prompt Token (per LLM call) Distribution of KAR³ on MuSiQue.

We appreciate reviewer's feedback. All tables are accessible via hyperlinks.

Q1-1. How large was the knowledge base in the experiments?

Table 1 provides detailed statistics. All chunks are derived from the context paragraphs of the sampled QA, with chunk count varying by dataset. Additionally, the count info is included in Appendix A.1.

Table 1: Chunk Statistics

Q1-2. How computationally intensive was the process of generating atomic questions?

Table 2 shows the computational cost. Per Appendix A.3, atomization is a one-time step requiring LLM calls equal to chunk count, with cost scaling linearly based on chunk sizes and generated questions.

Table 2: Preprocessing Token Consumption

Q1-3. Discussion of scalability and simpler preprocessing method.

Our approach mitigates scalability for larger datasets through three key features:

• Dynamic addition of atomic questions without structural changes
• Linear preprocessing cost scaling with corpus size
• Compatible with open-source LLMs (Table 2 in paper), reducing cost with only 3% performance drop on MuSiQue

Following the valuable suggestion to employ simpler preprocessing steps, we tested using plain-text sentence splitting via spacy instead of LLM-based atomization. Each sentence serves as an atomic question. After revising the selection prompt (Appendix line 1375-1396), results in Table 3 show that while performance drops 7% on MuSiQue, this method still outperforms most baselines from Tables 1 and 2 in paper. This demonstrates its effectiveness in case lower-cost preprocessing needed, offering a flexible performance-efficiency tradeoff alongside "dynamic addition".

Q2. Clarify whether baselines employed preprocessing or KB re-structuring? How might that affect the performance？

Most baselines use standard retrieval methods without preprocessing (details in Appendix Table 5, page 14). Only Self-Ask explicitly generates sub-questions that can query atomic questions using a similar retrieval path (sub-question -> atomic question -> chunk).

Testing atomic questions with Self-Ask, IRCoT, and Iter-RetGen shows slight performance improve for Self-Ask (1.6%) due to its natural retrieval path, but decreased performance for IRCoT and Iter-RetGen (see Table 4). This demonstrates that atomization alone cannot contribute significantly to performance.

Table 4: Abalation study of baselines with atomic questions

Q3. Awareness of tree-based or partial KB re-structuring methods ([Sarthi et al., 24]). How might those compare to your approach？

We compared KAR³ with other KB-restructuring methods like GraphRAG and RAPTOR (Tables 6, 14, 15 in appendix, and Table 5). Both underperform in F1 scores, likely due to their focus on summarization which can introduce redundancy. KAR³'s precise chunk atomization enables more accurate retrieval and reasoning, leading to better performance on complex questions in 2Wiki and MuSiQue.

As for complexity, the preprocessing cost of both KAR³ and RAPTOR scales linearly with chunk size, while GraphRAG incurs significantly higher cost due to its hierarchical KG construction. This highlights that KAR³ achieves outstanding performance with reasonable computational cost compared to RAPTOR and GraphRAG.

Table 5: Comparison of KB re-structuring methods

Q4. How can we ensure the quality of knowledge atomizer？

KAR³ reduces dependence on high-quality atomization in two ways:

• Atomic Questions Mainly as Multi-aspect Indexing: Generates multiple atomic questions from different perspectives, providing relatively comprehensive coverage without requiring perfect atomization.
• Separation of Retrieval and Reasoning: Uses atomic questions only for retrieval, with downstream LLMs handling reasoning, making the system robust to imperfect retrieval.

In Table 3 of paper, using Llama3 instead of GPT-4 for all components only reduces MuSiQue performance by 2.9%, demonstrating KAR³'s robustness to atomizer quality. While better atomization would improve performance, KAR³ can achieve this incrementally since it naturally supports adding new atomic questions to existing databases on the fly.

We appreciate the valuable feedback and insightful comments from the reviewer. Below, we address the raised concerns point by point.

Q1. Discussion on the important limitation of the method that it is a resource heavy method and can not be used practically when retrieving from dynamic data sources or large volume datasources such as the web.

While resource demands are an important consideration, our approach incorporates several key features and optimizations that mitigate these challenges and enhance practicality.

Dynamic Integration of Data Sources: Our method supports dynamic addition of data sources without requiring structural modifications. This contrasts with knowledge graph-based extraction methods, which necessitate updating the graph when expanding the corpus. This flexibility ensures seamless integration of new information.

Scalable Preprocessing Costs: The construction of our knowledge base involves a one-time preprocessing cost that scales linearly with the size of the corpus. This scalability ensures that our approach remains efficient and suitable for processing large-scale datasets. The detailed preprocessing cost of three benchmarks are provided in Table 12 in the Appendix (page 18).

Optimizations for Large-Scale Data: We acknowledge the need to further optimize preprocessing for extremely large datasets, such as web-scale data, where resource efficiency becomes critical. To address this, we propose alternative atomization strategies to reduce preprocessing costs while maintaining competitive performance.

1. Using Open-Source Models: We explore replacing computationally expensive LLMs like GPT-4 with more resource-efficient open-source models such as LLaMA 3 during the chunk atomization step. In additional experiments, this substitution significantly reduced preprocessing costs, with only a minor accuracy drop (~3% on MuSiQue dataset), as shown in Table 1.

2. Sentence-Level Segmentation: for scenarios requiring even lower-cost preprocessing, we propose using sentence-level segmentation as atomic units for retrieval. Although this approach reduces performance (55.2% on MuSiQue), it still outperforms the majority of baselines presented in Table 1 of the main paper, demonstrating its practicality in resource-constrained settings.

Table 1: Ablation study on the preprocessing method on MuSiQue

Q2. Thoughts on the trade-off between resource usage and efficiency, and how to effectively weigh the advantages?

The trade-off between resource usage and efficiency is an important consideration for our method, and we have designed it to balance these aspects effectively while offering flexibility based on specific use cases. To this end, we propose alternative atomization strategies that allow users to tailor preprocessing costs to meet their needs.

For example, substituting computationally intensive models like GPT-4 with lighter open-source models such as LLaMA 3 significantly reduces preprocessing costs, with only a minor performance drop (~3% on MuSiQue). Additionally, for resource-constrained scenarios, sentence-level segmentation can be employed, reducing preprocessing overhead while still outperforming most baselines (55.2% on MuSiQue).

The choice of strategy depends on the specific application: high-accuracy requirements may justify the use of more powerful models, while dynamic or resource-limited settings can benefit from cost-effective alternatives. These considerations ensure that our method remains adaptable and effective across a range of scenarios, balancing resource usage and performance as needed. The detailed experiment results are provided in Table 2 of main paper and Table 1 in this response.

In the revised manuscript, we will incorporate a detailed discussion on the trade-off between resource consumption and efficiency, accompanied by experimental results of alternative preprocessing methods.

Q3. Evaluation with diverse set of tasks and domain should be done.

Thanks for the constructive suggestion. We agree that evaluating our method on a broader set of tasks and domains is crucial to demonstrating its robustness and generalizability. To address this, we have included evaluations on two legal benchmarks in Appendix A.5，which further validate the effectiveness of our approach.

Furthermore, we have applied our method to develop RAG systems in specialized domains, such as manufacturing and healthcare, achieving consistent accuracy improvements of over 15% across a variety of tasks. Due to privacy restrictions, we are unable to release the data from these domains. To further validate and benchmark our method, we are actively exploring publicly available datasets in specialized domains to conduct more rigorous evaluations.

We appreciate the valuable feedback from the reviewer. Below, we address each concern point by point.

Q1. How KAR-RAG system would fare on questions that require parallel subtasks.

KAR³ is specifically designed to handle complex questions by decomposing them into multiple subqueries, enabling effective retrieval and iterative reasoning. This decomposition mechanism allows KAR³ to address both sequential and parallel subtasks. For instance, consider the parallel comparison question provided in Figure 5 of the Appendix (page 22): "Which film came out first, What Women Love or Ramudu Kadu Krishnudu?" KAR³ decomposes this question into atomic subqueries:

(a) "What is the release date of What Women Love?"
(b) "What is the release date of Ramudu Kadu Krishnudu?"

In the first iteration, KAR³ retrieves a chunk tagged with the atomic question "In what year was the film 'What Women Love' released?" relevant to subquery (a) and add it to the context. In the second iteration, using the updated context and the original question, the system regenerates subquery (b) and retrieves the relevant chunk tagged with the atomic question "In what year was the film 'Ramudu Kadu Krishnudu' released?" Through this iterative decomposition and retrieval, KAR³ resolves parallel subtasks.

Q2. What is the stopping criteria for the system when no high quality proposed questions (or possible answers) are available?

When no high-quality proposed queries or no relevant atomic questions, the atomic selector may return an empty or out-of-range atomic question index after evaluating the provided context, which consists of the retrieved atomic questions. This means that no additional relevant chunks can meaningfully contribute to answering the original question. In such cases, the decomposition loop terminates, and the system generates a final answer based on the information already accumulated in the context.

Q3. The paper can use more analysis of results and anecdotes to drive home the advantage of the system presented over other baselines.

Thank you for the constructive suggestion. We agree that incorporating additional analysis and case studies will help to emphasize the advantages of our method over the baselines. In the revised version, we will include more baseline analysis and case studies to highlight the strengths of our method.

Q4. The claim on LLMs' unawareness to technical terminologies may be outdated given newer evidence and missing a key RAG application which is equipping LLMs with knowledge crossing its cutoff date.

We appreciate the insightful feedback and the opportunity to clarify and strengthen our claims.

Potential outdated nature of our claim: While we acknowledge that recent advancements have improved LLMs' performance in specialized fields, challenges persist in areas requiring precise understanding of technical terminologies, particularly in dynamic domains with evolving jargon. For example, in OLED-related technologies, the term CSE is often misunderstood by LLMs as "Charge Spread Effect" or "Charge Sheet Effect" when it actually refers to "Channel-Shortening Effect". Such examples highlight the ongoing limitations of LLMs in accurately handling domain-specific acronyms and terminology, especially when context-specific disambiguation is required. We will refine our claim to acknowledge progress in LLMs while addressing areas where challenges remain.

RAG’s role in addressing knowledge cutoff issues: We agree that one of RAG’s critical advantages is mitigating knowledge cutoff limitations by retrieving up-to-date information. We regret the omission of this important point and will explicitly highlight it in the revised manuscript as an important benefit of RAG systems.

Q5. Differentiating between atomic questions from the original question and the atomic questions from the knowledge base.

Thanks for your valuable suggestions. Atomic query proposals decomposed from the original query are derived to break down the original query into subqueries that aid in addressing the query. In contrast, atomic questions generated from chunks (knowledge base) are the questions that are relevant and can be answered by the given chunk. In the revised manuscript, we will replace the term "atomic questions" with "atomic tags" to clearly distinguish it from "atomic query proposals" and include illustrative examples.

Q6. Suggestions: a) reducing unnecessary jargon, b) early use of running examples, c) properly organizing supplementary materials.

Thanks for your valuable suggestions. In the revised version, we will a) simplify technical language by reducing unnecessary jargon and using clear, concise terminology, b) introduce running examples early on to illustrate key concepts step-by-step, making the explanations more accessible and easier to follow, c) carefully review and reorganize the supplementary materials to address misalignments in tables and figures.