LightRAG: Simple and Fast Retrieval-Augmented Generation

Your thoughtful review has significantly helped strengthen our manuscript. Below, we provide comprehensive responses to each of your valuable comments:

1. Additional experiments with more evaluation datasets and diverse LLMs.

Response: We expanded our evaluation to include additional datasets, specifically the MultiHopQA dataset. This dataset tests retrieval and reasoning across multiple documents with metadata, simulating real-world retrieval-augmented generation (RAG) scenarios. The evaluation results below highlight the effectiveness of LightRAG in handling complex multi-document reasoning tasks, demonstrating a significant accuracy advantage over GraphRAG (43.7% vs. 35.4%), which underscores its robustness.

Additionally, we evaluated LightRAG's performance on mixed datasets using GPT models with varying capabilities, with the results presented in the table below. Across all three models, LightRAG consistently outperformed NaiveRAG, demonstrating its robustness.

Among the tested models, GPT-4o achieved the best performance due to its ability to extract richer information during the indexing phase, enabling more precise and detailed retrieval. The comparable performance of GPT-4o-mini and GPT-3.5-turbo further highlights that LightRAG's effectiveness is influenced by the underlying capabilities of the LLM. More powerful LLMs can extract richer contextual information, leading to improved retrieval and answer generation.

2. Further clarification of global information incorporatation into the query construction process.

Response: Our query construction methodology is based on the approach outlined in GraphRAG. Specifically, we utilize a dataset summary as a foundation to generate representative users and tasks associated with that dataset. These users and tasks are then employed to formulate queries that align with realistic application scenarios.

5. Additional temporal performance analysis and indexing cost evaluation.

Response: We evaluated LightRAG's dynamic adaptation capabilities through a rigorous testing protocol involving sequential document insertions. Our evaluation of LightRAG's adaptation capabilities yielded two key performance metrics:

i) Temporal Data Integration Performance: - Successfully processed five sequential document insertions - Maintained stable indexing times (418-561 seconds per document) - Preserved all existing document relationships

ii) Query Efficiency: - LightRAG: 11.2 seconds average response time - GraphRAG: 23.6 seconds average response time - Result: 52.5% faster query processing

These findings confirm LightRAG's ability to efficiently integrate new information while delivering significantly faster query responses.

6. Further clarification of the schema used for information extraction.

Response: LightRAG's information extraction process is guided by a structured schema that serves two key purposes: defining fundamental entity types (such as person, organization, location, and event) and specifying the relationships between these entities. We manually developed this schema based on domain expertise and practical application requirements to ensure comprehensive and relevant information capture.

7. Further clarification regarding the difference between LightRAG and GraphRAG.

Response: Thank you for requesting this clarification. LightRAG introduces several fundamental architectural improvements over GraphRAG, which are validated by comprehensive evaluation results. Here are the key distinctions:

i) Novel Retrieval Architecture

• LightRAG implements a dual-level retrieval paradigm that combines: • Entity-focused (low-level) retrieval for precise matching • Relationship-focused (high-level) retrieval for conceptual understanding This approach enables efficient handling of both specific and abstract queries, contrasting with GraphRAG's more resource-intensive community-based traversal method.

ii) Enhanced Processing Efficiency LightRAG's architectural innovations deliver several key advantages: • Direct query-to-entity matching eliminates community detection overhead • Lightweight processing enables faster query resolution • Improved scalability across varying dataset sizes • Maintained high retrieval accuracy while reducing computational costs

iii) Advanced Data Integration LightRAG introduces a dynamic update mechanism that significantly outperforms GraphRAG's static approach: • Incremental updates: New data seamlessly integrates into existing structures • No full regeneration: Avoids GraphRAG's need to rebuild community structures • Reduced computational cost: Updates require minimal processing resources • Enhanced adaptability: Efficiently handles dynamic, evolving datasets

iv) Performance Optimization Our experimental results demonstrate LightRAG's superior efficiency through: • Reduced Resource Usage: - Lower token consumption - Fewer API calls Minimal computational overhead • Enhanced Scalability: - Efficient processing of large-scale datasets - Better cost-effectiveness - Maintained performance with increasing data volume

3. Further discussion of adopted evaluation metrics in our LightRAG.

Response: Thank you for your constructive comment. While win rates serve as a straightforward quantitative measure, they cannot fully capture the nuanced differences in model performance. To provide a more comprehensive evaluation, we conducted detailed qualitative analyses (Section 4.2 and Section 4.3 Table3) and case studies (Section 4.4), particularly focusing on complex domains. Through this approach, we have consistently demonstrated LightRAG's superior capabilities in managing complex queries and large-scale datasets.

Regarding concerns about Diversity metrics potentially introducing noise, we have taken careful steps to define Diversity specifically as the meaningful richness and breadth of perspectives in responses. Our evaluation framework ensures that higher Diversity scores reflect genuine improvements in information quality and practical utility, not just superficial variations. We specifically measure how well responses capture relevant viewpoints while maintaining coherence and usefulness in real-world applications.

To promote transparency and facilitate broader community evaluation, we have open-sourced LightRAG with comprehensive documentation and deployment guidelines. This has enabled independent verification across diverse domains and led to successful implementations in various sectors, including electronic library platforms, secure data analysis systems, and financial data processing. These real-world applications provide substantial evidence of LightRAG's effectiveness beyond controlled experimental settings, complementing our formal metrics with practical validation.

In line with the anyoumous policy of paper submission, we will incorporate more diverse evaluation results into our revised version of our paper.

4. Further clarification of our dual-level retrival mechniasm

We appreciate your comment regarding the clarity of the practical implementation of the low-level and high-level retrieval mechanisms in Section 3.2. To clarify, the three-step process described under "Integrating Graph and Vectors for Efficient Retrieval" represents a combination of both low-level and high-level retrieval mechanisms. Specifically:

i). Query Keyword Extraction: This step is a preparatory phase for both low-level and high-level retrieval. It involves extracting local query keywords ($k^{(l)}$) for detailed, entity-specific queries and global query keywords ($k^{(g)}$) for more abstract, high-level queries. These keywords are essential for directing the retrieval process, whether for specific entities or broader relationships.

ii). Keyword Matching: This step involves matching local query keywords with specific entities (low-level retrieval) and global query keywords with broader relationships or themes (high-level retrieval). Therefore, this step addresses both low-level (specific) and high-level (conceptual) queries by appropriately linking query keywords to the relevant entities or relationships in the graph.

iii). Incorporating High-Order Relatedness: This step serves both low-level and high-level retrieval. By gathering neighboring nodes and edges from the retrieved graph elements, it enriches the context of the query. For low-level retrieval, this helps refine the focus on specific entities by incorporating relevant neighboring entities and relationships. For high-level retrieval, it broadens the context by including related entities and relationships, providing a more comprehensive understanding of the overarching themes or concepts. Therefore, this step enhances the relevance of results for both detailed, entity-specific queries and more abstract, conceptual queries.

Thank you very much for your time and insightful feedback. We have addressed your questions and comments with detailed clarifications and additional experiments. We would greatly appreciate it if you could reconsider your rating. Additionally, we are more than happy to answer any further questions you may have.

We are grateful for your thorough review and insightful comments. Your feedback has been instrumental in improving our work. We have carefully considered your feedback and provide our point-by-point responses below:

1. Further discussion on LightRAG's advancements compared to previous graph-based RAG systems, such as GraphRAG.

Response: LightRAG offers several key advancements over GraphRAG, enhancing its scalability and efficiency in dynamic data environments. It utilizes an incremental update mechanism, allowing seamless integration of new entities and relationships without disrupting the existing structure. Additionally, LightRAG employs a dual-level retrieval paradigm that combines low-level and high-level query processing, ensuring detailed and conceptually rich responses. Furthermore, it improves retrieval efficiency by using a streamlined keyword-based vector retrieval approach, significantly reducing token usage and API calls, making it more cost-effective for large-scale deployments. We elaborate on these dimensions further as follows:

1. Incremental Updates for Better Scalability in Dynamic Data Environments Unlike GraphRAG, which requires the regeneration of communities and their reports when new data is added, LightRAG employs an incremental update mechanism. This allows new entities and relationships to be seamlessly integrated into the existing graph without disrupting the pre-existing structure. As a result, LightRAG achieves greater scalability and responsiveness in dynamic environments.

2. Dual-Level Retrieval Paradigm for Comprehensive Query Handling LightRAG introduces a dual-level retrieval paradigm that effectively combines low-level and high-level query processing. Low-level retrieval targets specific entities and relationships, while high-level retrieval focuses on aggregating broader topics and themes. This hybrid approach ensures that responses are both detailed and conceptually rich, surpassing the single-level retrieval method used in GraphRAG.

3. Significant Improvement in Retrieval Efficiency LightRAG enhances the retrieval process by eliminating the need for extensive community-based traversal. Instead, it utilizes a streamlined keyword-based vector retrieval approach, which significantly reduces token usage and API calls. This optimization makes LightRAG more cost-effective and efficient for large-scale deployments.

2. Further discussion regarding the graph construction process, such as specific techniques for entity disambiguation.

Response: In our graph construction process, we utilize the entity disambiguation technique that identifies and merges identical entities extracted from different document chunks. This approach reduces redundancy and enhances the coherence of the knowledge graph. Additionally, the system employs LLM-based profiling to generate detailed descriptions for entities, further assisting in disambiguating those with similar names or roles by incorporating context-specific information.

3. Further description of how local and global query keywords are matched within the graph.

Response: In LightRAG, the dual-level retrieval process matches local and global query keywords within the graph as follows:

Local Query Keyword Matching: Local keywords extracted from the query are matched with specific nodes in the graph that represent entities. This is accomplished using an efficient vector database, where the keywords are compared against entity embeddings. The system retrieves entities with high similarity scores, ensuring precise matches for detail-oriented queries.

Global Query Keyword Matching: Global keywords, which represent broader themes or relationships, are matched with the edges in the graph. These edges are enhanced with high-level keywords during the graph construction phase. Global keyword matching identifies edges that encapsulate overarching concepts, allowing the retrieval process to effectively address abstract queries.

7. A more detailed complexity analysis of LightRAG, incorporating actual runtime data.

Response: In response to the reviewer’s comment, we conducted additional experiments to benchmark the performance of LightRAG's incremental update mechanism against GraphRAG. The detailed experimental results are presented in the following table. Our findings indicate that incremental updates significantly reduce computational costs, with indexing times consistently lower by 40-60%. Furthermore, we provided concrete runtime data for both the indexing and retrieval phases: LightRAG demonstrates near-linear scalability in indexing, with times ranging from 418 to 561 seconds across five documents, and achieves an average query time of 11.2 seconds—less than half of GraphRAG's 23.6 seconds. These evaluation results further emphasize the efficiency of LightRAG.

4. Additional experiments for performance comparisons and analyses of LightRAG's integration with various LLMs.

Response: Our evaluation on the Mix dataset systematically assessed LightRAG's performance across different LLM capabilities, yielding several key insights:

GPT-4o emerged as the top performer, leveraging its advanced capabilities to extract richer information during indexing, which led to more precise retrieval and higher-quality responses. LightRAG consistently outperformed NaiveRAG across all three tested models (GPT-4o, GPT-4o-mini, and GPT-3.5-turbo), demonstrating the robustness of our approach regardless of the underlying LLM.

Interestingly, we observed comparable performance between GPT-4o-mini and GPT-3.5-turbo, indicating that LightRAG's effectiveness scales with LLM capability. This correlation suggests that more powerful LLMs enhance the system's performance through better information extraction during indexing, which in turn improves both retrieval accuracy and response quality. This finding highlights the importance of LLM selection in deploying retrieval-augmented generation systems.

5. Further clarification of ensuring consistency with the pre-existing structure when handling the incrementally added knowledge graph.

Response: In LightRAG, we maintain the consistency and coherence of the knowledge graph through a unified processing pipeline. All input—whether from the initial set of documents or newly added data—is first segmented into chunks. Each chunk undergoes entity and relationship extraction, where entities are added as nodes and relationships as edges in the graph. For duplicate nodes, their descriptions are merged to create a consistent and enriched representation. This approach effectively manages version control and synchronization by consolidating overlapping information during the integration process.

6. Expanded Insights on Human Evaluation of LightRAG

Response: Given the constraints of time, obtaining objective and professional reviewers for a thorough human evaluation of LightRAG’s performance may prove challenging. To address this, we have made the LightRAG framework open-source, allowing for easy implementation in practical settings. This initiative empowers the wider community to incorporate LightRAG into their systems and assess its effectiveness based on their specialized knowledge.

LightRAG has already achieved considerable recognition and has been effectively utilized in a variety of real-world applications, such as creating electronic library platforms and conducting secure data analysis. These examples demonstrate its versatility and effectiveness across different fields. In line with our anonymous policy, we will incorporate additional information on human evaluations of LightRAG from various disciplines in our revised manuscript.

We sincerely appreciate your thorough review and valuable feedback. We have enhanced our manuscript by conducting additional experiments and providing comprehensive clarifications and detailed analyses that address each of your concerns.

In light of our revisions, we would be grateful if you could reconsider your evaluation. We are also available to address any further questions you may have.

6. Further discussion on human evaluation of LightRAG, focusing on insights gained from real-life scenarios.

Response: Due to time constraints, it may be difficult to secure objective and professional reviewers for human evaluation of LightRAG's performance. As an alternative, we have open-sourced the LightRAG framework, providing an easy-to-use solution for deployment in real-world scenarios. This enables the broader community to integrate LightRAG into their systems and evaluate its effectiveness using domain-specific knowledge.

LightRAG has already gained significant recognition and has been successfully applied in various real-world contexts, including the development of electronic library platforms and secure data analysis. These practical applications highlight its effectiveness and versatility across multiple domains.

Due to our anonymous policy, we will include additional content on human evaluation of LightRAG from diverse fields in our revised manuscript.

3. Further analysis of the time and space complexity of LightRAG in the indexing and retrieval stages, including an evaluation of update efficiency when adding new documents incrementally.

Response: We conducted additional experiments by incrementally inserting five documents, with token counts ranging from 41,224 to 73,989. The results clearly demonstrate that LightRAG consistently outperforms GraphRAG in terms of both time and space efficiency. For document indexing, LightRAG exhibits near-linear scalability, with insertion times between 418 and 561 seconds. In contrast, GraphRAG shows significantly longer insertion times, ranging from 642 to 953 seconds, indicating a heavier computational overhead due to its community detection mechanisms.

During the retrieval phase, LightRAG achieves an average query time of 11.2 seconds, which is less than half of GraphRAG’s 23.6 seconds. This advantage is attributed to LightRAG’s lightweight keyword-based retrieval approach. Moreover, LightRAG’s final storage usage is only 39.5MB, in stark contrast to GraphRAG’s 286.7MB, highlighting its superior efficiency in managing large-scale data. These findings reinforce LightRAG’s advantages in scalability and resource efficiency when handling dynamic and complex datasets. Detailed experimental results are presented as follows:

4. Further discussion on potential limitations or scenarios in which graph-based indexing may underperform compared to traditional methods.

Response: In LLM-empowered retrieval-augmented generation (RAG), graph-based indexing, while effective in capturing comprehensive context from textual data, may have limitations in certain scenarios. For instance, with very small datasets or texts that lack complex context, the intricacies of graph indexing may outweigh its benefits, making traditional methods more efficient. In cases where the data is sparsely connected, graph structures might not fully capitalize on their advantages, resulting in better performance from traditional text- or attribute-based retrieval methods, as comprehensive information may not be necessary for response generation.

Additionally, the initial cost of developing and optimizing graph structures can pose challenges in resource-constrained environments. Although constructed graph structures can effectively represent entity-wise interdependencies, there are still costs associated with knowledge graph generation.

In response to the reviewer's comment, we will incorporate this discussion into our revised manuscript.

5. Further discussion with examples to clarify how LightRAG handles queries that may not align well with the constructed graph structure, highlighting the system's robustness across diverse query types.

Response: LightRAG demonstrates robustness in handling queries that may not perfectly align with the constructed graph structure by leveraging semantic matching to bridge gaps between the query and the available graph data. This approach ensures that even if a specific concept is not explicitly represented during the graph construction phase, the system can still identify and retrieve semantically similar content.

For example, if the constructed graph contains nodes and relationships centered around "machine learning" but does not explicitly include the term "deep learning," LightRAG can still respond effectively. When queried about "deep learning," the system employs semantic matching to recognize that "deep learning" is closely related to "machine learning" and retrieves relevant content from the graph. This flexibility allows LightRAG to dynamically adapt to diverse query types, even when exact matches to graph entities or relationships are absent.

We are truly grateful for the insightful feedback from the reviewer. Below, we present our detailed responses to the comments received:

1. Additional experiments and performance analysis with more diverse LLMs in conjunction with LightRAG.

Response: In response to the reviewer’s comment, we evaluated LightRAG on the Mix dataset using GPT models of varying capacities to assess its performance across different model sizes.

As shown in the table below, LightRAG consistently outperforms NaiveRAG across all three tested models. Notably, GPT-4o-mini and GPT-3.5-turbo demonstrate comparable performance, while the advanced model, GPT-4o, achieves the highest results. This superior performance is attributed to GPT-4o’s ability to extract richer information during the indexing phase, facilitating more precise and fine-grained retrieval. These findings indicate that LightRAG's performance is partially correlated with the capabilities of the underlying LLM; more powerful models enhance both retrieval quality and response generation by extracting more comprehensive information.

2. Additional experiments for performance comparison with the suggested baseline methods.

Response: We recognize the importance of comparing our work with state-of-the-art methods such as FLARE, Chain-of-Note, and Self-RAG to ensure a comprehensive evaluation. However, we faced challenges in conducting these comparisons. Specifically, the source code for FLARE and Chain-of-Note was unavailable, and Self-RAG depends on locally fine-tuned models, which demand substantial time and computational resources. We aim to address these comparisons in future work to enhance the rigor and completeness of our evaluation.

Dear Reviewer cAFL,

We greatly appreciate your thorough and constructive critique of our manuscript. In response, we have implemented comprehensive revisions, including expanded methodology descriptions and additional empirical evidence to substantiate our research findings.

We would be happy to provide additional clarification should you have any further questions.

Dear Reviewer pdw5:

Thank you for your valuable feedback and the time you dedicated to reviewing our paper. We genuinely appreciate your constructive comments. In response, we have outlined our replies below:

1. Additional experiments and performance analysis on Closed QA benchmarks.

Response: In response to your comment regarding the assessment of performance on Closed QA tasks, we conducted an evaluation using the MultiHopQA dataset, randomly sampling 1,000 queries. Employing GPT-4 for the assessment, we compared the accuracy of LightRAG and GraphRAG. As shown in the table, LightRAG consistently outperforms GraphRAG, achieving higher accuracy and demonstrating superior efficiency in handling complex multi-document reasoning tasks.

2. New features added to LightRAG for graph database management.

Response: In light of your insightful feedback, we are excited to introduce new capabilities in LightRAG that significantly enhance graph database management. Users can now write directly to various graph databases, including Neo4j, which provides greater flexibility in data handling.

Furthermore, we have implemented a delete feature that allows users to effortlessly remove graphs as needed. These enhancements are designed to improve both the flexibility and efficiency of our proposed RAG workflow.

Thank you for your feedback. We have updated the manuscript to include the following dataset citation:

Tang, Y., & Yang, Y. (2024). MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries. arXiv preprint arXiv:2401.15391

We appreciate your thorough review and would be glad to provide any additional clarification needed.

Dear Reviewer pdw5

Thank you very much for your detailed and valuable feedback. We sincerely apologize for any confusion caused by certain aspects of our paper. We have carefully addressed each of your concerns and have provided additional explanations and experimental results to better illustrate the contributions of our work.

Please do not hesitate to contact us if you have any further questions. We would be happy to provide additional clarification.

Hi, We checked the current code and it does not contain anonymity violation. We do not think this is a reason for desk rejection. Best, Area Chair

Thanks for your prompt reply. In fact, after I posted this public comment, the authors quickly removed the personal information from the repo. However, this does not negate the fact that they violated the anonymity principle. I have several pieces of evidence to support my point.

1. The last update date of this anonymous repo conincides with the day I wrote this public comment.
2. Before writing this public comment, I took screenshot of the leaked personal information (evidence temporally taken down to avoid further anonymity issue)
3. And the current anonymous repo still contains the authors' affiliation HKU (https://sm.ms/image/SfgnwxjO9bW4NB7).

In conclusion, I believe this work clearly violates the double-blind policy. I sincerely hope the AC could reconsider this issue.

Best Regards