KGGen: Extracting Knowledge Graphs from Plain Text with Language Models

Thank you for your diligent assessment of our contributions. We address the weaknesses and limitations that you identified below.

Weaknesses

Could the resolution stage incorporate domain-specific constraints (e.g., medical ontologies) to reduce under-clustering of specialized terms?

Thanks for asking!

We conducted an experiment that found that while specifying an ontology allows the extractor to deterministically adhere to the ontology, it led to a less thorough extraction. Specifically, we ran the extractor over a family tree description and found that only 14/39 were identified with the family ontology whereas 39/39 related people were identified without one. This is currently the subject of a follow-up work, in which we are developing a version of KGGen that can ingest an ontology to better guide its extraction.

What is the throughput (tokens/second) of KGGen on 1M-token corpora, and how does cost scale vs. GraphRAG?

We currently do not have accurate cost numbers for GraphRAG aside from time required for extraction since GraphRAG makes LLM calls internally. However, we can provide this data for KGGen with Gemini. On a 1M-token corpus, here is the breakdown for cost and time, along with the throughput:

In MINE-1, why use an LLM for fact verification instead of human annotation? Could model biases affect scores?

This is a reasonable question. Scoring by hand is extremely labor intensive. However, we scored 60 questions by hand and compared our judgements to those of GPT-4o. We found a 90.2% agreement rate and a 0.80 correlation. Therefore, LLM-as a judge is fairly accurate for this application.

The authors claims KGGen has already enjoyed widespread community adoption. KGGen has received over 450 Github stars and has been downloaded as a python module over 11,000 times since its release in January. Thus, KGGen has already made an impact in the community that studies knowledge graphs. Does this violate the double-blind policy?

We deliberately redacted the package name. There are many python modules that have been downloaded 11,000 times in the last several months, so this is not a unique identifier of KGGen.

Limitations

Generalizability: Evaluation primarily on Wikipedia-style articles; performance on conversational or technical text is unknown.

We ran KGGen on some fictional works that contain dialogue as well. However, it is much more difficult to evaluate performance on non-factual genres. Moreover, the primary use-cases for knowledge-graphs involve factual text and databases. MINE-1 and MINE-2 incorporate a broad array of texts to help assess KGGen holistically. Mine 2 is evaluated on the public WikiQA dataset, which contains 20,400 questions based on 1,995 wikipedia articles, and the details of MINE-1 can be found below.

MINE1: Number of articles: 105 Mean Length: 592 Words Std. Length: 85 Words Min Length: 440 Words Max Length: 976 Words Topic Breakdown:

• Arts, Culture & Society: 24 essays
• Science: 27 essays
• Technology: 19 essays
• Psychology/Human Experience: 18 essays
• History & Civilization: 17 essays

Black-box dependencies: Entity merging relies on Gemini without error analysis (e.g., confusion rates for ambiguous entities).

This is a fair criticism, which is common to many LLM-applications today. We are committed to improving error analysis, and we believe that we will have more insight into the necessary steps towards this goal as KGGen enjoys more widespread adoption.

Thanks for your helpful suggestions for improvement. We address your concerns below, and we hope that you'll reconsider your score.

Weaknesses and Questions:

The paper's ultimate goal is to create more useful KGs. However, on the MINE-2 RAG task, KGGen only performs "comparably" to GraphRAG.

Although GraphRAG performed similarly on retrieval benchmarks, it does not conform to the basic tenet of knowledge graphs: a knowledge graph consists of subject-predicate-object triples, where the subject and object are nodes and the predicate is an edge. We chose 100 node-triple-node pairs from each of OpenIE, GraphRAG, and KGGen and manually evaluated the percentage of these triples that conform to the definition of a knowledge graph. The results were as follows:

KGGen: 98/100 GraphRAG: 0/100 OpenIE: 55/100

Despite GraphRAG’s comparable performance, it doesn’t extract something that closely resembles a knowledge graph, which is another major strength of KGGen over GraphRAG. As discussed in the paper, KGGen exhibits better scaling properties than GraphRAG as well.

What is the evaluation accuracy of the LLM-based scoring in the benchmark MINE-1?

This is a reasonable question. Scoring by hand is extremely labor intensive. However, we scored 60 questions by hand and compared our judgements to those of GPT-4o. We found a 90.2% agreement rate and a 0.80 correlation. Therefore, LLM-as a judge is fairly accurate for this application.

Providing a detailed comparison of efficiency and cost between different methods. AND Providing more detailed analysis to key modules of the proposed method.

This is a fair point. While OpenIE is not LLM-based and thus cost comparison is not applicable, we can provide some insight into how costs and graph characteristics scale for KGGen as well as GraphRAG. To provide some variety in terms of genre, we extract a KG using KGGen with Gemini from the novel Name of the Wind, by Patrick Rothfuss. As one can see, the number of relations grows sub-linearly, and the entity de-duplication ratio improves with scale, showing the effect of the clustering algorithm.

The costs for each phase of KGGen on the entire novel can be found in the table below. We hope that the reviewer will agree that these costs are quite reasonable given the corpus size.

For comparison, we also evaluated GraphRAG using the same corpus and model. Note that we do not have accurate token usage and cost data for GraphRAG since it makes LLM calls internally.

How to process knowledge conflict when creating KGs?

Currently we don’t have a general solution to this problem. Resolving knowledge conflict would likely require post-processing whose specifics vary based on the corpus.

Paying attention to the paper writing (repeated sections).

We apologize for this mistake. We forgot to reformat the paper until very near the deadline, and we were rushing to adapt the paper for the NeurIPS template: mistakes were made. While we are not allowed to upload a new manuscript during the rebuttal period, please be assured that if our work is accepted, we will ensure that the writing is cleaner and limitations only appear once.

REBUTTAL:

Thank you for the positive view of our work. We are glad that you appreciated the advantages of KGGen over past methods!

We hope that we can address some of your remaining concerns below:

W1: Both the evaluation approaches rely on LLMs which introduces the concern of hallucinations and bias.

That is a fair point. However, potential bias is a drawback of any method that uses LLMs or even NLP, so it is not a unique weakness of KGGen.

W2: While the work states the corpora size, it is the first occurrence of this and there is missing context on the corpora, its contents and how KGGen generalizes with the growth of the corpus/corpora.

Thanks for pointing this out. MINE-2 was evaluated on the public WikiQA dataset, which consists of 20,400 questions about 1,995 Wikipedia articles. The details of this dataset can be found publicly. MINE-1 was evaluated on 105 synthetically generated articles. We provide some general statistics about these articles and a breakdown of their topics below: Mean Length: 592 Words Std. Length: 85 Words Min Length: 440 Words Max Length: 976 Words Topic Breakdown:

• Arts, Culture & Society: 24 essays
• Science: 27 essays
• Technology: 19 essays
• Psychology/Human Experience: 18 essays
• History & Civilization: 17 essays

Q1: Does KGGen’s core components generalize better across other LLMs (Claude, GPT)?

To answer this question, we measured the performance of KGGen on MINE-1 using 2 other models:

As you can see, the KGGen generalizes well across many different foundation models.

Q2: What are some challenges observed with building KG from text that is domain-specific? Has this been explored by the authors? Domains like medical, financial can be quite challenging to build a KG on. This can also be thought of as a limitation since building a KG requires domain knowledge and often times the context human-evaluators/annotators carry.

MINE-2 aims to address this question by testing KGGen on factual articles that span many different topics. KGGen performed well on MINE-2, demonstrating its capability across domains. We agree that sometimes human domain-experts will outperform LLMs on their areas of expertise, but this is an inherent challenge in building a KG-extractor that generalizes to arbitrary text. Unfortunately, there are not enough domain experts to construct large-scale and comprehensive KGs, leaving the task up to methods like KGGen.

Q3: What happens if the clusterings are incorrect or inaccurate?

During the clustering process, sub-nodes of each cluster are stored. Therefore, if one suspects that a cluster contains an inappropriate member, one can easily remove and correct the error.

Q4: Does KGGen take into account ontologies or ontology-aware extractions?

Thanks for asking this question!

We conducted a preliminary experiment that found that specifying an ontology allows the extractor to deterministically adhere to the ontology, but led to a less thorough extraction. Specifically, we ran the extractor over a family tree description and found that only 14/39 were identified with the family ontology whereas 39/39 related people were identified without one. For this reason, a version of KGGen that allows for pre-specified ontologies is currently the subject of a follow-up work.

Thank you for taking the time to review our paper. While we appreciate and will integrate your suggestions for improvement, we disagree with your overall characterization of our paper. We address the identified weaknesses and questions below, and we hope that you'll reconsider your score.

The datasets introduced are not well described. Some statistics about the datasets would be useful.

We are not exactly sure what statistics would be interesting to readers. To describe our datasets better:

MINE1: Number of articles: 105 Mean Length: 592 Words Std. Length: 85 Words Min Length: 440 Words Max Length: 976 Words Topic Breakdown:

• Arts, Culture & Society: 24 essays
• Science: 27 essays
• Technology: 19 essays
• Psychology/Human Experience: 18 essays
• History & Civilization: 17 essays

Mine 2: Mine 2 is evaluated on the public WikiQA dataset, which contains 20,400 questions based on 1,995 wikipedia articles. We are happy to provide more specific details about this dataset from Microsoft if the reviewer is interested.

The experimental evaluation is done by verifying facts. It would also be good to evaluate the actual triples extracted. In fact, it is not clear why the authors can't evaluate this approach on existing knowledge graphs.

We can’t evaluate this approach on existing knowledge graphs because there is no good metric for whether two knowledge graphs are similar. As you point out in your review, there are no knowledge-graph foundation models due to lack of data, which leaves embedding similarity out of the question. While there exist methods of graph embeddings, these only compare graph structure without taking node-level data into account. Moreover, human-labeled knowledge graphs are not infallible: it is well-known that even the largest human-labeled knowledge graphs, such as WikiQA, have millions of missing nodes and relations. Fact-verification is an efficient and objective way to measure a KG-extractor’s effectiveness. We also note that past works, like GraphRAG and OpenIE did not perform verification via triples either. In fact, our evaluations are already far more extensive than past works in the field, which generally verify based on metrics like information-compression.

The results of the proposed approach are not better than those of one of the baselines, despite the improved scalability of the proposed approach.

We first note that improved scalability is very important for a method to be practical, and should not be downplayed.

Although GraphRAG performed similarly on retrieval benchmarks, it does not conform to the basic tenet of knowledge graphs: a knowledge graph consists of subject-predicate-object triples, where the subject and object are nodes and the predicate is an edge. We chose 100 node-triple-node pairs from each of OpenIE, GraphRAG, and KGGen and manually evaluated the percentage of these triples that conform to the definition of a knowledge graph. The results were as follows:

Despite GraphRAG’s comparable performance, it doesn’t extract something that closely resembles a knowledge graph, which is another major strength of KGGen over GraphRAG.

The paper is not very well written. The Limitations section is included in two places.

We apologize for this mistake. We forgot to reformat the paper until very near the deadline, and we were rushing to adapt the paper for the NeurIPS template: mistakes were made. Please be assured that if our work is accepted, we will ensure that the writing is cleaner and limitations only appear once.

Why couldn't you evaluate the proposed approach on existing triple datasets that are manually extracted?

Manually extracted KG’s are not unique, as two different humans will likely extract different KGs from the same text. Even if there were agreement on the “correct” KG for a given text, it is very difficult to measure similarity between two knowledge graphs given that there are no foundation models to provide embeddings.

Sentence: "The ambitious Eurasia exhibition arose from an idea by Achille Bonito Oliva."
Target Entities: 'exhibition', 'idea'
KGGen Entities: ['Eurasia exhibition', 'idea', 'Achille Bonito Oliva']
Result: Success (both target entities captured, with KGGen providing more specific entity identification)