ORAG: Ontology-Guided Retrieval-Augmented Generation for Theme-Specific Entity Typing

• The proposed framework consists of two main techniques: (i) Structured Retrieval Memory Construction (Section 3.2) and (ii) Ontology-Guided RAG (Section 3.3). Unfortunately, this paper does not investigate the effectiveness of each technique. Not a few readers would like to know to what extent each technique contributes to performance improvement.

1. The experiment setup does not quite reflect the problem statement. The problem statement aims to solve fine-grained entity typing under “”highly-specialized and fast-evolving themes” and yet, the experiment setup requires an information-rich ontology with type description from specific dictionaries and encyclopedias, aliases, KG triples, etc. as outlined in section 3.2.1. How is it possible to have this complete set of knowledge for a “fast-evolving theme”? It seems the paper makes a strong assumption that such information is readily available for emerging entity and entity types. In practice, it will be hard to satisfy this assumption.
2. The experiment section is not extensive.

• I understand that it is time consuming to curate annotated data, I’d still expect evaluation to be done on more datasets.
• Lack of experiments with open-sourced LLMs. The paper only used GPT-3.5 and GPT-4 for experiments which will be hard to reproduce. It’s worth exploring similar techniques for other open-sourced LLM so that experimental results can be reproduced.
• Lack of ablation studies. For example, what happens if the “Semantic-rich sub-unit” is removed from the ontology, or the descriptions and the aliases are removed? Having an ablation study would help to better understand the contribution of each piece of information in RAG.

After rebuttal The authors have addressed my questions. I have updated my scores accordingly.

There are several issues with the manuscript that reduce my enthusiasm for it.

First, the evaluation corpora are quite small: each is only 20 documents and the total number of entities (I think they mean mentions) is only 300. Thus, even with the substantial performance improvements described, these evaluations should not be considered definitive, even for the two domains described. The results for "hard cases" are even more uncertain, since they are based on only 20 mentions for each domain. I believe error bars on the evaluation (Tables 2 & 3) would help substantially, but even so, I think this work is best described as a pilot study - a successful one that clearly suggests the approach is worth large-scale evaluation, but still a pilot study.

Second, only two domains are considered. There are many others which this method should help, such as chemical, biological, and clinical research, which have substantial datasets that should be adaptable to FET. It is possible, however, that the authors do not consider these to be sufficiently specialized or fast moving. In any case, with only two domains considered, it is likely that there are substantial issues in some domains that would complicate its use or cause it to fail. Again, I think this work is best described as a successful pilot study.

It is also somewhat unclear in what applications this approach would be most useful. It requires a fairly comprehensive ontology, which must be created if it does not already exist, which implies a substantial effort. This effort itself would require substantial time and expense to evaluate examples from documents, which suggests that at least in the case of domains that do not already have an ontology, the approach should ideally be compared to creating a small training corpus and fine-tuning an LLM. The approach also requires one LLM prompt per document, mention pair, so likely cannot be applied to large-scale datasets with GPT-4: if we assume the prompt and response are 1000 tokens on average (750 words) and 100 mentions per document, then a dataset of only 10,000 documents would cost $45,000 (this analysis ignores the RAG component, which I assume would also cost tokens). Finally, fine-grained entity typing itself is probably best understood as an intermediate natural language processing task, one that would be useful in a pipeline but one that users are not likely to care about directly. The most straightforward application users might care about, building or expanding an existing knowledge base, would likely run into problems scaling. At the very least, it would be helpful to suggest some user-facing applications.

In addition to the evaluation being small, the manuscript provides only a very superficial description of the process for creating the evaluation datasets. Not described are: how the documents were selected, the annotation process (the number of annotators per document, how disagreements were handled, etc.), the annotation guidelines (what constitutes a mention and what does not), inter-annotator agreement, how the online game ontology was created (e.g. the number of entity types in the ontology is the same as the number in the corpus, suggesting that the ontology was created from the corpus), whether the knowledge units for the battery domain included all 20,000 entity types or only the 484 in the dataset. All of these are relevant for understanding the quality of the evaluation datasets.

The manuscript does not clarify how performance was measured. Is the performance for each mention (i.e., two mentions of "fruit platter" in a single document would count twice), or for each (document, mention text) pair, or something else? How are true positives calculated, given that each mention may be annotated with multiple entity types, each of which is hierarchical? The authors allude to a hierarchical evaluation in section 4 ("Stopping at correct but coarse grained...") but do not provide details.

Some may argue that the straightforward approach described is too incremental or obvious to be useful. While I know of related methods, I am not aware of any that combine ontologies with RAG so neatly and in a way that would be useful in a zero-shot setting. Thus, I specifically argue that any such argument against this manuscript should be accompanied by a citation of one or more works that allow an LLM to use RAG with an ontology in a zero-shot setting.

There are several clarifications or terminology changes that would be helpful:

Fine-grained entity typing (FET) is not as well-known as named entity recognition (NER) and entity linking (EL). A brief contrast between them would help the reader better understand the needs of the task. In this case, NER typically involves predicting span and rough entity type, while EL involves predicting a specific identifier from a controlled terminology or lexicon that is potentially very large (millions of entities). Fine-grained entity typing does not involve predicting the span, but does involve predicting a type from a set of types which are intermediate in size between the rough entity types of NER and the identifiers in EL. FET may also have a context dependency such that a particular aspect or role of the entity is emphasized, such as classifying a mention of a person that is a politician, business person and media personality as referring in context specifically to "person/politician"; in this sense FET may be more specific than even EL (see, e.g., Dai et al., EMNLP-IJCNLP 2019, Improving Fine-grained Entity Typing with Entity Linking https://aclanthology.org/D19-1643)

The authors use the word "entity" ambiguously, to refer to the type from the ontology and also to the text from the document that should be typed. The latter is more typically known as a "mention," and I strongly suggest updating this terminology since it makes the manuscript difficult to read and makes some of the descriptions unclear (e.g., doe the "entities" in Table 1 refer to mentions?)

The authors use the word "theme" where I believe that the field more typically uses the word "domain." This is not critical, since I don't think there is any ambiguity, but it would align the manuscript better with the field.

Finally, while the LLM literature uses the word "hallucination" rather liberally, this manuscript not only does not define it but uses it to describe almost any sort of error made by the LLM. This sort of imprecise language makes understanding the problems being addressed and analyzing the results more difficult. It also fails to make one of the strongest points in favor of this approach: Subbarao Kambhampati (e.g. https://twitter.com/rao2z/status/1772636256184520981) talks about LLMs being able to capture a distribution, but correctness is a property of an instance; by extension, "hallucination" is where the LLM returns something that sounds correct because it follows the distribution of the domain, but is not correct because the LLM has no knowledge of the entities within the domain. Thus, the approach described by the authors neatly allows the ontology to inform the LLM of the entities, which should dramatically reduce hallucinations, according to this definition, but the terminology used in the manuscript is not precise enough to support this point. (I'm also unsure whether the task would even support this sort of analysis.)

1. Evaluation is done on a very small dataset of 20 documents, raising questions on the generalizability of the results.
2. Even the choice of themes - a) redox-active organic electrode materials, and b) online game posts for Genshin Impact, seems arbitrary without proper justification on how learnings on those datasets showcase generalizability.