We sincerely thank the reviewer for their thorough comments. We have revised our manuscript and addressed your questions in the following way:

Q1: the authors speculate that the results are due to discrepancies in vocabulary; have they confirmed that?

We appreciate the reviewer’s comment and thank them for pointing out this weakness. To address the reviewer's concern, we chose to conduct further analysis to support our speculation on the effect of vocabulary discrepancies on the performance of the system. We conducted a simple analysis on the correlation between the ontological relevance of the statement (i.e. how many ontological concepts are detected in the statement) and the performance of OntoRAG on evaluating said statements. The results are given in the following table.

The results indicate an overall positive and strong correlation between ontological relevance and downstream performance. We again thank the reviewer for their question, and hope this addresses this point.

Q2: What is the verification process "to ensure fidelity to the source text" described in Section 4.1?

thank you for this, indeed we have not made it very clear. We have updated the manuscript to make it clear that this process is a string matching operation, to ensure all the concepts proposed by the LLM are indeed sourced from the papers given as context, making it possible to detect and remove hallucinations.

Q3: how are discrepancies handled when the above verification process encounters them? Can the authors give a specific example of how a discrepancy might be handled?

Since the verification process involves a string matching over the original text, discrepancies are handled by simply discarding the term from the vocabulary if it is not found. For instance, if the original text contains the term "carbon dioxide" but the LLM hallucinates "CO2", the latter term will be discarded from the vocabulary, even if it is a valid synonym. This is done as a countermeasure to avoid the introduction of hallucinated terms into the ontology.

Q4: What self-consistency techniques were used in Section 4.2?

We have included details in the Appendix, where we formally describe and cite self-consistency, and where we describe its application in our work:

"self-consistency is applied in the category generation step of Ontogen by generating multiple lists of categories and then taking the most frequent categories (i.e. the majority vote) as the final list. In the taxonomy extraction step, self-consistency is applied in the "query_relationships" function (see Algorithm 1 in Appendix C). In this case, a query is prompted $N$ times to the LLM (e.g. "Single-Atom Catalist isA ?"), and a taxonomic relationship (e.g. "Single-Atom Catalist isA Catalist") is extracted only if it is the answer in the majority of the $N$ queries (i.e. if a relationship appears in at least $(N + 1)/2$ answers).

Extra

Regarding some of the concerns you raised on the weaknesses sections, here are a few clarifications, that we would gladly incorporate in the manuscript if the reviewer finds it appropriate.

For the question "Can OntoGen produce ontologies that capture scient. significant patterns?" we have included slices of the generated Ontologies for SACs to show what patterns are encoded there, and analyze the patterns found there. We believe this analysis, along with the results from downstream applications (e.g. the SACBench) will be enough to show how good the results from OntoGen are.

We have also included the base LLM (under the name of ZeroShot) as one of out baselines in the updated manuscript. Thank you for pointing this out, it is indeed a clear miss in our original submission.

Finally, the biomedical benchmarks were run initially kind of as control experiments. No improvement in these would just show that using OntoRAG doesn't hurt, while it can help in tasks more related to scientific discovery, as we show with the SACBench.

Still, we have re-run the experiments on the medical benchmarks, this time also including the gene ontology, and we have updated the results as follows:

As can be seen in the new results, there's typically either an advantage or simply no improvement over the baselines (ZeroShot or CoT). With this, we update our analysis to account for this fact and make it clear that this works more as a control experiment.

I hope the reviewer finds these updates reasonable, and we're open to hear more feedback to improve our work.

    "Because the LLMs-generated ontologies cannot utilized to enhance the LLMs directly without the human’s curation, since the hallucinations still exist when generating the ontologies with LLMs, the manual validator is needed."

We sincerely thank the reviewer for their thorough and insightful comments. We have carefully considered your comments and have made appropriate changes as we describe below, along with answers with your questions and clarifications.

Questions:

Can authors add additional benchmarks that show improvement in performance?

Indeed, the original results we show were not a clear improvement over the baselines. From the time of submission of the original work, we have changed prompts and slightly optimized the pipeline. Our results for the same benchmarks are now as shown in the following results table:

As shown in the new results, OntoRAG performs typically on par or better than the zero-shot LLM, or even a CoT baseline. However please note that these benchmark experiments were included more as control experiments, to show that the use of OntoRAG does not deteriorate performance. The main results however are those on SACBench, which we augment and expand also in the updated version.

How does OntoRAG handle conflicting information from different retrieved sources within a single ontology or between multiple ontologies?

This is indeed a very good question, and we think it would be worth considering in a follow-up study. Indeed, conflicting information can be retrieved from different ontologies. However we argue that retrieved information from an existing ontology should be self-consistent (in the sense of [1]), already preventing these situations from their creation. In the case of ontologies generated with OntoGen, these situations are prevented by using a series of verification and control steps, where responses are required to be self consistent, as well as matching with the retrieved literature. We have elaborated and explained more of this in the manuscript thanks to this and other reviewer's comments.

How does the quality of automatically generated ontologies compare to expert-curated ones in established fields?

This is a very good question. We believe it would indeed be very interesting to e.g. generate a new gene ontology from collected papers in the field, however we note that the extensive amount of literature that has been produced for this field, and thus a massive number of terms associated, makes it a much more challenging undertaking, as compared against the field of SAC, and thus a comparison against the Gene Ontology is rather unfeasible. The question highlights a very good point however, and as such we have included a new analysis section in the Appendix, where we display slices of the generated SAC ontologies (by different LLMs) and include assessments of their quality by chemists experts in the field. We hope this update will address this concern, and we're very much looking forward to your comments on this.

Sensitivity OntoRAG to the specific choices made in the ontology retrieval and fusion steps?

This is a great question, that we tried to directly address in form of ablations in our original submission, however the presentation was not very clear. With the updated results (Table above) we can also respond to this question by looking at what each variation of ontorag is. In particular, we ablate the fusion step using two variants of fusion, namely simple and tm, which stands for "translation module". The method simple, consists of providing all the ontological context as a simple string in json format containing all information. The tm is an intermediate module that summarizes the raw ontological context, and is made to distill it down to the relevant information for the query. Although in this aspect the results are not very conclusive, there seems to be a net positive effect of using tm on these benchmarks. In fact, the most clear trend that we find here (and also in the SACBench results, that we are adding to the updated manuscript) is that OntoRAG+HyQ benefits the most from tm, while OntoRAG-simple works better without tm.

Extra

To your additional concern regarding increased overhead and computational cost, our method is indeed more costly than simply using ZeroShot inference. However it's important to note that the ontology is only generated once, and the goal of it is to condense a lot of the state of a field, including concepts and relationships. In that sense, the ontology can continue being useful for other tasks, without needing to carry this overhead for every use.

We hope our responses have addressed your concerns, and we stay open to further discussion if needed and appreciate any further feedback.

[1] ArXiv, abs/2203.11171

We sincerely thank the reviewer for their thorough and insightful comments. We have carefully considered your comments and revised the definitions in our manuscript accordingly. Below we present the revised definitions, and we hope we can iterate on this and give them an ideal shape for our work.

Ontology

For the definition of ontology, we now avoid the distinction between axioms and relationships. This aligns more with the implementation of ontology used in our work. In addition we have improved the notation by using $i:C, C \in \mathcal{C}$ to denote that instance i belongs to class C. Finally, we specify what are object properties (when defining I) and data properties (when defining P). Please find the full updated definition here:

---

---

An ontology is a tuple $\\{ \\mathcal{C}, \\mathcal{R}, \\mathcal{I}, \\mathcal{P} \\}$ where:

• $\\mathcal{C}$ is a set of classes $\\{ C_{1}, C_{2},...,C_{n} \\}$ present in the ontology.

• $\\mathcal{R}$ is a set of relationships present in the ontology.

  $\\mathcal{R} = \\{ (C_{i}, r_{s}, C_{j}) | r_{s} \in \\mathcal{R}_s \\}$, where $\\mathcal{R}_s$ is the set of all possible relations (object properties).

• $\\mathcal{I}$ is the set of all instances of classes present in the ontology

  $\\mathcal{I} = \\{ i_{1},i_{2},...,i_{m} \\}$; $i:C, C \in \\mathcal{C}$.

• $\\mathcal{P}$ is the set of all possible properties in an ontology.

  $\\mathcal{P} = \\{ p_{1}, p_{2},...,p_{l} \\}$ and $p:\\mathcal{I} \\xrightarrow{}\\mathcal{V}$ or $p:\\mathcal{C} \\xrightarrow{}\\mathcal{V}$; where $\\mathcal{V}$ is the set of all possible values for a property (data properties).

---

---

Regarding our definition of RAG, while we took it from reference [1], it is indeed inaccurate for our implementation and we have modified it as follows to correct for this. It is now formulated in terms of retrieval function $R$ and fusion operator $F$. This way we directly address the fact that we only take a limited number (k) of retrieved documents, and perform no weighting on them. The probability of generation of a response $y$ is then directly affected by a single prior $F(x, R(x) )$. Please find our updated definition below:

---

---

p(y|x) = p_\\theta (y|F(x, R(x))). (1)

with

$R(x) = \arg \max_{z\in Z} ^k \{r(z, x)\}$. (2)

Where:

• $p(y|x)$ is the probability of generating output y given input x.
• $R(x)$ hence defines a set of the $k$ most relevant documents to $x$ under relevance function $r$.
• $r$ is a document relevance function, such that $r(z, x)$ quantifies the relevance of document $z$ to query $x$.
• $F$ is a fusion operator.
• $p_\theta (y|w)$ is the probability of generating $y$ given context $w$ for a language model parameterized by $\theta$.

---

---

The definition for OntoRAG thus changes as follows:

---

---

$p(y|x) = p_\theta (y|F(x, R(x), R_O(x)) )$. (3)

with

$R_O(x) = \{ O(c): c \in C(x) \}$. (4)

Where Eq. 1 is modified in Eq. 2 to include:

• $R_O(x)$ is the ontological context relevant to query $x$, which depends on:
• $O(c)$ is some ontological context retriever, and
• $C(x)$ is a set of concepts found in text $x$.

---

---

We will address the other points you have raised in another response. Thank you very much again for your feedback, we would greatly appreciate your thoughts on these revised definitions, to improve our paper.

References

[1] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. ArXiv, abs/2005.11401.

We sincerely thank the reviewer for their thorough and insightful comments. We have carefully considered your comments and revised the definitions in our manuscript. Below we address the questions you have posed, hoping to improve the quality and clarity of our work:

Q1: OntoRAG and OntoGen vs existing DRAGON-AI and LLMs4OL?

Thank you very much for highlighting these works, indeed we have missed to reference them, and we would like to clarify the differences between our work and these others.

On the one hand, LLMs4OL focuses on the evaluation of LLMs rather than the generation itself of ontologies. In this work an LLM is prompted to assess whether a term is a subclass of another, without additional context. DRAGON-AI on the other hand tackles the task of inserting new terms into an existing ontology, so it requires an existing ontology. Our work, particularly OntoGen, takes a set of documents as an input, and generates an ontology by selecting and interconnecting terms into an ontology.

We address this in our updated manuscript by directly mentioning what they do and how our work is different, in the Introduction section. In particular, we add the following lines:

"Tools have been proposed recently to accelerate this process by inserting new terms into an already existing ontology Toro et al. (2023); Funk et al. (2023), or automating tasks such as term typing, taxonomy discovery, etc, under the frame of Ontology Learning Ciatto et al. (2024); Toro et al. (2023); Babaei Giglou et al. (2023). However, none of these works has attempted to generate full ontologies. To address this we propose OntoGen, an LLM-based method for automatic end-to-end generation of ontologies."

We hope this helps clarify the novelty issue you have raised.

Q2: Details of verification process and manual effort for LLM-generated ontologies?

Thank you very much for your comment, indeed some of these details have been left unspecified. For both the verification process and the comment on manual effort, we have added new sections to the appendix that elaborates further on the role and details of these processes. To clarify here, the verification process consists of a string match that checks whether each of the terms in the current list, indeed exists in the documents that were provided to the LLM as context.

Regarding the comment on manual effort, we clarify that this is only made with the "seed" terms used to initialize the taxonomy. This seed list has only few terms as this is automatically extracted and is composed of only the most common terms extracted by the LLM; in the case of the SAC ontology this was around 7 terms, namely Characterization, Physical properties, Synthesis methods, Reaction mechanisms, Structure, Applications, Reactions and Support. The "manual curation" we perform in this step involved selecting the following additional categories from the pool of generated categories, so as to make the ontology more aligned with our chemistry knowledge: Catalytic performance, Preparation methods, Theory and modelling , and Materials.

This is, the manual effort that is expected here to exclude one or more categories from the generated list or include additional categories if needed. Notice that this does not involve manually refining the whole taxonomy, but just the set of terms from the initial seed.

We hope this can help clarify any concerns regarding human involvement. Indeed, no extensive human curation is required at any point, only shortly at the early stages for more complete results. Regarding hallucinations, countermeasures have been taken through the generation process so as to minimize their impact. Just to recap, a verification step is performed after vocabulary extraction, while self-consistency is applied both during category generation and taxonomy extraction.

Q3: Analysis of the results that are reported in Table 2 and Table 3 and highlight how much the OntoGen and OntoRAG contribute to the final results?

As the reviewer noted, our manuscript falls short to analyze these two tables and the effect of these 2 components in the final results. Thank you for pointing this out. In our updated manuscript we conducted additional experiments to assess the effect of variations of OntoRAG (Table 2). In particular, we also add the final results on these benchmarks to assess the effect of the fusion operator (F) with two variations: concat, and translation-module. We append the table in a follow-up comment.

For Table 3, we again conducted additional experiments on a fixed version of OntoRAG, but using different ontologies (generated with different LLMs in OntoGen), to assess the effect of this part. In addition we have included more of the SACBench metrics for the sake of completeness, and we further analyze this in the manuscript.

We thank again the reviewer for their insightful comments, and look forward to productive discussion.