Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study over Open-ended Question Answering

We sincerely thank the reviewer for the thoughtful comments. Below, we address each concern in detail:

W1: The reviewer is concerned that the observation (2) that CoT and SC may cause bias and hallucination may not be consistent across different models.

Our findings (from line 460 to 467) show that using internal reasoning strategies like chain-of-thoughts (CoT) and self-consistency (SC) enhances response quality (i.e., correctness, comprehensiveness and empowerment), but do not consistently improve factuality and can sometimes reduce it. As the reviewer mentioned the performance over llama-3.1-8b is a case that using cot and sc improve the response quality but diminish the factuality. We appended the pivot table using the llama-3.1-8b model as follows for easier comparison (G-Eval measures to the quality of response, and FactScore measures the factuality).

W2: The reviewer is concerned about the statistical details of the human-in-the-loop process, such as evaluators' educational background, and the final score distribution agreement between humans and LLMs.

We have three evaluators participating in the manual assessment of query quality. All of the evaluators are computer science majors with fluent English skills. As the evaluation centers on various linguistic metrics such as naturalness, relevance, specificity, novelty, and actionability, we only require the evaluators to possess a fundamental understanding of English without restricting their majors. To maintain confidentiality, we will not reveal the identities of these evaluators during the rebuttal period.

The Pearson correlation coefficients between human and LLM scores are as follows:

As the rounds progress, agreement between humans and LLMs increases, suggesting that iterative feedback improves alignment.

W3: The reviewer is concerned that there still need more kinds of perturbation methods for OKGQA-p setting.

We have included more perturbation settings to mimic more real-world scenarios, such as (1) entity addition/removal: randomly adding or removing entities from KG to simulate incomplete or overpopulated KGs; (2) entity ambiguity: introducing misspelling, synonyms, or abbreviations to represent user query errors or inconsistent data entry (which could cause the query cannot be retrieved from the KGs as you mentioned). We have included the new experiments in the revised version.

W4: The reviewer mentioned several papers that need to be discussed and summarized in the paper.

We will extend our literature review to include the suggested references.

Q1: The reviewer want to see more details of the metrics used in Lines 292 to 295.

We utilize the G-Eval[1] framework with GPT-4o to evaluate metrics such as context relevance, comprehensiveness, correctness, and empowerment. Prompts and scoring rubrics are detailed in Appendix A.3 (Lines 835–858).

[1] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (https://arxiv.org/abs/2303.16634)

Q2: Can FS-SG reduce hallucination in LLMs using highly perturbed KGs, as shown in Figure 4 and 5, specifically at perturbation level=1.0?

Figures 4 and 5 demonstrate that KG-derived information effectively reduces hallucinations at perturbation levels up to 50%. Beyond this, performance declines due to severe KG contamination and performs worse than the baseline chain-of-thoughts. In practice, platforms like Wikidata undergo regular updates and community-based quality control, making such high perturbation levels less likely.

I appreciate the authors' response. However, I still have some concerns:

1. I am particularly curious about statistical metrics for human agreement, such as Cohen's Kappa coefficient.

2. It might also be helpful to update the revised PDF with relevant literature to provide a clearer understanding.

Based on these points, I am inclined to maintain the current score.

Dear Reviewer AR6A,

Thank you for your efforts reviewing this paper. Can you please check the authors' response and see if your concerns have been addressed? Please acknowledge you have read their responses. Thank you!

I appreciate the author's response. The current statistical information provides a deeper understanding of OKGQA and OKGQA-P. However, I still consider it to be a borderline paper, and I will maintain my current rating.

We sincerely thank the reviewer for the positive comments. Below, we address your concern in weakness in detail:

W1: The reviewer suggests exploring the generalizability of OKGQA-P’s findings and providing deeper analysis of method performance and limitations.

Our findings on OKGQA-P (mostly in Figures 4 and 5) illustrate that the effectiveness of KG-derived information diminishes with a perturbation level at 50%, surpassing this level leads to a further decrease in performance. We think that before this perturbation level at 50%, incorporating external knowledge from KGs can mitigate hallucinations in LLMs compared to baseline using CoT. In practical scenarios, platforms like Wikidata are less likely to be severely perturbed than 50% due to ongoing updates and community-based quality control, ensuring the relevance of our findings in real-world applications.

In addition, we find that using subgraphs can consistently outperforms using triplets, and paths, which demonstrates that by adding more structural information (or context), can improve the robustness as the perturbations increase. This aligns with our intuition that even when some links in the KG are disrupted, LLMs can still implicitly acquire knowledge through the graph’s topology or the shared semantics of neighboring nodes.

Dear Reviewer uGAz,

Thank you for your efforts reviewing this paper. Can you please check the authors' response and see if your concerns have been addressed? Please acknowledge you have read their responses. Thank you!

We sincerely thank the reviewer for the thoughtful comments. Below, we address each concern in detail:

W1: What are the unique contributions and distinctions of the proposed OKGQA compared to existing benchmarks?

OKGQA is, to the best of our knowledge, the first benchmark specifically designed to evaluate LLMs enhanced with knowledge graphs (KGs) in open-ended KGQA scenarios. It extends traditional closed-ended question-answering benchmarks to an open-ended setting, enabling the assessment of LLM hallucination tendencies.

Key distinctions of OKGQA include:

• Focus on open-ended scenarios: Unlike closed-ended benchmarks (e.g., WebQSP, CWQ), OKGQA evaluates models where responses are not restricted to a predefined set of entities, relations, or logical forms, providing a broader evaluation of hallucination tendencies.
• Integration of hallucination metrics: Standard metrics such as FActScore (Min et al., 2023) and SAFE (Wei et al., 2024) require open-ended responses, which OKGQA supports by phrasing questions as statements that demand longer, more detailed answers.
• Perturbation-based setup (OKGQA-P): OKGQA-P introduces deliberate perturbations to KGs, mimicking real-world scenarios with noisy or inaccurate knowledge graphs, thereby testing model robustness under less-than-ideal conditions.

By addressing the limitations of closed-ended benchmarks, OKGQA provides a more comprehensive evaluation of LLMs+KGs models in real-world, open-ended QA scenarios.

W2: Lack of discussion about the limitations and practical implications of referenced closed-ended benchmarks

As noted in the related work section (Lines 124–132), existing benchmarks for LLM+KG evaluation predominantly focus on closed-ended scenarios, where responses are restricted to predefined entities, relations, or logical forms. While useful, such benchmarks:

• Only test a narrow subset of an LLM’s tendency to hallucinate.
• Fall short in assessing complex, real-world scenarios requiring nuanced, open-ended responses.

In contrast, OKGQA addresses this gap by tailoring its design to open-ended KGQA, enabling the evaluation of hallucination tendencies under realistic conditions. For example, metrics such as FActScore and SAFE, specifically designed to measure hallucinations, require open-ended settings where answers involve richer reasoning and detailed responses.

By bridging this gap, OKGQA not only complements existing benchmarks but also provides a more practical framework for evaluating LLM+KG systems in real-world applications.

W3: Can KGs further reduce hallucinations in open-ended scenarios, even beyond RAG techniques?

While retrieval-augmented generation (RAG) systems can reduce hallucinations by retrieving unstructured information, their effectiveness is often constrained by the quality and specificity of the retrieved content. In contrast, knowledge graphs offer:

• Structured and explicit facts: KGs provide precise, disambiguated information, reducing ambiguity and enabling more robust reasoning.
• Logical reasoning capabilities: KGs excel in tasks requiring path traversal, contextually rich reasoning, and answering highly specific questions—scenarios where unstructured information retrieval may fail.

In open-ended QA, KGs can significantly enhance LLMs by ensuring factual grounding and logical coherence, surpassing the limitations of RAG-based approaches.

Q1: Why does the paper mention that perturbing the KG diminishes human comprehensibility?

We acknowledge that this phrasing was unclear, and we have revised it in the updated manuscript.

Our intention was to illustrate that KGs, typically annotated by humans, are generally accurate and meaningful. When introducing perturbations—such as mislabeled attributes, incorrect relations, or missing connections—they can disrupt the KG structure and mimic real-world scenarios where KGs may be noisy or of lower quality. These perturbations challenge the model’s ability to reason effectively with incomplete or inaccurate information.

We sincerely thank the reviewer for the thoughtful comments. Below, we address each concern in detail:

W1: How do the proposed G-retrieval and G-generator methods fundamentally differ from existing approaches like RAG and KG-augmented generation.

We would like to clarify that the primary objective of our work is not to propose state-of-the-art methods for close-ended or open-ended KGQA tasks but to establish a comprehensive benchmark to evaluate how well LLMs+KGs approaches perform under open-ended KGQA scenarios. The G-retrieval and G-generator methods are designed as simple baselines to represent two common workflows: (1) retrieving essential information from knowledge graphs and (2) subsequently leveraging that information to enhance the reasoning capabilities of LLMs during the generation process. We intentionally kept these methods straightforward to ensure clear ablation studies, focusing on whether integrating KGs can make LLMs more trustworthy.

W2 & Q1,2,3: How well do the generated queries authentically represent real-world questions, and how robust and reliable is the evaluation process, including the alignment between automatic and human assessments?

As mentioned in Appendix (Lines 861–894), we employed a human-in-the-loop process to ensure the generated queries authentically reflect real-world scenarios. The queries were refined iteratively based on human feedback, ensuring alignment with real-world use cases such as answering user-specific questions or conducting exploratory queries in knowledge graphs. The human feedback is reflected over multiple metrics, including naturalness, relevance, specificity, novelty, and actionability, to emulate characteristics of practical queries.

For the human evaluation process, three evaluators participated in the manual assessment of query quality. All evaluators are computer science majors with fluent English skills. Since the evaluation focused on linguistic metrics such as naturalness, relevance, specificity, novelty, and actionability, we required only a fundamental understanding of English, without restricting the evaluators by their academic background. To ensure consistency, a detailed rubric and illustrative examples were provided to guide the evaluation process. Each evaluator worked independently to ensure unbiased assessments.

For a sub-question “what is the correlation between the LLM’s automatic evaluations and human judgements of query quality? And how were discrepancies between the automatic scores ($s_auto$) and manual scores ($s_human$) addressed during the evaluation process?

We reported the score distribution agreement between the people and LLMs (we use pearson correlation coefficient to measure the agreement between human and LLM's labels) as follows:

It shows that as the number of human-in-loop rounds increases, the agreement between people and LLMs becomes closer.

W3: How do the four perturbation heuristics in OKGQA-P sufficiently represent the range of inaccuracies found in real-world knowledge graphs?

We appreciate the reviewer’s concern regarding the scope of perturbation heuristics in OKGQA-p. Identifying inaccuracies in real-world KGs is inherently complex, as they stem from diverse sources like link prediction errors, entity mislinking, and incomplete annotations. To address this, our benchmark focuses on link errors, which prior studies identify as the most prevalent and impactful type of KG inaccuracy.

The four perturbation heuristics—relation swapping, replacement, rewiring, and deletion—were carefully selected to represent key manifestations of link errors:

• Relation swapping simulates misclassified or mislabeled relationships.
• Replacement introduces spurious links to emulate noise.
• Rewiring reflects structural distortions in graph connectivity.
• Deletion models missing edges or incomplete knowledge.

These heuristics offer a practical and systematic foundation for studying the robustness of LLMs+KGs systems under common inaccuracies in KGs. While they may not capture every potential error type, they focus on the most statistically significant ones.