Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge Graphs

We sincerely thank you for your thoughtful and constructive feedback. We appreciate your recognition of the clarity of our writing, the novelty of validating relation paths through question constraints, and the effectiveness of our experimental results. We also value your insightful concerns regarding the manual definition of constraints, the scalability of our knowledge distillation process, and the fairness of model comparisons. Your comments have helped us better understand the limitations of our current submission and guided several improvements and clarifications in both the revised draft and the responses below.

In what follows, we address each of your questions and concerns in detail. We also provide additional experimental results to support our claims, clarify methodological decisions, and discuss possible directions for improving the generalizability of our approach.

W2 The primary goal of fine-tuning in the distillation stage is to enhance the faithfulness of LLMs in generating relation paths that align with the structural patterns of the target KG. This is critical not only because the subsequent instantiation module directly relies on these generated paths to retrieve and ground entities from the KG, forming complete reasoning trajectories, but also because high faithfulness is particularly indispensable in specialized domains such as healthcare and legal services—fields where the accuracy and trustworthiness of responses are paramount, as unreliable information can lead to severe consequences. Without such fine-tuning, while in-context learning (ICL) can indeed prompt LLMs to generate relation paths, their faithfulness is significantly compromised.

The root cause is that existing LLMs, despite broad knowledge, lack sufficient exposure to the specific structural nuances and relation semantics of a given KG. This leads to paths that are semantically plausible but structurally inconsistent with the KG, undermining downstream reliability.

Notably, our fine-tuning is designed to be cost-efficient. We employ LoRA during both SFT and KTO, which efficiently adapts large-scale parameters with minimal computational overhead. Furthermore, the training data for fine-tuning is automatically acquired: the weak supervision signals (question-to-path mappings) are extracted by identifying the shortest traversal sequences from topic entities to answer entities in the KG using the Dijkstra algorithm, eliminating the need for manual construction.

Empirically, our fine-tuning strategy yields substantial improvements in path generation quality: Table 4 shows that the Hit rate of our path generation reaches 94.0% after fine-tuning, confirming the enhanced faithfulness of the generated relation paths. This improvement directly translates to better overall performance, as reflected in Table 2, where our method outperforms ICL-based baselines by over 20 percentage points in F1 score across datasets.

Minor Issue: Thank you for pointing this out. We will carefully review the manuscript and revise any potential typos.

A1 Thank you for raising this important concern. We agree that including irrelevant paths during the distillation stage could introduce noise and affect the model’s reasoning quality. In fact, we have explored several filtering strategies during development, including semantic similarity filtering and LLM-based relevance prediction. However, we observed that these approaches tended to over-filter and remove semantically plausible but indirect reasoning paths. For instance, in the case of the question "What is the name of Justin Bieber’s brother?", both of the following paths can lead to the correct answer "Jaxon Bieber": 1.

["Justin Bieber", "people.person.sibling_s", "m.0gxnnwp"],  
["m.0gxnnwp", "people.sibling_relationship.sibling", "Jaxon Bieber"]

["Justin Bieber", "people.person.parents", "Jeremy Bieber"],  
["Jeremy Bieber", "people.person.children", "Jaxon Bieber"]

While the second path is logically valid and factually correct, it is often mistakenly filtered out due to low semantic overlap with the original question. As a result, these filtering methods, though effective at removing noise, risk eliminating valuable reasoning paths and reducing coverage of the answer space.

Although we ultimately chose not to apply a filtering mechanism during the distillation phase, this decision allows us to retain a diverse set of reasoning paths, including those that are logically valid but semantically distant from the question. We acknowledge that this choice inevitably introduces some irrelevant paths, which may interfere with downstream reasoning.

To address this, our framework incorporates path selection and constraint validation during the introspection stage. These components are designed to identify and promote paths that are both answer-relevant and constraint-compliant, thereby substantially reducing the negative impact of irrelevant paths on final answer generation.

A2 Thank you for raising this important point regarding model comparability. In our study, the main baselines we compare against are ToG, PoG, and DoG, as these works all follow the iterative reasoning paradigm similar to DP. These baselines represent the state-of-the-art in KGQA, and therefore serve as the most appropriate points of comparison for our proposed method.

To ensure fairness, we conducted experiments using ChatGPT(GPT-3.5) and GPT-4 as the backbone LLMs, aligning with the backbone choices in ToG, PoG, and DoG. These results are reported in Table 2 of our paper.

For clarity, we reproduce the relevant results from Table 2 below to directly compare the performance of DP with ToG, PoG, and DoG under the same backbone LLMs (W, C, and M denote the WebQSP, CWQ, and MetaQA datasets, respectively.):

A3 Thank you for highlighting this important issue regarding test set consistency. Indeed, we observed that the baseline methods vary in how they select their test sets—some use the full test set, while others use randomly sampled subsets. This inconsistency makes it difficult to directly align test samples across all prior works. In our case, due to the high cost of LLM inference, we sampled a fixed number of examples for evaluation:

• 500 examples each for WebQSP and CWQ,
• 600 examples for MetaQA. To ensure the fairness and reliability of our comparison, we selected DoG—a strong and representative baseline—and re-evaluated it on the exact same sampled test sets used for DP. This allows for a controlled, head-to-head evaluation using identical data. We present the comparison results below:

The "*" represents the original results in the paper, and "**" represents the results when using the same sampling set as DP. W, C, and M denote the WebQSP, CWQ, and MetaQA datasets, respectively.

As shown above, DP consistently outperforms DoG across all datasets when evaluated on the same sampled test sets, further validating the effectiveness of our method under controlled conditions.

A4 Thank you for raising this point. We would like to clarify that the performance of naïve LLMs was evaluated as part of our ablation study, specifically in Table 3, second row. This row corresponds to the setting where GPT-4.1 is used directly for question answering without any retrieval or reasoning augmentation, representing the baseline performance of a naïve LLM on these tasks. As shown in Table 3, the naïve GPT-4.1 model performs significantly worse than our framework on both WebQSP and CWQ datasets. This empirical result directly supports our claim that DP can enhance the reasoning reliability of various LLMs (W and C denote the WebQSP and CWQ datasets, respectively.).

We appreciate your attention to this aspect and hope this clarification helps demonstrate the effectiveness of our approach more clearly.

We sincerely thank you for your follow-up questions and for acknowledging the improvements in our rebuttal. Below, we provide our responses to each of the remaining points.

A1: Yes, this process can be automated. In fact, as shown in Table 3 of the paper (row “w/o CPD”), we have experimented with an automated approach where the constraint types are not manually predefined. Instead, we use a prompt-based method to have the LLM automatically infer constraint types from the input question. While this automatic variant slightly underperforms the version with predefined constraints, it still shows promising results (only an average 2.18% drop on WebQSP/CWQ). This may suggest that the manually defined constraint types align well with the specific characteristics of the benchmark datasets.

What's more, we have also provided a more detailed discussion on automated constraint extraction in our response to Reviewer iFUY.

A2: Thank you for pointing this out. To clarify, the computational cost for fine-tuning on 2x NVIDIA A800 (80GB) GPUs is: ~3 hours for the SFT stage (3 epochs) and ~2 hours for the KTO stage (1 epoch). Additional training details (e.g., learning rate, batch size, and LoRA config) are provided in Appendix B.1 (Training Detail).

We will include this information in the revised version to improve transparency.

A3: First, we need to point out the core difference in training paradigm between our approach and existing Supervised learning methods (e.g., RoG and AMAR):

• SL Methods: They directly fine-tune the LLM using gold answer labels provided in the KGQA datasets.
• DP: It does does not rely on any answer supervision. Instead, it uses only question–relation path pairs for offline training, guiding the LLM to generate faithful relation paths for KG-based retrieval. Importantly, in the online inference stage, including constraint extraction, path selection and constraint verification, no fine-tuning is applied—all steps are carried out by the LLM without additional training.

To ensure a direct and fair comparison, we followed your suggestion and aligned our base model with existing advanced SL methods. To maintain consistency with RoG and AMAR, which utilize LLaMA2-Chat-7B, we carried out additional experiments with DP using the same model.

This approach ensures that any performance differences can be primarily attributed to the framework's methodology itself, rather than to the use of a more advanced base model. We believe this is the most equitable comparison scheme under the current circumstances. Results on WebQSP (W) and CWQ (C) are:

As these results demonstrate, DP can achieve highly competitive performance without requiring any answer labels for training, relying only on relation path supervision.

A4: We now include the results of the naive prompting baselines using the LLMs mentioned. The table below reports their performance on the WebQSP and CWQ datasets using H@1 and F1 as evaluation metrics, under naive zero-shot prompting without fine-tuning or additional reasoning modules, which clearly shows that DP brings substantial performance gains to naive LLMs, confirming its efficacy.

Thank you for your careful review and constructive comments. We address your concerns point by point below:

Regarding the missing AMAR results on CWQ in A3: The AMAR method uses LLaMA2-13B for the CWQ dataset and LLaMA2-7B only for WebQSP, as reported in its original paper. Since our method uses LLaMA2-7B, we included AMAR’s WebQSP results for a fair comparison under the same model size. In Table 2 of our main paper, we see “LLaMA2-7B/13B” are employed in AMAR. Therefore, we omitted the CWQ results from AMAR in table A3 (in the official comment) to avoid comparing across different model sizes.

Regarding the absence of GNN-RAG: GNN-RAG is a hybrid method that combines graph neural networks (GNNs) with an LLM fine-tuning pipeline. Specifically, it fine-tunes LLaMA2-7B and also leverages additional GNN-based reasoning components. To ensure a fair and controlled comparison focused purely on LLM-based reasoning methods, we did not include GNN-RAG in table A3 (in the official comment).

On test set comparability: Thank you for pointing this out. Regarding the comparison with SL methods such as RoG and AMAR, we have re-evaluated our method (DP, with LLaMA2-7B) on the full official test sets of both WebQSP and CWQ, which aligns with the evaluation setup used by RoG and AMAR.

Below we provide the updated performance of DP under this consistent setup. We see that although DP is not trained using the training set, it also achieves SOTA results (evaluated by H). We need to clarify that DP is a few/zero-shot model from the answer generation view. The fine-tuning and reinforcement learning stages only employ the weak supervision to improve the faithfulness of relation generation. We do not use the answer label to fine-tune LLMs. In contrast, the baselines are fine-tuned using the answer label. We think these can demonstrate the superiority of DP to some extent.

We hope this clarifies the evaluation setup and appreciate your careful attention to the comparability of results.

We sincerely thank you for your thoughtful and constructive review. We greatly appreciate your recognition of the motivation and contributions of our work—particularly the trustworthiness of the DP framework, the practical value of reduced LLM calls, and the thoroughness of our experiments and ablations. Your comments and questions helped us reflect more deeply on both the strengths and limitations of the current design. Below, we address each of your concerns in detail and clarify several points that were unclear in the original version.

A1 You raise an important point regarding the scalability of our framework when constraint types are manually defined. We fully agree that enabling automatic constraint acquisition would significantly improve the practicality of our approach in real-world applications.

In fact, we conducted an ablation study (see Table 3, row “w/o CPD”) where constraints were not manually predefined. Instead, we used a prompt-based method to let the LLM automatically identify constraint types from the input question. While this automatic variant slightly underperforms the version with predefined constraints, it still shows promising results. This may suggest that the manually defined constraint types align well with the specific characteristics of the benchmark datasets. However, their effectiveness may not generalize to more diverse or open-domain settings, where more flexible and adaptive constraint extraction strategies would be required.

In future work, we plan to explore this direction through the following approaches:

• Leveraging prompt-based methods to guide large language models in analyzing representative questions and automatically identifying, abstracting, and generalizing common constraint patterns within a target domain.
• Exploiting the schema or ontology of the knowledge graph, which encodes valuable structural and semantic information (e.g., entity types, relation domains and ranges). By analyzing this metadata, the system can infer potential constraints. For example, if a relation’s domain is “Film” and its range is “Year,” any query involving this relation can be automatically linked to a constraint such as “the answer type must be a year.”

A2 In our setting, multiple shortest paths of equal length may exist between a given subject entity $e_s$ and answer entity $e_t$, and we account for this in our supervision process. For instance, for the question "What is the name of Justin Bieber’s brother?", both of the following reasoning paths can lead to the correct answer "Jaxon Bieber":

1. ["Justin Bieber", "people.person.sibling_s", "m.0gxnnwp"], ["m.0gxnnwp", "people.sibling_relationship.sibling", "Jaxon Bieber"]
2. ["Justin Bieber", "people.person.parents", "Jeremy Bieber"], ["Jeremy Bieber", "people.person.children", "Jaxon Bieber"]

To capture such diversity, we use Dijkstra’s algorithm to collect all shortest-length relation paths, rather than selecting just one. This helps us maximize the supervision signal and enables the model to learn from multiple semantically equivalent reasoning routes.

A3 While Table 3 shows that removing any of the three modules—constraint extraction, path selection, or constraint verification—results in a comparable performance drop, this does not imply that these modules contribute independently or equally to the overall performance.

Each of the three modules plays a distinct yet interdependent role in the DP framework:

• Constraint extraction provides semantic guidance by identifying meaningful conditions to be satisfied.
• Path selection ensures the reasoning process focuses on plausible paths that align with both the query and the extracted constraints.
• Constraint verification serves as a filtering mechanism that enforces answer validity by cross-checking constraints post hoc.

Although the degradation appears numerically similar across ablations, this is likely due to the fact that removing any single module breaks the coherence of the pipeline. For instance, without constraint extraction, the subsequent modules lack semantic grounding; without path selection, the search space becomes noisy and inefficient; and without constraint verification, spurious or weakly supported answers may be retained.

In this sense, the performance drops are not indicative of independent or redundant contributions but rather suggest that these components are synergistic—each is critical, and disabling one disrupts the end-to-end reasoning process.

A4 The higher token consumption observed with Qwen1.5-8B is mainly due to its verbose generation style. Qwen3-8B is optimized for reasoning tasks and tends to explicitly articulate its intermediate reasoning steps, resulting in longer outputs compared to other models like GPT or LLaMA. This verbosity, while potentially beneficial for transparency and interpretability, naturally increases token usage during inference.

We thank you for your thoughtful and constructive feedback. We appreciate your recognition of this work.

A1 We agree that the function definition in Line 125 could be made more precise and readable. To improve clarity, we will revise the statement in Line 125 to："we define a function $\mathcal{M}$ that maps a question $q \in \mathcal{Q}$ to a set of plausible relation paths...". This change emphasizes that the function operates on individual questions, rather than on the full question set. The complete input to the function $\mathcal{M}$ is clearly specified in Equation 1: $\mathcal{M}(q, e_s, \mathcal{G}, k)$ .

A2 When the reasoning depth is not available, our framework defaults to a general backup strategy: we perform shortest-path search over the entire knowledge graph. Specifically, the parameter k in our implementation represents the maximum allowed reasoning depth and is typically provided by the dataset. Extracting a k-hop subgraph centered at the topic entity serves to narrow the search space, significantly improving efficiency during path extraction and instantiation. However, in the absence of such a predefined depth, we fall back to querying the full graph for shortest paths between the topic and target entities. While this ensures correctness, it results in increased computational overhead and longer runtimes due to the expanded search space.

A3 You are absolutely right that $\mathcal{P}_w(q)$ represents a set, and using boldface notation would improve clarity and consistency. We will revise the notation accordingly in the final version to better reflect its set nature.

A4 Yes, the DP framework does include a feedback mechanism that improves the effectiveness of KG retrieval, though not in the traditional sense of updating the retriever module directly.

DP transforms the triple retrieval into relation path generation. Specifically, it first generates multiple candidate relation paths and then instantiates them over the KG to ground the answer. In this architecture, retrieval is guided by structured reasoning paths rather than by query-keyword matching, and path selection becomes the central mechanism of retrieval.

The introspection module plays a critical role in enabling feedback. It evaluates whether the instantiated answers satisfy the pre-extracted constraints. If a candidate path leads to an answer that violates these constraints or lacks sufficient information, the system triggers backtracking to select an alternative relation path from the candidate pool. This mechanism functions as an implicit feedback loop that iteratively refines the path selection process to improve final answer quality.

Looking forward, such feedback could inform KG completion (e.g., when certain paths repeatedly fail to instantiate) or guide refinement of the path generator (e.g., when paths consistently violate constraints).

A5 In our framework, the training is conducted in two distinct stages: SFT followed by KTO. We do not alternate between them during training.

• SFT teaches the model to generate plausible reasoning paths from input questions in a supervised manner, establishing the basic question-to-path mapping.
• KTO then refines this ability by using preference data to distinguish high-quality paths from suboptimal ones, encouraging the generation of semantically coherent and structurally valid reasoning.

As shown in Table 3, removing KTO leads to a clear performance drop, highlighting its essential role in enhancing the robustness and faithfulness of the overall DP pipeline.

A6 We agree that the use of the term semantic alignment was not clearly defined in the original text, and we will revise the description to improve clarity.

What we intended to convey is that semantic alignment refers to the initial relation path selection step, where the model selects the most semantically relevant path to the input question from a set of candidates. At this stage, no verification feedback is available yet—the model relies solely on the semantics of the question and the candidate paths. In subsequent iterations (after backtracking), the model can incorporate feedback signals from the constraint verification module to refine the selection process.

We will clarify this distinction between initial and feedback-informed path selection in the revised version and refer to Appendix B.2.3 for further details.

A7 It is indeed one of the core innovations of our framework. Measuring how frequently backtracking is triggered in practice provides useful insight into how often the system encounters and resolves constraint violations during reasoning.

We analyzed our test-time logs and computed the average number of backtracking steps per question for each dataset, summarized below:

GPT-4.1 triggers backtracking more often than GPT-3.5, likely due to its stronger instruction-following ability, which leads to stricter constraint checking. In contrast, GPT-3.5 may rely more on internal heuristics and skip verification steps. This also corresponds with Table 2, where GPT-4.1 yields better performance.

A8 In the final version, we will add a dedicated case study section to highlight the significance of these examples, along with relevant references and brief discussions in Sections 3.2 and 3.3 to further illustrate their connection to the strengths and limitations of our framework.

A9 We appreciate your detailed suggestions regarding Table 2. Below we address each point in turn:

1. You are absolutely right that reporting the std-dev directly in Table 2 improves clarity. The reason we placed it in the appendix originally was due to space limitations in the main paper. In our revised version, we will integrate the std-dev values into the main table for better accessibility.

2. This is a very helpful suggestion. We have reorganized the table to group methods based on the underlying LLMs, which makes performance comparisons more intuitive. Please see the updated version below (W, C, and M denote the WebQSP, CWQ, and MetaQA datasets, respectively.): |Model|Method|H(W)|H@1(W)|F1(W)|H(C)|H@1(C)|F1(C)|H(M)|H@1(M)|F1(M)| |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| |GPT-3.5|ToG|76.2|-|-|58.9|-|-|-|-|-| ||PoG|82.0|-|-|63.2|-|-|-|-|-| ||DoG|88.6|62.6|54.2|58.2|49.4|56.6|95.4|85.1|87.2| ||DP|89.7|86.9|79.2|80.0|72.6|69.2|96.7|95.4|90.8| |GPT-4|ToG|82.6|-|36.4|69.5|-|31.8|-|-|-| ||PoG|87.3|-|-|75.0|-|-|-|-|-| ||DoG|91.0|65.4|55.6|56.0|41.0|46.4|98.3|90.1|93.1| ||DP|90.4|86.7|81.7|85.6|74.6|71.1|96.7|95.4|94.8|

3. Our main experimental setting uses GPT-3.5 and GPT-4, which are also the backbones used in baseline works like ToG, PoG, and DoG. This ensures a fair comparison under consistent model conditions. The additional experiments with GPT-4o, GPT-4.1, and LLaMA3.1-8B are primarily meant to validate the generalizability and compatibility of DP across a range of modern models, rather than to compare with prior baselines under few-shot ICL.

4. Thank you for pointing this out—we acknowledge this omission and will correct it in the updated table.

5. Yes, the DP pipeline does involve a certain level of randomness, mainly due to using a sampling temperature of 0.8 when invoking LLMs during path generation and introspection. However, as shown in Table 7, the std-dev across runs remains below 1.0, indicating that DP produces stable and reliable results despite this inherent stochasticity.

A10 We agree that DP has fine-tuning costs, but in this analysis, we focus on the costs associated with generation and inference.

The training cost is a one-time cost, whereas the inference cost is incurred repeatedly with each query. Therefore, the total cost for generating responses is a combination of both the training and inference costs. As the number of queries (n) increases, the training cost can be spread over more queries, which makes it less significant in comparison to the growing inference cost.

To clarify this, we use the following formula to express the total cost: $C_{\text{total}} = \frac{C_{\text{train}}}{n} + n \cdot C_{\text{infer}}$.

As $n$ increases, the inference cost becomes more important, and the training cost for each query decreases. This means that as the number of queries increases, inference cost becomes the main factor and becomes increasingly important over time. This gives DP an efficiency advantage in real-world scenarios.

A11 We appreciate your attention to the evaluation details. There may be a misunderstanding regarding the hit rate (H) reported for CE in Table 4. Here, CE stands for Constraint Extraction, and the H metric measures how accurately the model extracts constraints from questions—not answer correctness. Specifically, we annotated 100 examples with gold-standard constraints and computed how often the model’s output matched them. The high score (~99) reflects strong extraction accuracy on this subset and is not directly comparable to the H scores in Table 2.

A12 Our current DP framework uses a set of predefined constraints, which work well in controlled settings. However, enabling automatic constraint acquisition is essential for broader applicability. In future work, we plan to:

• Use LLMs to analyze representative questions and generalize common constraint patterns within a domain.
• Leverage KG schema or ontology (e.g., entity types, relation domains/ranges) to infer likely constraints. For example, if a relation’s range is “Year,” the system can enforce type consistency for answers.

Additionally, as noted in our response to Q2, the fixed reasoning hop limit is used only during training to bound the subgraph for efficient path extraction. The framework does not rely on a fixed number of hops and can naturally scale to large KGs with flexible or unbounded reasoning depth.

Thanks for your valuable and constructive comments.

A1 Our DP framework does rely on a set of predefined constraints that are universal and effective. In Tabel 3, we conducted an ablation study where constraints were not manually predefined (row “w/o CPD”), but instead a prompt-based approach was used to enable LLM to automatically identify constraint types from the input problem. While this automatic variant slightly underperforms the version with predefined constraints, it still shows promising results (the average decrease in the three indicators on WebQSP and CWQ is only 2.18%). This may suggest that the manually defined constraint types align well with the specific characteristics of the benchmark datasets.

We fully agree that automatically extracting constraints is a crucial step to ensure that the framework generalizes to more diverse and specialized domains. In future work, we plan to explore the following aspects:

• LLM-based In-Context Induction This method leverages the powerful instruction-following and pattern-learning capabilities of LLMs. By designing specific prompts and providing a few (few-shot) examples of "question-constraint" pairs, we can guide the LLM to learn and generalize the patterns for identifying corresponding constraint structures in new questions. For instance, by showing the model that "Who was the first person to walk on the moon?" corresponds to an "ordinal constraint," the model can learn to identify similar ordinal logic in other questions.
• Semantic Parsing to Logical Forms This approach involves translating the natural language question into an intermediate, machine-readable logical form (e.g., λ-Calculus or a SPARQL query sketch). The resulting logical form inherently and explicitly contains the core constraints of the question. For example, the question "Which films were directed by Ang Lee and starred Chow Yun Fat?" could be parsed into λx.(directed_by(x, Ang_Lee) ∧ starred_by(x, Chow_Yun_Fat)). By analyzing this logical form, we can directly and accurately extract the two "multi-entity constraints."
• KG Schema and Ontology Analysis This method directly utilizes the structured a priori knowledge defined within the KG itself. By analyzing the KG's schema or ontology, we can automatically extract the rules embedded within. For instance, if a relation award_year has a "domain" of "Award" and a "range" of "Year," then any question involving this relation can be automatically associated with a type constraint specifying that "the answer must be a year." This is a high-precision method, though its coverage is limited by the completeness of the KG schema.

In more complex domains such as law, finance, or medicine, a question's constraints are often more nuanced and implicit. The methods above can work in synergy to address these challenges:

• Handling Implicit Constraints: For questions like, "What public offices did Roosevelt hold before he became president?", the temporal constraint ("before he became president") is relative and implicit. LLM-based induction is particularly effective in such scenarios due to its superior ability to understand the contextual and temporal logic of natural language.
• Ensuring Logical Rigor: In domains requiring high precision, such as law or finance, semantic parsing can translate complex clauses or questions into unambiguous logical forms. This ensures all conditions are captured accurately, avoiding reasoning errors that can arise from the ambiguity of natural language.
• Leveraging Domain-Specific Priors: In specialized KGs (e.g., a biomedical KG), the schema is often highly detailed. KG schema analysis can extract a large number of high-confidence, domain-specific constraints (e.g., "Protein A participates in Biological Process B"), providing a solid foundation of domain knowledge for the reasoning process.

A2 Based on our experiments and case analyses (as illustrated in Figures 9 and 10), the failures of the DP framework during the path generation phase can be attributed to the following key factors:

• Subtle Structural Variations in Complex Reasoning Scenarios: In some challenging reasoning cases, incorrect paths may exhibit high structural similarity to the ground-truth paths, differing only in a critical relation. For example, in the failure case shown in Figure 9, the model-generated path is nearly identical to the correct one, but a minor deviation ultimately leads to reasoning failure. This suggests that although our progressive knowledge distillation strategy (including SFT and KTO) effectively equips the LLM with a strong understanding of structural patterns in the knowledge graph, it still struggles to distinguish between such "high-fidelity" incorrect paths. A potential reason is the lack of sufficiently diverse and contrastive negative samples, which limits the model's ability to capture decisive semantic differences.
• Error Accumulation in Multi-Hop Reasoning: For tasks requiring long-chain reasoning, minor deviations in early steps of path generation can compound and result in significant divergence from the correct path. Although the DP framework incorporates introspection and backtracking mechanisms for error correction, their effectiveness is constrained when the initially generated candidate path set is of low overall quality. In such cases, even the subsequent selection and verification processes may fail to identify a completely correct path.

Future Directions:

• Developing Minimal Yet Informative Negative Sampling: Rather than introducing overly complex or synthetic negative examples, we aim to design simple but impactful negative sampling strategies. For instance, we can mine naturally occurring “borderline” paths within the KG that deviate from gold paths by only one relation or entity. These naturally occurring “near-miss” paths not only reflect realistic ambiguity but also improve the model’s robustness in a lightweight manner. This approach balances discriminative training and implementation simplicity.
• Enabling Lightweight and Targeted Path Revision: Instead of overhauling the entire reasoning path upon constraint violation, we advocate for a modular, token-level correction mechanism. This involves allowing the model to revise only the faulty component—such as a misaligned relation or misplaced intermediate entity—while retaining valid portions of the path. This incremental correction strategy reduces computational overhead, encourages reusability of partial reasoning, and fits well into few-shot or constrained-interaction settings.

A3 DP is intentionally designed with several mechanisms that enable it to handle the aforementioned issues to a certain extent:

• Decoupling of Path Planning and Instantiation: The DP framework first employs an LLM to generate candidate relation paths and then proceeds to instantiate these paths within the KG to identify concrete entities and facts. This decoupling provides several inherent advantages:
  • Handling Incomplete Knowledge: If a particular fact is missing from the KG, the corresponding path will fail during instantiation and be discarded during the verification phase. The framework then attempts to instantiate alternative candidate paths. Since the path generator typically outputs multiple candidates, this increases the likelihood of discovering a viable reasoning route even under incomplete knowledge conditions.
  • Mitigating Noisy Knowledge: If a spurious fact exists in the KG, leading to an incorrect yet superficially plausible reasoning path, the introspection module plays a crucial role. It evaluates each path against constraints extracted from the question (e.g., type constraints, multi-entity constraints). If the noisy path violates these constraints, it is identified and filtered out, triggering the framework’s backtracking mechanism.
• Multiple Candidate Paths with Semantic Selection: The framework generates multiple candidate paths and applies semantic relevance filtering to prioritize plausible ones. This design ensures that even if some paths are invalidated due to KG noise or incompleteness, the system can still fall back on other semantically valid and executable alternatives within the KG, increasing the robustness of answer retrieval.

Remaining Challenges and Future Directions:

Despite these design advantages, we acknowledge that the DP framework faces limitations when deployed on large-scale, noisy, and highly incomplete real-world KGs:

• Collective Failure of Candidate Paths: In scenarios where critical portions of the KG are severely missing, all generated candidate paths may fail during instantiation, resulting in the inability to derive an answer.
• Constraint-Evasive Noise: In rare edge cases, an erroneous fact may structurally satisfy all predefined constraints, allowing it to pass through the introspection mechanism undetected.

While such extreme conditions are uncommon, they highlight important areas for further enhancement:

• Integrating Knowledge Graph Completion: Although KG completion is a separate research domain, it can be incorporated into our pipeline. During the path instantiation stage, if a path fails due to missing links, a KG completion model can be invoked to predict the missing relations and potentially “repair” the broken reasoning chain.
• Incorporating Fact Verification Mechanisms: Before finalizing an answer, a fact verification step can be introduced to cross-check key facts within the reasoning path using external textual corpora or more trustworthy sources, thereby improving the framework’s resilience to noise.

In summary, DP demonstrates a degree of robustness in handling imperfect knowledge graphs through its modular design. However, continued research is necessary to further enhance its performance in complex, real-world applications.