KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge

We appreciate the reviewer's thoughtful comments and their recognition of our work's strengths. We address their concerns as follows:

---

[W1/W2] (Dependency on LLM knowledge and potential issues with limited domain coverage)

We appreciate the reviewer's concern about KG-FIT's reliance on LLM knowledge. While we view this as a strength that allows our method to leverage vast and continually improving knowledge, we acknowledge the potential limitations in highly specialized domains.

To address this, we propose several mitigating strategies:

1. Incorporating external knowledge: We can enhance LLM performance in specialized domains by using Retrieval-Augmented Generation (RAG) to incorporate domain-specific external knowledge bases during the description generation process.

2. Leveraging KG context: For domains where external knowledge is limited, we can use the context within the Knowledge Graph itself to generate more informative entity descriptions. This approach ensures that even without extensive domain knowledge, the LLM can still provide useful descriptions based on the relationships and attributes present in the KG.

3. Fallback to seed hierarchy: In cases where the LLM truly lacks domain-specific knowledge, our results show that the seed hierarchy alone still significantly improves KG embeddings. As demonstrated in Table 2 of our paper, KG-FIT with just the seed hierarchy (before LLM refinement) consistently outperforms base models across all datasets.

4. Domain-specific LLMs: When available, using domain-specific LLMs can provide more accurate and relevant knowledge for specialized fields.

These strategies ensure that KG-FIT remains effective and adaptable across a wide range of domains, from general knowledge to highly specialized fields. Our experiments on diverse datasets (FB15K-237, YAGO3-10, PrimeKG) demonstrate KG-FIT's robustness and broad applicability, even when dealing with domain-specific knowledge.

In future work, we plan to explore methods for automatically selecting the most appropriate strategy based on the domain and available resources, further enhancing KG-FIT's versatility and performance.

[W3] (Justification for agglomerative clustering and silhouette score)

We appreciate the reviewer's suggestion for additional justification. We chose agglomerative clustering for its natural ability to create a hierarchical structure without pre-specifying the number of clusters, which is crucial for our approach. The silhouette score was selected as it balances both cluster cohesion and separation, providing a robust measure of clustering quality.

To address the reviewer's concern, we will conduct additional ablation studies comparing different clustering methods:

1. Top-down agglomerative clustering: This variant will start with all entities in one cluster and progressively split them, potentially offering a different perspective on the hierarchy.
2. K-means: We will implement a recursive K-means process, where we first cluster all entities, then recursively apply K-means to each resulting cluster until a stopping criterion is met (e.g., cluster size or maximum depth). This will create a top-down hierarchical structure.
3. DBSCAN: We will use a similar recursive approach as with K-means, but DBSCAN's ability to detect noise points will allow us to create a hierarchy that potentially captures outliers at higher levels.

For evaluation metrics, alongside the silhouette score, we will also compare the Calinski-Harabasz index and Davies-Bouldin index.

We will include these results in the appendix of our revised paper, providing a comprehensive comparison of different clustering methods and their impact on KG-FIT's performance.

---

[Q1] (Clarification on P, L, R in line 134)

We apologize for the lack of clarity. P, L, and R refer to Parent, Left child, and Right child, respectively. We will add this explanation to the paper for better understanding.

[Q2] (Clarification on efficiency evaluation)

The reported training time in Table 4 indeed focuses on the fine-tuning stage for a fair comparison with baselines. The hierarchy construction (Steps 1-3) is a one-time preprocessing step. For transparency, we will add the following breakdown to Appendix H:

• Entity description & text embedding generation (with 15 threads): ~10 minutes for FB15K-237, ~1 hour for YAGO3-10, ~8 minutes for PrimeKG
• Seed hierarchy construction: ~2 minutes for FB15K-237, ~8 minutes for YAGO3-10, ~1.5 minutes for PrimeKG
• LLM-guided refinement (with 15 threads): ~12 minutes for FB15K-237, ~1 hour for YAGO3-10, ~10 minutes for PrimeKG

These preprocessing times are relatively small compared to the overall training process, especially considering they're one-time operations that can be reused for multiple experiments or model variations. Moreover, the LLM-guided refinement step shows good scalability with parallel processing, which can further reduce preprocessing time for larger datasets.

Thank you for your thoughtful review. We appreciate your recognition of our work's strengths and we address your concerns as follows.

---

[W1] "there is a lack of the sensitivity study for the hyperparameters in the loss function"

We appreciate the reviewer's concern about the lack of a sensitivity study. To address this, we conducted a sensitivity analysis, presented in Figure A in our rebuttal PDF. We have also presented our hyperparameter study in Appendix I in our paper.

Figure A demonstrates KG-FIT's robustness to hyperparameter variations:

1. Left plot (λ1, λ2, λ3): Performance metrics remain stable across different combinations, indicating that KG-FIT is not overly sensitive to these hierarchical clustering constraint parameters.

2. Right plot (ζ1, ζ2, ζ3): While there's some variation, performance remains consistently high over a range of ratios for these loss function component weights.

These results show that KG-FIT maintains good performance across various hyperparameter settings, addressing the reviewer's concern and suggesting that the model can be readily applied to new datasets without requiring extensive tuning.

[W2] "The comparable methods are old and lack the new ones in the last 2 years"

Thank you for suggesting these recent baselines. We have conducted additional experiments to compare KG-FIT with CompoundE [1], GIE [2], and DualE [4]. The results are presented in Table C in our rebuttal PDF.

These results demonstrate that KG-FIT not only compares favorably with but substantially improves upon recent state-of-the-art KGE methods. This underscores KG-FIT's effectiveness in leveraging LLM knowledge to enhance various KGE architectures.

Regarding KRACL [3], as it employs a GNN-based approach, integrating it with KG-FIT presents unique challenges. We have added the exploration of KG-FIT's integration with GNN-based methods to our future work list.

We appreciate the reviewer's suggestion to include these recent baselines, as it has allowed us to further demonstrate KG-FIT's capabilities and versatility.

[W3] "The paper lacks the analysis of time complexity as well as space complexity"

We appreciate the reviewer's attention to this important aspect of our model. Here's a clarification:

For time complexity: We have analyzed KG-FIT's time complexity both theoretically (Lines 206-210) and empirically (Table 4, Lines 296-303). Our results demonstrate that KG-FIT is 12 times faster in training than the best PLM-based method, while maintaining inference speed comparable to backbone KGE methods.

For space complexity: While we didn't explicitly state it in the paper, the space complexity of KG-FIT in terms of trainable parameters is the same as the backbone KGE models:

$O(|E| * n + |R| * m)$

Where $|E|$ is the number of entities, $|R|$ is the number of relations, $n$ is the entity embedding dimension, and $m$ is the relation embedding dimension.

This is because the main trainable components of KG-FIT are:

1. Entity embeddings: $O(|E| * n)$
2. Relation embeddings: $O(|R| * m)$

The additional components introduced by KG-FIT (entity text embeddings and cluster embeddings) are not trainable parameters, but fixed inputs used during the forward pass. They do consume memory during runtime but do not increase the model's parameter count.

This space complexity is significantly lower than PLM-based methods, which often require gigabytes of memory for model parameters alone. For example, on the FB15K-237 dataset, KG-FIT's trainable parameters would only occupy approximately 60MB of memory (assuming 32-bit floating-point numbers and $n = m = 1024$).

[W4] (Typos in the paper)

We appreciate the reviewer's attention to detail. We will correct these typos:

1. Page 5, line 163: "determined by the lowest common ancestor"
2. Page 6, line 195: "is the sigmoid function"

We will thoroughly proofread the entire manuscript to improve clarity and precision. Thank you for helping us enhance the quality of our paper.

---

References

[1] (ACL 2023) Compounding Geometric Operations for Knowledge Graph Completion.

[2] (AAAI 2022) Geometry Interaction Knowledge Graph Embeddings.

[3] (WWW 2023) KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion.

[4] (AAAI 2021) Dual Quaternion Knowledge Graph Embeddings.

Dear Reviewer dKZ2,

Thank you for your insightful comments and suggestions. In our author response, we have provided additional experimental results, analyses, and clarifications to address your concerns.

As the discussion period nears its end (in 24 hours), we would be grateful if you could take a moment to review our response and let us know if there are any remaining concerns or if our clarifications have adequately addressed your points.

We are grateful for the time and expertise you have shared in reviewing our work.

Sincerely,

The Authors

Thank you for your thoughtful review. We appreciate your recognition of our work's strengths and we address your concerns as follows.

---

[W1/Q1] (Reconsidering motivation)

If the retrieval process here refers to the "retriever" mentioned in KICGPT [1], it's important to note that:

• The retriever in [1] is actually a KG embedding (KGE) model (RotatE in their case) used to generate top-ranked candidate object entities $o$ given a query $(s, r, ?)$. The LLM is then used to re-rank these candidates.

• In this case, the final results rely heavily on the candidate generation handled by the KGE methods. This means that re-ranking methods like [1, 2, 3] can be used as an add-on to KG-FIT, where KG-FIT provides more accurate candidates, improving the final results. We demonstrate this by implementing KICGPT with KG-FIT and show the results in Table B of our rebuttal PDF.

However, if the retrieval process does not involve KGE methods:

1. Link prediction is a crucial task for evaluating knowledge discovery ability. To the best of our knowledge, there are currently no methods that use LLM with retrieval for this task.
2. Knowledge discovery on an existing KG typically requires the model to be fine-tuned on the KG. This is because (1) we need a comprehensive view of all possible object entities $o$ given a query $(s, r, ?)$, and (2) the model must learn underlying patterns from the existing knowledge.
3. Traditional retrieval methods without KGE cannot be directly applied for link prediction because they do not provide a systematic way to rank all possible object entities for a given query. They typically retrieve a small subset of relevant entities based on text similarity, which may miss many valid candidates. In contrast, KGE methods like KG-FIT learn to embed the entire KG structure, enabling them to score and rank all possible object entities for a given query.

We will clarify these points in the revised paper to highlight the unique strengths of KG-FIT and its potential to complement retrieval-based methods.

[W2/Q2] (Incorporating LLM-based baselines)

Thank you for this suggestion. We have added two recent LLM-based baselines: KICGPT [1] and KG-LLM [4].

The results are shown in Table B of our rebuttal PDF and will be added to Table 2 in the revised paper. Note that we only implemented KG-LLM on FB15K-237 due to its high computational cost.

[W3/Q3] (Why KG embedding methods)

We appreciate this question as it allows us to clarify the unique advantages of our approach. Let's compare different methods:

1. Fine-tuning-based methods:
   • Pros:
     • Can adapt LLMs to specific KG domains
     • Leverage the vast knowledge and reasoning capabilities of LLMs
   • Cons:
     • Computationally expensive, especially for large LLMs
     • May overfit to small KGs due to the large number of parameters
     • Difficult to update as the KG evolves, requiring retraining
2. Re-ranking (Retrieval)-based methods:
   • Pros:
     • Leverage LLM knowledge without expensive fine-tuning
     • Computationally efficient for inference
   • Cons:
     • Rely on pre-existing KG embeddings for candidate generation
     • May miss global patterns and relationships in the KG
     • Limited by the quality of the retrieval process and the retrieved context
3. KG-FIT and KG embedding methods:
   • Pros:
     • Capture the global structure and patterns of the entire KG
     • Computationally efficient
     • Easily updatable as new knowledge is added to the KG
     • Provide interpretable entity and relation representations
   • Cons:
     • Require a well-designed model architecture and training process to effectively incorporate LLM knowledge

To illustrate the importance of KGE methods for knowledge discovery, let's consider the following example.

[Example] Suppose we have a movie knowledge graph where entities represent actors, movies, directors, and genres, and relations represent facts like "acted_in", "directed_by", and "belongs_to_genre". Given a query $(s, r, ?)$ where $s$ is a specific actor, $r$ is "likely_to_collaborate_with", and the goal is to predict another actor $o$ that $s$ is likely to work with in the future, a KGE model like KG-FIT can learn from the global KG structure to infer patterns such as "actors who have worked with the same directors or in similar genres are more likely to collaborate". This allows KG-FIT to make informed predictions about potential actor collaborations, even if those specific actors have never worked together before. In contrast, a non-fine-tuned LLM-based method might have difficulty making such inferences if the actors' previous collaborations are not explicitly mentioned in the training set/retrieved context, as it lacks the global understanding of the interconnected nature of the film industry that KGE methods can capture.

It is also important to note that KG embeddings enhanced by KG-FIT are complementary to LLM/PLM fine-tuning and re-ranking methods. For example, PKGC [2] and TagReal [3] apply both fine-tuning and re-ranking, but they heavily rely on the "retrieved" results from their backbone KG embeddings. KG-FIT can improve the quality of these backbone embeddings, leading to better overall performance.

Moreover, KG-FIT strikes a balance between leveraging the strengths of LLMs and maintaining the desirable properties of KG embeddings. By incorporating LLM knowledge into the embedding space, KG-FIT can capture more semantic information and complex relationships while still being computationally efficient and interpretable.

---

References

[1] (EMNLP 2023) KICGPT: Large language model with knowledge in context for knowledge graph completion.

[2] (ACL 2022) Do pre-trained models benefit knowledge graph completion?

[3] (ACL 2023) Text-augmented open knowledge graph completion via pre-trained language models.

[4] (arXiv) Exploring large language models for knowledge graph completion.

Thank you for your thoughtful review. We appreciate your recognition of our work's strengths and we address your concerns as follows.

---

[W1.1] "There are several related studies on using LLMs to enhance text information in KGs [1,2] ..."

We acknowledge this oversight. In our latest draft, we have cited [1] and [2] in line 83:

• "Some methods [1,2] are proposed to improve the performance of the aforementioned methods by enhancing the text information of entities/relations in KGs."

Interestingly, these methods are complementary to Step 1 of KG-FIT. We conducted additional experiments replacing our entity descriptions with those generated by MPIKGC [1] (E&S strategy) and Contextualization Distillation (CD) [2] (ED strategy). For CD, we averaged the embeddings of all descriptions generated for each entity.

Results on FB15K-237 with RotatE and HAKE are shown in Table A in our rebuttal PDF, which demonstrates that KG-FIT can be further enhanced by incorporating these methods.

[W1.2] "Additionally, constructing a hierarchy seems redundant given the existing graph structure of the KG."

Constructing a hierarchy offers key benefits:

1. Different levels of abstraction: KGs represent direct relationships but not hierarchical relationships or varying abstraction levels. KG-FIT's hierarchy captures broader semantic relationships not directly encoded in the KG.
   • Example: In a biomedical KG, "Aspirin" and "Ibuprofen" might be directly connected as "pain relievers". However, an LLM-constructed hierarchy could group them under "NSAIDs", then "Analgesics", and finally "Pharmaceuticals", providing a richer semantic context.
2. Incorporating external knowledge: The LLM-constructed hierarchy in KG-FIT incorporates external knowledge absent from the original KG, enriching entity representations.
   • Example: "Apple" and "Microsoft" grouped under "Tech Giants", "Consumer Electronics", and "Fortune 500 Companies", incorporating broader market knowledge not explicit in the original KG.
3. Handling sparse connections and improved generalization: KGs often have sparse entity connections. Hierarchical structure bridges gaps between semantically related but unconnected entities and enhances generalization to unseen entities or relationships.
   • Example: In a medical KG, "Heart Disease" and "Diabetes" might not be directly connected, but both could be grouped under "Chronic Diseases". An LLM-constructed hierarchy could further classify them under "Cardiovascular Diseases" and "Metabolic Disorders", respectively. This organization bridges gaps and generalizes treatment or risk factors shared among similar diseases, providing a richer context for inference.

These hierarchies provide valuable semantic context beyond the explicit KG structure, enhancing overall embedding representational power.

We have also analyzed the effect of hierarchy in Table 3 and Figure 3 in the paper.

[W2] "The proposed method concatenates structural and textual embeddings to construct the hierarchy, ..."

This is a misunderstanding. Our method involves two distinct steps:

1. Hierarchy construction: We use only textual information - no structural embeddings. The enriched entity embedding $v_i$ is a concatenation of entity name embedding $v^e_i$ and description embedding $v^d_i$ (Equation 1).

2. Fine-tuning: Entity embedding $e_i$ is initialized as a linear combination of a random embedding $e'_i$ and sliced text embedding $v'_i$ (Equation 5). This allows the integration of LLM-derived semantics while adapting to KG structure.

Structural information from the KG is used only in the link prediction objective (Equation 8).

[W3] "Expensive use of LLMs and hierarchical refinement."

We appreciate this concern, but we respectfully disagree for two main reasons:

1. Significant improvements: In many cases the improvements are substantial. For example, KG-FIT-HAKE w/ LHR on PrimeKG achieves a ~0.07 improvement in Hits@1 over KG-FIT-HAKE w/ seed hierarchy.
2. One-time, reasonable cost: The use of LLMs is limited to the hierarchy construction, which is a one-time preprocessing step. As detailed in Appendix H.2, even for large-scale KGs like YAGO3-10, the cost remains reasonable when considering the deployment of KG-FIT in real-world applications.

[W4] "Several recent strong baselines for KG link prediction, such as NBFNet [3] and AdaProp [4] ..."

We appreciate the suggestion to compare with these strong GNN-based baselines. However, KG-FIT is designed for embedding-based methods, offering better scalability, interpretability, and lower computational requirements. Adapting KG-FIT to NBFNet or AdaProp would require substantial changes and might lose these benefits.

KG-FIT constructs a hierarchical structure of entity clusters using static embeddings, while NBFNet and AdaProp use dynamic entity representations through message passing. Integrating these fundamentally different architectures would essentially amount to developing a new hybrid model, beyond our current scope.

Nonetheless, we believe KG-FIT's underlying philosophy could inspire improvements in GNN-based methods in future work.

---

[Q1] "Issue of multiple entities having the same name"

Thank you for this question.

First, as shown in Fig. 7 in Appendix E.1, the prompt asks the LLM to generate descriptions with a "hint" (an entity description from the original KG dataset). This helps differentiate entities with the same name during this step. In our paper, only YAGO3-10 does not have such descriptions, but it does not have this issue.

Second, we can also mitigate this by using strategies like MPIKGC/CD (mentioned in W1.2). Feeding the LLM triples of entities from the training set helps provide context and differentiate entities within the KG.

Dear Reviewer hiNn,

We greatly appreciate your thoughtful review and the time you have taken to provide detailed feedback on our work. In our author response, we have addressed your valuable comments regarding related work, motivation, and the several technical aspects of KG-FIT.

As the discussion period nears its end (in 24 hours), we would be grateful if you could take a moment to review our response and let us know if there are any remaining concerns or if our clarifications have adequately addressed your points.

Thank you once again for your efforts in evaluating our submission.

Sincerely,

The Authors