Knowledge Graph Tuning: Real-time Large Language Model Personalization based on Human Feedback

Thanks for your review and constructive suggestions. We are happy to address your concerns.

Reliance on the quality and completeness of the knowledge graphs.

Instead of heavily relying on the quality of KGs, our algorithm is more of a method to improve the quality and completeness of the knowledge graph during the daily use of LLMs. Even if the KG is low-quality and incomplete, our method can inject and update the knowledge triples in the graph automatically based on the user’s interaction with the LLM. In addition, we want to emphasize that with the development of LLM, people realize that the structural knowledge base, such as knowledge graphs, is valuable to enhance LLM's reasoning. More and more knowledge graphs are developed and proven effective in cooperating with LLMs[1-3], including commonsense knowledge graphs[4] and some domain-specific graphs like Medical Knowledge Graphs (MKG)[5,6]. We believe that the quality and quantity of KG will improve continually, especially with the rapid development of LLMs and knowledge-enhanced inference technologies.

Reliance on the LLM’s ability to follow instructions.

It is true that our method, along with all the other retrieval-based methods, relies on the LLM’s ability to follow instructions. However, it is notable that there is a clear tendency for advanced pre-trained LLMs to have a stronger ability to follow instructions since these models are to be fine-tuned using Reinforcement Learning from Human Feedback (RLHF) [7]. Thus, our method is scalable to more advanced LLMs.

User effort.

Our algorithm does not require the user to provide an “explicit” and "correct" answer. All we need is to extract the personalized triples containing factual knowledge from the AI-user interactions. For example, when you use video platforms or shop online, you input some information or choose some entities/recommendations that contain your preference. These interactions contain personalized factual knowledge that our algorithm can utilize to edit the user's KG and realize personalization. Our algorithm does not require the user to provide any information in addition to normal interactions while employing LLM to extract necessary knowledge. A practical example of such interaction is shown in appendix B.1.

Conflicts with previously stored information in the knowledge graph.

Thanks for your insightful question, and this is what we intended to demonstrate via the evaluations on CounterFact datasets. For the CounterFact datasets, all the feedback from the user is contradictory to the knowledge graph we employed. The performance on such datasets shows the effectiveness of updating KGs using our algorithm.

[1] Ting-Yun Chang, Yang Liu, Karthik Gopalakrishnan, Behnam Hedayatnia, Pei Zhou, and Dilek Hakkani-Tur. Incorporating commonsense knowledge graph in pretrained models for social commonsense tasks. arXiv preprint arXiv:2105.05457, 2021.

[2] Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang. Human parity on commonsenseqa: Augmenting self-attention with external attention. arXiv preprint arXiv:2112.03254, 2021.

[3] Yunzhi Yao, Shaohan Huang, Li Dong, Furu Wei, Huajun Chen, and Ningyu Zhang. Kformer: Knowledge injection in transformer feed-forward layers. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 131–143. Springer, 2022.

[4] Speer, Robyn, Joshua Chin, and Catherine Havasi. "Conceptnet 5.5: An open multilingual graph of general knowledge." Proceedings of the AAAI conference on artificial intelligence. Vol. 31. No. 1. 2017.

[5] Wu, Xuehong, et al. "Medical knowledge graph: Data sources, construction, reasoning, and applications." Big Data Mining and Analytics 6.2 (2023): 201-217.

[6] Gong, Fan, et al. "SMR: medical knowledge graph embedding for safe medicine recommendation." Big Data Research 23 (2021): 100174.

[7] Ouyang, L., Wu, J., Jiang, X., et al. (2022). “Training language models to follow instructions with human feedback.”

[8] Lewis, Patrick, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020): 9459-9474.

Thanks for your review and constructive suggestions. We would like to respond to your concerns. Weakness1. It is true that retrieval-based methods have been explored intensively. However, we want to claim that the main contribution and novelty of our paper is to maintain and update the knowledge base rather than simply retrieving and reasoning. It is trivial to collect personal context through conversations and conduct retrieval-based reasoning based on that knowledge base nowadays, but it is not trivial to detect conflicting knowledge facts during the LLM-AI interaction and update the stale knowledge in the knowledge base, which was not explored by the reference you provided and is the focus of our paper. We still appreciate the reference you provided and have included it and the discussion in our revised draft’s related works.

Weakness2. We would like to address your concern about human effort first. Our algorithm does not require the user to provide an “explicit” and "correct" answer. All we need is to extract the personalized triples containing factual knowledge from the AI-user interactions. For example, when you use video platforms or shop online, you input some information or choose some entities/recommendations that contain your preference. These interactions contain personalized factual knowledge that our algorithm can utilize to edit the user's KG and realize personalization. We do not require the user to provide any information in addition to normal interactions. We employ LLM to extract that necessary knowledge, which can “unsupervised and automatically extract memory from conversational contexts for personalized outputs” as you claimed. A practical example of such interaction is shown in appendix B.1.
Regarding your concern about constructing KGs, we would like to emphasize that with the development of LLM, people realize that the structural knowledge base, such as knowledge graphs, is valuable to enhance LLM's reasoning. More and more knowledge graphs are developed and proven effective in cooperating with LLMs[1-3], including commonsense knowledge graphs[4] and some domain-specific graphs like Medical Knowledge Graphs (MKG)[5,6]. We believe that the quality and quantity of KG will improve continually, especially with the rapid development of LLMs and knowledge-enhanced inference technologies. Thus, we think it is a promising research direction to apply KG to LLM reasoning. [References sees reply for reviewer 5.] Our theoretical analysis is inspired by RAG[8]. However, this is the first work that conducts optimization on the knowledge base rather than only retrieving and reasoning, which is novel. Thanks for providing the reference; we have included it in our revised draft. [References sees reply for reviewer 5.]

Weakness 3 and Questions 1 & 2. We do not only evaluate the fact editing tasks. To evaluate KGT in a more realistic setting, we build and evaluate on a real personalization interaction benchmark PeInt in our paper. Thanks for your suggestion to evaluate the alignment tasks. We want to emphasize that our method focuses on the deployment phase, which is after the RLHF step, as shown in Figure 1. Our method focuses on the penalization that is realized by editing the specific factual knowledge rather than aligning the LLM to some general preference. Even though our paper does not focus on the alignment, we are still delighted to conduct experiments and show the results that are compared with the reference (DeAL) you provided. We conduct experiments on the HarmfulQ dataset with the MPT-7B model, which is not aligned with RLHF. The compared results are shown below:

It is shown that even though KGT does not focus on alignment tasks, it can also align the LLM to reduce harmfulness by indicating some harmful factual knowledge in the KG. Besides knowledge editing methods, we also compare KGT with K-LaMP (Baek et al., 2024) in our paper, which aims to personalize the responses of LLMs with the knowledge from the knowledge store.

We sincerely appreciate the reviewer's positive evaluation of our work. Specifically, the reviewer recognizes the soundness and reasonableness of our approach. Moreover, the reviewer acknowledges that our method surpasses the established baselines while incurring lower computational costs.

Regarding the reviewer's concern:

W1: more details of the interaction dataset.

We engaged five real users through our university's volunteer recruitment portal. These volunteers interacted with the LLM (Llama3-7b) on various topics tailored to their personal preferences and experiences. One example of interaction is like

User 1: "Can you update me on the latest publications related to quantum entanglement from the past month and also help me plan my day tomorrow? I have a full day at the lab."

AI Assistant: "I found five new publications that match your criteria, including two from the Quantum Information Journal. Would you like a summary of each, or should I focus on those with experimental results similar to your project? Also, I see you have a busy day at the lab tomorrow. There's flexibility in your afternoon schedule; how about scheduling some violin practice for the evening?"

User 1: "Please summarize the ones with experimental results, and download the full papers for later. Schedule violin practice at 7:00 PM and remind me to take Schrödinger for his vet appointment."

AI Assistant: "I've scheduled violin practice for 7:00 PM and set a reminder for Schrödinger’s vet appointment at 3:00 PM, with a 30-minute buffer for travel. The summaries of the selected publications will be ready in your inbox within the hour. Is there anything else I can assist with?"

User 1: "Thanks! Could you also check for any chess tournaments happening this weekend? I'd like to participate if possible."

AI Assistant: "I will look into the local chess club listings and let you know about the tournaments this weekend. Anything else you need?"

There are several personal factual knowledge contained in this example, including that user 1's research field is related to quantum physics, user 1's pet is called Schrödinger, and user 1's hobby is chess. We collect all these interactions and evaluate if our method and the baselines can adapt to these personalized knowledge bases.

W2: concerns on using knowledge editing benchmark dataset to evaluate personalization.

Our algorithm is to realize personalization by editing factual knowledge in the KG. So, generally, our algorithm is also a method of knowledge editing technique. Counterfactual knowledge is the "extreme and difficult case'' of personalized knowledge, and that is why we apply the counterfactual knowledge editing benchmark to evaluate our method. Further, we thoroughly considered concerns as such. Consequently, we built a real personalization interaction benchmark.

Thank you for your response. I have read it carefully and I still believe that the main evaluation datasets (about editing factual knowledge in KGs) are not perfectly aligned with the goal of LLM output personalization. Therefore, I will keep my score.

P.S. I have read the reviews from other reviewers including Reviewer YoVu, and (while I see some clear differences between the ROG paper and this work) I hope the authors will more carefully discuss this previous work in the revision.

Dear reviewer,

I am replying to your complain about the code and charge of plagiarism.

Code:

We totally agree with you about the importance of submitting the code. We had already submitted the code to the supplementary material before the submission deadline.

Plagiarism:

We appreciate the time and effort you dedicated to reviewing our work. However, we want to remind you that allegations of plagiarism are serious and should be supported by clear evidence. While constructive reminders of related work are always welcome, unfounded accusations undermine the professionalism and respect integral to the peer-review process. We respect your anonymity but firmly reject your claim of plagiarism. If you believe we have violated academic integrity, we suggest you raise this matter with the Area Chair (AC) or Program Chair (PC), even though I believe they have noticed your concern, for an impartial evaluation.

Sincerely,

Authors of submission 8863

We thank the reviewer for acknowledging that our method is efficient and interpretable. Upon the concerns of the reviewer, we will address them one by one.

W1: detailed comparison with MemPrompt.

We thank the reviewer for drawing attention to MemPrompt. Our work possesses distinct differences, detailed as follows:

1. Storage: Memprompt uses a dictionary where keys and values are input and user feedbacks, respectively. Both of them are strings in natural language. We extract and save the knowledge triples, enjoying advantages that a. We use less memory. b. We enjoy certain level of abstraction that allows better generalizability.
2. Dynamic updating on conflicting knowledge: Memprompt has trouble resolving conflicts (also mentioned in the limitation of the paper). It is the focus of our paper to detect conflicting knowledge facts during the LLM-AI interaction and update the stale knowledge in the knowledge base.
3. Retrieval augmentation approach: Memprompt directly concatenate the user feedback to the input to the LLM once the input triggers the additional memory. However, we employ a knowledge graph as additional enhancement for reasoning, providing a more structured augmentation.
4. Theoretical motivation: Memprompt largely relies on the in context understanding ability of the GPT3 model. Meanwhile, we provide a theoretical insights to motivate our approach under the framework of RAG. We have revised our draft to include this reference and discussion in related works.

W2: reliance on the quality and completeness of the knowledge graphs.

Instead of heavily relying on the quality of KGs, our algorithm is more of a method to improve the quality and completeness of the knowledge graph during the daily use of LLMs. Even if the KG is low-quality and incomplete, our method can inject and update the knowledge triples in the graph automatically based on the user’s interaction with the LLM. In addition, with the development of LLM, people realize that the structural knowledge base, such as knowledge graphs, is valuable to enhance LLM's reasoning. More and more knowledge graphs are developed and proven effective in cooperating with LLMs [1-3], including commonsense knowledge graphs[4] and some domain-specific graphs like Medical Knowledge Graphs (MKG) [5,6]. We believe that the quality and quantity of KG will improve continually, especially with the rapid development of LLMs and knowledge-enhanced inference technologies. [References sees reply for reviewer 5.]

W3: KG on more complex tasks.

We only focused on the one-hop setting (knowledge triples) in this paper because this is the first paper to adapt the knowledge graph to realize personalization. A multi-hop solution (more relations to detangle more complex tasks) is definitely a goal we aim to achieve for future research. However, we want to mention that even though the detailed techniques might be different, the general theoretical framework supports multi-hop solutions. We are delighted to provide the idea to realize the multi-hop KG editing.

When we need to edit the KG in the multi-hop setting, we can still utilize Equation 2 since it does not have restrictions on the one-hop setting. The difference is the sampling of $z$, $Q(z|q,a)$, and $\mathcal{H}(q,a,K)$. There should be a constant $I$ to define the most hops to cover in the editing. Then $z$ can be formulated as $(e_q, r_k^1, e_k^2,..., r_k^i, e_a )$ where $i\leq I$. When we need to cover the range of $I$ hops in the KG, we will sample $K$ $z$s for each $i\leq I$. We will use $\mathcal{H}(q,a,K,I)$ to denote the set of personalized triples, where $\mathcal{H}(q,a,K,I) = \bigcup_{i=1}^I\mathcal{H}^i(q,a,K),$ and $\mathcal{H}^i(q,a,K)={(e_q, r_k^1, e_k^2,..., r_k^i, e_a )}_{k\in [K]}$ is the set of personalized paths with $i$ hops. Then, $Q(z|q,a)$ is approximated as

$Q(z|q,a) \simeq \frac{1}{KI} \text{ if } z \in \mathcal{H}(q,a,K,I) \text{ else } 0.$

Then, we replace the $\mathcal{H}$ and $K$ in Equation 7 to derive the final loss function

$\mathcal{L} = \mathcal{L_{\text{retrieve}}} + \mathcal{L_{\text{reasoning}}} = -\frac{1}{KI}\sum_{z\in \mathcal{H}(q,a,K,I)} \log \left[P_{\theta,\mathcal{G}}\left(a | q,z\right)P_{\theta,\mathcal{G}}\left(z | q\right)\right].$

The optimization method can still be applicable to the multi-hop setting. The detailed prompts to derive personalized $z$ and the computation of $P_{\theta,\mathcal{G}}\left(z|q\right)$ need to be adjusted, but the basic idea is the same with the one-hop setting.

Our primary focus in this paper is to prove that editing KG to realize LLM personalization works. However, we realize that a multi-hop solution will definitely advance our algorithm under more complex settings, and we will include the discussion about the detailed multi-hop idea in future work!

W4: reference format.

We thank the reviewer for pointing this out. We have revised the reference list to make them consistent and updated.

Dear reviewers,

The ICLR author discussion phase is ending soon. Could you please review the authors' responses and take the necessary actions? Feel free to ask additional questions during the discussion. If the authors address your concerns, kindly acknowledge their response and update your assessment as appropriate.

Best, AC