Enhancing Conversational Recommender Systems with Tree-Structured Knowledge and Pretrained Language Models

We would like to thank you for your thoughtful and constructive feedback. Below, we address each of the points raised in your review:

Weakness 1

Thank you for your suggestion to enhance the motivation and rationale in the introduction. We will restructure the introduction to better highlight the innovative aspects of our method and how each component contributes to performance improvements. Specifically:

1. Knowledge Tree Enhanced Module: Current CRS models utilize RGCN to capture the relational information from KGs, and neglect the reasoning capability of PLMs for KG structure-based knowledges, so we introduced a Knowledge Tree Enhanced Module to use PLMs to capture the semantic relationships and structural information of KGs, providing background information for the recommendations.
2. User Preference Extraction Module: One key challenge in CRS is effectively capturing and utilizing user collaborative preference information, particularly from dialogue interactions. Traditional PLM-based methods often focus primarily on processing textual or contextual inputs, while overlooking the direct modeling of collaborative signals. To address this limitation, we have designed a User Preference Extraction Module that is jointly trained with the PLMs. This module captures collaborative preference signals directly from dialogue interactions, using them to guide the PLM's recommendation ability.
3. Multimodal Information Integration: Our model consists of multiple modules, so we introduce a more comprehensive Multimodal Information Integration to address the semantic inconsistency problem between features extracted by different modules.

These three modules, working together, significantly improve the performance of the model in generating more accurate and contextually relevant responses and recommendations.

Weakness 2

Thank you for your comment about the complexity of the proposed method. While the model incorporates multiple modules to process different types of information, we want to clarify that the computational complexity of our approach is not significantly higher than that of existing methods. To clarify, we conducted a computational complexity analysis of PCRS-TKA, which is outlined as follows:

• Feature Extraction Module: O(C² + M×R + N×R)
• Knowledge Tree Enhanced Module: O(T²)
• User Preference Extraction Module: O((C+E)²)
• Multimodal Information Integration: O(T² + E²)
• Prompt Learning Module: O(C² + I)

Where:

• C = min(max token length of PLM, max token length of conversations in a batch)
• T = min(max token length of PLM, max token length of knowledge tree texts in a batch)
• N = number of nodes in KG
• R = type of edges in KG
• M = number of edges in KG
• E = number of entities mentioned in the conversation
• I = number of candidate items

For M>N>I and E<<C≈T, the total complexity of PCRS-TKA can be expressed as O(M×R + C²), which is comparable to the complexity of the baseline model, UniCRS, which also scales as O(M×R + C²). Thus, while PCRS-TKA incorporates additional modules, the overall computational cost remains competitive with baseline models.

Weakness 3

Thank you for your suggestion regarding the exploration of larger language models beyond DialoGPT. Despite LLMs like Llama-3-instruct may offer better performance due to their larger parameter sizes and enhanced reasoning abilities, but we focus more on PLMs for their suitability for full-scale fine-tuning and practical deployment in real-world scenarios, where computational efficiency and scalability are key considerations. According to your advice, to prove that PCRS-TKA can be flexibly transplanted to different PLMs, we conducted experiments replacing RoBERTa with BERT and replacing DialoGPT with GPT-2. The results (shown in the table below) demonstrate that PCRS-TKA performs better than UniCRS across different PLMs, and the choice of RoBERTa and DialoGPT is the most suitable for the specific task and dataset.

Once again, thank you for your valuable suggestions. We will incorporate these improvements and update the experimental results in the next revision.

Thank you for your insightful comments.

Weakness1 & Question 1

Here we will clarify several key innovations and improvements that distinguish our approach from existing frameworks:

1. Knowledge Tree Enhanced Module: We introduced a Knowledge Tree Enhanced Module that leverages PLMs to mine the structural information from KGs. This module extracts deeper semantic relationships and contextual knowledge, which enhances the recommendation quality.
2. User Preference Extraction Module: We introduced a User Preference Extraction Module that is jointly trained with the PLMs in an end-to-end manner. This module captures collaborative preference signals from dialogue interactions, which are incorporated as a user feedback loss during training. By jointly optimizing the PLM with this feedback loss, the recommendation model is guided to better reflect user-specific needs and improve the quality of personalized recommendations.
3. Multimodal Information Integration: Since our model consists of multiple modules, we introduce a more comprehensive Multimodal Information Integration to address the semantic inconsistency problem between features extracted by different modules.

These innovations contribute to PCRS-TKA's superior performance compared to previous approaches and are integral to the model’s effectiveness.

Weakness 3 & Question 2

We appreciate your comments on the hallucination challenge. We specifically address hallucinations by deeply integrating knowledge graphs (KGs), which provide factual, structured knowledge that the model can rely on when generating responses. By grounding the model’s outputs in verified and relevant knowledge, we reduce the risk of generating speculative or incorrect content. To evaluate the effectiveness of our approach, we conducted a human evaluation experiment that included a new metric: “Information Inaccuracy”. Annotators assessed the factual accuracy and logical consistency of model outputs. As shown below, the results demonstrate that our framework reduces hallucinations more effectively than the baseline models:

Thank you for your insightful comments.

Weakness 1

Thanks for your comments. We would like to highlight several key innovations in our approach that distinguish it from prior work, particularly UniCRS [1].

1. Knowledge Tree Enhanced Module: As you noted, we introduced a Knowledge Tree Enhanced Module that leverages PLMs to mine the structural information from KGs. This module extracts deeper semantic relationships and contextual knowledge, which enhances the recommendation quality.
2. User Preference Extraction Module: We have introduced a User Preference Extraction Module that is jointly trained with the PLMs in an end-to-end manner. This module captures collaborative preference signals from dialogue interactions, which are incorporated as a user feedback loss during training. By jointly optimizing the PLM with this feedback loss, the recommendation model is guided to better reflect user-specific needs and improve the quality of personalized recommendations.
3. Multimodal Information Integration: Since our model consists of multiple modules, we introduce a more comprehensive Multimodal Information Integration to address the semantic inconsistency problem between features extracted by different modules.

These innovations contribute to PCRS-TKA's superior performance compared to previous approaches and are integral to the model’s effectiveness.

Weakness 2

Thank you for your comments regarding the choice of PLMs over LLMs. While it is true that LLMs have demonstrated superior performance in many tasks due to their larger parameter sizes and ability to capture complex relationships, PLMs are more suitable for full-scale fine-tuning and real-world deployment, particularly when practical resources are limited. Both PLMs and LLMs have their merits, and our work focuses on PLMs to provide a more efficient and deployable solution for real-world applications.

Weakness 3

We appreciate your suggestion to include a broader range of recent models in our comparison. In response, we have incorporated three state-of-the-art models [2, 3, 4] into our evaluation. The updated results demonstrate that PCRS-TKA outperforms most other models. This improvement can be attributed to the superior integration of knowledge graph (KG) information and the incorporation of the proposed User Preference Extraction Module, which captures collaborative preference signals from dialogue interactions. The updated comparison table is as follows:

References

[1] Wang, Xiaolei, et al. "Towards unified conversational recommender systems via knowledge-enhanced prompt learning." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.

[2] Dao, Huy, et al. "Broadening the view: Demonstration-augmented prompt learning for conversational recommendation." Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024.

[3] Yang, Ting, and Li Chen. "Unleashing the Retrieval Potential of Large Language Models in Conversational Recommender Systems." Proceedings of the 18th ACM Conference on Recommender Systems. 2024.

[4] Zhang, Chengyang, et al. "Improving conversational recommender systems via multi-preference modelling and knowledge-enhanced." Knowledge-Based Systems 286 (2024): 111361.

Thank you for your insightful comments. We appreciate the opportunity to further clarify and improve our work.

Weakness 1 & Question 1

Thank you for your insightful suggestion. As noted, in our initial approach, we followed the settings in [1], where the responses generated by the conversation task were used as part of the prompts for the recommendation task. Inspired by your comment, we conducted additional experiments where we unified the prompt for both tasks, using the recommendation task prompt ($P_{rec}$) as the input for both. The results, presented in the table below, demonstrate that this adjustment not only addresses the semantic misalignment issue but also unexpectedly enhances the performance of the conversation task. We found this approach to be highly effective, and we will include a detailed discussion of these findings in the revised manuscript. Thank you for helping us uncover this valuable improvement.

Additionally, we appreciate your comments on the hallucination challenge. To evaluate the effectiveness of our approach in mitigating hallucinations, we conducted a human evaluation experiment that included a new metric: “Information Inaccuracy”. Annotators assessed the factual accuracy and logical consistency of model outputs, lower values indicate better performance in reducing hallucinations. As shown below, the results demonstrate that our framework reduces hallucinations more effectively than the baseline models. We will add more discussions for revisions according to your comments.

Weakness 2 & Question 1

We also appreciate your comments on the "repeated item shortcut" problem. After reviewing your suggestions, we adopted the experimental setup from [2] and carried out separate evaluations for repeated items and new items. As shown in the results below, we found that while the PCRS-TKA model does exhibit the "repeated item shortcut" issue, it still outperforms the baseline models for both repeated and new items. These findings are important for understanding the robustness of our method in the presence of such data leakage, and we will incorporate these results into the revised manuscript.

Question 2

Thank you for raising the important question regarding the impact of knowledge tree depth and the embedding model on performance. We have conducted experiments with more complex embedding models, such as T-GNN [3], and tried various knowledge tree text structures containing different depths. However, our results indicate that neither the use of a more complex embedding model nor a different structuring of the knowledge tree text resulted in significant improved performance. This aligns with findings in the broader literature on knowledge graphs (KGs), which highlight that multi-hop connections often introduce additional noise that can outweigh their potential benefits [4]. According to your comments, we will add more discussions for revision.

Once again, we thank you for your valuable feedback, which has helped us improve our work. We look forward to incorporating these changes and discussing the results in the updated manuscript.

References

[1] Wang, Xiaolei, et al. "Towards unified conversational recommender systems via knowledge-enhanced prompt learning." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.

[2] He, Zhankui, et al. "Large language models as zero-shot conversational recommenders." Proceedings of the 32nd ACM international conference on information and knowledge management. 2023.

[3] Qiao, Z., Wang, P., Fu, Y., Du, Y., Wang, P., & Zhou, Y. (2020). Tree Structure-Aware Graph Representation Learning via Integrated Hierarchical Aggregation and Relational Metric Learning.

[4] Jay Pujara, Eriq Augustine, and Lise Getoor. 2017. Sparsity and Noise: Where KG Embeddings Fall Short. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1751–1756, Copenhagen, Denmark. Association for Computational Linguistics.