Language Representations Can be What Recommenders Need: Findings and Potentials

---

Comment 4: To support the claim of collaborative signals encoded in LMs, the performance of using original language representations without any transformations should be considered.

Thanks for your valuable suggestions. We have revised our paper in Section 4.2 accordingly.

We have added experiments using original language representations for recommendation. As shown in Table 3, directly using the language representations without training (i.e., EMB-KNN) consistently outperforms the heuristic method (i.e., Pop). Moreover, in most cases, EMB-KNN is comparable with MF and even surpasses it on Video Games. These findings are consistent with previous works [2]. Such experimental results further help us understand the possibility of collaborative signals encoded within language representations.

Table 3 Performance of directly using language representations for recommendation

[1] Hou, Yupeng, et al. "Bridging language and items for retrieval and recommendation." arXiv preprint arXiv:2403.03952 (2024).

[2] Ding, Hao, et al. "Zero-shot recommender systems." arXiv preprint arXiv:2105.08318 (2021).

---

Comment 2: The implementation of KAR and RLMRec should align with AlphaRec.

Thanks for your concern about the implementation of KAR and RLMRec. According to your suggestions, we have revised the corresponding description of KAR and RLMRec and added one more ablation experiment about RLMRec in Appendix C.2.

In our paper, the implementation of KAR and RLMRec has been closely aligned with AlphaRec. For these methods, the CL is introduced. Moreover, we all use the same language representations as in AlphaRec (i.e., item representations from titles and user representations from the average of items).

We have also tried another version of RLMRec, which is more similar to their original implementation (i.e., item and user profiles are generated by LLMs). For this version, we follow the same prompting approach in the original RLMRec paper to create user and item profiles.

As reported in Table 1 below, the performance of this version of RLMRec (LLM-generated profile) is lower than the implementation in our submission. Moreover, the performance of incorporating LLM-generated profiles is not always stable, which may even lead to performance degradation in some cases (e.g., Movies & TV). We attribute the performance gap to the possible noise and hallucination introduced by these generated profiles. If the language representation is sufficiently powerful, as we expected, the item titles should already capture the necessary information about profiles, making additional LLM-generated profiles redundant or detrimental. We believe these results can make our conclusion more convincing.

Table 1 Performance comparison of different versions of RLMRec

---

Comment 3: Evaluation on private and newly published datasets can help alleviateng the information leakage problem.

Thanks for your concern about the information leakage problem. Your suggestions about adopting private or new datasets are insightful. We have revised our paper in Appendix C.6 to add one new dataset Amazon Electronic 2023 [1].

Collecting private datasets is difficult for academic research. Therefore, we consider the latest version of Amazon 2023 [1] as the newly published dataset. This dataset was published in March 2024, which is the latest public dataset that we have access to. To further alleviate the information leakage problem, we only consider the interactions after 2022.

We conduct the same experiment as in Section 4 on this new dataset, and present the performance in Table 2 below. As shown in this table, in this new dataset, AlphaRec still consistently outperforms baselines. We attribute this result to the fact that language models understand the user preference behind item title descriptions rather than naively remembering the training data. Therefore, AlphaRec still works fine on the latest dataset.

Table 2 Performance of AlphaRec on Amazon Electronics 2023

We would like to thank you for your acknowledgment of our research topic and your valuable suggestions.

---

Comment 1: The performance gain seems come from the InfoNCE loss rather than language representations. Whether such a result can support the power of language representations.

Thanks for your careful reading and for raising this important question. We agree with your observation and would like to clarify a key point: naively incorporating language representations does not guarantee better in-domain performance in recommendations when compared with advanced ID-based recommenders, highlighting the importance of appropriate model design.

AlphaRec’s seemingly simple design plays a critical role in achieving its performance, highlighting its unique contribution compared to the linear mapping version.

To better address your concern, we conducted additional experiments on the Movies & TV, Video Games, and Books datasets with six baselines: (ID-based) LightGCN+BPR, LightGCN+InfoNCE; (contrastive learning-based) XSimGCL, XSimGCL$_{t}$ (replacing the ID embeddings in XSimGCL with language representations); AlphaRec+BPR, AlphaRec+InfoNCE. The results are illustrated in the following table.

The results reveal two key findings:

1. Language representations are powerful and have the potential to replace learnable IDs in recommender systems. This is evidenced by AlphaRec’s consistently strong results compared to both CL-based and LLM-enhanced CF methods, and its demonstrated abilities in zero-shot and intention-aware tasks in our paper.

2. However, without appropriate loss functions and model architecture, this potential cannot be fully realized:

• Loss function: As you noted, AlphaRec+BPR performs similarly to LightGCN+BPR. This demonstrates that simply replacing learnable IDs with language representations in traditional models does not guarantee improvements. Notably, InfoNCE loss outperforms pair-wise BPR loss by better handling multiple negatives, which is crucial for learning collaborative signals in recommendation. This enables a more robust and powerful mapping from language space to behavior space, unlocking the potential of language representations.
• Model architecture: Using language representations in XSimGCL without altering its architecture led to inconsistent performance improvement. We attribute this to the data augmentation technique in XSimGCL introducing noise, which may harm language representation quality.

These findings suggest that unleashing the potential of language representations requires careful and appropriate model design, including both contrastive loss (e.g., InfoNCE) and a tailored architecture. Simply replacing input features of existing advanced CF models and observing a performance drop risks underestimating the capabilities of language representations.

Dear Reviewer m7fA.

Thank you again for your positive feedback and insightful questions. Following your suggestions, we have added the necessary discussion and experiments about the comparison fairness. Moreover, to address the possible information leakage problem you have mentioned, we conduct experiments on the latest dataset Amazon Electronics 2023. We also present the performance of directly adopting language representations for recommendation to better showcase their power. About revisions have been added to the latest version of our paper.

We look forward to further discussion with you.

Best regards,

Authors.

Thanks for your positive feedback and acknowledgment of the importance of the research topic! Your suggestions have greatly helped us strengthen our submission.

---

Comment 1: Discussion with existing fine-tuned LLM-based approaches can strengthen the claims in this paper.

Thanks for bringing this high-quality paper and for your valuable comments! We fully agree that fine-tuned LLM-based approaches are highly inspiring and deserve further discussion. Following your suggestion, we added ELM [1] as a reference in the introduction section and added a detailed discussion in Appendix B.3 of the revised paper.

We would like to categorize the research line (i.e., fine-tuned LLM approaches), including ELM [1], as efforts to understand domain-specific representations by projecting them into an LLM and fine-tuning the LLM to generate explanations. This line of research also includes works we mentioned in our original paper in Line 77-79, such as RecExplainer [2] and RecInterpreter [3], which use LLMs for interpretability, and LLaRA and CoLLM, which utilize LLMs for direct recommendations. In these fine-tuned LLM approaches, the behavior representations are projected as a new modality into the token space of LLMs. The fact that LLMs can interpret such behavior representations indicates that the user behavior space learnt from domain-specific dataset can be aligned to language space at the token level.

AlphaRec and these approaches share a complementary goal: exploring the connection between language modeling and user behavior modeling from a representation space perspective. However, they differ in strengths and limitations:

Fine-tuned LLM-based approaches:

• Strengths:

1. Inject domain-specific representations into LLMs, allowing for interpreting the meaning of representations from the behavior space in a generative way.
2. Offer potential for user-friendly, explainable recommender systems, which is the future from my personal perspective.

• Limitations:

1. Domain-specific representations inherently carry the limitations of their domain-specific nature, often lacking the generalization ability required to effectively address zero-shot challenges in recommendations.
2. High inference time, which can hinder online implementation.

AlphaRec:

• Strengths:

1. Directly leverages language representations, allowing for easy adaptation to real-world recommendation platforms as a plug-and-play, low-cost solution.
2. Exhibits potential to capture short-term user intentions efficiently.

• Limitations: Lacks the intuitive interpretability for human understanding, as it does not generate natural language explanations.

We believe both research directions are highly significant and will continue to inspire each other. If you have additional insights, we would be delighted to discuss them further with you. Thank you again for your thoughtful question!

---

Comment 2: Personalized user-specific mapping is worth exploring.

Your suggestion to incorporate personalization into AlphaRec is insightful! It is true that while AlphaRec ensures the generalization across all users through a shared MLP, it overlooks the personalization of individual users, which is a current limitation and worth exploring. In response, we have revised the limitation (i.e., Section 6) to include a discussion on this limitation on personalization.

Moreover, inspired by your suggestion, we explore a minimal personalization approach by introducing a learnable parameter $p_{u}$ for each user (i.e., replacing $z_{u}$ with $p_{u}z_{u}$) to examine the effectiveness of personalization. We denote this adjusted model as AlphaRec-p. As shown in Table 1 below, this modification results in slight performance improvements on two datasets, demonstrating that even minimal personalization can bring benefits.

We also believe that more diverse and advanced personalization approaches, such as Mixture of Experts (MoE) [4], are worth exploring in the future.

Table 1 Performance of personalized AlphaRec

---

Comment 3: Dynamic query generation for user-specific intent modeling is worth exploring.

Thanks for your suggestion! We conducted an addtional experiment and revised our paper accordingly.

The goal of this section is to explore the potential of perceiving user intention. Therefore, we simplify the problem by only considering a fixed intention. However, it is true that user-specific intention is more practical in the real-world scenarios. Following your valuable suggestions, we consider generating dynamic and personalized intentions by adding the user's historical interactions in the prompt (i.e., "The user has interacted with the following items: <item_list>"). This naive personalized user intention method is denoted as AlphaRec (dynamic intention), while the original fixed intention is denoted as AlphaRec (static intention). As shown in Table 2 below, by considering the dynamic intention, the recommendation results are refined more precisely. We believe that incorporating personalized intentions is a very important and practical research problem deserving further exploration.

Table 2 Intention capture results when adopting dynamic user intention

[1] Tennenholtz et al. Demystifying Embedding Spaces using Large Language Models. ICLR 2024.

[2] Lei, Yuxuan, et al. "RecExplainer: Aligning Large Language Models for Explaining Recommendation Models." KDD. 2024.

[3] Yang, Zhengyi, et al. "Large language model can interpret latent space of sequential recommender." arXiv preprint arXiv:2310.20487 (2023).

[4] Hou, Yupeng, et al. "Towards universal sequence representation learning for recommender systems." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.

Dear Reviewer uzkZ.

Thank you again for your positive feedback and valuable suggestions. According to your suggestions, we have added the necessary discussion with fine-tuned LLM approaches. Moreover, we have also explored the personalization of AlphaRec and intention query generation. We have also revised our paper, adding discussions about these topics.

We look forward to further discussion with you.

Best regards,

Authors.

---

Comment 4: What is the performance of replacing the ID embeddings in XSimGCL with language representations?

Thanks for your question. Following your suggestion, we have conducted additional experiments replacing the ID embeddings in XSimGCL with language representations. We have also revised our paper in Section 4.2 according to your suggestion.

We denote this model as XSimGCL$_ {t}$. As shown in Table 2, we find that the performance of XSimGCL$_ {t}$ improves on some datasets but declines in others. We attribute this to a possible reason that the data augmentation technique by adding noise in XSimGCL may harm the language representations. This unstable performance phenomenon also aligns with existing studies [4, 5], which show that even the most advanced ID-based recommenders cannot achieve consistent performance gains through a naive input feature replacement with language representations alone. These findings further emphasize the importance of careful and appropriate model design to stimulate the potential of language representations for recommendation.

Table 2 Performance comparison when replacing ID embeddings with language representations in XSimGCL

[1] Yu, Junliang, et al. "Are graph augmentations necessary? simple graph contrastive learning for recommendation." Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 2022.

[2] Yu, Junliang, et al. "XSimGCL: Towards extremely simple graph contrastive learning for recommendation." IEEE Transactions on Knowledge and Data Engineering 36.2 (2023): 913-926.

[3] Wu, Jiancan, et al. "On the effectiveness of sampled softmax loss for item recommendation." ACM Transactions on Information Systems 42.4 (2024): 1-26.

[4] Yuan, Zheng, et al. "Where to go next for recommender systems? id-vs. modality-based recommender models revisited." Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2023.

[5] Li, Ruyu, et al. "Exploring the upper limits of text-based collaborative filtering using large language models: Discoveries and insights." arXiv preprint arXiv:2305.11700 (2023).

---

Comment 2: VQ-Rec is omitted in the zero-shot experiment.

Thanks for your insightful suggestion about the baseline of VQ-Rec. We have added the implementation of VQ-Rec in the zero-shot experiment part and revised our paper in Section 5.2. We report the updated zero-shot performance in the following Table 2. AlphaRec presents relatively high zero-shot performance compared with baselines across various datasets, and VQ-Rec presents improvements compared with ZESRec and UnisRec. Thanks for your suggestion.

Table 2. VQ-Rec performance in zero-shot settings

---

Comment 3: Simply adding MLP and GCN on the top of linear mapping does not lead to effective performance improvements.

Thanks for your careful reading and valuable concerns. We might not fully understand your question, and we are happy to discuss it with you if our response does not address your concerns.

In the linear mapping experiments, the CL has been adopted. CL has been proven to be one of the most important components for recommendation [1, 2, 3]. Therefore, the results in the ablation study that linear mapping with CL outperforms AlphaRec with GCN + MLP +BPR are reasonable and consistent with these previous findings [1, 2, 3]. Moreover, the difference between AlphaRec and linear mapping lies in the MLP and GCN, and the performance improvements exactly indicate the significant roles of these components. This phenomenon further highlights the importance of appropriate model design for stimulating the potential of language representations, as we have concluded in Section 4.2.

Thanks for your acknowledgment of the linear mapping method in our paper and the experimental comprehensiveness. We also highly appreciate your concerns and valuable suggestions about our paper. Your suggestions have greatly helped us refine our paper. We respond to your questions as follows:

---

Comment 1: The performance of AlphaRec heavily relies on the language model. The inference cost and API cost can not be overlooked.

Thanks for your valuable concerns about the performance and cost of AlphaRec with different language representations. We carefully revised our paper and tried to address your concerns as follows:

Firstly, we want to clarify that the non-linearity and neighborhood aggregation introduced in AlphaRec narrow the performance gap between different language representations. According to the suggestion of yours and reviewer V7NB, we have conducted an ablation study to analyze the effect of varying the input language representations for AlphaRec. As shown in Table 1 below, the performance gap of AlphaRec between different language representations narrows, compared with the linear mapping setting in our original paper.

Secondly, we would like to point out that the inference cost of LLMs and API cost are relatively affordable, since we only need to extract the language representations of item titles for one time. Extracting the representations with a 7B model on one single V100 gpu for 10000 items only takes around 6 minutes, and using the API to generate representations for 10000 items costs less than 0.2 US dollars. We have revised the paper to add Table 16 about the cost of extracting features. We think these costs are reasonable and affordable. After extracting these representations, we can store them and train a language-representation-based recommender. In this stage, the training cost is similar to traditional ID-based recommenders, as illustrated in Figure (3b) of the paper. The inference cost is relatively acceptable since the language representations are frozen.

Table 1 Ablation study of varying language representations for AlphaRec

Dear Reviewer QRqF,

Thank you again for your insightful suggestions. According to your suggestions, we have added necessary discussion about the cost of extracting features. Moreover, we have added the VQ-Rec under the zero-shot recommendation scenario. We also add a discussion about the role of MLP and GCN, and present the results of replacing ID embeddings with language representations in XSimGCL. All the revisions have been incorporated in our latest version of paper.

We look forward to further discussion with you.

Best regards,

Authors

We sincerely appreciate your positive feedback and valuable suggestions! Your suggestion about the ablation study on the adopted language representations in AlphaRec is important. We have conducted suggested experiments and revised the paper accordingly in Appendix C.4.

---

Comment 1: Ablation study of different text embedding models for AlphaRec can be considered.

Following your suggestions, we have added an ablation study varying language representations for AlphaRec and reported the results in Table 1 below. The results reveal a clear trend: advanced language representations, such as SFR-Embedding-Mistral and Text-embeddings-3-large, consistently outperform earlier BERT-style models across all datasets. This finding aligns with the observations from the linear mapping experiments, further reinforcing the critical role of high-quality language representations in recommendation tasks. Additionally, AlphaRec's architecture design, with its incorporation of non-linearity and neighborhood aggregation, reduces reliance on specific language models. This design narrows the performance gap between different language representations, as compared to the linear mapping setting in our original paper.

Table 1 Ablation study of varying language representations for AlphaRec

We sincerely appreciate your suggestion, which has helped us deepen our analysis and strengthen the claims in our paper. Thank you!

Dear Reviewer V7NB,

Thank you again for your positive comments and valuable suggestions. We have revised our paper according to your suggestions, adding one ablation study on the language representations of AlphaRec.

We look forward to further discussion with you.

Best regards, Authors