Language Models Encode Collaborative Signals in Recommendation

Response to Reviewer $\color{orange}{\text{EzpW}}$

Thanks for your acknowledgment of the importance of the topic we studied. We believe you are an industrial expert with a deep understanding of sequential recommendation. We greatly appreciate your careful reading of our paper and your responsible comments. Some of your comments can significantly improve our paper. Below are our responses to your comments. Due to the character limitation, results are presented in the attached PDF.

Comment 1: The discussion of CF paradigm

Thanks for your concerns. We do not aim to distort nor omit the previous important works using content information and embeddings, as we include these previous studies as the early explorations of the language-representation-based paradigm in line 209-212 and compare AlphaRec with previous works like UniSRec. The primary goal of the CF paradigm discussion here is to summarize this research line, including the previous important works, into a unified paradigm (since there is no acknowledged definition of such paradigm; although content-based recommendation has been used previously, it is not accurate for this paradigm), and showcase the advantage of using the advanced LM representations. In hindsight, we agree using the word "new" is a little glib. We wish to apologize for this overstatement.

Additionally, we still address the importance of the shift from BERT-style representations to LLM-based representations, for three key factors: more encoded user preference similarities, superior zero-shot recommendation ability, and user intention-aware ability.

Comment 2: Appropriate experiments to verify the claim

Thanks for your comments. The mentioned experiments will help make our claim more convincing. We would like to explain the setting differences of our paper with sequential recommendation first. It's true that sequential recommendation (or next item prediction) is important. However, general recommendation [1] (or direct recommendation [2]), which aims to discover the general preferences of users rather than predicting the next item, is still an innegligible issue even recently [3, 4]. They are deployed in domains like music recommendation [3]. The motivation for adopting general recommendation in our paper is reasonable: we do not wish to introduce any non-linearity for studying the representation space relationship, while the sequential relationship is usually captured by non-linear transformation such as attention [5] or RNN. As a result, the general recommendation is an appropriate task for our research goal.

1. Appropriate experiments and stronger baselines

   Considering the general recommendation setting we adopted, the baselines in our paper are the latest and SOTA methods (XSimGCL [3] and RLMRec [4]). We also compare the linear mapping method with these baselines in Table 13.

   We also consider your suggestion about comparing linear mapping with sequential models in the sequential setting. The results are presented in Table 14. AlphaRec is not compared due to the GCN module is not designed for the sequential recommendation, making it hard to adapt to sequential recommendation.

   It's worth noting that linear mapping does not contain any model design, but still maintains comparable performance with specially designed sequential models SASRec. These results are inspiring and make our claim more convincing.

2. Fix LLM type and vary parameter size.

   We fix the model of Llama2 family and change the parameter size. We report the performance in Table 15 in the PDF. A clear performance increase is observed as the model size increases.

3. Zero-shot performance

   In our zero-shot setting, no candidate set is adopted. While in LLMRank [6], a candidate set with the size of 20 is used, which leads to the reported high performance. We follow the same setting of LLMRank, equipping 19 random negative items for each positive item, to conduct the zero-shot experiments again. As indicated in Table 16, AlphaRec significantly outperforms LLMRank.

Comment 3: Improvements in the details

Once again, thank you for your meticulous reading of our article.

1. The task formulation

   It is true that "selecting the item with the highest reward" is more practical in industry. However, "selecting items that the user has interacted with" is also a widely used setting for academic research, such as the famous SASRec [4].

2. The particular K used and 2-3 Ks

   Thanks for your suggestion. We use K = 20 in this paper, and we will highlight this in the latest version of this paper. For more Ks, we have reported more Ks results in our rebuttal period and we will add more Ks results in the appendix of our latest version.

3. The Industrial dataset

   Thanks for your suggestion. This dataset is the Amazon Industrial dataset on Amazon Review. We use this citation because it is the official citation provided by the author of the dataset Amazon Review [7]. We will rename it as Amazon-Industrial for better clarification.

4. The contrastive loss

   We do not regard contrastive loss as the contribution of our paper, as we have clarified this in line 149-151 and line 335-337. The contribution of our paper is that we prove that only simple model design can arouse the potential of advanced language representations.

[1] Neural collaborative filtering. 2017 WWW.

[2] Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). 2022 RecSys.

[3] Representation Learning with Large Language Models for Recommendation. 2024 WWW.

[4] XSimGCL: Towards extremely simple graph contrastive learning for recommendation. 2023 TKDE.

[5] Self-attentive sequential recommendation. 2018 ICDE.

[6] Large Language Models are Zero-Shot Rankers for Recommender Systems. 2024 ECIR.

[7] Justifying recommendations using distantly-labeled reviews and fine-grained aspect. 2019 EMNLP.

Response to Reviewer $\color{blue}{\text{GXUC}}$

Comment 1: More discussion about the collaborative signals encoded in LMs

Thanks for your concern and suggestion. We respond to this question as follows.

First. We would like to restate the importance and correctness of the linear mapping method we used. It is acknowledged by the NLP community that a high performance of linear methods (i.e., linear mapping [1] and linear probing [2]) would suggest specific signals have been encoded in advanced LMs [3, 4], since the linear structure ensures the homomorphism [5] between two spaces. With such linear methods, lexical [6] and syntax [7] structures have been found inside LMs. More details about this research line can be found in previous works [2, 3] on top-tier NLP conferences. Therefore, a high performance of linear mapping would suggest that collaborative signals are encoded in LMs.

Second. "More semantics of the textual descriptions" does not fully explain the success of linear mapping. Linear mapping captures the user preferences beyond textual similarities. As shown in Figure 1c, homosexual movies such as My Beautiful Laundrette and Food of Love scatter in the language space, but gather together in the recommendation space. These movies are semantically dissimilar but share similar user preference similarities, which is captured by the linear mapping. Therefore, we can not simply attribute the improvement to more semantics.

Third. We add more experiments about linear mapping according to the suggestion of another reviewer (results are presented in the attached PDF). We also consider the sequential recommendation setting in Table 14 and the effect of LM size in Table 15. These results further indicates that user preference similarities are encoded in advanced LM, and the knowledge inside increases as the model size increases.

Comment 2: The novelty is limited

Thanks for your concern about the novelty of this paper. The novelty of this paper does not lie in replacing previous BERT-style embeddings with advanced LM embeddings, or deliberately designing a new CF module. We address the two key novelties of this paper.

1. We are the first work to use linear mapping to reveal how much knowledge about recommendation is encoded in language models. Although linear mapping and linear probing are widely used and recognized methods for exploring the knowledge embedded in language models in the NLP field [2, 3], relevant exploration in recommendation is rare.

2. We prove that advanced language representations exhibit excellent properties for designing recommenders with multiple abilities. With only a simple model design, language-representation-based CF models can surpass traditional ID-based models on multiple tasks. Moreover, we summarize the advantages of using current advanced LM representations: low training cost, superior zero-shot performance, and user intention-aware ability.

Comment 3: The computational cost is high

Thanks for your concern. We would like to reply to your concern from three aspects. First. We would like to kindly point out that the computational cost you mentioned does not equal the training cost in our paper. We state in our paper that AlphaRec is of low training cost rather than computational cost. Second. It is important to highlight that language representations can be computed and documented in advance, and there is only a one-time computational cost. Third. The time cost is comparable with ID-based methods. The training cost is relatively low, compared with LM-based recommenders. And the total time cost is comparable with ID-based methods. For quantitative analysis, we represent the training time cost comparison as follows.

Table 1 Training cost comparison

As indicated in Table 1, AlphaRec maintains competitive training costs with ID-based methods, which are much lower than LM-based methods.

Question 1: Elaborate more on the points in Appendix B3

Thanks for your concern. We assume that the improvement of advanced LMs comes from two possible ways, better feature encoding ability (such as more compact feature space or higher embedding dimension) and any other features or knowledge about recommendation. By shuffling the embeddings, we eliminate any other features or knowledge about recommendation. In this way, the performance only comes from the feature encoding ability. Therefore, the decline in performance proves that the improvement not only comes from better feature encoding ability, which is consistent with the claim in our paper.

To summarize, we follow the following points to prove user preference similarities (i.e., collaborative signals) are encoded in advanced LMs:

1. The linear mapping performance may come from feature encoding ability or knowledge about recommendation (e.g., textual similarity or preference similarity). Random shuffle results are poor -> The performance largely comes from the knowledge about recommendation.
2. The linear mapping performance is excellent -> Previous works in NLP [2, 3, 4, 6, 7] suggest user preference similarities may be encoded in advanced LMs.
3. Textually dissimilar items gather in the recommendation space -> user preference similarities beyond textual similarities are encoded in advanced LMs.

[1] Linearly mapping from image to text space. 2023. ICLR.

[2] Understanding intermediate layers using linear classifier probes. 2017 ICLR.

[3] Language models represent space and time. 2024 ICLR.

[4] Emergent world representations: Exploring a sequence model trained on asynthetic task. 2023 ICLR

[5] Linear algebra and geometry. 1969

[6] Probing pretrained language models for lexical semantics. 2020 EMNLP.

[7] A Structural Probe for Finding Syntax in Word Representations. 2019 NAACL.

Response to Reviewer $\color{green}{\text{tJs7}}$

We sincerely thank you for the positive feedback and valuable comments! To address your concerns, we present our responses as follows.

Comment 1: Why only encode titles There is more information within your used Amazon dataset including item descriptions.

Thanks for your concern. An important contribution of this paper is the investigation of the how much recommendation knowledge is encoded in LMs through linear mapping [1, 2, 3]. If the language model encodes sufficient world knowledge, it should be able to uniquely identify items using simple item titles. Therefore, we have ignored other descriptions of items to avoid introducing unnecessary noise.

Comment 2: Un-informative model names There is more information within your used Amazon dataset including item descriptions.

Thanks for your suggestion! The model name AlphaRec is also known for the adjustable hyperparameter \alpha for user intention capture, as introduced in Section 4.3. We believe that this is more reasonable for the model name we used.

Comment 3: More advanced baselines

Thanks for your great suggestions! We fully agree that considering more CF methods will further verify the effectiveness of AlphaRec, especially with the research line on the uniformity and alignment in recommendation. We compare AlphaRec with DiretAU and GraphAU. As shown in this table, AlphaRec presents relatively high performance compared with these methods across various datasets.

[1] Linearly mapping from image to text space. 2023. ICLR.

[2] Understanding intermediate layers using linear classifier probes. 2017 ICLR.

[3] Language models represent space and time. 2024 ICLR.

Response to Reviewer $\color{red}{\text{NfrM}}$

We sincerely thank you for your concerns about our paper.

Comment 1: Terminology The usage of the terminology "collaborative filtering"

Thanks for your question. We would like to kindly point out that collaborative filtering is a common paradigm in recommendation [1], and we only use this terminology to describe the paradigm we adopt in this paper.

Additionally, we do not use the terminology of collaborative filtering information in our paper, and there is no formal definition of collaborative filtering information in the literature. I guess you may mean collaborative signals [2] rather than collaborative filtering information. In our paper, we describe the collaborative signals as "user preference similarities between items", which is consistent with the definition in the famous paper NGCF [2].

In this way, our claim equals whether user preference similarities are encoded in advanced LMs. In this paper, we use linear mapping to study this claim. The linear mapping or linear probing method is widely used to explore the encoded feature in LMs [3, 4]. It is acknowledged by the NLP community that a high performance with a linear layer would suggest that specific knowledge has been encoded in LMs [5] (since linear mapping ensures the homomorphism [6] between two spaces). As indicated in our paper, linear mapping yields high recommendation performance, which suggests that user preference similarities may be implicitly encoded in advanced LMs. More details of the linear mapping and linear probing method can be found in previous works that have been published in top-tier NLP conferences [4, 7, 8].

According to the suggestions of other reviewers, we also conduct more experiments on sequential recommendation and model size effect of LMs in the attached PDF. These results can better clarify our claim.

Comment 2: Training data for LMs The interaction data is confident and open-source LMs won't be able to train on that data

Thanks for your concern. We would like to answer this question from two aspects.

First. All the datasets we adopted in this paper are public datasets, and everyone has access to these data. Open-source LMs have probably been trained on such user behavior data. It's important to highlight that most of the training data for open-source LMs are crawled from websites, such as the CommonCrawl data [9] used for training the Llama family [10]. Moreover, some LMs clearly state that Amazon Review data is used for training [11].

Second. Features in LMs do not require LMs to be trained on a specific type of data. Evidence for this comes from previous works that find that the lexical [8] and syntax [12] structure is encoded in LMs. Lexical and syntax data rarely appear in the training corpus explicitly, but LMs still learn such features through training on a huge amount of structured natural language data. From this aspect, LMs are also able to learn the preference similarities of items through training on the user behavior data.

Comment 3: The main claim in the paper seems questionable.

Thanks for your concern. We answer this question in our response to comment 2. Moreover, we use experiment results to verify our claim like previous papers [4, 5, 6, 7, 8].

Comment 4: More diverse datasets It would be beneficial if we could have results on more diverse datasets.

Thanks for your comments. However, we would like to kindly point out that six datasets from three platforms (i.e., Amazon, MovieLens, and BookCrossing) have been adopted in our paper.

The scale of this dataset, as well as the scale of our experiments, is quite large in the field of recommendation systems. Moreover, most recommender system papers (including highly influential papers) typically conduct experiments on only 2-3 datasets [13, 15, 16] and a single data platform (e.g., Amazon) [14, 15].

According to your suggestion and comments of reviewer $\color{orange}{\text{EzpW}}$, we add one more experiment on the steam dataset under the zero-shot recommendation setting and compare the performance with the LLM4Rec baseline LLMRank [16]. We follow the task setting of LLMRank, equipping 19 negative items for each positive item and conducting the selection task. As indicated in Table 1, AlphaRec significantly outperforms LLMRank on the Steam dataset.

Table 1 Zero-shot performance on Steam dataset

[1] Neural collaborative filtering. 2017 WWW.

[2] Neural graph collaborative filtering. 2019 SIGIR.

[3] Linearly mapping from image to text space. 2023. ICLR.

[4] Understanding intermediate layers using linear classifier probes. 2017 ICLR.

[5] Language models represent space and time. 2024 ICLR.

[6] Linear algebra and geometry. 1969

[7] Emergent world representations: Exploring a sequence model trained on asynthetic task. 2023 ICLR

[8] Probing pretrained language models for lexical semantics. 2020 EMNLP.

[9] CCNet: Extracting high quality monolingual datasets from web crawl data. 2020 LREC.

[10] LLaMA: Open and Efficient Foundation Language Models. 2023

[11] SFR-Embedding-Mistral: Enhance Text Retrieval with Transfer Learning. 2024

[12] A Structural Probe for Finding Syntax in Word Representations. 2019 NAACL.

[13] Self-attentive sequential recommendation. 2018 ICDE.

[14] Towards universal sequence representation learning for recommender systems. 2022 KDD.

[15] Recommender Systems with Generative Retrieval. 2024 NeurIPS.

[16] Large language models are zero-shot rankers for recommender systems. 2024 ECIR.