FACE: A General Framework for Mapping Collaborative Filtering Embeddings into LLM Tokens

We appreciate your comments. However, we need to emphasize that the main motivation and contribution of our paper.

We are the first to map hidden embeddings given by collaborative filtering methods into pre-trained LLM tokens, in order to improve the interpretability of arbitrary traditional CF models, and to this end, we propose a general and interpretable semantic mapping framework. This framework enables large language models to understand recommendation embeddings through semantic mapping, thereby improving interpretability as well as recommendation performance by leveraging LLM knowledge. For this, we have designed extensive experiments to demonstrate the quality of the mapped tokens and the interpretability beyond recommendation performance.

Response to Q1:

Thank you for your suggestions. In fact, orthogonally initialization is enough for the projection heads. Our model inherently tends to capture comprehensive preferences, automatically forcing disentanglement heads to capture different aspects, as the multiple mapped descriptors are required to reconstruct the original representation and align with semantic information. For validation, we design an experiment. We have implemented the de-correlation loss from the your listed method [2] to constrain the outputs of the multi-projectors. The result could be because it is difficult to balance this loss with the other losses, or this additional constraint hinders the semantic alignment.

Response to Q2:

In the original paper of RQ-VAE (Lee et al. 2022), the author proposed and adopted shared codebook across all quantization levels. This technique maximize the utilization of codebook, making "all code embeddings available for every quantization depth". And we follow this setting in our work. For our work, although LLM token embeddings do not have an explicit hierarchical structure, the vector arithmetic of their embeddings is meaningful (e.g., king - man + woman ≈ queen). When obtaining the final quantized vector $z_q = \sum_{i=1}^H c^{(h)}$, this property can be leveraged. Additionally, due to the nature of residual quantization, the first-level quantization result for each disentangled aspect captures the most significant information, which is then used for the interpretability experiments. For example, SPAE (Yu et al., 2023) in the image-text multi-modal domain, also employs the same LLM vocabulary for its multi-level quantization.

Response to L1:

First, to the best of our knowledge, our work is the first to map collaborative representations to pre-trained, fixed semantic tokens in the LLM for interpretability and is not merely for improving recommendation performance using textual information. Such a general framework also holds significant potential for many downstream tasks, such as conducting interpretable analysis, generating embeddings for cold-start items using the decoder, or leveraging the controllable recommendation capabilities of LLMs after mapping users and items to tokens.

Second, our framework can be applied to RLMRec, a collaborative filtering model that already incorporates textual information, to achieve further improvements in performance, even if the improvement is less compared to its application on traditional recommendation models that do not use textual information. This is the reason why we wrote, "the gains achieved by RLMRec model integrated with FACE are minimal," and we apologize if our imprecise wording caused any misunderstanding.

To fairly compare the ability of different frameworks to align with large language models, it is necessary to conduct experiments under the same base model (LightGCN) and with the same textual information. We have included the method you listed, CTRL (Wang et al., 2023), which aims at CTR prediction in collaborative filtering. Additionally, we have included another enhancement framework that utilizes large models and text, KAR (Xi et al., 2023), and we simultaneously investigate the effect of further applying FACE to these LLM-enhanced recommendation models.

The experimental results demonstrate that: 1. FACE achieves superior performance compared to other LLM-enhanced recommendation methods, meaning that LightGCN, which does not inherently use semantic information, outperforms existing LLM4Rec methods when integrated with FACE; 2. FACE can be effectively applied to various existing LLM-enhanced recommendation approaches to further improve their performance. These findings are consistent with those presented in the original paper, highlighting the advantages of the FACE framework in utilizing semantic mapping to enable LLMs to comprehend collaborative embeddings.

Response to L2:

Our work focuses on mapping hidden embeddings given by collaborative filtering methods into interpretable LLM tokens, which to our knowledge has not been explored. Existing LLM4Rec alignment works directly align collaborative space with textual space by contrastive learning merely for textual injection and performance improvement (RLMRec, CTRL, etc.), while our work enables LLM to interpret collaborative embeddings and improve interpretability. Therefore, besides the performance evaluation, we have designed various interpretability experiments (Section 4.4) to demonstrate the quality and the application of semantic mapping. For interpretability, we design item-retrieval, item-generation tasks and real-user study on LLM interpretation on interactions. These studies quantify the quality of semantic mapping, and suggest that our method constructs effective semantic mapping and improve the interpretability of latent CF embedding.

Response to L3:

• What's the challenge the authors would like to tackle? The challenge we aim to tackle is the semantic interpretability of user and item embeddings in traditional collaborative filtering models. To address this, we propose a general semantic mapping framework that is applicable to any CF model and any LLM.

• To what extent do existing approaches or naive methods resolve it? Traditional models do not address this issue at all. Most existing interpretable methods use language models to explain interaction outcomes, but they do not focus on providing interpretability at the embedding level. Knowledge-enhanced methods, on the other hand, typically leverage semantics solely to improve the recommendation performance of collaborative filtering, not for interpretability.

• Can the proposed module do better? How to measure or quantify it? Given that we are the first to tackle the semantic interpretability of user and item embeddings in traditional collaborative filtering models, besides recommendation performance evaluation, we designed various experiments to specifically quantify interpretability. These include item-retrieval and item-generation tasks, as well as a real-user study on LLM interpretation of interactions. These studies quantify the quality of the semantic mapping and suggest that our method improves the interpretability of latent CF embeddings by constructing this mapping.

Dear Reviewer,

We greatly appreciate the time and effort you have invested in evaluating our work. We have uploaded a detailed rebuttal addressing all the concerns and questions raised in your reviews.

We would like to clarify a minor phrasing issue in our original rebuttal. In the following passage: "Can the proposed module do better? How to measure or quantify it? Given that we are the first to tackle the semantic interpretability of user and item embeddings in traditional collaborative filtering models, besides recommendation performance evaluation, we designed various experiments to specifically quantify interpretability." We meant to emphasize that we are the first to tackle CF embedding interpretability through LLM vocabulary mapping. Our key contribution lies in proposing the novel approach of using pre-trained LLM token vocabulary to semantically interpret collaborative filtering embeddings, which represents a fundamentally different methodology from existing interpretability approaches.

Thank you for your understanding and we look forward to your feedback.

Dear Reviewer,

Thank you for taking the time to review our paper. We have provided detailed responses to address your concerns and questions. Please let us know if there are any remaining points that need clarification - we would be happy to discuss them further.

Best regards,

Dear Reviewer YJtU,

Thank you for your review. We have provided a detailed rebuttal addressing all your concerns, including:

1. Clarifying that FACE is the first framework to map CF embeddings into pre-trained LLM tokens for interpretability (not just another "inject LLM into recommendations" approach)
2. Providing additional experiments comparing with CTRL and KAR methods
3. Addressing your questions about disentanglement and quantization design choices

We would appreciate your thoughts on our responses, especially regarding the fundamental contribution of our semantic mapping framework. With the discussion deadline approaching, we hope to engage in productive dialogue to clarify any misunderstandings.

Thank you for your consideration.

Best regards,

We sincerely thank you for your recognition of our work, and we are encouraged by your recognition that “The novelty of the proposed method is notable”. Here are our replies to your concerns.

Response to W1: About dual objectives.

We are the first to propose a novel semantic mapping method that creatively maps collaborative filtering representations into LLM tokens. This direct mapping improves explainability as well as accuracy. The dual objectives of improving explainability and accuracy are not only the results of this semantic mapping but also serve to validate its effectiveness in turn. The improvement in recommendation performance provides initial evidence that the mapped tokens can be understood by the LLM, which leverages its textual knowledge to enhance model performance. Furthermore, we conducted extensive experiments, including item generation/retrieval and real-user ranking studies, to further demonstrate that the descriptors and the relationships between user/item descriptors can be comprehended by the LLM, which enhances the interpretability of the embedding-based collaborative filtering method.

Response to Q1: How the semantic alignment objective (contrastive learning) directly contributes to recommendation accuracy improvements?

Regarding the direct contribution of semantic alignment to accuracy, as demonstrated in RLMRec (Xi et al., 2023), maximizing the "hidden prior belief" beneficial for recommendation is equivalent to maximizing the mutual information between the CF embedding and the semantic representation. Our method can be viewed as a specific approach to maximizing this mutual information. Given the design of our semantic alignment, the alignment loss signal can be learned by the initial CF embeddings, thereby improving recommendation performance.

Empirically, our ablation studies in Section 4.6 (the "w/o align" variant) confirms that the accuracy improvement stems from semantic alignment. Additionally, the hyperparameter sensitives in Section 4.5 on the weight of the alignment loss shows that this parameter controls the intensity of semantic alignment (i.e., the amount of textual information injected). The recommendation metrics exhibit a trend of first increasing, linking the alignment objective to performance gains, and then decreasing as this coefficient rises, demonstrating too much information injection affects recommendation.

Response to W2 & Q2: How does FACE compare with other recent LLM-enhanced recommendation methods beyond RLMRec?

We have conducted new experiments comparing FACE with other recent LLM-enhanced recommendation methods: CTRL(Wang et al. 2023) leverages contrastive learning for fine-grained knowledge alignment and integration with collaborative and language model; KAR (Xi et al., 2023) combine LM representations with traditional recommenders using a hybrid-expert adaptor. We compared all methods using the same textual embeddings and LightGCN as the backbone. The new experimental results demonstrate that: 1. FACE achieves superior performance compared to other LLM-enhanced recommendation methods; 2. FACE can be effectively applied to various existing LLM-enhanced recommendation approaches to further improve their performance. These findings are consistent with those presented in our original paper, that proves the advantage of FACE.

Response to Q2: Discuss the computational overhead and scalability of the three-stage training process compared to simpler alternatives.

At first, we conduct a time complexity analysis of the proposed framework. Let $B$ denote the batch size, $D$ the number of disentangled descriptors, $C$ the size of the codebook for quantization, and $d$ the embedding dimension. The disentanglement module, involving a disentangled mapping and a Transformer encoder, has a time complexity of $O(B(D^2d + d^2D))$; The quantization module requires a nearest-neighbor search over the codebook, with a time complexity of $O(BCDd)$; The Alignment stage, using a Transformer-based language model and contrastive loss, has a time complexity of $O(B(D^2d + d^2D) + B^2d)$.

Considering that the batch size $B$ is much smaller than the total number of users and items $N=|\mathcal U|+|\mathcal I|$, the additional computational cost of the FACE framework is linear with respect to $N$. Additionally, the number of descriptors $D$ is typically set to a small constant, the actual increase in computational load is modest.

The three-stage training strategy was designed not only for better convergence but also to reduce computational time. In the first stage, we train the base model. The simple yet effective LightGCN model has a training complexity of $O(L⋅|\mathcal E|⋅d)$, where $|\mathcal E|$ is the number of interactions and $L$ the number of convolution layers. In the second stage, we only combine the AutoEncoder, without alignment. The full framework is trained in the final stage, which converges in just a few epochs due to effective pre-training—significantly faster than training the whole model from scratch. The time complexity of all the three stage is linear with respect to $|\mathcal U|, |\mathcal I|$ or $|\mathcal E|$ and is efficient in practical scenario where they grow rapidly.

For comparison, RLMRec based on LightGCN, introduces an additional complexity of $O(Bd^2 + B^2d)$, quadratic with respect to the batch size $B$, but the overall training overhead remains linear with respect to the total number of users/items $N$, similar to our FACE framework.

Response to W3 & Q3: Can the trained FACE framework be transferred across different domains or datasets without retraining?

Thank you for this insightful question regarding the framework's transferability. We propose a model-agnostic framework that can be integrated with various collaborative filtering methods. The transferability of FACE is contingent upon the capabilities of the underlying base model. If the base model supports cross-domain recommendation, FACE may inherit this transferability. We will continue to explore the performance of the FACE framework in cross-domain settings in our future work.

We sincerely thank you for your recognition of our work, noting that “The methodology is well-structured and technically sound.” Here are our replies to your concerns.

W1.1: It would be beneficial to provide a clearer explanation of "descriptor embeddings," including the distinction between descriptor and summary embeddings.

Thank you for your suggestions. Summary embeddings are derived from textual information in the dataset, and then processed by LLM enhancement and embedding. In contrast, descriptor embeddings originate from the collaborative embeddings. Specifically, a collaborative embedding is mapped to a set of discrete "descriptors" (LLM tokens), denoted as $z_d$. These tokens are then inserted into a prompt template to form a complete sentence, which is subsequently fed into the LLM to generate a final, semantically rich sentence embedding. We will provide a clearer explanation in the revised version.

W1.2: Clarification on the pseudo-inverse matrix and how it is obtained would be helpful.

Regarding the pseudo-inverse matrix: during quantization, we obtain the token embedding $c$ (i.e., $z_d$) after dimensionality reduction. It is required to recover the original token embedding $c_0$ while maintaining the gradient for the alignment stage. Considering the projection matrix $W_c \in \mathbb{R}^{d \times d_{LLM}}$, since $d<d_{LLM}$,, a standard inverse matrix does not exist. Therefore, we employ the pseudo-inverse to solve this problem. From the relation $c=W_cc_0$, we can derive the solution for $c_0$ as:

$c_0 =(W_c^TW_c)^{−1}W_c^T c$ ,

where we denote $(W^T_cW_c)^{-1}W_c^T$ as the pseudo-inverse matrix $W_c^{-1}$. While computing, this requires $W_c$ to have full row rank (to ensure $W_c^T W_c$ is invertible), this condition can be met in practice by adding a small perturbation to the weights for numerical stability. We really appreciate your suggestions, and will provide a clearer explanation in the revised version.

W2: A more detailed time complexity analysis of the proposed framework would be advantageous.

Thank you for your valuable suggestions. Let $B$ denote the batch size, $D$ the number of disentangled descriptors, $C$ the size of the codebook for quantization, and $d$ the embedding dimension.

• The Disentanglement module, involving a disentangled mapping and a Transformer encoder, has a time complexity of $O(B(D^2d + d^2D))$.
• The Quantization module requires a nearest-neighbor search over the codebook, with a time complexity of $O(BCDd)$.
• The Alignment stage, using a Transformer-based language model and contrastive loss, has a time complexity of $O(B(D^2d + d^2D) + B^2d)$.

Considering that the batch size $B$ is much smaller than the total number of users and items $N$, the additional computational cost of the FACE framework is linear with respect to $N$. This is important in practical scenario where $N$ grows rapidly, which shows the efficiency of our proposed framework.

Thank you for your valuable comments and for recognizing the novelty of our approach.

However, first of all, we would like to clarify that the main contribution our proposed framework, FACE, is a general framework mapping collaborative embeddings from any CF model into pre-trained tokens from any LLM, which enhances interpretability as well as recommendation performance. This is the first general enhancement framework that enables LLMs to directly understand collaborative representations in the form of LLM tokens. Below are our responses to your concerns.

Response to W1 & Q4: scalability & cold-start

The chosen LLM codebook in our framework has good scalability, and is enough for practical recommendation scenario. Let $|\mathcal{D}|$ denote the size of the LLM vocabulary, $n$ the number of disentangled descriptors, and $H$ the number of RQ layers. The number of possible token combinations reaches: $C^n_{|\mathcal{D}|} \times (A^n_{|\mathcal{D}|})^{H-1} = \frac{(A^n_{|\mathcal{D}|})^{H}}{A^n_n}$ Considering a semantic-filtered vocabulary size of approximately 2000 tokens, with $n=16$ disentangled descriptors and $H=3$ layers, this number is estimated to be over $10^{100}$. Even for $H=1$, it is approximately $10^{34}$. This capacity is sufficient to represent items in real-world recommendation scenarios. For complex or information-rich real-world item/user spaces, the number of tokens mapped per item can be increased for further scalability.

Our framework provides an effective cold-start solution for collaborative filtering models. However, in our paper, we primarily focus on semantic mapping and interpretability. We will incorporate this part in the revised version. Below, we present the analysis and experiment. In FACE, when using an AutoEncoder for disentangled mapping, our model simultaneously trains a decoder that generates original collaborative representations from descriptors. We can leverage textual information from the dataset to generate descriptors for cold-start items. These descriptors can then be fed into the well-trained decoder to generate collaborative representations for cold-start items. The same principle applies to cold-start users.

We conducted an item zero-shot experiment to illustrate the model's cold-start performance: For a given dataset, 1/5 of the items were designated as cold-start items and excluded during training. During testing, we utilized the textual summary information to allow the LLM to generate descriptors, mimicking the association between textual information and descriptors of other items. The embeddings of these generated descriptors were then fed into the decoder to produce collaborative representations. Item recommendations were then made by calculating the similarity with user representations. We compared our approach against AlphaRec, a model with strong cold-start performance, using the same base model and textual information. The results are presented below:

As shown, FACE facilitates a bidirectional mapping between collaborative filtering representations and LLM vocabulary, thereby empowering traditional collaborative filtering models with cold-start capabilities. It achieves better performance than AlphaRec, which relies on a mapping to initial collaborative representations. This demonstrates FACE's strong cold-start and scalability potential.

Response to W2: baselines & citations

Thank you for your suggestions. Our primary focus, as stated in the Limitations section, is on collaborative filtering, a mainstream and classic domain in recommendation, rather than sequential recommendation (e.g., TIGER (Rajput et al. 2023), S3-Rec (Zhou et al. 2020), SASRec (Kang et al. 2018)). Sequential recommendation operates under its own distinct problem settings. However, it's worth noting that our framework could be adapted for sequential recommendation tasks, for instance, by enhancing item embeddings of a sequential recommendation model or by leveraging LLMs for sequential modeling using the mapped descriptors that LLMs can understand, as discussed in Appendix G Broader Impact. We are keen to explore this in future work.

Regarding "SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs," this work is an image tokenization paper in the image-text multi-modal domain, whereas our work performs disentangled mapping in the collaborative filtering scenario. We will cite this paper and add it to our discussion.

Response to W3, Q1, Q2, & Q3: Implementation Details

Q1: Do contrastive loss for users and items separately or together? We conduct contrastive loss for users and items together within a single batch, similar to the approach in CTRL (Li et al. 2023), which leads to stable and robust learning during experiments. Performing the loss separately is also a valid approach. We will add this detail to the paper in revision.

Q2.1: What is recommendation objective? Our proposed framework is a model-agnostic plugin. Therefore, the recommendation objective is that of the base model. In our experiments, the collaborative filtering base models all adopted the BPR recommendation loss function as their objective and were evaluated under the all-ranking protocol (calculating similarity between users and all candidate items). These settings are consistent with the original papers to ensure fair comparison.

Q2.2: Do we optimize all the parameters of the LLM? In fact, our FACE achieves semantic alignment without fine-tuning LLMs, as stated in our Abstract and Introduction. This is because we semantically map the CF embeddings directly to the LLM's vocabulary, which allows the LLM to understand the representations and leverage its pre-trained capabilities to align without any fine-tuning. Indeed, this represents a significant advantage. Compared to approaches that require fine-tuning, our method demonstrates superior efficiency and generality. We apologize for any misunderstanding regarding LLM fine-tuning, and will emphasize this point more clearly in the next version of the paper.

Q3: Why Residual Quantization (RQ) if we only consider the first code? A large body of existing work has demonstrated that Residual Quantization (RQ) exhibits greater stability and superior performance compared to Vector Quantization (VQ). The hierarchical structure of RQ allows for different levels of semantic information to be captured, and summing across layers provides a fine-grained approximation of disentangled embeddings, as discussed in Section 3.2.2, For interpretability, we used only the descriptors from the first layer in our interpretability experiments because they are most similar to the original disentangled embedding and contains the most salient semantic information. In our experiments, we found that using only the first layer for alignment already yielded excellent results.

Response to W4: Where the improvement is coming from?

The performance improvement comes from leveraging the pre-trained capabilities of large language models (e.g., world knowledge, reasoning abilities) to effectively align the tokens mapped from collaborative embeddings with textual information in the dataset, thereby injecting textual semantic knowledge. Existing alignment methods (such as RLMRec) perform direct cross-modal alignment, which fails to make good use of semantic logical relationships. Our framework maps collaborative filtering representations to LLM tokens, effectively leveraging the pre-trained knowledge within LLMs. We designed a contrastive alignment loss and employed a series of measures to ensure proper gradient propagation, thereby enhancing model performance through the injection of textual information. Our ablation studies (Section 4.6) demonstrate that the tokenization mapping component alone provides minimal or negligible improvement to the base model (as shown in the "w/o align" part of our ablation experiments). The significant performance gains are observed when the semantic alignment is introduced, proving the effectiveness of our model in the CF representation-to-LLM vocabulary mapping, which enables utilization of LLM knowledge.

Dear Reviewer,

Thank you for taking the time to review our paper. We have provided detailed responses to address your concerns and questions. Please let us know if there are any remaining points that need clarification - we would be happy to discuss them further.

Best regards,

Dear Reviewer dB6G,

Thank you for your initial review of our paper. We have provided a detailed rebuttal addressing all your concerns, including:

1. Clarifying that FACE is an LLM-tuning-free solution
2. Providing scalability analysis and cold-start experimental results
3. Adding comparisons with additional baselines as requested
4. Clarifying implementation details

We would greatly appreciate your feedback on our responses. As the discussion deadline approaches, we look forward to engaging in further dialogue to address any remaining concerns.

Thank you for your time and consideration.

Best regards,

We sincerely thank you for your insightful feedback. We are particularly encouraged by your recognition of our work's significance: "The core idea of enabling direct interpretation of CF representations by LLM addresses a fundamental challenge in the field." Below, we address your suggestions and questions.

On Clarification and Readability (W1, W2)

We appreciate your constructive suggestions regarding clarity. In the revised manuscript, we will incorporate the following changes:

Clarifying Variables in RQ: We will expand Section 3.2.2 and the caption of Figure 1 to further clarify the roles of the latent variables.

• $z_e$: The disentangled collaborative embedding derived from the CF model.

• $z_d$: The quantized embedding corresponding to a descriptor token from the LLM's vocabulary. This variable plays a pivotal role in achieving semantic alignment and interpretability, as the alignment loss is backpropagated through $z_d$.

• $z_q$: The approximate disentangled collaborative embedding, constructed from the sum of different levels of quantization codewords.

Improving Figure Readability: We will increase the font size of the text in Figure 1 to ensure better readability for all readers.

Can the proposed strategy be extended to facilitate alignment across other modalities? (Q1)

Thank you for this question. We believe the core principles of our framework are generalizable. Our method establishes a bidirectional mapping between continuous representations (from CF models) and discrete semantic tokens (from an LLM). The efficacy of this mapping is ensured by the autoencoder's reconstruction architecture and continuous-to-discrete quantization, while semantic alignment is achieved via contrastive learning. Crucially, we maintain effective gradient flow from the discrete token space back to the continuous embedding space by using a pseudo-inverse mapping. This set of techniques is not inherently limited to recommendation systems. It could potentially be adapted for other cross-modal alignment tasks, such as aligning continuous image embeddings with discrete textual tokens. This is an exciting direction for future work that we are keen to explore. Thank you again for your valuable time and thoughtful review.