Towards Scalable Semantic Representation for Recommendation

1. The 3rd paragraph of Introduction section seems like the paraphrase of the 1st paragraph of section 3.1.

$**Introduction: **$"Notably, the LLM embeddings usually have very large dimensions, ranging from 4,096 to 16,384 (Dubey et al., 2024). When generating the embeddings for these codes, their dimension needs to match that of the recommendation IDs. However, the dimension in recommendation is usually small due to the Interaction Collapse Theory (Guo et al., 2023)."

$**Section 3.1: **$"The original LLM embeddings span a high-dimensional space, e.g., ranging from 1,024 to 416,384 dimensions. (Dubey et al., 2024). When we build a semantic representation based on the LLM embeddings to transfer their knowledge to recommendation, the dimension of these representation needs to match that of the recommendation ID embeddings. However, the dimension of recommendations is usually no more than 256 due to the Interaction Collapse Theory (Guo et al., 2023), and hence is much smaller than that of the LLM embeddings. "

I think the illustration of section 3.1 will lead to redundancy and therefore not needed.

2. The illustration on fusion module is not clear. It is just a self-gating network. Did authors compare with other fusion experiments (e.g., concatenation, single MLP, or cross-attention)?

1. The current experimental design does not sufficiently validate the authors' claims

a) The improvements are not significant. In table1, compared to the non-scaling version, the improvement is often only around 0.1%.

b) The necessary baseline comparison using IDs to scale embeddings without using codebooks is missing. If there is no significant improvement compared to this baseline, then there is no need to explore the use of semantic IDs.

c) Contradic to this paper's claim: MoC will help for scaling up, using the MoC method leads to a decrease in AUC metrics on some datasets. (i.e. in Table 1, model DCNv2, Toys dataset, and model AutoInt+, Beauty dataset). Moreover, MoC does not consistently outperform the baseline, and in some cases, the RQ-VAE method achieves the best metrics. (i.e. in Table 1, model DeepFM, Beauty dataset, 7x).

d) The components in MoC do not always contribute positively. removing the MoC fusion module actually led to metric improvements in certain cases. (Table2, MoC method, 2x)

e) Lack of ablation of main component of MoC. There lack a fair comparison with ME [1] with fusion.

2. The mathematical definitions in the article could be further improved.

a) There is a lack of definition for 1_{x_1} indicates the one-hot encoding of x_i in X_i in lines 096 and so does line 150

b) In lines 321, The authors have omitted the mathematical modeling of the activation layer.

[1] Guo, Xingzhuo, et al. "On the Embedding Collapse when Scaling up Recommendation Models." Forty-first International Conference on Machine Learning.

1. Taking RQ-VAE-based semantic IDs for semantic representations is impractical.
   1. RQ-VAE-based semantic IDs are structured to maintain sequential dependence across different levels, making them commonly used in generative retrieval or generative recommendation models. In these cases, an autoregressive model is trained to capture these semantic IDs in a sequence-to-sequence fashion. From my understanding, RQ-VAE is not well-suited for direct application as features or semantic representations for tasks like CTR. Additionally, I have not found any papers that describe the use of RQ-VAE-based semantic IDs as features (representation learning) for ranking, either in the references of this paper or in my own literature review. The author(s) cite several papers to suggest that RQ-VAE is widely used for deriving semantic IDs in recommender systems; however, the cited methods apply these semantic IDs in a fundamentally different way than in this paper (specifically, as features for ranking). Therefore, RQ-VAE-based semantic IDs appear to be unreliable as one of the two most important base methods for both analytical and performance experiments.
   2. When discussing RQ-VAE-based semantic IDs, I’d like to suggest two alternative approaches for learning semantic representations.
      1. Product/Vector Quantization [1]: This method can also encode input vectors into multiple semantic IDs but without sequential dependencies. It’s been used in existing literature as a semantic representation approach. Vector quantization is much simpler than the RQ-VAE-based method and is worth exploring further.
      2. Multiple Adaptors on LLM Embeddings [2, 3]: Instead of using semantic IDs, this approach directly maps input LLM embeddings to the recommendation space through adaptors (e.g., MLP). The scalability of this method can be explored by increasing the number of adaptors.
   3. In Section 3.1, the author(s) discuss the limitation of a single code. However, few methods actually use a single code. For example, TIGER uses 4 codes per item, and VQ-Rec [1] uses 32 codes per item. It would be better if more practical settings were adopted, like 4 -> 32 instead of 1 -> 3 semantic IDs, when drawing the observations in Figure 1.
2. Experiments are not solid enough.
   1. In Table 1, only methods using additional semantic representations (e.g., ME, RQ-VAE semantic IDs, MoC) are compared. Including a baseline method that relies solely on the original features, without any added semantic representations, would provide a clearer sense of the value these scalable semantic representations offer.
   2. Does "1x, 2x, 3x, 7x" in Table 1 mean "1, 2, 3, 7" levels of semantic IDs (or embeddings of ME)?
   3. It appears that there is no indication of repeated experiments or statistical significance testing, especially since the AUC metrics do not show large differences across all methods.
   4. Ablation study (Table 2).
      1. The effectiveness of applying the proposed fusion method on MoC seems not significant. On settings "2x" and "7x", the fusion module doesn't bring gains for MoC but downgrades the performance.
      2. Only one dataset is used for ablation study.
3. Presentation issues.
   1. In line 124, "416,384" -> "16,384"?
   2. In Eqn. (3) and (4), $h$ and $n$ are not defined. What are the connections between $n$ and $N$?
   3. In lines 201 - 202, what does "quantized representation $Q(r)$" actually mean? Does it mean the clustering centroids of RQ?
   4. What are the references of "a robust semantic representation should have higher top singular values" in Definition 3.2?
   5. Eqn. (8) doesn't match Figure 6. There is no activation function between $W_{down}$ and $W_{up}$ in Eqn. (8).
   6. Line 244, "low-frequency information" and "high-frequency" are not defined in the paper.
4. Code is not available during the reviewing phase.

[1] Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders. WWW 2023.

[2] Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited. SIGIR 2024.

[3] Towards Universal Sequence Representation Learning for Recommender Systems. KDD 2022.

1. The writing of this paper needs to be improved. For example, Sections 4.1 and 4.2 use the same symbol N to represent two different variables, which seems inconsistent. Besides, there are many experiments and analyses through the paper, and the authors should pay more attention to the relationships between them.
2. The use of discrete semantic codebooks in this paper is innovative, but before the authors further discuss how to use them better, they should first elaborate on the advantages and benefits of the basic method in detail (please refer to question 2).
3. This paper aims to scale up semantic representation, but the authors do not explain why scaling up is necessary and what’s the significance. I agree that “scaling up” could be impactful, but the authors should provide a detailed discussion.