Multimodal Quantitative Language for Generative Recommendation

Thank you very much for your insightful comments on our work！ We try to answer your questions as follows.

Q1: The authors should consider testing the statistical significance of MQL4GRec results.

A1: We conduct one-sample t-tests, and the results indicate that the improvements of MQL4GRec on the NDCG metric are statistically significant (p-value < 0.05).

Q2: To enhance the comprehensiveness of the "Multi-modal Recommendation" section in the related work, the authors could consider including more recent state-of-the-art multimodal recommender system papers, such as [2,3]. The field of multimodal codebooks from other communities should also be included in the related work section to clarify the distinctions between the proposed MQL approach and existing methods, such as those in [4, 5]。

A2: Thank you very much for your suggestions. We will read the relevant papers and incorporate them into the related work section of the next version of the paper as soon as possible.

Dear Reviewer rVKM,

We are truly overjoyed and deeply grateful to receive your kind and positive feedback. Your recognition of our efforts in addressing the concerns and the overall quality of the paper means a great deal to us. It is extremely encouraging and gives us a strong sense of accomplishment.

We sincerely hope that you can raise the rating score, as it would have a significant impact on the future prospects and dissemination of our research.

Best regards,

The Authors

Thank you very much for your insightful comments on our work！ We try to answer your questions as follows.

Q1: The model's high complexity poses significant computational and storage demands, which could lead to considerable costs in real-world deployment.

A1: The method primarily consists of three models: two RQ-VAE models for tokenization and a transformer model for generative recommendations:

• The RQ-VAE model includes an encoder, a decoder, and a quantization component. Both the encoder and decoder are implemented as Multi-Layer Perceptrons (MLPs) with ReLU activation functions. The encoder has six intermediate layers of sizes 2048, 1024, 512, 256, 128, and 64, culminating in a final latent representation dimension of 32. The time complexity of vector quantization is given by: $O\left(\sum_{i=1}^k\left(n_{i-1} \times n_i\right)\right)$. Here, $k$ represents the number of layers in the multi-layer perceptron, $n_0$ is the dimensionality of the item representation, and $n_i$ is the number of nodes in the $i$-th layer.

  • The specific training and inference times are related to the number of items. We set the batch size to 1024 and the maximum number of epochs to 500, and we use a Tesla V100 for training. Taking the Instruments dataset as an example, which has 6250 items, each epoch takes approximately 0.4 seconds, so training the vector quantization model takes about 4 minutes and occupies about 1.5GB of GPU memory. We perform one round of inference to obtain tokens for each item and save the distances from the residual vectors to the codebook, then handle item conflicts as described in the paper, a process that takes only about 1 second.
  • After training the RQ-VAE model, we can use it as a general-purpose tokenizer, and the model has only about 20 million parameters.

• We implement our transformer-based generative recommendation model using the T5 framework. Details of the model are introduced in Appendix C. This model has only about 8 million parameters.

• Most importantly, the size of our model does not increase with the number of items. Traditional methods that rely on item IDs or item representations need to maintain an embedding for each item, and when the number of items is very large, the model parameters and storage space will increase dramatically. In contrast, our model has a fixed-size vocabulary, and we only need to store the tokens for each item. Compared to storing embeddings, this cost is negligible.

• During the testing phase, MQL4GRec utilizes autoregressive decoding and beam search for inference. Its time complexity is given by $O\left(KNL\left(d N^2+N d^2\right)\right)$, where $L$ is the number of model layers, $N$ is the sequence length, and $d$ is the dimension of the hidden states, and $K$ is the beam size. Additionally, a beam search with $H$ steps introduces a time complexity of $O(NKV\log K)$, with $V$ denoting the vocabulary size. Traditional matrix factorization methods, which are the most efficient, require a time complexity of $O(Id)$ to complete all inner product computations and $O(I \log K)$ to identify the top-K recommendations for a single user, where $I$ is the number of items. This suggests that generative recommendations may reduce inference time in cases where $I \gg KNL\left(N^2+N d\right) > NKL$.

Q2: Figure 3 shows that the performance on the Games dataset slightly declines as the amount of pre-training data increases. Does this suggest that the pre-training strategy has limitations when applied to domains with substantial cross-domain differences?

A2: Figure 3, when using both text and image quantitative languages for pre-training, the performance on the Games dataset decreases as the size of the pre-training dataset increases. However, when pre-trained only with text quantitative language ($NIG_1$ w/ pre-training), the performance increases. We believe there might be two potential reasons for this: 1) When pre-training with both text and image quantitative languages, the model may overfit; 2) The domain difference between the pre-training dataset and the Games dataset is significant, leading to insufficient generalization of the pre-trained model.

Thanks again for spending your time on reading our response! We hope our answers are persuasive and have dispelled your doubts.

Thank you very much for your insightful comments on our work！ We try to answer your questions as follows.

Q1: Similarity to Existing Tokenization Approaches.

A1: Our tokenization approach indeed got inspiration from the well established RQ-VAE technology as other excellent and innovative works do, such as TIGER [1], LC-Rec [2] and LETTER [3]. We believe that our approach's innovation and originality lie in its unique application and integration of RQ-VAE within our model's broader framework, which distinguishes it from existing work. We would like to list some key points as follows.

• Unlike previous methods, we propose a new approach to handle collisions. Previous methods are all aimed at a single dataset and can not address the issue of large-scale item collisions across multi-domain datasets, and they would introduce semantically unrelated distributions. Our proposed method for handling collisions can perfectly solve the above problems.

• Moreover, the core innovation of our method lies in transforming the content of items from different domains and modalities into a unified quantitative language, thereby breaking down the barriers between them. This novel idea is precisely achieved through multimodal tokenization.

• After obtaining the multimodal quantitative language, we design a series of quantitative language generation tasks, and implement the transfer of recommendation knowledge across different domains and modalities through pre-training and fine-tuning. This method possesses novelty and originality.

Lastly, could it be that there was a typographical error in this section of the review, where MMGRec was mistakenly referred to as GenRet, given that GenRet does not incorporate multimodal information and RQ-VAE?

[1] Rajput S, Mehta N, Singh A, et al. Recommender systems with generative retrieval[J]. Advances in Neural Information Processing Systems, 2023, 36: 10299-10315.

[2] Zheng B, Hou Y, Lu H, et al. Adapting large language models by integrating collaborative semantics for recommendation[C]//2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 2024: 1435-1448.

[3] Wang W, Bao H, Lin X, et al. Learnable Item Tokenization for Generative Recommendation[C]//Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024: 2400-2409.

Q3: Why does the model not use multimodal information simultaneously to predict the next click item? What are the challenges or limitations that prevent this integration?

A3: Thanks for this great suggestion, and in the future, we will also explore a better way to simultaneously and effectively utilize multimodal information. Actually we use multimodal data for some specific tasks and combine the result together for item retrieval. We have to mention that using multimodal data in a more compact way may face some challenges.

• Joining the text tokens and visual tokens as a single tuple increases the length of representing an item, which in turn raise the complexity of the generation tasks and lead to a degradation in efficiency.
• There is a necessity to devise a more diverse array of generation tasks. While this enhances the variety of training samples, it also introduce the potential issue of overfitting.

However, exploring a better way of leveraging the multimodal information as suggested is a valuable direction and we will consider as our future work.

Thanks again for spending your time on reading our response! We hope our answers are persuasive and have dispelled your doubts.

Thank you very much for your insightful comments on our work！ We try to answer your questions as follows.

Q1: How do different training operations, such as collision handling and quantization, impact the computational time and resource requirements of the model?

A1: Our proposed MQL4GRec consists of four distinct steps: vector quantization, collision resolution, training of the recommendation model, and testing. Below, we will provide a detailed introduction:

• Firstly, we train RQ-VAE models using item representations extracted from pre-trained modal encoders. Both the encoder and decoder of the RQ-VAE are implemented as Multi-Layer Perceptrons (MLPs) with ReLU activation functions. The encoder has six intermediate layers of size 2048, 1024, 512, 256, 128 and 64 with a final latent representation dimension of 32. The time complexity of vector quantization is: $O\left(\sum_{i=1}^k\left(n_{i-1} \times n_i\right)\right)$, where $k$ is the number of layers in the MLP, $n_0$ is the dimension of the item representation, and $n_i$ is the number of nodes in the $i$-th layer.

  • Specifically, the training time is dependent on the number of items. We set the batch size to 1024 and the maximum epochs to 500, and use a Tesla V100 for training. For example, the Instruments dataset contains 6250 items, with each epoch taking about 0.4 seconds. Training the vector quantization model takes approximately 4 minutes and occupies about 1.5GB of VRAM.

• After training the RQ-VAE, we assign tokens to each item and resolve item collisions. We perform one inference pass to obtain the tokens for each item, and save the distances from the residual vectors to the codebook. We then handle item collisions using the method described in the paper.

  • Taking the Instruments dataset as an example, this process requires only about 1 second.

• After assigning tokens to each item, we carry out the pre-training and fine-tuning of the recommendation model, the details of which can be found in Appendix C. The time complexity of this model is $O\left(L\left(d N^2 + N d^2\right)\right)$, where $L$ is the number of model layers, $N$ is the sequence length, and $d$ is the dimension of the hidden states.

  • Vector quantization has only one parameter that affects the time complexity of the recommendation model, which is the number of levels in the vector quantizer. This determines the number of tokens assigned to each item. The more tokens per item, the longer the input and output sequences will be, leading to higher time complexity and resource requirements.
  • Collision resolution has no impact on the time complexity of the recommendation model.

• During the testing phase, MQL4GRec employs autoregressive decoding and beam search for inference. Its time complexity is given by: $O\left(KNL\left(d N^2 + N d^2\right)\right)$, where $K$ is the beam size. Additionally, a beam search with $H$ steps introduces a time complexity of $O(NKV\log K)$, with $V$ denoting the vocabulary size.

  • Similarly, the number of levels in the vector quantizer will affect the inference time and resource requirements, while collision resolution does not have any impact.

Q2: What methods or algorithms are employed to map these quantitative vectors back to specific items in the dataset?

A2: The purpose of vector quantization and collision resolution is to assign a unique set of tokens to each item, where the codebook vectors and residual vectors involved are only used to calculate the distances between vectors and are not used in the subsequent recommendation model.

• After assigning tokens to each item, we maintain dictionaries from items to tokens and from tokens to items.
• When training the transformer model, the embeddings corresponding to the vocabulary are randomly initialized, rather than being initialized with quantitative vectors.
  • This is because the quantitative vectors mainly contain semantic information of the items, and we hope to automatically learn word vectors suitable for recommendation tasks through various generation tasks.
• During transformer decoding, the last layer's hidden state will pass through a linear layer and a softmax layer to obtain the token ID. By looking up the vocabulary, we can get the decoded tokens. Finally, we look up the dictionary mapping from tokens to items.

Dear Reviewer AB8j,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As the author - reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns.

We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best regards,

The Authors