RecLM: Recommendation Instruction Tuning with Large Language Models

1. The current method generates user and item profiles mainly based on textual features and does not consider multimodal information, while the LLMRec compared in RQ5 considers multimodal information.

2. Although the paper tests on various publicly available datasets, the selected datasets (e.g., Netflix, MIND) are primarily text-rich, thus limiting the demonstration of performance on datasets that are text-sparse or dominated by multimodal information. Furthermore, while the comparison methods include several typical collaborative filtering and graph neural network models, they lack a broader selection of diverse recommendation algorithms.

3. For the training of reward models in reinforcement learning methods relies on high-quality positive and negative samples, where the positive samples are generated via ChatGPT, while the negative samples include profiles of similar users and multiple types of low-quality responses. If these negative samples do not effectively represent the low-quality outputs that the model may generate in real-world applications, does the reward model still have some validity.

1. The literature review in this paper is extremely insufficient, raising questions about the contribution of this work. On the one hand, there are only two references in the introduction, which is extremely unprofessional for an academic paper. Please refer to literatures for how to write the introduction in an academic manner correctly [1, 2]. On the other hand, while the main claims of this paper focus on the cold-start scenario, this paper lacks discussion with important and relevant reference papers [3, 4, 5, 6, 7] in this area. Moreover, progresses of adapting LLMs for cold-start recommendation are also totally ignored in this paper [8, 9]. The lack of literature review results raises the concern about the incorrect understanding of cold-start recommendation of this paper, therefore reducing the novelty and contribution of this paper.

2. The experiments in this paper fail to support the authors' claims. The authors state, “We propose the development of effective language models as profiling systems specifically designed for recommendation tasks, aimed at enhancing performance in cold-start recommendation scenarios.” However, this paper does not include any comparison or discussion with other important cold-start recommenders (e.g., Dropoutnet [6] and CCFCRec [7]).

3. Baselines for user profile modeling are missing. For instance, RLMRec [10] also models user profiles, while this paper does not compare with it as a baseline.

4. The experimental setup and results are questionable. The metrics for the baseline selected in this paper (i.e., LLMRec [11]) show a significant decline compared to those reported in the original paper. In LLMRec [11], the reported results on the Netflix dataset are Recall@20=0.0829 and NDCG@20=0.0347. However, in the reproduction of LLMRec results in this paper, Recall@20 is 0.0529 and NDCG@20 is 0.1721, indicating a significantly lower Recall result and an unusually high NDCG result. Additionally, it is noteworthy that the Recall@20 reported in the original paper of LLMRec, 0.0829, is higher than that of the RecLM method proposed by the authors, which achieved Recall@20=0.0788. Please specify the experimental setting difference between this paper and LLMRec.

5. The writing of this paper can be significantly improved.

   $\bullet$ The citation format in this paper is incorrect, making this paper extremely unprofessional. For citations that are not used as nouns, please use \citep instead of \cite for referencing. For example, in line 94, the correct citation format should be “methods, including NGCF (Wang et al, 2019), GCCF (Chen et al., 2020), and LightGCN (He et al., 2020).” For more details, please refer to the APA citation format on Wikipedia.

$\bullet$The introduction is poorly written, with little effort to summarize the current work before directly transitioning to the contributions. Moreover, important references are missing, as I have emphasized in the weakness 1.

$\bullet$The writing about the implementation details of RecLM is unclear. This paper aims to design a plug-in module, but the methodology section only explains how to fine-tune the module while failing to discuss how the module can be plugged into an existing recommendation model. Important implementation details such as the loss function for integration should be incorporated.

[1] He, Xiangnan, et al. "Lightgcn: Simplifying and powering graph convolution network for recommendation." Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 2020.

[2] Kang, Wang-Cheng, and Julian McAuley. "Self-attentive sequential recommendation." 2018 IEEE international conference on data mining (ICDM). IEEE, 2018.

[3] Schein, Andrew I., et al. "Methods and metrics for cold-start recommendations." Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval. 2002.

[4] Li, Jingjing, et al. "From zero-shot learning to cold-start recommendation." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.

[5] Wei, Yinwei, et al. "Contrastive learning for cold-start recommendation." Proceedings of the 29th ACM International Conference on Multimedia. 2021.

[6] Volkovs, Maksims, Guangwei Yu, and Tomi Poutanen. "Dropoutnet: Addressing cold start in recommender systems." Advances in neural information processing systems 30 (2017)

[7] Zhou, Zhihui, Lilin Zhang, and Ning Yang. "Contrastive collaborative filtering for cold-start item recommendation." Proceedings of the ACM Web Conference 2023. 2023.

[8] Sanner, Scott, et al. "Large language models are competitive near cold-start recommenders for language-and item-based preferences." Proceedings of the 17th ACM conference on recommender systems. 2023.

[9] Huang, Feiran, et al. "Large Language Model Interaction Simulator for Cold-Start Item Recommendation." arXiv preprint arXiv:2402.09176 (2024).

[10] Ren, Xubin, et al. "Representation learning with large language models for recommendation." Proceedings of the ACM on Web Conference 2024. 2024.

[11] "Llmrec: Large language models with graph augmentation for recommendation." Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 2024.

• Could include more comparisons with other LLM-based recommendation approaches. I acknowledge that most existing methods are not "model-agnostic", but it would be interesting to see whether the proposed method can achieve comparable results with other methods that utilize collaborative and semantic signals.
• The proposed method still may not effectively handle cold-start users, since it relies on user profiles.
• It would be better if the authors discuss more on the computational efficiency, since LLMs are much more expensive than traditional recommender systems. Constructing good SFT and PPO data and updating them timely may also be difficult for practical recommender systems.

W1: Scope of claim. The proposed method is an LLM-enhanced recommendation method, which utilizes LLM to augment external features to enhance the CF methods. However, the title "RecLM: Recommendation Instruction Tuning with Large Language Models" and the model name "RecLM" seem to claim an LLM-based recommendation method. This discrepancy appears throughout the paper, particularly in the introduction, related work, and methodology sections.

W2: Experiments. The baseline selection is far more sufficient as only one LLM-enhanced method (LLMRec) is compared.

W3: Presentation quality. The presentation of the paper needs to be improved, especially in sections covering motivation, mathematical formulation, and methodological descriptions.

W4: Literature reviews. The literature review is incomplete, with several relevant important references missing.