LLM-Rec: Personalized Recommendation via Prompting Large Language Models

We really appreciate the positive feedback from the reviewer. The suggestions about the limitations and generalizability are greatly appreciated which we will incorporate into our revision. They are helpful in improving our study.

We express our sincere gratitude to the reviewer for the valuable suggestions regarding the contribution of our study.

We concur with the reviewer's perspective on the importance of gaining a deeper understanding of how LLM-Rec enhances personalization, particularly concerning significant neighbors. This insight is indeed crucial. However, it is also important to emphasize that our study's primary objective is to showcase an alternative method of utilizing LLMs. Our approach leverages the extensive knowledge and commonsense reasoning capabilities of LLMs to address existing challenges in personalization.

As demonstrated in our experiments, LLMs have the potential to significantly improve recommendation performance. This improvement is primarily achieved by augmenting additional information to item descriptions that may initially be incomplete or insufficient. This augmentation aspect is a key differentiator in our approach, underlining the unique benefits of applying LLMs in the realm of personalized recommendations.

We appreciate the valuable and constructive feedback from the reviewer. We are actively incorporating these suggestions into our revision.

The main contribution revolves around prompt template design, which appears to be a manual fit for pre-trained LLMs. This approach lacks sustainability, considering the rapid changes in LLMs. Table 1 of the experiments highlights this issue, where even basic prompts outperform more intricate ones on Llama2. How can the authors address this limitation?

Thank you for pointing this out. We will add experiments of the prompts that have the same meaning but differ in wording to evaluate the consistency of LLM-Rec. The variant we use for Llama 2 is the Llama-2-Chat of 7B parameters. It is within our expectation that its generated text may not boost the performance of the recommendation performance as the generated text of GPT-3. However, the findings are still consistent. Adding extra information augmented by LLMs outperforms the model only takes the original item descriptions as input. Additionally, concatenating the augmented text can further improve the recommendation performance which allows single models such as MLP to achieve comparable and even better performance than other more complex feature-based models.

The application scope is restricted to textual-rich datasets. How does the proposed method fare in scenarios where datasets lack textual information, which is a common occurrence?

LLM-Rec aims to augment the input of the items that are relevant to rich textual content. The dataset itself does not necessarily have to contain rich textual information (The Movielens dataset does not contain movie description initially. We have shown how we can generate such description in the Appendix). The limitation lies in whether the item is associated with rich features, which we believe is another major issue in the recommendation area.

Why did the authors opt for partial ranking evaluation (1,000 items) instead of a full ranking approach, which might provide a more comprehensive assessment?

Thank you for this suggestion. We follow previous studies for this evaluation method. We will add experiments of the full ranking approach in the revision.

The comparison seems limited, especially considering the absence of a benchmark comparison with ID-based baselines such as DirectAU [1]. How does the proposed method fare against these established techniques?

Thank you for this suggestion. Our study aims to evaluate how LLMs can be used to improve the content-based recommendation which does not overlap with ID-based methods. Therefore, we did not compare our framework with the ID-based baselines.

We thank the reviewer for the insightful and constructive feedback which are helpful in improving our study. We will respond to the highlighted weaknesses and questions in a point-by-point fashion.

Some of the mistakes make it confused. (1) In Table 1, there is a wrong bold in the second column. (2) “Table 3”in the 7th page is typed as “Figure 3” by mistakes. (3) The citation at the end of the 5th page is not in the right format.

Thank you for catching these mistakes. We will fix them in the revision.

The evaluation of the prompting strategies could benefit from increased interpretability. For instance, there is no experiment validating whether engagement-guided prompting effectively utilizes neighbor information to guide the LLM. It may seem counterintuitive to assert that important neighbor information is considered merely by including item names without a more detailed description and categorization.

We believe there is a misunderstanding about the engagement-guided prompting. In fact, the engagement-guided prompting strategy does NOT focus on item names, it includes the descriptions of other important neighbors. In Figure 1 and Section 2 (the last sentence of the 4th paragraph), we show that the prompt of the engagement-guided prompting is “Summarize the commonalities among the following descriptions: ‘description’; ‘descriptions of other important neighbors’”. Here the ‘description’ refers to the description of the target item, and ‘descriptions of other important neighbors’ refers to the item descriptions of other important neighbors. It does not represent item names.

The paper lacks a detailed discussion on the potential limitations and challenges of using LLMs for personalized recommendation, such as computational complexity, model interpretability, and potential biases in the generated prompts.

We appreciate this suggestion and will include a detailed discussion on the potential limitations and challenges of using LLMs for personalized recommendation in Appendix. Here we briefly discuss the potential limitations and challenges:

Computational Complexity: The LLM-Rec framework focuses on enhancing input for personalized recommendations through generating item descriptions. This approach addresses the problem of original item descriptions often being incomplete or insufficient for effective recommendations. The primary computational load in LLM-Rec arises during the augmentation phase including the output text length, number of items, and the concatenation of text embeddings.

For output text length, our findings indicate that selecting important words for inclusion, rather than incorporating all response words, can lead to improved recommendation performance, as evidenced in Table 3. Future research could explore the balance between the number of words generated and the resulting performance enhancements.

For the number of items, to mitigate this, we propose potential strategies to reduce the need for augmentation across numerous descriptions. One approach is developing a model that evaluates whether an original item description already possesses sufficient information for recommendations. This avenue warrants further exploration in subsequent studies to decrease computational demands.

For the concatenation of text embeddings, as demonstrated in Table 2 and Figure 6, combining the embeddings of augmented text substantially enhances performance. However, this also increases the input dimension for the recommendation module, potentially raising computational costs. A solution could be to opt for a simpler model structure, such as an MLP, which we have implemented in LLM-Rec, to manage this increase in computational complexity.

Model Interpretability: LLM-Rec introduces an innovative approach to model interpretability by enabling direct comparisons between variations in generated prompts and corresponding shifts in recommendation performance. This relationship is illustrated in our research, as showcased in Figures 4 and 5, as well as Tables 3, and 9-16. Nonetheless, we propose that delving into a more granular, word-level analysis could yield deeper insights. By examining both the generated responses and the subsequent recommendation module, we can gain a more nuanced understanding of how specific textual changes influence overall model behavior, enhancing our comprehension of model interpretability.

Potential Biases in generated responses: The presence of potential biases in the prompts generated by LLMs is a significant concern, primarily because these biases are deeply ingrained in the data used to train the LLMs. This challenge calls for a multifaceted approach, including algorithmic adjustments, ethical guidelines, and continuous monitoring for bias, which are important future directions.