Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond

Thank you for your valuable review and suggestions. We deeply appreciate your thoughtful feedback and are grateful for the time and effort you've invested.

Q1: The dataset's limitation to Amazon data is acknowledged.

Thank you for your valuable suggestion. We primarily selected the Amazon dataset as our experimental platform due to its extensive and varied content, including reviews, ratings, images, interactions, and product attributes. This comprehensiveness makes it particularly suited for multi-modal, multi-task evaluations. Recognizing the importance of validating our findings across different datasets, we have also applied our UniMP method to the Netflix [1] and HM [2] datasets. For Netflix, the images are obtained by crawling movie posters from the website. The results, as detailed in our tables, affirm the effectiveness and adaptability of UniMP across these varied datasets.

While we are keen to broaden our multi-modal validation to encompass video and audio data, we currently face the limitation of not having access to publicly available datasets containing raw video or audio content. Despite this, our method holds the capability for extension to various other data modalities. For instance, we could process image frames from a video through our vision encoder and apply pooling techniques to derive representations for further analysis.

Q2: The term "universal" might be overstated. Would it be advisable to revise the title or abstract to specify "language and images" to better reflect the scope?

We appreciate the constructive feedback and valuable suggestions. We have updated the abstract to explicitly mention 'language and images,' ensuring it more accurately represents the scope of our study. Additionally, we've substituted the term 'universal' with 'unified' throughout the paper, as it more appropriately aligns with our work's focus and scope. We will also update the title to reflect these changes as soon as OpenReview enables this functionality.

Thank you for your valuable review and suggestions. We appreciate the opportunity to make the following clarifications.

Q1: Good experimental study and results. Consider multi-modal multi-tasks. But lack of novelty. The components are shown in the literature.

We greatly appreciate your recognition of our work, particularly our comprehensive experimental study, the robust results obtained, and our proposal of a multi-modal, multi-task approach. In terms of novelty, we are the first to propose the problem of unified multi-modal personalization for various tasks. Our method addresses the inherent challenges through a comprehensive approach that encompasses three complementary aspects: the formulation of input data, the fusion of modalities in network modeling, and the design of optimization objectives. The novelty of our work lies in the unique solution to each perspective and the seamless integration of these components to formulate an overall effective strategy. Next we will elaborate on how each component has been designed and adapted to create a distinctive approach.

For user input modeling, we are the first to emphasize and verify the feasibility of utilizing raw multi-modal content, as opposed to relying solely on single-modal data or product IDs to represent user sequences. This choice of content acts as a bridge, facilitating transferability and generalization across new domains and users. The comparative analysis of generalization capabilities, presented in Table 4 of our paper, highlights the strengths of our method.

For modal fusion within the network, while bridging pre-trained vision and language models and the concept of cross-attention is not new, our implementation and the specific challenges we address in our paper are unique. In previous works, CLIP [1] processes visual and textual data separately through modality-specific encoders and only combines them at the final layer to compute similarity scores. VIP5 [2], on the other hand, merges visual embeddings, obtained from a frozen visual encoder, with textual inputs right at the beginning for language modeling. We refer to these as "late fusion" and "early fusion" strategies, respectively. These methods focus on processing a single image-text pair. In contrast, UniMP is designed to utilize a user's entire history, involving multiple products, for the generation. To handle this, we proposed UniMP to align and integrate the visual information into the language model through the specific cross-attention design (instead of concatenation). The visual information of each product is conditioned exclusively on its corresponding textual data during fusion. We found that removing exclusive attention leads to a substantial performance decrease (0.0337$\rightarrow$0.0244). This fusion is also performed at every layer, allowing for a more nuanced and detailed understanding of user preferences. To demonstrate the effectiveness of this approach, we've included an experiment titled 'w/o Fine-grained' in Table 3 of our submitted paper, showing our method's performance without the cross-attention design. Additionally, we have conducted comparisons with Early Fusion, Late Fusion, and without Exclusive Attention strategies respectively in the below table, which further validate the effectiveness and rationale behind our proposed fusion method.

Furthermore, our approach to enhance multi-task optimization, which focuses on improving alignment and prioritization, coupled with the introduction of a new multi-modal personalization benchmark, provides critical insights into this emerging domain.

In summary, the integration of all these components presents a holistic solution for addressing real-world challenges in achieving unified multi-modal personalization. We believe our paper contributes valuable insights and advancements to the field, and we hope that the reviewer will consider these aspects of novelty and innovation in our work.

Thanks for your response. You have clarified my concerns and I raised my score accordingly.

Thank you for providing your valuable feedback, and we appreciate the opportunity to make the following clarifications:

Q1: The strategies of data fusion and user modeling are kind of straightforward. The effectiveness is not validated.

Thank you for raising the question. Regarding data fusion, for previous works, CLIP [1] processes visual and textual data separately through modality-specific encoders and only combines them at the final layer to compute similarity scores. VIP5 [2], on the other hand, merges visual embeddings, obtained from a frozen visual encoder, with textual inputs right at the beginning for language modeling. We refer to these as "late fusion" and "early fusion" strategies, respectively. These methods focus on processing a single image-text pair. In contrast, UniMP is designed to utilize a user's entire history, involving multiple products, for the generation. To handle this, we proposed UniMP to align and integrate the visual information into the language model through the specific cross-attention design (instead of concatenation). The visual information of each product is conditioned exclusively on its corresponding textual data during fusion. We found that removing exclusive attention leads to a substantial performance decrease (0.0337$\rightarrow$0.0244). This fusion is also performed at every layer, allowing for a more nuanced and detailed understanding of user preferences. To demonstrate the effectiveness of this approach, we've included an experiment titled "w/o Fine-grained" in Table 3 of our submitted paper, showing our method's performance without the cross-attention design. Additionally, we have conducted comparisons with Early Fusion, Late Fusion, and without Exclusive Attention strategies respectively in the table below, which further validate the effectiveness and rationale behind our proposed fusion method. We have also incorporated the results in Appendix A.6 of the revision.

In our approach to user modeling, we emphasize the use of raw multi-modal content, as opposed to relying solely on single-modal data or product IDs to represent user sequences. This choice of content acts as a bridge, facilitating transferability and generalization across new domains and users. The comparative analysis of generalization capabilities, presented in Table 4 of our paper, highlights the strengths of our method. Notably, UniMP demonstrates a substantial edge in transfer learning scenarios.

Q2: The presentation of this work should improved.

Thanks for pointing out the typos. We've addressed these corrections and polished the paper. We'll continue to improve the overall presentation and organization of the paper, aiming to enhance its readability.

Q3: How do you incorporate the behavioral data, like click, browse, and scroll?

Thanks for the question. For the Amazon dataset, there are purchase and review behaviors available. In the case of Netflix, it provides data on viewing behavior. To accurately model these behaviors, we incorporate the items that have been purchased or viewed, along with their corresponding attributes, into the user sequence. Moreover, review texts are utilized to represent review behaviors. If additional behavioral signals like clicks, browsing, and scrolling are available, we could assign specific behavior-type tokens to distinguish these various activities and describe the duration of each behavior in a textual format.

Q4: How do you define and characterize the multi-modal information? The category, brand, description, and price are all textual inputs; why are they defined as multi-modalities in the paper?

Apologies for any confusion caused. In our paper, we refer to "multi-modal information" as the combination of textual and visual content. Meanwhile, elements like category, brand, description, and price are classified as "heterogeneous information", which can be represented in a textual format.

Thank you for your valuable review and suggestions. We hope the following answers can address your concerns.

Q1: The rational behind the cross-attention design of its approach is not well-explained. It is important to discuss how this approach compares to other vision-language fusion techniques such as CLIP[1] and other multi-modal recommendation systems like VIP5.

Thank you for bringing out the question. Our fusion method's main innovation, compared to CLIP [1] and VIP5 [2], lies in our unique approach to fusion strategy and fusion position. CLIP processes visual and textual data separately through modality-specific encoders and only combines them at the final layer to compute similarity scores. VIP5, on the other hand, merges visual embeddings, obtained from a frozen visual encoder, with textual inputs right at the beginning for language modeling. We refer to these as "late fusion" and "early fusion" strategies, respectively. These methods focus on processing a single image-text pair.

In contrast, UniMP is designed to utilize a user's entire history, involving multiple products, for the generation. To handle this, we proposed UniMP to align and integrate the visual information into the language model through the specific cross-attention design (instead of concatenation). The visual information of each product is conditioned exclusively on its corresponding textual data during fusion. We found that removing exclusive attention leads to a substantial performance decrease (0.0337$\rightarrow$0.0244). This fusion is also performed at every layer, allowing for a more nuanced and detailed understanding of user preferences. To demonstrate the effectiveness of this approach, we've included an experiment titled "w/o Fine-grained" in Table 3 of our submitted paper, showing our method's performance without the cross-attention design. Additionally, we have conducted comparisons with Early Fusion, Late Fusion, and without Exclusive Attention strategies respectively in the table below, which further validate the effectiveness and rationale behind our proposed fusion method. We have also incorporated the results in Appendix A.6 of the revision.

Q2: The necessity of the context reconstruction loss is still unclear to me. I wonder the detailed design of this loss item and why it can benefit alignment. And about the token-level reweighting, I think it is important to give more definitions/explanations of easy/hard tokens and how to distinguish.

We apologize for the missing details. For the context reconstruction loss, this term involves predicting product attributes for each historical product image in a user's sequence. Unlike our primary task in the paper (e.g., recommendation), which utilizes the entire user history encompassing multiple products, context reconstruction loss focuses on individual products. This approach aligns with the pre-training objective of the model, such as generating detailed descriptions for each specific image. The integration of context reconstruction loss serves as a bridge, harmonizing the optimization of our model. Importantly, the inclusion of context reconstruction loss doesn't incur extra computational costs, yet it substantially enhances training effectiveness by enriching the information learned.

As for token-level reweighting, we recognize that reconstructing context is generally simpler than accurately predicting user preferences, which is a key motivation for our approach. To address this, we propose a reweighting term designed to automatically differentiate between tokens that are easier or harder to predict. We've also included the details and discussion about the loss in Section A.7 of the paper.