Barycentric Alignment of Mutually Disentangled Modalities

Q: A clear connection must be made to family of hybrid models (FM, hybridSVD, hybrid EASEr)

A: Our proposed model shares the intention with hybrid models (FM, hybridSVD, hybrid EASEr), combining techniques from content-based filtering and collaborative filtering to achieve higher recommendation accuracy. We would consider adding this discussion in the final version.

Q: Figure 1 lacks notations

A: We already provided key notations in Figure 1. Due to limited space, the detailed descriptions of weight matrices are presented in preliminary sections to ease the reading.

Q: The notation $(r^{uy}_k, o^{uy}_k)$ is not clear, is it a concatenation?

A: This notation means the output includes two terms. We follow MacridVAE [1] for consistent notations.

Q: MovieLens dataset statistics

A: As described in supplementary, the MovieLens dataset is a subset of MovieLens-10M. We only include items with textual content, and pre-process data following MDVAE [2].

Q: On the evaluation of computational efficiency

A: For clarification, to verify time efficiency we report the training time to run a single epoch in our GPU server in the main text. The reported time is averaged over multiple runs. The main observation is that the proposed BandVAE only requires slightly higher training and inference time than the closest baseline ADDVAE.

The table below shows the number of training epochs to achieve the reported results. We run each model 5 times with different random seeds on each dataset then report the averaged number of required epochs. For contrasting, we compare BandVAE against ADDVAE which uses textual content and MacridVAE which does not use textual content. There are two key takeaways. First, models using textual content does not always require more training epoch than models not using text content, which is shown by contrasting MacridVAE and ADDVAE. Second, BandVAE only requires slightly more training epochs than both MacridVAE and ADDVAE yet achieves higher recommendation accuracy, demonstrating the effectiveness and efficiency of BandVAE.

[1] Ma et al. Learning Disentangled Representations for Recommendations. NeurIPS 2019.

[2] Zhu et al. Mutually-Regularized Dual Collaborative Variational Auto-encoder for Recommendation Systems. The Web Conference (WWW) 2022.

Q: Smaller $\epsilon$ that favors interpretability leads to worse recommendations. This is in line with the overspecialization remark

A: We would like to clarify that the trade-off between interpretability and recommendation in our model is controlled by $\epsilon$ in Equation 4, which controls the sparsity of alignment probabilities in Algorithm 1.

When the alignment probabilities are close to uniform distribution (large $\epsilon$), it is difficult to find a one-to-one mapping between rating and text factors of a single user, thus decreasing the interpretability because we do not know which rating interest factors align with which text interest factors. When $\epsilon$ is small, the alignment probabilities are highly skewed, we can easily find a one-to-one mapping between rating interest factors and text interest factors. Therefore, we can understand the meaning of a rating interest factor via the aligned text factor by visualizing the output of text decoder corresponding to the aligned text factor.

We illustrate this concept in Figure 4 in supplementary materials.

Q: The proposed architecture still puts a lot of focus on interpretability

A: For clarification, we focus on disentangled representation learning to improve recommendation accuracy as this direction has been shown to be effective [1]. The interpretability is embedded inside disentangled representations by nature [2] as it discovers factors of variation underlying data. We transfer this concept from disentangled representation learning to recommender systems to discover user interest factors from ratings and texts. Thus, interpretability is naturally incorporated. Interpretability is not a new feature, rather we analyze it to further understand the proposed model, i.e., how the proposed aligning and fusing method relates to interpretability.

Q: Effect of $\mathcal{L}^t_u$ due to overfitting to specific factor (like popular words), or training/data-splitting artifacts that lead occasionally higher values

A: To clarify the understanding, turning off the effect of $\mathcal{L}^t_u$ means the model does not have the supervision signals from textual content. Then the model is free to embed anything into text factors. Table 5 demonstrates that when including $\mathcal{L}^t_u$, text factors are trained to capture interest factors from texts, which is more meaningful than freely embedding anything into them, leading to better recommendation accuracy. Regarding statistical significance, in Table 5, the numbers of Citeulike-a R@10 and MovieLens R@10 and N@10 are statistically significant. Table 5 brings two key takeaways. For one, textual signals are helpful to recommendation accuracy to a certain extent. For another, the improvement of BandVAE not only comes from simply reconstructing textual content but it is also (mainly) from the proposed aligning and fusing modules, which are the key contributions in this paper.

Q: no comparison is made with simpler modes, i.e. Factorization Machines (FM) by [Rendle 2010]

A: We did not compare BandVAE against FM as one of the baselines MDCVAE has outperformed FM. As the main target is to improve recommendation accuracy via disentangled representation learning, we involve recently developed disentangled recommendation models (and these models already demonstrated state-of-the-art results), including both those using textual content and those that do not, to verify the advantages of our model.

Q: The interpretability aspect cannot be considered novel here, although, the approach to obtain representations to be analyzed is different

A: As stated in the contributions, the key novelty in this paper is how to align and fuse disentangled user interest factors uncovered from ratings and texts. The interpretability is brought by disentangled representation learning by nature [2]. We analyze interpretability to further understand the inner working of the proposed model.

Q: The title is too general. Only textual factors are considered in the work, which is not the general case of arbitrary modalities

A: We consider two modalities ratings and texts. Ratings are the key modality in multi-modal recommender systems [3]. A more specific rephrasing of the title could be Barycentric Alignment of Mutually Disentangled Rating and Text Modalities.

Other modalities can be incorporated in our model. The alignment method can work for two mutually disentangled modalities. For example, one can replace the text channel with an image channel of which the inputs are image pixels. We consider textual content as it is available in chosen datasets while images are not available in Citeulike-a or MovieLens.

[1] Ma et al. Learning Disentangled Representations for Recommendation. NeurIPS 2019.

[2] Bengio et al. Representation learning: A review and new perspectives. TPAMI 2013.

[3] Truong et al. Multi-Modal Recommender Systems: Hands-On Exploration. ACM RecSys 2021.

Dear Reviewer ZCpy,

We would like to thank you for your time reviewing our paper. We really appreciate your detailed and constructive comments. Your concerns are addressed as follows

Q: the work implicitly relies on the assumption of the relatedness between behavioral data and side information, which in general is not supported by prior empirical evidence.

A: Recent studies [2, 3, 4] have shown that incorporating side information such as textual and/or visual content is beneficial to recommendation models, e.g., improving recommendation accuracy. As collected rating data is often sparse in nature, which could prevent learning expressive user/item representations, side information provides additional semantic information of users/items, which contributes to learn higher quality of user and item embeddings from both rating data and side information.

Q: Modalities may be non-alignable. … The work implicitly assumes that ratings are not expressive enough... Focusing on side features mostly leads to the so-called overspecialization, meaning that side features are not descriptive enough to provide any additional knowledge on top of what's hidden in the behavioral data and can only limit the expressivity of models, not improve it

A: We would like to make two clarifications here. First, recent studies [2, 3, 4] have demonstrated that side information helps alleviate rating data sparsity problem to achieve higher recommendation accuracy.

Second, in our model, textual content does not completely replace ratings to make recommendations. Rather, it acts as an accompaniment. We fuse user interest factors learned from textual content with those from ratings to make rating interest factors more expressive. That it achieves higher recommendation accuracy than baselines, which evidences that the modalities are aligned.

Q: Discussion on a classical study on the even few ratings are more valuable than metadata effect by [1]

A: In [1], the authors concluded that “... even 10 ratings of a new movie are more valuable than the best solely metadata-based representation. ... this is due to the large gap between the movie descriptions and the movies themselves: people rate movies, not their descriptions”

The experiment leading to this conclusion is that the authors of [1] compare the proposed model using ratings against another model that only uses metadata to make content-based recommendation. Again, we would like to emphasize that our model does not fully rely on textual content to make recommendations. We still rely on rating data to make recommendations, which shares the same intention with [1]. The novelty is we innovatively fuse user interest factors learned from texts with those learned from ratings in disentangled fashion using alignment method (Section 4.2.1) and mapping method (Section 4.2.2) to obtain more expressive user representations.

Q: On the argument with respect to cold-start scenario

A: We agree that side information, e.g., visual or textual content, can be used to resolve cold-start problem in recommender systems. However, the key research question in this paper is how to fuse disentangled user interest factors from texts with those learned from ratings to achieve a more accurate recommendation model. Recent studies on multi-modal recommender systems also focus on how to leverage side information to obtain more expressive representations, leading to higher recommendation accuracy.

As our focus is not resolving the cold-start problem, we do not dive into cold-start scenario to keep this paper focused on disentangled representation learning with textual content.

[1] Pilászy, I. and Tikk, D., 2009. Recommending new movies: even a few ratings are more valuable than metadata. In Proceedings of the third ACM conference on Recommender systems (pp. 93-100).

[2] Truong et al. Multi-Modal Recommender Systems: Hands-On Exploration. ACM RecSys 2021.

[3] Wu et al. A Survey on Accuracy-Oriented Neural Recommendation: From Collaborative Filtering to Information-Rich Recommendation. TKDE 2023.

[4] Zhou et al. A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions. https://arxiv.org/pdf/2302.04473.pdf

Dear Reviewer ZCpy,

This is a friendly reminder that whether our responses have addressed your concerns and questions.

We are happy to discuss if you have further questions.

Best Regards,

Authors.

Dear Reviewer esB1,

We would like to thank you for spending time reviewing our paper. We address your concern as following

Q: Lacking some real-world applications of this approach

A: The proposed approach in this paper is tested on the recommender system, which is a real-world research problem. Not only is it applicable for rating and text modalities but the proposed method also works for alignment problems between disentangled factors from other mutually disentangled data modalities, e.g., ratings and images, images and texts.

Dear Reviewer esB1,

May we know that whether we have addressed your concern?

If you need further clarification, we are happy to discuss.

Best Regards,

Authors.

Dear Reviewer Mi6r,

We appreciate your constructive comments and your time reviewing our paper. We address your concerns as follows:

W1: Ratings are not usually regarded as one modality in recommendation

A1: Ratings are considered as the key modality in multi-modal recommender systems [1]. Textual content, visual content, etc., are considered as other modalities to alleviate data sparsity problem, which helps recommendation models learn more expressive embeddings. As rating data is often highly sparse in nature, learning user/item representations from other data-modalities, e.g., texts or images, then fusing them with those learned from ratings has been shown to alleviate the data sparsity problem and obtain higher quality of user and item representations [1, 2].

[1] Truong et al. Multi-Modal Recommender Systems: Hands-On Exploration. ACM RecSys 2021.

[2] Zhou et al. A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions. https://arxiv.org/pdf/2302.04473.pdf

Q1: Why did the authors decide to adopt ratings as one of the modalities involved in recommendation?

A1: Ratings serve as key information to make recommendations. Existing recommendation models adopt ratings to learn user and item embeddings then make recommendations based on the learned embeddings. Our paper follows this typical paradigm in recommender systems research.

Textual content is widely available in popular benchmark datasets such as Citeulike, MovieLens and Amazon datasets (visual content is not available in Citeulike and MovieLens), so we adopt textual content to complement rating data, i.e., fusing user representations learned from texts with those learned from ratings to derive more expressive user representations.

A future work could extend our proposed method to incorporate visual content by finding its projection onto rating space via barycentric mapping strategy then combining the projected embeddings of visual content with those from texts and ratings (as described in Section 4.2.2. Mapping and Fusing) to derive user embeddings, which could be more expressive representations and has the potential to improve recommendation performance.

Q2: It seems that the improvement over the tested baselines is not sufficiently large? Can the authors provide justifications for the not-so-high improvement with respect to the tested baselines in Table 1?

A2: The results from Table 1, which are averaged over 10 runs, show that the proposed BandVAE achieves statistically significantly higher recommendation accuracy than the closest performing model on Citeulike-a and Cell Phones datasets, demonstrating the efficacy of BandVAE.

On MovieLens dataset, RecVAE achieves the second highest accuracy (after our method BandVAE). That RecVAE is based on the complex multi-layered encoder (which uses roughly 1.2M parameters (around 18% of RecVAE’s parameters) more than encoder of BandVAE) and the composite prior layer. On other datasets, RecVAE is much less competitive.

Dear Reviewer Mi6r,

Thank you for your explanation.

W1/Q1. We would like to clarify that BandVAE does not use any explicit feedback data, and explicit feedback data is not a side information (i.e., a multimodal feature) of BandVAE.

As described in Section 3 (PRELIMINARIES AND NOTATIONS), the inputs of BandVAE only include $y^u$, which is the binary rating vector (implicit feedback) of user $u$, and $t^u$, which is the textual content associated with user $u$. BandVAE only uses these information to make recommendations, which is fair when comparing with baselines.

We disentangle user preference factors from $y^u$ and $t^u$, then aligning them using the proposed optimal transport-based approach as described in Section 4 to derive more informative user representations, leading to better recommendation accuracy.

Binary rating data (implicit feedback) is also considered the key modality to make recommendations [1]. We, thus, also refer binary rating vector of users (implicit feedback) as one modality in BandVAE together with text modality.

[1] Truong et al. Multi-Modal Recommender Systems: Hands-On Exploration. ACM RecSys 2021.

Dear Reviewer Mi6r,

Thank you for your responses. We address your concerns as following

W1/Q1: For fair comparison, all models, including the proposed BandVAE and baselines, were trained using implicit feedback data in our experiments.

Although the proposed BandVAE and baselines can work directly on explicit feedback data, e.g., explicit ratings from 1 to 5, we choose implicit feedback data as the literature has suggested that a recommendation model with good rating prediction performance, measured by RMSE/MAE on observed ratings, might not perform well on recommendation task [1] in many cases. Therefore, one possible direction is to work on implicit feedback data and evaluate top-N recommendation performance using accuracy metrics such as Recall, which is what we did in this paper.

[1] Cremonesi et al. Performance of recommender algorithms on top-n recommendation tasks. ACM RecSys'10.

W2/Q2 We measure statistical significance using a paired t-test with p-value < 0.05, following ADDVAE's setting. There exist cases when numbers are statistically significant with smaller p-value, e.g., p-value < 0.01.

Dear Reviewer Mi6r,

We just want to post a friendly reminder that whether our responses have addressed your concerns.

If you have further questions, please feel free to post. We are looking forward to hearing from you.

Best Regards,

Authors