Federated Recommendation with Additive Personalization

Thank you for your time and comments! We address your questions and concerns in the following.

W1 (Clarity and Consistency in Writing)

We reported experimental results on six datasets in the paper and we will make the text consistent. We will move the results to Section 4.4 on Page 7.

W2 (Statement of Motivation)

In federated recommendation systems, clients or users have different preference to each item and they may focus on different attributes or charateristics of the item, resulting in data heterogeneity, i.e., the items and their ratings are different across clients. Hence, a global model cannot accurately predict the preference of every user, while a personalized model per user can better address the heterogeneity problem.

Most existing personalized federated recommendation systems focus solely on personalizing user embedding or score function, overlooking the potential impact of user preferences on different attributes of items. While PFedRec considers personalized item embedding, it neglects the knowledge sharing and collaborative filtering across users via global item embedding. To bridge the gap, in this work, we introduce a novel additive personalization that takes both the personalization of item embedding and sparse global collaboration in to account, leading to FedRAP that achieves significant improvement on the federated recommendation benchmarks.

We will improve the presentation of our motivation and strengthen the connection between the heterogeneity and the personalization proposed.

W3 & Q2 (Differentiating Heterogeneity from Personalization)

Heterogeneity of users in federated recommendation syetems is the challenge we aim to address and the additive personalization is our proposed approach to address the challenge. So they are not seperated. FedRAP learns a user-specific item embedding and a globally shared item embedding jointly to enhance the performance in federated recommendation.

W4 (Organizational Refinements)

We will improve the organization and logical flow of the paper as you suggested in the updated draft.

W5 & Q1 (Clarification on ‘Reconstruction Error’)

In our paper, the term "reconstruction" refers to predicting the user ratings given in the training data. The reconstruction or prediction of ratings is achieved by the inner product between each user embedding $\mathbf{u}_i$ and each item embedding, which adds the global item embedding $\mathbf{C}$ and local item embedding $\mathbf{D}^{(i)}$ together. The reconstruction error compares the prediction with the ground truth ratings by Eq. (1).

W6 (Questioning Novelty)

FedRAP addressed a key challenge in federated recommendation and achieved significant improvement compared to existing methods on standard benchmarks. Its performance is unprecedently on par with the centralized recommendation system performance. This is due to the following technical novelties: (1) Additive modeling combining a local item embedding with a sparse global item embedding, which has not been studied for the problem; (2) A curriculum learning approach with two regularizations, which is critical to achieve the proposed additive modeling.

W7&Q4 (Comparative Analysis)

We will cite both PerFedRec and PerFedRec++ and discuss them in our uploaded draft. We are still tuning the hyperparameters for PerFedRec and aiming to include it in our experimental comparisons. Since PerFedRec++ has not released its code and it is a very recent preprint (published later than our preprint), we will try our best to include it in the future version of our paper.

W7&Q4 (Comparative Analysis)

We conducted a comparative analysis of PerFedRec and FedRAP on ML-100K and ML-1M. The results, as shown in Table R3, indicate that FedRAP demonstrates competitive performance compared to PerFedRec. We are still tuning the hyperparameters for PerFedRec on more datasets.

Table R3. Experimental results of PerFedRec and FedRAP shown in percentages on the ML-100K and ML-1M dataset.

Dear Reviewer QwLj,

In the rebuttal, we have rephrased the motivation and novelty, highlighting the role of sparse constraints on $\mathbf{C}$. We have conducted a comparative analysis of PerFedRec and FedRAP on ML-100K and ML-1M, with the results presented in Table R3. Furthermore, we are refining the manuscript to enhance consistency in expression and reduce typographical errors. In future releases, we plan to cite and discuss PerFedRec++ extensively.

As we are approaching the midpoint of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Should there be any lingering points that require further attention, please rest assured that we are enthusiastic about the opportunity to provide comprehensive responses to any subsequent queries or comments you may have.

Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Thank you for your time and consideration.

Best,

Authors

Thank you for your constructive comments and support!

W1 (Unclear Motivations)

In Section 4.5, we introduced eight variants of FedRAP based on three ablation study tasks. Specifically:

1. To investigate the roles of global item embedding and local item embedding in FedRAP and validate the basic assumption of user preference influencing ratings, we designed FedRAP-C using only global item embedding and FedRAP-D using only local item embedding.
2. To explore the impact of different constraints on C, we introduced FedRAP-No, which imposes no constraints on C, and FedRAP-L2, which uses the Frobenius norm on C.
3. To examine the influence of different weight curricula on FedRAP's performance, we introduced FedRAP-fixed using two constants $v_1$ or $v_2$ in hyperparameters, FedRAP-sin using a sine function for hyperparameter variation, and FedRAP-frac using fractional changes in hyperparameters. Additionally, we presented FedRAP-square with a hyperparameter periodically transitioning between zero and constants $v_1$ or $v_2$.

These three ablation study aim to validate the contributions of three key innovations in FedRAP: two-way personalization, encouraging the sparsity of C, and a curriculum transitioning from full to additive personalization.

W2 (Improve Writing)

Thank you for bringing this to our attention! We will thoroughly review the paper and address any typos and grammar errors to ensure its clarity and quality. Your feedback is valuable in enhancing the overall presentation of the work.

Q1 (Ablation Study)

We introduce a new variant of FedRAP called FedRAP-S, where we remove the regularization $- \lambda_{(a, v_1)} ||\mathbf{D}^{(i)} - \mathbf{C}||^2_F$ encouraging the difference between the global item embedding $\mathbf{C}$ and local item embedding $\mathbf{D}^{(i)}$ from Eq. (3). We trained FedRAP-S five times on the ML-100K dataset and report the results in Table R1, where FedRAP-S shows a significant performance degradation compared to FedRAP. This degradation could be attributed to the absence of constraints between user-specific item information and globally shared item information in FedRAP-S. Without such constraints, the personalized model becomes overly individualized, leading to overfitting on user data and a decrease in generalization ability. Therefore, enforcing differentiation serves as a bridge between personalized and shared models and controls the complexity of the personalized component and preventing overfitting. This ensures a balance between personalization and generalization in the model.

Table R1. Experimental results of FedRAP and FedRAP-S shown in percentages on the ML-100K dataset.

Q2 (Typo)

It should be "FedRAP-No: Removes item sparsity constraint on C" instead of "diversity." This is indeed a typo. We appreciate your keen observation, and assure you that this error will be corrected in the future version of the paper.

Q3 (Explanation of Fig. 7)

In Fig. 7, the row-sparsity pattern in each user-personalized matrix, denoted as local item embedding $\mathbf{D}^{(i)}$, indicates non-zero elements corresponding to items the i-th user has interacted with. Conversely, all-zero rows signify items that the user has not accessed. To address this, global item embedding $\mathbf{C}$ is introduced to incorporate global item information, allowing FedRAP to predict whether a user will interact with items they have not engaged with before. It's noted that optimizing the storage of $\mathbf{D}^{(i)}$ can reduce client-side space cost. However, since FedRAP transmits only $\mathbf{C}$ between the client and server, optimizing the storage of $\mathbf{D}^{(i)}$ does not lead to a reduction in communication costs.

Q4 (Baseline Comparison)

We have evaluated the performances on the Movielens-100K (ML-100K) dataset, comparing FedMF, FedNCF, PFedRec, and FedRAP with and without Differential Privacy (DP). Due to space constraints in the main text, the detailed results of our experiments are provided in Appendix Sec. B.4, as illustrated in Tables 5 and 6. The results demonstrate that FedRAP maintains a performance advantage, even in the presence of privacy-preserving measures. These findings showcase the robustness and efficacy of FedRAP in privacy-preserving federated recommendation scenarios.

We would like to appreciate you for your constructive comments and support!