Unbiased Recommender Learning from Implicit Feedback via Weakly Supervised Learning

[W1] More discussion on scalability.

Response. We express our sincere gratitude for this valuable comment. We add a scalability analysis on the additional ML-1M dataset that is larger than the datasets involved in this paper. In the below table, we summarize the performance of PURL given varying ratios of unlabeled samples (denoted as P).

As P increases, the performance of PURL generally increases with some fluctuations. The best performance is achieved when P>0.9, which is consistent with the results reported in the main text and showcases scalability. Compared to the results on smaller datasets, the performance is less sensitive to P. This can be attributed to the fact that even a small P in a large dataset corresponds to a large number of unlabeled samples, which suffices to meet the demand of unlabeled samples in Theorem 2 to reduce variance and improve overall performance.

[W2] The direct impact of PPT on performance.

Response: ! In this work, we introduce two components: (1) the PURL loss, which effectively leverages unlabeled samples; and (2) the PPT strategy, which accurately estimates class priors for calculating the PURL loss. To quantify the individual contributions of these components, we conducted a series of ablative experiments.

• To discern the contribution of PURL, we varied the proportion of unlabeled samples (denoted as P) used in the PURL loss calculation, as illustrated in Figure 2. On both datasets, the performance consistently improves as P increases, which demonstrates the effectiveness of PURL.
• To discern the contribution of PPT, we compared the performance of PURL with PPT-estimated class prior and other values of class priors, in Table 3. The results show that the PPT-estimated class prior consistently led to the best recommendation performance. This indicates that the PPT class prior is effective in estimating the class prior and thus improving the ImplicitRec performance.

[Q1] Could PPT be extended to multi-class implicit feedback scenarios?

Response. We agree that PPT can be extended to multi-class scenarios. A plain approach would be treating the multi-value class labels as multiple binary classes. For example, if we have two positive classes (1, 2) and a negative class (0), we can treat them as two binary classes (0 v.s. 1 and 0 v.s. 2). The PPT can be applied to each binary class separately.

[Q2] Sensitivity to hyperparameter tuning.

Response. Once again, we sincerely appreciate the reviewer's meticulous and constructive comments. We discuss the sensitivity to hyperparameter tuning in the following two aspects:

• Firstly, for the recommendation performance, we conducted comprehensive experiments to investigate the sensitivity of key hyperparameters, including the ratio of unlabeled samples (Fig. 5), batch size (Fig. 7-a), learning rate (Fig. 7-b), and embedding dimension (Fig. 7-c). Different datasets exhibit varying sensitivity to these hyperparameters, which is largely dependent on the specific characteristics of the data. The detailed analysis is provided in Section 4.5 and Appendix B.3.
• Secondly, for the class prior estimation, the PPT approach is generally isolated from these hyperparameters. To showcase this claim, we investigate the logs of sensitivity analysis that record the class prior estimation results $\hat{\kappa}_p$. Below are the results with different batch sizes, averaged over 5 random seeds:

Across a wide range of batch sizes, the estimated class prior $\hat{\kappa}_p$ remains stable around 0.05 (i.e., the class prior producing the best recommendation performance in Table 3). It indicates that PPT is robust to variations in hyperparameters.

[References] MNF in sequential recommendation should be discussed.

Response. Thank you very much for your insightful comment. We agree that the MNF problem also matters in sequential recommendation. In the revised version, we will include discussions on key works in this area, including but not limited to [1,2,3].

[1] Modeling dynamic missingness of implicit feedback for recommendation. NeurIPS. 2018.

[2] Modeling dynamic missingness of implicit feedback for sequential recommendation. IEEE TKDE. 2020.

[3] Sequential learning over implicit feedback for robust large-scale recommender systems. PKDD. 2019.

[W1] Ablation study matters.

Response: Thank you for your kind comment! In this work, we devise two components: (1) the PURL loss; (2) the PPT strategy. We have conducted experiments to discern their individual contributions.

• To discern the contribution of PURL that leverages unlabeled samples, we have conducted ablative experiments by changing the proportion of unlabeled samples involved in Fig.2. The results show that on both datasets, the performance increases with the proportion of involved unlabeled samples, which demonstrates the effectiveness of PURL in leveraging unlabeled samples.
• To discern the contribution of PPT that estimates class priors, we have conducted ablative experiments in Table 3. The results show that the class prior estimate of PPT, with the smallest $\mathbb{W}$, corresponds to the best recommendation performance.

[W2, Q1] Challenge of propensity estimation.

Response: Thank you for your insightful comment. We agree that the propensity score estimation is a general challenge, but we demonstrate that this challenge is exacerbated in ImplicitRec, for the following reasons:

• In recommendation, the propensity score is typically the probability of a specific treatment given a user-item pair.
• In ImplicitRec, this probability is hard to estimate due to the absence of negative feedback. For example, in CTR prediction, the propensity score is the exposure probability of a user-item pair. However, ImplicitRec only provides positive (exposure & click) and unlabeled samples (non-exposure), lacking negative samples (exposure & non-click). This makes estimating exposure probability infeasible.
• This challenge is unique to ImplicitRec and significantly hampers the effectiveness of existing propensity-based methods in achieving unbiased recommendations in ImplicitRec.

[W2, Q2] Why does ImplicitRec model outperform some explicit models?

Response: Thank you for your insightful comments. The phenomenon is noticable and we justify it from two aspects.

• Theoretical aspect. The proposed PURL loss $R_\mathrm{purl}$ reduces the variance of the ideal binary loss $R_\mathrm{ideal}$ that uses explicit feedback, given $\kappa_p\leq 0.5$ (a naturally holding condition) and large number of unlabeled samples. The variance reduction property is justified in Theorem 2, which leads to better generalization and performance.
• Empirical aspect. Figure 5 shows consistent performance improvement with more unlabeled samples, validating the theoretical aspect above. Thus, more unlabeled samples decrease PURL variance and improve performance over explicit feedback methods.

[W2, Q3] The connection with noisy label recommendation.

Response. Yes, it is feasible to employ learning-from-noisy-label techniques to address the MNF issue within ImplicitRec. However, the weakly supervised learning framework used in this work is more suitable for two main reasons:

• Theoretical supports. Most noisy label methods are heuristic or rely on strong assumptions about the noise mechanisms. In contrast, the weakly supervised learning framework provides theoretical guarantees without such harsh assumptions.
• Empirical supports. One exemplar work approaching ImplcitRec as learning from noisy labels is T-CE [1]. Our experiments below show that PURL outperforms T-CE, demonstrating the effectiveness of using weakly supervised learning for ImplicitRec.

[W3] Originality of PPT.

Response: Thank you for the reminder. The PPT generalizes the canonical OT problem by relaxing the equality constraint for $\beta$ and changes its total mass from 1 to $w$. This formulation is specifically designed for ImplicitRec and, to our knowledge, has not been previously studied.

[W4] Project website.

Response: Thank you for your kind reminder. The repository has now been restored. Additionally, we offer a Docker file to facilitate quick reproduction.

[Reference and Other comments].

Response. Thank you for your meticulous comment. We will fix the typos, remove the invalid reference, adjust the related works and unify the reference formats in revision.

[1] Denoising implicit feedback for recommendation. WSDM. 2021.

Thank you very much for your kind and sincere words. It is impressive to meet a review that candidly express limitations while responsibly and analyzing the claims and details of the paper. We truly appreciate it!

We have noticed the question about the dataset generation process and are happy to provide a detailed response.

• Firstly, we chose the Yahoo and Coat datasets, which are commonly used in prior ImplicitRec studies such as CUMF [1], UBPR [2], and UPL [3].
• In the training set, we retained only user-item pairs with positive feedback and treated those with negative feedback as unlabeled. This reflects real-world implicit feedback scenarios where only positive interactions are observed, and the rest are unlabeled. In the test set, we preserve the explicit feedback which is necessary to calculate the metrics (NDCG, Recall, etc.), mirroring the online serving scenario of implicit feedback recommendation.
• This process aligns with common practices in ImplicitRec research [1-3].

Reference.

[1] Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. 2020. Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feed- back. In WSDM. ACM, 501–509.

[2] Yuta Saito. 2020. Unbiased Pairwise Learning from Biased Implicit Feedback. In ICTIR. ACM, 5–12.

[3] Yi Ren, Hongyan Tang, Jiangpeng Rong, and Siwen Zhu. 2023. Unbiased Pairwise Learning from Implicit Feed- back for Recommender Systems without Biased Variance Control. In SIGIR. ACM, 2461–2465.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address the raised concerns point by point and try our best to clarify any confusions.

[W1] The connection between mass weight $w$ to class prior estimation is not explicit.

Response. We agree that the connection needs clarification.

• Connection clarification. We estimate the class prior as the inverse of the mass weight, i.e., $\hat{\kappa}_p=w$.
• Rationale of the connection. The PT problem is designed to match the positive samples ($\alpha$) with a selective set including $1/w$ of the unlabeled samples ($\beta$). If there exist negative samples in the selective set, the PT discrepancy will be large. By finding the $w^*$ with minimum discrepancy, we can estimate the class prior $\hat{\kappa}_p = 1/w^*$. This process is visualized in Fig. 1.
• Toy example. We provide simulated results in Fig.4. The ground-truth class prior $\kappa=0.5$. When we set $w=2$, all positive samples are exactly matched with the proximal positive samples within the unlabeled population, producing the minimal transport cost.

[W2] The claim concerning BPR requires elaboration.

Response. Thank you for your insightful comment. We elaborate the claim as follows, and promise to add it in revision.

• The BPR loss randomly samples unlabeled samples as negatives. However, not all unlabeled samples are true negatives. Some may simply be items the user hasn't encountered yet. This misclassification makes BPR biased.
• In contrast, our proposed PURL loss is unbiased given accurate class prior estimation. This is supported by Theorem 1 in our paper, which provides a theoretical support for the advantage of the proposed PURL.

[W3] Computational time and memory requirement.

Response. Thank you for your insightful question. We add the analysis on computational time and memory consumption below.

• The complexity of solving the proposed PPT problem is mainly determined by the number of samples. We conducted an empirical study on its running time, revealing that, although the theoretical complexity remains not straightforward to derive, the running time scales linearly with the number of samples on a log-log scale. In the largest test with batch size of 2,048—exceeding typical batch sizes in most practices—the running time was under 0.1 seconds, rendering it negligible compared to model training.
• The memory consumption is also affected by the number of samples. We add experiments where we used the decorator profile in the memory_profiler package to test the memory consumption in solving the PT problem. According to the results below, in the largest setting of 2048 samples, the memory consumption is 52.9 MB, which is acceptable and even negligible compared to the memory cost of training typical recommendation system.

• We promise to adding these discussions in the revised manuscript.

[W4] The principle of dataset selection should be explained.

Response. Thank you for your kind reminder. We select the datasets based on the following principles:

• Firstly, we opt for the datasets that contain unbiased test set with explicit feedback. This is crucial for evaluating the performance of our method, where the explicit feedback is used to calculate the metrics (NDCG, Recall, etc.), and the unbiased set mirrors the online serving scenario.
• Secondly, we follow the practice of previous works in the ImplicitRec field. Specifically, the employed Yahoo and Coat datasets were used in CUMF (WSDM'20), UBPR (SIGIR'20), UPL (SIGIR'23), CDR (ICLR'23) and SDR (ICLR'23).
• Moreover, motivated by UIDR (ICLR'24), we add an additional dataset--Kuairec--to provide comprehensive evaluation of the proposed method on large-scale industrial dataset.

[W5] The derivation of Eq.7.

Response. Eq. 7 defines a proximal transport problem, which is a generalization of the canonical OT problem in Eq. 2.

• To derive Eq. 7, we replace the marginal constraint $\boldsymbol{\pi}^\top \boldsymbol{1}_n=\mathbf{b}$ with $\boldsymbol{\pi}^\top \boldsymbol{1}_n \leq w \mathbf{b}$ with $w\geq 1$, which acquires a semi-relaxed OT.
• On the basis, we add the constraint $\boldsymbol{1}_m\boldsymbol{\pi}^\top \boldsymbol{1}_n = 1$, which forces the full mass to be transported to be 1.
• Then, given the assumption that the mass in $\alpha$ and $\beta$ is uniformly distributed and sums to 1, we have $\mathbf{a}=\boldsymbol{1}_n$ and $\mathbf{b}=\boldsymbol{1}_m$. The derivation of Eq. 7 is completed.

[Other comments] Consistency of terms.

Response. Once again, thank you for your valuable comment. They refer to the same thing and we will use a unified term "user-item pair" in revision.