Be Aware of the Neighborhood Effect: Modeling Selection Bias under Interference

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[Q1] In practice, how do you specify the probability density function of $\boldsymbol{g}$.

Response: We thank the reviewer for the comment. In our experiments, we choose the one with the better debiasing performance among a uniform distribution or a Dirichlet distribution for $\boldsymbol{g}. In practice, some other distributions can also be chosen.

[Q2] In practice, what is the best choice between Epanechnikov and Gaussian kernels? Experiments seem to have been done only with the Gaussian kernel.

Response: We thank the reviewer for the question. Yes, in our original manuscript, the experiments are only done with Gaussian kernel. For exploring and comparing the Epanechnikov and Gaussian kernels, we add the experiments that our methods adopting Epanechnikov kernel on all three dataset (Page 8, Table 2) in our revised manuscript. The results are shown below, the best two results are bolded in IPS based, DR-JL based and MRDR-JL based methods, respectively.

Adopting either the Gaussian kernel or the Epanechnikov kernel in our methods is able to stably outperform the baseline methods in all metrics and these two kernels yield similar performance of our methods.

[Q3] Can you please detail again for the semi-synthetic experiments how do you define the set of $\mathcal{N}_{u,i}$ as the set of historical user and item interactions for the neighbors of (u,i) who do have an influence on user u?

Response: We thank the reviewer for pointing out this issue. In the semi-synthetic experiment, we suppose that all the $o_{u, i'}$ which $i' \neq i$ affects $r_{u, i}$. The intuition behind is that users who have seen better movies than the current movie are more likely to have a low rating towards the current movie than the average rating. It is intuitive and is verified in the case study (Appendix I) in our revised manuscript. Meanwhile, we suppose that all the $o_{u', i}$ which $u' \neq u$ affects $r_{u, i}$ because users may subjectively believe that movies that have been watched more times are of better quality. So we suppose $\mathcal{N}_{u, i}$ is {$({u'}, i')\neq (u, i) \mid u'=u~ \text{or}~ i'=i$} in the semi-synthetic experiment.

[Q4] How does $p_{u, i}=p \alpha^{\max \left(0, 4-r_{u, i}\right)}$ account for the neighboring effect too?

Response: We thank the reviewer for the question. In fact, $p_{u, i}=p \alpha^{\max \left(0, 4-r_{u, i}\right)}$ only taking MNAR effect into account. Neighborhood effect is considered by splitting $r_{u, i}$ into $r_{u, i}(1, g)$, introducing $\pi_{g}$ and setting the ideal loss to $\tilde{\mathcal{L}}_\mathrm{ideal}(\hat{\mathbf{R}})$.

[Q5] c is chosen to be the median of all g_u,i according to p. 7 but g_u,i is also defined depending on c.

Response: We thank the reviewer for pointing out this issue and we apologize for the typo here. Actually, $c$ is chosen to be the median of all $\sum_{ (u',i') \in N_{(u,i)} } o_{u',i'}$. We have fixed this typo in our revised manuscript.

[Q6] Does KuaiRec specify the MAR and MAR watching ratio records? If not, how to measure the neighboring effect?

Response: We thank the reviewer for the useful question. We manually split the MNAR and MAR set in KuaiRec dataset. Because KuaiRec is a fully exposed dataset, we uniformly sample 5% watching ratio records for each user to generate the MAR set. Meanwhile, we adopt the same technic as semi-synthetic experiments to manually set a propensity for each user-item pair to sample the MNAR set.

[Q7] Can you please elaborate: “we adjust p to ensure the total observed sample is 5% of the entire matrix”?

Response: We thank the reviewer for the detailed question. As reviewer mentioned, the definition of $p_{u, i}$ is $p_{u, i} = \textcolor{red}{p} \alpha^{\max \left(0, 4-r_{u, i}\right)} \cdot m_{u, i}$, where $\textcolor{red}{p}$ is a pre-specified constant. The expected observation number of user-item pair is $\sum_{(u,i) \in \mathcal{D}} p_{u, i}$. Following the previous studies [1-2], we adjust $\textcolor{red}{p}$ to ensure $\sum_{(u,i) \in \mathcal{D}} p_{u, i} = 0.05 * |\mathcal{D}|$ for different mask number $m_{u, i}$.

[W1] Some typos and unclear Figure 1.

Response: We thank the reviewer for pointing out some typos and unclear figure in our original manuscript. In our revised manuscript, we fixed those typos and add a notation table in Figure 1 to improve the readability.

---

We hope the above discussion will fully address your concerns about our work. We really appreciate your insightful and constructive comments to further help us improve the quality of our manuscript. Thank you!

---

Reference

[1] Tobias Schnabel et al. Recommendations as Treatments: Debiasing Learning and Evaluation. ICML 16.

[2] Xiaojie Wang et al. Doubly Robust Joint Learning for Recommendation on Data Missing Not at Random. ICML 19.

2. Assumptions

[W2.1] why $r_{u, i}$ is affected by $o_{u, i}$ ? In my opinion, $o_{u, i}$ is just a treatment to observe $r_{u, i}$ and does not affect the 'value' of $r_{u, i}$. (i.e., the value of $r$ is affected only by $x$ and observed only when $o=1$).

Response: In fact, $o_{u, i}$ is not just a treatment to observe $r_{u, i}$ and does affect the 'value' of $r_{u, i}$. We further clarify the definition of $o_{u, i}$ in Section 2 and add a notation table in Figure 1 to make the presentation clearer.

• Specifically, $x_{u,i}$, $o_{u,i}$, and $r_{u,i}$ are the feature, treatment (e.g., exposure), and feedback (e.g., conversion)} of the user-item pair $(u,i)$. These broad concepts can have different meanings in different recommendation scenarios.
• For instance, one may consider $o_{u,i}$ equals 1 or 0 represents whether the item $i$ is exposed to user $u$ or not, and $r_{u,i}$ is the conversion indicator. Thus it is plausible that the exposure of the item affects the conversion.

[W2.2] If $r_{u, i}$ is affected by $o_{u, i}$, I think the assumption 3 is not hold.

Response: We would like to distinguish here between the observed feedback $r_{u, i}$ and the potential feedback $r_{u, i}(1)$.

• We agree with the reviewer that $r_{u, i}$ is affected by $o_{u, i}$, e.g., the exposure of the item affects the conversion.

• Nonetheless, what assumption 3 states is that the potential feedback $r_{u, i}(1)$ is independent with the exposure indicator $o_{u, i}$.

• Since the definition of $r_{u, i}(1)$ is the potential feedback that would be observed if item $i$ had been exposed to user $u$ (i.e., $o_{u,i}$ had been set to 1). Thus it is plausible that $r_{u, i}(1)$ is independent with $o_{u, i}$.

[W2.3] In the paper, $g$ is a scalar (a continuous variable), not a representation vector.

Response: We thank the reviewer for pointing out this issue.

• On a theoretical level, our approach allows $g$ to be a vector. Noting that there is a significant difference in the statistical theory when $g$ is a vector and a scalar, we discuss in detail the new proofs and the theorems in main text for multi-dimensional $g$ in Appendix G.

• On a experimental level, as the reviewer captured, we only considered the cases when $g$ is a scalar (a continuous variable), but such implementation has already achieved significant performance improvement. In the future, we will conduct more experiments to explore the use of multi-dimensional $g$ to implement the proposed methods.

3. Minor concerns

• In the real-world experiment, the authors use 5% MAR test ratings for the propensity estimation. This process is unrealistic.

Response: Indeed, in our original manuscript, we implement the proposed methods using both Logistic Regression (LR) and Naive Bayes (NB) for estimating the propensities. LR does not require any MAR test ratings for the propensity estimation, whereas NB requires 5% MAR test ratings for the propensity estimation. We carefully revised the statement about propensity estimation in Section 6, as well as highlighting the results of the experiments using LR and NB in Table 2.

[W3.2] In the semi-synthetic experiment, the definition of neighborhood effect is the number of neighbor pairs with $o \geq c$. what does it mean? Since $o \in$ {$0$, $1$}, I cannot understand the $c$ is chosen to be the median of all $\boldsymbol{g}$.

Response: We thank the reviewer for the careful reading and apologize for the typo here. Actually, $c$ is chosen to be the median of all $\sum_{ (u',i') \in N_{(u,i)} } o_{u',i'}$. We have fixed this typo in our revised manuscript.

---

Finally, we would like to kindly remind the reviewer that he/she might have mistakenly flagged "Flag For Ethics Review" due to checking both "No ethics review needed" and "Yes, Responsible research practice (e.g., human subjects, data release)".

We hope the above discussion will fully address your concerns about our work. We really appreciate your insightful and constructive comments to further help us improve the quality of our manuscript. Thank you!

---

References

[1] Yu Zheng et al. Disentangling User Interest and Conformity for Recommendation with Causal Embedding, WWW 2021.

[2] Mouxiang Chen et al. Adapting Interactional Observation Embedding for Counterfactual Learning to Rank. SIGIR 2021.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

1. Motivation

Why do we need to eliminate the neighborhood effect? For example, existing recommenders can consider the neighborhood effect in the training phase and make recommendations with the neighborhood effect (e.g., similar users have similar embedding and thus get similar recommendations). Therefore, the neighborhood effect can be a rich information source for model training.

Response: The reviewer has raised an important point. We hope the following clarification addresses your concerns about the motivation.

• We agree with the reviewer that modern recommender systems consider the neighborhood effect in the training phase and make recommendations with the neighborhood effect.
• Nonetheless, we know that users tend to behave similarly to the others in a group, even if doing so goes against their own judgment, making the feedback in the observed data not reflect the true preferences of the users [1].
• We also notice another related work [2] referenced by Reviewer qJ35, in which the authors discuss the presence of interference in debiasing user feedback for the learning-to-rank (LTR) task.

Moreover, we also add a real-world example using the KuaiRec dataset demonstrating the necessity of eliminating the neighborhood effect for debiased recommendation in Appendix I, for a clearer comprehension of the motivation in this paper.

• For the same item interacted with different users, we use the user social network information to compute and compare the feedback similarity of friends and non-friends. The results are shown in Figure 3(a). Compared with the non-friend user pairs, it is clear that there is a higher similarity in the ratings of friends for the same item.

• For a given user with different items, we use timestamps to first select the $K$ most recent items this user has interacted with, then compute the average video viewing time among all users on the fully exposed dataset. As shown in Table 3 and Figure 3(b), we found that the lower the average viewing time of the items the user recently interacted with, the better the user's feedback on the current item will be, and vice versa. Such a feedback mechanism is reasonable: when users have previously observed more low-quality videos, they will provide better feedback on the current items.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we try our best to address these concern and update the manuscript accordingly.

[W1] Too many assumptions made the manuscript hard to follow.

Response: We thank for the reviewer for raising such concern. We would like to kindly remind the reviewer that all causal conclusions are based on a set of assumptions [1,2]. In this paper, we have adhered to common assumptions widely employed in causal inference.

It's important to highlight that the primary assumption underpinning our approach is Assumption 1 (neighborhood treatment representation). The remaining assumptions we utilize are standard in causal inference and kernel-smoothing estimation. To elaborate:

• Assumptions 2-3 are common assumptions (also called backdoor adjustment criterion in structural causal model, or unconfoundedness assumption in potential outcome framework) in causal inference to ensure the identifiability of the ideal loss (eq. (2)).
• Assumption 4 aligns with the standard assumption in kernel-smoothing estimation [3, 4, 5].
• Assumption 5 merely assumes the boundedness of propensities and imputed errors, which is a trivial assumption.

---

We hope the above discussion will fully address your concerns about our work. We really appreciate your insightful and constructive comments to further help us improve the quality of our manuscript. Thank you!

---

Reference

[1] Miguel A. Hernán et al. Causal Inference: What If. 2020.

[2] Guido W. Imbens et al. Causal Inference For Statistics Social and Biomedical Science. 2015.

[3] Jianqing Fan et al. Local Polynomial Modelling and Its Applications. 1996.

[4] Qi Li et al. Nonparametric econometrics. 2007.

[5] Wolfgang Härdle et al. Nonparametric and Semiparametric Models. 2004.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[W1] The breach of SUTVA and the presence of interference in debiasing user feedback were previously discussed in [1], where they also examined the interactions between propensity and implicit feedback on other items. I would like to see a discussion on it in this manuscript.

Response: We thank the reviewer for bringing up the relevant literature [1], and we have added a detailed discussion in the Related Work Section regarding the relation and difference between our paper and [1], and have added a clear citation to [1] as a key motivation of our paper in the Introduction Section.

• Despite both our paper and [1] exploring the influence induced by "other user-item interactions", it appears to be the sole similarity between the two studies.
• Firstly, for the problem settings, [1] focuses on the task of learning to rank (LTR), addressing position bias using implicit feedback data. In contrast, our paper focuses on eliminating selection bias in the rating prediction task using explicit feedback data.
• Secondly, for the basic ideas, [1] considers "other user-item interactions" as "confounders" (refer to the third paragraph in the Introduction Section). In contrast, our paper regards "other user-item interactions" as a new "treatment" from the perspective of interference in causal inference.
• Thirdly, for the proposed methods, [1] uses embedding as a proxy confounder to capture the influence of "other user-item interactions". In contrast, our paper formally formulates the influence of "other user-item interactions" as an interference problem in causal inference, and introduces a treatment representation to capture the influence. On this basis, we propose a novel ideal loss that can be used to deal with selection bias in the presence of interference. We also provide comprehensive theoretical guarantees.

[W2] A case study or a real-world example demonstrating the neighborhood effect would be valuable for a clearer comprehension of the underlying motivation.

Response: Thank you for the kind advice. As suggested by the reviewer, we add a real-world example using the KuaiRec dataset demonstrating the presence of the neighborhood effect in Appendix I, for a clearer comprehension of the motivation in this paper.

• For the same item interacted with different users, we use the user social network information to compute and compare the feedback similarity of friends and non-friends. The results are shown in Figure 3(a). Compared with the non-friend user pairs, it is clear that there is a higher similarity in the ratings of friends for the same item.

• For a given user with different items, we use timestamps to first select the $K$ most recent items this user has interacted with, then compute the average video viewing time among all users on the fully exposed dataset. As shown in Table 3 and Figure 3(b), we found that the lower the average viewing time of the items the user recently interacted with, the better the user's feedback on the current item will be, and vice versa. Such a feedback mechanism is reasonable: when users have previously observed more low-quality videos, they will provide better feedback on the current items.

[W3] To enhance the clarity, it is advisable for the authors to furnish pseudocodes delineating the procedural steps of the proposed estimator and the propensity estimation process.

Response: We thank the reviewer for pointing out this issue. As suggested by the reviewer, we add pseudocodes delineating the propensity estimation process in Alg. 1, as well as the procedural steps of the proposed N-IPS, N-DR, and N-MRDR in Alg. 2-4, respectively. Please kindly refer to Appendix H for the added 4 algorithmic details using pseudocodes.

---

We hope the above discussion will fully address your concerns about our work. We really appreciate your insightful and constructive comments to further help us improve the quality of our manuscript. Thank you!

---

Reference

[1] Mouxiang Chen et al. Adapting Interactional Observation Embedding for Counterfactual Learning to Rank. SIGIR 2021.