Debiased Collaborative Filtering with Kernel-Based Causal Balancing

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[W1] It is advisable to include a discussion of the relationships or distinctions between this work and prior research in the Related Work section.

Response: Thank you for your kind advice. As suggested by the reviewer, we add a discussion of the relationships and distinctions between our work and prior research in the Related Work section for both “Debiased Recommendation” and “Covariate Balancing in Causal Inference” parts.

[W2.1] Why balancing the kernel functions with maximal $|\alpha_{s, t}|$ can contribute the most to the $|e_{u, i}|$?

Response: Taking CBIPS as an example, in the last equation on Page 5, we use the universal property and the representer theorem to show the bias of CBIPS estimator is

$\operatorname{Bias}^2(\mathcal{L}_{\mathrm{CBIPS}}(\theta))$

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[W1 & Q1] How to update the weights for the problems where the number of items and/or the number of users are large?

Response: We thank the reviewer for raising an interesting concern.

• First, the balancing weights $w_{u, i}$ are parameterized using a deep neural network model $g(x_{u, i}; \phi_w)$, which ensures the model scalability and the model parameters $\phi_w$ could be much less than the number of user-item pairs when the number of items and/or the number of users is large.

• Second, we can write the primal optimization problem to the Lagrange dual problem, in which the number of parameters is the same as the number of constraints $O(J)$, and is independent with the user numbers and item numbers.

• Third, for the optimization problem, we can uniformly sample a batch in $\mathcal{D} =$ u_1, u_2, ..., u_m $\times$ i_1, i_2, ..., i_n $$ to compute $\tau^{(j)}$ for $j = 1, 2, ..., J$, instead of using all $m \cdot n$ user-item pairs that concerns the computational efficiency.

[W2] How to incorporate the proposed method into the modern deep learning (DL) based recommender system?

Response: We thank the reviewer for the useful comments. The proposed methods are model-agnostic, which means the balancing weight model $g(x_{u, i}; \phi_w)$ and the prediction model $f(x_{u, i}; \theta)$ can adopt the modern deep neural networks. To show this, we add the experiment on a large-scale industrial dataset Product using neural collaborative filtering (NCF) [1] as the base model to empirically show the performance of the proposed methods in Appendix B, Table 2. The results are shown below.

The experiment results show that the proposed kernel balancing methods achieve overall performance improvement compared to the baseline methods.

[Q2] Is it possible to adjust this method as a means of fine-tuning the item and/or user embeddings?

Response: Yes, it is possible to fine-tune the embeddings using our proposed methods. We illustrate this by using click-through rate (CTR) prediction and post-click conversion rate (pCVR) prediction in e-commerce recommendation as an example. From a debiasing formulation, the data formats and the quantity of interest in the "offline rating prediction" scenario and the "e-commerce conversion rate prediction" scenario are exactly the same [2-4]. We provide a comparison table as below.

Therefore, we can adopt the method proposed in this paper to adaptively choose $\phi(x_{u, i})$ as a kernel function $K(x_{s, t}, x_{u, i})$ with some fixed $(s, t)$ and then fine-tune the CTR model $\hat{p}(o = 1|x_{u, i})$. One possible way to fine-tune the CTR model is minimizing the directed Kullback entropy divergence [5] defined by $h(w_i) = w_i log({w_i}/{q_i})$ between the new and the original CTR model predictions as the objective function in the proposed entropy-based optimization problem. Moreover, we can further fine-tune the pCVR model based on the fine-tuned CTR model.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

1. Motivation

[W1] This paper lacks a sufficient motivation for the importance of carefully selecting the function class for balancing. The authors do discuss the computational cost associated with choosing numerous balancing functions in Section 3.2. However, the original paper [1] suggests that even selecting a single function for balancing can yield performance improvements. It is essential to elucidate why the choice of multiple balancing functions is necessary and why the specific type of function is critical. In my view, the motivation may be derived from Corollary 1 to some extent, which asserts that certain types of balancing functions introduce bias (correct me if I am wrong). Nonetheless, it would be valuable to provide a more insightful explanation in the introduction section.

Response: We thank the reviewer for the careful and insightful thoughts, and have revised the Introduction Section in our revised manuscript.

• First, we agree that “the original paper [1] suggests that even selecting a single function for balancing can yield performance improvements”, and our experimental results also prove this.

• However, on the one hand, it is not realistic to balance all possible functions for a specific model using only finite samples.

• On the other hand, balancing only a single function is not sufficient for IPS and DR methods to achieve unbiased learning (see Corollary 1 for the formal theoretical results).

• Thus, it is necessary to discuss which functions should be more favored to be balanced for the IPS and DR estimators, resulting in smaller estimation biases of the ideal loss and enhanced performance of unbiased learning.

2. Clarity

[W2] The optimization problems in Sections 4.2 and 4.4 are perplexing. The rationale behind minimizing the sum of $g(\cdot)\log g(\cdot)$ as opposed to a standard cross-entropy loss is unclear and confusing. The purpose of constraining the sum of $o_{u, i}g(\cdot)$ to be one and the meaning of the third constraint require more detailed explanation and motivation. The authors should clarify the significance and reasoning behind each statement in the proposed optimization problem.

Response: We thank the reviewer for pointing out this issue, and have added detailed explanations regarding the optimization problems in Sections 4.2 and 4.4. Both of the optimization problems consist of the following three features.

• First, the objective function is the empirical entropy of the balancing weights, by the principle of maximum entropy, it reaches the maximum value when the balancing weights are uniform, thus effectively avoiding high variance due to extremely small propensities.

• Second, the balancing constraints are imposed to equalize the selected covariate functions between the observed and missing samples.

• Third, two normalization constraints are imposed, which implies that the weights sum to the normalization constant of one, and the nonnegativity of the balancing weights, making the empirical entropy as the objective function well-defined.

• For the optimization problem in Section 4.4, we further introduce a slack variable $\xi_j$ for each balancing function and a pre-specified threshold $C$, since achieving strict balancing on all balancing functions is usually infeasible as the number of balancing functions increases.

[W3] Several theorems and corollaries, such as Theorem 1, Corollary 1, and Lemma 2, lack supporting proofs. In particular, the proof of Corollary 1, which appears directly relevant to the paper's motivation, is crucial.

Response: We really apologize for the lack of supporting proofs, and have added detailed proofs for Theorem 1, Corollary 1, and Lemma 2 in Appendix A.

[W4] The paper suggests that "balancing propensity can reduce the generalization bound", but this claim is not clearly evident from Theorem 2, which seems to provide a bound without a comparative analysis.

Response: We thank the reviewer for pointing out this issue, and have added detailed discussions in Section 4.5, explaining the reason why the derived generalization bound in RKHS is able to be greatly reduced when adopting the proposed adaptive KBDR learning approach as in Alg. 1.

• For the first term in the generalization bound, the prediction model minimizes the loss $\mathcal{L}_{KBDR}(\theta)$ during the model training phase.

• For the second term in the generalization bound, as shown in Theorem 1 and Corollary 1, the proposed adaptive kernel balancing method can automatically choose the balancing functions that most need to be balanced to reduce the bias of the KBDR estimator.

[Intuitive Example] While I still hope for an intuitive example or explanation illustrating why "balancing only a single function is not sufficient for IPS and DR methods to achieve unbiased learning", referencing Corollary 1 is enough for conveying the paper's motivation to readers.

Response: Let $\phi(X)$ be a vector of functions of $X$ to be balanced, such as $\phi(X)=\left(X, X^2\right)$. It assign a balancing weight $w_{u, i}$ to each observed user-item pair $\\{(u, i): o_{u, i}=1\\}$ by

where $\tilde{\phi}=\frac{1}{|\mathcal{D}|} \sum_{(u, i)\in\mathcal{D}} \phi\left(X_{u, i}\right)$.

• Intuitively, if $e_{u, i}$ is a linear function of $\phi(X)=\left(X, X^2\right)$, that is, $e_{u, i}=\phi\left(X_{u, i}\right)^T \beta+\epsilon_{u, i}, \mathbb{E}\left[\epsilon_{u, i} \mid X_{u, i}\right]=0$. Then the IPS estimator with the balancing weights $w_{u, i}$ is unbiased

where the approximately equal holds via the balancing constraints in the optimization problem.

• Instead, if $e_{u, i}$ is not a linear function of $\phi(X)=\left(X, X^2\right)$, then it is clear that the IPS estimator with balancing weights $w_{u, i}$ is not unbiased, and only balancing $\phi(X)=\left(X, X^2\right)$ is not enough.

Please kindly refer to the intuitive example we provided further above. We are glad to know that your concerns have been effectively addressed. We are very grateful for your constructive comments and questions, which helped improve the clarity and quality of our paper. Thanks again!