Unveiling Extraneous Sampling Bias with Data Missing-Not-At-Random

We thank the reviewer for efforts in evaluating our work and helpful suggestions. Below, we will try our best to address your concerns and questions.

W1 & Q1: Lack of detail on whether stabilization techniques (e.g., propensity clipping) were used, which affects reproducibility and variance control explanation.

Response to W1 & Q1: We thank the reviewer for pointing out these issues. Yes, we use a propensity clipping technique, and the clipping threshold is a hyperparameter tuned in [0.005, 0.05] on all three datasets. The best threshold is 0.0186 for Coat, 0.0127 for Yahoo! R3, and 0.0112 for the KuaiRec dataset. We will add these details in our revised version.

W2: The alternating optimization scheme increases overhead but is not analyzed or discussed in terms of cost-performance trade-offs.

Response to W2: We thank the reviewer for raising this meaningful concern.

• Note that the proposed method subtly modifies the imputation loss $\hat e$ by adding $\epsilon (o-p)$, which theoretically does not increase complexity compared to the original joint learning method, due to we only require one more gradient calculation and backward propagation for a single learnable parameter $\epsilon$.
• Empirically, the model structurally is also not more complex. Besides a single learnable parameter $\epsilon$, no new learnable parameters are introduced.
• We also add training time experiments on all three datasets under varying dataset scales (controlled by the ratio parameter $b$, varying in {0.1, 0.3, 0.5, 0.7, 0.9}, i.e., we randomly sample 100b% of users and items). We take DR-JL as the baseline. Thus, all results are 1 for DR-JL. The results show that the time complexity of our method is close to the doubly robust joint learning method, and much less than the previous SOTA DR-based methods, demonstrating the efficiency of our method.

W3: Lack of some important related work: [1] Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation [2] Cdr:Conservative doubly robust learning for debiased recommendation.

Response to W3: We thank the reviewer for the useful suggestions. We will modify the related work section to include these two important works. In addition, we add experiments to compare our method to these two methods. The results are shown below:

We find that our method still outperforms the baselines.

Q2: How were the inner loop steps and learning rates chosen in Algorithm 1?

Response to Q2: We thank the reviewer for the question. Following the previous implementation in [1-3], the step is fixed to 1. In addition, as shown in lines 250-251 in the manuscript, the learning rate is tuned in {0.001, 0.005, 0.01, 0.02, 0.05} for parameters in prediction, imputation, and propensity model, and in {0.01, 0.05, 0.1, 0.15, 0.2} for $\epsilon$. The best choice is {0.01, 0.15} for Coat, {0.005, 0.15} for Yahoo! R3, and {0.005, 0.1} for KuaiRec.

Q3: Why apply correction to the imputation model rather than the prediction output?

Response to Q3: We thank the reviewer for the interesting question. The conclusion is, applying the correction to either the imputation model or the prediction model will lead to exactly the same results! Here are the reasons.

• For updating the prediction model, we first recap the vanilla DR loss below:

  $\mathcal L_{\text{DR}}(\theta)=\frac{1}{|\mathcal{D}|} \sum_{(u, i) \in \mathcal{D}}\left[\hat e_{u, i}+\frac{o_{u, i}\left(e_{u, i}-\hat e_{u, i}\right)}{\hat p_{u, i}}\right].$ For updating the prediction model, note that whether adding the correction to the imputation model or the prediction model based on this DR loss will not affect the updating of the prediction model, because there is no gradient to the prediction model.

• For updating the imputation model, we recap the vanilla imputation loss function below $\mathcal L_e = \frac{1}{|\mathcal{D}|} \sum_{(u, i) \in \mathcal{D}} \frac{o_{u, i}\left(e_{u, i}-\hat e_{u, i}\right)^2}{\hat p_{u, i}}.$

Note that adding the correction to the imputation model or the prediction model based on this imputation loss also makes no difference when updating the imputation model and the parameter $\epsilon$.

Q4: Are MF-based features sufficient for generalization across disjoint user-item samples?

Response to Q4:

We thank the reviewer for raising this concern.

• We add experiments on all three datasets, demonstrating that MF-based features are sufficient for generalization. Specifically, for all three datasets, we randomly split the training data (i.e., MNAR data) into training / validation / test data, with a ratio 0.8 / 0.1 / 0.1. Then we use an MLP as the backbone to train a prediction model using the naive loss, i.e., loss on the observed user-item pairs. $\mathcal L_{\text{naive}}(\theta)=\frac{1}{|\mathcal{O}|} \sum_{(u, i) \in \mathcal{O}}e_{u,i}.$

The test performance on all three datasets corresponding to MSE and AUC is shown below:

• The results show that simple naive loss-based neural networks demonstrate excellent performance without data MNAR on all three datasets, proving MF-based features are sufficient for generalization.
• Furthermore, since KuaiRec is a more complicated industrial dataset, its lower results compared to Coat and Yahoo! R3 are in line with expectations.
• Finally, naive methods perform poorly with data MNAR (Coat and Yahoo! R3 have around 20% lower AUC, and KuaiRec around 10% lower AUC), highlighting the need for debiasing in real-world datasets.

We believe the above discussion has thoroughly addressed your concerns regarding our work. We would be grateful if you could kindly consider raising your rating. Thank you!

Thank you for your detailed response. My concerns have been adequately addressed, and I have decided to maintain my positive rating.

We thank the reviewer for efforts in evaluating our work and helpful suggestions. Below, we will try our best to address your concerns and questions.

W1. Assumption of Equal Joint Distributions: The super-population scenario assumes $P_{\rm train}(x,r)=P_{\rm test}(x,r)$; in practice, covariate drift or nonstationarity could violate this and undermine unbiasedness.

Response to W1: We thank the reviewer for the question. However, we think there is a huge misunderstanding.

• Note that we assume $P_{\rm train}(x,r)=P_{\rm test}(x,r)$ means that the target data distribution (not the observed data distribution) in training is the same as the distribution in test data. Then, due to the MNAR issue, the observed data distribution $P_{\rm obs}(x,r)$ is different from the test data.
• For example, the Coat dataset includes 290 users and 300 items with 6,960 MNAR observations, then we only assume target data distribution in training $P_{\rm train}(x,r)$ (the distribution of 290 * 300 = 87,000 user-item pairs) the same as the test distribution $P_{\rm test}(x,r)$, and we do not assume distribution on 6,960 observations $P_{\rm obs}(x,r)$ the same as the test distribution. We will make it clearer in our revised manuscript.

W2. Dependence on Propensity Pre-training: Requires a pre-trained propensity model (e.g., logistic regression) whose miscalibration could propagate through the correction step, yet sensitivity to propensity errors is not deeply explored.

Response to W2: We thank the reviewer for the question. The existing DR-based training algorithms are mainly divided into two types:

• The first is based on the joint training algorithm, which pretrains the propensity model, and then trains the imputation model and the prediction model, such as DR-JL [1], MRDR-JL [2], DCE-DR [3], etc.
• The second is based on the multi-task learning algorithm, which learns the propensity model, the prediction model, and the imputation model together, such as ESCM2-DR [4], SDR [5], DR-V2 [6], etc.

As suggested by the reviewer, we added experiments on all three datasets, Coat, Yahoo, and Kuairec, where the propensity model is 1 layer (i.e., logistic regression), 2 layers, and 3 layers neural networks, respectively, to explore the sensitivity to propensity errors of our method. In addition, we also implemented both joint learning and multi-task learning as the learning algorithm to investigate whether our method needs a pre-trained propensity model. The results are as follows:

The results show that our method is stable to the choice of propensity model and learning algorithm.

W3. Computational Overhead: Jointly learning three components (prediction, imputation, and $\epsilon$) may increase training time and tuning complexity compared to standard DR methods; scalability to very large item × user spaces warrants further study.

Response to W3: We thank the reviewer for raising this meaningful concern.

• Note that the proposed method subtly modifies the imputation loss $\hat e$ by adding $\epsilon (o-p)$, which theoretically does not increase complexity compared to the original joint learning method, due to we only require one more gradient calculation and backward propagation for a single learnable parameter $\epsilon$.
• Empirically, the model structurally is also not more complex. Besides a single learnable parameter $\epsilon$, no new learnable parameters are introduced.
• We also add training time experiments on all three datasets under varying dataset scales (controlled by the ratio parameter $b$, varying in {0.1, 0.3, 0.5, 0.7, 0.9}, i.e., we randomly sample 100b% of users and items). We take DR-JL as the baseline. Thus, all results are 1 for DR-JL. The results show that the time complexity of our method is close to the doubly robust joint learning method, and much less than the previous SOTA DR-based methods, demonstrating the efficiency of our method.

W4. Limited Domain Scope: Experiments focus on recommender-system datasets; applicability to other MNAR settings (e.g., causal inference, medical missingness) remains to be demonstrated.

Response to W4: We thank the reviewer for the question. As suggested by the reviewer, we further explore the effectiveness of our method on two scenarios, the medical field and business field, with two datasets, IHDP and Jobs.

• The semi-synthetic IHDP dataset [7] is constructed from the Infant Health and Development Program (IHDP) with 747 individuals and 25 covariates.
• The real-world Jobs dataset [8] is based on the National Supported Work program with 3,212 individuals and 17 covariates.

In this scenario, the DR estimator has the following form:

$\tau(x)=E[\frac{T(Y-m_1(X))}{\pi(X)}-\frac{(1-T)(Y-m_0(X))}{1-\pi(X)}+m_1(X)-m_0(X) | X=x],$

where $T \in$ {0,1} is the binary treatment indicator, $Y$ is the outcome, $\pi(X) = P(T=1|X)$ is the propensity model, and $m_0(X), m_1(X)$ are the imputation models for $T = 0$ and $T = 1$, respectively.

Following the previous study [9], we split the data into training/validation/testing sets with ratios 63/27/10 and 56/24/20, with 10 repeated times on the IHDP and the Jobs datasets, respectively. Note that each sample is already disjoint, and the training distribution is the same as the test distribution due to we randomly split the data. For evaluation, we use $\sqrt{\epsilon_{\text{PEHE}}}$ and $\epsilon_{\text{ATE}}$ for IHDP, and $R_{\text{pol}}$ and $\epsilon_{\text{ATT}}$ for Jobs. The definitions are shown below:

• $\epsilon_{\text{PEHE}} = {\frac{1}{N} \sum_{i=1}^N [(\hat {Y}_i(1) - \hat {Y}_i(0)) - (Y_i(1) - Y_i(0))]^2}$, where $\hat {Y}_i(1)$ and $\hat {Y}_i(0)$ are the predicted values for the corresponding true potential outcomes of unit $i$.
• $\epsilon_{\text{ATE}}=\frac{1}{N}| \sum_{i=1}^N ((\hat {Y}_i(1) - \hat {Y}_i(0)) - (Y_i(1) - Y_i(0)))|.$
• ${R}_{\text{Pol}}= 1-(\mathbb{E}[Y(1) \mid \hat {Y}(1) - \hat {Y}(0) >0, X=1] \cdot \mathbb{P}(\hat {Y}(1) - \hat {Y}(0)>0)+\mathbb{E}[Y(0) \mid \hat {Y}(1) - \hat {Y}(0) \leq 0, X=0] \cdot \mathbb{P}(\hat {Y}(1) - \hat {Y}(0) \leq 0))$, where $T, C, E$ are the indexes of treatment, control, and randomized sample set, respectively.
• $\epsilon_{\text{ATT}} = |\text{ATT} - \frac{1}{|T|} \sum_{i \in T} (\hat {Y}_i(1) - \hat {Y}_i(0)|$

where $\text{ATT} = |\frac{1}{|T|} \sum_{i \in T}{Y_i} - \frac{1}{|C \cap E|} \sum_{i \in (C \cap E)}{Y_i}|$.

The lower the metrics, the better the performance. See [9] for a more detailed definition of evaluation metrics. The performance on the test set is shown below:

Results on IHDP:

Results on Jobs:

The results show that our method can be applied to those datasets, and the performance is better than the DR-JL method.

We believe the above discussion has thoroughly addressed your concerns regarding our work. We would be grateful if you could kindly consider raising your rating. Thank you!

References

[1] Doubly Robust Joint Learning for Recommendation on Data Missing Not at Random. ICML 2019.

[2] Enhanced Doubly Robust Learning for Debiasing Post-Click Conversion Rate Estimation. SIGIR 2021.

[3] Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random. WWW 2024.

[4] ESCM$^2$: Entire Space Counterfactual Multi-Task Model for Post-Click Conversion Rate Estimation. SIGIR 2022.

[5] StableDR: Stabilized Doubly Robust Learning for Recommendation on Data Missing Not at Random. ICLR 2023.

[6] Propensity Matters: Measuring and Enhancing Balancing for Recommendation. ICML 2023.

[7] Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics 2011.

[8] Evaluating the Econometric Evaluations of Training Programs with Experimental Data. The American Economic Review 1986.

[9] Estimating Individual Treatment Effect: Generalization Bounds and Algorithms. ICML 2017.

Dear Reviewer gNuC,

Thanks again for your insightful comments and interesting questions. During the rebuttal period, we have validated time efficiency, insensitivity to the propensity model, and applicability to other domains (such as medical and business) of our method. In addition, we highlight that the previous method requires “exact match user and item” between the training set and test set, while our method only assumes $P_{\rm train}(x,r)=P_{\rm test}(x,r)$. To further validate our method's applicability in real-world scenarios (may have covariate shift or nonstationarity), we show our method is still valid and outperforms baselines if the assumption $P_{\rm train}(x,r)=P_{\rm test}(x,r)$ is violated.

We add experiments on the industrial dataset KuaiRec. To process the data, we first randomly split 80% users and items to construct the training set to make sure we do not see all users and items during training. Then we consider the scenario with $P_{\rm train}(x,r) \neq P_{\rm test}(x,r)$:

• We assign a propensity according to the item frequency and the user's active level (measured by the number of rated items) to split the dataset. This will lead to the first training set containing popular users and items, and the second containing non-popular users and items. We set the threshold to make the number of user-item pairs the same in the two datasets.

For all methods, we test on the MAR dataset. The results are shown below:

The results show that our method still outperforms the baseline DR-JL, especially under the second set (when the covariate shift is large). One possible reason is that our method deals with a super-population scenario, which is more robust toward a huge distribution shift.

A more practical scenario is that data may arrive sequentially with $P_{\rm train}(x,r) \neq P_{\rm test}(x,r)$, and we need to retrain our model efficiently on new data. To further show the applicability of our method, we evaluate whether our method can be efficiently applied in this scenario. We propose two ways to update our predictions efficiently:

• We store the parameters in models and optimizers at the end of training, and fine-tune them when new data comes (named "continue training").
• We freeze the parameters in prediction, imputation, and propensity model, and add a learnable shallow residual structure, which is zero-initialized when new data comes (named "adding residual").
• Also, we consider the full retraining of all parameters on both old and new data as the oracle baselines.

We also add experiments on the industrial dataset KuaiRec and first randomly split 80% users and items. Then we consider the following scenario:

• We randomly split the user set and item set into two parts with a ratio of 50/50, and train our method on the first dataset. Then we assign a propensity according to the item frequency and the user's active level, sampling popular users and items into the first time retraining, and the remaining into the second time retraining. Also, we adjust the threshold to make the number of user-item pairs the same in each retraining.

The results on the MAR test data are shown below:

• We find that our methods can efficiently adapt our predictions as new data becomes available.
• In addition, when there is a large gap between the new data and the original training data (such as non-popular users and items, as shown in the "Train Set3”), though the performance of our method will drop, our method significantly outperforms DR-JL.

By now, we hope we have fully addressed your concerns! We sincerely expect that the added extensive experiments and discussions on time efficiency, insensitivity to the propensity model, and applicability to other domains (such as medical and business) as well as the assumption $P_{\rm train}(x,r)=P_{\rm test}(x,r)$ is violated, will enable you to raise your rating!

Response: Thank you for your time and dedication in reviewing our manuscript! We are really glad that you are enjoying our paper and give our paper very positive comments! We believe that our study makes an important contribution to the field of debiasing in RecSys and causal inference, and we hope that you will support our research in subsequent discussions.

We thank the reviewer for efforts in evaluating our work and helpful suggestions. Below, we will try our best to address your concerns and question.

W1: Further clarification or empirical validation of the bound would strengthen the contribution even more.

Response to W1: We thank the reviewer for the useful suggestions.

• Further clarification of error bounds: Given the propensity model $\hat p_{u, i}$ and imputation model $\hat e_{u, i}$, satisfying $\hat p_{u, i}\geq K_{\psi}$ and $|e_{u, i}-\hat e_{u, i}|\leq K_{\phi}$, then with probability at least $1-\eta$, we have

$L_{\text{ideal}}(\theta) \leq \underbrace{{\frac{1}{|\mathcal{D}|} \sum_{(u, i) \in \mathcal{D}}\big[\hat e_{u, i}+\frac{o_{u, i}(e_{u, i}-\hat e_{u, i})}{\hat{p}_{u, i}}\big]}} _{\text{empirical DR loss}}$

$+ ~\underbrace{\frac{1}{|\mathcal{D}|} \sum_{(u, i) \in \mathcal{D}}\left|1-\mathbb{E}\left[\frac{o_{u, i}}{\hat p_{u, i}} | x_{u, i}\right]\right| \cdot\left|\mathbb{E}\left[e_{u, i} \mid x_{u, i}\right]-\mathbb{E}\left[\hat e_{u, i} \mid x_{u, i}\right]\right|} _{\text{empirical risks of imputation and propensity models}}$

$+ ~\underbrace{\left|\frac{1}{|\mathcal{D}|} \sum_{(u, i) \in \mathcal{D}} \operatorname{Cov}\left(\frac{o_{u, i}-\hat p_{u, i}}{\hat p_{u, i}}, e_{u, i}-\hat e_{u, i}\right)\right|} _{\text{the empirical covariance}}$

$+ \underbrace{\left(1+\frac{1}{K_\psi}\right)\left(2 \mathcal{R}(\mathcal{F})+K_\phi \sqrt{\frac{18 \log (4 / \eta)}{|\mathcal{D}|}}\right)} _{\text{tail bound controlled by empirical Rademacher complexity and sample size}},$

which includes four terms: the empirical DR loss, the empirical risks of imputation and propensity models, the empirical covariance, and the tail bound. Even if the $\hat p_{u,i} = p_{u,i}$ or $\hat e_{u,i} = e_{u,i}$, the empirical covariance does not vanish in the super-population scenario. The previous DR-based method can only control the first two terms, while our method can further control the empirical covariance by subtly modifying the imputation model (introducing one learnable parameter), without adding new modules and increasing time complexity.

• Further empirical validation of error bounds:
  • In our manuscript, we have demonstrated that our method can control empirical covariance on the observed samples on three widely used real-world datasets.
  • Furthermore, we validate empirical covariance control on the whole dataset by conducting a semi-synthetic experiment on the Movielens 100k dataset. Following previous studies [1-4], we use matrix factorization to generate the full binary rating matrix as the ground truth, the propensities for generating MNAR observed data, as well as the embedding for each user and item. Then we randomly select 50% / 70% users and items into our training data and train DR-JL, DCE-DR, and our method on the training data, finally calculating the empirical covariance (EC) on the whole dataset.

The results are shown below:

The results show that our method can control EC the most, which verifies the validity of the Controllability of Empirical Covariance (Thm 4.7), as well as we may a tighter error bounds.

Q1: Can this method be applied in a real-time setting where data arrives sequentially? In other words, is the proposed approach capable of efficiently adapting or updating its predictions as new data becomes available, without requiring full retraining?

Response to Q1: We thank the reviewer for the insightful question. This is a very common scenario in practice. For example, in the CTCVR scenario, the new MNAR data comes daily. To evaluate whether our method can be efficiently applied in this scenario, we propose two ways to efficiently update our predictions:

• We store the parameters in models and optimizers at the end of training, and fine-tune them when new data comes (named "continue training").
• We freeze the parameters in prediction, imputation, and propensity model, and add a learnable shallow residual structure, which is zero-initialized when new data comes (named "adding residual").
• Also, we consider the full retraining of all parameters on both old and new data as the oracle baselines.

We add experiments on the industrial dataset KuaiRec. To process the data, we first randomly split 80% users and items to construct the training set to make sure we do not see all users and items during training. Then we consider the following two scenarios:

• The new data has the same distribution compared to the training data (Consistent in distribution). We randomly split the user set and item set into three parts with a ratio of 50/25/25. First, we train our method on the data with 50% users and 50% items, and then retrain the model on the remaining data sequentially, i.e., twice in total.

• The new data has a different distribution compared to the training data (Inconsistent in distribution). We randomly split the user set and item set into two parts with a ratio of 50/50, and train our method on the first dataset. Then we assign a propensity according to the item frequency and the user's active level to sample the first half of users and items into the first time retraining, and the remaining into the second time retraining. We control the sampling process to ensure the sample number is the same in the two retrainings.

For all methods, we test on the MAR dataset. The results on the KuaiRec dataset are shown below. We bold the better results between ours and DR-JL.

We find that our methods can efficiently adapt our predictions as new data becomes available. An interesting finding is that when there is a large gap between the new data and the original training data (such as cold users and items, as shown in the "Train Set3" in the "Inconsistent in distribution" scenario), though the performance of our method will drop, our method significantly outperforms DR-JL. One possible reason is that our method deals with a super-population scenario, which is more robust toward a huge distribution shift.

We sincerely thank you for your feedback, and welcome any further technical advice or questions on this work. We will do our best to address your concerns.

References

[1] Recommendations as Treatments: Debiasing Learning and Evaluation. ICML 2016.

[2] Doubly Robust Joint Learning for Recommendation on Data Missing Not at Random. ICML 2019.

[3] Enhanced Doubly Robust Learning for Debiasing Post-Click Conversion Rate Estimation. SIGIR 2021.

[4] Be Aware of the Neighborhood Effect: Modeling Selection Bias under Interference for Recommendation. ICLR 2024.