Catch the Shadow: Automatic Shadow Variables Generation for Treatment Effect Estimation under Collider Bias

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[Q1] How is the CATE estimations evaluated on Twins dataset? The Twins dataset is usually used only for evaluating ATE estimations as there is no ground truth counterfactual outcomes.

Response: The reviewer raises an interesting concern. Following previous studies [1, 2], we use the Twins dataset to evaluate CATE estimation performance. Nonetheless, we agree with the reviewer that “the Twins dataset is usually used only for evaluating ATE estimations as there is no ground truth counterfactual outcomes”.

As suggested by the reviewer, we conducted experiments on two additional datasets with more comprehensive DGPs, namely Jobs and ACIC 2016, and removed CATE evaluation results on Twins dataset in Table 2. Since the Jobs dataset contains no counterfactual outcomes, we follow [3, 4] to use the policy risk for evaluating the CATE estimation performance, which measures the expected loss if the treatment is taken according to the CATE estimation. The results are shown as below.

From the above results, we observe that: (1) The performance of all the estimators is not as good as that on other datasets like the IHDP dataset because the data generation process of the ACIC 2016 datasets is more complex and comprehensive. (2) DR-CFR, DESCN, and ES-CFR show the most competitive performance among all baseline methods. (3) The proposed method stably outperforms all baselines addressing confounding bias or sample selection bias, since the previous methods are limited to the cases in which $X$ and $T$ cause $S$, and cannot handle the cases with addition $Y$ cause $S$. Instead, our ShadowCatcher and ShadowEstimator can address collider bias for accurate CATE estimation. (4) The proposed method on the Jobs dataset shows the lowest policy risk, which demonstrates the effectiveness of our methods in real-world applications.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[W1 & Q1] If the individual is not sampled, it’s not only Y that is not observed but also X (and T), in which case the shadow variable approach would not be applicable. How does this affect the usefulness in practice. Are there any representative real-world examples about situations in which they expect the method to be useful and where not?

Response: Thank you for the comment. We agree with the reviewer that there are also cases "if the individual is not sampled, it's not only $Y$ that is not observed but also $X$ (and $T$), in which case the shadow variable approach would not be applicable". Instead, the proposed method (and most methods on sample selection bias) is applicable when the target population is determined, which also covers a wide range of real-world scenarios.

For example, when studying the effect of vaccines ($T$) on disease prevention ($Y$), the target population is determined as, for example, the population of a country or region. Given the target population, governments or institutions would collect all information in terms of their covariates ($X$), and vaccine status ($T$). However, only a portion of the population will remain in follow-up ($S=1$), making the outcome ($Y$) able to be collected, while others will be lost ($S=0$), due to emigration, death, etc.

Another example is semi-supervised learning, in which one of the covariates is regarded as the treatment ($T$), and the rest of the covariates are denoted as $X$. Then the labeled samples are marked as $S=1$ with fully observed $X$, $T$, and $Y$, whereas the unlabeled samples are marked as $S=0$ with only $X$ and $T$ observed.

[W2] Overall, the method appears a bit hacky (a combination of multiple steps), but all the steps are well-motivated.

Response: Thank you for the comment. We agree that the proposed method is a bit hacky but well-motivated. To make the entire framework of ShadowCatcher and ShadowEstimator more connected and clearer, we briefly summarize the overall process of using ShadowCatcher and ShadowEstimator:

• ShadowCatcher first takes the fully observed $X$ and $T$, and the observed $Y$ of $S=1$ units as inputs to generate shadow-variable representations $Z$, and then tests whether the generated $Z$ satisfies Assumption 1.
• If the generated $Z$ does not pass the hypothesis test, ShadowCatcher should re-generate new shadow-variable representations until the generated $Z$ finally passes the test.
• Finally, ShadowEstimator uses the generated $Z$ to estimate treatment effects with observational samples.

We also add an overall flowchart as shown in Figure 4 on page 14.

[W3] When printed out, fonts in Figures 2 and 3 are barely readable.

Response: We thank the reviewer for pointing out this issue. We have adjusted them in our revised version on page 5 and Figure 3 on page 7 to make them clearer.

[Q2] There is a typo in the equation for the loss function in the “Hypothesis Test Phase” paragraph.

Response: We thank the reviewer for pointing out typos here. The correct equation should be $L_q = 1/n * \sum_{i=1}^n||(s_i / q(x_i, t_i, y_i) - 1) \cdot (x_i, z_i, t_i)||_2^2$, where $(x_i, z_i, t_i)$ is a vector and $||.||_2$ is the L2-norm. $L_q$ aims to learn a solution q of Q by Theorem 2 in Equation (6). Specifically, if $E[S / Q(X, T, Y) - 1 | X, Z, T] = 0$ (by Theorem 2 in Equation (6)), then $E[E[S / Q(X, T, Y) - 1 | X, Z, T] * (X, Z, T) ]=0$. The left hand side equals to $E[(S / Q(X, T, Y) - 1) * (X, Z, T)] = 0$. Then $L_q$ is just to minimize the square of the L2-norm of that Equation. We have corrected that on page 6, and added a more detailed explanation of $L_q$ in Appendix A.3.3 on page 18.

We hope the above discussion will fully address your concerns about our work. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[W1] The tradeoff between the efficiency and performance of the proposed method should be considered carefully in real-world applications.

Response: Thank you for your constructive suggestion. We have conducted additional experiments to explore the impact of different choices of $\alpha$ on the efficiency and performance. The results are shown as below.

The above results show that the performance of ShadowCatcher improves as the reject threshold decreases, because the hypothesis test gets more strict, and the constraint gets more reliable. However, the number of iterations required for ShadowCatcher to pass the hypothesis test also increases rapidly, reducing the efficiency of ShadowCatcher. Therefore, choosing an appropriate $\alpha$ is a trade-off between efficiency and performance, which depends on the real-world application scenarios.

[W2] The figure size is somewhat small.

Response: We thank the reviewer for pointing out this issue. We have adjusted them in our revised version on page 5 and Figure 3 on page 7 to make them clearer.

We hope the above discussion will fully address your concerns about our work. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[W1.1] Why do the $f(Z|...)$ functions not eliminate the need to calculate $\widetilde{OR}$ according to Equation (4)?

Response: Thank you for the comment. The necessity for further calculating $\widetilde{OR}(X, T, Y)$ given $f(Z | X, T, S)$ comes from that: for identification of $f(Y | X, T, S=0)$, we need to obtain both $OR(X, T, Y)$ and $E[\widetilde{OR}(X, T, Y) | X, Z, T, S=1]$ by Equation (2). The latter can be easily calculated by Equation (4) using $f(Z | X, T, S)$. However, the former still needs $\widetilde{OR}(X, T, Y)$ and $\widetilde{OR}(X, T, Y=0)$ by Equation (5). Thus, it is still necessary to know what $\widetilde{OR}(X, T, Y)$ as a function of $X, T, Y$ is to calculate $\widetilde{OR}(X, T, Y=0)$, and we need to further solve for $\widetilde{OR}(X, T, Y)$ as a function of $X, T, Y$ from the calculated $E[\widetilde{OR}(X, T, Y) | X, Z, T, S=1]$ by Equation (4).

[W1.2] Why is Equation (5) true?

Response: By Equation (2), $OR(X, T, Y=0) = 1$ (just substitute $Y=0$ into Equation (2)). Therefore, by the definition of $\widetilde{OR}(X, T, Y)$ that $\widetilde {OR}(X, T, Y) = OR(X, T, Y) / E[OR(X, T, Y) | X, T, S=1]$, the right-hand side of Equation (5) equals to $OR(X, T, Y) / OR(X, T, Y=0)$. Since $OR(X, T, Y=0) = 1$, we have $OR(X, T, Y) / OR(X, T, Y=0)$ equals the left-hand side of Equation (5) (i.e., $OR(X, T, Y)$). To make this clearer, we make a detailed explanation of Equation (5) in Appendix A.3.2 on page 18.

[W2.1] Would it be reasonable or more sensible to take random $Z$ as the implementation of $Z^-$ instead?

Response: Thank you for such insightful advice. It is absolutely reasonable and we have incorporated this suggestion throughout our paper. In fact, the original description of $Z^-$ in our article that "the opposite value of $Z$" is not accurate since we misuse the word "opposite." We have corrected it in our revised version (on Page 6) that $Z^-$ indeed denotes a value that differs from $Z$ to constrain the conditional dependence assumption, which any strategy can implement. We choose $-Z$ as $Z^-$ for continuous $Z$s just because this is easy to implement. We do agree that taking a random Z instead is a better idea since when $Z$ is close to 0, $-Z$ does not satisfy the requirement of $Z^-$. We use this new implementation to re-run all experiments and achieve obvious performance improvements. The results are shown as below.

The original implementation of our method is denoted as Ours (Old), whereas the new implementation of our method is denoted as Ours (New). Note that we add an additional real-world dataset - Jobs to better evaluate the performance, where the evaluation metric is the policy risk $\hat{R}_{\mathrm{Pol}}=1-(\mathbb{E}[Y(1)\mid \tau(\mathbf{x}) > 0,T=1] \cdot \mathbb{P}(\tau(\mathbf{x}) > 0) + \mathbb{E}[Y(0)\mid \tau(\mathbf{x}) \leq 0,T=0]\cdot \mathbb{P}(\tau(\mathbf{x}) \leq 0)$ instead of $\sqrt{\mathrm{PEHE}}$ for other datasets. The new implementation (taking a random $Z$ instead of $-Z$ as $Z^{-}$) shows obvious performance improvements on all datasets. We really appreciate your great advice!

[W2.2] Why will $h_r$, a function that aims to predict Z well, also help to move the $Z_0$ and $Z_1$ distributions towards each other?

Response: The reviewer raises an interesting concern. Please kindly note that $h_r$ itself is not the key to moving the $Z_0$ and $Z_1$ distributions towards each other. Instead, as $h_r$ is the estimate of $f(Z | X, T, Y, S=1)$, if $h_r$ can also predict $Z$ well for $S=0$ data, then $f(Z | X, T, Y, S=1) = f(Z | X, T, Y, S=0)$ and the assumption that $Z$ is independent of $S$ given $X, T, Y$ is satisfied.

[W2.3] What is the notation $\cdot (x_i, z_i, t_i)$ in the loss function for $Q$? it's not clear if there's a typo here.

Response: We thank the reviewer for pointing out typos here. The correct equation should be $L_q = 1/n * \sum_{i=1}^n||(s_i / q(x_i, t_i, y_i) - 1) \cdot (x_i, z_i, t_i)||_2^2$, where $(x_i, z_i, t_i)$ is a vector and $||.||_2$ is the L2-norm. $L_q$ aims to learn a solution q of Q by Theorem 2 in Equation (6). Specifically, if $E[S / Q(X, T, Y) - 1 | X, Z, T] = 0$ (by Theorem 2 in Equation (6)), then $E[E[S / Q(X, T, Y) - 1 | X, Z, T] * (X, Z, T) ]=0$. The left hand side equals to $E[(S / Q(X, T, Y) - 1) * (X, Z, T)] = 0$. Then $L_q$ is just to minimize the square of the L2-norm of that Equation. We have corrected that on page 6, and added a more detailed explanation of $L_q$ in Appendix A.3.3 on page 18.

[W2.4] why is distilling $h_{z_0} / h_{z_1}$ into a $\widetilde{or}$ function is necessary?

Response: Thank you for the comment, but we believe this question comes from [W1.1] in which Proposition 1 and Equation (5) confuse you. Please kindly refer to our response to [W1.1], it is necessary because we need to further solve for $\widetilde{OR}(X, T, Y)$ as a function of $X, T, Y$ from the calculated $E[\widetilde{OR}(X, T, Y) | X, Z, T, S=1]$ by Equation (4), so that we can obtain $OR(X, T, Y)$ from $\widetilde{OR}(X, T, Y)$ by Equation (5) and further identify $f(Y | X, Z, T, S=0)$ by Equation (2).

[W3] I thought there was an assumption around unconfoundedness. Is this still a fair comparison to other methods if de-confounding methods are used?

Response: The reviewer raises an interesting concern. However, the de-confounding way is just an implementation of the outcome estimators in ShadowCatcher and ShadowEstimator, which will not affect (either positive or negative) the ability to address collider bias. As you may noticed, we do need the unconfoundedness assumption on the target population (but not on only $S=1$ data because this assumption can be violated due to collider bias). This means we assume that there are no unobserved confounders, but there can possibly be confounding bias caused by fully observed covariates. The de-confounding method (an IPM to balance treated and controlled representations following [1]) we use is just to avoid the impact of such confounding bias on CATE estimation, thus we suppose the evaluation is fair to see each method's ability to address collider bias.

[W4] In the synthetic data in Table 1, some of the wins were not that big.

Response: We agree with you and have adopted your kind suggestion in [W2.1]. We think it mainly results from [W2.1] that the implementation of ShadowCatcher could be improved. We have adopted your suggestion to use a random $Z$ instead of $-Z$ as the implementation of $Z^-$. We re-run all experiments and achieve obvious performance improvements. Please kindly find the detailed results in the response to [W2.1].

[W5] There's a lot of experimental info missing from section 4 around what learning algorithm and models are used.

Response: We thank the reviewer for pointing out this issue. In the following, we report the missing implementation details, including what learning algorithms and models are used, as well as the choice of hyperparameters on different datasets in our revised version.

Specifically, we adopt 3-layer neural networks to implement each module in ShadowEstimator and ShadowCatcher. We use the Adam optimizer with batch normalization in the training process, and we use the Wasserstein distance as the Integral Probability Metric (IPM) to implement all the methods that need IPM to balance representations. We implement all the methods in the PyTorch environment with Python 3.9. The CPU we use is 13th Gen Intel(R) Core(TM) i7-13700K, and the GPU we use is NVIDIA GeForce RTX 3080 with CUDA 12.1. For the ease of reproducibility, the hyper-parameters of our methods on different datasets are detailed in the table below.