Generalizing Causal Effects from Randomized Controlled Trials to Target Populations across Diverse Environments

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point.

[Q1]

Regarding the testability of Assumption 3.2, we would like to clarify the following:

1. Assumption 3.2(1), i.e., $Z\notindep X^m |\widetilde{X^c}, S=1$, is testable because all three variables are observable in the dataset with $S=1$, where $S=1$ indicates that $X^m$ is observable in this dataset.
2. Assumption 3.2(2), i.e., $Z\indep S |X^m, \widetilde{X^c}$, is testable for the following reason. Although $X^m$ is unobservable in the dataset with $S=0$, we can still conduct the test based on Theorem 4.1, which does not require the values of $X^m$ in the dataset with $S=0$ since the only expression involving $X^m$ in the equation to be solved, i.e., $\frac{S}{Q(X^m, \widetilde{X^c})}$, is always zero when $S=0$.

[Q2]

Thank you for pointing out the issue caused by our accidental omission of the factor $\frac{P(S=1|X^m,X^c)}{P(S=1|X^c)}$ in the corresponding lines. However, we would like to clarify that this does not affect our final conclusion (the result on lines 674-675). The corrected proof is as follows:

$E\left[\frac{P(S=0|X^m,X^c)}{P(S=1 |X^m,X^c)}\cdot\frac{P(S=1|X^m=0,X^c)}{P(S=0|X^m=0,X^c)}\middle|X^c,S=1\right]$ $=\sum_{x^m}\frac{P(S=0|X^m=x^m,X^c)}{P(S=1|X^m=x^m,X^c)} \cdot\frac{P(S=1|X^m=0,X^c)}{P(S=0|X^m=0,X^c)}\cdot P(X^m=x^m|X^c,S=1)$ $=\sum_{x^m}\frac{P(S=1|X^m=0,X^c)}{P(S=0|X^m=0,X^c)}\cdot\frac{P(S=0|X^m=x^m,X^c)}{P(S=1|X^m=x^m,X^c)}\cdot\frac{P(S=1|X^m=x^m,X^c)}{P(S=1|X^c)}\cdot P(X^m=x^m|X^c)$ $=\frac{P(S=1|X^m=0,X^c)}{P(S=0|X^m=0,X^c)\cdot P(S=1|X^c)}\cdot\sum_{x^m}P(S=0|X^m=x^m,X^c)\cdot P(X^m=x^m|X^c)\\ =\frac{P(S=1|X^m=0,X^c)}{P(S=0|X^m=0,X^c)\cdot P(S=1|X^c)}\cdot\sum_{x^m}P(S=0,X^m=x^m|X^c)$ $=\frac{P(S=1|X^m=0,X^c)\cdot P(S=0|X^c)}{P(S=0|X^m=0,X^c)\cdot P(S=1|X^c)}$

Thank you again for your insightful feedback. We will address this issue in the revised version of the manuscript.

[Q3]

The identification of $OR(X^m,X^c)$ is guaranteed by Theorem 1 in [1], as referenced in Appendix D. The core idea relies on the completeness condition being satisfied. The detailed proof can be found in the Appendix (Proof of Theorem 1 on Pages 14-15) of [1]. Due to the character limit in the rebuttal stage, we regret that we are unable to provide the full proof here. We kindly ask that the reviewer refer to the original paper for further details. We will include this proof in Appendix D of the revised manuscript. Thank you for the suggestion.

[Q4]

Thank you for pointing out a typo in our manuscript. Indeed, it should be $T\indep X^m|R=1$, where $R=1$ is equivalent to $S=0$ in Setting 2. We will correct this typo in the revised version of the manuscript.

[Q5]

As cited in the manuscript, Theorem 4.1 is derived from Theorem 2.3 in [2], and the detailed proof can be found in Appendix A.3 (Page 12) of [2]. Due to the character limit in the rebuttal stage, we regret that we are unable to provide the full proof here. We kindly ask that the reviewer refer to the original paper for further details. We will include this proof in the appendix of the revised manuscript. Thank you for your valuable suggestion.

[Q6]

We employed mean value imputation to reduce the complexity of imputation and improve its practicality and efficiency. Performing distributional-estimation-based imputation would be perfect, but it requires additional estimation of quantities such as variance, higher-order moments, etc., or the use of generative methods, all of which would increase complexity. Mean value imputation is a commonly used technique to reduce complexity [3], and our experimental results demonstrate that it performs well. Of course, Theorem 3.3 supports distributional-estimation-based imputation, and we will explore it in future work. Thank you for your insightful suggestion.

[Q7]

Due to the character limit in the rebuttal stage, we are unable to include the full proof here. We promise to provide the consistency proofs for both doubly robust imputation and doubly robust ATE estimation in the appendix of the revised version. Thank you for your insightful suggestion.

---

We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score. Thank you!

References

[1] Miao, W., Liu, L., Li, Y., Tchetgen Tchetgen, E. J., & Geng, Z. (2024). Identification and semiparametric efficiency theory of nonignorable missing data with a shadow variable. ACM/JMS Journal of Data Science, 1(2), 1-23.

[2] d’Haultfoeuille, X. (2010). A new instrumental method for dealing with endogenous selection. Journal of Econometrics, 154(1), 1-15.

[3] Lin, W. C., & Tsai, C. F. (2020). Missing value imputation: a review and analysis of the literature (2006–2017). Artificial Intelligence Review, 53, 1487-1509.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point.

[Claims And Evidence 1]

First, we would like to clarify that the method we propose for automatically selecting shadow variables is theoretically guaranteed, based on the testability of Assumption 3.2:

1. Assumption 3.2(1), i.e., $Z\notindep X^m |\widetilde{X^c}, S=1$, is testable because all three variables are observable in the dataset with $S=1$, where $S=1$ indicates that $X^m$ is observable in this dataset.
2. Assumption 3.2(2), i.e., $Z\indep S |X^m, \widetilde{X^c}$, is testable for the following reason. Although $X^m$ is unobservable in the dataset with $S=0$, we can still conduct the test based on Theorem 4.1, which does not require the values of $X^m$ in the dataset with $S=0$ since the only expression involving $X^m$ in the equation to be solved, i.e., $\frac{S}{Q(X^m, \widetilde{X^c})}$, is always zero when $S=0$.

The shadow variables selected by the proposed method should pass the above hypothesis tests, ensuring that they satisfy Assumption 3.2.

Second, the existence of shadow variables can also be assessed through hypothesis testing. Moreover, as discussed in the manuscript, Assumption 3.2 is reasonable because not all variables in $X^c$ with predictive ability for $X^m$ are direct causes of $S$ in many real-world scenarios, such as the example of the AIDS study mentioned in the manuscript. Of course, in certain cases, shadow variables may indeed be weak or even absent. We have followed the reviewer's suggestion and conducted experiments in such cases.

[Claims And Evidence 2]

Our method, as most DR methods do, requires that at least one of the two models in both stages, either the regression model or the selection score model, be correctly specified to ensure the consistency of the imputation and ATE estimation. However, such requirement for correct model specification is very common in related work [1, 2], and we can leverage techniques such as neural networks to learn the correct model specification as accurately as possible. Of course, we value the reviewer's suggestion and have also conducted experiments to assess the impact of incorrect model specification due to violations of Assumption 3.2.

[Claims And Evidence 3]

Thank you for your insightful suggestion. Our method does not rely on specific CIT methods; any CIT method can be used for our hypothesis testing of Assumption 3.2(1). The choice of the method should be determined based on the specific application context. Due to the character limit in the rebuttal phase, we regret that we cannot provide a detailed discussion of the various CIT methods here. We promise to include the relevant discussion in the appendix of the revised manuscript.

[Methods And Evaluation Criteria]

Thank you for the suggestion. We have conducted experiments in cases where Assumption 3.2 is violated. Specifically, we conducted experiments on the synthetic dataset by

1. changing the coefficient of Z on S from 0 to {0.1, 0.3} to introduce weak dependence;
2. reducing the correlation coefficient between Z and $X^m$ from 0.8 to {0.3, 0.1}, so that Z only has a weak predictive ability for $X^m$.

The results demonstrates that while the performance of 2SDR does decline when Assumption 3.2 is violated, the decline is steady as the extent to which Assumption 3.2 is violated increases.

[Other Comments Or Suggestions]

Thank you for your valuable suggestions.

1. "Covariates affecting treatment effect heterogeneity" are not equivalent to confounders: Suppose there are three covariates, A, B, and C, and the generation process of Y is $Y=A\cdot T+B+C$. In this case, A is the only variable affecting treatment effect heterogeneity, as the value of $Y(1)-Y(0)$ changes with variations in A. We will improve the related definitions in the figure and its caption.
2. "Variables affect environmental shifts" refers to the variables in the causal graph that point to R (the causes of R). We will revise this statement to make it clearer.

---

We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score. Thank you!

References

[1] Miao, W., Liu, L., Li, Y., Tchetgen Tchetgen, E. J., & Geng, Z. (2024). Identification and semiparametric efficiency theory of nonignorable missing data with a shadow variable. ACM/JMS Journal of Data Science, 1(2), 1-23.

[2] Colnet, B., Mayer, I., Chen, G., Dieng, A., Li, R., Varoquaux, G., ... & Yang, S. (2024). Causal inference methods for combining randomized trials and observational studies: a review. Statistical science, 39(1), 165-191.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point.

[Other Comments Or Suggestions]

Thank you for your valuable suggestions. We will carefully revise our manuscript based on your recommendations.

1. We will provide more detailed descriptions of the practical examples for Settings 1 and 2.
2. We will improve the related definitions in the figure and its caption to make them clearer.
3. We will add a brief summary at the beginning of each proof step to explain what is happening in that step, and we will also provide a diagram to further illustrate it.

[Q1] How realistic is it to assume the existence of a shadow variable Z that satisfies Assumption 3.2?

Assumption 3.2 requires that among the common covariates shared by the two datasets, there exist variables that are correlated with the covariates missing in one of the datasets but do not directly influence environmental shifts (not a direct cause of R). Assumption 3.2 is reasonable in many real-world scenarios, as, typically, most covariates only indirectly affect R, with relatively few variables being direct causes of R. Moreover, Assumption 3.2 is testable:

1. Assumption 3.2(1), i.e., $Z\notindep X^m |\widetilde{X^c}, S=1$, is testable because all three variables are observable in the dataset with $S=1$, where $S=1$ indicates that $X^m$ is observable in this dataset.
2. Assumption 3.2(2), i.e., $Z\indep S |X^m, \widetilde{X^c}$, is testable for the following reason. Although $X^m$ is unobservable in the dataset with $S=0$, we can still conduct the test based on Theorem 4.1, which does not require the values of $X^m$ in the dataset with $S=0$ since the only expression involving $X^m$ in the equation to be solved, i.e., $\frac{S}{Q(X^m, \widetilde{X^c})}$, is always zero when $S=0$.

Therefore, we can assess whether Assumption 3.2 holds in real-world applications by conducting hypothesis tests on the covariates.

[Q2] Will results deteriorate smoothly w.r.t. the extent to which this Assumption 3.2 is satisfied?

Thank you for your insightful suggestion. We have conducted experiments in cases where Assumption 3.2 is violated. Specifically, we conducted experiments on the synthetic dataset under the following cases:

1. Case 1: Assumption 3.2(2) is violated. We changed the coefficient of Z on S from 0 to {0.1(Low), 0.3(High)} to introduce weak dependence between Z and S;
2. Case 2: Assumption 3.2(1) is violated. We reduced the correlation coefficient between Z and $X^m$ from 0.8 to {0.3(Low), 0.1(High)}, so that Z only has a weak predictive ability for $X^m$.

The results demonstrates that while the performance of 2SDR does deteriorate when Assumption 3.2 is violated, it deteriorates smoothly as the extent to which Assumption 3.2 is violated increases. We will include the results and analysis of this experiment in the revised manuscript.

[Q3] Typically, under environmental shifts, there may also be shifts in Y|X shifts. It would be beneficial if the authors discussed situations where we expect only X-shifts and where both X- and Y-shifts might occur, thereby clarifying where 2SDR can be effectively applied?

Thank you for your valuable suggestion. 2SDR relies on Assumption 3.1, which requires that $Y(t)\indep R|X^m,X^c$ holds. Therefore, the applicable situations for 2SDR can be summarized as follows:

1. X-shifts. As stated in Definition 2.1, the original definition of environmental shift in our work is X-shifts, i.e., $P(X|R=1)\neq P(X|R=0)$.
2. Both X- and Y-shifts caused by X. Although we did not explicitly state this in Definition 2.1, due to the fact that variables in X may be causes of Y, the distribution of Y will also shift along with the distribution of X, i.e., $P(Y|R=1)\neq P(Y|R=0)$, or alternatively, $P(X,Y|R=1)\neq P(X,Y|R=0)$.
3. $Y|X^c$ shifts. Given that $Y(t)\indep R|X$ holds, we have $P(Y|X,R=1)=P(Y|X,R=0)$. Therefore, our problem setting essentially assumes the absence of Y|X shifts. However, for the common covariates $X^c$ shared by the two datasets, there still exist $Y|X^c$ shifts, i.e., $P(Y|X^c,R=1)\neq P(Y|X^c,R=0)$. Assumption 3.2 required by 2SDR is still satisfied under $Y|X^c$ shifts, and therefore, 2SDR can address $Y|X^c$ shifts. However, existing methods rely on Assumption 2.8, which requires that $Y(t)\indep R|X^c$ holds. Under $Y|X^c$ shifts, Assumption 2.8 does not hold, and thus, in contrast to 2SDR, previous methods cannot address $Y|X^c$ shifts.

We will include the above discussion in the revised manuscript.

---

We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score. Thank you!