Doubly robust identification of treatment effects from multiple environments

We thank the reviewer for their thoughtful feedback and we appreciate the acknowledgment of our work’s clarity and novelty.

Below, we respond to the specific points raised.

[W1 Failure of Assumption 4.1] We acknowledge that Assumption 4.1 is relatively strong. However, we emphasize that it is implied by well-established assumptions in the invariance literature, most notably the simultaneous noise intervention assumption introduced in [1] (Theorem 2, Assumption iii).

Moreover, the assumption can be falsified in certain settings. For example, if practitioners know that an adjustment set is invalid (e.g., it excludes a known confounder), they can test whether the conditional means of and shift across environments. If the conditional means are not shifting then the assumption is falsified.

Nevertheless, we sincerely appreciate the reviewer’s suggestion (also raised by Rev r3F3) to examine the effects of violating this assumption. As suggested, we add a sensitivity parameter to represent the strength of heterogeneity of data sources, and then twist that parameter from zero (Assumption 4.1 fails) to high values (Assumption 4.1 is strongly satisfied). We have added these additional experiments in Appendix C.6 (of the revised version).

The main takeaway from these ablations is that both our method and previous methods relying on invariance (e.g., [2) perform poorly when the assumption is violated. However, our method outperforms IRM [2] even in this adversarial setting and is competitive with other baselines when the assumption is only weakly satisfied (i.e., small heterogeneity across environments).

[W2 Failure of Assumption 3.3] We agree with the reviewer that examining the impact of fully violating Assumption 3.3 is valuable. These results were already included in our paper (see Appendix C.2), where we show that, in cases of full violation, both our method and IRM fail due to the lack of invariance to exploit. The resulting error of both methods is significantly high, further suggesting that our simulation setting is not overly simplified.

[W3 Trimester of birth] We appreciate the reviewer's comments and agree that using the trimester of birth as the environment variable may not be ideal (i.e. it may not satisfy our identification assumptions). However, we stress that it is challenging to find publicly available real data with multiple environments (e.g., data from many different hospitals). Geographic location or hospital indicator could serve as better environment variables that introduce more significant shifts. Unfortunately, our data lack such information.

Finally, we stress that there are no repeated samples in our experiment: each data point represents a unique individual that gave birth in a specific trimester of the year.

[C1 Post-treatment and unobserved variables] That's a great question. Our setting accommodates simultaneously unobserved mediators, unobserved and observed colliders, and unobserved variables that are not confounders (as identifiability of the treatment effect would otherwise be fundamentally impossible). For instance, we allow for unobserved parents of the outcome, a scenario where previous methods like IRM [2] would fail.

[C2 Independence of noise] We apologize for a slightly imprecise formulation. Indeed, the exogenous noise variables in Assumption 3.1 are assumed to be independent. However, the DAG in Assumption 3.1 is over both observed and unobserved variables. This DAG induces a corresponding "observed" DAG with noise variables that might be dependent. Thus, the setting is more general than [1] where the graph is assumed to be fully observed.

Accordingly, we have added a footnote in line 215 to avoid any ambiguities.

[C3 Contribution] Thank you for pointing this out. We implicitly assume in this sentence that a valid adjustment set is not known (which is often the case in practice)—if it were, identifiability would follow directly by definition. We believe this phrasing is correct and highlights the novelty of our method, which, to our knowledge, is the first to identify the treatment effect in the presence of both post-treatment and unobserved variables when a valid adjustment set is not known.

[1] Peters, Jonas, Peter Bühlmann, and Nicolai Meinshausen. "Causal inference by using invariant prediction: identification and confidence intervals." Journal of the Royal Statistical Society Series B: Statistical Methodology 78.5 (2016): 947-1012.

[2] Claudia Shi, Victor Veitch, and David Blei. Invariant representation learning for treatment effect estimation. Uncertainty in Artificial Intelligence, 2021

We are grateful to the reviewer for recognizing the strengths of our paper, including the importance of estimating treatment effects in the presence of post-treatment and unobserved variables, the introduction of a novel double robustness property, and our extensive experimental validation.

We now address the raised weaknesses and questions.

[Q1 Advantages compared to (1) and (2)] Our method differs fundamentally from [1] and [2] in its objectives. While these neural network approaches focus on estimation given a covariate set that satisfy ignorability, our method addresses the prerequisite challenge of identifying an adjustment set that satisfies ignorability. Importantly, our approach is compatible with their estimation techniques - once a valid adjustment set is identified, the neural networks from [1] and [2] can be used in the estimation stage.

[Q2 Sensitivity to sample size] We agree that analyzing sensitivity to sample size is important. In the revised version, we have added additional synthetic experiments (Appendix C.5, Figure 9) focusing on the small sample size regime ($n=250$) since the synthetic experiments in our original submission used relatively large sample sizes.

[W2 Assumptions] We want to emphasize that our method does not require ignorability with respect to the full set of covariates, as the covariate set might contain e.g. colliders. This allows our approach to be applied in settings where traditional methods might fail due to the presence of post-treatment variables (e.g. [1] and [2]). Further, we have explicitly stated the positivity assumption in Theorem 1 rather than in the problem setting to be transparent about the assumptions required to identify the ATE.

[W1 Examples] We appreciate the suggestion to include concrete examples. While we focused on an abstract causal graph in the introduction for clarity, post-treatment variable bias appears frequently in real-world settings. The most notable example in healthcare is the birth-weight paradox: Studies found that among low birth-weight infants, those born to smokers had lower mortality risk than those born to non-smokers -- seemingly contradicting the overall relationship between maternal smoking and infant mortality. This paradox arose from inappropriately controlling for birth-weight, a post-treatment variable affected by maternal smoking.

[W4 Evaluation metrics] The focus of our paper is on identifying the average treatment effect (ATE), which is the causal quantity of interest in most scientific inquiries. While extending the evaluation to the conditional average treatment effect (CATE) is an interesting future direction, we believe that our current scope is not limited. Identifying and estimating the ATE is already a challenging problem in settings where ignorability (with respect to the full set of covariates) is not satisfied.

We thank the reviewer for their thoughtful review and positive assessment of our paper. We appreciate their recognition of the clarity of our writing, the thorough discussion of related work, and the connections we drew between our assumptions and existing invariance assumptions in the literature.

Below, we address the raised questions.

[Q1 Sample complexity] Our focus in this paper was on identifiability, which is a challenging problem in itself. However, we strongly agree that sample complexity is a valuable direction for future work. We believe that the standard results from the double machine learning literature should apply to our estimator—if the nuisances (propensity score and outcome function) are estimated at the classic $o_{\mathbb P}(n^{-1/4})$ rate, our estimator should be asymptotically normal, allowing for valid (asymptotic) confidence intervals.

[Q2 Descendant in Figure 2] In Figure 2, "Descendant" refers to the underlying causal graph where the node $X_c$ is a descendant of the outcome $Y$ but not of the treatment $T$ (to ensure that it is not a collider).

[Q3 Line 264] In line 264, we raise a crucial point regarding the behavior of our method when distributions are not shifting across environments. Specifically, if $\mathbb P^e = \mathbb P^f$ for all $e,f \in \mathcal E$, any adjustment set would minimize our objective in Equation 4. This occurs because $E_{\mathbb P^e}[V | Z_S] = E_{\mathbb P^f}[V | Z_S]$ holds true for any environments $e,f \in \mathcal E$ and any subset of covariates $S$.

Essentially, when the distributions are the same across environments, we cannot use the invariance principle to differentiate between valid and invalid adjustment sets (because everything is invariant).

[Q3 Line 278] In line 278, we discuss the number of environments required to satisfy the heterogeneity conditions. We note that if environments are generated through single-node interventions (i.e. each environment is sampled from the same distribution with an intervention on a different node), we would need a number of environments in the order of the number of nodes in the causal graph.

[Q4 Completeness] That’s a great question. We considered this and briefly discussed it in Appendix A.1. Unfortunately, our assumption is not minimal: in some cases, it might still be possible to find a valid adjustment set using the observed parents of either $T$ or $Y$ (or both), even if the full set of parents is not observed (see the causal graph in Figure 5, for example). However, this set cannot be recovered using invariance approaches, as neither $T$ nor $Y$ are invariant across environments when some of their parents are unobserved and their distribution shifts.

We thank the reviewer for their thoughtful feedback and recognition of our contributions. We appreciate the acknowledgment of our work’s clarity, scalability, and the importance of addressing bad controls in causal inference.

Below, we respond to the specific points raised.

[W1 Assumption 4.1] We acknowledge that Assumption 4.1 is relatively strong. However, we emphasize that it is implied by well-established assumptions in the invariance literature, most notably the simultaneous noise intervention assumption introduced in [1] (Theorem 2, Assumption iii).

Moreover, the assumption can be falsified in certain settings. For example, if practitioners know that an adjustment set $Z$ is invalid (e.g., it excludes a known confounder), they can test whether the conditional means of $Y|Z$ and $T|Z$ shift across environments. If the conditional means are not shifting then the assumption is falsified.

Nevertheless, we sincerely appreciate the reviewer’s suggestion (also raised by Rev 6ghW) to examine the effects of violating this assumption. Accordingly, we have added experiments in Appendix C.6 (of the revised version) showing the impact of both weak and strong violations of this assumption on our results.

The main takeaway from these ablations is that both our method and previous methods relying on invariance (e.g., [3]) perform poorly when the assumption is violated. However, our method outperforms IRM [3] even in this adversarial setting and is competitive with other baselines when the assumption is only weakly satisfied (i.e., small heterogeneity across environments).

[W2 Selected subsets] Due to the stochastic nature of optimization in our method, different subsets are selected for different initialization seeds.

The most frequently selected features (>70% of the seeds)—maternal age, alcohol consumption, maternal foreign status, and number of prenatal care visits—align with the adjustment sets suggested in epidemiology literature.

Less frequently selected features (<30% of the seeds) include marital status, firstborn status, and previous child mortality indicator (i.e. whether the mother had a child who died at birth). While a deeper epidemiological analysis would be needed, we suspect that the indicator for previous child mortality (which our method appropriately discards) is likely to be influenced by the treatment (i.e. smoking).

We refrain from making strong claims about this real-world example, as this is beyond our expertise. Our primary objective was to illustrate the practical use of our method.

Additionally, we emphasize that the main challenge in excluding bad controls based on domain knowledge is that, in this example, even experts cannot confidently determine whether certain factors were measured before or after smoking initiation.

[Q1 Descendant of $Y$] Great question. A concise explanation is that $Y$ acts as a collider, and conditioning on a descendant of a collider introduces bias (as long as $T$ has a causal effect on $Y$). For a more formal and complete explanation of why descendants of $Y$ introduce bias, we refer to the discussion in [2] (see Model 18).

[Q2 Standard errors in application] Because our method and IRM rely on an optimization procedure to identify a valid adjustment set, they select different subsets depending on the random initialization, leading to high standard deviations across the 100 seeds. Table 1 shows that IRM has a higher standard deviation than our method. In contrast, the ALL and NULL baselines have low standard deviations, as they don’t rely on an optimization procedure to find the adjustment set (which is pre-specified).

[Q3 Generality of our approach] Our method readily extends to other causal estimands, such as CATE and ATT. Extending our approach to continuous treatments may be more challenging, but we believe it is feasible.

[1] Peters, Jonas, Peter Bühlmann, and Nicolai Meinshausen. "Causal inference by using invariant prediction: identification and confidence intervals." Journal of the Royal Statistical Society Series B: Statistical Methodology 78.5 (2016): 947-1012.

[2] Cinelli, Carlos, Andrew Forney, and Judea Pearl. "A crash course in good and bad controls." Sociological Methods & Research 53.3 (2024): 1071-1104

[3] Claudia Shi, Victor Veitch, and David Blei. Invariant representation learning for treatment effect estimation. Uncertainty in Artificial Intelligence, 2021.

Thank you very much for your feedback.

1. We greatly appreciate the time you took to point out the inconsistencies in the notation of the appendix. We have uploaded a revised version where the notation is adjusted accordingly.

2. Regarding the environment heterogeneity in Appendix C.6:

   • For each environment, we sample $U \sim \mathcal{N}(0, \sigma^2 I_d)$ only once.

   • Then, we sample $X_i \sim \mathcal{N}(U_i, 0.5 + U_i)$ for $i=1,\ldots, d_x$.

   • If $\sigma^2 = 0$, $U = 0$ becomes degenerate and $X \sim \mathcal{N}(0, 0.5)$ has the same distribution across environments. Therefore, there is no heterogeneity across environments and Assumption 4.1 is violated.

   • If $\sigma^2 > 0$, the mean and variance of $X$ will shift across environments, with larger shifts as $\sigma^2$ increases. Therefore, the heterogeneity across environments increases as we increase the parameter $\sigma^2$, and Assumption 4.1 will be more likely to be satisfied.

We hope this clarifies the issue and remain open to further questions or clarifications. We have described the data generating process more carefully in the revised version.