Estimating Heterogeneous Treatment Effects by Combining Weak Instruments and Observational Data

Strengths

Thank you for your positive feedback. We appreciate your recognition of our work on leveraging observational and encouragement data with low compliance to predict CATEs, and for acknowledging the utility of our two-stage framework.

Re: CATE Identification

You are absolutely correct that in its current form, our work requires that monotonicity is added to Assumption 1. The issue stems from the fact that we noticed that monotonicity is not actually needed, but did not propagate the required changes to the rest of the paper. In order for the analysis to be fully sound without monotonicity, the following small and local changes need to be implemented in the current draft (and will be added to the final version):

• Add defiance indicator to Assumption 2:

Assumption 2 (Unconfounded Compliance). Define the compliance and defiance indicators $C$ and $D$ as $C:=\mathbb{I}[A^E(1)>A^E(0)]$ and $D:=\mathbb{I}[A^E(1)<A^E(0)]$, respectively. Then, $Y^E(1)-Y^E(0)\perp C \mid X^E$ and $Y^E(1)-Y^E(0)\perp D\mid X^E$.

• Revise definition of $\gamma(x)$ right after Eq. (3) $\gamma(x)=P(C=1 \mid X^E=x)-P(D=1\mid X^E=x)$

• Proof of Lemma 1 in Appendix B.1: Replace $\gamma(x)> 0$ with $\gamma(x)\neq 0$.

We will also add a derivation of Eq. 3 to the appendix. We believe that removing the monotonicity assumption strengthens our work without significantly altering the analysis.

Re: Related Work

There are several other works that tackle confounding in observational studies. The ones that target point estimation (rather than bounds) usually assume the existence of auxiliary data and/or structure (e.g. proxy variables, negative controls). We will extend our literature review discussion to include these works as well.

Re: Questions

• The paper does not require the additive noise assumption, although additive noise is one way in which we can ensure the validity of Assumption 1.
• Yes, the distributions of X and U are the same across the two datasets as the datasets are assumed to be from the same population/environment. We will make this explicit.
• Excellent observation! Under certain assumptions about the collection of the IV and observational data, the observational data can be treated as the $Z=0$ component of the IV study. This would allow us to set $Z=1$ in the experimental study, thereby potentially increasing the data available for analysis.

Strengths

Thank you for your thoughtful review. We appreciate your positive feedback on our method for estimating CATEs by integrating weak instrumental variables and observational data, and your acknowledgment of its novelty and effectiveness.

Re: Questions

• Assumption 2 states that the treatment effect is independent of compliance given the observed features, meaning there are no unobserved confounders affecting both compliance status and potential outcomes. This standard assumption (see [1, 2]) is crucial for identifying the CATE, rather than the CLATE (conditional local average treatment effect), from data with instrumental variables. We will include further explanations about this in the final version.
• An IV dataset is a dataset that includes an instrumental variable along with the treatment, features, and outcome. We refer to this as experimental data or an experimental dataset, as we assume the instrument is randomized (conditionally or marginally). Our focus is on an intent-to-treat/encouragement design (e.g., movie recommendation, exercise encouragement) that can be easily deployed.
• From Assumption 1, the instrument is relevant in some covariate strata, meaning it affects treatment uptake. The instrument propensity, defined as $\pi_Z(x) = P(Z^E = 1 \mid X^E = x)$, is independent of the treatment and can be estimated directly from the data.
• Since the experimental data refers to intent-to-treat data with an instrument, the actual observed treatment is not randomized and is influenced by each unit's compliance. In short, the instrument is randomized, but the treatment is not. As you correctly pointed out, experimental samples with fully randomized treatments are challenging to obtain due to ethical considerations, financial constraints, and the infeasibility of enforcing specific treatments. This challenge is precisely why we propose our method.

Based on these suggestions, we will include more clarifying discussions in our final version.

[1] Wang, L. and Tchetgen Tchetgen, E., 2018. Bounded, efficient and multiply robust estimation of average treatment effects using instrumental variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3), pp.531-550.

[2] Frauen, D. and Feuerriegel, S., 2022. Estimating individual treatment effects under unobserved confounding using binary instruments. arXiv preprint arXiv:2208.08544.

Strengths

Thank you for your encouraging feedback. We are pleased that you recognize the novelty of our approach for estimating conditional average treatment effects using IVs, particularly in scenarios with weak instruments. Your positive feedback on the clarity of our exposition, the intuition of our methods, and the robustness of our theoretical results is appreciated and encourages us to improve our work further.

Re: Comparison with Coussens & Spiess, J. (2021) and Abadie et al. (2024)

These works focus on improving the efficiency of the 2SLS estimator for the local average treatment effect (LATE) by incorporating weights based on heterogeneous compliance. Our work, however, aims to estimate the CATE $\tau(x)$, which coincides with the conditional LATE (CLATE) $\tau^E(x)$ under our assumptions. The compliance-weighting strategy in their works cannot be directly applied here and does not improve the efficiency of our CATE ratio estimator in Eq. (3). We build on the concept of compliance weighting to improve the estimator in Eq. (3) in scenarios of low or no compliance by weighting samples for bias correction using observational data, diverging significantly from these studies.

Re: 401(k) Dataset Analysis

We agree that the language around this analysis needs precision and will revise it in the final version. The motivation behind these experiments is to demonstrate that our method can interpolate accurately in regions of no compliance, which we artificially introduce. We chose the 401(k) dataset due to its high estimated compliance ($0.49-0.90$), allowing us to establish a reasonably accurate ground truth CATE using Eq. (3). The figures show projection onto one axis of heterogeneity (years of education), but the procedure involves learning the biased CATE using the full sample and bias-correcting across all covariates and their interactions (46 features total for $\phi(x)$). The statement that the learned CATE extension "tracks" the true CATE refers to the close alignment in this projection. To quantify our claims, we repeated the experiment over 100 data splits, calculating the means and standard deviations of the treatment effects by years of education for the two examples in the paper. These tables are included in the rebuttal PDF and will be in the final version.

Re: Theoretical Results

To our knowledge, this is the first work to estimate CATEs from IV data with weak or no compliance by leveraging an additional observational dataset. Thus, comparing our estimator's bounds with IV-only estimators is challenging/inappropriate. Our estimator reduces variance in Eq. (3) when compliance is low and provides valid estimates when compliance is zero, where Eq. (3) fails. This reduction in variance and handling of zero-compliance cases is evident in our simulations. You are correct that a good estimator for $\theta$ (and therefore $b(x)$) requires a reliable estimator for $\tau^O(x)$. We use the learned $\widehat{\tau}^O(x)$ with the IV dataset to determine $b(x)$, crucial for correcting the unobserved confounding bias in $O$. We will discuss these points in the paper.

Re: Nits

Eq. 4 follows from Lemma 1 by noting (as you also pointed out) $\tau(x)$ as $\tau^O(x)+b(x)$ from our assumptions. We will make this connection explicit. We emphasize that $E$ and $O$ are fundamentally different datasets. $E$ includes an instrument from an IV study or randomized experiment with non-compliance, while $O$ is an observational dataset that may have been collected passively. Depending on the collection method, $O$ could be considered part of an IV study with $Z=0$, but this is not necessarily the case.

Re: Questions

• Yes, we could consider other estimating equations that might reduce the variance of the estimator in Eq. (3) at the cost of estimating 4 additional nuisances (see [1] below). However, this approach will not address the variance caused by low compliance, our primary concern. It would only reduce variance from potentially poor compliance estimation.
• Our method is agnostic to the types of estimators used for $\tau^O(x)$. We can include additional evaluations in the experimental section using other estimators, such as the doubly robust estimator. It is important to note that we do not expect these alternative estimators to reduce confounding bias, as it is asymptotically irreducible.
• The distribution over covariates $X$ becomes the distribution over compliers’ covariates when weighted by compliance.
• Yes, several works assume a joint representation between causal inference learning tasks to enhance generalization across tasks such as counterfactual learning and bias correction in observational samples (e.g., see [2, 3, 4, 5] below). We discuss [2] and [3] in the related works section of our paper. Additionally, [4] and [5] are foundational works in joint representation learning for causal inference. While it is possible to generalize beyond linear representations and consider $b(x) = h \circ \phi(x)$, where $h$ is a hypothesis class (see [3] for an example), this is beyond the scope of our current work.

References:

[1] Frauen, D. and Feuerriegel, S., 2022. Estimating individual treatment effects under unobserved confounding using binary instruments.

[2] Kallus, N., Puli, A.M. and Shalit, U., 2018. Removing hidden confounding by experimental grounding. Advances in neural information processing systems, 31.

[3] Hatt, T., Berrevoets, J., Curth, A., Feuerriegel, S. and van der Schaar, M., 2022. Combining observational and randomized data for estimating heterogeneous treatment effects.

[4] Shalit, U., Johansson, F.D. and Sontag, D., 2017, July. Estimating individual treatment effect: generalization bounds and algorithms. In International conference on machine learning (pp. 3076-3085). PMLR.

[5] Shi, C., Blei, D. and Veitch, V., 2019. Adapting neural networks for the estimation of treatment effects. Advances in neural information processing systems, 32.

Strengths

Thank you for your insightful feedback. We appreciate your recognition of our novel approach to combining an observational dataset with an IV study with weak instruments. We are glad that our efforts to ensure the paper is both rigorous and accessible are apparent.

Re: More Experiments

• Baseline comparisons: Debiased CATE estimation methods are not ideal comparisons as they address finite sample bias rather than confounding bias. The confounding bias, typically found in observational studies, is an asymptotically irreducible bias:

$$b(x) = \mathbb{E}\left[(\mathbb{E}[Y^O\mid A^O=1, X^O, U] - \mathbb{E}[Y^O\mid A^O=0, X^O, U])\mid X^O=x\right] - (\mathbb{E}[Y^O\mid A^O=1, X^O=x]- \mathbb{E}[Y^O\mid A^O=0, X^O=x])$$

where U denotes the unobserved confounders. To the best of our knowledge, the only types of methods that address confounding bias either (1) use latent variable models to recover unobserved confounders, often from noisy proxies or multiple/sequential treatments (see [1,2]), or (2) combine observational and randomized data with perfect compliance (see [3,4]). For (1), additional data (e.g. multiple treatments, nosy proxies) that enable the latent modeling might not be available, and even if available, the unconfoundedness condition given the additional data cannot be tested in practice. For (2), randomized data with perfect compliance is difficult to obtain in practice in the settings we discuss, either due to implementation bottlenecks, ethical considerations, financial constraints, ets, which is what our paper addresses by considering encouragement/IV designs. We will add this discussion to the literature review section.

• Dataset sizes: The dataset sizes (denoted by $n_E$​ for the experimental/IV dataset and $n_O$​ for the observational dataset) do not need to be equal. In Theorems 2 and 3, we show how both sizes influence the algorithm's convergence rate. The theorems detail the dependency of the convergence rate on $n_E$​ and $n_O$, illustrating how a smaller IV dataset ($n_E$​) affects the convergence rate. Since this is not currently reflected in the experimental section, we have added additional experimental results in the rebuttal PDF. Specifically, in Figure 1, we performed parametric extrapolation simulations by varying the ratio $n_E/n_O$​ while keeping $n_O=10,000$ fixed throughout the experiments. We display the mean squared error $\pm$ standard deviation across 100 iterations. As expected, the error in the $\widehat{\tau}^E$ estimation increases significantly with smaller $n_E$​ (due to low compliance in the finite sample). However, the corrected $\widehat{\tau}$ still shows a smaller mean error than the observational $\widehat{\tau}^O$, and this error steadily decreases as the $n_E/n_O$ ratio increases. We will include these additional results in the experimental section of the appendix.

• High-dimensional settings: We conducted additional experiments by modifying the data-generating process (DGP) to include $d=10$ features, with both baselines and bias depending on all features:

Y = 1 + A + X + 2A\beta^T X + 0.5X^2 + 0.75AX^2 + U + 0.5\epsilon_Y $$

U \mid X=x, A=a \sim N\left(\gamma^T x\left(a-\frac{1}{2}\right), 1-\left(a-\frac{1}{2}\right)^2\right)

where the coefficients $\beta, \gamma\in [-1, 1]^{d}$ are set at random at the beginning of the experiment. In this scenario, the bias is given by $b(x)=-\gamma^T x$. We leave all other settings and parameters (including $n_O=n_E=5000$) unchanged and perform the parametric extrapolation described in the paper. For this high-dimensional setting, we obtain the following results for the mean squared error (MSE) and standard deviation (SD) across 100 iterations: | |$\widehat{\tau}^O(x)$ | $\widehat{\tau}^E(x)$| $\widehat{\tau}(x)$| |--|--|--|--| |MSE$\pm$SD|$3.25\pm 0.15$|$7.70\pm 1.54$|$1.25\pm 0.20$| The high MSE of the IV estimator ($\widehat{\tau}^E(x)$) indicates the difficulty in estimating compliance in high-dimensional settings. Similarly, the observational data estimator ($\widehat{\tau}^O(x)$) is clearly biased. However, the combined data estimator ($\widehat{\tau}(x)$) performs much better in this high-dimensional setting. We will include more results (such as varying $n_E$​ and $n_O$​) for the high-dimensional setting in the experimental section appendix. **Re: Other Questions** When realizability does not hold, our estimator may be inconsistent, exhibiting a bias that persists asymptotically. The magnitude of this bias depends on the extent to which realizability is violated, i.e., how far the true representation deviates from the assumed function class. In some cases, it may still be beneficial to perform this analysis despite uncertainty about realizability, as the bias from this violation could be substantially smaller than the confounding bias. References: [1] Kuzmanovic, M., Hatt, T. and Feuerriegel, S., 2021, November. Deconfounding Temporal Autoencoder: estimating treatment effects over time using noisy proxies. In Machine Learning for Health (pp. 143-155). PMLR. [2] Wang, Y. and Blei, D.M., 2019. The blessings of multiple causes. Journal of the American Statistical Association, 114(528), pp.1574-1596. [3] Kallus, N., Puli, A.M. and Shalit, U., 2018. Removing hidden confounding by experimental grounding. Advances in neural information processing systems, 31. [4] Hatt, T., Berrevoets, J., Curth, A., Feuerriegel, S. and van der Schaar, M., 2022. Combining observational and randomized data for estimating heterogeneous treatment effects. arXiv preprint arXiv:2202.12891.