Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation

Validity of conclusions in our experimental analysis

We agree with the reviewer and would like to state that we were extremely careful about such issues. We trained the various nuisance models for CATE estimators and metrics using cross-validation with AutoML to be robust to issues like different partitioning of the training and validation set. Further, we have used a large number of random seeds (10 per experiment) to be robust to discrepancy from initialization of model parameters. Finally, since the statistics of ITE estimates, covariate dimension, etc. can change across datasets (Table 4, 5, 6), to determine robust trends we look at a large collection of datasets (78 in total) and train a wide variety of CATE estimators per dataset.

Data requirements in the proposed framework

The proposed framework does not in principle need large amounts of data. Our goal was to conduct a comprehensive large-scale evaluation study to make a fair comparison between the various evaluation metrics used in the literature. This helps to create an informative prior for practitioners regarding which evaluation metrics maybe useful for them.

Further, the suggestions of a two-level model selection strategy and causal ensembling do not necessarily need lots of data. It is upto the practitioner to train as many CATE estimators as required for their application and then incorporate the two-level model selection and causal ensembling strategy. Also, with regards to the training of the nuisance models (outcome & propensity models), we are in the typical regime as used by the other works in this area, with the standard train/validation split for the various datasets consider in our work.

Missing relevant works

We tried to be as thorough as we could about the prior works and did cover the most relevant ones. We do appreciate the suggestion by the reviewer and are happy to talk about other related works concerning our study.

Regarding the work by Nguyen et al. pointed out by the reviewer, their focus is more on sensitivity analysis, which deals with measuring robustness to unobserved confounders, randomized treatments, etc. While sensitivity analysis is an important practical aspect of causal inference, our goal with model selection is more general since sensitivity analysis alone might not be able to result in a clear distinction between several CATE estimators.

We wish to remind the reviewer that there are still a few hours left before the rebuttal period ends. It would be great to get your opinions on our response.

Lack of Deep Learning based methods

We agree with the reviewer that it might be interesting to include deep learning-based CATE methods as well. However, since the scale of the study was already quite large and our primary goal was model selection rather than learning state-of-the-art CATE estimators we decided to focus on basic machine learning models for the nuisance models in the CATE meta-learners. Further, the datasets that we work with are mostly tabular datasets, hence basic machine learning techniques with appropriate hyperparameter tuning should have good performance as well. We use state-of-the-art hyperparameter tuning techniques like AutoML to learn well-performing nuisance models.

Cross-validation results with the ideal metric

We want to highlight that we compute results with the ideal cross-validation metric (PEHE) with ground-truth ITE estimates, as mentioned at the start of section 5. The ideal PEHE metric is used to normalize the reported scores so they can be made consistent across datasets, and the reported results in Tables 1, 2, and 3 demonstrate the percentage difference in the quality of the chosen estimator by a particular metric compared to the ideal metric. As expected, we can see that none of these metrics are better than the ideal PEHE metric as all the reported scores are greater than zero.

Also, if by the typical cross-validation method, the reviewer meant computing the cross-validation score of the nuisance models (like Mu Score in Schuler et al.), then such an approach is not feasible in our study since we already select over the nuisance models using AutoML and these are shared across the different CATE meta-learners in our experiment. Hence, the cross-validation score of the various nuisance models across CATE meta-learners would be similar.

Motivation behind new metrics

Although our results show that we do not gain much with these proposed metrics, Table 3 still shows that scores like TMLE, Qini, and Calibration (DR) are among the set of dominating metrics across all datasets. Hence, they could be considered as a good prior option for a new study and might be even more suitable for some applications that are tied to their specific use case, like learning under extreme propensity regime, etc.

The motivation for including adaptive propensity clipping and targeted learning metrics is the extreme propensity regime, where the metrics that depend on inverse propensity scores (IPW, DR) can become biased. Hence, these proposed metrics can handle the extreme propensity region (very small propensity for the observed treatment) by using estimates from the regression-based learners (S/T Learner) or adding inverse propensity correction (TMLE).

For the Calibration score, the intuition is to match the group ATE across subgroups denoted by the different percentiles of the CATE estimates. We compute the group ATE using DR/TMLE, etc. which is viewed as an unbiased sample of the group ATE, and then we compute the weighted error of the group ATE using the CATE estimator against the unbiased group ATE. For the Qini score, rather than computing the group ATE with the CATE estimator as in Calibration, here we compare the improvement with unbiased group ATE estimates versus the uniform sampling estimate. Intuitively, if there is heterogeneity in the data, then we should expect the policy of uniformly treating an individual to be worse as compared to the policy that assigns treatment to the top-k percentile of the CATE estimates. Hence, the Qini score is qualitatively different than most of the other metrics as it does not directly compare the CATE estimates, rather it evaluates whether the groups most likely to be treated would benefit from the treatment.

Conclusions from experimental results

We would like to point that we also report results on the multiple ACIC 2016 datasets (75 datasets) in Table 1, 2, and 3 in the main paper, where it does not happen that the set of dominating metrics is bigger than the set of non-dominating metrics. Note that while there is a single column for the ACIC 2016 dataset, it is a collection of 75 datasets, therefore, the trends reported for them are more informative than the Twins and LaLonde datasets combined. Hence, the conclusions that we draw are valid since we do not rely only on the Twin and LaLonde datasets, but we also include the multiple ACIC 2016 datasets.

We agree with the reviewer that we may overestimate the set of dominating metrics by only looking at the Twins and LaLonde datasets. However, to determine the set of globally dominating metrics we look at the metrics that are dominating across all the ACIC 2016 datasets as well as the Twins/LaLonde datasets. It is indeed possible for some metrics to be dominating on a few datasets like Twins and LaLonde, but they are not dominating when evaluated across more datasets like ACIC 2016. Specifically, metrics like Match Score and IPW (and its variants) are dominating on the Twins and LaLonde datasets but they are not dominating on the multiple ACIC 2016 datasets. Hence, for a more useful recommendation to practitioners, we suggest metrics that are dominating across a wide range of datasets. Further, the conclusions we draw about the two-level model selection strategy and causal ensembling improving over the single-level strategy are valid since they hold across all the datasets.

Number of random seeds used in the study

As the reviewer stated the scale of the study is quite large, hence 10 random seeds for 78 datasets and 415 CATE estimators per dataset implies a total of $323,700$ CATE estimators trained and evaluated. Therefore, we think that 10 random seeds are enough to determine reliable conclusions given the computational cost of carrying out the study.

However, we will generate results for another set of 10 random seeds (making it a total of 20 random seeds), and report them soon in a few days.

Intuition behind new metrics

The motivation for including adaptive propensity clipping and targeted learning metrics is the extreme propensity regime, where the metrics that depend on inverse propensity scores (IPW, DR) can become biased. Hence, these proposed metrics can handle the extreme propensity region (very small propensity for the observed treatment) by using estimates from the regression-based learners (S/T Learner) or adding inverse propensity correction (TMLE).

For the Calibration score, the intuition is to match the group ATE across subgroups denoted by the different percentiles of the CATE estimates. We compute the group ATE using DR/TMLE, etc. which is viewed as an unbiased sample of the group ATE, and then we compute the weighted error of the group ATE using the CATE estimator against the unbiased group ATE. For the Qini score, rather than computing the group ATE with the CATE estimator as in Calibration, here we compare the improvement with unbiased group ATE estimates versus the uniform sampling estimate. Intuitively, if there is heterogeneity in the data, then we should expect the policy of uniformly treating an individual to be worse as compared to the policy that assigns treatment to the top-k percentile of the CATE estimates. Hence, the Qini score is qualitatively different than most of the other metrics as it does not directly compare the CATE estimates, rather it evaluates whether the groups most likely to be treated would benefit from the treatment.

Although our results show that we do not gain much with these proposed metrics, Table 3 still shows that scores like TMLE, Qini, and Calibration (DR) are among the set of dominating metrics across all datasets. Hence, they could be considered as a good prior option for a new study, and might be even more suitable for some applications that are tied to their specific use case, like learning under extreme propensity regime, etc.

Reference for disagreement between CATE estimators

Several benchmarking studies (listed below) show this issue where multiple CATE estimators can generate different results, which can be explained since the choice of the nuisance model and their hyperparameters, along with the choice of estimation strategy will give rise to different empirical estimates.

• ACIC Competition: https://arxiv.org/pdf/1707.02641.pdf
• Meta Learners for CATE estimation: http://proceedings.mlr.press/v130/curth21a/curth21a.pdf
• Model Selection: https://arxiv.org/pdf/1804.05146.pdf

Please let us know if your concerns have been resolved, we have addressed the issues you expressed regarding the experimental conclusions with more datasets and random seeds. Our study contains the 75 ACIC 2016 datasets in addition to the Twins and LaLonde datasets, and we now have a total of 20 random seeds per estimator for each dataset, therefore making the conclusions robust. We are happy to address any further questions that you might have.

Thanks a lot for engaging with our work and for your suggestions!

Regarding the terminology and writing in the paper

We thank the reviewer for their detailed feedback regarding the presentation and writing! We agree and would make the writing better with clear definitions and clarifications. We have already updated the paper with these suggestions and the changes can be viewed in the new draft.

Specifically, we refer to X as covariates now as opposed to controls and have added a sentence in section 2 to specify exactly what we mean by metrics for model selection. Further, other suggested changes regarding the writing have been made in the updated draft.

Typo in CATE model selection metrics

We thank the reviewer for pointing this out, indeed there was a typo in the sentence before the equation and it should have been ($\tilde{\tau}$). The same has been corrected in the updated draft.

IPTW Switch equation and related description

We agree with the reviewer and have updated the related description. Indeed, the switch score would rely on IPW if the propensity is not very small. We set epsilon to very small values and hence IPW is used when the propensity for the observed treatment is not very small.

Final models in CATE estimators

By final model, we mean the regression model used for predicting CATE in direct meta-learners like DR Learner (eq 15), R Learner (eq 16). For example in DR Learner, we construct the doubly robust potential outcomes ($Y_{0}^{DR}, Y_{1}^{DR}$) and then predict the CATE by learning the regression model ( $f_{\theta} : X \rightarrow Y_{1}^{DR} - Y_{0}^{DR}$ ) that maps the covariates to doubly robust potential outcomes. This is referred to as the final (regression) model.

Details regarding the CATE estimators were pushed in section B in the Appendix due to space constraints, but we will try to add a few lines about them in the main text, ideally in section 2. For now, we have referred to the specific equation when mentioning the final model of CATE estimators in Section 4.

Details regarding the two-level model selection strategy

The details regarding the two-level model selection strategy are present in Section 4, and the same has been mentioned under the contributions section (bullet 2). But we agree the same could be presented in a better manner, and we have done so in the updated draft.

Indeed, the choice of metric for selecting hyperparameters in the two-level strategy depends on the CATE estimator. Specifically, the hyperparameters for CATE estimators in our study are the final regression models, hence we need to do this for only DR-Learner, R-Learner, X-Learner, and Projected S-Learner, as the others do not have hyperparameters. Now for each of the four CATE estimators mentioned above, we select the hyperparameters using a metric with similar inductive bias, i.e., DR Score for the DR-Learner; R Score for the R-Learner, X Score for the X-Learner, and S Score for the Projected S-Learner.

Essentially, the two-level model selection strategy reduces the population of CATE estimators to different CATE meta-learners, and the choice of metric for reducing the population of a specific CATE meta-learner (e.g. DR-Learner) is designed to maximize the similarity with the inductive bias of the CATE meta-learner, to do reliable hyperparameter optimization.

Motivation behind new metrics

Our motivation was to connect widely used ideas from related problems like policy learning, uplift modeling, etc. to have more good candidates for model selection. Although our results show that we do not gain much with these proposed metrics, Table 3 still shows that scores like TMLE, Qini, and Calibration (DR) are among the set of dominating metrics across all datasets. Hence, they could be considered as a good prior option for a new study.

As the reviewer stated, the motivation for including adaptive propensity clipping and targeted learning metrics is the extreme propensity regime. For the Calibration score, the intuition is to match the group ATE across subgroups denoted by the different percentiles of the CATE estimates. We compute the group ATE using DR/TMLE, etc. which is viewed as an unbiased sample of the group ATE. Subsequently, we compute the weighted error of the group ATE using the CATE estimator against the unbiased group ATE.

For the Qini score, rather than computing the group ATE with the CATE estimator as in Calibration, here we compare the improvement with unbiased group ATE estimates versus the uniform sampling estimate. Intuitively, if there is heterogeneity in the data, then we should expect the policy of uniformly treating an individual to be worse as compared to the policy that assigns treatment to the top-k percentile of the CATE estimates. Hence, the Qini score is qualitatively different than most of the other metrics as it does not directly compare the CATE estimates, rather it evaluates whether the groups most likely to be treated would benefit from the treatment.

To address the point raised by reviewer i3fD, we have updated the draft with results over another set of 10 random seeds, hence a total of 20 random seeds in Tables 10 and 11 in section E (Supplementary). Our conclusions with 20 random seeds are still the same as before with 10 random seeds, hence the conclusions are robust to different random parameter initializations.