Differentially private learners for heterogeneous treatment effects

• Connection of the two proposed approaches: Thank you for giving us the chance to elaborate on the connection between both approaches, especially their differences and similarities. As correctly noted, the functional approach can as well be employed to privately release a finite a-priori known number of CATE estimates. Therein, the approach coincides with the first approach. Nevertheless, the second approach can also be employed for iteratively querying the underlying function. This highly differentiates the approach from the finite setting. The differences in the two ways of how our functional approach can be used can be best seen by inspecting sampling procedure from the Gaussian process $G$. We discuss the connection between the two approaches (finite-query approach and functional approach) based on the type of query f in the following:

  1. Simultaneous finitely many queries: For querying the function only once with a finite amount of queries, sampling from a Gaussian process implies sampling from the prior distribution of the process. In empirical applications, this means that one samples from a multivariate normal distribution. Therefore, the noise added in the functional approach is similar to the finite-query approach. However, the approaches are not the same, as the noise added in the functional approach is correlated, whereas the noise variables in finite-query approach are independent. Still, both approaches guarantee privacy.
  2. Iteratively querying the function: In this setting, sampling from a Gaussian process implies sampling from the posterior distribution of the process. Specifically, if no query has been made to the private function yet, the finite-query approach proceeds by providing the first private CATE estimate of query $x_1$. Observe that the privatization of every further iterative query $x_i$ needs to account for the information leakage through answering former queries. Thus, sampling from a Gaussian process now relates to sampling from the posterior distribution. To do so, it is necessary to keep track and store former queries $x_1,\ldots,x_{i-1}$ and the privatized outputs. This setting is entirely different from our finite query approach in which we propose to add Gaussian noise scaled by the gross-error sensitivity.

  Action: We added a new supplement where we discuss the connection between both approaches (see our new Supplement C). Furthermore, we provide an alternative algorithm for implementing DP-CATE for functions (see ournew Supplement C.1).

[1] Alicia Curth and Mihaela van der Schaar. Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms. In Conference on Artificial Intelligence and Statistics (AISTATS), 2021.

[2] Stefan Feuerriegel, Dennis Frauen, Valentyn Melnychuk, Jonas Schweisthal, Konstantin Hess, Alicia Curth, Stefan Bauer, Niki Kilbertus, Isaac S. Kohane, and Mihaela van der Schaar. Causal machine learning for predicting treatment outcomes. Nature Medicine, 30(4):958–968, 2024.

[3] Donald. B. Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 2005.

Dear reviewer qbNM,

Thank you for your feedback on our manuscript! We took all your comments at heart and improved our paper accordingly. Below, we provide answers to the questions in your review. We updated our PDF and highlighted all key changes in blue color.

• Consistency of DP-CATE: Thank you for the suggestion. We are more than happy to discuss the consistency of our final private meta-learners in detail. Our proposed DP-CATE framework provides consistent CATE estimators as we will outline in the following. Note that our framework does not alter the estimation procedure itself. As we built upon the consistent doubly robust estimators, the CATE estimator without the Gaussian noise is consistent. Furthermore, observe that the amount of noise added decreases in the sample size $n$. Let $g^P$ denote the private estimator, $g$ the non-private base meta-learner and $\tau$ the true CATE. Then

Therefore, $g^P$ is a consistent estimator of $\tau$.

**Action**: We have **added a discussion** on the consistency of our framework in a separate appendix dedicated to this topic (see **Supplement D**)

• Clarification on causal assumptions: Thank you for highlighting that readers from fields outside of causal inference might not be familiar with the necessary assumptions for identifying causal effects from observational data. In the following, we describe the three assumptions in more detail: The estimation of causal quantities, such as the conditional average treatment effect $\tau(x) = \mathbb{E}[Y(1) -Y(0) \mid X = x]$ involves counterfactual quantities $Y(a)$, as only one outcome per individual can be observed. Therefore, identification of causal effects from observational data necessitates the following three assumptions common in the literature (e.g. [1],[2],[3]).

  1. Positivity/Overlap: The treatment assignment is not deterministic. Specifically, there exists a positive probability for each possible combination of features to be assigned to both the treated and the untreated group, i.e., $\exists \ \kappa > 0$ such that $\kappa <\pi(x) < 1-\kappa$ for all $X=x \in \mathcal{X}$.
  2. Consistency: The potential outcome $Y_i(a=k)$ equals the observed factual outcome $Y_i$ when individual $i$ was assigned treatment $A_i=k$.
  3. Unconfoundedness: Conditioned on the observed covariates, the treatment assignment is independent of the potential outcomes, i.e., $Y(0), Y(1) \perp A|X$. Specifically, there are no unobserved variables (confounders) influencing both the treatment assignment and the outcome. The assumptions are necessary for consistent causal effect estimation for all machine learning models. Then, CATE is identifiable as $\tau(x) := \mathbb{E}[Y(1) -Y(0) \mid X = x] = \mu(x, 1) - \mu(x, 0),$ where $\mu(x,a) = \mathbb{E}[Y \mid X=x, A=a]$.

  Action: We added a new section where we explain the underlying assumptions and give a general theoretical background on CATE estimation (see our new Supplement A3). We hope that this makes our paper more accessible to readers with a background in differential privacy but without in-depth knowledge of causal inference.

• Tightness of upper bound on sensitivity: Thank you for raising this question. Although tightness of the upper-bound is beneficial as loose upper bounds can lead to overly perturbed estimates, the task of deriving tight bounds is highly non-trivial. This is a common problem in differentially private estimation methods, which employ bounds on the global sensitivity as originally demanded by the DP. As in Avella-Medina (2021), it is not possible to prove tightness of the bounds. We thus leave this task for future research.

• Comparison of both approaches (finite and functional DP-CATE): Thank you for giving us the opportunity to explain the differences between the two proposed approaches in practical scenarios. If one only wants to release private CATE estimates once, both approaches are applicable. Nevertheless, the second approach called “functional approach” can also be employed for iteratively querying the function, which is especially of interest to medical practitioners aiming to assess the treatment effect of a drug for various patients with different characteristics. Put simply, when companies want to release a decision support system to guide treatment decisions of individual patients. Such treatment decisions are made based on the entire CATE model, then the “functional” approach is preferred. In contrast, the first approach (called “finite-query approach”) is preferred whenever only a few CATE values should be released. This is relevant for researchers (or practitioners) who may want to share the treatment effectiveness for a certain number of subgroups (but not for individual patients).

  The functional approach requires sampling from a Gaussian process. Depending on wheter one aims to report finitely many querie once through this approach or iteratively query the function, the sampling procedure from the Gaussian process $G$ differs. We highlight the differences of the type of queries in the following:

  1. Simultaneous finitely many queries: For querying the function only once with a finite amount of queries, sampling from a Gaussian process implies sampling from the prior distribution of the process. In empirical applications, this means that one samples from a multivariate normal distribution. Therefore, the noise added in the functional approach is similar to the finite-query approach. However, the approaches are not the same, as the noise added in the functional approach is correlated, whereas the noise variables in finite-query approach are independent. Still, both approaches guarantee privacy.
  2. Iteratively querying the function: In this setting, sampling from a Gaussian process implies sampling from the posterior distribution of the process. Specifically, if no query has been made to the private function yet, the finite-query approach proceeds by providing the first private CATE estimate of query $x_1$. Observe that the privatization of every further iterative query $x_i$ needs to account for the information leakage through answering former queries. Thus, sampling from a Gaussian process now relates to sampling from the posterior distribution. To do so, it is necessary to keep track and store former queries $x_1,\ldots,x_{i-1}$ and the privatized outputs. This setting is entirely different from our finite setting approach in which we propose to add Gaussian noise scaled by the gross-error sensitivity.

  Action: We included a new supplement where we discuss the connections between the two approaches (see our new Supplement C). Furthermore, we provide an alternative algorithm for implementing DP-CATE for functions (see our new Supplement C.1).

Dear reviewer Ucae,

Thank you for your positive and detailed review of our paper! We took all your comments at heart and improved our paper accordingly. We updated our PDF and highlighted all key changes in blue color.

Answer to weaknesses:

1. Relation to Hall (2013): Thank you for this suggestion. We cite Hall (2013) more carefully to offer context to our readers. We further discuss the differences to Hall (2013) as part of the section “Answers to questions” section below.

   Action: We referred to Hall (2013) more often to provide more context to our readers. Thereby, we also spell out the differences between Hall (2013) and our work.

2. Terminology (robustness): Thank you for giving us the chance to clarify the meaning of robustness and improve our terminology. Throughout our paper, we focus doubly robust estimators. This type of robustness implies that the estimators remain consistent even when either the outcome or the treatment selection model (i.e., the propensity model) is not correctly specified. The estimators are insensitive to perturbations of the nuisance models. We apologize that we also used the term robustness in other places to refer to the insensitivity to outliers. Upon reading your comment, we realized that this could be confusing to the reader. => We rewrote the respective texts and improved our terminology. In particular, we clearly distinguish between “double robustness” and flexibility or insensitivity, where appropriate.

   Action: We updated the terminology in our manuscript. Additionally, we included a new section where we give theoretical background on the double robustness property (Supplement A3).

3. Recalling definitions: Thank you for highlighting that readers from fields outside of causal inference might not be familiar with certain definitions and notations. We thus added more background materials throughout our manuscript. We hope that this makes our paper more accessible to readers without in-depth knowledge of causal inference.

   Action: We have introduce important the definitions throughout the manuscript. Additionally, we included a new section where we give theoretical background on CATE estimation and the underlying assumptions (Supplement A3).

Dear reviewer smF6,

Thank you for your positive evaluation of our paper! We took all your comments at heart and improved our paper accordingly. We updated our PDF and highlighted all key changes in blue color.

Answer to weaknesses:

1. Jargon: Thank you for giving us the chance to improve our terminology. Throughout the paper, we consider so-called doubly robust estimators for causal inference. Double robustness implies that the estimators remain consistent even when either the outcome or the treatment selection model (i.e., the propensity model) is not correctly specified. The estimators are thus insensitive to perturbations of the nuisance models, which is a great practical benefit.

   Action: We simplified the terminology in our manuscript. Additionally, we included a new section where we give theoretical background on the double robustness property (Supplement A3). We hope that this makes our paper more accessible to readers with a background in differential privacy but without in-depth knowledge of causal inference.

2. Comparison to baselines: Thank you for giving us the chance to show the superiority of our approach to other more restrictive methods. We highlight again that other DP methods are either not model-agnostic (Niu et al. (2019)), or restricted to RCT data or restricted to data with binary outcomes (Betlei et al. (2017), Guha & Reiter (2024)). Therefore, the baselines are not applicable to general settings as in our paper. In other words, powerful baselines with theoretical DP guarantees are missing.

   We therefore appreciate the reviewer’s suggestion about a very naive baseline based on k-anonymization. As we show in the following, such naive baseline has clear limitations in both theory and in our experiments. First, such k-anonymization essentially performs a data aggregation as part of the privatization technique, yet this can break the consistency assumption necessary for causal identifiability. Hence, this can lead to biased results. Further, it does not offer formal privacy guarantees as our method. Nevertheless, we agree with the reviewer that a comparison to such a more naive privacy method is of interest. Hence, we implemented the method and performed new experiments. We observe that our method provides superior CATE estimates (while further offering theoretical guarantees to ensure DP) for almost all privacy budgets. Hence, our new experiments confirm again the effectiveness of our proposed method.

   Action: We added a new experiment where we compare our method to a naive baseline based on k-anonymization, finding that our method is clearly superior (see our new Supplement E).

Answer to questions:

1. Domain of X: The only constraint on the domain of X is that it is bounded, which is very reasonable in practice. We do not make any further assumptions. In practice, the domain of X can vary from only very few features to high-dimensional settings. We aimed to cover both settings in our synthetic experiments on datasets with 2 and 30 confounders, respectively.

   Action: We stated the assumptions on the domains of the variables more clearly in our manuscript.

2. Characteristics of setting 1: Thank you for raising this question. We agree that further motivation would greatly improve comprehension, and we are thus happy to offer additional clarifications. In practice, the queries in this setting can range from a very small number, such as different age groups, as suggested by the reviewer, to very large numbers of CATE estimates, such as for various combinations of patient characteristics. In our experiments, the number of queries varied from 300 queries (synthetic experiments) to 1312 queries (MIMIC) and 2659 queries (TCGA).

   Action: We included more elaborations to explain our experimental setup.

Dear reviewer smF6,

We hope we sufficiently addressed all your concerns in our rebuttal. If you have any more questions or concerns, we are happy to answer them as soon as possible. Please let us know if this is the case. Otherwise, we would highly appreciate it if you could raise your evaluation score for our manuscript.

Best regards, The authors

Answer to Questions:

• Queries for setting 1 (Fig. 3+4): Thank you for spotting this. We apologize that we did not state the number of queries we evaluated our framework on. In the synthetic experiments (Fig. 3+4), we evaluated DP-CATE on 300 queries. The other experiments were evaluated on 1312 queries (MIMIC) and 2659 queries (TCGA). Action: We added this to the experimental results.

• Tightness of the sensitivity bound by the gross-error sensitivity: Thank you for raising this question. Although tightness of the upper-bound is beneficial as loose upper bounds can lead to overly perturbed estimates, the task of deriving tight bounds is highly non-trivial. This is a common problem in differentially private estimation methods, which typically employ bounds on the global sensitivity as originally demanded by the DP. As in Avella-Medina (2021), it is unfortunately not possible to prove the tightness of the bounds. We thus leave this task for future research.

• Question on functional DP-CATE in Fig.4: Thank you for this interesting question. The reviewer is right that in the shown plot DP-CATE underestimates the true CATE. However, there is no consistent under- or overestimation of the method for certain settings. Rather, this behavior is merely a result of the privatization approach in the functional setting 2. In contrast, in setting 1, we sample $d$ independent random normal variables, while, in setting 2, we sample from a Gaussian process through sampling a multivariate normal variable with the corresponding covariance. Thus, the added noise is correlated. This might result in what appears to be consistent under- or overestimation of the target. Action: We included a new supplement where we discuss the differences and similarities of the two approaches and give more intuition on the behavior of setting 2 (Supplement C).

• Y-axis in Fig. 6+7: In Figure 6+7, the y-axis represents the CATE, not the estimation error. Action: We stated the presented quantities more precisely in the updated version of our manuscript.

• Computation of the Lipschitz constant for the empirical evaluation: Thank you for noting that we did not specify how we calculated the Lipschitz constant in our empirical evaluation. In our settings, we employ the L2-loss on a bounded domain. Therefore, although the L2-loss itself is not Lipschitz, we can calculate L numerically as the gradient at the upper bound of the loss. Action: We stated the computation of the Lipschitz constant in our empirical evaluation more precisely in the updated version of our manuscript.

Dear Reviewer ukCV,

Thank you for your detailed feedback on our manuscript. We took all your comments at heart and improved our manuscript accordingly. We updated our PDF and highlighted all key changes in blue color.

Response to weaknesses:

• Knowledge of the Lipschitz constant: Thank you for raising this question. We agree that it might not directly become clear why the assumption of the knowledge of the Lipschitz constant is not restrictive. First, employing the lipschitz constant of the loss for post-hoc processing is very common across many fields (e.g.[1],[2],[3],[4]). Second, for many losses, the Lipschitz constant is data-independent and directly computable from the loss function. For example, for the L1 loss, the Lipschitz constant L equals 1; for the Huber loss L equals the loss parameter $\delta$, or for the truncated L2 loss, the constant equals the gradient at the truncation value. Action: We added a discussion of the Lipschitz constant to the updated version of our manuscript.

• Details on the experiments: Thank you! We apologize for not stating sufficient details on how we conducted our experiments. We greatly extended our experimental details. We also respond below to your specific questions and how we improved our paper as a result (see our “Response to question”). Action: We added more details on the experiments to our manuscript in the respective parts in the main paper and the corresponding supplement.

• Real-world dataset with heterogeneous treatment effect: Thank you for giving us the chance to show the applicability of DP-CATE to medical datasets with heterogeneous treatment effects. Upon reading your question, we realized that we chose a clinical example where the effect is constant (i.e., the effect of ventilation on clinical outcomes is known to have little variation across patients of different ages) and therefore comes without interesting clinical interpretation. As a result, we have opted for a different example with direct clinical interpretation. Specifically, we now present a new experiment in which we estimate the effect of ventilation on red blood cell count conditioned on hematocrit (see our new Fig. 6). Here, we should expect a nice positive relationship as stipulated by domain knowledge in medicine. Aligned with this, we indeed with an increasing effect across different hematocrit values. Action: We reworked our experiment for the MIMIC dataset to show a more meaningful example with clinical interpretation (see our new Figure 6).

Dear reviewer ukCv,

We hope we sufficiently addressed all your concerns in our rebuttal. If you have any more questions or concerns, we are happy to answer them as soon as possible. Please let us know if this is the case. Otherwise, we would highly appreciate it if you could raise your evaluation score for our manuscript.

Best regards, The authors