Conformal prediction for causal effects of continuous treatments

Dear Reviewer nAyf,

Thank you for your detailed feedback on our manuscript. We took all your comments at heart and improved our manuscript accordingly. We updated our PDF and highlighted all key changes in blue color.

Answer to weaknesses:

• Terminology: Thank you for helping us to improve the reader’s understanding of our manuscript by clarifying our employed terminology.

  Action: We improved the terminology according to your suggestions in the updated version of our manuscript.

• Relation to Lei and Candes (2021): Thank you for raising this concern. The reviewer is correct in that Lei and Candès (2021) provide an approach for CP with unknown propensity score through specifying a so-called coverage gap. Specifically, Lei & Candés (2021) consider the estimated propensity by incorporating the estimation error as a TV-distance term in the coverage guarantees. However, the TV-distance assumption contains several drawbacks compared to our assumption based on density ratios (DR): The bounded DR assumption is more universal. For example, the DR assumption implies boundedness of the TV-distance (see Bretagnolle and Huber inequality) or f-divergences.
  For large TV-distances (close to 1), Lei & Candes can only construct informative intervals with a very limited coverage $\alpha \in (0, 1 - TV)$), since, otherwise, the method yields non-informative intervals. In contrast, thanks to our DR assumption, we can infer informative intervals with any coverage ($\alpha \in (0, 1)$) for any $M$.

  We acknowledge that our statement is very strong and unclear, and we apologize for the confusion. We thus rewrote the corresponding passages to include the explanations above.

  Action: We updated the corresponding passages in the updated version of our manuscript.

• Relation to Gibbs (2023): The reviewer is correct that our work is built upon previous work outside the field of causal inference from Gibbs (2023). We apologize if this did not become clear in our manuscript. However, we note that our contribution lies in combining two research areas: causal inference and conformal prediction. Furthermore, Gibbs (2023) cannot be directly applied to our setting but only serves as an underlying tool which has to carefully be adapted to our setting (i.e., the propensity estimation and smoothing).

  Action: We referred to Gibbs (2023) more frequently in the appropriate parts of our manuscript. We also spelled out the differences more clearly.

• Under-coverage of empirical results: Thank you for letting us explain the variability in the coverage and for spotting a typo regarding the number of runs we conducted in our experiments. The reviewer is correct in that Fig. 4 and 5 show some instability for certain cases. This is likely due to the fact that the CP coverage guarantees are only marginal. Therefore, we might experience under- or over-coverage. However, we note that across all runs, our method, on average, achieves the desired coverage. These instabilities only occur in single settings. Furthermore, the variance in coverage of our method is much lower than the coverage variance of the baselines.

  Additionally, we would like to apologize for our mistake in stating the number of runs (10) in the text. This was due to inattention when transferring the text from a former version of the manuscript. Observe that in the figure descriptions, we reported the correct number of runs (50).

  Action: We added a discussion on the stability of our method to the updated version of the manuscript (see new Supplement C)

• Computational complexity: Thank you for raising this concern. As stated, our optimization procedure can be computationally expensive. Therefore, the proposed method might not be applicable to quantifying uncertainty in very high-dimensional queries. We agree that information about the time needed for calculating the prediction intervals is important to judge its benefit to large real-world applications.

  Action: We stated the time for calculating the prediction intervals in the updated version of our manuscript (see Supplement G). Furthermore, we added a statement on the computational complexity to the limitations section of our paper.

• Experiments: Thank you for giving us the chance to show the applicability of our method to higher dimensional datasets and show its superiority over further baselines. However, we note again that by choosing low-dimensional datasets, we can visually inspect the prediction intervals. Nevertheless, We later also show that our method scales to high-dimensional settings on actual medical datasets. To evaluate our method further, we conducted additional experiments on the semi-synthetic TCGA, which includes high-dimensional gene expressions as confounders. Furthermore, we evaluate our method against additional baselines. Specifically, we compare our method additionally against (i) vanilla CP and (ii) a Gaussian process estimator to show the usefulness and necessity of considering the propensity shift in CP and to assess the aleatoric uncertainty in our experiments.

Action: We will report our new results in our new Supplement C of our updated manuscript.

Answer to questions:

1., 2., 3. See our answers to the respective questions in “weaknesses”

• Typo: Thank you for noting our typo.

  Action: We corrected it in the updated version of our manuscript.

• Hard interventions in synthetic experiments: The hard intervention in the synthetic experiments was designed to depend on the covariates as this is more realistic in practice. For example, a doctor would never consider giving the same medication dosage to a child as to an adult. Therefore, we are interested in prediction intervals for interventions which depend on the covariates.

• Estimation error: Thank you for raising this question. Assumption 1 does not imply that the estimation error has to be the same for all samples. We only assume an overall upper bound. Therefore, the parameter $c_a$ does not depend on the covariates $x$.

• Optimal parameters: Thank you for giving us the chance to explain the meaning of the optimal parameters in more detail. The optimal parameter $\sigma^{\ast}$ represents a trade-off between the uncertainty in the prediction and the uncertainty in the interval construction: A small $\sigma^{\ast}$ resembles the propensity of the hard intervention best. Thus, with sufficient or even infinite data close to $a^{\ast}$, we could construct the narrowest CP interval. However, the smaller $\sigma$, the less data close to $a^{\ast}$ will be available to calculate the prediction interval in practice. As a result, many calibration data samples will be strongly perturbed during the calculation, which increases the uncertainty and, thus, the interval size. The parameter $c_a$ allows us to incorporate the estimation error in the propensity score. It represents a weighting of the propensity shift such that the $(1-\alpha)$-quantile of the non-conformity scores is increased with higher estimation error.

  Action: We present a discussion of the optimal parameters in the updated version of our manuscript (see Supplement D4)

• Lemma 4: Thank you for noting our precision. The reviewer is correct that Lemma 4 only applies if the non-conformity score equals the absolute residual. However, we note that this is the setting we considered throughout our paper. We apologize for being imprecise regarding the setting in Lemma 4.

  Action: We stated the setting in Lemma 4 more precisely in the updated version of our paper.

Dear Reviewer wfcX,

Thank you for your detailed feedback on our manuscript. We took all your comments at heart and improved our manuscript accordingly. We updated our PDF and highlighted all key changes in blue color.

Answer to weaknesses:

• Motivation: Thank you for giving us the chance to clarify the motivation for considering the different challenges in our manuscript. It is true that the problem of the unknown propensity score is present for discrete treatments as well as continuous treatments. These challenges are not unique to the setting of continuous treatments. However, challenge (b) is often neglected in existing works for CP for discrete treatments. More importantly, we want to emphasize that our paper bridges two different fields. It especially targets readers from the field of treatment effect estimation who might not be familiar with CP. Therefore, we aim to highlight the challenges in CP for this setting for these readers.

  Action: We have improved our motivation and now mention that some challenges also apply to discrete treatment settings but we highlight that unique to the continuous treatment setting is that we rarely see data points and thus must involve smoothing.

• Hard and soft interventions: Thank you! Indeed, the missing setting could also be of interest by studying the effect of soft interventions, even if the propensity score is unknown. Upon reading your comment, realized that we were missing this case and should extend our paper accordingly. We thus extended our CP algorithm to this scenario. Specifically, we derive corresponding coverage guarantees for soft interventions on the estimated propensity score.

  Action: We have clarified the applicability of soft interventions for both scenarios. Furthermore, we added new Theorem 4 which provides coverage guarantees for soft interventions on the estimated propensity score.

• Constant M: Thank you for raising this question. Indeed, one can view the parameter $M$ as a type of sensitivity parameter. This implies that one needs to make an assumption about the maximum error-bound. In employing this bound, we follow former work in causal inference which incorporates domain knowledge to specify the parameter $M$ (e.g., [1],[2],[3]). Another way of making use of the parameter $M$ is to observe how the intervals change for varying $M$. This indicates how much effect the propensity missspecification has on the prediction interval and can help in making reliable decisions. A third option to calibrate $M$ is to employ measures for epistemic uncertainty on top of the propensity estimate when there is no domain knowledge for specifying $M$.

  Action: We added a discussion on $M$ in the updated version of our manuscript (see new Supplement D.5)

• Results on the medical dataset: Thank you for giving us the chance to elaborate on the results of our experiment. We agree that only comparing the method based on width is difficult. To provide a better evaluation, we will further present results on semi-synthetic data (based on your suggestion) in the updated version of our manuscript. This allows us to also numerically evaluate the coverage of our method.

Answer to minor notes:

• Non-unique solutions of quantile regression: Thank you for noting that this might not be clear for the reader. Quantile regressions might have multiple optimal values, which can depend on the indices of the scores (e.g., see [8]). Therefore, we must restrict our theoretical analysis to solvers that are invariant to the data ordering. We note than commonly used solvers, such as interior point solvers, are invariant to the data ordering.

  Action: We included more explanation to motivate the choice of our solver in the updated version of our manuscript.

• Size of algorithm text: We thank the reviewer for noting that the text in our pseudo-code is difficult to read.

  Action: We increased the size of the text in the updated version of our manuscript.

• Parameter $\sigma$: Thank you for giving us the chance to elaborate further on the parameter $\sigma$. In the optimization problem, $\sigma$ acts as an optimization variable. Specifically, $sigma$ functions as a parameter describing the optimal trade-off between the uncertainty in the prediction and the uncertainty in the interval construction. By optimizing over $\sigma$, we make sure that we achieve the best approximation of the non-conformity scores and thus of the quantile of interest for constructing the prediction interval.

  Action: We extended our discussion on $\sigma$ in the updated version of our manuscript

Answer to questions:

• Semi-synthetic experiments: Thank you for noting that an evaluation on semi-synthetic data can be valuable to assess the performance of our method. We will thus provide further results for the TCGA covariate dataset.

  Action: We will add semi-synthetic experiments to the updated version of our manuscript.

• Calibration of M: Please see our answer to your question on $M$ above.

• Smoothness of the dose-response curve: Thank you for raising this questions. Indeed, we assume the dose-response curve to fulfill some smoothness criterion. This is common in causal inference with continuous treatments (e.g., [4],[5]). This also motivates the use of kernel smoothing for estimating the treatment effect. Again, this is common in causal treatment effect estimation (e.g., [6],[7]).

  Action: We provide a discussion on the smoothness of the dose-response curve in Supplement D.3.

[1] Dennis Frauen, Fergus Imrie, Alicia Curth, Valentyn Melnychuk, Stefan Feuerriegel, and Mihaela van der Schaar. A neural framework for generalized causal sensitivity analysis. In International Conference on Learning Representations (ICLR), 2024.

[2] Nathan Kallus, Xiaojie Mao, and Angela Zhou. Interval estimation of individual-level causal effects under unobserved confounding. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.

[3] Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476):1619–1637, 2006.

[4] Patrick Schwab, Lorenz Linhardt, Stefan Bauer, Joachim M. Buhmann, and Walter Karlen. Learning counterfactual representations for estimating individual dose-response curves. In Conference on Artificial Intelligence (AAAI), 2020.

[5] Jonas Schweisthal, Dennis Frauen, Valentyn Melnychuk, and Stefan Feuerriegel. Reliable off-policy learning for dosage combinations. In Conference on Neural Information Processing Systems (NeurIPS), 2023.

[6] Edward H. Kennedy. Nonparametric causal effects based on incremental propensity score interventions. Journal of the American Statistical Association, 114(526):645–656, 2019.

[7] Lokesh Nagalapatti, Akshay Iyer, Abir, and Sunita Sarawagi. Continuous treatment effect estimation using gradient interpolation and kernel smoothing. In Conference on Artificial Intelligence (AAAI), 2024.

[8] Isaac Gibbs, John J. Cherian, and Emmanuel J. Candès. Conformal prediction with conditional guarantees. arXiv preprint, 2023

Dear reviewer wfcX,

Thank you for taking the time to read our rebuttal and provide further constructive feedback. We are fully committed to remedying any remaining questions.

In our manuscript, we have reported experimental results for our method on the medical semi-synthetic TCGA dataset (see red color in Supplement C). We now have also compared our method against the baselines. To this end, we have now additionally computed all four baselines on the semi-synthetic dataset, which we present below. Every cell includes three entries that denote the mean empirical coverage for the three confidence levels: 0.05, 0.1, and 0.2. Again, we find that our proposed method is best.

Action: We will add the results from above to the camera-ready version of our manuscript.

Dear Reviewer xiCH,

Thank you for your positive evaluation of our manuscript and your detailed feedback. We took all your comments at heart and improved our manuscript accordingly. We updated our PDF and highlighted all key changes in blue color.

Answer to weaknesses

• Numerical evaluation: Thank you. We followed your suggestion closely and improved our empirical evaluation as follows. In our updated manuscript, we will add new experimental results on higher-dimensional covariates, i.e., the TCGA dataset including high-dimensional gene expressions. Furthermore, we evaluate our method against additional baselines: (i) vanilla CP, without adjusting for the propensity shift, and (ii) a Gaussian process prediction.

  Action: We will add our new experimental results to new Supplement F of our updated manuscript.

• New evaluation metric: Thank you for this question. The interval [-inf,inf] would indeed be a very trivial but unuseful solution. To properly evaluate our method, we now also report the width of the resulting prediction intervals. However, we note that coverage is a more important metric than interval width. In practice, one can easily decide on the usefulness of the resulting interval by looking at the interval range, whereas one can never make any conclusion on the coverage of a returned interval which does not fulfill mathematical guarantees.

  Action: We further evaluated our method in terms of the width of the prediction interval (see our new results in Supplement G).

• Stability of our method: Thank you for pointing this out. The reviewer is correct in that Fig. 4 and 5 show some instability for certain cases. This is likely due to the fact that the CP coverage guarantees are only marginal. Therefore, we might experience under- or over-coverage. However, we note that, on average across all runs, our method achieves the desired coverage. These instabilities only occur in singular settings. Furthermore, the variance in the coverage of our method is much lower than of the coverage variance of the baselines.

  Action: We added a discussion on the stability of our method to the updated version of the manuscript (see new Supplement D.6)

Answer to questions

• New experiments on medical data: Thank you. Upon reading your question, we realized that we have forgotten to rescale the outcome after the prediction. This led to very unreasonable results in terms of blood pressure. Furthermore, we realized that, from a medical perspective, estimating the effect of ventilation on blood pressure might not be of interest. Therefore, we will present an updated experiment in which we estimate the hematocrit level after ventilation instead of the blood pressure. We hope our new analysis is more meaningful and thus of larger interest.

  Action: We will revise our experiment to provide an example that is more clinically relevant.

Thanks for the authors' replies. I am overall satisfied with the rebuttal.

Dear Reviewer qNm2,

Thank you for your detailed feedback on our manuscript. We took all your comments at heart and improved our manuscript accordingly. We updated our PDF and highlighted all key changes in blue color.

• Existing work for uncertainty quantification for effects of continuous treatments: Thank you for the question. Yes, there exist a variety of work for uncertainty quantification (UQ) in causal effect estimation for continuous treatments. Existing methods for UQ of causal quantities are often based on Bayesian methods (e.g., [1],[2],[3]). However, Bayesian methods require the specification of a prior distribution based on domain knowledge and are thus neither robust to model misspecification nor generalizable to model-agnostic machine learning models. Other methods only provide asymptotic guarantees (e.g. [4],[5]). The strength of conformal prediction, however, is to provide finite-sample uncertainty guarantees. Overall, none of the methods tackles the problem of distribution-free uncertainty quantification for causal effect estimation of continuous treatments in finite sample settings.

  Action: We extended our related work to cite additional works on UQ in causal inferences but where we highlight the advantages of our conformal prediction (see our revised Supplement D.1).

• Smoothness assumption: Thank you for raising this question. Indeed, Eq. 14 performs kernel smoothing to approximate the indicator function. As inherent to kernel smoothing, this requires sufficient smoothness of the underlying functions, which, in our case, is the propensity function. However, we note that this assumption is standard in causal effect estimation of continuous treatments (e.g. [6],[7]). Hence, when estimating treatment effects, interpolation and kernel smoothing of the outcome function are commonly employed (e.g., [8],[9]).

  Action:: We highlight that the smoothness assumption is standard in causal inference for continuous treatments (see our revised Section 3). We further provide a discussion of the employed kernel-smoothing and the underlying smoothness assumption in Supplement D.3.

• Trade-off through the parameter $\sigma$: Thank you for giving us the opportunity to provide more information on the trade-off imposed by the parameter $\sigma$. The optimal parameter represents a trade-off between the uncertainty in the prediction and the uncertainty in the interval construction: A small $\sigma$ resembles the propensity of the hard intervention best. Thus, with sufficient or even infinite data close to $a^{\ast}$ , we could construct the narrowest CP interval. However, the smaller $\sigma$, the less data close to $a^{\ast}$ will be available to calculate the prediction interval in practice. As a result, many calibration data samples will be strongly perturbed during the calculation, which increases the uncertainty and, thus, the interval size.

  Action:: We added a discussion on the trade-off through parameter $\sigma$ in Supplement D.4.

• Empirical evaluation: Thank you. We followed your suggestion closely and improved our empirical evaluation as follows. In our updated manuscript, we added new experimental results on higher-dimensional covariates, i.e., the TCGA dataset including high-dimensional gene expressions. Furthermore, we evaluate our method against additional baselines: (i) vanilla CP, without adjusting for the propensity shift, and (ii) a Gaussian process prediction.

  Action: We will add our new experimental results to new Supplement F of our updated manuscript.