Estimation of Treatment Effects in Extreme and Unobserved Data

Thanks for the valuable feedback! Please see below for our detailed responses. We hope our responses can resolve your concern.

1. Assumption 3.1 appears somewhat restrictive. Can U be included in the covariate vector X to control for its confounding effects?

Thanks for the insightful question! The short answer is that we can include U in the covariate vector, but for estimation, we need to adapt our estimators. Following the same argument as Proposition 3.3, one can derive a similar identification formula. However, when we include U in the covariates, the domain of propensity is not compact. It is difficult to estimate the propensity function defined in the whole real space. If we don’t have access to the propensity function, which sometimes can be obtained from policy documents, we suggest using the DR estimator or the regression adjustment estimator for estimation in this setting.

2. My main concern lies in the explanation of the NETE. Provide a more detailed explanation, ideally supported by visual illustrations.

Thanks for the valuable suggestion. We now include a more detailed, intuitive discussion of the NETE in a simple setting:

Data‐Generating Process: Consider no covariates and let $U$ be a univariate Pareto with extreme value index $1/\beta$. We generate potential outcomes as $Y(d) = \bigl(d\theta_{1} + (1-d)\theta_{0}\bigr)\,U^{2}+ 0.1\,\sqrt{U}+ \varepsilon,\quad\varepsilon \sim N(0,\,0.1),$ where $0 < \theta_{0} < \theta_{1}$. In this case, $\alpha = 2$ so that

$$E\bigl[Y(d)/t^{2}\mid U > t\bigr]\sim c_{d} t^{2}$$

for constants $c_{d}$. By definition, the NETE is

$$\mathrm{NETE} = c_{1} - c_{0}= (\theta_{1} - \theta_{0}) \cdot \frac{\beta}{\beta - 1}.$$

Intuitively, NETE measures how much faster the treated outcome grows in the extreme tail compared to the control.

Interpretation :

• A large positive NETE occurs when:
  1. $\theta_{1} - \theta_{0}$ is large (treatment increases the leading coefficient), or
  2. $\beta$ is close to 1 (Pareto tail is very heavy).
• These two factors correspond to:
  1. Faster tail‐growth under treatment, and
  2. Longer tail of $U$.

Planned Illustration :We will add a figure plotting 10,000 simulated points of $\bigl(Y(1),U\bigr)$ and $\bigl(Y(0),U\bigr)$ with $\beta = \tfrac{10}{11}$, $\theta_{0} = 0.5$, and $\theta_{1} = 1$. The scatter will fan out around the curves $Y(d)=\theta_{d}\,U^{2}$. A steeper “fan” visually conveys a larger NETE, illustrating how the joint behavior of $(Y(d),U)$ encodes tail sensitivity.

We were unable to upload the figure here, but will include it in the revised manuscript.

3. Furthermore, the explanation of the results for the semi-synthetic dataset is limited. In most parts of the semi-synthetic application, the focus is primarily on the statistical properties of the estimators. However, more detailed interpretation of the results is needed to enhance the reader’s understanding of their practical implications.

Thanks for the suggestion. We will add result analysis in later versions. We can see from equation (4.1) that the treatment $D$ will only change the coefficient of $W^{\alpha_1}S^{\alpha_2}$, which is proportional to the NETE. As mentioned in the previous responses, a positive NETE means the treatment will make the tailed expected outcome grow faster with respect to the threshold t.

4. From a statistical perspective, extreme value theory is well-established and well-motivated. However, researchers in the machine learning or causal inference community may be less familiar with this framework. Therefore, a more accessible explanation and stronger motivation would help broaden the impact and understanding of the work. In particular, the meaning of \alpha may be important and warrants further clarification.

Thanks for the advice. We will add an extra section in the appendix to briefly introduce the extreme value theory. Besides, we will also add a paragraph in the introduction part to motivate EVT.

Thanks for your valuable review of our paper! We are sure your feedback will make our paper better! Please see the following paragraphs for our responses.

1. The appendix appears to be missing, which makes the review more challenging.

Thanks for asking. The appendix is in the supplementary material.

2. It would be useful to give more clarity on the relationships between U and Y, especially. e.g., do you think of Y as having a structural component and U noise? The synthetic dataset in Sec 4.1 is a useful step.

Thanks for asking! We agree that the name “extreme noise” can be confusing. We will make the discussion clearer in later versions. As in the example of line 124, we think U is also a structural component of the model, which represents extreme events. A more appropriate name for U may be “extreme factors”.

3. For the semi-synthetic dataset, does it correspond to anything concrete? Is there any notion of uncertainty that can help calibrate whether these point estimates are “large” or “small”?

Thanks for the question. In the semi-synthetic dataset, we use real-world data as the extreme noise U and generate the rest of the variables by equations (4.1). (4.1) does not correspond to a real-world example. We use this example to conceptually verify our theoretical results. For uncertainty quantification, please see our response for the next question.

4. Recognizing that this is challenging, there is no discussion of uncertainty quantification, which would be particularly important in a setting like this.

We thank the reviewer for highlighting the absence of uncertainty quantification. We agree that assessing the variability of our extreme‐treatment estimates is both important and nontrivial. A natural approach to estimating the sampling variability of our estimators is the bootstrap [1]. However, straightforward resampling of extreme observations can fail to reproduce tail dependence and may severely underestimate variance [2]. Recent work on adapting bootstrap methods to heavy‐tailed and extreme‐value settings [3] offers promising strategies—for instance, resampling only the exceedances above a high threshold or applying a parametric tail bootstrap based on the generalized Pareto approximation. While these approaches suggest viable adaptations, a full theoretical justification in our context would require substantial new analysis. We therefore view rigorous confidence‐interval construction for extreme‐treatment effects as a direction for future work and refrain from making formal claims about interval validity in the current manuscript.

[1] Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26.

[2] Cornea‐Madeira & Davidson (2015). “A Parametric Bootstrap for Heavy-Tailed Distributions.” Econometric Theory 31(3): 449–470.

[3] de Haan, Laurens, and Chen Zhou. "Bootstrapping extreme value estimators." Journal of the American Statistical Association 119.545 (2024): 382-393.

Thank you for your detailed evaluation of our paper, which will definitely make our paper clearer and better! Please see the following responses.

1. Would it be possible to provide deeper insight into what the NETE represents and what type of causal relationship it captures?

Please see the second bullet point of our response to the Reviewer iYgk for a detailed answer.

2. Similarly, could a synthetic experiment be constructed where the NETE identifies a causal relationship that would not be detected by a classical causal inference method (again, assuming the mean exists)?

In the example in the previous answer, the NETE captures how fast the expected outcome in the tail grows with respect to U. ATE can only capture the change in the overall expectation. While in theory one can estimate CATE and use CATE to capture similar properties, insufficient tailed samples will create a significant challenge for estimation.

3. Where is the proof for Proposition 3.3? Not sure why this holds.

Thanks for asking. The proof can be found in the appendix, which is in the supplementary material.

Clarify questions:

1. Line 13+: The term “extremities” is somewhat awkward, as it can imply an upper bound. Consider using “extremes” or “extreme events” instead.

Thanks for the suggestion. We will change the wording.

2. Line 84: The term “consistency” is unclear in this context. Please clarify whether it refers to the consistency of an estimator, or something else.

Thanks for pointing this out. ‘Consistency’ here refers to the consistency assumption in casual inference, which states that an individual's observed outcome is equivalent to their potential outcome under the treatment they actually received.

3. Lines 94 and 104: Is this the same constant c? If so, it should be explicitly stated; if not, the notation should be differentiated.

Thanks for pointing out the notation abuse. They are different constants. We will explicitly state this in the paper.

4. Lines 91 and 115: The mean is used before it is formally defined. Consider introducing the necessary definitions earlier for clarity. The orthogonality symbol is used without explanation. Please define this notation when it first appears.

Thanks for pointing these out. We will move the definition of expectation earlier and add the definition of the orthogonality symbol. The orthogonality symbol represents an independent relationship.

5. Line 94: The term “estimated propensity” is vague—does it refer to an estimated propensity score? If so, please make this explicit.

Thanks for asking. It refers to an estimated propensity score. We will make this explicit.

Thanks for the detailed feedback and valuable suggestions! Please see the following responses.

1. It would be helpful to state clearly in the abstract that the authors appear to propose a new definition of extreme treatment effects.

Thanks for the advice. We will add a sentence in our abstract to make this clearer.

2. The paper gives the impression that no prior work has modeled quantities beyond the population average (ATE). A discussion of these differences would strengthen the literature review.

Thanks for providing the related reference. We acknowledge that the expression of the sentence may be inappropriate. We will provide a more thorough literature review in later versions. Currently, there are two approaches in the literature to characterize the extreme treatment effect (ETE). The first approach is to use extreme Quantile Treatment Effect (QTE) [3, 4, 5, 6], which we discuss in detail in our paper. The other approach is to assume the outcome falls in the domain of attraction and use the change of Extreme Value Index (EVI) as a measure for ETE [1, 2]. In particular, [3] also considers using the ratio of tail expectation under different treatments as a measure. However, our setting is different from these works in that (1) We allow extreme exogenous variables and the outcome is a function of treatment, covariate and the extreme variables. (2) We model extreme events by a multivariate regularly varying variable, while all these works focus on univariate settings. Moreover, our work can also be placed in the broader context in the literature that studies causal effects of treatment beyond the mean, for example, distributional effect [7,8,9], Ratio‐ and Log‐Ratio‐Based Effects [10,11], CATE [12,13,14].

[1] Aloui, Ahmed, et al. "Treatment Effects in Extreme Regimes." arXiv preprint arXiv:2306.11697 (2023).

[2] Bodik, Juraj. "Extreme treatment effect: Extrapolating dose-response function into extreme treatment domain." Mathematics 12.10 (2024): 1556.

[3] Huang, Wei, Shuo Li, and Liuhua Peng. "Extreme continuous treatment effects: Measures, estimation and inference." arXiv preprint arXiv:2209.00246 (2022).

[4] Victor Chernozhukov and Songzi Du. Extremal Quantiles and Value-at-Risk, May 2006.

[5] Yichong Zhang. Extremal quantile treatment effects. The Annals of Statistics, 46(6B):3707–3740,December 2018.

[6] David Deuber, Jinzhou Li, Sebastian Engelke, and Marloes H. Maathuis. Estimation and Inference of Extremal Quantile Treatment Effects for Heavy-Tailed Distributions. Journal of the American Statistical Association, 119(547):2206–2216, July 2024.

[7] Abadie, A. (2002). Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models. J. Amer. Statist. Assoc., 97(457): 284–292.

[8] Gautier, Eric, and S. T. E. F. A. N. Hoderlein. "Estimating the distribution of treatment effects." arXiv preprint arXiv:1109.0362 (2013).

[9] Hohberg, Maike, Peter Pütz, and Thomas Kneib. "Treatment effects beyond the mean using distributional regression: Methods and guidance." PloS one 15.2 (2020): e0226514.

[10] Cole, S. & Hernán, M. (2002). Fallibility in Estimating Direct Effects. Epidemiology, 13(3): 297–306.

[11] VanderWeele, T.J. (2014). Policy‐Relevant Proportions for Direct Effects. Epidemiology, 24(2): 175–176.

[12] Shalit, U., Johansson, F. & Sontag, D. (2017). Estimating individual treatment effect: generalization bounds and algorithms. International Conference on Machine Learning (ICML).

[13] Wager, S. & Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523), 1228–1242.

[14] Abrevaya, Jason, Yu-Chin Hsu, and Robert P. Lieli. "Estimating conditional average treatment effects." Journal of Business & Economic Statistics 33.4 (2015): 485-505.

3. Do the authors see a natural extension of their work to this conditional setting?

Thanks for the great question. Indeed, our results can be extended to the conditional setting using a similar argument without much difficulty. The identification formula (Proposition 3.3) can be adapted to $\theta^{{CNETE}}(x) = \lim_{t \rightarrow \infty} \mathbb{E}[g (X, 1, U/\|U\|) - g (X, 0, U /\|U\|) \mid \|U\|> t, X = x] \cdot \lim_{t \rightarrow \infty} \mathbb{E}[\|U\|^{\alpha} / t^{\alpha} \mid \|U\|> t] .$ Similar to CATE estimation, one can design different estimators based on this identification formula, e.g., T learner, S learner, X learner, DR learner.

4. Clarification of Assumption 3.2.

Thanks for the question and sorry for the confusion. The correct statement should be that there exists some constant $\alpha$, functions $g(x,d,u), e(t)$ such that the inequality holds for all x, t, u, d. It is easy to see that $\lim_{t\rightarrow \infty} f(x,d,tu)/t^\alpha = g(x,d,u)$. Therefore, the constant $\alpha$ is unique. Otherwise, if there exists $\alpha’$ that satisfies the same relationship, we have $\lim_{t\rightarrow \infty} t^{|\alpha - \alpha’|} <\infty$. Since $\alpha$ is unique, by this relationship, $g$ is unique too. The error term $e(t)$ is not a function of $g$ and only depends on t. This assumption can be viewed as an asymptotic version of homogeneous functions.

5. Is $\alpha$ fixed for each $Y$? Please clarify.

Thanks for the question. Yes. $\alpha$ is fixed for each $Y(d)$. Furthermore, for different $d$, the alpha is the same.

6. Many functions $e(t)$ could verify the required property, even if they are nonzero but converge to zero as $t\rightarrow \infty$. Is my understanding correct?

Yes! Your understanding is correct. Since $e(t)$ is only an upper bound, it can be satisfied by many functions as long as they converge to 0 as t grows to infinity.

7. The claim that the definition “generalizes ATE to the setting of extreme events and aligns with the growth rate given by EVT” is somewhat misleading. It is not obvious how this is a generalization of ATE. Rather, it appears to be a separate definition. It may be more accurate to say it generalizes the unnormalized version (ETE), if anything. Or am I missing something?

Thanks for the suggestion! We intended to say that NETE provides a causal measure in the extreme setting just like ATE in the regular setting. Since this expression may cause confusion, we will change it in the later version.

8. Could the authors add error bars in Figure 2? The experiments are repeated over 50 different seeds, meaning the data changes each time, and reporting variance would be helpful.

Thanks for pointing that out. We will include error bars in our figures in later versions. Unfortunately, we did not find a place to show our figures here.

9. The authors mention that as the number of samples increases, the assumption becomes violated and the estimator becomes dominated by error.

Thanks for the question. We want to clarify that in the second experiment (Fig 2 in the paper), we design the data generation process such that assumption 3.4 (Pareto mixture) is violated all the time. Since our non-asymptotic analysis is based on this assumption, violation of this assumption may cause the error bound in Theorem 3.5 to be invalid. We will make this point clearer in our experiments section. There is a way to detect whether assumption 3.4 is violated. Please see the third point of our response to Reviewer GEJF for explanation.

10. What do authors expect regarding the performance of their estimator when conditional unconfoundedness is violated?

Thanks for the questions. When the conditional unconfoundness assumption is violated, our identification formula no longer holds, and thus our estimator is non-consistent. There have been many works in the literature discussing this assumption. One approach is to completely drop this assumption and consider using partial identification methods to obtain a bound for the estimand [1,2]. The other approach is to conduct a sensitivity analysis [3]. Following the reviewer’s advice, we examine what would happen empirically if this assumption is violated. Our data generation process is

$

X \sim \mathrm{Uniform}([0,1]^{10}),\ p_i = \frac{1}{1 + \exp\bigl(-\beta^\top (X \circ X)\bigr)}, \quad D_i \sim \mathrm{Bernoulli}(p_i), \quad

$

and $Y = \|U\|^2(D + \gamma^T(X \circ X) + U/\|U\| + \epsilon) + \|U\|,$ where $U,\epsilon$ are generated the same way as in the first synthetic experiment in the paper and $X \circ X$ is the element-wise multiplication. The following table shows that the MSE decreases as the observable dimension increases.

Thank you for the valuable feedback! Please see the following detailed response to your questions. We hope our responses can resolve your concern.

1. Observability of Extreme Noise U. How realistic is it to assume ∥U∥ is observed so that {∥U∥ > t} can be conditioned on? Could the method be adapted when only Y (or a proxy) identifies extremes?

Thanks for the great question. We can think of U as some factors that model the extreme events, e.g, speed and radius of a hurricane, which are often observable. In many cases, we are interested in the influence of extreme events rather than the events themselves. For example, we care about how much economic loss would be caused by the extreme events. That’s why in our model, the outcome is a function of U, X (covariate) and D (treatment). We will make this point clear in our paper. When only Y identifies the extremes, the identification formula (Proposition 3.3) no longer holds and we need to develop new identification techniques and change our framework. Indeed, in the literature, [1,2,3] consider such settings and develop estimation methods for their proposed estimands.

[1] Aloui, Ahmed, et al. "Treatment Effects in Extreme Regimes." arXiv preprint arXiv:2306.11697 (2023).

[2] Bodik, Juraj. "Extreme treatment effect: Extrapolating dose-response function into extreme treatment domain." Mathematics 12.10 (2024): 1556.

[3] Huang, Wei, Shuo Li, and Liuhua Peng. "Extreme continuous treatment effects: Measures, estimation and inference." arXiv preprint arXiv:2209.00246 (2022).

2. Independence Assumption. Please discuss robustness/sensitivity when this fails.

Thanks for the valuable feedback. We acknowledge that Assumption 2.1 may not always hold in practice, as policies are often influenced by the extreme event U. Regarding the situations where the independence assumption is violated, there are two cases. First, we have prior knowledge about which extreme variables influence the treatment assignment. In this case, we suggest including the extreme variables as covariates. Please see our response to the first question to Reviewer iYgk for detailed discussion. Second, we do not know which extreme variables will influence the treatment, but we treat U to be independent of treatment. In this case, the IPW estimator will be biased since the propensity estimation does not include (part of) U as inputs and the estimated propensity function will not converge to the ground-truth propensity. However, if all other assumptions in Theorem 3.5 still hold, the DR estimator is still consistent thanks to the double robust property.

3. Tail‑Model Specification. Can the authors provide a diagnostic or adaptive procedure to detect mis‑specification in practice?

Thanks for the important question. Unfortunately, in theory, we don’t have a good way to test whether Assumption 3.4 holds or not. In practice, we suggest plotting the spectral distribution $U/ \|U\| \mid \|U\| > t$ for a large threshold t. If Assumption 3.4 holds, by Lemma 2.1 in [1], we should be able to see points concentrate on a few clusters. If points spread evenly on the sphere, Assumption 3.4 may be violated.

[1] Zhang, Xuhui, et al. "Wasserstein-based minimax estimation of dependence in multivariate regularly varying extremes." arXiv preprint arXiv:2312.09862 (2023).

4. Scaling‑Exponent α. α is estimated by a log‑log regression; no theory is provided for its error propagation. Empirically, how sensitive is NETE to misestimated α?

This is a great question! We did an experiment to see the sensitivity of alpha. In particular, we compare the MSE of using the true $\alpha$ versus using our heuristic to estimate the $\alpha$ in the first synthetic experiment in our paper. Unfortunately, we did not find a place to show our figures here. The following table shows part of the results. We can see that actually the MSEs are closed and our heuristic for $\alpha$ estimation works well.

5. Compute & Reproducibility. Please provide (i) code, (ii) random seeds, (iii) hardware spec, and (optionally) (iv) wall‑clock times so reviewers can replicate Tables 1–2.

Thanks for asking. We will provide the code in the camera-ready version. Unfortunately, due to Neurips' policy, we cannot provide any link in the rebuttal stage.

6. Real‑world Case Study. A genuine observational dataset—e.g., hurricane damage with/without building codes—would greatly strengthen practical relevance.

Thanks for the suggestion. We want to mention that it is difficult to find realistic datasets that include covariates, extreme events and outcomes. To conduct such a real-world study, we need to collect data from several different data sources and construct a new dataset, which requires a substantial amount of time and effort. Given that the goal of this paper is to propose a new framework for estimating extreme effects, we think that synthetic and semi-synthetic experiments are enough to verify our theoretical results. We will work on real-world problems in later studies.

7. Why are only averages shown in Fig 2 although the experiment was repeated 50 times. Please add CI.

Thanks for pointing that out. We will add the error bar to all graphs. Unfortunately, we did not find a place to show our figures here.

8. A paragraph on Limitations is missing.

Thanks for the feedback. We briefly mention the limitation in the conclusion section in the appendix. We will add a more detailed explanation of the limitation.