Estimating Heterogeneous Treatment Effect with Delayed Response

• The method is fairly straightforward (and, to a large extent, similar to existing works).
• The motivation of this work is unclear: Medical studies typically consider a sufficiently large time window for the treatment response to be observed, such that in all relevant datasets the eventual outcome is known. Hence, there is no need for a method that considers delayed response time, as this is simply a data preprocessing issue.
• Therefore, it would be a much fairer comparison to train the baselines on the eventual outcomes. I doubt that CFR-DF outperforms baselines that are trained on correctly formatted data.
• This work is, as pointed out in the RW section, related to censored HTE estimation under time-to-event data. The authors claim that the difference of their work is that they consider observable outcomes $Y^t \in \{0,1\}$ that eventually have a positive outcome ("conversion in uplift modeling"), while the cited literature focuses on negative outcomes (such as death of a patient). However, I wonder how this makes any difference from a technical perspective, as this is purely interpretation of the user. Methodologically, this should not make any difference (e.g., whether $Y^t=1$ denotes death of a patient or a positive treatment outcome).
• The assumptions are fairly strong. Especially, Monotonicity and Principal Ignorability are quite restrictive for identifiability. Further, time independence seems unrealistic to me. Here, the authors claim that observation times may not affect potential response times, which seems plausible. However, the effect could be the other way around. In the dynamic treatment literature, it is a known issue that observation time are, in reality, intertwined with the patient's health condition (known as "informative sampling").
• There are methods available for long term outcome targeting where short term proxies are available (which is, admittedly, a slightly different setting). IMHO it would be fair to acknowledge this literature stream. I believe this scenario is more realistic in practice.
• A few minor concerns: a. The authors use an exponential distribution to model the response time in their method, such that the integral over the response time (the one w.r.t. $u$) reduces to a simple exponential function. If a different model for the response time is used (e.g., the suggested non-parametric), I wonder how the integral can be solved and the algorithm trained with reasonable computational complexity. b. In the text before equation (2), I think it should be "$T=t$" and not "$T=t>d$". If we assume that $t>d$, then $P(\tilde{Y}^t=0 \mid X=x, W=w, Y=1) = 0)$. I don't think this is what the authors intended. c. What is the function class in the IPM? d. Since the authors simulate the outcomes and response times on the AIDDS, JOBS and TWINS datasets, I would suggest to call these "semi-synthetic" instead of "real-word" datasets. e. A few of the cited arXiv papers have already been published ((Nagpal et al., 2022), (Nie et al., 2021), (Alaa et al., 2017)).

Novelty: The proposed model of delayed responses is well known in the literature under the name Mixture Cure Model, and it belongs to the broader class of Cure Models (see [1] for an overview). Much of sections 3 and 4 are already well established:

• Equation (4) is the definition of a mixture cure model (see equation 1 in [1]).

• Similar EM algorithms are commonly used for these models, see for example [2] and [3].

• Identifiability results for similar cure models are well established, see [4] and [5].

The primary novelty appears to be the application of the CFR to cure models. This is potentially a valuable contribution but is not how the paper is currently framed. Furthermore, a significant amount of related work is omitted (see, for example, many of the references cited in [1])

Experiments: The data generating process for the synthetic data closely matches the DGP for CFR-DF but none of the baseline comparison models, so superior performance is expected by construction. While useful as a confirmatory study for your implementation, the comparison to these models is misleading. Instead, comparison to models that assume delayed feedback, such as [6], [7], or a standard Cox PH or AFT linear cure model, would be more convincing of the efficacy of the proposed model.

References:

[1] Mailis Amico, Ingrid Van Keilegom. Cure Models in Survival Analysis. Annual Review of Statistics and Its Application. 2018.

[2] Yingwei Peng, Keith Dear. A nonparametric mixture model for cure rate estimation. Biometrics. 2000.

[3] Judy Sy, Jeremy Taylor. Estimation in a Cox Proportional Hazards Cure Model. Biometrics. 2000.

[4] Chin-Shang Li, Jeremy Taylor, Judy Sy. Identifiability of cure models. Statistics & Probability Letters. 2001.

[5] Leonid Hanin, Li-Shan Huang. Identifiability of cure models revisited. Journal of Multivariate Analysis. 2014.

[6] Matthew Engelhard, Ricardo Henao. Disentangling Whether from When in a Neural Mixture Cure Model for Failure Time Data. AISTATS. 2022.

[7] Rongqian Sun, Xinyuan Song. A Tree-based Bayesian Accelerated Failure Time Cure Model for Estimating Heterogeneous Treatment Effect. Bayesian Analysis. 2023

• $**The proposed model is almost the same as the cure model in survival analysis. The author should clearly explain what are new contributions.**$

• In the real-world data analysis, the paper relies only on covariates from that dataset, which might not yield meaningful results. It would be more appropriate to generate data only for the variables that contain missing values (e.g., the missing part of potential response times, potential observed outcomes, potential eventual outcomes).

• Compared to existing HTE estimation methods, obtaining suitable data for the application of this method is expected to be challenging. It is because data need to satisfy following conditions. (1) Data should include the time treatment is given (2) The outcome at that point should be zero (3) Data should include the time the status changed.

• While the EM algorithm equations appear correct, it would be helpful to include a detailed derivation of the expected log-likelihood in the appendix.

• While $\tilde{y}_i^t$ may represent the observed outcome, $\tilde{Y}_i^t$ may represent $\tilde{Y}_i^t | (T=t)$ which could cause confusion.

• Expression $P(D(0)=d|X=x, Y(0)=1)$ should be modified to have the pdf form.

• An explanation of why PEHE and $\epsilon_{ATE}$ were chosen as performance measures in the experiments or references to related literature is recommended.

• The description of notation and setup should be refined. For example, (1) outcome should be zero when the treatment is given for all unit. (2) The covariate does not change over time.

• The paper's problem setting appears to differ significantly from the typical causal inference scenario. Therefore, using status instead of outcome may be more appropriate.

• Given the differences in the setup, an description of how estimation was performed for other methods may be needed.

• Since the assumptions seem stronger compared to existing causal inference method, it would be nice to discuss how to check these assumptions.

• The empirical evaluations include only relatively small datasets. Although they are useful for benchmarking, it would be better to see how the model scales to larger datasets with more variables.
• The monotonicity assumption may hold for some applications, but not always. It would be great to see some analysis of the identifiability when this assumption does not hold.
• Figure 3 in the supplemental materials provides important insights into the proposed architecture and it would make the presentation much clearer to move it to the main text.