Evaluating and Learning Optimal Dynamic Treatment Regimes under Truncation by Death

We appreciate your insightful and constructive review. Our detailed, point-by-point responses addressing your comments on weaknesses and questions are provided below. We appreciate your understanding regarding our inability to include external links within this rebuttal. A list of references has been provided at the end.

Definition of $D$ and $N$

The terms $D = \mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)]$ and $N(\pi) = \mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)m_{\mu_2}^\pi(X_1)]$ are defined as denominator and numerator of Equation 1, respectively. These correspond to $D_K$ and $N_K(\pi)$ when $K=2$ as briefly mentioned in Section 8. We appreciate the reviewers' feedback regarding the omission of a formal introduction for these terms in the main text.

Monotonicity assumption

We recognize that Assumption 3, while standard in principal stratification literature, may not be appropriate in every context. However, its plausibility is high in studies where healthcare providers are precluded from assigning inferior treatments (Sommer and Zeger, 1991; Follmann, 2006). Furthermore, the assumption is automatically satisfied when $S_2^{a_1a_2}=0$ for $a_1+a_2\le1$. It's crucial to note that despite Assumption 3 being imposed, we accommodate scenarios where the outcome, for example, a quality-of-life measure, can worsen following treatment.

Details on the MIMIC-III application

For the selected subpopulation of sepsis patients, we consider an individual's condition to be largely uninfluenced by others. While patients received vasopressin concurrently with mechanical ventilation, 91% of these patients were treated with less than 0.5 mcg/kg/min of vasopressin, and 77% received less than 0.2 mcg/kg/min. The correlations between the maximum vasopressin dose and mechanical ventilation were 0.24 ($k=1$) and 0.14 ($k=2$). Moreover, the correlation between the maximum vasopressin dose and the outcome of interest was −0.069. Based on these observations, we assert that causal consistency is a reasonable assumption.

We confirmed the probabilistic monotonicity of the censoring indicator, demonstrating that $\mathbb{P}(C_2​=1|A_1​=a1​,A_2​=a_2​)\ge \mathbb{P}(C_1​=1|A_1​=a_1​)$. While the low mortality rate prevented empirical verification of survival indicator monotonicity, mechanical ventilation is widely recognized as a critical intervention designed to prolong survival in acute settings, frequently referred to as a "cornerstone of patient management" (Fan et al., 2017). This inherent purpose provides a strong basis for assuming monotonicity of the treatment effect.

MIMIC-III is a standard dataset for reinforcement learning applications, particularly for Markov Decision Process-type problems after suitable preprocessing (Komorowski et al., 2018). Our reliance on Komorowski et al.'s (2018) preprocessing justifies our assumption of sequential randomization. Additionally, we included all available covariates believed to be pertinent to sepsis patient conditions, aiming to control for confounding as thoroughly as possible.

Our analysis confirmed significant overlap in covariate distributions across the different treatment groups, thereby supporting the positivity assumption for the propensity scores. Additionally, employing a flexible classifier, such as a random forest, allowed us to estimate observation and survival probabilities that were consistently near 1. These empirical findings indicate that positivity is upheld, at least probabilistically.

Principal ignorability is a strong yet untestable assumption that has not been justified in previous literature. Thus, it is important to conduct sensitivity analysis to evaluate if the results are sensitive to violation of the assumption. We propose and provide result of a sensitivity analysis in the next section.

Given the unknown true models in this real-world data, evaluating model fit was limited. We chose flexible models (Random Forest and Generalized Additive Model) to reduce the risk of misspecification and underfitting. A comparison between our MR estimator and the principal Q-learning estimator (the plug-in version of Equation 1) on both training and testing sets consistently resulted in close values. This is implied when the outcome regression models are correctly specified, the scenario which guarantees the consistency of the proposed estimator.

Our findings from the MIMIC-III dataset indicate that the derived optimal policy largely favored 00. Treatment assignment was primarily restricted to patients exhibiting either high body weight (exceeding 100 kg) or advanced age (above 55 years). The results obtained using the conventional AIPW estimator were close to our own.

Sensitivity analysis

Under Assumptions 1, 3, 4, and 6-8, we can identify the always-survivor value function as $\mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)m_{\nu_2}^\pi(X_1)]/D$ with ${\nu_2}^{a_1a_2}(X_1) = {\mu_2}^{a_1a_2}(X_1) / \omega_{a_1a_2}(X_1)$ where $\omega_{00}(X_1) = 1$,

\omega_{01}(X_1) = \frac{ m_{p_2}^{00}(X_1) }{ m_{p_2}^{01}(X_1) } + \rho_{0101}^{01}(X_1) \left\\{ 1 - \frac{ m_{p_2}^{10}(X_1) }{ m_{p_2}^{01}(X_1) }\right\\} + \rho_{0111}^{01}(X_1) \frac{m_{p_2}^{10}(X_1) - m_{p_2}^{00}(X_1)}{m_{p_2}^{01}(X_1)},

\omega_{10}(X_1) = \frac{m_{p_2}^{00}(X_1)}{m_{p_2}^{10}(X_1)} + \rho_{011}^{10}(X_1) \left\\{ 1 - \frac{m_{p_2}^{01}(X_1)}{m_{p_2}^{10}(X_1)}\right\\} + \rho_{0111}^{10}(X_1) \frac{m_{p_2}^{01}(X_1) - m_{p_2}^{00}(X_1)}{m_{p_2}^{10}(X_1)},

\omega_{11}(X_1) = \frac{m_{p_2}^{00}(X_1)}{m_{p_2}^{11}(X_1)} + \rho_{0101}^{11}(X_1) \frac{m_{p_2}^{01}(X_1) - m_{p_2}^{10}(X_1)}{m_{p_2}^{11}(X_1)} + \rho_{0011}^{11}(X_1) \frac{m_{p_2}^{10}(X_1) - m_{p_2}^{01}(X_1)}{m_{p_2}^{11}(X_1)} + \rho_{0001}^{11}(X_1) \left\\{1 - \frac{m_{p_2}^{01}(X_1) + \lambda(X_1)}{m_{p_2}^{11}(X_1)} \right\\}.

Here, $\rho_{u}^{\bar a}(X_1) := \frac{\mathbb{E}[Y^{\bar a} \mathbf{1}\\{\pi(\bar X)=\bar a\\}|X_1,U=u]}{\mathbb{E}[Y^{\bar a}\mathbf{1}\\{\pi(\bar X)=\bar a\\}|X_1,U=1111]}$ and $\lambda(X_1) = \mathbb{P}(U=0011|X_1,A_1=1,C_1=0,S_1=1) - \mathbb{P}(U=0101|X_1,A_1=1,C_1=0,S_1=1)$ are sensitivity parameters. The former is sometimes referred to as a tilting model. Principal ignorability is satisfied if $\rho$’s are equal to one and $\lambda(X_1)=0$.

To simplify our analysis, we treated the sensitivity parameters as unknown constants, setting $\rho_{u}^{\bar a}(X_1) = \rho$, $\lambda(X_1) = \lambda$. We varied $\rho$ from 0.8 to 1.25 and $\lambda$ from -0.2 to 0. The negative range for $\lambda$ is motivated by recent evidence indicating that earlier mechanical ventilation may lead to better survival outcomes than later intervention (Kim et al., 2024)

We evaluated the policy using the new identification under varying values of sensitivity parameters. The maximum observed relative error of our estimates, when compared to the previously proposed estimates, was 0.12. This finding suggests that the MR estimator is not sensitive to the violation of principal ignorability.

Adapting the policy in practice

While experimental validation of these specific methods was not conducted, we anticipate that our proposed approach will perform effectively for data with well-represented always-survivor subgroups. This involves categorizing patients based on their estimated always-survival probability and applying policies. The restricted optimal treatment regime (Zhou et al., 2019) was suggested as an alternative, but exploring this method is beyond the current scope of our work and is reserved for future research.

References

• Sommer, A., & Zeger, S. L. (1991). On estimating efficacy from clinical trials. Statistics in medicine, 10(1), 45-52.
• Follmann, D. (2006). Augmented designs to assess immune response in vaccine trials. Biometrics, 62(4), 1161-1169.
• Fan, E., Del Sorbo, L., Goligher, E. C., Hodgson, C. L., Munshi, L., Walkey, A. J., ... & Brochard, L. J. (2017). An official American Thoracic Society/European Society of Intensive Care Medicine/Society of Critical Care Medicine clinical practice guideline: mechanical ventilation in adult patients with acute respiratory distress syndrome. American journal of respiratory and critical care medicine, 195(9), 1253-1263.
• Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018). The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(11), 1716-1720.
• Kim, G., Oh, D. K., Lee, S. Y., Park, M. H., Lim, C. M., & Korean Sepsis Alliance (KSA) investigators. (2024). Impact of the timing of invasive mechanical ventilation in patients with sepsis: a multicenter cohort study. Critical Care, 28(1), 297.

We appreciate your insightful and constructive review. We wish to clarify that the variables $A_k$’s are defined as treatment assignments in lines 72-73, and the bar notation is defined in lines 83-84 of the manuscript. We acknowledge the comment regarding the number of assumptions. We wish to emphasize that the imposed assumptions are standard within both principal stratification and dynamic treatment regimen literature.

Our detailed, point-by-point responses addressing your comments on weaknesses and questions are provided below.

Types of censoring

Our current work focused on a specific type of censoring to simplify the analysis while retaining practical relevance. Although we did not explicitly investigate other forms of censoring, we believe our framework remains applicable. Its core strength lies in handling truncation by death through principal stratification, a mechanism not inherently reliant on the specific type of censorship. Under an alternative censoring mechanism, the primary difference in the estimator would reside in the inverse probability of censoring weighting component, specifically impacting the definition of the nuisance function $\varphi_k()$.

Stability of estimator under varying consistency scenarios

We suppose what the reviewers refer to as sparsity is about the stability of estimators. Should this interpretation be inaccurate, please inform us so that we can elaborate. While not strictly proven, we anticipate the MR estimator will exhibit enhanced stability when the probability models ($p_1, p_2, \varphi_1, \varphi_2$) are correctly specified and their values are bounded sufficiently far from zero.

We appreciate your insightful and constructive review. Our detailed, point-by-point responses addressing your comments on weaknesses and questions are provided below.

Truncation by death

For individuals whose observation is subject to truncation by death, the counterfactual outcomes are not merely missing but are, in fact, ill-defined. This fundamental distinction separates such cases from conventional missing data scenarios, as assuming an outcome value is generally inappropriate when it is inherently tied to a mortality event. Furthermore, truncation by death leads to non-comparable treatment groups; individuals who survived under one treatment might have died under a different, counterfactual treatment, thereby rendering the population-level value function ill-defined. This distinction necessitates the development of methods specifically for truncation by death.

Principal stratification in $K\ge3$ cases

With truncation by death, the value function of the outcome collected at the end of the study is undefined. To address this, we should either restrict analysis to the always-survivor stratum, which is the only subgroup for which the final counterfactual outcome is well-defined. Our method still utilizes all available observed data to estimate the always-survivor value. Therefore, despite concentrating on the local average counterfactual outcome, no information is sacrificed.

Provided that Assumptions 3 through 6 are satisfied, principal stratification is a suitable framework for identifying the always-survivor value function. For any $K$, the condition analogous to Assumption 3 is often a natural fit for this problem setting. Similarly, the condition equivalent to Assumption 6 is standard in all multiple-decision DTRs and is inherently satisfied in randomized trials. Assumptions 4 and 5 represent other standard identification assumptions, though their validity necessitates a data-by-data verification.

Comparison to the doubly robust (AIPW) estimator

The proposed estimator fundamentally differs from conventional AIPW, primarily due to the inclusion of the principal score term. Existing doubly robust DTRs cannot handle truncation by death and will often be biased in such scenarios. While our estimator shares structural similarities with doubly robust DTRs—a consequence of their shared semiparametric theoretical foundation—we consider this an advantage, not a drawback. Indeed, it is a strength of our MR estimator that it results in a generalized form of the conventional AIPW estimator, offering broader applicability while addressing the challenge.

Given the fundamental distinction in how truncation by death is handled, a direct comparison between the multiple robustness of our proposed estimator and the double robustness of the conventional AIPW estimator is unjustified, as AIPW cannot account for such truncated outcomes.

Nevertheless, if death is (mis)treated as a censoring event, consistent with the AIPW approach, and we set $p_1(\cdot)=p_2(\cdot)=m_{p_2}(\cdot)= \hat p_1(\cdot)= \hat p_2(\cdot)= \hat m_{p_2}(\cdot)=1$, our estimator aligns with AIPW ((3) from Section 6) and is consistent whenever AIPW is. Therefore, under an interpretation where death is simplified as a censoring event, our estimator indeed provides double robustness. Crucially, our proposed estimator extends this to multiple robustness in more general settings where outcomes truncated by death are rigorously considered undefined.

The observed improvement of the MR estimator over the AIPW estimator in our simulation studies is anticipated, given that the data were specifically simulated to include observations subject to truncation by death. We employed a version of the conventional AIPW estimator for comparison, as it represents a standard approach for death-truncated data in existing literature, and to our knowledge, no other DTR model is currently capable of handling truncation by death.

IPW and outcome regression variant of the proposed estimator

The same simulation was also run with the IPW and Outcome Regression (OR) versions of our proposed estimator. The IPW variant, in addition to survival probability models, includes only inverse probability of censoring weighting (IPCW) components, while the OR variant contains outcome models.

All three estimators (MR, IPW, OR) yielded similar performance. The standard error for the MR estimator (0.029 for N=2000, 0.017 for N=5000) was positioned between the OR estimator (0.022 for N=2000, 0.014 for N=5000) and the IPW estimator (0.032 for N=2000, 0.021 for N=5000). This behavior is theoretically anticipated; under correct model specification, the OR estimator is known to be the most efficient. Our results confirm that the MR estimator achieves comparable efficiency to the OR estimator.

We appreciate your insightful and constructive review. Our detailed, point-by-point responses addressing your comments on weaknesses and questions are provided below.

Illustrative example

Our framework allows the outcome of interest $Y$ to be a measure of quality-of-life (QOL), functional status, or medical expenses. This often-overlooked aspect of survival-related DTR literature is critical, as patient-centered research increasingly prioritizes QOL and functional status as primary outcomes. Beyond clinical efficacy, the integration of expected medical expenses is crucial for cost-effectiveness analyses and health technology assessments.

Additionally, we provide additional interpretations in the context of MIMIC-III for some assumptions:

• Causal consistency holds if a patient's SOFA score is not affected by other patients’ scores, and no hidden variations or side effects are influencing the score.
• Monotonicity holds if mechanical ventilation benefits survival.
• Principal ignorability holds if the always-survivor characteristics match observed survivor characteristics, but not necessarily the opposite.

Comparison of the MR and AIPW Estimators

In our simulation experiments and the MIMIC-III application, the gap in off-policy evaluation did not appear to be directly influenced by the mortality rate. Instead, we speculate this gap widens when the covariate or outcome distributions of the always-survivor stratum significantly diverge from those of the observed survivors. While a low death rate likely increases the overlap between the always-survivor stratum and observed survivors, our proposed estimator can still substantially reduce bias even with low mortality if these two strata exhibit considerable differences.

Details about estimators in $K\ge3$ cases

In the main text, we introduced the identification of the always-survivor value function, which directly leads to a plug-in estimator. The EIF is then derived as follows, with additional notations introduced: $\\{ \phi_{N_K(\pi)}(O) - V_{\text{AS},K}(\pi) \phi_{D_K}(O) \\} / D_K(O)$ where

We appreciate your insightful and constructive review. Our detailed, point-by-point responses addressing your questions are provided below.

Definition of $D$ and $N$

The terms $D = \mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)]$ and $N(\pi) = \mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)m_{\mu_2}^\pi(X_1)]$ are defined as denominator and numerator of Equation 1, respectively. These correspond to $D_K$ and $N_K$ when $K=2$ as briefly mentioned in Section 8. We appreciate the reviewers' feedback regarding the omission of a formal introduction for these terms in the main text.

Details about estimators in $K\ge3$ cases

In the main text, we introduced the identification of the always-survivor value function, which directly leads to a plug-in estimator. The EIF is then derived as follows, with additional notations introduced: $\\{ \phi_{N_K(\pi)}(O) - V_{\text{AS},K}(\pi) \phi_{D_K}(O) \\} / D_K(O)$ where

Thank you for the rebuttal! You clarified all my questions, and I keep my initial positive score for the paper.