Path-specific effects for pulse-oximetry guided decisions in critical care

Thank you for your detailed review and valuable suggestions. We address each of your comments below.

Weaknesses

1. We will address weaknesses 1 and 2 together.

   In Table 1, [27], [44], and [46] investigate path-specific effects, but only [27] focus on effects isolated through a single mediator (as shown in the fourth column). [27] is conceptually similar to our non-self-normalized estimator which results in high variance (see Figures 3a and 3c), especially in small sample regimes like our application. [44] and [46] assume a single mediator, which could be applicable in other ICU settings, but is limiting in the pulse oximetry setting because many variables beyond the supplemental oxygenation discrepancy should be considered for valid fairness estimation. Nonetheless, we compare with this "misspecified" setting (i.e. one mediator), called the NIE$^*$, in Figure 5. We will add a discussion to address the research gap in the revisions.

4. As with most causal inference methods, our framework relies on a specified causal graph and thus makes assumptions about the data-generating process that can be challenging to verify in practice. To support these assumptions, we will expand the appendix to include a detailed discussion of their plausibility in the context of our ICU application. We will also emphasize that, although the validity of our methodology does indeed depend on the specified causal diagrams, using DAGs is advantageous because they clearly and explicitly represent the assumptions being made, and its importance has been highlighted in critical care literature as well [1]. These assumptions can be more easily verified or corrected compared to those in more opaque methods, making causal models relevant in medical (and other) domains.

   We agree that the variable $X$, representing race, should not be modeled as receiving a directed arrow from $Z$ alone. The more appropriate structure includes a hidden confounder between $X$ and $Z$, as demographic variables such as age may influence the distribution of racial groups. Importantly, introducing a bidirected arrow between $X$ and $Z$ does not affect our estimators or the theoretical analysis presented in Appendix A. We will revise the causal graph accordingly in the updated manuscript.

5. We agree with the reviewer and have included comparisons with three additional baseline methods in both the synthetic and semi-synthetic experiments. To the best of our knowledge, no existing work directly estimates the VDE beyond the canonical doubly robust estimator, which is challenging to interpret due to high variance. Therefore, we evaluate several state-of-the-art methods for similar effect estimation tasks. For details on the experimental setup and evaluation, please see our response 3 to Reviewer ScmX.

Questions

1. The ICU is a particularly complex clinical setting because of its longitudinal nature and the high dimensionality of patient data. These characteristics make it difficult to apply standard algorithmic fairness frameworks directly. This motivates us to incorporate domain knowledge and a clinically grounded data-generating process to more appropriately assess fairness in this setting.

We hope our response resolves your concerns and welcome any further questions you may have.

[1] - David J. Lederer, Scott C. Bell, Richard D. Branson, et al. Control of confounding and reporting of results in causal inference studies. Guidance for authors from editors of respiratory, sleep, and critical care journals. Annals of the American Thoracic Society, 16, 2019.

Hello reviewer vpLv - thank you again for taking the time to review this paper. A mandatory acknowledgement should only be done after you have had some discussion. Please comment on the paper to initiate the discussion. -Area Chair

Thank you for your thoughtful and constructive questions. We address all your concerns in the following response.

Weaknesses

1. In Table 1, [27], [44], and [46] investigate path-specific effects, but only [27] focus on effects isolated through a single mediator (as shown in the fourth column). [27] is conceptually similar to our non-self-normalized estimator which results in high variance (see Figures 3a and 3c), especially in small sample regimes like our application. [44] and [46] allow only a single mediator, which could be applicable in other ICU settings, but is limited in the pulse oximetry setting because all relevant patient variables beyond the supplemental oxygenation discrepancy should be considered for valid and unbiased fairness estimation. Nonetheless, we compare with this "misspecified" setting (i.e. assume only one mediator), called the NIE$^*$, in Figure 5. We will add a discussion to address the research gap in the revisions.

2. In Appendix A, we explicitly outline the identifiability conditions, which assume no unmeasured confounding between W and V in order to identify the VDE. This is reasonable in the context of our ICU application, and we will expand the appendix to include a discussion of the plausibility of this assumption based on the available covariates and the clinical context of our ICU application.

3. We thank the reviewer for identifying this point. To address the concern, we compare our proposed estimators, the canonical doubly robust estimator for the VDE and its self-normalized variant, with three additional baselines across synthetic (binary and continuous) and semi-synthetic (eICU and MIMIC-based) datasets. Since, to the best of our knowledge, no prior work directly targets the VDE beyond the canonical estimator, we include the following (somewhat misspecified) baselines:

   • X-learner, which estimates the ATE without accounting for the mediation structure (ignoring $W$ and $V$), and cannot isolate the target effect we want.
   • Single-mediator doubly robust estimator, which computes the effect through the mediators as if $W$ and $V$ were merged into a single variable (this is equivalent to the NIE$^*$ used in our real-world experiments).
   • Nested regression estimator, which corresponds to $\mu_0^1$ in Equation 6.

   We compare all five estimators in the table below, reporting the mean and standard error over 100 bootstraps, where each iteration samples a dataset of size 32,000. Relative errors are shown in parentheses, except for the semi-synthetic eICU setting, where the true VDE is very close to zero and relative error is not informative. Entries in bold denote the best performing model for each dataset.

   	Synthetic binary	Synthetic continuous	Semi-synthetic eICU	Semi-synthetic MIMIC
   Ground truth	$-0.125\phantom{+}$	$-0.163\phantom{+}$	$0.000$	$0.001$
   X-learner	$\phantom{+}0.364\ (292\\\%) \pm 0.001$	$-0.352\ (217\\\%) \pm 0.001$	$-0.007 \pm 0.000$	$-0.029\ (4819\\\%) \pm 0.000$
   Single-mediator doubly robust estimator [44, 46]	$-0.033\ (26\\\%) \pm 0.001$	$-0.312\ (192\\\%) \pm 0.008$	$\phantom{+}0.024 \pm 0.014$	$-0.027\ (4354\\\%) \pm 0.004$
   Nested-regression estimator (Eq. 6, $\mu_0^1(x_1, Z)$ term)	$\mathbf{\phantom{+}0.001\ (0.5\\\%) \pm 0.001}$	$-0.024\ (15\\\%) \pm 0.002$	$\phantom{+}0.004 \pm 0.001$	$\mathbf{-0.001\ (117\\\%) \pm 0.000}$
   Canonical doubly robust estimator ([27], Fig. 3a&c)	$\mathbf{-0.001\ (0.5\\\%) \pm 0.001}$	$\mathbf{\phantom{+}0.006\ (4\\\%) \pm 0.008}$	$\phantom{+}0.018 \pm 0.020$	$\phantom{+}0.025\ (4039\\\%) \pm 0.176$
   Self-normalized doubly robust estimator (Proposed, Fig. 3b&d)	$\mathbf{-0.001\ (0.5\\\%) \pm 0.001}$	$\mathbf{\phantom{+}0.006\ (4\\\%) \pm 0.006}$	$\mathbf{\phantom{+}0.003 \pm 0.007}$	$\phantom{+}0.004\ (622\\\%) \pm 0.004$

   The findings show that the self-normalized estimator typically yields the lowest error and maintains a $95\\%$ confidence interval that includes the true value. While the nested regression estimator performs comparably on semi-synthetic data, its confidence intervals are overly narrow. State-of-the-art methods for similar effect estimation tasks (i.e. the X-learner and single-mediator estimator) exhibit high error across all settings since they do not account for the correct mediation structure.

4. We have roughly 31,000 White and 4,000 Black patients in eICU, and around 4,000 White and 400 Black patients in MIMIC, thus capturing the imbalance you have correctly highlighted. This mainly reflects in the variance of the estimator, which we propose to mitigate using the proposed self-normalized variant. We also approximately preserve the imbalance in our semi-synthetic setup (the ratio of White to Black patients is 2:1). Figures 3c and 3d demonstrate that the self-normalized doubly robust estimator demonstrates good convergence behavior despite the imbalance.

   We also include an experiment demonstrating how varying levels of imbalance affect the performance of our estimator. Using the two semi-synthetic datasets, we threshold the predictions of $X$ at different values to induce varying proportions of $x_0$​ and $x_1$ subgroups. To quantify this imbalance, we define \eta = \frac{\\# \text{ of x_1 samples}}{\\# \text{ of total samples}}. Following the approach in Figure 3, we evaluate performance using the relative error for each causal query. The mean and standard error are reported over 100 bootstrap iterations, each sampling a dataset of size 32,000. Results are presented in the two tables below.

   eICU-based semi-synthetic data	$\quad\quad \eta \approx 0.5$	$\quad\\:\\:\\: \eta \approx 0.33$	$\quad\quad \eta \approx 0.2$	$\quad\quad \eta \approx 0.1$
   $\mathbb{E}[Y_{x_0}]$	$\phantom{+}0.1\\% \pm 0.1\\%$	$-0.1\\% \pm 0.1\\%$	$\phantom{+}0.1\\% \pm 0.1\\%$	$\phantom{+}0.0\\% \pm 0.1\\%$
   $\mathbb{E}[Y_{x_1}]$	$-0.1\\% \pm 0.1\\%$	$\phantom{+}0.0\\% \pm 0.2\\%$	$-0.5\\% \pm 0.2\\%$	$-0.5\\% \pm 0.3\\%$
   $\mathbb{E}[Y_{x_1, M_{x_0}}]$	$-2.8\\% \pm 6.6\\%$	$-2.7\\% \pm 3.5\\%$	$\phantom{+}7.2\\% \pm 6.7\\%$	$-9.1\\% \pm 6.6\\%$
   $\mathbb{E}[Y_{x_1, V_{x_0}, W_{x_1}}]$	$-1.2\\% \pm 1.6\\%$	$\phantom{+}1.2\\% \pm 1.7\\%$	$-0.8\\% \pm 2.8\\%$	$-3.9\\% \pm 2.8\\%$

   MIMIC-based semi-synthetic data	$\quad\quad \eta \approx 0.5$	$\quad\\:\\:\\: \eta \approx 0.33$	$\quad\quad\eta \approx 0.2$	$\quad\quad\eta \approx 0.1$
   $\mathbb{E}[Y_{x_0}]$	$-0.6\\% \pm 0.2\\%$	$-0.6\\% \pm 0.2\\%$	$-0.6\\% \pm 0.2\\%$	$-0.6\\% \pm 0.2\\%$
   $\mathbb{E}[Y_{x_1}]$	$-0.5\\% \pm 0.2\\%$	$\phantom{+}0.3\\% \pm 0.4\\%$	$-0.9\\% \pm 0.4\\%$	$-0.6\\% \pm 0.6\\%$
   $\mathbb{E}[Y_{x_1, M_{x_0}}]$	$\phantom{+}2.0\\% \pm 4.3\\%$	$-3.8\\% \pm 2.2\\%$	$\phantom{+}0.2\\% \pm 3.9\\%$	$-6.7\\% \pm 3.7\\%$
   $\mathbb{E}[Y_{x_1, V_{x_0}, W_{x_1}}]$	$-6.8\\% \pm 2.6\\%$	$\phantom{+}6.0\\% \pm 5.1\\%$	$-5.7\\% \pm 6.6\\%$	$-1.1\\% \pm 7.7\\%$

   As expected, the relative error and variance of the estimator generally increase with greater imbalance between the $x_0$ and $x_1$ subgroups (with the exception of $\mathbb{E}[Y_{x_1, V_{x_0, W_{x_1}}}]$ on MIMIC-based data). Nonetheless, the overall differences in estimator performance across levels of $\eta$ remain small, with relative error increasing by at most a few percentage points.

We hope that our response addresses your concerns. We are happy to clarify any other questions you may have.

Thank you for acknowledging the novelty of the method, and for your constructive comments and review.

Questions

1. Thank you for the interesting question. We focus on the specific research question of path-specific effects of rate and duration of ventilation. We agree that we cannot rule out either of the two possibilities, that either the clinicians successfully integrate other information, reducing the overall effect of pulse oximeter discrepancy, or that other outcomes need to be measured, which is an active area of research. Our study is designed in line with the outcomes chosen in prior studies. We will add these details to our discussion.

Please let us know if you have any additional questions and we are happy to clarify further.

Thank you for your thoughtful review and for raising constructive points, which we address below.

Questions

1. In our analysis, $V$ includes $\text{SpO}_2$, $\text{SaO}_2$, and their discrepancy, which is inherently defined by the joint behavior of $\text{SpO}_2$ and $\text{SaO}_2$. Isolating a path solely through the discrepancy is not as meaningful in our setting because race influences $\text{SpO}_2$ and $\text{SaO}_2$ directly, and the discrepancy arises from their measurements, not from how it is mathematically computed. While the VDE may, as the reviewer noted, include components like $\text{SaO}_2$ that are not inherently unfair, we include the full set of oximetry-related variables in $V$ to more comprehensively capture the mechanism by which racial bias in pulse oximetry mediates clinical decisions. We will clarify this modeling choice explicitly in the revised manuscript.

2. We have previously identified an issue in the hyperparameter search that incorrectly selected the number of estimators for the XGBoost model. After conducting a proper hyperparameter search across 64 settings for each propensity and regression model, both the eICU and MIMIC datasets show a positive VDE. Specifically, eICU has an effect of $0.8$ hrs ($95 \\%$ CI [$0.7$ to $0.9$]), while MIMIC shows an effect of $7.4$ hrs ($95 \\%$ CI [$7.2$ to $7.7$]). The results related to invasive ventilation rates are not meaningfully affected. We will update Figure 5b and the corresponding text in the revised manuscript to reflect the corrected estimates and clarify what the VDE measures.

3. Although the variance still decreases, the convergence rate may be slow because of the propensity score clipping. Nonetheless, we emphasize that the purpose of the self-normalized estimator is to mitigate variance compared to the canonical doubly robust version. Empirically, the behavior is better than one without self-normalization (see Figure 3a vs. 3b). Improving these properties further is certainly an active area of open research more broadly.

4. We agree that the variable $X$, representing race, should not be modeled as receiving a directed arrow from $Z$ alone. The more appropriate structure includes a hidden confounder between $X$ and $Z$, as demographic variables such as age may influence the distribution of racial groups. Importantly, introducing a bidirected arrow between $X$ and $Z$ does not affect our estimators or the theoretical analysis presented in Appendix A. We will revise the causal graph accordingly in the updated manuscript.

5. We agree and will expand the definition to the following: "Algorithmic fairness operationalizes computational methodologies to identify, measure, and address disparate behavior related to (automated or other) interventions such as decision-making processes. One prominent line of work evaluates differential performance of machine learning models across subpopulations…"

Limitations

1. We agree on the importance of clearly stating the assumptions underlying our approach. As with most causal inference methods, our framework relies on a specified causal graph and thus depends on assumptions about the data-generating process that can be challenging to verify in practice. Consequently, it shares common limitations related to potential misspecification of the causal structure. For example, if there is strong reason to believe that $W$ and $V$ have a latent common cause in a given application, the path-specific effect is unidentifiable. While our setup is more general than has been considered in typical causal fairness frameworks and more appropriate for studying bias in pulse oximetry applications, we will clarify these assumptions more explicitly in the "Limitations" section of the manuscript. This includes both assumptions specific to our ICU application in Section 4, and more generally, the limitations of potential conclusions drawn from our framework.

We hope this addresses the points you raised in your review. We are happy to clarify any additional questions you may have.