Quantifying Treatment Effects: Estimating Risk Ratios via Observational Studies

• Figure Caption Clarity:
  Thank you for pointing this out. We agree that the figure captions could be clearer. In particular, we will revise the caption to explicitly mention that the colors represent the "Sample Size".

• Doubly Robust Conditions:
  You are absolutely right that in classical doubly robust settings, it is sufficient for only one of the nuisance estimators (either the outcome regression or the propensity score) to be consistent for the overall estimator to remain consistent. This is commonly referred to as weak double robustness.

  As discussed in Wager's Causal Inference Book (Section 3), it is helpful to distinguish between weak and strong double robustness:

  • Weak double robustness refers to the property that the estimator is consistent if either the outcome model or the propensity score model is estimated consistently. However, weak double robustness does not, in general, guarantee asymptotic normality or valid confidence intervals under model misspecification.
  • Strong double robustness, on the other hand, refers to the setting where asymptotic normality can be achieved if both nuisance functions are estimated consistently at sufficiently fast rates. Specifically for the classical RD setting, this holds when both $\hat \mu_{(t)}$ and $\hat{e}$ are estimated with root-mean squared error rates decaying faster than $n^{-\alpha_{\mu_t}}$ and $n^{-\alpha_e}$ respectively, and when $\alpha_{\mu_t} + \alpha_e \geq \frac{1}{2}$ for each $t \in \{0, 1\}$. Then AIPW is asymptotically normal.

  In our paper, we establish strong double robustness for RR-AIPW. This result relies on both Equation (17) and Equation (14), which are strictly equivalent to the classical conditions for strong double robustness. It is worth noting that weak double robustness is also achieved for RR-AIPW but not for RR-OS since we always need $\hat{\mu}_{(0)}$ to be consistent.

• Low Baseline Risk:
  Thank you for raising this important point. When the baseline risk is extremely low, the risk ratio becomes inherently unstable, often resulting in wide confidence intervals. We have explored this empirically, and the results are shown in the table below.

  As the baseline risk approaches zero, the estimated RR can become extremely large, while the absolute risk difference remains small: different causal measures lead to different interpretations, thus suggesting that several causal measures should be used simultaneously to properly understand the impact of a treatment.

  This case also illustrates why it is important to report both an absolute measure (e.g., risk difference) and a relative one (e.g., risk ratio), as they provide complementary perspectives on treatment effects—particularly when baseline risks are extreme.

  Baseline	RR	Estimator	Coverage (mean ± std)	Length (mean ± std)
  2.5	1.77	Linear RR-G	0.878 ± 0.113	0.099 ± 0.002
  2.5	1.77	OLS RR-G	0.948 ± 0.050	0.120 ± 0.003
  2.5	1.77	Linear RR-AIPW	0.968 ± 0.047	0.185 ± 0.007
  0.045	15.3	Linear RR-G	0.992 ± 0.027	3.728 ± 0.513
  0.045	15.3	OLS RR-G	0.984 ± 0.037	3.037 ± 0.435
  0.045	15.3	Linear RR-AIPW	1.000 ± 0.000	4.734 ± 0.735
  0.0018	112.1	Linear RR-G	0.987 ± 0.022	884.427 ± 206.672
  0.0018	112.1	OLS RR-G	0.978 ± 0.032	1026.384 ± 214.754
  0.0018	112.1	Linear RR-AIPW	0.972 ± 0.036	2073.229 ± 509.709

  Table: Coverage and length of confidence intervals for different RR estimators across varying baseline risks.

• Clarifying the Novelty and Contribution:
  We thank you for acknowledging the thoroughness of both the theoretical development and empirical validation. We would like to take this opportunity to clarify the main contribution of the paper.

  One of the main contributions of this paper is to provide a comprehensive theoretical and empirical analysis of RR estimators, including new ones like RR-OS and RR-AIPW, in the context of observational studies. While RR is a well-established estimand, its estimation introduces distinct challenges due to its non-linearity.

  To the best of our knowledge, there has not been a unified and rigorous asymptotic analysis of IPW, G-formula, or AIPW estimators for the marginal RR in the literature. Our work aims to fill this gap by deriving explicit variance expressions, exploring efficiency bounds, and validating the estimators both theoretically and empirically in both continuous and binary outcome settings.

• On the Work by Rose and van der Laan (2014):
  The cited work by Rose and van der Laan focuses on Targeted Maximum Likelihood Estimation (TMLE) for the risk difference in case-control studies. While this is an important contribution, the setting considered there differs from the randomized and observational study framework analyzed in our work and focuses on binary outcomes. Furthermore, while TMLE is a flexible approach, our paper focuses on IPW, AIPW, and G-formula estimators, whose asymptotic properties have not, to the best of our knowledge, been rigorously established in the context of marginal RR estimation using observational data — also for continuous outcomes. We thank the reviewer for pointing out this reference; we will mention it in the conclusion as a potential research direction for finite sample studies.

• On the Mention of Kennedy’s Work:
  We are not entirely sure what specific paper by Kennedy you refer to. We have read several papers by Kennedy, three of which are cited in our work, but we did not encounter specific studies by Kennedy (or anyone else) analyzing different estimators of the marginal risk ratio. That said, it is worth noting that the conditional risk ratio is not directly collapsible. This means that estimating the conditional RR generally does not provide a valid estimator for the marginal RR. In contrast, our work specifically targets estimators for the marginal RR and provides a detailed asymptotic analysis under standard causal inference assumptions.

We hope that the above points help to dispel your doubts about the novelty of our work.

Thank you for your thoughtful and constructive review. We appreciate your positive comments on the clarity of the writing and the contribution of our work to the causal inference literature. Please find our responses to your comments below:

1. Regarding the assumption $E[Y(0) \mid X] > 0$:

   Our initial motivation for introducing it was to establish finite-sample bias and variance results, which are currently included only in the appendix. However, for the main theoretical results, we only need to assume $E[Y(0)] > 0$ to ensure that the risk ratio is well-defined.

   We agree that this assumption may not always hold, but in practice, many biological quantities are positive (e.g., blood concentrations). Besides, if the variable $Y(0)$ is binary, $E[Y(0)] > 0$ holds as soon as one observation satisfies $Y_i(0) = 1$, which seems to be a reasonable assumption.

   In the revised manuscript, we will move the stronger assumption $E[Y(0) \mid X] > 0$ to the appendix (where it is needed) and replace it in the main text with the weaker and more general condition $E[Y(0)] > 0$. We will also add the above discussion.

2. On model misspecification in simulations:

   We completely agree that comparing the estimators under various forms of model misspecification is important. Due to space constraints, we were unable to include this analysis in the main body of the paper, but we incorporated two relevant scenarios in Appendix 8.2:

   • Wager C model (Appendix 8.2.1):
     Both response functions are nonlinear, while the propensity score is logistic. In this setting, Logistic RR-IPW and Linear RR-AIPW and RR-OS (using linear and logistic regressions) remain asymptotically unbiased, while Forest RR-OS and Forest RR-AIPW are biased for small sample sizes due to the difficulty of forests in estimating logistic functions with few observations.

   • Wager A model (Appendix 8.2.2):
     Both response functions and the propensity score are nonlinear/non-logistic. All linear estimators are biased in this case, but estimators that use Random Forests to estimate the nuisance functions exhibit decreasing bias as the sample size increases.

   If the paper is accepted, we will use the additional page to move this analysis into the main paper.

3. On the implementation of variance estimators:

   Thank you for pointing this out. In fact, we designed variance estimation procedures to construct the confidence intervals reported in the experiments, but we recognize that the implementation details were not highlighted enough.

   In the current version, the relevant formulas are in the appendix (in Section 4.2). We will elaborate on this and add a dedicated section in the Appendix. In addition, we will better highlight in the main text that we have estimated all variances, as they can all be expressed using estimable quantities, and that we used these estimations in all our simulations.

4. Redundancy in Assumption 2.2:

   You are totally right. We will revise this assumption to remove the redundancy. Thank you for pointing this out!

Thank you very much for your positive review. We are glad that you found the paper well-written, well-structured with interesting results for practitioners. We also appreciate your suggestion regarding the use of histograms to illustrate the asymptotic normality of our estimators. We agree that this would provide a clearer visual understanding of the results and plan to incorporate it in the next revision.