Externally Valid Policy Evaluation from Randomized Trials Using Additional Observational Data

We thank the reviewer for the comments and provide our response below. We believe they may clarify some potential misunderstandings.

[W1] The motivation for problem setup is not clear. Why inferring $L_{n+1}$ for the additional $(n+1)$th data point rather than the new $1...n$?

Bounding $L_{n+1}$ quantifies the out-of-sample performance of $\pi$ with respect to a future individual drawn from the target population (after having sampled the covariates of $n$ persons), see lines 81-83.

In our assessment, it could be possible to extend the methodology to consider, say, $N$ future individuals, i.e., $n+1, …, n+N$ by leveraging permutations of independent samples. However, it appears to lead to a combinatorial explosion which quickly becomes intractable. We have therefore opted for clear intuitive results that are computationally feasible.

[W1, cont’d] This would be reasonable in clinical trial, $n$ new patents coming in with unknown outcomes.

It seems this could be a misunderstanding of the problem setting we are considering: The aim of the paper is not to infer outcomes in a clinical trial, but rather outcomes of a policy applied to a target population for which we only have covariate data.

[W2] the method relies heavily on heuristics, e.g. strong assumptions about the correctness of the estimates of unknown distribution of p(S|X,U) are made, and division of D into D' and D'' seems arbitrary.

A central point of the methodology we develop is precisely to avoid strong assumptions about $p(S|X,U)$ (see for instance lines 34-35, 69-80, 131-132, 290-291). We do so using a model $\hat{p}(S|X)$ and the sensitivity specification $\Gamma$ in eq. (3).

Sample splitting $\mathcal{D}$ into $\mathcal{D}'$ and $\mathcal{D}''$, as described in lines 168-169, is a standard procedure in statistical inference to ensure valid inferences. (In supervised machine learning, this occurs e.g. in train-test splits to evaluate out-of-sample risk of a learned prediction rule.)

[W3] The experimental setup looks overly simplistic, a basic two-d covariate space and a simple quadratic model for the loss.

The first experimental setup presented in Section 5.1 is indeed intended to be illustrations as the subheading indicates. This is so that the reader can obtain some intuition behind the method. For this reason we could either use one or two-dimensional covariates to visualize the setting. We chose two-dimensional covariates as in Figure 4.

Note that the method uses a nonparametric assumption for the loss distribution; and only uses a parametric model for the sampling pattern, i.e., $\hat{p}(S|X)$, along with $\Gamma$. The simplicity of the conditional loss distribution is therefore not relevant when illustrating the method.

The second experimental setup presented in Section 5.2 (and Figure 3) evaluates seafood consumption policies based on real data and involves 8 covariates. This is a considerably more complex problem for policy evaluation.

[Q1] Could you clarify the rationale behind inferring L_{n+1} for a single additional data point rather than considering the distribution of outcomes for multiple new observations (1...n)?

Please see our reply to W1 above.

[Q2] Is there a way in your experiments to evaluate the estimated p(S|X) and hence odds ratios (Figure 4)? Is proposed alg 1 sensitive to mis-estimation of p(S|X), which is very likely due to unobserved confounding U?

We are unsure about the first part of the question. Figure 4 belongs to the illustrative experiments in Sec. 5.1 in which there is a synthetic ground truth $p(S|X)$. In real experiments, our suggestion is to use ideas from sensitivity analysis (see lines 86-99 respective Figure 3a and 5a) to evaluate a credible range for $\Gamma$ that limits the odds ratios in eq. 3.

The second part of the question seems to restate the misreading above: A central point of Alg. 1 is that it is robust against mis-estimation of $p(S|X)$, not least when there is unobserved confounding $U$! Hence the (benchmarked) miscalibration degrees $\Gamma$ required as input.

We are very grateful for the reviewer’s past comments, which improved the revision substantially.

The informal benchmarking approach involves building intuition for plausible impacts of selection bias due to unobserved factors, but it didn't seem to speak to model misspecification or estimation error. This is not a major issue necessarily, but it might be worth softening some of the claims that the proposed approach handles all these other types of error, without some corresponding method for benchmarking these. [Q1] How to incorporate model misspecification or estimation error into the choice of Gamma?

This is a valid point: the benchmarking method addresses only the plausible impact of unobserved selection factors $U$. We will add a remark that points the reader to Appendix B.3, where we suggest using reliability diagrams to quantify bounds on model misspecification or estimation errors.

The possible joint impact of $U$ and model misspecification or estimation error is, of course, beyond the scope of our methodology.

The second weakness, which I'm less inclined to weigh heavily, is that the novelty of technical contribution is slightly unclear. [Q2] Are there any parts of the technical contribution the authors would highlight as particularly interesting from their perspective?

Yes, while the `mechanical’ aspects of the employed proof technique does build on Jin et al. 2023 and cited works developed in other problems areas, the contribution to the problem of establishing externally valid policy evaluation is novel. We have tried to break down our proof steps in Appendix A in a transparent manner and highlight those steps that invoke past results (e.g. lines 767, 776, 778). To make it clearer we will also refer to the relevant theorems in the cited work.

Regarding the minor feedback:

[1] Regarding the benchmarking, it would be worth giving the caveat that this is an informal approach to benchmarking that can yield unintuitive results, see [0]. Part of the contribution of Huang 2024 (cited here) was to give a more principled approach under a different sensitivity model. However, I don't view this as a major weakness, since this is generally an unresolved challenge as I understand it for Rosenbaum-like sensitivity bounds as used here.

We agree. While we have tried to be transparent about it, adding a reference could make it even more formal.

[2] There's a lot of work on combining observational & experimental data, and it might be worth highlighting differences to the settings considered. For instance, [1] deals with a similar setting, but one where outcome information is available from the observational data. There are other papers that deal with estimating causal effects in target populations from experimental data where observational data includes individuals not represented in the trial, e.g., [2], [3].

We have descriptions of the setting both in the introduction (lines 25-31) and the background (lines 109-117), but the difference to other settings may not be clear enough. We will review the references and try to clarify this.

[3] It's not entirely clear from the introduction what the "loss" $L$ entails, as the name suggests something like a prediction error. I think it would be useful to highlight upfront that $L$ can represent e.g., an outcome of interest (more typically rendered as $Y$), as done in the application. It is more intuitive to me to interpret the setup in light of the application, e.g., the goal is to bound the probability of extreme outcomes occurring.

We refer to it as “loss” as the opposite of “reward”, either way it is an ordered decision outcome that we want to bound. However, this is a good suggestion that will help the reader and we will try to clarify it already in the introduction.

[4] As an improvement, it seems like it should be fairly straightforward to derive lower bounds as well, no? E.g., replacing $L$ with $-L$ and applying the same machinery. That might be useful in applications where you want to ensure that an outcome stays within a certain range.

Replacing it with $-L$ will result in a lower bound. Although we have not considered this type of application, it is a valuable suggestion and we will add a remark about it.

We thank the reviewer for the question and are pleased that he/she acknowledges the importance of the problem we are tackling.

If I understand correctly, the policy Pi is given as an exogenous parameter. However, it is a common practice that we learn a policy using the RCT data and aim to know its performance on the target population.

Yes, this paper focuses on the evaluation of any given policy $\pi$, which could either be proposed by any clinical expert or learned using past data. It is therefore possible to set aside, say, $N$ past samples from an RCT study to learn a policy $\pi_N$ and then use the proposed methodology to evaluate its out-of-sample performance. We will add a remark in the manuscript to inform the reader about this possibility.

We appreciate the reviewer’s honesty and trust that the chair will discount this review.