Efficient Policy Evaluation Across Multiple Different Experimental Datasets

Thank you for your feedback. We agree with your comment and have added another experiment using a real-world dataset.

---

For example, the non-synthetic experiments are only conducted on ACTG 175 clinical trial dataset. Experimenting on other different datasets will enhance the empirical results.

We appreciate this feedback. We provide an additional experiment using the real-world dataset called "Project STAR" dataset (Krueger & Whitmore, 2001) [1] in the global response (attached pdf). We will add full details about the experiment in the revised version of the paper.

Thank you for your thoughtful review. Please find below a point by point answer to all questions and concerns. Please let us know if we can clarify anything further.

---

(1) line 71, ‘variable $\mathbf{Z}$ For example’ -> ‘variable $\mathbf{Z}$. For example’; (2) unify the notations if necessary, try to avoid writing $\Delta_{ij}$ and $\Delta_{i,j}$ simultaneously.

Thank you for pointing this out. We will fix these.

---

For the experimental section, please give detailed descriptions on the simulated experiment and the empirical experiment

Thank you for the feedback. We will add further description on the details of the simulations. For further reference, we have provided the details of the dataset leveraged in this paper in the appendix.

---

Honestly, the materials in the two studies are similar. Authors can unify the two studies: presenting the study when combining multiple experiments followed by the two examples (example 2 and example 3). This can release more space to discuss the core components.

Thank you for your suggestion. We designed the current structure of the paper since our theories are new to both the causal and RL communities. We wanted to introduce the idea starting from a canonical setting (combining two experiments) and then extending it to more general settings.

---

Suppose that all the datasets are synthetic together such that the resulting data is treated as a single dataset obtained from one experiment

We added the simulation under the suggested scenario, and the result is reported in the attached PDF (Figure 3) in the global response. The key observation in this figure is that the errors do not converge to zero and diverge as the sample size increases. This shows that the estimators are not consistent, unlike the proposed estimators that are consistent with the true effect of the policy in the target domain.

---

The computation complexity increases

The computational complexity of the proposed estimator is $O(m \cdot T(n,m))$, where $m$ is the number of variables in a causal graph, $n$ is the number of samples, and $T(n,m)$ is the time complexity of training ML models for estimating nuisances, which is commonly polynomial in $n$ and $m$. The complexity is $O(m \cdot T(n,m))$ because we are learning $2 \times L \times m$ nuisances for constructing the proposed estimator. This analysis shows that even if the number of source domains (upper bounded by $m$) increases, the time complexity increases linearly with $m$. Therefore, the proposed estimator is scalable with respect to the number of source domains.

---

To my realization, the distributions vary even in the examples presented in the paper.

We couldn't parse this question. In our setting, the distributions vary across domains. However, if some invariances between domains (such as the domain transfer for $Y$ and $W$ in Def. 4 and Def. 6) hold, our proposed method is applicable.

---

How does the proposed methodology handle situations where the underlying assumptions are violated?

If the assumptions are violated, the proposed estimator will not be consistent with the effect of the policy in the target domain and may deviate systematically from the true value. For example, as shown in the attached PDF in the global response, if we ignore the heterogeneity assumption and treat all datasets as if they are from the same population, the resultant estimates do not converge and even diverge.

Thank you for your extensive review, we really appreciate the feedback. In the following, we comment on the mentioned weaknesses and address outstanding questions and concerns separately.

---

(1) Same number of studies to horizon (2) While the paper studies multi-horizon dynamic treatment regimes, the simulations are conducted in a single stage setting.

We appreciate the careful reading of our setting and thank you for raising this point. Given that the paper tackles a novel problem setting, we have presented a canonical case of the T-horizon off-policy evaluation problem that assumes access to T studies. This assumption can be relaxed in practice, depending on the causal structure. For concreteness, in the attached PDF, we provide a simple example where a two-horizon OPE problem can be solved with access to a single source domain. More generally, the number of heterogeneous domains and the horizon are not necessarily matched if a given causal structure satisfies the proposed criterion formalizing the invariance of mediated and outcome variables across domains.

---

Knowledge of matching source population.

In the worst case, where every variable has arbitrarily different distributions from those of the target domain, the data obtained from the source population may not provide any clues for estimating the quantity defined in the target domain. To transfer knowledge, some form of invariance between sources and the target domain is essential.

In our proposed framework, the required knowledge about domain discrepancy is relatively minimal. Instead of quantitative information, qualitative information on which variables' distributions could differ from the source is sufficient to apply the proposed method. For example, if the age distributions in London and New York are different, we can use this information without needing to know the extent of the difference.

---

A more realistic approach would adaptively learn which studies are similar to the target distribution at each time based on the data.

Thank you for the intriguing proposition. In the setting we consider, however, we do not have access to data from the target population (except for baseline covariates) – and therefore, adaptive learning, which requires sources from the target population, may not be directly applicable to our problem setting.

We want to remark that our problem setting might be reasonable for safety-critical or costly applications where inference of causal effects is necessary before implementing a candidate policy of interest and collecting data in the target domain.

---

Although the outcome distributions between the source and target populations may be similar at each time, they are not necessarily identical.

We appreciate the references provided and will cite them. However, we respectfully disagree with the statement. The invariance of outcome distributions across domains stems from the ignorability assumption, which is present in all mentioned papers (e.g., Assumption 1 in the first paper) and in this paper (fourth bullet of Def. 4). This assumption establishes conditional independence between the outcome variable and domain-representing index variables, ensuring consistent outcome distributions across domains.

---

D4RL benchmark datasets or OpenAI Gym environments

We appreciate these suggestions. However, the proposed benchmark may not be suitable for our problem setting, as it lacks elements like heterogeneous population inference, unobserved confounding, or causal structures.

Instead, to further convey the practical utility of the proposed approach, we conduct an additional experiment using a real-world dataset from Project STAR dataset (Krueger & Whitmore, 2001). This includes multiple stages and a semi-synthetic set-up (in which the existing dataset is partitioned to form different environments). We include a full description in the global response and attached pdf.

---

For instance, one can ignore the differences in the outcome distributions and assume the multiple datasets come from a single population

We added the simulation under the suggested scenario, and the result is reported in the attached PDF (Figure 3) in the global response. The key observation in this figure is that the errors do not converge to zero and diverge as the sample size increases. This shows that the estimators are not consistent, unlike the proposed estimators that are consistent with the true effect of the policy in the target domain.

---

Real data

To enhance the practical implications of the proposed method, we added a real-world data simulation using the dataset from Project STAR (Krueger & Whitmore, 2001) in the attached pdf (Figure 2).

---

Related works.

Thanks. We will definitely cite the introduced paper.

---

Would it be possible to establish the semiparametric efficiency of the proposed estimators?

Yes. Since the proposed estimator is composed of heterogeneous datasets from multiple domains, multiple sampling distributions $P^i$ corresponding to each dataset should be leveraged to construct the semiparametric efficiency bound. Specifically, the partial influence function [2], an influence function defined relative to each $P^i$, can be constructed to provide an efficiency bound for the part corresponding to $\mathbb{E}_{\mathcal{D}_i}$. We will add a detailed discussion on the semiparametric efficiency using the partial influence function in the revised version of the paper.

[2] Pires, Ana M., and João A. Branco. "Partial influence functions." Journal of Multivariate Analysis 83.2 (2002): 451-468.

Thank you for your valuable feedback and insights. In the following, we aim to answer each one of your concerns in turn. Please let us know if we can provide more details.

---

(1) the empirical evaluations are very limited. There is little discussion on how this could impact realworld problems, (2) Please provide more empirical analysis on the proposed method ideally on more real-world datasets if possible. (3) Provide more empirical evidence on real-world data.

We appreciate this feedback. We provide an additional experiment using the real-world dataset called "Project STAR" dataset (Krueger & Whitmore, 2001) [1] in the global response (attached pdf). We will add full details about the experiment in the revised version of the paper.

[1] Krueger, Alan B., and Diane M. Whitmore. "The effect of attending a small class in the early grades on college‐test taking and middle school test results: Evidence from Project STAR." The Economic Journal 111, no. 468 (2001): 1-28.

---

there is not a detailed analysis on the empirical results of the results on ACTG175 dataset.

Please note that the data generation process relating to ACTG 175 is described in the Appendix. More detailed discussions on the results of the synthetic and ACTG 175 experiments will be added.

---

From figure 3.c, it can be seen a contradictory message that the proposed DML method does not perform better than OM on all datasets, though there is no discussion on that.

Please discuss the results on ACTG175 datasets and the different pattern observed in Fig3.c.

Thank you for this comment. This observation is expected. The construction of the DML estimator combines aspects of the OM estimator and the PW estimator. The error of the DML estimator can be represented as a product of the errors of nuisances for the OM estimator and the errors of the PW estimator. As a result, if the PW estimator has a large error, this can be reflected in a correspondingly larger error of the DML estimator. Since the error of the PW estimator is relatively larger than that of the OM estimator in Figure 3c, we expect the DML estimator to underperform compared to the OM estimator when the sample size is small. However, when both the OM and PW estimators converge to the truth, we expect that the DML estimator will outperform the other estimators as the sample size grows.

---

also authors state that "simulations on real and synthetic data are provided for illustration purposes only." which brings the question to what extend this approach is useful in real world scenarios.

Please consider the context in which this statement is located. The full sentence is in our Broader Impact Statement section as follows:

"Finally, we emphasize that simulations on real and synthetic data are provided for illustration purposes only. These results do not recommend or advocate for the implementation of a particular policy, and should be considered in practice in combination with other aspects of the decision-making process."

With these sentences, we do not question the usefulness of our methods, which we have justified through a series of theoretical and empirical evidence. Rather, we clarify that our paper should not be read as an endorsement of specific policies.

---