Optimistic Algorithms for Adaptive Estimation of the Average Treatment Effect

It would be beneficial to compare the proposed method with some of the many related works, if possible.

We did not compare our algorithm with Kato et al. (2020) because Cook et al. (2022) showed that ClipSDT significantly outperforms Kato’s algorithm, even under weaker assumptions (see Figure 2 in Cook et al.). Since we already compare against ClipSDT—the stronger of the two—it would be redundant to include Kato et al. (2020) in our evaluation.

As for Kato et al. (2024), the setting considered in that work is different from ours: their focus is on adaptively selecting which covariates to observe, whereas our problem does not involve covariates at all. As such, their method is not applicable to the setting we study.

What are the details of the experiments, including the data generation process for the synthetic dataset and the implementation details of the methods?

Each of our figures is based on simulations in a two-armed Bernoulli bandit setting, with different Neyman allocations. The mean rewards are set to $\mu_1 = 1/2$ and $\mu_0 \in \{0.05, 0.1, 0.2, 0.3, 0.4, 0.5\}$. At each round $t$, we sample $R_t(a) \sim \text{Bernoulli}(\mu_a)$ for $a \in \{0, 1\}$, and the algorithm observes the realized reward $R_t(A_t)$ corresponding to the chosen action $A_t$.

This interaction continues for $T$ rounds, after which we compute the estimated ATE. We repeat this process across multiple independent runs and report the variance of the resulting ATE estimates.

These experimental details are provided in lines 310–318 (right column) and lines 375–379. We agree that some parameters were only implicitly specified in Figure 2, and have revised the experimental section to explicitly list all relevant parameters, including the exact values used for the Bernoulli means, the number of rounds, and the number of repetitions. We hope this improves the clarity and completeness of the experimental description.

the authors mention many related works, but why are many of them not compared in the experiments?

We discuss this in the experiments section on Lines 303-307 (right column). Many of the prior works were designed for the IPW estimator, and so their performance in comparison to the AIPW estimator is so poor that it visually obscured the graphs. These existing algorithms do not converge to the asymptotically minimal variance.

It would also be helpful to include experiments on real-world datasets to validate the effectiveness of the method in real-world scenarios.

We have run simulations using the macro-insurance dataset from Dai et al. and will include the figure in the camera ready if accepted. Please see the table presented in our response to Reviewer kjqu for a summary of these simulations.

On line 144, what is the definition of $R(a)$?

We meant that $\Delta = \mathbb{E}[R_t(1) - R_t(0)]$ for any $t$ and we had omitted the subscript for brevity since the choice of $t$ is immaterial. However, we agree that this could be confusing. To clarify, we have revised the definition to $\Delta = r^\star(1) - r^\star(0)$, where $r^\star(a)$ was already defined just above line 135. This should eliminate the ambiguity.

Discussion on conditionally fixed mean and variance assumption

One concrete example where our assumption holds is the i.i.d. setting which is a standard assumption for RCTs [1]. More broadly, this assumption is either equivalent to or a generalization of those made in several prior works on adaptive experimental design (Kato et. al, Cook et. al, Neopane et. al) and is the standard assumption made in the bandit literature.

We also emphasize that these assumptions are necessary in a meaningful sense: the ATE is defined as the difference in mean outcomes under treatment and control, and without our assuming these means are conditionally constant, the ATE is ill-defined. Similarly, the Neyman allocation depends on the variances of the potential outcomes, so if the variances change across rounds, we no longer have a meaningful way to define an optimal allocation. Therefore, this assumption represents the weakest possible condition under which adaptive ATE estimation remains well-defined.

The paper's contribution is mainly theoretical...

We would like to clarify that our contributions are both theoretical and methodological. In addition to presenting new theoretical insights, we propose a novel algorithm that is not a minor variant of existing methods—it is based on distinct design principles. Our algorithm enjoys a greatly simplified analysis, achieves significantly stronger theoretical guarantees than prior work, and demonstrates superior empirical performance in simulations.

While our long-term motivation is to inform the design of adaptive clinical trials, we acknowledge that our current setting does not include covariates. However, it does capture a realistic and meaningful intermediate scenario where outcomes from previously treated patients are observed and used to inform future treatment decisions. The challenge of performing nonasymptotic analysis in this setting—without covariates—is itself nontrivial and had not been fully addressed in prior work.

Extending our framework to incorporate covariates is indeed an important next step toward practical applicability in real clinical settings. We view our current work as a foundational step toward that goal.

I believe this paper requires significant revision to clearly justify why the setting and assumptions are valid in real-world scenarios and to clarify why their theoretical results are exciting for a broader audience.

We respectfully disagree with this statement, as we believe it reflects a misunderstanding of the paper’s scope and contributions. Our goal is not to present a fully deployable solution for real-world clinical trials, but rather to address a fundamental theoretical problem that remains unresolved—even in the simplest setting without covariates. It is worth emphasizing that randomized controlled trials can, and often do, proceed without incorporating covariates into the treatment assignment, especially in early-phase trials, low-data settings, or when covariate information is unavailable or unreliable. Our work aims to rigorously understand this foundational setting, which has practical relevance in such scenarios, before tackling the added complexity of contextual information.

The setting and assumptions we consider are standard in the literature, have been adopted in prior foundational works. Our theoretical results are novel, conceptually clean, and advance the understanding of Neyman regret in nonasymptotic regimes. Importantly, these insights led to the development of a new algorithm that is not only theoretically sound but also empirically superior to existing methods.

While we agree that real-world applicability in full generality—particularly in large-scale or personalized settings—requires the incorporation of covariates, we view our work as a necessary first step. We will revise the introduction to clarify the scope, highlight the novelty of our contributions, and emphasize the broader relevance of our results beyond any single application.

Additionally, the paper should demonstrate that the proposed algorithm performs well on at least one real-world dataset.

We have run simulations using the macro-insurance dataset from Dai et al. and will include the figure in the camera ready if accepted. Please see the table presented in our response to Reviewer kjqu for a summary of these simulations.

[1] Imbens, G. W., & Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences (See chapters 1 and 6)

Thank you for your positive and thoughtful review. We appreciate your feedback and are glad that you found our work worthy of acceptance.

In response to your suggestion, we have incorporated a discussion of the concurrent work by Noarov et al. (“Stronger Neyman Regret Guarantees for Adaptive Experimental Design”) into our related works section. As this paper was posted on arXiv on 2/24/25—after the submission deadline—we were unable to include it in the original draft.

At a high level, this paper studies the fixed-design/adversarial setting introduced by Dai et al., and shows that, under additional assumptions, the ClipOGD algorithm from that work can be refined to yield stronger regret guarantees. However, their approach still relies on an IPW-based estimator. As a result, when adopting the definition of Neyman regret used in our paper, their algorithm suffers linear Neyman regret due to the inherent bias of IPW estimators in adaptive settings.

The claims are substantiated, although clarity and exposition could be greatly improved (e.g., Section 3 should be more clear and define the variables accordingly, the definition of Neyman regret seems off in eq. (5), result of Lemma B.1 is central to the algorithm and should be moved to the main text, etc.).

We agree Section 3 needed clearer definitions and better exposition. We’ve revised it to improve clarity and precision.

Regarding eq. (5), we understand the concern about the scaling mismatch. However, this scaling is intentional: multiplying the algorithm’s MSE term, $(\widehat \Delta_T - \Delta)^2$, by $T$ enables a direct comparison with the oracle’s MSE. We’ve clarified this by revising eq. (5), defining Neyman regret explicitly as the normalized excess MSE over the optimal policy, and recasting eq. (6) as a proposition to emphasize its connection.

We agree that more discussion on Lemma B.1 is warranted and have moved it to Section 5 and expanded the discussion to highlight how it supports Lemmas 5.1 and 5.2.

The authors focus on synthetic Bernoulli experiments. This is understandable as a first demonstration, yet adding semi-synthetic data could make the empirical study more realistic (see Dai et al. as an example).

We have run additional experiments using the macro-insurance intervention dataset used by Dai et al and will include them in the camera ready version. Our experiments on this real world dataset qualitatively align with our synthetic experiments. Below is a table summarizing the performance of various algorithms on this dataset.

However, the lack of contextual or covariate-based experiments is a gap that might be addressed in future work (as Kato et al. do incorporate covariates).

We agree this is an important direction. Kato et al. and Cook et al. address contextual settings but rely on asymptotic analysis. In contrast, our work is the first to provide nonasymptotic guarantees for adaptive reward estimation, even in the simpler noncontextual case. We see this as a foundational step toward handling contextual settings in a rigorous, finite-sample framework.

The main contribution of an optimistic approach for adaptive ATE estimation is novel and interesting.

We appreciate the recognition of our optimistic approach. In addition, our work is, to our knowledge, the first to address adaptive reward estimation with finite-sample guarantees—even without covariates. Our method also yields strong empirical performance while enjoying a simpler and more transparent analysis.

Exposition is occasionally confusing. For instance, Section 3 and 4.2 might be clearer if the difference between Eq. (1) and Eq. (2) were fleshed out more (particularly explaining $\widehat r_t$ in the A2IPW estimator).

Thank you for flagging this. We previously explained some differences between Eq. (1) and (2) in lines 154–161 but have now expanded this explanation for clarity. The distinctions are as follows:

• Eq. (1) uses marginal action probabilities $\mathbb{P}(A_t)$, while Eq. (2) uses conditional probabilities $\pi_t(A_t)$ based on past data.
• The standard AIPW estimator uses data-splitting, while A2IPW requires only that $\widehat r_t$ be predictable (i.e., adapted to the past). Beyond this, $\widehat r_t$ may depend freely on past data. We have added this explanation after Eq. (2). We also corrected a typo in Eq. (2) where $(r)$ should have been $(\hat{r})$.

Is the population assumed identical at each round (i.i.d.)?

Our assumption is more general than i.i.d. As noted in lines 133–136, we assume only that the means and variances of the potential outcomes are conditionally fixed over time. Other aspects of the data generating process may vary arbitrarily across rounds. This form of conditional stationarity is necessary for the ATE and Neyman allocation (and hence the problem) to be well defined.

How might one adapt OPTrack to the contextual setting such as in Kato et al.?

We agree this is a compelling direction for future exploration. Our primary goal here was to establish a rigorous nonasymptotic analysis of adaptive reward estimation in the simplest (tabular) setting, as no prior work had done so. Extending OPTrack to incorporate covariates, as in Kato et al., would introduce new complexities in controlling reward estimation errors across various function classes, providing a meaningful and challenging path for future research.