Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects

Dear Reviewer hoJN

We thank the reviewer for their kind words and enthusiasm for our paper. We address your concerns below.

Claims and evidence/Questions For Authors:

I have reservations about the experimental results. The authors claim that two-stage designs outperform single-stage approaches, yet it is unclear in what sense this improvement is measured. There is limited evidence provided that these designs enhance sample size efficiency or statistical precision—key factors in classical randomized trials, where the width of the asymptotic confidence intervals is the primary concern. Could you elaborate on why the experimental evaluation focuses only on the certificate as the performance metric?

The key point of our paper is that we are pruning arms, not trying to estimate the means of all interventions precisely. Our setting is designed for scenarios where the goal is to obtain a tight confidence interval for the best possible policy—not for all arms individually. In fact, our lower bound explicitly characterizes the width of the confidence interval for the best policy only. We intentionally do not aim to produce tight confidence intervals for lower-quality interventions, as they are ultimately discarded by our method.

It is not clear from the experiments how much sample size saving the method can offer. It would be helpful to see comparisons that directly measure the reduction in sample size needed to achieve a desired level of statistical precision (e.g. confidence interval width). How does the certificate translate into tangible sample size savings or improved statistical efficiency in real-world RCT settings?

Our method should be understood as providing a worst-best case scenario guarantee under a fixed budget. That is, given a fixed budget, we are able to offer a statistically valid guarantee on the performance of a high-impact treatment.

Can you provide insights or additional experiments that show the efficiency gains from the proposed method as the fixed sample size budget increases?

Figure 2 in the paper illustrates the average value of our certificate as a function of the available budget. As shown in the plot, increasing the total budget reduces the gap between the sample splitting approach and the two-stage policy—demonstrating that sample splitting increasingly approximates the performance of the two-stage design as more data becomes available.

Could you clarify the implementation details of the single-stage baseline used in your experiments? What estimator is used to compute the treatment effects in the single-stage setting?

The estimate is given by the sample mean of each arm. In the first stage, we use uniform allocation across all $n$ original arms. We then compute the estimator using all collected data—there is no data discarded or ignored in this process.

Have you considered or benchmarked the performance of alternative estimators, such as the AIPW estimator? In scenarios where a near-perfect model for is available, an AIPW estimator achieves the efficiency lower bound by imputing the outcomes assigned to the "bad" arms. How do you expect the performance of a single-stage method using AIPW to compare with your proposed two-stage design in this setting?

We do not incorporate features in our current design, and therefore the class of methods suggested by the reviewer does not apply in our setting. Nevertheless, we thank the reviewer for raising this point—it would indeed be interesting to explore how such methods compare in an extended version of this work where features/context are available.

We thank again the reviewer for their kind words and we look forward to clarify any extra question that arrises.

Dear Reviewer S1mJ,

We thank the reviewer for their kind words. We address your concerns below.

Questions/Weaknesses

Theoretical claims made in the paper are supported. They are mainly adaptations of existing results from bandit literature and not substantial.

We believe there are several novel theoretical contributions worth highlighting: First, to provide the bandit's approximation guarantee in Theorem 3.2, we developed a new characterization of optimal policies and proved how the problem can be reduced to a search over only top-k policies. Second, we would like to emphasize that our theoretical results supporting the Bayesian approach are not derived from bandit literature at all. Rather, these contributions are more closely related to submodularity and discrete optimization.

It seems like $s_1$ and $k$, [...] are super critical for the final output/success of the algorithm, but there is no recipe as to how to choose them. For that reason, I do not see how the existence of a top-k algorithm results are interesting/useful (i.e., how do I choose or is more important). It is not clear what "optimal k" is, and if the authors' algorithm finds this or not. [...] RCTs are already small in sample size and throwing some part of the data after the first stage does not seem like the best use of data..

To clarify, we demonstrate that the optimal policy, that is the policy that will yield the highest certificate, is a top-k policy. Here, the optimal $k$, which we denote as $k^*$, refers to the number of arms that such optimal policy selects for the second round. We agree with the reviewer that the selection of $s_1$ and $k$ is critical to the success of our algorithm. However, there appears to be a misunderstanding regarding the role of $k$: the purpose of our algorithm is precisely to automatically select a near-optimal value of $k$, rather than treating it as a user-specified input. To this end, we design our sample splitting algorithm to estimate a value of $k$ with approximation guarantees to the optimal $k$ ($k^*$).

Regarding the choice of the first-stage budget $s_1$: in Figure 3, we analyze how shifting the budget between the first and second stages affects certificate performance. We find that allocating 20–50% of the budget to the first stage yields the best results.

With this clarified, we would like to emphasize that concerns about statistical power in the first stage are somewhat misplaced. The point of our two-stage design is to improve statistical power in the second stage, which directly impacts the final certificate.

Algorithm 1 [...] the for loop goes from 1 to k, and then k is defined after that for loop, so that probably needs fixing We thank the reviewer for pointing out this typo. Instead of k in the for loop it should be $n$ -the original number of arms. It is not clear how this approach would affect subsequent subgroup analysis.

This is a very insightful question. Our view is that by concentrating samples on the more promising arms, we also gain statistical power for conducting subgroup analyses of heterogeneous effects within those treatments. In other words, the approach naturally prioritizes subgroup analysis for the arms with the greatest overall potential.

Methodologically, authors mention that sometimes it is not feasible to wait on the results of an RCT to guide next steps. This I think is another thing that limits the practical adaptation of the algorithm proposed in this paper.

As we point out in the paper, there are important scenarios—such as drug trials or government interventions with very slow feedback loops—where outcomes can only be observed after several months or even years. In such settings, fully adaptive trials are often infeasible. In summary, while there is indeed a large and growing literature on adaptive trial designs, our work addresses a key practical limitation in their applicability, offering a principled alternative when adaptivity is constrained.

Experimental Designs Or Analyses:

In Figure 4, UCB method seems to work the best by a margin. Can you comment on what is the disadvantage method? It is fairly simple method and complex at all. Is there a reason why some of the baselines in the real-world experiments (e.g., UCB) are not benchmarked against in the synthetic experiments?

Our designs are motivated by use cases where delayed outcomes make fully adaptive trials infeasible (e.g., deploying the UCB algorithm in practice). For this reason, UCB is not directly comparable to our setting, which is why it is not included in the benchmarks shown in Figure 1. However, in Figure 4, we do compare our approach to fully adaptive baselines in order to quantify the potential loss relative to an idealized, fully adaptive trial. This comparison highlights the performance tradeoffs while reinforcing the practicality of our method in constrained settings.

Dear Reviewer HuaZ,

We thank the reviewer for their kind words and enthusiasm for our paper! We address your questions below.

Questions/Weaknesses

The use of "treatment effect" in this paper is a bit confusing to me. Typically in RCT, there will be one control arm and many other treatment arms, so the treatment effect should be the gap between any treatment arm and the control arm. In this way, the calculation of the so-called treatment effect certificate should be adjusted --- If I understand the math correctly, the use of concentration inequality can be well extended given that everything is i.i.d; If I miss some key component and the adjustment is actually hard, the authors should change the notion of "treatment effect".

In the Bernoulli setting, the average treatment effect is precisely the mean from the distribution. This is a consequence of the outcome of the model being one (say for the treated) and zero (the untreated). Nevertheless, we do agree, about the need to be more precise in the others settings and we thank the reviewer for the feedback (we will add clarifications in the camera ready version)

In the numerical experiments, it seems that two-stage sample split is worse than two-stage TS, especially the case without prior. I think this observation raises some concerns. It basically says that if we limit the design to be two-stage, actually we should follow TS, though it is tailored for minimizing regret. In addition, I am curious about the performance of fully adaptive TS. The reason is that if fully adaptive TS has the best performance (better than UCB), then there must be some theoretical essence about TS for maximizing certificates.

Thank you for your insightful observation regarding the Thompson sampling experiments. Our intention with these experiments was to demonstrate that our two-stage approach offers sufficient flexibility to accommodate non-uniform sampling strategies. We appreciate the reviewer noting the effectiveness of Thompson sampling in computing a better certificate. We agree with this assessment and believe this likely occurs in scenarios where the variance in the distributions is substantial relative to the gaps between arms, which can indeed make rapid accumulation of empirical means challenging.

In the Appendix part C, the authors conduct multi-stage designs and obtain the certificate using all-stage data. I am wondering how the certificate is obtained given that the data is not i.i.d.

The final certificate was obtained as in the experiments with two stages with the caveat of only using data from the last stage. The reviewer is absolutely correct that in order to give a theoretical guarantee in this scenario we would need to correct for the dependency inherent of picking a particular subset arms at each stage. However, we believe that we could deliver such a guarantee through only slight modifications which account for the lack of i.i.d-ness, and that the final results would not differ significantly.

Other Comments Or Suggestions

We thank the reviewer for the thorough read of our paper and identifying formatting mistakes, and will update our manuscript accordingly.

Dear Reviewer Kupv,

We thank the reviewer for their kind words and enthusiasm for our paper. We address your concerns below.

Questions/Weaknesses

In Alg 1, there is data splitting procedure. How much is this impacting the efficiency of the algorithm? Is it possible to do a cross fit type of algorithm? Sampling splitting is typically an inferior choice as data points in many RCTs are pretty expensive.

We thank the reviewer for their valuable suggestion. We focus on the sample split algorithm for ease of presentation but it can be easily extended to a cross-fit style of approach. We run a small experiment with this, comparing the results of cross validation to that of sample split, using the original settings from Figure 1 (with equal budget between stage 1 and stage 2). We find that sample splitting performs as well, if not better, than cross validation. Using cross validation with k=2 performs slightly worse (~1% worse), while for k=3,4,5, we find that it performs ~4% worse compared to sample splitting. This occurs because cross validation sacrifices test data points for training ones; for example, doing k=4 folds results in ¼ of the data being used for evaluation, and ¾ for some analog of training. As a result, we find that the size of these evaluation data points are critical to reducing variability.

In practice, researchers are also making decisions about choosing $s_1$ and $s_2$. The author had some discussion on the tension between these two parameters, but what is the practical guidance from the theory for the best choice of $s_1$ and $s_2$ ?

The reviewer raises a very important inquiry. In Figure 3 of the paper, we compare the impact of shifting budget between $s_1$ and $s_2$ on the performance of various policies for certificate generation. We find that having the first stage occupy between 20-50% of the budget allows for optimal performance. This occurs because the first stage needs to have a large enough size to eliminate sub-optimal arms, while not being too large that the second stage is small. Intuitively, the first stage should be just large enough so that any suboptimal arms can be eliminated.

For incorporating the prior, there is always a $1 - 1/e$ gap guaranteed by the lower bound. Is that saying (theoretically) incorporating prior is a slightly worse strategy compared with sample split? This is counter-intuitive as I would assume prior information on the parameters can provide more guidance for policy designing.

We note that the $1-1/e$ bound is not against the absence of a prior, but rather the optimal selection of arms even with the prior. In essence, while our bound is worse, our comparison point is against a better opt value. Naturally the reviewer is right to point out that a better prior will lead to better finite sample guarantees for the Monte Carlo approximation and thus probably a less pessimistic lower bound.

For simulation (Compare against Adaptive designs), Sample Split is actually doing close to (a little bit worse than) Two-stage Thompson sampling (TS). Is this a general phenomenon or simply due to the experimentation design?

It is likely that for our particular scenario -- where the posterior properly captures the variance of the arms -- Thompson sampling is better than uniform sampling. However, this might not be the case if the gaps between arms are big enough. Such a scenario will lead to a faster accumulation of the empirical means than the convergence of Thompson sampling.

Other Comments Or Suggestions

We thank the reviewer for the thorough read of the paper and pointing out this formatting mistakes. We plan to correct these in our final version of the paper.