Dynamic Subgroup Identification in Covariate-adjusted Response-adaptive Randomization Experiments

• Thank you for your insightful questions regarding the design objective and for kindly pointing us to the reference.
  • We completely understand your concern about the practicality of identifying the best set of subgroups rather than all the benefitted ones. In many clinical settings, treating all the benefitted patients is indeed a natural and ethical goal from the practitioner's standpoint when the number of treatments is unlimited. Different from this setting, our design is tailored to situations where resources are limited, making it preferable to identify only the best set of subgroups. For example, during the onset of COVID-19, when the total number of vaccines was limited, only a subset of the population could be treated. The design objective of identifying the best set of subgroups thus becomes more relevant.
  • Furthermore, we would like to point out a subtle difference between our design objective and the design objective of maximizing overall welfare. Our work considers situations where assigning a treatment is costly, and there is an overall constraint on how many treatments can be deployed. In Fernandez-Loria and Provost (2022), for example, the goal is to ensure all the benefitted subjects are precisely assigned to the treatment arm while the cost of treatments is negligible. They aim to learn the optimal policy that maps a given covariate to the best arm for that patient profile. In our work, we consider scenarios where implementing the treatment can be costly per sample of a patient (experimentation unit). For example, in clinical settings, randomized experiments are often expensive due to the substantial costs of treatment medications. Consequently, the resource constraints in our problem are equivalent to the number of treatments that can be administered.
  • We appreciate the reference you provided, which we found very helpful. We agree that efficiently estimating the best causal effect can sometimes misalign with the best causal decision rule, especially when the best decision rule is to always treat a patient when there is a positive effect. In such cases, accurately estimating the magnitude of the treatment effect becomes less important. Since our primary objective is not to assign all subjects with positive treatment effects to the treatment arm, there could be a misalignment with the best causal decision rule discussed in Fernandez-Loria and Provost (2022). Nevertheless, if we are in a scenario similar to the previously mentioned COVID-19 vaccine example, where the best decision rule is to identify the most benefitted subgroups to prioritize treatment assignments, the accurate causal estimation aligns well with the best causal decision-making. We agree that it is important to be mindful of whether the design objective leads to the best causal decision rule, and the potential misalignment between causal effect estimation and causal decision-making should be carefully discussed. We hope to add this discussion to our revised manuscript.
  • While our current design is not tailored to welfare maximization, we shall discuss a potential approach to refining our design toward maximizing participant welfare. Consider a two-stage experiment with four candidate subgroups and their population treatment effects follow the order: $\tau\_1=\tau\_2 > \tau\_3 > \tau\_4$, where $\tau\_1 = \tau\_2 > \tau\_3 > 0 \geq \tau\_4$. At the end of Stage 1, our current procedure will be able to correctly identify subgroups 1 and 2 as the best set of subgroups with a high probability. If we were to maximize the patient's welfare, we would add an additional ``early stopping" step. That is, besides identifying the best set, we will also identify subgroups that exhibit significantly adverse treatment effects. To protect the patient's welfare, we will stop Subgroup 4 from enrollment in the next stage. In Stage 2, we not only avoid impairing the welfare of Subgroup 4 but also maximize the resources (treatments) available to the rest of the benefitted subgroups.
• Thank you for your careful reading of our manuscript. We shall clarify our optimization problem formulation as follows. Our original optimization problem is formulated as $\max_{ \mathbf{e}} \min\_{2\leq j\leq m^\ast\_{t}} \frac{( \hat{\tau}\_{t-1,(j)} - \hat{\tau}\_{t-1,(1)})^2}{2\big(\hat{\mathbb{V}}\_{t-1,(1)}(e\_{1}) + \hat{\mathbb{V}}\_{t-1,(j)}(e\_{j})\big)}, \text{s.t.}\ \sum\_{l=1}^{m^\ast\_{t}} \hat{p}\_{tl} e\_l \leq c\_1, \ c_2\leq e\_l \leq 1-c\_2,\ l=1, \ldots,m^\ast\_{t}$. The set of constraints includes the resource constraint: $\sum\_{l=1}^{m^\ast\_{t}} \hat{p}\_{tl}\leq c\_l$, and the feasibility constraint: $c_2\leq e\_l \leq 1-c\_2,\ l=1, \ldots,m^\ast\_{t}$. Because the original objective function takes the minimum of $m^*-1$ rate functions, the original optimization problem is nonlinear. We instead work with its equivalent epigraph representation: $\max\_{ \mathbf{e}} z$, s.t. $\min\_{2\leq j\leq m^\ast\_{t}}\frac{( \hat{\tau}\_{t-1,(j)} - \hat{\tau}\_{t-1,(1)})^2}{2\big(\hat{\mathbb{V}}\_{t-1,(1)}(e\_{1}) + \hat{\mathbb{V}}\_{t-1,(j)}(e\_{j})\big)} -z \geq 0, \ \sum\_{l=1}^{m^\ast\_{t}} \hat{p}\_{tl} e\_l \leq c\_1, \ c_2\leq e\_l \leq 1-c\_2,\ l=1, \ldots,m^\ast\_{t}$, which is Eq (1) in our current submission. The original formulation of the objective function was omitted due to space limits. We will clarify the objective function in our revision.

Per your suggestions, we plan to make the following updates to our manuscript:

• We will add a discussion on the potential gap between our design and the best causal decision rule in precision medicine scenarios in our revised manuscript.
• We will revise sentences and notations for clarity.
• We will provide more illustrations of Eq (1).

Reference:

• Fernandez-Loria, C. and Provost, F. (2022). Causal decision making and causal effect estimation are not the same... and why it matters. INFORMS Journal on Data Science, 1(1):4–16.

Thank you for your valuable questions and suggestions!

A main limitation of our proposed approach is the assumption that outcomes are observed instantaneously following treatment. This assumption, prevalent in adaptive experiments such as Hu et al. (2015) and Zhu and Zhu (2023), simplifies the modeling process and allows for quick adjustments based on the latest data. While this assumption is common, it may not always reflect real-world scenarios where response delays. To provide a more comprehensive understanding, we review the literature addressing delayed responses.

• The importance of incorporating delayed responses in adaptive experiments is well-recognized in the literature. Rosenberger et al. (2012) discuss the effects of delayed responses on response-adaptive randomization. Early work by Wei and Durham (1978) introduces the randomized play-the-winner rule, which updates the contents of an urn only upon receiving patient responses, thus naturally accommodating delayed responses. This approach offers inherent flexibility in managing delays and represents an improvement over more rigid methods. Bai et al. (2002) and Hu and Zhang (2004) establish asymptotic normality results under urn models with delayed responses, explicitly concerning the asymptotic normality of the fraction of patients assigned to each treatment arm. Their findings show that limiting distributions remain unaffected by delayed responses under reasonable conditions, providing a solid theoretical basis for handling such delays. Zhang et al. (2007) extend this work by introducing a generalized drop-the-loser urn model, demonstrating that asymptotic properties are preserved despite response delays. This generalized model offers broader applicability and enhanced flexibility for managing various delay mechanisms.

• Regarding the doubly-adaptive biased coin design (DBCD), Zhang and Rosenberger (2006) have shown through simulations that moderate delays in responses have minimal impact on the power and skewness of the DBCD. Their results suggest that the design remains robust even in the presence of delays, although the effects of more severe delays are less clear. Hu et al. (2008) study the asymptotic properties of the DBCD with delayed responses, showing that these properties remain unaffected by such delays. They also provide strong consistency results for the constructed variance estimator that incorporates delayed responses, enhancing the design's reliability in practical scenarios.

• In addition to response-adaptive randomization designs, some group-sequential designs address the challenges brought by delayed responses. Hampson (2013) proposes to incorporate short-term endpoints to enhance the efficiency of group-sequential tests when long-term responses are delayed. This method effectively balances immediate data needs with constraints imposed by delayed outcomes. Schuurhuis et al. (2024) expand this framework by suggesting the integration of pipeline data into group-sequential designs, enabling the trial to restart after a temporary halt. This integration provides added flexibility and robustness in managing trials with delayed responses.

• Overall, handling delayed responses requires challenging adjustments to our design. First, our objective function, based on the semiparametric efficiency bound of the subgroup treatment effect, must be revised. The difficulty stems from the function's current assumption of immediate responses, and modifying it to account for delays significantly complicates the task of ensuring accuracy. Second, our estimators need updating. Delayed responses impact both the treatment effect and variance estimators, necessitating an additional step to address these delays. While this step is crucial for maintaining precision and reliability, it also introduces complexities that must be managed to ensure a robust estimation process.

Per your suggestions, we plan to make the following updates to our manuscript:

• Present a more in-depth analysis of the literature in our revised manuscript.
• Discuss the critical limitation in the context of the existing literature in our revised manuscript.

Reference:

• Hu, J., Zhu, H., and Hu, F. (2015). A unified family of covariate-adjusted response-adaptive designs based on efficiency and ethics. Journal of the American Statistical Association, 110(509):357–367.
• Zhu, H. and Zhu, H. (2023). Covariate-adjusted response-adaptive designs based on semiparametric approaches. Biometrics.
• Rosenberger, W. F., Sverdlov, O., and Hu, F. (2012). Adaptive randomization for clinical trials. Journal of Biopharmaceutical Statistics, 22(4):719–736.
• Wei, L. and Durham, S. (1978). The randomized play-the-winner rule in medical trials. Journal of the American Statistical Association, 73(364):840–843.
• Bai, Z., Hu, F., and Rosenberger, W. F. (2002). Asymptotic properties of adaptive designs for clinical trials with delayed response. The Annals of Statistics, 30(1):122–139.
• Hu, F. and Zhang, L.-X. (2004). Asymptotic normality of urn models for clinical trials with delayed response. Bernoulli, 10(3):447–463.
• Zhang, L.-X., Chan, W. S., Cheung, S. H., and Hu, F. (2007). A generalized drop-the-loser urn for clinical trials with delayed responses. Statistica Sinica, 17(1):387–409.
• Zhang, L. and Rosenberger, W. F. (2006). Response-adaptive randomization for clinical trials with continuous outcomes. Biometrics, 62(2):562–569.
• Hu, F., Zhang, L.-X., Cheung, S. H., and Chan, W. S. (2008). Doubly adaptive biased coin designs with delayed responses. Canadian Journal of Statistics, 36(4):541–559.
• Hampson, L. V. and Jennison, C. (2013). Group sequential tests for delayed responses (with discussion). Journal of the Royal Statistical Society Series B: Statistical Methodology, 75(1):3–54
• Schuurhuis, S., Konietschke, F., and Kunz, C. U. (2024). A two-stage group-sequential design for delayed treatment responses with the possibility of trial restart. Statistics in Medicine.

1. Thank you for pointing us to the references. While both Guo et al. (2017) and ours are in an adaptive experiment setting, there are two major differences. (i)Their design is Bayesian, relying on prior specification, while ours is frequentist and model-free. (ii)They do not discuss theoretical properties for the identified subgroups, while we provide theoretical results justifying the reliability of our approach. Next, there are two lines of literature for subgroup identification: (1)post-hoc analyses using previously collected data, and (2)adaptive data collection through randomized experiments. Lee (2014) and Xu (2023) align with the first one, whereas our method focuses on the second. Lee et al. (2014) use clustering techniques based on data from prior randomized controlled trials, and Xu et al. (2023) identify subgroups from existing observational data using machine learning, both differing from our adaptive experiment setting.

2. In classical CARA design (so does our design), patient covariates are used to define subgroups. Concretely, denote the covariate as $X_{it}$ and assume the covariate space $\mathcal{X}$ is partitioned into $m$ regions: $\mathcal{S}_{j=1}^m$ (lines 98-100). CARA designs differ from RAR as RAR does not consider covariates at all, optimizing treatment allocation solely based on past treatment assignments and outcomes. This oversight can lead to less effective treatment strategies by ignoring valuable patient-specific information that could significantly influence treatment responses. We hope to note that in response to your Q7, we refined our approach using the augmented Inverse Probability Weighting estimator, allowing us to incorporate other covariates that are not used to define subgroups, enabling us to adjust treatment allocation accordingly.

3.We apologize for missing the definition of $\hat{p}_{tl}$. It is defined as $\hat{p}\_{tl}=\frac{\sum\_{s=1}^{t}\sum\_{i=1}^{n\_s}\mathbb{1}\_{(X\_{is}\in\mathcal{S}\_{(l)})}}{\sum\_{s=1}^{t}n_s}$. ${B}\_{t,b}$ is defined in Eq.6 in preparation for the calculation of the loss function in Eq.7. Additionally, $\Delta=\min(1,R\cdot(\frac{\sqrt{\sum\_{j=1}^{m}n\_{tj}\hat{\mathbb{V}}\_{tj}/m}}{\sqrt{\sum\_{j=1}^{m}\hat{\mathbb{V}}\_{tj}/n}})^{2*\gamma})\approx \min(1,R\cdot n^{\gamma})$, where $\gamma$ is a small tuning parameter to ensure $\Delta<1$. We choose $\gamma=0.05$ and show that our procedure is not sensitive to the tuning parameter(Table 1 in the attached pdf).

4. Due to the space limit in our first submission, we neglected technical details to justify Eq.1. In fact, the rate function in the optimization problem's objective is a monotone transformation of the correct identification probability in an asymptotic sense. Below is a brief derivation due to the space limit: $\lim\_{N\rightarrow\infty}\frac{1}{N}\log(1-\mathbb{P}( \hat{\tau}\_{\mathcal{T}\_1}\geq\max\_{j \notin\mathcal{T}\_1}\hat{\tau}\_j))=-\min\_{j\notin \mathcal{T}\_1}G(\mathcal{S}\_1,\mathcal{S}\_j;e\_1,e\_j)$. We are happy to provide more details if you raise additional concerns.

5. After inspecting our proof, we believe that our proof in the three places you pointed to us is correct. $\tau\_{t,l}=\tau\_l$ naturally holds by Assumption 1. This is because potential outcomes are independently identically distributed for $i=1,\ldots,n_t$, $t=1,\ldots,T$, implying $\tau\_{t,l}=\tau\_l$ in the proof of Theorem 1. This i.i.d assumption is commonly assumed in adaptive design literature. With the analysis in LN 451-453, we have $\lim\_{N\rightarrow\infty}\mathbb{P}(N^{\delta}|\tau\_{j}-\tau\_{\check{1}}|<C)=0$ and thus $\lim\_{N\rightarrow\infty}\mathbb{P}( N^{\delta}|\tau\_{j}-\tau\_{\check{1}}|+C< 2C)=0$. By Lemma 2 in the Appendix, we obtain $\lim\_{N\rightarrow\infty}\mathbb{P}(N^{\delta}|\hat{\tau}\_{tj}^*-\tau\_{j}| +N^{\delta}|\hat{\tau}\_{t,\check{1}}^*-\tau\_{\check{1}}|\geq 2C)=0$. Then we reach the conclusion in Eq.13. Similarly, with the analysis in LN 458-461, we have $\lim\_{N\rightarrow\infty}\mathbb{P}( N^{\frac{1}{2}}|\tau\_{j}-\tau\_{\check{1}}|<C)=1$. We also derive

$\lim\_{N\rightarrow\infty}\mathbb{P}(N^{\delta}|\hat{\tau}\_{tj}^*-\tau\_{j}|<C)=1$ and $\lim\_{N\rightarrow\infty}\mathbb{P}(N^{\delta}|\hat{\tau }\_{t,\check{1}}^*-\tau\_{\check{1}}|<C)=1$. With $\delta<\frac{1}{2}$, we establish the result in LN 455.

6. We have added an additional case study using the National Supported Work program data(Figure 1 in the attached pdf).

7. For the comparison with various contextual MAB algorithms, we now extend our proposed design to an augmented inverse propensity score weighting (AIPW) estimator incorporating contextual information (Figure 2 in the attached PDF). For the causal tree, the comparison may not be entirely fair since the causal tree identifies subgroups after data collection, whereas our method designs the data collection mechanism to identify the best subgroups accurately. Thus, our approach is expected to have higher accuracy in identifying subgroups. We have provided a comparison (Figure 2 in the attached PDF). The performance of the causal tree model is similar to contextual bandit algorithms, and our proposed algorithm has the highest probability of identifying subgroups.

8. The purity score is similar to correct selection probability, as both measure accuracy: purity score assesses how well clusters contain a single class, while correct selection probability evaluates identifying the best subgroups. Both range from 0 to 1, with higher values indicating better performance. Figure 1(c) (d) in the attached PDF compares the normalized mutual information for our proposed design and three competing methods.

Per your suggestions, we will update our manuscript as follows:

• Add a more thorough literature review on posthoc subgroup analysis
• Add a notation table
• Add additional simulation results, including contextual bandit, AIPW estimator, and normalized mutual information as an additional metric

The second comment below addresses your second concern:

Tuning parameter: We are sorry that the previous presentation of our procedure might have caused some confusion. In the following, we would like to clarify that our bootstrap procedure is not intended to select $\gamma$, and both $\gamma$ and $\Delta$ are only adopted to select the hyperparameter pair $(c\_`L`,c\_`R`)$, which determines the neighborhood for merging subgroups.

First, we will replace lines 159-170 with the following in our revision to clarify our procedure:

• Dynamic identification of the best subgroups (Algorithm 2):The dynamic subgroup identification algorithm involves a resampling step at each stage. Specifically, at each stage $t$, we generate bootstrap samples $\hat{\mathbf{\tau }}\_t^{\circ}$ from a Gaussian distribution centering around $\hat{\mathbf{\tau }}\_t$, which is estimated using the data collected up to stage $t$ using Eq (8). We then identify the best subgroups at Stage $t$ using

$

\hat{\mathcal{T}}_{t1} = \{k: w_{k,(1) }^{\circ} = 1,k = 1,\ldots,m\}, \ (5)

$where$w_{k,(1)}^{\circ}=\mathbb{1}\{-c^t_{L}\cdot N_t^{-\delta}\cdot \hat{\mathbb{V}}_{t,( 1)}^{\delta}\leqslant (\hat{\tau}_{tk}^{\circ}-\hat{\tau}_{t,(1) }^{\circ})\leqslant c^t_{R}\cdot N_t^{-\delta}\cdot \hat{\mathbb{V}}_{t,(1)}^{\delta }\}$.$N_t = \sum_{s=1}^t n_s$and$\delta = 0.25$. Note that the above formulation relies on a pair of hyperparameters$(c^t_{L},c^t_{R})$, which are selected data-adaptively. In what follows, we shall illustrate the algorithm for selecting hyperparameters.

• Hyperparameter selection (Algorithm 3): In line 7 of Algorithm 3, we adopt a bootstrap method and propose several alternative bootstrap methods in Algorithm 3(line 7) in Appendix (Section H).Algorithm 3 involves a resampling step that generates bootstrap samples $\hat{\mathbf{\tau}}\_t^*$ from a Gaussian distribution centering around $\mathbf{\tau}\_t^*$ at Stage 1. In line 2, we compute $\mathbf{\tau}\_t^*=(\tau\_{t1}^*,\ldots,\tau\_{tm\_t^*}^{* })^{\prime}$ as $\tau\_{tj}^*=\Delta\_t \cdot \frac{\sum\_{j=1}^{m\_t^*}\hat\tau\_{tj}}{m\_t^*}+(1-\Delta\_t) \cdot \hat\tau\_{tj}, \ j = 1,\ldots,m, \ (3)$ where $\Delta\_t =\min\\{0.99,\frac{\sum\_{j=1}^{m\_t^*}\hat{\mathbb{V}}\_{tj}}{ N\_t\sum\_{j=1}^{m\_t^*}(\hat{\tau}\_{tj}-\overline{\hat{\tau}}\_t)^{2}}\times N\_t^{\gamma}\\}$ and $\gamma \in (0,0.2)$. We choose $\gamma = 0.05$ in our simulation studies, and our procedure is shown not sensitive to $\gamma <1$. In line 8, we compute $\hat{\mathbf{\tau}}\_t^*=(\hat{\tau}\_{t1}^*,\ldots,\hat{\tau}\_{tm\_t^*}^*)^{\prime}$ at Stage $t$ for $t>1$ as

$

\hat{\tau}_{tj}^=\Delta_t \cdot \frac{\sum_{j=1}^{m_t^}\hat{\tau}_{tj}^{\circ}}{m_t^*}+(1-\Delta_t) \cdot \hat{\tau }_{tj}^{\circ}, \ (4)

$where$\hat{\tau}_{tj}^{\circ}$ is computed with the bootstrap samples as in Eq (8).

Through this revision, we hope to clarify that

• The final tie set is selected using Eq (5), not depending on $\Delta$ and $\gamma$.
• You are absolutely correct that $\Delta$ cannot be 1; we will revise our algorithm to put a lower bound of 0.99. Additionally, there was a typo in Table 1; the magnitude of $\gamma$ we have tested is actually from $0.00$ to $0.10$. We hope to provide additional simulation results below. Due to time limit, we are only able to provide results for reduced sample size with $n_t = 400$, $T=4$ and reduced $B=400$ on the correct selection probability, Monte Carlo bias, and variance for a various choice of $\gamma$:
  • $\gamma = 0.05$, CSP: $0.68, 0.81, 0.85, 0.86$; $\sqrt{N}$Bias: $35.20$; SD: $44.71$;
  • $\gamma = 0.10$, CSP: $0.68, 0.81, 0.85, 0.86$; $\sqrt{N}$Bias: $35.20$; SD: $44.71$;
  • $\gamma = 0.15$,CSP: $0.67, 0.81, 0.86, 0.86$; $\sqrt{N}$Bias: $35.21$; SD: $44.72$;
  • $\gamma = 0.20$, CSP: $0.68, 0.81, 0.86, 0.86$; $\sqrt{N}$Bias: $34.42$; SD: $44.75$.

The performance of our method in this response may not be as strong as the results in the submitted manuscript due to time constraints in addressing your concerns. The results still demonstrate that our method is not sensitive to the choice of $\gamma$.

We hope this revision of our manuscript will address your concerns!

• Thank you for your encouragement. Below, we provide our understanding of why adaptive designs can be preferable in identifying the best subgroups compared to other alternatives, why resampling and bootstrapping are critical in our procedure, and what technical challenges bootstrapping addresses.

  • To illustrate why adaptive designs can be preferable in identifying the best subgroups, we first review the literature on identifying subgroups and put our method into perspective. There are two lines of literature: The first line conducts a post-hoc analysis of already collected data from previous studies, while the second line focuses on adaptively collecting new data via randomized experiments to ascertain the best subgroup. Therefore, designing an adaptive data collection mechanism is the focus of the second line of literature, and our method aligns with the second line. We follow the second line for subgroup identification because While the first line of literature adopts post hoc subgroup analyses that do not require new data collection, they rely on untestable causal assumptions, limiting the credibility of causal conclusions (e.g., the unconfoundedness assumption is necessary for causal inference in observational studies, but unmeasured confounders can compromise these conclusions). Conversely, in randomized experiments, valid causal conclusions do not depend on such assumptions, but analysis of subgroup treatment effects may still face biases like the winner's curse, mainly if heterogeneous effects are selected from the data in an ad hoc manner (Guo and He, 2021;Andrews et al., 2019). Our approach aligns with the second line of literature and involves collecting data through randomized experiments to identify subgroups that benefit most from the treatment accurately. This approach is preferable because it avoids the limitations of analyzing existing data, which is the case with first-line approaches. It ensures that treatments are randomly assigned, thus eliminating the influence of unmeasured confounders and allowing for robust, valid causal inferences without the need for untestable assumptions—distinctly superior to using observational data. Moreover, our adaptive design method permits iterative adjustments based on new information, enhancing trial flexibility and efficiency. We also hope to gently note that we are not deliberately constructing a complex adaptive experimental design strategy. Given randomized experiments can be time-consuming, costly, and potentially result in adverse patient outcomes if unsuccessful, our approach aims to efficiently allocate experimental resources within a limited budget to identify the most beneficial subgroup.

  • Bootstrap and resampling-based methods play a crucial role when dealing with subgroups that have tied effect sizes in the population. To shed light on this issue, we consider a more straightforward scenario without adaptive data collection. In this simplified scenario, our objective is to identify the subgroup clusters (referred to as "tie sets" in our paper) and rank these clusters according to their average effect sizes with statistical confidence. Suppose we have in total $p$ subgroups forming $m$ clusters, and their population effect sizes and the averaged cluster effect sizes are $\underbrace{\beta\_1= \ldots =\beta\_k}\_{\text{Cluster 1}}, \beta\_{k+1}, \ldots, \beta\_{l-1},\underbrace{\beta\_{l} = \ldots = \beta\_{p}}\_{\text{Cluster m}}$, $\alpha\_{j} = \sum\_{l\in \text{Cluster j}} w\_l \beta\_l, \sum\_{l\in \text{Cluster j}} w\_l = 1, j = 1, \ldots, m$. Then, the bootstrap procedure proposed in our method helps us to achieve the following two goals simultaneously: Provide valid statistical inference (confidence intervals and consistent point estimates) on the ``ordered" averaged cluster effect sizes, that are $\alpha\_{(1)},\alpha\_{(2)}, \ldots, \alpha\_{(m)}$; Identify the clusters with a high probability. While numerous methods exist for subject clustering, providing confidence intervals for their ordered estimated mean effect sizes is a challenging task due to the winner's curse bias. This issue is well-known in the existing literature, where constructing confidence intervals for order statistics is particularly difficult. Therefore, our bootstrap procedure is important in identifying the tie set while delivering valid statistical inference on the identified best set of subgroups.

• Thank you for your question. When there are two arms with the same treatment effects but different variances, our algorithm can effectively manage the allocation efforts, avoiding the scenario when most efforts are allocated to the arm with zero variance. We shall illustrate this in a two-stage adaptive experiment. At the start of Stage 1, we adopt the same treatment allocation for all subgroups. At the end of Stage 1, if there are two arms (or subgroups) in $\mathcal{T}\_1$ with similar treatment effects but one with zero variance and the other with large variance, our algorithm will be able to identify these two competing subgroups and then merge these two subgroups into the best "set" $\hat{\mathcal{T}}\_1$. In Stage 2, since both of these two subgroups belong to $\hat{\mathcal{T}}\_1$, they will be assigned the same treatment allocation, denoted as $\hat{e}\_1^*$. Additionally, Theorem 1 in our manuscript provides an analysis of the probability of successfully identifying $\mathcal{T}\_1$. We show that as the sample size tends to infinity, we can always correctly identify the best set of subgroups. Therefore, we can guarantee the identification of everything in $\mathcal{T}\_1$ and ensure that nothing out of $\mathcal{T}\_1$ is included.

Reference:

• Andrews, I., Kitagawa, T., and McCloskey, A. (2019). Inference on winners. Technical report, National Bureau of Economic Research
• Guo, X. and He, X. (2021). Inference on selected subgroups in clinical trials. Journal of the American Statistical Association, 116(535):1498–1506.