Strategic A/B testing via Maximum Probability-driven Two-armed Bandit

We sincerely appreciate the reviewer’s thoughtful feedback and valuable suggestions. We also sincerely apologize for any difficulties or confusion arising from insufficient clarity in the presentation of some fundamental equations and references in the paper. Below, we will discuss the issues you have raised：

1. Effectiveness of the Proposed Statistic

• Notation Clarification:

We define $\bar{R}_n^{(\vartheta_i)}=\bar{R}_n^{(1)}\mathbb{I}(\vartheta_i=1)+\bar{R}_n^{(0)}\mathbb{I}(\vartheta_i=0)$, where $\bar{R}_n^{(1)}=\sum\_{i=1}^nR_i^{(1)}/n=-\bar{R}^{(0)}_n$ and $\mathbb{I}(\cdot)$ denotes the indicator function. Since our paper does not introduce or define the notation $R_j^{(\vartheta_i)}$ for $i \neq j$, we will explicitly define $\bar{R}_n^{(\vartheta_i)}$ as stated above, immediately following Equation (2).

• Intuitive Explanation and Theoretical Perspective:

Using $\sum\_{i=1}^n\bar{R}\_n^{(\vartheta_i)}/n$ instead of $\sum\_{i=1}^n R_i^{(\vartheta_i)}/n$ reduces the variance of the mean term while preserving the asymptotic properties of the statistic. This leads to a faster and more stable convergence, as also verified experimentally. Replacing $R_i^{(\vartheta_i)}$ with $\bar{R}_n^{(1)}\mathbb{I}(\vartheta_i=1)+\bar{R}_n^{(0)}\mathbb{I}(\vartheta_i=0)$ represents a key methodological improvement.

2. Motivation for Linking to the Two-Armed Bandit Framework

• Differences in Research Objectives:

The classical two-armed bandit model focuses on maximizing the average reward by balancing exploration and exploitation. In contrast, our model does not seek to identify the arm with the highest return. Instead, it aims to maximize the target probability through collaborative arm selection. Within the hypothesis testing framework, this maximized target probability corresponds to the tail probability $\mathbb{P}(|T\_{n, \lambda}(\theta_n)|>z_{1-\alpha/2}|\mathcal H_1)$.

• Clarification in Writing:

Given the distinct objectives of our proposed model compared to the classical framework, we appreciate the opportunity to improve clarity in our manuscript. To this end, we will explicitly define our proposed model following the second paragraph of Section 2.2, thereby clearly distinguishing it from the classical model.

3. Missing References

We sincerely apologize for the unintentional omission of the reference to “Z. Chen et al., Strategic Two-Sample Test via the Two-Armed Bandit Process” during the drafting process. This reference has been included in the revised manuscript along with a detailed comparison:

• Research Background:

Chen et al. addressed the one-sided two-sample testing problem under independent, batch-wise paired observations. While valuable, their approach is not designed to address hypothesis testing in more complex frameworks, such as causal inference scenarios involving missing data or confounding variables. In contrast, our proposed method incorporates advanced techniques, enabling its application to hypothesis testing within causal inference frameworks. This adaptability enhances its relevance to real-world research contexts.

• Theoretical Contributions:

Chen et al.’s study primarily focuses on the asymptotic properties of the test statistic, without investigating its behavior in finite samples. This limitation can result in inflated Type I errors in small-sample settings. We effectively control the Type I error under finite samples by modifying the mean term as the mean of $\bar{R}\_n^{(\vartheta_i)}$ and introducing a weighting factor $\lambda$. Their result represents a special case of our framework when $\lambda=0.5$.

• Algorithm Robustness:

Chen et al. compute the $p$-value from a single ordered sample sequence, resulting in unstable statistical power in both simulations and real-world experiments. To enhance robustness, we introduce a permutation-based meta-analysis, recalculating the $p$-value across multiple sample reorderings. This improvement significantly strengthens the algorithm’s reliability and practical utility.

In summary, while our work draws inspiration from “Strategic Two-Sample Test via the Two-Armed Bandit Process” in its use of bandit strategies for hypothesis testing, our study offers more generalized insights across research frameworks, theoretical advancements, and technical implementations. We hope this clarification underscores the distinct contributions of our work while acknowledging the foundational influence of Chen et al.’s research.

We sincerely appreciate the reviewer’s keen observation, which has enabled us to refine the manuscript and better contextualize our contributions within the existing literature.

We sincerely thank the reviewer for the comprehensive and insightful feedback. Below, we provide a point-by-point response to the issues raised:

1. The Use of “Exchangeability”

We appreciate the reviewer’s observation regarding the term “exchangeability” and fully agree that a clear definition is essential. In the revised manuscript, we have explicitly defined exchangeability here as referring to the rewards derived from the sequence of arm choices.

2. Presentation Adjustments

• Two-Arm Bandit Introduction:

We acknowledge the suggestion regarding the introduction of the full term “Two-Armed Bandit” before using its acronym. In the revised version, we ensure clarity by introducing the full term at first mention (e.g., “Two-Armed Bandit (TAB)”).

• Typo Corrections:

Line 193: We have corrected the typo from “mean” to “main”.

Line 410: We have changed the paragraph title from “Another simulations” to “More ML-based simulation studies” for clearer expression.

• Figure Improvements:

In Figure 2(b), we will add a detailed legend to describe each curve, ensuring that the visual representation is self-explanatory. The pink dashed, cyan solid, and orange dotted lines represent $\sigma=0.5, \sigma=0.6$, and $\sigma=1.0$, respectively. Additionally, we will update the caption of Figure 2(b) to “The empirical type I error rate across different $\lambda$ and $\sigma$, fixed $n=20000$”.

• Section Transitions and Introductions:

We agree that additional contextual transitions at the beginning of sections would improve readability. We will include brief overviews in Section 3.1 and elsewhere to better guide the reader through our arguments and experimental results.

• Lemma 2.1 Clarification:

We have revised Lemma 2.1 to explicitly define the alternative hypothesis $\mathcal{H}_1$ before applying it, thereby eliminating potential ambiguity.

3. Theoretical and Experimental Clarifications

Regarding the reviewer’s question about Lemma 2.1, its proof is provided in Chen et al.'s article (Z. Chen et al., “Strategy-driven limit theorems associated bandit problems,” Theorem 3.3). To improve the readability of our manuscript, we will include the full proof in the appendix of the revised version.

4. Ensemble Method (Stacking) Performance

We thank the reviewer for highlighting the performance discrepancies of the stacking method in the synthetic experiments. We identify two key factors contributing to this discrepancy:

• First, the current implementation uses a limited selection of primary learners. We are actively investigating the incorporation of additional machine learning models as primary learners to enhance the performance of the stacking method.

• Second, the choice of primary learners and their respective weights in the ensemble may not be optimal under all configurations, leading to suboptimal aggregation of predictions. We are exploring the use of more advanced meta-learners (e.g., random forests) instead of simple linear regression to better assign weights to different primary learners and further improve the stacking method’s performance.

5. Additional Evaluations on Real-World Data

We appreciate the reviewer’s suggestion regarding a more comprehensive evaluation on real-world data. To address this, we have conducted additional experiments using synthetic data based on real-world data. The results of these additional experiments are summarized in Table 1.

Table 1: Type I error rates and statistical power based on synthetic data derived from real-world dataset.

These results provide compelling evidence of the effectiveness of our proposed PWTAB method in real-world scenarios. When the null hypothesis holds, all methods maintain Type I error rates close to 0.05, preserving the reliability of statistical inference in practical settings. Under the alternative hypothesis, the proposed method consistently outperforms competing methods in terms of statistical power. PWTAB achieves the highest statistical power when used with LightGBM or XGBoost, and its performance is further enhanced when combined with the ensemble learning algorithm Stacking.

We sincerely appreciate the reviewer’s constructive comments, which have been invaluable in improving the clarity, rigor, and overall impact of our work.

We sincerely appreciate the reviewer’s careful evaluation of our work and the constructive feedback provided. Below, we address the key weaknesses and questions raised:

1. Adding a More Intuitive Explanation of Mathematical Densities

We thank the reviewer for this valuable suggestion. We acknowledge that the extensive mathematical derivations may pose a barrier to practitioners with limited statistical backgrounds. In the revised version, we have incorporated an intuitive explanation section to explain the form of different probability densities under different hypotheses. We believe this additional exposition will facilitate a broader understanding and practical application of our method.

To illustrate, consider the case when the null hypothesis holds with $\lambda$ fixed. Given that the optimal policy parameter $\vartheta_1^*$ has an equal probability of being 0 or 1, assume that $R_1^{(1)}$ is observed and that $T\_{1, \lambda}(\theta_1^*)\ge 0$. Consequently, according to the optimal policy, $\vartheta_2^*=1$, implying that $R_2^{(1)}$ will be observed. This process continues with $\vartheta_i^*=1$ until there exists some index $m$ such that $T\_{m, \lambda}(\theta_m^*)<0$. Under the assumption that the null hypothesis holds, it is likely that $T\_{2, \lambda}(\theta_2^*)<0$, resulting in $\vartheta_3^*=0$, which leads to the observation of $R_3^{(0)}$, a reward that is more likely to exceed 0. This brief discussion shows that the optimal policy $\theta_n^*$ will control the value of $T\_{n, \lambda}(\theta_n^*)$ to fluctuate around 0 under the null hypothesis, thereby concentrating its distribution around 0. A similar rationale applies when the alternative hypothesis holds.

2. Guidance on $\lambda$ Selection

We agree that the guidance on selecting $\lambda$ is crucial. The threshold value of 0.03 was derived empirically from our synthetic experiments. However, we are actively exploring more data-driven methods for selecting $\lambda$. We propose a data-driven approach for selecting $\lambda$ by first discretizing its range and then employing bootstrapping techniques to generate multiple datasets. For each candidate $\lambda$, we compute the type I error rate across these datasets. The optimal $\lambda$ is chosen as the one that maximizes statistical power while controlling the type I error.

3. More Real-world Applications

We appreciate the reviewer’s insightful query on the generalizability of our method to domains such as e-commerce. Although our current real-world validation is based on ride-sharing data, our preliminary experiments in other domains indicate that the method demonstrates strong potential. We are confident that our approach can be generalized to most companies conducting A/B testing. We intend to extend our experimental evaluation to include additional domains, such as a food delivery company and an internet technology company, thereby providing a more robust demonstration of the method’s versatility and robustness.

4. Breaking Exchangeability

We appreciate the reviewer’s interest in this aspect. We will clarify the advantages of breaking exchangeability in the revised manuscript by explicitly detailing how it contributes to enhanced performance. Traditional hypothesis testing methods based on the Central Limit Theorem (CLT) are inherently data-driven; once i.i.d. samples are observed, the construction of the test statistic is independent of the sample order, implying that the data are exchangeable. In contrast, our proposed testing framework is goal-driven—it seeks to progressively construct the test statistic from the available data to maximize statistical power. In our proposed two-armed bandit framework, earlier data actively influences the construction of the current test statistic, making the data non-exchangeable. This shift toward a maximum-probability objective enables the optimal construction of the test statistic, thereby enhancing testing performance.

Once again, we are grateful for the reviewer’s positive comments and valuable suggestions. We are committed to incorporating these improvements to enhance the clarity, interpretability, and impact of our work.