e-GAI: e-value-based Generalized $\alpha$-Investing for Online False Discovery Rate Control

Thank you for your helpful comments and suggestions. We would like to part-wisely respond to your comments.

In financial bubble detection, ω1 is set to 0.0001 without justification.

This application can be seen as a long-term time series (i.e., $T=10000$). As response to your W1&W2, we suggest choosing $\omega_1$ as $1/T$ approximately to avoid spending too much wealth in the early stages and resulting in $\alpha$-death.

The paper does not evaluate runtime or memory consumption...

We would like to make the following clarification.

• The update of $\omega_t$ and $\alpha_t$ can be expressed in a recursive form, so computation is highly efficient and memory-friendly. Specifically, updating $\omega_t$ is shown in (9) (line 195). To compute $\alpha_{t}$ for e-LORD (lines 165-177), we define remaining wealth $r_t^{\text{e-LORD}}=\alpha-\sum_{j=1}^{t}\frac{\alpha_j}{R_{j-1}+1}$ and update $\alpha_t=\omega_t r_t^{\operatorname{e-LORD}}(R_{t-1}+1)$. Similar recursive $\alpha_t$ for e-SAFFRON can be obtained.
• We also evaluated runtimes of various algorithms in Table 2 below. It shows that e-LORD and e-SAFFRON are computationally fast. We will add the recursive update form and present results in final version.

W1: The user is required to set the initial parameters (e.g., ω₁, λ)...

Thank you for your insightful comments! We would provide some suggestions about $\omega_1$ and $\lambda$.

• In general, a larger $\omega_1$ means more wealth will be assigned to each hypothesis testing, making it easier to discover alternatives, and meanwhile, it will be more possible to exhaust entile wealth at an early stage. Hence, we recommend a relatively small $\omega_1$ for a long period, e.g., $\omega_1\approx 1/T$ empirically.
• $\lambda$ affects initial wealth and the proportion of hypotheses counted into FDP's estimator. We recommend $\lambda=0.1$ to preserve wealth as discussed in Remark 3.5.
• We have simulation results with different ($\omega_1$,$\lambda$) in Appendix C.1. Additional results of powers of e-LORD and e-SAFFRON ($\lambda>0$) under an $\operatorname{AR}(1)$ model are shown in Table 1 below. All FDRs are controlled and omitted due to the space limit. We can see the choices of ($\omega_1,\lambda$) do not affect results too much. So, it enables users to integrate domain knowledge in real applications.

Table 1: Power (%) with different ($\omega_1,\lambda$) and $\alpha=5$%.

W2: Long-term performance of algorithm has not been analyzed. ...

As noted, excessive early-stage wealth consumption leads to "$\alpha$-death", halting rejections once $\alpha$-wealth is zero. It is a common phenomenon within GAI framework. So, we recommend using a small $\omega_1$ to preserve wealth as response to your W1.

This issue warrants further research, such as considering "activation" after reaching a certain condition or exploring scenarios where online sequence inherently exhibits block structure, among other possibilities. In addition, this work focuses on measuring the cumulative error by FDR. Exploring other metrics is also a valuable future direction.

Q1: ...what if there are some time-varying correlations...

We design a new time-varying $\text{AR}(1)$ model: $X_t=\rho_t X_{t-1}+\epsilon_t$ for $H_0$ and $X_t=4+\rho_t X_{t-1}+\epsilon_t$ for $H_1$ with $\rho_t=\frac{2}{1+\operatorname{exp}(-0.01(t-T/2))}-1$. The table below shows that e-LORD and e-SAFFRON control FDR while leading to relatively high power.

Table 2: FDR, power, and runtime with $\alpha=1$%. SupLORD is referenced by Reviewer k3m9.

Q2: How to consider long-term performance?

Here we focus on controlling online FDR, a common criterion in online multiple testing. To alleviate "$\alpha$-death" over a long period (potentially infinite), Ramdas et al. (2017) discussed controlling decaying memory FDR under independence. But, how to control it under dependence remains challenging due to additional complexity introduced by these dependencies. As response to your W1, another potential way is to consider other strategies or specific structures. We will include a careful discussion of long-term issues in a future revision per your suggestions. Thank you!

We appreciate your careful reading and constructive suggestions. Per your comments, we would like to clarify key contributions of our work as follows:

1. We propose the e-GAI framework for online testing with using e-values, which achieves FDR control under arbitrary dependence among valid e-values (i.e., satisfying condition (2)).
2. We designed a new e-value-based FDP estimator and provided theoretical guarantees for online FDR control building on it under the relevant conditions.
3. Within the e-GAI framework, we developed a new updating approach for allocating testing levels from a risk aversion perspective, aiming to save budget while achieving high power. Along this line, we proposed two new algorithms (e-LORD and e-SAFFRON) for implementation.

Thanks for your comments on our paper. We would like to part-wisely respond to your comments.

This claim (and variants) are central thrust of the paper and repeated throughout...

We sincerely apologize for any confusion. As you have pointed out, by "arbitrary dependence", we mean allowing any dependence structure among test statistics for each hypothesis, i.e., online conditionally valid p-values/e-values. In the revision, we will refine descriptions to minimize confusion. Further details on the experimental design will be clarified in the following. Thank you!

I don't believe LORD++ and e-LORD have any equivalence (nor SAFFRON and e-SAFFRON). ...

Thank you for your insightful comments. We agree with you that e-GAI is a different framework from GAI. The "equivalence" in Section 4 was to explore the connection between LORD++ and e-LORD when independent p-values are available. In such cases, we can design testing algorithms using a less conservative FDP estimator, i.e., $\sum_{j = 1}^t \alpha_j / (R_t \vee 1)$ such as the one in Theorem 4.1, which will coincide LORD++ and e-LORD. Previously, for notational simplicity, Theorem 4.1 did not explicitly highlight the change in the FDP estimator. The final version will revise the discussion in Section 4 for clarity and precision. Hope these interpretations can ease your doubts!

As referred to before, the design of the numerical simulation is unclear. ...

Thank you for pointing this out! We apologize for the unclear setup description. The covariance matrix assumes positive correlations between all elements, including nulls and alternatives. In simulations, we assessed our methods using oracle p-values/e-values, calculated from the known conditional distributions derived from the normal distribution. Furthermore, we conducted additional experiments by designing other dependence structures, and these results will be added into the future revision. For details, please see our response to Reviewer kFoC's Q1 due to the space limitation.

The real data experiments are lacking critical details. ...

We appreciate your feedback and offer the following clarification. In Section 5.2, the data is assumed to be independent and normally distributed under $H_0$ as discussed in (Ahmad et al., 2017), using p-values derived from the normal distribution. We employed KDE to estimate $f_t(x)$ and $f_0(x)$, taking their ratio as the e-value. Due to the independence assumption, these e-values are valid.

In Section 5.3, following [1] modeling framework for financial bubbles, we characterized the problem as a right-tailed unit root test in an $\text{AR}(1)$ model. We built valid e-values on normalized likelihood ratios by dividing its conditional expectation under $H_0$. Both this likelihood ratio and p-values are proposed by Dickey & Fuller's series works. We will clarify these details in the future revision.

Two References Not Discussed

Thank you for your recommendation! We have introduced and cited Fisher (2024) in Section 2.2 (lines 126-129 of right column) in the main text. Additionally, we include a comparison with the SupLORD algorithm (Xu & Ramdas, 2022) in Table 2 of our response to Reviewer kFoC's Q1 due to space constraints. In the new setting, SupLORD controls FDR but has low power, whereas our methods achieve FDR control with higher power. In future revisions, we will incorporate Xu & Ramdas (2022) and add comparative experiments.

Looking at the proof in Appendix A.1, is there anything in particular that is specific to e-values? ...

The proof in Appendix A.1 does indeed rely on a unique property of e-values: the larger the e-value, the more significant the evidence against the null. Note that $\delta_t=\mathbb{I}[e_t\geq 1/\alpha_t]\leq e_t\alpha_t$ deterministicly in the numerator of inequality (ii) of the proof. This, however, does not hold for p-values. Hence, e-GAI cannot theoretically guarantee FDR control when directly applied to p-values satisfying condition (1).

Ref:

[1] Phillips, P. C., & Yu, J. Dating the timeline of financial bubbles during the subprime crisis. Quantitative Economics, 2(3), 455-491, 2011.

Thanks for your comments on our paper. We would like to part-wisely respond to your comments.

First, the empirical argument would have been stronger in Section 5.1 with a simulated scenario that caused LORD++ to exceed the desired FDR bounds.

Thank you for your valuable suggestion! We have to acknowledge that we explored various simulation settings and did not get the results that LORD++ exceeds the target FDR level for now. This is mainly because the LORD++ algorithm itself is inherently conservative. However, we found that LORD++ does not perform well with $\widehat{\text{FDP}}=0.189$ in the real data analysis in Section 5.2. As we know, LORD++ lacks reliable theoretical guarantees with the dependence of p-values, which may compromise its safety and reliability in practical applications.

Section 2.2, it is stated that the "key idea in GAI...is that each rejection gains some extra $\alpha$-wealth", shouldn't this be the opposite, that accepting the null gains wealth, and rejections spend it?

We would like to make the following clarifications.

• The GAI algorithm is designed from an investment perspective: each hypothesis test is treated as an investment, where the significance level $\alpha_t$ is drawn from a wealth pool $W_t$. Thus, each test consumes a certain amount of "wealth", and if the test results in a rejection, it is considered a successful investment, thereby gaining additional "wealth" and increasing the cumulative wealth.
• In contrast, our e-GAI framework adopts a different design philosophy: e-GAI views the entire testing process as a risky investment and the whole wealth does not increase; instead, each test consumes a fixed proportion of the remaining wealth (e.g., $\omega_t$ in e-LORD). Therefore, each rejection introduces risk, and from a risk-averse perspective, it is necessary to reduce the investment proportion in subsequent tests to ensure long-term sustainability.
• The difference in design philosophy between e-GAI and GAI stems from the distinct estimators used for the FDP. For the GAI method, which allows for wealth accumulation, increased investment is advantageous. In contrast, for the e-GAI method, where total wealth cannot increase, each investment inherently entails risk.

I get the math, but had trouble following the conceptual explanation below Theorem 3.1, ... A little more explanation would have been helpful here.

Thank you for your suggestion! As you noted, the key role of $R_{j-1}+1$ here is that it serves as a $(j-1)$-measurable lower bound for $R_t\vee1$. Since $R_{j-1}+1\leq R_t\vee 1$ and it is placed in the denominator, it results in an overestimation of the oracle FDP. As the true $R_t\vee1$ is unobservable at time $j-1$, we use $R_{j-1}+1$ as a substitute, effectively "predicting" the value at time $t$ from the perspective of time $j-1$. This is why we used the verb "predict" in this context. Per your comment, we will provide a more detailed explanation in the final version to make this point clear.

The PRDS condition is mentioned in several places in the introduction / Section 2 (see page 1, "p-values with...positive regression dependence on a subset") but is not formally defined, might have been nice to at least include in the appendix for completeness.

We appreciate your attention to detail and your valuable feedback. We will include a formal definition of PRDS and relevant references in the appendix of the final version.

It may be worth taking an editing pass for grammar, e.g., Line 092, left-hand column ...

Thank you for pointing this out! We will carefully review the grammar throughout the paper and make the necessary corrections, including the sentence you highlighted.

Could the authors clarify how their results in Theorem 3.1 differ from those in the e-LOND paper? I didn't see this exact result anywhere in that paper on a very brief skim, but I imagine they need to have used some similar result to also achieve FDR control with arbitrary dependence?

Theorem 3.1 introduces an estimator for the FDP and proves that FDR control can be achieved when the expectation of this estimator does not exceed $\alpha$. Based on this key estimator, we subsequently propose algorithms for designing testing levels (e-LORD and e-SAFFRON). In contrast, the e-LOND algorithm follows a different approach. Specifically, e-LOND does not introduce an FDP estimator; instead, inspired by the LOND algorithm [1], it defines the testing levels $\alpha_t$ by directly leveraging a decay sequence that sums to 1 and the number of rejections. This approach allows for the cancellation of a common term in the numerator and denominator of the FDP, thereby achieving FDR control.

Ref: [1] Zrnic, T., Ramdas, A., and Jordan, M. I. Asynchronous online testing of multiple hypotheses. Journal of Machine Learning Research, 22(33), 1-39, 2021.

Thanks for your comments on our paper. We would like to part-wisely respond to your comments.

One small note is that maybe Figure 2 could benefit from some improvements in its display as currently the squares and dots are quite small and subtle (the latter could be fixed by different coloring I think).

Thank you for your advice! We have improved Figure 2 by increasing the size of the squares and dots and adjusting the coloring for better contrast. These changes will present in the future revision.

A potential weakness is that the paper doesn't go into too much depth on devising new methods, and mainly instantiates LORD and SAFFRON in their own framework.

Thank you for your comments! Our work aims to address the challenge of online FDR control under arbitrary dependence. To overcome the limitations of p-values in this context, we propose a novel framework leveraging e-values for hypothesis testing, along with two concrete and implementable algorithms, i.e., e-LORD and e-SAFFRON. Furthermore, under the independence assumption, we establish connections between our framework and existing methods, such as LORD++ and SAFFRON from the GAI framework. We have named our approach e-GAI, which reflects the key method is to build a new GAI framework with the use of e-values under arbitrary dependence.

One general small personal suggestion would be to not center the A/B testing for tech companies applications so much in the abstract and introduction, and instead showcase more exciting (possibly for social good) applications.

Thank you for this suggestion! We will revise the abstract and introduction to emphasize additional use cases, such as real-time monitoring of machine operation status in industrial settings. Indeed, we have applied the e-GAI algorithm to one relevant real example in Appendix C.2. We appreciate that this will better showcase the potential benefits of our framework.

Typos: add “The” before “e-LOND” in line 19, left column; add “the" before “GAI” in line 70 right column; “guarantee” should be “guarantees” in line 417 right hand side

Thank you for the careful reading! We will correct these typos in the final version.