Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection

Thank you for your insightful comments. Below we address your question point by point:

Claims And Evidence

• Missing Theorem: We will add the following theorem on the statistical validity of aLTT “Given a hyperparameter set $\\Lambda$, a reliability level $\\alpha \\in [0,1]$, and an error level $\\delta \\in [0,1]$, aLTT with an FWER or FDR-controlling selection rule returns a final prediction set $\\hat{\\Lambda}^{{\\rm aLTT},T}$ that satisfies $(\\alpha,\\delta)$-FWER control or $(\\alpha,\\delta)$-FDR control, respectively.” and provide a proof in the Appendix.

• Reliability of selected parameters Models in the predicted set have reliability above $\\alpha$ with a probability at least $1-\\delta$, ensuring valid hyperparameter selection. However, in our experiments, realized performance often exceeds the target reliability ( https://ibb.co/CpGgwVtP ). We agree that reporting the best model’s reliability is insightful and will include it in the revised version.

• Methods And Evaluation Criteria Training an agent online is not always feasible due to safety concerns (e.g., robotic surgery). Alternatively, training with off-policy data requires accounting for covariate shift using propensity scores [1]. Even with known propensity scores, valid performance guarantees must be established via policy rollouts, as done in aLTT.

[1] Uehara M, Shi C, Kallus N. A review of off-policy evaluation in reinforcement learning.

Experimental Designs Or Analyses:

• A single grid search has poor TPR and lacks FWER/FDR control, making it unsuitable for statistically valid hyperparameter selection. As an alternative, we benchmark validation-based selection, which tests hyperparameters on a grid and selects those with empirical risk below $\alpha$. Our experiments show that this approach results in high empirical FWER/FDR, making it unreliable.

• The reliability $\\alpha$ parameter must be chosen by the user based on the desired level of reliability for the application. Accordingly, the choice of this parameter must be informed by some knowledge about realistic values for the average risk. In the RL example, one could use off-policy data log to estimate a reasonable target reliability level $\\alpha$. To evaluate the impact of $\\alpha$ here we perform an ablation study by varying its value ( https://ibb.co/xq5fxXTn ).

Relation To Broader Scientific Literature: In the paper we mention various approaches based on Bayesian optimization [1-4]; however, we do not mention evolutionary strategies or PAC-Bayes methods. Thanks for pointing out this, in the revised paper we will add references to [5-8]. ( References https://ibb.co/twNJDDCd )

Weaknesses: Our approach extends to multiple correlated hyperparameters by associating a multivariate parameter with a single hypothesis. Appendix B.2 presents a wireless resource allocation experiment where hyperparameters include transmit power level and scheduling policy, which are inherently correlated. Even in such cases, aLTT significantly improves testing efficiency and system performance. If space allows, we will move this discussion to the main text.

Clarifying e-values: We acknowledge that readers unfamiliar with these concepts may need more guidance. In the revised version, we will explicitly direct them—already in the introduction—to Section 4.1, which explains e-values, e-processes, and their role in testing.

Questions For Authors:

• A simple grid search selecting the hyperparameter with the lowest empirical risk yields a maximum TPR of 1/|reliable hyperparameters|, which can be much lower than 1 when reliable hyperparameters are numerous. For instance, in our RL experiment, with nine reliable hyperparameters, grid search would achieve at most 0.11 TPR, far below aLTT’s 0.85. Moreover, this approach lacks FWER/FDR guarantees, as shown in our experiment.

• The reliability requirement is an expectation over multiple roll-out. The definition in (15) is the reward for a single roll-out, while the expectation of (15) over the roll-out distribution defines the reliable hyperparameters, i.e. $\\Lambda^{\\rm rel}=\\{\\lambda\\in\\Lambda: R(\\lambda)=\\mathbb{E}_{P_Z}[R(\\lambda,Z)]> \\alpha\\}$. We will clarify this in the main text as “The reliability for a single roll-out is measured via the cumulative reward obtained by a policy $\\lambda$ on the MDP $\\mathcal{E}$, which is defined as…”

• We choose to compare aLTT only against methods that are statistically valid, as the goal of this paper is to perform ‘statistically valid hyperparameter selection.’ To illustrate that methods without formal guarantees can violate the target FWER and FDR levels we consider a validation based baseline that picks an hyperparameter if its empirical risk is below $\alpha$. Figures: RL experiment ( https://ibb.co/xq2KnrGH ) and the APE experiment (https://ibb.co/QFxgkc0G ) .

Thank you for taking the time to review our paper and for your positive comments! Below, we address each comment point by point:

• "Why no main theorem/proof?...". - Thank you for the suggestion! In the revised version, we will add the following theorem on the statistical validity of aLTT “Given a hyperparameter set $\\Lambda$, a reliability level $\\alpha \\in [0,1]$, and an error level $\\delta \\in [0,1]$, aLTT with an FWER or FDR-controlling selection rule returns a final prediction set $\\hat{\\Lambda}^{{\\rm aLTT},T}$ that satisfies $(\\alpha,\\delta)$-FWER control or $(\\alpha,\\delta)$-FDR control, respectively.” and provide a proof in the Appendix.

• "It may be a good idea to reference selectivenet...". - Thanks for bringing this related work to our attention, we will include it in the related work.

Other Comments Or Suggestions

• We consider the stopping criterion $|\\hat\\Lambda^{\\rm aLLT,t}|>d$ or $t=t_{\\rm max}$ because we don’t make any assumption on the relation between the value of $\\lambda$ and its risk (e.g. monotonicity as in RCPS). For example, in case of the prompt selection, there is no clear notion of larger or smaller prompt. However, if some preference or information about the parameters is available, it is possible to include this information in the design of the stopping criterion. We will clarify this in the revised text.
• Yes, assuming that the loss function is monotone, one can use aLTT in a similar manner to RCPS. The e-processes in aLTT can be used to obtain upper confidence bounds on the risk, which can then be used to implement the selection procedure as in RCPS. The validity would follow as in the RCPS paper. However, note that our paper operates in the most general setting, where we remain agnostic to any relationship between risk and hyperparameters.
• Thanks for spotting the typo!

Thank you for your insightful comments. Below we address your question:

Questions For Authors: Yes, adaptive LTT (aLTT) substantially outperforms LTT, even when LTT uses fixed-time valid p-values. The table below reports final TPR for aLTT and LTT using p-values from Waudby-Smith & Ramdas (2024) and Hoeffding-Bentkus p-values (Angelopoulos et al., 2021). The latter is tight for 0-1 losses. In all cases, aLTT with adaptive acquisition ($\epsilon<1$) achieves significantly higher true positive rates (TPR), demonstrating its advantage stems from data-adaptive acquisition rather than LTT’s use of weaker p-values. We will include these results in the revised paper.

P-values based on the central limit theorem are asymptotically valid but unsuitable for rigorous finite-sample guarantees. Such p-values can inflate FDR and lack statistical soundness—for instance, prediction sets from these p-values yield an empirical FWER of 0.154, exceeding the intended $\\delta=0.10$.

Claims and Evidence

• We acknowledge the ambiguity in the term “testing” and will replace it with “evaluation” when referring to the data-adaptive acquisition policy. Specifically, at each time step $t$, a subset of hyperparameters $\\mathcal{I}^t \\subseteq \\Lambda$ is selected for evaluation, updating only the e-processes in $\\mathcal{I}^t$ (see Eq. 14). This clarifies that all hypotheses in $\\Lambda$ are tested, but only a subset is evaluated at each step.

• We believe Figure 3 aligns with theory, as both empirical FWER and FDR stay below $\delta$. It shows FWER (blue) and FDR (red) for an FWER-controlling procedure (dashed triangle) and an FDR-controlling procedure (solid square). Since FWER is stricter than FDR, for either scheme (triangle or square), the FWER curve (blue) is always higher than the FDR curve (red). Likewise, as the FWER-controlling procedure (dashed triangle) is stricter, its FDR and FWER curves are lower than those of the FDR-controlling procedure (solid square).

Methods and Eval

• RL and LLMs are widely used in high-stakes and high-risk applications. For example, in the field of robotics, RL algorithms are used for robotic surgery, autonomous navigation, robotic arm control, and more. In these cases, policies are often trained offline using simulators to mitigate the risks associated with training policies in the real world. However, due to the mismatch between simulators and the real world, the actual performance of these policies is often unknown. In Section 5.1 we exemplify how aLTT can be used to identify policies that satisfy real-world performance guarantees, while limiting the number of real-world interactions. Similarly, LLMs are used in critical fields such as medical diagnosis and legal applications. For example, Med-PaLM is a large language model (LLM) designed to provide high-quality answers to medical questions. The performance of these model can vary depending on how are prompted, rendering their use potentially harmful. In the automated prompt engineering example proposed in Section 5.2 we showcase how aLTT can be used to identify prompt templates whose accuracy falls above the minimum reliability level.

• The FWER criterion ensures high-confidence reliability, whereas FDR prioritizes statistical power in large-scale testing. FDR control is common in exploratory research, including genomics, neuroimaging, and social sciences, enabling broader discovery while accepting a higher false positive rate.

• We assume the standard episodic RL setting [2], where agent interactions occur in episodes with resets to the initial state. Risk (15) measures average episodic performance, and reliability requirements apply over the initial state distribution $P_{s^1}$ and policy roll-out.

[2] Sutton RS, Barto AG. Reinforcement learning: An introduction.

Related Work

• In the revised text, this relation will be further clarified as “This work aims to improve the data efficiency of LTT by leveraging recent advances in sequential multiple hypothesis testing (MHT) based on e-processes (Vovk & Wang, 2021; Waudby-Smith & Ramdas, 2024; Xu et al., 2021). Unlike LTT, which employs standard p-value hypothesis testing, we propose an adaptive learn-then-test strategy based on the hypothesis testing framework proposed in Xu et al. (2021) to perform statistically valid hyperparameter selection for machine learning models and generate finite-sample performance certificates.”
• Thanks for bringing to our attention Jamieson and Jain!

Thank you for taking the time to review our paper! Below is our response to your question and your comment regarding the absence of formal proofs:

• The intuition behind aLTT’s inability to discover all hyperparameters is that, in our experiments, we use finite datasets, which may sometimes be insufficient for aLTT to accumulate enough evidence to identify the entire set of valid hyperparameters.

• In the revised text we have decided to state formally the validity of aLTT with theorem as follows “Given a hyperparameter set $\\Lambda$, a reliability level $\\alpha \\in [0,1]$, and an error level $\\delta \\in [0,1]$, aLTT with an FWER or FDR-controlling selection rule returns a final prediction set $\\hat{\Lambda}^{{\\rm aLTT},T}$ that satisfies $(\\alpha,\\delta)$-FWER control or $(\\alpha,\\delta)$-FDR control, respectively.”

Thank you for your insightful comments. Below we address your question point by point:

Theorem and Stopped e-process: In the revised version, we will formally state the aLTT guarantee in a theorem and provide a proof in the Appendix, which follows from the validity of the e-processes. “Given a hyperparameter set $\\Lambda$, a reliability level $\\alpha \\in [0,1]$, and an error level $\\delta \\in [0,1]$, aLTT with an FWER or FDR-controlling selection rule returns a final prediction set $\\hat{\\Lambda}^{{\\rm aLTT},T}$ that satisfies $(\\alpha,\\delta)$-FWER control or $(\\alpha,\\delta)$-FDR control, respectively.” Regarding the validity of the e-BH procedure, we will add the following statement clarifying that e-BH maintains statistical validity under the scenario discussed in the paper, while for alternative scenario in which data is re-used at later testing round it is necessary to use the procedure of [1] “Finally, note that while the eBH procedure directly applies to the sequential testing of Section 2.3, where we allow risk estimates $\\mathcal{R}^t=\\{R(\lambda_i, Z_i^t)\\}_{\\lambda_i\\in \\mathcal{I}^t}t$ be arbitrarily dependent --- for example, by using the same data to evaluate all models in $\\mathcal{I}^t$ --- reusing the same data for evaluation of different hyperparameters at later stages can invalidate the e-process validity unless this is properly corrected [1]"

[1] Wang et al. Anytime-valid FDR control with the stopped e-BH procedure. arXiv preprint arXiv:2502.08539. 2025 Feb 12

Hyperparamater Definition: We consider a broad definition of hyperparameters, including train-time, architectural, and post-processing parameters. For example, in the RL experiment (Sec. 5.1), “We produce $N = 20$ control policies by setting the hyperparameter $\\lambda$ in the TD3+BC training objective.” The revised text will clarify that aLTT applies to both train-time and inference-stage hyperparameters.

Xu et al. : We agree that aLTT relies on the sequential multiple hypothesis testing framework based on e-processes introduced by Xu et al. (2021). In this sense, aLTT is to Xu et al. (2001) what the original LTT is to multiple hypotheses testing with FWER control. In the original text we wrote “This work aims at improving the data efficiency of LTT by leveraging recent advances in sequential MH based on e-processes (Vovk & Wang, 2021; Waudby-Smith & Ramdas, 2024; Xu et al., 2021)”. In the revised text, this relation will be further clarified as “This work aims to improve the data efficiency of LTT by leveraging recent advances in sequential multiple hypothesis testing (MHT) based on e-processes (Vovk & Wang, 2021; Waudby-Smith & Ramdas, 2024; Xu et al., 2021). Unlike LTT, which employs standard p-value hypothesis testing, we propose an adaptive learn-then-test strategy based on the hypothesis testing framework proposed in Xu et al. 2021 to perform statistically valid hyperparameter selection for machine learning models and generate finite-sample performance certificates.”

Alternative Data Acquisition: The $\\epsilon$-greedy strategy generalizes to a top-$K$ approach, where the $K$ largest e-processes are selected with probability $1-\\epsilon$, and $K$ random hyperparameters are chosen otherwise. We will revise the text accordingly and add an experiment ( https://ibb.co/d46Q2yPK ).

Why p-values from e-processes: While for FDR control there exists procedures that directly operate with e-values, for FWER e-values are first converted to p-values using e-to-p calibration and then FWER procedure are based on p-values are used. [1]

[1] Vladimir Vovk and Ruodu Wang. E-values: Calibration, combination and applications.

Highlighted box page 3: We clarify that while data for each hyperparameter is i.i.d., risk estimates across hyperparameters may be dependent due to shared data reuse (see also first point).

Experiments:

• Unless the experiments are conducted using synthetic data, the ground truth is generally unknown. In real-data experiments, the ground truth is estimated from large hold-out samples, which will be clarified in the revision.
• The average number of tested prompts is 66 across tasks.
• We consider tasks from the Instruction induction dataset; in Appendix B.3 we report the performance per tasks, specifying which task we consider. In the revised version we will subsample some interesting good/bad prompt example from the exhaustive list ( https://ibb.co/1GV2MHyj ).
• In the revised text, we will clarify that this statement is made in an empirical and average sense. In fact, due to the randomness of testing, it is possible that the discovered sets are not exactly subsets of one another. However, we find that a stricter requirement leads to a smaller predicted set of reliable prompts (on average) and an increase in the minimal instruction length (on average).
• Here we consider plot failure probability instead of minimal length (https://ibb.co/QSCN4sy).