Reliable Algorithm Selection for Machine Learning-Guided Design

We appreciate your feedback! We would like to refer the reviewer to the first 3 paragraphs of our response to reviewer EEWq, as we believe there has been a misconception regarding our primary contributions.

In particular, to our knowledge, our work is the first to formalize and propose a rigorous solution to the problem of design algorithm selection (DAS): a principled method for making the decisions that every practitioner of ML-guided design must make, and which might otherwise be biased by personal preference or convenience. None of the reviewers have suggested any alternatives, which testifies to the novelty and difficulty of DAS and the creativity of our solution.

Our responses to specific points follow.

• "Would [the Bonferroni correction] lead to overly conservative selection?" Empirically, this was not a problem. Our method had low error rates while maintaining high selection rates, compared to baselines (Figs. 3a,4a). In new results with a 5X-larger menu for the protein GB1 experiments (Fig. R1, see first comment to reviewer Bfc7 for details), our method actually has higher selection rates than with the original menu, as the expanded menu contains more configurations with higher mean design labels. If one is concerned about conservatism, however, a conceptual strength of our formalization of DAS as a multiple testing (MT) problem is that any MT procedure that controls FWER can be used instead of Bonferroni. E.g., one can use procedures that respect the hierarchical [1] or correlation structure between configurations [2], which may yield less conservative multiplicity corrections. Since our goal was to establish a first, general solution to DAS, we instantiated Alg. 1 with Bonferroni, as it doesn't require assumptions about how configurations are related.

• "Can you get theoretical guarantees on the selection rate?" No, because if the practitioner specifies an impossible success criterion, then the selection rate must be zero. That is, no non-zero lower bound exists in general.

• Potential issues with estimated density ratios (DRs): While the guarantees in the paper do not hold with DRE, this does not nullify the value of having guarantees when DRs are known, in contrast to baselines w/o guarantees at all. We appreciate your concern, however, and present new results in Fig. R1 showing that our method performs similarly in the protein GB1 experiments using estimated vs. known DRs. Both there and in our RNA experiments in Fig. 4, our method with DRE outperforms the baselines. Also note that DRE yields consistent estimators of the true DRs if the model class is correctly specified (flexible enough) [3], and is routinely used in ML applications involving importance weighting [4].

To further address the concerns regarding Bonferroni and DRE, we also present new RNA experiments in Fig. R2, which use DRE with an expanded menu of size 249 containing the hierarchical configuration space shown in Fig. R3. Our method performs well in these experiments.

• Thank you for suggesting the relevance to Bayesian optimization and active learning! We now mention them in "Related Work," condensed here: Bayesian optimization (BO) is a paradigm for iteratively selecting designs, acquiring their labels, and updating a predictive model in order to optimize a property of interest. In each round, BO chooses the design that globally maximizes some acquisition function quantifying desirability based on the model's predictions. A typical goal of BO is to converge to the global optimum as the rounds progress, under regularity conditions [5]. In contrast, the goal of DAS is to achieve criteria on the distribution of design labels to be imminently proposed. Such guarantees can help justify designs to stakeholders when acquiring labels for even one round is resource-intensive, and the priority is to achieve specific criteria within one or a few rounds. However, nothing precludes BO from being used as design algorithms within our framework: configurations of BO with, e.g., different HPs or acquisition functions can be on the menu.

Finally, we emphasize that our method empirically outperforms the baselines. Despite the impact of the DAS problem, no reviewer has suggested any alternative approaches. If the reviewer would like to suggest one, we are happy to include it!

[1] Bretz et al. A graphical approach to sequentially rejective multiple test procedures. Stat. Med. 2009.

[2] Dudoit et al. Multiple hypothesis testing in microarray experiments. Stat. Sci. 2003.

[3] Gutmann & Hyvarinen. Noise-contrastive estimation of unnormalized statistical models. JMLR 2012.

[4] Sugiyama et al. Density ratio estimation in machine learning. 2012.

[5] Srinivas et al. Gaussian process optimization in the bandit setting. ICML 2010.

Thank you for your feedback! We are glad the reviewer found the idea of selecting reliable algorithms to have strong potential to benefit the ML community. We believe there has been a misconception regarding our primary contributions.

[First to formalize + address DAS] To our knowledge, although ML has revolutionized the design of proteins, small molecules, and other modalities, this work is the first to formalize the problem of design algorithm selection (DAS) and propose a solution with any guarantees. None of the reviewers have suggested any baselines or alternative approaches, which testifies to the novelty and difficulty of DAS and the creativity of our solution.

[Not just HP tuning] This work is not simply a variant of hyperparameter tuning as it is performed in supervised learning. Tuning cannot be performed for design tasks the same way, because we never have the labels of designs needed to evaluate each design algorithm configuration. This is why our major technical innovation is to rigorously evaluate which configurations will be successful, in the complete absence of labels for their designs.

[Formalized population-level success criteria w/ high-probability guarantees] Another innovation is formalizing + achieving a common goal in practice: to produce a population of designs that satisfies a desired criterion (e.g., that at least 10% of designs' labels surpass some threshold, or that the average design label does so). Our method provides high-probability guarantees for a broad class of such population-level success criteria. Prior work has focused instead on uncertainty over individual designs, rather than population-level criteria often sought in practice.

We are glad the reviewer found the paper well-written, and appreciate the concerns regarding the method's practicality, addressed below.

• "Menus ... only contain roughly 100 choices": See Fig. R1 and Fig. R2 for new protein and RNA experiments with menu sizes 501 and 249, respectively, where our method still performs well. However, even for the original smaller menus, our method outperformed the baselines. Are there alternative approaches the reviewer might propose? We are happy to try them.

• "[Density ratios can be] impossible to estimate": Fig. 4 showed empirically that our method is still effective with a relatively simple classifier-based density ratio estimation (DRE) technique. Also see Fig. R1 for new results showing our method performs well with DRE in the protein GB1 experiments. Note that DRE is routinely used in ML applications involving importance weighting [1,2] and yields consistent estimators of the DRs if the model class is correctly specified (flexible enough) [3]. We agree that the guarantees don't hold with DRE, but this doesn't nullify the value of having guarantees when DRs are known, in contrast to baselines without guarantees at all.

• Requires "held-out labeled data (HLD)": Figs. 3&4 compared to baselines that do not require HLD (prediction-only and GMMForecasts), which instead used all the labeled data to train the predictive model. Our method outperformed these baselines. We also note that the entire literature on post-hoc calibration for uncertainty quantification (including e.g. conformal prediction) requires HLD. Indeed, there is a fundamental limit to how calibrated models can be w/o using HLD [4]. In practice, collecting some extra labeled data can be a reasonable investment when the method's guarantees can help justify costly resources for synthesizing designs to project stakeholders. Lastly, our method is amenable to cross-fitting variants that do not completely hold out data [5] (happy to elaborate).

• Can the method handle "categorical parameters and ... multiple parameters?": See Fig. R2 for new RNA experiments with an expanded menu of size 249, containing the more complex hierarchical configuration space illustrated in Fig. R3. Our method performs well here (similar error rates with higher selection rates), as the expanded menu contains configurations with higher mean design labels than the original menu.

• Fig 3B,4B: The "=" is an equals sign, indicating that the diagonal line is the y = x line. Results on/above the diagonal mean that selected configurations are successful.

• Line 275: Thank you for noting the broken link! Now fixed.

[1] Sugiyama et al. Density ratio estimation in machine learning. 2012.

[2] Grover et al. Bias correction of learned generative models using likelihood-free importance weighting. NeurIPS 2019.

[3] Gutmann & Hyvarinen. Noise-contrastive estimation of unnormalized statistical models. JMLR 2012.

[4] Bengs et al. Pitfalls of epistemic uncertainty quantification through loss minimisation. NeurIPS 2022.

[5] Zrnic & Candes. Cross-prediction-powered inference. PNAS 2024.

We are grateful for the positive evaluation of our work, and are glad the reviewer found our method and experiments well-motivated and appealing. Our responses to specific comments and questions follow.

• "The hyper-parameter range is also discretized at an arbitrary resolution of 100 values, but the menu size drives the performance of the algorithm due to the multiple testing correction." We appreciate your concern regarding this point. See Fig. R1 for new results where the menu for the protein GB1 experiments is discretized to contain 501 values (np.arange(0.2, 0.7, 0.001) instead of the original np.arange(0.2, 0.7, 0.005)). Our method still keeps error rates below the user-specified level, consistent with the guarantees. Interestingly, it does so with higher selection rates for greater values of tau (x-axis) than with the original smaller menu. That is, the multiple testing correction did not make the method more conservative with a 5X-larger menu; in fact, the method actually selected successful configurations more frequently, because the expanded menu contains more configurations with greater mean design labels.

• "I was surprised to find that the 'menu' corresponded to a range of values for a single real-valued hyper-parameter. There is a lot of structure across this menu that the method doesn't exploit." Thank you for this insightful observation. Please see Fig. R2 for new RNA experiments with an expanded menu of size 249, containing the more complex hierarchical configuration space illustrated in Fig. R3. Our method empirically performs well in these experiments, but we agree that it would be possible to better exploit the structure in the menu induced by the relationships between different configurations. This is a conceptual strength of formalizing design algorithm selection as a multiple testing problem: any multiple testing procedure that controls family-wise error rate (FWER) can actually be used in place of the Bonferroni correction in Line 6 of Alg. 1, meaning one can use FWER-controlling procedures that respect, e.g., the hierarchical organization of the configuration space [1] or the correlations between configurations [2]. We instantiated Alg. 1 with the Bonferroni correction for full generality, as it doesn't require any assumptions about how configurations are related, but replacing this with procedures that exploit these relationships is a great direction for future work. Note that using such structure-respecting multiple testing procedures would not improve the existing guarantees on the error rate, but might improve (i.e., increase) the selection rate by yielding a less conservative multiplicity correction.

• "For problems where the density ratio was tractable, it would have been helpful to see what the performance difference is between the case when using the true density ratio vs. an approximated one." Thank you for this suggestion. See Fig. R1 for new results where we estimated the density ratios (DRs) for the protein GB1 experiments. Specifically, we separately estimated the labeled distribution and the design distribution corresponding to each configuration. For the former, we performed maximum-likelihood estimation with Laplace smoothing (with pseudocounts of 1) to estimate the site-specific categorical distributions using the held-out labeled sequences; for the latter, we did the same using the design sequences. For a given sequence, we then took the ratio of its densities under these two estimated distributions as the estimated DR. The results from our method using these estimated DRs are very similar to the original results using the known DRs.

• Figs. 3A, 4A: Thank you for noting that the presentation was confusing. To clarify, would the reviewer prefer that we zoom-in the y-axes of plots closer to 0 for error rate and 1 for selection rate? Note that the methods have error rates and selection rates that span the entire [0, 1] range, because we wanted to thoroughly compare their performance across a wide range of success criteria (x-axis values) that practitioners might be interested in. However, we have zoomed-in the error rate plot of Fig. R1 closer to 0 as suggested, since the plotted rates are all less than 0.1.

[1] Bretz et al. A graphical approach to sequentially rejective multiple test procedures. Stat. Med. 2009.

[2] Dudoit et al. Multiple Hypothesis Testing in Microarray Experiments. Stat. Sci. 2003.