Risk-Controlling Model Selection via Guided Bayesian Optimization

Thank you for your helpful and constructive review.

On the rational behind the definition of the ROI and expected limitations

We assume that there is a limited budget for searching the configuration space since each evaluation is very expensive (requires retraining of the model). We want to focus the optimization efforts on finding configurations that are: (i) likely to pass the test (the risk is not overly high), and are (ii) effective with respect to the free objective. The defined $\alpha^\textrm{max}$ specifies the limiting value that, if observed during testing, leads to rejecting the null hypothesis and declaring that the configuration is valid. Since the objectives are conflicting, as we get close to this limiting value, we improve in terms of the free objective. Ideally, we would like to find the configurations whose expected value is exactly $\alpha^\textrm{max}$. We include in the region of interest all the configurations that are likely to be associated with this ideal configuration. Specifically, we exclude configurations if the probability to correspond to an expected value $\alpha^\textrm{max}$ is lower than $\delta’$. These configurations are not relevant since they are likely to fail the test or they do not best minimize the free objective. To improve clarity, we added clarifications around Eqs. (5), (6) and (7) and an illustration demonstrating these principles in Fig. E2.

Regarding the effectiveness, it depends on the given budget and the shape of the frontier. In the limit of a large budget, sampling the entire front or only a portion of it, will be comparable. In addition, if the derivative of the free objective with respect to the constrained objective is small, the improvement in the free objective as we get closer to the limit is small as well. However, even in these cases the proposed method is not supposed to fail or perform worse than the other baselines.

On non-conflicting objectives

Indeed, if the objectives are not conflicting the defined region of interest may not be relevant. This case can be trivially solved as a constrained optimization problem, and the solution can be statistically tested and verified without any concern that it will not pass the test. Only when the objectives are conflicting, the minimizing solution satisfies the constraint tightly, and we need to find a set of configurations around this point in order to increase the chances that we would be able to find a valid configuration.

On “expansive high-volume configuration spaces” not demonstrated

We corrected this to “expansive configuration spaces”. We indeed present examples with configuration space similar in size to what was previously shown. The major difference is that we demonstrate expensive configuration spaces that require retraining of large models per each new configuration. In previous works, the configurations were set by hyperparameters that affect the way the model is used after the model is already trained, which is significantly less expensive, and can be performed with a relatively large budget.

On the choice of $\delta’$

We added explanations about the meaning and the choice of $\delta’$ in the revised manuscript. This hyperparameter controls the size of the region around $\alpha^\textrm{max}$. We exclude configurations that correspond to $\alpha^\textrm{max}$ with probability that is less than $\delta’$. This is an empirical choice with a tradeoff between enlarging the region of interest to include other potential configurations and increasing the density of the configurations that are found close to $\alpha^\textrm{max}$.

We use $\delta’=0.0001$. Following your comment, we added an experiment to examine the influence of different $\delta’$ values (see Fig. C.5.). We observe that most of the time the exact choice of $\delta’$ does not lead to significant differences in the performance. Note that the width of the region is also influenced by the size of the validation data. When $k$ increases, the width decreases as we have more confidence in the observed empirical losses of being reflective of the actual expected loss.

On the chosen ranges of $\alpha$ values

We added clarifications on the choice of these values in the revised manuscript. We define the range according to the values observed for the initial pool of configurations that is generated at the beginning of the BO procedure. The minimum and the maximum edge points obtained for each task appear in table B.1 and are also observed in the examples shown on Fig. C.4. We choose values in between these extreme edge points, but not too close to either side, since too small values may not be statistically achievable and too large values are trivially satisfied (with tighter control not significantly improving the free objective). Note that in the chosen range even small differences in the limit values along the constrained axis, result in a significant difference for the free objective. As a general note, this choice is task and model specific, and for other tasks (for example pruning) the range is much wider.

On insignificant results for fairness and robustness

We show in the experimental results that the proposed method is consistently better compared to all other baselines, almost everywhere across tasks and $\alpha$ values. It is true that for certain tasks the naïve baselines perform well but these baselines perform much worse in other settings. The naïve baselines spread the configurations uniformly or randomly in the configuration space. The multi-objective baselines spread them all over the Pareto front. It can happen that in some cases the resulting set contains configurations that reside in proximity to the testing limit, hence leading to efficient testing outcomes. However, more often it happens that the configurations are far from the limit, resulting in reduced efficiency. The proposed method always finds a dense set of configurations around the desired limit, thus is much more stable and efficient across different settings. We added an illustration in Fig. E2. to demonstrate this principle.

On missing experimental details

Following your comment, we edited the experimental section and added further details and clarifications.

On the efficiency of the proposed method with respect to Pareto Testing

We can compare the efficiency of the proposed method with respect to Pareto Testing from two viewpoints. The first viewpoint compares testing efficiency (quantified via a better minimization of the free objective) for a given optimization budget. Pareto testing suggests first finding a sampled version of the Pareto front, and then orders and tests these configurations via sequential testing. We claim that most of the Pareto front is irrelevant, and we should focus only on the part of the front that is near the testing limit. For a given finite budget, our method yields a denser set of configurations near the limit, and therefore is expected to obtain tighter control. This is demonstrated in Fig. 3, 4, C1 and C2 and is illustrated in Fig. E.2. The second viewpoint is the budget required to match the ideal performance of Pareto Testing with a dense grid of configurations. As shown in Fig. C.3. with a budget of 50 configurations we can match the performance of Pareto testing with a dense grid of over 6K configurations. From this viewpoint, our method has a computational advantage and can be applied to tuning hyperparameters that affect the training of large models, which was not exemplified before in Pareto Testing.

Thank you for your helpful and constructive review.

On comparison to baselines that perform multi-objective-optimization without testing

Thank you for pointing this out. We understand we did not explain clearly the exact baselines that were compared. All baselines consist of two stages: optimization followed by testing. We do not include baselines that do not contain testing, since they were shown in [1] to lack valid risk control (they do not satisfy Eq. (1)). It was also shown in [1] that Pareto Testing is preferable over other testing mechanisms. In this paper, we compare our method to baselines that are all equivalent to Pareto Testing, while utilizing different methods for optimization in the first stage (and then perform testing in the second stage): uniform, random and different multi-objective optimization algorithms. Therefore, the testing mechanism is the same for all presented methods, and the improvement achieved by our method is related to the different optimization mechanism that is designed for improving testing efficiency.

[1] Laufer-Goldshtein, Bracha, et al. "Efficiently Controlling Multiple Risks with Pareto Testing." The Eleventh International Conference on Learning Representations. 2022.‏

On the performance without defining the region of interest

The baselines of the multi-objective optimizers correspond to the case that no region is defined, and the entire Pareto front is recovered.

On the performance with a single limit

Following your comment, we added an ablation study (see Fig. C.7.) to examine the case of a single limit instead of the defined region. This is shown to be inferior with respect to the full region in most cases, as expected.

On notation in Section 3.

We changed the notation following your suggestion.

On swapped limits in Eq. (7)

We corrected the typo.

On the evaluation of the loss in Algorithm D.1.

We use the validation data during BO. Following your comment, we added it as an input to the algorithm, and clarified its use for evaluating $\hat{\ell}$. The calibration data is used afterwards in the testing procedure.

On the comparison to MOBO baselines in Figs. 5 and 6.

Following the reviewers’ comments, we added these comparisons. See extended results in Figs. 3, 4, C1 and C2.

Thank you for your helpful and constructive review.

On the clarity of Eqs. (5), (6) and (7)

Following your comment, we added clarifications around Eqs. (5), (6) and (7), and additional derivations and mathematical details in Appendix E. In short, for testing we receive an empirical loss evaluated over the calibration data, and we need to determine whether the configuration is valid or not. For bounded losses, Hoeffding’s inequality quantifies the probability to obtain an empirical loss value $\hat{\ell}$ under a given expected loss $\ell$. In appendix E, we show how to derive Eq. (5) from Hoeffding’s inequality. Eq. (6) quantifies the value of $\hat{\ell}_1$ for which the p-value in (5) equals $\delta$. This is denoted as $\alpha^\textrm{max}$. In order to be valid the loss must be lower than $\alpha^\textrm{max}$. However, for effectively minimizing the free objective function, we should be as close as possible to $\alpha^\textrm{max}$ (since increasing $\ell_1$ implies decreasing $\ell_2$). Therefore, we define in (7) the region of all configurations that are likely to be obtained under an expected value $\alpha^\textrm{max}$. Specifically, we exclude configurations if their probability to be associated with $\alpha^\textrm{max}$ is lower than $\delta’$. The hyperparameter $\delta’$ controls the region's width and is unrelated to the testing confidence level $\delta$. It is an empirical choice that trade-off between including more potential values, and obtaining a set of configurations that is dense around $\alpha^\textrm{max}$. We added an experiment (see Fig. C.5.) showing that the method is generally insensitive to the choice of $\delta’$.

On the lack of proof for Theorem 5.1.

We added the proof on Appendix E.

On 3 out of 4 tasks with a single-dimensional hyperparameter

Our experiments include two settings with a multi-dimensional hyperparameter. Specifically, we including the following five tasks:

Classification fairness – 1 dimension

Classification robustness – 1 dimension

Selective classification and robustness – 2 dimensions

VAE - 1 dimension

Transformer pruning - 3 dimensions

Note that compared to previous work (Pareto Testing), we present four cases requiring model retraining for each new hyperparameter setting. This makes the optimization much more cost intensive and is enabled by our proposed approach, which focuses on a limited part of the Pareto front.

On missing baselines for $n=1$

It is important to note that in the single dimension case, varying the hyperparameter value and retraining the model, results in configurations that reside close to the Pareto front. Therefore running multi-objective optimizers for these cases is not expected to be significantly different from the Uniform sampling baseline (though the configuration distribution may change). The Random baseline for $n>1$ is based on Latin Hypercube sampling, which is a semi-random sampling method that generates well-spread points in the configuration space. This is very close in spirit to the Uniform grid defined for $n=1$. Nevertheless, following your comment, we added these baselines, as can be seen in Figs. 3, 4, C1 and C2.

On the chosen ranges of $\alpha$ values

We added clarifications on the choice of these values in the revised manuscript. We define the range according to the values observed for the initial pool of configurations that is generated at the beginning of the BO procedure. The minimum and the maximum edge points obtained for each task appear in table B.1 and are also observed in the examples shown on Fig. C.4. We choose values in between these extreme edge points, but not too close to either side, since too small values may not be statistically achievable and too large values are trivially satisfied (with tighter control not significantly improving the free objective). Note that in the chosen range even small differences in the limit values along the constrained axis, result in a significant difference w.r.t the free objective. As a general note, this choice is data and model specific, and for other tasks (for example pruning) the range is much wider.

Thank you for your helpful and constructive review.

On mismatch to ICLR

The paper belongs to the field of reliability of machine learning models, and how we can tune the model hyperparameters while satisfying multiple risk constraints. This is related to the broader statistical field of conformal prediction and risk control , which is the topic of many papers published in recent years in ICLR and other top ML conferences. In addition, we combine the fields of hyperparameter tuning, multi-objective optimization and Bayesian optimization, which are core techniques in building machine learning models. Our method is demonstrated on different applications involving classification and generative models on benchmark machine learning tasks.

On writing clarity

We made major changes to the paper’s writing and clarified points that were unclear to the reviewers.

On the sign of (7)

We corrected the typo.

On the practical use of the proposed method

We agree that performance or business metrics related to the objective function are crucial in many practical applications. However, in many cases there are tradeoffs between these metrics (accuracy vs cost, fairness, robustness etc.), and it is important to constrain certain metrics while optimizing others. Our method presents a simple-yet-effective approach to solve this by a new guided BO method combined with statistical testing. We guarantee $(\alpha,\delta)$-risk control, where $\alpha$ is the bound and $\delta$ is the significance level, reflecting the certainty level with respect to the randomness of the calibration data. If $\delta =0.1$, then we can guarantee that 90% of the time (over different calibration datasets) the risk does not exceed $\alpha$.

On experiments testing limited tasks compared to introduction

We demonstrate the performance in all the tasks mentioned in the introduction: algorithmic fairness, robustness to spurious correlations, generation capabilities in VAEs and cost reduction in Transformer pruning.

On the performance in high-dimensional hyperparameter spaces.

The proposed method consists of two stages: (i) Bayesian optimization, and (ii) multiple hypothesis testing. The first stage is affected by the dimensionality of the hyperparameter space. We adopt Bayesian optimization techniques, which are effective in optimizing large hyperparameter spaces. For extremely high-dimensional hyperparameter spaces we can incorporate more advanced methods for effective hyperparameter optimization [1]. The second stage of statistical testing is performed over a proportion of the Pareto frontier in the objective space, and, therefore, is not affected by the hyperparameter dimension.

[1] Daulton, Samuel, et al. "Multi-objective Bayesian optimization over high-dimensional search spaces." Uncertainty in Artificial Intelligence. PMLR, 2022.‏

On the risk of bias and sensitivity w.r.t. the defined “region of interest”

The region of interest is defined to include configurations that can be potentially verified by testing and are effective in minimizing the free objective. We demonstrate in the experimental results that introducing the “region of interest” is almost always preferable compared to the baseline methods that consider the full front. In addition, we added experiments showing that the specified region is preferable over a one-sided bound (Fig. C.7) and that the method is generally insensitive to the choice of $\delta’$ that controls the region’s width (Fig C.6).

On the balance between exploration and exploitation

The overall approach does not rely on the optimization mechanism, as it suggests to perform optimization over a portion of the Pareto front in order to improve statistical efficiency while selecting a risk-controlling configuration.
Here we implement this by defining a new reference point that encloses the region of interest. This can be coupled with several acquisition functions that are defined with respect to a reference point in the multi-objective case, such as hypervolume maximization, expected hypervolume maximization and probability of hypervolume improvement. These methods offer different exploration vs exploitation balancing possibilities. In our case, we found hypervolume maximization to be effective, where there is more emphasis on exploitation. It may be attributed to the fact that the region of interest narrows the search space focusing on a relatively small region.

On the computational overhead compared to hyperparameter optimization techniques

The computational overhead of the proposed method compared to traditional hyperparameter techniques is in the testing stage. We need to evaluate the configurations in the chosen set, which requires a forward-pass of the model per tested setting. This is negligible with respect to the BO part that requires retraining the model N times.