Efficient Hyperparameter Optimization with Adaptive Fidelity Identification

To Reviewer WVYX:

We appreciate the effort and time spent reviewing our submission, and we are grateful that you highlighted the wide generality of our method. We address the weaknesses and questions next.

“The coverage of related work is lacking, …, DyHPO, …, DPL, …”

We sincerely thank you for bringing DPL to our attention, which is a latest work that had not been accepted at the time of our paper drafting, limiting our awareness of its content. We have added additional experiments and discussions on DyHPO. It performs well on both LCBench and NAS-Bench-201, but exhibits slightly inferior performance on FCNet. We found these two works interesting and appreciate their contributions to the community. These important works have also been incorporated into the related work.

In contrast to the two methods that introduce new surrogates and use neural networks as support, our FastBO takes a different way. We opt for a more explicit way to incorporate learning curve information. Instead of using black-box neural networks, our method provides enhanced interpretability and ease of understanding. This is possible because the concepts of efficient and saturation points are transparent. Furthermore, our strategy can be easily extended to any single-fidelity method without the need for a redesign.

“The paper advocates that it is able to adaptively decide the fidelity for which every configuration should be evaluated, however, it incorporates fixed design choices, e.g $w$, the number of warmup steps necessary to have for every learning curve is 20% of the total curve (Although the authors have a heuristic that can stop the warmup phase if no improvement is observed).”

Because of the adaptive process of efficient and saturation points customization, our method has the capability to adaptively decide the fidelity for each configuration. This adaptive process is different from the warm-up process, where $w$ is used.

Regarding the concern on the fixed design choices of $w$, we have added experiments to study the impact of different choices of $w$. The results show that FastBO is not highly sensitive to the values of $w$, showing the robustness of our method. The fixed choice of $w$ does not compromise our key contributions. We intentionally kept a fixed setting for $w$ to avoid introducing unnecessary complexity by tuning a 'hyperparameter' within a 'hyperparameter' optimization algorithm. Fine-tuning $w$ is a possibility and can lead to a further improvement on performance.

“The combination of a surrogate model together with LC prediction for an individual hyperparameter configuration can be inefficient from my perspective.”

FastBO is in fact efficient. We added a comparative analysis between FastBO and DyHPO, a representative method that integrates LC prediction into the surrogate model. The results show that the overhead of FastBO is smaller than DyHPO.

“The best performance does not necessarily need to be at $r_{max}$ for the incumbent configuration, it could also be at an intermediate fidelity level. The proposed target of optimization is not correct and the rest of the paper is based on that. As the authors note later on: "However, we observe that an increase in fidelities does not always result in performance improvement."”

We acknowledge the confusion caused by the statement in Section 3 and have corrected it. However, the rest of the paper remains unaffected. In Section 4.3, where we state, "However, we observe that an increase in fidelities does not always result in performance improvement," this claim is correct and does not conflict with the multi-fidelity target. The purpose is just to present an implementation detail in post-processing: recording all intermediate results and selecting the final output from the intermediates. While it is common in implementation, we included it for the completeness of our method. We have rephrased Section 4.3.

“In section 4.2: $w$ is used, although it is defined only in 4.3.”

Thanks for your suggestion. We have updated Section 4.2 to avoid any confusion.

“Unless I missed it, the surrogate model is referred to as $M$, however, no information is given on what the surrogate model is, until Section 5 with FastBO uses a Matern $\frac{5}{2}$ kernel. I believe it should be clearly described in the manuscript that $M$ is a GP.”

Thanks for your suggestion. We have explicitly mentioned the used surrogate model as well as the acquisition function in Section 3. Our focus is not to introduce new surrogate models or optimize existing ones, and our method can be applied to any surrogate model. Therefore, we did not highlight this aspect earlier.

To Reviewer ZiMT:

Thank you for the in-depth review with valuable feedback and advice. We appreciate the significant effort spent reviewing our paper and recognizing the strengths of our method. We address the weaknesses and questions next.

“The method introduces additional hyperparameters \delta_1, \delta_2. These are not simple to interpret and hence might be difficult to set a-priori in practice. It would be also great if the paper could comment how important it is to set these values correctly.”

We appreciate your comments on the introduced hyperparameters. In order to avoid introducing complexity by tuning a 'hyperparameter' within a 'hyperparameter' optimization algorithm, we intentionally set the hyperparameters in a simple way. The setting of $\delta_1$ and $\delta_2$ is related to the performance metrics. We standardized the metrics to a scale from 0 to 1 before setting them. After that, the values (i.e., 0.001 and 0.0005) are just the common thresholds used for detecting minor performance variations. We encourage the users to directly use our default setting. Fine-tuning them is also a possibility and, if explored, may lead to further optimization on performance.

“The proposed learning curve model appears to be quite similar to that of Domhan et al., … I don't see a reason why it couldn't also be used to predict the efficient or saturation point.”

A crucial difference between FastBO and Domhan et al.’s method is the targets of learning curve modeling. Domhan et al. aim to predict how likely the current configuration's final performance beat the current best values. Thus, they need to obtain uncertainty by a Bayesian approach. In contrast, we aim to adaptively identify fidelities for each configuration, with the learning curve modeling only serving as a supporting tool. Thus, we don't need their complex parametric model (e.g., considering 11 learning curves) or to estimate uncertainty. Therefore, using Domhan et al.'s learning curve modeling module may not be a good choice for us, although it could also be used to predict the efficient or saturation point.

Given the efficiency goal of the HPO method, we simplified the parametric learning curve models to strike a balance between capturing essential learning curve shapes and prioritizing computational efficiency. Opting for complex learning curve models introduces a computational burden during parameter estimation, as the computational complexity of parameter estimation is proportional to the number of parameters in the combined learning curve model. The increase in computational demands increases the time required for each configuration, which runs counter to the objective of designing efficient HPO algorithms.

“The results seem somehow different to the results reported by Salinas et al. … Did you follow their experimental setup? If not, what has changed and why?”

We appreciate your observation regarding the experimental comparison. We followed the settings of A-CQR, but we mistakenly set up A-CQR runs with different approaches to estimate missing evaluation points on LCBench. This is a benchmark-related issue and may result in certain differences in datasets. Thanks for pointing it out! We have fixed it and found that A-CQR is generally better than A-BOHB. We will fix the corresponding figures and tables. However, we also found that different runs of A-CQR with the same random seeds sometimes led to unstable results. This is a potential reason for the different results.

“Do you factor in the post-processing time into your runtime plots?”

Yes, all the FastBO processing time are included in the wall-clock time.

To Reviewer gzF7:

We appreciate your effort and time spent reviewing our submission. We address the weaknesses and questions in the following.

“Overall, the proposed method consists of several simple heuristics whose justification as a general methodology is not fully clear, and for me, it is somewhat difficult to see significant technical contributions in the paper.”

In HPO research, it is common and unavoidable to use heuristics like expert intuition and domain knowledge. This is rooted in HPO's goal of automating the hyperparameter tuning process traditionally performed by human experts. Due to the NP-hard nature of HPO, some examples of applying “simple yet effective heuristcs” in HPO include Hyperband (JMLR’18), BOHB (ICML’18), ASHA (MLSys’20), DyHPO (NeurIPS’22), Hyper-Tune (VLDB’22), PASHA (ICLR’23), DPL (NeurIPS’23). They were all published in top venues in this area, recognizing the substantial contributions of the heuristics.

Similarly, FastBO also exploits heuristics. The heuristics are carefully designed and are non-trivial, resulting in strong improvement and making substantial technical contributions (the most important contribution is to adaptively identify appropriate fidelities for each configuration in the multi-fidelity setting).

“The model of the learning curve is too simple. I do not agree with that these three functions have enough flexibility to handle a variety of shapes of learning curves. ….”

The functions we chose have shown good fitting and predicting performance in existing empirical learning curve literature. General learning curve shapes can be captured, which is adequate for identifying the efficient and saturation points — The latter is our main focus, while the former serves only as a supporting tool. Our experiments in Section 5.3 demonstrate the effectiveness of our method built on the learning curve modeling.

Given the high efficiency goal of HPO, we simplify the parametric learning curve model to strike a balance between capturing general learning curve shapes and prioritizing computational efficiency. Opting for complex parametric models introduces a computational burden during parameter estimation. The computational burden translates into an increase in the time required for each configuration, and has an adverse effect on the fundamental objective of designing efficient HPO algorithms.

We acknowledge that learning curves can have diverse shapes, sometimes non-convex shapes. However, to the best of our knowledge, existing HPO methods that consider learning curves all exclude the consideration for the poorly-behaved learning curves, particularly those with non-convex shapes. Furthermore, in the SHA-based multi-fidelity methods, there is also an implicit assumption of well-behaved learning curves for hyperparameter configurations. The exclusion of poorly-behaved learning curves is a common practice, mainly due to their complexity and limited impact on the overall optimization process. Having said this, it is straight forward to update the curve fitting model to also include non-convex shapes (but it is believed to be unnecessary in the ML community).

Thanks for your valuable feedback. We have added a separate section in Appendix for an in-depth discussion on the choice of parametric learning curve models.