A SYSTEMATIC STUDY ON EARLY STOPPING CRITERIA IN HPO AND THE IMPLICATIONS OF UNCERTAINTY

We sincerely thank Reviewer wZMM for the valuable suggestions. Below we would like to give detailed responses to each of your comments.

Fundamental questions and Motivation

We use the concept of reliability to assess if a specific performance metric shows statistically significant superiority over others. As we explained in Section 4.1, model uncertainty is a technical term that refers to a lack of precise knowledge or confidence in a model's predictions, typically manifested as variations in predicted outcomes.

The nature of early stopping revolves around the idea of monitoring a model's performance during the training phase and halting the training process when certain criteria or performance metrics are met. This paper aims to explore the impact of using different performance metrics in the early stopping mechanisms on the final results. We conducted extensive experiments to explore the commonly used metrics - training/validation loss/accuracy - and found that training loss significantly outperforms the other metrics, especially for complex tasks with slow convergence. To this end, we showed through experiments and analysis that uncertainty and data size are possible reasons for the poor performance of validation metrics. Further, we explored a series of strategies that utilize uncertainty to help metrics improve reliability.

Thanks for the suggestion. Following your advice, we have added explanation in Section 1.

Answers to the fundamental questions

This paper addresses the three fundamental questions across Sections 2 to 4, providing distinct insights into each:

1. Section 2 empirically compares commonly used criteria. It reveals that different criteria may have significantly different impacts on the HPO performance, and training loss tends to suit complex tasks better than validation loss.

2. Section 3 analyzes the factors that influence early stopping, offering insights for Section 2's findings. The nature of early stopping revolves around the idea of monitoring a model's performance during training and halting the training process when certain criteria or performance metrics are met. This practice essentially involves “a trade-off between the HPO cost and the effectiveness of the final models”. Criteria or metrics are responsible for estimating and ranking the models' capabilities. According to Section 3, the factors that affect the validity of a criterion are the size of the dataset on which the calculation is based and whether or not the criterion contains uncertainty that affects discrimination. As we analyzed in Section 3, “while a larger training dataset may contribute, the primary justification lies in the training loss serving as a superior indicator of model expressiveness, yielding greater stability and consistent model ranking. In contrast, the validation loss prioritizes the aspect of generalization and tends to favor models with lower capabilities that converge early. ”

3. Section 4 investigates the impact of model uncertainty on early stopping. Besides the commonly considered measures, other early stopping criteria include the combination of training and validation metrics validated in Section 3 and Appendix D, and a collection of methods that incorporate uncertainty explored in Section 4. Model uncertainty has some important impact on the effectiveness of early stopping. As stated in Section 4.1, “if an early stopping decision is made at a point of peak uncertainty, stronger models may be prematurely terminated.” By examining a series of criteria that introduce uncertainty, this paper concludes that “a careful combination of model uncertainty with conventional early stopping criteria can yield significant benefits for HPO, especially when dynamic trends of the training process are considered.”

Thanks for raising this question. We have given a summary of our findings near the Introduction for readers to grasp the message.

New HPO algorithm

Thanks for the suggestion. We further conduct experiments with two widely recognized algorithms: BOHB [1] and the state-of-the-art Sub-sampling [2]. The results and detailed settings can also be found in Appendices B and E in the updated version of our paper. It can be seen that the training loss significantly outperforms other metrics across all HPO algorithms. Furthermore, introducing uncertainty into metrics demonstrates enhanced reliability—an aspect we delve into within our findings.

[1] https://arxiv.org/abs/1807.01774

[2] https://arxiv.org/abs/2209.12499

Terminology

Thanks for raising this concern. We clarify the terminology used in our work as follows:

1. "Gaps" in Insight 2 refers to the differences observed in performance metrics across various models, essentially reflecting the concept of performance regret. A more discriminative metric leads to clearer differences between model performances, thereby enhancing reliability.

2. The lower plots in Figure 1 depict the discrepancy between the final test accuracy achievable by the remaining model candidates after each instance of early stopping and the best final test accuracy achievable by the whole set of samples as training progresses. The term "fraction of budget" denotes the proportion of allocated budget utilized during training. These results are derived from a scenario where R=150 and the filtering ratio equals 3. Each data point within these plots signifies a round of early stopping. The "fraction of budget" serves as an indicator of the early stopping point, specifically denoting the epoch at which early stopping is executed, calculated based on the filtering ratio ($\eta$) and R. Furthermore, the count of retained model candidates is determined according to the filtering ratio in each early stopping iteration.

We have added explanations in Section 2 and Insight 2.

We sincerely thank Reviewer F2DD for the positive feedback and valuable suggestions. Below we would like to give detailed responses to each of your comments.

HPO tasks

Thanks for the suggestion. We have followed your advice to add a description of the 9 HPO tasks and the 2 newly added benchmarks in Appendix A, including their model structures, number of configurations, hyperparameters, loss function, etc.

Explanation of Fig. 4

Thanks for the suggestion. Figures 4(a) and (b) show three randomly selected models from the ImageNet task, where 4(a) mainly depicts the fluctuations in validation loss between neighboring epochs, while 4(b) mainly illustrates the model uncertainty contained in different runs. Figure 4(c) shows the relationship between early-stage fluctuations and final test loss for all models in ImageNet, Cifar10, and Cifar100, intended to illustrate that models with large early uncertainty tend to have good final capabilities. We have updated Figure 4 to make it clearer.

We sincerely thank Reviewer dT38 for the valuable suggestions. Below we would like to give detailed responses to each of your comments.

The "early stopping" terminology

Thanks for pointing out this concern. The term "early stopping" was inherited from the Hyperband paper [1], where it broadly signifies determining when model training should halt based on predefined metrics and strategies. We agree with you that in the context of Hyperband, “early discarding” is a more accurate expression. Yet, we would like to clarify that early discarding represents just one case covered by early stopping. Numerous HPO algorithms, such as Sub-sampling [2], rely on early stopping mechanisms where halted model candidates remain active and might resume training in subsequent stages.

The performance metrics investigated in our study serve to rank model candidates, providing basis for early stopping mechanisms to make decisions. However, whether the early stopping mechanisms entail discarding falls outside of this paper's focus. To validate the broad applicability of our findings, we have introduced experiments on two additional HPO algorithms, BOHB and Sub-sampling, detailed in Appendices A, B, and E.

[1] https://arxiv.org/abs/1603.06560

[2] https://arxiv.org/abs/2209.12499

Contribution and read path

Thanks for raising this valuable question. This work belongs to an insight paper, aiming at exploring the impact of diverse performance metrics within the early stopping mechanisms of HPO algorithms. Our experiments centered around commonly used metrics, including training/validation loss/accuracy. The results highlighted the superior efficacy of training loss, especially in intricate tasks with slower convergence. This finding is intriguing, considering that validation loss has been widely adopted. We substantiated this observation through experiments and analysis, identifying factors - model uncertainty and dataset size - being potential reasons for the inferior performance of validation metrics. Furthermore, we explored strategies leveraging uncertainty to enhance the reliability of these metrics.

We have followed your advice to improve the organization of this paper. In particular, we have given a summary of our findings in the Introduction for readers to grasp the message. We also revised some sentences in Section 1 to make our motivation clearer. We've expanded the experimental scope in Appendices A, B, and E, introducing new benchmarks (MLP and LogReg from HPOBench) and incorporating additional HPO algorithms (BOHB and Sub-sampling), to enrich the data basis of this work.

Decision variable and motivation

We fully agree with you that in most cases the metrics are determined by user needs. As we pointed out in Section 1, the choice of metrics within HPO is based on the practitioners' personal preferences. Regarding decision variables, our study has shown that delving into the reliability of metrics yields benefits not only for users but also for HPO tools. From the user's perspective, if there is a priori knowledge of a model's convergence behavior, one can choose between training and validation losses based on whether overfitting occurs at early stopping points. Specifically, if there is overfitting, validation loss is preferred; otherwise, it is the training loss. For the HPO tools, they can request both training and validation losses from users, thus incorporating an overfitting detection to avoid unfair comparisons, and making decisions based on which metric contains less uncertainty. Additionally, they can introduce uncertainty, as shown in Section 4, to design more robust metrics for more reliable decision-making. We have outlined general guidelines in Section 5.

Regarding the motivation of this paper, we have followed your suggestion to replace "criterion" with "performance metric" and made a clear distinction in Section 1. In addition to comparing loss and external metrics (e.g., accuracy), we also seek to compare the differences between training and validation metrics. We have modified the expressions in Section 1 and Insight 1 to make the motivation clearer.

Final objective

We use the concept of reliability to assess if a specific performance metric shows statistically significant superiority over others. In our experiments, cross-entropy serves as the loss function, and we choose test accuracy as the final objective based on following considerations:

First, we use balanced accuracy throughout our experiments, except for ImageNet. However, our findings demonstrate that using both test loss and test accuracy yields consistent results. Through an analysis of the relationship between final test loss and accuracy across all models, we unveil a significant linear correlation. The difference in accuracy among models with nearly identical losses (differences around 1e-6) is less than 3%. Conversely, for models achieving the same accuracy, the difference in losses is within 0.4. Notably, in the case of ImageNet, the achieved differences in accuracy and loss stand at 1.7e-7% and 0.08, respectively. Further empirical evidence along with clarifications regarding the balance of benchmark classes can be found in Appendix A.

Second, the test dataset itself might be subject to diverse biases and uncertainties. Consequently, neither test loss nor test accuracy can be deemed as definitive assessment metrics. Cross-entropy loss, sensitive to subtle differences between predicted and true distributions, might respond differently to slight variations in class probabilities. While test loss offers a more nuanced model ranking, it might also contain additional noise or uncertainty. In our comparison scenario, we need only focus on overall significant differences to determine model reliability.

Third, test accuracy, due to its simplicity and intuitive nature, aligns more closely with practical application goals. Real-world scenarios such as image classfication competitions [1] and LLM leadearboards [2] predominantly favor test accuracy as the final objective. We have added experiments in Appendix A with test loss as the final objective. The results show that the significance of the differences obtained using test loss and test accuracy as the final objective are generally aligned. These results consistently highlight the superiority of training loss and emphasize the advantage of incorporating uncertainty.

[1] https://paperswithcode.com/sota/image-classification-on-imagenet

[2] https://crfm.stanford.edu/helm/latest/

Dataset selection in Figure 1

Because the models in LCBench all converge very quickly, producing overfitted and disordered results, we randomly chose two datasets for illustration. We have added results for the other datasets in Appendix B.1.

Explanation to the theoretical part

We fully agree with the comment that D can be a training or validation set, and that model complexity plays a role in HPO efficacy. However, we would like to clarify that our discussion is in the context of "one early-stopping based HPO", which implies a fixed DL task, similar model complexity (with different hyperparameters), and fixed training, validation, and testing sets. Our goal is to allow users or HPO tools to select the best metric when faced with an HPO task, where there is no need to compare between multiple HPO tasks. We have added a detailed description in Section 3 to provide further clarity.

Section 3 aims to delve into strategies for minimizing decision errors, which we define as the probability of the optimal model's loss being greater than the loss of a suboptimal model. By minimizing the probability, it supports the conclusion that "using a larger dataset or choosing more discriminative metrics can yield more reliable results". The training metrics stand out due to their foundation on a larger database, offering more stable values, as demonstrated in our experiments on Nas-Bench-201 and the new benchmarks outlined in Appendix A, B, and E. While validation metrics outperform training metrics on certain datasets like Fashion-MNIST and jasmine, this discrepancy does not contradict our statements. Instead, as explained in the second paragraph of Section 3, it stems from lower model complexity leading to overfitting.

We have refined Insight 1 and highlighted certain statements in Section 3 to enhance the clarity of our findings. We have also updated Section 4.2.1 and Appendices A and E with additional technical details.