NAP2: Neural Networks Hyperparameter Optimization Using Weights and Gradients Analysis

We would like to begin by thanking the reviewer for their time and effort. Below we address the weaknesses raised by the reviewer.

1. "The paper does not cover many related works in this area which have used similar techniques. See [1], [2], [3] among others. Even if the above related work did not exist, it is not clear to the reviewer whether the contribution of the paper is sufficiently novel or useful to give a score of accept."

The reviewer provided three representative studies from different domains (HPO, pruning, and NAS). We argue that NAP2 has significant novelties compared to these studies, and those similar to them. We will briefly describe each study, then elaborate on our method’s novelties.

[1] is a relatively early (from 2014) HPO study, that analyzes the convergence of the training curves throughout training to predict final performance. The method uses Bayesian Optimization to decide which architectures are worthy of additional resources. In essence, this approach is a precursor of current SOTA methods that use similar principles (the methods that are our baselines). It is important to note that the architectures are not treated as `white-boxes’, and that there is no modeling of the weights/gradients of individual layers.

[2] is a study in the field of network pruning. Unlike the majority of studies in the field, this study attempts to prune the network before it is fully trained. Similar to the large majority of pruning studies, the authors analyze the weights of individual neurons in the layer to determine which ones need to be pruned (the search is guided by sub-network similarity). Decisions are reached based on neuron weight and sub-network similarity.

[3] is NAS study that analyzes the gradients of architectures to efficiently predict their performance. The gradients are evaluated at the architecture level, thus treating the neural net as a black box.

Compared to these studies (and those similar to them), NAP2 contains multiple novelties:

a) We are the first to analyze changes in weights and gradients over time, and show that it is highly effective for the task of HPO. We are the first to analyze the evolution of the architecture throughout its training. Furthermore, this is the first time that such a white-box approach has been proposed to the tasks of HPO (or NAS, to the best of our knowledge).

b) Unlike [2] and other pruning-based approaches that analyze weights using a simple heuristic, we develop a set of features that capture various traits of each layer. Additionally, we are the first to analyze the gradients at the layer level rather than the architecture as a whole. Additionally, we propose ways to represent this information using feature maps and compress it so that it is more usable.

c) Unlike all other learning-based solutions (in the domains of HPO, NAS, and pruning), we are the first to propose a model that can be trained on one set of architectures, then applied on another disjoint set. As shown in our experiments, our model is transferable also to architectures that are trained on other datasets. Because our model is reusable, our approach is highly efficient in an online setting (one only needs to extract the feature maps once every 100 mini-batches).

2. "The paper's validation on CIFAR-10 and CIFAR-100 may not be sufficient evidence for it to be shown to be useful in practice on larger neural networks than toy networks."

We conduct our evaluation on architectures sampled from the NAS-Bench 101 dataset, which is the same dataset used in all leading SOTA studies. Given that, we follow the standard evaluation in this domain.

The evaluated architectures are by no means “toy networks”: as shown in Table 2, we evaluate NAP2 on hundreds of architectures with more than 30 layers, with hundreds more above 50 layers.

Conclusion: The reviewer requested that we elaborate on our contributions and the robustness of our evaluation. We analyzed the studies mentioned by the reviewer, and explained that our work has novel aspects that they do not have: a) temporal analysis of weights and gradients; b) multiple features that represent various aspects; c) transferability between architectures and datasets. Additionally, we explained that our evaluation is based on the dataset used for evaluation in current state-of-the-art.

We will be happy to answer any other questions the reviewer may have.

We would like to begin by thanking the reviewer for their time and effort.

We first address the weaknesses raised by the reviewer. This is particularly important because the reviewer has MISREAD the scale on Figure 6 (0.86 instead of 0.086), which led to a serious misunderstanding on the reviewer’s part as to the performance of our approach (see #2):

1. “It is more reasonable to compare it with performance predictors like BANANAS[2], NPENAS[3], or AG-Net[4]”

While the reviewer is correct in stating that previous studies (Bananas, Npeans, etc.) used performance prediction, we believe the reviewer overlooked a critical point: all mentioned studies require the learning model to be trained on the evaluated architectures. NAP2, on the other hand, can be trained on one set of architectures, then applied to another set – even one trained on another dataset (CIFAR-100 in our case) without any additional training.

This difference has significant implications: first, we can train the prediction model `offline’, without any connection to the test set architectures. Then, we can use the trained model in multiple experiments, even on other datasets. Secondly, for our model to work at test time, we only need to extract our meta-features once every 100 mini-batches (the interval size we used in our evaluation). Coupled with the fact that we sample values from each layer, this process has negligible computational cost. This means that our setup is the same as the HPO baselines to which we compare: we begin our evaluation with a fixed set of architectures on which we have no knowledge.

2. "Figure 6 shows that the mean square error between performance prediction and final accuracy is about 0.015, which is larger than the mean absolute error of other performance prediction results[5] in NAS-Bench-101 (mean absolute error < 0.01). Also, the result MSE > 0.86 in CIFAR-100 doesn't substantiate NAP2’s transferability if the scale of accuracy is 0 to 1.”

We would like to begin with the second point: the reviewer has MISREAD Figure 6: the MSE error for CIFAR-100 is 0.086, NOT 0.86. This order of magnitude makes all the difference in understanding our results, since they show that a model that was trained on CIFAR-10 architectures is transferable to CIFAR-100.

Secondly, the reviewer is correct that [5] achieved an error of <0.01, but that was achieved after multiple epochs of training and the use of several training seeds. Our approach, on the other hand, achieves MSE of about 0.015 AFTER A SINGLE EPOCH on the evaluation set architectures. Additionally, as shown in Figures 2-5, NAP2 requires less than an epoch (100-300 mini-batch steps) to significantly outperform all the baselines.

Summary: To conclude our response to the weaknesses raised by the reviewer, our model is different from NAS-oriented solutions in critical ways: out model is trained offline (without any connection to the evaluated architectures/search space), it is transferable to other architectures and datasets, and its white-box setup enables it to produce highly accurate predictions after than less of epoch of training, compared with tens of epochs required by existing solutions (for example, the curve-based methods mentioned in [1] require at least a full epoch of training to get any data points).

We would like to begin by thanking the reviewer for their time and effort.

1. "The experiments are limited to neural architecture search only on CIFAR-10 and CIFAR-100"

Our experiments are conducted on the NAS-Bench dataset, which is the current standard for HPO and NAS-based tasks (see recent studies from both domains in [1], [2]). NAS-Bench 101 contains large and complex architectures with dozens of layers (that are trained on CIFAR). We refer the reviewer to Table 2 in our paper for information about the architectures’ sizes.

Given that this dataset is used by practically every study, we maintain that our choice of dataset is justified.

2. "As they do not provide any theoretical perspective of this approach, they need more experiments in terms of tasks and datasets to support their claim."

We use the NAP2 prediction model in a Successive Halving setup [3]. As such, we can rely on the same theorem-backed performance guarantees as this algorithm (see Section 3.1 in the relevant paper). Additionally, we compare NAP2 to well-known and widely-used state-of-the-art baselines, and outperform them by highly significant margins.

3. "Gradient-based hyperparameter optimization, e.g., [1], can also be regarded as a white-box approach. Are there any comparisons with this line of works?”

The study mentioned by the reviewer is NOT a white-box approach. That study uses gradient descent to modify the hyperparameters used for training of an architecture (e.g., the Momentum optimization value), then trains the architecture until convergence. This means that this approach, like all other methods we review and compare NAP2 to, treat the architecture as a black-box. They do not analyze weights, gradients, or their evolution over time. As we explain in our study, no existing HPO or NAS approaches analyze the weights and gradients in a similar way to NAP2.

Moreover, a comparison to this line of work is not consistent with the current state-of-the-art. While policy gradient-based approaches were used in the past for HPO and NAS (see the highly influential work by [4]), they have a prohibitively high computational cost when applied in scale. This led to the adoption of other, more efficient techniques, the most common of which being Bayesian optimization.

4. "The paper only presents mini-batch steps, but how long does it take for architecture search in wall-clock time?"

Our per-architecture running time is nearly identical to those reported by our baselines, because all methods are evaluated on NAS-Bench 101 dataset architectures. The only additional computational cost of NAP2 compared to the baselines at test time is the extraction of the meta-features once every 100 mini-batches (which is efficient because we sample values from each layer), and using the already-trained NAP2 model to predict the architecture’s performance.

When considering the computational cost, it is important to differentiate between the computational cost of NAP2 during training and evaluation. The training of the NAP2 requires computational resources because we train multiple architectures to convergence, extract meta-features and the train the prediction model. This process, however, can be done `offline’, and on a different set of architectures and datasets than those used for the evaluation (which is our experimental setup). At test time, the meta-features only need to be extracted once every 100 mini-batches, and the small pre-trained prediction model is used once to produce a prediction (requires less than a second). In a sense, our trained model can be thought of as a LLM: studies in the field of natural language processing use extremely large models trained by others (e.g., OpenAI), and do not consider the training of those models because they were done by other.

[1] Mellor, Joe, et al. "Neural architecture search without training." International Conference on Machine Learning. PMLR, 2021.

[2] Wistuba, Martin, Arlind Kadra, and Josif Grabocka. "Supervising the multi-fidelity race of hyperparameter configurations." Advances in Neural Information Processing Systems 35 (2022): 13470-13484.

[3] amieson, Kevin, and Ameet Talwalkar. "Non-stochastic best arm identification and hyperparameter optimization." Artificial intelligence and statistics. PMLR, 2016.

[4] Zoph, Barret, and Quoc V. Le. "Neural architecture search with reinforcement learning.",2016.

We would like to begin by thanking the reviewer for their time and effort. Below we address the weaknesses raised by the reviewer.

1. Weakness #1

The reviewer touched on a point that is critical to understanding the strength and re-usability of our model. The reviewer would be correct if the prediction model had to be trained anew for each task. This is NOT the case: our prediction model can be trained once, then used in other prediction tasks. We demonstrate the re-usability of our trained model in two ways, with varying degrees of difficulty:

a) Training our model on a set of architectures that were trained CIFAR-10, and then performing HPO on a another (disjoint) set of architectures trained on the same dataset.

b) Training our model on a set of architectures that were trained CIFAR-10, then performing HPO on a set of architectures that were trained on CIFAR-100.

While the second setup is more difficult for our model, because of the need to generalize, NAP2 performs very well in both scenarios.

It is precisely for the reasons mentioned above, that we include in our contributions (see Section 1) the release of the trained model along with our dataset and code. The inclusion of the model is designed to enable other researchers to utilize our approach more easily.

We will update our paper to better emphasize this point.

2. Weakness #2

The cost of training our model is not included in our evaluation. Our reasoning is as follows:

a) As explained in our response to the previous comment, because the idea behind our model is “train once, use multiple times” (including on other datasets), we do not consider the training cost to be relevant to the evaluation.

b) The training of the NAP2 model can be done ‘offline’, a long time before the online evaluation.

c) Conceptually, our work is similar to studies that use pre-trained LLMs for various tasks. These LLMs require huge amounts of time and compute resources, but these costs are not factored into the evaluation, because they are used `as-is’. The same logic applies to the training of our model.

We will update our paper to better emphasize this point.

3. Weakness #3

The explanation for the baselines’ poor performance on CIFAR-100 is simple, and comprises of three reasons:

a) The number of architectures is larger: in CIFAR-10 we have 1160 architectures but use a four-fold evaluation. In CIFAR-100, we have 440 architectures, and all are used for evaluation. The larger number of architectures makes detecting the top performers more difficult (esp. a small number like top-3 or top-5).

b) Performing well on CIFAR-100 is more challenging because of the larger number of classes (see Figure 7 in the appendix for performance distribution). As a result, the baselines (which do not have NAP2 predictive capabilities) perform more poorly.

c) We show the results for small number of epochs, because NAP2 does not need more in order to perform well. We elaborate on this further in our response to question #5 below.

It should also be noted that none of our baselines used the top-3/5 precision in their reported evaluation. We include it in our evaluation specifically because it is a very challenging task that only NAP2 manages to do well.

4. Weakness #4

Because our goal was to create a generic and re-usable model, we did not make any adaptations to our prediction model: we trained it on CIFAR-10 architectures, then evaluated it on the CIFAR-100 architectures. Our model was able to achieve good results on CIFAR-100, which strongly supports our claim of our model being generic. However, Figure 6 shows that compared to CIFAR-10, our model is far less accurate on CIFAR-100: after one epoch, our model’s MSE is x4.5 higher, and after seven epochs the difference is x5. This lower accuracy is also reflected in our results (Figures 2-5), where NAP2 underperforms on CIFAR-100 compared to CIFAR-10 (but still far better than the baselines).

As stated by the reviewer, the difference in architecture accuracy distribution likely contributed to the different accuracy. The different training dynamics because of the need to classify x10 labels is likely a larger contributing factor.

5. Weakness #5

The reason for the lack of improvement by the baselines is simple: 100 mini-batches are about a quarter of an epoch (see Section 4.2). For CIFAR-10, we use four-fold cross validation, so one epoch for all evaluated architectures (before pruning architectures) is ~110,000 mini-batches. This means that we present results for about four epochs of training.

This small number of epochs is more than enough for our approach, which begins to be effective after as little as 100 mini-batches (a quarter of an epoch). This is the result of our approach being a white-box approach, and its ability to obtain insights mid-epoch (as opposed to, for example, curve extrapolation prediction methods). The other approaches are too slow to succeed in such a short time frame.