Towards Neural Architecture Search through Hierarchical Generative Modeling

We are deeply grateful for your positive evaluation and insightful comments on our manuscript. Your feedback not only encourages us but also enhances the quality of our research. Regarding the issues you have raised, we hope the following answers are satisfied. Please let us know if you see any further issues in the paper that are unclear or need to be addressed.

If time permits do you think you could provide results on a different modality, e.g. an NLP task?We

Since generalizability was a common concern among reviewers, we decided to include experiments on NB360. We hope this should be more than enough to satisfy this request. Please see the common reply summarising new experiments.

To better understand the variance of your method, could you rerun the architecture selection multiple times and also report conf. intervals over the variance of each architecture when run for 3-5 random initializations?

Please see the common reply summarising new experiments.

As stated in their limitations, the method largely relies on ZC proxies for both G-VAE and the generation of the macro architecture.

Recent work in zero-cost metrics has shown that zero-cost metrics-based NAS can produce promising performance networks compared with the training-based NAS approach. Admittedly, the zero-cost metrics performance can vary when ranking models from different tasks and different search spaces. However, in our generator, we are not strictly based on the assumption that model performance ranking strictly follows the zero-cost metrics ranking, but ‘similar zero-cost metrics will lead to relatively similar performance’. This is different from existing zero-cost metrics NAS.

How do you estimate the number of components for the GMM?

To decide the number of components for the GMM we increase their number, starting from 16, and calculate the Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC). We stop when they plateau.

Why is the cost of the ES for training the SG such a major bottleneck when using ZC proxies? Did you parallelize ES?

Please note we report GPU-hours, i.e., cost and not wallclock time — to avoid confusion, we will change the term used in the paper accordingly. This cost is subject to parallelization, following common practices. Since all steps can be parallelized trivially, we can achieve almost linear speedup if we use multiple GPUs, e.g., with 4 GPUs we can finish everything within ~8h.

Regarding the relative cost of ES to the rest of the system. The bulk of the cost comes from calculating proxies for models, so it is going to be directly proportional to the amount of “labelled examples” mentioned in Tab. 5 (compared to calculating proxies, operating on graphs is very fast, e.g., when training a G-VAE or running a metrics predictor). As such, getting the training data for G-VAE and the SG should take similar time (50k vs 60k), but note that the micro model uses only up to 64 channels (Eq. 2) and our macro search space allows for <=1024, so the average model is much larger in the macro stage.

Finally, we did not dedicate a lot of time optimising our code and there are some opportunities to achieve non-negligible speedups. To give an example, when computing the vector of proxies we currently compute each of: NASWOT, SNIP-SSNR and FLOPs completely independently from each other, resulting in 3 forward passes and 1 backward pass per model, but it clearly should be possible to optimise it such that only 1 forward and 1 backward is needed per model. This would reduce the time needed for data generation by up to 2x.

Thank you for dedicating your time and review our manuscript. We appreciate your comments and acknowledge the concerns you have raised, we want to make further clarification as follows:

The proposed method still heavily depends on human prior knowledge (e.g., reference design, zero-cost metric)

Regarding the reference design, we would like to point out that in our experiments we used cluster centres for that, which are decided automatically and do not require any prior knowledge. Although we discussed the possibility of improving results further if a good reference point was known, we did not rely on that when running our method. Therefore, we believe the reviewer’s point is reaching too far in that aspect.

Regarding zero-cost proxies, we agree we rely on them, which involves certain risks, as mentioned in the limitations. Still, the proxies are a way of probing into the characteristics of neural networks, like #params or #flops; if anything, we would consider our approach of clustering cell designs according to ZC proxies to be rather automated and require very little human input. We strongly disagree with the implied statement that building on top of the existing knowledge should be considered a weakness.

Having said that, we understand the reviewer is also making a point about the applicability of our method to less-studied tasks. While we disagree with the points above, we do agree with concerns about generalizability, considering our original submission focused on well-studied tasks only. To address this, we have run our method on a suite of tasks from NB360, please find the details in the common reply summarising new experiments.

The generalization of the proposed method is a big question.

Please see the results on NB360 in the common reply.

The idea of using a hierarchical design to reduce the space is not novel.

We never claim the idea of using a hierarchical design is novel on its own — if the reviewer got an impression that’s the case, please let us know which part of the paper made them think so and we will revise it. The novelty of the paper lies in designing a hierarchical generative system, which is the first of its kind, to the best of our knowledge.

I feel the current experiment design is not a good fit for this paper. I am more interested in seeing experiments on less studied domains instead of these standard benchmark datasets.

Please see the results on NB360 in the common reply.

Thank you for the time and effort you have invested in reviewing our manuscript. We greatly appreciate your insights and acknowledge the points you have raised. Regarding your concerns, we have provided additional clarification as follows and we are committed to enhancing our experiments and manuscript, hope that the revision will satisfy your concern.

The design conflicts the target pain point.

We apologise for not making our point sufficiently clear — this is a side effect of limited space, so please let us explain our reasoning in more detail.

First, it is impossible to have an operational codebase that would be able to train models without defining its ramifications. This means any NAS system requires a search space that is designed manually — even if this search space is implicit — because of the need to write functioning code that would be able to execute networks. This is a fundamental issue and we do not see a way around it. We want to make this point clear because the way in which the reviewer raised their point (“this paper still manually designs a search space and develops an algorithm upon it”) directly touches upon it; there is no way to perform NAS differently than designing a search space and developing an algorithm upon it, any deviations from this rule are likely to be nothing more than semantic nuances (e.g., what exactly is called a search space).

Instead, what is usually understood behind the term “manually designed” search space is the amount of manual effort that is needed to construct a search space in which good models can be found fast enough. The trade-off between search space size and time needed to search is again fundamental — we can easily think about a search space of all possible models that can be expressed in, e.g., Pytorch, but searching in such a search space is going to be extremely difficult. Consequently, every NAS method performs some kind of regularisation of this huge search space by considering only its subset, to make the problem tractable. The question becomes: how big can the subset be and if there are any constraining factors. This leaves us with a spectrum, where a method can be considered more “automated” if it retains strong searching performance without limiting the number or the qualitative properties of models in the search space. We would argue that our method advances in both aspects. On one hand, our search space is extremely expressive, to the best of our knowledge the largest among any works that achieve competitive ImageNet performance. On the other hand, we do not impose constraints such as the ability to construct a supernet (with the size of our search space, we would run out of GPU memory very fast if we tried that), etc. As such, the effort to design our search space is rather minimal — we simply include as many operations from the literature as possible, expose as many parameters (channels, stride) as possible and only make sure any network can be executed correctly. Considering all that and the fact that we can obtain strong performance fast, we would argue it is justified to say our method is “more automated” than the existing ones and that it helps minimise manual effort (we will revise our paper to make it clear it is not our goal to completely eliminate manual design, since this cannot be done).

We thank the reviewer for the valuable and constructive comments on our work. We hope to have answered all of your questions satisfactorily below. Please let us know if you see any further issues in the paper that are unclear or need to be addressed.

I recommend comparison with black box nas methods on exactly the GraphNet space(eg: regularized evolution [1], hierarchical nas [2])

We have reported the performance of regularized evolution with T-CET in our GraphNet space in Tab. 2 (c.f. Evo(T-CET)). As suggested by the reviewer, we ran additional experiments using regularized evolution with a reduced training scheme (50 epochs training, same as NAS-Bench-201). We refer to this new baseline as Evo(reduced-training) in the following. Under a similar search budget with Evo(T-CET) (~20 GPU hours), we were able to train ~40 models using Evo(reduced-training), and the results are shown in the table below.

To further demonstrate the effectiveness of our approach, in addition to running black box NAS methods directly on the GraphNet space, we also conducted experiments where we run Evo(reduced-training) on an optimised search space produced by our method (SG+CCNF+Evo(reduced-training) in the following table), under a similar search budget (~20 GPU hours). In this setting, we first use our sequence generator (SG) to generate the macro structure while for each cell we use CCNF to propose 5 candidates. We then construct a search space for running Evo(reduced-training) by considering any combination of the 5 generated cells (note: the macro models used 4 cells in this case). In total, the optimised search space is of size 5^4.

We observe that under a similar search budget, regardless of requiring training or not, regularized evolution, i.e. both Evo(reduced-training) and Evo(T-CET), struggle to find good performing models on the large GraphNet space. On the other hand, when equipped with our hierarchical generative approach, the optimized search space significantly bolsters performance of both the standard and the zero-cost evolution. Interestingly, SG+CCNF+ZCNAS achieved a bit higher result in this experiment, compared to running reduced training in the optimized space. This should not be a surprise, however, if we recall that reduced training is known to be a somewhat bad proxy, and T-CET is known to be especially good for datasets like CIFAR-10. Still, reduced training did reasonably well, and much better than when running in a full space; we include it for the sake of completeness. We also suspect it might be a better choice for tasks on which T-CET might fail.

The method uses a pretrained GPT-Neo-125M model. Hence the actual search cost also implicitly inclues the pre-training cost of this model, which should ideally be added to the search cost computation (table 1,2,3).

We agree there is a hidden cost in needing a pre-trained GPT model - we will include a note about that in the paper for the sake of completeness (note that the authors of GPT-Neo did not disclose the time needed to train their models, so we will rely on how costly it is to train similar ones). However, we are not sure if a straightforward attribution of this cost to our method is completely fair. GPT models are trained to perform NLP tasks and there is no clear relation between those and NAS. This is in contrast to some NAS methods that might require starting from, e.g., an open-sourced pretrained supernet - in this case, the connection to performing NAS is clear and including supernet pretraining cost is justified. The point is: if we consider GPT training cost to be part of the search cost, it gives a false impression of having to pay it each time we search for a new model (which is clearly not the case); even if we consider it a pre-cost, it gives a false impression that it has to be paid each time we want to extend/change our NAS settings (such as including more models in the search space, etc.) - again, this is not the case. As such, we believe it should be classified as “pre-pre-cost”, but there is no clear consensus within the NAS community on how (or even if) such cost should be reported, as far as we know. In particular, many existing methods might have paid analogous cost (i.e., with similar level of indirection) but did think about reporting it. Of course, because of that we are happy to engage in discussion with the reviewer to work out the best way of doing so - please let us know your thoughts about our arguments and the current way in which we include the cost in our paper.