Multi-objective Differentiable Neural Architecture Search

We sincerely thank you for your review of our submission and the positive feedback on several aspects like the zero-shot transferability and a unified multi-objective differentiable framework of MODNAS. We are also glad to hear that you found our paper well written, well organized and our experiments thorough. We address your concerns and questions in the following.

My only concern is that training differentiable NAS with multi-objective hardware metrics can often be unstable and may struggle to converge, leading to issues with reproducibility. While the authors have shared the hyperparameters and plan to release the implementation, it would be beneficial to include a discussion of the training techniques in the Appendix to address these challenges.

Thank you for highlighting this critical point. We concur that differentiable NAS and hypernetworks often demand careful tuning of hyperparameters (discussed briefly in the limitations paragraph of Sectoin 5 as well). For hyperparameters specific to the supernetwork, we align them exactly with those used in the original search spaces and benchmarks. Overall, we find the hypernetwork to be reasonably robust to the hyperparameter choices. While there is always room for improvement, in our experiments, there were 3 components that made MODNAS robust and to work reliably across benchmarks:

1. The choice of MetaHypernetwork update scheme: this played a pivotal role in the performance of MODNAS. While other gradient update strategies (see lines 406-412) underperformed or started diverging (Figure 6), MGD converged relatively quickly to a hypervolume close to that of the global Pareto front. The convergence of MGD to a pareto stationary point is discussed in Desideri [1] and Zhang et al. [2]. The convergence of MGD in bilevel optimization is an open research topic (see recent results from Ye et al. [3] and Yang et al. [4]). One potential scenario when MGD could fail is when the gradient directions of the objectives it is optimizing point in different opposing directions, however this becomes practically unlikely, especially as the number of objectives grows (in our case we use it to find the common gradient across devices, which is for instance 13 devices on NB201).
2. The choice of gradient estimation method in the Architect: In Section 3, lines 270-278, we discuss our choice for the method that enables gradient estimation through discrete variables (as architectures are discrete variables). We noticed that the ReinMax [5] estimator always outperformed previous estimators such as the one in GDAS [6] (Figure 15 in the appendix), so we believe this choice is crucial.
3. Weight entanglement vs. weight sharing in the Supernetwork: In early experiments on NB201 we noticed that weight sharing in the Supernetwork, was not only more expensive, but much more unstable as well when compared to weight entanglement [7, 8], even yielding diverging solutions quite often (common pattern seen in differentiable NAS with shared weights as you mention; see Zela et al. [9] for instance).

We hypothesize that all design choices mentioned above play an implicit regularization effect on the upper level optimization in the bi-level problem, leading to a faster convergence and robustness [9, 10, 11]. We have updated the paper with Appendix J containing the points we discussed above.

Could you provide the training and validation loss curves?

We have added the training and validation loss curves for NAS-Bench-201 in Appendix L. At each mini-batch iteration we plot the average cross entropy loss across all devices. As expected both training and validation cross entropy go down and we do not notice any overfitting. The high noise is common for sample-based NAS optimizers, since a sampled different architecture is activate at each mini-batch iteration.

In Figure 13, do you quantize the probability of $r_m$ to obtain the vector in the embedding layer $e_m$​? As I understand it, embedding layers typically take indices as inputs.

Yes, this observation is correct. We quantize the continuous sampled $r_m \in [0, 1]$ to the discrete $[0, 1, … 100]$ interval (see line 14 in https://anonymous.4open.science/r/MODNAS-1CB7/hypernetworks/models/hpn_nb201.py). We have added a sentence (highlighted in blue) in Appendix E.2 to clarify this in the updated version of the paper.

In Figure 13, if I understand correctly, $e_{\phi_0}$ is a linear layer that maps the device feature vector to a k-dimensional vector. If this is the case, referring to it as an 'embedding' layer may be somewhat confusing, given the use of $e_m$ for embeddings.

Yes, that is correct. We apologize for the confusion and we have already fixed this in the text by referring to it as a linear layer instead. Thank you.

Thank you for the detailed review of our work. We appreciate that you highlight different positive aspects of our work including our motivation to extend zero-shot NAS to the HW-aware settings, as well as appreciating our thorough experimental evaluation. We respond to each of your questions below.

...results shown are calculated using the estimated results from the "MetaPredictors"

We would like to provide a clarification here. While we indeed use the MetaPredictor to guide the differentiable search, the final hypervolume and Pareto-Front results presented throughout the paper are actually computed using the true accuracy, latency and energy values corresponding to different the benchmarks [1,2,3,4] that we use for evaluation.

What assumptions do the MetaPredictors make about the underlying devices when estimating latency and energy requirements?

We use precomputed latency and energy benchmarks from other papers [1,2,3,4,5] that have already been peer-reviewed. The authors of these benchmarks carefully control for number of allocated CPU cores to ensure consistent profiling across hardware devices (see HW-GPT-Bench [4] for example). Our MetaPredictor architecture itself makes no assumptions about the procedure followed in profiling these hardware metrics. However, this could be an interesting idea for improving the predictive performance of the MetaPredictor by appending such features to the hardware embedding.

Could the approach proposed by the authors be extended to include memory?

We thank you for raising this important point. The short answer is: Yes. MODNAS is agnostic to the type of hardware metric objective as long as we use a predictor to estimate the metric. Supporting a new objective like memory consumption is about replacing the existing latency/energy predictor with a GPU memory predictor. Following your suggestion, we are currently running experiments using MODNAS for memory-perplexity optimization on the GPT-L space from HW-GPT-Bench [4]. We will report back when this experiment finishes.

How exactly are "preference vector" and a "hardware device feature vector" defined?

• Preference vectors: During the training stage of MODNAS we sample a scalarization uniformly at random from the probability simplex (e.g. for 2 objectives a scalarization can be [0.24, 0.76])-- scalarizations are in [0,1] and their sum is 1. During inference, we sample 24 fixed points which are uniformly spaced on a circle (for 2 objectives) or a hypersphere (for >2 dimensions). Preference vectors are quantized ([0.24, 0.76] →[24, 76]) before being passed as input to the MetaHypernetwork.
• Hardware device embeddings: We use a simple scheme similar to HELP [5] to define the hardware embedding. We sample 10 fixed architectures from a search space, compute their hardware metric value (latency/energy) on a particular device and use this to compute the hardware embedding vector of size 10.

Which FPGA did the authors use?

For the experiments using latency and energy usage in FPGA, we utilize the precomputed values from HW-NAS-Bench [1], which describes the FPGA data collection procedure in their Appendix D.6. To quote the authors: “We then compile all the architectures using the standard Vivado HLS toolflow (Xilinx Inc., a) and obtain the bottleneck latency, the maximum latency across all sub-accelerators (chunks) of the architectures on a Xilinx ZC706 development board with Zynq XC7045 SoC (Xilinx Inc., b).”

Fig. 5: Why is MODNAS shown as an upper threashold line (+/- variance)?

Figure 5 depicts MODNAS as a single line because we do only one search run and in the end of it we evaluate 24 architecture samples using the same MetaHypernetwork conditioned on hardware devices. On the contrary, other black-box methods based on accuracy and latency predictors, sample a single new architecture at every optimization step and they need to run on every device independently since they do not incorporate hardware information during the search.

We would like to thank you again for reading our paper and your feedback. We hope that we were able to address all your concerns and that you will consider increasing your score after reading our responses. Otherwise, if you have additional questions, we are happy to follow up the discussion.

Thank you for carefully reading our paper, your detailed feedback and the positive score. We are encouraged to see that you identify several positive aspects of the work. We address your concerns and respond to each of your questions below.

Regulating the trade-off among user preferences with scalarization: In Figure 4, solutions on the Pareto frontier can be achieved by modulating user preferences. However, in HDX [1], which is a one-shot NAS with hardware constraints, claims that modulating hyperparameters linearly doesn’t lead to linearly distributed results. Is MetaHypernetwork free from this problem? Can real experimental results be plotted like Figure 4 to substantiate the integrity of MetaHypernetwork?

Following your suggestion, we have now plotted the Pareto front with the respective preference vectors in the same plot (https://anonymous.4open.science/r/MODNAS-1CB7/pareto_rays.png). We utilize one of our runs on the NAS-Bench-201 test devices, namely Eyeriss. As seen in the plots, the preference vectors and the points on the Pareto-Front are very aligned with each other, hence substantiating the integrity of the MetaHyperNetwork. Moreover, we have added a new Appendix K in the updated PDF describing this experiment. Thank you very much for this suggestion. Ultimately, this experiment helped us make our case stronger by validating the integrity of the generated solutions.

”network architectures the same as or near ground truth solutions may hard to be reach with, where other works can reach with huge search costs.” and “What the review wonders is whether can other works find near-GT solutions with a larger time budget?”

We agree with you that blackbox multi-objective optimizers can potentially reach the global Pareto front if the compute resources are not a concern and given enough time, it is not practical to train or even evaluate these architectures, especially for larger model sizes (e.g. Transformer spaces from HW-GPT-Bench). Sometimes in practice, the user wants to get a quick estimation of the Pareto front, and this is the use-case where MODNAS shines. Given enough budget, even random search will find a near optimal solution. For instance in NB201, the size of the search space is K=15625 architectures. The optimal theoretical number of random search steps $n$ to achieve a success probability $\alpha$ is approximately: $n \geq K ln(1/1-\alpha)$, therefore for random search to have a success probability more than 0.5 it requires $n \geq 10781$ iterations in theory. For the other guided search methods, this number is even smaller, though similar to MODNAS, they have the same limitation that they can converge to a local minimum. Nevertheless, we conducted the same experiment as the one in Figure 3 in the paper, but this time with the baselines given 4x more budget than MODNAS. You can find the results in this link: https://anonymous.4open.science/r/MODNAS-1CB7/radar_hypervolume.pdf. We updated the paper with Appendix M with this new result and the above discussion.

How are user preference and hardware device embeddings designed?

• Preference vectors: During the training stage of MODNAS we sample a scalarization uniformly at random from the probability simplex (e.g. for 2 objectives a scalarization can be [0.24, 0.76])-- scalarizations are in [0,1] and their sum is 1. During inference, we sample 24 fixed points which are uniformly spaced on a circle (for 2 objectives) or a hypersphere (for >2 dimensions). Preference vectors are quantized ([0.24, 0.76] →[24, 76]) before being passed as input to the MetaHypernetwork.
• Hardware device embeddings: We use a simple scheme similar to HELP [4] to define the hardware embedding. We sample 10 fixed architectures from a search space, compute their hardware metric value (latency/energy) on a particular device and use this to compute the hardware embedding vector of size 10.

Thank the authors for their kind answers and additional experiments.

I believe that the paper satisfies my acceptance threshold.

Therefore, I decide to maintain the rating of acceptance.

We thank you for carefully reading our paper and your detailed feedback on our work. We appreciate your recognition of the ability of MODNAS to adapt to a wide variety of hardware deployment scenarios. Below, we address each of your questions:

“Using Hypernetworks for NAS is well known but doesn’t seem a promising solution. It is like a heuristic solution.” and “Using hypernetwork for the NAS and pareto-frontier learning is well known.”

Thank you for referring to the different works which adopt hypernetworks for NAS. However, we would like to point out the following key differences in the way [A], [B] and [D] use hypernetworks compared to MODNAS: [A], [B] and [D] propose amortizing the cost of architecture training in NAS by learning a hypernetwork to directly generate the weights $w$ of a given architecture [A, D] or hyperparameter configuration [B]. MODNAS instead generates the parameters defining the distribution of an architecture i.e. the $\alpha$ values which are only a handful, e.g. 30 parameters for the NB201 supernetwork. Given the much lower dimensionality of the architecture distribution space $\alpha$ compared to the network parameter space $w$, the issues with scalability are greatly reduced. Furthermore, the hypernetwork itself is a simple embedding layer with very few parameters, e.g., 0.03MB for NB201 and this makes optimizing them and initializing them easier. Similarly, HyperST-Net [C] derives the parameter weights in a cascading manner from temporal and spatial characteristics. Again, since it generates network parameters which are usually much larger in number, it faces the same issues as mentioned above.

Hypernetwork is not performing well on unseen networks. https://arxiv.org/abs/2110.13100 With that, Could you elaborate on any specific constraints or limitations when transferring MODNAS to devices significantly different from the ones used in the training phase?

We refer you to the t-SNE plots (Figure 12 in the appendix) of computed hardware bank similarity vectors for the 19 devices (13 train and 6 test) on NB201. We observe that the embeddings indeed reflect that devices with similar properties are co-located. We study MODNAS on qualitatively different hardwares (see Table 4 in appendix for a complete list) during test (unseen) and training stages and the observation of transfer across different hardware devices does hold in practice. In addition, MODNAS shows good generalization performance with even a smaller subset of training devices (check figure 25 in appendix). We attribute this to the conditioning of the hypernetwork on the hardware embedding and the attentive device pool in the architecture of the MetaHypernetwork.

MiLeNAS paper (CVPR’20) shows that the gradient errors caused by approximations of second-order methods in bi-level optimization results in suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution. However, it seems that the authors still use a bi-level optimization technique. It seems using advanced techniques like MiLeNAS might be more useful.

We use the Reinmax [1] method, which is a straight-through estimator achieving second-order accuracy by integrating second-order numerical methods. Reinmax does not require Hessian computation, thus having negligible computation overheads. As it is impractical to employ methods that do not use discrete architectural samples during the search, especially on larger search spaces such as the Transformer ones, we chose Reinmax (NeurIPS’23), a state-of-the-art gradient estimation method for discrete variables. Nevertheless, since the type of architecture optimizer we use is a modular component, Reinmax can be replaced with the mixed-level formulation of MileNAS in our configurable experimental framework. We conducted such an experiment and added the result of MODNAS + MiLeNAS in the plots of Figure 15 (Appendix), where we compare it to Reinmax and GDAS as well. Reinmax is still the best performing optimizer.

Can the MODNAS be applied to object detection tasks as well?

Yes, the MODNAS algorithm is versatile if given an architecture search space and a set of hardware devices. In our paper we have selected benchmarks from previous works such as HELP [2], namely NB201, MobileNetv3 [3] for ImageNet classification, Transformer space for machine translation [4], where the ground truth latencies had been already profiled; process that took a significant effort from other researchers. We have also included the recent HW-GPT-Bench [5] for language modeling. If you are aware of any object detection benchmark which contains latency or energy measures on various hardware devices, we would be happy to include it in our experiment suite.