MOTE-NAS: Multi-Objective Training-based Estimate for Efficient Neural Architecture Search

Thank you very much for your valuable feedback and positive evaluation of our work.

Answers to Questions:

A1. Yes, by adjusting the K value in MOTE-NAS to extend the search time, we can achieve better architectures. While the current architecture obtained through our method does not have the highest accuracy on ImageNet, the significantly lower time cost is a considerable advantage.

Answers to Weaknesses:

AW1. We have included the core code of MOTE in the supplementary materials, and the complete code will soon be available on GitHub (the link is not shown due to the policy of NIPS 2024).

AW2. Please refer to A1.

AW3. Generally, current NAS research focuses on finding the architecture with the highest accuracy within a given search space (NASBench-101 and NASBench-201) as quickly as possible. Other factors such as parameters, FLOPs, latency, and other metrics can be considered in our cost function in a hybrid form.

AW4. The reduction strategies serve as acceleration methods for MOTE. When MOTE applies RD+RA, it does not experience significant performance loss in terms of "accuracy." MOTE can achieve substantial acceleration with minimal performance loss and still outperform other NAS methods, as demonstrated in Fig. 7 and Tab. 1 of our paper. Therefore, we emphasize the robustness of MOTE rather than the novelty of the reduction strategies. In TABLE III of the attached file, we replaced the random selection-based RD-RS, and MOTE still performed well. This further demonstrates that the novelty of the reduction strategies is not a primary factor affecting MOTE's performance.

Thank you for your feedback. The discrepancy arises due to differences in the search space and other hyperparameter settings. The configuration in our Tab. 2 was based on the experimental setup from ZiCO [1] (refer to their Table 2). Some of the values marked with an asterisk in our Tab. 2 are experimental results that we have reproduced.

[1] Li, Guihong, et al. "Zico: Zero-shot nas via inverse coefficient of variation on gradients." arXiv preprint arXiv:2301.11300 (2023).

Thank you very much for your valuable feedback.

Answers to Questions:

A1. To determine whether MOTE is affected by random weight initialization, an additional experiment was conducted, as shown in TABLE II of the attached file. The results indicate that the effect of random weight initialization on MOTE's architecture search is minimal. This is mainly because the landscape and speed terms used in MOTE are derived from a training procedure, making MOTE insensitive to weight initialization.

A2. To assess the sensitivity of MOTE to the use of skip connections, we selected four architectures from NASBench-201 to observe changes in accuracy and MOTE values with and without skip connections. These four architectures include Res-Conv(3x3) and Res-Conv(1x1), which use conv-3x3 and conv-1x1 with skip connections, respectively, and NoRes-Conv(3x3) and NoRes-Conv(1x1), which do not include skip connections. As shown in TABLE V and Fig. 2 of the attached file, the speed term, landscape term, and MOTE accurately reflected the changes in accuracy across the four architectures. When the accuracy increases, the values of the two terms and MOTE also increase, and vice versa. These experiments demonstrate that MOTE effectively captures the nonlinear architectures of DNNs.

A3. The specific process of RD is discussed in the global rebuttal. We encode each image in the original dataset using a VGG network trained on ImageNet (not fine-tuned on CIFAR-100), inspired by the Inception Score method. Although VGG can be replaced, its inherent knowledge as a simple and high-capacity CNN makes it suitable for image encoding.

A4. The MOTE also apply on the open search space(as shown Tab. 2 of our paper), where MOTE-NAS shows significant performance. To evaluate whether MOTE is suitable for other search spaces, an ablation study was performed on NASBench-301 (based on DARTS), as shown in TABLE I of the attached file. In this study, MOTE continues to outperform TSE and Synflow.

Answers to Weaknesses:

AW1. Please refer to A1.

AW2. Please refer to A2.

AW3. RD is a method used to accelerate MOTE. The selection method used in RD can be replaced by other methods. The ablation studies, shown in TABLE III of the attached file, indicate that MOTE performs well even if RD is replaced by other methods. Determining the best reduction method will be explored in future research.

AW4. In Fig. 7 and Tab. 1 of our paper, most comparison methods such as NASWOT, TE-NAS, KNAS, Zen-Score, and ZiCO are training-free (or zero-shot) NAS methods. However, the term "free" only applies to the search stage; these methods still require GPU time in the evaluation stage. The core idea of MOTE focuses on the time spent in the evaluation stage, which is more time-consuming than the search stage.

AW5. Please refer to A4.

AW6. MOTE-NAS-RS is a version that replaces the evolutionary algorithm with random sampling (please see Table A4 in the supplementary materials).

AW7. The limitations and contributions of this paper have been addressed in the global rebuttal and will be added to the camera-ready version if this paper is accepted.

Many thanks for your high appreciation of our work. We greatly value your feedback.

Answers to Questions:

A1. If our paper is accepted, all the typos you mentioned will be corrected and eliminated in the camera-ready version.

Answers to Weaknesses:

AW1. The RA design might be achievable by other algorithms, which remains an open question. However, the proposed RA is simple yet achieves promising results in NABench-101 and NASBench-201, demonstrating its effectiveness across different datasets.

Dear Reviewer XpJH ,

Please respond to authors and engage in discussion as soon as possible.

AC

Thank you very much for your valuable feedback. However, we noticed that you might have misunderstood the core idea of MOTE. Specifically, the landscape term does not reduce the proportion of $\theta_{init}$ (initial model weights) or increase the proportion of $\theta$ (trained model weights) as epochs increase. MOTE is composed of a macroscopic landscape term and a microscopic speed term. The formula for the landscape term is:

$\theta(g)=(\frac{g}{G})\theta_{init}+(1-\frac{g}{G})\theta,$

where $g$ ranges from 0 to $G$ and is independent of the number of epochs. Here, $G$ is a hyperparameter (default $G=10$), which indicates the granularity of the linear combination between $\theta_{init}$ and $\theta$. Therefore, even if the epochs for training candidate architectures increase, the proportion between $\theta_{init}$ and $\theta$ remains unchanged; only the position of $\theta$ in the loss landscape changes. The landscape term is designed to observe the loss landscape on a macroscopic scale rather than focusing on the minute variations in training loss (as TSE and the speed term do).

Answers to Questions:

A1. The number of samples ranges from dozens to hundreds. Since we used an evolutionary algorithm for dynamic sampling and conducted each experiment independently ten times to take the average, the specific numbers vary slightly each time. For instance, in NASBench-201, the number of samples for MOTE-NAS-EF ranges from 60 to 100. Some results of our paper in Fig. 1, Fig. 5, Fig. 7 (SynFlow, TSE), and the predictors in Tab. 1 and MobileNetV2/V3 in Tab. 2 are our reproduced experimental results. In contrast, other results directly refer to their corresponding papers. During the MOTE generation process, we did not use any data augmentation techniques (refer to the "GetTrainData" function in the "mote/gen_mote.py" code in the supplementary materials). Not all NAS algorithms follow a "search then evaluate" process; for example, WeakNAS does not follow this process. Therefore, the top-k hyperparameter does not apply to all NAS methods, making it impossible to categorize every NAS method as "evaluation-free" or "top-k".

A2. Plotting 2D or 3D loss landscapes with more information is computationally intensive [1], which is not conducive to efficient NAS. Indeed, the linear combination between two points does not fully capture the loss landscape, but it provides a convenient and quick assessment method. This method slices a 1D section through the loss landscape using two sets of model weights, and we can glean some of the landscape features by observing this slice [2]. The increase in epochs does not affect the proportion of $\theta$; please refer to the begin sentence. Our implementation of TSE already includes removing the initial 20% loss values (similar to what TSE-E and TSE-EMA do). In short:

1. The landscape term does not increase the proportion of $\theta$ when increasing epochs.
2. Since early loss values (20%) are eliminated in TSE, the proportion of $\theta$ increases with the epoch. Therefore, the increase in the proportion of $\theta$ is not why our proposed landscape term performs better than TSE. MOTE aims to describe the loss landscape from a microscopic perspective (speed term) and a macroscopic perspective (landscape term). The speed term can be considered a version of TSE that is aware of time variations.

A3. Detailed explanations of RD are addressed in the global rebuttal. The ablation experiments between the original dataset and the reduced one are shown in Fig. 5 of our paper. As the sampling hyperparameter $r$ decreases (RD equals the original dataset if $r=100$), the performance of both the landscape and speed term will degrade, but less than the "accuracy". The core idea of RD is simple and intuitive (see the global rebuttal). Other methods can build RD, the attached file shows the ablation study in TABLE III. Based on the study, MOTE still works well if RD is built by other methods. The best reduction method will be explored in the future.

A4. Cell-based search spaces require the combination of the predefined meta-architecture and the candidate cell to form a candidate architecture. The meta-architecture predefines the number of layers, downsampling method, other hyperparameters, etc. Different search spaces have distinct meta-architectures (e.g., NASBench-101 and NASBench-201) with huge time complexities. The goal of RA is to reduce redundant layers as much as possible, except for the cell layers. Although RA results in a simplified structure different from the actual candidate architecture (original meta-architecture + cell), it can construct an extremely lightweight and compact architecture to serve as a new meta-architecture for NASBench-101 and NASBench-201.

A5. MOTE-NAS-EF completely discards the evaluation stage to minimize total time consumption (search time + evaluation time). Since many training-free estimates are unreliable during the search stage, the evaluation stage needs to verify more architectures, leading to higher time consumption (e.g., KNAS). Currently, known training-free estimates are not truly "free" NAS methods.

Other ablation experiments to evaluate the performance of the landscape term and speed term are addressed in the attached file (see Fig. 1 and TABLE IV).

Answers to Weaknesses: Due to space limitations, these answers will be moved to global rebuttal.

[1]. Li, Hao, et al. "Visualizing the loss landscape of neural nets." Advances in neural information processing systems, 31 (2018).

[2]. Goodfellow, Ian J., Oriol Vinyals, and Andrew M. Saxe. "Qualitatively characterizing neural network optimization problems." International Conference on Learning Representations (2015).

I thank the authors for their response. To begin with, while I understand the idea of the MOTE-NAS paper, i seem to have misunderstood the way the linear combination of \theta and \theta_{init} is computed to arrive at \theta_{g}. Thank you for clarifying and correcting my understanding regarding g and G. In that case, how often do you evaluate \theta_{g} during the training? While I am familiar with the cell based search space, i now understood what you meant by the term meta-architecture.

A3. I understand that there are various methods to reduce the dataset. I wanted to know how the proposed k-means method compares to other subset selection methods that I pointed out. I see that you compared RD against random sampling in the ablation study.

A5. Algorithms such as TE-NAS, NASWOT, Zico, Zero-cost proxies for lightweight NAS rely only on the inference of a few batches during the search phase and evaluate the best architecture found. Like I mentioned earlier, the architecture found by MOTE-NAS-EF is worse than TE-NAS in table 1. Zico is also not included in Table 1.

While all the search spaces require the algorithm to search for a cell, some search spaces are more complicated than others. NASBench 101, 201 etc are all reduced search spaces when compared to the original DARTS search space. TransNASBench 101 is also a challenging search space. That was the reason, it is important to evaluate how well the proposed method performs on these search spaces

The authors do point out that the correlation of the landscape term and the speed term don't deteriorate as much as the accuracy term in the reduced setting, albeit on 1k architectures. It is important to see how well they can discern between the top few architectures that are close in performance. However, computing the landscape term and the speed term is still computationally expensive.

I thank the authors for answering most of my questions and I would like to increase my score