Encodings for Prediction-based Neural Architecture Search

We would like to thank you for your extremely thorough and insightful review. We agree with several of the concerns you have raised, and have done our best to address them. Specifically, we had made unsupported claims about the predictor design, that have now been fixed in our writing throughout the manuscript. We have also added further results for your consideration. Please find our full response below.

it is unclear whether the DGF module used actually solves the oversmoothing mechanism at all.

You are indeed right to point out that this issue affects GNNs in general, and that DGF does not necessarily solve the oversmoothing mechanism. We completely agree with this, and instead use empirical evidence to state that the skip connection in DGF may aid performance, and GAT ensemble seems to help. We believe there are 2 places where we make these claims that are not fully supported, we have now edited the text to remove these parts. Our paper’s key focus is on encodings for accuracy predictors within NAS, and our notes on predictor design were mainly done as a step to create a predictor to test out the performance of different encodings.

there is no guarantee an architecture encoding will be unique as it possible for two different architectures to have almost the same ZCP scores/latency/FLOPs/etc. but be different structurally. '

We completely agree. In fact, MultiPredict solely used ZCPs to encode architectures. These encodings do not provide uniqueness guarantees. This is one of the key reasons why, throughout our paper, we use these as supplemental encodings. These supplemental encodings are used in conjunction with the adjacency and operation matrices to build our accuracy predictors.

the use of "unique numerical indices" limits FLAN to a small number of predetermined search spaces

The training time for our predictor is very small (7.5 minutes). Thus, it is easy to simply re-index the operation matrices and re-train a predictor to transfer to a novel search space. To avoid confusion, we have also added a short sentence discussing the cost of re-indexing a predictor such that it does not need to incur large compute costs on new search spaces. We also add a small section detailing this in the Appendix (A.4).

"However, search on a global graph structure—one that encompasses all NNs—can be an intractable search problem due to its size." is very weak. and related work in this field called CDP [3] that is older than Multi-Predict/GENNAPE

We agree, thank you for bringing this to our attention. We have removed this weak claim and instead explicitly state our focus on cell-based NAS. We also acknowledge the work done in CDP for cross-domain prediction. We will try to compare to the CDP work–we are looking into it now.

I am skeptical of making strong claims like that on a t-SNE plot as its very hard to interpret what each axis is measuring

Thank you for bringing this to our attention. We have shifted this to the appendix. Further, we have regenerated the T-SNE plot with fewer search spaces, and scaled the marker sizes to correlate with parameter counts. We have also added visualisation for ZCPs. Additionally, we removed any strong conclusions we made from this graph, and instead validated our finding further by adding comments on explicit correlation measurements in the Appendix.

FLAN_{ZCP} sweeps performance for NB101 and ENAS, yet loses to TAGATES at the lowest data percent

Upon request by reviewer nwAh, we repeated these experiments with 9 trials instead of 3 (Table 2,3,4). We have added results for NASBench301 as well. These two modifications were to be more directly comparable to TAGATES as requested by reviewer nwAH.

Please note that out of the 12 tests comparing our performance with TAGATES, we only underperform by 0.0032 on the lowest data-percent on NASBench-201. In all other cases, FLAN ZCP outperforms TAGATES. We use this to show that we have a robust baseline predictor, and we offer further improvements by combining transfer learning and unified encodings.

you would expect it to either dominate quite conclusively…mutes the usefulness of FLOW.

We find that ZCP dominates most often, while sometimes CAZ may offer additional information by virtue of providing CATE and Arch2Vec encodings. Further, we justify/offer insights on the lukewarm result for NASBench-101 and NASBench-201 in the Appendix (Table 9) by empirically testing the kendall tau when using a single source space and multiple target spaces (Amoeba) and comparing those results with our current results in Table 4.

We would like to thank you for your insightful and thorough assessment of our manuscript. We believe our revision addresses several of your key concerns. As we approach the final stages of the author-reviewer discussion phase, we request you to check our responses and paper revision and let us know if there are any additional comments or questions that we can address.

Here is a quick summary of our response:

-- Address concerns about oversmoothing claims; edited manuscript to clarify the role of skip connections and GAT-DGF ensemble and focused on encodings for accuracy predictors within NAS.

-- Agree on limitations of ZCP/CATE/Arch2Vec not guaranteeing unique architecture encodings; our paper uses these as supplemental encodings alongside adjacency and operation matrices.

-- Discussed re-indexing and re-training of FLAN for new search spaces to address limitations in operation matrices and search spaces; added details in Appendix A.4.

--Revised weak claims and T-SNE interpretations, removing strong conclusions and focusing on empirical evidence; updated figures for clarity and re-ran key experiments with more trials for robust comparison with TAGATES and other methods.

-- Improved presentation of key tables and figures.

As the author-reviewer discussion deadline is closing soon, we would like to request your reconsideration for our manuscript, following significant revisions based on reviewer feedback. We have made an effort to address several concerns raised, by refining our claims, enhancing the clarity of the paper and providing more robust empirical evidence. I am pleased to inform you that reviewer nwAh has updated their assessment to a positive acceptance, and reviewer oKjh has decided to retain their score after reviewing our updates. Your feedback is crucial in guiding the final version of our paper.

As the author-reviewer discussion deadline nears, we kindly request your reassessment of our manuscript. We have further made a final revision for the rebuttal period, where we re-format our tables to look more consistent in formatting, which we hope further improves the quality of presentation.

We would like to thank the reviewer for their response before the rebuttal deadline. We understand several of your concerns, but we believe we have tried to address them previously, and we provide further elaboration below.

The response to these criticisms was to concede that their claims were false

The review process helps us refine our claims, and potentially identify unsupported statements. With your help, we have identified claims that are unsupported (not necessarily false), and adjusted them appropriately.

comment them out (DGF/GENNAPE/CDP) or rephrase what was said while moving it to the Appendix (t-SNE plots). In doing so the authors undermine themselves,

We benefit from the reviewer feedback on unsupported claims, and adjust our paper such that it better serves the NAS community. We move the plots to Appendix to make space for additional results requested by other reviewers. With your aid, we regenerated improved figures, and thus are able to have a more robust discussion in the appendix, and we add to the body of knowledge, and not to undermine ourselves. We also do not undermine our contributions, as we focus on encodings and their role in cell based predictor-based NAS, not necessarily designing the best graph neural network or addressing all forms of NAS.

If the benefit of DGF is just empirically higher performance, are there not other, simpler mechanisms by which this can be achieved?

Yes, we propose a new encoder that outperforms prior methods, we do not claim it is the best method possible in the field of graph neural network research. We empirically focus on neural network encodings.

(GAT got published by showings its better than GCN)

Again, we empirically focus on neural network encodings. The merit of adding GAT to the ensemble is not the sole contribution of this work.

'Score-based' being a better descriptor for FLAN's architecture representation was not convincing and half-hearted.

It is not a better descriptor, it is a supplementary encoding that was empirically shown to work well, and we use it as such.

'We have removed this weak claim and instead explicitly state our focus on cell-based NAS'

The claim was that global NAS is difficult, we only make this claim once, and thus, we removed it only once. This is part of our wider research question of enabling open-ended NAS but is not specifically addressed in this paper on NN encodings. Thus, we removed this sentence to keep the manuscript focused.

A similar rhetoric trick occurs when the author's respond to my question about the 7290 samples for NAS-Bench-101.

This is not true, the other tables present results on ALL NASBench-101 architectures. 7290 is explicitly to compare with TA-GATES and we still report full results.

If the claim about GENNAPE's global graph structures leading to intractable search is false, why use the unique numerical indices proposed with FLAN, which must be re-designed for new search spaces?

This is false, it does not need to be re-designed, re-indexing is trivial (counting and converting to one-hot) and search-space agnostic. We can indeed use ZCP or other metrics, but indexing is our selected method differentiating between operations in search space, and it works well as shown by our experiments. We definitely believe that this warrants further investigation in NAS.

Why not just concat ZCP/CATE/Arch2Vec info to their existing architecture encodings?

We explicitly do this, it is “CAZ”. However, as you mentioned (and we acknowledge), these representations are not unique, whereas indexes are unique by construction.

instead scattering NNs according to their computational complexity" has been replaced with "This proximity is influenced by the similarities in their parameter counts".

Computational complexity == FLOPs/Parameter Count, as described in the CATE [1] paper.

FLAN has a lot of instability, especially with the NAS-Bench-101 and NAS-Bench-201 plots.

This is explained in the Appendix, along with results for Amoeba.

"FLAN to enhance its accuracy by up to 64%" now being reduced to 47%, is a huge negative change.

We do not believe it is a huge negative change, as it still shows considerable improvement relative to prior work. This revision is the result of running more seeds as requested by other reviewers.

We thank the reviewer for their detailed and constructive feedback. We hope you will reconsider your score, keeping in mind that our work primarily focuses on the evaluation of neural architecture encodings in NAS. We believe that many of your remaining concerns are not regarding our core contribution on studying DNN encodings, rather the predictor design (created as a vehicle to study encodings) and some (valid and helpful) nitpicks on result presentation.

[1] Yan, S., Song, K., Liu, F., & Zhang, M. (2021). CATE: Computation-aware Neural Architecture Encoding with Transformers.

We would like to thank the reviewer for their constructive review of our work, and appreciate the valuable suggestions on further experiments to validate and justify our results. We have taken your concerns fully into consideration and added more results to address them in the paper and in the Appendix (highlighted in red in the revised manuscript). Below, we summarize the main changes and respond to your questions.

From table 4, this doesn’t seem to be the case, and each target space has its own source space

In the main paper, we hope to demonstrate that our framework supports transfer from an arbitrary source space to a target space. We believe that demonstrating this ability is key to universal sample efficient NAS. However, we also acknowledge the importance of demonstrating the transfer of a predictor from a single search space to multiple target spaces. With this in mind, we have added a table to the Appendix (Table 9), where we take Amoeba as our choice of source space, and transfer the trained predictor to NASBench-101, NASBench-201, DARTS, DARTS Fix W-D, ENAS, ENAS Fix W-D. Our conclusion further emphasises the trend which favours ZCP and CAZ supplementary encodings during transfer. We hope this helps in evaluating the effectiveness of predictor transfer.

Experiments on sample-based NAS are limited to NB101. Extending this (for example to NB201) would give a better assessment of the performance of FLAN.

We completely agree. We evaluate our NAS method on three search spaces in the main paper (Figure 5) and on all search spaces in the appendix (Figure 10). Please note that Table 6 is formatted in that manner to be consistent with prior literature (e.g. BRP-NAS). which report results on this NAS algorithm variant only on NASBench-101, and contextualise our NAS algorithm with respect to other relevant work.

To make a better comparison with TA-GATES, NB301 would be a natural choice to include in table 2.

We agree, one of the reasons why we did not include NB301 is because we were not able to trace back the networks used for training TA-GATES on NB301 to the original NB301 library. We have now added the NB301 results in Table 2, and FLAN ZCP performs consistently very well in our tests.

In Figs.5,6 the source search spaces ... specified. ...same as Table.4?

Thank you for bringing this to our attention. Yes, it is indeed the same as Table 4. We have added a clarification to the caption to explicitly state this.

In Table.15, for Amoeba, NB301 and TB101, what are the source search spaces used for pre-training FLAN^T?

Thank you for bringing this to our attention, we have added another table that shows the source to target search space mapping. These experiments will also be in ready-to-go scripts in our code for further clarity.

In the experiments of Table.2 (or Table.16) the ... of trials used for TA-GATES reported in the tables?

Thank you for bringing this to our attention. To fairly evaluate our work with TA-GATES, we have now re-evaluated our main tables for 9 trials. All our conclusions remain the same despite minor numerical differences.

Is Table.15 supposed to generalize ... See e.g. FLAN_ZCP for 128 samples on NB101.

There is a slight numerical mismatch because the larger table has fewer trials. As mentioned above, to match the trials with TAGATES, we updated our main paper results to be reporting on 9 trials.

predictor accuracy by 64% on average”, how can we see this from the tables?

These calculations are made as shown below:

>>> flanbase = [0.4424,0.3898,...,0.4876,0.5252]
>>> flanzcp = [0.4996,0.5655,...0.5231,0.5511]
>>> sum([100*(flanzcp[idx] - flanbase[idx])/flanbase[idx] for idx in range(len(flanzcp))])/len(flanzcp)
>>> 64.47

The numbers in the flanbase and flanzcp list represent the kendall tau correlation for tests in the table (full list excluded due to character limit but can be provided upon request), and we calculate the average of the percentage improvement in base v.s. ZCP supplementary encoding.

is the entire source space used for pre-training?

No, only 512 samples are used for pre-training on the source space. We detail this in the Appendix, but will also add this to the main paper for clarity. Thank you for pointing this out.

“Our FLAN^T_{CAZ} is able to ... by 2.12× .... shouldn’t this be FLAN_{CAZ}?

Thank you for the correction, you are indeed right. We have made the appropriate changes.

the search method Zero-Cost NAS (W) ... DARTS_{LRWD} is not described.

We have added a citation for this work. We have added the description in the appendix, thank you for bringing this to our attention.

In summary, we have added some cross-dataset and task results (Table 5), provided further context and clarification based on the suggestions and repeated experiments with more trials, which further support our findings.. Please let us know if we have address all of your comments, or if you have any further questions/concerns. Thank you.

We would like to thank you for your insightful and thorough assessment of our manuscript. We believe our revision addresses several of your key concerns. As we approach the final stages of the author-reviewer discussion phase, we request you to check our responses and paper revision and let us know if there are any additional comments or questions that we can address.

Here is a quick summary of our response:

-- Expanded the scope of transfer experiments to include Amoeba as a source space transferring to multiple target spaces, as detailed in the new Table 9 in the Appendix, demonstrating the effectiveness of predictor transfer across various search spaces.

-- Addressed the need for broader NAS evaluations by adding NASBench-301 results in Table 2, showing consistent performance of FLAN ZCP across different benchmarks. Also informed reviewer of our NAS evaluation in Figure 10.

-- Clarified the source search spaces used in Figures 5 and 6, and added a new table outlining source to target search space mapping for further clarity.

--Updated experimental trials to match those of TA-GATES for a fair comparison, and clarified numerical discrepancies and methodological details in response to specific queries about Tables 2, 15, and other results.

Thank you for your thoughtful and constructive review of our work. We appreciate your positive comments on the scale and significance of our study. We have taken your comments into consideration and added several results that further support our conclusions in the paper (highlighted in red in the updated manuscript).. Below, we respond to your questions and summarize our changes to the paper..

this is done only on CIFAR-10, right?

In this paper, we generate encodings for a large set of search spaces, one of them being TransNASBench-101 Micro, which covers several vision tasks. Additionally, our NDS results can be extended to ImageNet as well. In an effort to address this weakness, we have added two tables in the main paper (Table 5) which demonstrates the effectiveness of transferring predictors across tasks within TransNASBench-101 Micro, as well as across datasets (NDS CIFAR-10 to ImageNet).

Demonstrate the runtime of performance predictor

We believe that runtime performance is very important when prototyping search spaces for accuracy. Our training time for a transfer experiment is 7.5 minutes, and the transfer time (to adapt to a new domain) is less than 1 minute. Further, for inference, we are able to evaluate an average of 160 neural architectures per second. We have added this to the appendix (A.4).

deteriorates as you increase the number of samples.

We test this in more depth, specifically on the task of NDS CIFAR-10 to NDS ImageNet transfer, and find that in Table 5, more samples generally improves performance (On NDS DARTS, ENAS, NASNet and PNAS). There is some ambiguity when there are fewer than 8 samples (As seen on TB101), and this is because for such low sample-counts, the predictor performance variance could also be largely dictated by the specific NN architectures used for adaptation, and not necessarily just the learning algorithm. Future work could look at finding methods to sample better NN architectures for few-shot learning.

the authors concatenate a unique search space index to every operation. As far as I understand this requires to know apriori the number and type of search spaces

This observation is correct. However, we do not think that it is a major concern due to the efficiency of predictor training. One can simply re-index the search spaces and re-train the FLAN model in under 10 minutes on a single consumer GPU. Generating the Arch2Vec and CATE representations will take 20 and 40 minutes respectively. We believe that due to the simplicity and effectiveness of this approach, it is very reasonable to adapt the predictor to more search spaces. We have further added comments on this in the paper (Section 4.3)

Did authors evaluate their simple NAS method in section 5 (Table 5) on other search spaces except NAS-Bench-101?

Yes, we evaluate our NAS method on three search spaces in the main paper (Figure 5), and on all search spaces in the appendix (Figure 9 and 10). Please note that Table 6 is formatted in such a way to be consistent and directly comparable with prior literature (that also used the same format) in a clear manner.

In summary, we have made several improvements in the writing and graphs of the paper. Further, we have generated transfer results on TransNASBench-101 and NDS Imagenet, as well as benchmarked runtime performance in the Appendix. Upon request by Reviewer nwAh, we have updated the primary results in paper to average over 9 trials instead of 3. This is done to decrease variance and to be directly comparable to prior work (TAGATES) which also used 9 trials. All our conclusions stand unchanged. We believe this paper is of significant value to the NAS community, as we generate and catalogue encodings for over 1.5 million neural networks, and build a state-of-the-art predictor. We then take the predictor and combine encodings to improve sample efficiency of accuracy predictors by transfer learning. We demonstrate this effectiveness across search spaces, data-sets (CIFAR-10 to Imagenet) as well as computer vision domains (TransNASBench-101 Micro tasks). We also demonstrate NAS on several search spaces. We will open source our code to support future research in this field. We hope you have addressed your concerns, and would love to hear further feedback. Thank you!