Automated Search-Space Generation Neural Architecture Search

We thank the reviewer for the constructive reviews and recognizing the motivation, potential and soundness of our work. Please see our below early responses. We are preparing a revised manuscript that hopefully adequately address your mentioned concerns.

[W1. Revise Section 3 to improve the clarity by introducing more formal definitions.] Thanks, we will provide a revision following the suggestion.

We appreciate your suggestion and will provide a revision with better clarity and definitions to the mentioned terms. The preliminary explanations/definitions are presented as follows.

Valid/normal network: You are right. A valid network must be a directed acyclic graph, yet additionally require its layers obeying dependancy across adjacent operators. For example, the below CNN is a directed acyclic graph yet not a valid DNN since the output of 1st conv layer with 64 channels can not feed into the 2nd conv layer.

conv2d(3, 64)->conv2d(32, 128) # define in the manner of Pytorch

Saliency score and modular proxy: Upon the discovered search space of a general DNN, the saliency score calculation is flexible to various scenarios. We intentionally did not focus on a specific setting to avoid constraining the usage of the proposed framework. In general, the saliency score is designed upon the need of downstream tasks. For example, if fidelity is the main requirement, the score could measure from the optimization perspective. If efficiency on hardware is the main requirement, the score could be specifically designed to favor more on hardware.

Redundant groups: In our focused NAS scenario, the search space is defined as a set of removal structures. Our task becomes finding out important and redundant structures to form a (sub)-optimal network. It is achieved by grouping the trainable variables of each removal structure and yield hierarchical sparsity upon them. Then the redundant groups refer to the redundant removal structures with all trainable variables projected onto zero after sparse optimization by H2SPG.

Cosine similarity: We aim at identifying the redundant groups, which variables $x$ are projected onto $0$. One criteria for redundant identification is that after such projection from $x$ to $0$, the objective function does not regress. The cosine similarity between negative gradient $-\nabla f$ and the projection direction $-x$ measures the possibility of objective function regression. Lower similarility implies the projection will dramatically regress the objective functions thereby current group/structure is more important. Such score was discussed in the DHSPG paper, thereby we omitted detailed explanation in our work, yet we will include the explanation into our revision.

Trace graph: A trace graph for a DNN is a graph tracking the data flow of forward pass, serving as a detailed visualization of the DNN computational graph. We obtained it by calling Pytorch API as follows.

trace_graph, _ = torch.jit._get_trace_graph(model, dummy_input)
trace_graph = torch.onnx._optimize_graph(trace_graph, torch.onnx.OperatorExportTypes.ONNX)

Meanwhile, the visualizations of trace graphs are depicted in Figure 5-8(a) in our manuscripts.

Joint vertex: Joint vertex refers to the operators that aggregates multiple input tensors in the trace graph, such as add, concat, mul, etc.

Half-Space projection: Half-space projection is a projector that yields zero upon groups of variables without sacrificing objective function. We did not explain it in depth since it was introduced in other literatures of HSPG sparse optimizer.

[W2. Proposed procedure still requires a starting super-network.] We agree and will better position our work in the revision.

We agree that our work focuses on given a general DNN, how to automatically find out a (sub)-optimal subnetwork. Our approach indeed requires a starting point DNN that encompasses all candidate operators and connections. To more accurately reflect this focus and to eliminate any potential ambiguity, we will meticulously revise the title, naming and content of our paper. This change will better align with the core objective of our research.

[W3. Lack discussions to other automatic search space generation works.] We discussed and will move Appendix C into the main body.

Thanks for the comment. We respectfully note that we have discussed with prior automatic search space generation works in the Appendix C, except the latest works empowered by LLMs. The discussion was initially placed in Appendix C due to their search spaces defined distinctly to our scenarios. We will integrate this discussion to the main body in our revision.

Thanks,

Authors

Dear reviewer,

We have uploaded a revision with the suggested clarity improvements. We would appreciate if you could take some time to go through and assess it.

Meanwhile, here is our answer to the question.

Are there options to lift the requirement of pre-specifying a super-network or design a super-network also in an automated fashion?

The requirement could be lifted by

• using the existing popular human-designed DNNs, such as ResNet, DenseNet, etc,
• using the networks searched / generated by other NAS works, such as OFA,
• incorporating with LLMs to generate such super-networks. (not quite sure, since it is a new area.)

Thanks for your valued time and efforts.

Authors

I would like to thank the authors for taking my review into account. I have read their response and the changes to the paper. Related work on automatic search space generation works was added to the main paper, which is good. Unfortunately, some issues are still not fully addressed:

• clarity: while further textual explanation was added, the authors did only a minor revision and did not take my proposal of formally defining terms into account. In the current form, Section 3 still remains lengthy and confusing, lacking conciseness and clarity
• starting with a super-network: while the authors now reflect the limitation of having to start from a supernetwork in title and body of the work, it would be more desirable to make the approach actually open-ended. Sub-network search in a given supernetwork is closely related to structured pruning and would have to be compared to such approaches as well.

In summary, I keep my original assessment and encourage the authors to make a major revision of their work in terms of clarity and resubmit.

We appreciate the reviewer for reading and assessing the revision.

• We would like to say that we followed the suggestions of clarity by the reviewer to our largest extent. But due to the nine-page limitation, we had to make a trade-off and balance between making spacious formal definitions in the main body as follows

Definition 1. Valid Network A network is valid if and only if it is a direct acyclic graph...

or a new term followed by its textual definition in the revision. Our situation is difficult to include such formal definitions into the main text due to the space limitations and the large quantity of new terms, especially since our approach is a fresh NAS framework with a lot of difference to the existing NAS methods. In the future revision, we will include a section in Appendix to provide such formal definitions for each term Definition 1, 2, 3.

• Regarding discussion with pruning, we actually included two related automated pruning methods, OTOv2 (Chen et al., 2023), and TorchPruning (Fang et al., 2023) into the related works. We summarize them as searching over channel level inherent hyper-parameters.

We thank the reviewer for the insightful comments and suggestions. Our early responses are presented to the raised questions and suggestions. A revision is under preparation to hopefully adequately address the concerns.

[W1. Depend on a predefined super-network] We agree and will clarify our scope in the revision.

We agree that our framework requires a pre-defined super-network as starting point. We focus on how to automatically find out a (sub)-optimal subnetwork given a general super-network. To better position our work and clarify the scope and target scenario, we will carefully revise the title, naming, and related content in our manuscript.

[W2. Discussion with NAO and NAT.] Thanks, we will include.

Thanks for pointing out. We are familiar with these popular approaches, yet did not discuss with them due to the difference of the target scenarios. We agree that both methods and ours optimize the neural architecture and will include them in the revision.

[W3. Should compare with penalty methods that consider the sparsity constraints.]

We respectfully note that the H2SPG is the first stochastic sparse optimizer that considers and effectively resolves the hierarchical structured sparsity problem in DNN applications. In the classical deterministic optimization, the hierarchical sparsity constraint requires introducing additional latent variables then augments some regularization term into the objective function, while is impractical for DNN applications. Such impractice of introducing additional variables (doubling memory consumption) prohibits us from comparing with them.
Meanwhile, DHSPG is a hybrid optimizer that performs as a penalty method yet without consideration for the hierarchical constraint. DHSPG equips with a new projector to more effectively yield sparsity, while classical methods, such as proximal methods, are observed ineffectively in terms of sparse exploration for deep learning applications.

[W4. Trace graph creation is missing.] We call PyTorch API to obtain trace graph.

We ultilize PyTorch API to obtain the trace graph.

trace_graph, _ = torch.jit._get_trace_graph(model, dummy_input)
trace_graph = torch.onnx._optimize_graph(trace_graph, torch.onnx.OperatorExportTypes.ONNX)

The subsequent end-to-end pipeline and APIs include visualization, segment graph construction, variable partition, graph-aware sparse optimization, and sub-network construction are fully developed by us from scratch, which is noticeably engineering and algorithmic challenging.

[W5. More experiments on OFA and NAT] We are conducting them.

Thanks for the suggestion. We are conducting the asked experiments. Upon the experimental timeline, we will try to include them in the revised manuscript by the end of rebuttal, otherwise will post the numerical results here later.

[W6. The paper could benefit from a hyperparameter analysis.] We will include in the revision.

Thanks for the suggestion. We will include the sensitivities analysis regarding hyper-parameter selection in the revision.

[W7. Why the authors select HSPG rather than other sparse optimizers.] We select HSPG due to its superiority.

We select HSPG due to its superiority in sparse optimization comparing with other ones, such as proximal method or ADMM. The superiority of HSPG presents as maintaining competitive objective yet producing sparsity more effectively than others, and has been well recognized by the optimization community.

[1] An Adaptive Half-Space Projection Method for Stochastic Optimization Problems with Group Sparse Regularization

Thanks,

Authors

Dear reviewer,

We have uploaded a revision to tackle your comments/suggestions. The revision includes

• A section to discuss with neural architecture optimization in Section 2.

• More ablation studies against other sparse optimizers equipping with our hierarchical search phase in Table 4.

• More hyper-parameter sensitivity analysis over the length of hierarchical search phase and learning rate in Appendix D.

• Preliminary enhancement on OFA network in Appendix D. Due to limited bandwidth during this heavy period, we have not found bandwidth yet to complete the experiments over NAT, while would like to include it into the next revision.

Thanks again for your valued and constructive feedbacks.

Authors

We thank the reviewer for the constructive reviews, as well as for recognizing the motivation of our task and the presentation and readability of our manuscript. We provide the below early responses and are preparing a revised manuscript that hopefully adequately address your mentioned weaknesses.

[W1. The definitions for general DNN and search space are vague.] Thanks, we will highlight the definitions.

We thank the reviewer for the suggestions. We will highlight the definitions and add more illustrations in the revision.

General DNN: The general DNN serves as the starting point of our approach. As stated in Introduction, it refers to the DNN that covers all candidate operators and connections. Our goal is to search a sub-network given such a general DNN. It could be the DARTS widely studied in the gradient-based NAS realm, or other popular DNNs, such as ResNet, DenseNet, etc.
Search Space: As described in Section 3.1, the definition of search space is varying upon distinct NAS scenarios. In our context, the search space is defined as a set of removal structures upon a given general DNN. Our task then becomes how to find important components from the search space to form a (sub)-optimal network.

[W2. Missing important prior related works.] We respectfully disagree.

We respectfully note that our paper does include the key prior works you mentioned. In fact, AutoSpace was discussed in Appendix C, and RegNet was included in our numerical experiments. Our findings indicate that our method can even refine a crafted RegNet to find a better sub-RegNet, thus showcasing the effectiveness of our approach. In response to your feedback, we will integrate Appendix C into the main body, which were initially placed in the appendix due to their distinct search spaces compared to our focused scenario.

[W3. Results are not sufficient.] We will add more recent baselines.

We appreciate your suggestion and will include additional recent baselines in our revision. Furthermore, we will make our code publicly available to facilitate replication.

Thanks,

Authors

Dear reviewer,

We have uploaded a revision with the suggested clarity and related work improvements.

For the numerical comparison, we have updated upon recent results and added comparisons with more recent works (2023). One preliminary experiment on OFA searched network suggested by reviewer dxVy could be found in Appendix D.

We would like to highlight that our goal is to demonstrate the advantages of automation and ease-of-use, and reach competitive performance to the existing baseline methods. We actually did not pursue the best accuracy, thereby, there is no need for us to cherry-pick the compared methods.

Thanks for your valued time and efforts.

Authors

We thank the reviewer for the insightful comments. Please see our below early responses. A revision with more numerical results is under preparation.

[W1. Performance is not superior to other NAS methods in ImageNet DARTS.] We agree on accuracy, yet our method is scored higher in automation.

Thanks for the insightful question. We agree that our current accuracy on ImageNet DARTS is though competitive, not outperforming other competitors. To better support the efficacy, as suggested by Reviewer dxVy31, we are conducting more experiments starting from the networks searched by OFA and NAT to see if our framework could refine them further on ImageNet. If so, that could serve as additional evidence to demonstrate the efficacy of our approach.

Meanwhile, we would like to note that our approach is scored higher in terms of ease-of-use and automation. In particular, given a general DNN, the existing gradient-based NAS methods have to manually design and develop a search/training pipeline at first, which is not convenient. Ours is user-friendly with minimal engineering efforts from the end-users.

As the quote, there's no such thing as a free lunch, the superiority of automation for our framework sacrifices the accuracy performance at present. In particular, we automatize the search space discovery, while not yet automatize the introduction of auxiliary architecture variables. As a result, we are currently not performing multi-level optimization as the existing gradient-based NAS to achieve higher accuracy.

But, we are optimistic to the future and potential of our NAS framework. We believe that our future version would have more algorithmic and engineering breakthrough to outperform the existing gradient-based state-of-the-arts. Meanwhile, we sincerely hope to benefit the community and stimulate a new area in the NAS realm to search over automated discovered search space given a general DNN.

[W2. Does the approach apply to do fine-grained NAS.] Yes, it does.

Thanks for the great question. Yes, our framework is flexible to integrate with fine-grained NAS and can even jointly conduct layer/operator-wise and fine-grained NAS simultaneously. We did not discuss the fine-grained NAS in the manuscript in order to avoid the distraction from the main objectives, i.e., automatic layer/operator-wise NAS given a general DNN.

Thanks,

Authors

Dear reviewer,

We have uploaded a revision that tackle the suggestion.

In particular, we updated the recent numerical result along with one preliminary enhancement upon OFA's searched network in Appendix D. In the general response, we further explain that our automatically discovered search space actually serves as a super-set of DARTS. Therefore, there should exist better architecture in our context than the existing DARTS baselines. The current single-level optimization and saliency score design may be the bottleneck to effectively discover them out, which will be left as future work.

Regarding the integration of inherent NAS methods, we have explicitly highlighted it in Section 2 of the revision.

Thanks again for your valued and constructive feedbacks.

Authors