Neural Architecture Retrieval

Q1 "Page 5 Section 3.4 Eq 4: ... without two stages."

A2 While sampling the highest-frequency motifs might seem reasonable in some contexts, this approach has limitations when dealing with architectures that share similar motifs. Relying solely on the most frequent motifs could lead to a lack of diversity in the learned representations, making it challenging for the model to discern the nuanced differences between various architectures. Our methodology, as described in the paper, addresses this potential issue. By considering all sampled motifs in the first stage, we ensure a comprehensive understanding of the architectural diversity. The subsequent concatenation of these motifs into macro graph embeddings, post motifs pre-training, allows for a more holistic and detailed representation. This approach not only captures the essence of the most frequent motifs but also incorporates the subtleties of less common yet potentially significant ones. This strategy ensures a richer and more effective learning process, as opposed to a singular focus on high-frequency motifs.

Q2 "Page 7 Section 4.3: The details of the evaluation part are lacking. Although it covers the mainstream information retrieval scores, the parameters of the computational score are still unclear. For example, when testing the NDCG, is graded relevance or non-graded relevance used? It would be better to provide a formula here."

A2 We reference these metrics due to their widespread application, and we employ non-graded relevance, which is prevalent in information retrieval tasks. Further details on the evaluation metrics are provided as follows: For $i$-th query:

• MRR = $\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{1}{r_i}$, where $|Q|$ is the total number of queries, $r_i$ is the rank of the first relevant architecture;
• MAP = $\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{1}{m_i}\sum_{k=1}^{m_i}P(k)$ where $m_i$ is the number of relevant architectures, and $P(k)$ is the precision at cut-off $k$ in the ranked sequence of architectures;
• NDCG = $\frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{Z_i} \sum_{k=1}^{n_i} \frac{2^{R_i^k} - 1}{\log_2(k+1)}$ where $R^{i}_{k}$ is the relevance of the result at position $k$, $n_i$ is the number of retrieved architectures and $Z_i$ is a normalization factor.

Q3 "Page 5 Section 3.4 Eq 4: What role does the context graph play?"

The roles of the context graph $G_c$ in the paper:

• The context graph $G_c$ is crucial for accurately representing motifs, particularly for generalizing to out-of-distribution (OOD) motifs. It extends the representation beyond individual motifs to include their contextual information.

• $G_c$ plays a key role in the contrastive learning setup. The context graph of a motif $G_s$ is considered a positive sample $G^{+}_c$, and the context graphs of other motifs serve as negative samples $G^{-}_c$. This setup is vital for learning distinct features of motifs.

• The context graph is used in the motifs-level pre-training objective. The contrastive loss function, which utilizes the embeddings from the GCN networks $F_s$ and $F_c$, is designed to effectively distinguish between $G^{+}_c$ and $G^{-}_c$, thereby improving the motif representation learning process.

Q4 "Page 6, Section 4.1: How were the real-world repositories collected, and how were the models obtained? Will the 12k real-world computational graphs, along with corresponding labels, be made available for follow-up work?"

We collect datasets from the Huggingface Hub and PyTorch Hub. To obtain the computational graphs, we forward the model and trace every operator in the neural network. Yes, the dataset will be made public for use.

Q1: "The proposed NAR appears to focus...Could the authors provide additional insights and elaborations on this matter?"

A1 We are genuinely grateful for your observation and find the reviewer's question to be extremely insightful. We agree that there is an inherent relationship among architectures, tasks, and datasets, and acknowledge that some architectures perform differently in various tasks. This paper provides an important first step in gathering architecture designs to build a database for retrieving similar architectures, laying the groundwork for further studies on their impactful insights. A general research study into their inner connections may be beyond the scope of this paper. However, we propose a potential solution: observing their overlap scores in clustered embeddings from different perspectives (architectures, tasks). For instance, we can first retrieval a set of neural architectures embeddings doing object detection and compute the center point of these embeddings as the approximated neural architecture embedding for object detection task. Now we derive a set of similar ResNet variants with their embeddings through neural architecture retrieval. Through computing the similarity between ResNet variants embeddings and the approximated neural architecture embedding for object detection task, we can find the ResNet variant that is suitable for object detection task.

Q2 "The authors introduce ... Could the authors provide further clarification and justification for this aspect of their methodology?"

A2 In our approach, we initially identify frequent subgraphs as motifs, enabling us to recognize identical motifs. When choosing negative context graphs, we extract them from context graphs with motifs at their centers that differ from the current one. Since the center motifs are different, the negative context graph cannot include the same motif. In this way, we can avoid this problem.

Q3 It is unclear ... Please provide more details.

A3 We are grateful for your observation. We express our sincere gratitude for your detailed and insightful review. To clarify, the ground truth comprises the fine-grained classes demonstrated in Fig 4 (left and right bars). We describe the ground truth on page 6 in the 'Data Processing' section, as follows: 'The key hints... are treated as the ground truth label of the neural network architecture.

Q4 "In the "Related Work" section...NAS methods, as referenced in [A-G]."

A4 We deeply appreciate your provide literatures.

We will cite the insightful literatures in the NAS section and here is the discussion:

In RL-based NAS, the search for an optimal neural network architecture is framed as a sequential decision-making process, where an RL agent learns to make decisions (e.g., choosing layers, connections, or hyperparameters) to maximize a reward signal, typically linked to the performance of the generated architectures on a validation set. This approach allows for automated and efficient discovery of high-performing neural network architectures tailored to specific tasks or datasets. However, the focus of the NAR problem is to directly retrieve architectures from an existing set. Integrating insights from both RL-based NAS and NAR could yield promising ideas, but such a synthesis might extend beyond the scope of this paper.

Q5 ​"In Page 2 line 8, “exact” should be “exactly”. In Page 2 line 11, “through” should be “by”."

A5 Heartfelt thanks for bringing this to our attention. We will correct in the revised version.

Q1 "...more application scenarios?"

A1 Heartfelt thanks for bringing this to our attention; the reviewer's question demonstrates great insight. Here are some applications scenarios:

• Architecture Hub for Diversity & Inclusion. NAR collect architectures from different source, help professionals locate and engage with existing diversity-focused architectures support for corrective project, to usage inspiration on reference architectures. When scaling up to millions of architectures, it still could quickly retrieve architectures accurately. It could serve as a search engine for finding related network designs, research studies, and current practices based on architecture designs.

• Architecture Application Discovery. In a typical scenario, a researcher or professional designs an architecture and wishes to understand the potential applications of this new design. Our pipeline can be utilized to retrieve similar models based on the architecture and obtain the performance scores of these retrieved similar neural networks. In the appendix, we provide a potential application. This demonstrates how users can retrieve similar architectures to determine the potential tasks for which their design could be applied or studied.

• Aid in Professional and Academic Search Architecture search also benefits the retrieval of related classifications, aiding in finding detailed tags for professionals and researchers associated with these similar architectures. Examples of such tags include 'FLOPs,' 'model size,' 'model parameters,' and 'operator statistics distribution.' These benefits are crucial in modifying designs. It is helpful in many professional fields where a deep understanding of architecture is essential. (Consider how an image-based Google search returns a list of tags at the top as hits to narrow down the search purpose.)

• Architecture Data Modeling and Storage. Pre-training architecture embeddings process serves as a new data storage format, allowing us to perform architecture manipulation similar to SQL schema. For instance, users can manage architectures via “group by” clause which clusters similar architectures together via measuring similarity among pretrained embeddings.

• Architecture Integration. With both retrieval and storage in NAR, architecture integration can be achieved. A new architecture database can be built up to meet personal ized requirements through collaborating with other applications. For example, in recommendation systems, subscribed users will receive push notification for the latest architecture innovations, based on their specific interests.

Q2 "Some papers[1-3] ... should be discussed."

A2 We are truly thankful for the list of papers provided. These works are insightful and valuable. We will cite these papers in the revised version of our paper. Below is a comparison discussion:

• [1] To obtain topological embeddings of neural architectures, it relies on auxiliary information such as the numbers of layers, channel expansion ratios, and kernel sizes. However, these details may not provide a comprehensive and precise description of the architecture. In contrast, our approach takes a more direct route by utilizing the graph structure itself to describe each motif in a neural network. This method avoids the limitations of incomplete auxiliary information, providing a more accurate and detailed representation of the architecture's topological characteristics.

• [2] This paper proposes to tackle graph link prediction via a meta-learning framework to select model with better performances on unseen graph. one of the concerns in METAGL is the lack of modeling node features. Different from the graph learning in METAGL, our work takes architectures as graphs and operations as nodes, which enables us to perform feature extraction of architectures from a perspective of motifs. Thus, this method provides a more reliable representation learning of network architectures, involving operations, connections, and motifs.

• [3] The paper introduces RGCF, a method using meta-learning for efficient retrieval of GNN architectures in collaborative filtering, focusing on architectural search and rapid adaptation to new scenarios. In contrast, our paper proposes a unique approach for Neural Architecture Retrieval, employing contrastive learning to cluster neural architectures based on design similarities, and focusing on identifying connections between different designs. Our method utilizes the computational graph structure of neural architectures, dividing them into motifs for a more accurate and detailed representation, totally distinct from the RGCF's reliance on meta-features and task-level data augmentation.

[1] Task-Adaptive Neural Network Search with Meta-Contrastive Learning. NeurIPS 2021.

[2] MetaGL: Evaluation-Free Selection of Graph Learning Models via Meta-Learning. ICLR 2023

[3] Retrieving GNN Architecture for Collaborative Filtering. CIKM 2023