Structuring Benchmark into Knowledge Graphs to Assist Large Language Models in Retrieving and Designing Models

We sincerely thank the reviewer for the insightful comments and constructive feedback. We appreciate the reviewer for recognizing our novel idea and sufficient method design.

We have carefully considered the comments and revised the paper accordingly, revisions are in blue color. In the following reply, indecies of definitions, sections and equations refer to the revised manuscript.

Progress towards finishing the rebuttal:

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] JG3d-W1: Finished.

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] JG3d-W2: Finished.

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] JG3d-Q1: Finished.

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] JG3d-Q2: Finished

JG3d-W1: Labeling errors in Table 2

Reply to JG3d-W1:

We appreciate the reviewer for pointing out the labeling errors, we have corrected the notations in Tab. 2.

JG3d-W2: Limited experiments in the GNN domain

Reply to JG3d-W2:

Our framework is extensible to different data modalities and different application tasks. Because we propose a comprehensive ontology that clearly specifies how to organize the dataset features, model architectures, and the performance information into a knowledge benchmark graph (KBG) without requiring domain-specific knowledge. And our dataset similarity metric and model relevance metric are only dependent on the constructed KBG.

To better illustrate that our framework is extensible to various scenarios, we conduct an experiment on the NATS-Bench dataset [1]. NATS-Bench is a benchmark dataset on computer vision domain, containing performance records on 3 image datasets and 15,625 CNN topological architectures. Especially, we extract the features of image datasets by transforming the statistical information of raw images into a set of vectors and summarizing each statistical feature across all samples. The other construction procedure remains unchanged.

Currently, the experiment is ongoing, and we will provide preliminary results within the rebuttal period.

[1] 2021 TPAMI - NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size

JG3d-Q1: Lack of detailed description of large models

Reply to JG3d-Q1:

We have polished Sec. 7.6 and added Appendix A.6 for better clarification.

We utilized GPT-4 in our experiments, in line with other LLM-based AutoML baselines in our experiment.

When combining KBG and LLM to suggest model designs, we position LLM as a model design optimization tool that could understand and leverage the information retrieved from KBG. The detailed procedure is as follows:

1. Given a KBG and unseen dataset $D^u$, find similar datasets $\mathbb{D}^c$ to $D^u$ with dataset similarity score $\mathcal{S}(\cdot, \cdot)$.
2. Retrieve promising candidate models $\mathbb{F}^c$ with the model relevance score $\mathcal{R}(\cdot)$ based on $\mathbb{D}^c$.
3. Provide information of $D^u$, $\mathbb{D}^c$, $\mathbb{F}^c$ and historical results and instruct the LLM to infer (design a new model based on $\mathbb{F}^c$) or select (pick the most suitable model from $\mathbb{F}^c$) a model $F'$ for $D^u$.
4. Evaluate and record the performance of $F'$ on $D^u$.
5. Retrieve another set of $\mathbb{F}^c$ from KBG based on: 1) relevance $\mathcal{R}(\cdot)$ to $D^u$ and 2) similar model architectures to $F'$.
6. Repeat steps 3-5 until the computation budget is exhausted or the performance of $F'$ is satisfactory.

JG3d-Q2: Misleading name of Knowledge Benchmark Graph

Reply to JG3d-Q2:

We greatly thank the reviewer for pointing out the potential confusion caused by the name "Knowledge Benchmark Graph". We will polish our name in the final version.

eVNQ-W1.2: A case study can better illustrate how the algorithm succeeds

Reply to eVNQ-W1.2:

We have added the case study in Appendix A.7

We appreciate the suggestion that a case study can better demonstrate our method. To help understand our process, we first briefly introduce the pipeline of retrieving initial model designs from KBG for the unseen dataset below.

1. Given a KBG with becnhmark datasets $\mathbb{D}^b$ and unseen dataset $D^u$, we calculate the dataset similarity score $\mathcal{S}(D^b, D^u)$ between $D^u$ and each $D^b \in \mathbb{D}^b$ based on their statistical features.
2. Select the similar datasets $\mathbb{D}^c = \{\mathcal{S}(D^b, D^u) \geq \delta | D^b \in \mathbb{D}^b\}$ to $D^u$ up to a threshold $\delta$.
3. Calculate the relevance score $\mathcal{R}(\cdot)$ of the benchmark models $\mathbb{F}^b$ towards $D^u$ based on their performances on $\mathbb{D}^c$.
4. Select the candidate models $\mathbb{F}^c = \{\mathcal{R}(F^b) \geq \tau | F^b \in \mathbb{F}^b\}$ for $D^u$ up to a threshold $\tau$.
5. Recommend $\mathbb{F}^c$ to $D^u$.

For a case study, suppose cora dataset is the unseen dataset $D^u$, the more concrete steps are:

1. Calculate the dataset similarity between cora and other benchmark datasets, whose similarities are listed below.

1. Then set the threshold $\delta$ to 0.7 and obtain the candidate datasets $\mathbb{D}^c = \{\text{computers}, \text{photo}\}$.
2. Calculate the model relevance $\mathcal{R}(F;D^u)$ of the benchmark models towards cora based on their performances on $\mathbb{D}^c = \{\text{computers}, \text{photo}\}$. Suppose a model $F$ has the relative performance of 0.8 on computers and 0.7 on photo, then $\mathcal{R}(F;D^u) = 0.8 \times 0.735902 + 0.7 \times 0.724709 \approx 1.095$.
3. Select models with relevance scores higher than $\tau$.
4. Recommend the selected models to cora.

eVNQ-W1.3: There should be more insights and research issues beyond the similarity of datasets, like what features the algorithm should capture and consider.

Reply to eVNQ-W1.3:

Our work focuses more on dataset similarity because our method holds the assumption that similar datasets prefer similar models. This assumption sets the effectiveness of dataset similarity at the core of our framework. However, we agree that there are more insights or research issues that could be considered in the recommendation process, which we have presented in Reply to eVNQ-W1.1.

Currently, our method does not assign weights in utilizing the raw features and the features are directly extracted using graph statistical measurements. But we do agree that how to weight the dataset and model features in calculating the dataset similarity and model relevance is an interesting research issue. Because the importance of different features may vary under different application tasks. Regarding the wide possibility of unseen datasets, building an automatic feature selection mechanism is essential to improve the recommendation performance.

eVNQ-W2: Application scenario is relatively limited

Reply to eVNQ-W2:

Our framework is extensible to different data modalities and different designs of models. Because we propose a comprehensive ontology that clearly specifies how to organize the dataset features, model architectures, and performance information into a knowledge benchmark graph (KBG) without requiring domain-specific knowledge. And our dataset similarity metric and model relevance metric are only dependent on the constructed KBG.

To better illustrate that our framework is extensible to various scenarios, we are conducting an extensive experiment on the NATS-Bench dataset [1]. NATS-Bench is a benchmark dataset on the computer vision domain, containing performance records on 3 image datasets and 15,625 unique topological architectures. Especially, we extract the features of image datasets by transforming the statistical information of raw images into a set of vectors and summarizing each statistical feature across all samples. The other construction procedure remains unchanged.

Currently, the experiment is ongoing, and we will provide preliminary results within the rebuttal period.

[1] 2021 TPAMI - NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size

We sincerely thank the reviewer for the insightful comments and constructive feedback. We appreciate the reviewer for recognizing our novel idea and the completeness of our method.

We have carefully considered the comments and revised the paper accordingly, revisions are in blue color. In the following reply, indices of definitions, sections and equations refer to the revised manuscript.

Progress towards finishing the rebuttal:

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] eVNQ-W1: Finished.

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] eVNQ-W2: FInished.

eVNQ-W1: Lack some academic insights

Reply to eVNQ-W1:

We have revised our problem definition in Sec. 3 and evaluation on the dataset similarity in Sec. 6 to better present the insights of our work

We appreciate the reviewer’s feedback and would like to emphasize the unique contributions of this work, particularly in addressing the open challenge of leveraging benchmark knowledge for automated model retrieval and design.

1. Structured Knowledge Representation with the Knowledge Benchmark Graph (KBG): The Knowledge Benchmark Graph (KBG) offers a novel, structured approach to organizing benchmark data, encoding relationships between datasets, models, and their performance. This enables efficient retrieval and reasoning, addressing the limitations of brute-force model search in existing AutoML methods.
2. Dataset Similarity as a Core Insight: By introducing and formalizing dataset similarity $\mathcal{S}(\cdot,\cdot)$ , we provide a principled way to guide model selection based on performance relationships. This metric aligns with the intuition that similar datasets prefer similar models and offers a practical tool for improving model transferability.
3. Comprehensive Validation: The framework is thoroughly validated through experiments across various datasets and conditions, demonstrating its robustness and practical utility. This includes evaluating the effects of dataset similarity, hyperparameters, and KBG structure, providing empirical support for the proposed methods.

These contributions establish a strong foundation for further exploration.

eVNQ-W1.1: Controling and varying the hyperparameters bring limited insight

Reply to eVNQ-W1.1:

We acknowledge that hyperparameter variation, such as $\Delta$, $\epsilon$ and $\mathbb{D}^c$, primarily serves to validate the robustness of our framework and help understand how the dataset similarity can be evaluated.

While the current analysis focuses on the empirical behavior of the framework, we recognize the importance of linking these results to deeper insights. For example, although the model architectures do not have obvious relationships with their performance on different datasets, our experiments demonstrate that such relationships exist (Like in Figure 4 (a), better performing models tend to maintain its advantage across datasets). This observation motivates further research into the underlying factors that influence model performance across datasets.

For another example, simply aggregating information from KBG with our simple dataset similarity and model relevance metrics (results in Tab. 2 and Tab. 4) already yields promising results. This intrigues future work to further explore more effective and efficient retrieval methods for KBG to enhance the recommendation process.

The article is indeed very inspiring, but it still has some shortcomings at present. Even after the rebuttal, the issues were not completely resolved, so I will maintain my current rating.

We sincerely thank the reviewer for the insightful comments and constructive feedback. We appreciate the reviewer for recognizing our clear presentation and thorough experimental evaluations.

We have carefully considered the comments and revised the paper accordingly, revisions are in blue. In the following reply, indices of definitions, sections, and equations refer to the revised manuscript.

Progress towards finishing the rebuttal:

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] zioe-W1: Finished.

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] zioe-W2: Finished.

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] zioe-W3: Finished.

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] zioe-Q1: Finished.

[⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛] zioe-Q2: Finished.

zioe-W1: Data similarity score is too simple

Reply to zioe-W1:

We want to emphasize that how to design the dataset similarity score is not the focus of our work, as noted in Sec. 5. Our contributions are to identify the novel problem in AutoML of efficient model selection for unseen datasets using existing benchmark knowledge (clarified in Sec. 3), construct a novel knowledge graph ontology for organizing benchmark data (described in Sec. 4), and provide clear problem definition and comprehensive evaluation metric for designing future dataset similarity scores (clarified in Sec. 6). We have revised Sec. 3 to Sec. 6 to better clarify our contribution and the necessity of our design.

Given our method framework, we intentionally designed a simple dataset similarity score to validate our novel problem while aligning with our objective of efficient model retrieval. Designing an effective and efficient dataset similarity score is indeed an open challenge, and our work offers a practical starting point for future research. We acknowledge that leveraging the relational information in the KBG is a plausible way to enhance the quality of dataset similarity score, and have provided a related experiment in Reply to zioe-Q2 to support subsequent studies.

zioe-W2: The name model relevance score is confusing

Reply to zioe-W2:

We have revised Sec. 5 for a clearer introduction to the model relevance score.

After identifying benchmark datasets similar to the unseen dataset $D^u$ , our goal is to retrieve models from the Knowledge Benchmark Graph (KBG) that are likely to perform well on $D^u$ . This retrieval process can be intuitively understood as evaluating how relevant the models are to $D^u$ . Accordingly, we name this retrieval score $\mathcal{R}(\cdot)$ “model relevance” to properly reflect its objective.

When calculating $\mathcal{R}(\cdot)$, we exclude architectural information because there are no existing performance relationships between benchmark models and $D^u$. As a result, model architectures provide limited insight into determining relevance to $D^u$. Instead, based on the assumption that “similar datasets prefer similar models,” the historical performance of models on benchmark datasets serves as a proxy for their effectiveness on $D^u$, provided the datasets are sufficiently similar. Therefore, we define $\mathcal{R}(\cdot)$ using the historical performance of a model on similar benchmark datasets to $D^u$, weighted by the dataset similarities.

zioe-W3: Vague description of LLM+KBG method

Reply to zioe-W3:

We have polished the description of the LLM+KBG method in Sec. 7.6 and provided more details in Appendix A.6.

When combining KBG and LLM to suggest model designs, we position LLM as a model design optimization tool that could understand and leverage the information retrieved from KBG. The detailed procedure is as follows:

1. Given a KBG and unseen dataset $D^u$, find similar datasets $\mathbb{D}^c$ to $D^u$ with dataset similarity score $\mathcal{S}(\cdot, \cdot)$.
2. Retrieve promising candidate models $\mathbb{F}^c$ with the model relevance score $\mathcal{R}(\cdot)$ based on $\mathbb{D}^c$.
3. Provide information of $D^u$, $\mathbb{D}^c$, $\mathbb{F}^c$ and historical results and instruct the LLM to infer (design a new model based on $\mathbb{F}^c$) or select (pick the most suitable model from $\mathbb{F}^c$) a model $F'$ for $D^u$.
4. Evaluate and record the performance of $F'$ on $D^u$.
5. Retrieve another set of $\mathbb{F}^c$ from KBG based on: 1) relevance $\mathcal{R}(\cdot)$ to $D^u$ and 2) similar model architectures to $F'$.
6. Repeat steps 3-5 until the computation budget is exhausted or the performance of $F'$ is satisfactory.

Specifically, Infer refers to the prompt instruction for LLM to design a new model based on the observed patterns among $\mathbb{F}^c$, while in the Select process, LLM is asked to select the most suitable model from $\mathbb{F}^c$ based on the context.

zioe-Q1: Detailed explanation of LLM-assisted method

Reply to zioe-Q1:

We provided more method details in Appendix A.6.

We adopt an intuitive and effective prompting practice without excessive tuning because we aim to validate the effectiveness of our KBG as extra knowledge in assisting LLMs design models. Continuing from the example process in Reply to zioe-W3, our answers to the questions are listed as follows:

(a) Following [1], [2], [3], our prompt design is organized into five textual components:

1. Task description
2. Model space description
3. Optimization trajectory
4. Candidate models from KBG
5. Role-play instruction for Infer/Select strategies

These textual segments are updated (specifically, components (3) and (4)) and combined into a single prompt template during each iteration before sending to GPT-4 API for the response.

(b) The output of LLM, phrased using OpenAI’s structured output, contains three fields (using NAS-Bench-Graph as an example):

1. Structure topology of the recommended model
2. Layer operations of the recommended model
3. Reason for the recommendation.

After receiving the response, we extract the structure topology and layer operations to build the model automatically. Then, the recommended model is tested on the unseen dataset to obtain the performance feedback, which will be further combined with the model details and appended to the optimization history in the prompt for the next iteration. More importantly, the recommended model will serve as the anchor for retrieving the next round of similar models from KBG, thereby improving the model performance over time.

(c) The only constraint applied to LLMs is that they cannot recommend a model already tested in the optimization history, which can be enforced by proper instruction. Please note that our KBG can capture the similarity between datasets and models, which is fundamentally a filtering mechanism that only retrieves the most relevant and effective knowledge before sending them to LLMs for reference. Therefore, we did not apply further filtering to LLM’s suggestions so that the effectiveness of our KBG in this study could be directly observed.

The corresponding details are added to the revised manuscript in Appendix A.6.

[1] 2023 - Graph Neural Architecture Search with GPT-4

[2] 2023 - Heterogeneous Graph Neural Architecture Search with GPT-4

[3] 2024 - Computation-friendly Graph Neural Network Design by Accumulating Knowledge on Large Language Models

zioe-Q2: It's good to include more complicated design of the two scores.

Reply to zioe-Q2:

We appreciate the suggestions for incorporating more complex information in the data similarity score and model relevance score.

More complex dataset similarity

We realize more dataset similarity metrics should be included for a more comprehensive ablation study.

Thus, we propose to apply Personalized PageRank (PPR) algorithm to the relations among unseen dataset $D^u$ and benchmark datasets $\mathbb{D}^b$ to better identify the similar benchmark datasets to $D^u$. The original similarities $\mathcal{S}(D^b, D^u)$ calculated by the dataset features serve as the initial weights of edges. Considering that the unseen dataset $D^u$ does not have model performance information, we neglect the performance relations between $\mathbb{D}^b$ and benchmark models $\mathbb{M}^b$ as they are less helpful for dataset similarity.

The experiment results are shown below, where the reported values are the relevance linearity score (RLS) proposed in Sec. 6.2 for evaluating the dataset similarity metrics. A higher RLS indicates that the dataset similarity metric can better find a similar dataset. The best values are shown in bold and second best in italic.

Results show that, when further considering the edge information, the PPR algorithm significantly improves the dataset similarity quality over the naive methods. This further validates the essence of constructing the knowledge benchmark graph. However, the linearity is still not high enough (maximally 1), indicating that there is still room for improvement in the dataset similarity metric design.

More complex model relevance score

We included the utilization of architectural information during model refinement in Appendix A.6.

Following the response to Reply to zioe-W2, architectural information is unsuitable for the initial stage of model retrieval. However, when KBG is combined with LLM to iteratively refine models, the suggested models offer a chance to include architectural information. And we have already combined the model-architecture similarity with historical-performance similarity to give a more comprehensive recommendation, which is described in step 5 of the example in Reply to zioe-W3.

The architectural similarity $\mathcal{W}(\cdot)$ is calculated by the average number of similar architecture design elements between benchmark models and the currently suggested model $F'$. And we use $\beta \mathcal{R}(F; D^u) + (1-\beta) \mathcal{W}(F, F')$ as the score to rank benchmark models and pick the candidate models $\mathbb{F}^b$ for further recommendation, where $\beta$ is the weight hyperparameter and set to 0.8 in practice.

Dear Reviewer,

Thank you for your efforts in reviewing our paper and providing thoughtful feedback. In case you missed our previous responses, we briefly summarize our responses to your comments to help you catch up quickly.

First, we want to clarify a misunderstanding about our primary contribution. Our work aims to identify and validate a novel problem in AutoML, specifically the challenge of efficient model selection for unseen datasets using existing knowledge. While we propose initial methods to address this problem, our main contribution lies in bringing attention to the problem itself, formating a clear definition and demonstrating its importance through empirical validation.

Then, regarding to your two major concerns, our responses are summarized below:

1. Dataset Similarity and Model Relevance Scores

We appreciate your suggestions on enhancing the dataset similarity and model relevance scores. Our current simple designs for $\mathcal{S}(\cdot)$ and $\mathcal{R}(\cdot)$ are intentional, serving as baseline methods to initiate discussion on this new problem. We aimed for efficiency and practicality, enabling quick model recommendations without extensive computations. Incorporating complex metrics, such as edge-level information or architectural details, could indeed improve accuracy but may not align with our goal of proposing an accessible starting point for future research.

To address your concerns and facilitate further studies, we have:

• Clarified the purpose of our simple scoring methods in Reply to zioe-W1 and Sec. 5, emphasizing their role as foundational tools rather than final solutions.
• Added a more complex dataset similarity metric using the Personalized PageRank algorithm in Reply to zioe-Q2, demonstrating potential improvements and encouraging exploration of more sophisticated methods.
• Clarified the difficulty in using architectural information in the initial model retrieval due to the lack of performance relationships, as explained in Reply to zioe-W2. And described how architectural information is incorporated during model refinement in Reply to zioe-Q2 and Appendix A.6.

2. Details on LLM-Assisted Model Selection

Regarding the LLM-assisted method, we acknowledge that the initial description lacked detail. Our intent was to illustrate how Large Language Models (LLMs) can be integrated into the model selection process as a complementary tool rather than the primary focus.

In response, we have provided a detailed description on the LLM integration in Reply to zioe-W3, Reply to zioe-Q1, and Appendix A.6, outlining the prompts used, how outputs are processed, and any constraints applied.

We are grateful for your insights, which have helped us improve the clarity and depth of our manuscript. Please feel free to reach out with any additional questions or suggestions.

We sincerely thank the reviewer for the insightful comments and constructive feedback. We appreciate the reviewer for recognizing our contribution to reducing time and expertise required for designing models by proposing a comprehensive graph dataset modeling data-model-performance relationships, well-articulated theoretical framework, and thorough experimental evaluations.

We have carefully considered the comments and revised the paper accordingly, where revisions are in blue. In the following reply, indices of definitions, sections, and equations refer to the revised manuscript.

NmMt-W1: The theoretical explanations could be improved with additional background to aid reader comprehension.

Reply to NmMt-W1:

We have reorganized the manuscript to present our theory and methods more clearly and intuitively in Sec. 3 to Sec. 6.

• Sec. 3: Summarizes the overall problem definition, providing a foundation for the subsequent sections.
• Sec. 4: Introduces the construction of the Knowledge Benchmark Graph (KBG) and highlights its advantages in organizing benchmark data.
• Sec. 5: Details our proposed model retrieval method based on the KBG.
• Sec. 6: Focuses on the core of our framework, dataset similarity. We formally define the problem of designing a dataset similarity score (formally Def. 1) and propose a novel similarity evaluation metric.

NmMt-W1.1: An overview of existing problem formulations in model transfer or AutoML is helpful

Reply to NmMt-W1.1:

We have included the problem definition of AutoML in Sec. 2 and provided a comparison between our problem formulation and existing AutoML and model transfer methods in Sec. 3.

Briefly, our method focuses on retrieving models for unseen datasets using benchmark data, avoiding the computational overhead of model training. In contrast:

• Existing AutoML methods require training models as part of the search process.
• Existing model transfer methods address data scarcity by reusing a specific model trained on a source domain for a target domain.

To compare our problem with existing AutoML and transfer learning methods with greater detail and clarity, we first briefly present our definition. Given an unseen dataset $D^u$, searching for an optimal model $F^*$ from possible models $\mathbb{F}^b$ that has the best performance $\mathcal{M}(F;D^u)$ can be formulated as:

To address the efficiency issue in searching models, we adopt the intuition that similar datasets prefer similar models and leverage information from benchmark data to simplify the problem, which contains a set of models $\mathbb{F}^b$ with known performances on datasets $\mathbb{D}^b$. Based on the benchmark data, we transform the problem into evaluating the similarity between $D^u$ and $\mathbb{D}^b$ and retrieving the most relevant model from $\mathbb{F}^b$ for $D^u$. Our problem is defined as:

where $\mathcal{S}(\cdot, \cdot)$ is the similarity score between datasets.

Comparison with AutoML

The AutoML methods focus on designing good model searching algorithms $\pi (\cdot)$ that samples models from the model space $\mathbb{F}^b$ to find $F^*$, which is formulated as:

where $\text{E}[\cdot]$ is the expectation over the model performances. The optimization process of the problem is formulated as:

where $\mathbb{F}^c$ is the candidate model set for improving $\pi(F)$.

During optimization, a larger $|\mathbb{F}^c|$ increases the model recommendation quality from $\pi(F)$. However, enlarging $\mathbb{F}^c$ also increases the computation overhead of training models to obtain $\mathcal{M}(F,D^u)$. Additionally, $\pi(F)$ faces the cold start issue to search models from scratch for unseen datasets.

Our problem differs from AutoML methods in reducing the cost of obtaining $\mathcal{M}(F, D^u)$ by approximating it with retrieved information from benchmark data, leading to a zero training cost for model recommendation. Besides, the output of our problem provides a good initial model for AutoML methods to ease the cold start issue.

Comparison with Model transfer

The existing model transfer methods focus on transferring the model parameters $\omega$ from the source domain $D^s$ to the previously unseen domain $D^u$ to reduce the data sparsity issue. The model transfer problem can be formulated as:

where $F^s_{\omega}$ is the model trained on $D^s$ and $\mathcal{L}(\cdot)$ is the transfer learning loss function. Designing an effective $\mathcal{L}(\cdot)$ to better adapt $F^s_{\omega}$ to $D^u$ is the major focus of model transfer methods. Different from them, while our method focuses on efficiently recommending model designs to $D^u$ based on the benchmark model performances.

NmMt-W1.2: Explain the motivation of defining the problem with a probability lower bound and its relation to practical model transfer

Reply to NmMt-W1.2:

We have revised the presentation of dataset similarity theory definition in Sec. 6 to enhance clarity and motivation.

Designing an effective dataset similarity metric is an open challenge with various potential approaches. Relying solely on the transfer performance of a single model can lead to significant bias and fail to capture the broader relationship between datasets. To address this, we consider the transfer performances of a set of candidate models $\mathbb{F}^c$ to provide a more comprehensive evaluation of dataset similarity.

Intuitively, similar datasets should support the successful transfer of a large number of models with high performances. This relationship can be quantified by a higher probability $\Delta$ of successful transfers within a small performance drop tolerance $\epsilon$. To reflect this, our definition of dataset similarity incorporates a probability lower bound, which provides a minimum level of confidence for successful model transfer within the specified transfer loss tolerance.

In practice, the model retrieval performance heavily depends on whether the models of benchmark datasets perform well on unseen datasets. Our probability lower bound definition can robustly reflect such characteristic and provide guidance for future similarity metric designs.

NmMt-W1.3: Indicate whether this problem formulation is novel or based on existing methods, and specify its advantages

Reply to NmMt-W1.3:

Our problem of suggesting promising models for unseen datasets by retrieving from benchmark data without training is novel problem and distinct from existing AutoML and model transfer methods. In particular, our definition and evaluation of the dataset similarity metric in Section 6 are key contributions to the field.

There are prior AutoML methods [1] [2] that utilize the benchmark data to train a surrogate model for predicting the performance of candidate models on unseen datasets. These surrogate models are trained by using the dataset and model features as the input and the model performances as the ground truth. However, as discussed in Sec. 2 of the paper, the computation cost of building surrogate models is still unavoidable.

Moreover, these methods also operate under the assumption that “similar datasets prefer similar models,” which makes their effectiveness heavily dependent on a robust dataset similarity measurement. Unfortunately, they lack a formal definition of dataset similarity and overlook the exploration of how dataset similarity impacts the performance of recommended models. This gap limits their comprehensiveness and highlights the novelty of our approach.

In contrast, our method bypasses the need for training by directly retrieving relevant models from benchmark data. This design makes our approach significantly more efficient and scalable to much larger benchmark datasets, setting it apart from surrogate model-based methods.

[1] 2013 ICML - Collaborative hyperparameter tuning

[2] 2021 NeurIPS - Automatic Unsupervised Outlier Model Selection

NmMt-W2: In Section 4.3, the intuition behind the transferability score could be further clarified.

Reply to NmMt-W2:

We have polished Sec. 6.1 to offer a clearer background of the transferability score and added a conceptual explanation in Sec. 6.2 to address the whole Weakness 2.

In our paper, we define two datasets as similar if many models perform well on both. To evaluate different dataset similarity metrics, it is crucial to obtain a ground truth for how models transfer between datasets. To this end, based on our definition of dataset similarity in Sec. 6.1, we introduce the transferability score $\mathcal{T}(\cdot)$ to measure the success rate of high-performing transfers between benchmark and unseen datasets.

NmMt-W2.1: A conceptual explanation of the transferability score with an example would be beneficial

Reply to NmMt-W2.1:

Simply saying, given a set of candidate models $\mathbb{F}^c$ selected from benchmark dataset $D^b$ and transferred to unseen dataset $D^u$, a higher transferability score $\mathcal{T}(\mathbb{F}^c)$ reflects a higher success rate $\Delta$ of transferring $\mathbb{F}^c$ to $D^u$ within a smaller performance drop tolerance $\epsilon$. (Please refer to Reply to NmMt-W1.2 for a detailed introduction of $\Delta$ and $\epsilon$.)

An example of estimating $\hat{\mathcal{T}}(\mathbb{F}^c)$ is presented as follows.

Suppose when transferring $\mathbb{F}^c$ from $D^b$ to $D^u$, the curve between the success transfer rate $\Delta$ and the performance drop tolerance $\epsilon$ is empirically evaluated on $\epsilon = \{0.01, 0.02, 0.03, 0.04, 0.05\}$ as follows:

Then the estimated transferability score $\hat{\mathcal{T}}(\mathbb{F}^c)$ is calculated as:

$\hat{\mathcal{T}}(\mathbb{F}^c) = \frac{1}{5} \sum^{0.05}_{\epsilon=0} \Delta(\epsilon) = \frac{1}{5}$ (15% + 40%+ 70% + 80% + 90%) = 0.59.

If $\mathbb{F}^c$ has a larger transferrability, it means that $\Delta$ gains a higher value at each $\epsilon$, e.g. $\Delta$ increases from 15% to 20% at $\epsilon =0.01$. Ideally, when $\mathbb{F}^c$ is perfectly transferrable, $\Delta$ should be 100% at all $\epsilon$, meaning that $\mathbb{F}^c$ perform equally well both $D^b$ and $D^u$.

NmMt-W2.2: Comparing this score with existing metrics for evaluating model transfer effectiveness could further clarify its utility

Reply to NmMt-W2.2:

Given the novelty of our problem, there lack of related metric designs for evaluating model transfer effectiveness. So we compare our metric with two intuitive methods:

Performance-difference based metric $\mathcal{T}_{diff}(\cdot)$ measures how performances of $\mathbb{F}^b$ on $D^u$ differ from those on $D^b$:

$\mathcal{T}_{diff}(\mathbb{F}^c) = \frac{1}{| \mathbb{F}^c |} \sum ( \mathcal{M} (F;D^u) - \mathcal{M} (F;D^b) )$,

where the summation is for $F \in \mathbb{F}^c$. $\mathcal{T}_{diff}(\mathbb{F}^c)$ becomes larger when $\mathbb{F}^c$ performs better on $D^u$. The average performance difference is vulnerable to outliers when a few models perform significantly better or worse than the others. Thus fails to reflect the overall landscape of model transferability.

Rank-correlation based metric $\mathcal{T}_{rank}(\cdot)$ measures how the performance rankings of $\mathbb{F}^c$ on $D^u$ and $D^b$ are correlated, which may take the form of Spearman's rank correlation coefficient. $\mathcal{T}_{rank}(\mathbb{F}^c)$ becomes larger when $\mathbb{F}^c$ orders more similarly in performance on $D^u$ and $D^b$. Unfortunately, $\mathcal{T}\_{rank}(\cdot)$ overlooks the scale of performance differences. When two models order differently on $D^u$ and $D^b$ but have a neglectable performance difference, they should still be consistent, yet $\mathcal{T}\_{rank}(\cdot)$ will give a low score.

Compared with the two metrics, our metric reduces the effect from outliers by focusing on the successful transfer rate and handles performance-difference scales by considering multiple performance drop thresholds. Thus, our metric is more robust to different situations and can more honestly reflect the model transferability.

NmMt-W3.3: Comparing this approach to other recent methods that incorporate LLMs for AutoML or model selection would provide a useful context for the reader

Reply to NmMt-W3.3:

We have discussed the differences between our method and existing LLM-based AutoML methods in the second half of Sec. 2. We apologize for our previous disorganization that may have caused this confusion. Briefly speaking, the existing LLM-based AutoML methods focus on how to better utilize LLMs as model design optimizers (i.e., $\nabla_{\pi} \log \pi (F^i)$ in Reply to NmMt-W1.1), and neglect the organization and utilization of benchmark data. In contrast, our method focuses on how to efficiently retrieve promising models from benchmark data for unseen data (i.e., efficiently approximate $\mathcal{M}(F^i; D^u)$ in Reply to NmMt-W1.1). Thus, we are focusing on orthogonal problems that can aid the development of each other. Existing LLM-based methods can be integrated into our framework as an alternative model design optimizer to further improve the model design performance.

Besides, we kindly remind the Reviewer that in Tab. 5 at Sec. 7.2 we have already presented the experimental comparisons between our KBG and recent LLM-based AutoML methods (GPT4GNAS, GHGNAS, and DesiGNN) in proposing initial model designs. To better present the performances, we reorganize the results and present them below. Bold means best performance and italics means second best performance

NmMt-W3: The paper’s discussion on integrating LLMs into the proposed framework could be more comprehensive

Reply to NmMt-W3:

We have polished the corresponding experiment description in Sec. 7.6 and provided more experiment details in Appendix A.6.

NmMt-W3.1: The authors might expand on the specific role of LLMs in their approach

Reply to NmMt-W3.1:

When combining KBG and LLM to suggest model designs, we position LLM as a model design optimization tool that could understand and leverage the information retrieved from KBG. Following our discussion about the AutoML in Reply to NmMt-W1.1, LLM is an alternative of $\nabla_{\pi} \log \pi (F^i)$ that refine the model designs. The detailed procedure is as follows:

1. Given a KBG and unseen dataset $D^u$, find similar datasets $\mathbb{D}^c$ to $D^u$ with dataset similarity score $\mathcal{S}(\cdot, \cdot)$.
2. Retrieve promising candidate models $\mathbb{F}^c$ with the model relevance score $\mathcal{R}(\cdot)$ based on $\mathbb{D}^c$.
3. Provide information of $D^u$, $\mathbb{D}^c$, $\mathbb{F}^c$ and historical results and instruct the LLM to infer (design a new model based on $\mathbb{F}^c$) or select (pick the most suitable model from $\mathbb{F}^c$) a model $F'$ for $D^u$.
4. Evaluate and record the performance of $F'$ on $D^u$.
5. Retrieve another set of $\mathbb{F}^c$ from KBG based on: 1) relevance $\mathcal{R}(\cdot)$ to $D^u$ and 2) similar model architectures to $F'$.
6. Repeat steps 3-5 until the computation budget is exhausted or the performance of $F'$ is satisfactory.

NmMt-W3.2: Examples illustrating the LLMs' role in the process would be helpful, as well as discussing related challenges and solutions.

Reply to NmMt-W3.2:

The example process of integrating KBG with LLM is given in Reply to NmMt-W3.1.

During integration, one challenge is that LLMs have difficulty understanding the multiple information from models and datasets due to the hallucination in processing numerical values. This is reflected by the inferior performance of LLM inferring models from $\mathbb{F}^c$ in Tab. 5. To address this, we require LLM to directly select models from the candidate set $\mathbb{F}^c$ instead of inferring new models. Such change eases the burden of LLM in understanding the model information, reduces the risk of hallucination, and brings a better performance.

However, Finding Global Optimal Model is still a challenge in suggesting models with LLM + KBG. On the one hand, always selecting models from $\mathbb{F}^c$ reduces hallucination, but can easily stuck at a local optima. Inference, on the other hand, brings more diversity, but may also introduce more noise to the model design. Thus, it is important to balance the infer and selection strategies in the LLM + KBG framework so as to effectively optimize the model design towards a global optimum.

We will further investigate and address this challenge in future work.

We thank you again for the insightful comments from all the reviewers. Currently, we still have not heard feedback from Reviewer zioe. In case our replies are missed by Reviewer zioe, we would like to briefly summarize our responses:

First, we want to clarify a misunderstanding from Reviewer zioe about our primary contribution. Our work aims to identify and validate a novel problem in AutoML, specifically the challenge of efficient model selection for unseen datasets using existing knowledge. While we propose initial methods to address this problem, our main contribution lies in bringing attention to the problem itself, formating a clear definition, proposing a comprehensive evaluation method and demonstrating its importance through empirical validation.

Then, regarding the two major concerns from Reviewer zioe, our responses are summarized below:

1. Dataset Similarity and Model Relevance Scores

We appreciate the suggestions on enhancing the dataset similarity and model relevance scores. Our current simple designs for $\mathcal{S}(\cdot)$ and $\mathcal{R}(\cdot)$ are intentional, serving as baseline methods to initiate discussion on this new problem. We aimed for efficiency and practicality, enabling quick model recommendations without extensive computations. Incorporating complex metrics, such as edge-level information or architectural details, could improve accuracy but may not align with our goal of proposing an accessible starting point for future research. In fact, we point out that how to design an effective similarity metric is an open question in Sec. 5.

To address your concerns and facilitate further studies, we have:

• Clarified the purpose of our simple scoring methods in Reply to zioe-W1 and Sec. 5, emphasizing their role as foundational tools rather than final solutions.
• Added a more complex dataset similarity metric using the Personalized PageRank algorithm in Reply to zioe-Q2, demonstrating potential improvements and encouraging exploration of more sophisticated methods.
• Clarified the difficulty in using architectural information in the initial model retrieval due to the lack of performance relationships, as explained in Reply to zioe-W2. And described how architectural information is incorporated during model refinement in Reply to zioe-Q2 and Appendix A.6.

2. Details on LLM-Assisted Model Selection

Regarding the LLM-assisted method, we acknowledge that the initial description lacked detail. In response, we have provided a detailed description on the LLM integration in Reply to zioe-W3, Reply to zioe-Q1, and Appendix A.6, outlining the prompts used, how outputs are processed, and any constraints applied. We intended to illustrate how Large Language Models (LLMs) can be integrated into the model selection process as a complementary tool rather than the primary focus.

We are grateful for the insights from Reviewer zioe, which have helped us improve the clarity and depth of our manuscript. Please feel free to reach out with any additional questions or suggestions.