LLM Performance Predictors are good initializers for Architecture Search

Thanks for the careful review and for the helpful comments. We respond to quoted comments below:

• Weakness 1: The hybrid search algorithm is based on a simple yet novel idea: combine the strengths of LLM and supernet based performance predictors. The simplicity in the algorithm design allows hybrid search algorithms to be dropped into various state-of-the-art NAS frameworks for NLP to accelerate the search time.
• Weakness 2: Our evaluation setup of using machine translation benchmarks is borrowed from the NAS for NLP literature [Wang ACL'20, Jawahar ACL'23], which also uses only machine translation tasks. Our contribution is still non-trivial and significant, which as you’d pointed out is “innovative”, “valuable approach to reduce the cost of using LLMs for performance prediction”, “can be a substantial advantage for practitioners looking to optimize their deep learning models.” Exploring other NLP tasks, computer vision, speech recognition tasks is orthogonal to our main research questions, but will be an important future direction.
• "Is the prediction result provided by LLM repeatable?" We compute the predictions for 8500 randomly sampled architectures using LLM-PP GPT-4 three times and compute the standard deviation of the three predictions for each architecture. The mean of the standard deviation for 8500 architectures is very low: 0.21, 0.27, 0.27 BLEU for WMT’14 En-De, WMT’14 En-Fr, and WMT’19 En-De respectively. Thus, LLM-PP provides fairly robust performance predictions. For all our search experiments, we use a single estimate from LLM-PP.
• "...how do you ensure that LLM understands the DNN framework rather than reading memory..." Below, we provide evidences on why the high performance of LLM-PP is highly unlikely due to “memorization”:
  • Since the test set is created by randomly sampling from the search space, the probability of a test architecture to be part of the LLM training data is extremely low (e.g., 1 / 6000000 as the WMT’14 En-De search space has 6 million architectures). Note that we need a large number of resources (e.g., training a single architecture on 4 NVIDIA V100 GPUs takes 2 days) to train even a few 100s of architectures from scratch to convergence to get their true performance. Hence, it’s infeasible to have non-negligible coverage of architectures (along with its corresponding true performance) in the LLM training data.
  • If the performance of LLM-PP is largely due to “memorization”, LLM-PP should achieve competitive performance even when the number of demonstrations in the prompt is minimal. Consider the table below.

From this table, it is clear that LLM-PP did not achieve competitive performance compared to the SOTA supernet baseline when the number of demonstrations in the prompt is minimal (e.g., less than or equal to 3 demonstrations). This result highlights that the performance of LLM-PP might not be largely due to “memorization”, and is likely due to other LLM capabilities such as “understanding” and “reasoning”. As the training data of LLMs like ChatGPT and GPT-4 are not public, it is impossible to conclusively say that the training data do not contain any of our test architectures. Despite this, we have shown strong evidence on why the performance of LLM-PP is highly unlikely to largely come from “memorization”.

• "why not use a PP that performs better than LLM-PP for distillation?..." LLM-PP achieves the best performance (SOTA MAE, marginally worse Kendall-Tau but doesn’t have a big negative impact for the NAS task, as detailed in Section 5.2, second paragraph) compared to the supernet based performance predictors (see Table 1). Hence, LLM-PP-Distill is built by distilling the performance predictions from LLM-PP. In A.6.4, we show that LLM-PP-Distill that distills from HAT (marginally better Kendall-Tau than LLM-PP) leads to search hours reduction of at least 40% compared to HAT, but at the expense of big increase in latency and GFLOPs (9.7%, 32% respectively). This motivates why we distilled from LLM-PP.

We hope we have answered your questions to your satisfaction and hope you are willing to raise the score. Please let us know if there are any further questions that we can address during the interactive rebuttal period.

References:

• [Wang ACL'20] HAT: Hardware-Aware Transformers for Efficient Natural Language Processing. Wang et al., ACL 2020.
• [Jawahar ACL'23] AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation. Jawahar et al., ACL 2023.

Thanks for the careful review and for the helpful comments. We respond to quoted comments below:

• "The effectiveness of the proposed method largely depends on how much information GPT-4 has "memorized...." Thank you for the questions. While WMT’14 and WMT’19 are popular MT benchmarks, it’s highly unlikely that the high performance of LLM-PP can be largely attributed to “memorization” of the underlying LLM for the following reasons:
  • Since the test set is created by randomly sampling from the search space, the probability of a test architecture to be part of the LLM training data is extremely low (e.g., 1 / 6000000 as the WMT’14 En-De search space has 6 million architectures). Note that we need a large number of resources (e.g., training a single architecture on 4 NVIDIA V100 GPUs takes 2 days) to train even a few 100s of architectures from scratch to convergence to get their true performance. Hence, it’s infeasible to have non-negligible coverage of architectures (along with its corresponding true performance) in the LLM training data.
  • If the performance of LLM-PP is largely due to “memorization”, LLM-PP should achieve competitive performance even when the number of demonstrations in the prompt is minimal. Consider the table below.

From this table, it is clear that LLM-PP did not achieve competitive performance compared to the SOTA supernet baseline when the number of demonstrations in the prompt is minimal (e.g., less than or equal to 3 demonstrations). This result highlights that the performance of LLM-PP might not be largely due to “memorization”, and is likely due to other LLM capabilities such as “understanding” and “reasoning”. Due to lack of time in the rebuttal phase, it might not be possible to experiment with other suggested datasets (we will send an update response if we could experiment with other datasets within the timeline).

• "It is very possible that the author is testing the model that it had been trained on the test dataset" As the training data of LLMs like ChatGPT and GPT-4 are not public, it is impossible to conclusively say that the training data do not contain any of our test architectures. Despite this, we have shown strong evidence on why the performance of LLM-PP is highly unlikely to largely come from “memorization”.

We hope we have answered your questions to your satisfaction and hope you are willing to raise the score. Please let us know if there are any further questions that we can address during the interactive rebuttal period.

I thank the authors for the response.

I believe the authors should agree with the point that 'GPT-4 only works on the dataset (architecture information and its results) that it has seen before.' This is the fundamental reason why it also works on the common machine translation data (e.g., WMT14). This is the 'memorization' I am discussing. It predicts the model's performance based on what is most likely, with the restriction only on commonly used data (like WMT'14). That is also the reason I strongly disagree with the claim in the paper that 'LLMs have a ‘general understanding’ of DNN architectures.'

The biggest flaw in this paper is that the authors fails to show the generalization of this method on uncommon data/architectures/languages (see my points in the initial review above). This method, under my understanding, is difficult to generalize and only applies to very commonly used data. Without showing the limitations on the uncommon data (or, say, generalization), this study should be a clear reject.

Thanks for the careful review and for the helpful comments. We respond to quoted comments below:

• "I would especially point out that recently, the extremely simple baseline, "number of parameters" ... this would be great to add as a baseline e.g. in Table 1." Thanks for these suggestions. The main reason to compare against supernetwork-based performance predictors only is that they are the state-of-the-art performance predictors in NAS for NLP literature, including machine translation (e.g., HAT [Wang ACL'20], AutoMoE [Jawahar ACL'23], Mixture-of-Supernets [Jawahar arXiv'23]) and BERT language modeling (e.g., NAS-BERT [Xu KDD'21], AutoDistil [Xu NeurIPS'22], Mixture-of-Supernets [Jawahar arXiv'23]). We will modify the SOTA statement in the revision as follows: “In NAS for NLP literature, the SOTA approach for building performance predictors (f_T ) is to train a weight-sharing supernet model on the task T.” Based on your suggestion, we experiment with the simple baseline “number of parameters”. The kendall-tau for this simple baseline is 0.42, 0.51, 0.54 for WMT’14 En-De, WMT’14 En-Fr, and WMT’19 En-De respectively, which is clearly worse than LLM-PP GPT-4 (0.65, 0.75, 0.65, as shown in Table 1 of the paper). Due to lack of time in the rebuttal phase, it might not be possible to experiment with other suggested baselines (we will send an update response if we could run other baselines within the timeline), but our work has compared to established SOTA performance predictors in NAS for NLP literature.
• ".. the authors only test their performance predictor on a single NAS framework; the one from HAT. . There are other NAS frameworks too, for example, Bayesian optimization or BOHB." We show the efficacy of HS-NAS in two SOTA NAS for MT frameworks: HAT and Mixture-of-Supernet (MoS) NAS framework. Due to lack of space, HS-NAS HAT results are in the main paper, while HS-NAS MoS results can be seen in Section A.6.3. Generalizing HS-NAS to other search frameworks such as bayesian optimization is orthogonal to our main research questions, but will be an important future direction.
• "Finally, the paper could also be more impactful if it tested search spaces / tasks beyond machine translation. ..." Our evaluation setup of using machine translation benchmarks is borrowed from the NAS for NLP literature [Wang ACL'20, Jawahar ACL'23], which also uses only machine translation tasks. Our contribution is still non-trivial and significant, which as you’d pointed out is "interesting", "fairly novel", "fits the intuition that LLMs are strongest at performance prediction early on, but are no match for computational-based methods after a handful of iterations". Due to lack of time, other NLP tasks, computer vision, speech recognition cannot be included, but will be interesting future directions worth exploring.
• "It is surprising that LLM-Distill-PP performs better than LLM-PP. ..." We believe there are two major reasons for this.
  • LLM-Distill-PP is a linear regression model with only 486K parameters, while LLM-PP GPT-4 is a LLM with several billions of parameters. Due to smaller size, it is less likely that LLM-Distill-PP overfits compared to LLM-PP, thereby leading to better performance.
  • LLM-Distill-PP is a specialist model as the model parameters are trained for performance prediction task with a few thousands of examples. On the other hand, LLM-PP is a generalist model which performs in-context learning with PP prompt and 10 demonstrations.
• "The authors mention that they share their code, but I couldn't find it. Can the authors share their code, e.g. with https://anonymous.4open.science/?" Thanks for the suggestion. Please find the code in this link: https://anonymous.4open.science/r/llmpp-code-FF36
• "If the authors can address some of the points in the weakness section, I would be open to raising my score. Specifically, comparing to baselines would be important." Thanks. We hope we have answered your questions to your satisfaction and hope you are willing to raise the score.

Please let us know if there are any further questions that we can address during the interactive rebuttal period.

References:

• [Wang ACL'20] HAT: Hardware-Aware Transformers for Efficient Natural Language Processing. Wang et al., ACL 2020.
• [Jawahar ACL'23] AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation. Jawahar et al., ACL 2023.
• [Jawahar arXiv'23] Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts. Jawahar et al., arXiv:2306.04845, 2023.
• [Xu KDD'21] NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search. Xu et al., KDD 2021.
• [Xu NeurIPS'22] AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models. Xu et al., NeurIPS 2022.

Thanks for the careful review and for the helpful comments. We will improve the presentation aspects of the paper as per your suggestions in the revision. We respond to quoted comments below:

• "How does the LLM-Distill-PP model's efficiency and accuracy compare to other existing models?" The following table shows the efficiency (time taken to predict performance for 10 architectures) and accuracy (MAE, Kendall-Tau, as shown in Table 1) for supernet-based PP (HAT, Layer-wise MoS, Neuron-wise MoS), LLM-PP (GPT-4), and LLM-Distill-PP (GPT-4) for the WMT'14 En-De task. | Perf. Predictor (PP)| MAE | Kendall-Tau | Prediction Time (seconds) | | -- | -- | -- | -- | | HAT | 1.14 | 0.71 | 10.5 | | Layer-wise MoS | 1.05 | 0.81 | 13.9 | | Neuron-wise MoS | 0.97 | 0.56 | 13.3 | | LLM-PP GPT-4 | 0.28 | 0.65 | 11.9 | | LLM-Distill-PP GPT-4 | 0.22 | 0.64 | 0.01|

It is clear that LLM-Distill-PP provides the best efficiency-accuracy tradeoff with on par (if not better) accuracy as LLM-PP but significantly faster prediction time (0.01s vs. 11.9s).

• "Could the authors elaborate on the rationale and design process behind the PP prompts used for LLMs?" The goal of creating PP prompts is to precisely communicate the performance prediction task to the LLM. We selected machine translation benchmarks due to their extensive study in the NAS for NLP literature. The design process began by examining crucial elements of the machine translation task, commonly used model architectures, and relevant efficiency metrics. Initially, we presented only demonstrations, borrowing hyperparameter wording from HAT’s configuration file. Subsequently, we added the role and definition of each hyperparameter, using wording from HAT’s helper description. Moving forward, our aim was to craft instructions enabling the LLM to grasp essential tasks, architecture, and metric details. Most instructions are prefixed with 'You should' to encourage strict adherence. Five instructions were incorporated. The first specifies the dataset, translation direction, and quality metric. The second provides examples randomly sampled from the training set, presented with generic prefixes ('Input:' for source sentence, 'Output:' for target sentence). The third outlines the architecture, citing the 'Attention Is All You Need' paper, assuming the LLM is familiar with this popular work. Standard settings and optimization algorithms are noted for training the architectures. The fourth identifies the efficiency metric in the demonstrations. The final instruction aims to summarize the relationships the LLM should learn to solve the task effectively.
• "Is the proposed Hybrid-Search algorithm scalable for larger datasets or more complex architectures, and if so, how?" Thanks for the question. By design, the search algorithms (hybrid-search, baseline search) are scalable to: (i) larger datasets as the search time is agnostic to the size of the dataset and depends only on the hyperparameter settings (e.g., num-iterations, num-mutations, num-crossover, num-parents) of the search algorithm, and (ii) complex architectures as long as the PP prompts are adapted (e.g., third instruction in the PP prompt should contain the details of the complex architecture, and demonstrations in PP prompt contain the hyperparameter values of that architecture).

We hope we have answered your questions to your satisfaction and hope you are willing to raise the score. Please let us know if there are any further questions that we can address during the interactive rebuttal period.