LLM-Assisted Fast and Customized Model Generation: A Preliminary Exploration

W1: The Model Generator section presents pre-defined choices for tasks, scales, and architectures, especially with limited architecture options per modality (as shown in Appendix D, there are fewer than three models for each task). This raises concerns about FLAME’s effectiveness and generalization across more complex tasks, particularly considering that FLAME’s goal is to generate customized small models.

W2: The experiments lack important baseline comparisons. Although the authors claim in Section 4.1.1 that FLAME is the first framework to translate user data or descriptions into model parameters, it would be valuable to include comparisons with state-of-the-art LLM-based frameworks that also generate AI/ML solutions from user requirements, such as AutoM3L [1] and AutoMMLab [2]. Additionally, comparisons with traditional AutoML systems are missing, which would provide a more thorough evaluation of FLAME’s performance.

W3: The experiments only evaluate GPT4-turbo’s performance on the benchmarks, which leaves open the question of whether the proposed method relies heavily on GPT4-turbo’s capabilities. This raises concerns about the technical soundness and generalizability of the approach when applied to other LLMs.

W4: One of the significant concerns with integrating LLMs into AI solution generation is the risk of data leakage. Since LLMs are pre-trained on large amounts of publicly available data, including many common ML datasets, this overlap could lead to biased evaluations [3]. The paper does not address how this potential issue is mitigated.

Minor Comments:

The motivation for using hypernetworks to generate model parameters in this work needs further explanation.

[1] Luo D, Feng C, Nong Y, et al. AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models[J]. arXiv preprint arXiv:2408.00665, 2024.

[2] Yang Z, Zeng W, Jin S, et al. AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks[J]. arXiv preprint arXiv:2402.15351, 2024.

[3] Jeong D P, Lipton Z C, Ravikumar P. Llm-select: Feature selection with large language models[J]. arXiv preprint arXiv:2407.02694, 2024.

1. With the development of LLM, the problems mentioned in the paper can be solved by LLMs themselves in a more efficient way

This paper mainly focuses on three types of data: text, table, and image. Problems with these three types of data can already be solved by multimodal LLM, and there are already some powerful open source multimodal LLMs. These LLMs can be deployed locally to avoid privacy issues and can be fine-tuned to support customization. The method proposed in the paper requires (1) using LLM to generate metadata, then (2) using metadata to generate models, and finally (3) using the generated model to solve the problem. If a multimodel LLM is used, you only need to adjust the prompt in the first step, and the next two steps can be omitted. It would be better to add a discussion about the trade-offs between using LLM directly and using FLAME.

2. The proposed method has limited scope of application

The paper divides the tasks into three categories, and only considers classification and regression tasks. Other types (such as audio) are not supported yet. In addition, this classification by data type will make some tasks involving mixed data of multiple modalities impossible to complete, which will become an obvious limitation of the architecture.

(W1) Although the authors do not explicitly state this in the paper (perhaps intentionally?), this work firmly overlaps with the established areas of meta-learning and transfer learning. As such, the key challenge here is generalizability to unseen datasets. I was not able to find details about which datasets were used for training the FLAME framework and which ones were used for inference. Without this, it becomes hard to judge how well the proposed method can generalize to datasets it never encountered. The authors were pretty open about this being a shortcoming of the current approach (e.g. in Section 3.2.1 and Section 5) which is commendable. However, I would argue that generalizability is actually both the hardest and the most interesting part. Furthermore, without it, the entire framework can be seen as an interesting idea without much real-world use.

(W2) Leaning on the above point, I think this paper should be placed in the context of other work related to meta-learning and transfer learning, both in terms of methodology (see W1) and in terms of updating the introduction and related work sections.

(W3) Given that the authors apply HyperNetworks to obtain model weights, it's easy to wonder how much the proposed method relies on the parameter generator simply memorizing the weights for all of the dataset/task pairs it has been trained on, as opposed to being able to acquire meta-knowledge that is transferable between tasks. In other words, do we even need hypernetworks or can we just get away with a collection of pre-trained model weights? There are certain claims made in e.g. Section 4.2 about that topic, but this is more of a post-hoc interpretation rather than an empirically validated claim. More convincing empirical evidence would be to include results where we have a collection of pre-trained models (the same collection as the one FLAME ends up learning to parameterize during pre-training) and test if having an LLM pick the most appropriate set of weights from the stash could improve performance.

(W4) (minor issue) There are a small handful of writing issues and missing clarifications that could easily be addressed:

• (Abstract) "high-performance models", and in general usage of the word "model" is a little ambiguous as it is unclear if the authors are talking about LLMs or deep learning models, or even classical ML models. This becomes clear later on but could have been clarified earlier.
• (Section 1, page 2) "In real-world scenarios, data often is limited or lacks insufficient supervisions" -- Firstly, "supervisions" -> "supervision". Secondly, the authors likely want to say "lacks sufficient supervision". Thirdly, it is unclear what the authors even mean when they say the data lacks sufficient supervision. This should probably be reworded.
• (Section 4.1.2) "while Relative Efficiency scales this runtime against the worst-performing method" -- worst-performing in terms of runtime or model quality? I think the authors mean runtime but this could be clarified.
• (Section 5) "we aim to paves" -> "we aim to pave"

1. Lack of Practicality: The paper presents an ambitious concept of "generating AI models for all users." However, its applicability is limited to only three tasks and models: NLP (Distil-BERT), CV (ResNet-50), and Tabular (MLP). This limitation diminishes its practical significance in real-world applications.
2. Lack of Innovation: The paper primarily focuses on using hypernetworks with traditional models (Distil-BERT 2019, ResNet 2016, and earlier MLPs), where the role of large language models (LLMs) is merely to optimize descriptions through prompting. This limited integration offers little novelty or practical enhancement, particularly in light of recent advancements in the integration of LLMs and hypernetworks, e.g., using hypernetworks with LLMs for domain adaptation [1], and hypernetwork-based multi-objective fine-tuning for LLM alignment [2].
3. Manual Intervention & Generalizability: The use of FLAME highly relies on manual selection of datasets, models, and parameter adjustments. For instance, when addressing new tasks, it requires manually selecting the most similar task head (line 248). Additionally, the weight for each task must be manually set (line 296). Consequently, I am concerned that the strict constraints on corresponding tasks and models may hinder the generalization capability of FLAME.
4. Background and Premise Concerns: The paper's premise that users lack sufficient data, time, and resources (lines 16, 44), which contrasts with its use of small models (max 66M, line 366), questioning the validity of its foundational assumptions about resource constraints.
5. Efficiency Trade-offs: The paper highlights a significant speed increase (270x, line 99), which is an important metric from user's perspective. However, comparing only the inference speed without considering the pretraining costs (Appendix A) could result in a less comprehensive analysis.

[1] Hypernetwork-Assisted Parameter-Efficient Fine-Tuning with Meta-Knowledge Distillation for Domain Knowledge Disentanglement

[2] HyperDPO: Hypernetwork-based Multi-Objective Fine-Tuning Framework

1. The objective of the paper is to make a layman capable of generating a ready-to-use model to analyse their own data. With a layman assumption, the simple data and task description input may be ambiguous and not have an one-to-one mapping to the target model. This may happen when the requirement is underspecified as the user lacks of knowledge about ML models and the terminology to describe their data and tasks. The proposed method does not seem to discuss the complexity in requirement generation.

2. The paper argues its difference to HINT is that the proposed method applies broadly to different networks other than those used in NLP. However, The HINT approach does not have to be limited to NLP models. It aims to generate model parameters to attach to a pre-trained model. In this sense, the novelty of the proposed method seems is more on the requirement generation part, which is a bit ad-hoc because of weakness 1. In addition, the prompts for model architecture generation ask LLMs to choose from a limited and pre-defined model architectures before applying a similar approach for parameter generation. This is a trivial extension to HINT.

3. In order to generate parameters for non-transformer models, the paper disables batchnorm and other normalisation layers, which seems to remove the functionalities of functional models simply to fit for the parameter generation purpose. It might not be a good practice for designing a software system.