AutoM3L: Automated Multimodal Machine Learning with Large Language Model

Your comprehensive review and astute suggestions are truly appreciated and we're thankful for the guidance you've provided to improve our research.

W1:

Thank you for your feedback. We'd like to clarify that our paper aligns with the scope of papers accepted by ICLR, which includes articles similar to technical reports [1]. In our submission, we not only introduce an LLM-driven multi-modal AutoML framework but also propose a systematic set of evaluation metrics. This encompasses both objective quantitative analysis and subjective user experiment evaluation metrics. We approach the design of our AutoML system from the perspective of Human-Computer Interaction, conducting human-machine interaction experiments to showcase the framework's usability and intelligence. These aspects go beyond the typical content found in a standard technical report.

[1] Rogozhnikov A. Einops: Clear and reliable tensor manipulations with Einstein-like notation [C]//International Conference on Learning Representations. 2021.

W2:

Due to the current scarcity of open-source multi-modal datasets encompassing images, text, and tabular data, we proactively gathered four datasets covering classification, regression, and retrieval tasks. While the number of open multi-modal datasets is limited, these datasets accurately mirror real-world business scenarios where various modalities coexist. Examples include product images paired with textual descriptions and categorical/numerical information, as well as financial datasets storing user photos, names, ages, addresses, transaction details, credit ratings, and more.

Our decision to exclusively compare with AutoGluon in the paper was intentional, aiming to highlight the efficacy and intelligence of our approach in handling multi-modal data. In multi-modal scenarios, the ability to distinguish between different modal inputs and adapt suitable data preprocessing methods and models becomes indispensable. The overall framework design becomes more intricate and challenging, underscoring the importance of introducing AutoML in such contexts. Single-modal experiments were omitted from the paper as AutoGluon has demonstrated optimal accuracy in single-modal scenarios [2]. We have also benchmarked AutoM3L against H2O AutoML on the single-tubular modality datasets from OpenML(https://www.openml.org/), which cover regression and binary/multiclass classification tasks. The results demonstrate AutoM3L's strong performance even in single-modality settings.

These additional experiments reinforce that in addition to handling multimodal data, AutoM3L also serves as an effective general-purpose AutoML solution.

It's crucial to emphasize that our primary focus is not on accuracy improvement alone. The key contribution of our method lies in elevating the automation level and user-friendliness of the pipeline. This aspect transcends dataset accuracy measurement. Therefore, we designed user experiments, evaluated from a human perspective through human-machine interaction. The experiments substantiated that our framework exhibits higher usability.

[2] Gijsbers P, Bueno M L P, Coors S, et al. Amlb: an automl benchmark[J]. arXiv preprint arXiv:2207.12560, 2022.

Thank you for your insightful and detailed review. Your thoughtful suggestions are instrumental in refining our research, and we're grateful for the expertise you bring to the reivew process.

W1:

NAS is a pivotal component of AutoML that has garnered significant attention in recent years. In AutoM3L, we match specific modal inputs with corresponding pretrained models and merge multi-modal features. We consider NAS as an extensible aspect of our framework for the following reasons:

A. Scalability through model_zoo:

Considering the extensibility of the model_zoo we introduce, we can describe different neural network structures in the NAS search space as corresponding model_cards. For instance, different-scale convolutional layers, dilated convolutions, etc. In our framework, user requirements, including training resources, deployment devices, etc., can interact with LLM in natural language. This prompts LLM to search and assemble structures in the search space based on prompt engineering.

B. HPO-LLM in AutoM3L:

In AutoM3L, we have designed HPO-LLM to infer the hyperparameter search space based on the training configuration file. The configuration file inputted into LLM can also include model configuration containing parameters such as num_trans_blocks, num_atte_head, ffn_dropout, etc. Leveraging the characteristics of HPO-LLM, we can infer the search space for model parameters and search them alongside other training hyperparameters (e.g., learning rate).

W2:

We supplemented the comparison experiment with autokeras framework in multimodal scenarios. Autokeras framework is an earlier work, but the multi-modal training is a recently launched feature. Considering that Autokeras requires manual predefinition of data types, which is different from our task of parsing structured table data, we did not compare with it before.

We only conducted classification and regression experiments as AutoKeras does not support retrieval tasks. The primary focus of AutoKeras is on searching for network structures. In our analysis, we attribute the lower accuracy of AutoKeras to the network structures obtained within its limited search space, which lacks pretraining on large-scale datasets. In contrast, our approach leverages the strength of pretrained models by linking with open-source communities such as HuggingFace and Timm. This integration allows us to access more powerful pretrained models, contributing to the improved performance demonstrated in our work.

Besides, we have also benchmarked AutoM3L against H2O AutoML on the single-tubular modality datasets from OpenML(https://www.openml.org/), which cover regression and binary/multiclass classification tasks. The results demonstrate AutoM3L's strong performance even in single-modality settings.

We agree with the reviewer that accuracy is not the only crucial metric. AutoM3L's critical advantage lies in enabling the entire ML pipeline to be automated through natural language interaction. This substantially reduces the manual effort and learning curve for users, as validated quantitatively in our user studies. Such intuitive human-AI interaction and usability are lacking in other AutoML approaches.

I appreciate the authors' efforts in the rebuttal, which helped to improve the paper's quality. However, based on the current form, I decide to keep my score and encourage the authors to further improve the work, e.g., by realizing the incorporation of NAS more thoroughly (e.g., differential NAS such as DARTS is more complicated and the proposed approaches in the rebuttal seem not to be able to incorporate) and packing the proposed library in a more mature format.

We are grateful for your meticulous evaluation and constructive suggestions. Your professional perspectives have played a pivotal role in improving the depth and precision of our research.

W1:

We appreciate the reviewer raising this valuable point about the scope of our evaluation datasets and validation methodology. In this work, our key focus was introducing and demonstrating the capabilities of our novel AutoM3L framework on representative multimodal datasets, rather than an exhaustive benchmarking across a wide array of datasets. Nonetheless, we recognize the merit of more comprehensive evaluations.

Regarding the datasets, we strategically selected four multimodal datasets that cover common tasks like classification, regression, and retrieval. These datasets exhibit diversity in terms of size, modality types, labels, and complexity. While not exhaustive, we believe they enable reasonable assessments of AutoM3L's automation capabilities.

We agree that using a single train-test split limits robustness. Ideally, techniques like k-fold cross-validation would enable more rigorous performance validation. Hence, we have now incorporated 10-fold CV experiments on all datasets:

To compensate for dataset size limitations, we employed stratified sampling to create train-validation-test splits that preserve label distributions. We additionally introduced perturbations like random feature masking and noisy data injection to further stress test AutoM3L.

W3:

Your suggestion makes a lot of sense. In our framework, randomness primarily stems from LLM inference and hyperparameter search.

In the hyperparameter search experiment, we combined the search space recommended by HPO-LLM with the ray.tune tool, which supports repeated experiments to explore the most suitable hyperparameters. We conducted 256 repetitions in our hyperparameter search experiment.

In addition, the 10 fold cross-validation table in W1 indicates the relative stability of our framework. Regarding the stability assessment of MS-LLM, we conducted the following experiment:

Exp Setting: We utilized GPT3.5 to generate 10 sentences expressing the same user command in different ways. The objective was to examine whether MS-LLM could successfully retrieve the appropriate model.

Results: All 10 sentences were successfully indexed to Model:{"google/flan-t5-small",'mobilenetv3_large_100","categorical_mlp","numerical_mlp"} by MS-LLM.

W2:

Thank you for your suggestion. We have supplemented the 10-fold cross-validation results on 4 multi-modal datasets. Please refer to the response of W1 for details.

In addition, we supplemented the significance test of the method performance on 4 multi-modal datasets, as follows:

We formulate null hypotheses:

• H1: On the PAP dataset, the performance of AutoM3L does not significantly exceed that of AutoGluon.
• H2: On the PPC dataset, the performance of AutoM3L does not significantly exceed that of AutoGluon.
• H3: On the GMRR dataset, the performance of AutoM3L does not significantly exceed that of AutoGluon.
• H4: On the SPMG dataset, the performance of AutoM3L does not significantly exceed that of AutoGluon.

Normality Testing: To ensure the validity of our subsequent statistical analyses, we conducted a normality test on AutoM3L's performance on 4 datasets using Q-Q plots, as depicted in (https://i.ibb.co/2SxcdDC/Whzr7-OLUY5.jpg).

We performed paired two-sample t-tests (essentially one-sample, one-sided t-tests on differences) for the aforementioned variables across two experimental conditions: AutoGluon and AutoM3L. These tests were conducted at a significance level of 5%. The hypothesis testing results from paired two-sample one-sided t-tests as below:

Experiment results show that AutoM3L does not significantly surpass AutoGluon in all datasets. It should be noted that accuracy is not the only crucial metric. AutoM3L's critical advantage lies in enabling the entire ML pipeline to be automated through natural language interaction. This substantially reduces the manual effort and learning curve for users, as validated quantitatively in our user studies. Such intuitive human-AI interaction and usability are lacking in other AutoML approaches.

Thank you for your thoughtful review and valuable suggestions. We appreciate your professional insights, which are crucial for enhancing the quality of our research.

W1:

We understand the concern that our work may seem like a straightforward application of LLMs to AutoML. However, we believe there are several key technical contributions and innovations in AutoM3L that go beyond simply combining existing methods:

1. Adaptability and Scalability: Our framework is designed to be highly adaptable to new models and techniques. Adding a new model to AutoM3L simply involves appending a new model card to the model zoo. In contrast, expanding the capabilities of rule-based AutoML systems like AutoGluon often requires extensive code changes. This adaptability comes from AutoM3L's core use of LLMs to dynamically assemble pipelines based on user needs.

2. Automated Feature Engineering: To our knowledge, AutoM3L is the first AutoML system to leverage LLMs to automate feature engineering for multimodal data. This includes filtering irrelevant features and imputing missing values, which are challenging for rule-based methods. Our ablation studies demonstrate the benefits of LLM-powered feature engineering.

3. Interactive Customization: A key advantage of LLMs is enabling intuitive human-AI interaction through natural language. AutoM3L allows users to customize pipelines through simple directives at each stage, rather than grappling with configuration files. This interactivity and ease of use is a substantive differentiation from existing AutoML systems.

4. Generalizability: We demonstrate AutoM3L's capabilities across a diverse range of tasks (classification, regression, retrieval) and modalities (text, image, tabular). The strong performance across these datasets underscores AutoM3L's general applicability as an AutoML solution.

The technical challenge that AutoM3L solves lies in making LLM more effectively drive each component of AutoML. For instance, in MI-LLM and AFE-LLM, we leverage In-context learning to enable LLM to learn how to perform specific tasks with minimal samples. In the model retrieval stage, we generate model descriptions (model_cards) using LLM-based document reading tools like ChatPaper, automating the continuous addition of model_cards to the model library (model_zoo) for the latest models, thereby improving the framework's scalability and practicality. In HPO-LLM, to enhance LLM's inference of the parameters to be optimized, we use LLM to generate textual descriptions of configuration file-related parameters in the training context.

Another difficulty in applying LLM to AutoML arises from the challenge of assessing the framework's usability and intelligence. This is a common issue faced by many LLs-Agent-related works. To address this, we designed user experiments from a human-computer interaction perspective. The experimental results demonstrate that our framework, compared to existing ones, exhibits higher usability, significantly reducing user learning costs. This user-friendly aspect is particularly beneficial for users without a background in model training and underscores the significance of AutoML.

Additionally, this research looks into the novelty beyond the applications of LLM but also from the perspective of human-computer interaction. Specifically, our method simplifies user involvement, and eliminates the need for intensive manual feature engineering and hyperparameter optimization. Simultaneously, users can interact with the framework in natural language to customize pipelines, enhancing both the usability and intelligence of the framework.