ProtoLLM: Training and Example-free LLMs for Few-shot Tabular Learning

Results on regression tasks

We agree with you that the tabular regression task is a significant application area for tabular learning. One of the possible directions to apply our ProtoLLM to regression tasks is viewing our generated features as a data augmentation tool. For example, we can concatenate all generated values feature-by-feature to form a virtual labeled sample. Together with few-shot labeled samples, we can improve the regression performance of baselines. We here report the root mean squared error (RMSE) score on abalone and California datasets in Table 2. We find that our ProtoLLM improves the baseline by a large margin in most cases. This demonstrates the potential of ProtoLLM for regression tasks. Note that it is a simple attempt to apply our ProtoLLM to regression tasks, we leave it as future work for more systematic and in-depth analysis.

Table 2: RMSE results on regression task (lower is better)

The selection of $\tau$

We did not explicitly select the $\tau$ parameter. Instead, we just set it to 1 as a fixed value in our experiments.

Impact of the number of parameters of the base LLM

Thank you for your insightful question. We have investigated the impact of the number of parameters of the base LLM on overall performance, where we consider Llama-3B, Llama-8B models, GPT-3.5 and GPT-4O. The detailed results can be found in Table 3. In general, the performance of Llama-8B is better than that of Llama-3B, but the difference is not substantial. Moreover, we find that Llama-3B still achieves good results, indicating that even smaller models can perform well with ProtoLLM.

Table 3: Comparison of AUC scores with different LLMs

Training and test time

Following your advice, we report the training and testing time of different methods in Table 4. We find that our ProtoLLM requires higher training time costs. However, our ProtoLLM shows faster inference time than others (especially Tablet and Tabllm) thanks to the efficient prototype-based classification strategy we designed.

Table 4: Training and Inference Time of Different Methods

We appreciate the reviewer’s feedback and have provided the following responses to address the concerns raised about our paper. Please also check out the revised version of our manuscript (we have highlighted the revisions in blue), where notations, typos, and statements have been refined and revised for better understanding.

While the proposed approach is novel and interesting, the contribution is relatively thin for a conference like ICLR

Firstly, we appreciate the reviewer's positive comment about our novelty. We hold the belief that our ProtoLLM in this paper gives an interesting insight for introducing LLMs into the zero and few-shot tabular data analysis. We highlight our contributions as follows:

• Few-shot tabular learning has long been a popular research topic, due to its wide variety of applications. However, the features in tabular data are often heterogeneous, and their inter-relationships are not inherently sequential. This complexity poses significant challenges for tabular data learning, particularly in real-world few-shot constraints, That is to say, proposing an effective method for few-shot tabular learning is necessary but non-trivial.
• Recently, the impressive performance of LLMs has highlighted their broad knowledge and potential in instruction following and low-shot understanding. This motivated recent attempts to integrate LLMs with tabular data learning and our method belongs to this group, which is a novel and interesting solution and usually outperforms conventional tabular learning baselines.
• However, existing LLM-based tabular few-shot methods usually prompt the LLM with the few-shot samples. Our work observed an unusual phenomenon: directly using LLMs for data augmentation or rule generation by feeding a few examples significantly degrades the reasoning ability in tabular data understanding.
• To this end, we propose an example-free framework to leverage the inherent knowledge of LLMs. Our key idea is to prompt the LLM for feature value generation based solely on task and feature description. Without such example pollution, each output feature value is treated as a standard guideline, and they together act as a prototype for each class. To transfer the LLM’s knowledge to a given task, we further design an efficient fusion strategy to integrate the prototype with examples, which requires no learnable variable, i.e. neither fine-tuning LLM nor learning a classification model.
• To sum up, querying LLM with our designed prompt is zero-shot, and applying ours to tabular data classification is training-free. It is non-trivial since most of existing methods usually need to depends on the training samples or even finetune the LLM. Therefore, ours is a novel example-free and training-free framework for tabular data learning, which will provide a new way about how to integrate the LLMs into tabular data learning.

Computing cost

Thank you for your comments. We would like to clarify that our approach requires only $d \times K$ queries, where $d$ is the number of features and $K$ is the number of iterations. That is, each query prompts the LLM to generate feature values for all categories simultaneously. Moreover, when using an LLM like GPT-3.5, there are no hardware requirements since the queries are handled through the corresponding API. Furthermore, as shown in Table 1, we report the average number of tokens per query. We find that our token consumption is low and efficient even when using local LLMs. For datasets with many features, previous work, such as TabLLM and FeatLLM, requires all features to be fed into the LLM, which may exceed the model's limitations on input length. Our ProtoLLM can query the LLM feature by feature, providing an alternative for such datasets.

Table 1: Average number of tokens per query across datasets

Novel feature generation

We first want to note that employing LLMs for few-shot classification is gaining more and more research attention recently[1-2]. Pre-trained on web-scale tokens and a huge number of GPUs, LLMs show human-like abilities for understanding and reasoning. We in this paper assume that LLMs have relevant knowledge in most daily applications. Given the fact that Employing humans to generate the prototypes can be more costly in terms of time and money, which is out of the scope of this paper. Fortunately, recent research [3] finds that LLMs outperform human experts in most basic tasks.

[1] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning.

[2] TabLLM: Few-shot Classification of Tabular Data with Large Language Models.

[3] Jones, Nicola. "AI now beats humans at basic tasks—new benchmarks are needed, says major report." Nature 628.8009 (2024): 700-701.

Results of Fig.5

We kindly remind you that the standard deviations and the full results of Figure 5 were reported in Table 9 of the appendix. Additionally, following your suggestion, we have performed a Wilcoxon signed-rank test to compare the performance of our method with the other baselines. The results show statistically significant differences with $p < 0.05$, corresponding to a 95% confidence level. The details of this statistical analysis are provided in Table 5.

Table 5: Wilcoxon Signed-Rank Test

Results on noisy samples

Following your suggestion, we analyze the impact of spurious correlations. Specifically, we conduct experiments by randomly selecting columns from the Adult dataset and applying them to the Bank dataset. The Adult dataset predicts a person’s income and its features may introduce spurious correlations that could affect the task in Bank, which involves predicting the likelihood of a person subscribing to a term deposit. For this analysis, we set the number of shots to 4 and included 0-5 columns from the Adult dataset. The performance changes are shown in Table 6. As the number of spurious columns increases, the performance of ProtoLLM decreases gradually. This is because the spurious columns negatively impact the model's performance. However, as proposed in Section 4.3 of our paper, by assigning weights queried from LLMs to each feature (i.e., ProtoLLM-weighted), we can mitigate this effect and reduce the influence of spurious correlations.

Table 6: The Impact of Spurious Correlations on AUC Results

Results on Arrhythmia dataset

Following your advice, we reported results on Arrhythmia, as listed in Table. 7, We find that our models are still available on such datasets with many features and outperform baselines. Unlike previous works such as TabLLM which need to feed all features into LLMs, which is restricted by the input length, our approach queries LLMs feature-by-feature and thus can work on datasets with a large number of features.

Table 7: Results of Different Methods on Arrhythmia dataset

Note: N/A indicates that the method cannot process this dataset.

LLM-based prototypes

Thank you for your suggestion. In Figure 6 of our paper, we illustrated the relationship between performance (y-axis) and the number of queries (x-axis). $K = 0$ in the X-axis means that we only use the training samples to construct the prototype, exactly the setting described by the reviewer. We can find that, for different few-shot settings, the performance of $K = 0$ is lower than that of $K > 0$, showing the effectiveness of the prototype generated by the LLM.

The missing baselines in Fig.5

Thank you for your observation. The absence of certain baselines in Fig. 5 is primarily due to the limitations of some baseline frameworks. For example, many LLM-based frameworks, e.g., In-context, TABLET, and TabLLM, have restrictions on the maximum number of input tokens they can handle. This limitation makes it challenging or even infeasible to include these baselines in scenarios with a higher number of shots or more complex features. Additionally, TabPFN cannot run on datasets with a large number of features. Instead, ours has the ability to handle scenarios with a higher number of shots or more complex features because our designed example-free prompt is efficient. We appreciate you pointing this out and will clarify this in the revised manuscript to avoid such confusion.

More comparison on zero-shot tasks

Thank you for your valuable feedback. We appreciate your suggestion regarding zero-shot baselines, and we here incorporate additional zero-shot experiments in Table 3. Specifically, DSPy and P2T have been included as baselines. We find that our ProtoLLM outperforms DSPy and P2T in all cases, which shows the efficacy of the proposed model on zero-shot tabular data analysis.

Table 3: Zero-shot results with more baselines. GPT-3.5 is used for all the baselines.

TabLLM (GPT-3.5) VS ProtoLLM on zero-shot settings

We list the numerical results of ProtoLLM and TabLLM(GPT-3.5) in Table 4 for clear comparison. We find that 1) Although ours is inferior to TabLLM (GPT3.5) on 4 of 9 datasets, the performance of ProtoLLM is still comparable to TabLLM (GPT3.5), especially on Adult and Diabetes; 2) Our ProtoLLM beats TabLLM by a large margin on 5/9 datasets. These findings prove the effectiveness of ours in the zero-shot setting.

Table 4: Comparison of AUC scores between ProtoLLM and TabLLM (GPT-3.5), including improvement of ProtoLLM.

TabLLM GPT-3.5 results for few-shot settings

TabLLM is proposed to solve tabular few-shot classification tasks, which requires fine-tuning LLM with few-shot samples and utilizes T0 as the LLM backbone. For the zero-shot setting, as TabLLM is also feasible, we report the zero-shot performance of TabLLM with GPT-3.5 for a fair comparison. For the few-shot setting, TabLLM involves the fine-tuning of LLM but GPT-3.5 is a closed-source LLM, making it irrealizable to provide few-shot results for GPT-3.5. However, we kindly remind you that, in Fig. 5 in the manuscript, we reported the few-shot classification results of TabLLM with T0 being the backbone.

Minor issues

1) Thank you for pointing this out. We have revised Fig. 5 to make it clearer to read; 2) The links to each dataset are added in the revision; 3) Other typos and notations are corrected in the revision, please check out our revised version for details.

Continuous features

Unlike FeatLLM which directly uses the numerical values, we add the z-score normalization for the generated continuous features to complete the prototypes.

How many tokens are consumed

Our token consumption is demonstrated in Table 5. We compare our method with FeatLLM, which uses LLMs to extract rules (FeatLLM-rule) and functions (FeatLLM-function), respectively. Due to the designed example-free prompt, ProtoLLM achieves significantly lower token usage compared to both FeatLLM, highlighting its efficiency.

Table 5: Average number of tokens per query across datasets

Distance function used in ProtoLLM

ProtoLLM specifies the distance function as Euclidean distance by default. We also performed ablation experiments on Manhattan and cosine distance. The results in Table 2 of the submission show the robustness of our ProtoLLM to different distance functions.

Logistics at line 99

We have replaced "logistics" with "prediction probability" in the revision: All we need to calculate the prediction probability is the Euclidean distance between the input samples and obtained prototypes.

We appreciate the reviewer’s feedback and have provided the following responses to address the concerns raised about our paper. Please also check out the revised version of our manuscript (we have highlighted the revisions in blue), where notations, typos, and statements have been refined and revised for better understanding.

Additional results on open-source LLMs

Thank you for your valuable feedback. We understand the concerns surrounding the use of closed-source models like ChatGPT. However, we would like to highlight that a range of recent studies have successfully applied closed-source large models for few-shot classification [1-3] and other tasks such as feature engineering [4-5]. In response to your suggestion, we have expanded our experiments to include open-source LLMs, specifically LLaMA-3B and LLaMA-8B, to further validate our approach. However, as noted by Hugging Face, the T0* models are not well-suited for code-related tasks, which is why they were not included. More details on these experiments can be found in Table 1. We find that our ProtoLLM still achieves comparable results on open-source LLMs, showing the robustness of different base LLMs.

Table 1: Comparison of AUC scores with different LLMs

The datasets are small and non-standard

Our method is specifically designed for few-shot classification tasks, and we follow previous work[3] for dataset selection. Besides, the datasets used in ProtoLLM have also been used in other prior studies[2,6]. We also would like to highlight that our ProtoLLM beats LogReg and other LLM-based methods in most cases on NHANES and Cultivars datasets on both zero-shot (learning-based algorithms, such as LogReg fails to work) and very few-shot settings (len then 32). This demonstrates the effectiveness of our approach in zero or few-shot tasks, even on novel datasets.

Following your suggestion, we report the results on Arrhythmia, Soybean, and Cmc from TabZilla Benchmark in Table 2. We find that our ProtoLLM still achieves the best results in most cases when compared to traditional and LLM-based models, which increases the credibility of the proposed method.

Table 2: AUC score of different models on TabZilla Benchmark, where N/A means the method fails to run on such datasets due to the many examples or classes.

[1] Language models are weak learners, NeurIPS 2023

[2] Tabular Transfer Learning via Prompting LLMs, COLM 2024

[3] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning, ICML 2024

[4] Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering, NeurIPS 2023

[5] Optimized Feature Generation for Tabular Data via LLMswith Decision Tree Reasoning, NeurIPS 2024

[6] TabLLM: Few-shot Classification of Tabular Data with Large Language Models, AISTATS 2023

Thank you very much for your positive feedback and detailed comments on our response. Your valuable response helped us identify the limitations of our response at the first stage. This gives us an opportunity to improve the quality of our submission once again. We are grateful for the time and effort you have invested in reviewing our work.

Results on tabZilla benchmark

We apologize for misinterpreting your suggestion during the first rebuttal stage. We only reported results on three datasets rather than the whole tabZilla benchmark. We hope the following explanation will address your contents:

• First, we kindly remind you that we did not cherry-pick datasets at the first rebuttal stage. We experimented on the Soybean, Arrhythmia, and Cmc datasets, mainly because these datasets fulfilled the conditions of multi-class and multi-feature, among others, and were in line with the recommendations of the other reviewers.
• Most LLM-based models, such as FeatLLM and our ProtoLLM, require feature descriptions to improve the LLMs' understanding. We find 18 datasets within the tabZilla benchmark that have this type of information attached to them. We are in the process of comparing more of these datasets and will report the results as soon as possible (in 4 days).

Response to limitations

Thanks for your suggestion. We have added the discussions about the limitations in Appendix A.13 due to the limited space and we will add the section in the main paper in the camera ready. Below, we provide the details about limitations for your convenience: This work introduces an example-free and training-free ProtoLLM for zero and few-shot tabular classification task. It prompts the LLM for feature value generation based solely on task and feature description, making it difficult to apply to datasets without such a describing information. Besides, our ProtoLLM prompts the LLM without the training examples and designs the training-free prototype for classification instead of training a classifier. Therefore, it is more suitable for the zero and extreme few-shot regimes. With the increasing number of training samples, our approach is gradually closer to or inferior to other methods that depends on training samples and a learnable classifier, which is an interesting direction and can be considered in future work.

Response to comparison with TabLLM

For T0 models, they are not well-suited for code-related tasks {https://huggingface.co/bigscience/T0}. To achieve automatically feature value extraction and make the whole algorithm end-to-end, we ask LLMs to generate structure answers with the markdown code snippet formatted, including the leading and trailing "json" and "", as shown in Figure 3 in our submitted version. Therefore, using T0 as the backbone is not suitable for ours. We agree with you that GPT-3.5 is more powerful than T0. To address your concerns further, we consider conducting experiments on a relatively fair backbone, such as Llama3B. It is in the process and we will report the results as soon as possible (in 4 days).

Response to others

We have added the results of zero-shot baselines in Appendix A.5 in revision. Besides, we will add the missing datasets by the camera ready deadline.

Response to Benchmark

Following your suggestion, we further conducted experiments on the remaining datasets from tabZilla benchmark, where we consider datasets that contain the necessary descriptive information, such as feature names and descriptions. We report the results at Table 1 (Due to the complex table structure, please refer to the anonymous page for detailed results{https://anonymous.4open.science/r/anonymous-2D04/rebuttle_results.pdf}.

**Table 1: Additional datasets on tabZilla benchmark

Note: The bold text denotes the best performance, while the underlined text indicates the second-best performance. N/A indicates that the model has limitations preventing it from running on the dataset.

To summarize, from the results, we find that 1) Overall, Our ProtoLLM beats traditional and LLM-based models in most cases, achieving rankings of 1.62, 1.92, and 2.0, respectively; 2) Because of the efficiency of the introduced training and example-free strategy, our approach can be easily applied in most datasets (Other LLM-based models sometimes fail to run at complex datasets). This demonstrates the flexibility of our ProtoLLM.

Response to comparison with TabLLM

To address your concerns about a fair comparison with TabLLM, in the second rebuttal phase we found that fine-tuning TabLLM with Llama-3B often leads to poor results, so we decided to report our results for ProtoLLM based on the T0 model, as you had previously suggested. To adapt ours to T0, we designed slightly different prompts than those used in our original paper, as T0 is not capable of generating code. The prompt template is as follows:

Answer choices:{answer choices}

{task for a class}. Given a list of {feature name}({feature description}): {feature value list}. Which {feature name} should this belong to ?

Answer:

Below is an example of a prompt from the Adult dataset, where we query the possible feature values of the relationship feature for the class corresponding to individuals earning more than 50,000 dollars per year.

Answer choices:Own-child|||Husband|||Not-in-family|||Unmarried|||Wife|||Other-relative

The person earns more than 50000 dollars per year. Given a list of relationship (what this individual is relative to others): Own-child, Husband, Not-in-family, Unmarried, Wife and Other-relative. Which relationship should this belong to?

Answer:

For numerical features, we uniformly sample several feature values from the range provided by the few-shot samples to be used as answer choices. For categorical features, we list all the categories as answer choices.

Once we obtain the generated feature values from T0 with the designed prompt, we adopt them to build the prototype with Eq.(2) or Eq.(3) and then classify the test samples with Eq.(4) in our submitted version. The results are also provided in {https://anonymous.4open.science/r/anonymous-2D04/rebuttle\_results.pdf}. As shown in table 2, when taking T0 as the backbone like TabLLM, our proposed method with the degraded prompt performs better than TabLLM. It proves the effectiveness and robustness of ours, which avoids fine-tuning LLM and is available for different backbones. We will add the results into our camera ready. Thanks for your suggestions, which are helpful for improving the quality of our work.

We appreciate the reviewer’s feedback and have provided the following responses to address the concerns raised about our paper. Please also check out the revised version of our manuscript (we have highlighted the revisions in blue), where notations, typos, and statements have been refined and revised for better understanding.

Writing issues

We thank the reviewer for the helpful suggestion. We have revised our manuscript, where the feature value generation has been mathematically formulated, the captions in Figures 2-4 have been revised, and the "Oracle feature'' has been replaced with "LLM-generated feature''.

What is the value of z? For a dataset with $D$ features, our ProtoLLM aims to make use of the prior knowledge behind LLMs and ask LLMs to generate the feature values. Let $z_{c,d}$ denote the generated value for $d$-th feature in class $c$:

        $z_{c,d} = \text{LLM}(P_{d})[c]$,     if the $d$-th feature is a numerical feature,

        $z_{c,d} = \text{One-shot}(\text{LLM}(P_{d})[c])$,     if the $d$-th feature is a categorical feature,

where $P_{d}$ denotes the prompt input for $d$-th feature, and $\text{LLM}(\cdot)[c]$ denotes the output values of LLMs for $c$-th class. We directly use the output values of LLMs for numerical features and post-process the categorical features with the $\text{One-shot}(\cdot)$ function to convert the output class index to a one-hot vector.

colors in Figure 3, 4 Figure 3 shows an example of our designed prompt for ProtoLLM, which consists of two parts: <Meta-Info> (red color) and <Query> (green color). the blue-colored sentences denote the response formation. Figure 4 shows an example of the output of our ProtoLLM, where the blue-colored words denote the class names, and the red-colored sentences denote the corresponding output values of LLMs.

Experiments on multiclass datasets

To test the proposed model on zero and few-shot datasets, we follow previous works[1,2] for dataset selection and metric calculation. We find that most previous LLM-based tabular few-shot methods focus on binary classification or multiclass classification with a limited number of classes [1, 3]. Therefore, we initially evaluated our method on these commonly studied datasets.

Following your suggestion, we here report additional comparisons on three multiclass datasets from TabZilla Benchmark [5] and STUNT [4]. Table 1 and Table 2 report the statistics of datasets and results of different models respectively. From the results, we would like to note that: 1) Our ProtoLLM outperforms baselines in most cases, which shows the efficiency of the prototype-based strategy in multiclass settings. 2) TabPFN often works on small datasets, while FeatLLM inputs all examples into LLM to obtain decision rules, and thus tends to exceed the input length limit on large datasets. As a result, we only report the results of TabPFN and FeatLLM on the Cmc dataset. In opposite, our ProtoLLM queries LLMs feature-by-feature and thus it can process datasets with many features. This shows the flexibility of our approach.

Table 1: Statistics of multiclass datasets

Table 2: Results of different methods on multiclass datasets (mean values with 15 different runs. N/A means the method fails to run on such datasets).

AUC calculation on multiclass setting

we follow previous works[1,2] to calculate the AUC score. We find that some datasets are imbalanced. This makes AUC a more appropriate metric for evaluating model performance in these cases, as it better reflects the model's ability to distinguish between classes despite the imbalance. For multiclass classification, we compute AUC using sklearn.metrics.roc\_auc\_score with multi_class='ovr'. This computes the AUC of each class against the rest.

[1] TabLLM: Few-shot Classification of Tabular Data with Large Language Models.

[2] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning.

[3] Tabular Transfer Learning via Prompting LLMs.

[4] STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables.

[5] When Do Neural Nets Outperform Boosted Trees on Tabular Data?.

There are existing works focused on feature generation

We agree with you that feature generation is an efficient method for few-shot learning. Here we want to note that our method focuses on feature value generation using large language models, which is fundamentally different from feature generation. Feature generation creates new features, such as feature crossing or embeddings, to expand the feature set and is typically applied in scenarios with abundant data. In contrast, our method generates values for existing features without introducing new ones and is specifically designed for few-shot tasks, where data is scarce.

Comparison with LLM used Tabular learning frameworks

Thank you for highlighting the importance of additional baselines. We have conducted experiments to compare our method with the approaches mentioned. The detailed results are provided in Table 5. Specifically, LIFT-ICL corresponds to the method from [1], while P2T is based on [2]. Unfortunately, as the code for [11] is not publicly available, we were unable to include it in our evaluation. As shown in this table, ours achieves the best performance on most of the datasets and achieves comparable performance on Diabetes, showing the effectiveness of ours.

Table 5: AUC scores comparison with more baselines, where GPT-3.5 is used for all the baselines.

[1] LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks, NeurIPS 2022.

[2] Tabular Transfer Learning via Prompting LLMs, COLM 2024.

[3] Language models are weak learners, NeurIPS 2023.

Experiments on regression tasks

We in this paper mainly focus on integrating LLMs into tabular data understanding and following previous works conducted zero and few-shot classification tasks on various tabular datasets. Extensive results demonstrate the improvements and efficiency of the proposed model.

We agree with you that the tabular regression task is a significant application area for tabular learning. One of the possible directions to apply our ProtoLLM to regression tasks is viewing our generated features as a data augmentation tool. For example, we can concatenate all generated values feature-by-feature to form a virtual labeled sample. Together with few-shot labeled samples, we can improve the regression performance of baselines. We here report the root mean squared error (RMSE) score on abalone and California datasets at Table 3. We find that our ProtoLLM improves the baseline by a large margin in most cases. This demonstrates the potential of ProtoLLM for regression tasks. Note that it is a simple attempt to apply our ProtoLLM to regression tasks, we leave it as future work for more systematic and in-depth analysis.

Table 3: RMSE results on regression task (lower is better)

Results with variance

We kindly remind you that we had provided the detailed results, including standard deviation, in Table 9 of the appendix in the original submitted version. We find that our ProtoLLM yields smaller variance in most cases, showing better stability.

full-shot results

Following your suggestion, we report the full-shot results in Table 4. As shown in Table 4, Our ProtoLLM with few-shot samples achieves the desired performance when compared to the full-shot performance, where the average performance of 64-shot is very close to the performance of full-shot. Besides, using our LLM-generated feature values on some datasets (Blood, Cultivars, Myocardial) in the few-shot scenario can even outperform the full-shot setting. This highlights the effectiveness of the LLM-generated feature values in improving performance even with limited data. We have added the full-shot performance in our revised version.

Table 4: AUC scores comparison of ours with full-shot samples.

truly novel datasets

To generate the representative value for each feature, we query LLMs with detailed feature descriptions, this helps ProtoLLM to understand the novel features and thus has the ability to predict the correct values even for new features. Moreover, our ProtoLLM can be easily built upon different LLMs. As LLMs are updated iteratively, our ProtoLLM is able to understand newly created features. We are glad to test our ProtoLLM on novel features if the reviewer provides such datasets.

I thank the author for their time and efforts in writing the rebuttal. I have read the updated paper, which I believe is much clearer and better written. Also, I appreciate additional experiments on multi-class datasets, regression, and comparisons with LLM-based methods. In the final revision, I think it would be great to add some baselines on full-shot just to highlight the current method's limitations for the future researchers. Furthermore, I think it is valuable to discuss the difference between feature generation in the related work in the revision. While only focusing on a few-shot can be seen as a limitation, I think this work is still valuable, and future works should work on more challenging scenarios (e.g., I hope to see LLMs surpass boosting tree methods on a full shot as well). Overall, I think it is a solid paper with a small limitation (e.g., the main focus is few-shot). Therefore, I have adjusted my score accordingly (to 6). I thank the authors again for their time and efforts.

Prototype generation

To help you with how to prompt LLMs and reproduce the reasonable output, we provide our conversations with GPT-40 mini below (4/10 outputs the same values as Figure 4):

https://chatgpt.com/share/674eea54-537c-8007-be2f-ed602b1040d4

https://chatgpt.com/share/674eea9b-6bd8-8007-95b8-fab49068524a

https://chatgpt.com/share/674eea8c-d7b4-8007-8b63-61c85b906849

https://chatgpt.com/share/674ed472-65fc-8007-a08c-a5316290abd0

https://chatgpt.com/share/674eea63-ea84-8007-b54f-f5649f0012e8

https://chatgpt.com/share/674eea74-0db0-8007-b0e0-1f7cda093bef

https://chatgpt.com/share/674ee58c-fecc-8007-bbbd-e1a06eb8124e

https://chatgpt.com/share/674eeabd-8960-8007-9f27-f16d29f3fc31

https://chatgpt.com/share/674eea40-5534-8007-b324-9a9011253618

https://chatgpt.com/share/674ee61d-ee48-8007-915e-76a8b4d7321c

Besides, we also want to note that Fig. 3-4 is an example of our ProtoLLM. It is reasonable that the feature values you reproduce are not exactly the same as in Figures 4 and 9. We find that we can produce high quality prototypes as long as the output of LLMs makes sense. We suggest you calculate the AUC score before making your comments.

instability

We have reported the variance of our ProtoLLM in the appendix. The variance is a measure of model instability. Please take the time to check it.

Missing values

As for missing values, we in this paper mainly focus on querying LLMs to prototype generation and thus follow previous works [1-3] (including FeatLLM) in dealing with missing values. We do not think this is a technical weakness of this paper.

Finally, We thank all reviewers again for their time and valuable comments. The reviewers have provided our article with various high-quality suggestions. As experts in this field, I trust that they will make fair judgment.

[1] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning. ICML 2024

[2] The prevention and handling of the missing data, KJA 2013

[3] When and how should multiple imputation be used for handling missing data in randomised clinical trials–a practical guide with flowcharts, BMC Medical Res. Methodol. 2017

W7:Uncertainty about few-shot definition.

Please see the baseline details of appendix A.4, where we stated For each random seed, 20% of the datasets are designated as the test set. We then perform a balanced sampling of K instances from the remaining data, following the methodology in [1, 4]. The relationship between the K used in ProtoLLM (denoted as KP rotoLLM ) and traditional N-way K-shot setting can be formulated as: $K_{ProtoLLM} = N × K$. During rebuttal, the reviewers suggested testing ProtoLLM on the TabZilla benchmark, and we found several datasets with an odd number of categories. Therefore, we used the N-way K-shot setting.

W9:Lack of Theoretical Support.

This work provides a novel method to integrate the LLM into tabular few-shot classification, which we believe is interesting and useful for tabular learning. To sum up, our proposed method can be reproduced and we further provide the updated code for your convenience, where we provide more detailed descriptions for you. Our settings are generally following the closely-related work FeatLLM. Thanks for your time.

To sum up, our proposed method can be reproduced and we further provide the updated code for your convenience, where we provide more detailed descriptions for you. Our settings are generally following the closely-related work FeatLLM. Thanks for your time.

First, we are glad that you are interested in our approach and point out your concerns about the reported results. For a clearer and more correct reproduction of our results, we have reorganized and updated our code (https://anonymous.4open.science/r/anonymous-2D04/protoLLM\_code.zip). Please check and refer to the readme.md file for step-by-step instructions. Besides, we also provide all generated features of 10 datasets in the oracle_features folder.

W1: Prototype generation

• We agree with you that the generated feature values are unstable due to the randomness brought by LLM. Notably, it is one of the reasons that we query the LLM multiple times. Besides, we also provided the variance in the results in the submitted version.

• Figures 3-4 and 8-9 show the example responses rather than the only response from LLM. It makes sense that the feature values you reproduce are not exactly the same as in Figures 4 and 9. We suggest you reproduce the AUC score rather than the output feature value for more reasonable doubts.

• Possible inspection points. Please make sure that the version of LLM is gpt-3.5-turbo-0613 and the used prompts. We rerun our code and find that the same answer in Fig.4 appears 14 times out of 30. We also tested GPT-4o mini and find that the same answer in Fig. 4 was generated 3 out of 10 times.

W2: Reproducibility of AUC Results

Please carefully check out the version of LLM and the used prompts. You can refer to the oracle_features for the generated feature values or rerun run.py to generate values by yourselves. This makes sure the version of LLM and other hyperparameters remain the same as ours.

W3: The reported results for baseline methods exist issues

• We kindly remind you that before using the first few samples in each class, we have shuffled all samples of each dataset (line 96 in data.py). Therefore, our ProtoLLM also adopts a random sampling.
• You mentioned that you achieved better results with our protocol. We are curious about on which data this improvement is observed and by how much, and whether it falls within the range allowed by variance, as statistically, we consider it to be the same sampling strategy.

W4: the reported results of LogReg, XGBoost, TabPFN are diiferent from that reported in [1]

Because we needed to conduct ablations (Fig. 7 in the submission) using the traditional approach, we reproduced these methods and reported their results by ourselves. We summarize the results reported by FeatLLM and those by ours in Table 1. We can observe that no matter using the reported results from FeatLLM [1] or the results reproduced by us, our proposed ProtoLLM always outperforms baselines. We will cite these results reported by [1] additionally in our camera ready if necessary.

Table 1: Results of baselines in TfeatLLM and ProtoLLM. For each result in (a, b), a denotes the results in FeatLLM and b denotes the results in ProtoLLM.

W5: Missing values

For missing values, we follow FeatLLM [1] and other previous work [2-3] and simply handle them by removing the corresponding columns directly. We agree with you that dealing with missing values is a key aspect of feature heterogeneity, but this is beyond the scope of this paper.

[1] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning. ICML 2024

[2] The prevention and handling of the missing data, KJA 2013

[3] When and how should multiple imputation be used for handling missing data in randomised clinical trials–a practical guide with flowcharts, BMC Medical Res. Methodol. 2017

We respect that ICLR has an open review environment where anyone can comment to a paper. To help build a transparent and collaborative research community, we tried our best to response to your comments with detailed explanations and additional results, although we believe many of them are subjective, unfair and even incorrect. However, the quality of our paper is evaluated based on the professional reviews from our assigned reviewers, who have given professional and impartial comments to our paper. We trust that they will continue doing so without being biased by noises. We have decided not to response to you further. We thank you again for your interest in our paper and wish you have a nice day.