Language-Interfaced Tabular Oversampling via Progressive Imputation and Self-Authentication

We sincerely appreciate the reviewer tYE6’s constructive feedback. We address the reviewer’s concerns below.

[W1, W2. Understanding the semantics of feature names.]

As the reviewer has mentioned, the performance of LLMs may vary based on the semantic meaning of column feature names, as observed in [1]. However, even in the absence of column names, the language model does not break down in performance but rather is capable of processing the given data via its general sequential modeling capability originating from the pretrained weights when it is fine-tuned on the tabular dataset. For example, [2] leverages such capabilities to process time series data, which do not contain linguistic meaning.

This capability is further observed in our main experiments. In the datasets used in our experiments, there are cases where column names do contain semantic meaning (e.g., Shoppers, Diabetes), contain semantics but lack context (e.g. Default), contains abbreviations (e.g., Sick, Obesity), or do not contain any meaning (e.g. Satimage). Below, we have listed selected column (feature) names for each dataset:

Default: ['LIMIT_BAL', 'PAY_2', 'BILL_AMT1', 'PAY_AMT1', 'PAY_AMT2', …]

Shoppers: ['ExitRates', 'PageValues', 'SpecialDay', 'Month', 'VisitorType', …]

Diabetes: [’Glucose’, ‘BloodPressure’, ‘Insulin’, ‘BMI’, ‘Age’, …]

Sick: [’TSH’, ‘T3’, ‘TT4’, ‘T4U’, ‘FTI’, …]

Obesity: [’FAVC’, ‘NCP’, ‘CAEK’, ‘CH2O’, ‘FAF’, … ]

Satimage: [Aattr, Cattr, A1attr, D10attr, D28attr, …]

Even presented with a wide and versatile range of semantics levels for feature names, the main experiments in Section 4 show that our method surpasses other methods in the oversampling task. In Appendix C.1.1 of the revised manuscript, we provide the feature-wise histogram visualizations of synthetic samples w.r.t. the real distribution, where our method exceeds other baselines in modeling individual columns of varying levels of semantics (e.g. PAY_2, Month, T4U, D10attr, Cattr), even for datasets such as Satimage where feature names are devoid of meaning.

[W1. Reliability of the synthesized samples.]

Since the backbone tabular language model(TLM) of LITO is first fine-tuned on a given tabular dataset, the TLM can successfully learn the given tabular data and the relative relations of columns, as shown in [1, 3]. However, as the reviewer has mentioned, recent observations in language models have shown that they can generate counterfactual texts. This phenomenon is exacerbated when the availability of minority-class samples is low, which is verified in our experimental observation where a TLM baseline(GReaT) produces samples that intrude into the distribution of other classes even when it is prompted to generate a certain minority class, which can be considered as counterfactual generation. This is in fact why we employ self-authentication to filter out the counterfactual samples. Moreover, our importance-aware imputation method only re-samples the important columns, keeping other columns in the real data distribution - reducing counterfactual noise.

Visualizations in Appendix C.1.2 show that our method effectively reduces counterfactual generation(intrusion into other classes) compared to baselines. Quantitative analyses in Appendix C.2 indicate that LITO produces meaningful samples(coverage) while suppressing counterfactual samples(spillage).

[Q1. classification accuracy of the language model on original data.]

In the table below, we report the classification performance of the backbone TLM on the test dataset:

[1] Dinh et al. "LIFT: Language-interfaced fine-tuning for non-language machine learning tasks." NeurIPS 2022.

[2] zhou et al. “One Fits All: Power General Time Series Analysis by Pretrained LM.” NeurIPS 2023.

[3] Borisov et al. "Language models are realistic tabular data generators." ICLR 2023.

[W3. Using different language models e.g. GPT-3 or ChatGPT.]

Since the general capabilities of language models increase with their number of parameters, their capability to model tabular data can also increase with model size, as observed in [6, 7]. Since our method utilizes a language model backbone, the performance of our method can be further improved when equipped with a larger language model. This is observed in Section 4.5 of the main paper, where we adopt LITO to ChatGPT-3.5-turbo via in-context learning (ICL) and conduct experiments on the Diabetes dataset for imbalance ratio 20. We briefly summarize the results in the table below:

Experimental results show that utilizing larger language models results in a further performance increase of our method.

[1] Grinsztajn et al. "Why do tree-based models still outperform deep learning on typical tabular data?." NeurIPS 2022.

[2] Kim et al. "SOS: score-based oversampling for tabular data." KDD 2022.

[3] Han et al. “Borderline-smote: a new over-sampling method in imbalanced data sets learning.” ICIC 2005.

[4] McInnes et al. UMAP: Uniform Manifold Approximation and Projection. Journal of Open Source Software, 2018.

[5] Naeem et al. "Reliable fidelity and diversity metrics for generative models." ICML 2020.

[6] Dinh et al. "LIFT: Language-interfaced fine-tuning for non-language machine learning tasks." NeurIPS 2022.

[7] Borisov et al. "Language models are realistic tabular data generators." ICLR 2023.

[W2. Quality of the synthesized data.]

As observed in our main experiments(Section 4.2, 4.3) of the main paper, the classification performance increase of various downstream machine learning classifiers (logistic regression, xgboost, ... etc.) augmented with LITO oversampling reflects that our method generates quality samples that are effective in increasing classification performance compared to other baselines.

To further examine the qualities of the synthesized data, we undertake both qualitative and quantitative analysis of synthesized samples. The detailed figures and descriptions are provided in Appendix C.1 and C.2 of the revised manuscript.

In our qualitative analysis, following the methodology of previous work[2], we present feature-wise histograms and UMAP[4] visualization plots for synthesized samples for various datasets, which are included in Appendix C.1. These histograms reveal that LITO's column features more closely match the actual minority distribution than other comparison baselines, demonstrating the quality of the samples synthesized with our method. Additionally, we construct UMAP diagrams to visually compare the manifolds occupied by both the training data and the synthetic minorities created by our method and other baselines. The results demonstrate the ability of LITO to accurately represent the true minority distribution while avoiding intrusion into the distribution of other classes.

To quantitatively assess the quality of synthesized samples, we employ the Coverage Score[5] and measure the overlap of synthetic minority samples with the real minority samples. However, focusing solely on coverage w.r.t. the real minority samples may neglect the fact that naive high coverage can lead to overlap with the majority class distributions, reducing the effectiveness of minority oversampling and adversely affecting the performance of downstream classification tasks. Consequently, we also calculated the coverage score between the synthesized minorities and real majority samples, a metric we refer to as "spillage" temporarily. For relative reference, we also measure the natural spillage of real minority data into the real majority data and report its difference. The provided table contrasts the coverage and spillage scores of LITO with those of recent baselines. LITO demonstrates a balance of high coverage and low spillage, whereas GReaT exhibits significant spillage and SOS has limited coverage. These findings suggest that our approach not only generates a diverse set of samples but also maintains their quality. This pattern is also observed in the UMAP visualizations in Appendix C.1.

To further address the reviewer’s concerns, we record the acceptance ratios of self-authentication. The table below shows the sample acceptance ratio for the datasets used in the main experiment. As the imbalance ratio increases, the acceptance ratio is lower, maintaining the sample quality.

We sincerely thank the reviewer gyRK for their insightful comments. We address the reviewer's concerns below.

[W1. Novel contribution of the proposed method.]

While data augmentation strategies via language models are on the rise, our work focuses on whether language models are practically applicable for tabular data generation under class-imbalanced situations, which are frequent in the tabular domain. To this end, we propose a novel oversampling method with two components: progressive importance-aware imputation and self-authentication. In comparison to prior works, our paper introduces novel contributions in both its findings and methodology, which include:

1. Experimental observations (Section 4.2, 4.3, and Appendix C of the revised manuscript) show that naive oversampling with a tabular language model(GReaT) fine-tuned on imbalanced data to oversample minority class samples results in low-quality samples that degrade downstream classification performance(as in samples that overlap into the distributions of other classes), due to the scarcity of minority class samples.
2. To filter these samples and enhance sample quality, we propose self-authentication, which utilizes the language model itself. Our experiments in Section 4.4 show that self-authentication is an effective component for increasing the quality of the generated samples, which is reflected in the increase in downstream classification performance.
3. We propose progressive importance-aware imputation, a novel borderline sampling strategy that integrates the domain characteristics of tabular data and the capabilities of language models.
   • Unlike other data modalities(such as vision, speech, …) where class-relevant information is spread across features, the features of tabular data each hold distinctive information, where there exists columns important to the given task, and also relatively unimportant columns[1]. However, previous borderline sampling strategies such as [2, 3] manipulate all features simultaneously - introducing potential noise in the unimportant columns thereby degrading sample quality.
   • Using this domain characteristic, we propose to iteratively select, puncture, and re-impute the important columns, using the conditional sampling capability of the language model. This ensures the feature quality of the unimportant columns as they stay in the real distribution, while important columns are resampled for diversity.
   • To identify the instance-wise important columns, we propose to leverage the self-attention maps of the language model to measure the feature importance.

Thank you for your valuable insights and reconsideration of scores. We are pleased that the explanations and additional experiments have addressed the reviewer’s concerns. We sincerely appreciate your time in reviewing our response and paper.

[W2. Novelty of the proposed method.]

Although [4] has demonstrated the possibility of language models for the generation of tabular data, our work focuses on the practical applicability of such tabular language models - whether they can be effectively adopted for oversampling to remedy the class-imbalance problem, which is a frequent and substantial issue in the tabular domain. To this end, we propose a novel sampling strategy comprised of two components: progressive importance-aware imputation and self-authentication. Compared to previous works, our proposed work contains the following novel contributions:

1. Our experimental observations, including Section 4.2, 4.3, and Appendix C of the revised paper, show that naive oversampling with a tabular language model(GReaT) fine-tuned on imbalanced data to oversample minority class samples results in ill-generated samples (for example, synthetic samples that intrude into distributions of other classes) that degrade downstream classification performance, due to the scarcity of minority class samples.
2. To filter out such samples, we propose self-authentication, which utilizes the capabilities of the language model itself. Our experiments in Section 4.4 show that self-authentication serves as an effective filter for maintaining the quality of the tabular samples generated by a tabular language model.
3. We propose progressive importance-aware imputation, a novel borderline sampling strategy that combines and leverages the characteristics of the tabular domain and the capabilities of language models simultaneously.
   • Unlike other data modalities(e.g. vision, speech) where class-relevant information is spread across features, the column features of tabular data each hold distinctive information, where there exist columns important to the given task, and also relatively unimportant columns[5]. However, previous borderline sampling strategies such as [1, 6] manipulate all features simultaneously - introducing potential noise in the unimportant columns thereby degrading sample quality.
   • Inspired by this characteristic, we propose to select, puncture, and re-impute the important columns iteratively, using the conditional sampling capability of the language model. This ensures the feature quality of the unimportant columns as they stay in the real distribution, while important columns are resampled for diversity.
   • We propose to utilize the inherent self-attention maps of the language model to identify and measure the importance of column features instance-wise.

Due to possible technical issues with the website, citation [2] mentioned by the reviewer in the review text is not visible to us. To this end, we politely request the reviewer to re-inform us of the name of the paper for citation [2] in the review.

[1] Kim et al. "SOS: score-based oversampling for tabular data." KDD 2022.

[2] McInnes et al. UMAP: Uniform Manifold Approximation and Projection. Journal of Open Source Software, 2018.

[3] Naeem et al. "Reliable fidelity and diversity metrics for generative models." ICML 2020.

[4] Borisov et al. "Language models are realistic tabular data generators." ICLR 2023.

[5] Grinsztajn et al. "Why do tree-based models still outperform deep learning on typical tabular data?." NeurIPS 2022.

[6] Han et al. “Borderline-smote: a new over-sampling method in imbalanced data sets learning.” ICIC 2005.

We sincerely thank reviewer puAc for constructive feedbacks. We address the reviewer's concerns below.

[W1, Q1. Diversity of the generated synthetic minority samples.]

In order to examine the diversity of the synthetic minority samples generated by LITO(ours), we conducted additional qualitative and quantitative analyses, where detailed figures and descriptions are added in Appendix C of the revised manuscript. For qualitative analyses, we follow previous work[1] and plot the individual feature-wise histograms and UMAP[2] visualizations of synthesized minority samples for multiple datasets. The resulting figures are presented in Appendix C.1.1 and C.1.2, respectively. The individual feature-wise histograms show that the column features generated by LITO best resemble the ground truth minority distribution compared to other baselines, showcasing sampling diversity and quality. Also, we plot UMAP diagrams to visually observe the manifolds occupied by the training data and synthetic minorities generated by our method and other baselines. As seen in the figures, our method effectively covers the ground truth minority distribution while refraining from intruding into the distribution covered by other classes.

To quantitatively measure the diversity of the synthesized minority samples compared to the ground truth minority distribution, we measure the Coverage Score[3] between the real minority samples and synthesized minority samples (Appendix C.2). However, measuring coverage only against the real minority samples neglects the quality aspect of the synthetic samples, as a distribution with high coverage only may simply ‘spill’ into the distribution of majority class samples, degrading minority oversampling quality thereby lowering downstream classification performance. To this end, we also measure the Coverage score between real majority samples and synthesized minorities (for convenience, we temporarily term this as "spillage"). For relative reference, we also measure the natural spillage of real minority data into the real majority data and report its difference. The table below shows the coverage and spillage scores of LITO and recent competing baselines. LITO exhibits high coverage while maintaining low spillage, while GReaT shows high spillage, and SOS shows low coverage. These results indicate that our method generates diverse samples while maintaining quality. This tendency is also visually confirmed in the UMAP visualizations in Appendix C.1.2.

We also analyze the label distribution of synthesized samples. Since we cannot know the oracle true labels for synthetic samples, we approximately infer their labels using an XGBoost classifier trained using the full training data. Since the test accuracy of the XGBoost classifier is not 100%, there exist potential discrepancies. The table below shows the inferred label distribution: the percentage of synthesized samples produced with our method and competing baselines that are predicted as the minority class, where our method surpasses or is on par with other baselines.

XGBoost bAcc: 71.19(Default), 78.83(Shoppers), 82.61(Sick), 68.23(Diabetes)

Finally, as observed in our main experiments(Section 4.2, 4.3) of the main paper, the increase in classification performance of various downstream machine learning classifiers (logistic regression, xgboost, ... etc.) trained with LITO-oversampled data also reflects the quality and diversity of the synthetic minority samples generated by our method, compared to other baselines.

Thank you for reviewing our response. We sincerely appreciate your time and effort in reviewing our paper and giving constructive feedback.