Making Pre-trained Language Models Great on Tabular Prediction

We profoundly thank you for the positive evaluations on the quality and significance of our paper, and propose constructive questions.

Q1: Can we add experiments on multi-class datasets?

A1: Thank you for your suggestion. We did not include the multi-class dataset for several reasons mentioned in the end of Sec. 2.3 (lines before Sec. 3). We would like to add some experiments on downstream multi-class datasets in the final version. Overall, we find our TP-BERTa outperforms deep model baselines and is comparable with GBDTs on multi-class datasets as well. Here are rank results (average values and standard deviations) on 32 extra multi-class datasets from the database, the TP-BERTa here is the version pre-trained on 101 binary classification datasets. Detailed results will be sorted into our final version.

Q2: Is it possible to perform RMT without an added loss (regularization loss) term and hyperparameter ($\lambda$) (e.g., by carefully designing the embeddings)?

A2: In our experiment, the hyperparameter $\lambda$ is fixed at 0.1 during pre-training, in fine-tune stage we did not use the regularization (i.e., $\lambda = 0$ in fine-tune stage). Therefore, strictly conducting ablations on $\lambda$ needs several pre-training groups, which is out of rebuttal time limit and our cost limit, because we pre-trained TP-BERTa on the rented commercial computing server, with each pre-training round taking 72～143 hours and hundreds of dollars (detailed pre-training budget is in Appendix D).

In order to explore the impact of the added loss, we would like to conduct the ablation by fine-tuning a non-pre-trained TP-BERTa with different $\lambda$s in substitution, where all magnitude tokens are randomly initialized at the beginning. Here are the AUC performance changes on 80 binary classification datasets by comparing the group to the one with no added loss (i.e., difference between a group and the group “$\lambda = 0$"), the value in column “$|\Delta| \le 3\\%$" denotes the number of datasets with AUC variation less than 3%.

It can be seen the most datasets are not significantly impacted by the added loss or the specific value of $\lambda$, but for several datasets (the datasets in the "$\Delta > 3\\%$" case) a regularization in pre-training is helpful. We further inspect these datasets and find most of them are very small (less than 1000 samples), which may be not sufficient to learn the reasonable magnitude token embeddings from scratch. Therefore, our designed regularization on RMT is to help fast convergence for the magnitude token learning, especially helpful for downstream small-scale datasets which have no sufficient data diversity. For normal-size datasets (in data volume and feature quantity), removing the added loss does not affect the performance.

Is it possible to drop the regularization (added loss)? We think it is possible to remove the $\lambda$ by careful initialization for magnitude token embeddings (like the initialization of LM position embedding[1]). Besides, we have supplied the t-SNE visualization (in current PDF revision, Appendix F, page 14, Fig. 4) of the 256 magnitude token embeddings, which are placed on a highly regular manifold (refer to Q5/A5 of the first reviewer(AREw, link) for detailed analysis). Based on the observed regular pattern of the magnitude token distribution, it is possible to design suitable initialization for these tokens with regular math functions, by adhering to the principle of “the closer two numerical values are, the more similar they should be” rather than using the added loss.

Q2: The insights into the computational complexity introduced by RMT and IFA, especially on the larger datasets?

A2: Assume $F$ is the feature number, $N$ is the data scale, $L$ is the layer number of the LM. In summary, RMT introduces extra CPU computational complexity of $\text{O}(FN\text{log}N)$, which is equivalent to training an $F$-feature C4.5 decision tree (which is known to be very efficient). Adding IFA gives a total LM computational complexity of $\text{O}(NF) + \text{O}(NLF^2)$, while without IFA, the complexity is at least $\text{O}(4NLF^2)$. Therefore, employing IFA is beneficial for optimizing computational resources.

In RMT process, for a dataset with $F$ numerical features and $N$ samples, we will fit $F$ single-feature C4.5 decision trees (DTs) to the labels $y$ for each numerical feature, i.e., fitting C4.5$^{(i)}(x_i)$ to $y$. Since learning C4.5 DTs is CPU operation, and RMT takes place in the tokenization process, in our implementation it was incorporated with the Tokenizer class in the transformers package as a preprocessing step. The extra CPU computational complexity is from training $F$ single-feature C4.5 DTs, i.e., $\text{O}(F) \times \text{O} (N\text{log}N \times 1) = \text{O}(FN\text{log}N)$, equivalent to training an $F$-feature C4.5 DT.

For IFA mechanisms, given a dataset with $F$ features and $N$ samples, we assume each feature name or feature value takes up 1 token (e.g., “Gender Female”, “Height 183.2”, each feature uses 3 tokens (including one $[\text{CLS}]$ token)), then for each sample the IFA complexity is $\text{O}(F \times 3^2)$, giving a total one of $\text{O}(NF \times 3^2)$. Notably, if introducing IFA, the computational complexity of the subsequent $L$-layer LM process becomes $\text{O}(NLF^2)$, hence giving $\text{O}(NF \times 3^2) + \text{O}(NLF^2)$ computational complexity for the whole process. But if removing IFA, feature-wise $[\text{CLS}]$ tokens can be removed and each feature uses 2 tokens, the whole process complexity is $\text{O}(NL(2F)^2) = \text{O}(4NLF^2)$. It can be seen using IFA asymptotically achieve 4 times better computational complexity when $F$ is very large, such theoretical acceleration effect supports the observation of 32% more average training time (the lower half of Table 2) after removing IFA. Besides, IFA achieves name-value matching before all features are fed into the LM, which may further accelerate convergence and alleviate training burden.

Q3: Are there dataset types or domains the TP-BERTa faces challenges?

A3: As analyzed in A1, for the TP-BERTa, there are still several challenging data scenarios:

(1) In the extreme class imbalance cases, the TP-BERTa exhibits a relative performance decline like other common tabular deep models for the data insufficiency of the minority class, in which case GBDTs with specific design for those issues play an active role.

(2) When facing extreme feature absence situations (a high feature missing rate), the TP-BERTa appears different degrees of performance decline upon the characteristics of missing features.

(3) Besides, since our TP-BERTa relies on the semantic knowledge of LMs to transfer feature patterns from pre-training feature names to downstream ones, it implicitly requires meaningful and clear feature semantics. However, there may exist some tables with unclear feature names or values, e.g., quantum physics experiment tables containing lots of uncommon quantum descriptor feature names and medical domain substituting specific feature values with meaningless encoding to protect patient privacy. At present, the current version of TP-BERTa does not adeptly handle this situation. We will investigate this issue in future work.

We will add the above data challenges into our limitations in the final version. Thank you!

We sincerely thank you for your insightful questions and constructive suggestions to make our work more thorough and comprehensive.

Q1: How does TP-BERTa perform when handling imbalanced datasets or missing values?

A1: In summary, without hyperparameter tuning, TP-BERTa can still hold its superiority in moderate class-imbalance or feature absence conditions. The detail experiment results and analysis are as follows.

Imbalanced-class scenario: to inspect the performances on imbalanced datasets, we create pivot tables by filtering datasets whose minor-class proportions, i.e., p = \frac{\text{min}(\\# \text{positive}, \\# \text{negative})}{\\# \text{samples}}, are less than 1/3 (32 datasets), 1/5 (18 datasets), 1/8 (12 datasets), 1/20 (4 datasets) from 80 binary classification datasets. We report the average ranks (standard deviations); “(d)” and “(t)” denote default and tuned hyper-parameters.

It is obvious that TP-BERTa outperforms baselines in moderate class-imbalance situations. In the extremely imbalanced situations (4 datasets, with $p <$ 0.05), GBDTs showcase dominating performances, but TP-BERTa still outperforms most DNN approaches. Perhaps this is attributed to TP-BERTa leveraging the transferability and semantic understanding capabilities of the language model, thus resulting in consistent performance across various levels of data imbalance. Notably, AutoInt[1] exhibits promising performances in the imbalance-class scenario, which is originally designed for click-through rate prediction (a classical binary classification task). We think this may due to its specific combinatorial feature design that helps to capture patterns for minority class by manually combining original features to generate new features. We will sort the pivot tables and analysis on imbalanced datasets into our final version.

Missing-value scenario: on the 32 imbalanced binary datasets ($p <$ 1/3), we further conduct missing-value experiments by randomly dropping a proportion of features for each sample (including training, validation and test sets), i.e., Xs[np.random(Xs.shape) < missing_rate] = np.nan. We conduct experiment on datasets with missing rates of 0.1 (10 % features dropped) and 0.3 (30 % features dropped) and compare with several representative baselines (default and fine-tuned). For missing rates exceeding 0.3 (e.g., 0.5 or higher), we observed that the performances of most baselines on various datasets closely resemble random guessing. Thus, we opted to exclude such extreme missing rates from our analysis. We employ the mean value for imputing missing numerical features and utilize the most frequent value for filling in missing categorical features. The average performance rank (standard deviations) among models are reported as follows, “(d)” and “(t)” denote that the default and tuned hyperparameters are used.

In summary, TP-BERTa inherently has a strong ability to handle imbalanced data and missing values, possibly stemming from the language model's robust generalization. However, it cannot surpass algorithms specifically optimized for such extreme scenarios (e.g.Catboost). We intend to delve deeper into this aspect in our future research. Thank you!

Q4: Can the RMT be fine-tuned or adaptively adjusted (e.g., bin size) based on the domain or the data to potentially yield better results?

A4: Will bin size of RMT affect the performance? From the comparison experiment of different magnitude token numbers (i.e., bin size) in Section 3.5 (detailed analysis in Appendix C and Table 5), when we decrease bin size from 256 to 128, in 80 binary classification datasets, significant performance decline appears in 26 datasets, while improvement is witnessed in 12 ones. When decreasing from 256 to 32, the quantities are 40 and 13. Overall, the performance changes indicate that each dataset prefer a specific bin size, and gradually increasing bin size to 256 can benefit the most datasets. Actually, this is a similar question in NLP tasks, where different text datasets perfer different embedding size. However, to leverage pre-training benefits, a uniform embedding size is a standard setting, and it also works well.

Can I manually add some extra relative magnitude tokens in fine-tune step, even though the bin size is kept fixed (i.e, 256) in pre-training? Will manually added bin size in fine-tune step hurt performance? Although the default TP-BERTa provides 256 pre-trained magnitude token embeddings, these tokens just represent a relative relationship of number magnitudes. Extra randomly initialized magnitude tokens in fine-tune will learn their embeddings locally, and the weights of pre-trained tokens (bins) still keep the relative relationships. It may lead to a slower convergence, but does not necessarily hurt the performance.

Adaptively adjust RMT according to data characteristics? In theory, each dataset prefers a specific bin size. However, in our experiment, performance improvement was observed on most datasets when we increased the bin size from 32 to 256, the performance decline empirically appeared in small-scale datasets. Due to the necessity of a fixed bin size during the pre-training step, we choose a common value of 256. Actually, this is a similar question in NLP tasks, where different text datasets have different embedding size (i.e., hidden dimension size) preference. To leverage pre-training benefits, a uniform embedding size is a standard setting, and it also works well.

Q5: Any visualization or analysis to assess how effectively the TP-BERTa captures the representations of numerical values?

We have added the visualization of the TP-BERTa‘s 256 magnitude tokens (before and after pre-training) by directly applying t-SNE algorithm (using scikit-learn package and default function parameters) on their embeddings (in Appendix F “Interpretability of RMT” of the current PDF revision, page 14, Fig. 4). Interestingly, even if we have randomly placed these 256 tokens in the language space at the beginning, after pre-training they captured a clear inherent distribution, all token embeddings lie on a highly regular manifold and successfully maintain an intuitive assumption that the embedding of a numerical value should close to the embedding of a nearby one. This visualization empirically demonstrates the TP-BERTa is sensitive to the numerical value magnitudes and benefits from the captured regular relationship among the relative magnitudes, which interprets its significant progress on the existing LM-based tabular models (e.g., directly treating values as raw strings). We hope the design philosophy of the TP-BERTa can stimulate broad reflection in the field of LM adaption to tabular data, and open up the mind for LLM enhancements.

Reference

[1] Song, Weiping, et al. "Autoint: Automatic feature interaction learning via self-attentive neural networks." CIKM. 2019.

[2] Prokhorenkova, Liudmila, et al. "CatBoost: unbiased boosting with categorical features." NeurIPS. 2018.

Thank the authors for the detailed response which well addressed my main concerns! I raised my score accordingly.

Q1: It is unclear if some tabular data distribution patterns can be captured with the LMs, like word meaning or grammars captured in text pre-training paradigm (e.g., auto-encoder LMs with the masked language modeling, auto-regressive LMs with the autoregressive one)?

A1: Thank you for your concern. We will clarify the question from 3 aspects: previous literatures, main contributions, and the potential values of our work.

Previous literatures: some previous works also use LM adaption for tabular data related tasks (e.g., GReaT[1], TapTap[2] and LUNA[3]), and they have proved the capability and feasibility of LMs to model the tabular patterns. Notably, none of these LM-based tabular models is tailored for high-precision tabular prediction tasks, they all directly treat numerical tables as template texts and are designed for generation tasks, such as synthetic table generation, the LM generated tabular data shows great distribution similarity with the original data, implying LM's potential in modeling tabular data patterns[1].

Main contributions: our primary endeavor is (1) “making” pre-trained LMs (e.g., RoBERTa) great on tabular prediction tasks; (2) making cross-table pre-training on heterogeneous tabular datasets, such pre-training feasibility is based on the LM ability of associating heterogeneous features in the language space, i.e., when we transform features into texts and maintain LMs' sensitivity to the numerical values, they can transfer tabular patterns across tables. We will introduce our efforts in tailoring LMs to tabular data patterns, and demonstrating their superiority on tabular prediction tasks.

(1) Technical contributions

• To make LMs sensitive to numerical values, we propose RMT adaption technique. It is the first attempt to represent relative magnitudes of numerical values in the language space (and has interpretability with clear visualized distribution, please refer to Q2/A2 for details), conveying a numerical processing philosophy in LM adaption to tabular data. RMT significantly outperforms the numerical encoding strategy of existing LM-based tabular models (12.45% average AUC promotion compared to directly treating values as raw strings, in the upper half of Table 2), it enhances the LM sensitivity to the numerical value, which is a common and critical in high-precision tabular prediction tasks (e.g., regression).

• To assist LMs in learning tabular feature organization, we propose the IFA mechanism to explicitly match tabular feature name-value pair for a better contextual understanding. IFA can also significantly save training time (for the analysis of IFA theory computational complexity, please refer to the Q2/A2 of the reviewer AREw, which makes the TP-BERTa reproduction-friendly and easy to follow.

(2) Empirical contributions: we conduct extensive evaluations to demonstrate the traditional LMs, with suitable adaptions, has the potential to be comparable with or even exceed the strong baselines on tabular prediction tasks (in Table 1), which indicates LMs are great on tabular prediction and shows a promising new choice for transferring knowledge across tables (it was challenging for the heterogenous nature of tabular features, traditional tabular transfer models [4, 5] rely on overlapped features between pre-training datasets and downstream ones, while LMs can inherently associate features in the language space). Besides, we explore the feature appetite of the LMs in handling tabular data, which can help architecture selection based on feature type characteristics in future tabular prediction studies.

Potential values: the TP-BERTa is not a “naive SOTA work”, our primary focus lies in two facets: (1) “making”, i.e., conveying the design philosophy of the TP-BERTa (RMT and IFA) to make LMs aware of numerical values and tabular feature organization. The proposed methodology helps LMs to learn tabular patterns and improve efficiency (performance and training speed) on tabular prediction tasks, we hope TP-BERTa can stimulate broad reflection in the edge-cutting fields like LM adaption to tabular data, tabular foundation models and LLM enhancements. (2) the TP-BERTa sends a significant signal that pure NLP LMs can also outperform in high-precision tabular tasks, which provides a completely brand perspective in the tabular transfer learning field.

Overall, we make novel technical endeavor in adapting LMs to tabular patterns and extensively demonstrate their promising tabular prediction performances.

We really thank you for finding our work novel and distinguishable from the traditional tabular deep models, and propose insightful questions!

Q1: The impact of pre-training dataset diversity?

A1: Empirically, when using less diverse pre-training datasets, the model performance is likely to asymptotically degraded to the performance of using RoBERTa weights in Table 3. We additionally conduct TP-BERTa (single task) pre-training with 3, 10, 30 datasets (randomly selected from 101 binary classification datasets) respectively. In 10-dataset pre-training experiment, we repeated 3 groups (group $a$, $b$, $c$) with disjoint 10 datasets in each group. The results are reported as follows, the value in “$|\Delta| \le 0.5\\%$" group denotes the number of datasets with AUC variation less than 0.5%, “Avg. diff.” means the average performance difference on significantly changed datasets, “TP-BERTa($n$)” means the TP-BERTa pre-trained on $n$ datasets.

When pre-training dataset diversity is insufficient (3-dataset group), the TP-BERTa performance is almost close to the non-pre-trained one (RoBERTa initialization). Even all use 10 datasets, the 3 10-dataset groups exhibit different downstream performances since their different seen features in pre-training, some of which hold more benefit in downstream datasets. Besides, randomly sampled pre-training datasets varies in data volumes, i.e., different 10-dataset groups are still different in data diversity. Therefore, our primary contribution focuses on “making” LMs great on the tabular prediction (in the title), i.e., RMT and IFA adaption techniques, in which RMT elevates LM-based tabular models to an entirely new level (compared to the numerical processing strategy of existing LM-based tabular models[1, 2, 3]), especially in high-precision tabular prediction tasks. We hope the design of TP-BERTa can stimulate broad reflection in this edge-cutting field and tabular foundation models (e.g., tabular LLM design, LLM enhancement on table processing).

Q2: Compared to RMT, the IFA method seems marginally novel and not showing significant performance boost?

A2: Thank you for finding our IFA novel! The function of IFA is to reduce theory computational complexity introduced by the large language model and provide potential advantage of fast convergence (for detailed complexity and advantage analysis, please refer to the Q2/A2 of the reviewer AREw, which is also important under the conditions of limited computation resource or time, making it reproduction-friendly and easy to follow. It takes 32% more average training time after removing IFA (lower-half of the Table 2).

Reference

[1] Borisov, Vadim, et al. "Language Models are Realistic Tabular Data Generators." ICLR. 2022.

[2] Zhang, Tianping, et al. "Generative Table Pre-training Empowers Models for Tabular Prediction." arXiv. 2023.

[3] Han, Hongwei, et al. "LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-training." arXiv. 2022.