Language Models Are Good Tabular Learners

The authors thank the reviewer for comments. Here is the one-to-one response:

• Weakness 1: We focus on relational tables where the number of cells per column or per row does not change. In other words, given a table, the number of cells per column does not change and the number of cells per row does not change either. We do not consider cross-table tasks, so it is permissible if the number of cells changes when moving to other tables. Essentially different datasets correspond to different models. Here different models have two meanings: (1) the model weight is different. (2) the sequence length for numerical columns (or binary columns) is different. There will not be a problem when changing the table since we do not pre-train the model on one table and do fine-tuning on another table.

• Weakness 2: Both SCL and SSCL can be used in the pre-training process. As indicated in Eq. 11 and Eq.12, SSCl considers positive pairs for only the table row and its corresponding corrupted counterpart. SCL considers positive pairs for the table row and the corrupted rows that have the same label as that row.

• Weakness 3: We try to get the performance on the reference that the reviewer points out. However, there are more than 3 bugs in the official repo (https://github.com/jyansir/tp-berta) preventing us from obtaining the performance. Bugs we find include:

1. bin/tpberta_modeling.py line 79 indicates that embeddings has the same sequence length as input_scales. In lib/data_utils.py, the embeddings is related to num_fix_part and input_scales is related to num_input_scales. On line 379 of data_utils.py, the length of num_fix_part is equal to len(efn) + 2 because original code is [cls_token_id] + efn + [mask_token_id]. One line 384 of data_utils.py, the length of num_input_scales is 1 + len(efn). When we changed 1 + len(efn) to 2 + len(efn), the error message was gone.

2. In scripts/pretrain/pretrain_tpberta.py, the method of the Regulator class is called on line 154 regulator.trainer_end(args, model). However, in the definition of the Regulator class, the definition of this method is def trainer_end(self, model: nn.Module) on line 461 of lib/optim.py. When we changed it to def trainer_end(self, args, model), the error message was gone.

3. After pre-training, when loading the pre-trained model weights, one of the related code lines is state_dict['tpberta.embeddings.position_ids'] (bin/tpberta_modeling.py). When we examined the checkpoint, there is no position_ids in the checkpoint. What the checkpoint has is:

odict_keys(['tpberta.embeddings.word_embeddings.weight', 'tpberta.embeddings.position_embeddings.weight', 'tpberta.embeddings.token_type_embeddings.weight', 'tpberta.embeddings.LayerNorm.weight', 'tpberta.embeddings.LayerNorm.bias', 'tpberta.intra_attention.W_q.weight', 'tpberta.intra_attention.W_q.bias', 'tpberta.intra_attention.W_k.weight', 'tpberta.intra_attention.W_k.bias', 'tpberta.intra_attention.W_v.weight', 'tpberta.intra_attention.W_v.bias', 'tpberta.intra_attention.W_out.weight', 'tpberta.intra_attention.W_out.bias', 'tpberta.encoder.layer.0.attention.self.query.weight', 'tpberta.encoder.layer.0.attention.self.query.bias', 'tpberta.encoder.layer.0.attention.self.key.weight', 'tpberta.encoder.layer.0.attention.self.key.bias', 'tpberta.encoder.layer.0.attention.self.value.weight', 'tpberta.encoder.layer.0.attention.self.value.bias', 'tpberta.encoder.layer.0.attention.output.dense.weight', 'tpberta.encoder.layer.0.attention.output.dense.bias', 'tpberta.encoder.layer.0.attention.output.LayerNorm.weight', 'tpberta.encoder.layer.0.attention.output.LayerNorm.bias', 'tpberta.encoder.layer.0.intermediate.dense.weight', 'tpberta.encoder.layer.0.intermediate.dense.bias', 'tpberta.encoder.layer.0.output.dense.weight', 'tpberta.encoder.layer.0.output.dense.bias', 'tpberta.encoder.layer.0.output.LayerNorm.weight', 'tpberta.encoder.layer.0.output.LayerNorm.bias', 'tpberta.encoder.layer.1.attention.self.query.weight', 'tpberta.encoder.layer.1.attention.self.query.bias', 'tpberta.encoder.layer.1.attention.self.key.weight', 'tpberta.encoder.layer.1.attention.self.key.bias', 'tpberta.encoder.layer.1.attention.self.value.weight', 'tpberta.encoder.layer.1.attention.self.value.bias', 'tpberta.encoder.layer.1.attention.output.dense.weight', 'tpberta.encoder.layer.1.attention.output.dense.bias', 'tpberta.encoder.layer.1.attention.output.LayerNorm.weight', 'tpberta.encoder.layer.1.attention.output.LayerNorm.bias', 'tpberta.encoder.layer.1.intermediate.dense.weight', 'tpberta.encoder.layer.1.intermediate.dense.bias', 'tpberta.encoder.layer.1.output.dense.weight', 'tpberta.encoder.layer.1.output.dense.bias', 'tpberta.encoder.layer.1.output.LayerNorm.weight', 'tpberta.encoder.layer.1.output.LayerNorm.bias',
... <deeper layers>

We have contacted the authors from the mentioned reference. If we have the code work before the deadline, we will update the manuscript.

Due to the time limit, we are sorry that we might not be able to provide the performance on the reference the reviewer mentioned although we have spent long time trying to make it work. When we tried to debug, we needed to make sure not only there was no bug but also that the modification would not cause the wrong calculation.

• Question 1: We use the default hyperparameters for XGBoost. When we did experiments across 76 tables, we used the same hyperparameters. Ideally, 76 tables might correspond to 76 sets of optimal hyperparameters. We want to make sure the model complexity is the same across different datasets, so we do not do hyperparameter tuning.

• Question 2: CTA adds positional encoding to the embedding of categorical columns as indicated by Eq. 10. The categorical column embeddings serialize columns into natural language sentences. Hence, we add positional encoding. The numerical columns and binary columns are similar to sentence embedding where the embedding aggregates information from all sequence embeddings. Hence, we do not add positional encoding to enforce the permutation invariance of tables. In the ablation study, we do find removing positional encoding can have a remarkable effect on the embedding space as shown in Figure 4. We would like to emphasize that CTA does not entirely remove positional encoding. It only removes the positional encoding for numerical column embeddings and binary embeddings.

Our method does not work on cross-table tasks. Besides, the embedding is generated per row. Four rows in two example tables are the same. We do not think the embedding will be the same as the prediction done by feeding row embedding z_i to the classification head.

• Question 3: Yes, our method does take column names into consideration. That is the strength of our method which utilizes the power of language models for semantic understanding. Most recent tabular deep learning works use the embedding layer to encode cells which does not consider column name and semantics of cell values. Our method does consider them. Please refer to the section 3.1 for column embeddings.

• Question 4: PLE depends on the distribution of cell values within the same column. Before PLE, we normalize cell values in the data preprocessing. We tested z-norm and min-max normalization. For the min-max normalization, the scaling does not change normalized values. Hence, PLE does not change. For the z-norm, the scaling has a limited effect on the normalized values.

• Question 5: Using LLM will inevitably introduce a lot more number of trainable parameters, which makes it less competitive compared to tree-based methods such as XGBoost. We tested RoBERTa [2] as the backbone model (the number of layers and the hidden dimension are much smaller than the default parameters). The result is reported in the Appendix A.2.

Could the reviewer give examples of simple LLM baseline?

[1] Chen, Tianqi, and Carlos Guestrin. "Xgboost: A scalable tree boosting system." Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016.

[2] Liu, Yinhan. "Roberta: A robustly optimized bert pretraining approach." arXiv preprint arXiv:1907.11692 364 (2019).

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

We thank the reviewer for the feedback. Below is the one-to-one response:

• Weakness 1: our method can handle arbitrary-sized tables.

We concatenated the embeddings of different types of columns as indicated in Equation 8, which allows different numbers of columns.

We tested 76 tables. The number of numerical columns, categorical columns and binary columns are shown in Appendix A.3 (please refer to Supplementary Material). The number of columns do vary in those 76 tables.

• Weakness 2: The contrastive learning essentially builds positive and negative pairs. In our method, we make a prediction for each table row based on the embedding of that row as indicated in the Task formulation in Section 3. Pairs are generated by the corruption of table rows. For example, if there are table rows $r_1$ and $r_2$ in a table, the corrupted counterparts of these two table rows are $\tilde{r}_1$ and $\tilde{r}_2$. In SCL, if $r_1$ and $r_2$ have the same label, ($r_1$, $\tilde{r}_1$) and ($r_1$, $\tilde{r}_2$) are considered as positive pairs. Whereas in SSCL, ($r_1$, $\tilde{r}_2$) are considered as negative pair. Hence, in Equation 12 for SCL, we have a summation for $k \in P(i)$ while there is no such summation in Equation 11 for SSCL.

• Weakness 3: When we ran baselines, we use the default fixed hyperparameters. When changing hyperparameters such as the model depth, the model complexity will vary. We want to make sure the model complexity is the same across datasets (the model complexity of different methods is shown in Appendix Figure 7). Hence, we use fixed hyperparameters instead of the optimal hyperameters for each dataset, i.e., the hyperparameters for 76 tables are the same.

• Question 4: Table (1) and Table (2) will have different embeddings as the rows are different (one table has rows of [0, 0] and [1, 1] while the other one has [0, 1] and [1, 0]). The prediction is based on the table row as indicated in Task formulation in Section 3.

• Additional Comment: Our method has only 32.31 M trainable parameters as shown in Appendix Figure 7 while the prevalent LLM such as Llama 7B [1] has 7B trainable parameters. In addition, our method trains the model from scratch and does not rely on prior knowledge from the massive pre-training datasets. Hence, we did not compare our method against the prompting method on pre-trained LLM.

We are happy to provide further clarifications or modifications if the reviewer have more concerns.

[1] Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

I thank the authors for providing a response. I am really sorry for the late response. I am not increasing scores.

I'm not sure I'm understanding parts of the response.

• Weakness 1: do you simply mean that the tasks do not require capability to handle arbitrary sized tables?
• Weakness 2: I don't understand the following sentence "SCL considers positive pairs for the table row and the corrupted rows that have the same label as that row."
• Weakness 3: was there any hparam tuning for the proposed method at all? if the answer is positive, why wouldn't your tuning methodology work for the baseline as well?
• Question 4: Based on your response, it seems to me that tables (1) and (2) would have the same representation. If so, this makes me doubt of the practical usefulness of the model, regardless of whether permutation invariance is useful in the experimental benchmark.
• LLM baseline: few/zero-shot prompt with examples and list of of output classes.

The authors thank the reviewer for feedback. Here is the one-to-one response:

• Weakness 1: The reviewer’s concern makes sense. We made this statement based on the existing works [1-4]. The spectral analysis [2] provides the experimental evidence to support the claim that transformer-based architectures are incapable of interpreting heterogeneous data.

• Question 1: We added the discussion on computational costs in updated appendix A.5. The updated part is marked in blue color.

• Question 2: PLE depends on the distribution of the values in numerical columns. When scaling numerical values, the bin indices of numerical columns does not change. When b_{t-1} <= x <= b_{t}, xi_i is invariant of scaling.

• Question 3: We thank the reviewer for the good suggestion! We added the experiment results on different batch sizes in the updated manuscript. Please refer to Table 5, Table 6, Section 4.3 ablation study on batch size. We marked the updated part in blue color. We found that when decreasing batch size, the performance for the multiclass classification task decreases while that for the binary classification task nearly does not change.

[1] Shwartz-Ziv, Ravid, and Amitai Armon. "Tabular data: Deep learning is not all you need." Information Fusion 81 (2022): 84-90.

[2] Rahaman, Nasim, et al. "On the spectral bias of neural networks." International conference on machine learning. PMLR, 2019.

[3] Beyazit, Ege, et al. "An inductive bias for tabular deep learning." Advances in Neural Information Processing Systems 36 (2024).

[4] Gorishniy, Yury, Ivan Rubachev, and Artem Babenko. "On embeddings for numerical features in tabular deep learning." Advances in Neural Information Processing Systems 35 (2022): 24991-25004.

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

The authors thank the reviewer for comments. Here is the one-to-one response:

• Weakness 1. TAPAS is an approach for tabular question answering as opposed to tabular prediction. It directly flattens tables while our method proposes different ways to process different types of columns. Compared to existing methods, (1) we propose three drastically different encoding strategies for three different types of columns: categorical, numerical and binary columns; (2) we propose a corruption strategy designed for tabular data; (3) we propose a column-type aware positional encoding designed for our encoding strategies. This type of positional encoding encourages the permutation invariance of tables and ensures the sequence order of natural language sentences.

• Weakness 2: The deep learning baselines in our study are SubTab (2021) [1], Scarf (2022) [2] and SwitchTab (2024) [3]. We looked into the survey paper that the reviewer recommended. We find many baselines are not suitable for the task in our study. The survey paper covers prediction, data generation and table understanding. The task in our study is prediction. We do use the baselines mentioned in the prediction category: XGBoost, CatBoost and TransTab. Some other baselines mentioned in the survey paper are not up-to-date baselines. For example, TabNet [4] and TabTransformer [5] mentioned in the survey paper are earlier than most deep learning baselines we used. Can the reviewer elaborate on what specific baselines should the authors add for the comparison?

• Weakness 3: We added the computational cost comparison in the updated Appendix A.5. The updated part is marked in blue color.

• Question 1: The survey paper covers prediction, data generation and table understanding. Our model is used for the classification tasks, which falls into the prediction category. We use 76 datasets to examine the performance. Can the reviewer elaborate on what kind of experiments are needed to examine the performance of TDTransformer?

[1] Ucar, Talip, Ehsan Hajiramezanali, and Lindsay Edwards. "Subtab: Subsetting features of tabular data for self-supervised representation learning." Advances in Neural Information Processing Systems 34 (2021): 18853-18865.

[2] Bahri, Dara, et al. "Scarf: Self-supervised contrastive learning using random feature corruption." arXiv preprint arXiv:2106.15147 (2021).

[3] Wu, Jing, et al. "Switchtab: Switched autoencoders are effective tabular learners." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 14. 2024.

[4] Arik, Sercan Ö., and Tomas Pfister. "Tabnet: Attentive interpretable tabular learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 8. 2021.

[5] Huang, Xin, et al. "Tabtransformer: Tabular data modeling using contextual embeddings." arXiv preprint arXiv:2012.06678 (2020).

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

The authors thank the reviewer for the response. Can the reviewer elaborate on which aspects remain unclear or unconvincing?

The reviewer suggested to add baselines and experiments based on [1]. We tried to add baselines and experiments from the reference [1] that the reviewer gave, but we were not sure which baseline or experiment to add. We used SwitchTab (2024) as one of our baselines. Can the reviewer give more details on which baselines or experiments should the authors add to improve the quality of the work?

We tested one baseline SAINT (2021) that is included in [1]. The performance for the binary classification task is:

The performance for the multiclass classification task is:

[1] Fang, Xi, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Jane Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. "Large language models (LLMs) on tabular data: Prediction, generation, and understanding-a survey." (2024).

Dear Reviewer,

Thank you for the review of our submission. Following your suggestion, we have chosen one baseline SAINT [2] from the reference [1] you mentioned. Results and details are reported in the previous reply. We would like to kindly ask do you have feedback regarding the previous reply?

We would greatly value more specific details about which aspects of our work were unconvincing or unclear. If there are still concerns, could you provide additional insights into your concerns? For instance:

• If you think we need to do more experiments, could you provide more details on what type of experiments are needed to improve the quality of our work?
• If you think more baselines are needed, could you list the recommended baselines?

Your feedback would help us improve our work. We are happy to provide further clarifications or modifications based on your input.

Authors

[1] Fang, Xi, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Jane Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. "Large language models (LLMs) on tabular data: Prediction, generation, and understanding-a survey." (2024).

[2] Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C.B. and Goldstein, T., 2021. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342.

The authors thank the reviewer for feedback. Here is the one-to-one response:

• Weakness 1: Yes, compared to tree-based methods such as XGBoost [1], the tabular deep learning methods are computationally more expensive. We do consider the computational costs so we intentionally choose LM encoder instead of LLM encoder. The total number of trainable parameters of TDTransformer is merely 32.31 M whereas the number for large language models is much larger (e.g. Llama 7B has 7B parameters). The high computational cost of deep learning methods is an inherent drawback compared to traditional machine learning methods. However, the large language model has its unique strength compared to traditional machine learning methods: the ability of understanding semantics. This is particularly helpful in the tabular learning task when tables contain rich semantics. Traditional machine learning methods have difficulty encoding column names. Using our proposed column encodings and the power of the language model, the performance is pronouncedly higher than traditional machine learning methods as shown in Table 2 and Table 3.

• Weakness 2: TDTransformer uses three different types of column embedding to process categorical columns, numerical columns and binary columns. These three types of columns constitute all types of columns in relation tables. We focus on relation tables as indicated by Section 3 “Input data are relational tables that have a unique column given a column name”. For each type of column, we use drastically different encoding methods to enhance the capability of transformer models to process heterogeneous data.

• Question 1: Yes, we considered a more powerful LM encoder, but as the reviewer points out, the computational costs of deep learning methods are higher than traditional machine learning methods such as XGBoost [1]. We intentionally use LM encoders in lieu of LLM encoders. Our current model complexity is much lighter than Llama or Mistral. The total number of trainable parameters is just 32.31 M whereas Llama 7B [2] has 7B parameters and Mistral [3] has 7B parameters.

[1] Chen, Tianqi, and Carlos Guestrin. "Xgboost: A scalable tree boosting system." Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016.

[2] Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

[3] Jiang, Albert Q., et al. "Mistral 7B." arXiv preprint arXiv:2310.06825 (2023).

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

The authors thank the reviewer for comments. Here is the one-to-one response:

• Weakness 1: $z_i$ is obtained after feeding E in Eq.10 to the LM encoder as shown in Figure 1 (a) and extract i-th row. We explained the meaning of $z_i$ on page 5 of the original manuscript “$z_i$ is the hidden representation for $i$-th table row”. To clearly show their relation, we added a paragraph to illustrate the relationship between $z_i$ and $E$ in our updated manuscript marked on page 5 “After the contextualization in LM encoder…”. The updated part is marked in blue color.

The numerical column embedding depends on the distribution of scalars in the column, so in section 3.1, we mention column embedding. A table row i consists of all columns at i-th row, so we mention different columns in section 3.2.

The task is to make a prediction for each row as shown in Section 3 Task formulation. We train the model to learn semantically meaningful embedding $z_i$ to represent i-th row and make a prediction for that row. The i-th row contains cell values of multiple columns at this row. We treat different columns differently to solve the heterogeneous issue in tables. After getting the embedding, we want to combine all embeddings to obtain a sequence of embedding for each row, which is the combination process.

• Weakness 2: The fine-tuning process is in a supervised fashion as shown in Section 3 Task Formulation. The model is learned to maximize the probability of predicting the correct label as shown in Eq. 1. We added a paragraph for the fine-tuning process on page 6 "The model weight after the pre-training process is used as the initialized weight for the fine-tuning...".

• Weakness 3: Examples of tree-based methods include XGboost [1] and CatBoost [2]. The overview for the comparison between tree-based methods and deep learning methods can be found in [3-4]. We added specific examples for tree-baed methods on page 1 of the modified manuscript “However, they continue to lag tree-based methods such as XGBoost”.

• Weakness 4: Yes, we appreciate the typo pointed out by the reviewer.

• Question 1: Categorical columns are essentially columns with string datatype. Hence, TDTransformer is able to handle columns with string datatype. The way of handling categorical columns is reported in Section 3.1 column embedding.

• Question 2: We replace the missing values with the most common values in the column. We will add this detail in Section 4.1 Experimental details.

• Question 3: TDTransformer does the preprocessing (e.g. handling missing values and normalizing numerical values in the same column) at the same time. For the encoding process, TDTransformer converts multiple rows in a table to embeddings. The number of rows is equal to the batch size.

• Question 4: Different subsets are selected to examine the performance on different aspects. For example, TDTransformer proposes the embedding method for numerical columns. The embedding process of categorical columns is the same as traditional tabular deep learning methods using serialization and embedding. Hence, we select a subset of tables to examine the effectiveness of TDTransformer.

In addition to the performance on subsets, we reported the average performance over all 76 datasets.

[1] Chen, Tianqi, and Carlos Guestrin. "Xgboost: A scalable tree boosting system." Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016.

[2] Prokhorenkova, Liudmila, et al. "CatBoost: unbiased boosting with categorical features." Advances in neural information processing systems 31 (2018).

[3] Shwartz-Ziv, Ravid, and Amitai Armon. "Tabular data: Deep learning is not all you need." Information Fusion 81 (2022): 84-90.

[4] Grinsztajn, Léo, Edouard Oyallon, and Gaël Varoquaux. "Why do tree-based models still outperform deep learning on typical tabular data?." Advances in neural information processing systems 35 (2022): 507-520.

[5] Borisov, Vadim, et al. "Deep neural networks and tabular data: A survey." IEEE transactions on neural networks and learning systems (2022).

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair