ExcelFormer: Making Neural Network Excel in Small Tabular Data Prediction

Thank you for helpful comments. We trust that our responses will effectively address all your concerns.

Q1: Many orthogonal changes are proposed at once? Need more ablation studyies.

A1: Thank you for your concerns. We would like to clarify it from 2 aspects:

(1) More detailed studies. We have included more ablation studies and addictive studies to inspect the performances of proposed components.

1. Addictive study: on a vanilla Transformer (Tra), we show that all of our proposed modules, SPA, IAI, and GLU projection/embedding (GLU projection and embedding share the idea, and both of them process features through the GLU) make positive contributions to tabular data prediction tasks, and the contributions are cumulative.

2. Compare our data augmentation with Mixup and CutMix. Ablation study in Figure 4 show CutMix and Mixup perform worse than our proposed data augmentation. To analyze the difference of cutmix and Feat-Mix carefully, in Table 6, we present the results of cutmix and Feat-Mix on several datasets. Theoretically, the difference between cutmix and feat-mix lies in the use of feature importance. Empirically, we find that:

• In general, Feat-Mix outperforms Cutmix due to its ability to provide more accurate labels by considering feature importance.

• When noise exists in the data, Feat-Mix is less affected by the noise while cutmix is more heavily impacted. Since tabular datasets have been shown to contain many uninformative features[1], we believe Feat-Mix is a superior choice.

The detailed results:

3. Compare Hid-Mix, Feat-Mix with Feature resample (an augmentation for tabular data) on MLP:

The results are presented below. We observe that Hid-Mix outperforms Feature Resample and all these approach dominate different parts of datasets.

4. An ablation study is shown in Figure 4 that the performances of Excelformer drops when removing one of those components.

(2) Why propose those components?

1. We would like to point out a specific issue of the tabular data prediction domain: Tabular datasets are highly diverse and heterogeneous. Unlike other fields (such as computer vision) where a single model can dominate most datasets, different previous tabular data prediction models only excel on a part of datasets (see Table 2: excluding Excelformer, the best models vary in different groups). Excelformer is the first work that realizes such dominance.

2. Given the diversity of datasets in this field and the no-free-lunch principle, finding a single component that is universally applicable to all types of datasets is challenging (previous works have faltered in this aspect). In fact, the orthogonality of methods is indeed the cornerstone of our contribution. When addressing the diversity of tabular datasets and tasks, the proposal and integration of orthogonal methods augment the robustness and efficacy of Excelformer. Then, we achieve:

• Using default hyperparameters (it is very convenient in application!), ExcelFormer outperforms previous hyperparameter-tuned models;

• ExcelFormer achieves robust performance across various types of datasets / tasks (see Table 2)

We argue that AI development aims to facilitate practical applications. The combination of versatility and the convenience of not requiring hyperparameter tuning positions Excelformer as an excellent practical tool for tabular data prediction tasks, especially for users with limited computer science proficiency or computational resources. Please note that no previous work attains performances as dominating as Excelformer, making this a key contribution.

References:

[1] Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS, 2022.

Q2: Novelty?

A2: Thank you for this concern. I believe the evaluation of novelty is quite objective. I suggest considering the changes an approach brings as a reference for novelty creation. Let's examine what changes we made by introducing Excelformer:

• BEFORE EXCELFORMER:

  • The overall performance of deep learning approaches is inferior to GBDTs.
  • Different previous models excels in specific types of datasets / tasks (see Table 2). It is challenging to determine a "good" model before conducting an evaluation.
  • Fine-tuning hyperparameters is crucial for attaining state-of-the-art results across all previous models. This poses challenges for users with limited proficiency in computer science or access to computational resources.

• AFTER EXCELFORMER IS PROPOSED:

  • Excelformer (a DL approach) beats GBDTs and various DL competitors;
  • Excelformer outperforms previous models on various types of datasets / tasks (see Table 2). It is easier for users to choose a model (Excelformer) now.
  • Although fine-tuning hyperparameters is beneficial for Excelformer, using its default hyperparameters can achieve top results.

• To achieve this, THE COST WE INCUR IS MINIMAL:

  • Obviously, the computational efficiency of SPA is the same as vanilla self-attention module.
  • Interaction Attenuated Initialization is almost computational free.
  • As analyzed in Sec. 2.2, the computational complexity of GLU projection is equivalent to that of vanilla FFN.
  • Feat-Mix and Hid-Mix are nearly computational free.

We will highlight the contribution of Excelformer in the final version.

Considering the changes that Excelformer has brought to the field of tabular data prediction, we would appreciate it if you could reconsider your evaluation of novelty. Thank you!

Q3: In my opinion, the storyline around rotation invariance should be extended with specific analysis/experiments/results.

A3: Thank you for your constructive suggestion! In the initial version, we conducted an analysis of the rotation invariance property, i.e. the subfig (a) in Figure 5. Following your suggestion, we have taken a step further by providing an additive study and an ablation study to analyze the effects of SPA and IAI on breaking the rotation invariance of deep learning models.

• Additive study setting: We use SPA and IAI on the backbone of FTT [1]

• Ablation study setting: We exclude SPA and IAI from the architecture of Excelformer

We find that:

• With SPA and IAI, our Excelformer shows better non-rotationally invariance, compared with previous works.

• On the FTT backbone, the addition of SPA and IAI makes positive contributions to break the rotational invariance property.

• On the Excelformer architecture, removing SPA and IAI weakens Excelformer's rotational invariance property.

• The effects of adding or removing SPA and IAI are cumulative.

Q4: Sharing the codes?

A4: We would like to make all codes publicly available, including detailed implementations, model training and experiment examples, along with user-friendly APIs. Given the substantial volume of implementation and experiment codes, we chose not to include them in the initial submission to allow more time for the improved organization and presentation of this project. The code is now included in the supplementary materials, and we will continue to refine the project to offer more comprehensive documentation and unified APIs.

Q6: How exactly is the mutual information computed for the continuous features and for regression labels?

A6: In the implementation, we utilize the scikit-learn package to calculate mutual information. Specifically, for classification tasks, we employ the "feature_selection.mutual_info_classif" function, and for regression tasks, we use the "feature_selection.mutual_info_regression" function. Thank you for your reminder! We have depitcted the implementation in Appendix F.

Q7: Paper polishment.

A7: Thank you for your careful suggestions. We will do careful proof-reading for the final version.

Thanks again for your time and feedback. We've incorporated your suggestions into the manuscript, which is very helpful to improve the quality of our paper. We kindly request you to reconsider the ranking, considering the added ablation studies, additive studies and the practical benefits of ExcelFormer.

Sincerely,

Authors of paper 4888

References:

[1] Revisiting deep learning models for tabular data. NeurIPS. 2021.

Q7: It would be interesting to see an ablation of the proposed augmentation methods against maybe cutmix or cutout to observe the overall improvement.

A7: Thank you for your suggestion. We compare the our data augmentation approaches with cutmix in two aspects:

(1) We have added results of Excelformer architecture + cutmix into Figure 4, which shows that cutmix performs better than vanilla mixup but is still not as well as our proposed data augmentation approaches. To further analyze the difference between cutmix and Feat-Mix carefully, in Table 6 of revised version, we present the results of cutmix and Feat-Mix on several datasets. Theoretically, the difference between cutmix and feat-mix lies in the use of feature importance. Empirically, we find that:

• In general, Feat-Mix outperforms Cutmix due to its ability to provide more accurate labels by considering feature importance.

• When noise exists in the data, Feat-Mix is less affected by the noise while cutmix is more heavily impacted. This is clearly due to our use of feature importance when synthesizing new samples. Since tabular datasets have been shown to contain many uninformative features[1], we believe Feat-Mix is a superior choice.

The detailed results are as follow:

Q8: An interesting addition would be to include the plain architecture of ExcelFormer in the investigation, then with every suggestion included one at a time (SEA, IAI), then both. This would show how the architecture gets more non-rotationally invariant as the different components are added compared to the beginning. Comparing against FTT is interesting, but it does not separate the impact of the overall differences in the architecture vs (SEA, IAI).

A8: Thank you for your constructive suggestion. We conduct an additive study and an ablation study (see Fig. 5 of the revised version) to inspect the benefits introduced by SPA and IAI on breaking the rotational invariance property of DL models. We find that:

• With SPA and IAI, our Excelformer shows better non-rotationally invariance, compared with previous works.

• On the FTT backbone, the addition of SPA and IAI makes positive contributions to break the rotational invariance property.

• On the Excelformer architecture, removing SPA and IAI weakens Excelformer's non-rotational invariance property.

• The effects of adding or removing SPA and IAI are cumulative.

Thank you once again for your valuable time and constructive feedback. We have taken your feedback into careful consideration and have made revisions to the manuscript accordingly. We kindly request you to reconsider the ranking, considering the improvements (especially the detailed ablation and additive studies) made and the significant practical value that ExcelFormer offers.

Sincerely,

Authors of paper 4888

References:

[1] Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS, 2022.

Thank you for recognizing the good structure, comprehensive experiments, and the interesting aspects of our components. We trust that our responses will effectively address all your concerns.

Q1:

(i) Release codes?

(ii) Why previous works are debunked to be not a real sota?

(ii) Why propose several components?

(iii) Why XGboost outperforms Catboost?

A1: Thank you for your thoughtful considerations. We would like to address the aforementioned questions together as they are closely related.

(i) Certainly, we intend to make all codes publicly available, including detailed implementations, model training and experiment examples, along with some user-friendly APIs. We are confident that Excelformer can serve as a convenient and versatile solution for tabular data prediction. Given the substantial volume of implementation and experiment codes, we chose not to include them in the initial submission to allow more time for the improved organization and presentation of this project. The code is now included in the supplementary materials, and we will continue to refine the project to offer more comprehensive documentation and unified APIs.

(ii) Why previous SOTA be debunked? Without intending any offense, we would like to explain why you find that recent approaches (e.g., TabNet, Node, SAINT) initially claimed state-of-the-art performance but were later "debunked". Actually, tabular datasets are highly diverse and heterogeneous. Unlike other fields (such as computer vision) where a single model can dominate most datasets, different previous tabular data prediction models only excel on a part of datasets (see Table 2: excluding Excelformer, the best models vary in different groups). In this paper, Excelformer tries to realize such dominance. Previous works ignored such dataset diversity, and most of them (e.g., TabNet, Node, SAINT) only validated their claims on limited datasets. Similarly, the conclusions in the paper titled Tabular data: Deep learning is not all you need may also not be entirely convincing, as it relied on only 11 datasets. Hence, in our paper, we extensively utilized 130+ diverse datasets, and also presented results categorized by characteristics in Table 2. See Table 2, one may observe that different previous methods exhibit preferences for distinct datasets, but Excelformer obtains top performances in all the groups.

(iii) Why propose several components? Given the diversity of datasets in this field and the no-free-lunch principle, it is challenging to find a component widely applicable to all the dataset types (under default hyper-parameter setting). Following your suggestion, we compare the Excelformer (without data augmentation) against previous models, and found Excelformer (without data augmentation) achieves the best performance (see Q4 & A4). However, as an application-oriented research, we prioritize both model performance and user convenience. Thus, in this paper, we introduce those useful components, and we find they can make Excelformer:

• exempt from hyper-parameter tuning (using default hyperparameters is very convenient in application!)

• capable of robust performance across various types of datasets and tasks (see Table 2)

We believe that the purpose of AI research is to facilitate practical applications. The combination of versatility and the convenience of not requiring hyperparameter tuning positions Excelformer as an excellent practical tool on tabular data prediction tasks, especially for users with limited computer science proficiency or computational resources.

(iv) Why XGBoost outperforms CatBoost?

• Firstly, in our experiments, XGBoost does not consistently outperform CatBoost. See Table 3, under default parameters, CatBoost outperforms XGBoost. Additionally, see Table 2, CatBoost performs better than XGBoost in classification and the case of #. Features $\ge$ 16. Besides, CatBoost typically outperforms XGboost in multi-class classification (see Table 7 of the revised version). Such distinction is due to the diversity of datasets and tasks.

• A key contribution of CatBoost is reducing the workload for users in terms of feature engineering, visualization, and hyper-parameter tuning. This enables CatBoost to outperform XGBoost when using default parameters. However, after conducting extensive hyperparameter search for GBDTs (see Tables 6 and 7 - we provide more estimator candidates (4096) than previous works), we find that XGboost can outperform CatBoost on over 50% of datasets.

In summary, Excelformer is an application-oriented research. It achieves robust performance across diverse datasets using default hyperparameters. The proposed components work together and make Excelformer superior on the diversity of tabular data with almost no added cost.

Q5: How to compute mutual information?

A5: In the implementation, we utilize the scikit-learn package to calculate mutual information. Specifically, for classification tasks, we employ the "feature_selection.mutual_info_classif" function, and for regression tasks, we use the "feature_selection.mutual_info_regression" function. Thank you for your reminder! We have depitcted the implementation in Appendix F.

Q6: You compare ablated variants to the fully tuned baseline, are the ablated variations also tuned?

A6: Yes. For fair comparison, the ablated variations are tuned under the same setting.

Q7: How long were the models trained for? Was early stopping used during training? How the number of steps compare across deep models?

A7: We implemented early stopping with a patience of 32, and the maximum epoch for Excelformer was set to 500. To ensure a comprehensive tuning of XGBoost and CatBoost for optimal performance, we conducted 500 tuning iterations and increased the number of estimators to 4096 (most previous works used less than 2000). Excelformer underwent fine-tuning for 50 iterations across all datasets. For other deep learning approaches, on large-scale datasets, we followed the settings outlined in [1] or [2]. On small-scale datasets, all deep learning approaches were tuned for 50 iterations, as we found this to be sufficient for small-scale datasets and is fair among DL approaches. We did not set a time limit because fine-tuning on small-scale datasets does not consume an excessive amount of time. We have included these information in the revised version, thank you!

Q8: How ranks are computed?

A8: We performed 5 runs with different random seeds and calculated the average results for each dataset. Additionally, we computed the overall rank across datasets for comparison. Average rank is given to tied values. These details were included in Appendix F of the revised version. Thank you!

Q9: In the figure 3 hid-mix is called hidden-mix (only in the figure and nowhere else)

A9: Thank you for bringing this to our attention. We have revised Figure 3 in the updated version.

Thank you again for your time and your constructive feedback, which we've carefully considered and incorporated into the revised manuscript. We kindly request a reconsideration of the ranking, taking into account the improvements made and the substantial practical value of ExcelFormer.

Sincerely,

Authors of paper 4888

References:

[1]Revisiting deep learning models for tabular data. NeurIPS, 2021.

[2]Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS, 2022.

Q3: The table with various datasets aggregations looks redundant...Not sure why grouping by classification vs regression and the number of continuous/categorical features ...

A3: Thank you for your concern. We present performances on these subgroups separately because tabular datasets exhibit significant diversity, and thus different previous models only excel on specific portions of them (see Table 2: excluding Excelformer, the best models vary in different groups). This contrasts with domains like computer vision, where a single model can dominate most datasets. Excelformer aims to achieve a similar impact in tabular data prediction, but no previous works achieved this. We have made some modifications and try to clarify this point in the introduction.

This diversity sparks ongoing debates within the community about the state-of-the-art (SOTA) model. Users often find model selection challenging, and reviewers J7k8 also raises questions about the inconsistency in previous works being initially stated as SOTA and later "debunked". Similar to drug studies analyzing various characteristics (such as race, age, pregnancy) to determine applicable scenarios, our Table 2 experiments aim to investigate the universality of compared models. The proved universal applicability is a key advantage of Excelformer, not achieved by previous works, including GBDTs (as evident in Table 2, where GBDTs perform worse than FTT when the feature count is lower than 16). In essence, we argue that AI development aims to facilitate practical applications: the amalgamation of versatility and the convenience of not requiring hyperparameter tuning positions Excelformer as an excellent practical tool for tabular data prediction tasks, and Table 2 is helpful to users (especially those with limited computer science proficiency or computational resources) in model selection.

Honestly, we are open to discussions and welcome further exploration of these considerations.

Q4: Limited ablations and comparisons to baselines. Comparisons for augmentations and ablations for SPA and IAI should be presented.

A4: Thank you for your thoughtful question. We would like to clarify this points from two aspects: the selection of ablation study datasets; presenting more ablation results.

• The selection of tabular datasets. While some literature claims that GBDTs "dominate" tabular data prediction tasks, it's essential to clarify that this is not an absolute truth. See Table 2, TabPFN often outperforms GBDTs in classification on small-scale datasets, and GBDTs may perform worse than other DL models when the feature count is lower than 16. Thus, surpassing GBDTs is not the sole objective in building Excelformer; beyond defeating GBDTs, we also aim to safeguard the advantageous niche of DL models (we have made some changes to clarify this point in the introduction section; thank you for bringing it to our attention). To simulate real-world applications, our ablation studies are based on all datasets (including those GBDT-best datasets).

• More ablation study results.

(I) Key Structure Components

1. Following your suggestion, we conduct additive study to examinate the performances of the key structure components on a vanilla Transformer (Tra). We find that all the components make positive contributions to tabular data prediction tasks, and the contributions are cumulative.

2. Additionally, we perform an additive study and an ablation study (refer to Fig. 5) to examine the advantages introduced by SPA and IAI in disrupting the rotational invariance property of DL models. We find:

• On the FTT backbone, the addition of SPA and IAI individually to some extent breaks the rotational invariance.
• On the Excelformer architecture, removing SPA and IAI weakens Excelformer's rotational invariance property.
• The effects of adding or removing SPA and IAI are cumulative.

(II) Compare Hid-Mix, Feat-Mix, and Resample on MLP:

The results are presented below (in Table 5 of the revised version). We observe that the performances of feature resampling fall between Feat-Mix and Hid-Mix. We notice that each approach excels on different datasets. We will provide more discussions on this aspect in the final version.

All the additional results have been added into the revised version. Thank you for your constructive suggestion!

Q2: Results on multiple challenging-for-DL datasets where ExcelFormer significantly outperforms the DL competitors should be demonstrated.

A2: Thank you for your insightful suggestion. We agree that the tabular datasets are diverse, making parts of them specifically challenging for DL (i.e., GBDTs achieve relatively better performances while DL models obtain lower performances). In Table 2, we inspected the model performances on diverse types of tabular datasets / tasks. Additionally, in response to your suggestion, we introduced a subset known as "GBDT-best" datasets. The datasets in the "GBDT-best" group are selected based on: ignoring the results of Excelformer, XGBoost or CatBoost achieves the best performance. These results are included in the revised version (Table 2) with detailed analysis, and are also presented below.

Though these datasets pose challenges for deep learning approaches, we observed a reversal in performance with the changes introduced in ExcelFormer:

• Excelformer still outperforms GBDTs on "GBDT-best" classification datasets.

• Excelformer surpasses all DL competitors and achieves competitive performances with CatBoost. Importantly, this doesn't imply that Excelformer lags behind XGboost in regression, since these datasets are GBDT-friendly. In fact, Excelformer outperforms GBDTs on regression tasks (please refer to Q1 & A1).

• On those datasets initially "won" by GBDTs, Excelformer successfully outperforms GBDTs on 11 out of 15 GBDT-best classification datasets and outperforms GBDTs on 8 out of 22 GBDT-best regression datasets.

Here we present the results on GBDT-best datasets (top 3, including GBDTs, are marked in bold):

(1) Binclass Classification Tasks on GBDT-best datasets, measured by: Rank {$\downarrow$} (Ave Normalized AUC {$\uparrow$})

(2) Regression Tasks on GBDT-best datasets, measured by: Rank {$\downarrow$} (Ave Normalized nRMSE {$\uparrow$})

Thank you again for your constructive suggestion, which helps further validate the contributions of Excelformer. All the results with discussions are included in the revised version.

Thank you for acknowledging the clarity and significance of our work in this field and expressing your appreciation for our paper. We trust that our responses will effectively address all your concerns.

Q1: Using reduced rank as the main and only metric of model performance.

A1: Thank you for your suggestions, and we acknowledge the limitations of mean-rank. We utilized the performance rank following previous works (e.g., [1]). Additionally, we compute the average normalized scores (introduced in [2]) to aggregate the model performances across datasets. The results are presented in Table 2, Table 7, Table 8 in the revised version. The average normalized scores linearly scale the model's performance, providing insights into the performance differences among models beyond just ranking. The definitions are presented in Appendix F. Interestingly, we observe that average normalized score and performance rank tend to offer consistent evaluations of the models. This consistency may be attributed to the substantial volume of datasets we employed. Both performance rank and average normalized scores consistently demonstrate that our model outperforms previous works, on various datasets.

The detailed results are summarized below:

(1) The performances on 96 small-scale datasets:

• Binary-class Classification (51%; metric: Rank{$\downarrow$} (Average Normalized AUC {$\uparrow$}); best are in bold):

• Regression (49%; metric: Rank{$\downarrow$} (Average Normalized nRMSE {$\uparrow$}); best are in bold):

(2) On 21 larger-scale datasets, we compute the normalized average scores of ACC, AUC, and nRMSE ($\uparrow$) on multi-class classification datasets, binary-class classification datasets, and regression datasets, respectively. It is evident that Excelformer performs best.

(3) Following the suggestions of the reviewer J7k8, we also provide the results on 16 additional multi-class classification datasets (refer to Table 7 in the revised version), and both performance rank and average normalized ACC indicate that Excelformer performs best:

References:

[1]XTab: Cross-table Pretraining for Tabular Transformers. ICML, 2023

[2]Learning hyperparameter optimization initializations. DSAA, 2015