TabM: Advancing tabular deep learning with parameter-efficient ensembling

In particular, it's unclear in how far the initial adapter and last layer are coupled. How does the performance change if the best performing last layer is used for all the initial adapters? I.e. are the initial layer and last layer co-adapted or do they work independently?

Perhaps, the best visual answer is the new illustration of the model in Figure 9.

First, we quickly remind that TabM is just an efficient parametrization of $k$ MLPs, so it can be explicitly materialized as $k$ separate MLPs. However, due to the almost complete weight sharing, it is possible to pack all $k$ MLPs in one model (which gives TabM). At the same time, mathematically, the forward passes of the $k$ implicit MLPs are still independent (while in practice, they are efficiently performed in parallel).

Now, regarding the question. The $i$-th adapter is strictly coupled with the $i$-th prediction head, because both of them belong only to the $i$-th ensemble member, and not to any other member. So randomly combining $i$-th adapter with $j$-th prediction head ($i \ne j$) is the same as swapping layers of different models, i.e. this is not expected to work "correctly" without any additional assumptions.

Are these all the outliers that are clipped to the limits of the figure?

Yes, this is correct. We clarified this in the caption in the new revision.

Table 2, MLP on the Maps dataset is missing units in duration.

This is now fixed, thanks! It is 10 minutes.

The title of section 5.3 should read "How does the performance of TabM depend on k?" ... Line 257: "requires a special care" should be "requires special care"

This is now also fixed, thanks for pointing to this!

Are number of layers for MLP and TabM tuned as hyper-parameters? What are typical values for each?

Yes, they are tuned, as specified in "Implementation details" in Appendix. The allowed values are from 1 to 6 for MLP and from 1 to 5 for TabM.

Why is k not included in the hyper-parameter search for TabM?

$k$ implicitly affects the effective batch size by cloning the input $k$ times. Tuning batch size and learning rate simultaneously is usually suboptimal, because the optimal values of these hyperparameters are coupled in a non-trivial way, which makes the job harder for TPE-like hyperparameter tuning algorithms. So we fix both $k$ and batch size, and tune only the learning rate.

How do you ensure diversity for the MLPs in the ensemble in Figure 6?

By using different random seeds, which affects the random initialization, training batch sequences, dropout activations, etc. In general, for deep ensembles, it is known that different initializations are already enough to obtain meaningfully diverse models.

How do you initialize the last layers in TabM to ensure diversity?

For TabM, the diversity comes only from the random initialization of the non-shared weights, including the prediction heads. There are no additional tricks aimed at increasing diversity. To clarify, our approach is in line prior work on parameter-efficient ensembles.

Is it possible that TabM_mini outperforms the real ensemble because of more diversity in the initialization?

In TabM-mini, the $k$ implicit ensemble members share almost all of their weights, so by construction, the initialization of the $k$ models in TabM is less diverse than the initialization of $k$ fully independent models.

Is it correct that when doing the full ensemble, adding an adapter into each model would lead to an equivalent ensemble?

Yes, this is correct.

I assume random initialization of weights for each member of the ensemble makes the adapters irrelevant.

Yes, our reasoning was that the adapters in the beginning randomly initialized with $\pm 1$ do not make a difference for plain MLPs.

We thank the reviewer for the extensive feedback.

Two quick comments before we start:

• There are many questions in the review, so our reply takes multiple posts.
• (minor) It seems that some of the weaknesses are closer to questions in spirit.

The technical novelty of the paper is somewhat small

Please, see the "Big picture" part of the global post. While our method is easy to implement (which we see as its strength), it has been missing for many years, and our main focus in this work is to bring the significant missing value to the field of tabular DL. And this still required a non-trivial exploration of technical ideas (see the details below the "Big picture" part of the global post).

The paper doesn't describe the relationship to dropout, even though dropout was original describes as an efficient ensemble approach. Given the simple nature of TabM_mini, there seem to be some obvious parallels between drop-out and TabM_mini that I think are worth discussing.

We should admit that we do not fully understand the question. A network trained with dropout can indeed be viewed as an efficient ensemble, known as "dropout ensemble", and for example is already mentioned in Section A2. However, there are many other ensembling methods, and it is unclear, why dropout should be highlighted separately, how this is connected to the simplicity of TabM-mini, and what is implied by the "obvious parallels". We are open for discussion here, but currently, we are uncertain how to interpret the request.

In particular, TabM_mini actually contains a dropout layer, and it would be interesting to evaluate how the two forms of ensembling complement each other.

To answer the question, we evaluate MLP and TabM-mini of the same depth 3 and width 512 with and without dropout. Plus, we include $\mathrm{MLP}^{\times 32}$ -- a traditional deep ensemble of 32 MLPs without dropout as an additional baseline. The table below shows the ranks and relative improvements over MLP with dropout on 18 datasets.

Conclusion: the deep ensembling and "mini" ensembling are significantly more impactful than dropout.

The comparison doesn't include TabPFN, an extremely strong baseline. McElfresh showed that even subsampling data to at most 3000 datapoints, TabPFN outperforms most other models.

The original submission already includes TabPFN, and with even larger number of subsampled objects (10K) than suggested (3K). However, it is included only in Appendix F, because:

• TabPFN is not competitive on our benchmark, which is expected, because we experiment on datasets orders of magnitude larger (up to millions of objects) than the target scope of TabPFN.
• TabPFN has too many limitations for our benchmark, which prevents a proper evaluation (not applicable to regression tasks, not applicable to more than 100 features, not applicable to large datasets).

Given the focus on improving MLP performance, a comparison with regularization cocktails and "Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data" might be relevant but not essential.

Both papers are already cited in the original submission, because indeed, they are tangentially relevant and can serve as an indirect inspiration, showing that improving MLPs is generally possible. However, our scope is specifically architectures (for example, this is communicated at the beginning of the abstract). So we do not compare with non-architectural techniques that are actively used in the suggested papers.

Dear reviewer, we will be glad to hear if our rebuttal addresses the concerns. We performed the dropout-related experiments and answered all other questions.

consider incorporating the datasets from Tabzilla

(TL;DR) The results on 15 datasets from TabZilla are in line with our paper:

(Details) After removing small datasets (<2K objects, as mentioned earlier), the datasets already used in our paper, duplicated datasets, and non-tabular datasets (e.g. board games), we are left with 15 out of 36 datasets. The above table is obtained using these 15 datasets.

In Table 2 (p.10), the last column of the MLP row lists "10." Could the authors confirm if it is measured in minutes, as there’s no unit mentioned?

Yes, this is 10 minutes, and is now fixed in the new revision, thanks!

Could the authors clarify how they define “detrimental” in this context and provide further explanation?

This part is fully inherited from "TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023", ICLR 2024. But in fact, it applies only to one dataset. As a quick test, we evaluated MLP on this one dataset with the normalization turned on, and indeed observed performance degradation from 0.819 to 0.806 (accuracy), which is significant. Again, this part is inherited, but from our quick test, it seems to be reasonable.

Could the authors provide more detailed diversity metrics for the predictions of the 32 base models?

TL;DR. On average over 18 datasets, the diversity of $k = 32$ predictions for $\mathrm{TabM_{mini}}$ is $\times 1.88$ higher than for the standard ensemble of $k$ MLPs.

Technical details. We define the diversity of $k$ predictions as the mean deviation of the individual predictions from their collective mean prediction (i.e. prediction of an ensemble). For regressions, it is simply the standard deviation of predictions. For classification, it is the mean KL divergence between individual predictions and the collective mean prediction:

• $y^k_i$ is the prediction of the $k$-th head/model for the $i$-th sample
• $\bar{y_i}$ is the prediction of ensemble (deep or TabM)
• Regressions: $\frac{1}{KN}\sum_{i}^N \sum_{k}^K (y^k_i - \bar{y_i})^2$
• Classification: $\frac{1}{KN}\sum_{i}^N \sum_{k}^K KL(y^k_i || \bar{y_i})$

Would an ensemble of 32 independently and undertrained MLPs yield similarly strong performance?

By definition, TabM is an efficient parametrization of 32 MLPs, so in theory, it should be possible to train them separately in a way that makes them no worse than TabM. But how to do that may be a non-trivial question. Below, we evaluate a naive way to "undertrain" MLPs on 18 datasets from our benchmark. To obtain an ensemble of 32 undertrained MLPs, we train each MLP only for a fraction of the optimal number of epochs (defined by the full run with early stopping). The results are provided below and show that the naive undertraining leads to worse performance compared to the standard ensemble of fully trained MLPs. Meanwhile, $\mathrm{TabM_{mini}}$ remains superior. Notation: $\mathrm{MLP[0.25]}$ denotes that MLP is trained for a quarter of the optimal number of epochs.

We thank the reviewer for the detailed feedback. The strengths mentioned in the review resonate a lot with what we try to achieve with our project. So we did our best to address the remaining concerns below.

more detailed comparison with other related approaches for tabular data, such as Trompt

Trompt is already cited (Section 2) and evaluated in our submission in Appendix F. However, it has multiple problems, so it is presented only in Appendix F and is not included in the main text:

• There is no official implementation of Trompt.
• The performance is not competitive, especially given the following point.
• Trompt is extremely slow to train. Hyperparameter tuning is impossible beyond small datasets.

Comparisons with standard ensemble methods are insufficient. Incorporating results from methods like model soup could help position TabM's performance.

• The standard ensemble is already evaluated as $\text{MLP}^{\times 32}$ in Section 5.
• Section A2 already discusses why we start exactly from BatchEnsemble and not from any other ensembling method. In short, BatchEnsemble is the most architecture-agnostic, practical, and easy-to-use option among all available methods, so it is a reasonable choice for our project. Plus, full-fledged comparisons of ensemble methods across many domains are already performed in dedicated studies.
• The suggested "model soup" is not an ensemble method, it is a weight averaging method: "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time". It means that it produces one model making one prediction without additional inference costs. By contrast, TabM imitates an ensemble of models producing multiple predictions, and it has additional costs compared to plain single models.

Dataset scope is somewhat limited

We share the reviewer's enthusiasm for better benchmarks. However, it seems that our benchmark takes a step forward compared to prior studies. Our perspective is as follows:

• For example, let's consider large datasets. Our study already covers more datasets with >1M objects than 15 out of the 19 papers on GBDT and tabular DL cited in the "Related work" section. The paper on LightGBM is the only paper that uses more (5) than our paper (3). The papers on XGBoost, AutoInt, and NODE use 3 large datasets, and all other papers use less, with many papers using zero large datasets.
• Let's consider datasets with a large number of features. Our study already covers more datasets with >100 features than 19 out of 19 papers on GBDT and tabular DL cited in the "Related work" section.
• Furthermore, we are one of the first studies integrating the modern challenging TabReD benchmark [1], with more objects and features than in many prior benchmarks, and with real-world time-based splits. Please, see the details in the "Datasets" paragraph of Section 3.1, and the corresponding results on the right side of Figure 4.
• Limitations: our study does not cover the niches of small datasets (intuitively, <2K objects) and datasets with a high n_features / n_objects ratio. These niches can be different in terms of typical baselines, hyperparameter grids, and model ranking, and can require a different experiment setup (e.g. our train-val-test split and our hyperparameter tuning protocol may need changes for small datasets).

particularly for classification tasks

With the benchmarks from prior work that we used, regression tasks indeed happened to be more presented. However, given the below evaluation on TabZilla, it seems that our results generalize well on unseen classification tasks.

(TabZilla is covered in the next post)

We thank the reviewer for the comments, and we are looking forward to the discussion.

Contribution: 2 ... The proposed method is not much novel ... this method is already very simple and not much novel ...

Please see the “Big picture” part of our global post. While our method is easy to implement (which we see as its strength), it has been missing for many years, and our main focus in this work is to bring the significant missing value to the field of tabular DL. And this still required a non-trivial exploration of technical ideas (see the details below the "Big picture" part of the global post).

there is no discussion why TabM helps with reducing overfitting ... it seems like a general benefits of ensemble models.

Yes, in Section 5, we show that the usual mechanics of ensembles work as intended and are highly effective for TabM. We adjusted the wording in the abstract to better align readers' expectations with the content.

Baselines missing ... compare it to other naive ensemble method

This comparison is already presented in the paper. Namely, the figure in Section 5.1 contains $MLP^{\times 32}$ -- the naive ensemble method.

For instance, how much speed up it provides compared to normal ensemble

If we compare the training time between TabM with $k = 32$ and single MLP multiplied by $32$, then we get $\times 13.7$ speed-up on average across datasets. As for the inference throughput on CPU, the speed-up is $\times 14$. This is a naive approach, since, in theory, the training of MLPs can be performed in parallel. However, this approach gives the right intuition that TabM is more efficient than naive ensembles, and the overhead is less than $\times k$ compared to single MLP.

in Table 2 ... only three baselines, moreover, only TabMmini has been evaluated

As explained in Section 4.4, the goal of the experiments reported in Table 2 is to test the ability of various model families to scale to large datasets, which does not require evaluating all models.

Datasets ... how about high-dimensional datasets and large datasets ...

We share the reviewer's enthusiasm for better benchmarks. However, it seems that our benchmark takes a step forward compared to prior studies. Our perspective is as follows:

• Regarding large datasets, our study already covers more datasets with >1M objects than 15 out of the 19 papers on GBDT and tabular DL cited in the "Related work" section. The paper on LightGBM is the only paper that uses more (five) than our paper (three). The papers on XGBoost, AutoInt, and NODE use three large datasets, and all other papers use less, with many papers using zero large datasets.
• Regarding high-dimensional datasets, our study already covers more datasets with >100 features than 19 out of the 19 papers on GBDT and tabular DL cited in the "Related work" section.
• Furthermore, we are one of the first studies integrating the modern challenging TabReD benchmark [1], with more objects and features than in many prior benchmarks, and with real-world time-based splits. Please, see the details in the "Datasets" paragraph of Section 3.1, and the corresponding results on the right side of Figure 4.
• Limitations: our study does not cover the niches of small datasets (intuitively, <2K objects) and datasets with a high n_features / n_objects ratio. These niches can be different in terms of typical baselines, hyperparameter grids, and model ranking, and can require a different experiment setup (e.g. our train-val-test split and our hyperparameter tuning protocol may need changes for small datasets).

(TabZilla is discussed in the next post)

I suggest the authors to use TabZilla hard benchmark

(TL;DR) The results on 15 datasets from TabZilla are in line with our paper:

(Details) After removing small datasets (<2K objects, as mentioned earlier), the datasets already used in our paper, duplicated datasets, and non-tabular datasets (e.g. board games), we are left with 15 out of 36 datasets. The above table is obtained using these 15 datasets.

Dear reviewer, we are looking forward to discussion. In the rebuttal, we:

• Performed the evaluation on the Tabzilla benchmark, as requested.
• Commented on all other concerns, and hopefully addressed them.

Thank the authors for the thoughtful response. I am willing to increase my score to 6.

We thank the reviewer for the positive feedback.

the captions of Figure 2 and Figure 3 are long and blend with the core manuscript text

We note two things:

• Perhaps, the reviewer will be interested in the new illustration of the models in Figure 9. We are thinking of using Figure 9 instead of both Figure 1 and Figure 3. We are open to further feedback.
• Our approach is to make figures somewhat self-sufficient, without constantly referencing the main text, which requires putting a bit of extra information to captions.

Table 2, for the MLP (Maps Routing dataset) the time unit is missing. I am guessing it is 10 minutes.

This is fixed in the new revision, thanks! Yes, it is 10 minutes.

Dear reviewer, is there anything preventing the verdict from being more positive? We will be glad to receive further feedback to improve the paper.