Better by default: Strong pre-tuned MLPs and boosted trees on tabular data

We thank the reviewer for the constructive feedback.

Major weaknesses:

• We meant to write “is competitive with GBDTs in terms of benchmark scores”, we will update this sentence. Of course, GBDTs are faster on most datasets with our settings. But note that lower is better in terms of benchmark scores in Figure 2 and RealMLP is better than GBDTs on all benchmarks in terms of scores in Figure 2.
• The simple “TabPFN+subsampling” version from [1] is outperformed by TabR and FT-Transformer in the recent large benchmark in https://arxiv.org/pdf/2407.00956, both of which are included in our Figure 3 and partially in Figure 2. Regarding versions of TabPFN that scale to large datasets, many recent scaling works are concurrent works. There is TuneTables from February, but still it (1) will be substantially slower than 1 second when scaled to larger datasets, and (2) is limited to classification. TabPFN-based results will probably be outdated very soon anyways since TabPFNV2 is supposed to be out soon.
• Figure 3 contains no confidence intervals because we used the codebase of Grinsztajn et al. to compute results and it does not provide confidence intervals with the same meaning as the confidence intervals in Figure 2 (in general, producing such confidence intervals for normalized benchmark scores as done in Figure 3 would be statistically questionable, as the normalization of the metric depends on the benchmark results itself.)
• Your last claim is incorrect: RealMLP outperforms FT-Transformer in terms of speed and benchmark scores for regression on the Grinsztajn et al. benchmark (and performs slightly better for classification as well).

• Regarding the meta-train benchmark: The benchmark consists of datasets that were available to us when we started experimenting. While it is not established, we cannot change it post-hoc since we already used it to design our methods.
• Regarding the meta-test benchmark: The benchmark consists of the AutoML benchmark and the CTR23 benchmarks, which are well-curated tabular benchmarks. The AutoML benchmark is well-established in AutoML, and [1, 2] also use it for their dataset collection. The contained datasets span a large range of sizes and dimensionalities, allowing us to test meta-generalization of tuned defaults. We removed some datasets according to well-defined criteria in Appendix C.3.2, mainly datasets that are already included in the meta-train benchmark or are too small.
• The benchmark suite of [1] is interesting but was released concurrently to when we started experimenting with the meta-test benchmark, and is mainly covering classification. As far as we can see, [1] doesn’t provide clear criteria by which they have selected their datasets, and their TabZilla subset has selection criteria prefering datasets that are hard for GBDTs, which would be unfairly favorable to neural networks such as our RealMLP.
• [2] contains many datasets but mostly small ones, which we do not design our methods for, and which result in noisier evaluation due to small test sets. They also use some datasets from the AutoML benchmark, which we already use.
• We use the Grinsztajn et al. benchmark datasets separately in Figure 3, to have a common benchmark where many baselines can run. We chose not to use these datasets for the meta-test benchmark since there are less of them (less than half of what we have now after excluding the meta-train datasets), and they are more limited in terms of number of features etc.

3. We do report alternative aggregation metrics in Figure B.7 – B.15 and reference some of them from Section 5.2. We also report normalized accuracies on the Grinsztajn et al. (2022) benchmark (Figure 3), to conform with the original benchmark. We agree that all aggregation strategies have some upsides and downsides.

Minor weaknesses:

• We increased the linewidth of the error bars, see the rebuttal PDF.

Questions:

• See above.
• See above, we do report multiple aggregation metrics.
• Because the meta-test benchmark is so expensive to run, see the global response. We already have the Grinsztajn et al benchmark for this purpose. Another issue is the lack of easily usable interfaces for most other published deep learning methods, including details such as how to pass a separate validation set, how to change the early stopping metric, how to pass categorical data, how to select the correct GPU device, etc. In contrast, we make scikit-learn interfaces available that allow configuring all of this easily.

We thank the reviewer for the constructive feedback.

Weaknesses:

• Thank you for the suggestion. We have something like this in Figures B.12-B.15, D.2, D.6, D.7, or do you have a specific suggestion for an overview plot?
• Random search 1) is used in the Grinsztajn et al benchmark, 2) is more convenient to run since all steps can be run in parallel and resource constraints (RAM usage) can be adjusted per parameter configuration, 3) allows to change the optimization metric post-hoc as we do for AUROC in Figure B.4 since the sampled configurations do not depend on the optimization metric. Random search has also been used in the benchmark of McElfresh et al. (2023), for example. We have a comparison to TPE (in hyperopt) for GBDTs in Figure B.6 and the differences are rather small.
• Thank you for the comment, we will try to improve this in the next version.

Questions:

• Thanks for the question, we also looked at this. It seems that the results are about halfway between Best-TD and Best-HPO (whose difference is quite small already). We were not sure if including this would complicate the message too much.
• See above.
• Not a lot, but a reason could be that when the MLP continues to improve slowly but the last few validation accuracies are identical, the last one is still more promising for the test set.
• To make them look nicer and more memorable in paper/code. (We checked that it doesn’t significantly hurt the performance.)

We thank the reviewer for the constructive feedback.

Weaknesses:

1. What is the source for your claim? Which AutoML tool would these be in? Are you referring to some of the color-coded improvements or only to the gray ones? There are some non-novel components in Figure 1 (c) since we wanted to start from a “vanilla MLP”, but as we color-coded in the figure, there are many new and unusual components. We also tried out the two NNs in AutoGluon and their default settings performed worse than MLP-D on the meta-train benchmark. Regarding your second point, please see the global response.

2. While we understand the desire to have every method on every benchmark, unfortunately the meta-test benchmark is much more expensive to run than the other two benchmarks since it contains larger and higher-dimensional datasets. For example, FT-Transformer runs into out-of-memory issues on some datasets, trying to allocate, e.g., 185 GB of RAM. We did not try SAINT since it’s even more expensive.

3. Continuing on the point above, TabR-HPO would potentially take 5-20 GPU-months (based on extrapolating the runtime of TabR-S-D) to run on meta-test on RTX 3090 GPUs. We are now running it on the Grinsztajn et al. benchmark to have it on at least one benchmark. We did not run ResNet-HPO before since in most papers and Fig. 3 it is quite similar to MLP-HPO while being slower (takes around two GPU-weeks on our benchmarks in Fig. 2). We are currently running it and provide preliminary results in the uploaded PDF, which are confirming our expectations.

4. We included the AUROC experiments after the main experiments were finished to include an imbalance-sensitive metric. However, since the whole meta-learning and model development process was performed with accuracy as the target metric, and the best-epoch selection was also performed with accuracy as the target metric, we think that the results in Figure 2 are more suited for the main paper as they are more appropriate to demonstrate the effect of meta-learning default parameters. We do refer to the AUROC experiments in the main paper, though, and will try to emphasize these experiments more in the next version.

Questions:

• See above.
• This is an interesting question, but would deserve its own paper. This would also be challenging to do without too much overfitting, given the limited number of meta-train datasets. TabRepo, perhaps the most large-scale study on portfolio learning, still only considers static portfolios.

I would like to thank the authors for the reply. I have carefully read the other reviews and all the responses from the authors.

• Regarding: "What is the source for your claim? Which AutoML tool would these be in? Are you referring to some of the color-coded improvements or only to the gray ones? There are some non-novel components in Figure 1 (c) since we wanted to start from a “vanilla MLP”, but as we color-coded in the figure, there are many new and unusual components. We also tried out the two NNs in AutoGluon and their default settings performed worse than MLP-D on the meta-train benchmark. Regarding your second point, please see the global response."

  The gray components would be an example, but additionally, the highlighted label smoothing is not something surprising. As mentioned by another reviewer, the paper provides a set of results that give intuition on the different method performances. As such, it would be beneficial for techniques that can be applied to the MLP to be added to the other baselines. It might be that the components hurt performance, it might be that the components improve performance.

  One simple test would be to reuse the defaults for the MLP for another deep baseline (for what component can be applied), while this might not be conclusive as the hyperparameters probably need to be tuned per method, it would provide some insights.

• Regarding: "While we understand the desire to have every method on every benchmark, unfortunately, the meta-test benchmark is much more expensive to run than the other two benchmarks since it contains larger and higher-dimensional datasets. For example, FT-Transformer runs into out-of-memory issues on some datasets, trying to allocate, e.g., 185 GB of RAM. We did not try SAINT since it’s even more expensive."

  In my perspective, the already considered list of methods is extensive. For example, running TabNet from my perspective is not interesting, the method is outperformed significantly from many methods. However, the already included baselines in the comparisons should be present in all results, since they leave "holes" in the results.

  One might worry that some methods might achieve more competitive performance and that is the reason why the authors do not include them. I personally do not think the authors are hiding results, however, the concerns should be addressed, especially considering the work lies towards the benchmarking and experimental side.

  If a method fails on certain datasets, the authors can point it out in the work.

• Regarding: "We included the AUROC experiments after the main experiments were finished to include an imbalance-sensitive metric. However, since the whole meta-learning and model development process was performed with accuracy as the target metric, and the best-epoch selection was also performed with accuracy as the target metric, we think that the results in Figure 2 are more suited for the main paper as they are more appropriate to demonstrate the effect of meta-learning default parameters. We do refer to the AUROC experiments in the main paper, though, and will try to emphasize these experiments more in the next version."

  While that is true, the main attention is invested in the main manuscript, as such, the results can be potentially misleading. AUROC is a better metric that considers class imbalance. Personally, I find the work interesting, whether it beats CatBoost or not.

• As a last question, was TabR run with embeddings or without? My understanding is the latter. If that is the case, it is questionable, since the main version of TabR contains embeddings. Additionally, the MLP is run with embeddings.

I thank the authors for the response.

• Regarding: "The label smoothing is highlighted as “unusual”, and we would argue that it is indeed unusual in the tabular context. We tried TabR-S-D with our preprocessing + beta_2=0.95 + scaling layer + parametric Mish + label smoothing on meta-train-class and it was slightly better than with just the preprocessing, which is in turn better than with the original preprocessing (as we showed in the Appendix). At this point the message of this investigation is unclear and would require more ablations, tuning, etc., opening a big rabbit hole."

  I am not sure about the practitioners, but personally, I have used label smoothing in my experiments. Nevertheless, this is a paper for the community and I would need to review the other papers in the literature to confirm this very minor point. A recipe from the authors could be beneficial for the community. I agree that it is more complicated in the scenario the authors are pointing out for other baselines. One could maybe frame it as a general recipe or set of defaults for deep learning methods but it is not trivial and I am not sure if that is the message the authors want to convey.

• Regarding: "TabR-S-D was run without numerical embeddings, since this is the version that was proposed in the paper for default hyperparameters. While this might deteriorate performance, it also improves the training time, which we also report. The new TabR-HPO results accidentally didn’t use numerical embeddings, we will rerun them."

  I understand, in that case, I would agree with the authors on using TabR-S-D for the default case and the one with numerical embeddings for the HPO case.

In my perspective, the work is in a good position, it needs a bit more work regarding the inconsistent baseline usage, but overall it has extensive results. Based on this, I will increase my score from 3 to 4.

We thank the reviewer for the constructive feedback.

Weaknesses:

1. [1] was uploaded on 13th of July 2024, so this paper cannot be used to question our novelty. (Besides, it also uses a mix of architectures, while we consider algorithm selection / ensembling.) Regarding ensembles, we do not claim to rediscover ensembles. Rather, we try to see how meta-learning better defaults affects the trade-off between ensembling / algorithm selection and HPO. While the ExcelFormer paper claimed to have very good defaults, ExcelFormer performed poorly in the experiments at https://github.com/pyg-team/pytorch-frame. So either the original results were problematic or the method is difficult to use correctly, both of which don’t make it attractive for us to compare to ExcelFormer. Moreover, as a transformer with inter-feature attention, ExcelFormer would likely result in out-of-memory errors on some high-dimensional datasets of the meta-test benchmark (as FT-Transformer does). We compared to TabR-S-D since their paper also claimed to have better defaults than GBDTs. We can, however, acknowledge the missing ExcelFormer comparison as a limitation.

2. We consider dataset sizes of 1K to 500K samples and up to 10K features, which is quite a large range, larger than the Grinsztajn et al. benchmark, for example. Smaller datasets are also interesting but hard to evaluate due to noisy test scores, and might be much more affected by details in early stopping etc. In addition, they are easier to handle for TabPFN or LLM-based methods, necessitating different baselines and making MLPs less attractive.

3. We agree that additional efficient baselines can help to substantiate our claim, but we think that the mentioned methods are not particularly promising based on recent results from the literature (see below). Instead, we evaluate MLP-PLR [5], an MLP with numerical embeddings that was presented as better than FT-Transformer from the group that created FT-Transformer, and performed well in recent benchmarks [6, 7]. MLP-PLR is faster than RealMLP but does not match RealMLP’s benchmark scores. Here are some reasons why we think that the mentioned NNs are not very promising:

• TabNet performs poorly both in recent benchmarks as well as in our preliminary experiments. In McElfresh et al. (2023), TabNet performs much worse than MLP or FT-T while also being slower than both of them. In the FT-Transformer and SAINT papers, TabNet is also worse than MLP. The same holds for https://arxiv.org/abs/2407.09790 and https://arxiv.org/abs/2407.00956
• Net-DNF performed slightly worse than TabNet in Shwartz-Ziv et al. (2022). It did perform a bit better than MLP on 4 datasets in Borisov et al. (2022). It was also outperformed by TabCaps in the TabCaps paper.
• TabCaps performs worse than MLP in a recent large benchmark (https://arxiv.org/abs/2407.00956), see Tables 19 and 20 therein. The TabCaps paper also doesn’t contain a lot of datasets, so their own claims are probably not too reliable.

Questions:

• We do not aim to address tabular pre-training or zero-shot scenarios. In cases where one can pre-train on a dataset with the same columns, RealMLP might be a good option.

[5] https://proceedings.neurips.cc/paper_files/paper/2022/hash/9e9f0ffc3d836836ca96cbf8fe14b105-Abstract-Conference.html

[6] https://arxiv.org/abs/2407.02112

[7] https://arxiv.org/abs/2406.19380

Thank you for your reply.

I’m aware that [1] was uploaded on July 13th, 2024, and that it is completely different from your paper. As I mentioned in my initial comment, it was only an example to illustrate that this paper revisits an old conclusion. I’m not questioning the novelty of your work based on this example.

Regarding the performance of Excelformer, I believe you should use the official code. I’ve tried using the code from PYG, but it seems to have many bugs, and the hyperparameters were clearly misused.

Additionally, in my experience, Net-DNF often outperforms both TabNet and NODE.

But, the performance of previous works and the similarity to [1] are not the main points! We all know that a paper should aim to present new scientific findings.

You said: "we try to see how meta-learning better defaults affects the trade-off between ensembling / algorithm selection and HPO". Essentially, integrating different methods (such as MLP + GBDT) has long been recognized as an effective approach in Kaggle tabular prediction competitions. Besides, achieving good results in these competitions typically does not require extremely rigorous HPO (that may leads to over-fitting), which are well understood by tabular data prediction experts.

We thank the reviewer for the constructive feedback.

Weaknesses:

1. While we agree that such experiments would be interesting, they would be very costly and are not central to the research question we want to answer in the paper (how good can we make MLPs and GBDTs with default parameters?). We would argue that the existing results should already be interesting to readers and that our paper's ability to pose many interesting questions and potential follow-up studies should be viewed as a strength rather than a weakness.

2. We can mention this in the main paper. We already mentioned in Appendix B that this is likely due to the very high weight decay value (we did an earlier ablation without weight decay for regression and the differences were much smaller). So we would argue that the sensitivity to some hyperparameters is not a big issue as it can be fixed with lower weight decay or by just using our provided defaults.

3. We double-checked our implementation and we also checked MLP-PLR and MLP-PL with the original libraries and observed the same effect for defaults on meta-train (we didn’t run MLP-PL-HPO). So it appears that PLR is better than PL on the 11 datasets of its own paper with HPO, but is worse with fixed hyperparameters on our 118 meta-train datasets. The PLR library documentation also mentions that PL can allow for a lower embedding dimension (perhaps because there is no information loss due to the ReLU). So this seems to be a weakness of PLR embeddings and not of our paper.

4. TabR-S-D is a non-parametric baseline. Unfortunately, TabR-HPO would be way too expensive to run on the meta-test benchmark (easily 5-20 GPU-months on RTX 3090 GPUs extrapolated from the runtime of TabR-S-D). We did also include TabR-HPO on the Grinsztajn et al. benchmark in the rebuttal. Since other non-parametric methods like SAINT are even more expensive, we cannot afford to run them in Figure 2 (but SAINT is in Figure 3). We did include MLP-PLR now. The defaults are on the level of MLP-D (the paper does not propose defaults, but the library does). MLP-PLR-HPO comes close to RealMLP-HPO on meta-test-reg but underperforms RealMLP-TD on the other three meta-benchmarks. Given the results, optimizing the embedding dimension could be interesting for RealMLP-HPO as well, although the larger embedding dimensions appear to considerably increase CPU runtimes. (MLP-PLR-HPO has about equal runtime to RealMLP-HPO, despite using early stopping.)

Questions:

1. See weaknesses.

2. We did not mention this in the paper since it is a one-time cost, but: Around 100-300 steps for GBDTs (could have probably matched this with 20-30 manual tuning steps). For RealMLP, we did not use automated tuning, since we repeatedly implemented new components and tried them, so it is hard to say but several hundred configurations were tried.

3. We have measured inference efficiency but did not want to report it since it could be quite sensitive to implementation details (inference batch size, data conversions, etc.). For example, we heard from AutoGluon developers that they could greatly improve AutoGluon’s inference times just by running its preprocessing in numpy instead of pandas. Nonetheless, here are some of our measurements (inference time per 1K samples):

• CatBoost-D_CPU: 0.00273057 s
• XGB-D_CPU: 0.00297026 s
• RF-SKL-D_CPU: 0.016169 s
• RealMLP-TD_CPU: 0.00501239 s
• RealMLP-TD-S_CPU: 0.00377973 s
• MLP-D_CPU: 0.0187927 s
• TabR-S-D_CPU: 0.125827 s

As you can see, RealMLP-TD is a bit slower than GBDTs but still fast, while for MLP-D there seems to be a suboptimal implementation (we don’t know why it would be so slow otherwise) and TabR is very slow.

I carefully read the rebuttal and would like to thank the authors! Also, I apologise for the late reply.

1. In my opinion, the idea of making MLP as strong as possible is really valuable for the field but the lack of "fair" comparison is a big weakness of the paper. ("fair comparison" = other models with the proposed techniques). Though I understand the high cost of the tuning, I think even results for TabR-S-D+some_tricks would be beneficial for the paper.
2. Thanks for the answer. It should be added to the main text since stability to hyperparameters is important.
3. I think this "issue" could also be related to optimization stability (or sensitivity to hyperparameters).
4. Sorry, I made a typo, TabR is the non-parametric baseline. I meant that the paper lacks a strong parametric baseline since FT-T is quite expensive, and ResNet is just bad. MLP-PLR is a good baseline thanks for the provided results.
5. Inference time seems reasonable, thanks for the results.

Overall, in my opinion, the main weakness (the first one) remains and, thus, I keep my score unchanged.

We just noticed that our pdf uses an older version of the meta-train/meta-test plots where ResNet-HPO is not included. The results for ResNet-HPO are similar to the Grinsztajn et al. benchmark: a bit better than MLP-HPO for classification, a bit worse for regression, and about twice as slow.