TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks

Thank you for your detailed review. We appreciate that you found our paper well organized and easy to follow, and our problem well-motivated. We address each of your questions below:

W1: On novelty

Thank you for raising this point. We note that our novelty lies in applying prompt tuning to PFNs, and that we fill a major gap for PFNs: the issue of scaling beyond their various capacity limits. We respectfully note that the NeurIPS reviewer guidelines includes novel combinations of well-known techniques to be included in originality, and overall there has been a trend towards emphasizing impact rather than novelty.

W2: Computational expense

Thanks for bringing this up. We discuss this point as a limitation in Sec. 8 of our paper. However, we also note that, given the novelty of our method and the ongoing efforts to optimize the runtime of transformer-based architectures such as TuneTables, we can reasonably expect future research to further improve accuracy-runtime tradeoff of our method. Moreover, our method shows complementary strengths with GBDT, as each method outperforms on specific datasets.

W3: Presentation of results

We thank the reviewer for pointing out the improvements that can be made to Figure 2. We have created a wider version of Figure 2 such that the vertical lines for the algorithms no longer overlap, which we added to the paper. We also added the following to the caption of Figure 3:

The colorbar on the y axis represents the comparative change in per-dataset accuracy between two algorithms (A: blue, B: red). Positive numbers represent the absolute gain in accuracy of B w.r.t. A, negative numbers represent the absolute gain in accuracy of A w.r.t. B.

Q1: Assuming attributes

We do assume the basic type of each feature (numerical or categorical) on our tabular data.

Q2: Data summarization

We thank the reviewer for this question on data summarization. While preparing this rebuttal, we have completed additional experiments on several different methods for feature summarization using our own benchmarks (please see our global response part C), as this provides a realistic estimate of how feature summarization methods perform in this particular setting. Our results for sketching summarization are discussed in Appendix Table 5; there, we observe that no sketching method outperforms random search for most dataset-algorithm pairs. We hope this extension of results addresses your concern.

Q3: Bias mitigation

We thank the reviewer for making this suggestion. ​​In Sec. 5 of our paper, we cite fairness optimization as one particular example of a more general class of problems to which TuneTables is applicable, namely, multi-objective optimization problems. While in the particular case of fairness it might be possible to attain better results with TabPFN by debiasing the training data, this would not give us more insight into multi-objective optimization in TuneTables, as not all multi-objective optimization problems are amenable to such interventions. We nevertheless consider this an important and useful direction for future research into fairness with TuneTables.

Q4: Fixed number of classes

Thanks for this question. The TabPFN authors provide three reasons in their paper why they choose to fix the number of classes at 10; see below.

“We focus on small datasets because (1) small datasets are often encountered in real-world applications (Dua and Graff, 2017), (2) existing DL methods are most limited in this domain (Grinsztajn et al., 2022) and (3) the TabPFN would be significantly more expensive to train and evaluate for larger dataset.”

Thank you very much, once again, for your excellent comments. We respectfully ask that if you feel more positively about our paper, to please consider updating your score. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!

Thank you for your detailed review. We appreciate that you found our idea interesting, and that you appreciated the large number of experiments we provide. We address each of your questions below:

W1: Presentation

We thank the reviewer for a comprehensive set of suggestions on how to improve the presentation of our work. We especially agree with you that the main paper should be complete by itself. We have thought through and made all of the changes to make this true and to address all of your comments. Because NeurIPS does not allow us to upload a modified version of our paper as part of the discussion, we cannot provide you with a highlighted manuscript with our changes; instead, we summarize the major points below.

• We significantly compressed Sections 5 and 6 by having just one section titled “TuneTables extensions”, with mitigating bias and dataset understanding as two subsections, and substantially cutting down the text to have only the main points.
• Even with the compression of Sections 5 and 6, we cannot add all of Tables 6-12 to the paper. In order to reduce the need for the reader to flip between pages while reading, we added one-sentence summaries of the contents of the main appendix tables referenced (6,7,9), where they are referenced in the Sec. 4. Here is an example of such a change:

In Appendix Table 6 we show that despite the large divergence from the PFN’s pretraining, TuneTables achieves a high rank, second only to XGBoost. -> In Appendix Table 6 we show that despite the large divergence from the PFN’s pretraining, TuneTables achieves a mean rank of 2.52, ahead of CatBoost and a ResNet, second only to XGBoost, whose mean rank is 2.0.

• We move much of the ablation discussions in Sec 4 into the appendix next to tables 11 and 12, giving a high-level summary in Sec. 4
• We now name the benchmark in our Table 1 as the “tabzilla benchmark suite” to be consistent with the language of recent work.
• We also changed "reduces or reverses TabPFN's limitations" -> "addresses TabPFN's limitations", per the suggestion
• Thanks for the suggestion; we renamed the “TuneTables-Hard” dataset to LargeScaleTables, to reduce confusion between the algorithm and the dataset
• We have also made the smaller edits you suggest, including directly referencing neural processes earlier in the paper – we now write, “A recent breakthrough, prior-data fitted networks (PFNs) [33, 52] are a specific type of neural process (Garnelo et al., 2018) which learn to perform approximate Bayesian inference in a single forward pass using in-context learning.”

Q1: Overfitting

Thanks for the question. In our experiments, we found that TuneTables would sometimes attain higher 3-fold cross-validation accuracy compared to zero-shot TabPFN, but lower test accuracy, particularly when val and test sets were small. TabPFN, which was pretrained on large quantities of synthetic data and did not optimize using the val set, exhibited low val/test divergence on all datasets. We will add this comment to the final version of our paper.

Q2: Zero-shot TabPFN

Thanks for the question. Zero-shot TabPFN is indeed used as part of the TuneTables grid search on such datasets. The reason why we do not only use TabPFN is that the exact size at which TuneTables outperforms TabPFN is dataset-dependent (with an average transition of around 800 samples). We will add this comment to the final version of our paper.

Q3: Too many features

Thanks for the question. By ‘too many features’, we mean more than the 100 features which TabPFN can utilize without feature selection methods. We will clarify this in our final draft as well.

Q4: Batch size 1

Yes! The statement is assuming a batch size of 1. We conduct the relevant experiments on NVIDIA RTX8000 GPUs with 48GB of VRAM per GPU. We will clarify this in our final draft.

Q5: Memory requirements

We thank the reviewer for raising the topic of FlashAttention. Indeed, FlashAttention is memory-linear in sequence length. However, please note that it has not yet been implemented in the public release of TabPFN -- it may be integrated into a future release, and any benefits will carry over to our method. Please also note that while Flash Attention is extremely useful, it still has other limitations: inference time continues to be quadratic, and there are other overheads on GPU memory. Overall, there remains a considerable need for new algorithmic methods which can be scaled up to the sequence lengths required by large tabular datasets. We will add a clarification in our paper regarding this point.

Thank you very much, once again, for your excellent comments. We respectfully ask that if you feel more positively about our paper, to please consider updating your score. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!

Thank you for the rebuttal. Although I am unable to verify the presentation improvements, I believe the authors' commitments and stated changes. I lean towards accepting the paper and have updated my score (4 -> 6) to reflect this.

Thank you for your detailed review. We appreciate that you found our paper important and novel! We address each of your questions below:

W1: Hyperparameter spaces

We thank the reviewer for raising this point. We add TuneTables hyperparameters in the PDF attached to the global comment. For all baselines, we use the same hyperparameter ranges as the Tabzilla paper with one exception: we increased the range of tree depth by a factor of 100, since a high depth is much more appropriate for very large datasets. The tabzilla authors point the reader to this file in order to see hyperparameter ranges. We are happy to add all of this information directly into our paper.

W2: Arrangement of the paper

We thank you for drawing our attention to the indirection in Sec. 4. We will update this so that it refers to Table 5 directly. Regarding how to interpret Table 5, we clarify below:

The columns labeled SKT / FTS / SMP list the best performing method for sketching, feature subsampling and label-aware sketching technique, respectively. Label-aware sketching refers to a strategy where we either sample instances proportionate to their labels, or we oversample minority classes with replacement to create a class-balanced distribution. While the choice of label-aware sketching strategy is often impactful (and we use it in TuneTables), and the choice of feature subselection method can be important for some datasets, in all but one case, no sketching method we test outperforms random sampling.

The comment that “all sketching and feature selection methods plateau in performance well before approaching parity with GBDTs” is referring to the fact that in Table 5, TabPFN fails to match CatBoost’s performance on seven datasets, all of which are very large.

In the final version of our paper, also as you suggest, we will incorporate a table of contents for our appendix, and we will add the above discussion near Table 5.

W3: Ablation aggregate scores

Thanks for this note! We will add relevant aggregate scores to all results tables in the main paper.

W4: TT-medium and TT-light

We thank you for raising this point about reporting TuneTables-medium and TuneTables-light results for all of TabZilla, including runtimes, in order to better understand the time-performance pareto frontier of our method. While preparing this rebuttal, we have completed these experiments on the 98 datasets from the tabzilla benchmark suite; (please see our global response; section A). We hope this extension of results helps to address your concern.

Q1: Memory requirements

We thank the reviewer for raising the topic of FlashAttention. For our reply, please refer to our comment on review 7Ntu, Q5.

Q2: Grid search

The space over which we conduct grid search in TuneTables is conditioned on metadata; # features, # samples, # classes. The feature and class splits are set by the limits of the particular TabPFN checkpoint we optimize. When we reach a leaf node, TuneTables-standard and TuneTables-medium conduct a grid search over a fixed range of configurations. The size of our search space is always < 30, and usually < 10. For feature-large datasets, TuneTables-light conducts additional optimization in the feature space prior to grid search. See the PDF attached to our global response for further details. For the final version of our paper, we will add this clarifying material to the appendix.

Q3: TuneTables-Hard criteria

We curated datasets with far more samples and features than in the standard datasets from tabzilla (max 45,000 samples) from OpenML to better understand the scaling challenges of PFNs, omitting image classification datasets. Finally, because we intended to include TabPFN in our analysis and every dataset we had curated was beyond the recommended limits of that model, we added several small datasets from tabzilla; since TabPFN generally outperformed boosted trees on these smaller datasets, and TuneTables extended TabPFN, we heuristically selected smaller datasets so as not to favor either TabPFN or boosted trees. For the final version of our paper, we will add these details to the appendix.

Q4: Table 11

We thank the reviewer for this question. In Table 11, we compare different variants of our method and entire backbone fine-tuning (TabPFN-FT) to TuneTables itself. For all methods in this table except for TuneTables, there is no hyperparameter optimization (HPO). For TuneTables, we use our standard grid search. For the final version of our paper, we will clarify this.

Q5: Ensembling

We thank the reviewer for this clarifying question about ensembling in TuneTables. In a TuneTables ensemble, each ensemble member fits its own tuned prompt to the data. Variance in the ensemble members is introduced by differences in the random initialization of the tuned prompt, as well as permuting the order of features and labels, a la TabPFN, one time before each tuned prompt is fitted. For the final version of our paper, we will clarify the sentence you cited in your analysis.

Q6: Coresets

For efficient coreset selection, we use a variant of Farthest Point Sampling; after selecting an initial set of n=5 random points, we compute the distance of each point in the dataset to the set of already selected points. Next, we add to the set of selected points the point whose distance to the selected points is maximal. Finally, we update the distances of all the points according to the updated selected set; and continue iteratively. We will add these details to the final version.

Thank you again for your excellent comments. We respectfully ask that if you now feel even more positively about our paper, to consider slightly increasing your score. We are happy to continue the discussion any time until the end of the discussion period. Thank you!

Thank you for your detailed review. We appreciate that you found our paper to be presented well, and our methodology sound. We address each of your questions below:

W1: Chosen benchmarks

We thank the reviewer for raising this clarifying point about our benchmarks. Our main results in Tab. 1 are reported on a standard benchmark from the literature, tabzilla. In the tabzilla benchmark, SAINT outperforms FT-Transformer; this is also the case in the largest-scale comparison in the well-known Grinsztajn benchmark (see Fig. 1), where SAINT outperforms FT-Transformer outright on regression tasks and on classification tasks with few iterations of random search (there is no statistically significant difference after up to 100 iterations of random search on classification tasks). The benchmark introduced in our paper, TuneTables-Hard, was curated in part to illustrate TabPFN’s scaling limitations – for further discussion on this point, please see our response to FY4B, Q3.

W2: ExcelFormer

We thank you for raising this point about ExcelFormer, a recently published method. We have implemented this method for our rebuttal, and completed experiments on around half of the 98 datasets from the tabzilla benchmark suite; the initial results are intriguing, and we hope to extend them after the rebuttal period. Please see our global response -- tables B1 and B2; we hope this extension of results addresses your concern.

W3: Hyperparameter tuning

Experimental setup for hyperparameter tuning. This is a good question; thanks for allowing us to clarify! All of the algorithms come with their default set of hyperparameters used in the official implementation, and we used all of these settings. It is common in tabular papers to further improve performance for each dataset by conducting hyperparameter tuning (via k-fold cross validation) on a per-dataset basis. To be consistent with prior work such as Tabzilla, TabPFN, and Grinsztajn et al, we chose the best configuration among the default parameter sets and the additional 29 sets. We will clarify this in the paper, and we are happy to answer any further follow-up questions about our experimental setting.

W4: Regression tasks

We thank you for raising this point about regression tasks. TabPFN does provide a server to run regression tasks, but to the best of our knowledge, no regression results have yet been made public by the authors. We have completed extensive preliminary regression experiments on TuneTables for this rebuttal; please see our global response, Table D. We hope that this significant extension of results helps to address your concern.

Thank you very much, once again, for your excellent comments. We respectfully ask that if you feel more positively about our paper, to please consider updating your score. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!