TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems

Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:

---

Q1: Sensitivity Analysis

---

A1: We sincerely appreciate the reviewer’s valuable suggestion. To address the sensitivity of BETA’s performance to the number of bootstrapped samples and encoder paths, we have conducted additional experiments.

• In Appendix D.3, we provide an analysis of the number of encoders and other key components in BETA.

• Additionally, we include performance rankings for BETA across various datasets from the Tiny Benchmark2 classification dataset [1], where we vary the number of bootstrapped samples. A table summarizing the results is presented below:

  size	500	1000	2000	3000	tabpfn
  average rank	3.39	2.87	2.57	1.91	4.24

• We also summarize the performance of Beta with different number of encoders and their corresponding average training time across the datasets of Tiny Benchmark2. Specifically, we show the average percentage change in performance relative to the original TabPFN.

  number of encoders	1	2	4	8	16
  relative improvement	1.7%	2.6%	3.19%	3.23%	3.76%
  finetuning time (s)	31	48	73	126	203

The result demonstrates that both the number of bootstrapped samples and the number of encoders lead to better performance, but they also increase computational costs.

We hope these additional details address the reviewer’s concerns. Additionally, we will provide a brief summary of the experimental details and sensitivity analysis in the main text of the final version of the paper.

---

Q2: Recent Work on Contextual Embeddings

---

A2: We sincerely thank the reviewer for the suggestion. We have noted that the paper referenced by the reviewer, [2], provides a detailed analysis of the combination of ensemble methods with advanced embedding techniques. We will incorporate a discussion of this work and related research in the final version of the paper. Additionally, we plan to explore how these techniques could be integrated with BETA in our future work.

We appreciate the reviewer’s insight and will ensure this topic is adequately addressed in the final version of the manuscript.

---

Q3: Extending BETA to Regression Problems

---

A3: Thank you for your insightful comment. We applied BETA to regression datasets by using a pre-trained regression version of PFN, where the final classification head has an output dimension of 1, while keeping the other modules unchanged. The only modification was replacing the loss function with MSE loss and fine-tuning it on downstream regression datasets.

We evaluated the regression version of BETA on 100 regression datasets from [1] and compared the average rank to other tuned methods (we only display a subset of the methods for clarity) presented in [1]. Our results show that the regression version performs well, although its performance is slightly lower compared to its classification counterpart.

We will clarify these findings and potential directions for future work in the final version of the paper. Thank you again for your valuable suggestion!

---

Q4: Response to Testing BETA under Distribution Shifts

---

A4: Thank you for your thoughtful question. We have tested BETA on three classification datasets from TabRed [3] and integrated the split method (Beta-split) and temporal embedding techniques (Beta-temporal) discussed in [4] into BETA. The table below presents the AUC results of BETA on these datasets, showing that the plugin techniques are effective for improving BETA’s performance. Additionally, these results highlight the potential of ICL methods for addressing temporal shift scenarios.

We will include these findings and further clarify this in the final version of the paper. Thank you again for the valuable suggestion!

[1] A Closer Look at Deep Learning Methods on Tabular Datasets. 2024

[2] Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers. 2024

[3] Tabred: A benchmark of tabular machine learning in-the-wild. 2025

[4] Understanding the Limits of Deep Tabular Methods with Temporal Shift. 2025

Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:

---

Q1: What is the validation set used for? Were the models early stopped to maximize accuracy?

---

A1: We follow [1, 2, 5, 6], where the validation set is used only for hyperparameter tuning and early stopping, not for training. For example, for baselines like LightGBM, XGBoost, and FT-Transformer, we use the validation set for early stopping. For BETA and TabPFN variants, we use it for early stopping but do not refit on the combined training and validation set.

---

Q2: Why accuracy?

---

A2: We followed the evaluation metrics used in [1, 2, 5, 6]. We also computed AUC-based average ranks (see below). RealMLP's lower AUC ranking may be linked to label smoothing, aligning with findings in [3].

---

Q3: Bagging

---

A3: We agree that other baselines may benefit from bagging. However, since TabPFN can change context without retraining, we only applied inference-stage bagging for BETA. As a result, for other baselines, bagging would significantly increase the cost—scaling nearly linearly with the number of base models.

To evaluate bagging’s impact on other models, we trained 16 classifiers per dataset with different random seeds on subsets of size max(0.8 × train_size, 1000) for CatBoost, LightGBM, and RealMLP, ensuring that each subset was at least as large as the ones used by BETA during inference.

We appreciate your suggestion and will further enhance the discussion on Bagging in the final version.

---

Q4: HPO

---

A4: We followed the hyperparameter tuning approach of [1, 2, 5, 6]. We carefully read the article you referenced [4] and implemented the 5-fold CV evaluation protocol from [4] (Section 2). However, we found this protocol to be significantly more computationally intensive.

Due to computational constraints, this approach was applied only to XGBoost, RealMLP, and Beta on classification datasets with fewer than 3,000 rows (from [1]).

For XGBoost and RealMLP, we performed HPO using 5-fold CV, then averaged the predictions from the 5 trained models. For BETA, we maintained its default hyperparameters but trained five models on different training folds and averaged their predictions.

The results, shown in the table below, indicate that cross-validation does improve model performance, even for methods like BETA that do not require HPO, though the gain for BETA is relatively modest.

We sincerely appreciate your valuable feedback on HPO strategies, and in the final version of the paper, we will conduct a more detailed evaluation of these effects.

---

Q5: I agree with the authors that TabPFNv2 is concurrent work, however I think it would elevate the paper if it is possible to include a version of Figure 4 with TabPFNv2 results in the appendix.

---

A5: In the final version of the paper, we plan to include additional baselines. Due to time constraints, we evaluated TabPFN-v2 and TabICL, both using default settings.

To ensure a fairer comparison, we also tested subsampled variants (TabPFN-sub, TabICL-sub) with 16 training subsets and different preprocessing strategies.

Results show that full versions of TabPFN-v2 and TabICL approach BETA’s performance, while subsampled versions perform significantly worse. Our method is currently limited by memory constraints, but a well-optimized TabPFN-v1 could allow larger context sizes, potentially further improving results. This suggests that our method is currently constrained by engineering-related memory limitations. If a highly optimized version of TabPFN-v1 were available, we believe our approach could scale to even larger contexts, potentially further improving performance.

---

Q6: Sensitivity Analysis

---

A6: Please refer to A1 for reviewer 2vtp.

[1] Revisiting deep learning models for tabular data. 2021

[2] A Closer Look at Deep Learning Methods on Tabular Datasets. 2024

[3] Better by default: Strong pre-tuned mlps and boosted trees on tabular data. 2024

[4] Unreflected Use of Tabular Data Repositories Can Undermine Research Quality. 2025

[5] On Embeddings for Numerical Features in Tabular Deep Learning. 2022

[6] TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023. 2023

Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:

---

Q1: Limitation of Novelty

---

A1: BETA builds on existing techniques, with its novelty lying in breaking key limitations of TabPFN while maintaining inference efficiency. Simplicity and effectiveness are highly valuable, especially for tabular data:

1. Overcoming TabPFN’s Limitations:
   • (a) Handling Arbitrary Feature Dimensions: We introduce multiple lightweight encoders, incorporating batch ensemble techniques, to map raw features into a unified dimension while keeping TabPFN’s parameters frozen.
   • (b) Scaling to Large Dataset: Bagging extends TabPFN to datasets with more samples.
   • (c) Handling Large-Class Problems: Assigning ECOC codes to different encoders overcomes TabPFN’s class limitations.
2. BETA remains compatible with PFN-style batching, ensuring minimal extra model inference computational cost.
3. An effective combination of techniques for TabPFN that addresses key weaknesses and significantly outperforms widely used, finely tuned baselines is a meaningful contribution worthy of further study.

---

Q2: But the authors claims that on more other datasets, BETA demonstrates superior performance desipite that this experiment verify the BETA should perform better on large scale datasets.

---

A2: This experiment aims to demonstrate BETA’s improved generalization on large datasets. As shown in Figure 5, BETA’s performance gain over TabPFN grows with dataset size.

However, on the largest datasets, LocalPFN performs better because it assigns a unique context to each sample, preventing context sharing and making it incompatible with the highly parallelizable batching strategy used in PFN. While this boosts performance, it comes at the cost of much lower efficiency—as shown in Table 3, inference time increases significantly.

LocalPFN is a valuable contribution, but this highlights a trade-off on large datasets: BETA offers better efficiency, while LocalPFN prioritizes local adaptation at a high computational cost.

---

Q3: Figure 2 introduce some settings based on TabPFN, missing the variants of table 1.

---

A3: In Section 2.2, we categorize major TabPFN variants into Context Selection and Fine-tuning. As shown in Table 1, all listed variants fall into these two categories.

As stated in Section 2.3, we selected TabPFN-KNN and TabPFN-Finetune as the most representative methods for these categories, with other approaches being minor modifications or combinations of these two. To ensure clarity and avoid redundancy, Figure 2 presents results for these key methods. The following table shows the relative percentage changes in bias and variance (negative values indicate reduction) of Beta and other variants compared to TabPFN, averaged over the dataset sizes used in Figure 2, on the Adult and Bank datasets. Since TabPFN's absolute variance on these datasets is smaller than its bias, the percentage changes in bias are much smaller than those in variance.

---

Q4: No evidence to show the claims in Table 1.

---

A4: These claims are derived from the specific improvement strategies each method applies to TabPFN. For example, none of the other methods adjust TabPFN’s input representation, meaning they cannot handle high-dimensional datasets. Additionally, evidence supporting the claims about bias and variance can be found in Figure 2 and the table in A3 above.

We greatly appreciate your suggestion and will further clarify Table 1 in the final version to ensure the claims are well-supported.

---

Q5: Why are the encoders in fine-tuning stage placed before TabPFN instead of after it?

---

A5:

1. Placing encoders before TabPFN allows it to handle arbitrary feature dimensions by mapping raw features to a fixed dimension.

2. Our preliminary explorations found that adding encoders after TabPFN provided almost no improvement.

3. Moreover, placing them before TabPFN integrates effectively with batch ensemble techniques, enhancing performance.

---

Q6: Some foundation tabular model is missing

---

A6: CARTE models tabular data as a graph, embedding column names and entries with a graph-attentional network. UniTabE encodes column names, data types, and cell values as tokens using an encoder-decoder architecture with contrastive learning. Unlike TabPFN, which relies on a pre-trained Transformer without semantic modeling, these models incorporate semantic information. We will enhance the discussion on tabular foundation models in the final version.

Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:

---

Q1: Finetuning time plus inference time compared to other methods.

---

A1: In Table 3, we have provided a comparison of inference time, average rank, and the number of learnable parameters.

To address this point more comprehensively, we include the fine-tuning/training time for BETA as well as other methods listed in Table 3. Notably, since Beta only fine-tunes the encoder parameters, it has a clear advantage in both fine-tuning and inference efficiency compared to other variants.

Additionally, it is important to highlight that methods like FT-T and TabR underwent an extensive hyperparameter tuning process in [2], spanning 100 trials. As a result, their actual tuning time is approximately 100 times their reported training duration.

---

Q2: Claim: Response to "Scales to Large Datasets"

---

A2: We refer to datasets with a large number of samples as large datasets.

TabPFN is inherently limited by its attention mechanism over instances, leading to high memory overhead. As a result, it typically requires subsampling the training set to remain computationally feasible. To mitigate the impact of context size on model performance during inference, we introduce bagging for TabPFN.

Additionally, since TabPFN is pretrained only on synthetic datasets with fewer than 1000 rows, we incorporate multiple lightweight encoders to reduce bias, enabling the model to better scale to large datasets.

---

Q3: About TuneTables: Formulation seems strange to me, TuneTables only optimizes the prompt, so what does this "either finetuning the entire model" what do you mean here?

---

A3: In classification tasks, TuneTables employs prompt tuning, where “which adjusts only a small set of parameters, reducing resource consumption” specifically refers to prompt tuning. However, in their extension to regression tasks, they adopt end-to-end fine-tuning. This is why we mentioned "fine-tuning the entire model." We will clarify this distinction in the final version.

---

Q4: I don't see how it "ensures" "diverse representations", there is nothing enforcing that as far as I see, so ensure seems a misleading formulation here (as one could imagine losses that would ensure it).

---

A4: We appreciate the reviewer’s observation. The goal is to ensure that each encoder generates TabPFN-compatible latent representations, while diversity is encouraged through different initializations.

Additionally, TabM [1] has discussed diversity in base models, and we further analyze BETA’s encoder diversity through embedding visualizations in Appendix Figure 9, showing that the learned representations exhibit diversity. We will provide a detailed explanation of diversity and TabPFN-compatible latent representations in the final version.

---

Q5: Response to Figure Presentation.

---

A5: We appreciate the reviewer’s helpful suggestions.

• Figure 3: We will adjust the alignment of "Inference Stage", "Mean", and "Loss" to ensure proper centering within their respective boxes.
• Figure 5: We will modify the x-axis to directly represent actual dataset sizes and consider using a scatter plot of dataset size vs. relative improvement for better clarity. Additionally, we will correct the typo in the y-axis.

These improvements will be incorporated into the final version.

We sincerely appreciate the reviewer’s careful and constructive feedback. We believe these valuable suggestions will further enhance the quality of our paper, and we will incorporate the necessary revisions accordingly.

[1] Tabm: Advancing tabular deep learning with parameter-efficient ensembling. 2025.

[2] A Closer Look at Deep Learning Methods on Tabular Datasets. 2024.