A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities

We sincerely thank Reviewer xoSe for the positive feedback highlighting our paper’s extensive empirical evaluation, in-depth model analysis, and clear presentation.

We respectfully address the two concerns as follows:

Q1: On the novelty of our contributions:

While we do not introduce a new model architecture, our central contribution lies in systematically evaluating, understanding, and extending TabPFN v2—a model that is rapidly becoming foundational in the tabular domain. As argued in the main text, principled analysis and adaptation of such pre-trained models is increasingly impactful and necessary, especially as foundation models become more opaque and widely adopted. Our extensions (e.g., TabPFN v2*) are not just reapplications of known techniques like bagging, but carefully tailored adaptations that address the scalability limitations of TabPFN v2 without retraining, enabling its use in previously unreachable domains like high-dimensional or many-class tasks.

Q2: On the connection to Chain-of-Thought (CoT):

We acknowledge that the CoT analogy is conceptual rather than mechanistic. As clarified in Section 7, the idea is that sequential model reuse (e.g., using TabPFN v2 in a divide-and-conquer manner) mimics the layered reasoning process in CoT prompting. We agree this could be better framed and will revise the language to avoid overstating the parallel.

We hope this clarifies the intention and value of our work as a comprehensive, principled effort to advance the use and understanding of tabular foundation models.

We sincerely thank Reviewer pvYD for the thoughtful review and for recognizing the importance and timeliness of our work, the clarity of our paper, and the rigor of our evaluation methodology.

We respond to the specific questions and comments below:

Q1: Statistical significance and use of a single validation/test split

Thanks for the comment. We followed standard evaluation protocols from prior work (e.g., [Gorishniy et al., 2021]) to ensure comparability, using a 64/16/20 training/validation/test split. We agree that multiple train/validation/test splits would provide stronger statistical guarantees; however, this would incur a prohibitive computational cost due to the need for full hyperparameter tuning across all baselines and 273 datasets.

To assess robustness, we performed an additional comparison using 5-fold cross-validation on 10 classification datasets from the Talent-tiny benchmark. We observed consistent relative rankings across both hold-out and cross-validation settings:

These results confirm that the superiority of TabPFN v2 reported in our paper is robust to the evaluation protocol. We will include this consistency check in the final version.

Q2: Novelty of insights on randomized tokens and attribute token learning

We appreciate your comment. We want to first reiterate that our novel insight lies in understanding how “randomized tokens” work and enable TabPFN v2 to handle data heterogeneity (Lines 48 - 50). We certainly are aware that both TabPFN and TabPFN v2 adopt in-context learning and a single forward pass for inference (Line 89). However, their mechanism to handle data heterogeneity are drastically different (section 3).

The original TabPFN v1 relied on padding features to a fixed dimension (e.g., 100) to preserve their original semantics, whereas TabPFN v2 introduces randomized tokens, sampled at inference time, for each attribute. This design eliminates the need to define attribute- or dataset-specific tokens, syntactically enabling direct application of the pre-trained model across heterogeneous datasets (section 5.1).

Our novel insight is that, although randomized tokenization appears to discard attribute semantics, TabPFN v2 can still consistently infer meaningful attribute relationships through in-context learning. Specifically, we show that randomized tokens act like high-dimensional “continuous one-hot encodings” that reliably differentiate attributes across instances (Line 245). Pre-training on diverse tasks enables TabPFN v2 to infer co-occurrence patterns and relative attribute importance purely from the statistical structure of each dataset, without pre-defined attribute semantics.

Q3: Linearly separable embeddings and leave-one-fold-out strategy

We appreciate your comment. Before diving into the claim in line 60, we kindly invite the reviewer to revisit our preamble in lines 54 - 56. Just like what the reviewer argued, after seeing the superior performance of TabPFN v2, we expect to see a linear separable feature space produced by TabPFN v2, similar to other deep learning models. However, verifying, visualizing, and extracting it is non-trivial and not obvious (Lines 56 - 58).

This is because TabPFN v2 is a non-standard in-context learner, and naive extraction of embeddings from its output tokens leads to surprisingly discrepant features between training and test instances—stemming from their different label embedding types (true vs. dummy). As demonstrated in Figure 4(b), embeddings extracted in this way create divergent clusters for training (darker crosses) and test (lighter circles) points, leading to poor generalization when used in a linear classifier.

Our novelty lies in a carefully designed approach to extract features from TabPFN v2 (section 6.2). We propose a leave-one-fold-out embedding extraction strategy to resolve this issue: training instances are iteratively placed in the query set with dummy labels, while the remaining folds form the support set with true labels. This aligns the roles of training and test embeddings, resulting in a comparable embedding space. We empirically show that this strategy significantly improves linear probing performance (Table 2) and yields embeddings that match TabPFN v2’s in-context performance.

In sum, our contribution lies not merely in observing separability but in identifying and addressing a fundamental inconsistency in naive embedding extraction that was not discussed in prior work.

Q4: Extensions may overlap with existing methods (e.g., Random Forest, TabDPT)

We appreciate your comment, and we emphasize the differences and our contributions as follows. First of all, the purpose of section 7 is to demonstrate that the limitations of TabPFN v2 (Lines 13 - 15 and 65 - 71) can be effectively addressed without re-training the model, reminiscent of test-time scaling. In our humble opinion, our key novelty and contribution lie in pointing out and demonstrating this concept, not the particular methods we applied. Indeed, we deliberately choose not to propose a completely new mechanism but borrow and adapt existing mechanisms originally proposed in different contexts to TabPFN v2.

• Large-scale datasets:

The decision tree integration in TabPFN v2’s original paper is used for small-and medium-scale datasets suffering from within-dataset heterogeneity. We adapt this approach for a different purpose—handling large-scale datasets by partitioning them into smaller subsets where TabPFN v2 can operate effectively.

• Many-class classification:

We have provided the detailed implementation and the difference between ours and TabDPT (denoted as PFN-DPT in our paper) in Appendix C lines 92-99. While decimal encoding (TabDPT) is an existing idea, our TabPFN v2∗ improves robustness by randomly permuting class-to-digit mappings √C times and ensembling predictions. This mitigates artificial correlations introduced by fixed decimal encodings. We ensure fair comparison by matching the number of inferences of TabPFN v2 when ensembling their results in different methods.

• High-dimensional datasets:

We adapt Random Forest-style feature subsampling and bagging specifically for TabPFN v2 (Section 7.1), a direction unexplored in the original paper.

In light of our further clarification, we would appreciate it if the reviewer would reconsider the rating. Thank you.

Dear Reviewer,

Thank you for taking the time to review our submission. We sincerely appreciate your feedback and have done our best to address your comments and concerns in the author response.As the discussion period is ongoing, we would like to kindly check in to see if you had a chance to review our rebuttal and whether you have any further questions or require clarifications. We are happy to provide additional information or engage in discussion to help improve the quality and clarity of the paper.

Thank you again for your time and effort in the review process.

Best regards,

Authors

We sincerely thank Reviewer vnsZ for the thoughtful and constructive feedback. We appreciate your recognition of the value in analyzing and understanding tabular foundation models, particularly in clarifying the internal mechanisms of TabPFN v2. We address your specific questions and comments below:

Q1: Attribute relationships and attention maps

This is a very good question. In our humble opinion, understanding the inner workings of transformers—especially in the tabular context—remains an open and actively evolving research area, encompassing topics such as interpretability, explainability, and mechanistic understanding. In this work, we primarily rely on the most widely adopted approach: analyzing attention maps. While this provides some qualitative insights into how TabPFN v2 models feature interactions, we agree that attention alone may not fully capture the model’s reasoning process.

We plan to explore more advanced interpretability techniques in future work. Potential directions include probing-based analyses, attribution methods (e.g., SHAP adapted for in-context learning), and layer-wise relevance propagation. However, these approaches are still under development for transformer-based tabular models, and challenges remain in defining reliable ground truth for feature importance and interpreting high-dimensional interactions in models trained across diverse datasets. We appreciate the reviewer’s encouragement to pursue these avenues.

Q2: Using the same context and query without partitioning

Indeed, we experimented with this variant where the context contains all training data, and the query set includes both training and test points. This approach improves upon the vanilla feature extraction baseline but still underperforms compared to our proposed leave-one-fold-out method. We refer to this as the “non-partitioned” strategy, which suffers from role ambiguity—query points that appear in the support set (with dummy or ground-truth labels) are treated inconsistently. The network may not fully differentiate between training and test roles, degrading feature consistency. The performance ranking (lower is better) on classification tasks is as follows:

Q3: Does the naive strategy improve in later layers?

Yes, embeddings from the vanilla extraction do become smoother and more clustered in deeper layers. However, their quality, as measured by downstream linear probing, still lags behind both TabPFN’s native outputs and our leave-one-fold-out embeddings. This suggests that deeper layers alone cannot fully mitigate the role ambiguity caused by naive embedding extraction.

Q4: Are the ensemble sizes in Figure 5b controlled?

Yes. For a fair comparison in many-class classification (Figure 5b), all methods, including TabPFN v2*, TabPFN v2-DPT, and our ensemble extensions, use the same number of ensemble members. For instance, TabPFN v2* uses √C ensembles, where C is the number of classes.

Q5: Is “divide-and-conquer” a fair term for 5a?

Thank you for pointing this out. While our method does not follow a recursive divide-and-conquer paradigm, we use the term more broadly to describe the process of decomposing complex problems (e.g., many-class classification) into simpler subproblems (e.g., binary digit classification), inspired by ideas in test-time strategies and Monte Carlo ensembling. We agree that clarification is needed and will revise the terminology in the final version.

Q6: Is compute controlled in 5c (TabPFN v2-DF)?

Yes, we carefully controlled compute budgets to ensure a fair comparison across all methods. Specifically, we set the number of ensembles to four for each method in the large-scale classification setting, including our PFN*, TabICL, PFN-Bagging, matching the cost of four forward passes of TabPFN v2. We will clarify this further in the final version of the paper.

Below, we report a subset of average rank results across large-scale classification datasets, demonstrating that our proposed PFN* remains competitive under controlled compute:

These results confirm that PFN* achieves strong performance even with strict compute constraints. We sincerely thank the reviewer for raising this point and helping improve the clarity and rigor of our experimental design.

In light of our further clarification, we would appreciate it if the reviewer would consider increasing the rating. We would be happy to address additional questions you may have in the author-reviewer discussion phase.

Dear Reviewer,

Thank you for taking the time to review our submission. We sincerely appreciate your feedback and have done our best to address your comments and concerns in the author response.As the discussion period is ongoing, we would like to kindly check in to see if you had a chance to review our rebuttal and whether you have any further questions or require clarifications. We are happy to provide additional information or engage in discussion to help improve the quality and clarity of the paper.

Thank you again for your time and effort in the review process.

Best regards,

Authors

We sincerely thank Reviewer wWLo for the thoughtful and constructive feedback. We address each point below:

Q1: Comparison with TabICL and internalized attribute token learning

Thank you for the suggestion. We were aware of TabICL and have cited it [35], and we will extend the discussion. Both TabPFN and TabICL leverage in-context learning for tabular prediction, but they adopt distinct approaches to attribute tokenization. (Even TabPFN v1 and v2 adopt quite different tokenization mechanisms; see Lines 90 - 93 & sect. 3.) TabPFN v2 uses randomized attribute tokens (RTs), re-sampled per inference, which internalize feature relationships without requiring semantic alignment (Lines 207 - 214). In contrast, TabICL employs set transformers to encode distributional properties of an attribute, combined with RoPE for positional differentiation. While TabICL requires specifically learning distribution-aware tokens, TabPFN v2’s RTs enable plug-and-play generalization across heterogeneous tasks. We will expand this comparison in the final version to highlight the conceptual and practical implications of each approach.

Q2: Regression results and extensions for feature reuse

We have extended our feature extraction strategy to regression tasks. The embeddings generated via our leave-one-fold-out method show simplified and structured distributions, similar to the classification case. (Figures will be included in the final version.) To assess their utility, we trained linear regression (LR) on raw features, vanilla embeddings (c.f. section 6), and our leave-one-fold-out extracted embeddings. The average rank across regression datasets is summarized below:

Our embeddings lead to better regression generalization than raw features and vanilla TabPFN features. These results confirm that the extracted embeddings can serve as high-quality, reusable features.

Q3: Integration with domain adaptation methods

We agree this is a promising direction. Our feature extraction method offers a way to generate robust embeddings without retraining, which could complement domain adaptation techniques and enable cross-domain generalization. We will highlight this in future discussions.

Q4: Fine-tuning TabPFNv2

Thank you for the question. We note that a recent concurrent study ([Rubachev et al., 2025]) explores fine-tuning TabPFN v2 and demonstrates improvements using simple full finetuning. In our analysis (Table 2), we observe that combining embeddings from multiple layers enables strong linear probing performance—outperforming the original TabPFN output. This suggests that probing can serve as a lightweight, interpretable alternative to fine-tuning.

However, fine-tuning TabPFN v2 remains challenging for high-dimensional or many-class tasks due to (a) the scarcity of high-dimensional datasets and (b) the architectural constraint that TabPFN v2 is designed as a 10-class output format. To adapt it to many-class problems, one must jointly fine-tune both the label encoding scheme and the output head—adding substantial complexity and reducing generality. In contrast, our post-hoc divide-and-conquer strategies handle these challenges effectively without retraining, aligning with recent trends in test-time scaling in LLMs.

For large-scale datasets, fine-tuning is an effective choice given enough training time. Due to rebuttal time constraints, we fine-tuned TabPFN v2 on two large-scale datasets and compared it with our post-hoc divide-and-conquer strategies.

These results show that while fine-tuning can yield marginal performance gains, it comes with a significant computational cost. In contrast, our PFN v2*-DF approach achieves competitive performance with substantially lower overhead by taking advantage of its "divide-and-conquer" strategy, offering a practical alternative for scaling TabPFN v2.

Q5: Comparison with TabICL on many-class classification

We followed the setup in Figure 5 (middle) to evaluate TabICL on multi-class datasets (>10 classes). The average accuracy of TabICL is 0.6901, while our method achieves 0.7093. These results suggest our strategy scales more effectively to complex label spaces. We will extend this discussion in the final version.

Q6: Comparison with TabICL on large-scale datasets (>10K samples)

Using the configuration in Figure 5 (right), we report the average ranks on the large-scale classification datasets:

• PFNv2*-DF: 4.50 (with 4 ensemble models)
• TabICL: 6.25 (with 4 ensemble models)

Our method demonstrates stronger generalization in large-scale settings, highlighting its scalability advantages. We will extend this discussion in the final version.

In light of our further clarification, we would appreciate it if the reviewer would consider increasing the rating. We would be happy to address additional questions you may have in the author-reviewer discussion phase.

Dear Reviewer,

Thank you for taking the time to review our submission. We sincerely appreciate your feedback and have done our best to address your comments and concerns in the author response.As the discussion period is ongoing, we would like to kindly check in to see if you had a chance to review our rebuttal and whether you have any further questions or require clarifications. We are happy to provide additional information or engage in discussion to help improve the quality and clarity of the paper.

Thank you again for your time and effort in the review process.

Best regards,

Authors

Thank you for the thoughtful suggestion and for your encouraging feedback. Following your advice, we replaced the original Linear Regression with RidgeCV from scikit-learn for regression tasks based on the extracted features. We are pleased to report that this regularization indeed yields notable improvements (for all feature extraction methods). More importantly, using our feature extraction method still yields the lowest error, suggesting its consistent effectiveness. Below are the updated average ranks across benchmark datasets:

We will incorporate these updated results into the final version of the paper. Once again, thank you for your valuable input—it has helped strengthen the overall quality of our work.