We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer finds (1) the paper covers an important problem, (2) the experiments are comprehensive, and (3) the paper is well-written. We address the reviewer’s concerns as follows.

Reasons of performance gains after ensemble and fine-tuning.

We explain below why Mitra shows performance gains after ensemble and fine-tuning, which is a direct consequence of our design choices. Our central goal is to build a foundation optimized for maximum adaptability, providing the most practical path to state-of-the-art performance with efficient adaptation. Mitra’s ICL performance is comparable to TabICL [1] (concurrent work) as shown in Table 2, and by fine-tuning from a strong and efficiently adaptable foundation, Mitra ultimately achieves the state-of-the-art performance.

1. Performance gap re: ICL vs ensemble + fine-tuning: We acknowledge the performance gap between ICL and ensemble. We attribute this to (a) Mitra’s pretraining uses fewer features, and (b) TabPFNv2’s use of comprehensive feature preprocessing such as quantile transform and SVD that diversifies its ensemble members. Preliminary attempts to naively increase Mitra’s pretraining features or adopt TabPFNv2’s preprocessing did not yield gains out-of-the-box, highlighting a complex interplay between prior mixture, architecture, and training dynamics. We leave a systematic exploration of this as future work. Note that our primary focus is getting the best performance with minimal adaptation, to that end, Mitra sets a new state-of-the-art with fast fine-tuning, significantly outperforming TabPFNv2 (+e/+ef) by ~30-50 Elo in Table 2 with std only +4/-4 and ~100 over the top ICL performance.
2. Performance gap due to more efficient feature space:
   1. Mitra was pre-trained with a maximum of 16 features for pretraining computational efficiency, whereas TabPFNv2 was pre-trained with an order of magnitude more features (up to 160 features). Despite this discrepancy, Mitra achieves ICL performance comparable to TabPFNv2 on downstream datasets with up to 100 features. This demonstrates the strong generalization ability and sample efficiency of our proposed prior mixture, allowing the model to learn more robust patterns that generalize better to unseen, larger feature spaces.
   2. To further validate that the performance gap is not an effect of our prior mixture, we evaluate Mitra and TabPFNv2 ICL on small datasets with <= 16 features and <= 2k rows. On these datasets, Mitra shows better ICL performance than TabPFNv2. There are concurrent papers on exploring architectural advances to scale up rows and features such as TabICL [1] and TabFlex [2]. Our focus in this work is on developing an effective mixture of priors, and we plan to integrate these scalable architectures as future work to further improve scalability.
   3. Mitra also has better sample efficiency as demonstrated in Section 4.4 and Table 5, making it highly effective in real-world scenarios with limited labeled data and an overall best choice across both low- and high-label regimes.

3. Pretraining on our priors enables effective fine-tuning: Our mixture of priors provides a critical foundation that enables effective fine-tuning. To assess the impact of pretraining, we compare Mitra (+ef) with an identical model trained directly on the downstream dataset without any pretraining (denoted as Mitra-no-pretrain (+ef)). For Mitra-no-pretrain (+ef), we conducted a hyper-parameter search over training epochs {50, 100, 200} and maximum learning rates {0.01, 0.001, 0.0001}, reporting its best performance as follows. Training from scratch performs significantly worse, highlighting the importance of pretraining on our proposed mixture of priors.

4. Diverse priors create a better model foundation for efficient fine-tuning: The synthetic priors, particularly tree-based ones, are designed to capture decision boundaries. As suggested by prior works (e.g., TabForestPFN [3], Attic [4]), these are not perfect replicas of real-world datasets but are ideal for creating a foundation that can be effectively adapted. The diversity of our mixture gives rise to a better model foundation than TabPFNv2, and the gap could not be closed by TabPFNv2 even after extensive fine-tuning, as noted by its fine-tuning performance in Table 2 and confirmed with its authors. More importantly, this aligns with what users care about: achieving the best predictive performance on their datasets efficiently. Mitra fine-tuning is not only highly effective but also computationally efficient, running ~3.5× faster than CatBoost and faster than RealMLP as shown in Figure 14, delivering state-of-the-art performance with minimal adaptation cost.
5. Additional results on TabArena also show Mitra’s SOTA performance: We have run additional comparison on a recent benchmark TabArena [5]. Due to character limit, we only show the top methods. The full table can be referred to in Response to Reviewer Uza4. We show that
   1. Mitra achieves Pareto efficiency both in training time and inference time, with TabPFNv2 (HPO + ensemble) being significantly slower.
   2. Mitra is the strongest single model, outperforming all methods even when they perform hyperparameter tuning for 200 iterations. Mitra is only outperformed once the hyperparameter configurations of TabPFNv2 are ensembled together. We are still working on constructing a search space for Mitra, and thus do not have a HPO and HPO + ensemble result at this time.
   3. Mitra performs favorably to all non-foundational methods, including TabM and RealMLP.

Evaluation consistency.

Table 3 evaluates 1D attention model. The TabForestPFN baseline does not offer native ensemble logic, and the TabPFNv1 baseline does not offer native fine-tuning logic. For these reasons, we report results under the capabilities available in the respective baselines to ensure a fair comparison. We present comparisons that are meaningful and applicable to each method, and are not hiding any results. We will make this distinction on the 1D vs. 2D models clearer in the final version.

Training details.

We discussed the dataset size, model size, epochs in Figure 5 and more details in Appendix B.3. We pretrain Mitra for 22,000 steps on 45 million synthetically generated datasets. Mitra is built on a Transformer with 12 layers, 512 embedding size and 4 attention heads. The resulting model contains 72M parameters. We will adjust the paper flow to introduce these pretraining details earlier before discussing fine-tuning and evaluation.

On the second-best prior.

We have explained and compared the second and the third best prior in Line 164 - Line 169. The major reason is that the third-best prior ET shows better diversity and distinctiveness compared to the second-best prior DSRF. We also revisited this in the ablation study section in Line 277 - Line 279. We will make this more explicit in the final version.

Prior importance color in Table 1.

We use a color gradient in Table 1 to visually indicate the relative quality of each prior. The best-performing priors are shown in the darkest green, while the worst-performing priors are shown in the darkest red. Specifically, for diversity score on the diagonal of the generalizability matrix, lower values are better, with the lowest being the darkest green and the highest being the darkest red. For performance vector scores, higher values are better, with the highest being the darkest green and the lowest being the darkest red. We will make this clearer in the final version.

[1] TabICL: A Tabular Foundation Model for In-Context Learning on Large Data

[2] TabFlex: Scaling Tabular Learning to Millions with Linear Attention

[3] Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers

[4] Attic: A New Architecture for Tabular In-Context Learning Transformers

[5] TabArena: TabArena: A Living Benchmark for Machine Learning on Tabular Data

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer finds (1) our benchmarks diverse, (2) our results achieve state-of-the-art performance, and (3) our paper well written. We address the reviewer’s concerns as follows.

Novelty.

We acknowledge that prior works such as TabForest and TabICL have used tree-based priors. However, these works adopt tree-based priors in a heuristic manner, without explaining why incorporating these priors are beneficial. We are the first to systematically study what makes a mixture of priors effective and identify three key factors (performance, diversity, and distinctiveness) for better generalization. In Table 1 and Table 6, we show that following our principled analysis, our proposed mixture of priors performs better than the priors used in TabForestPFN (SCM + DT) and TabICL (SCM + GB), which only contain 1 tree-based method.

On fine-tuning.

We thank the reviewer for the thoughtful feedback. We acknowledge that one motivation for TabFMs is to introduce a powerful ICL model, but it is not the only motivation or benefit. While ICL performance is crucial, a truly foundation model for practical use must also be an excellent starting point for efficient adaption. Our central goal is to build a foundation optimized for maximum adaptability, providing the most practical path to state-of-the-art performance with efficient adaptation. Mitra’s ICL performance is comparable to TabICL (concurrent work) as shown in Table 2, and by fine-tuning from a strong and efficiently adaptable foundation, Mitra ultimately achieves the best performance.

1. Performance gap re: ICL vs ensemble + fine-tuning: We acknowledge the performance gap between ICL and ensemble. We attribute this to (a) Mitra’s pretraining uses fewer features, and (b) TabPFNv2’s use of comprehensive feature preprocessing such as quantile transform and SVD that diversifies its ensemble members. Preliminary attempts to naively increase Mitra’s pretraining features or adopt TabPFNv2’s preprocessing did not yield gains out-of-the-box, highlighting a complex interplay between prior mixture, architecture, and training dynamics. We leave a systematic exploration of this as future work. Note that our primary focus is getting the best performance with minimal adaptation, to that end, Mitra sets a new state-of-the-art with fast fine-tuning, significantly outperforming TabPFNv2 (+e/+ef) by ~30-50 Elo in Table 2 with std only +4/-4 and ~100 over the top ICL performance.
2. Mitra challenges SOTA in ICL mode with fewer pretrained features:
   1. Mitra was pre-trained with <= 16 features for pretraining computational efficiency, whereas TabPFNv2 was pre-trained with <= 160 features. Despite this discrepancy, Mitra achieves ICL performance comparable to TabPFNv2 on downstream datasets with <= 100 features. This demonstrates the strong generalizability and sample efficiency of our prior mixture, allowing the model to learn more robust patterns that generalize better to unseen, larger feature spaces.
   2. To further validate that the performance gap is not an effect of our prior mixture, we evaluate Mitra and TabPFNv2 ICL on small datasets with <= 16 features and <= 2k rows. On these datasets, Mitra shows better ICL performance than TabPFNv2.
   3. Mitra also has better sample efficiency as demonstrated in Section 4.4 and Table 5, making it highly effective in real-world scenarios with limited labeled data and an overall best choice across both low- and high-label regimes.

3. Diverse priors create a better model foundation for efficient fine-tuning: The synthetic priors, particularly tree-based ones, are designed to capture decision boundaries. As suggested by prior works (e.g., TabForestPFN, Attic), these are not perfect replicas of real datasets but are ideal for creating a foundation that can be effectively adapted. The diversity of our mixture gives rise to a better model foundation than TabPFNv2, and the gap could not be closed by TabPFNv2 even after extensive fine-tuning, as noted by its fine-tuning performance in Table 2 and confirmed with its authors. More importantly, this aligns with what users care about: achieving the best predictive performance on their datasets efficiently. Mitra fine-tuning is not only highly effective but also efficient, running ~3.5× faster than CatBoost and faster than RealMLP as shown in Figure 14, delivering state-of-the-art performance with minimal adaptation cost.
4. Pretraining on our priors enables effective fine-tuning: Our mixture of priors provides a critical foundation that enables effective fine-tuning. To assess the impact of pretraining, we compare Mitra (+ef) with an identical model trained directly on the downstream dataset without any pretraining (denoted as Mitra-no-pretrain (+ef)). For Mitra-no-pretrain (+ef), we conducted a hyper-parameter search over training epochs {50, 100, 200} and maximum learning rates {0.01, 0.001, 0.0001}, reporting its best performance as follows. Training from scratch performs significantly worse, highlighting the importance of pretraining on our mixture of priors.

HPO for baselines and add TabM.

We have added results on a recent benchmark TabArena [5], where all baselines are performed with HPO, including TabM. We show that Mitra remains the state-of-the-art tabular models on TabArena. Due to character limit, we only show the top methods. The full table can be referred to in Response to Reviewer Uza4. Specifically,

1. Mitra achieves Pareto efficiency both in training time and inference time, with TabPFNv2 (HPO + ensemble) being significantly slower.
2. Mitra is the strongest single model, outperforming all methods even when they perform hyperparameter tuning for 200 iterations. Mitra is only outperformed once the hyperparameter configurations of TabPFNv2 are ensembled together. We are still working on constructing a search space for Mitra, and thus do not have a HPO and HPO + ensemble result at this time.
3. Mitra performs favorably to all non-foundational methods, including TabM and RealMLP.

About RF prior.

As we explained in Line 279 to Line 282, while RF shows good diversity and distinctiveness, it also shows the worst performance on real datasets, so it is not as beneficial in the mixture. We will explain this more clearly in the final version.

Code availability.

The fine-tuning and inference code is already publicly available (at the time of submission the code was going through our institutional legal review as we mentioned in the checklist). However, given the rebuttal policy we are not allowed to provide an anonymous external link. We will ensure the code repository is included and accessible in the final version. Our pretraining code is currently under institutional legal review process, and we plan to release it once this review is complete.

The reason of training with 16 features instead of 100 features.

Mitra was pre-trained to maximize fine-tuning performance with larger model capacity compared to TabPFNv2. Given computational constraints scaling up to larger features during pretraining requires smaller batch size and eventually leads to worse training efficiency and performance. There are concurrent papers on exploring architectural advances to scale up rows and features such as TabICL and TabFlex. Our focus in this work is on design of effective mixture of priors, and we plan to integrate these scalable architectures as future work to further improve scalability.

Related work.

We will revise the related work section to better cover deep learning methods and add references [1][2][3][4] in the final version.

Prior ratio for TBP.

As stated in Line 193, we evenly split the ratios among tree-based priors, each taking 10%. We will explain this more explicitly in the final version.

Typo.

We will revise “all three datasets” to “all three benchmarks” in the final version.

[5] TabArena: A Living Benchmark for Machine Learning on Tabular Data

I would like to thank the authors for the reply. I have read the other reviews from the fellow reviewers and the authors rebuttal to all of them. I observed that additional reviewers have raised similar concerns to mine. Below are my answers to the author's rebuttal:

• Regarding the RF prior:

  The authors did not answer my concern regarding the RF prior. The point of the criteria and one of the major contributions of the work is on how to build an informative prior mixture. The authors deviate from the criteria with the argument that the prior did not improve the performance, if that is the case, then every component can be investigated in the same manner and one can skip the criteria completely. We can just observe the inference performance on the test benchmark.

• Regarding training on less than 16 features:

  I am not really fond of the reply from the authors that training on less than 16 features improved training efficiency. Since TabPFN trained on around 100 features and achieves better results compared to Mitra from a non-fine tuned perspective, this should have been investigated. Usually one starts from a prior common practice and then based on the performance modifies from that point. The authors of TabICL additionally train on more features gradually, so I am surprised on this design choice. The authors use this point as an argument for the lesser in-context learning performance, however, the authors themselves deviated from this prior established procedure.

• Regarding the fine-tuning time:

  I observed the results in the Appendix related to the running time of Mitra, however, the authors run Mitra on 8 A100 GPUs and advocate that the proposed method is 3.5x faster than CatBoost and RealMLP. That is a significant computational resource, how is that resource being used by gradient-boosted decision tree methods and by the simpler neural networks? The comparison is not fair and missleading. Providing results on a single GPU would have been more informative.

We sincerely thank the reviewer for the continued engagement and for the thoughtful follow-up comments. We address the reviewer’s questions as follows.

RF prior.

Our prior analysis in Table 1 serves as an in-depth study to provide insights into what makes effective priors, rather than a deterministic algorithm for selecting the mixture. Our proposed three key characteristics—performance, diversity, and distinctiveness—are further validated through our extensive ablation study that shows that the priors contributing most to the mixture align with this analysis. For example as we discussed on Lines 285 - 291, combining ET with SCM significantly boosts performance, yielding an Elo improvement of 63. Similarly, we see that adding GB to the mixture further improves the performance. Adding the remaining priors DT, DSRF, RF leads to a few configurations with similarly good performance on real datasets, aligned with our findings in Table 1 that they are less important due to either lower performance on real datasets or higher diagonal or off-diagonal values. Importantly, this prior analysis part is not a part of the actual pretraining process itself. Compared with previous works such as TabForestPFN which uses SCM + DT and TabICL which uses SCM + GB, we are the first to perform such a comprehensive study on prior importance which explains why our mixture of priors empirically achieve very strong performance. We will further clarify this in the final version. We agree that developing a deterministic algorithm is an interesting direction for future work, and we believe our study opens the door to such developments.

Regarding training on less than 16 features.

We did not include these preliminary findings in the paper as they are only initial observations and not to the level of a comprehensive ablation study. We present below our preliminary investigation of scaling Mitra to a larger number of features for ICL, and we can add these results to the Appendix to help better guide future work and help motivate our design choice. We explored two approaches:

1. Curriculum learning (TabICL-style): We continued pre-training from the current Mitra checkpoint while increasing the feature size up to 160 (same pre-training size as TabPFNv2). To accommodate the higher dimensionality, we reduced the batch size from 64 to 6 per device. We trained for 500 steps, saving checkpoints every 100 steps, and reported their ICL performance respectively, denoted by Curriculum (Step 100), Curriculum (Step 200), Curriculum (Step 300), Curriculum (Step 400) and Curriculum (Step 500) in the following table.
2. Feature grouping (TabPFNv2-style): We grouped features (2, 3, or 4 features per group) and learned a feature embedding for each group, scaling the input to 32, 48, and 64 features respectively, denoted by Group2 (Features 32), Group3 (Features 48), and Group4 (Features 64) in the following table. We pre-trained for the same number of 22,000 steps as Mitra and reported their ICL performance respectively as below.

In this preliminary study, we already observed a performance drop in ICL performance and therefore did not proceed to evaluate fine-tuning performance at that time. We are currently running the fine-tuning experiments and will report the results once finished (within the discussion period).

Based on these findings we chose to pre-train Mitra on 16 or fewer features. While pre-training with more features could potentially yield improvements, it is not something we have investigated in detail at this time. Empirically, the current design led to strong results, and we anticipate further exploration in this area to lead to even better results.

We provide response to fine-tuning time in the following comment.

I would like to thank the authors for engaging in an insightful discussion.

• RF prior:

  I think there was room for further investigation from the generalization and performance matrices that the authors generated. However, it is true that so far this is among the most detailed investigations so far in the domain regarding the mixture of priors.

• Pre-training on more than 16 features:

  I thank the authors for the additional information. I believe this information is interesting for future works and does provide validation to the argument from the authors.

• Regarding the training/inference times:

  I am glad the authors do not shy away from listing the cons of the proposed method. I am in favor of editing the runtime information to reflect a fair comparison between all methods. I additionally think the provided information relating to the train/inference times would be useful to include in the appendix.

Based on the provided information, I believe the proposed work is of interest to the community, given that it provides the prior and a more-detailed investigation. Most of my concerns are addressed and I will recommend acceptance with the condition that the authors open-source the implementation, model weights and prior. It would be great if the authors acknowledged the last point.

We sincerely thank the reviewer for the valuable feedback.

We additionally provide the ranking and aggregated results focusing only on Mitra (+ef) and Curriculum (Step 400) (+ef) as follows. Note that ranking-based metrics may not fully reflect the differences when comparing only two models, particularly when they exhibit minimal performance differences. To illustrate this, we simulated a scenario where model A outperforms model B by only 0.00001 on 60% of 660 datasets and underperforms model B by 0.00001 on the remaining 40% datasets. Although the differences are negligible, model A leads model B in Elo by a significant margin of 100. This is why we used a larger set of baselines with diverse performance to help provide a more meaningful comparison for ranking-based metrics. For reference, we also provide the aggregate metrics which confirm that the overall aggregated results are very close.

Aggregated results:

Regarding the prior release, we strongly believe in the importance of open source for the benefit of the community. We are fully committed to this principle and are actively working through the institutional approval process.

We thank the reviewer again for the support and thoughtful feedback!

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer finds the paper (1) comprehensive and systematic in its evaluation; (2) robust in baseline comparisons; (3) broad and insightful in its analysis; and (4) clear in motivation and practical impact and well-written. We address the reviewer’s concerns as follows.

Running time of post-training techniques.

While the running time of Mitra with ensemble and fine-tuning is higher than TabPFNv2 and Mitra in ICL mode, it is still reasonable and practical for tabular prediction models. For instance, it is ~3.5 times faster than CatBoost and also faster than RealMLP as shown in Figure 14. Mitra is targeted especially for smaller tabular datasets which makes the practical running time fast. In addition, Mitra (bagging) is faster than TabPFNv2 with post-hoc ensemble, while achieving the best performance with advanced ensemble techniques (Figure 3).

On performance improvement.

We added a new experiment showing that Mitra offers substantial rather than marginal performance gains on a recent tabular benchmark TabArena [1]. We show that

1. Mitra achieves Pareto efficiency both in training time and inference time, with TabPFNv2 (HPO + ensemble) being significantly slower.
2. Mitra is the strongest single model, outperforming all methods even when they perform hyperparameter tuning for 200 iterations. Mitra is only outperformed once the hyperparameter configurations of TabPFNv2 are ensembled together. We are still working on constructing a search space for Mitra, and thus do not have a HPO and HPO + ensemble result at this time.
3. Mitra performs favorably to all non-foundational methods, including TabM and RealMLP.

Evaluate under class imbalance settings.

To evaluate robustness to class imbalance, we computed the imbalance ratio (max class : min class) across TabRepo datasets and selected 14 datasets with a ratio > 10 as imbalanced datasets. Mitra remains state-of-the-art performance on the class-imbalanced datasets, as shown below, for both aggregate and ranking metrics.

Model specification.

1. Categorical feature encoding: We use ordinal encoding for categorical features.
2. Normalization: In Appendix B.3.2, we mentioned min-max normalization. We would like to clarify that this applies only to regression targets and not to input features. We will make this clear in the final version. To normalize features, we apply uniform quantile transform based on the support set, followed by standard normalization based on the mean and standard deviation from the support set. To normalize targets for regression task, we use min-max normalization based on the support set. The normalization is consistently applied for both pretraining datasets and inference datasets.
3. Likelihood: We will clarify the likelihood formulation in the final version: the population version of the likelihood takes a form of an expectation over the table $D$, with each table sampled from the prior mixture. Given the sampled table, the model assumes $q$ query rows are conditionally independent given the in-context examples, so that the log-likelihood within the expectation is decomposed into a sum of $q$ individual terms, one per query row. When the query label corresponds to a classification task, its (conditional) distribution is assumed to be categorical, making the training objective equivalent to minimizing the cross-entropy loss. When the query label corresponds to a regression task, its (conditional) distribution is assumed to be Gaussian and the training objective corresponds to minimizing the Mean Squared Error (MSE) loss.

Large values in test data.

As described in model specification part, to normalize features, we apply uniform quantile transform based on the support set, followed by standard normalization based on the mean and standard deviation from the support set. To normalize targets for regression task, we use min-max normalization based on the support set. The normalization is consistently applied for both pretraining datasets and inference datasets.

We observed that the model is able to extrapolate and generate predictions beyond [0, 1] for regression predictions. For example, on Tabrepo regression 167210 across 10 folds, 6 entries are out of distribution. Mitra successfully extrapolates 4 of these 6 entries, achieving a 67% success rate, with a mean absolute error of 0.03 and mean squared error of 0.001 in the normalized space. We leave other preprocessing techniques for improved extrapolation as future work.

Up to how many rows and features to scale.

There are concurrent papers on exploring architectural advances to scale up rows and features such as TabICL [2] and TabFlex [3]. Our focus in this work is on design of effective mixture of priors, and we plan to integrate these scalable architectures as future work to further improve scalability.

Revise figure format.

Thank the reviewers for the comment. We will reformat Figure 3, 4, 5 to make them larger and include only the zoomed in versions in the final paper.

[1] TabArena: A Living Benchmark for Machine Learning on Tabular Data

[2] TabICL: A Tabular Foundation Model for In-Context Learning on Large Data

[3] TabFlex: Scaling Tabular Learning to Millions with Linear Attention

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer finds (1) the proposed criteria for prior selection innovative and conceptually sound; (2) the procedure to derive the mixture novel; (3) the empirical evaluation extensive with detailed ablation study; (3) the paper very well written and easy to follow. We address the reviewer’s concerns as follows.

Mitra’s ICL performance relative to TabPFNv2 does not undermine the quality of our proposed prior mixture but reflects our design choices.

We clarify below why Mitra’s ICL performance relative to TabPFNv2 reflects our design choices rather than a limitation of the prior mixture. Our central goal is to build a foundation optimized for maximum adaptability, providing the most practical path to state-of-the-art performance with efficient adaptation. Mitra’s ICL performance is comparable to TabICL [1] (concurrent work) as shown in Table 2, and by fine-tuning from a strong and efficiently adaptable foundation, Mitra ultimately achieves the state-of-the-art performance.

1. Performance gap re: ICL vs ensemble + fine-tuning: We acknowledge the performance gap between ICL and ensemble. We attribute this to (a) Mitra’s pretraining uses fewer features, and (b) TabPFNv2’s use of comprehensive feature preprocessing such as quantile transform and SVD that diversifies its ensemble members. Preliminary attempts to naively increase Mitra’s pretraining features or adopt TabPFNv2’s preprocessing did not yield gains out-of-the-box, highlighting a complex interplay between prior mixture, architecture, and training dynamics. We leave a systematic exploration of this as future work. Note that our primary focus is getting the best performance with minimal adaptation, to that end, Mitra sets a new state-of-the-art with fast fine-tuning, significantly outperforming TabPFNv2 (+e/+ef) by ~30-50 Elo in Table 2 with std only +4/-4 and ~100 over the top ICL performance.
2. Performance gap due to more efficient feature space:
   1. Mitra was pre-trained with a maximum of 16 features for pretraining computational efficiency, whereas TabPFNv2 was pre-trained with an order of magnitude more features (up to 160 features). Despite this discrepancy, Mitra achieves ICL performance comparable to TabPFNv2 on downstream datasets with up to 100 features. This demonstrates the strong generalization ability and sample efficiency of our proposed prior mixture, allowing the model to learn more robust patterns that generalize better to unseen, larger feature spaces.
   2. To further validate that the performance gap is not an effect of our prior mixture, we evaluate Mitra and TabPFNv2 ICL on small datasets with <= 16 features and <= 2k rows. On these datasets, Mitra shows better ICL performance than TabPFNv2. There are concurrent papers on exploring architectural advances to scale up rows and features such as TabICL [1] and TabFlex [2]. Our focus in this work is on developing an effective mixture of priors, and we plan to integrate these scalable architectures as future work to further improve scalability.
   3. Mitra also has better sample efficiency as demonstrated in Section 4.4 and Table 5, making it highly effective in real-world scenarios with limited labeled data and an overall best choice across both low- and high-label regimes.

3. Pretraining on our priors enables effective fine-tuning: Our mixture of priors provides a critical foundation that enables effective fine-tuning. To assess the impact of pretraining, we compare Mitra (+ef) with an identical model trained directly on the downstream dataset without any pretraining (denoted as Mitra-no-pretrain (+ef)). For Mitra-no-pretrain (+ef), we conducted a hyper-parameter search over training epochs {50, 100, 200} and maximum learning rates {0.01, 0.001, 0.0001}, reporting its best performance as follows. Training from scratch performs significantly worse, highlighting the importance of pretraining on our proposed mixture of priors.

4. Diverse priors create a better model foundation for efficient fine-tuning: The synthetic priors, particularly tree-based ones, are designed to capture decision boundaries. As suggested by prior works (e.g., TabForestPFN [3], Attic [4]), these are not perfect replicas of real-world datasets but are ideal for creating a foundation that can be effectively adapted. The diversity of our mixture gives rise to a better model foundation than TabPFNv2, and the gap could not be closed by TabPFNv2 even after extensive fine-tuning, as noted by its fine-tuning performance in Table 2 and confirmed with its authors. More importantly, this aligns with what users care about: achieving the best predictive performance on their datasets efficiently. Mitra fine-tuning is not only highly effective but also computationally efficient, running ~3.5× faster than CatBoost and faster than RealMLP as shown in Figure 14, delivering state-of-the-art performance with minimal adaptation cost.
5. Additional results on TabArena also show Mitra’s SOTA performance: We have run additional comparison on a recent benchmark TabArena [5] and show that
   1. Mitra achieves Pareto efficiency both in training time and inference time, with TabPFNv2 (HPO + ensemble) being significantly slower.
   2. Mitra is the strongest single model, outperforming all methods even when they perform hyperparameter tuning for 200 iterations. Mitra is only outperformed once the hyperparameter configurations of TabPFNv2 are ensembled together. We are still working on constructing a search space for Mitra, and thus do not have a HPO and HPO + ensemble result at this time.
   3. Mitra performs favorably to all non-foundational methods, including TabM and RealMLP.

Computational complexity and stability for prior analysis.

1. Computational complexity: Mitra’s pretraining cost is comparable to TabPFNv2, taking around 60 hours on 8 Nvidia A100 GPUs. Our prior analysis (Table 1) and ablation study (Table 6) are used for analyzing the prior importance in our mixture of priors in Mitra and demonstrating the robustness of our principled method, and they are not part of the actual pretraining process itself. Compared with prior works such as TabICL [1], we are the first to perform such rigorous study on prior importance.
2. Stability: Mitra and its ablations are stable across runs. For example, the average AUC differences is within 0.003 across different restarts.

[1] TabICL: A Tabular Foundation Model for In-Context Learning on Large Data

[2] TabFlex: Scaling Tabular Learning to Millions with Linear Attention

[3] Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers

[4] Attic: A New Architecture for Tabular In-Context Learning Transformers

[5] TabArena: TabArena: A Living Benchmark for Machine Learning on Tabular Data