Attic: A New Architecture for Tabular In-Context Learning Transformers

Dear Reviewer,

Thank you for your review. In general, we followed the experimental setup of TabForestPFN and only reported differences in our setup, which might have lead to a lack of details in some parts of our paper. We would like to clarify any remaining points here.

Can the authors provide values instead of variables for architecture details: Token dictionary size L, what is the value of k, dimension sizes, what values of did you test?

All architectural details are presented in Table 1. The model has 12 layers, each with a dimension of 512. We kept the same dimensions as Tab(Forest)PFN and did not test other values.

How is the tokenization performed for each observation x feature?

Each observation × feature passes through the same linear layer, which has dimension 1x512.

What are the model sizes for the -M and -L versions of TabForestPFN?

These details are also presented in Table 1. The -M version has 12 layers with a dimension of 512, and the -L version has 24 layers with a dimension of 2048.

You claim in L153: “We changed this because we believe this formulation is more natural, but performance-wise, it has little impact.” Please provide an ablation describing and proving this claim.

When digging through the embedding of TabPFN, we found two unfamiliar design choices. First of all, TabPFN embeds the target label $y \in \cal{N}$ with a linear layer, as if it were numerical. Secondly, TabPFN uses feature count scaling: it multiplies all $x$ values with $\frac{100}{k}$, where $k$ is the number of features in $x$. In contrast, Attic does not feature count scale and embeds $y$ with a natural embedding layer for each class as is done in large language models.

As requested, below is the ablation. The table corresponds to Table 3 in our paper. We change the embedding of TabForestPFN to the Attic embedding without feature count scaling. Due to the inclusion of the new model, the normalized accuracies for the original TabForestPFN and Attic are slightly different from Table 3. All in all, we consider the change of embedding insignificant.

The authors need to highlight more clearly for the reader that this work and its findings are limited to datasets with 10 or fewer classes.

We will explicitly add this information to Table 1.

Do the authors re-run TabPFN for the Tabzilla benchmark? or do they take the TabPFN results from the Tabzilla paper/results?

We re-ran TabPFN for the TabZilla benchmark. The TabZilla authors ran TabPFN on only 57 out of 94 datasets (those with fewer than 1,250 observations). In contrast, we ran TabPFN on all 94 datasets, including those with more than 1,250 observations.

Dear Reviewer,

Thank you for your review. We agree with the strengths you presented and will address the questions raised below.

Weaknesses

However, when looking at Table 2 TabForestPFN is only marginally worse than Attic (ZeroShot) but much more cost effective due to the more abstract representation.

We do not believe the performance difference is marginal. The median rank improves from 10.0 to 6.2, and the average accuracy improves from 80.9% to 83.4% across 94 datasets. This difference is significant; see the answer to the question about significance scores.

Moreover, there are some points in the paper that are unclear to me. (Regarding TabZilla benchmark)

The TabZilla dataset is a benchmark consisting of 176 classification datasets; there are no regression datasets included. However, the authors of TabZilla use only 98 datasets because many algorithms take too long to run on larger datasets, leaving incomplete results. Therefore, although TabZilla presents 176 datasets, it is more accurate to say the benchmark effectively includes 98 datasets. Since all ICL-transformers are limited to multiclass classification with a maximum of 10 classes, we remove 4 additional datasets, resulting in 94 datasets in total. The number 28 in the table refers to the number of methods tested, not the number of datasets.

In terms of hyper-parameter optimization - what was the experimental setup?

We followed the experimental setups described in the benchmark papers (TabZilla and WhyTrees). For non-ICL transformers, training was performed on the training dataset, with early-stopping/validation on the validation dataset. The reported test accuracy corresponds to the test accuracy of the best validation run. For ICL-transformers, early-stopping for fine-tuning was performed on the validation set without hyperparamer search.

At the end of Section 3 it is stated: "We believe that this dependency on feature order leads to training inefficiencies. The cell-token ICL-transformer treats each feature the same and learns how to construct relationships between features." And while I agree with this intuition, why not perform an ablation study on it (e.g., shuffling columns and see if it impacts the performance)?

We do not believe an ablation study can effectively demonstrate this inefficiency. For inference, shuffling the features does indeed lead to different predictions for feature-variant models. However, the focus of our argument in section 3 is about pretraining. Both Tab(Forest)PFN and Attic are pretrained on synthetic data generators. Shuffling the features during pretraining would make no difference because the synthetic data generator already outputs an infinite stream of datasets with randomly ordered features. To show that the pretraining of Tab(Forest)PFN is inefficient due to the feature-variance, one approach is to switch the feature-variant module for a feature-invariant module and compare the performance. This approach effectively boils down to creating Attic.

Questions

Is it possible to show if the differences in performances shown in Table 2 are statistically significant?

We will add the critical difference diagram as done by Tabzilla to the appendix. Just like the Tabzilla authors, we use multiple Wilcoxon signed-rank tests with a Holm-Bonferroni correction on the log loss values. As a result, Finetuned Attic is significantly different from Finetuned TabForestPFN and Zero-Shot Attic is significantly different from Zero-Shot TabForestPFN.

Figure 4 is interesting - but isn't it the case that as the decision boundaries become more and more detailed, the generalization capability of the approach is reduced at the same time?

This is a great question, the answer is: not necessarily. If the decision boundaries of the model are more detailed than the “ground truth” decision boundaries, it indicates overfitting and harms the generalization performance. However, in Figure 4, the “ground truth” decision boundaries are inherently complex, so a good model should match that complexity. Figure 4 demonstrates that fine-tuned Attic is capable of creating complex decision boundaries when a specific dataset requires them, which in turn leads to better generalization (higher test accuracy). We want to show this figure because standard neural networks seem to struggle to create these complex decision boundaries.

In terms of computational complexity is there a way to show computational cost (e.g., as a new column in Table 2)?

We report run times in Appendix A.4, which we consider the appropriate metric for computational cost. Using FLOPS would be unfair, as some models use FlashAttention while others do not. Similarly, memory costs and token counts are incomplete metrics because datasets that do not fit into memory can still be processed using a limited context size.

If there is anything else that remains unclear, please do not hesitate to ask further.

Dear Reviewer,

Thank you for your review. To address your last question, yes, all the code will be open-source. This includes everything: preprocessing scripts, exact training settings, loss curves, and figure-drawing scripts.

Weaknesses

Overall, I think some more in-depth analysis into the runtime and memory tradeoffs Attic makes to achieve its superior performance can greatly benefit the paper.

While we do agree that efficiency is an interesting and relevant direction, we believe that high focus on maximum attainable accuracy is more important. In the development of large language models, the first objective was to make a model that achieves high quality, and after researchers were sufficiently pleased with the performance, the field largely shifted to efficiency.

We believe we should approach tabular neural networks in the same way. First focus on achieving maximum accuracy, and after we as a field are satisfied with the accuracy, we can use classical tools like quantization, pruning, distillation and efficient attention mechanisms to improve the fine-tuning and inference run time metrics.

Because Attic introduces a new dimension over the feature counts, I imagine the runtime costs greatly increase. How much does this increase with respect to the number of features?

For $n$ observations, $k$ features and model dimension $d$, the computation complexity of Attic is given by $O(n^2kd + nk^2d + nkd^2)$ due to the observation attention, feature attention and linear layers respectively. As often $n > k$, we expect the term $n^2kd$ to dominate. In contrast, the computation TabForestPFN scales $O(n^2d + nd^2)$. However, TabForestPFN is limited to a 100 features. If we want to pretrain TabForestPFN for 200 features for example, we likely need more model dimension space $d$ to encompass all features, so the complexity still indirectly depends on the number of features.

In Appendix A.4, we show the runtime for every single dataset in the benchmark for both Attic and TabForestPFN to provide a clearer picture of how the number of features affects runtime.

Similar to the previous question. How does the memory costs of Attic rise with respect to the number of features?

For the TabForestPFN, the attention mechanism has a memory footprint of $O(n^2d)$ and the activations have complexity $O(nd)$. As Attic switches to FlashAttention, which has linear memory cost scaling, the complexity of the activations and attention mechanisms are both $O(nkd)$. As often $n > k$, we argue the memory costs scaling of Attic is not inferior to that of TabForestPFN.

My understanding is TabPFN overcomes feature order invariance through ensembling across multiple feature shufflings. Could you discuss the tradeoff between using larger TabPFN ensembles and Attic, as both incur additional runtime costs?

Ensembling is a technique used in the zero-shot inference of TabPFN. Because the zero-shot inference of Tab(Forest)PFN is lightweight, ensembling is indeed a cost-effective way to mitigate feature order variance. However, this ensembling technique is less advantageous during fine-tuning. During fine-tuning, we keep the feature order fixed, so after fine-tuning, the model can only use the feature order it was trained with. Ensembling cannot be applied without re-fine-tuning the model for each ensemble member. Since fine-tuned Attic is, on average, 2× slower than fine-tuned TabPFN, creating large ensembles of fine-tuned TabPFN would be significantly slower than using Attic.

Intuitively, I feel larger feature count datasets would benefit the most from Attic. Empirically, are there trends on what specific datasets Attic most improves upon baseline algorithms?

We agree with this intuition, but the empirical trend is slightly different. Based on Figure 5, which plots the TabZilla benchmark datasets, the number of observations is actually more important than the number of features. Fine-tuned Attic outperforms fine-tuned TabForestPFN on all datasets with more than a thousand observations, even those with relatively low feature counts.

If there are any more questions, please don’t hesitate to ask.

Thank you for your rebuttal! I maintain that the rough runtime is an important metric to report. Even if it is not good, you could mention it in the Limitations section, along with TabPFN's feature count limitations. I maintain my score based on my original review and ATTIC's impressive performance results.

Dear Reviewer, thank you for you review. We generally agree with the strengths and weaknesses outlined by the reviewer. Below, we address a few specific weaknesses and respond to the questions raised.

Weaknesses

2. The authors note that Attic is significantly slower and more memory-intensive than TabForestPFN for datasets with many features. This scalability issue could limit Attic's practical applicability to large, high-dimensional datasets.

It is true that Attic’s scalability is limited with respect to high-dimensional datasets. However, we would like to point out that TabPFN and TabForestPFN are also significantly limited in handling high-dimensional datasets due to their acceptance of a fixed maximum number of features (maximum of 100). If we want to pretrain Tab(Forest)PFN from scratch to be able to use a larger number of features, we likely need a larger model dimension to properly represent a larger number of features. Therefore, Tab(Forest)PFN has a similar scalability issue.

5. More detailed analysis of failure cases where tree-based methods outperform Attic. Why, in datasets with less than 500 observations, does Attic underperform?

6. The paper does not thoroughly explore potential limitations or failure cases of Attic, which would provide a more balanced view of the method's capabilities.

While analyzing the failure cases of Attic would indeed be very interesting, it is also a challenging task. In Figures 5, 7 and 8 we can identify on which datasets Attic fails, but it is hard to analyse why it fails. With language and vision models, the output can be visualized or read for qualitative analysis, but with tabular data, it is hard to do any meaningful qualitative analysis due to the high dimensionality of the feature space. Many techniques like PCA or other statistical techniques often give limited insightful information, due to their dependence on linearity or other assumptions. If the reviewer has suggestions for suitable approaches or knows of relevant literature from tabular neural network papers that provide such analyses, we would greatly appreciate the guidance.

Questions

1. Any potential approaches to reduce memory and computational requirements of Attic?

We believe this will be an active area of future research. The answer also depends on which stage of the pipeline we aim to optimize for memory and computational efficiency. Pretraining will inherently have a high computational cost, but fine-tuning and inference could be improved using classic techniques such as quantization, pruning, distillation into a smaller network, or using more efficient attention mechanism. Additionally, caching the query-set observations for successive inferences could help reduce runtime.

2. How does Attic handle missing data or categorical variables? Are there any special preprocessing steps required?

Currently, there is no special handling of missing values implemented. If preprocessing encounters a missing value, it fills the value with the column mean. Since the benchmark datasets used in this study do not contain missing values, this approach does not affect the reported performance metrics. However, handling missing values could be implemented relatively straightforwardly by pretraining Attic with a special missing-value mask. Due to the lack of established tabular missing-value benchmarks, we leave this direction open for future work.

Categorical variables are not subject to any special preprocessing steps. Attic treats categorical variables as numerical variables, without applying one-hot encoding.

3. Can you elaborate on Attic's performance in regression tasks compared to classification?

We observed that pretraining is less stable during regression tasks compared to classification tasks. While pretraining under the cross-entropy loss behaves similarly to training large language models, pretraining under the mean-squared error loss occasionally exhibits extreme outliers, where the loss sometimes spikes to values is high as 10^8 and crashes regularly. We are uncertain why the mean-squared error loss behaves differently from the cross-entropy loss, but we included these regression results to remain transparent about the issue.

If there are any more questions, please don’t hesitate to ask.

We don't have an open-source code link. Quickly after the ICLR decision, TabPFNv2 came out which has the exact same architecture as us. Also, during the ICML rebuttal, we discovered a bug in the bfloat16 TabPFN baseline, and we lost access to our GPUs so we could not fix the paper. In the end, we decided not to put our paper on arxiv and to not give the code link for these reasons. If you are interested in the code of the architecture, please use the TabPFNv2 repo or dig in our code attached in the supplementary materials, as that code should fully work.