There are many limitations on the datasets requirements, e.g. number of features, number of classes, tasks

We agree. There are ways to go beyond them while still using the same architecture though. For instance one can perform feature selection very efficiently as the forward pass of TabPFN is very fast. Furthermore, all multiclass problems can always be reduced to binary classification problems. Finally even regression can be cast as classification as we demonstrated in the general message But we share your feelings and are currently investigating training a better base model which does not suffer from these limitations. This is however a different technical contribution on top on which the present method can be applied as well.

No other deep learning or transformer based models as a baseline

We address this point in the paragraph “Deep learning model comparisons” in Section 4.2 which points to Table 5 in appendix. Essentially deep learning models are usually slower to train and we could only obtain results on a subset of the datasets. We still report their performance and ours on those. We believe the contrast is very clear.

Sec 2.4 is not quite clear in finetuning steps, though the general ideas are appreciated.

This section is indeed the most technical part of the paper and we’d be happy to answer any specific question and update the main text. In the meantime to explain differently the main point: Let’s say we want to use a context of size N_ctx=1000 we wish to classify N_qy = 500. With vanilla TabPFN, we simply construct a sequence of size 1500 and tell the model which points are ctx and query through the attention mask. However when using a local context, now each of the 500 queries has its own 1000 context. We now have to fit $(1000+1) * 500$ points in the GPU! For doing inference this is fine, but when having to do backprop, and that over many steps, it just becomes too slow. What we do instead is find a way to select points which are all “local” to each other so that we can use a context within that neighborhood and share it for all points, even though it might not be the exact neighbors of each query. We find that this works very well and efficiently in practice.

L32, memory scales quadratically in the size of the dataset. Do the authors mean exactly the calculation of attention matrix? If so, it seems this memory limit s general and what the authors do is to leverage the limited context length better?

Yes, this is correct for the TabPFN-kNN method. Note that this poses some challenges on how to implement this efficiently but your understanding is correct.

In Table 1, is the gain from loCalPFN w.r.t TabPFN-kNN coming from only finetuning per dataset?

Yes this is correct.

Just make sure I understand correctly, do the authors fine tune on each dataset of 95 benchmarks?

Yes. (see L43, L132)

How are tree methods evaluated? Any hyperparameter optimization?

We use the results from TabZilla. All baselines used HPO indeed. Please refer to Appendix A.2.1 for more details.

What is the training time and inference time on each dataset and how this time scales with dataset size?

We provided some runtime results in the attached pdf. The main takeaway is that retrieval is not the bottleneck here, it is the cost of doing the finetuning.

What is the selection criteria of 95 datasets? How are they chosen, preprocessed, etc.

We tried to address this question in the first paragraph of the experiment section “The 95 datasets are filtered from TabZilla to meet TabPFN’s architectural requirements by ensuring that each dataset has at most 100 features, at most 10 classes, does not contain NaN values, and has at least one instance per class for each split”. These are essentially conditions so that the datasets can be used without modification by the base model, TabPFN. The list of datasets is available in Appendix A.1

Thank you very much for your thorough review and helping us improve the paper.

How are runtimes computed for figure 11? Which hardware? Which hyperparameters?

The time reported is the average time to perform training+inference for a single run (averaged over parameters/datasets/seeds). As such it doesn’t indeed account for the HPO for baseline methods which used one. As reported in Appendix A.2.2 the hardware is as follows: All experiments for our proposed methods can be run on a machine with a single NVIDIA RTX 6000 GPU Ada Generation, 995Gi RAM, and AMD Ryzen Threadripper PRO 5995WX 64-Cores CPU.

I'm not sure whether LocalPFN undergoes some HPO or not (I think not). If it's not the case, I think it should be made clearer, as it makes the comparison to tuned GBDT more impressive. If it does, the hyperparameter space should be provided.

We conducted some minor experiments with HPO and saw no difference except for learning rate which we tuned by hand. Thus, we kept the default parameters we had initially (from TabPFN repo). We believe this, on top of other choices not having much effect, such as using embedding vs raw space or euclidean distance vs inner product, shows the approach is quite robust. Indeed we will make this clear in the main text, thanks!

I think TabPFN-1k-32ens-int (maybe less than 32 ens if it's slow?) would be an interesting baselines as TabPFN is optimized for smaller datasets than 3K.

We do not have the exact numbers but generally TabPFN generalizes well to larger context and on average using larger context helps. The performance of 1k-32ens-int was above 1k-32ens but below 3k-32ens-int.

I think more aggregation metrics should be provided (for instance mean rank, mean normalized score (z-score or other rescaled scores)..).

We provide the mean rank, normalized AUC, z-scores for the algorithms of Table 1 in Table 3 of the attached pdf. Please let us know if there are some metrics you’d particularly wish to see and we’ll try to include them during the discussion phase. We made the choice to avoid normalized measures as the main metrics in our paper as we believe it can hinder the reproducibility of the results as the scores are now dependent on an exact set of algorithms. However we agree that they are also interesting and will include them in the paper.

figure 4 lacks details: which datasets are used? Are the different datasets subsampled to reach smaller sample sizes? I also don't understand why absolute mean AUC is decreasing when the dataset size is increased after a certain point in Figure 7. Is it because the list of datasets is changing?

Good question, we will clarify it in the main text. Your second guess is correct. In this figure we have not subsampled or tampered with the datasets in any way, but only binned them in the appropriate size range. This also means that the mean AUC of each bin is not directly comparable to another bin as the datasets within one bin are totally disjoint from the one in another bin. It may be that datasets in the range 1000-3000 for instance are on average (in Tabzilla) harder than those in the range 3000-10000. This is the main reason why we use a standard algorithm as a baseline so that some phenomena (such as TabPFN’s performance decreasing) with dataset size become visible.

The code is not currently available.

Following NeurIPS policy we are reaching out to the AC in order to be allowed to share an anonymous repo link with you. Note that we intended to release our code later (as we would like to see our method being used, we are particularly hopeful TabPFN-kNN could be used as a replacement for TabPFN), and it is currently not optimally refactored/documented.

For inference, you have to use a different context for each points to predict, right? How long / memory intensive is it? What is the runtime repartition between finetuning and inference?

TabPFN-kNN is essentially LocalPFN without finetuning. By looking at Figure 11 it can be seen that LocalPFN takes an order of magnitude longer (or more) compared to TabPFN-KNN. This is due to LocalPFN having to do finetune. Thus finetuning takes an order of magnitude more time compared to inference.

Do you think you can finetune with a smaller k than what you use for inference?

We suspect this approach would work (refer to previous point about robustness). However, it would cause a disparity between finetune and inference and might have negative effects. In terms of smaller k helping with finetune time, most of time is spent for model parameter update not retrieval.

I assume they are always available expect for TabPFN variants? I also think that the fact that your using the hyperparameter spaces from [35] should be said in the main text (if that's indeed the case).

We thank the reviewer for the observation, we will update the paper to reflect these more clearly. Most of the values were indeed present but we needed to re-run TabPFN (missing many datasets) and perform HPO for catboost on one dataset it was missing as well.

Which distance are you using in faiss? I'm wondering if using the euclidean distance would improve the performance of the ordinal encoded features for the knn (compared to using one-hot-encoding).

We also had intuition on which encoding/distance would work best. We tried multiple approaches and they made little difference thus we ended up using the simplest variant. Please refer to the general message for more details.

Indeed, we had only reported the full training+eval loop runtime for the algorithm but not the inference time specifically before. Let's get into more details.

To classify $N_\text{qy}$ examples given $N_\text{ctx}$ points, TabPFN would contruct a tensor of size ($L=N_\text{ctx}+N_\text{qy}, B=1, d$) with appropriate masking. As we do not share the context across queries, with TabPFN kNN we need to have a specific context for each point and as such our input tensor is size ($L=N_\text{ctx}+1, B = N_\text{qy}, d$) where the $+1$ is the query point to classify in each batch dimension.

For instance, for 512 query points and 1000 context size/neighbors, the inference speed of TabPFN would be about 0.01s. A naive TabPFN-kNN is about 1s. However using bf16 precision we can lower it to about 0.4s (0.008s for TabPFN). If we used 128 queries, the number would be 0.008s (TabPFN) vs 0.1s (TabPFN-kNN). So as we observe, we are slower than TabPFN.

In the 71 datasets in the attached pdf on which the deep learning baselines were able to run according to the Tabzilla design, the largest of the 95 datasets are typically excluded (this is because Tabzilla allocate a time limit for each algorithm and dataset, as such many --slow-- DL baselines do not run on the largest datasets in the given time budget). This is why TabPFN-kNN is really fast. On fig 12, including all datasets the average speed is about 10x slower but still fast considering there is no training. The fact that TabPFN-kNN is slower (and takes a lot more memory, especially when having to perform backprop as we do when performing fine-tuning end-to-end with retrieval) is why we introduced the approximate NN context sharing for fine-tuning. All in all, TabPFN-kNN is not the algorithm with the fastest inference speed, TabPFN is much faster and so is XGBoost etc...

However we believe this algorithm still has very important strengths: it scales extremely well with the dataset size (compared to TabPFN whose performance degrades strongly), so in the case of production ML models which have access to a very large dataset but only have to classify new queries at a lower rate, it is a very well suited algorithm. Furthermore, indexing on large datasets is usually much faster than retraining any ML algorithm, as such it can adapt very fast to changes in data distribution (such as covid for instance) without needing expensive retraining.

Note that the evaluation runtime was never the bottleneck in our research, so this is not what we focused on, but there are other simple ways one could improve upon it. For example, we can use the clustering capabilities of faiss to split the training data into a fixed number of contexts. This way we can ensure many queries will share the same context and harness the speed of TabPFN. While it may degrade the accuracy of the NN search, we think it can be an interesting trade-off if inference speed is an issue.

Thank you for the interesting comment, we will include numbers or a figure specifically concerning the inference speed for clarity.

Have you received the anonymized code link from the AC?

We thank the reviewer for their feedback.

“Although LoCalPFN shows improved performance, the fine-tuning process increases computational complexity and runtime, especially with large datasets.”

Yes this is true, however note that the complexity added by the finetuning process is not specific to LoCalPFN but a cost we also must pay when fine-tuning with TabPFN (vanilla). Note that despite the additional runtime, our method has similar runtime to many other deep learning approaches while having very high performance.

“The paper's reliance on TabPFN as the base model restricts the generalizability of the proposed method. While LoCalPFN demonstrates significant improvements, it remains unclear whether these benefits would transfer to other in-context learning models for tabular data.”

Please refer to the general message. The headline is that we expect the techniques and concepts from this paper to generalize to future tabular foundation models.

“The current implementation of LoCalPFN is constrained by TabPFN’s limitations on the number of features and classes, as well as its incompatibility with regression tasks.”

This is correct. Our contribution is a post-training one so we don’t retrain a base model. However we agree with the reviewer concerning the limitations of TabPFN and are currently working towards removing those. Please refer to the general message for more discussion regarding the possibility of other models.

“Have you tested the retrieval and fine-tuning techniques used in LoCalPFN on other in-context learning models?”

We are not aware of tabular in-context learners except for TabPFN. Are you referring to general in-context learners (i.e. LLMs)? There are multiple issues arising:

1. Since each row cannot be considered as a token but as a (potentially very long) sequence of tokens of varying length, comparing rows for retrieval is not straightforward.
2. How to embed rows is actually not obvious in the first place; since LLM perform best on text, most successful methods ([https://arxiv.org/abs/2206.06565 ]) rely on describing each row as a sentence.
3. Because each row would have many tokens, our context size would be much more limited: 1-2 orders of magnitude below what TabPFN can do [https://arxiv.org/pdf/2406.12031 ].
4. Despite all of this, LLMs for tabular data prediction still lag behind traditional methods and are harder to evaluate as they have memorized many well-known tabular datasets [https://arxiv.org/abs/2403.06644 ].

“Given the constraints on the number of features and classes due to TabPFN, how do you plan to extend LoCalPFN to handle datasets with more features and classes?”

Even though the current model is limited (100 features, 10 classes, no regression), it is still possible to extend it: for instance we can easily perform feature selection/averaging as the forward pass is fast. Any multiclass problem problem can be turned into multiple binary classification problems. And as we showed in the main message we can even perform regression as classification. However we also wish to go beyond those limitations and are actively towards designing a new architecture and training a new model that does not suffer from those.

Regression

Please refer to the general message

We appreciate the time you took to review our paper. We hope to clarify some points of confusion below in our response.

"Recent advancements [...] have struggled to scale to larger and more complex ones." is inaccurate -- depending on the dataset and task definitions.

Can you please clarify this point? This sentence is meant to be understood in the context of tabular data, as the previous one begins with “Tabular data”. In any case, we will update it to “have shown promise on smaller and less complex tabular datasets“ to remove ambiguity.

However, if this sentence is being challenged in the context of tabular data, several works have indeed shown TabPFN to be a competitive model [e.g., https://arxiv.org/abs/2305.02997, https://openreview.net/forum?id=XctSyEsBzx] however it doesn’t translate as well to large datasets because of the limited context size [https://arxiv.org/abs/2402.06971 Fig 3, or Fig 2 of our paper].

“There are many such examples in the paper, which make it hard to understand the scope of the paper. It raises concerns about the overall reliability of the paper's findings.”

We would be happy to have such examples pointed out so that we can better clarify the paper and convince you that our results are strong and reliable.

“The paper fails to clearly define the complexity of tabular datasets, making it difficult to assess the relevance and significance of the proposed method (LoCalPFN) in addressing the challenges of complex datasets.”

We would like to address the comment with several points.

Complexity of a dataset is one of the reasons we invoked when explaining why retrieval can help, even when the dataset size is small. Note that the main motivation for our paper is still to be able to use such in-context models on large datasets. Neither our method nor our results (Table 1 and 6-7) are affected by how we measure complexity.

There isn’t one way to measure data complexity and it is still an open question. Many of the usual measures are based on a notion of “compressibility” dating back to Kolmogorov complexity. Many practical methods are based on a measure of discrepancy between the data distribution (usually $p(x)$ or $p(y|x)$) and a model of known capacity. For instance, let’s consider a linear model and a more complex non-linear model (e.g., polynomial). If both the linear and non-linear model provide a similar approximation to $p(y|x)$ (i.e., low $\text{KL}(p(y|x)\mid\mid q(y|x))$ or equivalently cross-entropy), then we can argue that the classification problem is not complex as a simple linear model is enough to approximate it. On the other hand, if the discrepancy between the two models is large, then a non-linear model is necessary to explain the data and as such it is more complex. This is very much in line with our intuition and how the term is used in machine learning, where some tasks are considered “hard” (such as mathematical reasoning) as small models don’t perform well but more advanced ones (e.g., adding more parameters or some search capability) perform better. Thus we argue that measuring the discrepancy between the loss/performance of different algorithms is both intuitive and widely used. Indeed we see examples in the tabular domain [https://arxiv.org/abs/2207.08815, 3.1 “not easy” paragraph] where the gap is measured, or [ https://arxiv.org/pdf/2305.02997 ] where datasets are considered “hard” if a simple baseline does not achieve top performance. We are therefore very much in-line with the standards of the field, including the tabular sub-domain.

“The paper does not address the important practical considerations of the model's cost and latency at inference time, especially retrieval is used. This omission is a significant limitation, as it hinders the understanding of the model's real-world applicability.”

We believe you are referring to our mention of the runtime (L328) and Figure 11 in the Appendix. You can see there that the non-finetuned method (TabPFN-kNN) is faster than all tree based methods but does not significantly outperform XGBoost or CatBoost. As a reminder, TabPFN-kNN represents the raw cost of inference for our technique when only retrieval – not fine-tuning – is performed, which should hopefully assuage concerns about the runtime of the retrieval component. Meanwhile, LoCalPFN is slower (by a factor 10-25x on average) compared to a single XGBoost run, but achieves significantly better performance. Depending on the exact budget and the hardware available to a specific person, TabPFN-kNN, XGBoost, CatBoost, or LoCalPFN might be the best choice. Note that, as GPUs are improving and inference is made more efficient by the day (e.g., better flash attention, quantization, etc.), LoCalPFN’s inference speed should improve over time. Please see the general message for more details; we have included a runtime comparison against popular deep learning techniques there as well, showing that LoCalPFN leads on performance while remaining competitive on runtime.

The proposed LoCalPFN method is heavily reliant on TabPFN as the base model. This raises concerns about the generalizability of the approach to other base models and task types (e.g., regression).

Please refer to the general response.

Dataset Size and Model Quality: We empirically notice that for the datasets & models we have, the task tends to be a bit harder as dataset size increases.

Query Example: Could you clarify what kind of example you would like to see? For all queries, we first standardize its features using training statistics and then retrieve its neighbours in the training set using L2 distance. We then pass the (context, query) vectors to TabPFN.

Overall: Given the strengths you pointed out (comprehensive analysis and extensive ablations, novel method), and that we have hopefully addressed all your concerns, we ask you to re-consider your rating of 2. This is generally reserved for papers that are technically wrong.

Hi,

We would like to again thank you for taking the time to review our paper, but would also very much appreciate if took the opportunity to engage in discussion if time permits, considering that we have clarified some of the more negative points of your review both in the rebuttal to you and the shared rebuttal to all reviewers (that one has a PDF attached).

Thanks!