We thank the reviewer for their insightful review and recognition of our work.

---

1. More comparison results against the more recent models based on TabPFN ?

Thanks for these relevant works. Due to limited computational resources, we prioritized methods based on code availability.

• TuneTables: Official code is available.
• A Closer Look at TabPFNv2: No official code was available at the time of evaluation. Moreover, the paper was released online after the ICML submission deadline.
• MixturePFN (Mixture of In-Context Prompters): No code was publicly available.
• LocalPFN (Retrieval & fine-tuning for in-context tabular models): Code is found.

Based on these considerations, we decided to evaluate TuneTables and LocalPFN across 186 datasets with $\le 10$ classes on the TALENT benchmark. Their average ranks (based on accuracy), along with other methods, are summarized below:

---

2. A brief comparison between TabPFNv2 and your model on the method aspect ?

In the updated version, we will include a methodological comparison between TabPFNv2 and TabICL in the related work section. Below is a brief summary:

---

3. The benefit of using Set Transformer compared with the weight sharing ? ...

We admit that we may not fully understand the question. We guess that this question may refer to Lines 194–196 in our paper: "Instead of sharing the same weight and bias for all samples in a column as in a classic linear layer, each value is projected using its own weight and bias."

During column-wise embedding (feature embedding), we map each individual cell in a column $c_j$ into an embedding vector. Specifically, we compute:

$W, B = \text{SetTransformer}(c_j)$
$e_j = W \odot c_j + B$

Here, we consider two choices:

• Shared weights and biases: All cells in column $c_j$ share the same weight and bias
• Cell-specific weights and biases (our choice): Each cell $c_j^i$ is associated with its own weight $W_i$ and bias $B_i$.

We opt for the second design because, in tabular data, cells within the same column often serve distinct statistical roles—e.g., being the minimum, maximum, mode, or an outlier. Using cell-specific parameters allows the model to adaptively embed each value based on such roles.

Importantly, both options rely on the set transformer to generate the projection parameters, so the architectural backbone remains the same. The key difference lies in how the output of the set transformer is used.

How much computation overhead it can bring ?

Using separate weights and biases for each cell introduces only a mild overhead, as the set transformer still processes the entire column as a set, and the per-cell projections are implemented efficiently as element-wise operations.

How much it makes the performance better ?

In our early experiments, we observed that using per-cell weights and biases improved performance over shared ones. However, we acknowledge that we have not conducted a thorough ablation study to quantify this gain precisely, and we will consider including such analysis in future work.

---

4. Why diversifying activation functions can help the training? How you perform the random rescaling ?

Our intuition for diversifying activation functions is that it is generally helpful to increase the diversity of generated datasets. In particular, some of the activation functions can generate functions of different smoothness (e.g., the GP functions), periodic functions (sine), (semi-)discrete feature distributions (e.g. ReLU/signum), and heavy-tailed distributions (e.g. exp). We will add this to Appendix B.1. The random rescaling is already described in Appendix B.1: $x_i \leftarrow \exp(2a)(x_i + b)$ with $a, b \sim \mathcal{N}(0, 1)$.

---

In the revised version, we will include LocalPFN and TuneTables into the benchmark.

We appreciate the reviewer's valuable feedback. We now respond to the reviewer's questions.

---

1. Why should the training be invariant to permutation of columns?

In relational data, column order is typically arbitrary. The very visible publication [Why Tabular Foundation Models Should Be a Research Priority] has put forward that tabular methods are expected to be invariant to column order. This is, however, not our battle.

If there is a 'sum' column, will it be sensitive to column order?

We may not fully understand the question, but in principle, the position of a "sum" column should not be sensitive to column order.

---

2. It gives me a contradictory feeling ...

Yes, this is indeed a challenging issue. In fact, both TabPFNv1 and TabPFNv2 face the same mode collapse problem and address it similarly—by first breaking feature-order invariance and then restoring it via ensembling. They differ in how the invariance is broken: TabPFNv2 uses random feature embeddings and feature grouping, while TabPFNv1 uses fixed feature embedding layers.

Will mode collapse reoccur after restoring permutation invariance?

No, this will not happen. Ensembling aggregates predictions from independently permuted feature orders. Each permutation receives unique positional encodings via RoPE, leading to diverse, non-collapsed representations. Ensembling effectively averages out positional biases (e.g., effects of column order). This is analogous to data augmentation — diverse permutations provide complementary "views" of the data, enhancing robustness of TabICL. In short, RoPE prevents local collapse (within each permutation), and ensembling restores global invariance without reintroducing collapse. The excellent performance of TabICL also confirms this empirically.

---

3. Since the performance of the proposed method is so close to TabPFN2...

While TabICL and TabPFNv2 achieve comparable overall performance, TabICL offers key advantages in handling large datasets, computational efficiency, and scalability — crucial for real-world industrial applications. We acknowledge the need to better clarify these contributions. In the revised manuscript (Related Work, L143–148), we will explicitly emphasize:

• Methodological differences: TabPFNv2 uses alternating column-row attention, while TabICL adopts a two-stage design (fixed row embeddings + ICL) to reduce complexity.
• Computational complexity: Detailed comparison is provided in our response to the question 3 of the reviewer rh11.
• Efficiency and scalability: Including memory optimizations, support for arbitrary class counts, faster inference, etc

---

4. The comparison of inference speeds with other methods are not listed in paper...

We focus on inference speed comparison with TabPFNv2, as both are tabular foundation models that perform well without hyperparameter tuning—unlike most other tabular methods. Unfortunately, the TALENT codebase does not report pure inference times excluding tuning and fitting.

When comparing inference speed alone, neither TabPFNv2 nor TabICL shows an advantage. For instance, as noted in the TabPFNv2 paper, on a dataset with 10,000 rows and 10 columns, TabPFNv2 takes 0.2s per sample, while CatBoost (with default settings) requires only 0.0002s for inference.

Their real strength of tabular foundation models lies in the elimination of hyper-parameter tuning and cross-table transferability — qualities that are critical for real-world industrial applications and emphasized in [Why Tabular Foundation Models Should Be a Research Priority].

---

5. The reviewer wonder the concrete examples of the in-context learning in tabular classification...

In-context learning for tabular classification can in principle solve all of the applications that regular tabular classification models can solve. For example, some applications of TabPFN are listed at https://github.com/PriorLabs/awesome-tabpfn.

Note that synthetic datasets are only used for pre-training. Yes, our proposed TabICL can deal with real-world datasets. As shown in the paper, TabICL achieves SOTA results on the TALENT benchmark comprising 200 real-world datasets collected from 13 distinct domains, such as education, healthcare, and finance.

---

6. Suggest to indicate that the red bar ...

Thank you for your feedback.

• In the caption of figure 5, we will indicate that the red bar represents the median relative accuracy improvements over MLP across all datasets for each method.

• The black dots outside the whiskers represent outliers—datasets where the relative accuracy improvements of a method deviates significantly from the majority of results.

• Horizontal Bars (Whiskers): The horizontal bars (whiskers) extending from the box indicate the range of typical data distribution (approximately 1.5 × the interquartile range from the quartiles). They do not represent average accuracy.

We thank the reviewer for their insightful review, and for mentioning that “the results are really great”.

Next, we respond to the reviewer's questions.

---

1. A major drawback is ... the code or the model was released upon submission

We confirm that all of TabICL will be made fully public, including model weights, training details and prior data generation code. We have already released the inference code after consolidating it to a python package with a fully scikit-learn compliant interface. Here is an anonymous version: https://anonymous.4open.science/r/tabicl-A70B or download code from https://limewire.com/d/opefZ#jPuHwDXBUg. The pre-training code and data generation will be released upon acceptance.

We agree with the reviewer and believe open-sourcing our work will significantly boost its impact—especially since TabPFNv2 remains partially closed, which limits the community’s ability to reproduce or build upon it.

---

2. I am not quite convinced by the positional invariance claims...

Neither TabPFNv1, TabPFNv2, nor TabICL achieves permutation invariance of columns. All these models need to break this invariance in some way to solve the learning collapse issue when some features have similar statistics:

• TabPFNv1 relies on fixed, feature-specific embedding layers to implicitly encode column identities.
• TabPFNv2 introduces random feature embeddings and feature grouping.
• Our TabICL adopts RoPE.

We do not claim permutation invariance; Instead, we just address the issue differently from the other two models.

---

3. There are a few newer DL models [1, 2] missing ...

Thanks for these relevant works. Due to limited computational resources, we prioritized methods based on their reported performance and availability.

• TabM has been accepted at ICLR and showed promising results across 46 datasets. Its code is open-source, so we evaluated TabM on TALENT.

• Mambular shows equal performance compared to CatBoost on 12 datasets, according to both its ICLR rebuttal and the latest arXiv version, so we have deprioritized it for now.

• Attic does not provide publicly available code, is not on arXiv and was only visible on OpenReview (submitted and now rejected from ICLR).

TabM performance rank is as follows:

---

4. the benefit of the set transformer given RoPE ?

We’re unsure we fully understand the question. The set transformer is used for column-wise embedding and is row-order invariant, while RoPE later breaks column-order invariance to mitigate learning collapse. They are not directly related.

We chose the set transformer for its linear complexity with respect to sequence length (see response to Reviewer rh11, Q3), which is more efficient than standard self-attention—especially on large tabular datasets.

---

5. Given the recent TabPFNv2, the contributions do seem [...] combining ...

We would like to emphasize that TabICL was designed, developed, and pretrained independently of TabPFNv2. TabPFNv2 was published very shortly before ICML submission deadline, leaving little time to include it as a baseline. However, we still rigorously compared against it in our paper.

TabICL and TabPFNv2 differ in many ways:

Additional innovations in TabICL:

• Hierarchical classification for >10 classes
• Curriculum pretraining (1K→60K samples)
• Memory optimizations (5GB GPU for 100K samples)

The reviewer questions whether these contributions are "minor", but the empirical results demonstrate otherwise: TabICL is faster, more scalable, and also delivers better performance.

---

6. New references that should be included
   Thanks for these recent references. We will include them.

---

7. Why exactly do you use the 4 cls tokens ?

The choice of 4 [CLS] tokens was heuristic—we considered it a reasonable trade-off but did not conduct ablation studies with fewer or more [CLS] tokens.

---

8. How RoPE can mitigate the embedding collapse problem ?

RoPE mitigates embedding collapse by breaking feature permutation symmetry. After column-wise embedding, features with similar distributions may have indistinguishable embeddings. RoPE injects positional information via rotational encoding for row-wise interaction, ensuring distinct representations even for permuted columns. This is validated in Figure 4 (Section 3.3).

---

9. Other invariance strategies ?

Yes, we tried CNN to process feature embeddings after column-wise embedding, but RoPE is better.

---

In the revised version, we will include TabM.

We appreciate the reviewer's valuable feedback.

We kindly invite the reviewer to refer to the figures available at https://limewire.com/d/FhJbU#LPxKd0hlro, which may help better illustrate our responses. The link only contains figures.

We realize that there may be a misunderstanding of how column-wise embedding and row-wise interaction operate. So, we decide to first explain them.

For tabular data with $n$ rows and $m$ columns:

• Column-wise embedding applies attention across all $n$ samples within each column, mapping scalar values to embeddings
• Row-wise interaction applies attention across all $m$ features (plus 4 [CLS] tokens) within each row

We adopted "column-wise"/"row-wise" from BISHOP (ICML 2024), but acknowledge this may cause confusion. Therefore, we will rename:

• Column-wise embedding → Column-wise inter-sample embedding
• Row-wise interaction → Row-wise inter-feature interaction

We welcome any alternative naming suggestions from the reviewer.

Next, we respond to your questions.

---

1. Why use column-wise embedding and row-wise interaction instead of the opposite ?
   The core idea of TabICL is to encode each row into a fixed-dimensional embedding before applying in-context learning (ICL). Row-wise interaction is used to aggregate row-level information into a single embedding, which positions it between column-wise embedding and ICL.

---

2. In Figure 1 ... shouldn’t it be 4+n?
   As explained above, row-wise inter-feature interaction involves the attention between $m$ features as well as 4 [CLS] tokens within each row. Therefore, it is $4+m$.

---

3. What is the overall complexity formulation?
   Column-wise embedding $TF_{col}$ uses induced self-attention blocks (ISAB), while row-wise interaction $TF_{row}$ and dataset-wise in-context learning $TF_{icl}$ use vanilla self-attention blocks (SAB).

• SAB has quadratic complexity : $O(L_{seq}^2)$
• ISAB has linear complexity: $O(L_{seq} \ k)$, where $k$ is the number of inducing vectors

In summary,

• $TF_{col}$ has linear complexity w.r.t. $n$
• $TF_{row}$ has quadratic complexity w.r.t. $m$, but $m \ll n$ in most tabular datasets
• $TF_{icl}$ has quadratic complexity w.r.t. $n$, but it is computed only once because $TF_{row}$ collapses the feature dimension

In contrast, TabPFNv2 uses SAB for sample attention $O(n^2 \ m)$ and feature attention $O(m^2 n)$. Since it never collapses columns, sample attention becomes prohibitive for large $n$ and moderate $m$.

---

4. For the tree-based synthetic data generator, ... ?
   We fit a XGBoost regressor to Gaussian noise targets $y_i$. In practice, we limit the max depth to 4 to reduce both fitting time and model capacity, preventing it from fully capturing the randomness. This forces XGBoost to balance between noise and its limited capacity, resulting in a practical randomized initializer—validated by our experimental results.

---

5. Are you using 8 A100 GPUs ?
   We used three A100 GPUs, as mentioned L278-281.

---

6. The paper misses ablation studies on key components

• Ablation study on curriculum learning
  We evaluate the pre-trained checkpoints after each stage of curriculum learning. The table below shows the average relative accuracy improvements over the stage 1 (Acc Imp) and the average rank:

Stages 2 and 3 effectively increase the rank of TabICL from 10.52 (7th) to 6.30 (1st).

• Ablation study on tree-based generation
  We compare TabICL pre-trained with and without tree-based generation, using the stage 1 setup for 20K steps (to reduce training time). The average relative improvement of tree vs. no tree is as follows: | Accuracy | AUC | Log Loss | | --- | --- | --- | | 0.37% | 0.18% | 2.5% |

---

7. Did you compare with the "data grouping based on random forest" method from TabPFNv2?
   We applied this random forest extension to both TabPFNv2 and TabICL on datasets with over 10K samples (the reported limitation for TabPFNv2). Results show performance gains for both models:

TabPFNv2 improves from 10.89 (see Figure E.5) to 9.14, while TabICL becomes the best-performing method on large datasets !

---

In the revised version, we will include complexity analysis, ablation studies, and random forest extension.