Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation

The reviewer's constructive comments are highly valued and appreciated. Due to space constraints, our response will be divided into two parts.

Q1: The issue of feature heterogeneity remains partly unresolved, as there may not be a latent state that aligns to all decision boundaries with similarity measure. The top-K operation on prototypes helps to mitigate this problem to some extent but doesn't completely address it. For example, any transformations applied to the features within the tables, like replacing a feature 'x' with '1/x' (such transformation is reversible, and the information has not been altered or lost), might make the hidden states fail.

A1: Thanks. We will respond from three aspects.

First, the feature heterogeneity is the main difficulty in training a model over multiple tabular datasets. Some existing methods deal with the problem by padding the feature, learning dimension-agnostic transformations, and using large language models. The uniform representation in this paper only encodes the common states of different models, while may lose the specific information.

Second, for the given example, if we replace a feature 'x' with '1/x' (such transformation is reversible), the semantic meaning of this feature is changed. While in TabPTM, the mutual information between the feature and the label remains stable, and the feature will be emphasized in calculating the meta representation.

Third, it is the relationship between instances, i.e., the distance between features, that determines the values in meta-representation. If two samples are close to each other under positive feature x, it will still be close when we reverse the feature. The meta-representation is robust to part of the transformations in practice.

Q2: This model might only exhibit robustness on certain datasets, and it would be beneficial for the authors to clarify this point and clarify when the model is effective and when it is not. Use benchmark datasets.

A2: Thanks for the suggestions. In addition to the classification results in the paper, we validate the robustness of the model in regression tasks, based on the benchmark datasets in the mentioned paper.

In a regression task, the label $y$ of an instance $\boldsymbol{x}_i$ is a continuous value, so that we modify the meta representation of an instance $\phi(\boldsymbol{x}_i)$ as the concatenation of its distance with neighbours as well as their corresponding labels. In detail, if the $K$ nearest neighbours (based on the Euclidean distance) of instance $\boldsymbol{x}_i$ is ${\boldsymbol{x}_1, \dots, \boldsymbol{x}_K}$, we define the meta representation for regression as:

$\phi(\boldsymbol{x}_i) = [dist(\boldsymbol{x}_i, \boldsymbol{x}_1), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_j), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_K), y_1, \ldots, y_j, \ldots, y_K] \in \mathbb{R}^{2K}$.

Then, we can use a top-layer regressor to predict the label, i.e., $\hat{y_i}=\mathbf{T}_\Theta(\phi(\boldsymbol{x}_i))$.

If $\mathbf{T}_\Theta$ is a linear model, the prediction could be implemented based on a neighbourhood manner by weighted averaging the labels of the neighbours.

We implement $\mathbf{T}_\Theta$ as a MLP. Similar to the protocol of classification we used in our paper, we collected 12 regression datasets. 9 of 12 datasets are sourced from Grinsztajn's benchmarks. We pre-train our TabPTM on half of them (house_sales, california_housing, MiamiHousing2016, ailerons, abalone, pole), and evaluate the learned model on the remaining datasets (analcatdata_supreme, wine_quality, house_16H, sulfur, yprop_4_1, houses).

We set $K = 8$. By utilizing both Euclidean and Manhattan distance, the dimension of meta representation is $4K$. XGBoost and MLP are tuned on a single dataset. TabPTM is tuned on 6 pre-training datasets and generalizes to 6 unseen datasets without fine-tuning. The experiments are listed in the table below:

We will add more datasets in the future, and analyze the robustness of TabPTM.

Hi, thank you for your answer.

Your approach appears to solely rely on the distance to neighboring data, but it discards the direction. Could you explain (experiments are not necessary) on the potential issues arising from discarding direction? I am a bit curious about this because intuitively, I feel that direction might be crucial, although it could introduce significant computational costs.

Additionally, on irregular decision boundaries, situations arise where samples with shorter distances may have completely different labels. This suggests potential theoretical shortcomings in the method. I guess if we encode the entire dataset, relying solely on distance is acceptable. However, in cases where the dataset is extensive, and encoding the entire dataset is impractical, or when there are changes in data distribution (or some data becomes inaccessible) in reference, challenges may arise. Is this correct?

Thanks for the valuable feedback the reviewer have shared with us.

Q1: The "related work section" does not cover k-nearest neighbors and metric-based variants.

A1: Since the target of this paper is to learn a joint model over multiple heterogeneous tabular datasets, we do not emphasize metric-based methods a lot since the kNN or other metric-based methods cannot be applied across datasets. We discuss the relationship between ours and other meta-feature methods in "remarks" in Section 4.1. We will add more discussions in the related work part in the final version.

Q2: Selecting class-representative items in a dataset is a -- strong and costly -- form of training, just as selecting the support vectors for a SVM is a form a training. To my opinion, this proposal is merely shifting the "not-that-hard" classification problem to the "not-that-easy" metric-learning and representative-choice problems.

A2: Thanks. In the paper, we first describe the idea of learning a general model with meta-representation, and the meta-representation aims to encode the relationship between an instance and its nearest prototypes in a particular class. We have tried various strategies, and in our experiments, we set the whole training set of a class as the prototype set. In other words, all training instances in a class are treated as prototypes. Therefore, the computational burden mainly lies in the similarity search, which could be easily accelerated in practical implementations. We show the training time comparison between various models, including the non-deep method and deep methods, in Table 2. The results indicate the training-free method makes predictions on a target dataset fast.

Q3: The class-probability outputs of more robust classifiers like XGBoost or SVM are better and cheaper "universal representations" ?

A3: We can use the prediction results as the representation of an instance based on some robust classifier. However, there are several difficulties.

First, it depends on the training quality of the classifier, and sometimes, it requires careful tuning of the hyper-parameters. As demonstrated in Table 1, XGBoost and SVM can not exhibit particularly outstanding performance even with tuned hyper-parameters.

In addition, using XGBoost and SVM also requires training time (as shown in Table 2). So such a kind of representation cannot be obtained in a "cheaper" way.

Furthermore, since different datasets have different sizes of classes, the "universal representations" have different dimensions, which impedes the sharing of knowledge across datasets.

The main idea of this paper is to learn a general model over multiple heterogeneous datasets, and TabPTM represents instances in the same form.

Thanks again for the comments!

Thanks for your feedback. As the rebuttal period is concluding, we would like to provide a brief response.

• We have explored sampling the dataset instead of using the whole dataset. During training, a sampling approach was employed, where only a subset of the dataset was sampled in each iteration (as shown in Table 8). During inference, the entire dataset was utilized. Efficient algorithms for nearest neighbor search exist. The problem is a common challenge for all metric-based methods.

• We have experimented with these two simple baselines, but they did not yield satisfactory results. Thank you for pointing out, and we will consider incorporating them in the future.

We are grateful for the valuable suggestions.

Q1: Lp space may be insufficient to represent tabular data.

A1: Thanks. Since diverse tabular datasets have different sets of features and classes, we investigate how to represent all the tabular datasets in a uniform form to support the training of a shared model. In our meta-representation, distances between one instance and its same-class nearest neighbours are utilized. We show the general idea of meta-representation and its basic form by using the L1 and L2 distance. Our experiments indicate that learning with a shared model can make up for information loss using the Lp norm-based metric. Furthermore, we can also concatenate other kinds of distance in meta-representation to improve its representation ability.

Q2: Two heterogeneous tables can have different label distributions, fitting one table's distribution to that of another table may not work well.

A2: That's right. The main challenge of learning a shared model over multiple datasets lies in the fact they have heterogeneous feature and label distributions.

The main idea of using meta-representation to learn a general model across datasets is to capture the intrinsic classification characteristic over datasets. The meta-representation encodes the similarity distribution of a class. If an instance belongs to a class with high probability (e.g., near the center of a class or in the high-density area of the class), most distance values to its nearest neighbours are small. In contrast, the distance value between an instance and the non-target classes should have larger values. The top-layer module $T_\Theta$ captures such a kind of similarity distribution for classification, which could be shared across datasets. We also illustrate this idea in Figure 1 in the paper, and a discussion of what could be shared across datasets in Section 4.2.

Q3: Whether the proposed model is for extracting representations or classifying data labels.

A3: Thanks. Our model TabPTM is a classification model based on the meta-representation. The meta-representation formulates an instance into a $K$-dimensional form no matter how many dimensions it originally had. Meanwhile, the model $T_{\Theta}$ captures the shared pattern (the pattern in the similarity distribution in the last answer) across datasets.

Q4: The experimental results are insufficient to demonstrate the superiority of the proposed method.

A4: Firstly, the baseline methods we compare are tuned specifically on individual datasets, making them more prone to achieving good performance, with computational costs increasing linearly as the dataset size grows. However, TabPTM is a model applied directly to all downstream tasks without the need for additional fine-tuning. Secondly, our TabPTM achieves the best performance in 3 of the 10 datasets in Table 1, and the two variants obtain the best two average results over the 10 downstream datasets. In Table 2, TabPTM achieves promising results with very short training/prediction time (300 times faster than XGBoost). We also validate the effectiveness of TabPTM on regression tasks (as shown on Respond to Reviewer XUXC A1).

Q5: How to classify unseen data for which classes are also unknown?

A5: In our experiments, we evaluate our TabPTM on datasets with unseen classes and features, and the results validate the ability of TabPTM over datasets with unseen classes.

Given a new dataset, we calculate the meta-representation of an instance w.r.t. each one of the unseen classes. For example, given $C$ unseen classes, we obtain $C$ meta-representations with $K$-dimension. Then we apply the learned model $T_\Theta$ over them to obtain the $C$-dimensional class confidence vector without additional training. The instance is classified as one of the unseen classes by selecting the index with the maximum confidence.

Thanks again for the feedback!

Q4: TabPFN should be added to the comparison on larger datasets.

A4: It is mentioned in the TabPFN paper that the method can only support small-scale tabular datasets. In our experiments, most large datasets contain over 10K samples. We do try TabPFN on larger datasets, while there exist computational issues on some datasets. Thanks for the suggestion. We also noticed that there are some improved variants of TabPFN recently, and we will compare some of them in the final version.

Q5: Add results on the benchmark from [1].

A5: There are 22 classification datasets and 35 regression datasets in [1]. In our paper, we evaluate the results of TabPTM on 22 classification datasets with varying scale (some additional results are provided in Table 12 in Appendix d). Thanks for the suggestions. We evaluate TabPTM on 12 regression datasets (9 of the 12 datasets are sourced from benchmarks [1]), as described in Q1&A1. We will include additional datasets in the future.

We appreciate the feedback provided by the reviewer.

Thanks very much for the valuable suggestions provided by the reviewer. Due to space constraints, our response will be divided into two parts.

Q1: Supporting regression with a similar approach.

A1: Thanks for the suggestion. The meta representation of an instance encodes its distance with same-class nearest neighbours, and the idea could be extended to the regression scenario. In a regression task, the label $y$ of an instance $\boldsymbol{x}_i$ is a continuous value, so that we modify the meta representation of an instance $\phi(\boldsymbol{x}_i)$ as the concatenation of its distance with neighbours as well as their corresponding labels. In detail, if the $K$ nearest neighbours (based on the Euclidean distance) of instance $\boldsymbol{x}_i$ is ${\boldsymbol{x}_1, \dots, \boldsymbol{x}_K}$, we define the meta representation for regression as:

$\phi(\boldsymbol{x}_i) = [dist(\boldsymbol{x}_i, \boldsymbol{x}_1), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_j), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_K), y_1, \ldots, y_j, \ldots, y_K] \in \mathbb{R}^{2K}$.

Then, we can use a top-layer regressor to predict the label, i.e., $\hat{y_i}=\mathbf{T}_\Theta(\phi(\boldsymbol{x}_i))$.

If $\mathbf{T}_\Theta$ is a linear model, the prediction could be implemented based on a neighbourhood manner by weighted averaging the labels of the neighbours.

We implement $\mathbf{T}_\Theta$ as an MLP, and experiments validate its effectiveness to become a general regression model across datasets. Similar to the protocol of classification we used in our paper, we collect 12 regression datasets, pre-training our TabPTM on half of them (house_sales, california_housing, MiamiHousing2016, ailerons, abalone, pole), and evaluating the learned model on the remaining datasets (analcatdata_supreme, wine_quality, house_16H, sulfur, yprop_4_1, houses). It is notable that these two sets of datasets do not share the same set of features or regression ranges.

We set $K = 8$. By tilizing both Euclidean and Manhattan distance, the dimension of meta representation is $4K$. XGBoost and MLP are tuned on each dataset separately. TabPTM is tuned on 6 pre-training datasets and generalizes to 6 unseen datasets without fine-tuning. The experiments are listed in the table below:

TabPTM obtains the best performance on half of the datasets, and the second-best performance on the remaining datasets in a training-free manner.

Q2: Provide more detailed info on how to tune and evaluate the baselines.

A2: We described the implementation details, including the hyper-parameter tuning, in Section B.2. For some recently proposed methods such as TabCaps, we conducted hyperparameter tuning using their official code. We will provide more details in the final version. For example, hyper-parameters of XGBoost and MLP are selected by Optuna library with Bayesian optimization over 30 trials. Hyper-parameters of TabCaps are selected by hyperopt library over 30 trials. The hyperparameter spaces are as follows:

• XGBoost: {'model': {'alpha': ['?loguniform', 0, 1e-08, 100.0], 'colsample_bylevel': ['uniform', 0.5, 1.0],'colsample_bytree': ['uniform', 0.5, 1.0], 'gamma': ['?loguniform', 0, 1e-08, 100.0], 'lambda': ['?loguniform', 0, 1e-08, 100.0], 'learning_rate': ['loguniform', 1e-05, 1], 'max_depth': ['int', 3, 10],'min_child_weight': ['loguniform', 1e-08, 100000.0], 'subsample': ['uniform', 0.5, 1.0]}}
• MLP: {"model": {"d_layers": ["$mlp_d_layers", 1, 5, 1, 512], "dropout": ["?uniform", 0.0, 0.0, 0.5 ], },"training": {"lr": ["loguniform", 1e-05, 2e-3],"weight_decay": ["?loguniform", 0.0, 1e-06, 1e-3]}}
• TabCaps: {'lr': uniform('lr', 4e-2, 2e-1), 'sub_class': randint('sub_class', 1, 5), 'init_dim': randint('init_dim', 32, 128), 'primary_capsule_size': randint('primary_capsule_size', 4, 32), 'digit_capsule_size': randint('digit_capsule_size', 4, 32), 'leaves': randint('leaves', 16, 64)}

Q3: Proposed method uses augmentations applicable to other models, but MLP, Transformers and DANets are evaluated without augmentations.

A3: We investigated several strategies to obtain a $K$ dimensional meta-representation when the number of examples per class is limited. Data augmentation is one of the strategies we have tried, and finally, we find padding the meta representation with its last value performs the best, which is mentioned in Section 4.1 ("..., so we set the padding strategy as our default choice"). In our main experiments, the comparisons with other methods are fair.

Thanks to the AC for encouraging our responses. We appreciate the valuable feedback from each reviewer. We have responded to all reviewers. Hope our response may address their concerns and engage in further discussions.