PTaRL: Prototype-based Tabular Representation Learning via Space Calibration

Thanks for your comments. We respond to your questions as follows and hope they can address your concerns.

Q1: Unclear definition of the concepts of global data structure information and sample localization.

R: In the context of any tabular dataset, we have observed global data structure information comprises two different components: (i) the global feature structure and (ii) the global sample structure.

Considering the feature structure, traditional and deep tabular machine learning methods utilize all features or a subset of features as input, allowing them to model inherent interactions among features and thereby acquire a comprehensive global feature structure. In addition, there also exists the sample structure given a tabular dataset. Traditional methods (e.g., boosted trees) can effectively model the overarching relationships between data samples. Specifically, in XGBoost, the dataset undergoes partitioning by comparing all the samples, with each node of a decision tree representing a specific partition, and each leaf node corresponding to a predictive value. The iterative splitting of nodes during training empowers decision trees in XGBoost to learn the distribution of all the samples across distinct regions of the data space, capturing global sample structure.

However, we note that deep tabular machine learning methods typically rely on batch training to obtain data representations within a batch. These methods do not explicitly consider the structure between samples within a batch. Furthermore, they fail to capture the global structure between samples across different batches. This limitation presents challenges in comprehensively capturing global data distribution information, consequently impeding overall performance.

Our methods rebuild the representation space with global prototypes (P-Space) in the first stage. Then in the second stage, the original data representation by deep tabular machine learning methods is projected into P-Space to obtain projection representation with global prototypes. On the one hand, by minimizing the Optimal Transport distance between the two representations, we could represent each sample with global prototypes, and in the meanwhile encode the global feature structure in learning global prototypes, considering backbone models can inherently learn the interactions among features. On the other hand, the global prototypes are learned by directly modeling all the data samples and thus the complex data distribution could be obtained by global prototypes to capture the global sample structure.
Therefore, PTaRL is able to capture both the feature and sample structure information by prototype learning. Considering previous deep tabular machine learning methods can only acquire the representations limited by the batch training, we use the concept of sample localization to encapsulate this limitation. We also provide the detailed explanation in Appendix A.8.

Q2: Doubts about the Fig. 1 and Fig. 4.

R: Thanks for your mention. We also note that there exists certain overlap in Fig. 1. After our careful analysis, we attribute the differences between Fig. 1 and Fig. 4 to the different characteristics of distinct datasets.

Specifically, Fig. 1 is based on the Adult dataset, while Fig. 4 is derived from the Higgs and Jannis datasets. In comparison to the Higgs dataset (98050 samples, 28 features) and the Jannis dataset (83733 samples, 54 features), the Adult dataset (48842 samples, 14 features) has a smaller number of samples and features. Additionally, the Adult dataset includes both numerical and categorical features, whereas the Higgs and Jannis datasets only contain numerical features. Considering the smaller size of the Adult dataset for training and its more intricate feature relationships, it poses a more challenging task for deep tabular machine learning methods to learn disentangled representations. Our proposed PTaRL offers a viable solution to enhance the disentanglement of representations by deep tabular machine learning methods.

Q3: Unclear description of the diversification in Fig. 5.

R: We apologize for any confusion and have enhanced the clarity of Fig. 5 for improved illustration. The top and bottom rows respectively depict scenarios without and with the Diversifying Constraint. The first and second columns represent the mean coordinate values of two distinct categories, while the third column illustrates the difference between the two categories. The first row shows that without Diversifying Constraint, the average coordinates between two different categories are similar, posing challenges in effectively distinguishing representations belonging to distinct categories. Contrastingly, the second row, which introduces the Diversifying Constraint, showcases a more pronounced difference in coordinates between categories, thereby promoting diversification in the coordinates of distinct categories.

Many thanks to the authors for their thorough and insightful responses. I am happy to increase my score to 8.

We sincerely appreciate your efforts and instructive advice to improve the manuscript. We respond to each of your comments as follows:

Q1: Unclear description of the optimization process of Eq. 4.

R: We are sorry for the confusion and we would like to give a more clear description.

The Optimal Transport (OT) problem is usually to find the most cost-effective way to transform a given distribution to another distribution, which is typically achieved by calculating the specified transportation plan that minimizes the total transportation cost, while the minimized cost is usually called OT distance. In our paper, we minimize the distance between original representation distribution over each sample $P_i$ by deep tabular models and the corresponding projection representation distribution $Q_i$ in P-Space with global prototypes, in order to preserve original data information (of $P_i$) in $Q_i$. We follow the typical setting of OT problem to first estimate the transport plan to obtain the OT distance between $P_i$ and $Q_i$. Then, the obtained OT distance is further used as loss function to jointly learn the two representations.

Specifically, after initializing the global prototypes $B$ of P-Space, we project the original data samples into P-Space to learn the representations with global data structure information. To better illustrate the optimization process, we revise the Eq. 4 in the original paper to make it more readable. In Eq. 4, the $i$-th sample representation by deep tabular model is denoted as $G_f(x_i;\theta_f)$, the empirical distribution over this sample representation is $P_i = \delta_{G_f(x_i;\theta_f)}$, the projection representation distribution is denoted as: $Q_i = \sum_{k=1}^{K} r_{i}^{k} \delta_{\beta_{k}}$, where $r_i$ is coordinates. To capture the shared global data structure information, we formulate the representation projecting as the process of extracting instance-wise data information by $G_f(x_i;\theta_f)$ to $P_i$, and then pushing $P_i$ towards $Q_i$ to encourage each prototype $\beta_k$ to capture the shared global data structure information, a process achieved by minimizing the OT distance between $P_i$ and $Q_i$. The OT distance between $P_i$ and $Q_i$ could first be calculated by: $**OT**(P_i, Q_i) = \min_{**T**_i\in \Pi (P_i, Q_i)} \langle **T**_i, **C**_i \rangle$,

where $C_{ik} = 1-cos(G_f(X_i;\theta_f), \beta_k)$, the average OT distance between $P_i$ and $Q_i$ over train sets could be viewed as loss function $L_{projecting}(X, B)$ to be further optimized:

$\min\frac{1}{n}\sum_{i=1}^{n} **OT**(P_i, Q_i) = \min_{\theta_{f}, \gamma, B} \frac{1}{n}\sum_{i=1}^{n} \min_{**T**_i\in \Pi (P_i, Q_i)} \langle **T**_i, **C**_i \rangle$, we use gradient descent to update $\theta_f, \gamma, B$. We also provide the detailed description in Appendix A.7 in the original paper.

The idea of applying OT distances as loss functions has been employed in multiple applications. For example, [1] proposes using this idea for semantic segmentation, and [2] applies this idea to outlier detection in data.

[1] Importance-aware semantic segmentation in self-driving with discrete wasserstein training (Liu, Xiaofeng and Han, Yuzhuo and Bai, Song and Ge, Yi and Wang, Tianxing and Han, Xu and Li, Site and You, Jane and Lu, Jun) [AAAI 2020]

[2] Outlier-robust optimal transport (Mukherjee, Debarghya and Guha, Aritra and Solomon, Justin M and Sun, Yuekai and Yurochkin, Mikhail) [ICML 2021]

We express our sincere gratitude for your valuable efforts and instructive advice to improve the manuscript. We respond to each of your comments as follows:

W1: Further experimental results of applying PTaRL to recent deep models for tabular representation.

R: Thanks for your valuable suggestion. We have applied PTaRL to recent deep tabular model SAINT [1], making it one of our baseline models. We provide additional experiment results in Section 6 Table 1, Appendix Table 5 and Table 6. The main results as follows:

$\uparrow$ represents higher evaluation metric is better for classification and $\downarrow$ represents lower evaluation metric is better for regression. The results show that PTaRL has the ability to generally improve deep tabular models' performance, including recently proposed models such as SAINT.

[1] SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training (Gowthami Somepalli and Avi Schwarzschild and Micah Goldblum and C. Bayan Bruss and Tom Goldstein) [NeurIPS 2022 Workshop]

Q1: Can the proposed PTaRL be applied to generative models?

R: Thanks for your constructive suggestion. We hope that our proposed method could offer valuable insights to enhance generative model representations and it will be an important direction for our future work. Take the research line of deep generative models for representation learning as an example. Deep generative models excel in discerning the true data distribution by learning the distribution of input data within the latent space. In this context, the representation of data in the latent space serves as a comprehensive data representation. It appears that PTaRL offers a potential solution to incorporate global data information through prototypes. Leveraging such global data information to enhance the representation of latent structures aids in better modeling the true data distribution, consequently improving the generative model's performance.