LATABLE: TOWARDS LARGE TABULAR MODELS

Dear Reviewer YeUJ, many thanks for your insightful review. We thank you for acknowledging and summarizing succinctly the strengths of this paper. The questions you raise are important, and we hope the community recognizes the significance of discussing these questions. Here are our responses to the points you have raised:

Weaknesses:

1. Indeed, the notation is rather overwhelming and complex; even though this comes with the advanced setting (cross-dataset generation with tabular transformers and diffusion models), we agree the readability and clarity can be improved. In particular, we have included pseudo-code for training as well as generation in Appendix A.
2. You correctly note that Hollmann, Yak, and Zhu build tabular foundation models trained on multiple datasets, so do (Zhang et al, 2023) and (Dinh et al., 2022). However, we note that these approaches focus on supervised learning, and not generative learning. This implies we cannot compare directly to them, as our method does not provide predictions, and it is non-trivial to extend their method to generation. We believe the suggestion may stem from us using downstream model performance (i.e. TSTR a.k.a. Model Efficiency), which we could compare to their works. However, we note this comparison is not entirely valid: the TSTR metric is dependent on which downstream classifier architecture we use—finding the best classifier is beyond the scope of this work and we use the standard CatBoost model. Consequently, any generative model’s TSTR performance (including LaTable) could in practice be improved further, and comparing this to a supervised foundation model performance would not be entirely fair. To the best of our knowledge, the only generative tabular foundation model is currently GReaT.
3. Thank you for raising this. Although LaTable can be extended to conditional generation, the scope of this work is only unconditional generation. We agree that conditional generation is also promising for prediction. Future work could consider how conditional generation can be made efficient (i.e. conditional generation models $P(Y|X)$, whereas standard supervised learning algorithms are only interested in a single statistic $E(Y|X)$), and whether a generative approach to prediction can provide heuristic uncertainty (e.g. even for numericals we have access to $P(Y|X)$ and could thus approximate variance or other statistics).
4. Thank you for raising this. The discriminator metric used in (Borisov et al., 2023) we do not include: this is not very insightful for the few-shot setting. This metric achieves an almost perfect score (AUC>0.98) on both LaTable and baselines, by finding simple generation artifacts. These generation artifacts are deemed to occur, since the generative models have only seen a small number of real samples, and cannot therefore be expected to approximate the distribution perfectly. On the other hand, the discriminator used in the metric is trained on the real test set and synthetic sets (each a few thousand), and will be able to find and exploit artifacts easily. The metric could be changed to a simpler model to make it more fair. For example, we tested a linear regression model (with categoricals one-hot encoded), for which we found discriminator performance: 0.64, 0.62, 0.63 for TVAE, GREAT, and LaTable, respectively. These numbers fall do not show significant differences, hence we unfortunately find these not insightful enough for the paper.

We hope this response has addressed your central concern. Please see our general comment for more details about the experiment setup. We will continue to expand on the limitations outlined in the paper.

Dear Reviewer SWVR, many thanks for your constructive feedback. We thank you for clearly raising your confusions and concerns when reading the paper. Indeed, the notation is rather overwhelming and complex; even though this comes with the advanced setting (cross-dataset generation with tabular transformers and diffusion models), we agree the readability and clarity can be improved. In particular, we have included pseudo-code for training as well as generation in Appendix A. We hope this response has addressed your central concern. Please see our general comment for more details about the experiment setup. We will continue to expand on the limitations outlined in the paper.

Dear Reviewer avjV, many thanks for your constructive feedback. We thank you for clearly raising your confusions and concerns when reading the paper. Here are our responses to the central question you have raised:

We wish to clarify one key misunderstanding about our experiment setup, namely that we operate in the few-shot setting. Empirically, we notice that asking the model to generate synthetic data without any real data samples is unrealistic. Therefore, we make a weaker assumption, and assume that our application scenario is in the low-data regime. We show the model a very small portion of the calibration data, which is emphasized on both lines 300-308 as well as line 347 on page 6 of the manuscript. The whole picture of our experiment procedure is summarized as follows:

1. We begin with a tabular model trained to completion and a calibration dataset. We will use the dataset Adult to illustrate our point.
2. We perform minor fine-tuning of the tabular model on 100 data samples from Adult (which is very small for synthetic data generation models), and we generate synthetic data given the metadata from Adult.
3. We fit a well-known classifier model to the generated synthetic data, and evaluate on real data using downstream performance metrics such as accuracy, f1 score, and rocauc. We use the CatBoost classifier in this paper.

We hope that we have clarified the experiment setup. The metrics’ significance should be clear now. Even though the model has seen a few samples of real data, memorizing those data will not help the model generate the entire calibration dataset. Understanding about the dataset is still required from the model because the generated synthetic dataset is orders of magnitude larger than the few-shot data.

Regarding your question, "WhereIsAI/UAE-Large-V1" refers to the Universal Angle embedding model [1] (also shown in the footnote of the paper). We hope this response has addressed your central concern. Please see our general comment for more details about the experiment setup. We will continue to expand on the limitations outlined in the paper.

[1] https://huggingface.co/WhereIsAI/UAE-Large-V1

Dear Reviewer pV4v, many thanks for your constructive feedback. We thank you for clearly raising your confusions and concerns when reading the paper. Here are our responses to the questions you have raised:

We clarify that TabDDPM is one of our baselines, and TabDDPM achieves good performance in our experiments. We plan to release the codes upon paper acceptance. We thank you for raising questions about other synthetic data metrics. Aside from your suggestions, additional metrics for synthetic data include coverage, density, and the Discriminator metric suggested by another reviewer. We remark that there is a reason that we don't include these metrics, namely that they are not very insightful for the few-shot setting. For example, the Discriminator metric used in (Borisov et al., 2023) achieves an almost perfect score (AUC>0.98) on both LaTable and baselines, by finding simple generation artifacts. These generation artifacts are deemed to occur, since the generative models have only seen a small number of real samples, and cannot therefore be expected to approximate the distribution perfectly. On the other hand, the discriminator used in the metric is trained on the real dataset and synthetic sets (each a few thousand), and will be able to find and exploit artifacts easily. The metric could be changed to a simpler model to make it more fair. For example, we tested a linear regression model (with categoricals one-hot encoded), for which we found discriminator performance: 0.64, 0.62, 0.63 for TVAE, GREAT, and LaTable, respectively. These numbers fall do not show significant differences, hence we unfortunately find these not insightful enough for the paper.

We hope this response has addressed your central concern. Please see our general comment for more details about the experiment setup. We will continue to expand on the limitations outlined in the paper.