CardiCat: a Variational Autoencoder for High-Cardinality Tabular Data

Thank you for taking the time to review our paper. We appreciate your feedback, which has greatly contributed to the enhancement of our work. Your comments have been invaluable in refining the content and strengthening the overall quality of the paper.

That’s correct: the deviation from a traditional ELBO formulation is due to (a) the categorical embeddings parameterization, and (b) the regularization term in the loss. Our categorical embeddings are treated in the loss as the the other numerical features, which in effect do not translate into probabilities directly. This in return deviates from a purely generative probabilistic model in behalf of ”pseudo”-likelihoods. We believe that the traditional ELBO can be still preserved by normalizing the embedding space for each categorical feature (e.g., using a softmax function) so the embedding loss component represent the mean of the normalized deviations from the ”true” embedding (similarly to Gulrajani & Hashimoto (2024)).

Our framework is implemented on a simple VAE architecture in order to cleanly demonstrate the advantages of such embedding framework. Recent diffusion-based models have been showing competitive results against the more conventional VAE and GAN methods (Kotelnikov et al., 2023; Zhang et al., 2023; van Breugel et al., 2024). We believe these recent models do not take away from CardiCat’s contribution because their contribution is parallel. For example, CardiCat can be applied to latent-space diffusion-based models as well (such as in the architecture used by Zhang et al. (2023)), and in combination of other competing models (such as in (Xu et al., 2019; Stoian et al., 2024). Additionally, many of these publications do not assert one model to be a clear winner in all evaluation metrics, with the more ”conventional” approaches still performing well in some (van Breugel et al., 2024).

There are two substantial difficulties with learning distributions with high cardinality variables: a. resampling from the full distribution, including rare categorical values, and b. learning the dependencies between categorical values and other variables. We show substantial gains in both of these elements by looking at marginal and bi-variate distributions. Related works usually include low-order statistics evaluation metrics (by including various marginal and bi-variate density estimation and diversity metrics) and machine learning efficiency metrics (by evaluating the synthetic data against the real data using a classification/regression task). We chose to focus on low order statistics and provide an additional ’mixed’ bi-variate evaluation metric to emphasize the need to assess also mixed-type interactions. We also demonstrated that concentrating on one aggregative score that averages the evaluation metric can be misleading: our results show that different models excel in different metrics. Machine learning efficiency metrics can be misleading as well because they concentrate on a discriminant function and not on the joint distribution efficacy of the generative model.

Once again, thank you for your valuable feedback. Sincerely.

Thanks very much for your response.

Unfortunately my main concern that the baselines are outdated was not resolved, and it is not clear to me that the proposed method would outperform modern models for tabular data (such as the diffusion or GAN approaches you cite). You suggest that you have chosen specifically to demonstrate these changes on "a simple VAE architecture", which I did not find convincing. Vanilla VAEs are known to have plenty of issues, such as posterior collapse and poor sample quality, and therefore it does not feel reasonable to claim that improving over them is an empirical contribution---you should compare to modern baselines and beat them. For example, even if one proposes an improvement to MLPs, they should demonstrate it in some setting of practical interest, such as plugging their method into a modern Transformer architecture and showing reasonable results.

Because this issue is unresolved, I will keep my score unchanged.

Thank you for taking the time to review our paper. We appreciate your feedback, which has greatly contributed to the enhancement of our work. Your comments have been invaluable in refining the content and strengthening the overall quality of the paper.

ANSWER 1 (The relevance of the proposed method):

Ma et al. (2020) is based on a two- stage structure where the first stage individual features are homogenized by learning an independent VAE for each feature. Then, the separately learned factorized latent variables from the first stage models are used as inputs to a second stage VAE. The first stage independent models are trained to learn the marginal distribution of each feature. The second stage model is expected to ”recover” the joint distribution, using only information from the marginal distributions. However, if the features are not independent, there is much more information in a joint distribution than can be captured by its marginal distributions. We tested that empirically, but unfortunately, because the authors’ original code-base is not functional we were not able to use their code directly and had to test their framework using our own implementation of their model. Because of that, and the subsequent subpar performance, we didn’t include the results of our own implementation.

ANSWER 2 ( the objective to diverge from the traditional ELBO used in VAEs):

That’s correct: the deviation from a traditional ELBO formulation is due to (a) the categorical embeddings parameterization, and (b) the regularization term in the loss. Our categorical embeddings are treated in the loss as the the other numerical features, which in effect do not translate into probabilities directly. This in return deviates from a purely generative probabilistic model in behalf of ”pseudo”-likelihoods. We believe that the traditional ELBO can be still preserved by normalizing the embedding space for each categorical feature (e.g., using a softmax function) so the embedding loss component represent the mean of the normalized deviations from the ”true” embedding (similarly to Gulrajani & Hashimoto (2024)).

ANSWER 3 ( The baselines employed are insufficient and outdated):

Our framework is implemented on a simple VAE architecture in order to cleanly demonstrate the advantages of such embedding framework. Recent diffusion-based models have been showing competitive results against the more conventional VAE and GAN methods (Kotelnikov et al., 2023; Zhang et al., 2023; van Breugel et al., 2024). We believe these recent models do not take away from CardiCat’s contribution because their contribution is parallel. For example, CardiCat can be applied to latent-space diffusion-based models as well (such as in the architecture used by Zhang et al. (2023)), and in combination of other competing models (such as in (Xu et al., 2019; Stoian et al., 2024). Additionally, many of these publications do not assert one model to be a clear winner in all evaluation metrics, with the more ”conventional” approaches still performing well in some (van Breugel et al., 2024).

Once again, thank you for your valuable feedback. Sincerely.

Thank you to the authors for their responses. However, most of my concerns, some of which are shared with Reviewer xrEX, remain unaddressed. Specifically:

• Unless I am misunderstanding something, in the VAEM approach, when the data dimensions are not independent, their corresponding marginal posteriors should preserve these interdependencies. This technique is designed as a more balanced approach to effectively capture such relationships, making it a very robust alternative.

• The related work section is outdated. While this can be resolved by rewriting the literature review, including more recent baseline methods is essential to justify the significance of this paper. Comparing CardiCat to older methods does not seem fair or sufficient.

• The objective deviates from the ELBO, not only due to how categorical data is processed but also because the reconstruction term differs from the conditional likelihood. To properly denote an objective as an ELBO, it should ideally form a lower bound on the log-evidence:

  $$\mathbb{E}_{q(z|x)}[\log p(x|z)] - KL(q(z)||p(z)) \leq \log p(x)$$

For these reasons, I will keep my score unchanged.

Thank you for taking the time to review our paper. We appreciate your feedback, which has greatly contributed to the enhancement of our work. Your comments have been invaluable in refining the content and strengthening the overall quality of the paper.

ANSWER 1 (Lack of state-of-the-art comparative methods):

Our framework is implemented on a simple VAE architecture in order to cleanly demonstrate the advantages of such embedding framework. Recent diffusion-based models have been showing competitive results against the more conventional VAE and GAN methods (Kotelnikov et al., 2023; Zhang et al., 2023; van Breugel et al., 2024). We believe these recent models do not take away from CardiCat’s contribution because their contribution is parallel. For example, CardiCat can be applied to latent-space diffusion-based models as well (such as in the architecture used by Zhang et al. (2023)), and in combination of other competing models (such as in (Xu et al., 2019; Stoian et al., 2024). Additionally, many of these publications do not assert one model to be a clear winner in all evaluation metrics, with the more ”conventional” approaches still performing well in some (van Breugel et al., 2024).

ANSWER 2 ( similar or worse performance than tGAN):

The results of the conditional generators can be found in the supplementary material (the table in page 3). While the marginal reconstruction scores are indeed a bit below the tGAN scores, the bi-variate scores all exceed those of tGAN’s. In addition, tGAN employs a ”training by sampling” scheme during training, which can artificially boost the marginal reconstruction of categorical features (where, the conditional and training data are sampled according to the log-frequency of each category). It is evident from the bi-variate metrics that CardiCat was able to learn better the joint-distribution between features.

ANSWER 3 (What if the number of cardinalities is greater than the number of samples?):

If the cardinality of a categorical feature is equal to or almost the number of samples of the data, than this feature is degenerate in the sense that it acts more of an index than a feature that adds a valuable signal. Unlike many other in the literature, we demonstrate our model also on non-trivial datasets in terms of cardinality. This type of high cardinality is often overlooked by many tabular generative models, and we believe our model makes a concrete contribution towards modeling tabular mixed-type data. There is an interest in exploring the cases where the total cardinality of the data/feature is very high relative to the number of samples, but in this paper we don’t focus on solving these type of scenarios.

ANSWER 4 (The evaluation metrics seem to be limited):

There are two substantial difficulties with learning distributions with high cardinality variables: a. resampling from the full distribution, including rare categorical values, and b. learning the dependencies between categorical values and other variables. We show substantial gains in both of these elements by looking at marginal and bi-variate distributions. Related works usually include low-order statistics evaluation metrics (by including various marginal and bi-variate density estimation and diversity metrics) and machine learning efficiency metrics (by evaluating the synthetic data against the real data using a classification/regression task). We chose to focus on low order statistics and provide an additional ’mixed’ bi-variate evaluation metric to emphasize the need to assess also mixed-type interactions. We also demonstrated that concentrating on one aggregative score that averages the evaluation metric can be misleading: our results show that different models excel in different metrics. Machine learning efficiency metrics can be misleading as well because they concentrate on a discriminant function and not on the joint distribution efficacy of the generative model.

Once again, thank you for your valuable feedback. Sincerely.

Thank you for taking the time to review our paper. We appreciate your feedback, which has greatly contributed to the enhancement of our work. Your comments have been invaluable in refining the content and strengthening the overall quality of the paper.

ANSWER 1 (Seemingly inaccurate claim of contribution):

Indeed, TabSyn ( (Zhang et al., 2023)) utilizes a feature tokenizer that converts each column (both numerical and categorical) into a d-dimensional vector. However, they also apply a detokenizer to the recovered token representations in the encoder. There are three main key differences between CardiCat’s embeddings and TabSyn’s tokenizer that makes CardiCat’s embeddings superior. (1) CardiCat’s loss calculation is done directly on the embedding space while TabSyn is done on the original features. (2) The dual-embeddings used in CardiCat is trained to have the same embedding (i.e., weights) between the encoder and decoder (hence the ”dual”), while TabSyn enforces no such parity ending up with two different tokenizers for each feature. (3) CardiCat’s avoids one-hot encoding the categorical features while TabSyn tokenizer expect one-hot encoded categorical features, which can be very expensive in terms of the model’s required learned parameters ( especially for categorical features with high cardinality).

ANSWER 2 (incomprehensive comparison to benchmark methods):

Our framework is implemented on a simple VAE architecture in order to cleanly demonstrate the advantages of such embedding framework. Recent diffusion-based models have been showing competitive results against the more conventional VAE and GAN methods (Kotelnikov et al., 2023; Zhang et al., 2023; van Breugel et al., 2024). We believe these recent models do not take away from CardiCat’s contribution because their contribution is parallel. For example, CardiCat can be applied to latent-space diffusion-based models as well (such as in the architecture used by Zhang et al. (2023)), and in combination of other competing models (such as in (Xu et al., 2019; Stoian et al., 2024). Additionally, many of these publications do not assert one model to be a clear winner in all evaluation metrics, with the more ”conventional” approaches still performing well in some (van Breugel et al., 2024).

ANSWER 3 (valuation metrics are not comprehensive):

There are two substantial difficulties with learning distributions with high cardinality variables: a. resampling from the full distribution, including rare categorical values, and b. learning the dependencies between categorical values and other variables. We show substantial gains in both of these elements by looking at marginal and bi-variate distributions. Related works usually include low-order statistics evaluation metrics (by including various marginal and bi-variate density estimation and diversity metrics) and machine learning efficiency metrics (by evaluating the synthetic data against the real data using a classification/regression task). We chose to focus on low order statistics and provide an additional ’mixed’ bi-variate evaluation metric to emphasize the need to assess also mixed-type interactions. We also demonstrated that concentrating on one aggregative score that averages the evaluation metric can be misleading: our results show that different models excel in different metrics. Machine learning efficiency metrics can be misleading as well because they concentrate on a discriminant function and not on the joint distribution efficacy of the generative model.

ANSWER 4 (unclear descriptions of conditional CardiCat):

The results of the conditional generators can be found in the supplementary material (page 3).

ANSWER 5 (Code is a bit hard to go through):

Thank you for bringing this under our attention. This specific issue was a result of an unnoticed failed doc-string compiler, which is fixed now. As oppose to many published open-source code-bases, we are proud to adhere to pep8 standards in addition to provide rigorous documentation

Questions: Why not treat binary features as categorical features?

In therms of the loss, the embedded categorical relationship between the binary values will stay fixed regardless of the resulting embedding map.

Once again, thank you for your valuable feedback. Sincerely.

I would like to thank the authors for their feedback on my comments. And I acknowledge that their response resolves some of my concerns about the paper. Below are the remaining ones:

1. If CardiCat can work as a pluggable module of any VAE-based models, I would suggest the authors to support the claim with empirical evidence. Otherwise, it is not sufficiently convincing (also noted by Reviewer xrEX).

2. I am unsure if I fully agree with the authors on their feedback for evaluation metrics. I agree with the authors that downstream accuracy is not the optimal metric to demonstrate how well the generative model captures the joint distribution. However, it is still complementary to the low-order metrics, as many downstream models need to attend to higher-order relationships for better generation quality. Therefore, the downstream accuracy can empirically serve as a good "proxy" for the generators' utility. And "imperfection" may not be convincing enough to justify why the authors do not evaluate the downstream performance.

3. The authors' feedback on "Why not treat binary features as categorical features?" remains unclear to me. What does "therms of the loss" refer to? Why does "embedded categorical relationship" stay fixed?

Looking forward to the authors' response.