Enhanced Model-agnostic Training of Deep Tabular Generation Models

My main worry about the paper is that it seems to have been heavily inspired by the works that presented CTGAN and Stasy, however, it never mentions (nor compares itself to) the shared features of GADGET and the above models. For example, for CTGAN the authors wrote: "CTGAN approximates the discrete variables to the continuous spaces by using Gumbel-softmax". While this is true, one could argue that one of the major contributions of CTGAN is the proposal of modeling the distribution of each continuous feature with a variational Gaussian mixture model. This intuition seems to have heavily influenced this work, however, this is never mentioned. In the same way, the authors of StaSy also propose a self-paced learning algorithm. However, the one presented in this paper is never directly compared to the StaSy one. Could you please provide some insights on the differences?

Also, could the authors explain why they used 10,000 in the denominator to define $\mathbf{v}_k$? Does the number of epochs need to be exactly 10,000? Why?

How many modes were set for CTGAN? If I remember correctly the default number is 10. Would it make sense to train it why more than 1?

I think an ablation study to check what is the impact of the self-paced learning would be really useful to evaluate the impact of each of the two contributions.

Since GADGET can be applied to any model, it is more interesting to check what is its impact on each model rather than comparing SGM-GADGET with all the standard models (e.g. MedGAN). As SGM on its own does already better than MedGAN, what sense does it make to compare SGM-GADGET with it? It would be much more interesting to compare MedGAN vs MedGAN-GADGET.

Minor comments:

• introduce the abbreviation SGM before using it
• page 4: "Itô"?
• page 5: $\mathcal{T} k$ should be $\mathcal{T}_k$
• page 7: why does SGM-GADGET have the best performance among CTGAN-GADGET, FFJORD-GADGET and SGM-GADGET? What is FFJORD-GADGET?

Next I'll list some weaknesses. Altogether, I have found that the weaknesses outweighed the strengths. I'll elaborate below (again not necessarily ordered by importance):

1. There is a reason that tabular data has been underserved by the generative modelling community: it is typically hard to do. Several have argued that this is because you cannot easily anticipate the inductive biases of tabular data, unlike say images or natural language, and that successful approaches to -- and architectures for -- discriminative tabular data tasks are not easily ported into the generative domain. A lack of discussion on these points is a major weakness of the paper: it makes it hard to believe that the techniques used in GADGET will generalize to a wider range of tabular datasets or work particularly well in the wild, including on higher-dimensional data.
   • Further to this point, it may be worth discussing why this technique has not been used for image-based or language-based modelling, as there seems to be no reason why it wouldn't also apply there.
2. On this same line, I think other important discussions / references are missing:
   • It was recently noted that several advances in tabular data generation are often outperformed by SMOTE (Synthetic Minority Over-sampling Technique), which is a simple interpolation-based sampler for tabular data generation. As such I believe that all generative models for tabular data should compare against SMOTE.
   • If Score-Based Generative models are being used, it would make sense to compare against TabDDPM, as it is the most recent and performant SGM-based generative method for tabular data.
   • It also took me a while to realize the deep link between the Gaussianization piece and score-based modelling, which itself uses Gaussian distributions throughout the modelling process and conceivably works better on Gaussianized data. It is perhaps possible to reframe the paper through this lens, as it may be able to answer some questions about why we would expect the method to perform well.
   • Previously, there have been attempts at incorporating Gaussian mixtures into generative modelling. For example, masked autoregressive flows considered using a Gaussian mixture for the latent space distribution to achieve improved expressiveness. However, discussions related to approaches using Gaussian mixtures in generative modelling are sorely missing from this work.
   • The recently-proposed "union of manifolds hypothesis" also appears relevant here.
3. Another weakness of the related work is that the related work is simply presented, but not at all analyzed. I understand there are space constraints but there are other parts of the paper that can be arguably trimmed down or streamlined to accommodate some improved analysis here.
4. I have several thoughts on the clarity and presentation of the paper, which overall make the paper lack polish. The impression I get from reading the paper is that the work was hastily put-together, and that not enough time was taken to produce quality writing. I'll list my thoughts along these lines below:
   • There are a decently high number of typos throughout the manuscript.
   • The exposition at the end of page 3 feels not only unnecessary, but is also quite confusing. The first paragraph of 3.1 should have been sufficient if you were short on space. The discussion may also have been streamlined by presenting through the lens of conjugate priors.
   • Starting on Page 6, the discussion in Section 4.2 becomes incredibly confusing. I don't believe $v_{k, scaled}$ was actually introduced anywhere yet was presented in (7). Then, the paragraph immediately following is almost entirely sentences which are unclear. Figure 4 is also not very helpful. This section needs to be overhauled.
   • In Section 5.2, the "Generative Learning Trilemma" is apparently a focal point of the experimental section, but is not actually introduced. It is awkward to have such a fundamental part of the exposition be completely deferred to the cited reference.
   • RNODE and FFJORD appear to be used interchangeably, between 5.2 and 5.3.
5. There is no limitations section. This suggests a lack of perspective on how this work sits within the overall literature.
6. It would have been nice to see more quantitative discussions about training time and memory usage in the experimental section, considering you are training and storing $K$ times as many models. It is briefly mentioned that you have fewer learnable parameters, but it would be nice to quantify that.
7. Figure 5 appears to be cherry-picked.
8. This paper is all about making Gaussianized sub-problems for easier generative modelling of tabular data. However, the quality of the Gaussianization was never assessed. It would be nice to see some metric for how well the mixture itself models the data, and relate that to the overall quality of generative model based on downstream metrics.
9. An ablation on self-paced learning would be interesting here.

The idea of structuring tabular data into different Gaussian components follows the right direction, but it is certainly not novel. As a reference, see the paper by Ma et al. 2020: "VAEM: a Deep Generative Model for Heterogeneous Mixed Type Data," which should be included in the experimental settings.

The paper does not read well and lacks soundness in referring to and evaluating the different features of the proposed method. For instance, in section 2, the authors merely use a single example to argue that decomposing into K Gaussians is better. In Section 4 I do not see where exactly the method is explained (just in Figure 3?), and it contains definitions that do not define anything! Check for instance Definition 1.

Empirical results are not conclusive, only one database is included in Table 2. Also, how do you extend this to tabular data containing different data types? how do you tackle missing data?

• An important component of the GMM is the number of mixtures K: after reading the paper, I am not super clear on the effect of different K.