Exploring Data Distillation for efficient generation of Tabular Data

Weakness 1. I do not agree with the claim that privacy protection of the proposed method is stronger than privacy-preserving synthetic data generators. My reason is that if we use the definition of differential privacy (DP) to evaluate privacy protection, then once the synthetic data generator is trained on a synthetic dataset, no matter how many synthetic samples are generated, the privacy protection remains unchanged due to the immunity to post-processing property of DP. However, for the proposed method, the privacy risk will accumulate under differential privacy protection. Using the proposed method 10 times on the same dataset to generate 10 distilled datasets weakens the privacy protection. Therefore, I believe the authors overclaim the advantage regarding privacy protection.

Weakness 2. This paper has some writing issues. The most unacceptable thing in this paper is that there are some \ref appearing as question marks. See Page 6, Lines 317-322. Second, in this paper, the authors frequently mention image data and the corresponding techniques, which gives me the impression that they simply transfer techniques from the domain of computer vision to a new scenario involving tabular data. As a result, the key contribution is obscured in the writing. Third, the authors did not use mathematical notations properly. For example, Equation (1) in Page 3, $[t_0,\ldots, t_N] \in \mathbb{R}^{N\times D}$ makes no sense at all. From my understanding, each $t_i$ is a column vector and $[t_0,\ldots, t_N]$ should be $D \times (N+1)$.

Weakness 3. The main contribution is to tackle the difficulty of heterogeneity of tabular data. However, the technique of employing a unified embedding space is common in the literature. Therefore, this paper is more like a experimental work validating the effectiveness of some common techniques in tabular data.

Weakness 4. In the domain of machine learning, there usually exists a theoretical understanding behind each significant experimental improvement. This paper highlights significant improvements over existing methods but lacks theoretical support. At least the authors should provide some existing results regarding using gradient matching or distribution matching for data condensation.

• The paper shows impressive efficiency gains, but it’s unclear how it performs on extremely large datasets. Could the authors provide further information on how their approach scales in terms of memory and latency?
• The privacy is evaluated through distance-to-closest record (DCR), but this metric may not fully capture potential risks of re-identification in real-world scenarios. Including other privacy metrics or discussing potential vulnerabilities would add depth to the privacy claims.
• The experimental evaluation is limited compared to previous tabular works. It would be beneficial to evaluate the proposed method on a broader range of datasets to demonstrate its generalizability and robustness.

1. The pre-processing method the paper used for pre-processing the features, a mix of numerical and categorical features, is just following related work Zhang et al. It is not clear if using one-hot encoding and linear transformation of numerical features is good enough. It will be good to perform more studies on the effect of pre-processing methods, like piecewise linear encoding in Gorishniy et al., 2022, etc.
2. The main novelty comes from adapting existing data distillation methods used for image domain to a tabular data context. And other components in the proposed framework are directly from related work TabSyn. This makes the contribution a bit incremental.
3. There is no formal definition of privacy. How to measure privacy leakage of the distilled dataset? No membership inference attack evaluation is performed to verify the possible privacy leakage from model trained on distilled dataset.
4. Evaluations only run for 7 datasets. It is not clear how the authors selected the datasets from UCI. It will be good to see more results on datasets from OpenML.
5. More ablation studies are needed to evaluate how different selection of hyperparameters may affect the final performance of the proposed framework.

Reference:

[Gorishniy et al., 2022] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On Embeddings for Numerical Features in Tabular Deep Learning. Advances in Neural Information Processing Systems, 35:24991–25004, December 2022

1. The proposed method requires a pre-trained transformer auto-encoder, which introduces additional computation overhead compared to vanilla dataset distillation approaches. The authors should discuss the trade-off between computation efficiency and efficacy. At the very least, the authors should mention the detailed architectures of the autoencoder. However, I cannot find the details anywhere in the paper, even in the appendix.
2. Following (1), one of the main applications claimed by the authors is efficient parameter parameter search. However, there is no detailed computation cost analysis. I doubt that pre-training a transformer autoencoder may cause more computation costs than directly performing a search on the original dataset. The analysis in L413 is unclear. What is the x and y axes in Figure 5? What would be the FLOPs required by pre-training, distillation, and parameter search?
3. The so-called "privacy" metric DCR is not well-defined. The main paper does not provide sufficient information, making it difficult to judge. It is also unclear why the authors do not report the distance to the closet sample but a modification.
4. As the authors motivate in L104, tabular datasets are often imbalanced. However, the authors only report accuracy in the experiments. I'd recommend the author report F1 and AUC. Otherwise, it is unclear whether the proposed method overfits some classes.
5. How do the authors conduct core-set distillation on tabular datasets in Figure 4? Since the authors have already trained a VAE, K-means can be an informative baseline.

Overall, the work can be interesting and useful in some applications; however, the current form lacks details and contains several editorial flaws.

Editorial comments:

1. The authors misuse \citet and \citep in the entire paper.
2. Many hyperlinks are broken.
3. Figure 3 is never mentioned in the paper.
4. L209 $S_{enc}$ is not defined in Sec. 3-2. There are several typos like this. The authors are encouraged to carefully proofread the paper.

1. The numbering of figures and tables is often incorrect or missing, making the paper difficult to follow. Additionally, some sentences in the appendix are incomplete, suggesting the paper is not fully polished.
2. The captions lack sufficient detail for the reader to understand the figures and tables. For instance, the compression ratios (5% and 10%) in Table 1 are not explained.
3. Although the preliminary section claims that the proposed method considers class imbalance, there are no experimental results to support this. Evidence, such as visualizations or results, is lacking to substantiate this claim.
4. There is confusion around the term “DPC.” It appears in sections 2, 4.1, and Figure 4, but does not seem to be used consistently. A clear explanation of this term would have been helpful.

1. The quality of writing needs a lot of improvement. The paper was very difficult to follow at many places and there were trivial mistakes like all the table references appeared as ??. Particularly, the details of how the distillation strategies are adopted to tabular data are quite difficult to understand. Consider re-writing them more clearly and with mathematical details.
2. The preliminaries would benefit from a formal mathematical definition of what dataset distillation is.
3. Please report all experiments need to be repeated multiple (~10) times and report standard errors.
4. Given the authors claims about their method being private, it would be important to experimentally compare privacy loss against generative models.