On Learning Representations for Tabular Dataset Distillation

Weakness 4: Comparative analysis of TDColER against existing methods

We apologize for the lack of clarity and thank you for raising this point. Our proposed method, TDColER, by itself is not a distillation algorithm, so the main scalability of the distillation process depends on the core distillation method being applied. Previous work that compared distillation methods [6,7] place emphasis on the downstream model performance on the distilled data, which is why we also focused on downstream predictive performance. We would also like to note that the cost of distillation is a one-time occurrence, as the distilled data can be re-used for multiple downstream tasks.

The additional resource constraint TDColER adds comes from the encoding and decoding process. As detailed in Appendix A.4.3, all three of our encoder architectures' parameter count linearly scales with $D$, the dimension (number of columns) of the original dataset. Similarly, the encoding and decoding steps also scale linearly with respect to $D$, as the parameter size of all autoencoders are used once during forward pass. We also empirically examine the difference between the autoencoder architectures after HPO on the datasets in Figure 4.

Weakness 5: Theoretical foundations of TDColER

We thank you for this suggestion. This is indeed an important aspect that we have not considered in this work. However, this is a complex topic that requires a thorough analysis and our work focuses on the downstream models' performance. We believe that the effectiveness of TDColER can be attributed to the ability of the column embeddings to capture the relationships between different columns, which is crucial for tabular data analysis and the ability to adapt to any distillation methods that are not tied to a particular model architecture.

Thus, some examples of existing lines of work that can be incorporated to theoretically explaining the effectiveness of TDColER in future works are the following: feature heterogeneity and distribution shift between the generating and downstream models. Feature heterogeneity is a well-known challenge in machine learning [8]. In tabular data, this is particularly challenging as the features can be of different types (categorical, numerical, etc.) and have different scales. In addition, data distillation works proposed for vision datasets make the assumption of NN-based downstream classifiers, which is not applicable in tabular data. Thus, we believe that existing literature on distribution shift may be a good starting point to understand the shift that occurs between the models that are used to generate the data and the models that are trained on the generated data.

References

[1]: "Why Tabular Foundation Models Should Be a Research Priority." ICML 2024 Position Paper

[2]: "Why do tree-based models still outperform deep learning on typical tabular data?" NeurIPS 2022

[3]: "On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm." CVPR 2024.

[4]: "Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching." ICLR 2024.

[5]: "Dataset Distillation by Matching Training Trajectories." CVPR 2022.

[6]: "DC-BENCH: Dataset Condensation Benchmark." NeurIPS Datasets and Benchmark Track, 2022.

[7]: "New Properties of the Data Distillation Method When Working With Tabular Data." AIST, 2020.

[8]: "Deep Neural Networks and Tabular Data: A Survey." IEEE transactions on neural networks and learning systems, 2022.

We thank you for your detailed review and insightful comments. We have addressed each of your concerns pointed out in the weakness below. It would be great to hear your feedback on our responses.

Weakness 1: Unclear motivation for tabular data distillation

Thank you for pointing this out. We apologize for any lack of clarity in our motivation behind this work. We believe that data distillation for tabular data is an important aspect that has many implications, such as storage/operational cost saving, as well as a privacy mechanism. [1] also points out the need for more research into tabular data, a modality that has been overlooked in comparison to images and text in recent days. Thus, the main rationale behind our work is the following: "Can we apply data distillation techniques developed on image datasets directly to tabular data? If not, what adjustments are needed/helpful?". The main challenges in applying these methods are as mentioned in Section 2: Feature heterogeneity and model agnosticity, where the model may not be a differentiable (NN-based) model. We also further clarify our usage of these two terms in the next response. Our goal is the examine whether these challenges are indeed valid when applying distillation methods directly on tabular data -- and our experiments show that this is indeed the case, where the traditional SotA model (XGBoost) does not gain as much from NN and architecture-specific distillation algorithms. Our results also show that the feature heterogeneity is indeed another challenge for all distillation algorithms, demonstrating a significant improvement in downstream performance when applying TDColER (latent space projection) in combination with the distillation algorithms.

Furthermore, we show that there is indeed a merit in applying data distillation on tabular data in our HPO use case, which showed that conducting HPO with the distilled data on averages uses only 21.84% of the runtime compared to using the full data, while yielding up to 98.37% of the performance. This can be a critical benefit in time-sensitive scenarios where some trade-off between computation time and performance is acceptable. We hope that this clarifies our motivation and contributions.

Weakness 2: Lacking clarity in key terms and concepts

Thank you for pointing this out. We apologize for any confusion in the terms we used in this work. To further clarify what we meant, below are a more elaborate description of the two issues that are particular to tabular data:

Feature heterogeneity: In image data, each feature corresponds to a pixel. In language, token. But in tabular data, it is not as clear as some features may be categorical/ numerical or even binary, and have missing values. Thus, a feature in tabular datasets can mean many different things in terms of its semantics, data type, and also its magnitude. Past work has identified that this is one of the key factors of tabular data that causes NNs to struggle when compared to gradient boosting models [2].

Non-differentiable model: We realize that we never formally defined this term. When we say non-differentiable learning model, we are referring to any ML model which cannot be used to produce gradients of the prediction output with respect to the input (i.e. cannot be learned through back-propagation). We mainly use this term to differentiate between NN-based models and those that are not NN-based, which essentially refers to XGBoost, K-Nearest Neighbors and Gaussian Naive Bayes in our work. We apologize any confusion in using this term and will update the manuscript with a more clear definition.

Weakness 3: Is distillation of tabular data alone worth it?

Thank you for suggesting these works. While [3] is not directly applicable to tabular data due to their usage of image patches, which is not a concept that translates natively to tabular data, we have incorporated [4], and its predecessor [5] in to our additional analysis. Please find the detailed report that was added to Appendix D.1.

In addition, we would like to note TDColER itself is not a distillation algorithm -- rather, it is a wrapper on top of an arbitrary distillation algorithm to improve its application on tabular data. This was the reason we only considered off-the-shelf methods and test the difference in their effectiveness with and without TDColER. Our experiment results also show that distillation methods developed for vision data do not apply very well on tabular data, especially when the downstream classifier is not an NN model. We would also like to highlight that another main contribution of this work -- the TDBench python package -- allows for a quick prototyping and implemention of new distillation methods and can be easily extended by the community in the future as well.

We thank you for your detailed review and insightful comments. We have addressed each of your concerns pointed out in the weakness below. It would be great to hear your feedback on our responses.

Weakness 1: No comparison to existing methods designed for tabular data

Thank you for bringing up this point. This was in fact one of the main motivations behind this work. We are not aware of any established works that tackle dataset distillation/condensation and proposed a novel method specifically for tabular data. In fact, our goal is to establish such a baseline in tabular data, similarly to [1] in computer vision, for future researchers to propose and test potential state-of-art methods using our benchmarking suite.

Weakness 2: Missing details of column embeddings

Thank you for pointing this out. Column embeddings are indeed a crucial part of our proposed TDColER, and was adapted from the feature tokenzier method seen in [2]. We further discuss the details behind the training objective, fine-tuning process and the architecture of the encoders in Appendix A.4, which also discuss the column embedding module in more detail. We also include visual descriptions of the architectures in Figures 8,9,10,11 of the Appendix. In particular, Figures 8 and 9 describe how we handle the column embeddings explicitly. We will also edit the main manuscript to make this detail easier to find.

Weakness 3: Is distillation of tabular data alone worth it?

Thank you for raising this question. This is indeed a critical point to address when examining data distillation solely from the lens of uni-modal tabular data. We believe that the merit of dataset distillation is not limited to storage space. For example, our HPO use case experiment shows that conducting HPO with the distilled data on averages uses only 21.84% of the runtime compared to using the full data, while yielding up to 98.37% of the performance. In time-sensitive scenarios where some trade-off between computation time and performance is acceptable, using the distilled data can make a critical difference. In addition, there are also cases such as those involving privacy or copyright infringement, where the original data cannot be stored indefinitely, and eventually need to be discarded. In such cases, distilled data can also prove to be a valuable proxy that can provide a compromise between regulation/privacy and utility (e.g. downstream performance). We would also like to point to the fact that while many publicly available tabular datasets are smaller in storage compared to other modalities, there are still cases of large tabular data that exists in internal resources of many organizations. One good example is web data, where there is a constant stream of user data every day. We argue that one of the main roadblocks that prevent a full utilization of such large data is the scale, which dataset distillation can help with.

References

[1]: "DC-BENCH: Dataset Condensation Benchmark." NeurIPS 2022.

[2]: "Revisiting Deep Learning Models for Tabular Data." NeurIPS 2021.

Weakness 3: More recent distillation methods

We apologize for any confusion in the naming convention. Our results contain all results seen in [1], including that of Table 1. However, we have modified how we refer to some of the methods (e.g. NNC $\to$ KNearestNeighbors) that may have lead to some confusion. Additionally, although we have results for Naive Bayes Classifier (NBC), we drop it from the analysis because the classifier shows inferior performance on full data, which lead to an inflated performance on the distilled datasets when aggregating with other classifiers. This can be verified in the results csv files that are included in the supplementary materials.

During this rebuttal phase, we also have added four representative deep-coreset-selection methods from [2] as well as a more up-to-date gradient-based distillation methods from [3] and [4] during this rebuttal period. We would also like to highlight that the TDBench framework package allows for quick prototyping of new distillation methods and can be easily extended. The detailed results from these additional comparisons can be found in the modified Appendix D.1.

Question: Which part of the design is specifically applicable for tabular data?

The design of TDColER is specifically tailored to address the challenges unique to tabular data by using of column embeddings to represent both categorical and numerical features. The column embedding-based representation learning allows the autoencoder to capture the relationships between different columns. By learning the latent projection of the dataset, TDColER enhances the effectiveness of the distillation algorithm, reducing the data size while preserving the essential information. This is particularly important for tabular data, where the number of features can be large, and the data size can be substantial. In addition, our analysis shows that the conversion of the dataset from a sparse one-hot representation to a dense space is beneficial for both distance-based distillation algorithm (clustering) and NN-based algorithms (GM, KIP, MTT, DATM, deepcore).

Another key aspect we consider is that the best performing models for tabular data are usually not NN-based models [5,6]. Another key contribution in our work is the ablation tests to check which distillation methods are most applicable to such models. Our finding shows that while the learned latent space is useful, NN architecture-specific distillation methods developed for vision data are not so useful for SotA models like XGBoost, which we believe is a valuable contribution to the community.

References:

[1]: "Effective Data Distillation for Tabular Datasets." AAAI, 2024

[2]: "Deepcore: A comprehensive library for coreset selection in deep learning." International Conference on Database and Expert Systems Applications, 2022.

[3]: "Dataset Distillation by Matching Training Trajectories." CVPR, 2022.

[4]: "Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching." ICLR 2024.

[5]: "Why do tree-based models still outperform deep learning on typical tabular data?" NeurIPS Datasets and Benchmark Track, 2022.

[6]: "When Do Neural Nets Outperform Boosted Trees on Tabular Data?" NeurIPS 2023.

We thank you for your detailed review and insightful comments. We have addressed each of your concerns pointed out in the weakness below. It would be great to hear your feedback on our responses.

Weakness 1: Limited novelty, difference from previous work

Thank you for pointing this out. In addition to all the methods and pipelines covered in [1], this work considers an additional transformer-based autoencoder architecture and and the Gradient Matching (GM) distillation method. In particular, while our finding also shows that using the latent space is useful, we further back this finding empirically by conducting additional ablation studies on affects of different binning schemes. It is also worth noting that [1] is a two-page student abstract and has no code artifacts, while we provide a much more in-depth analysis and a complete python package that can be used by the community. We have also added a detailed comparison of the difference between this work and [1] which can be found in Appendix D.3.

Weakness 2: Unclear how the proposed framework addresses challenges for tabular data

Our proposed method, TDColER, aims to address feature heterogeneity by employing a uniform binning strategy to process the input data. TDColER ensures that the model can effectively learn from the data, regardless of the feature type. The column embeddings of TDColER allows for native handling of numerical and categorical data, as well as missing data. We also show empirically that using the column embeddings greatly improves the downstream performance of all families of distillation methods -- clustering, gradient-based and kernel-based -- considered in benchmark. Additionally, the model agnosticity challenge is addressed by leaving a flexible choice of distillation algorithm. In addition, depending on the type of models, we also leave the option to use either the dense (embedding) representation or the original representation for downstream tasks. This flexibility allows for a fair comparison between differentiable and non-differentiable models, and allows users to choose the most suitable representation based on the specific requirements of their application. Our experiments show that while the usage of latent embeddings for downstream classification is shown to be helpful in most cases, model-specific distillation methods borrowed from computer vision are consistently outperformed by simpler clustering-based baselines, again highlighting the differences we note between tabular and vision data. Our results justify the need for further development of model-independent distillation schemes that are targeted towards tabular data.

We thank you for your detailed review and insightful comments. We have addressed each of your concerns pointed out in the weakness below. We would love to hear your feedback on our responses.

Weakness 1: The evaluation is based on a custom metric, relative regret, rather than on common metrics such as accuracy or mean squared error.

Thank you for pointing this out. We agree that using regret as the primary evaluation metric may not be intuitive for all readers. However, we chose this metric because it allows us to make a fair comparison between the performance of different distillation methods across multiple datasets with respect to random sampling. We initially conducted our analysis by averaging the raw balanced accuracy score, but found that the results can be misleading in certain cases. This approach is inspired by [2], which benchmarks AutoML packages and aggregates the per-dataset performance scaled by the performance of random forest model to model the performance gain that occurs from different packages. Our regret metric is defined as the following: $(A_F-A)/(A_F-A_{R10})$, where $A_F$ is the performance from full-data and $A_{R10}$ is the performance on 10 randomly sampled points. The normalizing term $A_F-A_{R10}$ is used to scale the difference between the full data and random sampling performance. Similar to [2], the intuition is as the following: since random sampling is the simplest form of distillation, we should be aware of how well it performs in that particular setting when looking at other distillation methods. This prevents cases where a distillation method's downstream performance appears artificially high. In our analysis, we found that simple averaging over all of our 23 datasets can lead to misleading results. We will make this more clear in our footnote when introducing this metric.

For example, consider the following case:

• On dataset A, the full-data performance is 0.9, random sampling performance is 0.5, and method X's performance is 0.6.
• On dataset B, the full-data performance is 0.7, random sampling performance is 0.5, and method Y's performance is 0.6.

At a glance, it may appear that method X and method Y yield the same performance. However, this is not the case in reality, as dataset B is a much harder dataset. By computing the regret instead, we are able to get a fairer understanding of how each methods perform with respect to the lower (random sample@IPC=10) and upper (full data) bounds.

Weakness 2: Analysis of impact on inter-column correlations

We appreciate you bringing up this angle, and we agree that this is a very interesting topic to further branch into. In this work, we mainly focus on downstream performance based on the assumption that if there are correlations between features that are useful for the downstream task, losing such correlations will be reflected in degraded downstream performance. In addition, we also report very high reconstruction accuracy from our autoencoders, which also suggests that the correlations are indeed picked up by the encoder and decoders. Another complication is that the best downstream performance occurs when the distilled data is in the latent space, which makes it difficult to conduct a 1-on-1 comparison against features in the original space. This said, we do agree that an in-depth analysis to confirm whether this indeed happens is valuable. We conducted some preliminary analysis into whether our best performing pipelines preserve such pairwise feature correlations and have added it to Appendix D.2. Please refer to the modified text for further details into this analysis.

Weakness 3: Security, privacy and bias analysis of TDColER

We thank you for raising this valuable point. This is indeed an interesting direction for data distillation and a very natural extension of this work. However, we believe that the scope of this question can be quite broad as it can be examined from many possible use cases and components to be examined (leakage from embeddings, distillation method, decoding step, etc.) and that it deserves a dedicated work to be fully answered. This work aims to serve as a starting point for data distillation in the tabular data domain. Following suite from previous work that examined data distillation methods [3], we focus on comparing the downstream performance of the distilled data. Extending upon this, future works can examine best performing methods identified from our framework from the lens of privacy/bias mitigation against downstream utility. Simple analysis includes examining robustness against attacks like membership inference (privacy), or examining and mitigating bias on sensitive attributes by modifying distillation algorithms to use weighted sampling methods. These are just some ideas we could brainstorm, but we believe that there is much more to these questions that can reveal interesting insights. Again, thank you for raising this question, and we hope to answer it in our following works.

Weakness 4: Lack of comparison with existing data condensation methods (Deepcore)

Thank you for this suggestion. We have incorporated 4 representative methods from [1] (forgetting, GraNd, Glister and Graph Cut) and 2 additional gradient trajectory-matching methods [5,6] to our benchmarks and show the results in Appendix D.1. Due to limited time during the author-reviewer discussion, we only consider XGBoost classifier with TF-SFT encoder. We find that the results among the four methods on our framework is consistent to the results reported in [1] -- Graph Cut shows best performance overall while other methods are indistinguishable over different datasets. However, we also observe that the Deepcore methods, which are based on specific NN architectures, do not generalize well for non-differentiable models like XGBoost and also fall behind the gradient-matching methods from [4,5,6].

We would also like to bring to attention that one of our contributions -- the TDBench framework -- was designed with extensibility in mind and will be open to public and anyone can add their own methods to compare against our results (as we were able to incorporate these 4 deepcore distillation schemes and 2 trajectory-matching schemes).

References:

[1]: "Deepcore: A comprehensive library for coreset selection in deep learning." International Conference on Database and Expert Systems Applications, 2022.

[2]: "Amlb: an automl benchmark." Journal of Machine Learning Research, 2024.

[3]: "DC-BENCH: Dataset Condensation Benchmark." NeurIPS Datasets and Benchmark Track, 2022.

[4]: "Dataset Condensation with Gradient Matching.", ICLR, 2021.

[5]: "Dataset Distillation by Matching Training Trajectories." CVPR 2022.

[6]: "Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching." ICLR 2024.

Thank you for your feedback and engagement with our work. We appreciate your recognition of the new experiments we have conducted on feature correlations. We agree that reporting the raw values can provide a more intuitive way to understand the results. We have updated the appendix to include additional comparisons using our raw metric (balanced accuracy score) to the appendix to further back demonstrate the effectiveness of 1. using TDColER to distill, 2. distilling with simpler clustering baselines compared to sophisticated vision-based methods. Tables 19~24 of Appendix D.4 show the raw values and each method's win rates vs. random sampling and the number of instances each method ranks best for the dataset/classifier setting. Please note that only table 19 includes the methods that were added during the rebuttal period due to time constraints. We will run the rest of the experiments and add them in the future. These tables show that applying TDColER instead of naive distillation indeed raises the win-rate of all distillation methods against random sampling. It also shows that agglomerative and $k$-means clustering show superior performance to vision-based distillation methods. These results are consistent with our aggregate results shown using the regret score in the main manuscript. We hope that these additions address your concerns and improve the overall quality of our submission. We appreciate your time and feedback, and thank you for adjusting your score. We would appreciate it if you could share whether you feel there are other outstanding issues with our submissions that can be amended.