Understanding and Mitigating Memorization in Diffusion Models for Tabular Data

[W3] The methodology is naive. One example is when there is a violation of feature dependencies — In the Adult dataset, Education and Education-Num are related. Swapping one without the other can create inconsistent samples.

A3: Thank you for raising this insightful comment. We have thoroughly addressed this issue through three perspectives.

• Hyperparameter Control in TabCutMix for Balancing OOD Risk and Data Utility. TabCutMix provides a hyperparameter, the augmented ratio, to control the number of augmented samples included in training. This allows for careful tuning to balance between reducing memorization and preserving the utility of the augmented data.

• Proposed TabCutMixPlus with Adaptive Feature Exchange. To further mitigate the risk of creating OOD data, we developed TabCutMixPlus, which incorporates an adaptive feature exchange strategy. Specifically, we calculate feature correlations using metrics like Pearson’s correlation coefficient (for numerical features), Cramér’s V (for categorical features), and the squared ETA coefficient (for numerical-categorical pairs). These measures allow us to cluster highly correlated features, ensuring that features within the same cluster are exchanged together during augmentation. This process preserves the structural integrity of the data and significantly reduces the likelihood of generating OOD samples.

• OOD Detection Experiments for Quantifying OOD Risk. To assess the OOD risk introduced by TabCutMix and TabCutMixPlus, we conducted dedicated OOD detection experiments. Following [1], the OOD detection experiments framed the problem as a classification task, treating normal samples as negative and OOD samples as positive, where the positive OOD samples were synthesized by scaling a randomly selected numerical feature by a factor of $𝐹=100$ or choosing a random value for categorical features from existing categories. A multi-layer perceptron (MLP) was trained on the original data and evaluated on augmented samples generated by TabCutMix and TabCutMixPlus to assess the proportion of samples classified as OOD. As shown in Table. 4, TabCutMixPlus consistently reduces the OOD ratios compared to TabCutMix across all datasets. For example, in Adult dataset, OOD ratio reduced from 2.06% (TabCutMix) to 0.36% (TabCutMixPlus). These results demonstrate that TabCutMixPlus significantly mitigates OOD risks while maintaining robust classification capabilities, reinforcing its utility in synthetic data augmentation workflows.

The table below shows the F1 scores and OOD ratios (with standard deviations) for TabCutMix and TabCutMixPlus across various datasets.

TabCutMixPlus shows a significant reduction in OOD ratios compared to TabCutMix, particularly in datasets such as Default and Cardio, while consistently maintaining high F1 scores.

[W4] Limited Datasets.

A4: The proposed TabCutMix method relies on exchanging feature samples within the same class label, which inherently limits its applicability to classification tasks. Consequently, we are unable to validate the method on regression tasks due to this methodological constraint.

To address your concern and broaden the scope of evaluation, we have added three additional classification datasets to the benchmark: Churn, Cardio, and Wilt. Please see Appendix E.4 for additional results on these datasets.

We thank the reviewer for the constructive comments.

[W1] Which features are you removing specifically for the examples? The features you remove can potentially affect the memorization ratio. Would removing highly correlated features be more impactful on the memorization ratio than less correlated features?

A1: Thank you for raising this important question about the impact of feature removal on memorization ratio. In the current experiments, features were removed randomly without considering their correlation or importance. While this approach simplifies the analysis, we acknowledge that removing highly correlated or influential features could have a more pronounced effect on memorization ratios. Features with strong correlations often capture key patterns in the dataset, and their removal might significantly alter the data manifold, potentially reducing memorization further.

[W2] Theoretical Analysis in this section and the referred appendix is nice but trivial. The following information could be inferred from the EDM paper which is what TabSyn uses. Additionally, there is no citation of the EDM paper either.

A2: We want to clarify that our analysis is independent of the EDM paper and is specifically designed to address memorization phenomena in tabular diffusion models, which is a distinct research focus. While the EDM paper [1] explores the design space of diffusion models for image generation and emphasizes efficiency and quality improvements, it does not investigate memorization issues, particularly in the context of tabular data.

Our theoretical contribution is unique in explaining why memorization occurs in tabular diffusion models, providing insights into mitigating these issues. These aspects are unrelated to the scope of the EDM paper, which focuses on architectural and sampling improvements in diffusion models for vision tasks.

To avoid confusion, we will cite EDM paper and add more discussion on the differences.

[1] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." Advances in neural information processing systems 35 (2022): 26565-26577.

Thank you for the experiments. I have raised my score from a 3 to a 5. However, I still remain conservative as the method from a theoretical perspective is trivial based on existing diffusion papers such as EDM.

We thank the reviewer for the constructive comments.

[W1] TabCutMix contaminates the pair-wise correlation in the training dataset. Therefore, I am skeptical about the Trend Score in Table 1, which shows applying TabCutMix has very little effect on the pairwise correlation, which is pretty counter-intuitive.

A1. TabCutMix provides a hyperparameter, the augmented ratio, to control the number of augmented samples included in training. This allows for careful tuning to balance between reducing memorization and preserving the utility of the augmented data.

Additionally, we design OOD detection experiments for quantifying OOD risk. To assess the OOD risk introduced by TabCutMix and TabCutMixPlus, we conducted dedicated OOD detection experiments. Following [1], the OOD detection experiments framed the problem as a classification task, treating normal samples as negative and OOD samples as positive, where the positive OOD samples were synthesized by scaling a randomly selected numerical feature by a factor of $𝐹=100$ or choosing a random value for categorical features from existing categories. A multi-layer perceptron (MLP) was trained on the original data and evaluated on augmented samples generated by TabCutMix and TabCutMixPlus to assess the proportion of samples classified as OOD. As shown in Table. 4, TabCutMixPlus consistently reduces the OOD ratios compared to TabCutMix across all datasets. For example, in Adult dataset, OOD ratio reduced from 2.06% (TabCutMix) to 0.36% (TabCutMixPlus). These results demonstrate that TabCutMixPlus significantly mitigates OOD risks while maintaining robust classification capabilities, reinforcing its utility in synthetic data augmentation workflows.

The table below shows the F1 scores and OOD ratios (with standard deviations) for TabCutMix and TabCutMixPlus across various datasets.

TabCutMixPlus shows a significant reduction in OOD ratios compared to TabCutMix, particularly in datasets such as Default and Cardio, while consistently maintaining high F1 scores.

We thank the reviewer for the constructive comments.

[W1&Q1] Please compare the proposed mixed-distance against l2 distance with one-hot encoding for categorical variable, and explain the essential difference between the two metrics. The proposed metric may still suffer from issues such as sensitivity to outliers, a limitation common to other distance-based measures.

A1: Thank you for raising the need to compare the proposed mixed-distance metric with $L_2$ distance using one-hot encoding for categorical variables and for highlighting the issue of sensitivity to outliers in distance-based measures. To clarify, $L_2$ distance with one-hot encoding for categorical variables is mathematically equivalent to the indicator distance in our mixed-distance metric, differing only by a constant factor. Specifically, the $L_2$ distance between two one-hot encoded vectors equals $\sqrt{2}$ when 0 when they are the same, which aligns with the binary indicator distance. The key distinction of our approach lies in combining the indicator distance for categorical features with normalized $L_2$ distance for numerical features, ensuring compatibility across mixed-type tabular datasets. Additionally, our metric normalizes distances for all features into the same range to prevent any single feature type from dominating the overall distance calculation. This normalization is essential for ensuring balanced contributions from both numerical and categorical variables during distance computations.

We agree that sensitivity to outliers is a limitation shared by most distance-based metrics, including the proposed mixed-distance metric. This is a valid concern that warrants further research. Future work could explore robust scaling techniques or distance-weighted adjustments to mitigate the impact of outliers, particularly in mixed-type tabular datasets. However, we would like to highlight that the primary contribution of this work is to deepen the understanding of memorization in tabular diffusion models and to propose methods for mitigating it. The mixed-distance metric is introduced as a practical tool within this scope, rather than as a general-purpose distance metric. We acknowledge the importance of outlier sensitivity as a promising direction for future investigation.

[W2] Limited Dataset Benchmarking. More datasets are needed to make a convincing conclusion.

A2: The proposed TabCutMix method relies on exchanging feature samples within the same class label, which inherently limits its applicability to classification tasks. Consequently, we are unable to validate the method on regression tasks due to this methodological constraint.

To address your concern and broaden the scope of evaluation, we have added three additional classification datasets to the benchmark: Churn, Cardio, and Wilt. Please see Appendix E.4 for additional results on these datasets.

[W3] More Baselines: To validate TabCutMix’s usefulness, the paper should compare it to other established “shallow” perturbation techniques, such as SMOTE or Mixup, to better illustrate its advantages. A3: We have implemented the baseline methods, SMOTE and Mixup, and provided their evaluation results in Table 1 and Table 5. The results demonstrate that our proposed TabCutMix and TabCutMixPlus achieve more effective memorization ratio mitigation in most cases while offering a superior trade-off between data generation quality and memorization control.

[Q3] In the case study: does the resulting sample of TabSyn + TabCutMix now closely resemble another real example? Or is the distance to close real example increased?

A4: Based on Table 2, the sample generated by TabSyn + TabCutMix does not closely resemble another specific real example. Instead, it increases the distance to the nearest real sample by altering features like age (e.g., changing from 47.0 to 36.0) while preserving key categorical relationships such as workclass ("Private") and marital status ("Divorced"). This demonstrates that TabCutMix introduces diversity and reduces the risk of memorization.

[Q4] The tested dataset have 10k+ samples and are still fairly large with 10% subsampling. Does the same behaviors hold for more realistic small-scale dataset with <= 200 samples? Consider repeating the analysis in section 3 on the following benchmarking datasets: Insurance, Indian Liver Patient, Titanic, Obesity. You can also try smaller subset percentages such as 0.1%, 1%.

A5: We have conducted more experiments for an additional 3 datasets and smaller subset percentages 0.1%, 1%. After conducting experiments on smaller percentages datasets, we revise our observation in Section 3.3 and find the consistent higher memorization ratio for all datasets. Please see Figure 3 and Table. 5 for detailed experimental results.

We sincerely thank the reviewer for their constructive comments and for raising their score. We greatly appreciate your acknowledgment of our efforts to address the evaluation principles, the effectiveness of TabCutMix, and its advantages over SMOTE in balancing memorization reduction and generation quality.

Regarding your remaining concerns about the completeness of memorization metrics, we acknowledge that standard distance-based metrics, including ours, can face challenges such as sensitivity to outliers and the curse of dimensionality. These are indeed important issues, and we agree that incorporating inference attack-based methods could further enhance the benchmarking of memorization. We added the discussion in future work (Appendix G).

However, we respectfully disagree with the assertion that our mixed-type distance metric closely resembles standard DCR metrics except for a normalizing constant. As detailed in Appendix D.5, DCR’s dependence on the holdout set limits its robustness, as the results can vary with changes in the holdout set composition. Additionally, the mixed-type distance metric we use is equivalent to the $L_2$-distance applied to one-hot encodings for categorical features, except for the constant factor. This approach ensures that our metric handles categorical and numerical features in a unified manner while maintaining interpretability. Unlike DCR, our memorization metric directly assesses overfitting within the generative process and is independent of holdout set composition, addressing a key limitation of DCR.

Once again, we thank the reviewer for the thoughtful suggestions and the revised rating. We are encouraged by the feedback and will continue to refine and enhance our approach to address these open challenges in future research.

I appreciate the detailed and thoughtful responses, and I apologize for the delay in providing my feedback. My concerns regarding the evaluation principles and the effectiveness of TabCutMix have been addressed. The additional comparison with SMOTE demonstrates that TabCutMix achieves a better balance between reducing memorization and preserving generation quality.

That said, my concerns with the completeness of memorization metrics remain. Based on the details provided by the authors, the mixed-type distance metrics closely resemble standard DCR metrics, with the exception of a normalizing constant. Memorization tests in synthetic data generation that rely solely on standard distance-based metrics are known to be susceptible to issues such as the curse of dimensionality and sensitivity to outliers. Incorporating inference attack-based methods could further enhance the quality and reliability of benchmarking.

Taking all responses into account, I am revising my rating from 5 to 6.

[Q3] What's the implicit hypothesis TabCutMix about data manifold, explicit cases in which it might "fail" by creating OOD data, and explicit cases in which they expect it to perform well? Even better, a practical example of a dataset on which the method failed.

A5: Thank you for raising this insightful question regarding the implicit hypothesis underlying TabCutMix and the scenarios where it may succeed or fail.

Implicit Hypothesis of TabCutMix: The underlying assumption of TabCutMix is that the data manifold of tabular data can be effectively approximated through random feature-level exchanges between samples. Specifically, TabCutMix assumes that the resulting augmented samples will remain close to the original data manifold if features are independently meaningful and can be combined without disrupting their inherent dependencies. This hypothesis aligns well with tabular datasets where features are relatively independent or loosely correlated. We have added this limitation discussion in Appendix. F.

Explicit Cases Where TabCutMix May Fail: TabCutMix may fail when the assumptions about feature independence or loose correlation do not hold. For example: (a) Highly Correlated Features: If features are tightly coupled (e.g., a dataset with strongly dependent numerical and categorical attributes), swapping individual features without accounting for correlations may disrupt the structural relationships in the data, potentially leading to OOD samples. (b) Imbalanced Datasets: In datasets with severe class imbalance, TabCutMix may inadvertently create augmented samples that disproportionately represent minority classes, introducing noise or unrealistic distributions. (c) Sensitive Domains: In sensitive applications such as medical data, feature interactions may carry domain-specific meanings (e.g., age and diagnosis), and arbitrary exchanges may result in implausible or nonsensical combinations.

Explicit Cases Where TabCutMix Performs Well: TabCutMix performs well on datasets where features are either independent or exhibit weak correlations.

To address the failure cases mentioned above, we introduced TabCutMixPlus, which adapts feature exchanges based on feature correlations. By clustering highly correlated features and swapping them together, TabCutMixPlus mitigates the risk of creating OOD samples in datasets with strong feature dependencies, as demonstrated in the reduction of OOD ratios across all datasets.

[Q1]. Why the memorization ratio is independent of the diffusion model used?

A3: While our results demonstrate that both TabSyn and TabDDPM converge to similar memorization ratios for the same dataset, this observation is based on experimental findings. It suggests that the memorization ratio may be predominantly influenced by data-centric factors, such as dataset complexity, sparsity, and feature redundancy, rather than model-centric factors, such as the specific diffusion model architecture. These data-centric considerations warrant further investigation to better understand the relationship between dataset properties and memorization behavior in tabular diffusion models.

We acknowledge that this question lies beyond the primary scope of this paper, which focuses on evaluating existing tabular diffusion models and proposing methods to reduce memorization risks. To address this, we have included a discussion of this promising direction in the future work section (Appendix F). We believe such investigations could provide valuable insights into the interplay between dataset characteristics and generative model behavior in the context of tabular diffusion.

[Q2] Obs 2 of Sec 3.3. I would expect smaller dataset to yield higher memorization rates (easier overfitting). (a) Why is it not changing on some datasets? (b) I suspect that the definition you use in l230 is biased in this regard: since it relies on distance ratio, smaller datasets leads to more sparsely populated space, so absolute distances are increased but maybe relative distances remain unchanged.

A4: Thank you for the insightful observation regarding Obs. 2 in Section 3.3. We have carefully examined both parts of your comment.

Regarding (a): Smaller datasets and memorization ratio trends. We agree with your expectation that smaller datasets are more prone to higher memorization due to easier overfitting. To investigate further, we conducted experiments with extremely small training sizes (1% and 0.1% of the original data) and observed that smaller datasets indeed result in higher memorization ratios across all four datasets, as shown in Figure 3. This confirms the expected trend and is now explicitly discussed in the revised Section 3.3 of the manuscript.

Regarding (b): Potential bias in the memorization ratio definition. Our answer is No bias. We would like to clarify that the memorization ratio in our framework is based on relative distance ratios rather than absolute distances. While it is true that smaller datasets lead to increased absolute distances due to sparse space, the memorization ratio metric depends solely on the relative distance ratio. This ensures consistency across different dataset sizes, as the relative distances remain comparable regardless of the dataset's "absolute density". Therefore, the definition of memorization ratio is not biased with respect to dataset size.

[W2] TabCutMix is at high risk of creating OOD data that might overlap the categories

A2: Thank you for raising this insightful comment. We have thoroughly addressed this issue through three perspectives.

• Hyperparameter Control in TabCutMix for Balancing OOD Risk and Data Utility. TabCutMix provides a hyperparameter, the augmented ratio, to control the number of augmented samples included in training. This allows for careful tuning to balance between reducing memorization and preserving the utility of the augmented data.

• Proposed TabCutMixPlus with Adaptive Feature Exchange. To further mitigate the risk of creating OOD data, we developed TabCutMixPlus, which incorporates an adaptive feature exchange strategy. Specifically, we calculate feature correlations using metrics like Pearson’s correlation coefficient (for numerical features), Cramér’s V (for categorical features), and the squared ETA coefficient (for numerical-categorical pairs). These measures allow us to cluster highly correlated features, ensuring that features within the same cluster are exchanged together during augmentation. This process preserves the structural integrity of the data and significantly reduces the likelihood of generating OOD samples.

• OOD Detection Experiments for Quantifying OOD Risk. To assess the OOD risk introduced by TabCutMix and TabCutMixPlus, we conducted dedicated OOD detection experiments. Following [1], the OOD detection experiments framed the problem as a classification task, treating normal samples as negative and OOD samples as positive, where the positive OOD samples were synthesized by scaling a randomly selected numerical feature by a factor of $𝐹=100$ or choosing a random value for categorical features from existing categories. A multi-layer perceptron (MLP) was trained on the original data and evaluated on augmented samples generated by TabCutMix and TabCutMixPlus to assess the proportion of samples classified as OOD. As shown in Table. 4, TabCutMixPlus consistently reduces the OOD ratios compared to TabCutMix across all datasets. For example, in Adult dataset, OOD ratio reduced from 2.06% (TabCutMix) to 0.36% (TabCutMixPlus). These results demonstrate that TabCutMixPlus significantly mitigates OOD risks while maintaining robust classification capabilities, reinforcing its utility in synthetic data augmentation workflows.

The table below shows the F1 scores and OOD ratios (with standard deviations) for TabCutMix and TabCutMixPlus across various datasets.

TabCutMixPlus shows a significant reduction in OOD ratios compared to TabCutMix, particularly in datasets such as Default and Cardio, while consistently maintaining high F1 scores.

[1] Mohammad Azizmalayeri, Ameen Abu-Hanna, and Giovanni Cin ´a. Unmasking the chameleons: A benchmark for out-of-distribution detection in medical tabular data. arXiv preprint arXiv:2309.16220, 2023.

Reviewer DwHs

We thank the reviewer for the constructive comments.

[W1] Add more limitation discussion. Data augmentation TabCutMix changes the input distribution and produces OOD data that act like regularization.

A1: We have added the limitation discussion as follows:

• OOD generation. TabCutMix may disrupt meaningful feature correlations by switching parts of features between samples. This disruption could lead to out-of-distribution (OOD) issues and reduce the utility of the generated data, especially in applications where feature dependencies are critical.
• Lack of Theoretical Analysis: While the method shows empirical success in mitigating memorization, the underlying reasons for its effectiveness are not fully explored. A theoretical framework explaining why TabCutMix works and its impact on the learned representations would strengthen its scientific contribution.

On the other hand, aside from some cosmetic suggestions, the reviewers have brought up the following major concerns:

• OOD generation risk by Reviewers DwHs, 2Ann, and 8dDK
  • We tackle this issue by (a) clarifying that augmented ratio hyperparameters can control the tradeoff between reducing memorization and preserving the utility of the augmented data. (b) Proposing an enhanced TabCutMix (called TabCutMixPlus) that can mitigate OOD risk; (3) Conducting OOD detection experiments to quantify OOD risk.
  • Please see the details in Sections 4 and 5, Appendices D.4.6, and E.1.
• More dataset benchmarking by Reviewers j6Xu, 2Ann, and 8dDK.
  • We conduct experiments on three additional datasets Churn, Cardio, and Wilt.
  • Please see the details in Appendix E.6.
• More baselines such as SMOTE or Mixup by Reviewers j6Xu, 8dDK
  • We conducted three baselines SMOTE, Mixup, and IJF.
  • Please see the details in Section 4 and Appendix E.7.
• More generative models by Reviewer 8dDK
  • We conduct experiments on CTGAN and TVAE.
  • Please see the details in Appendix E.8.
• More evaluation metrics by Reviewer 2Ann
  • We included two evaluation metrics C2ST and DCR.
  • Please see the details in Section 5, Appendices D.4, D.5, D.6, and E.6~E.8.
• Implicit hypothesis TabCutMix does about data manifold by Reviewer DwHs
  • We discuss the implicit hypothesis of TabCutMix and explicit cases where TabCutMix may fail/perform well.
  • Please see the details in Section 4, Appendices E.5 and F.
• Limitations discussion by Reviewer DwHs
  • We have added limitation discussion on independent feature assumption, sensitivity to outliers, and lack of data-centric insights.
  • Please see the details in Appendix F.
• Choice of Memorization Metric by Reviewer j6Xu
  • We clarify that our metric is based on normalized $l_2$ distance and is equivalent to $l_2$ distance using one-hot encoding. We also acknowledge the limitation of this metric.
  • Please see the details in Appendix F.
• More discussion with previous memorization metrics by Reviewer 2Ann
  • We have added the discussion between memorization ratio and DCR.
  • Please see the details in Appendix D.5.

We argue all the above suggestions are factually rooted, and we have responded to them in a point-blank fashion and should have addressed them all. However, the following concern is rather a question of preference, and we would like to highlight it fairly by quoting our reviewer:

• Reviewer 8dDK: The methodology is naive.

We acknowledge the OOD risk associated with TabCutMix and we further propose TabCutMixPlus as a mitigation strategy. More broadly, we respectfully argue if a simple but effective solution works well for a relatively new task — which is the case for TabCutMix (and TabCutMixPlus) — it may not be necessary or well-motivated to pursue a more complex or "perfect" solution at this stage. Instead, such work sets the foundation for follow-up research to further refine and enhance performance in this emerging area.

During the rebuttal period, we have made significant improvements to our work, incorporating the valuable feedback provided by the reviewers. We respectfully argue our contributions exceed the bar of venues like ICLR, and we kindly request the reviewer reconsider their evaluation. Again, while we agree with the reviewer that our technical solution can be "more perfect", such advancing technical refinements & customizations are natural directions for future research. Our primary contributions lie in:

• Identifying the critical issue of memorization in tabular diffusion models,
• Analyzing the underlying causes of this phenomenon, and
• Proposing a simple yet effective solution to address it, while paving the way for future advancements.

We believe these contributions are highly impactful and represent a meaningful step forward in the field of tabular generative models.