Understanding and Mitigating Memorization in Diffusion Models for Tabular Data

We thank the reviewer for the constructive comments. All responses and corresponding revisions have been incorporated into the updated manuscript, available at: https://anonymous.4open.science/r/TabCutMix-3F7B/TabCutMix.pdf.

[Q1] The paper's theoretical claims about memorization in diffusion models for tabular data lack novelty and are insufficiently substantiated. Propositions 3.1 and 3.2 and found them to be essentially identical to results already established in [Gu et al., 2023] for image generation.

Ans: We acknowledge that the theoretical results presented in Proposition 3.2 build on existing foundations (e.g., [1]). Our goal, however, is not to claim novelty in the theoretical derivation itself, but rather to contextualize and apply these results specifically to tabular diffusion models. As discussed in the manuscript (see the paragraph surrounding Proposition 3.2), we leverage these theoretical insights to explain why memorization can arise in the latent space of tabular data models such as TabSyn, and why it does not necessarily lead to exact duplication in practice—due to factors like imperfect score matching and the stochasticity of VAE decoding.

Our primary contribution is the first empirical investigation of memorization in tabular diffusion models, a setting that has been underexplored in the literature. Furthermore, we propose two novel and effective mitigation strategies, TabCutMix and TabCutMixPlus, which address memorization in a model-agnostic and structure-aware way. We have revised the manuscript to more clearly distinguish between the reused theoretical insights and our original empirical and methodological contributions.

[Q2] The evidence for the effectiveness of TabCutMix and TabCutMix-Plus in mitigating memorization regarding a specific distance based metric is presented, but its unclear if this metric is well-suited for the task especially given the VAE's latents are used for the final generation.

Ans: Prior work on image generation [1,2] commonly uses L2 distance to assess similarity in the input space. Building on this and mixed-type data clustering techniques [3], we adopt a standard mixed-type distance metric suitable for both continuous and categorical features in tabular data.

We also clarify that our memorization metric is computed in the original input space, not in any model’s latent space. Although TabSyn uses a VAE, our method is model-agnostic, and the evaluation remains unaffected. We will clarify this further in the revised manuscript.

[1] Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop. [2] On memorization in diffusion models. arXiv:2310.02664. [3] An improved k-prototypes clustering algorithm for mixed numeric and categorical data. Neurocomputing 2013.

[Q3] The method for determining memorization in tabular data is problematic. Why use 1/3 to judge if a sample belongs to be memorized.

Ans: While we initially use 1/3 as a representative threshold to determine whether a sample is memorized, we agree that threshold selection can influence interpretation. To address this, we conducted a correlation analysis across multiple thresholds (e.g., 1/4, 1/3, and 1/2) and with Mem-AUC (which averages memorization ratios over all thresholds). As shown in Figure 11, the memorization ratio at threshold 1/3 exhibits extremely high correlation (e.g., >0.99) with both Mem-AUC and other thresholds, demonstrating that 1/3 serves as a reliable and representative choice. We have added this discussion and the supporting analysis to the revised manuscript.

[Q4] The proposed mitigation method relies solely on data augmentation rather than addressing model architecture or training procedure issues, which limits its applicability as memorization can still occur with the augmented data.

Ans: We respectfully disagree with the concern that our method is limited in applicability due to its focus on data augmentation. Our approach is model-agnostic and has been successfully applied across a diverse set of generative models and training procedures, including StaSy, TabDDPM, TabSyn, CTGAN, and TVAE. This demonstrates its broad compatibility and effectiveness regardless of the underlying architecture.

Furthermore, we view our method as orthogonal to model design and training objectives—it can be readily integrated into more advanced generative models or used alongside other memorization control techniques. In fact, we believe it can further enhance models that already include architectural or optimization-based memorization defenses. We have included a discussion of these complementary directions in Appendix G (Future Work) of the revised manuscript.

We thank the reviewer for the constructive comments. All responses and corresponding revisions have been incorporated into the updated manuscript, available at: https://anonymous.4open.science/r/TabCutMix-3F7B/TabCutMix.pdf.

[Q1] The results in Table 1 show that, across different datasets, the memorization ratio decreases by only a few percentage points. [Figure 6. Add the ratio for different thresholds]

Ans: We respectfully disagree that the gains on memorization ratio reduction are not significant. In Table. 1, we report the performance with balanced tradeoff in memorization ratio and data utility. Our proposed methods can achieve significant memorization ratio reduction. For example, in default dataset Tabsyn model, TCM and TCMP reduce memorization ratio by 16.16% and 12.48%, which is significant compared with mixup and smote baseline (2.65% and 6.93%). Additionally, as shown in Table 3 and Figure 6 with different augmentation ratio, TabCutMix provides a clear and tunable reduction in memorization via its augmentation ratio from 0% to 100%. For example. In Default dataset, the memorization rate drops from 20.11% to 15.34%, achieving a relative reduction of 23.7%.

[Q2] The theoretical result in Section 3.2 is somewhat straightforward, as it directly follows from the assumed delta distribution.

Ans: We have trimmed down the theoretical part and add more details and reference for additional experiments in appendix. Our goal, however, is not to claim novelty in the theoretical derivation itself, but rather to contextualize and apply these results specifically to tabular diffusion models.

We clarify that our main contribution lies in the empirical investigation of memorization phenomena within tabular diffusion models—a previously unexplored area. Moreover, our proposed novel methods, TabCutMix and TabCutMixPlus, effectively mitigate memorization issues specific to tabular data. We have clarified this distinction explicitly in the revised manuscript to better highlight our empirical findings and methodological contributions.

[Q3] Have you experimented with adding a flag to the model to indicate whether it is processing real or augmented data? This type of augmentation conditioning has been explored in EDM networks [1].

Ans: We thank the reviewer for this insightful suggestion. We have not yet experimented with adding an explicit flag or conditioning signal to indicate whether a sample is real or augmented. We agree this augmentation-aware conditioning, as explored in EDM networks [1], could help the model better distinguish between real and synthetic patterns and improve both generalization and memorization control. We view this as a promising direction and will include it in the revised manuscript under Future Work discussion.

[Q4] How do you handle discrete features for the diffusion models as the framework is mainly designed for continuous signals?

Ans: Our proposed methods, TabCutMix and TabCutMixPlus, are model-agnostic and operate by exchanging feature values across samples within the same label. This augmentation strategy applies equally well to both discrete and continuous features, and we have successfully implemented it across various tabular diffusion models, including TabSyn and TabDDPM.

We acknowledge that diffusion models are primarily designed for continuous data, and different models adopt different strategies for handling discrete features. For example, TabSyn uses a VAE to encode mixed-type features into a continuous latent space, where diffusion is applied. These techniques enable continuous diffusion models to effectively represent and generate mixed-type tabular data. We will clarify this point in the revised manuscript to avoid potential misunderstandings.

[1] Karras T, Aittala M, Aila T, Laine S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems. 2022 Dec 6;35:26565-77.

We thank the reviewer for the constructive comments. All responses and corresponding revisions have been incorporated into the updated manuscript, available at: https://anonymous.4open.science/r/TabCutMix-3F7B/TabCutMix.pdf.

[Q1] I think studying memorization in tabular data might be beneficial for other domains as well.

Ans: We agree that our study on memorization in tabular data can have broader implications for other domains. We have added the discussion in Future work (Appendix. G) in the revised paper.

[Q2] The techniques may generate out-of-distribution samples, which can pose challenges in certain practical scenarios.

Ans: While data augmentation may lead to out-of-distribution (OOD) samples, our method TabCutMixPlus is designed to mitigate this risk through structure-aware augmentation at the feature-group level. As shown in Table. 5 (Appendix D.4.6), TabCutMixPlus consistently yields lower OOD detection ratios (e.g., 25.44% vs. 39.47% on Default, and 0.70% vs. 1.58% on Shoppers). We highlight OOD detection experiments more clearly in the main text of the revised version.

[Q3] Currently, TCM/TCMP focus on classification tasks and may not generalize as readily to regression settings, which limits the scope of the paper.

Ans: TabCutMix and TabCutMixPlus rely on exchanging features between samples of the same class, which inherently limits their applicability to classification settings. As a result, extending our methods to regression tasks remains a challenge and is beyond the scope of this paper. We have acknowledged this limitation in Limitation Discussion (Appendix. F) of the revised version.

[Q4] It would be helpful for the authors to evaluate OOD ratios for Mixup and SMOTE as well, providing a more direct baseline comparison alongside TCM/TCMP (Section D.4.6) Table. 4.

Ans: We thank the reviewer for the suggestion. As requested, we have added OOD detection results for Mixup and SMOTE (now included in Table 5). The “Ratio%” column represents the proportion of samples detected as out-of-distribution. While Mixup shows minimal OOD risk, it performs poorly in mitigating memorization compared to our methods. In contrast, TabCutMixPlus achieves a favorable balance—substantially reducing memorization while maintaining low OOD ratios (e.g., 0.36% on Adult vs. 2.06% for TabCutMix and 0.75% for SMOTE), making it a more effective and reliable augmentation strategy for tabular diffusion models.

We thank the reviewer for the constructive comments. All responses and corresponding revisions have been incorporated into the updated manuscript, available at: https://anonymous.4open.science/r/TabCutMix-3F7B/TabCutMix.pdf.

[Q1] I would suggest including (or at least referencing) important metrics such as precision and MLE in the main body of the paper rather than leaving them to the appendix.

Ans: We have added more descriptions on evaluation metrics and the reference for these important metrics in the main text.

[Q2] The Results in Proposition 3.1 are not new, it has been shown in e.g., [1].

Ans: We have moved Proposition 3.1 in Appendix A and explicitly cited relevant references to prior works. Our goal, however, is not to claim novelty in the theoretical derivation itself, but rather to contextualize and apply these results specifically to tabular diffusion models.

Our main contribution lies in the empirical investigation of memorization phenomena within tabular diffusion models—a previously unexplored area. Moreover, our proposed novel methods, TabCutMix and TabCutMixPlus, effectively mitigate memorization issues specific to tabular data. We have clarified this distinction explicitly in the revised manuscript to better highlight our empirical findings and methodological contributions.

[Q3] The paper's organization could be improved. Table 1 occupies too much space, pushing important details (for example, the introduction of the evaluation metrics) to the Appendix. And Figure 4 appears after Figure 5, etc.

Ans: We have reorganized the paper to improve clarity and readability. Specifically, we have moved Magic dataset results from Table 1 to Table 8 (Appendix E.6). We have added evaluation metrics details and reference, in the main text.

[Q4] I'm a bit confused about the motivation for using MLE (machine learning efficiency evaluation) as a metric for data fidelity.

Ans: MLE evaluates data fidelity by measuring how well a model trained solely on synthetic data performs on real data (e.g., via AUC on a downstream task), which directly reflects the utility of the synthetic data in practical applications. While traditional fidelity metrics like shape score and trend score assess how closely the synthetic data's distribution matches that of the real data, MLE provides an end-to-end evaluation that incorporates task-specific performance, offering a complementary perspective on whether the generated data preserves meaningful patterns for model training.