Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapes

Relation between the trace of the Jacobian and the norm of the score function in the non-Gaussian case

Thank you for the insightful comment regarding the relation between the trace of the Jacobian and the score norm in the non-Gaussian case.

We found that Lemma 4.1 is indeed generalizable beyond the Gaussian case under mild boundary conditions [1] (e.g., $\lim_{|\mathbf{x}| \to \infty} p(\mathbf{x}) = 0$ and $\lim_{|\mathbf{x}| \to \infty} p(\mathbf{x}) \nabla \log p(\mathbf{x}) = 0$), under which the identity

$\mathbb{E}[\||s(\mathbf{x}) \||^2] = -\mathbb{E}[\mathrm{tr}(H(\mathbf{x}))]$

holds in general where the trace is now in expectation.

We also appreciate the suggestion to consider Tweedie's formula for Lemma 4.3. We are currently revisiting the proof and will consider incorporating it into the formulation. Regarding Lemma 4.2, which quantifies the gap between conditional and unconditional distributions, we chose Gaussian assumption to ensure tractability and interpretability. This choice is supported by prior work [2] and the empirical trends shown in Figure 4, making our approximation both practical and well-justified.

• [1] Hyvärinen, "Estimation of Non-Normalized Statistical Models by Score Matching", JMLR, 2005.
• [2] Wang et al. "The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications." TMLR, 2024.

Connections with modern Hopfield networks

We sincerely thank the reviewer for highlighting important theoretical connections between our paper and recent work on modern Hopfield networks. As pointed out by the reviewer, [1] demonstrates that large class of diffusion models asymptotically yield energy landscapes equivalent to modern Hopfield networks, where memorized states correspond to local minima. [2] similarly presents an insightful perspective by interpreting diffusion models as a extension of associative memory retrieval, showing that the iterative denoising process parallels recurrent energy minimization in Hopfield-like networks. Their framework highlights how the dynamic updating of latent variables in diffusion models can be cast as an attractor-based process, reinforcing the broader view of diffusion as a form of associative memory.

Our paper shares the central theoretical idea-that memorized states are sharp minima-but differs in scope and methodology. While these referenced studies focus on establishing theoretical equivalences and analyzing associative memory capacity, our contribution lies in explicitly quantifying sharpness via Hessian eigenvalues and score norms as an early-stage utility tool to detect and mitigate memorization during model inference. We greatly appreciate the reviewer's insightful suggestion to clarify these theoretical connections, as this strengthens the context and clarity of our work.

• [1] Ambrogioni "In search of dispersed memories: …", Entropy, 2024
• [2] Hoover et al. "Memory in Plain Sight:...", NeurIPS Workshop, 2023.

Connection with the eigenbasis framework

We sincerely appreciate your valuable suggestions, particularly regarding closely related studies that enrich the theoretical context of our work. The concurrent work [1] rigorously characterizes geometric memorization through analysis of the score function's Jacobian eigenstructure. The authors identify memorization onset when spectral gaps between singular values close, indicating the manifold's tangent-space structure has collapsed. This eigenbasis approach aligns with our central idea-that memorization corresponds to increased local curvature, which implies very small local variance in the density. We find their elegant mathematical formulation very insightful and acknowledge the deep theoretical contributions they've made to understanding generative model memorization.

However, while their detailed eigenanalysis can be computationally prohibitive for large-scale diffusion models like Stable Diffusion, our approach simplifies the process by using the Hessian score product as a summary statistic. This streamlined method maintains theoretical coherence while efficiently monitoring and mitigating memorization in high-dimensional generative models.

Similarly, [2] demonstrates how strong generalization naturally emerges through geometry-adaptive harmonic representations. Their work shows that optimal denoisers implicitly operate within a geometry-adaptive basis, explaining how diffusion models achieve generalization without exponentially large datasets. While they characterize the adaptive basis underlying generalization, our contribution focuses specifically on sharpness as a direct, practical measure of memorization, complementing their basis-oriented theoretical analysis. We are grateful for these insightful references to the reviewer and will update our manuscript accordingly.

• [1] Achilli, Beatrice, et al. "Losing dimensions: ….", arxiv, 2024
• [2] Kadkhodaie et al. "Generalization in diffusion models arises …", ICLR, 2024.

Connection to [1].

We thank the reviewer for highlighting the connection to [1], which we will properly acknowledge. While both works study memorization through geometric properties of the density, there are key differences:

• [1] focuses on first-order smoothness via comparisons between trained and oracle scores, whereas our analysis emphasizes second-order geometry through the trained Hessian and explicitly measures sharpness by observing distribution of eigenvalues.
• We also demonstrate how specific initial latents lead to memorization by consistently mapping into high-curvature regions, further amplified by text conditioning.

We believe these distinctions clearly position our contributions as complementary to [1], enriching the understanding of memorization in generative models.

Evidence of key motivation

We appreciate the concern about the generalizability of our observed phenomenon. While we use MNIST for clarity and intuition, our experimental design extends well beyond this toy dataset. As shown in Figure 3, the key phenomenon, sharpness correlating with memorization, is present in modern, large-scale models like Stable Diffusion.

This progression from simple to complex domains actually strengthens our motivation by showing that the behavior persists across different datasets. Figure 5 further supports this finding, revealing consistent patterns in the Hessian spectra between memorized and non-memorized samples.

Works like [1,2] already used curvature properties to discuss memorization.

We acknowledge that prior works [1, 2] have explored memorization and data complexity through curvature-related concepts. However, our contribution advances beyond these studies in both scope and methodology. While [2] examines curvature only at the final generation stage, we introduce a dynamic framework that analyzes sharpness throughout the entire diffusion process across all timesteps. This continuous perspective enables earlier detection and mitigation of memorization, which we believe are novel and practically valuable contributions.

Gaussian Assumptions

Empirical evidence from recent work [3] shows that diffusion models exhibit approximately Gaussian score behavior in early and intermediate steps, supporting the validity of our Gaussian-based assumptions.

As shown in Figure 4, key metrics like the negative Hessian trace align well with the score norm and Hessian-score product, confirming that our theoretical approximations hold in practice.

Small differences in eigenvalues

We understand your concern. While eigenvalue differences at the initial step are small, they become effective when aggregated (e.g., sum or cubed sum). As shown in Table 1, both our metric and Wen’s metric, which measure these statistics, perform well at Step 1. This confirms the usefulness of early-stage eigenvalue signals.

Regarding LID, we excluded it from our detection experiments since it requires full sampling steps, making direct comparison unfair. For reference, we provide our LID results at Step 50:

• SD v1.4: AUC = 0.974 (n=1), 0.992 (n=4); TPR@1%FPR = 0.184 (n=1), 0.824 (n=4)
• SD v2.0: AUC = 0.972 (n=1 and n=4); TPR@1%FPR = 0.470 (n=1), 0.216 (n=4)

Compared to our method in Table 1, LID yields significantly lower TPR@1%FPR, especially in SD v2.0. This highlights the effectiveness of our sharpness-based approach, which is essential to the success of our mitigation strategy (Figure 6).

Performance & Time cost compared to Wen's metric

Thank you for pointing this out. While our method involves computing a Hessian-score product, the additional cost is minimal due to efficient JVP implementations in standard libraries.

Our metric provides clear computational benefits. As Table 1 demonstrates, it achieves equal or superior AUC compared to Wen's metric with fewer sampling steps ("steps") and generations ("n"). Using SD v1.4, we compare the runtime (in seconds) between these metrics below, with bold entries indicating where both methods achieve equivalent AUC.

Our metric is also critical for SAIL. While Wen’s metric is effective for detection, we found that using it as SAIL objective (line 360) led to optimization failure. In our case, the amplified early-stage differences enabled stable convergence and effective mitigation.

SAIL may not be applicable to SDE.

While most of prior works focus on ODE samplers for analysis, we believe SAIL can still be applied to SDE samplers in expectation by selecting good initializations. We agree this is a valuable direction for future work.

[1] Wang et al. "On the discrepancy and connection...", ICML Workshop, 2024
[2] Kamkari et al. "A geometric view of data...", ICML Workshop, 2024
[3] Wang et al. "The Unreasonable Effectiveness...", TMLR, 2024.

Evaluate methods on more datasets.

Thank you for the suggestion. For our Stable Diffusion experiments, we adopted the established benchmark of known memorized prompts introduced by [1], which has become a standard dataset in the current literature [2-4]. In line with prior work, we made an effort to comprehensively include all verbatim categories within this benchmark.

• [1] Webster, "A reproducible extraction …" arXiv. 2023.
• [2] Wen et al. "Detecting, explaining, and mitigating …", ICLR. 2024.
• [3] Ren et al. "Unveiling and mitigating memorization …", ECCV, 2024.
• [4] Chen et al. "Exploring local memorization …", ICLR, 2025.

Empirical results in Table 1 (Comparison with Wen’s metric at step 1)

Thank you for the suggestion. We would like to highlight the advantages of our upscaling method from two perspectives: detection and mitigation.

Computational Cost

Our metric offers clear computational benefits in detection. As shown in Table 1, our approach consistently achieves comparable or superior performance while requiring substantially fewer sampling steps (“Steps”) and fewer simultaneous generations (“n”) compared to Wen’s metric. This efficiency is especially valuable in practical scenarios such as real-time detection.

Below, we present a comparison of computation time (in seconds) between Wen’s metric and ours using Stable Diffusion v1.4. We highlight in bold the entries where both methods achieve equivalent AUC performance.

As shown above, our method achieves similar or better performance at a clearly lower computational cost. We also expect further efficiency gains if the JVP operation is optimally integrated into libraries such as Hugging Face.

Harder Detection Cases in SD v2.0:

Unlike Stable Diffusion v1.4, which includes both Exact and Partial memorized samples, v2.0 contains only Partial memorized prompts, making the detection task more challenging. The more pronounced performance gap between our method and Wen’s metric in this setting empirically demonstrates the effectiveness of our upscaling approach.

Mitigation - SAIL Optimization:

Our metric is also crucial for SAIL. Although Wen's metric works well for detection, using it in the SAIL objective (line 360) resulted in optimization failure. Our approach, however, successfully achieved stable convergence and effective mitigation by amplifying differences in the early stages.

How to solve SAIL in Section 5.1

Thank you for the question. While we provide detailed pseudocode for SAIL in Appendix D.2, we are happy to explain it here as well.

The SAIL objective involves two forward passes at the initial sampling step ($t=T-1$).

• In the first pass, we compute the score difference $s_{\theta}^\Delta(\mathbf{x}_T)$.
• We then perturb the initialization slightly in the direction of this score difference, i.e., $\mathbf{x}_T + \delta \cdot s\_\theta^\Delta (\mathbf{x}_T)$,
• and perform a second forward pass to obtain $s_{\theta}^\Delta\bigl(\mathbf{x}_T + \delta \cdot s\_{\theta}^\Delta(\mathbf{x}_T)\bigr)$.

With these components, we optimize the initial noise $\mathbf{x}_T$ sampled from an isotropic Gaussian using the SAIL objective (refer to line 373).

This optimization is lightweight and typically converges within two to three iterations on average, depending on the threshold hyperparameter $\ell_{\text{thres}}$. SAIL also supports batch-wise execution, allowing multiple initializations to be optimized in parallel. Please feel free to let us know if any part requires further clarification.

In Figure 6 (left), for SAIL, why can the CLIP score not reach larger values like the other baselines?

We appreciate your valuable question. It is indeed possible to achieve higher average CLIP scores with SAIL by adjusting its threshold hyperparameter $\ell_{\text{thres}}$.

However, we would like to clarify that, empirically, algorithms achieving high CLIP scores alongside high SSCD scores (e.g., > 0.35) often produce outputs that are only superficially altered. These include blurry or partially contaminated memorized images, which are difficult to consider genuinely mitigated.

For all baseline methods, we carefully selected hyperparameters based on their original papers to ensure a fair comparison. In contrast, for SAIL, we prioritized configurations that provide strong memorization mitigation. Notably, across extensive hyperparameter sweeps, SAIL consistently achieved the lowest points on the SSCD -CLIP performance trade-off curves in both Stable Diffusion v1.4 and v2.0.

In the Figure 3 (right), did you miss the negative sign in the y-axis?

Thank you for pointing this out. The reviewer is correct and we will revise it accordingly in the final version.

Thank you for your positive review and for recognizing our contributions to this topic. We sincerely appreciate the time and effort you dedicated to reviewing our work. Please do not hesitate to reach out with any further questions or suggestions.