A Geometric Framework for Understanding Memorization in Generative Models

Thank you for your positive review, and for your comment that the MMH “offers a fresh perspective on memorization, … and adding depth to existing theoretical understandings.” That was very much appreciated. To address the listed weaknesses and questions:

• (Weakness 1) We agree with your comment that “the reliance on geometric and measure theory could make it challenging for a broader audience.” While this is partly unavoidable when proving theorems about it, the understanding of memorization expressed by the MMH is intuitive enough to be accessible to a broad ML audience (much like the manifold hypothesis itself).

  To make this clearer in the paper, we have identified that the theory in the former Section 3.1 is mostly unnecessary for the core purpose of the paper and have therefore moved it to Appendix A. The section’s key takeaways have been distilled into discussion at the end of Section 2 (L200-229). Since this was probably the most technical section, we expect this will improve readability.

• (Weakness 2) “There should be more experiments.” While more experiments are always better, we highlight that our comparison spans 4 models (including both diffusion and GANs) across 3 data regimes (with special focus on Stable Diffusion) with 5 proxies for LID and 3 differentiable scalars for inference-time mitigation. We believe this is sufficient to both validate our hypothesis and demonstrate its utility.

• (Weakness 3 and Question 2) There is indeed some overlap between memorized and non-memorized samples in LID estimates. In our experience, no existing LID estimator is completely accurate, especially for high-dimensional, natural data which tends to be very noisy, so we anticipate that there is room to better apply the MMH via more precise LID estimates.

• (Question 1) Any extension of the MMH to discrete models would be non-trivial - the manifold hypothesis and all the relevant theory do not immediately apply. Intuitively, because the manifold hypothesis expresses the “compressibility” of high-dimensional data, there may be an information theoretic analogy to LID that can be used to detect memorization, but we leave such an exploration for future work

• (Question 3) In addition to LLMs, text, and discrete modelling in general, the MMH does not apply to any modelling regime that does not satisfy the manifold hypothesis, such as simple models that cannot learn manifold structure.

Thank you for your time, effort, and feedback! Thanks for pointing out the typo - we have addressed that and any other typos we could find in the manuscript. In response to your concerns:

• Your main concern (Weakness 2, Question 3) was that our contributions were not clear enough. We appreciate this point, and have restructured the manuscript to make the contributions clearer (descriptions of manuscript changes are bolded below).

  Indeed, the work’s contributions go well beyond DD-Mem. Our work is structured scientifically around a hypothesis: we propose it, show why it explains past observations, and then test it empirically. We then present a downstream application of our theory. Every step of this process is a contribution, and we have updated our introduction to explicitly enumerate them. We have also relegated the former Section 3.1 connecting the MMH to definitions of memorization to the appendix because it is tangential to this story.

  The manifold memorization hypothesis is a fully novel lens from which to interpret memorization in DGMs. As a hypothesis, its value can be understood in two categories:

  • Explanatory: The explanatory power of the MMH, while clear in hindsight, is based on several non-trivial connections we develop in Section 3. For example, your comment in Weakness 2, that Proposition 3.2 (3.5 in the former version) describes the results of Wen et al. (2023), is only clear after also seeing the connection between LID and the classifier-free guidance norm developed later in the section. We believe that altogether the explanatory power of this hypothesis and supporting theory constitute a substantial contribution over and above just summarizing existing work.
  • Practical: To summarize the practical implications of the MMH, we have added and reorganized some content into a series of paragraphs at the end of Section 2: “Why is the MMH Useful?” (L200-229). In comparison to past theoretical frameworks, it is useful because (1) its geometric perspective leads to computationally tractable tools (particularly in comparison to Bhattacharjee et al. (2023) (Question 3), which is not tractable at the scale of Stable Diffusion); (2) it is the only framework that also encompasses the “reconstructive memorization” of Figure 2, or in our terminology, memorization with $LID_\theta(x) > 0$; and (3) it distinguishes between OD-Mem and DD-Mem, leading to a differentiated understanding of the different types of memorization.

  Finally, validating this hypothesis (Section 4.1) and devising inference-time mitigation approaches based on it (Section 4.2) are each further contributions of the paper.

• (Question 1) To clarify, the earlier identity (L135-134 in the updated manuscript) is the definition: (A) $\text{LID}(x)$ is defined as the dimensionality of the manifold at $x$. All later descriptions of LID are meant to be heuristic interpretations of this definition. We have reworded the manuscript to clarify this (L143).

  To address your specific question about the relationship between two further ways of describing LID:

  • (B) The number of degrees of freedom at $x$.
  • (C) The number of variables needed in a minimal representation of the data.

  Both are common heuristic descriptions of LID, and as a first approximation, (A) = (B) ≈ (C). Locally, the local dimensionality of the manifold at $x$ (A) is equal to the number of degrees of freedom or the “number of independent dimensions you can move” along the manifold at $x$ (B), which is in turn equal to the minimal number of variables required to describe data on the manifold locally near $x$ (approximately C).

  However, we don’t actually use the exact wording of (C) in our work, because if taken literally, it would imply some fixed number of variables required to encapsulate a global representation of the dataset. Datasets typically vary in complexity by region, so some regions need fewer variables to describe them (hence the word “local” in LID).

Thank you for your detailed review and helpful feedback! We were especially thrilled by your comment that we provide an “interesting and valuable perspective about memorization.” We address your comments one by one below (apologies for the LaTeX rendering issues - we had to write some LaTeX in plaintext):

• (Weakness 1 and Question 3) You expressed general concern about the clarity of Figure 6 and the comparison of LID metrics.

  • First, we note that all metrics shown are either estimates of LID or proxies for LID; in particular, our analysis from the end of Section 3 predicts that the CFG vector norm varies inversely with LID. This connection was not explicit in Section 4 of our original manuscript; we thank you for raising this and have updated the manuscript to state this explicitly.
  • Second, we note that the correct overall interpretation of the figure is a validation of the manifold memorization hypothesis: all estimators assign lower LIDs to memorized than non-memorized samples. We have slightly reworded part of this section to make this clearer.
  • The memorized samples are those of Webster (2023), and some are available in Figure 11 in Appendix D.5 for reference.

• (Question 6) You asked about comparing the accuracy of two estimators. First, it is important to point out that, while these are both indeed estimators of LID, they measure the LID of different things: \widehat{LID}_* estimates the LID of the ground truth manifold, while $\widehat{LID}_\theta$ estimates the LID of the model manifold. Since each estimates a different quantity and neither ground truth quantity is known for realistic datasets, it is difficult to authoritatively state which one is more accurate.

• (Weakness 2) You mentioned that it is hard to quantitatively differentiate between OD-Mem and DD-Mem. We agree and believe this is a promising area for future work. There are a couple of practical avenues for differentiating these:

  • If one can estimate both \widehat{\text{LID}}_* and \widehat{\text{LID}}_\theta, their relative values can be compared to differentiate between OD-Mem and DD-Mem. We do this for CIFAR-10 in Figure 5. Samples we label as exact memorization tend to have low values for both, which suggests that they are mostly undergoing DD-Mem. Unfortunately, $\widehat{\text{LID}}_*$ estimators do not scale well with dimensionality or dataset size (Tempczyk et al., 2022).
  • In general, if memorization occurs for duplicates in the training data or overly simple images, this is suggestive of a low ground truth manifold dimension and therefore is likely DD-Mem.

• (Question 1) You asked why it is necessary that $\lambda > 1$ in condition (2) of the definition of memorization by Bhattacharjee et al. (2023) (which we have now moved to Appendix A). The reason for this is that, if $\lambda = 1$, then the definition would label situation $P_\theta(B) = P_*(B)$ as memorization. This means the model could learn perfectly ($P_\theta = P_*$) and still be considered memorizing. While that could make sense in a DD-Mem scenario, we believe the original authors of this definition were aiming to capture overfitting behavior. (We remark that we have concluded that the former Section 3.1 on definitions of memorization is tangential to our thesis and have moved it to Appendix A.)

• (Question 2) You asked about the difference between $s_\theta$ and $s_\theta^\text{CFG}$ in Appendix D.2. They are actually related by the definition of $s_\theta^\text{CFG}$ (Equation 2), which we have added to the appendix to clarify:

  $s_\theta^\text{CFG}(x; t, c) = s_\theta(x; t, \emptyset) + \lambda \big(s_\theta(x; t, c) - s_\theta(x; t, \emptyset)\big)$

• (Question 4) Thank you for asking about this - we are currently compiling the memorized examples. Once we have these next week, we will update the appendix with additional figures and provide a drive link of all our labelled generated and training sample pairs for each CIFAR-10 model along with a CSV containing the metrics for each. These will also be made available on GitHub when our code is released.

• (Question 5) To clarify, we did not directly use a threshold on any quantitative metric to label memorized samples (L351-359). We found the two most common memorization metrics (SSCD of Pizzi et al. (2022) and calibrated L2 distance of Carlini et al. (2023)) were unreliable for our models and data. Instead, we used the method outlined in the “CIFAR10 Memorization” part of Section 4: with each metric, we filtered the top 300 “most memorized” images, and then labeled the top 300 pairs for each metric (600 total) by hand (as promised above, we will make all of these labels publicly available). (The corresponding thresholds before manual labeling were SSCD > 0.6854 or calibrated L2 < 0.8594 for iDDPM and SSCD > 0.6677 or calibrated L2 < 0.8750 for StyleGAN.)

Thank you to the authors for taking their time to respond to my questions and comments. I am satisfied with their responses, which help clear up most of my misunderstandings. I am happy to say that I will raise the score of this paper to 8. To justify my decision, I would like to provide some clarifications.

The main goal of this paper, to the best of my understanding, is to advocate for and connect manifold memorization hypothesis (MHN) to the understanding of memorization in deep generative models. I believe that the paper does demonstrate this aspect very well as the authors provide a very cool and new perspective on memorization, which separates it from the naive thinking that memorization is overfitting. This is a very wonderful perspective. Additionally, the paper also introduces two new metrics based on LID, which help prevent memorization in class-conditional diffusion models and show the potential usefulness of LID. Hence, the score of 8 is acceptable for me.

Anyhow, I would like to continue a short discussion for my own understanding, which may help others. On page 7, in the CIFAR10 memorization section, once you have generated the 50k samples. You utilized SSCD distance and l2-distance metrics from Carlini et al. (2023) to retrieve the most similar 600 images w.r.t the 50k set. What is the reason behind choosing the closest 300 images from both of the two metrics, respectively? Why not use just one of the two metrics?

Thank you very much. I suggest that you should add this reason/statement to your manuscript for clarification on the reason of using both metrics and to highlight their imperfection.

Best of luck!