Image-level Memorization Detection via Inversion-based Inference Perturbation

It has nothing to do with the specific methods or evaluation settings, which won't be taken into consideration until a research problem is defined. The concentration should be the motivation of this work, i.e., how do we reach such a problem. The motivation of this work seems exactly the same as Ma et al.

What I'm really concerned with is that, although the authors have noticed previous work conducted by Ma et al., who clearly pointed out the limitation of prompt-based analysis methods and proposed the image-only setting, only a short description is included in line 512, Sec 5.2: "Ma et al. (2024) further analyze memorization across conditional and unconditional diffusion models". This description is vague and misses the main point of their work. Also, it should not be stated together with the prompt-based methods.

I request that the authors clarify the problem-level relationship of these two works.

Thanks for your attention and comments on our work. We would like to clarify the main difference with [1] as follows:

Memorization Scope and Comprehensive Setup

• Our work addresses both exact duplication and partial replication, expanding the scope of memorization detection. The reviewers agree that exploring various degrees of image-level memorization detection substantially enhances the practical importance of our work.
• We have made significant efforts to establish a comprehensive evaluation setup for image-level memorization detection. Specifically:
  • We have constructed a much larger and more practical benchmark, simulating diverse real-world memorization scenarios. This includes memorized images under varying degrees of replication, ensuring a more practically meaningful evaluation.
  • Additionally, we extend our approach to detect memorized images from both training and non-training images, as well as robustness evaluation under data augmentation attacks, to better address real-world scenarios.
• In contrast, [1] focuses solely on exact duplication and conducts its experiments with only 78 memorized images and 100 non-memorized images. As denoted by Reviewer dGYC and Reviewer 8Z99, this setting has limited practical significance because only a relatively small subset of training images is likely to be memorized.
• In summary, our primary contribution lies in establishing this comprehensive setup for image-level memorization detection, which we believe will foster broader discussions on infringement inspection in text-to-image diffusion models, addressing real-world scenarios and practical applications.

Different Motivation

• While both methods involve inversion-based detection, the approach in [1] and our proposed Inversion-based Inference Perturbation (IIP) differ significantly in their objectives and implementation.
  • In [1], the focus is on using inversion to derive model prediction errors as clues for memorization. Specifically, [1] optimizes the distribution of noise $\epsilon$ and further compares it to a Gaussian distribution to evaluate whether memorization has occurred.
  • In contrast, our motivation is to leverage inversion to obtain latent codes at smaller timesteps that retain rich semantic information about the original images. This enables us to detect memorization by analyzing distinct patterns in both the similarity of the original and reconstructed images, as well as the magnitude of text-conditional noise predictions (MTCNP) after perturbing the inference stage.

New Detection Clues and Novel IIP Framework

• We discover that memorized images exhibit lower similarity to their original versions and maintain larger magnitudes of text-conditional noise predictions (MTCNP) after perturbing the generation procedure.
• Building on these findings, we propose our Inversion-based Inference Perturbation (IIP) framework, which first employs DDIM inversion to derive latent codes containing rich semantic information from the original images and then optimizes random prompt embeddings to perturb the inference process effectively.
• In contrast, [1] focuses on the clue model's prediction error and compares it on the target images to safe unconditional models. —none of which are part of the motivation or core of our method. Our approach is developed independently of [1] and introduces a novel perspective on memorization detection.

Effectiveness and Efficiency

• As mentioned earlier, we have constructed comprehensive setups to rigorously demonstrate the effectiveness of our IIP framework. Specifically, we evaluate it on SD v1.4, v1.5, and v2.1, using hundreds to thousands of memorized images across various degrees of memorization. This extensive validation ensures the effectiveness and applicability of our IIP method in real-world scenarios.
• Additionally, our IIP framework is highly efficient, processing a single image in just 3.5 seconds. However, the prompt inversion in [1] seems considerably more time-consuming.
• Moreover, Our method does not rely on other models while [1] involves distribution comparisons with unconditional model.

Acknowledgment:

• We would like to express our gratitude for this comment. We will revise the discussion of [1] in the next version to provide a clearer comparison.
• In addition, we look forward to the open-source release of [1], which will enable further comparisons and improvements in future work.

Thank you for your continued feedback. While we acknowledge the similarity in the research problem, we would like to clarify that specific methods and evaluation settings play a significant role in assessing the contributions of a research work. We emphasize that our work addresses a broader range of memorization including both exact duplication and partial replication. This significantly extends the applicability and practical importance of our task and detection method. Additionally, the two works are distinct in detection clues and the framework, as well as effectiveness and efficiency under extensive experimental evaluations, as outlined in our previous response.

However, we recognize the value of [1]. As the paper’s revision deadline has passed, we will ensure to include a clearer explanation in the next version, highlighting its main point and fully discussing it.

Thank you again for directly pointing out your concerns and for your interest in our work—this truly reflects the value of open peer review.

Thank you for your interest in our work! The GitHub repository is correct—we have just opened this repository. We are also in the process of organizing the dataset and code and will update them as soon as possible. Thanks!