Exploring Local Memorization in Diffusion Models via Bright Ending Attention

Response to W3: Exploring more evaluation metrics.

We appreciate the reviewer for highlighting the potential to explore additional evaluation metrics beyond the first and only localized memorization metric introduced in this paper. While our proposed metric is simple and has room for further refinement, we agree that this represents a promising direction for future work. As noted in the "Limitations and Future Works" section, one potential strategy worth exploring is training self-supervised methods similar to SSCD. By tailoring the dataset preparation and training strategy specifically for localized copy detection, such approaches could enable model embeddings for masked images to compute meaningful cosine similarities with embeddings from other masked images. This would provide a more advanced way to integrate our localization insight with the robustness of SSCD's object-level similarity.

We also greatly appreciate the reviewer's recognition of our multi-fold contributions, as noted in the "Summary" and "Strengths" sections of the review. These contributions include introducing a novel localized perspective on memorization by identifying performance gaps in existing strategies, pioneering the new task of locating localized memorization regions, uncovering the novel BE phenomenon for efficient and effective memorization mask extraction, and improving various tasks such as memorization detection, mitigation, and evaluation by simply integrating our BE mask. Among these, the improved evaluation metric represents just one of the benefits enabled by the BE mask. We are encouraged by the reviewer's acknowledgment and will build on these findings to propose more advanced evaluation metrics in future work. Thank you again for your recognition and valuable suggestions!

Response to Q1: Can BE's effectiveness be affected by thresholding choices of memorization size or intensity?

Thank you for this insightful question! BE's effectiveness is not impacted by the memorization size or intensity because its mechanism for incorporating the BE mask into existing strategies, such as memorization detection, does not rely on any thresholding hyperparameters. Instead, it solely depends on the pre-trained model's overfitted memory, making it robust and adaptable.

Specifically, as discussed in Section 4, the bright ending (BE) attention map inherently serves as an automatically extracted memorization mask. This mask directly utilizes the BE attention scores as its values, eliminating the need for thresholding to convert it into a binary mask of only 0s and 1s. By doing so, it avoids the added variability introduced by thresholding hyperparameters, which may be less effective under certain threshold settings, thereby improving robustness. The approach remains effective regardless of whether the memorized regions are subtle or abstract, as it solely relies on the pre-trained model's overfitted memory to extract the mask. This eliminates the dependence on factors such as the degree of subtlety or abstraction of memorized regions or thresholding based on size or intensity of memorization. For a qualitative evaluation of the mask, please refer to Figures 1 and 4, where brighter patches indicate higher attention scores and darker patches indicate lower ones.

We find addressing this question highlights a unique advantage of our method that was not previously emphasized in the paper. As a result, we have included additional explanations in lines 314-323 of Section 5.1 (highlighted in blue). We appreciate the reviewer's valuable contribution to improving the clarity of our work!

Response to W1 and Q3: Additional discussions and suggestions regarding efficiency.

We sincerely thank the reviewer for suggesting further demonstration of BE's efficiency and for the interesting question about optimizing the process for greater efficiency. These points are highly relevant and may interest many audiences.

We wish to highlight BE's efficiency through its two main capabilities:

• First, BE is the first method to accurately localize the memorization regions in generated images without requiring access to training data or performing a nearest-neighbor search over all ground truth images. Instead, BE requires only a single inference pass, taking just a few seconds on a consumer-grade GPU. This efficiency makes BE practical for real-world applications.
• Second, BE can be seamlessly integrated into existing memorization detection, mitigation, and evaluation strategies, achieving new state-of-the-art results. As discussed in the "Limitations and Future Works" section, this integration may increase the computational cost compared to baseline strategies because BE requires an additional inference pass (50 denoising steps, approximately 2 seconds using an RTX-4090) to extract the mask before incorporating it into the strategies. However, we believe the additional computational cost is well justified by the significant contributions BE makes to studying local memorization in diffusion models.

Regarding the reviewer’s question on optimizing efficiency, we are pleased to share that the 'bright ending (BE)' phenomenon is observable even in earlier inference steps, though it is not as pronounced as in the final denoising step. This stems from the same insight discussed in Section 4: as the denoising process progresses, non-memorized generations’ end tokens typically exhibit a dark cross-attention map containing only low-attention finer edges or scattered random points. Since the end token summarizes the semantic meanings of the prompt, this behavior indicates that, in later inference steps, the denoiser’s attention shifts from focusing on semantic meanings to concentrating on fine details as the image becomes cleaner. In contrast, memorized generations follow an inflexible overfitted trajectory, resulting in their end token’s attention scores displaying a distinct anomaly — the 'bright ending'. While leveraging BE from earlier steps could improve efficiency, the anomaly becomes most pronounced at the final step. Thus, we focus on the final-step BE to ensure robustness and effectiveness.

Given the significant practical contributions demonstrated throughout this paper, we argue that the marginal increase in computational cost for using the final-step BE compared to earlier-step BE is well worth the benefits in accuracy and utility. We appreciate the reviewer for raising this insightful question, which we believe will interest a wide audience. To address this issue thoroughly, we have added Section 8.2 in the Appendix. Thank you for your valuable suggestion!

Response to W2 and Q2: Generalizability of BE to other types of diffusion models.

Thank you for this thought-provoking question, which has inspired us to reflect on BE’s broader implications. We believe the BE phenomenon, rooted in the distinct denoising patterns of overfitted generations, has potential generalizability to certain other diffusion models that have the following two attributes: (1) suffer from the memorization issue, (2) use a denoising framework.

For video diffusion models, especially the text-conditional ones, BE could extend naturally by analyzing aggregated or sequential attention patterns across time, leveraging temporal consistency. For unconditional image generation models, it can be challenging to observe BE due to the lack of conditioning, but it may manifest as latent space anomalies or distinct noise prediction patterns that are worth investigating. However, for flow-matching-based models, which bypass denoising by learning trajectories between distributions, we are skeptical about BE’s applicability due to their fundamentally different design.

We also observe that the degree of overfitting likely influences BE’s effectiveness. Text-conditional models, such as Stable Diffusion, are particularly prone to overfitting because text-conditioning acts as a "key" to the model's memory, creating more specific training data associations. Unconditional models lacking this specificity may exhibit less severe overfitting and, therefore, a subtler BE effect.

This intriguing direction offers exciting avenues for future research, and we will investigate its feasibility in subsequent work. We appreciate the reviewer for bringing this valuable perspective to light!

Response to W1: Clarification for Table 1.

Thanks for this valuable question that contributes to clarifying the purpose of Table 1. As the baseline detection strategy in Equation 7 incorporates a hyperparameter $T$, where the detection signal is defined as the average magnitude over $T$ inference steps, Table 1 evaluates performance across different values of $T$ (1, 10, and 50), which serves as the difference choices of hyperparameter values and is adapted from baseline's setup. This also allows for the demonstration of BE's ablation effects under different choices of the baseline's hyperparameter values. Specifically, under such design, Table 1's results can demonstrate the consistent advantage of Ours (magnitude with BE mask) over Baseline (magnitude without BE mask) under varying baseline hyperparameter settings across all metrics, validating (1) the BE mask's effectiveness in accurately identifying memorized regions and (2) the importance of adopting a localized perspective for studying memorization, which remains under-explored as you also kindly mentioned in the "Strength" section of your official review. Differently, the "step" in our description of the BE mechanism refers to its implementational detail of extracting such a BE mask. When testing its effectiveness in Table 1, it is already available and can be directly used in the computation in Equation 8 to obtain the masked magnitude.

In summary, the "steps" in Table 1 and our description of BE serve different purposes; hence, using the exact wording can be confusing. We really appreciate the reviewer for pointing out this question, and we have correspondingly refined its presentation in our revised paper: (1) We have modified the headers in Table 1 to $T=1$, $T=10$, and $T=50$ to clarify that they only serve as a set of different hyperparameter choices adapting from the baseline's evaluation setting; (2) We have provided additional clarifications in lines 404-413.

Response to W2: Details regarding how to extract memorization mask from BE.

Thanks for this important question. The bright ending (BE) attention map itself inherently serves as the automatically extracted memorization mask. Specifically, it directly utilizes the BE attention scores as its values, eliminating the need for thresholding to convert it into a binary mask of only 0s and 1s. By doing so, it avoids the added variability introduced by thresholding hyperparameters. For a qualitative evaluation of the mask, please refer to Figures 1 and 4, where brighter patches indicate higher attention scores and darker patches indicate lower ones. Although we discussed in line 295 of Section 4 that the BE "effectively functions as an automatic memorization mask," and in lines 332–335 that "Unlike methods requiring hyperparameter tuning, such as setting thresholds to convert attention scores into a binary mask, our approach uses the attention scores directly as weights. This reweights the relative importance of each pixel in the magnitude computations," we sincerely appreciate the reviewer's valuable suggestion to include more details in Section 5.1, which can indeed improve this paper's flow. We have now included more explanations in Section 5.1. Please check out the blue text for the added details.

Response to Q1: Implementational details regarding EOS token in Stable Diffusion. Thank you for the reviewer’s acknowledgment of Stable Diffusion's implementational details and for raising this question regarding its text tokenizer with concrete examples! This has made the question very specific and easy to address. We wish to clarify that the tokenizer provided by Stable Diffusion can, in fact, automatically handle the EOS token. Specifically, Stable Diffusion leverages the CLIP tokenizer, which automatically appends the <startoftext> (aka. SOS, token ID: 49406) and <endoftext> (aka. EOS, token ID: 49407) tokens during text tokenization. This ensures the model knows where the meaningful text starts and ends, and then either adds padding tokens or truncates to the required length of 77.

For example, for a prompt containing $N = 15$ tokens, the decoded tokens will include <startoftext>, followed by the $N = 15$ tokens, then <endoftext>, and finally $77 - 1 - N - 1 = 60$ padding tokens. Our proposed bright ending (BE) strategy utilizes the cross-attention map from the <endoftext> token, as visualized in Figure 4.

Response to W1: More explanation for Section 5.1, Table 1, and Figures 6/7.

Thank you for the reviewer’s valuable suggestion that Section 5.1, Table 1, and Figures 6/7 could benefit from further elaboration. We completely agree with this observation. In response, we have included detailed explanations in these three subsections. Specifically, the blue texts in the revised manuscript elaborate on Section 5.1 (lines 314-323), the explanation of Table 1 (lines 404-413), and the interpretation of Figures 6 and 7 (lines 421-425 and 431-433). We believe these updates significantly improve the overall presentation of the manuscript, making the paper easier to understand, and we sincerely appreciate the reviewer’s constructive feedback and contribution.

Response to W2 and Q2: More details about the manual labeling process and the dataset.

Thank you for pointing this out! We previously provided only a brief explanation of the motivation and feasibility for such manual labeling in lines 375–377 of Section 5.2, stating: "Due to current metrics' failure to accurately identify memorized cases, we manually label the ground truth for each generated image from the memorized prompts. Webster (2023) organizes the nearest training images for these generations, making it easy to observe local memorization cases with minimal subjectivity in manual labeling."

The reviewer’s suggestion to include more details about the manual labeling process and the dataset is highly beneficial for ensuring reproducibility. To address this, we have added an additional sub-section (Section 8.1) in the Appendix, which provides a detailed illustration of the labeling process. This includes pointing out the sources from which we downloaded the datasets and qualitative examples in Figure 9 to demonstrate how the manual labeling was conducted. These additions highlight that the labeling process is straightforward and objective and allows for quick and accurate labeling.

Response to W3 and Q3: Sharing the code and dataset.

Thank you for this important question. We completely agree with the reviewer that open-sourcing these resources would significantly enhance the reproducibility of our work. To this end, we would like to assure the reviewers that we will release both the code implementing our method and the dataset used in our experiments upon the acceptance of this paper. This will include comprehensive documentation and example scripts to help the community easily replicate our results and apply our method to other tasks. We believe this commitment will make our contributions more accessible and beneficial to the research community.

Response to Q1: Motivation for incorporating SSCD similarity into the metric.

Thank you for this insightful question. The motivation behind incorporating both SSCD similarity and masked L2 distance into the metric lies in leveraging the strengths of both approaches. Masked L2 excels in its ability to localize memorization regions, providing pixel-level similarity/distance metrics, akin to traditional L2 distance. However, it remains constrained to pixel-level comparisons. On the other hand, SSCD quantifies object-level similarity, offering robustness against transformations such as scaling and shifting. Nevertheless, SSCD relies on embeddings from a pre-trained model designed for global image copy detection, making it less effective for local memorization scenarios, as illustrated in Figure 2 and discussed in Section 3. Additionally, SSCD cannot compute meaningful similarity scores for masked images.

Our method integrates the localization capabilities of masked L2 with the robustness of SSCD's object-level similarity, effectively combining the benefits of both approaches. This simple yet effective integration allows us to achieve accurate evaluations across both global and local memorization cases, addressing the limitations of each individual metric.