Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models

Thank you for your insightful comments. We've carefully considered them and provided responses. Please let us know if you need any clarification or have additional questions.

W1: On the motivation for the work and the CFG strength.

• The motivation for the work

As noted in Section 4.1, Stable Diffusion v2 shows fewer memorization issues, but they are not entirely resolved. Prior works [1, 2] confirm that while verbatim memorization has decreased, template memorization remains common.

Given the potential risks of leaking sensitive or copyrighted content (Section 1), even rare cases of memorization are concerning. Addressing them remains an important objective.

• On the CFG strength

As you noted, lowering CFG strength can reduce memorization, but the reduction is modest. We evaluated this using CFG strengths {2, 3, 5, 7}, with results shown in the table below.

While SSCD scores decrease at lower strengths, the effect is limited—for example, CFG = 2 still shows substantial memorization (SSCD = 0.4908).

Importantly, applying Per-sample consistently leads to much lower SSCD scores across all CFG values, demonstrating its robustness and effectiveness.

W2 & Q2: Performance results under SDv2.0.

To demonstrate the robustness and generalizability of our proposed method, we conducted experiments using SDv2.0 and the Euler discrete scheduler. The performance results are shown in the table below, where Per-sample achieves the best Pareto front. Batch-wise also performs comparably well.

W3 & Q7: On the effectiveness of the proposed methods.

• Performance of the proposed method

To enable a more comprehensive evaluation, we varied the mitigation strength for both the baselines and our proposed methods using the settings from Table 1. Due to the character limit, we kindly refer you to our response to Reviewer XBNj’s Question 2 for the corresponding performance table (related to Pareto front results).

• On combining Per-sample with Prompt Embedding Adjustment

The table below presents results for combining Per-sample with Prompt Embedding Adjustment, using the same settings as Table 1. In the combined approach (e.g., l_target = 5 for Prompt Adjustment and 0.9 for Per-sample), prompt embedding is first adjusted until $||\tilde{\epsilon}_\theta(x_T, T, y)||_2 < 5$, followed by noise adjustment until it falls below 0.9.

As shown, Per-sample alone achieves the best Pareto fronts, while Prompt Embedding Adjustment performs the worst. The combined method improves upon Prompt Embedding Adjustment but still falls short of Per-sample. This may be due to prompt modifications disrupting alignment, whereas Per-sample maintains alignment by keeping the prompt unchanged.

W4: Pareto fronts across different hyperparameters.

As shown in our response to Weakness 3, Per-sample, our proposed method, consistently achieves the best Pareto fronts across a range of hyperparameter settings.

W5: Suggestion to include FD_Dinov2.

We agree that FD_Dinov2 offers a more semantically meaningful evaluation due to its stronger vision backbone. If resources permit, we plan to include it in the revised version. We appreciate the suggestion.

Q1: A quantitative study across a whole dataset.

We evaluated transition point shifts across 5,000 samples generated from 500 memorized prompts, varying initial noise $x_T$ and adjustment strengths ($\tilde{\gamma} = 0.5$, $1.0$, $1.5$).

For each $x_T$, we measured the change in transition point relative to the baseline ($\tilde{\gamma} = 0$). Positive values indicate earlier transitions.

The table below shows average changes, all positive, confirming that initial noise adjustment consistently advances the transition point, with larger $\tilde{\gamma}$ producing greater shifts.

Q3: SNR based transition point analysis.

Following prior work [3], we analyzed transition points using their corresponding log SNR values. Using a whole dataset, we measured how varying $\tilde{\gamma}$ ($0.5$, $1.0$, $1.5$) shifts the transition point relative to the baseline ($\tilde{\gamma} = 0$). Negative values indicate earlier transitions.

As shown in the table, higher $\tilde{\gamma}$ consistently leads to earlier transition points, aligning with the trends observed in the timestep-based analysis.

Q4: On the regions in the initial noise space.

The region of the initial noise space where $x_T$ has a sufficiently small $||\tilde{\epsilon}_\theta(x_T, T, y)||_2$ value—small enough to eliminate the transition point—can be considered outside the attraction basin. Investigating how to directly sample initial noise from this region would be a promising direction for future research.

Q5: initial noise displacement.

As shown in our response to Reviewer JH8S’s Question 1-2, JSD between 5,000 adjusted and unadjusted samples was only 0.0665, indicating minimal distribution shift. This suggests that the overall adjustment magnitude is likely small.

Q6: Computational Cost Analysis.

Due to the character limit, we kindly ask you to refer to our response to Reviewer JH8S's Question 1-1 for further details.

Q8: On the high CFG.

We followed prior works [2, 4, 5] in using a high CFG weight (e.g., 7.0–7.5). As noted in our response to Weakness 1, lower CFG weights improve FID but still produce high SSCD scores, indicating limited effectiveness in mitigating memorization.

[1] Webster et al. "A reproducible extraction of training images from diffusion models". arXiv 2023.

[2] Ren et al. "Unveiling and mitigating memorization in text-to-image diffusion models through cross attention". ECCV 2024.

[3] Kingma et al. “Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation”. NeurIPS 2023.

[4] Jain et al. “Classifier-Free Guidance Inside the Attraction Basin May Cause Memorization”. CVPR 2025.

[5] Hintersdorf et al. “Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models”. NeurIPS 2024.

We are very grateful for your constructive comments. We have provided answers to each comment. Please let us know if you need any clarification or have additional questions.

W1 & W2: Computational Cost Analysis.

To ensure a fair comparison, we measured GPU memory usage and inference time per image. As shown in the table, Batch-wise incurs minimal overhead, while Per-sample requires slightly more resources.

Nonetheless, as addressed in our responses to Reviewer tNu7’s Weaknesses 2 and 4, Per-sample consistently achieves the best Pareto fronts (SSCD vs. CLIP) across SDv1.4 and SDv2.0 compared to other baselines.

These findings suggest that Per-sample offers superior memorization mitigation performance and robustness across different models. As discussed in Section 6, effective mitigation of memorization plays a crucial role in enhancing privacy and intellectual property protection while significantly reducing potential risks. Considering this, we believe that the additional cost of the Per-sample method is a reasonable trade-off.

W3: Used timesteps in the experiments.

Following the experimental setup of prior work [1], we report our results in Figure 3. Using 50 inference steps for denoising results in a step size of 20, meaning the timesteps used are [981, 961, 941, …, 1].

W4: On the missing inference time mitigation baseline (Hintersdorf et al.)

Our proposed method is an inference-time mitigation approach that does not require access to training data. Therefore, as described in Section 4.1, we selected baseline methods that also operate at inference time and do not rely on training data, to ensure a fair comparison.

To the best of our knowledge, NeMo [2] requires access to training data to compute activation statistics on unmemorized samples.

Q1: On the calculation of FID score.

We computed the FID using 10000 randomly sampled images from the LAION dataset and 5000 generated images produced from memorized prompts. We suspect that the number of generated and reference images, along with the exclusive use of memorized prompts for generation, may have influenced the scale of the FID score.

Q2: The reason why mitigation methods improve FID score.

Without mitigation, diffusion models tend to generate identical or highly similar images for memorized prompts, which significantly reduces diversity. In contrast, applying mitigation alleviates this issue, increases diversity, and is likely to improve the FID score as a result.

The table below extends Table 1 by including LPIPS (a diversity metric) and ImageReward (a metric for quality and prompt alignment), where higher values are better for both. As shown, applying mitigation leads to higher LPIPS scores, confirming that diversity indeed improves.

[1] Jain et al. "Classifier-Free Guidance inside the Attraction Basin May Cause Memorization". CVPR 2025.

[2] Hintersdorf et al. "Finding nemo: Localizing neurons responsible for memorization in diffusion models". NeurIPS 2024.

We thank the reviewer for the constructive comments. We have provided answers to each comment. Please let us know if you need any clarification or have additional questions.

Q1: Comparisons of performance with and without adjustment for the Per-sample strategy.

As described in Section 3.2.2, the Per-sample strategy involves adjusting the initial noise $x_T$ and applying CFG from the beginning of the sampling process. Without the initial noise adjustment, hence, the Per-sample strategy becomes equivalent to the No Mitigation.

Therefore, the performance comparison between No Mitigation and Per-sample reported in Table 1 allows us to assess the effect of applying the initial noise adjustment. Additionally, Appendix D.3 presents the performance of the Per-sample strategy under different adjustment strengths. The comprehensive results are summarized in the table below, demonstrating that initial noise adjustment effectively mitigates memorization.

Q2: The effectiveness of the proposed methods.

To conduct a more comprehensive performance evaluation, we varied the mitigation strength for both the baselines and our proposed methods. The results are summarized in the table below, which also includes evaluations using LPIPS (diversity) and ImageReward (quality and prompt alignment). All other experimental settings are consistent with those in Table 1.

Our main finding is that adjusting the initial noise $x_T$ to minimize $||\tilde{\epsilon}_\theta(x_T, T, y)||_2$ leads to an earlier transition point and improves mitigation performance. Based on this, we proposed two strategies—Per-sample and Batch-wise—which share the same core idea.

Batch-wise involves three hyperparameters ($\tilde{\gamma}$, $\rho$, and $M$), while Per-sample uses one (l_target). Since mitigation strength is closely related to the value of $||\tilde{\epsilon}_\theta(x_T, T, y)||_2$, we report the performance of Per-sample across different l_target values.

Using the data in the table, one can plot SSCD vs. CLIP curves and observe that Per-sample achieves the best Pareto fronts. Batch-wise also shows strong performance, comparable to Cross-attention Scaling. These results demonstrate the effectiveness of our proposed methods.

While we are unable to include figures in this rebuttal, we will provide graphical results in the revised version of the paper.

Q3: A more comprehensive comparison over different privacy levels.

Through the above response, we provide a comprehensive comparison across different privacy levels. The fact that Per-sample achieves the best Pareto fronts further demonstrates the effectiveness of our proposed method.

Q4: The experimental settings used in Figure 4.

The experimental setup in Figure 4 follows the same configuration as in Table 1; however, the No adjustment shown in Figure 4 does not correspond to any of the baselines reported in Table 1. Before explaining this further, we would like to clarify the purpose of Figure 4.

Our goal was to examine whether initial noise adjustment meaningfully affects mitigation performance. For this, we used the Batch-wise strategy. The mitigation performance of Batch-wise is primarily influenced by two factors: (1) whether initial noise adjustment is applied, and (2) the starting timestep for applying CFG (denoted as $\tau$).

We observed that increasing $\tau$—that is, applying CFG earlier—tends to improve both SSCD and CLIP scores, indicating a trade-off between the two. Importantly, this trend holds consistently regardless of whether initial noise adjustment is used. This is likely because the point at which a sample exits the attraction basin varies depending on the seed (i.e., $x_T$) and the prompt; thus, applying CFG too early (i.e., at higher $\tau$) increases the chance of generating memorized images.

For this reason, we varied the $\tau$ values under both settings—with and without initial noise adjustment—and reported the results in Figure 4, labeled as With adjustment and No adjustment, respectively. As shown, initial noise adjustment yields a more favorable trade-off between SSCD and CLIP scores compared to the No adjustment. This demonstrates that initial noise adjustment indeed enhances memorization mitigation performance.

(Note: In Figure 4, the rightmost data point on the No adjustment curve corresponds to $\tau = 780$, and the leftmost to $\tau = 720$. Similarly, the rightmost and leftmost data points on the With adjustment curve correspond to $\tau = 900$ and $\tau = 840$, respectively.)

Thank you for your insightful comments. We've carefully considered them and provided responses. Please let us know if you need any clarification or have additional questions.

Q1-1: Computational Cost Analysis.

To ensure a fair comparison, we measured GPU memory usage and inference time per image. As shown in the table, Batch-wise incurs minimal overhead, while Per-sample requires slightly more resources.

Nonetheless, as addressed in our responses to Reviewer tNu7’s Weaknesses 2 and 4, Per-sample consistently achieves the best Pareto fronts (SSCD vs. CLIP) across SDv1.4 and SDv2.0 compared to other baselines.

These findings suggest that Per-sample offers superior memorization mitigation performance and robustness across different models. As discussed in Section 6, effective mitigation of memorization plays a crucial role in enhancing privacy and intellectual property protection while significantly reducing potential risks. Considering this, we believe that the additional cost of the Per-sample method is a reasonable trade-off.

Q1-2: The choice of AdamW and the effect of weight decay.

• On the choice of AdamW

As shown in Section 3.1.2, we observed that initial noise with a smaller conditional guidance leads to an earlier transition point, which in turn improves memorization mitigation performance. Therefore, our goal is to effectively reduce the norm $||\tilde{\epsilon}_\theta(x_T, T, y)||_2$.

We experimented with minimizing this norm using both SGD and AdamW. The results show that AdamW reduces $||\tilde{\epsilon}_\theta(x_T, T, y)||_2$ more effectively than SGD. This leads to a substantial improvement in mitigation performance, as demonstrated in the table below. As shown in the table, AdamW achieves the best mitigation performance, and we therefore adopt it as our optimizer.

• The effect of weight decay

We believe that weight decay plays a partial role in preventing adjusted initial samples from entering an out-of-distribution (OOD) state. This belief is grounded in the following reasoning. According to prior works [1, 2], the L2 norm of the initial noise $x_T \sim \mathcal{N}(0, I)$ follows a chi distribution:

$||x_T|| = \sqrt{\sum_{i=1}^d {x_T^i}^2} \sim \chi^d = ||x_T||^{d-1}e^{-||x_T||^2/2}/(2^{d/2-1}\Gamma({d \over 2}))$,

where $d$ is the dimensionality of $x_T$ and $\Gamma(\cdot)$ is the Gamma function. Based on this, to prevent the adjusted initial noise from deviating into an OOD state, one could consider minimizing the objective $||\tilde{\epsilon}_\theta(x_T, T, y)||_2 -w\log{p(||x_T||)}$, where $p$ is the chi distribution and $w$ is a weight (hyperparameter). Since $p$ is the chi distribution, this objective can be rewritten as:

$||\tilde{\epsilon}_\theta(x_T, T, y)||_2 + c - {w (d-1)\log{||x_T||} + {w{||x_T||^2 \over 2}}}$,

where $c$ is a constant independent of $x_T$. We argue that AdamW's weight decay implicitly minimizes the final term, ${||x_T||^2 \over 2}$, and therefore helps prevent adjusted initial samples from becoming OOD. To empirically validate this, we conducted an experiment comparing 5000 unadjusted and 5000 adjusted initial noise samples. We measured the Jensen–Shannon divergence (JSD) between the two resulting distributions, varying the weight decay parameter $w$. The results, shown below, indicate that as $w$ increases, JSD decreases. This suggests that stronger weight decay encourages the adjusted samples to remain closer to the original (in-distribution) noise distribution.

Finally, to examine the effect of weight decay on mitigation performance, we varied the weight decay parameter $w$ as described above and measured the corresponding performance. The table below presents the results of this experiment.

Across the experimental settings we used, the performance remained relatively consistent regardless of the value of $w$. We believe this is because the default value used in our experiments, $w=0.01$, does not cause the adjusted initial noise to significantly deviate from the original distribution $\mathcal{N}(0, I)$. Indeed, as shown in the JSD results table above, the JSD at $w=0.01$ is 0.0665, indicating minimal distributional shift.

Q1-3: A reason why this type of regularization is not applied.

We did not explicitly incorporate a regularization term to ensure that the initial noise $x_T$ remains within the high-likelihood region of the Gaussian distribution. However, as mentioned in our response to the previous question, the experimental setting used for performance evaluation in this paper (AdamW with $w = 0.01$) does not cause the adjusted initial noise to deviate significantly from the original Gaussian distribution. In fact, it results in a low JSD value.

Moreover, even when we increased the value of $w$—thereby encouraging the adjusted initial noise $x_T$ to stay closer to the high-likelihood region of the Gaussian distribution—no significant performance improvement was observed. These findings suggest that the adjusted initial samples are unlikely to fall into an out-of-distribution (OOD) state that might trigger "reward hacking" behavior.

That said, as you pointed out, explicitly regularizing the adjusted initial noise $x_T$ to remain in-distribution—for example, by adding the regularization term $-\log{p(||x_T||)}$—could contribute to safer deployment of diffusion models. Since

$-\log{p(||x_T||)}= c - { (d-1)\log{||x_T||} + {{||x_T||^2 \over 2}}}$,

and weight decay already minimizes the ${{||x_T||^2 \over 2}}$ term, it should be straightforward to incorporate the full regularizer by augmenting the existing metric $||\tilde{\epsilon}_\theta(x_T, T, y)||_2$ with the additional term $- { (d-1)\log{||x_T||}}$.

Q2: Decomposing evaluation into quality and diversity.

As suggested, we evaluated performance using LPIPS and ImageReward. For both metrics, higher values are better. Despite similar CLIP and ImageReward scores, methods like Adding 4 Random Tokens, Prompt Embedding Adjustment, and Per-sample show notable differences in SSCD and LPIPS, underscoring the strength of our approach.

Q3: Synergy with other methods.

As shown in [3], $||\tilde{\epsilon}_\theta(x_T, T, y)||_2$ is typically higher for memorized prompts. Our method reduces this value by adjusting the initial noise $x_T$ for such cases.

This design is compatible with other noise optimization techniques aimed at enhancing visual quality or prompt alignment, as long as they do not increase $||\tilde{\epsilon}_\theta(x_T, T, y)||_2$. Integration can be done by simply adding our term to existing loss functions.

Q4: Generality of the Framework.

If an analogous attraction basin could be defined based on unconditional noise prediction $\epsilon_\theta(x_t, t)$ and large $||\epsilon_\theta(x_t, t)||_2$ values similarly indicate strong steering, our method could be adapted accordingly.

[1] Samuel et al. “Norm-guided latent space exploration for text-to-image generation”. NeurIPS 2023.

[2] Ben-Hanmu et al. "D-Flow: Differentiating through Flows for Controlled Generation". ICML 2024.

[3] Jain et al. "Classifier-Free Guidance inside the Attraction Basin May Cause Memorization". CVPR 2025.

Thank you for the detailed rebuttal. I have some concerns about the optimization methodology that I'd like to clarify.

Missing Critical Regularization Component:

Your rebuttal reveals for the first time that weight decay in the AdamW optimizer is intended to serve as an implicit regularizer to keep the optimized noise latent $z_0$ in-distribution. You correctly argue that this involves minimizing $||z_0||^2$. This is not a minor implementation detail; it is the theoretical bedrock of your optimization strategy as can be seen by the SGD numbers.

You mention that this regularization approximates the likelihood of the chi-squared distribution, but there's an important connection you don't explicitly state: minimizing $||z_0||^2$ is equivalent to maximizing likelihood under the Gaussian prior as for $z_0 \sim \mathcal{N}(0, I)$, the log-likelihood is: $\log p(z_0) = -\frac{d}{2} \log(2\pi) - \frac{||z_0||^2}{2}$.

The regularization through weight decay is not a minor implementation detail but a fundamental component that governs the theoretical soundness of your entire approach. Without this regularization, your method risks producing out-of-distribution noise samples that could lead to unpredictable generation behavior. This component is essential for understanding why your method works reliably and safely, yet readers of your paper would have no knowledge of this critical aspect. The omission of such a core theoretical element from the original submission indicates that the methodological presentation could benefit from more comprehensive documentation of key design choices.

Adam Numbers Discrepcancy: Your table shows AdamW and Adam achieve identical results across all metrics (0.2265 SSCD, 0.2647 CLIP, 111.7207 FID). This would suggest that actually the weight decay has no impact? Could you please clarify this discrepancy?

Recommendation: This regularization framework should be prominently featured in any revision as it's essential for understanding the method's theoretical foundations. I believe the framing should include the goal of keeping $z_0$ in-distribution and then either maximize likelihood under the initial Gaussian, or the Chi-squared, and use the corresponding regularization term. Ideally including an ablation of which of the impact of these terms, and that this is crucial to the method working as suggested by the SGD numbers.

LPIPS computation: What exactly are you computing LPIPS between here?