Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency

Thank you for your review and for your assessment that our experimental evaluation is extensive. To add to this, we add the additional experiments you have requested below, namely a runtime analysis and SPELL’s performance under text prompts of varying length. We also provide further results, including qualitative examples, in the responses to the other reviewers.

The SPELL method introduces additional computational overhead due to the repellency terms, potentially making it less efficient than simpler diffusion models.

A runtime analysis can be found in Table 5 in the appendix, we also post it here for your convenience. It shows that the generation time with and without SPELL is equal in the diversity setup. This is because SPELL only adds one matrix operation to calculate distances and one to add the repellency terms, which is small compared to the computational cost of the diffusion backend.

The authors should compare the performance of the proposed SPELL method with varying text prompts

Thank you for the suggestion, we have added your suggested experiment here https://imgur.com/a/T6KdRAR . We sliced the CC12M dataset into groups of prompts with increasing length. As can be seen in the plot, SPELL consistently increases the diversity of both short and long prompts compared to the baseline model, without losing on the CLIP-Score, which measures prompt adherence.

The paper lacks organization, making it difficult to follow and identify the key points.

Thank you for the feedback, we will add intuitions with the additional space of the camera-ready version of the paper.

Thank you again for your feedback. We would be happy to learn if the additional experiments and explanations resolve your concerns and update your evaluation of our paper.

Thank you for your review and for your interest in the theory behind SPELL that enables its outperformance. We delve into the theoretical connections between SPELL and other methods below, as well as the experimental results for pixel-space diffusion. We also provide further results in the responses to the other reviewers.

I believe the paper needs to validate the proposed method with pixel-space diffusion to enhance its visibility.

We already validate on a pixel-space diffusion model, Simple Diffusion, as noted in lines 239 and 292. Please refer to Table 1 for its results. In general, SPELL works independently of the space the diffusion model diffuses in, whether it is a latent VAE space or a direct RGB diffusion, and whether it is an unguided, classifier-guided, or classifier-free guided model. This sets it apart from alternative approaches like CADS and IG.

This paper doesn’t rigorously offer a theoretical framework for negative guidance using diffusion models. However, their conditions and the magnitude of negative guidance are based on a geometric interpretation.

SPELL’s repellency directions and magnitudes are based on a theoretical framework, which is explained in appendices A to C. The general problem we tackle is that we want to bring the probability / density that a diffusion model generates (an r-Ball around) a point $z_k$ to exactly zero, i.e. $P_0(B_r(z_k)) = 0$. The interesting part about this proof is that we need to choose the direction and magnitude of the intervention despite not knowing the density at $P_0(z_k)$ of the original diffusion model, since diffusion models do not provide density estimates. In Appendix B we use Tweedie’s formula for a theoretical derivation of the exact direction and magnitude required to achieve this, leading to Equation (5). While in the main paper, we start with a geometric interpretation of Equation (5), we also give a conservative-field interpretation in Appendix C to connect SPELL to other negative-guidance based frameworks, and show SPELL’s theoretical relation to Diffusion Posterior Sampling in L165-219. We hope that these rigorous theoretical analyses help future researchers connect and develop SPELL as well as negative guidance.

The proposed method consistently improves recall metrics, but the FID score is not maintained. However, the paper does not provide an explanation for this issue.

We attribute the decrease in FID to the fact that it is a reference-based metric that compares distributional similarity with a reference dataset. As SPELL increases the diversity and explores more of the image manifold, its output distribution is intentionally broader than the reference dataset that the diffusion models without SPELL are trained to match as closely as possible. This issue occurs with all reference-based metrics (precision, coverage, recall, density), which is why we report reference-free metrics like the Vendi and CLIP score. SPELL’s trade-off is, however, better than for other diversity-inducing metrics (CADS, IG, PG in Fig. 3), even for reference-based metrics. To demonstrate the preservation of image quality, we’ve generated additional high-resolution SD3 + SPELL images here https://imgur.com/a/Dsh7gxb https://imgur.com/a/Uz0LhOz https://imgur.com/a/VGcvqNE https://imgur.com/a/zrZRuV2 https://imgur.com/a/cuSAADZ .

It would be beneficial to review the paper [1] that discusses negative guidance aiming for mitigation of memorization. Furthermore, [2] also addresses dynamic negative guidance while preserving generation performance and effectively negating a portion of the data distribution.

Thank you for those interesting pointers! We will discuss them in the camera-ready version of the paper (this year's reviewing form does not allow to upload revised pdfs).

Thank you again for your time during the busy rebuttal period. We would be happy to learn if the additional experiments and explanations resolve your concerns and update your evaluation of our paper.

Thank you for your review and for acknowledging that SPELL tackles meaningful practical applications while remaining training-free. We are happy to provide the experimental results for high-resolution images, complex prompts, and compute overhead that you have requested below.

Examples and analysis under the SD3 model with higher resolution images and more complex prompts

We have added 1024x1024 images with prompts of varying length, generated both via SD3 and FLUX, under the following links: https://imgur.com/a/Dsh7gxb https://imgur.com/a/Uz0LhOz https://imgur.com/a/VGcvqNE https://imgur.com/a/zrZRuV2 https://imgur.com/a/cuSAADZ https://imgur.com/a/oHouASK https://imgur.com/a/3mI2ynU https://imgur.com/a/knqPFhc https://imgur.com/a/FwJvkhG https://imgur.com/a/qigDza4 . We’ve generated the images one-by-one from the same seeds like in Figure 5. The results show that while the first samples of the unconditional model are already diverse enough (top rows) and SPELL does not act due to its sparsity, it starts diversifying the images in the later generations (bottom rows). This holds both for simple and complex prompts.

The experiments in our paper also already use complex prompts, with the CC12M prompts ranging from 15 to 491 characters. We have added an ablation on SPELL’s diversity for different prompt lengths here https://imgur.com/a/T6KdRAR . It shows that SPELL increases the diversity of the generated images even in the longest 90-100% percentiles of the prompts compared to the baseline model, without compromising on the prompt adherence measured by the CLIP score.

I believe the authors should express their content using more accessible, standardized, and logically structured narratives.

Thank you for the feedback! We will highlight the narrative in the camera-ready version (this year's reviewing form does not allow to upload revised pdfs).

Figure 1 only shows results for Simple Diffusion and MDTv2, with no results for Stable Diffusion 3.

We apologize for this editing issue. We provide a Stable Diffusion 3 example here, https://imgur.com/a/qNYX5tX , on top of the other SD3 examples linked above.

Experiments on larger resolution images (greater than 256*256)

We have added results for 1024x1024 images, see the links above.

More complex scenarios beyond "a dog plays with a ball,". Can more diverse generations could be achieved in such cases?

The “a dog plays with a ball” prompt was only used in Figure 5 to give a qualitative example. Our quantitative experiments already use CC12M prompts with 15-491 characters, and we have added qualitative results for longer prompts in the links above, following your recommendation. We will outline this in the camera-ready version.

Is there additional computational overhead introduced, and could they provide an analysis of this overhead?

We provide an analysis of computational overhead in Table 5 in the Appendix for the diversity use-case, and also reposted here. As can be seen, SPELL does not increase the generation time during diverse generation, since it adds only one matrix operation to calculate distances and one to calculate the repellency direction (and in many timesteps does not even do this thanks to its sparsity (see Appendix H)) which is a small computation compared to the diffusion backbone.

Could the authors distill the paper's greatest contribution

We provide SPELL, a sparse and training-free mechanism to push diffusion trajectories away from already-generated or protected images. We will make this clearer in the abstract and introduction, thank you for your feedback!

Thank you again for your reviewing time. We would be happy to learn if the additional experiments and explanations resolve your concerns and update your evaluation of our paper.

Thank you for your review and for recognizing that SPELL is a simple way to tackle two current problems in diffusion models at once. We are happy to share explanations and your requested FLUX experiments below.

Can image protection with SPELL be applied to text-to-image models?

Yes, there is no difference in applying SPELL for image protection on class-to-image, noise-to-image, or text-to-image diffusion models. Unlike approaches like Interval Guidance and CADS, SPELL is generalized away from the conditioning signal, i.e., it does not rely on classifier- or classifier-free guidance, but acts on the diffusion trajectory and the output space of any diffusion model itself, even if the conditioning signal tries to push the diffusion trajectory close to a protected image. SPELL can thus protect any-to-image diffusion models. The reason why we demonstrate this on the class-to-image model MDTv2 in Section 4.6 as opposed to a text-to-image model is simply that MDTv2’s exact training dataset and preprocessing are public (ImageNet-1k) so that we can protect the exact training data, whereas text-to-image models like SD3 are trained on proprietary dataset splits.

Why does SD3 + SPELL have significant degradation in FID and FD_DINO? Does this imply a drop in image quality?

We attribute the decrease in FID to the fact that it is a reference-based metric that compares distributional similarity with a reference dataset. As SPELL increases the diversity and explores more of the image manifold, its output distribution is intentionally broader than the reference dataset that SD3 without SPELL is trained to match as closely as possible. This issue occurs with all reference-based metrics (precision, coverage, recall, density), which is why we report reference-free metrics like the Vendi and CLIP score. SPELL’s trade-off is, however, better than for other diversity-inducing metrics (CADS, IG, PG in Fig. 3), even in reference-based metrics. We’ve also generated additional high-resolution SD3 + SPELL images, generated one-by-one with the same seeds as in Figure 5, to show the image quality https://imgur.com/a/Dsh7gxb https://imgur.com/a/Uz0LhOz https://imgur.com/a/VGcvqNE https://imgur.com/a/zrZRuV2 https://imgur.com/a/cuSAADZ

Can the SPELL method be applied to rectified flow-based models? For example, FLUX.

Yes, SPELL can be applied to any diffusion model. We have added examples for FLUX1.schnell in 1024x1024: https://imgur.com/a/oHouASK https://imgur.com/a/3mI2ynU https://imgur.com/a/knqPFhc https://imgur.com/a/FwJvkhG https://imgur.com/a/qigDza4 Since FLUX1 already generates more diverse images by itself compared to SD3, SPELL’s sparsity means that it is activated on fewer pictures, and in more details, like background objects or colors. If you are interested in one-step diffusion models, we denote that SPELL should be used with a number of diffusion steps greater than 1, because it is applied after each diffusion step. If there is only one step, it will be applied effectively after the generation has already ended, which could introduce artifacts. We will add this discussion to the camera-ready version of the paper to outline SPELL’s applicability.

Which model does Figure 3 compare?

Figure 3 shows results for Latent Diffusion, as denoted in the caption.

The assumption of $B_k$ being disjoint in L165-183, page 4, limits the range of $r$ ( if $r \rightarrow \infty$, $B_k$ will inevitably overlap). Can the authors provide a principal way for choosing $r$?

In the protection case, the question of how big of a shield a user wants to create around a protected image is ultimately a user choice. Should they use a radius that large that shields start overlapping, and should a trajectory point exactly to the middle of two shields (and inside the radius of both shields), their terms cancel each other out, as we openly address in L179-180. In this case, the overlapping shields $z_1$ and $z_2$ can be merged with a combined radius of $r_\text{merged} = r + d(z_1, z_2)$ and $\frac{z_1+ z_2}{2}$ as shield center to retain the protection guarantee, as mentioned in L673 in the appendix. However, in practice we did not observe this to be necessary – even in the protection experiment on all 1.2M ImageNet-1k images, we did not have issues with overlapping shields. Due to the high-dimensional space, it is unlikely that a trajectory points exactly in the middle of two shields, where their terms would cancel each other out exactly. Rather, their terms point in slightly different directions, so that SPELL guides the trajectory points away from both shields throughout the diffusion. So, a user is able to choose a shield radius as large as it suits their needs. As a side note, for the diversity use case, we provide a principled way of choosing $r$ in L779 in the appendix.

Thank you again. We would be happy to learn if the additional experiments and explanations update your evaluation of our paper.