SPARKE: Scalable Prompt-Aware Diversity and Novelty Guidance in Diffusion Models via RKE Score

We would like to thank Reviewer W1na for the thoughtful feedback and are glad that the reviewer found our paper to “present a careful mathematical derivation” and our numerical result to “demonstrate consistent gains in diversity metrics”. Below is our response and clarifications regarding the questions and comments in the review:

1- runtime comparison of SPALE guidance with the baselines

We thank the reviewer for pointing this out. To address the comment, we evaluated SPALE's computational efficiency by comparing its runtime and peak memory with those of the baselines over 1000 text-to-image generations (512×512 resolution) using a 50-step diffusion process on an NVIDIA RTX 4090 (24GB VRAM). The results are in the following table, indicating that SPALE requires significantly lower runtime and less memory (as it functions in the latent space). We will include this table in the revised manuscript.

2- Additional quantitative comparison on state-of-the-art diffusion models

Following the reviewer’s comment, we have performed additional numerical evaluation and here provide quantitative comparisons of the state-of-the-art models, SD-XL and PixArt-$\Sigma$, in the following table. These quantitative results show SPALE results in an increase in diversity scores (Con-Vendi, Vendi, In-batch Sim, scores) with only a minor decrease in the evaluated CLIPScore. We also note that the kernel distance (KD) evaluation score (evaluating both quality and diversity) and the Authenticity score (with MS-COCO being the reference dataset) have both improved in the guided sample generation.

We would like to thank Reviewer THSZ for the thoughtful feedback and are glad that the reviewer found our method to be “new” and with “solid theoretical foundation”. Below is our response and clarifications regarding the questions and comments in the review:

1- Number of Prompts in the Numerical Evaluation

First, we think there might be a misunderstanding of the number of MS-COCO prompts in our experiments; The actual number is 10,000, which is stated to be only 1000 in the reviewer’s comment.

To further investigate this, we have numerically validated that the diversity represented by the selected 10,000 MSCOCO prompts can reflect the diversity of the entire MS-COCO validation set. To do this, we measured the Recall [1] and Coverage [2] scores of the selected prompts with respect to the reference set of the whole MSCOCO validation set: The Recall score was 0.91, and the Coverage score was 0.96, suggesting that a major fraction of the prompt clusters in MSCOCO are present in the selected set.

2- Quantitative evaluation on additional diffusion models

To address the reviewer's comment, we provide the quantitative evaluation results on the state-of-the-art text-to-image diffusion models, SDXL and PixArt-$\Sigma$, in the following table. The numerical results indicate a similar trend to the Stable diffusion cases in Table 2 of the main text.

3- Quality and Diversity trade-off in the SPALE guidance

Regarding this comment, we would like to refer to the Appendix, Figure 11 (Qualitative results with different CFG scales) and Figures 8, 9, and 10 for a quantitative comparison of the quality/diversity trade-off with precision/recall, density/coverage, and Vendi Score/KD. Furthermore, we have included randomly selected samples generated by the PixArt-$\Sigma$ and SDXL diffusion models in Figure 14 (PixArt-$\Sigma$ results) and Figure 15 (SDXL results) in the appendix.

Also, we would like to highlight that CLIP-Score may not fully reflect the semantic alignment of a text and image pair in the presence of multiple components in the input image. As noted in Reference [3] (refer to Figure 1 in this paper), the image encoder of CLIP could favor larger objects, while its text encoder could prioritize first-mentioned objects. Similarly, Reference [4] states that “CLIP puts a large focus on the few main objects of an image”. Such biases in the CLIP embedding may potentially lead to slightly lower CLIP-Scores when diverse components are also present in the image.

4- Reliability of the score in early diffusion steps

We thank the reviewer for pointing this out. We would like to highlight that one advantage of applying guidance in the latent space, rather than in the ambient space, is that it does not require the latent representation to be decoded into an image, which can be especially useful in early diffusion steps. This feature of latent RKE diversity guidance avoids the need for computing scores on noisy intermediate images during the early diffusion steps, which might lead to unreliable evaluated scores.

[1] Kynkäänniemi et al. Improved Precision and Recall Metric for Assessing Generative Models. In NeurIPS 2019

[2] Naeem et al. Reliable Fidelity and Diversity Metrics for Generative Models. In ICML 2020

[3] Abbasi et al., CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation. In CVPR 2025

[4] Stein et al. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.In NeurIPS 2023

We would like to thank Reviewer HGHP for the thoughtful feedback on our work. We are glad that the reviewer finds our experiments “rigorous and comprehensive” and the paper “clear and well-written”. Below is our response and clarifications regarding the questions and comments in the review:

1- Analysis on diversity guidance scale $\eta$

We appreciate the reviewer’s point on the diversity guidance scale hyperparameter. We would like to kindly refer the reviewer to Figure 6 in the appendix in the supplementary material, where we qualitatively compared the effect of the guidance scale ranging from 0 to 0.1 on the prompt-guided sample generation using SDXL.

To further address the reviewer’s comment, in the following table, we provide a quantitative comparison of the performance with different $\eta$ values on class conditional Diffusion Transformer (DiT-XL/2), which shows the higher diversity of the generated data with larger values of $\eta$ with only a minor reduction in the Precision/Density in sample generation. Specifically, our numerical results highlight that with a choice of $\eta$ in the range of $[0.02, 0.06]$, the diversity score can significantly grow while preserving the Precision/Density of generated data. In the revision, we will include Figure 6 and the following table in the main text.

2- Qualitative comparison of SPALE guidance

Regarding this comment, we would like to refer the reviewer to kindly view the randomly selected samples from different diffusion models in Figure 11 (Qualitative results with different CFG scales), Figure 14 (PixArt-$\Sigma$ results), and Figure 15 (SDXL results) in the appendix.

To further address the reviewer’s comment, we also provide quantitative comparisons of these state-of-the-art models, as shown in the following table, which shows an increase in diversity scores (Con-Vendi, Vendi, In-batch Sim, scores) with only a minor decrease in the evaluated CLIPScore. We also highlight that the kernel distance (KD) evaluation score (evaluating both quality and diversity) and the Authenticity score (with MS-COCO being the reference dataset) have indeed improved in the guided sample generation, indicating that the scores evaluating both quality and diversity have increased.

3- Qualitative comparison of Latent (ours) vs Ambient-based diversity guidance methods

We would like to highlight that one key property of our proposal is that we apply guidance on the latent space in contrast to the previous works that guide diversity scores in the ambient space (after passing through the CLIP or DINO embeddings). We observed that in large-scale diffusion models, the latent diversity guidance prevents generation artifacts and will generate semantically diverse concepts. In Figure 3 (in the main text), we showed the comparison of Vendi score guidance on the latent space vs. CLIP ambient space on SDXL, which shows that ambient guidance will lead to noticeable noise in the image, rather than true semantic diversity. Additional qualitative results are provided in Figure 12 of the Appendix, which further support this point.

We sincerely thank Reviewer HGHP for the feedback on our response and are pleased that our rebuttal was able to address most of the reviewer’s concerns. We will revise the manuscript accordingly to reflect the clarifications provided in our responses.

Regarding the reviewer’s point on the potential for further improvement in the results, we would be grateful if the reviewer could point us to any specific baseline on diversity guidance in diffusion models that they may have in mind. We would like to highlight that our method demonstrates improvements over existing diversity guidance baselines, including CADS [1], Particle Guide [2], and c-VSG [3], as highlighted in quantitative comparisons in Table 3 (Appendix) and supported by the runtime/GPU usages referenced in our response to Reviewer W1na. If Reviewer HGHP has a particular baseline in mind, we would be happy to provide additional experimental results—time permitting—to address it.

[1] Sadat et al. “CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling”, ICLR 2024

[2] Corso et al. “Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Models”, ICLR 2024

[3] Askari Hemmat et al. “Improving Geo-diversity of Generated Images with Contextualized Vendi Score Guidance”, ECCV 2024

We would like to thank Reviewer wxtJ for the constructive and thoughtful feedback and are delighted to hear that the reviewer found our work “clearly presented” and “well-supported by theoretical proofs and large-scale experiments”. Below is our response and clarifications regarding the questions and comments in the review:

1- Literature review for related works on guidance

We thank Reviewer wxtJ for mentioning the related works on guidance in diffusion models. As pointed out in the review, a more comprehensive discussion of guidance methods, particularly those based on Sequential Monte Carlo (SMC) and optimization formulations, will definitely position our work better in this literature. We will revise the related work section to include and discuss the mentioned references.

2- Relation to other kernel-based sampling methods

We thank the reviewer for the valuable feedback and for raising the connection to other kernel-based sampling methods. We appreciate this opportunity to clarify the theoretical distinctions and contributions of SPALE.

Our answer to the reviewer’s question is positive, and there is a gradient-flow-based interpretation of our method. SPALE, along with Stein Variational Gradient Descent (SVGD)[1], Particle Guidance (PG)[2], and the ensemble-based/kernel-based sampling methods in [3, 4], can be understood as an interacting particle system that follows a gradient flow to minimize a specific energy function: $\mathrm{d}\mathbf{z}^{(n)} = \left[ - \mathbf{f}(\mathbf{z}^{(n)}, t') + \frac{g^2(t')}{2} \left( \nabla_{\mathbf{z}^{(n)}} \log p_{t'}(\mathbf{z}^{(n)}) - \eta \nabla_{\mathbf{z}^{(n)}} L_{\text{Cond-IRKE}}(\mathbf{z}_1, \dots, \mathbf{z}_n) \right) \right] \mathrm{d}t + g(t') \mathrm{d}\mathbf{w}$ While previous methods define a gradient flow to optimize for sample quality and prompt-unaware diversity, our proposed SPALE defines a flow to minimize a Conditional Inverse-Rényi Kernel Entropy (Cond-IRKE). The core idea of SPALE lies in the novelty of the energy function it minimizes, thus represents a forward by introducing prompt-awareness and improving scalability.

Prompt-Awareness. Previous methods like Particle Guidance (PG)[10] employ a prompt-unaware repulsion mechanism. Their guidance potential $\log \Phi_{t'}(\mathbf{z}_1, \dots, \mathbf{z}_n)$ treats all samples in a batch of different prompts equally, applying a uniform repulsion force between every pair regardless of their corresponding prompts. This uniform repulsion limits the computational efficiency, and it is not completely aligned with the goal of generating diverse outputs within related contexts, i.e., prompt categories. SPALE introduces a prompt-aware guidance mechanism, which is the key difference from previous methods. By optimizing the Conditional Rényi Kernel Entropy during sampling, our guidance potential is defined by:

Linear-Time Computational Scalability. We would like to clarify that in studying the scalability of the proposed SPALE method, we specifically compare it with other diversity-score-based guidance methods, including the Vendi score guidance. The computation of the Vendi score and its gradient requires $O(n^3)$ computations for $n$ generated samples, which enforces considering short windows for applying the diversity guidance in generating a moderately large number of data. On the other hand, the proposed SPALE method switches to the RKE score whose gradient can be calculated with $O(n)$ computations, which allows us to scale the diversity score-guided sample generation to several thousands. We would like to highlight that as shown Theorem 1 in [5], the RKE (entropy order-2) is the only entropy score with such a property in the family of Renyi entropy measures. We will make this point on the scalability of SPALE in comparison to the Vendi score diversity guidance clear.

3- Insights of theoretical analysis of RKE Guidance

As previously discussed, our guidance algorithm follows the gradient-flow-based sampling formulations, and we provide a theoretical insight into interpreting how Rényi Kernel Entropy (RKE) score[6] and Conditional RKE score[7] enhance the sample diversity.

The main insight is that the order-2 RKE score serves as a differentiable objective for counting the modes of a particle distribution, and [6] provides a theoretical analysis about the RKE score of the kernel covariance matrix of mixtures of Gaussian and sub-Gaussian components. Furthermore, the Conditional RKE score in [7] quantifies the internal diversity conditioning on the prompt categories.

SPALE defines a gradient flow that optimizes the Conditional RKE score. This process sequentially drives samples to cover distinct modes, while the conditional mechanism ensures the prompt-aware diversity guidance potential. The dynamic nature of this sequential generation scheme makes it analytically challenging to characterize the final joint distribution of the samples. In the revised version, we will include the above discussion on how SPALE achieves prompt-aware diversity via RKE score and Conditional RKE score for future analysis.

[1] Liu, Q., & Wang, D. (2016). Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29.

[2] Corso, G., Xu, Y., De Bortoli, V., Barzilay, R., & Jaakkola, T. S. Particle Guidance: non-IID Diverse Sampling with Diffusion Models. In The Twelfth International Conference on Learning Representations.

[3] Wild, V. D., Ghalebikesabi, S., Sejdinovic, D., & Knoblauch, J. (2023). A rigorous link between deep ensembles and (variational) bayesian methods. Advances in Neural Information Processing Systems, 36, 39782-39811.

[4] Li, L., Korba, A., Yurochkin, M., & Solomon, J. Sampling with Mollified Interaction Energy Descent. In The Eleventh International Conference on Learning Representations.

[5] Ospanov, A., & Farnia, F. Do Vendi Scores Converge with Finite Samples? Truncated Vendi Score for Finite-Sample Convergence Guarantees. In The 41st Conference on Uncertainty in Artificial Intelligence.

[6] Jalali, M., Li, C. T., & Farnia, F. (2023). An information-theoretic evaluation of generative models in learning multi-modal distributions. Advances in Neural Information Processing Systems, 36, 9931-9943.

[7] Jalali, M., Ospanov, A., Gohari, A., & Farnia, F. (2024). Conditional Vendi Score: An information-theoretic approach to diversity evaluation of prompt-based generative models. arXiv preprint arXiv:2411.02817.