An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models

We thank Reviewer f3gh for his/her thoughtful feedback on our work. We are glad to hear that the reviewer finds the idea proposed in our work “sound”. Regarding the reviewer’s comments and questions:

1-”What's new of your conditional-Vendi compared with the conditional entropy proposed by Sanchez Giraldo et al., besides the exponential?”

We believe that our technical contribution extends beyond merely proposing the application of the matrix-based conditional entropy for evaluating model-induced diversity in conditional generative models. Specifically, we provide an interpretation of the conditional entropy in Theorem 1, which establishes a connection between the conditional entropy defined by Sanchez Giraldo et al. in [1] and the average of the (unconditional) entropy over the components of a mixture text distribution $P_T$. To the best of our knowledge, this theoretical result is novel and has not been previously addressed in the literature.

For more details, please refer to Item 1 in our general response.

2- “Follow which equation can the conditional-Vendi score be calculated?"

The equation in Line 263 combined with entropy definition in Equation (3) gives the formula for computing the Conditional-Vendi score.

3- "It's better to write an explicit algorithm, perhaps in the appendix.”

Following the reviewer’s suggestion, we have added Algorithm 1 to the revised Appendix B.6. We thank Reviewer f3gh for the suggestion.

4-”What do you want to convey by Section 5 (the statistical interpretations)?”

In Section 5, we first derive the statistic estimated by the matrix-based conditional entropy function defined in [1]. Note that the conditional entropy definition $H(x_1,\ldots , x_n|t_1,\ldots ,t_n)$ in [1] is a complex function of the samples, and it is not clear what target statistic this score aims to estimate as the sample size $n$ approaches infinity. In Proposition 1 and Corollary 1, we derive the target statistic to reveal what function of the joint (text,image) distribution $P_{T,X}$ is estimated by the matrix-based conditional entropy defined in [1].

In Theorem 1, our goal is to interpret the matrix-based conditional entropy $H(X|T)$ defined by Sanchez Giraldo et al. Please note that the matrix-based conditional entropy measure $H(X|T)$ defined in [1] is not equal to the average of instance-specific entropies $H(X|T=t)$ over $t \sim P_T$, which means

$H(X|T) \text{ } {\large\neq} \text{ } \mathbb{E}_{t \sim P_T}[H(X|T=t)]$

Theorem 1 addresses this gap and interprets the matrix-based conditional entropy $H(X|T)$ as averaging the entropies $H(X|C=i)$ for the components of a mixture text distribution $P_T$. These components are defined based on the bandwidth parameter $\sigma$ of the text Gaussian kernel function, which determines the threshold for identifying two embedded text vectors as “close” or “far.” For more details, please refer to Item 1 in our general response.

[1] Sanchez Giraldo et al., “Measures of entropy from data using infinitely divisible kernels”. IEEE Transactions on Information Theory, 2014

We thank Reviewer Uq1Z for his/her thoughtful feedback on our work. We are glad to hear that the reviewer finds the problem studied in our work “interesting and useful ”. Regarding the reviewer’s comments and questions:

1- Technical Contribution of our work

We believe that our technical contribution extends beyond merely proposing the application of the matrix-based conditional entropy for evaluating model-induced diversity in conditional generative models. Specifically, we provide an interpretation of the conditional entropy in Theorem 1, which establishes a connection between the conditional entropy defined by Sanchez Giraldo et al. in [1] and the average of the (unconditional) entropy over the components of a mixture text distribution $P_T$. To the best of our knowledge, this theoretical result is novel and has not been previously addressed in the literature. For further details, please refer to Item 1 in our general response.

2- “the idea of constructing conditional diversity score based on these results does not compose of sufficient contribution”

Please note that the matrix-based conditional entropy measure $H(X|T)$ defined in [1] is not equal to the average of instance-specific entropies $H(X|T=t)$ over $t \sim P_T$, i.e.,

$H(X|T) \neq \mathbb{E}_{t \sim P_T}[H(X|T=t)]$

As a result, our proposal for model-induced diversity quantification is not simply the averaging of entropies $H(X|T=t)$ for data generated for each prompt $t \sim P_T$. In Theorem 1, we interpret this process as averaging the entropies $H(X|C=i)$ for the components of a mixture text distribution $P_T$. These components are defined based on the bandwidth parameter $\sigma$ of the text Gaussian kernel function, which determines the threshold for identifying two embedded text vectors as “close” or “far.”

3- “so what about just calculate the unconditional entropy for each cluster of prompts, and then use some weighted average to combine them?”

We implemented the reviewer’s suggestion to define a golden metric, “GroundTruth-Cluster-Vendi,” which computes the average Vendi scores over text clusters assumed to be provided by an oracle. Please refer to Item 2 in our general response for the numerical results showing the correlation between the GroundTruth-Cluster-Vendi and our defined Conditional-Vendi scores.

While the intuition behind the Cluster-Vendi score suggested by the reviewer is meaningful, note that, in a general evaluation task, we do not have access to the ground truth clusters. Consequently, computing sample clusters to evaluate the Cluster-Vendi score becomes necessary. This task can be intractable in general cases, as we may not know the number of clusters or whether the samples are well-clustered. Even if the number of clusters is known, the standard K-Means algorithm can converge to different solutions depending on its initialization, as the K-Means optimization problem is highly non-convex. In our numerical experiments, we also observed that the KMeans-Cluster-Vendi score exhibits significant variance due to the random initialization of K-Means clustering. In contrast, our defined Conditional-Vendi score can be computed deterministically through the eigendecomposition of kernel matrices, which can be computed with no error using the solution to the tractable eigendecomposition task.

4- Measuring the conditional diversity using a clustered Vendi score

We agree with the reviewer that the GroundTruth-Cluster-Vendi can serve as a valuable golden metric for evaluating the Conditional-Vendi score. For further details on this suggestion, please refer to Item 2 in our general response. We thank the reviewer for this great suggestion.

5- "If prompts cannot be easily clustered and come from some complex distributions… then the concrete meaning of this measure is unclear to me."

In a general scenario where the text samples may not be well-clustered, the Conditional-Vendi score relies on the eigenvectors of the kernel covariance matrix to define soft clusters over the data. Note that, given a kernel function $k$, the kernel covariance matrix is a positive semi-definite (PSD) matrix with orthogonal eigenvectors. In our proposed diversity evaluation, each eigenvector of the kernel covariance matrix represents a soft cluster of the data.

If the data distribution $P_T$ consists of well-separated components, then with high probability, the drawn samples will form well-defined clusters, and the soft clusters derived from the eigenvectors will closely resemble hard clusters. However, even when the samples are not well-clustered, the interpretation of soft clusters based on the eigenvectors of the kernel covariance matrix remains applicable to the proposed diversity calculation.

[1] Sanchez Giraldo et al., “Measures of entropy from data using infinitely divisible kernels”. IEEE Transactions on Information Theory, 2014

We thank Reviewer awod for his/her thoughtful feedback on our work. We are glad to hear that the reviewer finds the work “well-organized and logically fluent”. Regarding the reviewer’s comments and questions:

1- “it should at least demonstrate some (new) important properties or use cases”

We believe that we have demonstrated novel and important properties of the matrix-based conditional entropy as defined by Sanchez Giraldo et al. in [1]. It is important to note that the matrix-based entropy measure $H(X|T)$ defined in [1] lacks the standard interpretation of Shannon conditional entropy, which corresponds to the averaged instance-specific entropy $H(X|T=t)$ over $t \sim P_T$. As a result, the relationship between $H(X|T)$ as defined in [1] and the average of instance-specific entropies $H(X|T=t)$ remains unclear.

Our main technical contribution, Theorem 1, aims to address this gap and provides an interpretation of the matrix-based conditional entropy $H(X|T)$. We believe this contribution offers useful insights into the properties of matrix-based conditional entropy measures. For more details, please refer to Item 1 in our general response.

2-“In such a case, the only novelty I can admit is that the paper demonstrates in experiments that the conditional entropy behaves normally in generative models.”

We would like to emphasize the theoretical results presented in Section 5, which are, to the best of our knowledge, entirely novel. Theorem 1 provides an operational interpretation for the conditional entropy $H(X|T)$ defined by [1]. As far as we know, this has not been addressed in the existing literature. For more details, please refer to Item 1 in our general response.

3- “No golden metric, or even any proxy metric, has been introduced for comparison.”

As discussed in Item 2 of our general response, to the best of our knowledge, our work is the first to explore the diversity decomposition of conditional generative models. Consequently, there are no established metrics in the literature to separately quantify model-induced and prompt-induced diversity of generated samples that we could use as a golden metric.

In the revision, we have followed the suggestion of Reviewer Uq1Z to define and utilize the GroundTruth-Cluster-Vendi metric as a surrogate golden metric. The GroundTruth-Cluster-Vendi assumes knowledge of sample clusters and computes the average (unconditional) Vendi score of samples within each cluster. In the nine simulated settings where we had access to the ground truth sample clusters, the GroundTruth-Cluster-Vendi score exhibited strong correlation with the conditional-Vendi score. For additional details, we kindly refer the reviewer to Item 2 of the general response.

4- “the range of experiments in the paper is still limited”

In the revision, we conducted the diversity evaluation for 9 additional settings, combining three categories with three SOTA text-to-image models. For more details, please refer to Item 3 of the general response.

5- “All experiments are simply measuring the scores with varying amounts of input clusters (groups)”

As discussed in Item 3, to the best of our knowledge, there are no baseline metrics in the literature to quantify model-induced diversity. Consequently, we could not assess the Conditional-Vendi score by measuring its correlation with an existing baseline metric. In the original submission, we simulated numerical settings where the ground truth ranking of model-induced diversity across several models was known, and demonstrated that the Conditional-Vendi score accurately reflected this ground truth ranking.

In the revision, we also included a numerical comparison with the newly defined golden metric, GroundTruth-Conditional-Vendi. However, it is important to note that computing the GroundTruth-Conditional-Vendi score requires knowledge of the underlying clusters. For a general dataset, this computation becomes intractable due to the non-convex nature of K-Means and other standard clustering optimization problems.

[1] Sanchez Giraldo et al., “Measures of entropy from data using infinitely divisible kernels”. IEEE Transactions on Information Theory, 2014

We thank Reviewer 3Epu for his/her thoughtful feedback on our work. We are glad to hear that the reviewer finds the paper “well-written”. Regarding the reviewer’s comments and questions:

1-Technical contribution of our work

To explain our technical contribution, we note that without Theorem 1, the connection between the (unconditional) Vendi score and the Conditional-Vendi score (defined using the matrix-based conditional entropy) would remain unclear. This is because the conditional entropy $H(X|T)$, as defined in Sanchez Giraldo et al. (2014), does not correspond to the averaged entropy $H(X|T=t)$ over text instances $t \sim P_T$. It is important to note that the Vendi score relates only to instance-specific entropies $H(X|T=t)$.

As explained in Item 1 of our general response, Theorem 1 in our paper addresses this gap. Specifically, under a mixture distribution on text $T$ with minimal assumptions, Theorem 1 provides an interpretation of the Conditional-Vendi score as the aggregation of Vendi scores across the different components of the mixture text distribution.

2- contextualization and interpretation of Theorem 1

We have included additional discussion in the revised introduction and section 5 to better explain the role of Theorem 1.

3- “showing some of the generations from the models in Figure 4”

We thank the reviewer for the suggestion. We have added the pictures generated by the models in Figure 4 of the main text to improve the clarity of the qualitative evaluation. The results can be found in Figure 25 in the revised Appendix B.7

4- “in light of Figure 6, it seems that it would be possible to achieve a high conditional Vendi score by a model that has poor alignment with the prompt.”

We would like to clarify that the video generation models in Figure 6 have generated videos that are aligned with the provided prompts. Due to space constraints, we included only 5 of the text prompts corresponding to the 16 generated videos displayed per cluster in Figure 6. As a result, 11 text prompts associated with the displayed videos were not shown in Figure 6, which may have led to the impression that the generation models produced poorly-aligned videos. We have clarified this point in the revised manuscript, and in the revised Appendix Figure 24, we have provided the complete set of prompts for the samples displayed in Figure 6.