Towards a Scalable Reference-Free Evaluation of Generative Models

We would like to thank Reviewer ymdX for his/her time and constructive feedback and suggestions on our work. The following is our response to the reviewer’s comments and questions.

1. Variance of estimating entropy scores using FKEA

Re: To address the reviewer’s question, we measured the standard deviation of estimated FKEA entropy scores over 10 trials of running the experiment. We performed the variance estimation three times under: 1) independently drawn random Fourier features and fixed generated samples (across trials), 2) fixed (across trials) random Fourier features $\omega$ and independently drawn generated samples, 3) independently drawn random Fourier features and generated samples.

The estimated entropy scores and their standard deviation are presented in Table 2 of the Rebuttal PDF. The results indicate that FKEA with $r=4000$ and $r=8000$ can yield approximations with relatively low variance.

2. Selection of bandwidth $\sigma$ and Fourier feature size $r$ hyperparameters

Re: We would like to clarify that the kernel bandwidth hyperparameter $\sigma$ is a specification of kernel function which is separated from the FKEA method designed to approximate the Vendi [1,3] or RKE [2] scores with the evaluator’s chosen kernel function. Therefore, in our experiments we chose the $\sigma$ parameter following References [1-3] that originally proposed the scores. Also, note that, based on the theoretical results in [2], a proper choice of hyperparameter $\sigma$ should be greater than the maximum standard deviation of the modes of a mixture distribution and be smaller than the minimum Euclidean distance between the means of different modes.

On the other hand, the number of Fourier features $r$ is a hyperparameter of the FKEA method. Based on Theorem $1$, the selection $r=\frac{8(\log(n) + \log(1/2\delta))}{\epsilon^2}$ will be sufficient to guarantee an $\epsilon$-bounded approximation error for the kernel matrix’s eigenvalues with probability $1-\delta$. In our experiments, we observed that the choice of $r=8000$ for standard image datasets results in an estimation whose standard deviation is bounded by 1% of the estimated entropy.

3. Merits of reference-free metrics

Re: As discussed in the introduction, reference-free metrics are highly useful when the evaluator does not have access to a proper reference dataset. This can happen in scenarios where generative models produce outputs that are not represented in the standard datasets. To illustrate this, we included a toy example in Figure 1 (Rebuttal PDF) using a synthetic dataset of elephants generated by the Stable-Diffusion-XL text-to-image model. When the dataset consists of regular elephants, a reference set of “elephant” samples of ImageNet provides an accurate assessment of diversity. However, when the model is prompted to generate elephants with unrealistic colors, the internal diversity increases—something that ImageNet-based reference-dependent evaluations could not capture.

In this scenario, reference-dependent metrics seemed to report either lower score (Coverage) or no significant change (Recall). In contrast, the FKEA-computed Vendi and RKE scores could identify the increased diversity, suggesting its effectiveness in evaluating generative models in contexts where traditional datasets do not fully capture the generated data's diversity. We will include this example in the revised text.

4. Computational costs of FKEA

Re: Table 1 in the included figures (see attached PDF to global response) summarises the improved time complexity compared to Vendi/RKE scores computed by standard PyTorch eigendecomposition method using one NVIDIA GeForce RTX 3090 GPU. The FKEA method could significantly reduce the compute and memory costs, while providing an accurate estimation of the original metric with low variance (Table 2 in Rebuttal PDF). Moreover, RKE/VENDI were too expensive to compute with moderately high sample size e.g. 40k; however, FKEA remained applicable with much higher sample sizes such as 250k.

[1] Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. In Transactions on Machine Learning Research, 2023.

[2] Mohammad Jalali, Cheuk Ting Li, and Farzan Farnia. An information-theoretic evaluation of generative models in learning multi-modal distributions. In Advances in Neural Information Processing Systems 2023.

[3] Amey Pasarkar and Adji Bousso Dieng. Cousins of the vendi score: A family of similarity-based diversity metrics for science and machine learning. In International Conference on Artificial Intelligence and Statistics 2024

We would like to thank Reviewer NG2d for his/her time and constructive feedback and suggestions on our work. The following is our response to the reviewer’s comments and questions.

1. FKEA computational costs compared to VENDI/RKE

Re: To address the reviewer’s comment, we have measured the time taken by FKEA-based and non-FKEA-based (via PyTorch eigendecomposition) computation of Vendi and RKE scores. Table 1 in the Rebuttal PDF summarizes the time complexity of FKEA evaluation compared to the baseline eigendecomposition-computed scores on one NVIDIA GeForce RTX 3090 GPU. While the baseline computation of RKE/VENDI were unaffordably expensive in terms of memory and compute power for sample sizes above 30k, FKEA-based entropy computation remained feasible at much larger sample sizes, e.g. 250k. We will add the table to the revised paper.

2. Effect of Embeddings on FKEA entropy evaluation

Re: Based on this comment, we performed numerical experiments and compared the Vendi/RKE diversity scores across various embeddings, including DinoV2, CLIP, SwAV, InceptionV3 for images and text-embedding-3-large/small, BERT and Gemini for text data. Figures 2a and 2b (in Rebuttal PDF) illustrate the numerical results. The results indicate that the proposed FKEA method could approximate the Vendi/RKE metrics across different embedding spaces.

3. Correlation between FKEA-approximated scores and other diversity metrics

Re: To address this point, we refer to Table 1 in the main text that compares FKEA-Vendi and FKEA-RKE with standard assessment metrics for diversity across several image-based generative models. The results indicate that FKEA-evaluated entropy scores correlate with existing metrics e.g. Recall and Coverage.

We would like to thank Reviewer ohKy for his/her time and constructive feedback on our work. The following is our response to the reviewer’s comments and questions.

1. Datasets in the numerical evaluation

In the main text, we have discussed the numerical results on the following benchmark datasets:

• Images: MNIST, ImageNet, StyleGAN3 generated FFHQ
• Text: Synthetic countries and landmarks dataset
• Video: UCF101

Due to the 9-page space limit, we had to defer our numerical results for the other datasets to the Appendix. We would like to refer the reviewer to sections B1, B4, B5 in the Appendix where we report our numerical results on several more large-scale datasets including FFHQ, AFHQ, MSCOCO, F-MNIST, CNN/Dailymail 3.0, Wikipedia, CMU Movie Corpus and Kinetics-400.

2. Comparison to other evaluation metrics in the literature

Re: We would like to clarify that our main contribution is to provide a computationally efficient method for approximating the existing reference-free Vendi and RKE evaluation scores. The Vendi and RKE metrics have been already proposed and analyzed in the literature (References [1-3]); however, they could be expensive to compute in practice. In this work, we leverage the framework of random Fourier features to reduce the computational expenses of evaluating generative models with RKE and Vendi metrics. The merits of these two metrics have been already studied in References [1-3], and as discussed in the introduction, their main application is in evaluation settings where no reference datasets are available for assessment. We refer the reviewer to Figure 1 in the Rebuttal PDF providing a toy example where the reference-based metrics did not perform satisfactorily, while the reference-free FKEA-Vendi and FKEA-RKE resulted in the proper ranking of the generative models.

[1] Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. In Transactions on Machine Learning Research, 2023.

[2] Mohammad Jalali, Cheuk Ting Li, and Farzan Farnia. An information-theoretic evaluation of generative models in learning multi-modal distributions. In Advances in Neural Information Processing Systems 2023.

[3] Amey Pasarkar and Adji Bousso Dieng. Cousins of the vendi score: A family of similarity-based diversity metrics for science and machine learning. In International Conference on Artificial Intelligence and Statistics 2024.