SelfEval: Leveraging the discriminative nature of generative models for evaluation

We thank the reviewer for their valuable time and effort. We resolve some confusions and address your concerns below.

Clarification about SelfEval

• We wish to clear the confusion of the reviewer regarding their statement that a stronger text-to-image model can be used to evaluate other models. SelfEval eliminates the need for any external model by converting a generative model ($p(x,y)$) to a discriminative model ($p(y|x)$) and evaluate the generative capabilities using its discriminative performance (proportionality via Bayes’ Rule). We DO NOT use one text-to-image model to evaluate others in this work.

Weakness

• We point out some important distinctions of our work from CLIP R-Precision. CLIP R-precision measures the text retrieval performance given a generated image whereas in our case, we use the ground truth image-text pairs for evaluation. Unlike such embedding-based metrics, our evaluation is absolute, since models are always evaluated against ground truth. This along with the splits introduced in our work, provide a diagnostic tool for analyzing generative models. Another key takeaway and novelty of SelfEval is the fact that a generative model can be used to evaluate itself. To the best of our knowledge, this capability has never been shown before and differentiates our work from other evaluation metrics proposed in literature.

• Limitations of CLIP were highlighted in MID [2] but their analysis is limited and doesn’t show clearly how this affects the final evaluation. In our work, we show the limitations of CLIP’s text understanding and how using different CLIP models affects the actual evaluation.

• We believe [4] was made public AFTER the ICLR paper submission deadline. Except method [2], none of the other works are published in literature. Moreover, methods [1-3] use external models (like CLIP) for evaluation and as mentioned throughout our paper, using external models requires additional error analysis to prevent any biases. With SelfEval, we aim to eliminate any external model for evaluating text faithfulness of generative models and we believe it is unfair to compare SelfEval against these methods. We will add a separate section in the related works to discuss these methods.

Thanks for the clarification, but I mean that's a strong point in your work since you repurpose the text-to-image generative model to a discriminative model for evaluating its generative capabilities; if you're targeting the stronger text-to-image generative model ($p(x,y)$), you can have more strong evaluation quality by assuming that its discriminative ($p(y|x)$) performance is stronger.

I acknowledged that the related works [1,3,4] have not been published yet, although LLMScore [3] will be published in NeurIPS 2023, and the ICLR 2024 reviewing guidelines have loose rules for it. Still, you could compare your work with MID [2] along with CLIP, and the failure of this undermined the comparative effectiveness of your work. Nevertheless, I highly recommend faithfully comparing with those related works.

I appreciate your endeavor to provide the requested comparison and acknowledge that MID requires a sufficient number of comparative samples like FID.

• I need more elaboration on the details of the experiments.

  • "In both the tables, for MID, lower score is better" -> for the MID score, the higher wasn't better?
  • "SelfEval is the accuracy over chance" -> how do you get the accuracy? What brings you to this conclusion?
  • I only understand that Humans A/B in the tables means that the number of humans voting for the generations from the "A" model is A, and for the generations from "B" is B.

• Do you assume that since B is more voted than A by humans, the evaluation metrics should vote for B? A or B cannot give consistent generation performance for every sample, so you should consider the sample-wise preferences of humans. For this reason, many related works resort to Kendal-tau correlations to see if a model follows human judgments for the samples. Since the reports missed key details and ambiguity, my interpretations are limited. Please correct me if there's a misunderstanding.

In Tbl. 3 of the MID paper, they compared in accuracy. In a similar way, you could get the accuracy using MID.

As Sec. 3.3 in this manuscript stated that you measured the accuracy by picking the correct caption for a single image $x$ from a set of captions $\\{c_i\\}$, you could measure the MID scores between the single image $x$ and each one of the captions $\\{c_i\\}$. Then, you can pick the highest MID score among them as the prediction and measure the accuracy. Notice that MID can give you a pair-wise score like the CLIP-score.

In this way, CLIP and MID scores can be converted to the accuracy having the interpretability you argued.

Since the accuracy metric is based on a set of probing captions, it could be sensitive toward the choice of candidate captions inevitably. Notice that the accuracy also has weaknesses.

• Your argument that Likert scale-type ratings can be subject to bias is reasonable, although a sufficient number of samples would show better human judgment correlations for the better model.

• I agree with the authors that the proposed model has advantages over previous works, including MID, since the evaluation is not biased from the other pretrained models when there's no significant bias in the generative model itself.

• Unfortunately, I cannot see that the proposed method outperforms or is on par with MID. In the above Tbl 1 & 2, if I'm correctly interpreting, both MID and SelfEval roughly follow human voting tendencies. However, since MID and SelfEval's metrics are different, the MID score and the Accuracy Over Chance, respectively, we cannot compare directly. Couldn't you convert MID scores to the Accuracy Over Chance compatibly? Please don't mix up different metrics in one table. Because the authors argued that MID is subject to the pretrained model's bias, you can easily show that SelfEval may outperform MID, but the current experiment cannot show it, making the work incomplete.

• After discussion, I cannot help to raise my score but remain leaning toward weak rejection due to the above concern.

We thank the reviewer for recognizing the simplicity and effectiveness of SelfEval for evaluating text faithfulness of generative models. Please find our responses to your concerns and questions below.

Weaknesses

• Using Eq. 4-9 (in the main paper) one can compute the likelihood of the data $x$ given conditioning $c$. Throughout the derivation we do not make any assumptions on the shape/format of the conditioning variable $c$. We chose to proceed with text-to-image generation because evaluating the performance of the model on the image-text matching task is correlated with its generation capabilities. If the reviewer has specific conditioning variables or downstream tasks in mind, please let us know and we can provide a specific and detailed answer.

• We believe that proper evaluation metrics are necessary to make progress in any field. Given the exponential progress in generative modeling, there is a need for developing strong evaluation metrics. We argue that research on evaluation metrics is not narrow but rather necessary to propel the advancement of generative models. Moreover, [1] proposed CLIP score, widely used for evaluating text faithfulness while [2] proposed FVD a metric to measure quality of generated videos, both of which are widely used by the generative community making such research very valuable.

• We re-iterate that the contribution of our work is broadly two fold. 1) Generative models can themselves be used to evaluate their generations and 2) a principled approach to achieving this with text-to-image diffusion models and repurposing discriminative ground truth image-text pairs. We believe that 1) is an important contribution with far reaching consequences and is not specific to diffusion models. Generative models can inherently be used to evaluate themselves using Bayes’ Rule and this applies to any future developments in this area. We agree with the reviewer that showing results on additional models is beneficial but after an extensive review we fail to obtain any text-to-image GAN or VAE models which are publicly available making it impossible to evaluate using this approach (see common response above). Instead we chose four popular models, with enough details in their respective papers for re-implementation, from two families of diffusion models to validate our hypothesis. We urge the reviewer to provide specific details (papers, codes, checkpoints) of good text-to-image generative models we can include in our analysis.

[1] Jack Hessel et al. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. ”EMNLP (2021)”.

[2] Unterthiner, Thomas et al. “FVD: A new Metric for Video Generation.” DGS@ICLR (2019)

We thank the reviewer for recognizing SelfEval’s ability to eliminate the use of external models and its strong agreement with human evaluations. We will address your concerns and questions below.

Weaknesses

• We agree with the reviewer that the limitations of the evaluating model affects the performance of the evaluation itself but as repeatedly mentioned throughout the paper, this is an issue only when an external model is used. SelfEval uses the generative model itself for evaluation. For example, if the generative model is weak in “counting”, then as shown in the paper, the scores for this task are lower. We show that this is the case with four popular text-to-image models and request the reviewer to provide the details (paper, code and checkpoints) of other generative models they deem useful.

• SelfEval relies on the sampling of the generative model to compute the scores. So the limitations of the sampling process of a generative model affect SelfEval. Unlike other black-box evaluation methods which only require the generations from the model, SelfEval requires the model definition and its checkpoints for evaluation. We can provide more details if necessary and will add this discussion to the next draft.

Questions

• SelfEval is evaluated using two different families of diffusion models, pixel and latent and across 4 different datasets and shows strong agreement with human evaluators.
• Any likelihood-based generative model (i.e. ones that model the data distribution $p_{\text{data}}(x)$ , like VAEs) can be used with SelfEval. For implicit generative models, like GANs, methods like GAN inversion [1] can be used to obtain likelihood scores from them as described in [2]. After an extensive survey of existing methods, we failed to obtain public repositories of text-to-image GANs or VAEs with checkpoints. We request the reviewer to point us to relevant text-to-image generative models (With code and checkpoint) for further experiments.
• We request the reviewer to clarify what additional analyses/experiments they consider valuable to this work.
• We show the effect of different text encoders in Sec. 4.2 of the main paper.

[1] W. Xia, Y. Zhang, Y. Yang, J. -H. Xue, B. Zhou and M. -H. Yang, "GAN Inversion: A Survey," in IEEE T- PAMI, 2023.

[2] Eghbal-zadeh, Hamid and Gerhard Widmer. “Likelihood Estimation for Generative Adversarial Networks.” 2017.

Dear Reviewer, We perform additional experiments comparing SelfEval with MID[1] as suggested by Reviewer M8yh. MID uses a CLIP model to compute mutual information divergence and we observe that it suffers from the same drawbacks as CLIP-Score (Fig. 2 in the main paper). We show the results in Table-1 below. The second and third column in Table-1 compute MID using ViT-B/32 and ViT-L/14 backbones respectively. Similar to CLIP score, depending on the evaluation model used, LDM-CLIP (with ViT-L/14 text encoder) ranks higher/lower than LDM-CLIP (with ViT-H/14 text encoder). We believe this issue exists regardless of the metric as long as it relies on external models. For more details refer to the review posted to Reviewer M8yh.

Table-1: Drawback of MID

Please take a look at our posted rebuttal and additional discussions and provide us with more details. We believe this discussion can convince the reviewer to improve their score.

[1] Kim, J.-H. et al. (2022). Mutual Information Divergence: A Unified Metric for Multimodal Generative Models. Advances in Neural Information Processing Systems 35.