A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models

We were able to derive the bias-variance-covariance decomposition by using a quadratic form in the definition of kernel scores. This quadratic form is in general not present for other losses, which prohibits deriving a covariance term.

Kernel scores are loss functions giving good values when the prediction is close to the ground truth and bad values when not. In Figure 4 and 6, the Pearson correlation between the kernel entropy (our uncertainty measure) and the loss (kernel score) shows that kernel entropy is very effective in predicting the loss. The nlg results in Figure 7 confirm that this holds for the loss used to evaluate question answering datasets. Part of the explanation for our good results is that the embedder transforms the generated answers into a semantically meaningful vector space, in which the kernel then compares the similarities. Vector embeddings combined with the cosine similarity have a long history in the NLP domain, which underlines their effectiveness (Camacho-Collados et al, 2018).

References:

Jose Camacho-Collados and Mohammad Taher Pilehvar. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63:743–788, 2018.

Dear reviewer ox7H,

Thank you very much for taking the time to engage with our paper. Below, we have tried to address all of your feedback and questions. Specifically, we give reasoning for our strong performance in natural language generation and explain why a covariance term cannot be derived for other losses. Please take a look and let us know in case you would like additional clarification on any of these points.

• Q: “Practitioners may find it challenging to apply this method to their case.”

A: We added a description of our procedure to compute the kernel entropy in Algorithm 1 Appendix C.1. Note that our final approach is simpler to implement than semantic entropy introduced by Kuhn et al (2023).

• Q: “In figure 4, what are the points being plotted? The 20 different models?”

A: Each point in Figure 4 corresponds to a digit.

• Q: “Why is only a single model used to compute kernel entropy? Are there any advantages of using multiple models to compute the entropy?”

A: We compute the kernel entropy based on single models to stay as truthful as possible to practical constraints: Generative models are usually computationally very expensive and closed-source access is almost always provided only to single models. It is likely that the average kernel entropy of multiple models gives even better results because this improves the estimation.

• Q: “How is the bias calculated in the mode-collapse experiment of section 5.1? Is the original distribution used to compute the bias, or the modified distribution with reduced frequency of class 0 used?”

A: The MMD, bias, and variance values in Figure 5 are only computed with respect to digit ‘0’ (which has reduced frequency in the training data). The other digits in the mode-collapse experiment have similar lines as in Figure 3 (no reduced frequency in the training data).

References:

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.

The discrepancy of the kernel entropy behavior between the image and audio experiments can be explained via the architecture and task setup. For image generation, the initial diffusion model already generates volatile and noisy samples. This leads to a roughly constant kernel entropy throughout training. In contrast, the audio model initially generates very short audio instances (c.f. Figure 10 Appendix C.2.2). This means that most of the entries in the audio vector are zeros, which results in a small kernel entropy. Consequently, the kernel entropy increases as soon as the model learns to produce longer audio instances.

We compare the AUROC of the kernel entropy for various choices of kernels in Figure 14 in Appendix C.2.3. The differences between RBF and polynomial kernel as well as cosine similarity are marginal, while the Laplacian kernel performs worse. This is in line with common practice in NLP, where cosine similarity is the most often used kernel to compare text embeddings (Camacho-Collados & Pilehvar, 2018).

References:

Jose Camacho-Collados and Mohammad Taher Pilehvar. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63:743–788, 2018.

The RBF kernel is a common kernel for low-to-medium dimensional data, such as low resolution images like MNIST (Schölkopf, 1997). However, the RBF kernel does not scale well to higher dimensions (Binkowski et al, 2018). This is problematic since audio instances are represented by vectors of length 100,000 or more in our experiments (c.f. Figure 10 Appendix C.2.2). Consequently, we use the Laplacian kernel in the main paper but also add an evaluation with the RBF kernel in Figure 11 in Appendix C.2.2. As expected, the results are more erratic compared to the Laplacian kernel but show a similar trend. Specifically, the kernel score and kernel entropy increase and decrease at the same training iterations, however the absolute Pearson correlation between them is only between 0.8 and 0.9.

References:

Bernhard Schölkopf. Support vector learning. PhD thesis, Oldenbourg München, Germany, 1997.

Mikołaj Bínkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.

Dear reviewer 27VK,

We appreciate your thorough review of our paper. In response to your feedback and questions, we have made efforts to address them comprehensively below. Specifically, we update Appendix C.2 with evaluations of the RBF kernel for the audio experiments and the Laplacian and polynomial kernel for the language experiments. We discuss the results in the following and also explain why kernel entropy stays constant in the image experiments but increases in the audio experiments. Last, we give answers to the other miscellaneous questions. If you require further clarification on any specific points, please feel free to let us know.

In general, our theoretical results and Figure 2 indicate that more samples are always better. However, at some point the improvements are too marginal to justify the computational costs. To highlight this point, we plot the AUROC of the kernel entropy in case of Opt-13b and CoQA for different sample sizes (=number of text generations) in Figure 15 in Appendix C.2.3. As can be seen, more samples are always better, but the improvement flattens for larger sizes. In the main paper in Figure 7, we use 20 generations for CoQA and 10 generations for TriviaQA, which follows Kuhn et al (2023).

References:

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.

Dear reviewer vzDu,

We appreciate your thorough review of our paper. We have carefully addressed all the feedback and queries you provided below. Specifically, we discuss the effect of different sample sizes and the benefits of an explicit covariance term in the bias-variance decomposition. Feel free to reach out if you need further clarification on any of these matters.

In previous work, the bias-variance decomposition is often used to motivate ensemble approaches. Specifically, it is assumed that the individual ensemble members are independent, which then reduces the variance term and also the generalization error. However, it is not clear how negative the effect on the generalization error is when this assumption is violated. Our variance-covariance decomposition allows us to quantify the negative impact of correlated ensemble members via the covariance term. In Figure 3 we use the normalized covariances (=correlations) for better interpretability. As we can see, the correlations between epochs 20 to 40 are very high, which means that creating an ensemble based on iterative gradient steps, like stochastic weight averaging Gaussian (Maddox et al, 2019), strongly violates the independence assumption. Specifically, we can now quantify how much this correlation/covariance impacts the generalization error. This underlines the practical relevance of our theory.

References:

Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., & Wilson, A. G. (2019). A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems, 32.

We now clarify how the experiments for image and audio are connected to the natural language generation (nlg) ones in terms of uncertainty estimation. In general, a meaningful measure of uncertainty is able to predict the loss for a given prediction. The literature formulates nlg uncertainty estimation as a binary classification problem with a binary loss (Kadavath et al, 2022; Kuhn et al, 2023), i.e. a generation is either correct or wrong. Consequently, it makes sense to use the AUROC as a summary measure of how well an uncertainty measure predicts the correctness (i.e. is correlated to the loss). For image and audio generation we use a kernel-based loss function (kernel score), which has a continuous range of how good a prediction is. Consequently, we cannot use the AUROC anymore. Instead, we use the Pearson correlation (range between -1 and 1) to assess how well an uncertainty measure can predict the loss. Absolute values close to 1 indicate a very good uncertainty estimate. In Figure 4 and 6, we can see that kernel entropy is very predictive of the loss (absolut Pearson correlation >0.9), making it an excellent measure of uncertainty in these cases. The nlg experiments in Figure 7 confirm that kernel entropy also gives state-of-the-art results for the binary loss function proposed by Kuhn et al (2023).

References:

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.

We agree with the reviewer that the provided evidence is not sufficient to back our strong claims about mode collapse. We now soften our claims/formulations and clarify that our bias-variance-covariance decomposition can be used as a tool to diagnose mode collapse or overfitting and elucidate its nature in terms of bias and variance. We now also provide additional experiments to showcase the practical utility and repeat the same mode-collapse experiment of Figure 5 with other digits reduced (digit ‘2’ and ‘3’, c.f. Figure 9 Appendix C.2.1). Based on our bias-variance decomposition, we discovered that mode collapse does not appear in these cases. However, the lack of training data is still expressed in an increased bias term. Consequently, we are still convinced that our decomposition is a useful tool to gain insights into the fitting behavior of generative models.

Dear reviewer XNge,

Thank you for your time and attention. We appreciate your thorough review of our paper. We have carefully considered all your feedback and questions in the following responses. Specifically, we communicate how we adjust our claims about mode collapse and how image, audio, and language experiments are connected with uncertainty estimation. If you need further clarification on any of these points, please feel free to let us know.