Uncertainty Quantification with Generative-Semantic Entropy Estimation for Large Language Models

We thank the reviewer for their detailed and comprehensive feedback. In reply:

1. Yes, this is a fair distinction. Our comment was meant to imply that the GSEE algorithm is, in point of fact, independent of the latent embedding locus (as any model embedding can be used in principle) — however, the quality of the UQ will be sensitive to the choice of model and locus, naturally.
2. Here we simply meant to indicate by a nominal “forward pass”, i.e., with respect to an <end> token, multiple generations can be rendered; we address this computational cost in reviewer comments above. Thank you for your comment, we agree that this idea should be stated more clearly.
3. By “mathematically-principled” we underscore that GSEE corresponds with the effective rank of the covariance matrix of the generated responses, a well-established statistical measure of uncertainty.
4. Fair point, however, there are extant, publicly-available replications of GPT-3 + openAI tokenizers used in the HF community.
5. Because semantic similarity is inherently difficult to measure consistently, yielding misleading conclusions regarding UQ specifically, we provide prompts, “bad” ones, that constrain the problem domain into objectively similar/dissimilar semantics settings, which yields a qualitatively different measure of UQ “correctness”. Using PCC in the latter setting would reintroduce the reliance upon noisy semantic similarity measures, e.g., Rouge-L and perplexity, so we instead opted to demonstrate the quality of good/bad prompting UQ cluster separation.
6. Thank you for the question. We use the average of the generated responses to evaluate correctness; we will make this point clearer in the final version of the paper.
7. Thank you for this important comment. The ability of LLMs to capture latent semantics is, to the authors’ knowledge, well-established, please see: Tennenholtz et al., “Demystifying Embedding Spaces using Large Language Models”, ICLR 2024.
8. We did not include these results in the table, as we felt they were provided few insights, as indicated by the attendant comment in the paper. Nevertheless, we are happy to include them in the final version of the paper.
9. This is a reasonable point. We did not generate non-length normalized results, but this is certainly something that we could include in a supplemental section of the paper.
10. Yes, the eigenvalues are normalized to yield a proper probability distribution; the entropy is calculated with respect to the full matrix.
11. We divide the entropy to reduce length sensitivity, but the reviewer makes a cogent point that this step introduces length information; a viable alternative could be to apply GSEE to processed outputs (e.g., reduced verbosity) and to subsequently remove the averaging according to mean length.
12. We will make a concerted effort to improve the streaming of word choice, thank you for the suggestion.

We greatly appreciate the reviewer’s feedback. In reply to the stated weaknesses:

1. Ideally, the GSEE-based UQ could be leveraged to help detect and mitigate hallucinations, as you mention, or to provide better calibrated model confidence estimates, or to augment RAG and other prompting techniques, or to provide a guidance in continual learning paradigms to address knowledge gaps in the LLM, etc. Our current experiments reflect strong correlations with GSEE-based UQ and generative “correctness” (measured with respect to Rouge-L and Perplexity metrics), in addition, we devise a less metric sensitive experimental setting using in-distribution vs. OOD prompts to further validate the effectiveness of GSEE as a bellweather for correct predictions. We believe that GSEE can be leveraged additionally to optimize prompt engineering (e.g., design prompts to minimize uncertainty) to further improve model prediction, but we leave this exploration to future work.
2. Thank you for the question. In fact, we compare lengthy-normalized entropy as introduced in Kuhn et al., in our experimental baseline comparisons.
3. Yes, GSEE can be sensitive to the number of generated responses and temperature; we explore this sensitivity in several of our ablation experiments, please see: Table 6.
4. Thank you for the comment. While matrix entropy has been introduced in previous work to assess dataset complexity, to the best of our knowledge, it has never been previously applied for LLM and related model UQ purposes.

Thank you kindly to the reviewer. In reply to the stated weaknesses:

1. We agree with the reviewer that the uncertainty quantification can be critically dependent upon the manner and locus of latent embedding extraction. For our experiments, we chose to follow research “best practices” (as referenced) by averaging token likelihoods based on the penultimate layer model embeddings. In order to achieve an apples-to-apples comparison with other baseline UQ methods, we used this same embedding extraction method across all relevant methods. While we agree that an ablation study with regard to embedding extraction methods can provide some additional insights — particularly for determining a best extraction point for a given LLM/LMM — we nevertheless felt that this exploration was orthogonal to directly validating the effectiveness of GSEE vis a vis other LLM UQ methods, as the choice of embedding extraction locus is more suited as an application decision.
2. Indeed, this is a fine point. While creating multiple generations (M) for each model input can be parallelized through a single forward pass, the memory cost can be significant, particularly as GSEE is a white-box UQ method. This memory cost scales effectively at O(M); however, in practice, we found that using a reasonably small generation size, e.g., M=5, 10, produced favorable UQ results, which can be accommodated in many cases by a single GPU for many SoTA LLM/LMM models. With regard to covariance calculations, the reviewer’s concern is also well-founded, as SVD complexity is generally O(M^3); nonetheless, in our case, as the generation size is reasonably small and the covariance matrix calculations are with respect to MxM matrices, so we found this calculation to be close to negligible in comparison to model inference cost.
3. Please alert us to how we can make the presentation clearer, thank you.

We thank the reviewer for their valuable feedback. In reply to the stated weakness: more formally, we wish to estimate the uncertainty of the posterior distribution p(y|x,t), where x denotes the input data (e.g., image+prompt), t is the model temperature, and y is the generated output. Effectively, each generated output is a sample drawn from p(y|x,t), and thus the dispersion of this sample (in our algorithm measured with matrix entropy) corresponds with predictive uncertainty; in general, this framework for modeling predictive uncertainty is well-established in literature, including Bayesian Neural Networks, see:

Andrew Gordon Wilson and Pavel Izmailov. 2020. Bayesian deep learning and a probabilistic perspective of generalization. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS '20). Curran Associates Inc., Red Hook, NY, USA, Article 394, 4697–4708.