Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

Please start by reading the first part of our rebuttal to your review.

The performance of SEPs is unsatisfactory, and the author should include more competitive baseline methods. According to Fig. 7, SEPs fall short by 5-10 points compared to SE. Furthermore, SE is not the state-of-the-art (SOTA) method for hallucination detection [2,5]. It is recommended that the author include comparisons with the SOTA methods. I acknowledge that SEPs aim to balance performance and computational efficiency, I would suggest the authors to discuss this trade-off more explicitly in relation to SOTA methods.

Thank you for acknowledging that SEPs present a ‘best-in-class’ performance given their low computational cost. We would like to again emphasize the dramatic difference between SEPs and SE in terms of their test-time cost to quantify uncertainty. The cost of computing SEPs is basically negligible: some back-of-the-envelope math shows that SEPs add only about 0.0000002% in terms of FLOPS compared to the cost of an average generation for Llama-2-70B. This is compared to full SE, which increases test-time generation costs by a factor of K, with K=10 typical, and further require significant computational overhead to run the semantic clustering algorithm.

We are further unsure how [2,5] show that SE is not a SOTA method. [5] targets long paragraph-length generations with multiple facts and focuses on how to decompose uncertainty across facts. As such, it does not compare to SE. Further [2], also does not compare to SE. Also note that [2] seems to only study very short generations, see their examples in Appendix H, which is significantly less complex than the full sentence-length generations that we study following Farquhar et al.

We would agree that the field of hallucination detection is currently lacking standardized definitions, experiment scenarios, and comprehensive benchmarks of methods. To us, it is unclear which of these methods is best at extracting model uncertainty! We think this is a problem in the field, and we would welcome future work that clearly compares all of the existing methods. We further think that using other types of uncertainty metrics as probing targets is exciting future work. In Appendix A (Figure A.11), we already present some interesting results when using the p(True) uncertainty method as a supervisory signal.

Note that we already clearly state that SEPs do not match the performance of full SE in lines 259 and 465. However, following your feedback, we have added a limitations paragraph to section 8, where we make clear again that SEPs cannot match the performance of full SE.

Please read the third part of our rebuttal to your review next.

Please start by reading the first three parts of our rebuttal to your review.

α represents the optimal threshold for partitioning raw SE scores into high categories, and I'm interested in how this threshold varies across different datasets.

Following your question, we have updated the draft to show the best-split objective for Llama-2-7B across tasks in Appendix A (‘Different Binarization Procedures’, Figure A.15). We find that the objective behaves quite similarly across tasks. This complements our previous results in Figure A.14 which show the behavior of the global (across tasks) best-split objective for all models. Note that, for all our experiments, we rely on a single global threshold per model that is fixed across tasks.

Additionally, I'm uncertain about the generalizability of trained SEPs across these datasets.

Could you perhaps clarify what we can do to alleviate your concerns about the generalizability? We feel that Figures 7 and 8, as well as Table 2, show clearly that SEPs generalize significantly better to new tasks than accuracy probes.

Would using hidden state information to predict SelfCheckGPT scores [1], Eigenscore [2], or other uncertainty scores potentially lead to better results?

We think that using other uncertainty metrics, such as those proposed in [1] or [2], to supervise linear probes is a very interesting direction for future research. On a more abstract level, we suspect that LLM latent spaces generally capture uncertainty over semantic meaning. We do not know which uncertainty method, be it SE, [1], [2], or something else, best captures this underlying ‘ground-truth’ semantic uncertainty. Therefore, we think it is highly interesting future work to supervise linear probes with other uncertainty methods. In fact, in Appendix A (Figure A.11), we already present some interesting preliminary results when using the p(True) uncertainty method as a supervisory signal.

Again, thank you very much for your review! Please let us know if you have any remaining questions or concerns and we will do our best to respond to them as soon as possible.

Please start by reading the first two parts of our rebuttal to your review.

There seems to be no clear explanation for why predicting SE yields better performance compared to predicting labels. I would appreciate a more detailed analysis explaining why predicting SE outperforms label prediction.

We agree this is a really interesting question that we could have spent more time on in the paper. Following your feedback we have reworked and improved our discussion in Section 8 of why SEPs generalize better to new tasks than accuracy probes and present a more extensive version of this for your convenience below.

Our intuition here is that accuracy probes are more likely to latch on to spurious correlations between model predictions and accuracy in the dataset that will not generalize to new datasets, whereas SEPs will learn to robustly predict SE, which predicts accuracy well across tasks.

More concretely, we believe that largely unavoidable problems with assigning accuracy labels to ultimately vague linguistic statements are to blame for this behavior. While semantic uncertainty should generally be a good predictor of model accuracy, there may be some cases where uncertainty is low despite the model answer being judged as incorrect. (This mostly happens due to shortcomings in measuring accuracy, e.g. the model answer not being precise enough, for example, just giving the year when the gold answer is looking for an exact date.) We believe these cases may lead to accuracy probes learning worse predictors than SE probes, and that this becomes noticeable on OOD data.

In this scenario, for the SE probe, everything works according to plan: the label will be ‘high semantic uncertainty’, which we assume can be robustly predicted from latent space as we have discussed above. So if the probe is trying to find the direction in latent space that is predictive of ‘high semantic uncertainty’, the information it is getting with this datapoint agrees with previous observations. We further know that SE works robustly across datasets, so if the probe finds a direction in latent space that robustly represents SE, it will generalize well.

For the accuracy probe, however, this datapoint is much more problematic. For it, the ‘SE uncertainty signal’ in latent space is not helpful for predicting the accuracy label anymore. Therefore, the accuracy probe is much less likely to learn to rely on the ‘SE uncertainty’ direction in latent space. Instead then, we believe that the accuracy probe likely relies on other features of the question embedding to achieve good predictive performance. Given the complexity of the hidden space, the accuracy probe may rely on features that identify questions the model cannot answer (e.g. by their topic, such as ‘historical facts’) instead of using the SE direction. However, these features would often be brittle and not generalize well to new datasets, as they conflate model capability with the labeling process. For example, we often seen models answering at the wrong level of granularity or missing implicit assumptions about for a particular subset of questions. For the OOD dataset, answer granularity and implicit assumptions may be different, such that these features will not generalize!

(For example, on dataset A, the accuracy probe may learn that the LLM cannot answer historical questions, because the LLM always responds only with a year, when the gold answer is a full date. The accuracy probe will then learn to associate ‘historical’ questions with low accuracy – it will not rely on the SE signal as the model is confident in its answers here. However, on dataset B, the gold answer is only years and not full dates on similar historical questions. This will lead to failure of the accuracy probe. However, SEPs will work well on dataset B as they are much more likely to capture the underlying SE signal.)

In summary, our results support the idea that semantic uncertainty can be captured robustly from latent space, with SEPs generalizing similarly to SE. Accuracy on the other hand depends on an ‘external’ labeling process, which the hidden states do not capture. Therefore, probe learning from accuracy signals is subject to conflicting signals and spurious correlations, which lead to worse performance in generalization regimes.

Please read the fourth part of our rebuttal to your review next.

Dear reviewer eRJ8,

Thank you for your hard work and helpful feedback. We have gladly incorporated many of your excellent suggestions into the draft and look forward to engaging with you during the discussion period to clarify any remaining points. Please read our comment above, addressed to all reviewers, first.

The related work section lacks several crucial references [1,2,3,4]. For example, ref [1] also adopt a supervised method for uncertainty estimation of LLMs. ref [2,3] explores the internal states of LLMs for hallucination detection. [...] Therefore, I suggest the authors to add a paragraph discussing internal state-based and probe-based hallucination detection methods, highlighting key differences between those approaches and SEPs

Thank you for suggesting these related papers. We have updated the draft to add a discussion of them in relation to our work, and have added a slightly more extensive discussion below for your convenience.

In [1], the authors train probes supervised by accuracy metrics, and we will cite them as another example of an accuracy-based probing method. They use random forests instead of linear models. We have investigated non-linear probing methods in Appendix A, but find no benefit from them. Further, the paper sometimes uses a different model for probing than is used to generate the answer, which we have certain objections with. Concerningly, this seems to rely entirely on knowledge being highly correlated between different models. This has also been highlighted in concurrent reviews of this work.

In [2], the authors predict hallucinations by computing a so-called eigenscore from the latent states of multiple model generations. In other words, like full semantic entropy, [2] requires the sampling of multiple model generations. Therefore, this method does not call for direct comparison to SEPs given it incurs much higher computational costs. (It would be interesting to compare this method to SE though, and then perhaps try to approximate [2] from single generations similar to our work here.)

[3] is another example of an accuracy-based probing method and we will cite them as such. They also present interesting experiments on predicting training set membership from latent states. We don’t think their probing method contains any methodological changes that would require us adding to/changing our accuracy probing baseline.

[4] targets word-level factuality. It is unclear how to apply their method to the task of assigning overall factuality scores in the open-ended question-answering settings that we study in this paper. However, we think it would be interesting for future work to extend SEPs to word-level factuality and then to compare against [4].

Please read the second part of our rebuttal to your review next.

Please start by reading the first part of our rebuttal to your review.

What is the actual potential time efficiency benefit of the method as compared to SE, for example, as measured in in wall clock time? And how much would the possibility to generate samples in batch-mode affect those numbers?

Following your feedback, we have collected the actual time difference between computing SE and SEPs for Llama-2-7B on TriviaQA. We find that, on average, computing SE for 1,000 samples takes 168 $\pm$ 19.11 minutes. In contrast, the computation of SEP predictions requires only 7.13 $\pm$ 0.14 seconds in the same setting. Consequently, SEPs are able to speed up hallucination detection by 1413-fold compared to full SE. We have added these results to Appendix A.

Note that, in terms of pure FLOPS, SEPs are much more than 1400 times cheaper than SE. Some back-of-the envelope math suggests that SEPs add only about 0.0000002% in terms of FLOPS over naive generation, suggesting our current wall-clock speedup could be improved significantly.

As for parallelizing over batches, SEPs are just linear layers and so they can be computed in parallel without any problem. For SE, while the additional generations can be performed in parallel, the semantic clustering algorithm and NLI computations are run sequentially (iteratively over the set of generations for each question).

The proposed method is very simple and seems somewhat "obvious". I'm a bit surprised this hasn't been studied before, but not being an expert in this work I give the paper the benefit of the doubt and consider this simplicity a plus.

Great! We agree that the method is simple and would also consider this a strength. We also think that some of the ideas around SEPs generalizing better than accuracy probes are perhaps less obvious and may explain why SEPs have not been considered before.

Again, thank you very much for your review! Please let us know if you have any remaining questions or concerns and we will do our best to respond to them as soon as possible.

Dear reviewer BxzX,

Thank you for your hard work and helpful feedback. We have gladly incorporated many of your excellent suggestions into the draft and look forward to engaging with you during the discussion period to clarify any remaining points. Please read our comment above, addressed to all reviewers, first.

The accuracy of the semantic entropy prediction is quite significantly behind the sampling-based estimate.

We agree that SEPs’ performance lags behind SE and other sampling-based approaches. We believe that closing this gap is exciting future work. For now, SEPs offer a “best-in-class” performance considering their extremely low computational cost.

In addition, it seems hard to quantify how much prediction accuracy for hallucination detection would actually impact any practical applications. Since there are many possible avenues to improve accuracy, some investigation in this direction - even just in a preliminary fashion - would make this work stronger.

This is a great suggestion! We have added accuracy rejection curves to Appendix A (Figure A.16). These curves show that SEPs can improve the overall accuracy of the model by refusing to predict on those inputs where SEPs predict the highest uncertainty. This is only possible because high a high predictive value from SEPs correlates well with model accuracy, such that low uncertainty inputs (as judged by SEPs) are less likely to be correct. We believe that such rejection accuracy curves are a great way to showcase the positive practical impact of SEPs and plan to add a more extended showcase for the camera ready version of the paper.

Would it make sense to use significantly higher N during training as a way to improve SEP accuracy?

We here follow Farquhar et al., who typically set N=10, which robustly gives good performance across tasks in their experiments. They also provide an ablation which shows that SE performance does not significantly improve for N larger than 10. As SEPs use only the SE value as a supervisory signal, we don’t expect them to benefit from significantly larger N, as such increases in N don’t benefit SE itself either.

In this work, probes are applied only to separate layers. Have the authors considered combinations of multiple layers? This would make sense if one assumes that different layers may contribute different evidence for model uncertainty.

Yes, we have. In fact, some of the results in the current version of the draft do concatenate hidden states across layers for probe training. We detail this in Appendix B.4 (‘layer concatenation’) and have revised the main version of the draft to point more clearly to the appendix here. We generally find that layer concatenation helps moderately improve performance. Following your feedback we have added a direct comparison of results with and without layer concatenation in Figure B.3.

How OOD is the OOD setting really? How (dis-)similar are the different datasets from one another?

That’s a great question! All the datasets we consider are free-form question-answering datasets. TriviaQA has trivia questions, NQOpen real-world questions people asked Google, SQuAD is a reading comprehension task (but its questions can usually be answered without the context), and BioASQ targets medical and biological domain knowledge. Although this is difficult to quantify, we would say that, from a human perspective, TriviaQA, NQOpen, and SQuAD are all quite similar in terms of the type of questions that they contain and that BioASQ is the most unique. In a sense, it is therefore even more surprising that accuracy probes can struggle significantly to transfer between these datasets.

I assume the values in Figure 2 are for held-out not training data. Is that correct?

Correct! This is data that was not used to train the probes in any way.

Please read the second part of our rebuttal to your review next.

Please start by reading the first part of our rebuttal to your review.

Can authors further discuss why "the discrete variant of SE" should be used instead of the plain version Eq (1)? How is the "fraction of generations in the cluster" related to "avoid problems for generation of different length" (L193)?

Both the discrete and plain version of SE could be used with SEPs. We simply prefer the discrete version following the recent publication of Farquhar et al., who find it yields comparable performance while being simpler conceptually and allowing application to models where tokens probabilities are not accessible.

The discrete SE variant uses a simple counting measure to estimate the probability of a particular semantic meaning (e.g. the probability of semantic meaning A is given by the fraction of generations the clustering algorithm associates with that meaning). It does not use the token probabilities of the generated sequences.

This is in contrast to the original SE variant introduced by Kuhn et al. which does rely on the token probabilities to estimate SE. A problem that comes up when doing this naively is that longer sequences will typically have lower probability because the probability of a sequence is a product of the conditional probabilities (<=1) of the individual tokens. Empirically, what works well is to compute the average of the log token probabilities instead of their sum, i.e. divide the sum of log probabilities by the length of the generation (see citations at the end of Section 3). However, this is wrong from a probabilistic perspective as we are no longer working with the true joint log probability of the sequence. The discrete SE variant allows us to estimate semantic cluster probabilities without using token sequence probabilities. Thus we can avoid making a difficult choice regarding length-normalization of log sequence probabilities.

In Sec 5, it seems that the prompt "single brief but complete sentence" also promotes short responses. Does a Chain-of-Thought style prompt work for the long-form answer generation?

Thank you for this suggestion! Following your feedback, we have prompted Llama-3 70B to ‘think step by step’ when giving answers on TriviaQA. Firstly, we have found that CoT prompting does not significantly change accuracy on this task ($0.885$ before vs $0.888$ after). Interestingly, however, the in-distribution performance of SEPs actually seems to improve with CoT prompting, whereas the performance of accuracy probes degrades to such a degree that SEPs seem to outperform accuracy probes even in-distribution.

We think these results are very interesting and plan to investigate CoT prompting further for the camera ready version of the paper.

Again, thank you very much for your review! Please let us know if you have any remaining questions or concerns and we will do our best to respond to them as soon as possible.

Dear reviewer toQ8,

Thank you for your hard work and helpful feedback. We have gladly incorporated many of your excellent suggestions into the draft and look forward to engaging with you during the discussion period to clarify any remaining points. Please read our comment above, addressed to all reviewers, first.

Motivation of SEP efficiency: [...]

Thanks for bringing this to our attention! It is absolutely true that SEPs need to be trained while SE can just be calculated directly at test time without any training and, further, the initial training of SEPs does require SE computation for many training sequences. However, we do believe that our claims of SEPs lower computational costs at test time can be extended to apply to the overall cost, including SEP training cost, in many practical settings.

For one, the cost of computing SEPs at test time is basically negligible: some back-of-the-envelope math (we’re happy to add this) shows that SEPs add only about 0.0000002% in terms of FLOPS compared to the cost of an average generation for Llama-2-70B. This is compared to full SE, which increases test-time generation costs by a factor of K, with K=10 typical, and further requires significant computational overhead to run the semantic clustering algorithm.

So for the comparison that you suggest, one would essentially find that SEPs become cheaper than SE as soon as the size of the test queries is larger than the size of the training set. We use training sets with between 1000 and 2000 samples, which we believe is a number that should quickly be reached in large-scale deployment scenarios of LLMs.

Following your feedback, we have added a new discussion of limitations in Section 8, where we clearly state that SEPs are only compute-efficient overall if the number of training samples is much smaller than the number of test queries. (Which again, we believe should be quite a reasonable assumption in most cases.)

Improvement in Long Generation Experiments: [...]

Thanks for raising this concern! With our ‘short’ and ‘long’ generation scenarios, we directly follow Farquhar et al. (2024) in both notation and experimental setting.

We agree that extending our work to paragraphs of generations, i.e. multi-sentence response that contain a number of factual statements, is exciting work. However, we note that, as far as we are aware of, most approaches for detecting hallucinations in paragraphs boil down to first extracting individual factual statements and then testing each factual statement on its own (see e.g., Farquhar et al., 2024, arXiv:2303.08896, arXiv:2305.15852, arXiv:2309.11495, 2210.08726). As such, we believe that research into methods such as ours, which improve the factuality detection of individual statements, continues to be valuable, as they can be combined with many of the paragraph-length methods to serve as the core hallucination detector.

Further, a lot of other recent approaches for hallucination detection also focus on the setting of detecting hallucinations where model responses are as long or shorter than in our publication, see for example arXiv:2404.15993, arXiv:2402.03744, arXiv:2407.03282, arXiv:2307.01379, arXiv:2305.14613, arXiv:2302.09664. There is therefore clearly an interest in the community for detecting hallucinations in this setting. Note that our long generation prompt leads to model answers with an average length of 96 characters, which we suspect is much longer than many of the above citations.

Following your feedback, we have further investigated chain-of-thought prompting for our setup (see below).

To calculate semantic entropy (L185), how is the number of clusters K determined?

We here rely on the clustering algorithm proposed by Farquhar et al., see their Methods section and Extended Data Fig. 1. In this algorithm, the number of clusters K is not fixed for a given question. Instead K will correspond to the number of distinct semantic meanings found by the algorithm when given N sampled model generations.

Intuitively, the algorithm proceeds as follows. For each model generation, compare the generation to all existing clusters. If the generation has the same semantic meaning as one of the existing clusters, add it to that cluster. Else, create a new cluster containing that generation.

We have updated the draft to clarify that K is not fixed and corresponds to the number of distinct semantic clusters found by the algorithm.

Please read the second part of our rebuttal to your review next.

Please start by reading the first two parts of our rebuttal to your review.

Most of the experiments compare the proposed method with accuracy-based probing methods. However, there are insufficient details provided to describe these counterpart methods. It would be more helpful to provide a comprehensive introduction to these methods for readers who are unfamiliar with them.

Thank you for the suggestion! Following your feedback, we have expanded our discussion on accuracy probing in the related works section.

1. Do authors have a clear [...]

If we’re not mistaken, our answer to your first point above already addresses these questions.

One of the advantages of SEP is that it does not rely on ground truth labels from the dataset. However, as mentioned by the authors, there are also existing works [1,2] that predict the trustworthiness of LLMs from internal states using unsupervised methods. I am curious about how these methods would perform under the same settings as SEP.

Thanks for bringing this to our attention! With regard to [1], their proposed CCS method can only be applied to binary yes/no questions if we understand correctly. This setting is more restrictive than the general question answering scenario that we discover here. There further seem to be some problems with the robustness of this method as discussed by Farquhar et al. (2312.10029). [2] does present a number of very interesting ideas but note that their method is actually not fully unsupervised as they rely on labels to ‘identify the layer and direction’ of the probes for truthfulness prediction (Section 4.1). We have added a discussion of these work to our related work section.

Again, thank you very much for your review! Please let us know if you have any remaining questions or concerns and we will do our best to respond to them as soon as possible.

Please start by reading the first part of our rebuttal to your review.

Why will SEPs generalize better to OOD settings than accuray probes? This is a very good question, too. Our intuition here is that accuracy probes are more likely to latch on to spurious correlations between model predictions and accuracy in the dataset that will not generalize to new datasets, whereas SEPs will learn to robustly capture a semantic entropy signal from the latent space which predicts factuality well across tasks.

More concretely, we believe that largely unavoidable problems with assigning accuracy labels to ultimately vague linguistic statements are to blame for the deficiencies of accuracy probes. While semantic uncertainty should generally be a good predictor of model accuracy, there may be some cases where uncertainty is low despite the model answer being judged as incorrect (Farquhar et al., 2024). (This mostly happens due to shortcomings in measuring accuracy, e.g. the model answer not being precise enough, for example, just giving the year when the gold answer is looking for an exact date.) We believe these cases may lead to accuracy probes learning worse predictors than SE probes, and that this becomes noticeable on OOD data.

In this scenario, for the SE probe, everything works according to plan: the label will be ‘high semantic uncertainty’, which we assume can be robustly predicted from latent space as we have discussed above. So if the probe is trying to find the direction in latent space that is predictive of ‘high semantic uncertainty’, the information it is getting with this datapoint agrees with previous observations. We further know that SE works robustly across datasets, so if the probe finds a direction in latent space that robustly represents SE, it will generalize well.

For the accuracy probe, however, this datapoint is much more problematic. For it, the ‘SE uncertainty signal’ in latent space is not helpful for predicting the accuracy label anymore. Therefore, the accuracy probe is much less likely to learn to rely on the ‘SE uncertainty’ direction in latent space. Instead then, we believe that the accuracy probe likely relies on other features of the question embedding to achieve good predictive performance. Given the complexity of the hidden space, the accuracy probe may rely on features that identify questions the model cannot answer (e.g. by their topic, such as ‘historical facts’) instead of using the SE direction. However, these features would often be brittle and not generalize well to new datasets, as they conflate model capability with the labeling process. For example, we often seen models answering at the wrong level of granularity or missing implicit assumptions about for a particular subset of questions. For the OOD dataset, answer granularity and implicit assumptions may be different, such that these features will not generalize!

(For example, on dataset A, the accuracy probe may learn that the LLM cannot answer historical questions, because the LLM always responds only with a year, when the gold answer is a full date. The accuracy probe will then learn to associate ‘historical’ questions with low accuracy – it will not rely on the SE signal as the model is confident in its answers here. However, on dataset B, the gold answer is only years and not full dates on similar historical questions. This will lead to failure of the accuracy probe. However, SEPs will work well on dataset B as they are much more likely to capture the underlying SE signal, which leads to better generalization.)

In summary, our results support the idea that semantic uncertainty can be captured robustly from latent space, with SEPs generalizing similarly to SE. Accuracy on the other hand depends on an external labeling process, which the hidden states do not capture. Therefore, probe learning from accuracy signals is subject to conflicting information and spurious correlations, which lead to worse performance in generalization regimes.

Again, thank you for raising both of these questions. Following your feedback, we have improved our discussion of this aspect of SEPs in Section 8 in the current version of the draft.

An obvious limitation is the applicability of the proposed method to black-box LLMs. I recommend that the authors discuss this in the limitations section.

We agree that this is a limitation of SEPs (and probing methods in general), and have added a discussion of this to a new paragraph on limitations in Section 8.

Please read the last part of our rebuttal to your review next.

Dear reviewer 6ThF,

Thank you for your hard work and helpful feedback. We have gladly incorporated many of your excellent suggestions into the draft and look forward to engaging with you during the discussion period to clarify any remaining points. Please read our comment above, addressed to all reviewers, first.

I think the main weakness of this work is the lack of clear motivation and explanation for the outcomes. The SEP method appears to simply switch the prediction goal of previous internal state detection methods from accuracy to semantic entropy (SE), which seems like a simple 'a+b' combination. [...] why is it possible to use internal states to predict SE? And why does switching the prediction goal from accuracy to SE result in better performance in out-of-distribution settings?

We think these are really interesting questions! Fortunately, we can actually offer some intuitions and insights into why SE can be predicted from latent space and why SEPs would consequently generalize better than accuracy probes, and we are happy to detail these below. In fact, these intuitions were the reason we originally explored SEPs in the first place, and we see our results as supportive evidence of these intuitions. However, we agree that further work could be done to fully answer these questions and would welcome any concrete suggestions that you may have in this direction.

Why can we predict SE from latent space? For the general-purpose QA setting that we study here, we think the following toy example provides helpful intuition to answer this question. Let’s say we ask the LLM “When was the Hoover Dam built?”, and let’s further assume the LLM does not know the answer to this question. In our experience, which is supported by Farquhar et al. and others, this will lead to high semantic uncertainty in the answers. Concretely, when sampling from the model, we will obtain answers such as “It was built in 1930”, “1931”, “In 1932”, “1930”, “The hoover damn was built in 1930”, “In 1932”, or “1932”. The sampled answers vary in both syntax and semantic meaning. In terms of syntax, we have the variants “It was built in X”, “X”, and “In X”, where X is a year. And in terms of semantic meanings, we observe the years “1930”, “1931”, and “1932”, indicating high semantic uncertainty.

Now, when looking at only the distribution of p(next_token|”When was the Hoover Dam built?”), based on the above generations, we know that the model assigns probability mass to tokens such as [It, The, In, 1930, 1931, 1932] (depending on the exact tokenizer of course). Crucially here, models almost always consider giving extremely short answers (which are just directly the year in this case). From this, we can conclude that the latent state before answering contains knowledge of the semantic meaning of the generated answer as well as the semantic uncertainty over it, even if the model ends up generating a longer-form answer such as “The hoover dam was built in X”. Note that this idea is more general than it may seem at first, as there is rarely a question which cannot be answered in just a few tokens.

So, in summary, the latent space here must contain semantic-level information about the answer. It would be highly likely for the model to learn an efficient representation that separates semantics and syntax, such that it can keep track of high level information across tokens. Further, for the probabilities of the model to be calibrated (which is something that should be promoted from training with a proper scoring rule such as negative log likelihood loss), this latent state should correctly model uncertainty over the semantic meaning. By inspecting the token distributions, one can see that, for any given syntactic instantiation (such as “It was built in X”), the token probabilities are dispersed when semantic uncertainty is high. It seems to us that the simplest way to achieve this for a given model is to represent semantic meaning and uncertainty in the latent space separately from syntax. This is in line with work such as Belrose et al. (arXiv 2303.08112) and Pal et al. (arXiv 2311.04897) who show that hidden states are predictive of outputs multiple tokens ahead with high accuracy, as well as represent and control higher level functions of LLM behavior (see our literature review).

Please read the second part of our rebuttal to your review next.