Shifting Attention to Relevance: Towards the Uncertainty Estimation of Large Language Models

Q1: Although there is stable improvement over the baseline models, it seems marginal.

Thank you for acknowledging our consistent improvements! We would like to present statistics demonstrating that the advancements in SAR are indeed substantial. Our paper includes approximately 21 experiments: 8 on pre-trained Large Language Models (LLMs), 8 on instruction-tuned LLMs, and 5 on medical datasets. We report the enhancements achieved by SAR in comparison to state-of-the-art methods (such as PE, LN-PE, SE), including average improvements, average improvements per setting, and the number of experiments in which SAR outperforms others by at least 2%.

It is shown that our SAR outperforms state-of-the-art experiments with >2% margins for most experiments. Therefore we would like to mention that the improvement is rather substantial and notable.

Q2: I have a minor question since I am not an expert in this area: why would the unimportant tokens pose a higher uncertainty? In Figure 1, it seems that the unimportant tokens have relatively less possibility to convey information since the model should have been pretty certain about these tokens in my understanding. However, as shown in Figure 3, this is actually the opposite since the token-level mean of UP is actually higher when the relevance is less. Could you please give an explanation for this?

R2: Thank you for your insightful comment! Why would the unimportant tokens pose a higher uncertainty is exactly the question that inspired our method.

We believe this is due to the intrinsic properties of LLMs, such as data imbalance in training corpora. For example, if the LLM frequently encounters the phrase "according to" during training, the likelihood of the word "to" following it becomes very high (meaning the uncertainty of "to" is very low) when the preceding token is "according". If the LLM has never encountered "density of" (as shown in Fig. 1 of our paper), the probability of the word "of" following "density" is low (with high uncertainty). Thus, the certainty of LLMs regarding these tokens should be determined by their training. In Figs. 2 and 3, we demonstrate that many irrelevant tokens are assigned lower probabilities by LLMs, which we believe impedes accurate uncertainty estimation.

Our motivation is that important tokens should dominate uncertainty estimation, rather than less significant ones. To achieve this, we quantify the importance of tokens and refocus on these significant tokens during uncertainty quantification, leading to improved performance.

We hope these kindly address the reviewer's concerns. We would like to provide responses if the reviewer has further questions.

Dear Reviewer VX9S,

Thank you again for your insightful comments. This is just a gentle reminder as the discussion period ends soon. We would like to know whether we have addressed your concern. Thank you!

Authors of paper 3019

Q3: The tables in the paper are very difficult to parse; the font size is small, and presentation could be improved.

Thank you for pointing this out! We have modified the formats and font sizes of tables in our new revision.

Q4: Is removal the right way to test for change in semantics (rather than just considering likely replacements)?

Thank you for your insightful comment! We believe that removal is an appropriate approach for our method for the following reasons:

1. Removal allows for the quantification of per-token relevance and is easy to implement.
2. Removal strategies are widely employed in existing works[1-2].
3. The grammar mistakes or fluency issues caused by removal are not significant for current sentence similarity measurements. For example, consider the following SciQ example, which demonstrates the changes in similarity after the removal of each token, as measured by the cross-encoder-roberta-large model:

The similarity significantly changes only when key tokens (such as Mesophilic, food, cheese, yogurt, beer, wine) are removed. This is conceptually aligned with human understanding and also meets our requirements effectively.

Using replacements could be less effective because if we substitute a token with synonyms, the semantic impact is minimal, making it challenging to quantify token relevance accurately. Additionally, finding suitable replacements for some words or tokens, like definite and indefinite articles, prepositions, etc., can be difficult. We are open to conducting evaluations if the reviewer can suggest specific strategies for replacements.

Reference:

[1] Lee, Joosung. "Stable style transformer: Delete and generate approach with encoder-decoder for text style transfer." arXiv preprint arXiv:2005.12086 (2020).

[2] Liu, Junyi, et al. "TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction." arXiv preprint arXiv:2310.15556 (2023).

We hope these kindly address the reviewer's concerns. We would like to provide responses if the reviewer has further questions.

Q1: The sentSar and tokenSar methods are conceptually quite different, and don’t really make sense to package together (especially since while they are compared, sentSar requires more computation than tokenSar).

Thank you for your insightful comment. We agree that sentSAR and tokenSAR differ conceptually, as they focus on distinct aspects of generation. However, as outlined in Sec. 5.2, they are complementary and can be integrated to address uncertainty quantification effectively at both the token and sentence levels. Consequently, we consider sentSAR and tokenSAR as two separate contributions of our paper. These distinctions are emphasized in our new revision (Sec. 4.3).

Q2: The sentSar method is more computationally expensive than baselines.

Thank you for your insightful comment. We agree that sentSAR is more computationally expensive than the baseline methods. However, it's important to highlight that sentSAR is more generation-efficient. It surpasses baseline methods under significantly smaller computational constraints. As demonstrated in Fig. 2, SAR, with just 2~3 generations, outperforms all other methods that use 10 generations.

We have quantified the time consumed for each step in the overall uncertainty quantification pipeline. This includes sequence generation, computing logits, semantic clustering for SE, and sentence similarity for sentSAR. We exclude the time taken for aggregating logits/scores as it is negligible (less than 0.001 second for all methods). The average time consumed per question, based on an evaluation of 1000 questions from the Vicuna-13b + SciQ dataset, is provided. These measurements were taken using an AMD EPYC 7302 16-Core CPU and a 1xA40 GPU server:

We present two scenarios for our sentSAR: 1) It takes 7.86 seconds when utilizing 5 generations; 2) The time reduces to 2.64 seconds when using just 2 generations, which is the minimum number of generations required for sentSAR.

Then we compare the 2-generations sentSAR with 5-generations baseline methods:

Our sentSAR still surpasses the baseline methods while consuming less than half the time, demonstrating its greater generation efficiency. We have included these results in the updated version of our paper (Appendix C.5).

Dear Reviewer UFP5,

Thank you again for your insightful comments. This is just a gentle reminder as the discussion period ends soon. We would like to know whether we have addressed your concern. Thank you!

Authors of paper 3019

Q1: I believe there is a typo in Equation 9. According to the definition of R_S given in Equation 4, the sum in the second line should be over $g(s_j, s_k)p(s_k|x)$ not $g(s_j, s_k)p(s_j|x)$.

R1: Thank you for pointing this out! It is a typo. We have corrected this in our new revision.

Q2: "The reason we normalize relevance score in Eq. (6) is two-fold: a) to make tokens comparable within a sentence" - this doesn't make sense, the relative token relevances are the same before and after the normalization, they're all being scaled by the same factor

R2: Thank you for pointing this out! You are right, this is a typo. We mean with normalization, token relevances are comparable across sentences. We have fixed this statement in our new revision.

Q3: Table 1 is very crowded (multiple models, multiple datasets, multiple Rouge-L cutoffs, multiple baselines, multiple SAR methods). Maybe stick with just 1 RL cutoff and if you want to include the other cutoff, put it in the appendix.

R3: Thank you for your suggestion! We have split Table 1 into two parts and only keep the results of RL-0.5 in the main body of our new revision. The results of RL-0.3 are placed in Appendix C.4.

Q4: "The area under the receiver operator characteristic curve (AUROC) metric is equivalent to the probability that a randomly chosen correct answer has a higher uncertainty score than a randomly chosen incorrect answer." - Shouldn’t this be the opposite? If the model’s uncertainty is high, then it implies that the model is more likely to get the answer wrong, no?

R4: Thank you for pointing this out! We indeed follow the setting from the SE paper and we didn’t realize this typo before. We have corrected this description as the following (also in our new revision):

Following prior work (Kuhn et al.), we evaluate uncertainty estimation by predicting the correctness of the model's generations regarding a given question, i.e. to what extent the generated answers can be trusted. The area under the receiver operator characteristic curve (AUROC) indicates the probability that a random correct generation has a lower uncertainty than a random incorrect generation, predicted by uncertainty estimation methods. AUROC equals 0.5 means the assigned uncertainty is no better than random guessing, i.e., they can not differentiate between correct and incorrect generations. AUROC equals 1 means all the correct generations are assigned lower uncertainty than all incorrect generations.

We hope these kindly address the reviewer's concerns. We would like to provide responses if the reviewer has further questions.

Dear Reviewer i37c,

Thank you again for your insightful comments. This is just a gentle reminder as the discussion period ends soon. We would like to know whether we have addressed your concern. Thank you!

Authors of paper 3019