Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models

Dear Reviewer MD6P,

Thank you so much for your detailed insights and the time you took to review our paper. We are extremely excited that you find our findings novel, and find that our work can inspire many future works! At a high level, we provide new results that

• (i) we show that the errors that attention and confidence make are different
• (ii) this lets us come up with a method that combines confidence and attention, which works better than confidence alone.

We further incorporated the suggestions, clarified the contribution of our framework compared to the existing literature, and resolved ambiguities in the text.

1. The proposed method is not better on factual error prediction than the simple confidence baseline. It seems to me it does not bring any new information beyond the confidence score

Do confidence and attention result in the same predictions? Combining Attention and Confidence: In Appendix E Figure 15, we compare the predictions by Attention vs Confidence for all three models in the Llama family. Overall, while they do correlate, there are many cases where both predictors disagree. For instance, the model can be overconfident and attention can correctly identify factual errors (see top left). Because the errors are not identical, we can combine them to build a better system.

We extended our results with a Combined predictor, where we add confidence in addition to the attention features, and perform logistic regression. This model mostly performs the best overall, in terms of AUROC (Figure 6), and risk in either tail (bottom 20% and top 20% most confident predictions, Figures 9 and 10).

Why is the finding that model attention can be used to predict constraint satisfaction and is comparable to confidence important\interesting? We would like to highlight that the LLM is optimized using confidence as the objective (e.g. probability of the next token). We find it to be an extremely interesting finding that the process that the LLM uses to produce an output, without information about the objects it is acting on (predicted probability distribution), gives almost as much information as the output. We believe our findings could lead to several follow-up works in studying factual errors and the model’s reliability. Why do we have more attention to popular entities? How can we manipulate attention to increase constraint satisfaction and steer the model behavior? Our framework and findings open interesting questions and can pave the way for future research in this area.

Important difference between attention and confidence: As we emphasize in the paper, note that confidence is only associated with the generated tokens and cannot be traced or mapped back to individual constraints for queries with multiple constraints. This is a crucial functional advantage of SAT-Probe and becomes even more important as the query complexity increases.

2. Although the findings are interesting to know, I doubt the practicality of the proposed method

Constraint availability: We agree that our experiments focus on settings that assume access to constraints. Nevertheless, constraint extraction from user queries is an interesting language task to study by itself and can be considered as a pre-processing step prior to any factual verification method. Relevant methods may rely on entity recognition, previous work [a,b] on parsing declarative queries from natural text, or innovations in chain-of-thought methods that first extract main constraints from queries before generating factual answers. We added this note in our conclusion section.

[a] Elgohary, Ahmed, Saghar Hosseini, and Ahmed Hassan Awadallah. "Speak to your parser: Interactive text-to-SQL with natural language feedback." arXiv preprint arXiv:2005.02539 (2020).

[b] Yaghmazadeh, N., Wang, Y., Dillig, I., & Dillig, T. (2017). SQLizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages, 1(OOPSLA), 1-26.

3. I think this paper focuses on and only studies a narrow notion of factuality, where constraint entities exist as in a world knowledge domain

Generality of the framework: We agree with the reviewer that the CSP framework does not cover all potential queries to LLMs. Our work lays out a framework for mechanistic interpretability to analyze a broader class of queries, thus contributing to the literature: existing work in factuality and mechanistic interpretability in LLMs (e.g. Meng et al. NeurIPS 2022, Geva et al. EMNLP 2023) only considered single-constraint queries with well-defined, yet restrictive (subject, relation, object) structure. While there are still limitations, it creates further opportunities regarding what we can understand with a mechanistic lens, in particular by directly mapping attention patterns of individual constraints to model performance. We would appreciate it if the reviewer could consider the context of the contribution here.

Further, we do not only look at CounterFact and we expand the set of datasets further to questions that are verifiable and that can also be reused in future research. We further agree with the reviewer that there is more work to do around mathematical knowledge and reasoning, and we believe these are interesting avenues for future work. We added this to the conclusion section, thank you for your suggestions.

4. From Figure 14-23, all the datasets look simple and toy.

As further shown in Table 1 and Figure 6a, our datasets also contain queries with double constraints (which were previously not covered in related work). Most of the related work on mechanistic interpretability for factual queries has used CounterFact (which we also study) for single-constraint queries. Here, we expanded the set of datasets, and hope that this can serve as a resource to the community to study more complex query forms.

5. does that mean you need a training dataset with labels to learn the parameters w and b?

In each of the datasets, we split the dataset into 50% train -50% test splits and train on the training set. Thus, the number of data points used to train a probe is half of the number of queries in Table 1 when testing on respective datasets.

Overall, we do believe we are studying the problem of factual error prediction from a novel point of view, our findings around attention and factual error prediction, datasets, and the framework could prove useful to the literature. If you also find similar value in this research, the future work that can build on our insights, and contextualization in the literature, we would appreciate it if you could consider increasing your score. Thank you very much once again, and we are happy to follow up with any additional questions you may have!

Dear Reviewer m32R,

Thank you so much for your kind words and the time you took to review our paper. We are truly excited by your support, and we appreciate that you found no major weaknesses; and thanks for your insightful questions!

At a high level, we provide new results that

• (i) show that the errors that attention and confidence make are different
• (ii) this lets us come up with a method that combines confidence and attention, which works better than confidence and attention alone.

Below we respond to the individual points you raised.

1. ...suggests that when LLM is confident it is often right. How does this generalize in cases shown in prior work when LLMs confidently hallucinate?, In your experience, is there any way to differentiate between attention scores (or some other variable) for correct vs incorrect confident answers

Differences between confidence vs attention scores: In Appendix E Figure 15, we compare the predictions by Attention vs Confidence for all three models in the Llama family. Overall, while they do correlate, there are many cases where both predictors disagree. For instance, the model can be overconfident and attention can correctly identify factual errors (see top left). Thus, there is value in considering both predictors.

Combining attention and confidence: We extended our results with a Combined predictor, where we add confidence in addition to the attention features, and perform logistic regression. This model mostly performs the best overall, in terms of AUROC (Figure 6), and risk in either tail (bottom 20% and top 20% most confident predictions, Figures 9 and 10).

2. In Figure 5, it looks like attention is summed across layers, its unclear if the final score was normalized by number of layers?

Normalization in Figure 5: We sum the values across layers, and normalize all numbers by the maximum observed value as indicated in the caption of Figure 5. Indeed, for models with a larger number of layers, the unnormalized values are larger. We wanted the visualization to be more interpretable within the context of the model being investigated. We added further clarification to the caption (see in red), and thank you for raising this point.

3. Were there any insights as to how attention patterns vary across different layers, especially for different-sized models? for e.g., higher attention scores for hallucinating tokens towards the later layers perhaps?

Attention Across Layers: Very interesting question! We find that scaling matters here. In Appendix Figure 7, we discuss the impact of early stopping. Interestingly, we observe that we can make failure predictions in the earlier layers as well as the later layers for smaller models (left two plots). However, for larger models, we need to observe the attention in the later layers to make better predictions. We believe one can hypothesize that attention scores in later layers are more predictive for the larger models than smaller models, based on these findings. Other than this, we did not observe stronger findings around larger attention values in later layers indicating factual errors, but we believe our tools (e.g. tracking the dynamics of attention to constraints) can lead to future research on this question.

Overall, we do believe we are studying the problem of factual error prediction from a novel point of view, our findings around attention and factual error prediction, datasets, and the framework could prove useful to the literature. If you also find similar value in this research and future works that can build on these findings, we would really appreciate it if you could consider increasing your score. Thank you very much once again, and we are happy to follow up with any additional questions you may have!

Hello Reviewer KSCt,

First, thank you for your time reviewing our work! We believe the above review belongs to another paper, and the review for our paper is possibly elsewhere. Would you mind updating your review?

Thank you so much!

2. I do wonder about the generality of thinking of factuality as a CSP

Generality of the Framework: We agree with the reviewer that the CSP framework does not cover all potential queries to LLMs. Our work lays out a framework for mechanistic interpretability to analyze a broader class of queries, thus contributing to the literature: existing work in factuality and mechanistic interpretability in LLMs (e.g. Meng et al. NeurIPS 2022, Geva et al. EMNLP 2023) only considered single-constraint queries with a well-defined, yet restrictive (subject, relation, object) structure. We would appreciate it if the reviewer could consider that in the context of mechanistic understanding of LLMs, moving to multiple constraints is novel and allows broader investigations. While there are still limitations, it creates further opportunities regarding what we can understand with a mechanistic lens, e.g. in this case by directly mapping attention patterns of individual constraints to model performance.

Thanks for pointing out the analogy to how SQL and other declarative language model constraint-based search! Indeed, previous work (such as “Conjunctive-query containment and constraint satisfaction” from Kolaitis & Vardi, 1998 but also others) have used the same framework to model similar queries, albeit not for machine learning models. The analogy is a great argument for leveraging the same framework but with an ML perspective and by using architectural information native to trained models (rather than stored information in database systems). If we missed any other works here, please let us know and we will be happy to discuss further and cite further work on this topic.

3. Is there a breakdown of wall-clock time for each of the datasets?

Wall-clock time: Our comments are based on the fact that the inference time scales linearly in the number of layers, and we can sometimes use less than 50% of the layers to predict failure as successfully as using the entire network. While the exact inference time will depend on the prompt length and the hardware, it will scale linearly in the number of layers.

4. Table 7 examines the ability of SAT probe to isolate which constraint was violated when there are multiple constraints. Some more detail is necessary here: in the dataset, when 1 constraint is violated, how often is it the case that both constraints are violated?

The numbers vary across datasets – sometimes only one fails and sometimes both do. Below is a table summarizing the statistics for Llama-2 70B performance in the first train/test split as an example:

Thus, indeed there are datasets where only one of the constraints fails, and attention can identify the ones that do (shown by Table 7).

Insights into how to fix errors: This is an important question that warrants future research. While we do not have experiments regarding this point, our findings can lead to future investigations. E.g. It would be interesting to intervene in attention to constraints to modify the model behavior. However, our setting is akin to Selective Classification, where we focus on abstention as the action to take when reliability is expected to be low. We added further suggestions in the conclusion section on this point, thank you for the suggestions.

Overall, we do believe we are studying the problem of factual error prediction from a novel point of view, our findings around attention and factual error prediction, datasets, and the framework could prove useful to the literature. If you also find similar value in this research and future works that can build on these findings, we would really appreciate it if you could consider increasing your score. Thank you very much once again, and we are happy to follow up with any additional questions you may have!

Dear Reviewer KSCt,

Thank you so much for your detailed insights and the time you took to review our paper, and also for updating your review promptly, we really appreciate your help! We are extremely excited that you find our framework novel, and experiments insightful and convincing!

At a high level, we provide new results where

• (i) we show that the errors made by attention and confidence are different
• (ii) this lets us come up with a method that combines confidence and attention, which works better than confidence or attention alone.

We further incorporated the suggestions, clarified the generality of the framework compared to the analyses in the existing literature, and resolved ambiguities in the text.

Below we respond to the individual points you raised.

1. My main point of contention is that the SAT probe actually underperforms confidence classification by a statistically significant margin, However, broadly speaking, are there ever cases of the LM being confidently wrong?

Confidence vs Attention: We give two sets of new results to address this question:

Do confidence and attention result in the same predictions? Combining attention and confidence: In Appendix E Figure 15, we compare the predictions by Attention vs Confidence for all three models in the Llama family. Overall, while they do correlate, there are many cases where both predictors disagree. For instance, the model can be overconfident and attention can correctly identify factual errors (see top left subfigure). Because the errors are not identical, we can combine them to build a better system.

We extended our results with a Combined predictor, where we add confidence in addition to the attention features, and perform logistic regression. This model mostly performs the best overall, in terms of AUROC (Figure 6), and risk in either tail (bottom 20% and top 20% most confident predictions, Figures 9 and 10). Thus, there is value in considering both predictors, in addition to providing fine-grained predictions per individual constraints and early stopping with attention.

Why is the finding that model attention can be used to predict constraint satisfaction and is comparable to confidence important\interesting? We would like to highlight that the LLM is optimized using confidence as the objective (e.g. probability of the next token). We find it to be an extremely interesting finding that the process that the LLM uses to produce an output, without information about the objects it is acting on (predicted probability distribution), gives almost as much information as the output. We believe our findings could lead to several follow-up works in studying factual errors and the model’s reliability. Why do we have more attention to popular entities? How can we manipulate attention to increase constraint satisfaction and steer the model behavior? Our framework and findings open interesting questions and can pave the way for future research in this area.

• This should be moved up and highlighted as one of the main advantages of this method., This seems to undermine “we find that SAT PROBE mostly performs comparably to and sometimes better

We adjusted the tone of the claims about that table per se and highlighted the advantage of attention in being more efficient and allowing us to interpretably isolate the constraint that failed. Thank you for these suggestions.

• it would be good to note what type of error bars are being plotted and how many trials this was over/what the source of randomness was

Error bars are discussed in the caption of Figure 6, i.e. “Error bars show the standard error over 10 different random train/test splits.”