Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models

Thank you for the constructive review of our paper. We are particularly encouraged by your positive assessment of our experimental methodology and results.

W1. Lack of comparison to methods other than SAEs

Linear probing can serve as a method to identify meaningful directions within neural representations. However, the strength of SAEs lies in their ability to perform unsupervised discovery, making them a powerful exploratory tool in interpretability research (Templeton et al., 2024, Kissane et al., 2023). Unlike linear probes, SAEs allow for the identification of novel patterns without requiring a specific training labeled dataset.

Moreover, linear probes may indicate correlations between model representations and the features, but it can’t tell us whether the feature is involved in the model predictions (Belinkov et al., 2022). In contrast, SAEs circumvent this issue: the found latents contribute directly to the reconstruction of the representation, providing greater confidence that the identified directions are genuinely used by the model. While SAEs remain a relatively underexplored approach in interpretability research, our work demonstrates their potential as a powerful tool for uncovering unexpected phenomena, as it enabled us to discover the unexpected features presented in this work, offering a valuable case study in their application.

Regarding the sparsity constraint within the SAE, this allows the reconstruction to be a linear combination of a small subset of the learned dictionary features ($W_{\text{dec}}$). It is therefore typically used as a proxy for interpretability. Forcing sparsity is “based on the intuition that natural data, (such as text or images) may generally be described in terms of a small number of structural primitive”, we refer to “Why sparseness” section in Olshause et al., 1997.

W2. Lack of model diversity

We have run experiments on Llama 3.1 8B, using the SAEs suite from LlamaScope (He et al., 2024). Following the methodology detailed in Section 3, we successfully identified both known and unknown entity latents. The layerwise analysis of Top 5 latents follows patterns consistent with our previous observations, with the most effective and generalizable latents emerging in the middle layers. When applying steering with the top unknown entity latent, we also observe increased refusal rates in the instruction-tuned model, as well as a reduction when orthogonalizing the model with that direction. In line with previous results with Gemma, the decrease when steering with the top known entity latent is not clearly observed. Finally, we replicate the observations regarding the disruption of attention to the entity tokens. There is a decrease in attention scores when steering with the unknown entity latent, and an increase when steered with the top known entity latent. All the results can be found in Appendix Q.

The replication of our key findings from Gemma in Llama 3.1 8B gives us increased confidence in the robustness of our methodology and the generalizability of our results. Especially, given the architectural differences between these models, and their associated SAEs.

We thank you for the suggestion, we believe the comparison with other models/SAEs has helped improve our manuscript.

Q1. Can you provide more ablation study to show that it's really necessary to use both 1) sparse and 2) auto encoder modules to get the interpretability?

See response to W1.

Before the discussion period ends, we wanted to ask the reviewer whether we have addressed your concerns with our work?

We appreciate your feedback, and as a result have added results on LLaMA 3.1 8B and clarified that SAEs played a role in exploratory discovery of the existence of entity recognition, in addition to helping find the entity recognition direction. Exploratory discovery of concepts we didn't expect is a unique advantage of SAEs over probing. We would appreciate clarification on any remaining concerns you have, and ask if you might be open to increasing your score if we have addressed all of them.

We appreciate your positive review and encouragement.

W1. The experiments are now only for Gemma models. It would be interesting to see the model behavior on other LLMs.

Thank you for this suggestion. We have expanded our analysis to include Llama 3.1 8B using the LlamaScope SAEs release (He et al., 2024). The patterns we observed in Llama 3.1 align closely with our findings from the Gemma models, reinforcing the generalizability of our conclusions. The results can be found in Appendix Q.

Q1. How does the model size influence the conclusion of the study?

We observed particularly pronounced effects in the smaller Gemma 2 model (2B parameters). However, drawing definitive conclusions about the relationship between model scale and our findings requires further investigation.

We thank you for your thorough review. We appreciate the positive comments, as well as the thoughtful points raised.

W1. Definition of known and unknown entities. It does leave open the possibility that what is currently categorized as unknown is actually known by the model and can perhaps be elicited with a better prompt or few-shot examples.

The primary objective is not to create perfect distinctions between known and unknown categories. Rather, we aim to verify that these latents can distinguish between the two types of entities. We believe minor classification errors are acceptable, as they don't impact the core conclusions of the paper. We have now raised this point in the paper. As you suggested, to validate the robustness of the entity recognition latents we identified, we conducted additional testing using a list of 283 songs released after August 2024 (post-model training). The frequency of activation of the entity recognition latents across models are:

We acknowledge that this validation set may contain repeated song titles from previous years that overlap with pre-training data. However, the consistent activation patterns across models support the robustness of our identified latents in distinguishing knowledge states.

W2. Simpler explanation which currently can’t be ruled out is that these latents are activated on high likelihood tokens (known) and low likelihood tokens (unknown).

Thank you for this suggestion. We agree that token likelihood represents a compelling potential confounder, as known entities are inherently more predictable than unknown ones. To address this concern, we have conducted an extensive investigation detailed in Appendix S ("Token Likelihood Hypothesis").

The entity recognition latents discussed in the paper were selected after filtering out those that activate on more than 2% of random tokens from the Pile. To further evaluate the Token Likelihood Hypothesis, we analyzed these latents' activations on a sample of the FineWeb dataset. Our analysis reveals that these latents are highly sparse, activating on only 0.1-2.6% of tokens, and show minimal correlations with token prediction probabilities (Table 9).

We have also investigated more complex dependencies by examining relationships with prediction entropy and perplexity of the surrounding tokens, finding no substantial correlations. We believe that these results provide sufficient evidence to conclude that the latents we find encode a more sophisticated form of entity recognition beyond simple token predictability.

W3. Hard to directly interpret the results in Fig 2. How high should that (min-max) value be to conclude the existence of generalized latents?

Unfortunately, there is no specific min-max value which we can take as reference, especially since the number of labeling errors are unknown. However, a particularly noteworthy finding was the systematic progression of scores across layers in all models, which suggests non-random, structured formation of entity recognition features. Our study includes approximately 35,000 distinct entities across four types. However, the entity recognition capabilities we observed extend well beyond these core categories. As demonstrated in Tables 4 and 5 (Appendix B), the identified latent representations largely activate across a diverse range of entities, including corporations, scientific figures, artists, and cultural institutions, suggesting high generalizability.

Q2. On average how many attributes does each entity have?

On average each entity type has 4.75 attributes. We refer to Table 3, Appendix A.

Q3. I also couldn’t find details about how many entities of each type was used in the analysis.

This information is available in Table 3, Appendix A.

Q4. Have you tried changing the prompt in 10 (line 359) to see if the intervention has a bigger effect in that case?

Thanks for the recommendation! As suggested in the paper, we will add few-shot examples and check if the effect is more pronounced.

Q6 and Q7.

Thanks for the suggestions, we have included them in the paper.

We appreciate your careful review and validation of our experimental approach and findings.

W1. I think the choice to describe these latents as representing "self-knowledge" is a little bit dicey.

We refer to knowledge awareness as ‘detecting whether the model can recall facts about the entity’ (lines 80-81). Our observations suggest this awareness mechanism causally affects the refusal behavior of the chat models. However, we emphasize that our definition of knowledge awareness focuses specifically on the detection process, not the subsequent response behavior. We approach the term "self-knowledge" with appropriate caution and, in line with your feedback, we have tempered our claims regarding “self-knowledge” in the revised version of the paper.

W2. The maxmin metric makes sense for relative comparisons but is difficult to understand on an absolute level. Indeed, when you select the "most general" latents based on this metric it's unclear if these features are actually general across more than the four entity types you use.

While we did the quantitative analysis on four types of entities, we observe the generality of entity recognition latents extend beyond those. Looking at examples of sentences generated by Claude, included in Tables 4 and 5 (Appendix B), we see how these latents pick up on all sorts of entities—everything from companies and scientists to artists and museums.

W3. In the mechanistic analysis… It's not clear to me from the text if this is averaging across all attention scores or some subset.

We apologize for the confusion, we refer to the aggregated attention scores to the entity tokens of all heads downstream from the latent. We have clarified this in the revised version of the manuscript and shown the differences in attention scores for all downstream heads in Appendix M.

There should be some straightforward measure here which you can use to quantify the suppression effect here.

Since the magnitude of change varies with the steering coefficient, we have assessed the statistical significance of attention score changes by comparing steering with entity recognition latents versus with random SAE latents using the same layer and steering coefficient. We conduct t-tests where the null hypothesis states that both steerings would yield identical mean attention scores differences across downstream attention heads. The alternative hypothesis is that known entity latents would increase mean attention scores, while unknown entity latents would decrease them. We tested against 10 different random SAE latents using 100 distinct prompts.

Results indicate that for Gemma 2 2B, Gemma 2 9B and Llama 3.1 8B, steering with the top known entity latent shows statistically significant larger average attention score when compared to random SAE latents on 10/10, 10/10 and 7/10 cases respectively. When steering with the top unknown entity latent it shows statistically significant lower average attention score when compared to random SAE latents on 9/10, 1/10 and 10/10 cases respectively. As shown in Figure 19, Gemma 2 9B top unknown entity latent doesn’t show strong reductions. However, the second unknown entity latent shows significant differences in 9/10 tests.

These results demonstrate consistent patterns in how steering affects attention scores across different model architectures, with particularly strong effects observed in Gemma 2 2B and Llama 3.1 8B.

W4. The results in Figure 5 seem dubious. They are really hard to read because of the overlap of all of the confidence intervals

The results in Figure 5 show a moderate but consistent correlation between the effect of steering with entity recognition latents and model self-knowledge expressions across all entities. While the effect size is modest, the directional movement in yes/no motivates further investigation. As you suggest, we plan to validate these findings through significance testing against random baselines using an expanded dataset.

Q1. The main text doesn't say which direction d_j you orthogonalize out on p.6.

We orthogonalize the model with the top unknown entity latent. We have now made it more clear it in the paper.

Q2. Equation 11's use of the word model is weird.

We refer to the token ‘model’, which is one of the end-of-instruction tokens in Gemma chat models. The example (11) is the exact prompt presented to Gemma. We have substituted the red highlighting to avoid confusion.

Q3. Do you have a succinct/clear and empirically grounded explanation of why you use (your formulation of) latent separation scores for the first analysis and t-statistics for the second?

Our initial analysis on entities examines a binary distinction between known and unknown entities. This is motivated by its applicability to hallucinations, as an ideal model should refuse to answer if presented with an unknown entity. Later, in Section 7, we explore a more nuanced aspect: the model's degree of uncertainty. Rather than treating uncertainty as a binary state, we conceptualize it as existing along a continuous spectrum. For instance, the model might still express some uncertainty even when answering correctly and therefore, we believe measuring the difference in activation means through the t-static seems more appropriate.

Q4. Can you clarify in the paper how you use the train/test split in the analysis in Section 7?

We divide the dataset of prompts into train/validation/test sets (50%, 10%, 40%). We use the training set to compute t-statistics and select the top latent. We use the test set to extract the left panel of Figure 6 and the AUROC and F1 scores. The validation set is used to choose the optimal decision threshold for computing the F1 score. We have clarified it in the paper.

Q5. Do you think you can add some discussion in the paper about how your approach can be generalized to other kinds of features?

We appreciate your suggestion. Our methodology for feature detection from SAEs can indeed be generalizable across different feature types. The framework we present in Section 3 enables the identification of binary features, while the use of the t-statistic score (in Section 7) instead of the latent separation scores extends this capability to features that can be considered continuous, such as model uncertainty. We have incorporated these clarifications in the revised manuscript to better emphasize the broad applicability of our approach.

This all sounds great. Thanks for your clarifications and improvements to the manuscript. Upping my Soundness rating to 4 and overall rating to 10. The overall score does feel a bit extreme as I would normally reserve that for groundbreaking work and the contribution here is a bit more focused. But the work seems super solid to me and I think is worth considering at least for a spotlight.

Do you think the sparse autoencoder-based approach in this paper can be extended to detect or mitigate hallucinations in situations involving known event https://pokerogue.io knowledge, not just entity recognition?