From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers

[GENERAL] We thank the reviewer for their thoughtful comments and their positive assessment of our experiments. To further support these findings, we have added targeted experiments with three new text datasets as well as an analogous assessment for the CLIP vision transformers to directly address the reviewer’s points (cf. W1 and W2 below).

[W1] We recognize that the experiments conducted in this work cover a lot of territory. We first show that transformer models strongly impose semantic structure on inputs, even when they lack true semantic structure (Section 3). This semantic framing is deconvolved from the internal model representations using SAE-identified concepts. We then demonstrate, in images and text, that this semantic imposition increases as inputs become increasingly unstructured and incoherent (Section 4). Finally, we directly link these SAE-identified concepts in the input space of the transformer to hallucinations in its output (Section 5). We desired to test a putative mechanism for the hallucination phenomenon observed in transformers in general, and, after validating this mechanism in images and language, to demonstrate the practical implications for acting on these insights (e.g. model steering of spurious concepts).

As the reviewer points out, there are many kinds of “hallucination” [1]. We tested the “faithfulness” of a summarization task, which is not dependent on the LLM’s background knowledge, as all required information is contained within the prompt. This is therefore a direct test of the effects that the transformer's internal representational structure has on its generated output. For more variety, see the tables presented in W2 for hallucination prediction evaluation on TruthfulQA (Lin et al. 2021), TriviaQA (Joshi et al. 2017), and CoQA (Reddy et al. 2019), as well as comparisons to established baseline hallucination prediction methods. We also perform a new experiment directly linking the L0 and patterns of semantic concepts to the misclassification of image samples (analogous to "hallucination" in the CLIP setting), effectively bridging the vision transformer analyses and the hallucination experiments. Please see the reply to Reviewer e9fj for more details and results (not reproduced here due to space limit).

[W2] We acknowledge that on its face, an $R^2$ of 0.27 may appear low in some contexts. However, we would like to note that this is a conservative out-of-sample result, averaged over 10 different cross validation folds, predicting a continuous degree of hallucination. This explained variance strengthens our core point that internal layer concept activations can be taken to play a role in output hallucinations. In Section 5, we reported a stronger 73.0% accuracy when classifying examples by whether their hallucination score is above or below the median. Our goal with these experiments was not to build a SOTA “hallucination detector”. Rather, we wanted to show a link between the concept activations and a concrete hallucination metric, and to provide evidence that intervening on these activations can suppress hallucinations to a measurable degree. We show this using a simple linear model. Sophisticated non-linear methods and larger training datasets would likely improve performance. Further, we are predicting output hallucination based on the internal pattern of concept activations derived from the input alone.

We acknowledge the reviewer’s point that additional baselines would improve credibility to our findings and conclusions. We have added the results of several new experiments in comparison with our PLS-Concepts approach. To provide a fair comparison, we run each hallucination benchmark on Gemma-2B-IT using the widely used “Language Model Evaluation Harness” framework (Gao et al. 2024). This framework provides standardized prompt templates and evaluation metrics for many benchmark datasets. Each hallucination prediction method is evaluated on the same 10 cross validation folds, and the mean AUC ± 1 s.d. across folds is reported. For each task, the hallucination detection method (rows of the tables below) must predict whether new, unseen samples are hallucinated, as labeled by the metric associated with each task (columns of the tables below). Note that different tasks evaluate the "hallucination" or "correctness" of an output using different metrics.

TruthfulQA — Generation

TriviaQA — Generation

CoQA — Generation

TruthfulQA — Multiple Choice

* = our model Based on multiple benchmarks of hallucination classification performance, we note that our method is at least comparable to, if not slightly superior to, many established hallucination prediction tools. We also note that these methods all require some input signal from the LLM-generated content, whereas our method only relies upon concept activations that are internally triggered by the input prompt. As well, Lexical Similarity, Normalized Entropy, and EigenScore require comparisons between multiple LLM generations in order to form their predictions (in the above benchmarks, we set the generation budget to 5). This is why they are not included in the Multiple Choice benchmarks.

We will add these useful results accompanied by a more fulsome description of the experimental details and evaluation protocols in the Appendix.

[W3] We thank the reviewer for making us aware of the lack of clarity surrounding these terms. We acknowledge that some of these terms could indeed be made more concrete.

[W3.1] In Section 2.2, $f_i(\mathbf{x})$ corresponds to the activation strength of concept i for the input sample $\mathbf{x}$. Thus, the vector $f(\mathbf{x}) \in \mathbb{R}^{d_\mathrm{SAE}}$ corresponds to all the SAE concept activations for the input sample $\mathbf{x}$; we refer to a single element of this vector as $f_i$. In this context, $f$ represents the function of the encoder $F: \mathbb{R}^{d_\mathrm{model}} \to \mathbb{R}^{d_\mathrm{SAE}}$. In response to this comment and the comments from reviewer LyaA, we provide an updated Section 2.2 enhanced for clarity (please see the reply to LyaA for the updated section, not reproduced here due to space limit).

[W3.2] In the mechanistic interpretability literature, “a concept is activated” for a given input if $f_i(\mathbf{x}) \geq 0$ for concept $i$. That is, there is a non-zero entry for concept $i$ in the SAE-encoded vector of the input sample.

[W3.3] We define the concept quality metrics semantic purity and steerability more formally in Section 2.4. Semantic purity measures the extent to which the semantic labels associated with image samples that activate for a given concept are similar to one another. A concept is "pure" if the top-$k$ ($k$=16) maximally activating images for that concept have an average cosine similarity of 0.75 across their semantic label embeddings. Steerability is the capability of flipping a neutral image's semantic label to that of the concept, by clamping the concept's activation to an artificially high value. If $n=32$ out of $32$ images from a randomly sampled batch of neutral images flip to the concept's label after clamping, then this concept is said to be "steerable". These conservative metrics are used to validate that the concepts extracted from the noise activation-trained SAE are meaningful. We will update Section 2.4 to reflect the clarified definitions.

[CLOSING] We believe that these experiments and clarifications significantly strengthen the results and conclusions introduced in our paper. We thank the reviewer for suggesting these improvements.

[1] Huang et al. "A survey on hallucination in large language models" 2023

[2] Ren et al. "Out-of-distribution detection and selective generation for conditional language models." 2022

[3] Liu et al. "Energy-based out-of-distribution detection." 2022

[4] Lin et al. "Towards collaborative neural-symbolic graph semantic parsing via uncertainty." 2020

[5] Malinin and Gales. "Uncertainty estimation in autoregressive structured prediction." 2020

[6] Chen et al. "INSIDE: LLMs' internal states retain the power of hallucination detection." 2024

[GENERAL] We thank the reviewer for their positive assessment of our work and for their constructive comments. We particularly appreciate the reviewer highlighting the novelty and significance of our noise activation-trained SAE experiments and our mechanistic insights into internal transformer concept dynamics. In response to this reviewer comment and the comments from Reviewer pJz5, we have added several new hallucination benchmark datasets and have compared our hallucination detection approach with established baselines. We have also added a new experiment linking the vision SAE-identified concepts to an analogous notion of hallucination in the image setting.

[W1] We recognize that our hallucination results could benefit from proper contextualization with additional datasets and existing hallucination detection methods. Note that our main goal here was not to develop a state-of-the-art hallucination detection method. Instead, we aimed to reveal a direct link between SAE-identified concepts in the internal representations of transformers and their propensity to produce confabulated outputs. In this sense, the "faithfulness" of summarization is an ideal task as it is agnostic to the internal "knowledge" of the LLM [1]. However, we have conducted additional benchmarks with TruthfulQA [2], TriviaQA [3], and CoQA [4], comparing with established hallucination prediction baselines. These short form Q+A tasks expand the scope of hallucinations studied with our present framework. Please see the reply to Reviewer pJz5 for the full results tables and experimental setup (not reproduced here due to space constraints).

We also recognize that the connection between the noise-input experiments (Sections 2 & 3) and the hallucination experiments (Section 4) could be made stronger. It is somewhat challenging to find exact parallels between vision and language model hallucinations, which is why we focused solely on language hallucinations in the original text. However, to this end, we have conducted additional experiments testing the utility of the L0 metric in predicting an analogous notion of hallucination in the CLIP model: misclassification of semantic image labels. We train a simple linear classifier to predict whether or not an image will be misclassified, based solely on the L0 metric for that image. We source the concept activations and calculate the L0 metric from layer 6 of the vision transformer, as this layer yielded the largest spike in mean L0 differences between normal and shuffled images in Section 4. We train our linear classifier on L0 counts of SAE latents extracted from 9000 images, and verify the results across 10 independent cross validation folds. We find that we can predict whether or not an image will be misclassified solely using its associated L0, robustly above random chance. We also attempt to use the pattern of concept activations, extracted by a partial least squares (PLS) model, to predict the same misclassification. While still robustly above chance, the results for the PLS model are slightly inferior to the raw L0 results (see table below). As well, the natural versus shuffled images seem to yield equal prediction performance; this is likely because although the rate of hallucination (misclassification) is higher in the shuffled images, it is not easier to predict whether a misclassification will occur for a given sample. We believe that this new experiment provides a solid bridge between the vision transformer experiments and the direct hallucination prediction tasks with the LLMs.

Mean AUC (percentage) ± 1 s.d., from ViT layer 6 concepts

[W2] We will add qualitative examples to the Appendix exploring the additional concepts activated for shuffled images over their natural counterparts.

[Q1] This is an interesting point. Based on the results presented in our paper, we would draw the conclusion that noisy input data is one clear way to increase the likelihood of hallucination. This is evident in the extraneous concept activations that arise as the transformer imposes structure on inputs with little true semantic content (Sections 3 & 4). However, there are likely many other means of inducing hallucinations (such as increased context length, certain topics, etc) [5]. We would consider the experiments presented in this paper to speak only to aleatoric uncertainty. The extraneous concepts activated for pure noise inputs are not a result of "lack of knowledge", but instead a result of the representational systems, shortcuts, and biases instilled in the transformer during its training regimen. This is also why we chose to test hallucinations induced during summarization, as this directly addresses the issue of confabulation despite all the necessary information to complete the task being provided in the prompt. In this way the evaluation of hallucinations is divorced from the inherent "knowledge" of the model and only tests if its internal representational structure of the input has some bearing on the faithfulness of its output.

[Q2] Our interpretation, which we attempt to put forward in the Conclusion, is that vision and language transformers alike attempt to impose structure on the input data even in the absence of true structure, as a function of inductive biases derived from massive pre-training. We do not notice a specific "noise" concept activated for the shuffled image examples. Instead, we qualitatively see a wide variety of concepts being activated, often with little relation to the true semantic label associated with the natural image.

[Q3] Thanks, we added a sentence in the conclusion, mentioning these key aspects as targets for future research. We would expect that increases in temperature and context length would be accompanied by increased hallucination, as there is more room for conceptual wandering to occur [6]. However, we note that for the hallucination experiments in the original text we set the temperature to 0 (as stipulated by the Vectara benchmark that we implemented).

[CLOSING] We are confident that the additional experiments suggested by the reviewer greatly strengthen the robustness and scope of the paper, and provide a more natural connection between the results presented in the main text. We will add these experiments to the Appendix.

[1] Huang et al. "A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions." 2023

[2] Lin et al. "TruthfulQA: Measuring how models mimic human falsehoods." 2021

[3] Joshi et al. "TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension." 2017

[4] Reddy et al. "CoQA: A conversational question answering challenge." 2019

[5] Kang et al. "Unfamiliar finetuning examples control how language models hallucinate." 2024

[6]Liu et al. "Lost in the middle: How language models use long contexts." 2023

[GENERAL] We thank the reviewer for their thorough reading of the paper and their enthusiastic appraisal of our work.

[W1] We will merge references [35] and [36] as per the reviewer’s helpful comment.

[GENERAL] We thank the reviewer for confirming the validity of our experiments and conclusions. We appreciate that the reviewer finds the problems addressed in our paper important, and the conclusions we draw from our experiments to be sound. We also appreciate that the reviewer considers our experiments a proper validation for the mechanistic framework of hallucinations put forward in the paper.

[W1] We note that the Appendix is included as a PDF named “appendix.pdf” in the “Supplementary Material” zip file attached to the OpenReview submission. Within the Appendix, we have provided full details on model specifications, hyperparameters, and training procedures for each transformer model and SAE used in the paper (Appendix A). In addition to the Preliminaries section in the main text, which describes our modeling setup and motivations for each experiment, we have provided additional details for our hallucination experiments in Appendix E.

[W2] Directly addressing the reviewer's point, Appendix A provides the number of input image samples used to train the layer-wise vision SAEs (1.3M images from the ImageNet-1k training set), and the number of tokens used to train the layer-wise language SAEs (204.8M tokens from the FineWeb-Edu corpus). In Section 3 we state the number of validation image samples sourced from the ImageNet-1k validation set: 50,000. For better readability, we will add additional columns to the SAE hyperparameter tables in Appendix A corresponding to the size of the validation sets (50,000 image samples in the case of the vision SAEs, and around 5.12M tokens in the case of the language SAEs).

[Q1] In Section 2.1, we define the “residual stream activations” as follows:

For all transformer models, the object of investigation is the activations in the residual stream: the hidden state of the deep neural network, updated by attention and multi-layer perceptron (MLP) blocks at each layer. The residual stream is the core conduit for information flow and representation refinement to which layers read from and write to, in the transformer architecture.

However, we understand how this may be an unclear definition to readers less familiar with the recent mechanistic interpretability literature. More directly, the residual stream activations refer to the token-level embedding vectors extracted from the transformer model following each layer block [1][2][3]. We will add this additional clarification to Section 2.1.

[Q2] We believe that the reviewer is referencing the number of random Gaussian noise images sampled to train the noise SAE in Section 3. We chose 1.3M as a training set size for this experiment to mirror the size of the ImageNet-1k training set, which is used to train SAEs on natural image inputs for the experiments in Section 4. We will add an extra clarification to the appendix justifying this training set size in the noise setting.

[Q3] The $\mathbf{d}\_i$ are learned. The $\mathbf{d}\_i$ defined in Section 2.2 refer to the columns of the decoder matrix: $D: \mathbb{R}^{d\_\mathrm{SAE}} \to \mathbb{R}^{d\_\mathrm{model}}$. All parameters of this decoder are learned in conjunction with the encoder, using the same loss signal defined in Section 2.2.

[Q4] We thank the reviewer for pointing out this oversight on our part. The dimensionality of both $\mathbf{d}\_i$ and $\mathbf{b}$ should be $\mathbb{R}^{d\_\mathrm{model}}$ not $\mathbb{R}^{d\_\mathrm{SAE}}$. With these fixes, the equation for the reconstruction of the activation sample $\mathbf{x}$ is correct, as $f\_i(\mathbf{x})$ is a scalar and we sum over all $i$ SAE dimensions. We will update the paper to reflect this change, as well as clarify the role of the decoder and its training to address [Q3]. Here is the updated Section 2.2:

For each SAE formulation, we work with SAEs whose parameters are trained to produce the following approximation for a given transformer model activation vector $\mathbf{x}\in \mathbb{R}^{d_\mathrm{model}}$ at a particular layer:

$$\mathbf{x} \approx \sum\_i^{d_\mathrm{SAE}}f_i(\mathbf{x})\mathbf{d}\_i+\mathbf{b}\,$$

that is, each transformer layer’s intermediate activation can be approximated as a (sparse) linear combination of unit direction vectors $\mathbf{d}\_i\in \mathbb{R}^{d\_\mathrm{model}}$ scaled by concept activation coefficients $f_i(\mathbf{x})\geq0$, with $\mathbf{b}\in \mathbb{R}^{d\_\mathrm{model}}$ being the bias term. Practically speaking, this SAE is a simple neural network with a learned encoder $F: \mathbb{R}^{d\_\mathrm{model}} \to \mathbb{R}^{d\_\mathrm{SAE}}$ projecting each transformer activation into a sparse concept space, typically of much larger dimensionality than the transformer model activation space. For instance, the ViT embedding dimensionality is $d\_\mathrm{model} = 768$ with the corresponding SAE concept space dimension being $d\_\mathrm{SAE} = 49,152$. Conversely, the learned decoder $D: \mathbb{R}^{d\_\mathrm{SAE}} \to \mathbb{R}^{d\_\mathrm{model}}$ component of the SAE aims to reconstruct the original transformer residual stream activation solely from this pattern of sparse concept activations.

[CLOSING] Experimental details provided in the Appendix, as well as the clarifications of terminology and math notation detailed above should address the reviewer’s outstanding concerns. Thanks to the reviewer, the additional clarity and modeling details will greatly improve the readability of the paper.

[1] Elhage et al. "A mathematical framework for transformer circuits." 2021

[2]Elhage et al. "Privileged bases in the transformer residual stream." 2023

[3] Cunningham et al. "Sparse autoencoders find highly interpretable features in language models." 2023

We are happy to provide concrete details for our modeling architectures and analysis setup, which we will summarize in our revised NeurIPS paper. We recognize that we could have made more explicit the distinction between the model under study (the transformer) and the experimental tool we use to explore it (the sparse autoencoder).

The “main models” of study in this paper are OpenCLIP ViT-B/32 (vision transformer) and Pythia-deduped-160M / Gemma-2B-IT (language transformer). Note that these transformer models were not trained or finetuned as part of our work, but were instead sourced as pre-trained models from Hugging Face:

At the output of each transformer layer, the token embeddings $\mathbf{x} \in \mathbb{R}^{d_\mathrm{model}}$​ are known as “residual stream activations” in the LLM explainability literature. These dense embeddings are hard to interpret directly due to entangled (“overlapping”) internal representations. We therefore use sparse autoencoders (SAEs) to map residual activations into a higher-dimensional sparse concept space, $\mathbb{R}^{d_\mathrm{model}} \to \mathbb{R}^{d_\mathrm{SAE}}$ with $d_\mathrm{SAE} \gg d_\mathrm{model}$, i.e., an overcomplete latent space rather than a bottleneck. The encoder applies a linear map followed by ReLU: $f(\mathbf{x}) = \mathrm{ReLU}(F\mathbf{x})$. The entries of $f(\mathbf{x})\geq 0$ constitute the concept activation vector: component $f_i$ reports the presence and magnitude of concept $i$ contained in the activations $\mathbf{x}$. The linear decoder $D$ reconstructs the activation, $\hat{\mathbf{x}} = Df(\mathbf{x}) + \mathbf{b}$, whose columns $\mathbf{d}_i$ define the concept directions. We train SAEs end-to-end with the loss

$$\mathcal{L} = ||\mathbf{x} - \hat{\mathbf{x}}||^2_2 + \lambda ||f(\mathbf{x})||_1,$$

where the first term ensures accurate reconstruction of the original input activation and the $L_1$ term (tunable hyperparameter $\lambda$) encourages sparsity in the concept space. This yields a representation in which each residual activation is expressed as a sparse linear combination of learned concepts. (Shapes: encoder $F \in \mathbb{R}^{d_\mathrm{SAE} \times d_\mathrm{model}}$, decoder $D \in \mathbb{R}^{d_\mathrm{model} \times d_\mathrm{SAE}}$, bias term $\mathbf{b}\in \mathbb{R}^{d_\mathrm{model}}$)

To address our goal of unpacking the internals of our transformer models (cf. table above), each SAE is trained independently on a single transformer layer’s residual stream activations—e.g., for a 12 layer transformer, we train 12 distinct SAEs on the residual stream activations output from the corresponding layer. In this work, we train 3 sets of SAEs, and source one set of pre-trained SAEs from Hugging Face.

Below, we provide the training and evaluation details for each set of SAEs. Specific hyperparameters for training all SAEs can already be found in Appendix A.

Data used to fit SAEs

[Section 3]: Concepts identified in noise activations from OpenCLIP ViT-B/32. SAEs trained on 1.3M i.i.d. Gaussian noise image (224×224) activations, validation on 50k ImageNet-1k images, 12 SAEs for all 12 layers. $d_\mathrm{SAE}=49,152$.

[Section 4]: More concepts from transformer activations of increasingly disordered inputs. OpenCLIP ViT-B/32—trained on 1.3M ImageNet-1k activations, validation on 50k ImageNet-1k activations with shuffled patches of various sizes, 12 SAEs for all 12 layers. Pythia-deduped-160M—trained on 204.8M FineWeb-Edu tokens of normal text, validation on 5.12M FineWeb-Edu tokens with shuffled $n$-grams of varying $n$, 12 SAEs for all 12 layers. $d_\mathrm{SAE}=24,576$.

[Section 5]: Hallucination prediction and suppression from concepts identified in activations of Gemma-2B-IT. SAEs pre-trained on 1.23B token activations from FineWeb, validation on 1,006 Vectara benchmark document activations, pre-trained SAEs sourced from Hugging Face for layers $1,7,11,13,17$. $d_\mathrm{SAE}=16,384$.

We appreciate the careful review. We believe that this compact description of our main modeling pipeline will improve reader clarity.