NoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Models

Q2: It is unclear what makes attention heads special in this context. Similar experiments could potentially be conducted using MLP layers.

A2: Thank you for your suggestion. We are happy to present our experiments on both MLP (FFN) and QKVO states. The results show an average 23.9 point loss in accuracy when using MLP norms. This is improved to an average 12.5 point loss when using multi-headed states Q and V. Head norms outperform all representations in all datasets.

There are two reasons why attention head norms are good in this context.

1. Multi-Heads are Fine-Grained A measure as coarse and broad as the L2 norm would work much better on more fine-grained (multi-headed) representations of each token, rather than monolithic ones. This is supported by experimental results, which shows lower accuracies for representations that are not split into multiple heads (K, O, FFN1, and FFN2), with FFN and O vector norms having the lowest scores.

2. Heads are the Final Representation of Information The head is the final output of the information retrieval process described by the QKV operation. Here, Q and V are intermediate outputs used to retrieve relevant information; their states are not important in isolation. This is supported by experimental results, which shows better accuracies due to their multi-headed nature, but still lower than the head.

$^{1}$ Experiments are conducted on Mistral-7b-Instruct, which does not use multi-headed K vectors.

Q3: The authors only found a correlation between the norms of the heads and truthfulness. However, it is not explained why such a correlation exists or if there is any causal relationship between the two.

A3: There are three reasons why such a correlation could exist:

1. Proposition Completion We find evidence that head norms are correlated with truth because of the completion of factual propositions. We see that head norms spike significantly in token positions where the proposition is complete (Lines 389-402, 430-435, 954-1011). Taking the head norms prematurely before completion results in catastrophic loss of performance (Lines 516-527).

2. Factual Association We find evidence that head norms are correlated with truth because of the attention towards specific factual associations, such as between ‘pigs’ and ‘fly’. We see that head norms spike significantly in token positions where these associations occur (Lines 389-402, 416,420, 954-1011). Perturbing these associations with character insertions causes catastrophic loss of performance, and vice versa (Line 522).

3. Self-organisation in High-Dimensional Space Representations are found to self-organise in human-meaningful clusters, with mnist digits clustering based on stroke similarity [1], and word representations clustering based on human constructs such as gender [2]. It is plausible that truthful statements are self-organised as well. Small semantic shifts in truth can lead to large but organised movements in high-dimensional space [3], causing points to lay in a predictably different hypersphere surface. This mechanism is a possible reason why truth and L2 head norms are correlated.

We will supplement these analyses in the revised version.

[1] Self-classifying MNIST Digits, accessed online at https://distill.pub/2020/selforg/mnist/.
[2] Efficient Estimation of Word Representations in Vector Space, Arxiv 2013.
[3] Semantics in High-Dimensional Space, Frontiers 2021.

Q1: This approach has very limited applicability, being useful only for multiple-choice tasks. Therefore, it should be noted that this is not a generic hallucination reduction technique like existing methods such as Truthx.

A1: MCQ tasks are not a limitation of our method but rather are used as an evaluation tool to explore and confirm any fundamental internal state misalignments. There are four applications of our solution beyond MCQ tasks:

1. Ranked Generation Our method can be applied beyond MCQs to other task types, such as ranking multiple candidate text spans either generated, or retrieved, by the LLM. Our analysis reveals that head norms spike significantly (up to 83%) in token positions where the proposition is complete (Lines 516-527, 389-402, 430-435, 954-1011). This phenomenon can be used to rank the truthfulness of candidate text spans during open-ended generation.

2. In-situ Hallucination Detection Our findings can be applied beyond MCQs to showcase a more fundamental problem: common and fluent falsehoods are strongly indicative of internal state misalignments. TruthfulQA is a benchmark specifically crafted to mislead LLMs to output common and fluent falsehoods. We show an average 24 point accuracy gain across eight models$^{1}$ in TruthfulQA with head norms, over the language likelihood (Lines 291-298). This difference can be used in hallucination detection on-the-fly, due to its internally encapsulated and lightweight nature, especially for factually incorrect but fluent outputs.

3. Adversarial Defence Our method can be applied beyond MCQs to improve model robustness against textual attacks. An average 6.6 median accuracy gain across four models on all six subtasks of AdversarialGLUE showcase the potential of head norms for building textual adversarial defence (Lines 279-282, 287, 317-323).

4. Finetuning Our method can be applied to improve finetuning accuracy on general tasks beyond MCQs, such as during the alignment stage for better open-ended generations. Experiments on head-norm-finetuning with DeBERTa show an average 0.9 point accuracy gain over standard feature vector fine-tuning, across all nine datasets.

$^{1}$ The evaluation of four new models have been added, doubling the original number from four to eight.

Q5: Out-of-distribution experiments are missing. The authors should conduct experiments where NoVo is calibrated on one dataset and tested on another.

A5: We conduct out-of-distribution (OOD) experiments on NoVo-Mistral-7B-Instruct. Here, NoVo is calibrated on one dataset and tested on another. We calibrate on ten datasets: Arc-Easy, CQA2, QASC, SWAG, HSwag, SIQA, PIQA, Cosmos, CICv1, and CICv2 in a pairwise manner, and record the standard deviations of accuracies. The standard deviation reflects the spread of the OOD accuracy of a dataset when calibrated on various other datasets.
 

The accuracy standard deviation show that OOD performance is mostly dependent on the evaluated dataset, rather than our method. For example, our method can be calibrated significantly OOD on multi-turn CICv1 dialogues, sentence completion SWAG, or factual-style Arc-Easy, and evaluated for CQA2 commonsense yes-no questions with only a 0.59 standard deviation. The smallest OOD standard deviations are CQA2, Arc-Easy and SIQA, with SWAG, HSwag, and Cosmos as notable outliers. We note that we specifically chose to maximise the OOD difference between each dataset, by manually checking that question topics and formats are highly varied, to ensure that our method can truly generalise. We also note that the SOTA performance achieved by NoVo on TruthfulQA was calibrated OOD on Arc-Easy.

Q6: I think the authors should choose between two options. They can either convert their observation about norms and truthfulness into a generic algorithm that works for open-ended generations, or they can deeply explore the reasons behind the observed correlation and write an interpretability paper. Another option might be to reformulate the problem as addressing selection bias or improving multi-choice scenarios, similar to [1].

A6: We are happy to note that our paper’s contributions are well-aligned with your suggestion. NoVo addresses the selection bias and improves multi-choice scenarios (Lines 85-87). NoVo enables LLMs to do better in multi-choice scenarios via two mechanisms: First NoVo circumvents the selection bias problem identified in [1] by operating on entire text spans, rather than letter options. Second NoVo improves robustness to common falsehoods, by avoiding the popular use of log likelihoods in multi-choice scenarios (Line 55). If we placed our paper side-by-side with [1]:

Zheng et al. ICLR 2024 [1]

1. Viewed poor MCQ performance through the lens of selection bias.
2. Formulated the selection bias problem as an a priori probabilistic problem.
3. Proposed PriDe to separate the prior and prediction distribution, to improve MCQ, resulting in ~1.5% improvement in 2 datasets.
4. Provides an interpretive exploration of the selection bias problem.

Our Paper

1. View poor MCQ performance through the lens of hallucinations.
2. Formulated the hallucination problem as an internal state misalignment problem.
3. Proposed NoVo to operate directly on internal states (head norms) to improve MCQ, resulting in ~10% improvement across 21 datasets.
4. Provides an interpretive exploration of the internal misalignment problem.

Zheng et al. contributed to the selection bias problem using MCQ tasks to demonstrate their hypothesis. We contributed to the internal-state misalignment problem using MCQ tasks to demonstrate our hypothesis.

We will clarify this concern in the revision of our related works section.

[1] Large Language Models Are Not Robust Multiple Choice Selectors, ICLR 2024

Q4: The direct relationship between the norm of a head and truthfulness is not well-explained. Why should a correlation be expected in the first place? And why use the L2 norm?

A4: We list three reasons why a correlation was expected, and why to use the L2 norm.

1. Evidence of Correlation in Prior Works Previous studies by Burns et al. [1], Azaria et al. [2], Zou et al. [3], Li et al. [4], and Chen et al. [5] demonstrated that the latent representations of LLMs can be linearly classified into true-false clusters. While many of these works focused on the MLP, Li et al. extended this to individual attention heads, confirming that certain heads encode truth-related features. These findings strongly suggest that some LLM hidden states self-organise around truthfulness, motivating us to explore whether L2 norms of specific heads similarly reflect this pattern.

2. Use of L2 Norms in Other Domains In computer vision, studies have shown that the L2 norm of feature vectors in the final layer of CNNs correlates with perceived image quality [6,7]. Poor-quality images result in weaker, sparser vector representations [8]. Drawing from this analogy, we hypothesised that intermediate L2 norms in transformers might similarly capture the strength of features relevant to "truthfulness" in language models, even across layers and architectures.

3. Conceptual Basis of Representation Learning Transformers encode diverse language features in their intermediate representations, which often self-organise along meaningful dimensions [9-11]. It is plausible that one of these dimensions reflects the alignment of propositions with reality (i.e., truthfulness) [12]. For instance, features related to coherent concepts (e.g., "plane - passenger - ticket") might express stronger magnitudes in these representations, leading to higher (or, if negatively correlated, lower) L2 norms in certain attention heads.
    

[1] Discovering Latent Knowledge in Language Models Without Supervision, ICLR 2023
[2] The Internal State of an LLM Knows When It’s Lying, EMNLP 2023
[3] Representation Engineering: A Top-Down Approach to AI Transparency, Arxiv 2023
[4] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, NeurIPS 2023
[5] Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning, AAAI 2024.
[6] Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework, CVPR 2017
[7] Deep Objective Quality Assessment Driven Single Image Super-Resolution, IEEE-TMM 2019
[8] Wang, Z., & Bovik, A. C. (2006). Modern Image Quality Assessment. In Synthesis lectures on image, video, and multimedia processing. https://doi.org/10.1007/978-3-031-02238-8.
[9] Girolami, M. (1999). Self-Organising Neural Networks. In Perspectives in neural computing. https://doi.org/10.1007/978-1-4471-0825-2
[10] Li, Ping. "Language acquisition in a self-organising neural network model." Connectionist Models of Development. Psychology Press, 2004. 112-142.
[11] Self-classifying MNIST Digits, accessed online at https://distill.pub/2020/selforg/mnist/
[12] Efficient Estimation of Word Representations in Vector Space, Arxiv 2013

Q8: May you also share the full table of results instead of only stds?

A8: We are happy to present the full OOD table replicated from the paper. We note that calibration (i.e. zeroth order training) works within reasonable OOD limits (i.e. TruthfulQA is calibrated on Arc-Easy) and requires very little data. There are obvious limitations when calibrating to a significantly OOD dataset (multi-turn dialogue CICv1 to ambiguity-focused PIQA). This limitation has been clarified in the paper.

Q9: It would be really good if you included [1] in your benchmark.

A9: We have included [1] in our evaluations in a new revision of our paper. We reproduce the table here for your convenience.

Q10: I will be happier if the authors bravely discuss all the limitations of their proposed algorithm in the paper.

A10: We have added a limitation section in the paper per your suggestion:

We dedicate this section to clearly explain the limitations of our work. (1) While ranked generation is possible, it is currently not yet possible to apply head norms to single-span generation. Our formulation of head norms is relative, and therefore requires several candidate texts to work. (2) NoVo is not a replacement for generic generative REAR methods such as ITI and DoLA. NoVo only outperforms on the multiple-choice aspect. (3) It is unclear if the success of DeBERTA head-norm-finetuning (+NoVo) applies to decoder or encoder-decoder architectures. (4) Despite our novel interpretation of continuous truthful hyperspheres in attention heads, we do not claim to narrow down any specific truthful subspace for further interpretation. (5) As mentioned in Appendix B, Norm Selection only works within reasonable out-of-distribution (OOD) limits, with the best being in-domain samples.

May you also share the full table of results instead of only stds?

It would be really good if you included [1] in your benchmark.

I will increase my score to 5 but I am still on the rejection side (you can think my score is 4 but there is no such an option) unless other reviewers and/or authors convince me. I appreciate the efforts of the authors and the detailed experimental results in their paper. I will be happier if the authors bravely discuss all the limitations of their proposed algorithm in the paper. A good paper doesn't have to solve all problems in an area. A paper showing its limitations with comprehensive benchmarks is what our community needs. I know we feel pressure to make big improvements in the paper to get it accepted that's why we tend to overclaim in the papers but this only hurts the scientific progress.

Dear Reviewer a4Cj,

The uploaded PDF will highlight all changes from the initial version, so kindly allow us to summarise the second-round revision of the paper here, based on your second-round feedback.

Line Number: Comment
053: explicit clarification that our method is for multi-choice scenarios.
086: explicitly stated that ranked generation is for future works.
098: clarified REAR methods that are generic hallucination mitigation techniques, to avoid confusing readers.
110: clarified REAR methods that are generic hallucination mitigation techniques, to avoid confusing readers.
113: explicit clarification that our method is for multi-choice scenarios.
377: explicitly stated that ranked generation is for future works.
380: explicitly stated that finetuning on decoder models for alignment is for future works.
1138: Explained why Out norms factuality performs falters when compared to head.
1179: Included evaluation for [Large Language Models Are Not Robust Multiple Choice Selectors].
1183: Included a section detailing limitations of our work.

Q4: In Table 2, very different results are obtained for CICv1 and CICv2. Is there any explanation for why the results differ so much here, or in general why the results vary so much across datasets?

A4: The reason why results vary across datasets can be attributed to three types of questions:

1. Questions with Strong Stereotype Assumptions The first type of question requires strong stereotypes to disambiguate equally likely answers (Section 4.3). We find that our method avoids assuming strong stereotypes, which can be helpful in ambiguous questions. For example, in PIQA, when asked whether a book or milk should be placed on the shelf. Both are correct, but the strong assumption here (and ground truth answer) is ‘book’. A larger proportion of these questions can hurt results.

2. Questions with Identical Answers The second type of questions admits nearly identical answers, for example: choosing between “no”, and “no, he did not”. For these questions, the correct answer can vary widely, which causes unpredictable result fluctuations.

3. Questions with Misleading Phrasing The third type of questions contains misleading phrasings and tries to elicit fluent falsehoods, such as is mostly found in TruthfulQA. Our method avoids assuming strong stereotypes and hence performs well for these types of questions. A larger proportion of these questions can improve results.

Q5: What is the variability of results when using different sets of randomly drawn samples for norm selection?

A5: Random variations experiments are conducted over 200 runs, across all ten datasets (TruthfulQA, CQA2, QASC, SWAG, HSwag, SIQA, PIQA, Cosmos, CICv1, and CICv2) and across all four models (10 x 4 = 40 reports, each 200 runs). Standard deviations are all within 1.5 points, with the exception of Llama2-7b-Cosmos at 1.64, and Vicuna-7b-QASC at 1.53. Interquartile ranges are all within 2.3 points. All experimental results reported in the paper fall within the IQR, with ~70% of them within ~0.5 points from the median. Our random variation experiments show that there is no over-reporting of results. As the full experiment table is quite large, we kindly direct the reviewer to Appendix D for more details.

Q2: These experiments are done exclusively on multiple-choice QA datasets, and it seems that it would not be possible to use this method to reduce hallucination in open-ended generation.

A2: Our method can be reformulated to adapt to open-ended generation. During generation, the LLM can either decode or externally retrieve multiple candidate text spans. Our analysis reveals that head norms spike significantly by up to 83% when compared to a different candidate text, in token positions where the proposition is complete (Lines 516-527, 389-402, 430-435, 954-1011). This phenomenon can be used to rank the truthfulness of candidate text spans during open-ended generation.

In addition, experiments on AdversarialGLUE where our method achieves an average 6.6 median accuracy gain across all models on all 6 subtasks, and head-norm-finetuning on DeBERTa, where our method achieves an average 0.9 point gain across all models on all 9 datasets, show that our method can play a key role outside of MCQ tasks: improving adversarial defence and finetuning accuracy during alignment, for more desirable open-ended generations.

Q3: Evaluating on a wider range of models would help provide more confidence in the method, especially testing on currently popular models like Llama 3.

A3: Thank you for your suggestion. We are happy to present additional evaluations on four more currently popular models, doubling the number of evaluated models from 4 to 8. We included varying sizes ranging from 3.8b to 9b, and ensured a good mix of instruction-tuned, chat-tuned, and based pretrained models. Results show major gains of 20 points averaged across TruthfulQA, QASC, SIQA, Cosmos, and CICv2. Moderate gains of 3.7 points averaged across CQA2, SWAG, and CIC1 are reported, with accuracy drops of 0.7 points averaged across HSwag and PIQA being reported. We note that these performance characteristics are largely similar to the original models.

Q1: While the method is simple, I find the conceptual motivation unclear. The authors posit that for some heads, the L2 norm is correlated to the truthfulness of a sequence. Why? What makes this a reasonable thing to expect, conceptually?

A1: We list three reasons why this was a reasonable thing to expect, conceptually.

1. Evidence from Prior Works Previous studies by Burns et al. [1], Azaria et al. [2], Zou et al. [3], Li et al. [4], and Chen et al. [5] demonstrated that the latent representations of LLMs can be linearly classified into true-false clusters. While many of these works focused on FFN outputs, Li et al. extended this to individual attention heads, confirming that certain heads encode truth-related features. All these findings strongly suggest that some LLM hidden states self-organise around truthfulness, motivating us to explore whether L2 norms of specific heads similarly reflect this pattern.

2. Insights from L2 Norms in Other Domains In computer vision, studies have shown that the L2 norm of feature vectors in the final layer of CNNs correlates with perceived image quality [6,7]. Poor-quality images result in weaker, sparser vector representations [8]. Drawing from this analogy, we hypothesised that intermediate L2 norms in transformers might similarly capture the strength of features relevant to "truthfulness" in language models, even across layers and architectures.

3. Conceptual Basis in Representation Learning Transformers encode diverse language features in their intermediate representations, which often self-organise along meaningful dimensions [9-11]. It is plausible that one of these dimensions reflects the alignment of propositions with reality (i.e., truthfulness) [12]. For instance, features related to coherent concepts (e.g., "plane - passenger - ticket") might express stronger magnitudes in these representations, leading to higher (or, if negatively correlated, lower) L2 norms in certain attention heads.

[1] Discovering Latent Knowledge in Language Models Without Supervision, ICLR 2023
[2] The Internal State of an LLM Knows When It’s Lying, EMNLP 2023
[3] Representation Engineering: A Top-Down Approach to AI Transparency, Arxiv 2023
[4] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, NeurIPS 2023
[5] Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning, AAAI 2024.
[6] Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework, CVPR 2017
[7] Deep Objective Quality Assessment Driven Single Image Super-Resolution, IEEE-TMM 2019
[8] Wang, Z., & Bovik, A. C. (2006). Modern Image Quality Assessment. In Synthesis lectures on image, video, and multimedia processing. https://doi.org/10.1007/978-3-031-02238-8.
[9] Girolami, M. (1999). Self-Organising Neural Networks. In Perspectives in neural computing. https://doi.org/10.1007/978-1-4471-0825-2
[10] Li, Ping. "Language acquisition in a self-organising neural network model." Connectionist Models of Development. Psychology Press, 2004. 112-142.
[11] Self-classifying MNIST Digits, accessed online at https://distill.pub/2020/selforg/mnist/
[12] Efficient Estimation of Word Representations in Vector Space, Arxiv 2013

Dear Reviewer GDLP,

Thank you again for your comments, effort and time spent reviewing this paper.

We gently remind the reviewer that the discussion stage is ending. We deeply appreciate your valuable insights in helping us improve our manuscript's quality. Once this stage ends, we will be unable to respond.

Kindly let us know if you have further comments about this paper. We look forward to your suggestions!

Regards,
Authors of Submission 3392

Q3: Authors claim "Hallucinations in Large Language Models (LLMs) remain a major obstacle, particularly in high-stakes applications where factual accuracy is critical". Are the benchmarks used in this study representative of the same?
A3: Yes, our diverse range of benchmarks is representative of the different facets of factual hallucinations, a critical issue in high-stakes applications [1-3]. Our method was designed to be generalisable and easily applied to a wide range of specialised datasets, such as those designed for strategic reasoning in multi-turn dialogues (Line 257), textual adversarial attacks (Line 259), domain-specific knowledge testing (Lines 255, 260-263, and causal reasoning over narrative contexts (Line 256). The benchmarks used in this study show that our method enables LLMs to elicit the correct factual knowledge under adversarial or misleading contexts, which is crucial for high-stakes applications.
 
[1] Capabilities of GPT-4 on Medical Challenge Problems, Arxiv 2023.
[2] Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digital Health 2, 2023.
[3] Large language models encode clinical knowledge, Nature 2023.

Q2: The motivation to solve MCQ type questions is not clear. Why is it very important to solve?
A2: There are five reasons why solving MCQ type questions is important and critical.

1. A Fundamental Task According to Bloom’s Taxonomy of cognitive pedagogy [1], recognising and understanding the correct answer are the two most fundamental learning outcomes. These are present in MCQ tasks. To analyse, apply and synthesise are higher-level outcomes which are present in open generation. Fundamental MCQ benchmarks like TruthfulQA remain unsolved even with models like GPT4. Furthermore, there is scrutiny in a LLM’s true ability to solve generative tasks [2]. Before moving up to more advanced outcomes, It is critical to solve MCQ tasks to determine an LLM’s fundamental capabilities.

2. Real-life Applicability MCQ tasks are used in the real high-stakes applications as a key component of standardised assessments for professional certifications and regulatory compliance [3-6]. These MCQs require specific formatting to ensure clarity and fair comparability, whereas open-ended generation could introduce interpretative ambiguity. Solving MCQ tasks in LLMs is an important step towards their use in real-world, high-stakes scenarios, where standardised testing is necessary for regulatory compliance and trustworthiness.

3. Academic Significance MCQ tasks allow us to objectively evaluate the factual knowledge, reasoning, and understanding skills of an LLM with a direct and unified metric (accuracy). By choosing MCQ tasks as an evaluation tool, we are able to assess our observations of internal state misalignment across significantly more varied benchmarks, models, and methods, without needing any nuisance control over prompting and evaluation techniques. It is important to solve these MCQ benchmarks to directly and objectively validate our approach versus the rest.

4. Technically Challenging MCQ tasks are equally challenging to solve when compared to generative tasks. Well-designed MCQs can include answer distractors, ambiguity, and misleading phrasing to confuse LLMs. The constrained output space of MCQ tasks might seem easier when compared to generating novel responses, but performance on benchmarks do not reflect this. Measuring letter option probability in the logit space is prone to selection bias [7], while finetuning on MCQ examples causes overfitting and forgetting [8]. The canonical way is to select the answer option with the highest language likelihood [9], but accuracy is still far from human performance in some benchmarks like TruthfulQA [10]. It is important to solve these technical challenges posed by MCQs to better understand and overcome these fundamental limitations of LLMs.

5. Benefits for Generation Success on MCQ tasks reflects an LLM's ability to recognise coherent, reasonable and factual statements, under challenging questions. These abilities form the foundation for open-ended generation, whereby similar capabilities are required but without the explicit constraints of predefined options. Solving MCQ tasks thus serves as a stepping stone and evaluation method for enhancing performance in more complex generative scenarios.
    

[1] Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals, Handbook I: Cognitive Domain.

[2] GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models, Arxiv 2024.

[3] Occupational Safety and Health Administration, Hazard Identification and Assessment, accessed online at https://www.osha.gov/safety-management/hazard-identification.

[4] Med-HALT: Medical Domain Hallucination Test for Large Language Models, EMNLP 2023.

[5] Multistate Bar Examination, National Conference of Bar Examiners, accessed online at https://www.ncbex.org/exams/mbe.

[6] Graduate Record Examinations, Educational Testing Service, accessed online at https://www.ets.org/gre.html.

[7] Large Language Models Are Not Robust Multiple Choice Selectors, ICLR 2024.

[8] An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning, Arxiv 2024.

[9] Language Models are Unsupervised Multitask Learners, OpenAI Blog 2019.

[10] TruthfulQA MC1 https://paperswithcode.com/sota/question-answering-on-truthfulqa.

Q4: It is not very clear how the benchmarks help in high-stakes applications and it is very difficult to assess the impact of the work.

A4: Our contributions go beyond specific benchmark performances to showcase and tackle fundamental issues that can arise in high-stake scenarios. The impact of our work with these benchmarks can be summarised in four points

1. Using Benchmarks to Demonstrate Internal Misalignments Our results on various benchmarks show that internal misalignments contribute to factual hallucination across a wide and diverse range of topics. We also show that the log likelihood tends to favour fluent but incorrect output, and that this limitation can be addressed with head norms. Our benchmark results indicate that this undesirable phenomenon can arise even in high-stake scenarios, where the cost of hallucinations is much higher.

2. Tackling Inherent Weaknesses in LLMs on an unsolved benchmark TruthfulQA MC1 remains unsolved at 60% even with GPT4, due to its misleading questions. Our method showcases that this limitation of LLMs can be addressed with internal states (head norms). Accuracy gains on this benchmark is not only academically significant but also serves as a critical solution in high-stakes scenarios, where misleading questions can have expensive and harmful consequences.

3. Applicability to High-Stakes Testing In high-stakes applications, it is crucial for LLMs to undergo standardised testing for regulatory compliance as part of the trustworthiness framework [1]. These tests often contain MCQ questions to ensure clear and fair evaluations [2-5]. Our results on a wide variety of benchmarks showcase the generalizability of our method on different MCQ domains and formatting styles, which is useful when translating to high-stakes, real-world unconstrained MCQ styles.

4. Potential for Adversarial Defence We report an average 6.6 median accuracy gain across four models on all six subtasks of AdversarialGLUE. These results indicate the potential for head norms to be used to enhance textual attack robustness. This benchmark is particularly relevant in high-stakes scenarios, where bad-faith actors can inject adversarial prompts to elicit undesirable outputs.

Q5: Another question that arises is - how critical is MCQ based tasks? should we even solve it?

A5: MCQ based tasks are critical to solve, because they are:

• Fundamental tasks that reflect low-level learning outcomes.
• Applicable in real-life standardised testing for regulatory compliance of LLMs.
• Academically significant in objective comparison between different approaches.
• Technically challenging tasks that probe at a LLM’s limitations.
• Offer foundational benefits for open-ended generation.

We gently refer the reviewer to the more detailed Answer 2 (A2) to Question 2 (Q2).

 
[1] Biswas, A., & Talukdar, W. (2023). Guardrails for trust, safety, and ethical development and deployment of Large Language Models (LLM). Journal of Science & Technology, 4(6), 55-82.
[2] Occupational Safety and Health Administration, Hazard Identification and Assessment, accessed online at https://www.osha.gov/safety-management/hazard-identification.
[3] Med-HALT: Medical Domain Hallucination Test for Large Language Models, EMNLP 2023.
[4] Multistate Bar Examination, National Conference of Bar Examiners, accessed online at https://www.ncbex.org/exams/mbe.
[5] Graduate Record Examinations, Educational Testing Service, accessed online at https://www.ets.org/gre.html.

Q1: While the paper presents detailed analysis and shows impressive gains across benchmarks, the applicability of the solution beyond the MCQ type of problems is not obvious.

A1: MCQ tasks are not a limitation of our method but rather are used as an evaluation tool to explore and confirm any fundamental internal state misalignments. There are four applications of our solution beyond MCQ tasks:

1. Ranked Generation Our method can be applied beyond MCQs to other task types, such as ranking multiple candidate text spans either generated, or retrieved, by the LLM. Our analysis reveals that head norms spike significantly by up to 83% when compared to a different candidate text, in token positions where the proposition is complete (Lines 516-527, 389-402, 430-435, 954-1011). This phenomenon can be used to rank the truthfulness of candidate text spans during open-ended generation.

2. Hallucination Detection Our findings can be applied beyond MCQs to showcase a more fundamental problem: common and fluent falsehoods are strongly indicative of internal state misalignments. TruthfulQA is a benchmark specifically crafted to mislead LLMs to output common and fluent falsehoods. We show an average 24 point accuracy gain across eight models$^{1}$ in TruthfulQA with head norms, over the language likelihood (Lines 291-298). This difference can be used in hallucination detection tasks, especially for factually incorrect but fluent outputs.

3. Adversarial Defence Our method can be applied beyond MCQs to improve model robustness against textual attacks. An average 6.6 median accuracy gain across four models on all six subtasks of AdversarialGLUE showcase the potential of head norms for building textual adversarial defence (Lines 279-282, 287, 317-323).

4. Finetuning Our method can be applied to improve finetuning accuracy on general tasks beyond MCQs, such as during the alignment stage for better open-ended generations. Experiments on head-norm-finetuning with DeBERTa show an average 0.9 point accuracy gain over standard feature vector fine-tuning, across all nine datasets.
    

$^{1}$ The evalutions of four new models have been added, doubling the number of models from the initial four to eight.

Dear Reviewer 3eKd,

Thank you again for your comments, effort and time spent reviewing this paper.

We gently remind the reviewer that the discussion stage is ending. We deeply appreciate your valuable insights in helping us improve our manuscript's quality. Once this stage ends, we will be unable to respond.

Kindly let us know if you have further comments about this paper. We look forward to your suggestions!

Regards,
Authors of Submission 3392

Dear Reviewer 3eKd,

We understand you might be busy! We would be deeply grateful if you could kindly take a moment to share your thoughts. Your acknowledgment and further feedback would mean a great deal to us.

• We summarise our responses and paper revisions to address all your four (4) concerns:

1. Applicability Beyond MCQs?
Response Our solution can potentially be applied beyond MCQ tasks to ranked generation, hallucination detection, adversarial defence and finetuning for alignment.
Revision Detailed how our solution can be applied beyond MCQ tasks in Sections 1 and 4.4.

2 Why Solve MCQs?
Response Solving MCQs is crucial because they probe fundamental and low-level learning outcomes, are commonly used in high-stakes-scenario standardised assessments for regulatory compliance, provides a unified and objective metric across a wide range of benchmarks for fair comparison against other methods, are technically challenging tasks that reveal fundamental limitations of models, and forms the foundational towards more complex generative scenarios.
Revision Detailed our motivation to solve MCQ tasks and explained why MCQs are critical in Section 1.

3 Do your Benchmarks Represent High-Stake Scenarios?
Response Yes, our diverse range of benchmarks comprehensively probes different facets of factual hallucinations, a critical issue in high-stakes scenarios. This includes strategic reasoning in multi-turn dialogues, textual adversarial attacks, general and scientific knowledge, causal reasoning over narratives, social and physicality reasoning, and responding under misleading contexts. Good performance in these benchmarks reflect crucial skills for high-stakes scenarios.
Revision Explicitly showed how our benchmark performance are representative of high-stakes applications in Section 1.

4 Impact of Your Work?
Response We demonstrate a fundamental problem: internal state misalignments with the language likelihood, which impacts future works in deeply addressing hallucinations. We achieve a significant 30-point accuracy gain in an unsolved benchmark and generalize to 20 more datasets, which impacts future efforts in devising more accurate and generalizable methods. Our benchmark results can impact real-world high-stakes scenarios, where models must undergo standardised MCQ testing for regulatory compliance as part of the trustworthiness framework. Our results on AdversarialGLUE and finetuning may impact future works in adapting our method for adversarial defence and safety-alignment finetuning.
Revision Added an impact statement which contextualized our benchmark performances to real-world high-stake scenarios in Section 1.
$~$
We’d be delighted to receive any further thoughts and perspective from you.
Thank you so much for your time and consideration.

Authors of Submission 3392