Investigating Factuality in Long-Form Text Generation: The Roles of Self-Known and Self-Unknown

Thank you for the review and helpful comments!

The statistical observations are based on an existing dataset of biography from Min et al., 2023, which impairs the contribution of this work.

We appreciate the reviewer’s concern regarding the use of an existing dataset, and we understand the importance of distinguishing our contributions. While we did use the dataset from Min et al. (2023) for our experiments, we believe our work still makes a significant contribution in the following ways:

• Our primary contribution lies not in the dataset itself but in the novel findings to the factuality evaluation of long-form text generation. By analyzing self-known and self-unknown metrics and their correlation with factuality errors, we provide new insights into how LLMs handle factuality across different segments of long-form text, which has not been explored in this manner before.

• The proposed mathematics linking from factuality to self-known and self-unknown scores, and the related empirical results — offers a quantifiable method to assess the accuracy of generated content. This approach provides a fresh perspective on evaluating LLM outputs and sets the stage for future improvements in error detection and correction.

• We agree with the reviewer that validating the findings on additional datasets strengthens the robustness of the conclusions. However, there is a limited availability of human-annotated, long-form LLM factuality datasets. We aim to apply this to more diverse datasets in future work to strengthen the generalizability of our findings.

Factuality scores tend to decline in later sentences is obvious and not exciting.

We appreciate the reviewer’s observation regarding the tendency of models to exhibit more factuality errors in the later parts of their outputs. Our findings confirm that current state-of-the-art LLMs, such as GPT-4 and Claude-3-Opus, still face this issue. Our work aims to advance this understanding by providing a quantitative analysis that correlates these errors with Self-Known and Self-Unknown scores, as detailed in Equation (3). This analysis introduces a novel perspective by mathematically linking these metrics to factuality, offering deeper insights into the underlying causes of these errors and potential mitigation strategies. Additionally, we observe that for advanced LLMs, the Self-Known scores show minimal variation across positions. However, factuality remains lower in later sections, suggesting a strong positional bias in generation. In the RAG experiments, even when all relevant knowledge is incorporated, LLMs still struggle with lower factuality in later generations. This highlights the need for alternative decoding algorithms specifically designed for long-form generation tasks.

This work does not provide any insightful reasons

We developed a mathematical framework linking Self-Known and Self-Unknown scores to factu- ality, providing deeper insights into their relationship. Both empirical and theoretical results demon- strate that higher Self-Known scores correspond to improved factuality, while higher Self-Unknown scores are associated with reduced factuality. Moreover, even without significant changes in the models’ self-judgment (Self-Known and Self-Unknown), the number of unsupported claims may still increase, likely reflecting the inherent challenges of long-form generation.

Why reporting both the percentage and number of supported and unsupported atomic claims in Figure 1

Figure 1(a) illustrates the percentage of supported and unsupported claims across different relative positions, while Figure 1(b) presents the absolute number of supported and unsupported claims at these positions. Both views are provided to account for variations in the total number of claims across different relative positions.

Necessity of introducing RAG.

In the RAG experiments, even when all relevant knowledge is incorporated, LLMs still struggle with lower factuality in later generations. This highlights the need for alternative decoding algorithms specifically designed for long-form generation tasks.

Correction of Errors and proof reading

We have corrected the phrasing in lines 60 and 85 to ensure consistency with our actual findings. Specifically, we state that "sentences generated later in the sequence tend to exhibit lower factuality scores.

We have conducted a thorough proofreading of the manuscript to identify and resolve similar inconsistencies, ensuring accuracy throughout.

We appreciate the reviewer’s feedback on the clarity of the paper’s presentation and the logical flow of the methodology.

Clarifying the Logical Flow

In Section 3, we explore three ways to evaluate the Self-known and Self-Unknown. The beginning of Page 4 shows the three approaches, and three corresponding prompt templates provided in Appendix Section A.2, which is already shown.

In Section 4, we provide a novel perspective by mathematically linking Self-known and Self-Unknown scores to factuality, offering deeper insights into the underlying causes of these errors and potential mitigation strategies.

In Section 5, we provide the empirical results on different LLMs on Self-Known, Self-Unknown scores and factuality.

Clarifying Factuality Score Usage

We have clarified in Section 4 that the estimation of the factuality score provides a theoretical foundation, while in Section 5.1, automatic tool is used for practical implementation. Additionally, at the beginning of Section 5, we have explicitly stated which factuality score is used in subsequent experiments, ensuring consistency and transparency throughout the paper.

Novel Insights and conclusion

We appreciate the reviewer’s observation regarding the tendency of models to exhibit more factuality errors in the later parts of their outputs. Our findings confirm that current state-of-the-art LLMs, such as GPT-4 and Claude-3-Opus, still face this issue. Our work aims to advance this understanding by providing a quantitative analysis that correlates these errors with Self-Known and Self-Unknown scores, as detailed in Equation (3). This analysis introduces a novel perspective by mathematically linking these metrics to factuality, offering deeper insights into the underlying causes of these errors and potential mitigation strategies. Additionally, we observe that for advanced LLMs, the Self-Known scores show minimal variation across positions. However, factuality remains lower in later sections, suggesting a strong positional bias in generation. In the RAG experiments, even when all relevant knowledge is incorporated, LLMs still struggle with lower factuality in later generations. This highlights the need for alternative decoding algorithms specifically designed for long-form generation tasks.

Correction of Errors and proof reading

We have corrected the phrasing in lines 60 and 85 to ensure consistency with our actual findings. Specifically, we state that "sentences generated later in the sequence tend to exhibit lower factuality scores.

We have conducted a thorough proofreading of the manuscript to identify and resolve similar inconsistencies, ensuring accuracy throughout.

Assumption on log-likelihood of generation (true and false)

We thank the reviewer for raising the issue regarding the assumption on log-likelihood in our model. To clarify, our framework makes the following assumptions about the log-likelihoods of both true and false generation outputs: Since the true and false parts of the whole generated output come from the same model, their log-likelihoods should be similar, as shown below:

$

P(y \mid x) \approx P(\mathrm{y_{correct}} \mid x) \approx P(\mathrm{y_{wrong}} \mid x) \nonumber

$

Thank you for the review and helpful comments!

The difference between Figure 1 (a) and Figure 1 (b) is not illustrated.

Figure 1(a) illustrates the percentage of supported and unsupported claims across different relative positions, while Figure 1(b) presents the absolute number of supported and unsupported claims at these positions. Both views are provided to account for variations in the total number of claims across different relative positions. Please see the caption in the Figure 1.

“Time-consuming to do evaluation for every atomic statement” and “a bit limited as it can only be applied to factuality tasks which can be separated into atomic statements.”

Yes, evaluating every atomic statement can be time-consuming, but it provides a fine-grained assessment, especially since LLM responses often contain a mix of both true and false information. Currently, this level of atomic evaluation is the standard approach for assessing long-form factuality, as demonstrated in works such as [1,2].

[1] FACTSCORE: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP 2023

[2] Long-form factuality in large language models. Neurips 2024

“There is not a way to adjust or convert the fact errors into correct facts. The proposed method only covers fact error detection.”

We thank the reviewer for highlighting this important point. Indeed, the primary focus of our work is on detecting and analyzing factual errors rather than directly correcting them. The goal of the proposed method is to provide a robust framework for identifying and quantifying factual inaccuracies in long-form generation, which we believe is a critical step toward improving factual consistency in LLM outputs.

“Missing results for the LLAMA family makes the results less comprehensive.”

Please see Figure 9 in the appendix. More related results are already shown.

Thank you for the review and helpful comments!

"The Finding where model tends to have more factuality errors in later part is no surprising"

We appreciate the reviewer’s observation regarding the tendency of models to exhibit more factuality errors in the later parts of their outputs as a well-known phenomenon. Our findings confirm that current state-of-the-art LLMs, such as GPT-4 and Claude-3-Opus, still face this issue.

Our work aims to advance this understanding by providing a quantitative analysis that correlates these errors with Self-Known and Self-Unknown scores, as detailed in Equation (3). This analysis introduces a novel perspective by mathematically linking these metrics to factuality, offering deeper insights into the underlying causes of these errors and potential mitigation strategies.

Additionally, we observe that for advanced LLMs, the Self-Known scores show minimal variation across positions. However, factuality remains lower in later sections, suggesting a strong positional bias in generation. In our RAG experiments, even when all relevant knowledge is included, LLMs still struggle with factuality issues. These observations highlight the need for alternative decoding algorithms tailored specifically for long-form generation tasks.

Steps from Eq (1) to Eq (2)

Thanks for point about the confusion. Following is the rewritten steps:

$$\begin{aligned} & \sigma * P(y \mid x) \\\\ =& P(d=1 \mid y, x)* P(y \mid x) = P(d=1,y \mid x) \nonumber \\\\ =& P(d=1, \mathrm{y_{correct}} \mid x) \sigma + P(d=1, \mathrm{y_{wrong}} \mid x) (1-\sigma) \nonumber \\\\ =& P(d=1 \mid \mathrm{y_{correct}} ) P(\mathrm{y_{correct}} \mid x) \sigma + P(d=1 \mid \mathrm{y_{wrong}} ) P(\mathrm{y_{wrong}} \mid x) (1-\sigma) \end{aligned}$$

These updates can be found in Section 4 on page 5 of the revised manuscript. We hope these changes address the reviewer’s concerns and make the derivation more comprehensible.

Definition of "y_correct" and "y_wrong"

We acknowledge that the definitions of 'y_correct' and 'y_wrong' were not sufficiently clear in the original manuscript. To address this, we have revised the definitions to explicitly state that 'y_correct' refers to model outputs aligned with the ground truth and 'y_wrong' refers to outputs that are wrong.

Assumption of No 'Overconfidence

We appreciate the reviewer’s comment regarding the assumption of no "overconfidence." While this assumption was introduced to simplify the mathematical model, we agree that it does not fully capture the complexities of real-world scenarios. To address this: We have clarified the scope and limitations of this assumption in Section related work. Some existing work[1,2,3] already demonstrated that LLMs are capable of assessing the validity of their own claims and predicting their ability with answering true/false questions accurately.

[1] Language models (mostly) know what they know. Anthropic. 2022.

[2] A token-level reference-free hallucination detection benchmark for free-form text generation. ACL 2022

[3] Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. EACL 2023

Validation on More Than One Dataset

We agree with the reviewer that validating the findings on additional datasets strengthens the robustness of the conclusions. However, there is a limited availability of human-annotated, long-form LLM factuality datasets.

To address this limitation: we have included a footnote in Section 4 on page 6 about the current scarcity of such datasets and include details for research reproducibility and validation. Future work will aim to address this gap as more annotated datasets become available.

Self-Known is also called, True Negative Rate and Self-Unknown" will be True Positive Rate

Yes, they look similar. This is another way to interpret these two scores.