FABLES: Evaluating faithfulness and content selection in book-length summarization

We appreciate the reviewer's insightful comments and would like to address their concerns below. First, the reviewer wonders about the annotation quality:

Annotation quality control

Great questions! The reviewer is correct that we only hire one annotator per book, as in practice it is difficult to find 2-3 people who have read the same book. That said, we believe that our annotators had not only read the books but also carefully conducted the task, as the process of gathering evidence and giving comments (in addition to the T/F label) forces them to frequently revisit the book.

We perform two additional small-scale analysis experiments that demonstrate the high quality of our dataset: (1) inter-annotator agreement on a subset of claims where we had access to another annotator who also read the book, and (1) self-consistency of annotations (i.e., how often a single annotator assigns the same label to claims with the same semantic content generated by different models). We will add a detailed description of these experiments to the next version of our paper.

Inter-annotator agreement: For two books in our dataset, we were able to hire another annotator who had also read those books and provided overlapping annotations. This process resulted in 115 re-annotated claims, and we find the agreement rate between the original annotator and the new annotator is 91.30%, with Cohen’s Kappa of 0.621 (p < .0001), indicating substantial agreement. Unfortunately, re-annotating the entire dataset with multiple annotators is prohibitively expensive, costing around $200-$250 per book and requiring ~10 annotation hours.

Self-consistency: For each book, an annotator analyzed five summaries, each generated by a different model. To assess self-consistency (intra-annotator agreement), we randomly selected five books and compared the annotations made on the first and last summaries (as per annotation order) for claims with the same semantic content. Consistent labels would suggest that they maintained a stable judgment throughout the process. Out of 127 claims from the first summary, 46 have semantically-equivalent claims in the last summary, all 46 of these claims were consistently labeled.

Recall of claim decomposition

The Reviewer wonders about the recall of the claim decomposition step. As this concern is shared with Reviewer FiuX, we kindly refer the Reviewer to our response to Reviewer FiuX.

We would like to thank the reviewer for their insight feedback and address the concerns mentioned in the review below. The reviewer asks about why our automatic evaluation metric fails:

"A more in-depth analysis of where and why the automatic evaluation methods fail would be welcome. One valuable insight is that unfaithful claims often require multi-hop reasoning to be detected: it would useful to estimate the percentage of cases where this is the case. The authors note themselves that the sample size of unfaithful claims is very small -- so it would probably be possible to even do the in-depth analysis manually."

This is an insightful point. As we show in Table 3, 50.2% of unfaithful claims require multi-hop (or indirect) reasoning to be invalidated. Following the reviewer’s suggestion, we look at predictions from our auto-rater experiment (on seven books) for which Claude 3 and GPT-4 Turbo incorrectly mark a false claim as true, and we annotate the type of reasoning required to verify these claims. The results in the table below show that ~75% of the failure cases require multi-hop reasoning over the book, which is significantly higher than the data distribution of 62.8% with seven books and suggests that our auto-raters struggle with multi-hop reasoning. We will include a more in-depth discussion and analysis in the next version of our paper.

We appreciate the detailed feedback and would like to address the reviewer's concerns.

Choosing the books

We did not select books ourselves; rather, we hired annotators who self-report books they have read that match our criteria (>2023 fiction), as noted in Section 2. That said, our dataset has multiple genres (see Table 7).

Annotation quality control

Great point! We perform two small-scale analyses to show the annotations’ high quality: (1) inter-annotator agreement, and (2) self-consistency. We will add them to the next version of our paper.

Inter-annotator agreement: For two books, we hired another annotator who had also read them, resulting in 115 claims with overlapping annotations. The agreement rate between the two annotators is 91.30%, with Cohen’s Kappa of 0.621 (p < .0001), indicating substantial agreement (https://aclanthology.org/J08-4004.pdf). Unfortunately, we cannot do this for the entire dataset, as finding multiple workers who have read the same book is challenging and expensive, costing ~$250 per book and 10 annotation hours.

Self-consistency: We randomly selected five books and compared the annotations made on two of the summaries for semantically similar claims. Consistent labels would suggest that labels were not arbitrarily assigned. Out of all 127 claims in the first set of summaries, 46 had semantically equivalent claims in the second set, all 46 claims were consistently labeled.

Claim taxonomy

The taxonomy was created via manual coding of collected annotations and aligns with prior work (https://arxiv.org/pdf/2303.01432). See Table 15 for details. In short, indirect reasoning is multi-hop reasoning over different parts of the book, subjective reasoning calls for subjective judgment (usually related to themes), and extra info requires meta information about the book (e.g., publication date). We will clarify Table 3 in the next version.

Future directions

We thank the reviewer for pointing this out and will add it in the next version. One future direction is to address content omissions via planning (e.g., generate important arcs, characters, and relationships, and use them as a basis for the summary). Another is to build more reliable auto-raters, as our work shows that existing ones are insufficient. Once auto-raters are improved, we can also use their signals to fine-tune LLMs to become better summarizers (similar to factuality tuning in https://openreview.net/pdf?id=WPZ2yPag4K).

We thank the reviewer for the insightful feedback and address their concerns below.

Limited methodological innovation

We believe evaluation is crucial and complements the development of new methods. The field lacks robust benchmarks for assessing long-context LLMs with extremely long inputs due to data contamination and the challenge of evaluating tasks requiring familiarity with entire books. Our work aims to fill this gap.

Future directions

The reviewer is concerned about how to improve the auto-rater. Please see our response to Reviewer AyeL.

Annotation quality control

As this concern is shared with Reviewer BatK, we kindly refer the reviewer to our response to Reviewer BatK due to space limit.

Recall of claim decomposition

In Figure 2, the omitted part is a generic statement at the beginning of the summary, but the romantic relationship between the characters is indicated in claims 4, 7, 9, 12, 13, 14, and 18. For a detailed analysis of recall, please see our response to Reviewer FiuX.

Claim vs subclaim-level annotation

In footnote 8, the phrase “claims sometimes contain multiple subclaims” refers to claims making multiple statements. During coding, we consider all subclaims, so a single claim may be assigned multiple labels to ensure each aspect is classified.

Correlation of Qualitative Observations with Faithfulness

This question is relevant to Section 5, where we find that the least faithful models tend to be more generic. In addition, while all models omit important information according to annotators, Claude 3 Opus (most faithful) is noted for making the fewest omissions.

Different Prompting

Unfortunately, with book-length input few-shot prompting is impossible with the evaluated models. As per wording of the prompt, we conducted further experiments requesting (1) the answer-only, (2) answer and then explanation, and (3) explanation and then answer. We observe ~1% difference in accuracy for these prompts.

Difference Across the Books

We measured the percentage of incorrect claims across books. Most variation we observed was due to the models; for example, Claude 3 Opus consistently performs well. There is also some variation across books, but it’s unclear how much of it is due to the difference in book content. Due to space constraints in the rebuttal, we are unable to include a detailed table of results by book. We will add it along with a relevant discussion in the next version of the paper.

Thank you to the reviewer for the valuable comments and insightful feedback. We would like to address the concerns raised in the review below.

"Intermediate steps for human evaluation are automatically performed, especially the claim extraction, which might also introduce errors."

Great point! While our paper contains a discussion of the high precision of the claim extraction in Section 2, we did not include a discussion of recall. In response, we closely analyze the extracted claims on a subset of 20 summaries (371 sentences, 450 extracted claims), from which we conclude that the claim extraction process is not responsible for the omissions that our annotators observe. We will include details about this experiment in the next version of our paper.

Recall analysis: For each of the 20 summaries, we manually evaluate the quality of the extracted claims against the content of each summary. Calculating recall is difficult since it is unclear what granularity to calculate it against (e.g., sentences, clauses). 3.8% of the 371 sentences in the 20 summaries were omitted in extracted claims. Of these, 85.7% were generic statements, and 14.3% were minor details. We also qualitatively observe a small percentage of omitted details at the sub-sentential level (e.g., clauses), none of which impacts the narrative. These omissions can be broadly categorized into two types:

1. Generic statements lacking substantive content. For instance, “The narrative unfolds with intrigue, danger, and treacherous encounters” appears in the summary but is omitted in extracted claims. Note that this sentence only addresses things already covered by other extracted claims in a generic way, so omitting it has few consequences.

2. Insignificant details that contribute little to the narrative. For instance, “Altha, a 17-century woman, stands trial unjustly accused of witchcraft due to her remarkable healing abilities which are misunderstood by her village” appears in the summary, but “misunderstood by her village” is omitted in the claims. However, this is only a minor detail with little impact on the narrative.

Importantly, we confirm that none of these discrepancies between the content of the summaries and claims were reasons for criticism of omissions, chronological errors, and factuality issues in the annotators’ summary-level free-form comments. In the next version of the paper, we will run this qualitative check on all summaries and their extracted claims.