BooookScore: A systematic exploration of book-length summarization in the era of LLMs

We thank the reviewer for their insightful comments and for appreciating the applicability and other strengths of our work.

> The evaluation of human annotations focuses only on precision and does not investigate recall. However, the authors are open about this limitation.

In order to measure recall, we would require ground truth annotations with all possible coherence errors in the summaries. However, prior studies (SNaC and Scarecrow) have discussed that this is challenging for subjective fine-grained evaluation tasks where annotators generally display high precision but low recall.

One solution could be to collect annotations from multiple annotations per summary, and consider their union as the exhaustive set of errors. Scarecrow and SNaC employ 10 and 3 annotators per summary respectively, but such a scheme would be very costly for our setting, as annotating 100 summaries already costs $1.5K. Annotation precision is more economical to evaluate, while also serving as an informative metric. We have updated Section 3 to make it clearer. We believe future work in human evaluation that makes recall-centric annotation more tractable is a promising area that could complement the precision-centric measurement we propose here.

> Some of the examples of types of errors shown in Table 1 seem to be of questionable quality…

This is a valid concern, and we appreciate the reviewer for pointing it out. We acknowledge that sometimes annotators or GPT-4 may ask questions that are also unanswered in the book, and there is no way to confirm without reading the source text. Using the reviewer’s example, if the identity of the mysterious man is not revealed in the book, the summary could say something like “A mysterious man whose identity is never revealed.” Since our focus is on coherence, the nature of our evaluation protocol is to have readers naturally raise questions as they read through a summary. Checking whether a question can be answered by the source text and finding answers to these questions would fall under faithfulness evaluation, which we leave to future work.

> This may be a side effect of offering a monetary reward based in part on the number of annotations (cf. Appendix G).

We appreciate the reviewer for going through the appendix. For the first evaluation task where we collected span-question pairs, we actually paid $15 USD per summary regardless of how many annotations were provided as mentioned in Section 3.

Appendix G (now J) is for the second human evaluation where we validated annotations collected from the first one. For this evaluation, we paid based on how many annotations from the first task an annotator had to annotate, which also did not incentivize them in any way. The annotators were informed of the second evaluation task a few weeks after they completed the first one, so payment for the second task could not have biased annotators to create more annotations for the first. We understand that this may be unclear, and have made clarifications in Appendix J.

> The sentence-level score may disproportionately favor summaries that contain a large number of (short) sentences.

We understand this concern. In practice, we do not observe significant differences in sentence length. We computed the average number of tokens per sentence for the 100 summaries collected under each configuration (10 in total). The minimum is 21, maximum is 28.3, and standard deviation is 2.5. Future work may decompose the summary into atomic units (like FactScore) before running fine-grained evaluation. However, this decomposition may be hard to automate.

> Furthermore, the evaluation of different models does not further investigate whether some of the score differences can be explained by the different length of the summaries.

This is a valid concern. We have added a plot of BOOOOKSCORE vs. summary length in Appendix F, which shows no correlation between the two.

> Table 2: Why is there no entry for LLaMA2 in the incremental update section?

This is explained in Section 5, we could not get LLaMA 2 to perform incremental updating.

> Doesn't the choice of p = 1 for nucleus sampling disable it completely?

You are correct, we have rephrased it in the paper.

> The choice of a large temperature also seems rather unusual…have you investigated how stable the results are when the temperature is varied?

We found that setting temperature to 0 affected generation quality in our preliminary experiments. Plus, prior to the recent OpenAI update, setting temperature to 0 for these closed-source models did not guarantee deterministic output. We did not study the effect of temperature on BOOOOKSCORE due to budget constraints.

> shouldn't "lower" be "higher"?

That is correct, we have changed it in the paper.

> “Highly extractive" seems to contradict the large proportion of novel trigrams…

“Highly extractive” is relative to other models. We have updated the paper with clarification.

We extend sincere gratitude to the reviewer for their insightful comments and for greatly appreciating the value and importance of our work.

> It is however unfortunate that the evaluation framework did not consider faithfulness. This could have been done by asking annotators to assess each fact and search for it inside the book to verify its factuality.

We thank the reviewer for proposing a possible experimental setup for evaluating faithfulness. Coincidentally, we did try the task ourselves using the exact setup as described in the reviewer’s comments, and found it incredibly challenging. Some facts may span multiple pages, even chapters, of a book, and verifying a single fact could take 10-20 minutes, let alone a full summary’s worth. Considering the amount of time, money, and energy it would take to complete a full-scale human evaluation of faithfulness, we decided to stick to coherence and acknowledge that our work is complementary in nature. Prior work like LongEval and FactScore have established guidelines and methods for evaluating faithfulness, and our evaluation protocols can always be adapted to or combined with these work for faithfulness evaluation. We focus on coherence as it emerges as a noteworthy problem on its own when it comes to evaluating very long summaries of book-length documents.

> How did you choose the number of O in BOOOOKSCORE?

There are 4 O’s in BOOOOKSCORE because we have four authors on the paper!

We thank the reviewer for their insightful comments and for appreciating the importance of our work to future research in summarization.

> One point I miss in the experiments is a baseline without hierarchical merging or incremental updating.

This is indeed a valid concern, we appreciate the reviewer for bringing it up. To address this issue, we evaluated GPT-4 on the SQuALITY dataset, which contains short sci-fi stories that are 4000-6000 words long. Results from these experiments have been added to Appendix E. Here, we show that the baseline method (without merging or incremental updating) produces slightly better summaries than incremental updating (in terms of ROUGE-L score), and we are currently running BOOOOKSCORE on this dataset as well for a more specific analysis of coherence. Additionally, qualitative inspection of these summaries (added in Appendix E.1) shows that they are not only superior to those produced by incremental updating but also human reference summaries from the dataset, which is aligned with results from recent work on LLM-based short-document summarization: https://arxiv.org/abs/2309.09558. Our experiments overall suggest that approaches like incremental updating can introduce coherence errors into LLM-generated summaries.

> Not finding results online by no means imply that memorization is impossible.

It is true that closed-source models may have been trained on publicly available data, we thank the reviewer for pointing this out. We have adjusted our wording in the paper to be more precise. That said, there is still a very low chance that these models have seen actual summaries of those books given how recently the books were published. All books in our dataset were published after the beginning of 2022, and 87% of them were published in 2023. For our experiments, we used the gpt-4 2023-03-15 and gpt-3.5-turbo-0301 checkpoints on Microsoft Azure. Both models have a training data cutoff date of Sep. 2021. Anthropic unfortunately does not disclose checkpoint information, but our summaries were all obtained via their Claude 2 API in September 2023. We also note that most of the books are not (yet) widely read, making it even less likely that readers have written high-quality summaries. Overall, we made diligent efforts within our control to avoid testing on the training set.

> However, your experimental findings show that hierarchical merging results in more coherent summaries. Do you have an explanation for this observation?

Prior to running our experiments, our theory was that incremental updating should enable the model to take more context into consideration when composing summaries. We understand that our phrasing in the paper is too definitive, and have revised it to clarify that this was our hypothesis.

Our results show the opposite, and we hypothesize some reasons for this in Section 5 of the paper. Essentially, incremental updating is inherently a more complex task, as it requires the model to maintain a running summary as it steps through chunks of a book. When deciding what to include from the current book chunk and what to remove from the current summary, the model needs to juggle texts of different levels of granularity. Plus, the model must compress the summary whenever it exceeds the final summary limit, which could further affect coherence. For the chunk size = 2048 setting, models need to compress 17.6 times per book on average, while each book has 68 chunks on average. Hierarchical merging, on the other hand, only requires the model to summarize individual chunks and recursively merge them together. If a single summary gets too long, it would often be enough to simply have the model regenerate. This approach does not involve working with texts of different granularity levels or compressing summaries while maintaining coherence.

We thank the reviewer for their insightful comments and for appreciating our extensive experiments, clear writing, and innovative evaluation metric design.

> The approach of weighing all sentences equally may overlook the varying importance of different parts of the text for overall coherence.

We acknowledge that weighing all sentences equally may overlook the relative importance of different parts of the text to overall coherence. One way to mitigate this problem is to assign severity weights to different error types, as in MQM for machine translation. However, this step introduces more subjectivity into the evaluation procedure (e.g., see https://arxiv.org/abs/2305.18201 for similar concerns in long-form QA), which is why we chose to equally weight all errors. It is unclear how the weights should be chosen, and whether a single weighting scheme makes sense in all scenarios. Future work can further explore correlations between different weighting schemes and human judgments of overall summary coherence.

> there is an absence of a consistency check for the evaluation metric.

We understand the importance of the consistency check (i.e., if you resampled summaries from each LLM, would you obtain the same ranking of systems?), and appreciate the reviewer for bringing up this issue. That said, running multiple iterations of full-scale experiments is expensive (it can cost roughly $1-2K to collect 100 summaries from closed-source LLMs, depending on which model and chunk size are used). Instead, we ran a bootstrapping analysis where given 100 summaries generated under one configuration, we generated 1000 random samples with replacement using a sample size of 100, then computed BOOOOKSCORE for the 1000 samples. The scores have a standard deviation of 0.015, which indicates that BOOOOKSCORE is stable across varying random samples. We have added this analysis in Appendix G.