CLIPPER: Compression enables long-context synthetic data generation

We thank the reviewer for acknowledging the strength and interpretability of our pipeline.

LLM-as-a-judge is used to check whether generated CoTs are grounded or not. While effective for cost and scale, it is unclear how good the judge would be at this task. On a small subset of data, authors need to correlate model judgments with human judgments, before asserting that such models can be effectively used as judges.

We would like to point the reviewer to section A.6 and Table 11 in the appendix, where we report how each LLM judge aligns with human annotations on 66 CoT samples. The LLM judge that we eventually choose, DeepSeek‑Distill‑Llama‑70B, has the highest agreement (>90%) with human annotators and provides clear explanations.

We thank the reviewer for acknowledging the thoroughness of our experiments.

The way to generate reliable claims is basically leveraging hierarchical summarization. So the novelty is not that high, but the experiments and human evaluation is really thorough.

Our approach goes beyond summarization to produce claims that are intentionally challenging for models to verify. In this sense, we adapt hierarchical summarization to the specific task of long-form claim verification, which we think is a valid and exciting way to demonstrate how established methods can be repurposed to solve challenging problems.

The authors should add performance margin such as in Table 2 to show whether the improvement is significant. It appears that the improvements on NarrativeQA and MuSR are minor.

We present below statistical test results for the benchmarks reported in Table 2. For CLIPPER-test, NoCha, and MuSR, which return binary True/False predictions, we use McNemar’s test. For NarrativeQA and InfiniBenchQA, which return ordinal scores ranging from 0 to 3, we use the Wilcoxon signed-rank test.

Fine-tuning on CLIPPER yields statistically significant improvements across all models on CLIPPER-test and NoCha. For MuSR, both Qwen and LLaMA show significant gains, while ProLong does not. For InfiniBenchQA, Qwen demonstrates a statistically significant improvement. For NarrativeQA, no models exhibit a significant improvement.

We emphasize again that while improvements on NarrativeQA, MuSR, and InfiniBenchQA are modest, these results represent performance on OOD tasks in our paper. NarrativeQA and InfiniBenchQA focus on question answering over narrative contexts, while MuSR consists of algorithmically generated reasoning problems. Therefore, significant performance gains on these tasks would be nice to have, not expected.

Table 1. Test statistics comparing fine-tuned and baseline models across benchmarks

Table 2. p-values for statistical significance (p < 0.05)

The reviewer is curious about the performance of specialized claim verification models such as those trained for detecting errors in text summarization.

We agree that specialized claim verification models are interesting baselines. However, our baselines, Qwen‑2.5‑7B‑Instruct, LLaMA‑3.1‑8B‑Instruct, and ProLong‑512K‑8B‑Instruct, are already among the strongest open‑weight models for claim verification, as shown in the NoCha benchmark. We believe that our baseline provides a meaningful upper bound for our task.

We also provide results for OpenAI o1‑mini and Gemini 1.5‑Flash 8B, which are proprietary models of the comparable size as our CLIPPER models. These models outperform LLaMA and ProLong baseline but still lag behind Qwen‑2.5‑7B by around 3% and the worst CLIPPER model by 25% on our test set.

When constructing book-level claims, would those claims simply be conjunctions of claims for each chapter?

We perform a quick heuristic check on 20K claims by looking for coordinating conjunctions (e.g., "and," "but," "or," "nor," "so," "for," "yet") in the claim texts. We find that 77.12% of true claims and 74.19% of false claims are conjunctions of multiple chapter events.

While this is somewhat expected, we note that the number of naturally occurring combinations is still significant, with up to approximately 5K such claims for each type in the dataset.

We thank the reviewer for acknowledging the novelty of our approach and the thoroughness of our experiments.

The compression stage could largely affect the quality of the output dataset. This part is difficult to tune and subject to model biases and potentially hallucination. It is also difficult to verify because of the volume.

The compression stage, where entire books are compressed into summaries and chapter outlines, could indeed be challenging to properly tune and validate. However, prior research has demonstrated that LLMs are capable of producing high-quality summaries of long documents [1][2].

In addition, these compressed representations could still provide a strong foundation for claim generations, as most of CLIPPER claims are grounded in the original book (as determined by our human validation on a subset of 66 claims). Therefore, we consider this compression stage adequate for our purposes.

Have you considered using different LLMs to generate claims to promote diversity / avoid model specific biases?

This is a good point. At the time of writing, Claude-3.5-Sonnet is the only model we find capable of generating diverse claims that remain grounded in the source texts. We expect that newer LLMs with stronger instruction-following capabilities could also be used to diversify claim generation, and we will leave this to future research to explore.

[1] Chang, Yapei, Kyle Lo, Tanya Goyal, and Mohit Iyyer. "Booookscore: A systematic exploration of book-length summarization in the era of llms." https://openreview.net/forum?id=7Ttk3RzDeu

[2] Kim, Yekyung, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. "FABLES: Evaluating faithfulness and content selection in book-length summarization." https://openreview.net/forum?id=YfHxQSoaWU

We thank the reviewer for their detailed review and for acknowledging the strength of our method.

A more rigorous baseline, such as a simplified version of CLIPPER based on passage retrieval and fine-tuning on passage-claim-response pairs, would help better isolate the contribution of the proposed method.

This baseline sounds like a retrieval-augmented generation (RAG) pipeline, where a model retrieves relevant passages and verifies claims using only the retrieved evidence. We quickly implement one version of such pipeline, using BM25 to retrieve the top 50 relevant book passages (each no longer than 256 words) for a given claim and prompting our original baselines with these passages instead of the full book text.

As shown in the table above, these RAG baselines (denoted by BM25 + model name) outperform our original LLaMA and ProLong baselines (+9%), but not the Qwen baseline (-15%). Compared to our CLIPPER models, however, these RAG baselines still lag by 22-50%.

It is important to note that RAG approaches do not consistently outperform long-context models in long-form claim verification. For instance, Karpinska et al., 2024 [1] and Kim et al., 2024 [2] benchmark a similar RAG setup where GPT-4o is provided only with BM25-retrieved passages. As shown in Table 3 of [1], the RAG versions (k = 5, 25, 50) consistently underperform the setting where GPT-4o has no retrieval support.

Certain conclusions, such as the one in Section 4.3 ("Short-context claim data is less helpful"), lack robust empirical support, especially as the baselines referenced are not instruction-tuned or optimized for the task.

We would like to clarify that our claim specifically concerns the limitations of short-context claim verification data in the context of long-form claim verification. We choose our baselines for this specific task based on their performance on the NoCha benchmark (https://novelchallenge.github.io/). We are not trying to show that the claims made by Dubey et al., 2024, and Gao et al., 2024 are entirely incorrect; rather, we want to highlight a long-form task where their findings may not apply. While we maintain that our claim is valid, we will revise the section header to indicate that this limitation specifically concerns long-form claim verification.

Some sections suffer from a lack of logical flow, making them difficult to follow. For instance, Section 2.2 ("Naïve claim generation using book texts") appears abruptly and is followed immediately by the introduction of CLIPPER without sufficient contextual buildup.

We present Section 2.2 as a straw-man proposal, which is a straightforward and intuitive approach but does not work well. We put this section before the introduction of CLIPPER to have a proper motivation for our method. That being said, we will revise the first paragraph of Section 2.2. And 2.3 so that there is a clearer flow.

The Related Work section references a large number of prior studies but does so in a cursory fashion, often without elaboration or critical analysis. This weakens the reader’s understanding of how the current work builds on or diverges from existing research.

Despite space constraints, we did try our best to list all relevant work. Throughout the paper, we cite and engage with these works where appropriate. In a future revision, we will expand on the concluding sentence of each related work subsection to provide further clarification.

[1] Karpinska, Marzena, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. "One thousand and one pairs: A" novel" challenge for long-context language models." https://aclanthology.org/2024.emnlp-main.948/

[2] Kim, Yekyung, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. "FABLES: Evaluating faithfulness and content selection in book-length summarization." https://openreview.net/forum?id=YfHxQSoaWU