Synthetic continued pretraining

This paper addresses how to train LLMs in a data scarce regime, given that LLMs require O(1000) examples of a fact to actually "learn" it. This has applications both to niche corpora (e.g., mathematics textbooks) as well as to training larger models once all human text is exhausted. The authors propose to use a pre-trained LLM to (1) extract entities and summaries from a comparatively small, niche corpus, and (2) use the extracted entities to generate rephrased assertions about those entities, to facilitate learning by a second (here, smaller) LLM. They experiment with a 1.3M token reading comprehension dataset, and test the approach against several baselines, including closed-book tests on the LLM used to extract entities and the rephrased text used to train the second LLM. Finally, the authors present a mathematical model through which they attempt to understand the training behavior of this data augmentation system.

The paper proposes "synthetic continued pretraining" to enhance language models' domain adaptation efficiency, particularly when adapting to a small corpus of domain-specific documents. The authors introduce EntiGraph, a synthetic data augmentation algorithm that constructs a knowledge graph from the entities in the original corpus and synthesizes diverse text to enhance the pretraining process. The approach was evaluated using a reading comprehension dataset, QuALITY, showing notable improvements in model performance with both closed-book and open-book scenarios.

This paper proposes a method to continue pretraining LLM with a synthetic data augmentation method. The method is based on expanding the training corpus with many verbalizations of the entity graph present in the training corpus. It moves from a sparsely verbalized entity graph to a more densily verbalized one by using only the source documents and prompting LLMs to generate the new tokens.

The paper shows that the method is beneficial for downstream tasks in closed- and open-book QA as well as RAG.

Overall, I think the paper is worthy of acceptance, it propose a clean method with good results and the experiments are fairly convincing.

This paper addresses the problem of data inefficiency in pretraining language models. Current pretraining corpora may not generalize effectively and models may benefit from structured, repeated, diverse representations of knowledge.

The proposed is a two-step process that (1) extracts entities from the corpus and then (2) extracts relationship information amongst a subset of the entities.

Experimentation uses the QuALITY corpus and dataset, which is a benchmark for long-document reading comprehension. Evaluation compares with relevant baselines like training on the original QuALITY corpus and a corpus containing rephrasings.

We thank the reviewers for their hard work and detailed feedback.

We were glad that you support our motivation, mentioning that “the problem… is important” (9zM9) and that the EntiGraph approach is “well-motivated” (wsFi). Further, we are happy you found that our method is “clean” (ipvv) and that the theoretical model “provides insights into the mechanics of synthetic $data$ ” (9zM9), “aligns well with empirical observations and provides a deeper understanding of $EntiGraph’s$ scaling properties” (wsFi), and “provides some good intuition” (nNRV).

We were also glad to see you highlight our empirical evaluation, mentioning that the experiments/evaluations are “fairly convincing” (ipvv), “convincing” (nNRV), and “comprehensive” (wsFi), and the downstream performance improvements are “notable”, “clear”, and “significant” (wsFi). Lastly, we are excited that the paper was “clear and well-written” (9zM9) and “clear and easy to read” (nNRV).

Based on your feedback, we conducted three additional experiments to better understand EntiGraph and its properties. Full results, setup details, and discussion are in Appendix I (last 3 pages) of the manuscript, and we will add pointers in the main text. We summarize the results below:

Rebuttal Experiment 1: Ablation with weaker synthetic data generator (Appendix I.1)

First, based on the feedback of ipvv, wsFi, and nNRV, we conducted experiments replacing the strong GPT-4-Turbo-based synthetic data generator with a weaker Llama 3.1 8B Instruct model. This accounts for the concern that the gains of synthetic CPT arise from distilling a stronger model.

We observe consistent gains with the Llama 3.1 8B generator up to 334M synthetic tokens. Moreover, the slope of the scaling trend is similar to EntiGraph’s trend using the GPT-4-Turbo generator. In contrast, the log-linear slope is smaller for the Rephrase baseline (with Llama 3.1 8B Instruct, the same weaker synthetic data generator) as in our main scaling results in Figure 2 (Section 4.2). Altogether, this experiment demonstrates that the gains of Synthetic CPT with EntiGraph are significant and reproducible even using standard open-source models.

Rebuttal Experiment 2: Factuality and lexical diversity of generated synthetic data (Appendix I.2)

Second, reviewers ipvv, wsFi, and nNRV suggested we obtain quantitative, intrinsic measures of our synthesized corpora. We conducted a human evaluation of the factuality of the EntiGraph corpus, and measured lexical diversity by computing n-gram overlap statistics between EntiGraph- and Rephrase-generated text and the source documents.

Factuality: we have paper authors annotate random sentences from EntiGraph and find that 94.4% of non-subjective sentences are factually supported by the source document. Contrasting with GPT-4’s poor closed-book performance on QuALITY (~51%), this experiment shows that grounding with source documents helps prevent hallucination.

Lexical diversity: for both EntiGraph and Rephrase-generated text, we find that only a small percentage of n-grams are exactly copied from the source documents. For example, less than ~0.5% of 8-grams and ~0.2% of 16-grams in the EntiGraph and Rephrase synthetic corpora appear in the source documents. This suggests that both methods produce lexically diverse knowledge representations.

Rebuttal Experiment 3: New dataset beyond QuALITY (Appendix I.3)

Lastly, reviewers 9zM9, wsFi, and nNRV suggest that we test our method on another domain, ideally in a “complex field” (9zM9) or a “harder topic” (nNRV). To that end, we **conducted scaling experiments using the Coursera Exam QA dataset $1$ **, which contains 15 lecture transcripts and exam questions from advanced technical courses. This setting is even more data-scarce, with only 124K raw tokens (compared to 1.3M in QuALITY). We find that EntiGraph consistently delivers log-linear improvement in accuracy, outperforming the base model and Rephrase baseline.

Altogether, these carefully constructed evaluations show that Synthetic CPT robustly enables an LM to learn from niche domains, and that these gains arise from diverse representations of knowledge rather than distillation.

Thanks again for the helpful feedback. We hope to engage with you to clarify any remaining points!

$1$ An et al., 2023. L-Eval: Instituting Standardized Evaluation for Long Context Language Models.

The paper proposes a data-synthesizing method for continued pretraining for adapting an LLM to a specific domain, where the proposed approach is based on EntiGraph including entities and their relations. In addition, the authors derived bounds for their scaling trends.

All reviewers agree that this is an interesting and solid paper. Reviewers have raised a few minor concerns and questions, which may be addressed in the revision.

Accept (Oral)