How to Synthesize Text Data without Model Collapse?

We are grateful for your positive feedback and insightful comments. Below, we give detailed responses to your questions.

[Q1] : The solution authors provided is to use semi-synthetic data instead, which seems deviated from what the title suggests.

We will revise the wording and provide further clarification to avoid misunderstandings. An example is provided below:

Based on prior works, model collapse is related to multiple factors, including self-generated processes, data quality, and others. Our proposed method is inspired by the statistical analysis on synthetic data in Sec. 3. Specifically, we focus more on the model collapse phenomenon caused by data quality. Therefore, we try to improve data quality then further indirectly prevent model collapse. In other words, we are not directly addressing model collapse, and thereby indirectly prevent it by improving data quality.

These statements will be included in the Introduction and Related work sections to better clarify our method.

[Q2] : If the issue of synthetic data is the coverage and lack of long tails, could we generate tons of data using LLM and only preserve those long tail ones? Will this method be better than token editing because token editing still require human data?

1. Could we generate tons of data using LLM and only preserve those long tail ones?

For now, generating long-tail samples is currently difficult for language models. The reason lies in the sampling strategy of LLMs [1]. Current LLMs adopt top-p, top-k or other sampling strategy for better performance. However, these sampling strategy will lead to cut-off distribution. When the data synthesizing scale up, this drawback will finally scaling law cut-off on synthetic data. However, human corpus data follows a Zipf distribution, $p_i \propto i^{-\beta}, \quad i = 1, 2, \dots$ [2]. The truncated output distribution causes the LLMs to nearly fail to sample long-tail samples. In other words, it is currently difficult to induce long-tail samples fromLLMs that are as diverse as human data.

On the other hand, if we force the language model to generate long-tail samples, these may contain both noisy and high-information samples, which are like two sides of a coin, both distributed in the long tail of the data [3]. This necessitates further filtering of the high-information samples. Unfortunately, such samples are challenging to automatically identify in practice and may require extensive human annotation [4].

[1] Dohmatob E, Feng Y, Yang P, et al. A tale of tails: Model collapse as a change of scaling laws[J]. arXiv preprint arXiv:2402.07043, 2024.

[2] Zipf G K. The psycho-biology of language: An introduction to dynamic philology[M]. Routledge, 2013.

[3] Swayamdipta S, Schwartz R, Lourie N, et al. Dataset cartography: Mapping and diagnosing datasets with training dynamics[J]. arXiv preprint arXiv:2009.10795, 2020.

[4] Lin Z, Gou Z, Gong Y, et al. Rho-1: Not all tokens are what you need[J]. arXiv preprint arXiv:2404.07965, 2024.

2. Will this method be better than token editing because token editing still require human data?

Yes, we agree with you on this idea. If we had sufficiently powerful LLMs, we could generate a large amount of long-tail data and quickly filter out the irrelevant ones, which would significantly boost the performance of current LLMs. However, these conditions are difficult to meet at present, and much work remains to be done.

We are grateful for your enthusiastic feedback. Below, we give detailed responses to your questions.

[Q1] : References Not Discussed

We will include a discussion in the revised version as follows:

[1] demonstrates that without enough fresh real images, future generative models will gradually decline. [2] develops a rigorous framework to demonstrate the importance of real data in maintaining the stability of iterative training. [3] illustrates that real data in the iterative training process can slow the decline of LLMs, but cannot fully prevent it. [4] shows that the quality and diversity of generated images degrade over time.

[1] Self-Consuming Generative Models Go MAD.

[2] On the Stability of Iterative Retraining of Generative Models on their own Data.

[3] Large language models suffer from their own output: An analysis of the self-consuming training loop.

[4] Towards Understanding the Interplay of Generative Artificial Intelligence and the Internet.

[Q2] : Maybe a small example to illustrate their method.

We add an input-output example below. The complete workflow is also provided in Lines.

Case 1: code reasoning sample in Magicoder-Evol-Instruct-110K:

[Q3] : Does ToEdit change properties of data in a desired way ?

Yes, we present the statistical analysis of edited data below. As shown in Supplemented Tables 1 and 2, ToEdit meets our expectations by preserving the original long-tail distribution. Additionally, Table 14 (Line 1100) illustrates that tokens above the threshold gradually decrease as the iterations progress.

1. KL Divergence Between Distributions (gen_0, gen_1, gen_2)

2. Sample Distribution Across PPL Intervals (gen_0, gen_1, gen_2)

[Q4] : Does ToEdit help in an iterative process?

Yes, the following supplemented table shows effectiveness in an iterative process, with slight improvements in performance across generations.

3. Performance in an iterative process on Instruction tuning data.

[Q5] : Can ToEdit help with already strongly collapsed data or is a minimum quality of the data necessary?

This is a very interesting and insightful question. The ToEdit algorithm was initially designed to preserve the long-tail distribution during the data generation process, thereby avoiding model collapse. For already collapsed data, the variance is typically very small, and enhancing diversity is crucial. We can also adjust the threshold $p$ to introduce more randomness in data. Through this operation, we can inject randomness into the collapsed data. However, this is a theoretical scenario, and as we know, data situations are highly complex. In practice, there will be many more challenges to address.

We appreciate your valuable feedback. In the following, we will address your concerns accordingly.

[Q1] : References Not Discussed

We will include a discussion on all the provided references as follows:

[1] develops a rigorous framework to demonstrate the importance of real data in maintaining the stability of iterative training. [2] theoretically demonstrates that the impact of data curation can be formalized as an implicit preference optimization mechanism. [3] reveals the detailed training dynamics of model collapse under three different training workflows. Of course, there are also some remarkable studies that successfully used synthetic data. [4] proposes the Self-Instruct data generation framework, enhancing instruction-following capabilities. [5] employs the self-talk method to generate high-quality data. ReST [6] uses a policy model to generate datasets and then employs offline RL to fine-tune LLMs on generated datasets. [7] demonstrates that self-training with binary feedback filtering can reduce reliance on real data.

[1] On the stability of iterative retraining of generative models on their own data.

[2] Self-consuming generative models with curated data provably optimize human preferences.

[3] Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World.

[4] Self-instruct: Aligning language models with self-generated instructions.

[5] Ulmer, Dennis, et al. "Bootstrapping llm-based task-oriented dialogue agents via self-talk.

[6] Gulcehre, Caglar, et al. "Reinforced self-training (rest) for language modeling.

[7] Beyond human data: Scaling self-training for problem-solving with language models.

[Q2] : The first finding is already established information.

We agree with you that the potential for synthetic data to cause model collapse has been demonstrated in prior notable works [8]. However, our contribution of statistical experiments provides a more fine-grained analysis of textual data features, specifically examining the underlying reasons behind the failure of synthetic data, e.g., overconcentration of n-grams.

[8] Position: Model Collapse Does Not Mean What You Think

[Q3] : The authors focus on improving the quality of the real-data training via editing.

Yes, we agree with you on this point. Based on insights from prior works [1], model collapse is associated with multiple factors, including self-generated processes, data quality, and others. Our proposed method is inspired by the statistical findings on synthetic data (as stated in Sec. 3). Therefore, we attempt to prevent model collapse by improving data quality.

Furthermore, we will clarify in the paper that we are not directly addressing model collapse, but rather indirectly preventing it by improving data quality. Additionally, in our experiments, the data we edited is not entirely real data. We also conduct experiments with synthetic data, such as (1) datasets used in continual pretraining (e.g., Biomed, Finance) , (2) OSS-Instruct-75K and Evol-Instruct-110K also contain samples synthesized by ChatGPT.

Please refer to Appendix G, Q7 for more discussion.

[9] Instruction Pre-Training: Language Models are Supervised Multitask Learners (Cheng et al., EMNLP 2024)

[Q4] : The authors should elaborate further on the results.

As shown in Table 3, our method improves performance across both OLMo-1B and Llama-3-8B in successful tasks. In Biomedicine, OLMo-1B’s average score improves from 38.83 to 40.89, and Llama-3-8B from 56.04 to 56.48. Similar improvements are seen in Finance and Math domains. Additionally, Table 2 shows the effectiveness of our approach in general pre-training, with OLMo-1B’s average performance rising from 32.75 to 33.11. In Table 4 (SFT), ToEdit enhances FLAN v2 from 70.18 to 70.65 and boosts task performance in Natural Instructions. However, in more challenging tasks, such as Math, the improvements are limited or negligible. This indicate that data modification alone has limited impact on harder reasoning tasks.

Please refer to Appendix C for detailed discussion.

[Q5] :

1. Some of the experiment settings are missing.
2. The information about the details of the ToEdit experiments is missing.

Please refer to:

1. Appendix F (Line 412) for detailed experimental settings, including pre-training and fine-tuning.
2. Section 5.1 (Line 381) for the ToEdit experiment settings and cost.

Additionally, we add iterative experiments on instruction tuning data in Rebuttal to Reviewer Qb5k.

[Q6] : How does the editing mechanism ensure the quality of the semi-synthetic data?

We conduct pre-ablation on experiments on throhold $p$ to control the editing process and ensure quality of output data. As shown in Table 5, we choose best parameters for main experiments.

[Q7] : How are algorithm metrics selected?

We select our evaluation metrics based on established practices. Please refer to Appendix F.2.

We sincerely thank you for your critical feedback and valuable suggestions. Below, we will strive to address your concerns and refine our paper accordingly.

[Q1] : I think the problem with "synthetic data" is with a particular synthetic data (Cosmopedia) only. This paper can lead to a false impression that all synthetic data suffer from the same issues.

We would like to clarify that our analysis is not meant to imply that all synthetic datasets suffer from the same issues. The analysis and theoretical validation are grounded in previous research on model collapse [1,2,3,4], a potential risk associated with training using synthetic datasets. Specifically, we use Cosmopedia as a representative example to highlight these concerns.

To avoid any potential misunderstanding, we will include the following explanation and clarification in the revision:

1. Cosmopedia is currently the largest open-source synthetic dataset, and it comes with detailed statistical information. We use Cosmopedia as a representative example to illustrate failure modes when using synthetic data. We agree that issues identified may vary depending on the specific dataset or generation pipeline used. However, the results obtained on Cosmopedia should also have a certain level of representativeness.
2. While our analysis primarily focuses on Cosmopedia, we do not intend to imply that all synthetic datasets suffer from identical problems. Rather, we aim to highlight general principles and cautionary lessons about synthetic data generation and evaluation.
3. The tremendous success of synthetic data does not conflict with potential issues, such as model collapse. We provide several examples of the success of synthetic data in the Introduction and Related Work. In Line 33-36 and 799-812, there are listed numerous high-quality synthetic datasets, such as UltraChat, UltraMedical, and so on. Furthermore, famous Phi-1/2 series models are basiclly on synthetic data.

[1] AI models collapse when trained on recursively generated data. Nature 631, 755–759 (2024).

[2] A tale of tails: model collapse as a change of scaling laws. ICML'24

[3] Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating, arXiv, 2024

[4] Position: Model Collapse Does Not Mean What You Think. 2025.