ToEdit: How to Synthesize Text Data to Avoid Model Collapse?

[Q3] : Please introduce all these terms in the Figure caption

Thank you very much for pointing this out. We have updated the following points in the figure caption.

The data $(x_{0},y_{0}) \sim P_{\Sigma,w,\sigma^2}$ on $\mathbb{R}^d \times \mathbb{R}$, $f_0$ is first generation trained model, and $y_1$ is synthetic data, $m_1$ is matrix to specify which need to edit. $T$ and $d$ denote the number of data points and the dimensionality, respectively. Further detailed definitions and derivations can be found in Sec. 3.

[Q4] : What are the key differences between the iterative and non-iterative model collapse? Where are they similar? Why is one more realistic than the other and why? Please focus on that.

What are the key differences ?

• From a setting perspective: the difference between the two lies in their scope: non-iterative model collapse is not confined to training on self-generated data, enabling it to uncover broader properties of synthetic data. For instance, in our paper, we train GPT-2 using the Cosmopedia dataset in a single generation, which was generated by Mixtral-8x7B-Instruct-v0.1. In contrast, iterative model collapse requires training on self-generated data over multiple generations.
• From a property perspective: the non-iterative model collapse highlights the gap between human data and general purely synthetic data, particularly in terms of distributional properties and n-gram features. The iterative model collapse illustrates the iterative evolution of the model, resembling a self-play process. This process illustrates the gradual evolution of self-generated data without involving an analysis of the differences in nature between self-generated and real data [1,2,3].

Where are they similar? They both ultimately lead to model collapse, driven by the same cause—synthetic data, while they investigate different aspects of synthetic data.

Why is one more realistic than the other ? In common model training scenarios, we do not deliberately use self-generated data to train the model itself. The most common setting is training model on a mixture of human and synthetic data, where the synthetic data is not generated by the model itself, even we don't know where this synthetic data comes from. Moreover, numerous popular datasets, such as UltraChat [9] and OpenOrca [10], have already been used to combine synthetic and real data for training. Therefore, studying synthetic data under the non-iterative model collapse setting is more realistic.

We further elaborate on this issue in the revision.

[Q2] : how did authors choose the p=0.99 as their threshold for resampling with token-level editing and what happens if one varies this threshold.

how did authors choose the p=0.99

In our experiments, the value of $p$ was selected through a comparative analysis of hyperparameters. As the table below shows the various performance of different $p$, we selected hyperparameter $p$ by conducting experiments on the validation set of BioMed. As discussed in our paper, an effective synthetic dataset generation method needs to balance real data distribution and modification extent. By weighing the performance and the proportion of tokens involved, we ultimately selected a value of 0.99.

what happens if one varies this threshold

A larger $p$ indicates fewer tokens will be resampled, while a smaller $p$ results in more tokens being resampled. Only moderate $p$ yield optimal results, while excessive changes degrade performance, i.e., model collapse. We present example statistics of the BioMed dataset, detailing the number of tokens and their respective percentages, as follows.

We further supplement more experiments with more detailed ablation on hyperparameters. The following tables present experiments on different sampling strategies and sampling sizes. From these experiments, we ultimately selected top-k as the sampling strategy, considering both the efficiency and effectiveness of the sampling process.

We are grateful for your enthusiastic feedback and insightful comments. Below, we give detailed responses to your questions.

[Q1] : Add way more nuance to results with pretraining and SFT.

To further validate the effectiveness of our method, we incorporate more 3 general instruction tuning tasks and 2 code reasoning-related SFT tasks. As for general instruction tuning tasks, we adopt instruction tuning datasets from [2], including CoT [3], FLAN V2 [4], and Open Assistant 1 [5]. As for code-related reasoning tasks, we utilize OSS-Instruct-75K and Evol-Instruct-110K.

The experimental results demonstrate that the ∆ToEdit consistently improves performance across both general instruction tuning and code-related SFT tasks. For code-related tasks, the improvements are particularly evident in ARC-Challenge and GPQA, indicating better reasoning and code comprehension. From the trends observed, our method demonstrates partial improvements across multiple datasets, indirectly validating its effectiveness.

Regarding pretraining, due to the significant time and storage cost of downloading large-scale data and performing pretraining from scratch, related experiments are still being supplemented.

1. General Instruction tuning.

2. Code SFT tasks; OSS-Instruct-75K and Evol-Instruct-110K are code-related supervised tasks, containing python, cpp, java and php languages.

[Q10] : what percentage of the original tokens are edited (on average per dataset) when you use token-level editing with p=0.99?

We present the token probability statistics as percentages below. As we mentioned above, a larger $p$ indicates fewer tokens will be resampled, while a smaller $p$ results in more tokens being resampled. Specifically, for $p=0.99$ at generation 1, 12.5% tokens are resampled.

[Q8] : One important question here is how did you choose this number of 0.99, and what would happen if this number were different? How do we generalise any findings (or can propose any future work) based on your results?

As we mentioned in [Q1], we conduct a series of experiments on the validation set, then select the suitable hyper-parameter $p$. A larger $p$ indicates fewer tokens will be resampled, while a smaller $p$ results in more tokens being resampled. As stated in line 238-244, we need to refine the data while preserving the distribution of the real data. Based on Figure 6 and Supplementary Tables in [Q1], we select the optimal hyperparameter using the validation set and apply it consistently across all experiments.

propose any future work based on your results The discussion in our paper and above highlights: the key to improving the quality of synthetic data lies in balancing long-tail distribution preservation and optimizing the synthetic data mix ratio. In other words, we should focus on how to generate more informative synthetic data and how to integrate it with real data for better performance. Building on this foundation, future improvements can focus on two aspects: first, obtaining more information gain, designing more efficient generation mechanisms to inject valuable information into the synthetic data; second, optimizing methods to reduce noise during the synthesis process. This approach ensures that synthetic data retains its authenticity while enhancing its utility in practical tasks.

[Q9] : In line 211, you may want to explain how the DSIR sampling works and why you use it in your data analysis.

DSIR [11] works by estimating importance weights for each data sample to measure its relevance to the target distribution. This involves three main steps: first, we leverage n-gram models to fit two distributions of human and synthetic data, $q_{feat}$ and $p_{feat}$, which represent the target and raw distributions, respectively. We use them to compute the likelihood ratio for each sample. Next, we calculate the importance weight for each sample $w_i = \frac{p_{feat}(z_i)}{q_{feat}(z_i)}$, which quantifies how well the sample aligns with the target distribution. Finally, we perform importance-weighted sampling without replacement to select examples.

We use DSIR in our data analysis as it allows for principled and computationally efficient selection of synthetic data points that align with the target distribution. And the importance weight also reflects the alignment between the n-gram features of synthetic data and human data. Using DSIR, we can analyze the differences between synthetic and human data across n-gram feature distributions and data matching. As shown in Fig.5, It is challenging to select synthetic data that matches human data characteristics under the significant distribution difference. To obtain high-quality synthetic data, it is essential to focus on improving the data synthesis method.

[Q5] : One stark difference in Fig.2 you do not discuss is a bimodal distribution of ppl on human data. Or is that an artifact from the binning strategy you were using (if any)?

Thank you for pointing this out. The bimodal distribution of PPL on human data is likely a property of the underlying data distribution rather than an artifact of the binning strategy. We tested this behavior using the other model and observed a similar distribution (Figure 9), which suggests it is inherent to the dataset itself. On the other hand, this phenomenon does not impact our main conclusions, as our findings remain consistent across different models and settings.

[Q6] Why did you choose to use Figures to motivate your work instead of using more objective metrics, like those that measure the similarity between probability distributions ?

Visualizing data statistics offers a direct and intuitive way to highlight the differences between human and synthetic data. We think this can help readers quickly understand our key findings and research questions.

Not sure I fully agree with the discussion around Fig.2 (lines 190-198). I find the plots for 1M human texts and 1M synthetic texts very similar.

I am assuming you are referring to Fig. 4 (lines 190-198)?

If so, the bottom plot in Fig.4 shows the differences in the n-grams distribution between huamn and synthetic data. If the values in the bottom plot approach zero, the two distributions are nearly identical. However, across 10,000 hash buckets, the differences in log probabilities fluctuate around $y=0$, indicating that the human texts and synthetic texts are not similar.

If you have further questions, we will provide more clarification.

[Q7] : I do not think that a line plot is the best setting for Fig.5B, you would probably be better off with a bar plot here. The lines hint at the wrong intuition that they have meaning, whereas only the dots have meaning.

Thank you for your advice. We have updated a bar plot for Fig.5B in our revision.

We greatly appreciate your further questions and feedback, your feedback is highly important to us, and we will do our best to address your concerns below.

[Q1] Typos

Thank you for pointing out the typos. We have carefully reviewed the entire paper and corrected typos in the revision.

[Q2] In Figure 1, there are still many things to be improved.

Based on your suggestions, we have made the following updates:

• We have annotated all the processes and symbols at the bottom of the Figure 1 to facilitate quick reference for readers.
  • Source Real Data: $(x_0, y_0)$
  • Synthetic Data: $(y_1, y_2, \dots, y_n)$
  • Number of Iterations: $i \in \{1, \dots, n\}$
  • Test Error: $E_{\text{test}}$
  • Trained Model: $f_i$
  • Input Dimensions: $d$
  • Data Size: $T$
  • Label Noise Scalar: $\sigma$
  • Editing Operation Matrix: $m_i$
• We have revised the caption to quickly explain the figure in conjunction with the symbols.
  • "Specifically, starting from real data $(x_o, y_o)$, the test error $E_{test}$ increases as $f_0$ is iteratively trained on synthetic data $(y_1, y_2, \dots, y_n)$. Our method, ToEdit, utilizes $f_0$ and an operation matrix $m_i$ to edit the data, achieving a fixed upper bound."

[Q3] I just do not think authors succeed in linking these findings to actionable insights that can guide researchers in their future work, which to me would make this a paper with impact.

we must assume we have data which is 100% human authored,

We do not need to assume that the data is 100% human authored.

In experiments, some datasets used in our experiments include partially synthetic data.

• Datasets used in continual pretraining (e.g., Biomed, Finance) include partially synthetic data, which has been reformatted into a reading comprehension structure [15].

• OSS-Instruct-75K and Evol-Instruct-110K also contain samples synthesized by ChatGPT.

In the theoretical framework, synthetic data is generated iteratively through an $n$-generation process. (1) If the starting point is a real distribution, our method preserves most of the initial distribution to generate higher-quality data. (2) If the starting point is a mixture of synthetic and real data, the modifications are minimal, ensuring the original distribution remains largely unaffected. Therefore, applying our method in any generation $i$, we can further avoid issues, such as reduced variance and diminished diversity, which are key factors contributing to model collapse.

In other words, whether the current data is fully real or a mix of real and synthetic, using it as an anchor to synthesize data, our method builds upon the current data distribution to achieve improvements, rather than causing model collapse.

In summary, we aim to improve the data synthesis method, specifically focusing on how to obtain higher-quality data from the existing datasets. We do not need to assume that the data at hand is 100% human-generated. Our algorithm is designed to minimize excessive distribution truncation of the original data.

[15] Cheng D, Huang S, Wei F. Adapting large language models via reading comprehension[C]//The Twelfth International Conference on Learning Representations. 2023.

and struggle to see how it would impact future work.

Based on the above discussion, our approach needn't assume the original data is 100% human-authored. In contrast, our approach can be applied to optimize the current data, even if it is a mixture of real and synthetic data. From the findings and proposed method in our paper, we can influence future research in the following aspects:

• Potential applications of our work: (1) Data optimization. We can quickly modify and optimize the current data, using a trained language model with a single forward pass. (2) Regularization in data synthesizing process. When synthetic data becomes excessive, we can introduce real data as an anchor to balance the issues of excessive homogeneity and tail distribution cut-off in synthetic data, thereby preventing mode collapse.
• Lessons from our work: The key to improving the quality of synthetic data lies in balancing long-tail distribution preservation and optimizing synthetic data approaches. In other words, we should focus on two questions: how to generate more informative synthetic data and how to effectively integrate it with real data. Building on this foundation, future improvements can focus on two aspects: first, obtaining more information gain by designing more efficient generation mechanisms to inject valuable information into the synthetic data; and second, optimizing methods to reduce noise during the synthetic process. This approach ensures that synthetic data retains its authenticity while enhancing its utility in practical tasks.

[Q4] : The paper uses terms like "non-iterative model collapse" and "coverage collapse." It would be helpful broader audience to better understand these terms if it was clearly defined.

non-iterative model collapse We provide a detailed explanation of non-iterative model collapse as below, and offer a comparison with orignal iterative model collapse.

We define “non-iterative model collapse” as the performance degradation caused by directly mixing general synthetic data without iterative training. Theoretically, without additional regularization constraints to guide data generation, the variance of the model-generated data gradually decreases during this process. The diversity of the generated data diminishes over time, ultimately leading to the collapse of the model itself.

Difference with iterative model collapse

From a setting perspective: the difference between the two lies in their scope: non-iterative model collapse is not confined to training on self-generated data, enabling it to uncover broader properties of synthetic data. For instance, in our paper, we train GPT-2 using the Cosmopedia dataset in a single generation, which was generated by Mixtral-8x7B-Instruct-v0.1. In contrast, iterative model collapse requires training on self-generated data over multiple generations.

From a property perspective: the non-iterative model collapse highlights the gap between human data and general purely synthetic data, particularly in terms of distributional properties and n-gram features. The iterative model collapse illustrates the iterative evolution of the model, resembling a self-play process. This process illustrates the gradual evolution of self-generated data without involving an analysis of the differences in nature between self-generated and real data [1,2,3].

They both ultimately lead to model collapse, driven by the same cause—synthetic data, while they investigate different aspects of synthetic data.In common model training scenarios, we do not deliberately use self-generated data to train the model itself.

Non-iterative model collapse is more realistic. The most common setting is training model on mixture of human and synthetic data, where the synthetic data is not generated by the model itself, even we don't know where this synthetic data comes from. Moreover, there are already numerous popular datasets that combine synthetic and real data for training, such as UltraChat [4] and OpenOrca [5]. Therefore, studying synthetic data under the non-iterative model collapse setting is more realistic.

Coverage collapse

“Coverage collapse” refers to a phenomenon where synthetic data is distributed within a much narrower range compared to human data, even when the data sizes are the same. For instance, as shown in Figure 3, the PPL range of synthetic data is confined to [0, 14], whereas the PPL range of human data spans [0, 100]. Despite this disparity, the integral of the two distributions remains the same. This significant distribution gap is what we define as “coverage collapse.”

[Further Suggestion]

Thank you very much for your insightful suggestions. These are highly valuable and align closely with the directions we are currently exploring. We agree that using an ensemble of models with varied pretraining sources could improve the robustness and diversity of token selection. To address this, we plan to explore ensemble-based strategies, including model voting, to enhance token selection quality and minimize single-model biases. These investigations will help determine whether ToEdit’s effectiveness requires adaptations for specific configurations. Relevant experiments are currently underway.

[Q3] : Although common, it might still be useful to expand that PPL stands for perplexity. It will be helpful for new researchers.

PPL stands for perplexity, a commonly metric in NLP to evaluate the quality of language models. It measures how well a probabilistic model predicts a given dataset, with lower values indicating better performance. Formally, the perplexity of a language model is calculated as:

$\text{PPL} = 2^{-\frac{1}{N} \sum_{i=1}^{N} \log_2 P(x_i)}$

Alternatively, it can also be expressed as:

$\text{PPL} = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(x_i)\right)$

Where $N$ is the number of tokens in the dataset, $P(x_i)$ is the predicted probability of the $i$-th token. Perplexity essentially represents the exponential of the average negative log-likelihood of the predicted tokens, indicating how “perplexed” the model is when making predictions. In our implementation, we use the following code to calculate PPL:

train_outputs = model(**model_inputs)
neg_log_likelihood = train_outputs.loss
ppl = np.exp(stabilized_loss)

[Q2] It would help if the paper provided further clarification on the specific threshold values used for token-level editing (e.g., the probability threshold $p=0.99$.

We supplement 4 experiments on hyper-parameter $p$, including: (1) ablation studies of values , (2) token percentage statistics, (3) comparisons of sampling strategies , and (4) an ablation study on sampling size.

As frist table below shows different $p$ influences on BioMed, different values lead to fluctuations in data performance. The second table below presents the distribution percentages across different probability value ranges. As stated in line 241-244, we need to refine the data while preserving mainly source distribution. As shown in Figure 3, a larger $p$ indicates fewer tokens will be resampled, while a smaller $p$ results in more tokens being resampled. Balancing performance and the preservation of data distribution, we set $p=0.99$ as threshold for our experiments. The third below shows the results of different sampling strategies. Specifically, to control variables, we set $k=8$ for top-k sampling and $p=0.99$ for top-p sampling. We use reject sampling implementation in [11]. The results of reject sampling, top-p, and top-k are comparable. However, top-p involves a dynamic sampling range, and reject sampling requires multiple rounds of computation, leading to increased overhead. Considering computational efficiency, we chose top-k for sampling. This aligns with our original objective of maintaining minimal computational overhead. This aligns with our initial objective of minimizing computational overhead as much as possible. The final table shows the ablation study on sampling size of top-k. The improvement achieved with larger values is relatively small. Therefore, we chose $k=8$ in our experiments.

Performance impact of different $p$

Token distribution across different probability ranges.

Results of different sampling strategies.

Ablation study on sampling size on $k$ of top-k.

We sincerely appreciate your positive comments and support for our acceptance. In the following, we will address your concerns and refine our paper accordingly.

[Q1] ToEdit would work on other data types (e.g., code, tabular data)?

To further validate the effectiveness of our method, we incorporated 2 code reasoning-related SFT tasks and 3 general instruction tuning tasks. As for general instruction tuning tasks, we adopt instruction tuning datasets from [7], including CoT [3], FLAN V2 [4], and Open Assistant 1 [5]. As for code-related reasoning tasks, we utilize OSS-Instruct-75K and Evol-Instruct-110K.

The experimental results demonstrate that the ∆ToEdit consistently improves performance across both general instruction tuning and code-related SFT tasks. In general instruction tuning, ∆ToEdit enhances average performance across CoT, FLAN V2, and Open Assistant, with notable gains in BoolQ, SIQA, and Winogrande. For code-related tasks, the improvements are particularly evident in ARC-Challenge and GPQA, indicating better reasoning and code comprehension. These findings validate the effectiveness of the proposed modifications in enhancing both generalization and task-specific performance.

1.Code SFT tasks; OSS-Instruct-75K and Evol-Instruct-110K are code-related supervised tasks, containing python, cpp, java and php languages.

2. General Instruction tuning.

[Q8] : Can you elaborate the relationship between perplexity improvements and actual model capabilities.

We further include 22 validation datasets from the Pile and 7 general understanding downstream tasks, to analyze the relationship between perplexity improvements and actual model capabilities. The testing framework follows [4]. Specifically, as shown in first table below, the PPL decreases as the proportion of purely synthetic data decreases. In second table , the performance on downstream tasks similarly exhibits a gradual improvement with the decline in synthetic data. Although the improvement in PPL does not directly translate to an equivalent increase in downstream task accuracy. It can be observed from the trend that improvements in PPL are positively correlated with enhancements in downstream task performance. The relationship between PPL improvement and downstream performance gain depends on more complicated factors, like instruction tuning methods and base model capabilities.

PPL evaluation results on 22 validation using the testing framework in [1].

Downstream tasks evaluation using the testing framework in [1].

[Q6] : Results Interpretation

The improvements shown in Tables 1-3 could use more discussion about statistical significance

To further validate the effectiveness of our method, we incorporate more 3 general instruction tuning tasks and 2 code reasoning-related SFT tasks. As for general instruction tuning tasks, we adopt instruction tuning datasets from [2], including CoT [3], FLAN V2 [4], and Open Assistant 1 [5]. As for code-related reasoning tasks, we utilize OSS-Instruct-75K and Evol-Instruct-110K.

The supplement tables below demonstrate that the ∆ToEdit consistently improves performance across both general instruction tuning and code-related SFT tasks. For code-related tasks, the improvements are particularly evident in ARC-Challenge and GPQA, indicating better reasoning and code comprehension. From the trends observed, our method demonstrates partial improvements across multiple datasets, indirectly validating its effectiveness.

More discussion of Tables 1, 2, and 3: Experiments in our paper demonstrate the effectiveness of our method across the three critical stages of language model training: continual pre-training, pre-training, and fine-tuning. Our method enhances model performance on downstream tasks without increasing data size, showcasing its ability to tap into the potential of existing data. Additionally, these results confirm that semi-synthetic data serves as a viable approach for generating high-quality data and improving model generalization.

As for Table 1, our method achieves consistent improvements over the baseline data for both OLMo-1B and LLaMA-3-8B models. For example, in the Biomedicine domain, OLMo-1B improves from 38.83 to 40.89, while LLaMA-3-8B increases from 56.04 to 56.48. These improvements demonstrate that our ToEdit strategy effectively enhances learning across diverse domains.

As for Table 2, it further supports the robustness of our approach in general pre-training tasks. The average performance of OLMo-1B increases from 32.75 to 33.11, reflecting improved generalization capabilities. While the improvement is modest, the consistent trend across tasks like PIQA, BoolQ, and ARC-c highlights the broader applicability of our method.

As for Table 3, it illustrates the benefits of our method in fine-tuning tasks. The average performance of LLaMA-3-8B improves from 69.34 to 69.70, with notable gains in tasks like Winogrande and SIQA. Similarly, in following table, ToEdit improves other data such as FLAN V2 and Open Assistant 1, with average performance increasing from 70.18 to 70.65 for FLAN V2. These improvements, demonstrate the adaptability of our method to instruction-tuning tasks.

1. General Instruction tuning.

2. Code SFT tasks; OSS-Instruct-75K and Evol-Instruct-110K are code-related supervised tasks, containing python, cpp, java and php languages.

[Q7] : Can you please provide justification for choosing p =0.99 and k =8

We have supplemented additional experiments and conducted a detailed analysis; please refer to [Q2].

[Q5] : Experimental Setup

Figure 5 shows results about "DSIR-Selected Data" but details about this selection process aren't well explained

DSIR [11] works by estimating importance weights for each data sample to measure its relevance to the target distribution. This involves three main steps: first, we leverage n-gram models to fit two distributions of human and synthetic data, $q_{feat}$ and $p_{feat}$, which represent the target and raw distributions, respectively. We use them to compute the likelihood ratio for each sample. Next, we calculate the importance weight for each sample $z_i$ as $w_i = \frac{\hat{p}_{\text{feat}}(z_i)}{\hat{q}_{\text{feat}}(z_i)}$, which quantifies how well the sample aligns with the target distribution. Finally, we perform importance-weighted sampling without replacement to select examples.

We use DSIR in our data analysis as it allows for principled and computationally efficient selection of synthetic data points that align with the target distribution. And the importance weight also reflects the alignment between the n-gram features of synthetic data and human data. Using DSIR, we can analyze the differences between synthetic and human data across n-gram feature distributions and data matching. As shown in Fig.5, It is challenging to select synthetic data that matches human data characteristics under the significant distribution difference. To obtain high-quality synthetic data, it is essential to focus on improving the data synthesis method.

Details about the continual pre-training setup across different domains (Biomedicine, Finance, Math) are sparse

We follow [13] to conduct continual pre-training on Bio, Finance, and Math domains. Specifically, PubMed Abstracts from the Pile are utilized as the pre-training corpora for the biomedicine domain. For the finance domain, financial news data covering over 7,000 stocks from May 2022 to May 2023 is collected using the FinGPT framework. We continue pre-training OLMo-1B and LLaMA-3-8B on each domain. For implementation, we utilized the official training framework for OLMo-1B, leveraging Fully Sharded Data Parallel (FSDP) for continual pretraining. For LLaMA, we adopted the LLaMA-Factory framework to carry out the continual pretraining process.

[Q4] : Distribution Analysis

The U-shaped distribution in Figure 6 needs better explanation of why it's a good indicator for token editing

From the perspective of information theory, we can analyze the filtering potential of U-shape distribution as follows:

We utilize the U-shape distribution (Figure 6) to re-sample tokens in the high-probability region, aiming to adjust the U-shaped distribution toward a uniform distribution. By doing so, we can maximize the information entropy. According to information theory,

Lemma 1: Let $X$ be a discrete random variable with $n$ possible outcomes. If the probability of each outcome is uniform, i.e., $P(x_i) = \frac{1}{n}$ for all $i \in \{1, 2, \dots, n\}$, the Shannon entropy is maximized, given by:

$H(X) = -\sum_{i=1}^{n} \frac{1}{n} \log \frac{1}{n} = \log n.$

This represents the maximum uncertainty achievable, implying that the dataset carries the maximum possible information content. Loosely speaking, the uniform distribution possesses the maximum information entropy.

From the perspective of language model learning, our method emphasizes the importance of poorly learned data. Specifically, we resample easy tokens and encourage the model to focus on learning more challenging ones. Our method can enhance under-learned data learning by resampling high-probability tokens.

The "coverage collapse" phenomenon mentioned multiple times could be better defined and explained

“Coverage collapse” refers to a phenomenon where synthetic data is distributed within a much narrower range compared to human data, even when the data sizes are the same. For instance, as shown in Figure 3, the PPL range of synthetic data is confined to [0, 14], whereas the PPL range of human data spans [0, 100]. Despite this disparity, the integral of the two distributions remains the same. This significant distribution gap is what we define as “coverage collapse.”

"Feature over-concentration" and model performance isn't clearly explained

"Feature over-concentration" refers to the excessive repetition of n-gram features overly repeat and higher similarity in latent space. As shown in Figure 4, 9, and 10, the feature distribution of real data differs significantly from that of synthetic data. In particular, n-grams in synthetic data are heavily concentrated in high-frequency features. This phenomenon results in low data diversity, which intuitively hinders the model’s generalization ability. Consequently, training on such low-diversity data leads to a decline in model performance.

[Q3] : Progressive Decrease in Editing : editing operations decrease with a fixed ratio η seems mathematically convenient but lacks empirical validation, No real-world evidence is provided to support this pattern of decay.

We present the percentage statistics of edited tokens in table below, demonstrating that the edited tokens indeed exhibit a progressive decrease. Specifically, We observe that the percentage of edited tokens (above the threshold $p>0.99$) decreases as the generation number increases. Theoretically, this is a process of distribution shifting. When tokens ($p>0.99$) are resampled, randomness is introduced. The sampling process can select tokens with lower probabilities. Then, tokens ($p>0.99$) are replaced, leading to a reduction of edited tokens in subsequent generations. The following table provides real-world evidence for this pattern of decay.

Percentage of tokens requiring edits in the Natural-Instructions dataset. The total number of tokens is 4,671,834.

[Q2] : Token Resampling Strategy.

The choice of top-k sampling with k=8 seems arbitrary. We conduct experiments on a validation set to choose a sampling strategy and sampling size. The first below shows the results of different sampling strategies. Specifically, to control variables, we set $k=8$ for top-k sampling and $p=0.99$ for top-p sampling. We use reject sampling implementation in [14]. The results of reject sampling, top-p, and top-k are comparable. However, top-p involves a dynamic sampling range, and reject sampling requires multiple rounds of computation, leading to increased overhead. Considering computational efficiency, we chose top-k for sampling. This aligns with our original objective of maintaining minimal computational overhead. This aligns with our initial objective of minimizing computational overhead as much as possible. Table 4 shows the ablation study on sampling size of top-k. The improvement achieved with larger values is relatively small. Therefore, we chose $k=8$ in our experiments.

Results of different sampling strategies.

Ablation study on sampling size on $k$ of top-k.

We deeply appreciate your positive comments and valuable feedback. In the following, we will address your concerns and refine our paper accordingly.

[Q1] :Parameter Sensitivity, There's no ablation study showing how different threshold values affect performance.

We supplement 4 experiments on hyper-parameter $p$, including: (1) ablation studies of values , (2) token percentage statistics, (3) comparisons of sampling strategies (see in Q2), and (4) an ablation study on sampling size (see in Q2) . As the first table below shows different $p$ influences on BioMed, different values lead to fluctuations in data performance. The second table below presents the distribution percentages across different probability value ranges. As stated in lines 241-244, we need to refine the data while preserving mainly source distribution. As shown in Figure 3, a larger p indicates fewer tokens will be resampled, while a smaller p results in more tokens being resampled. Balancing performance and the preservation of data distribution, we set p=0.99 as the threshold for our experiments.

The ablation studies of $p$ on BioMed.

Statistics of token percentages and counts across probability ranges.

[Q2] Flawed Experimental Design for Main Claims

We provide the domain statistics for Dolma, demonstrating that It is not domain mismatching problem. We supplement the testing results followed your suggestion above, which are consistent with the findings reported in our paper.

It is domain mismatching problem We can't fully agree on this point, it is not domain mismatching problem. As shown in below table, C4 and wikipedia account for approximately 6.86% and 0.14%, respectively, of this dataset. Specifically, Dolma (v1.6) comprises 7 subdomains: C4, Wikipedia, Wikibooks, Common Crawl, Reddit, PeS2o, Project Gutenberg, and The Stack. This small portion of in-domain data has minimal impact on PPL testing.

Cosmopedia's data distribution differs significantly from web-crawled data, making the comparison with RedPajama and RefineWeb potentially unfair. From a pretraining perspective, Cosmopedia was generated by Mixtral-8x7B-Instruct-v0.1, which was likely trained on the same open-source datasets as well. Moreover, Cosmopedia is currently the largest publicly available open-source general synthetic dataset, created by diverse prompts. It comprises 7 subdomains: web_samples, stanford_edu, wikihow, openstax, khanacademy, and automathtext.

Additionally, we are not claiming that synthetic data is ineffective; rather, we emphasize that different synthetic data generation methods lead to varying outcomes. Therefore, we compared Cosmopedia and other data mixture using the same general PPL testing. To provide a more thorough explanation, we apply the testing framework from [1], which includes 22 validation datasets and 7 downstream tasks. The additional experimental results are consistent with the findings reported in our paper.

Dolma dataset statistics (v1.6) (Quoted from here)

Downstream tasks evaluation using the testing framework in [1].

We sincerely thank you for your critical feedback and valuable suggestions. Below, we will strive to address your concerns and refine our paper accordingly.

[Q1] : Significant Literature Gap and Contradictory Findings

This fundamental contradiction needs to be addressed and explained

We should emphasize that the conclusions in [1] do not contradict ours; on the contrary, the findings in [1] are consistent with ours.

Cosmopedia [12] is produced by pure synthetic data method (prompting LLMs), but both Rephrasing the Web[1] and ours are semi-synthetic data methods (reformatting and token replacing). The former cut-offs long-tail distribution [7], while the latter preserves and adjusts it.

Specifically, both Rephrasing the Web [1] and token-level editing (ours) aim to refine data while preserving the original source distribution, producing semi-synthetic data. In contrast, purely synthetic data in Cosmopedia lacks the long-tail distribution (Figure 3) and overly concentrates on n-gram features (Figure 4). Ultimately, semi-synthetic data enhances training performance, whereas purely synthetic data results in model collapse. In other words, replacing the whole real sample with synthetic data can damage the performance.

We provide a detailed comparison of Cosmopedia, Rephrasing the Web [1] and ours in the below table. The primary distinction between Cosmopedia, [1], and our approach lies in the degree to which the original human data distribution is preserved.

I would suggest to follow the evaluation framework from [1] for better comparison

We have followed your suggestion to conduct evaluation framework from [1]. The results are consistent with our findings. Specifically, the PPL increases as the proportion of purely synthetic data grows, while the performance on downstream tasks similarly exhibits a gradual decline with the increase in synthetic data.

PPL evaluation results on 22 validation using the testing framework in [1].

[1] Maini P, Seto S, Bai H, et al. Rephrasing the web: A recipe for compute and data-efficient language modeling[J]. arXiv preprint arXiv:2401.16380, 2024.

[2] Xia M, Malladi S, Gururangan S, et al. Less: Selecting influential data for targeted instruction tuning[J]. arXiv preprint arXiv:2402.04333, 2024.

[3] Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in neural information processing systems, 2022, 35: 24824-24837.

[4] Longpre S, Hou L, Vu T, et al. The flan collection: Designing data and methods for effective instruction tuning[C]//International Conference on Machine Learning. PMLR, 2023: 22631-22648.

[5] Köpf A, Kilcher Y, von Rütte D, et al. Openassistant conversations-democratizing large language model alignment. CoRR, abs/2304.07327, 2023. doi: 10.48550[J]. arXiv preprint arXiv.2304.07327.

[6] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631(8022):755–759, 2024.

[7] Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. arXiv preprint arXiv:2402.07712, 2024a.

[8] Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws. arXiv preprint arXiv:2402.07043, 2024b.

[9]Ding N, Chen Y, Xu B, et al. Enhancing chat language models by scaling high-quality instructional conversations[J]. arXiv preprint arXiv:2305.14233, 2023.

[10] Lian W, Goodson B, Pentland E. OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces[EB/OL].(2023)

[11] Xie S M, Santurkar S, Ma T, et al. Data selection for language models via importance resampling[J]. Advances in Neural Information Processing Systems, 2023, 36: 34201-34227.

[12] Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, 2024. URL https://huggingface.co/datasets/HuggingFaceTB/ cosmopedia.

[13] Cheng D, Huang S, Wei F. Adapting large language models via reading comprehension[C]//The Twelfth International Conference on Learning Representations. 2023.

[14] Liu T, Zhao Y, Joshi R, et al. Statistical rejection sampling improves preference optimization[J]. arXiv preprint arXiv:2309.06657, 2023.

[Q5] The observed probability distribution may be natural rather than indicating filtering potential.

We appreciate you for providing an excellent reference. We provide the following discussion for U-shape distribution indicate filtering potential:

From the perspective of language model learning, our method emphasizes the importance of poorly learned data, as phenomena like linguistic entropy you provided. Specifically, Our method can enhance under-learned data learning by resampling high-probability tokens.

Furthermore, we can analyze the filtering potential of U-shape distribution through the lens of information theory. According to information theory:

Lemma 1: Let $X$ be a discrete random variable with $n$ possible outcomes. If the probability of each outcome is uniform, i.e., $P(x_i) = \frac{1}{n}$ for all $i \in \{1, 2, \dots, n\}$, the Shannon entropy is maximized, given by:

$H(X) = -\sum_{i=1}^{n} \frac{1}{n} \log \frac{1}{n} = \log n.$

This represents the maximum uncertainty achievable, implying that the dataset carries the maximum possible information content. Loosely speaking, the uniform distribution possesses the maximum information entropy. To leverage this property, we utilize the U-shape distribution (Figure 6) to re-sample tokens in the high-probability region, aiming to adjust the U-shaped distribution toward a uniform distribution. By doing so, we can maximize the information entropy.

[Q4] Weak Empirical Results

Baseline performance is concerningly low. The baselines model (OLMo-1B) is a relatively weaker model of reasoning, which is trained on open-source corpora. However, it provides more open-source data and code, facilitating us to conduct research.

To further validate the effectiveness of our method, we incorporate more 3 general instruction tuning tasks and 2 code reasoning-related SFT tasks. As for general instruction tuning tasks, we adopt instruction tuning datasets from [2], including CoT [3], FLAN V2 [4] and Open Assistant 1[5]. As for code-related reasoning tasks, we utilize OSS-Instruct-75K and Evol-Instruct-110K.

The experimental results demonstrate that the ∆ToEdit consistently improves performance across both general instruction tuning and code-related SFT tasks. In general instruction tuning, ∆ToEdit enhances average performance across CoT, FLAN V2, and Open Assistant, with gains in SIQA, and Winogrande. For code-related tasks, the improvements are particularly evident in ARC-Challenge and GPQA, indicating better reasoning and code comprehension. These findings validate the effectiveness of the proposed methods in enhancing both generalization and task-specific performance.

3 more tasks of general Instruction tuning.

2 more Code SFT tasks; OSS-Instruct-75K and Evol-Instruct-110K are code-related supervised tasks, containing python, cpp, java and php languages.

Results of different sampling strategies.

Ablation study on sampling size on $k$ of top-k.

[Q3] Methodology Concerns: The non-autoregressive token replacement approach may compromise text coherence and grammatical correctness

Thank you very much for pointing out such an important point. We will address this in detail.

Since data synthesis is a prerequisite for model training, we aim to ensure that the cost of data synthesis does not exceed the cost of training itself. When designing data synthesis algorithms, we must balance synthesis efficiency and effectiveness, considering both autoregressive and non-autoregressive approaches. Autoregressive methods leverage the LMs to generate coherent text sequentially. In contrast, non-autoregressive methods resample individual tokens based on their probability distributions.

Specifically, our ToEdit modifies data using the probability distribution generated in a single forward pass. For instance, if the generated sequence length is 1024, the computational cost of autoregressive methods would be 1024 times higher than ours. This efficiency advantage is why our method can run effectively on GPUs like the 3090 or 4090 series.

However, we acknowledge that non-autoregressive methods may introduce the semantic inconsistency problem. Resampled tokens may not fit seamlessly into a given sentence. To address this issue, we determine the sampling threshold through validation experiments, aiming to reduce aggressive replacements. We perform sampling in high-probability regions, focusing on tokens that are relatively easier to predict. And we manually verify and tune threshold on a validation set before applying it. A larger $p$ indicates fewer tokens will be resampled, while a smaller $p$ results in more tokens being resampled.

We supplement more experiments and discussion about hyper-parameter $p$. As table below shows different $p$ influence performance on BioMed, different values lead to the fluctuations in data performance. The second below presents the distribution percentages across different probability value ranges. Balancing the sampling rate, token proportion distribution, and performance, we selected $p=0.99$. The final two tables respectively present the results of different sampling strategies and the ablation study on hyperparameters.

Performance impact of different $p$

Statistics of token percentages and counts across probability ranges.

Thanks for your response, and I really appreciate the authors' additional experiments and clarifications. However, these don't address my original concerns about:

• The claim in Line 83 contradicting previous work
• Data leakage in Figure 2's evaluation and domain issues
• The problematic and insufficiently explained non-autoregressive method for generating better synthetic data
• The lack of statistical significance in the evaluation results (In the updated manuscripts, both Table 2 and Table 3 show very small improvements with no mean/variance reported)

So I would keep my initial rejection position.

1. The claim in Line 83 is challenged by previous work WRAP[1], which wasn't discussed in the initial version. Moreover, this claim is now also challenged by this submission's new experiments:

• The new results show that different synthetic datasets yield different conclusions about performance impact: Cosmopedia degrades model performance, while WRAP[1] and ToEDIT improve it. This challenges the submission's fundamental claim in line 83 that "we find that directly mixing general synthetic data, without iterative training, leads to performance degradation."

• The definitions of semi-synthetic and pure-synthetic lack scientific rigor. In fact, Cosmopedia, WRAP, and TOEDIT synthesize data conditioned on different levels of real data:

  • Cosmopedia expands real data and generates data without real data
  • WRAP generates paraphrases of real data
  • TOEDIT performs token editing on real data These three methods clearly modify real data at large, medium, and small scales respectively. The concept of pure synthetic data is ambiguous, as WRAP is also fully generated by an LLM - couldn't it be considered pure synthetic? Additionally, Figure 1 uses "complete synthetic data" - do you mean "pure synthetic data"?

2. The experiments in Figure 2 remain the primary issue warranting rejection:

The authors overlooked that C4 is a subset of CommonCrawl. The table in Response-2 clearly shows that CommonCrawl is the largest portion of Dolma. Therefore, training with Dolma and evaluating with C4 is clearly an in-domain evaluation. C4 and Wikipedia are part of Dolma, which is the pretraining data. Testing with C4 and Wikipedia should be avoided. This isn't an in-domain versus out-of-domain issue - it's a test data leakage issue.

As shown in WRAP[1], equations (1) and (2) demonstrate that it's unfair to compare the perplexity of models trained on synthetic data with those trained on real data, as the latter are optimized to minimize perplexity over real data. Thus, they should naturally achieve lower perplexity than models trained on a mixture of real and synthetic data.

Consequently, I remain concerned about all results and conclusions from Figure 2, which I consider a significant issue in ML research methodology.

3. Regarding non-autoregressive token replacement:

• While I agree this is computationally efficient,
• The new results contradict the selection of p=0.99. According to this table, p=0.1 performs best.
• It's unclear why replacing words that the model is highly confident about (which are likely common words) would be beneficial. What do these tokens get replaced with? A key concern is that pronouns and articles typically have high probability - why would replacing them make sense? Could you explain what tokens with p=0.99 probability would be replaced with? In my understanding, if one words get >0.99 probability, the rest choices are all lower than 0.01 which means they are improper to be here.

4. For the results in Response-4 and the updated draft, could you report means and variances? The improvements remain very small and inconsistent.

Finally, I am sorry I late in responding as it has been took me some time to read all these new results and discussion. But luckily the discussion period has been extended and I will response quickly in the next several day.

Thanks for your examples.

1. Can you answer my question about pronoun and articles?
2. Can you answer my question about what's the exact resampling strategy TOEDIT is using? Also, I think this is an important detail but why it is missing from the submission?
3. Can you answer my question about how p=0.99 was selected? According to the ablation results during rebuttal, p=0.99 is not the best. You proposed a hypothesis about keeping long-tail property, but it is contradicted to the experiment results.
4. The examples are all whole word replacement, but in practice, the replacement would happen for subwords. For example, given a word A, if it is split into subwords B and C. C's probability would be very high when conditioned on B but it is weird to replace C but keep B unchanged. how does TOEDIT handle this case?
5. Can you also provide a sequence of subwords and the probabilities of each subword for any of the examples you posted here?

We would like to kindly bring your attention to the responses posted:

• We have presented the resampling strategy at line 424 in Sec. 4. Discussions and ablation experiments of the resampling strategy are presented in Appendix E and Table 7,8. We also discussed with other reviewers in Reponse -a and Reponse - b . The same reponse were also presented to you in Reponse-3.2 .
• We have discussed why we choose $p=0.99$ empirically and theoretically in follow-up reponse 4 and Appendix E.
• " A hypothesis about keeping long-tail property" is not new. We have mentioned this motivation repeatedly in the Introduction (line 052), Sec 2.2, Sec 3.1 Method, Appendix F.7, and Figure 3.

---

We truly acknowledge and respect your comments. Our paper focuses on "How to synthesize text data to avoid model collapse ?" The model collapse problem is our primary concentration. Around this topic:

• We have demonstrated that our method (1) theoretically avoids model collapse, with a fixed upper bound proved in Section 3 and Appendix B, and (2) empirically enhances performance in pre-training, continual pre-training, and SFT, as shown in Tables 1, 2, and 3.
• We also present ablation experiments, including different $p$ values, sampling strategies, and percentage statistics (Tables 4, 7, 8, and 9), along with case studies (Follow-up Response-4).
• Further detailed discussions in the Appendix includes the differences between iterative and non-iterative model collapse, filtering potential, coverage collapse, the DSIR mechanism, potential applications, and a rigorous definition of synthetic data (Follow-up Response-1).

As for the part of speech (POS) of token within samples (e.g., pronouns or articles), we treat all words equally and do not implement specific designs for different POS. This does not affect the effectiveness of our method. Below, we present our source code, where you can directly refer to the implementation details. Since the extended discussion phase nears its close, we will update the analysis for probabilities of each subword and a supplementary discussion for POS in Appendix.

We will include all additional results in the revision.

Thank you once again for your invaluable suggestion.

def resampling(prob_dict, num_samples=1, beta=1):
        words_candicate = list(prob_dict.keys())
        prob_scores = np.array(list(prob_dict.values()))

        adjusted_probs = softmax(prob_scores / beta) 
        accepted_token = np.random.choice(words_candicate, num_samples, p=adjusted_probs)
    
        return accepted_token