On the Diversity of Synthetic Data and its Impact on Training Large Language Models

In the cluster generation step, the clustering process is localized to the number of samples that can be fit in the context window of the LLM. This can lead to random and unpredictable cluster formations leading to unpredictable diversity scores. Did the authors try any approaches that can mitigate the issue?

We acknowledge that the clustering process is limited by the context window of the LLM, which can constrain the number of samples processed at a time and potentially introduce variability in cluster formation. To mitigate this, our method employs multiple iterations of clustering using K-sized random samples, ensuring that the diversity score reflects a comprehensive representation of the dataset rather than being reliant on any single iteration. This iterative approach allows the model to capture shared patterns and characteristics across samples, leading to more stable and robust cluster formations.

Additionally, the self-verification module acts as a safeguard by identifying and filtering out invalid or inconsistent clusters, which further enhances the reliability of the clustering process. As demonstrated in our results, a significant proportion of clusters flagged by the LLM were corroborated as invalid through human evaluation, confirming the effectiveness of this step.

While the clustering process does not explicitly enforce uniformity across iterations, the robustness of the LLM Cluster score is supported by our ablation studies, which show consistent results across varying $K$ values and repeated runs. This consistency indicates that the iterative pipeline compensates for context window limitations, producing reliable diversity measurements without compromising on computational efficiency.

How is GPT-40 prompted to extract a hierarchy of topics and a set of keywords covered in the page's content?

We use an additional prompt for gpt-4o, feeding the content of the webpage and extract the relevant hierarchical topics and keywords. We also set the expected format of the hierarchical in the prompt with examples.

In the Synthetic Data Generations section, it is unclear how the downweighting of Cosmopediav0.1 and Cosmopediav0.2 was done.

We adjust the sampling weight of Cosmopedia v0.1 and v0.2's to make them effectively 20B. This is achieved via data sampler of our implemented data loader.

There is a brief explanation, but it is still unclear why topic-style persona almost always outperforms multi-topic-styles persona. Intuitively, the latter is supposed to provide more diversity.

The multi-topic-styles persona prompt may introduce redundancy in the synthetic data generation. The redundancy may exist not only in each generation but also across generation, since the multiple topic candidates can overlap.

What would be the average price for carrying out such an experiment due to the presence of multiple proprietary LLM calls?

The average price for each call of the pipeline would be around 0.05$ to get the cluster score.

From the ablation experiments, it looks like when K = 10 and N =10K, the diversity score stabilizes. Do the authors report all their experiments with this configuration?

All main results are reported with K = 10 and N = 5K. Running for larger N would make the results more robust but also more expensive.

Would be interested to know the comparison between the following method say KMeansDiversityScore, where we identify...gathering step has a positive effect in computing good diversity scores.

Thanks for this great advice. We made some initial attempts on this method, i.e., using K-means to identify C clusters and get the initial cluster size from K randomly sampled data before input to our LLM cluster pipeline (without metadata and metrics). The results are comparable to our pipeline We believe diversity is indeed a multi-dimensional factor and combining several metrics together would make the diversity measurement more robust.

Typo at line 093: s\per-training\pre-training

Thanks for pointing our this typo. We have fixed it in our revised paper.

Missing citation ?? at line 1055

Thanks for catching this error! We have fixed it in our revised paper.

If you find our above response well address your concerns, please consider raising the score. If you have further questions, please do not hesitate to let us know.

Thanks for the reviewer's time and efforts on giving suggestions revising our paper. We now address the concerns as follows.

The primary drawback I observe...

It is indeed important to incorporate iterative diversity checks during the generation process to ensure higher-quality outputs. While the current pipeline measures diversity post-generation, intermediate assessments, such as evaluating diversity metrics at smaller intervals (e.g., every 100M tokens), could help identify low-quality data early, allowing adjustments to the generation process accordingly. This approach was not implemented in the current work due to cost overhead and the need to maintain consistent experimental settings. However, it is a promising direction for future work. We plan to explore this refinement in follow-up studies to enhance the efficiency and reliability of synthetic data generation, and also the method to utilize this metric to create more diverse synthetic data on the fly.

Another significant limitation of this approach is its heavy reliance on large pre-trained LLMs to evaluate various criteria.

To ensure the stability and reliability of the generated metrics and metadata, we have implemented several measures. The metadata and metrics generation step is repeated $M$ times, with different $J$-sized samples randomly selected in each iteration. This iterative approach ensures that the results are robust and not biased toward specific subsets of the dataset. Additionally, all experimental results include standard deviations coming from multiple runs with different seeds, demonstrating the reproducibility and internal stability of the proposed pipeline.

In the cluster verification stage, filtered clusters identified by the self-verification module were compared with human evaluations. The results, shown below, demonstrate a strong agreement between LLM-based filtering and human-verified invalid clusters.

This agreement between our framework and human-evaluation confirms the robustness of the self-verification step.

In addition to that we conduct human-study on synthetic data generated using different prompts. Ten evaluators rated the diversity of 50 samples from six synthetic data variants on a scale of 1–6. The results showed strong consistency between human diversity scores and the LLM-based diversity metrics. For instance:

The strong correlation ($r = 0.91, p = 0.011$) demonstrates a statistically significant positive relationship between the Human Diversity Score and the LLM Cluster Score, supporting the validity of our metric.

These findings provide further confidence in the alignment between human judgments and the diversity captured by our pipeline.

To address the concern about inter-rater reliability, we conducted experiments using different LLMs for metadata and metric generation. The comparison of self-verification results with various models is shown below:

The results demonstrate that more capable models, such as GPT-4, are better at identifying invalid clusters, and even smaller open-source models like Llama-3.1 show reasonable performance, albeit with a slight drop in reliability. These findings underscore that the pipeline is adaptable to different models based on cost and resource constraints.

We believe that the combination of human evaluation, iterative verification, and inter-rater reliability tests validates the robustness of the proposed pipeline. The agreement between human judgments and LLM-generated metrics provides additional reliability of the our pipeline.

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We would be happy to provide further clarification if you have any additional questions or concerns.

-- Authors

We thanks for the reviewer's time and effort in reviewing our work, and the constructional feedback on refining our manuscript. Please find our response below.

The number of samples is a cause for concern. (J=5, K=10)

The number of samples is mainly decided by our ablation study and earlier iteration of the prompt. We experiment with multiple values of $J = (3, 5, 10, 15, 30, 50)$ and $K = (5, 10, 15, 20, 50, 100)$. With different values of J in proper range, we find that the metadata and metrics extracted from the data by our LLM agent are overlapped and specific as shown in B.2. Setting J and K to extremely high values like 50 and 100 would lead to performance degradation due to the long context length (when K=50, the context length is roughly 50K): the metadata and metrics becomes more general and the clusters degenerates to certain values, as the results updated here.

We have updated the results with higher J and K values in the revised Appendix of our paper.

Can a cluster score over 10 samples definitely yield an idea of the diversity of the dataset? This could benefit from more supporting evidence.

We need to highlight that the model does not rely on only J and K samples to capture the distribution. For the metadata and metrics gathering, we run for M rounds and and let LLM refine the final metadata and metrics from gathered and ranked results. For the clustering, we also run for sufficient rounds to make sure the cluster score robustly reflect the distribution (K=10 and K=15 gives similar results). All the results are obtained from multiple running of the proposed LLM-cluster pipeline and baseline metrics for different random seeds.

Other clustering metrics in Ablation tables

We also provide the ablation study of perplexity with different models and K-means cluster score with different number of clusters here.

From the results, perplexity present the similar trend as in the main paper regardless of the underlying distribution. And K-means is more sensitive to the hyper-parameter setting of the number of clusters.

-- Was there any human evaluation performed in order to verify the outcomes of the self-verification module, and in order to find the optimal verification prompt?

The self-verification module and earlier iteration of the verification prompt is indeed human-verified. Here we provide a more formal human evaluation on the filtered invalid clusters.

From the results, one can observe that a large proportion of the filtered cluster is indeed invalid.

-- Using the same model to verify its own reasoning chain and clustering judgements may lead to a positive bias; the verification module would benefit from utilizing an LLM that is not part of the earlier steps of cluster score generation pipeline.

Thanks for this great advice. In appendix B.2, we indeed provide the clustering results of using different LLMs for the entire baseline. Here, we provide an additional ablation study of using different LLMs for the self-verification module.

We can observe that GPT-4 present better verification and the smaller models are more incapable to discriminate the invalid clusters. The trend of the results are similar to use different models for the entire pipeline.

These results are included in our updated Appendix.

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We would be happy to provide further clarification if you have any additional questions or concerns.

-- Authors

Thank you for your detailed response. 

• The number of samples still a concern - the sample size variations in the authors' response do not seem to reflect a sufficiently high range of values to determine that the parameter values are optimal. Additionally, the perplexity test as well is not entirely convincing, given that 10/20/30 are close within the same order of magnitude.
• Thank you for providing insight into human verification, as well as including the ablation study over different LLMs for verification.
• Despite the authors' clarification, in the opinion of this reviewer, metadata and metric are still confusingly similar - while the intention to separate quantitative and qualitative measures is clear, this does not seem to be manifesting in practice. For example: how are text complexity, a metadata, and conceptual density, a metric, qualitative and quantitative respectively? How is interdisciplinary integration quantifiable?
• This reviewer's concern regarding lack of consistency in how an LLM categorizes each cluster based on the same metadata and metrics still stands. Given that this is a cornerstone of the approach, this may be a significant flaw. If this concern is addressed more convincingly, either through quantification of scores for a set of clusters over different runs, or some qualitative insights, it would go a long way towards justifying an improvement in the review score.
• Thank you for updating the diagrams, this change would likely help interpretability and comparison significantly.

In lieu of improved clarity in communication re: the diagrams and ablation studies to better support some of the insights, the presentation and soundness scores have been updated.

We first thanks the reviewer's detailed suggestions and feedback. And we address the raised concerns and question in the following.

The paper criticizes previous methods for context length limitations (lines 192-195), but the proposed method also faces similar constraints

We thank the reviewers for highlighting the context length limitation. But we want to clarify that we are not criticizing that previous methods have context length limitations and ours not. All model-based methods will be subjected to the context length of the model used. For traditional model-based methods, i.e., perplexity and K-means, the metric value is directly related to the context length of the models. Thus we have to segment the inputs to calculate these metrics.

For our LLM-cluster method, it also subjects to the context length, and with the over-lengthy inputs also cast performance degradation, as shown in Appendix B.3. However, we show that, with small values of K, we are able to capture the clusters of the underlying data distribution by iterating the clustering process over sufficient rounds of N. Moreover, as shown in the paper and our response to Reviewer qfg7, our metric demonstrates the best correlation with the performance of LLMs, showing the advantage over the baseline metrics. We hope this can make the confusion on the context length more clear.

The findings are extrapolated to larger scales, but the experiments are limited to 1.4B parameter models. Please discuss theoretical reasons or acknowledge the limitations of extrapolating to much larger models.

We acknowledge the limitation that we cannot verify the proposed method and large-scale data and model due to the computational limit. However, from the results shown in Figure 1, we indeed demonstrate that our metric correlates more strongly with the larger 1.4B model, compared to the 350M model. We have tuned down the statement in the conclusion part and hope this will help resolve the concern.

The metric assumes that LLMs' clustering perfectly captures true data diversity, which may degrade for highly complex distributions.

We agree that the reliance on LLMs' clustering capabilities assumes a level of semantic understanding that may degenerate for highly complex or noisy data distributions. The self-verification step addresses some of this concern by removing clusters that do not align with the model’s generated criteria. However, we will emphasize that the quality of clustering depends on the LLM's inherent capabilities and is not immune to limitations. Besides, the synthetic data explored in this work is highly practical, i.e., textbook style synthetic data are widely used in training LLMs, and we showed that the proposed metric is robust to the complex distributions and domain-specific knowledge.

The metric provides a single diversity score, but diversity is inherently multi-dimensional.

Diversity is indeed a multi-dimensional factor. In the design of our method, we use metric and metadata to capture the multi-dimensional diversity of synthetic data. Thus our metric captures an aggregate view rather than individual facets. However, we will clarify in the paper that the LLM Cluster-agent metric is designed to provide a high-level, interpretable summary of diversity but does not replace metrics that focus on specific facets (e.g., n-gram diversity, self-repetition score). In our future work, we will explore the combination of different metrics as the more comprehensive measurement of the diversity.

Why not use a more capable "teacher" model for self-verification? Would this improve reliability? What was the rationale for using the same model for both clustering and verification?

Thanks for this suggestion. A more capable "teacher" model will indeed improve the self-verification step. We use the same GPT-4o model mainly for the cheaper cost and consistency consideration. Here, we provide an ablation study on using different models for self-verification.

The results shows that the more capable GPT-4 indeed captures more invalid clusters and gives a slight better cluster score. The eventual selection of the models may subject to the cost consideration in practice.

We first thanks the reviewer for the comprehensive and detailed feedback on our paper. We address the weakness and questions raised as follows.

The work does not sufficiently demonstrate...

We appreciate the reviewer’s concern and recognize the need to provide stronger evidence of the superiority of the LLM Cluster score over baseline metrics. To address this:

1. We have included quantitative correlation metrics (e.g., Pearson correlation coefficients) between pre-training/downstream performance and all baseline diversity metrics, as well as the LLM Cluster score as shown below. We also included it in the revised Appendix.

Pearson coefficients (with p-value)

2. We also present diagrams similar to Figure 1 for baseline metrics, plotting their correlation with downstream performance. This allows for direct comparison with the LLM Cluster score, clarifying how traditional metrics such as perplexity or K-Means clustering align less consistently with performance trends compared to the LLM Cluster score.

3. While Cosmopedia v0.2 performs similarly to v0.1 and Topics in Figures 7a, 7c, and 7d, it is important to note that the observed performance parity is despite Cosmopedia v0.2 having a significantly lower LLM Cluster score. This aligns with our hypothesis that lower diversity can still yield comparable results under specific conditions, but the LLM Cluster score generally provides better predictive diversity across varied datasets.

Can the authors show quantitative correlation metrics or diagrams to demonstrate that the LLM Cluster score is a better indicator of performance than other baseline metrics?

Please refer the above response for the updated quantitative correlation metrics, and updated diagrams for correlation in Appendix B.2.

Is the model trained on real data matched in size to the synthetic tokens? Would it be useful to quantify the efficacy of synthetic data training?

For our current results, the model is trained on real data only, with 34B tokens in total. For synthetic trained models, the model is trained on real and synthetic data, with more tokens. It is true that training the models under exactly the same amount of tokens can further justify our results. However, due to the time and computation required for training the models, we have to left this for our future work.

Is this technique viable given the cost of using GPT-4o?

We acknowledge the valid concerns regarding the cost of using models like GPT-4o, particularly when applied to large datasets. We shown in Appendix B.2 (now B.3), that open-source models can also be used in our LLM-cluster pipeline, whereas more capable models give more robust results. In this paper, we use GPT-4o for consistency, but it is fully viable to use other models, especially the open-sourced ones in the LLM-Cluster metric.

Typos and missing citations We thank you for pointing out the grammatical issues and the missing citations. We have these issues thoroughly addressed and corrected in the revised version of the paper.

In section 2.1, it is unclear how capturing the underlying distribution of clusters and cluster sizes implies that this measure originates from the principle of entropy. What does this mean?

This is because the metric entropy itself measures the cluster sizes and number of clusters of the underlying distribution, which is the same purpose as the proposed LLM-cluster. We will make this more clear.

Seems to be an extra space on line 309 and line 422.

This is mainly caused by the warp table.

We appreciate for your detailed feedback and would like to provide further clarification regarding the remaining concerns.

analysis on why k-means.

The negative correlation of k-means trend can be attributed to several inherent limitations of the K-means clustering algorithm, particularly when applied to high-dimensional and complex text datasets like those used in our experiments. For example, the distance metric in kmeans on text embeddings is often inadequate for capturing the semantic and contextual nuances of large-scale datasets and also make them subject to the embedding models. This geometric approach can lead to clusters that fail to reflect meaningful semantic distinctions, as it prioritizes proximity over contextual relationships.

In addition to that, k-means assumes that clusters are isotropic and of roughly equal size, a limitation that often misrepresents the underlying data distribution. In practice, dense regions of data with subtle variations may be split into multiple clusters, while sparse but diverse regions may be merged into a single cluster. This imbalance results in cluster configurations that are misaligned with the true diversity of the dataset, leading to poor correlation with performance.

In contrast to KMeans, the LLM-cluster agent leverages the semantic reasoning capabilities of large language models to generate clusters based on metadata and metrics that capture multi-dimensional diversity. This approach aligns more closely with the nuanced attributes that influence model performance, resulting in a stronger positive correlation. These findings underscore the importance of incorporating contextual and semantic understanding in clustering methods for evaluating data diversity in LLM pretraining.

Regarding the costs, I think that it is important for the research community at large to see what the costs of generating new data with gpt-4o would be for this technique to work, as an overhead on normal synthetic data generation. In particular, if it is relatively cheap, some people may default to gpt-4o for ease of use. I would like to see some quantitative estimates of the cost overhead in the final paper.

We acknowledge the importance of providing quantitative estimates of the cost overhead associated with using GPT-4o for our pipeline. We use both GPT-4o for LLM cluster agent and synthetic data generation. For LLM cluster agent, each clustering call is roughly 0.005 (K=10, average input tokens=4000) for input tokens and 0.04 for output tokens (average output tokens=4000). To obtain the LLM cluster score, we run the clustering for N=5000, making the total cost of diversity measure as 225. In our main experiments, we repeat this process for 10 rounds, but in practice this can be achieved with fewer rounds.

For the synthetic data generation part, it is much more expensive since we generate 10B tokens for 4 variants in our main experiments. Our average input tokens for each generation are roughly 4000 and output tokens are roughly 1000, making each generation 0.015. To obtain 10B tokens, the total cost would be approximately 15000. But fortunately most of the synthetic data generation are using the API credits.

While this cost might not be feasible for everyone, GPT-4o-mini provides a cost-effective alternative with comparable performance as reported in other works. For the same dataset, gpt-4o-mini would cost 75 for input tokens and 30 for output tokens, totaling 105. By using the Batch API, which offers a 50% reduction in cost, the input cost further reduces to 37.50 and the output cost to 15, bringing the total to 52.50. Although we did not test GPT-4o-mini in our experiments, it could serve as a viable low-cost alternative for researchers working with tighter budgets.

Will the code for LLM-cluster agent be open-sourced? This could be a valuable tool for the research community at large.

We greatly appreciate the interest in our LLM-cluster agent and recognize its potential value to the research community. To support further advancements in this area, we are committed to open-sourcing the LLM-cluster agent, providing the OpenAI API code and full prompt templates used in our paper to enable researchers to apply the pipeline to their own datasets. Additionally, we will release all the seed topics and configuration details used in our experiments, ensuring transparency and facilitating further exploration of our method. Furthermore, if possible, we plan to share the synthetic data generated through our pipeline, making it accessible for use in pre-training or fine-tuning large language models. We believe the synthetic data of 10B scale would indeed be valuable to the community. By releasing these resources, we aim to help the community, benefiting researchers working on diverse aspects of LLM training and evaluation.

We hope the above response could resolve the remaining concerns.

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We would be happy to provide further clarification if you have any additional questions or concerns.

-- Authors

We thanks for the reviewer's detailed feedback, and address the raised weakness and questions as follows.

Lack of a rigorous human evaluation study...The error in this initial step would be propagated to later steps and would potentially make the metric unreliable.

Thanks for your advice on human verification of the proposed method. Here, we provide a human study on the synthetic data generated from using different prompt. We sample 50 data from the 6 synthetic data variants, ask human to score the diversity from 1-6, and report the average score across 10 human evaluators.

The strong correlation (r = 0.91, p = 0.011) demonstrates a statistically significant positive relationship between the Human Diversity Score and the LLM Cluster Score, supporting the validity of our metric. The results show consistency between the human diversity score and the LLM cluster score, which strongly correlates with the LLM performance.

In addition, the reliability of the proposed pipeline is guaranteed by multiple rounds of iteration of each module. For the metadata and metric generation module, we repeat the process for M times and each time with randomly sampled J data. The final metadata and metric are gathered from this iterative process. This is similar for the clustering module. In our experiments, we also report the error bar from multiple runs, and show that the results are robust across different runs.

Assessing cluster purity...clusters formed and how do we assess cluster purity?

We recognize the importance of assessing cluster purity to evaluate the quality of the clusters formed. In our pipeline, the self-verification module plays a crucial role in ensuring the reliability of the clustering process by filtering out invalid clusters that do not meet the LLM-defined criteria. Additionally, the robustness of the clusters is evaluated through multiple iterations with varying $K$-sized samples, ensuring that the clustering results are consistent and representative of the underlying data distribution.

While we do not include a detailed statistical analysis of cluster purity in the current work, we conduct human evaluations that align well with the clusters identified by the LLM. These evaluations provide support for the validity of the formed clusters. Please refer to human evaluation results above.

Lack of control on the number of clusters formed

Instead of controlling the number of clusters in the prompt, we believe that let LLM fully decide the clusters, as in our method, is a better approach. Since the optimal number of clusters also vary significantly by scenarios in practice, and selecting it by human would not be an ideal choice.

Moreover, in our pipeline, we have a self-verification module to valid the formed clusters. As shown in our response to other Reviewers, the filtered clusters from the self-verification module can be validated by human.

Parameter Sensitivity: The performance of the clustering may depend on the choice of parameters like K (number of samples for clustering), N (number of clustering iterations), J and M (for metadata/metric generation). Optimal parameter settings may need to be determined.

It is true that the performance of the clustering depends on the choice of parameters. In practice, these parameters may need to be determined by running small ablations and there would not be optimal values universally usable for different scenarios. We show in our ablation, as long as J and K are within proper range, the metadata/metrics generation and cluster results are very consistent.

How do you decide on the optimal number of clusters? Is there a cluster evaluation technique (similar to elbow curves, cluster homogeneity metrics for traditional clustering techniques) in place?

The optimal number of clusters is decided by LLM itself. As shown in our earlier response, human-decided optimal number of clusters would also be very sensitive and laborious. The self-verification module in the proposed method ensures the validation the the formed clusters.

Can we have a human eval study included to address the metric reliability issues?

Please refer to the above response for the human evaluation study.

If you have further questions, please let us know.

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We would be happy to provide further clarification if you have any additional questions or concerns.

-- Authors

Thank you for your response, I am generally satisfied the changes made. Thank you for providing the human eval study, that was really helpful. Make sure you include them in the paper as well. I'll be bumping the score up to 8.