DataGen: Unified Synthetic Dataset Generation via Large Language Models

Q: The proposed framework is complex, and in the ablation study presented in Table 4, the analysis may be too simplified to fully validate the effectiveness of each module.

A: Thank you very much for your suggestion! Due to the structure of our paper, which focuses on the procedule by order of the framework's modules, our ablation studies are not limited to Table 4. For instance:

• In Figure 3, we present the ablation results of self-reflection module.
• In Figure 8, we analyze six ablation scenarios from a cost perspective.

We appreciate your feedback and have updated the caption of Figure 3 and the title of Section 3.6 to reduce any confusion.

Additionally, to address your concern thoroughly, we conducted additional experiments based on your suggestion, including code-based mathematical validation, RAG module evaluation, diversity measurement and iteration number of self-reflection. The results are as follows:

Code-Based Mathematical Validation:

RAG Module Evaluation:

Diversity Measurement:

Iteration Number of Self-Reflection (%):

From the above results, we observe:

1. The DataGen framework exhibits strong generalizability. Regardless of whether the model is open-weight or proprietary, DataGen performs effectively across all modules.
2. DataGen significantly enhances the truthfulness of data generation for many models. For example, DataGen reduces mathematical generation errors for Llama3-70b, Claude-3, and Llama3-405b by over 50%. Additionally, the RAG module effectively mitigates factual errors for Claude-3 and Llama3-70b, with a correction rate exceeding 20%.

Thank you again for your valuable suggestions. We hope our responses address your concerns thoroughly and effectively!

Thank you so much for your valuable feedback! We truly appreciate your thoughtful suggestions and will address your concerns as follows:

Q: In Table 1, the authors list a series of current dataset generation frameworks, highlighting that DataGen considers a broader range of factors. However, for downstream applications, especially the data augmentation section, none of these methods are compared, which limits the demonstration of their effectiveness.

A: Sorry for the confusion. It is worth emphasizing that in Table 1, we conducted a broad and comprehensive comparison of current dataset generation frameworks across eight dimensions. DataGen stands out because its capabilities span all eight dimensions:

1. It not only supports data augmentation but also dynamic evaluation, introducing new knowledge to the generated datasets without requiring manual intervention.
2. We designed several modules to ensure high-quality data generation, focusing on truthfulness and diversity.

Compared to other dataset generation frameworks, the key advantage of DataGen lies in its strong generalizability and flexibility. It is not solely limited to data augmentation, as we aim to strike an optimal balance between generalizability and effectiveness. For example, while frameworks like MetaMath and Dyval can be used for data augmentation, they are primarily focused on reasoning datasets rather than all types of datasets.

In summary, we plan to explore further optimization of this balance in future work. Thank you very much for your suggestion!

Q: As shown in Figure 4(a), while the length distribution of the generated data tends toward a normal distribution, longer-length samples are missing for HellaSwag and MMLU. In Figure 5, although the generated examples align with the original datasets, it is evident that the generated dataset represents only a partial subset of the originals.

A: Thank you very much for pointing this out! To address your concern, we injected specific instructions into the dataset constraints to encourage LLMs to generate longer samples. The results of this approach are as follows:

It can be seen that by introducing constraints, LLM effectively generate longer text, which means the controlability of DataGen. This operation can be achieved by modifying the configuration file in our toolkit (e.g., datagen/examples/generation_config.yaml).

Q: Table 7 shows that, without difficulty enhancements, LLMs perform better on generated benchmarks compared to the originals, which reduces DataGen's effectiveness and practical value.

A: Thank you very much for your suggestion. By default, it is challenging to make LLMs generate more difficult data, which is why we introduced difficulty enhancements. It is important to emphasize that difficulty enhancements are an integral part of the DataGen framework, not an isolated feature. Users can configure these enhancements based on their specific needs (as README in toolkit shows). It may not be entirely fair to evaluate benchmark difficulty solely by the presence or absence of difficulty enhancements. This might have been a writing issue in our paper, and we have now clarified this point in the revised version. Thank you again for pointing this out!

Dear Reviewer,

Thank you very much for taking the time to review our paper. If you have a moment, could you kindly confirm whether our responses have addressed your concerns? Thank you so much!

Q: Have you tried any experiments with open-sourced LLMs such as LLaMA 405B being the generating LLM? Would it be as beneficial as GPT-4 or Claude?

A: We sincerely apologize for any confusion caused. Due to previous computational constraints, we were unable to conduct experiments with Llama3-405B. However, we did evaluate open-sourced models, such as Llama3-70B, on DataGen, and the benchmark results are presented in the appendix as follows:

We highly value your suggestion and subsequently acquired additional computational resources to conduct experiments with Llama3-405B. The results are as follows:

As shown, open-sourced models effectively achieve dynamic evaluation under the DataGen framework. Many models show significant performance drops on generated data, revealing potential overfitting to test datasets [4]. This demonstrates the effectiveness of DataGen in dynamic benchmarking. If the paper is accepted, we will include these results in the camera-ready version.

Thank you again for your valuable feedback, especially your insightful analysis regarding GSM8K overfitting. We sincerely hope our responses address your concerns!

[1] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1658–1677, Bangkok, Thailand. Association for Computational Linguistics.

[2] Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. In Findings of the Association for Computational Linguistics: ACL 2024, pages 963–981, Bangkok, Thailand. Association for Computational Linguistics.

[3] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, Singapore. Association for Computational Linguistics.

[4] Mirzadeh, Iman, et al. "Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models." arXiv preprint arXiv:2410.05229 (2024).

[5] Zhu, Kaijie, et al. "Dyval: Dynamic evaluation of large language models for reasoning tasks." The Twelfth International Conference on Learning Representations. 2023.

[6] Zhang, Hugh, et al. "A careful examination of large language model performance on grade school arithmetic." arXiv preprint arXiv:2405.00332 (2024).

Thank you very much for your valuable feedback! We will address your concerns one by one:

Q: RAG-based validation is very high in cost (raising cost from 0.038 to 0.19, almost a 5x increase). However, it is unclear how it affects the final data generation quality (like the results in Table 7). In other words, it would be better to ablate the modules in terms of metrics in Table 7, instead of the current reports in Table.

A: This is an good suggestion! Regarding your concern about the high cost of RAG-based validation, there are indeed several alternative solutions:

1. Using a cheaper model: We can substitute the current GPT-4o with a more cost-effective model like GPT-4o-mini, which scores 82% on MMLU and currently outperforms GPT-4 on chat preferences in the LMSYS leaderboard. This substitution could reduce RAG costs by 16x, making the cost decrease from 0.19 to 0.048, which is entirely acceptable. We have also included options for different models in the RAG module of our toolkit.
2. Prompt compression techniques: There are many established and effective techniques for prompt/document compression [1][2][3], which we plan to integrate into our toolkit to further reduce RAG costs.

Regarding your concern about the unclear impact of RAG on the results in Table 7, we want to emphasize that Table 7 presents benchmark results for each model, where RAG's effects may not be directly reflected. Nevertheless, we conducted ablation studies on the RAG module on four models and found that adding RAG reduced factual errors significantly, which is critical for improving dataset quality. Here are the results:

Q: I am not convinced that the performance decline on GSM8K in your experiments can be concluded to the claim that many LLMs may be overstated and overfit on the GSM8K dataset. Would you please elaborate more on this?

A: Sorry for the confusion! We believe that LLMs exhibit overfitting on GSM8K, which, in other words, implies a risk of data leakage. This has been highlighted in many recent studies [4], and similar research has reached conclusions consistent with ours [5, 6]. We will add more citations to this finding in PDF.

Thank you for your thoughtful feedback and valuable suggestions! We hope the following responses can address your concern:

Q: Data formatting.

A: We sincerely apologize for the misunderstanding. The "data format error" mentioned does not only refer to non-compliance with JSON but also to issues such as improper output formatting or missing information, such as generating only the question without the answer. Regarding the resolution of JSON formatting issues, we mentioned in line 407 of our paper that we employed "an integrated framework like LangChain." Thank you for your suggestion! We are in the process of updating our toolkit, and once the paper is accepted, we will release a new version of the toolkit. This update will support three formatting frameworks: Guidance, LangChain, and OpenAI's native JSON interface.

Q: Effectiveness of Modules in DataGen.

A: The Remote-Clique Score (RCS) is a robust metric because it directly measures semantic cohesion among texts using embeddings, focusing on closely related pairs that exceed a similarity threshold. Here's how it is calculated:

1. Get Embeddings: Each text is converted into a vector $E_i$ using a model (in our case, OpenAI's text-embedding-ada-002).

2. Cosine Similarity:

   $S(i, j) = \frac{E_i \cdot E_j}{\|E_i\| \|E_j\|}$

   Similarity between text pairs is computed.

3. Form Cliques:

   • A threshold $\theta$ is defined.
   • Texts are considered part of the same clique if $S(i, j) > \theta$.

4. Compute RCS:

   $\text{RCS} = \frac{\sum_{i < j, S(i, j) > \theta} S(i, j)}{N}$

   Here, $N$ is the number of text pairs above the threshold.

We referenced prior work [1], which also used this metric to evaluate text diversity. Additionally, we incorporated other diversity metrics to ensure comprehensive evaluation, such as APS and INGF. The results, presented in Appendix Table 14, are as follows:

Regarding why HellaSwag shows a larger difference in the remote-clique score compared to other datasets: the difference here does not indicate improvement but rather the diversity difference between the original and generated datasets. The 8% difference shows that HellaSwag has the largest diversity gap, though this is still objectively less than 10%, which is acceptable. This larger gap might be attributed to the dataset's longer average text length. During generation, the model may have sacrificed some diversity to maintain length distribution.

Q: Benchmarking LLM.

A: We apologize for any confusion caused. In our experiments, we used the same prompt for all generators to ensure a fair comparison (this clarification has been revised in the PDF to reduce misunderstandings). One reason Claude-generated questions might be perceived as harder is that many models are fine-tuned or pre-trained on synthetic data generated by GPT-4, making them more familiar with GPT-style content, which could create difficulty discrepancies. If user would like to experiment with different prompts, they can modify the configurations in our toolkit at datagen/utils/prompt.py.

Thank you once again for your valuable feedback. We hope our responses address your concerns effectively!

[1] Li, Zhuoyan, et al. "Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations." The 2023 Conference on Empirical Methods in Natural Language Processing.

Thank you very much for your suggestions! We will address your concerns one by one:

Q: There could be more side-by-side of questions from the original dataset and each generated dataset.

A: We are sorry for the confusion. We did not include side-by-side generation in our experiments mainly for the following reasons:

1. Token Consumption. The efficiency of generating side-by-sides is relatively low, increasing our framework's token consumption during the generation process.
2. The Same Goal as the Current Pipeline. The generation of side-by-side shares the same goal with our current generation process. Both approaches focus on generating questions that deviate from the original dataset's goals and task objectives while introducing new knowledge.
3. Still Supported By DataGen. Despite these considerations, our framework still supports side-by-side. By modifying the configuration file in our toolkit (e.g., datagen/examples/generation_config.yaml), you can specify the requirement for side-by-sides in the constraint section (generation_hint/dataset_constraint). The LLM can then generate side-by-side data as per the requirements. We have demonstrated the LLM's performance on instruction-following under given constraints (for single and multiple constraints) in the appendix:

From these results, it is evident that under the DataGen framework, LLMs perform exceptionally well in instruction-following, which shows potential capability on side-by-side generation.

Q: What is the performance of each module given different generator models?

A: Thanks for your suggestion! Based on your feedback, we supplemented relevant experiments. Specifically, we evaluated code-based mathematical validation, diversity measurement, RAG module, and the number of iterations in self-reflection. The results are as follows:

Code-Based Mathematical Evaluation:

Diversity Measurement:

RAG Module:

Iteration Number of Self-Reflection (%):

From the above results, we can observe the following:

1. The DataGen framework exhibits strong generalizability. Regardless of whether the model is open-weight or proprietary, DataGen demonstrates excellent performance across various modules.
2. DataGen effectively enhances the truthfulness of data generation for many models. For example, DataGen significantly reduces errors in mathematical problem generation for Llama3-70b, Claude-3, and Llama3-405b (improvements exceeding 50%). Additionally, DataGen's RAG module effectively mitigates factual errors for Claude-3 and Llama3-70b (correction rates exceeding 20%).

We deeply appreciate your thoughtful feedback and valuable suggestions, which have provided insights to refine our work further. Thanks a lot!

Thank you for your thoughtful feedback. If there are specific aspects of your concern that remain unresolved, we would greatly appreciate it if you could elaborate further so we can address them more effectively.