DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models

Thank you for your constructive comments and the related work you provided. We address your concerns as follows:

Fine-tuning Baselines:

We use GReaT as the baseline because it is the current SoTA tabular synthesis method based on PLMs (GPT-2). However, fine-tuning larger LLMs (i.e., LM with more than 7B parameters) for structured data synthesis is not trivial. We exploited GReaT with stronger LLMs by replacing GPT-2 with Mistral, but it did not effectively generate data with the desired structure. The synthetic sample is shown in follow. We hypothesize that controlling an LM by fine-tuning it with supervised data cause catastrophic forgetting for LLMs, as suggested by Luo et al[1]. Specifically, the "key is value" data constructed by the GReaT method caused internal knowledge collapse when continuing to pre-train Mistral. In summary, this not only impaired the model's inherent knowledge but also failed to learn data synthesis effectively.

Synthetic data generated by GReaT with Mistral:

<s> income is <=50K, $300/month  income is >50K, $600/month  I am trying to figure out how to implement the above logic with the following table I have:  | ID | Amt | |----|-----| | 1  | 450 | | 2  | 650 | | 3  | 500 | | 4  | 150 | | 5  | 700 | | 6  | 400 |  I have tried using `IF ELSE` function in Excel but I am not sure how to apply it to this particular situation.......</s>

Regarding the training parameters, we reproduced the tabular results using the code released by the original paper, ensuring that all hyperparameters and settings were consistent with the original implementation. All results were almost identical to those reported in the TabSyn paper; therefore, we used the results reported in TabSyn in Table 1 to ensure a fair comparison.

Justification for the Choice of Diffusion Models:

In our early experiments, we used only a VAE to learn the latent space. During data synthesis, however, only about 10% of the samples had structures consistent with the training data (e.g., in tabular data, the synthetic data contained all required columns). This indicates that the standard VAE representations were often ignored by the LLM decoder, leading to poor structural consistency in the generated data. Latent diffusion addressed this sampling failure, increasing the success rate of synthetic data to approximately 97%.

As for more expressive prior distributions like a mixture of Gaussians, we referred to previous tabular synthesis works[2] in designing our method and chose a more direct diffusion approach to address the discrepancy in the latent space. We believe that a trainable denoising network can help learning a stronger latent space, and this technique is also commonly used in the current computer vision field.

Explanation of DCR, Figure 5:

DCR (Distance to Closest Record) is used to measure the closest distance between the target data and the training data. In Figure 5, we present the DCR histograms of the test data and the synthetic data. Intuitively, a better DCR distribution should be closer to that of the test data. From the figure, we can see that the synthetic data generated by DiffLM did not copy the training data (all DCRs are greater than 0), while maintaining a DCR distribution similar to that of the test data compared to the finetuning based method, i.e., GReaT.

Ablation experiments:

Thank you for pointing this out. We are currently working on ablations for the denoising network. Regarding the fine-tuning baselines, they performed poorly with LLMs like Mistral, as mentioned above. We will add the necessary descriptions and analyses to the paper to clarify these points.

Related Works:

Thank you for the references you provided. We will add them to the paper and discuss how they relate to our work.

Reference:

[1] Luo et al. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. arXiv 2023.

[2] Zhang et al. Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space. ICLR 2024.

Thanks for your constructive comments and recognition of our work. Please see below for our responses to your concerns.

R1: Missing related work.

A1: We have conducted a more in-depth survey on LLMs with auto-encoders and plan to include more relevant references on language modeling task[1,2,3,4], unconditional text generation[5,6] and control text generation[7,8]. We would appreciate it if you could specify the important related works that we missed.

R2: Limited contribution.

A2: We respectfully disagree with the assessment that our contribution is limited. Our contributions and insights are as follows:

A2.1: Motivation and Challenge: We pointed out the principles and challenges for high-quality structured text data synthesis, i.e., producing scalable and diverse synthetic data with reasonable relevant knowledge while maintaining the same data requirements (structure, topic, domain, etc.) as the target data. Existing prompt-based and fine-tuning methods cannot achieve both objectives at a low cost.

A2.2: Framework: To address these goals, we propose the DiffLM framework, which decouples the task of learning the requirements of the data to be synthesized from the language modeling task. We model these requirements in the latent space and inject them into an unaltered LLM, enabling it to generate desired and realistic data. The synthetic data combines LLM's broad knowledge with specific data patterns learned from the training data, leading to enhanced performance on downstream tasks.

A2.3: Techniques: For implementation, we propose a training recipe that keeps the LLM decoder fixed and only trains the VAE encoder and projector. We also validate different latent knowledge injection methods.

Responses to Questions:

Q1: [Role of VAE encoder] Do we need to train different VAE encoder? If yes, can we still call it as plug-and-play module?

A3: Yes, for different data types, we train different VAE encoders. It is still plug-and-play because by swapping different VAE encoders and injectors, we can generate text data that meets the corresponding requirements without altering the generative model (LLM).

Q2: How to guarantee the generated samples are consistent?

A4: We designed simple post-processing methods to validate the structure of synthesized data (e.g., required columns for tables), and we found that only ~3% of the synthesized data has the wrong format, which demonstrates the consistency of the generated samples.

Q3: Major contribution

A5: As addressed in A2, our major contribution lies in proposing the DiffLM framework, which combines PLMs with diffusion techniques to synthesize data that merges the LLM's broad knowledge with specific data requirements learned from the training data.

Q4: Is it possible to use this method to do natural text generation?

A6: Yes. You can replace the training data with free text from a specific domain or topic, train DiffLM, and achieve generation of natural language in that specific domain or topic. You may refer to [9] for detailed data construction and training processes. Compared to [9], we have added latent diffusion techniques to address the discrepancy between the encoder's posterior and prior. Since our work focuses on synthesizing structural data, we did not include experiments in these scenarios due to space limitations and the main focus of the paper.

Reference:

[1] Lin et al. Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise. ICML 2023.

[2] Yuan et al. Text Diffusion Model with Encoder-Decoder Transformers for Sequence-to-Sequence Generation. NAACL 2024.

[3] Reid et al. DiffusER: Diffusion via Edit-based Reconstruction. ICLR 2023.

[4] Zhang et al. DiffuSum: Generation Enhanced Extractive Summarization with Diffusion. ACL (Findings) 2023.

[5] Yu et al. Latent Diffusion Energy-Based Model for Interpretable Text Modeling. ICML 2022.

[6] Chen et al. A Cheaper and Better Diffusion Language Model with Soft-Masked Noise. EMNLP 2023.

[7] He et al. DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models. ACL 2023.

[8] Austin et al. Structured Denoising Diffusion Models in Discrete State-Spaces. NeurIPS 2021.

[9] Li et al. Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space. EMNLP 2020.

Thanks for your constructive comments. Please see below for our responses.

R1: Finetuning Baselines - GReaT.

A1: Yes, we attempted to validate the GReaT method on Mistral but found it could not directly and effectively generate data with the desired structure. GReaT organizes tabular data in a "key is value" format and uses a smaller PLM (i.e., GPT-2) for continued pretraining. However, when applied to larger models like Mistral, GReaT struggled to effectively generate the desired structured data. The sample generated by GReaT with Mistral is shown in follow. We hypothesize that controlling an LM by fine-tuning it with supervised data cause catastrophic forgetting for LLMs, as suggested by Luo et al[1]. Specifically, the "key is value" data constructed by the GReaT method, when used to continue pre-training Mistral, causes internal knowledge collapse - both undermining the model's existing knowledge and failing to do effective data synthesis. Additionally, training the adult dataset on GReaT for 200 epochs (default settings) requires approximately 50 hours on 8 A100 80G GPUs, which is resource-intensive. In contrast, DiffLM under the same training settings requires only about 7 hours.

Synthetic data generated by GReaT with Mistral:

<s> income is <=50K, $300/month  income is >50K, $600/month  I am trying to figure out how to implement the above logic with the following table I have:  | ID | Amt | |----|-----| | 1  | 450 | | 2  | 650 | | 3  | 500 | | 4  | 150 | | 5  | 700 | | 6  | 400 |  I have tried using `IF ELSE` function in Excel but I am not sure how to apply it to this particular situation.......</s>

R2: DiffLM does not consistently outperform TabSyn.

A2: We want to emphasize that our goal is a unified structured data synthesis framework that supports various domains like tabular data, codes, and tools, and tabular data generation in our work is just a subdomain of synthetic data generation. As a comparison, TabSyn is not applicable to more complex data synthesis tasks, such as code generation and tool usage, which involve generating longer content, more complex data types, and highly structured data, while our DiffLM can handle complicated scenarios. The results of tabular data synthesis are to demonstrate that our method possesses generality and can achieve on-par results with domain-specific models without being specifically tailored to a particular domain.

R3: How the quantity of real & synthetic data affects downstream task performance?

A3: We experimented with increasing the amount of data synthesized by DiffLM and combining real data with DiffLM-synthesized data. As shown in Table A, adding more synthesized data further improves around 0.2% MLE performance in the tabular scenario. Since our method can synthesize unlimited amounts of data and we did not design any complex post-processing method, the performance improvement brought by DiffLM-synthesized data in downstream tasks still has significant room for growth. Additionally, combining real and synthetic data generated by DiffLM can improve downstream performance; all results exceed >0.2% of those using only DiffLM data. Notably, on the Beijing and Shoppers datasets, the combination of real data and DiffLM synthetic data surpasses 0.6%-3% of the performance of training on real data alone.

Table A: Tabular MLE performance with varying quantity of real and synthetic data. Performance on the Beijing dataset is evaluated using the RMSE metric, where lower values indicate better performance.

Reference:

[1] Luo et al. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. arXiv 2023.

We greatly appreciate the time and effort you have invested in our paper. Hope the following responses will help improve the paper's score.

1. Finetuning LLMs like Mistral may lead to knowledge collapse, but GPT-2 would not.

As noted in Luo et al.[1], "Our findings revealed that the CF (catastrophic forgetting) problem is generally prevalent in the continual fine-tuning of various LLMs. Moreover, as the model scale increases, LLMs exhibit a more severe degree of forgetting in domain knowledge, reasoning abilities, and reading comprehension skills." In our experiments, we similarly observed that larger LLMs (e.g., Mistral) often struggle with effective synthesis due to CF during the GReaT fine-tuning process. We agree with your point that better fine-tuning strategies could mitigate this issue. However, we believe our proposed DiffLM introduces a novel angle by leveraging diffusion techniques to effectively address challenges like knowledge collapse, providing a complementary alternative to existing methods such as GReaT.

2. MLE performance with GPT-4 prompting methods.

We understand your concerns regarding the use of few-shot prompting with GPT-4 for data synthesis. In Table B, we provided results where tabular data was synthesized using GPT-4 with 5-shot prompting approach. As the table shows, prompt-based methods consistently underperform compared to our DiffLM approach across all datasets. Specifically, in the Shoppers dataset, the prompt-based method lags behind DiffLM by 7.7%. While prompting methods may achieve adequate results in specific cases, they lack guarantees for generating diverse, large-scale synthetic datasets. We plan to include additional details about prompt designs and extend results across the other two datasets in the Appendix of the revised version.

Table B: Tabular MLE performance with different synthetic data generation methods. Rho stands for column-wise distribution density estimation task, where lower values indicate better performance.

3. Controllability.

We would like to clarify that our primary goal is to propose a data synthesis strategy, rather than deliver a unified model. Specifically, our objective is to develop an effective framework for generating structured text data suitable for training downstream models. In this context, the produced data holds greater value than the model itself, so making a unified model is not a must-have in our task.

Furthermore, our method does more than "just changes the training data". DiffLM innovatively integrates external data requirements into the generation process via a latent space without modifying the LLM’s parameters, thereby preserving LLM's general knowledge capabilities. Therefore, our method is not a trivial framework that only sets random values, but it leverages the LLM's rich knowledge to synthesize data that, in some downstream tasks, surpasses the performance of real data, which is a notable advancement over prior work.

We acknowledge that our current approach does not support unified generation across diverse data structures, as it requires training both a denoising network and an injector network on domain-specific datasets. A potential solution is to introduce task-specific tokens, enabling unified task learning via a shared framework (e.g., training with task-specific soft prompts for different data synthesis tasks in a unified training process). Due to time constraints during this rebuttal phase, we will explore this direction in future work.

We hope this addresses your concerns and clarifies the contributions of our work. Thank you again for your valuable feedback.

Reference:

[1] Luo et al. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. arXiv 2023.

Dear reviewer P77K, considering the Rebuttal deadline is approaching, we sincerely hope to receive your response. If you have any questions or concerns regarding our explanation, please do not hesitate to contact us. Your response is of great significance to us for improving this work, and we look forward to hearing from you.

Thank you for your valuable comments. We appreciate Reviewer hoMx's recognition of our work on DiffLM for high-quality data synthesis. We address your concerns below:

R1: [Controllability Clarification] Can the authors clarify what exactly the controllability refers to?

A1: We define the controllability of text data synthesis as the ability to generate text that satisfies desired requirements (e.g., structure, topics, domains)[1,2]. Existing methods for structured textual data synthesis often struggle with controllability. On one hand, LLM prompt-based methods relying on prompt engineering or few-shot inference cannot guarantee the diversity and scalability of synthetic data, even with complex human-crafted processes[3]. On the other hand, controlling a LM by fine-tuning it with supervised data (SFT, RLHF) is not only expensive but might also degrade the LLM's general capability[1,4]. Our method addresses these challenges through sampling in the latent space while maintaining data structure due to LLM's instruction-following ability. Thanks for pointing this out, we will update related parts in the next manuscript.

R2: [Analysis on Synthetic Data Outperforming Real Data] It would be more helpful to provide more analysis and empirical insights on why and how this (the synthetic data can outperform real data for continued pre-training) is made possible.

A2: Our motivation arises from observing that many works attempt to use LLMs for data synthesis but often face difficulties in efficiently generating desired and realistic high-quality data[3] at scale. We propose DiffLM, a framework designed to steer LLMs in data generation by decoupling the task of capturing the characteristics of the target dataset from the core language modeling process. We model the nature of the target dataset in the latent space and then inject it into the unaltered and carefully aligned LLM, enabling it to generate desired and realistic data. As shown in Figure 2 of our paper, DiffLM synthesizes data that leverages the learned target data latent representation and the LLM’s internal knowledge, resulting in better judgment scores than the real data. We believe this is because the synthetic data combines LLM's broad knowledge with specific data patterns learned from the vast quantity of training data, leading to enhanced performance on downstream tasks.

R3: Interpolation of Latent Space and Role of Diffusion Model

A3.1: In Figure 6, we provide t-SNE visualizations of the latent space learned by DiffLM across multiple datasets. The latent space exhibits clear clustering and hierarchical structures. Specifically, for the Beijing dataset, samples with larger numerical values are distributed in the upper part of the 2D t-SNE space, while those with smaller values are in the lower part, forming a gradient from top to bottom.

A3.2: Regarding the role of the latent diffusion model, in our early experiments using only a VAE to learn the latent space, only about 10% of the synthetic samples had structures consistent with the training data (e.g., in tabular data, the synthetic data included all the required columns). The latent diffusion model addressed this sampling failure problem, increasing the pass rate to about 97%. As mentioned in our reply to Reviewer rfz8, Question 3, we will include the relevant information in the next version.

R4: Human Evaluation

A4: We agree that automated metrics like DCR and downstream task performance may not fully capture the nuances of data quality for complex structured data. In fact, we have used GPT-4 to rate and perform preference judgments on synthesized tools and real tools. The results in Figure 2 and Table 3 demonstrate the quality of our synthesized data. As per your suggestion, we have conducted human evaluations on the tool usage data. Specifically, we compared 100 pairs of randomly selected DiffLM-generated data and real data within the same category. As shown in Table A, our synthetic data is preferred by human annotators.

Table A: Human Evaluation

Reference:

[1] Keskar et al. CTRL: A Conditional Transformer Language Model for Controllable Generation. arXiv 2019.

[2] Li et al. Diffusion-LM Improves Controllable Text Generation. NeurIPS 2022.

[3] Long et al. On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. ACL 2024.

[4] Borisov et al. Language Models are Realistic Tabular Data Generators. ICLR 2023.

Dear reviewer hoMx, considering the Rebuttal deadline is approaching, we sincerely hope to receive your response. If you have any questions or concerns regarding our explanation, please do not hesitate to contact us. Your response is of great significance to us for improving this work, and we look forward to hearing from you.