Synthetic Text Generation for Training Large Language Models via Gradient Matching

We thank the reviewer for acknowledging the novelty of our work and our thorough experiments.

---

1. Unverified assumption $\xi \leq |g_t|$ in theory.

The following figures confirm the validity of our theoretical assumption, by showing that the gradient error, i.e. $\| \nabla \mathcal{L}(\theta_t) - \nabla \mathcal{L}^s(\theta_t) \| \leq \| \nabla \mathcal{L}(\theta_t) \| = \| g_t \|$ at the pretrained parameters and this relation indeed holds during fine-tuning. Crucially, the data generated by GRADMM has a much smaller gradient error compared to the zero-shot baseline during fine-tuning, which is the reason for its superior performance.

We thank the reviewer for acknowledging the novelty of our method, our theoretically-rigorous framework, our well-supported claims and extensive experiments.

---

Experimental validation:

• Problem formulation: gradient matching in Eq 3 applies to both the data scarce regime (Fig 1) and Dataset Distillation (DD) (Table 1). In both, the synthetic data is generated by matching the average gradient of the few available examples (Fig 1) or a larger training data (Table 1).

• Evaluation: while accuracy is an accurate metric to measure the performance in both cases, we include the following table for Fig 1a (SST2) which shows the divergence (FID) between the (i) training data distribution, (ii) distribution of the few available real examples, (iii) distribution of the 100 GRADMM synthetic data, (iv) distribution of 100 zero-shot synthetic data. Our synthetic data has a smaller FID, confirming its more similar distribution to that of real training data, compared to the baselines. This corroborates the superior performance of GRADMM.

• FID is not generally used to evaluate DD (Table 1), which aims at generating a small subset of synthetic data with similar dynamics to a large training data. This is because: (i) FID requires both distributions to have a large sample size such that they resemble gaussian distributions, and otherwise is less accurate. (ii) distribution of the small generated data is not comparable to the large real training data (in particular, their variances are not comparable due to their very different sizes). Hence, FID is not commonly used for DD (c.f. the DD related work in Sec 2.1), but is more common when generating large data with diffusion models.

Theoretical bounds:

• The following figures confirm the validity of our theoretical assumption, by showing that the gradient error, i.e. $\| \nabla \mathcal{L}(\theta_t) - \nabla \mathcal{L}^s(\theta_t) \| \leq \| \nabla \mathcal{L}(\theta_t) \|$ at the pretrained parameters, and this relation holds during fine-tuning. Crucially, the GRADMM generated data has a much smaller gradient error compared to the zero-shot baseline during fine-tuning, corroborating its superior performance.
• We generated 200 synthetic data in Figure 1a. The accuracy of the model fine-tuned on this larger subset is 90.1 $\pm$ 0.1, which is higher than 89.8 $\pm$ 0.4 of training on the original 100 synthetic data, and is closer to the last column in Table 1.
• We compared the $L_2$ normed difference in model parameters when trained on real training data vs 100 synthetic data generated for SST2. The normed difference for GRADMM is 1.99, which is smaller than the 2.27 for zero-shot. This further confirms the validity of our theory.

Privacy:

• Thanks! We will add discussion of the importance of privacy, data leakage and benefits of synthetic data to our revised version.
• GRADMM is the first dataset distillation method able to generate human readable text. DD methods provide differential privacy guarantees [1]. Specifically, for two datasets that differ in only 1 example, the parameters of the models trained on the distilled version of the two datasets are provably highly similar [1]. Intuitively, as dataset distillation methods generate data by techniques such as matching the mean of the data distribution or average gradient of a real training data, as long as synthetic data is not initialized by real training examples, there is no information leakage about individual training examples [1]. GRADMM does not initialize synthetic data with real training examples, and hence effectively preserves the privacy of the training data.
• We conducted the loss-based MIA to the model trained on GRADMM synthetic data for SST2. Specifically, we select N=100 member samples from the training subset and N non-member samples. Then, we find the optimal threshold that maximizes the advantages (2 x (acc - 50%)) on these 2N samples. Finally, we test the loss-based MIA with optimal threshold on another 2N samples consisting of N members and N non-members and report the advantage score. We repeated the whole process 10 times and the advantage score (%) is only 1.75 $\pm$ 1.4, demonstrating the effectiveness of GRADMM against loss-based MIA.
• The following figures shows the histogram of the distances of synthetic examples to their closest real training data. None of the synthetic examples generated by GRADMM are very similar to the real training examples, further confirming that our synthetic data is not identical to real examples.

We will add the above discussion and results for all the datasets to our revised version. We hope that our rebuttal has addressed the reviewer's concerns and they can consider further supporting our work to be accepted.

We thank the reviewer for supporting our work and acknowledging our convincing experiments.

---

1. Can this scheme used to generate math, logic, and code?

The idea of our work (generating synthetic text via gradient matching using ADMM) can be applied to math, logic or code. However, this requires incorporating additional structure in the generated text. Controllable text generation which requires the text to follow a particular structure, syntactic or semantic properties has been a topic for several recent research, including [Li et al’22, Gong et al’22, Zhou et al’24 (references are in Line 133 from the paper)] to generate text in the form of tables, code, etc. Such techniques can be incorporated in our method to generate structured synthetic text. Considering that our method is the first to show the applicability of gradient matching to generate readable synthetic text, it lays the ground for many exciting future work, including controllable text generation via gradient matching.

We thank the reviewer for their feedback. However, we disagree with their evaluation of our work, as we discuss below.

---

Data scarce regime: Fig 1 provides strong evidence for the applicability of our method to the data scarce regimes. The 100 synthetic data generated by GRADMM based on as few as 5 to 20 randomly selected real examples already reach 98%, 89%, 96% the performance of training on the full training data (last col in Tab 1), and outperform training on the 5-20 real examples with a big margin.

The following table compares (for SST2, Fig 1a) the divergence (in terms of FID) between the (i) training data distribution, (ii) the distribution of the few available real examples, (iii) the distribution of the 100 synthetic data generated by GRADMM and (iv) the distribution of 100 synthetic data generated using the zero-shot approach.

Our synthetic data has a smaller FID, confirming that it has a more similar distribution to that of real training data, compared to baselines. This corroborates the superior performance of GRADMM. While the effectiveness of GRADMM depends on the diversity of the available real examples, our empirical results show that a small number of randomly selected examples can be leveraged to effectively reduce the expected loss. We do not claim that GRADMM perfectly minimizes the expected loss with limited information (a few available real examples), but it is considerably more effective than other baselines in the data scarce regime.

Privacy concerns: Using real examples as a guide to generate synthetic data does not compromise the privacy of real training examples. As discussed in Sec 2, GRADMM is the first dataset distillation (DD) method able to generate human readable text. DD methods provide differential privacy guarantees [1]. Specifically, for two datasets that differ in only 1 example, the parameters of the models trained on the distilled version of the two datasets are provably highly similar [1]. Intuitively, as dataset distillation methods generate data by techniques such as matching the mean of the data distribution or average gradient of a real training data, as long as synthetic data is not initialized by real training examples, there is no information leakage about individual training examples [1]. GRADMM does not initialize synthetic data with real training examples, and thus effectively preserves the privacy of the training data.

To confirm, we conducted the loss-based Membership Inference Attack (MIA) to the model trained on synthetic SST2 data generated by GRADMM. Specifically, we select N=100 member samples from the training subset and N non-member samples. Then, we find the optimal threshold that maximizes the advantages (2 x (acc - 50%)) on these 2N samples. Finally, we test the loss-based MIA with optimal threshold on another 2N samples consisting of N members and N non-members and report the advantage score. We repeated the whole process 10 times and the advantage score (%) is only 1.75 $\pm$ 1.4, demonstrating the effectiveness of GRADMM against loss-based MIA.

Finally, the following figures show the histogram of the distances of synthetic examples to their closest real training example. Indeed, none of the GRADMM synthetic examples are very similar to the real training examples, further confirming that our synthetic data is not very similar to real examples.

Theoretical assumption: Our gradient matching approach ensures a high similarity to the target gradient. The following figures confirm the validity of our theoretical assumption, by showing that the gradient error, i.e. $\| \nabla \mathcal{L}(\theta_t) - \nabla \mathcal{L}^s(\theta_t) \| \leq \| \nabla \mathcal{L}(\theta_t) \|$ at the pretrained parameters and this relation holds during fine-tuning. Crucially, GRADMM synthetic data has a much smaller gradient error compared to the zero-shot baseline during fine-tuning, which is the reason for its superior performance.

Empirical evidence: The following figures show std for our experimental results (based on three runs) and confirms the strong performance of GRADMM on the 3 dataset in our paper in addition to two new datasets, namely IMDB [Maas et al., 2011] and Sentence polarity [Pange et al., 2005].

We hope that our rebuttal could address the reviewers’ doubts. Our work is the first to show the applicability of dataset distillation to generate readable synthetic text via gradient matching with strong empirical performance, and provides a promising avenue for future research. Thus, we hope that the reviewer considers supporting our work to be accepted.

[1] Privacy for free: How does dataset condensation help privacy? ICML’22.