Synthesizing Privacy-Preserving Text Data via Finetuning *without* Finetuning Billion-Scale LLMs

We appreciate your encouraging remarks that our method is reasonable, our experimental design is both proper and comprehensive, and that our analyses are well-supported with most claims being appropriately justified.

Privacy Budget Allocation

Our budget allocation follows [1], our "Post-Generation Resample" baseline, which also has both DP-histogram and DP-finetune steps. For DP-histogram, we adopt the same -- adding a Gaussian noise 10 on every histogram item. The overall privacy budget (ε) is computed as a composition of both steps using the standard dp_accouting package.

We also show individual ε values below on PubMed, under composed $ε=4,2,1$. DP-histogram only takes a small portion of the overall budget. This aligns with observations in [1] (page 5) and highlights our CTCL-topic design achieves strong performance while consuming only a small portion of the privacy budget.

We also find that the impact of changing the allocation is insignificant. For example, in $ε=4$ setting on PubMed, increasing histogram's noise from 10 to 20 only alters the DP-finetuning budget $ε$ marginally—from 3.96 to 3.99.

[1] Privacy-Preserving Instructions for Aligning Large Language Models, ICML 2024

Generator design choices

We use a seq2seq generator because its encoder is well-suited for understanding the conditions. For size, the 140M model is to balance the efficiency with generation ability, making it practical for resource-constrained scenarios.

We agree other model selections might perform better, but we do not extensively explore all designs because our intuitive choice already demonstrates strong downstream performanceas in our experiments. Moreover, same as all pretraining research, due to the cost and scale, it could be super challenging to exhaustively search over all possible configurations.

We would also like to emphasize that the core contribution of our work is the overall CTCL framework. Each module within the framework remains flexible and open to further design improvements.

140M model can learn 430M data well? The large dataset is collected based on the common intuition that larger pretraining datasets typically lead to stronger models. The goal of pretraining is not for the open-ended pretraining task, but for a model with strong controllability supporting downstream tasks. Moreover, large data even on comparatively small models is a common practice. For instance, LLaMA-3 with 3B or 8B parameters are trained on 15T tokens, where the data samples also significantly exceeds the number of model parameters.

Topic model design choices

We choose Wikipedia for our topic model because it is large-scale, high-quality, and semantically diverse. In practice, it is rare to encounter texts that are entirely outside of its scope. The table below with sampled topics of our five diverse datasets illustrates that a broad range of universal topics are captured:

The "out-of-domain" text is also considered and processed in our experiments--in our topic model, there is an “unclassified” bin for samples that are not close to any topic. The table below shows the details

We can see the number of unclassified samples is neglectable across all datasets, and those unclassified samples typically contain little to no substantive content, confirming the broad coverage of our topic model.

Determine the number of clusters

First, the number should not be too small, so as to keep the topic conditions meaningful. Second, it should not be too large, as that would amplify per-item DP-histogram noise in downstream tasks. To strike a balance, we set a threshold that each cluster must contain at least 100 out of 6M Wikipedia articles, resulting in 1,300 clusters.

Quality guarantee of the proposed method

Our framework introduces a universal topic model and controllability pretraining, which enables the model to capture valuable high-level, topic-wise information in the private domain—beyond the word-level representations learned by the vanilla DP-finetuning baseline. As a result, the data quality in our method should be guaranteed no worse than that of vanilla DP-finetuning. This quality improvement is empirically supported by the performance gains demonstrated in our experiments and further validated by our ablation study in Section 4.3.2.

Thank you for your encouraging remarks that our claims are well-supported by clear experiments and detailed analyses, and our proposed method is reasonable.

Generator design choices

We use a seq2seq generator because its encoder is well-suited for understanding the conditions. For size, the 140M model is to balance the efficiency with generation ability, making it practical for resource-constrained scenarios.

We agree other design choices might perform better, but we do not extensively explore all designs because our intuitive choice already demonstrates strong downstream performanceas in our experiments. Moreover, same as all pretraining research, due to the cost and scale, it could be super challenging to exhaustively search over all possible configurations.

We would also like to emphasize that the core contribution of our work is the overall CTCL framework. Each module within the framework remains flexible and open to further design improvements.

Qualitative Analysis

Thank you for the suggestion! We will include more qualitative examples in the next version of the paper in addition to those in Table 6. As noted in Section 4.3.3, we emphasize qualitative results because our primary usage of synthetic data is to support downstream model development. Therefore, the surface form of the synthetic text is less critical. For instance, Table 6 shows that PE-based generation using ChatGPT consistently produces fluent outputs. However, its downstream model performance is only comparable to (and sometimes worse than) that trained on less fluent synthetic data generated by DP-finetuned BART-base.

Topic model design choices

We choose Wikipedia for our topic model because it is large-scale, high-quality, and semantically diverse. In practice, it is rare to encounter texts that are entirely outside of its scope. The table below with sampled topics of our five diverse datasets illustrates that a broad range of universal topics are captured:

The "out-of-domain" text is also considered and processed in our experiments--in our topic model, there is an “unclassified” bin for samples that are not close to any topic. The table below shows the details

We can see the number of unclassified samples is neglectable across all datasets, and those unclassified samples typically contain little to no substantive content, confirming the broad coverage of our topic model.

Privacy Budget Allocation

Our budget allocation follows [1], our "Post-Generation Resample" baseline, which also has both DP-histogram and DP-finetune steps. For DP-histogram, we adopt the same -- adding a Gaussian noise 10 on every histogram item. The overall privacy budget (ε) is computed as a composition of both steps using the standard dp_accouting package.

We also show individual ε values below on PubMed, under composed ε=4, 2, or 1. DP-histogram only takes a small portion of the overall budget. This aligns with observations in [1] (page 5) and highlights our CTCL-topic design achieves strong performance while consuming only a small portion of the privacy budget.

We also find that the impact of changing the allocation is insignificant. For example, in ε=4 setting on PubMed, increasing histogram's noise from 10 to 20 only alters the DP-finetuning budget $ε$ marginally—from 3.96 to 3.99.

Nucleus Sampling

Our choice of nucleus sampling with top-p = 0.95 follows the standard practice introduced in [2], which has been shown to provide a strong balance between diversity and coherence, outperforming alternatives such as top-k or beam search. Also, since this standard hyperparameter setting already yields strong performance, we chose not to exhaustively tune this particular top-p value.

[1] Privacy-Preserving Instructions for Aligning Large Language Models, ICML 2024

[2] The Curious Case of Neural Text Degeneration, ICLR 2020.

Thank you for your supportive and detailed comments! We are encouraged by your feedback that our approach is novel, our experiments and analyses are comprehensive, and that our method makes significant advancements in privacy-preserving data synthesis.

Privacy Attacks

According to the post-processing property of differential privacy (DP), our synthetic data has the same formal DP guarantees—$\epsilon=1,2,4$—as our generator, and these privacy budgets are widely accepted in the machine learning community, with $\epsilon < 10$ considered reasonable and $\epsilon \approx 1$ regarded as offering strong privacy protection (Ponomareva et al. 2018).

DP already offers provable protection against membership inference and other, typically weaker, attacks such as reconstruction or attribute inversion attacks. This is supported by extensive empirical evidence, e.g., Steinke et al. 2023, Andrew et al. 2023, and Nasr et al. 2023.

Furthermore, recent works such as Yue et al. 2023 and Yu et al. 2024’s experiments on secret sharer-style attacks (Carlini et al. 2018) in the DP synthetic data setting further support the privacy protection offered by DP.

As practical attacks are usually weaker than the worst-case attacks that DP is designed to protect against, based on empirical privacy auditing and attack studies in recent years, we skipped auditing in this paper. That being said, we would be happy to include relavant experiments if you have a specific attack in mind that is beyond our consideration and feasible in our setting!

Thank you for your thoughtful comments and encouraging feedback that the problem we investigate is important, that our method is well-motivated and matching real-world applications, that our experiments are thorough, and that our paper is well-written!

Pretraining

We would like to clarify that our pretraining does not start from random initialization. As noted in Sec. 3.1, we perform continual pretraining on top of BART-base [1], a model already pretrained on a general corpus and it possesses basic language understanding abilities. Our continual pretraining is to endow the model with controllability.

We agree that knowledge distillation could be an alternative to our pretraining, while it still falls within our overall framework, as it also needs properly constructed data in order to achieve controllability. Moreover, it should require more meticulous design choices and more computation due to the large teacher model.

We would also like to emphasize that the core contribution of our work is the overall CTCL framework. Each module design within the framework remains flexible and open to further optimization and design improvements.

[1] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, ACL 2020

Discrepancy in constructed conditions between pretraining and finetuning

Why not topic model for pretraining? The distinction between pretraining and finetuning conditions is intentional for two purposes: enhancing model generalizability and adhering to privacy constraints. In pretraining, using conditions from 2B LLM is for the diversity and flexibility of the aspects that the model can handle, enabling stronger controllability. Otherwise, “keywords” would become the only aspect in the pretraining.

why not Gemma 2B for private data? If Gemma 2B were applied directly to private data, the extracted conditions would be in free-text format. This raises our concerns about sending the private information to the synthetic data generation process. In contrast, our framework ensures differential privacy by using a noised, topic-wise histogram instead. This preserves meaningful high-level topic information of the private domain while satisfying privacy requirements.

To demonstrate the value of the flexible conditioning in pretraining, consider one may have a bit prior knowledge about the downstream domain, e.g., knowing the data is dialogues. In our framework, such prior knowledge can be incorporated, providing a better initialization for DP-finetuning.

Below are the model generations controlled by doc type using the same keywords (Surfing, World Championship, Young Athletes), showcasing the better initialization:

• (Doc Type: News): A member of WSA and WSL, won the World Championships for the first time in his career. [...]
• (Doc Type: Chat): How do you feel about the current state of surfing in the U.S. right now? \n There are a lot of great young athletes [...]
• (Doc Type: Blog Post): I've been fortunate enough to have the opportunity to meet the most talented young athletes [...]

This flexible design also intends to encourage broader and more creative uses of our model.

Adapting condition generation to PE methods

We agree our condition generation could be potentially adapted by PE, for example, within the variation prompts. However, as noted in the paper, a key limitation of PE methods is not learning fine-grained info from private data. Applying our condition generation into the PE framework might not address this key limitation of theirs.

Topic Model on Wikipedia and Out-of-Domain Generalization

We choose Wikipedia for our topic model because it is large-scale, high-quality, and semantically diverse. In practice, it is rare to encounter texts that are entirely outside of its scope. The table below with sampled topics of our five diverse datasets illustrates that a broad range of universal topics are captured:

The "out-of-domain" text is also considered and processed in our experiments--in our topic model, there is an “unclassified” bin for samples that are not close to any topic. The table below shows the details

We can see the number of unclassified samples is neglectable across all datasets, and those unclassified samples typically contain little to no substantive content, confirming the broad coverage of our topic model.

We sincerely appreciate your recognition of the adequate discussion in our paper writing, the accurate identification of limitations in prior approaches, the efficiency of our lightweight final model, and our efforts to reduce reliance on prompt engineering.

Public Data capturing private domain info.

We agree a key intuition behind our design is the information learned from large-scale, diverse public data can benefit downstream learning in private domains. This aligns with the principle behind the pretraining–finetuning paradigm adopted across most LLM research.

Topic model design choices

We choose Wikipedia for our topic model because it is large-scale, high-quality, and semantically diverse. In practice, it is rare to encounter texts that are entirely outside of its scope. The table below with sampled topics of our five diverse datasets illustrates that a broad range of universal topics are captured:

The "out-of-domain" text is also considered and processed in our experiments--in our topic model, there is an “unclassified” bin for samples that are not close to any topic. The table below shows the details

We can see the number of unclassified samples is neglectable across all datasets, and those unclassified samples typically contain little to no substantive content, confirming the broad coverage of our topic model.

Privacy Budget Allocation

Our budget allocation follows [1], our "Post-Generation Resample" baseline, which also has both DP-histogram and DP-finetune steps. For DP-histogram, we do the same -- adding a Gaussian noise 10 on every histogram item. The overall privacy budget (ε) is computed as a composition of both steps using the standard dp_accouting package.

We also show individual ε values below on PubMed, under composed $ε=4, 2, 1$. DP-histogram only takes a small portion of the overall budget. This aligns with observations in [1] (page 5) and highlights our CTCL-topic design achieves strong performance while consuming only a small portion of the privacy budget.

We also find that the impact of changing the allocation is insignificant. For example, in $ε=4$ setting on PubMed, increasing histogram's noise from 10 to 20 only alters the DP-finetuning budget $ε$ marginally—from 3.96 to 3.99.

[1] Privacy-Preserving Instructions for Aligning Large Language Models, ICML 2024

Computational Cost

Our pretraining has 3 steps:

1. 2B LLM inference on a 430M documents.
2. Training a 140M LM on 430M (condition, document) pairs.
3. Generating embeddings for 6M Wikipedia docs with a 20M embedding model, followed by a HDBSCAN clustering.

Among these, Step 1 dominates the computation due to the 2B LLM. This remains lighter than or comparable to the computation of pretraining a 2B LLM.

Notebly, we will release our pretrained models, so users don't take the pretraining cost. This corresponds to the pretraining of the LLMs behind the APIs in PE-based approaches.

For the downstream stage, PE methods often have significant user-side costs. For instance, AUG-PE synthesizes 2,000 samples in T=10 evolution iterations, each with L=4 variations, totaling 80K (2K x 10 x 4) ChatGPT requests. This represents a substantial API cost, whereas our approach applies finetuning on the lightweight model on local devices, which has no significant cost overhead.

Use of Gemma

As shown in our paper Table 1 and mentioned in Sec 3.1, unlike PE’s reliance on strongest LLMs like ChatGPT, our framework only requires basic instruction-following ability of the LLM, since aspects are extracted from existing documents without relying on LLM’s creativity.

Notably, the LLM we use, Gemma-2-2B, is already one of the smallest LLMs. Larger models, such as its 9B and 27B variations, would likely yield even stronger results. We don’t experiment with more LLM choices because the small 2B LLM already has strong performance.

We would also like to emphasize that the core contribution of our work is the overall CTCL framework. Each module design within the framework remains flexible and open to further design improvements.

Pseudo-code and Limitations section

Thank you for the suggestions! While ICML does not require a Limitations section, we will include both our pseudo-code and a limitation section into our next paper version for improved clarity.