Differentially Private Model Compression via Selective Pretraining

We thank the reviewer for the encouraging and constructive feedback. We have taken your suggestions to improve our submission. Please find our response below.

Q: PRV accountant was chosen for the DP analysis. Could the authors elaborate on the rationale behind this selection.

A: We use the PRV accountant because it provides tighter bounds on privacy parameters $\varepsilon$ and $\delta$ than moments accountant. We have added an explanation regarding this choice in Section 2.

Q: For clarity and consistency, it would be beneficial to include the sequence size for OpenWebText.

A: The sequence size for OpenWebText is 512, which aligns with the sequence length mentioned in the original GPT paper [1]. This is shorter than the sequence length in the GPT-2 paper [2] (1024). We set the sequence size to be 512 to reduce the computational expenses. As per your suggestions, we have added these details in the revision.

Q: Figure 1 and Figure 4 have variations in the y-axis and gridline spacing.

A: Thanks for the suggestion. We will make the y-axis spacing in Figure 1 and Figure 4 to be the same. For displaying the perplexity, both figures now use a spacing of 2. For displaying the accuracy, both figures now use a spacing of 1.

Q: It appears that the tiny model's performance is slightly lower in terms of accuracy (approximately 33.9% for Sel. PT+DP as opposed to 35.1% for GPT-XL) and exhibits a higher perplexity (around 49 for Sel. PT+DP compared to 44 for GPT-XL).

A: We thank the reviewer for the careful reading. Yes, your interpretation aligns with our findings, and it was a typo. We have revised this paragraph to the following. “Observe that the performance of a small model with only 45 million parameters is comparable to the zero-shot performance of GPT2-XL public model with 1.5 billion parameters.”

Q: In Section 1.2, could you specify the particular English corpus used to train the EnglishGPT-124M model? Additionally, for the EnglishGPT-82M's training data, is it a subset of the aforementioned corpus, or is it an entirely distinct dataset that bears distributional similarities to the Enron dataset?

A: The pre-training corpus for the EnglishGPT-124M model is the full OpenWebText. The EnglishGPT-82M’s pre-training data is a subset of OpenWebText, which is selected using the algorithm in Section 3.1.

Q: Specifically, is each sequence treated as a single datapoint such that its addition or removal wouldn't impact the algorithm's output? If so, does the sequence length have an influence on the privacy budget?

A: For the GLUE tasks, the unit of privacy protection is a single training sample in the original datasets. For the Enron email datasets, we break the dataset into sequences of length 256 and the unit of privacy protection is a single sequence (we discussed this design choice in Section 4.1.1).

This design choice is based on the observation that most of the emails in the Enron email dataset are shorter than 256. For long emails, it is possible that it may be splitted into multiple training sequences, and hence the privacy budget for that email would increase. Another option is simply to use a random subset of 256 contiguous tokens. However, we agree that in real world applications, it is important to carefully bound the maximum contribution of a single email/user.

Thanks for your suggestion, and we will add this discussion to our revision.

Q: In Appendix C, when discussing the Data Selection for the Enron Email, there seems to be a mention of the data being "6 times larger" than the target data (likewise 6 times smaller). However, in Section 3.1, it's indicated that negative examples are "five times larger than the number of positive examples".

A: Thanks for the question. Our description in Appendix C is consistent with our description in Section 3.1. The training set of the domain classifier contains 6N samples, where N is the size of the target data. The training set contains 5N negative samples and N positive samples, which is 6 times larger than the size of the target data. We will revise the corresponding descriptions to clarify this.

[1]: Improving Language Understanding by Generative Pre-Training

[2]: Language Models are Unsupervised Multitask Learners

Thanks for the detailed and fair review. We appreciate your time and effort, and we will certainly incorporate some of your feedback in the final version of the paper. Below we address your specific questions.

Q: Privacy guarantees of public data and generalization concerns.

A: These are great questions. Consider our experiments on the Enron email dataset. To ensure that there is no data contamination in the selected pre-training data, we did a 10-gram deduplication between the Enrom emails and the full OpenWebText before the selection. The deduplication follows the algorithm described in Section 4 of the GPT-2 paper [1].

Conceptually, the improvement of our algorithm could be interpreted as follows. Writing styles of users vary according to the context. For example, writing emails is different from writing academic papers which is different from language usage in text messages. While particular emails of users are private, the use of language in writing emails among the population has some common “stylistic similarities” that are preserved by data selection algorithms. Putting it another way, our hypothesis is that, while emails themselves are private, the style and language usage in writing emails is not. The goal of selective pretraining is to first teach this style of writing emails to the model.

Our industry deployments in multiple applications also gives confidence that our framework generalizes to more real-world settings, and this generalization is not due to examples similar to downstream datasets being there in public datasets.

Q: More experiments on scaling behavior.

A: We agree with you that more experiments are needed to understand the scaling behavior of the models. This is one of the main reasons we did not emphasize this aspect as much as we wish to. Performing experiments to establish scaling behavior (in a spirit similar to the Chinchilla paper) is extremely expensive, and we hope to do that in the future. Unfortunately, however, we may not be able to do that for this paper. But we think that this is a great research direction and one of our fav open problems.

Q: Comparing with other compression techniques.

We agree that our framework can be combined with other compression techniques. However, as has been shown by Mireshghallah et al, a black box application of Knowledge Distillation algorithm in DP setting is challenging and can lead to huge performance loss. On the other hand, pruning techniques typically lead to unstructured sparsity, which does lead to better inference time. These methods were explored in Mireshghallah et al paper.

Having said that, one possible approach to further compress our models is to take our selectively pretrained language model, do model compression using the non-private KD algorithm on the public data, and then followed by DP fine-tuning on the private data. It is reasonable to believe that this could further improve compression ratios.

Goal of this paper was to initiate these conversations, and to show that private data can be used at all stages of the LLM training pipeline in novel ways. While performing all the experiments you suggest may not be possible within the scope of this paper, we hope that our work sets benchmarks that others can improve upon. Moreover, our work already gives enough evidence to the main technical question we set forth to answer: Can small DP models achieve good utility vs privacy tradeoffs as that of larger ones, which was an intriguing open problem in this space.

Different Data Selection Policies:

This is a great question. We agree that more sophisticated data selection policies may lead to better results. However, we chose a simple classifier-based approach for two reasons: 1) Our approach needs no extra work to make data selection step DP, is simple and effective. 2) We wanted to show the power of the overall framework rather than the data selection algorithm itself. We hope that our work gives motivation to explore more sophisticated data selection algorithms that you mention, and study them in the context of DP.

Reproducibility:

Our source code is in the supplementary, which contains a detailed readme to reproduce the main findings in the submission. We will also make the source code available online as soon as the review process is over. We are happy to take any further suggestions regarding our code.

[1]: Language Models are Unsupervised Multitask Learners

We thank the reviewer for careful reading of the paper and giving us valuable feedback. We appreciate your time and effort.

It appears that there is a misunderstanding of our framework, which made you think that our DP guarantees are wrong. We can assure you that our framework is provably and rigorously (epsilon, delta)-DP with respect to the private data as defined in Section 2.

Before we elaborate, let us first try to understand where the confusion is. From your comment, it appears that you expect DP guarantees to hold on the public data, a subset of which is selected for pretraining? If this is the case, then you are right that our framework does not provide any DP guarantees on the public data set. However, this is by our definition.

Let us recall our problem setting as described in Section 2: Input to our problem is a private dataset D(priv) corresponding to a downstream task T, a model M of size p, privacy parameters epsilon > 0, delta > 0, and a public dataset D(pub). Our goal is to train M on public and private datasets with the aim of maximizing the downstream performance on the task T. The entire process should be (epsilon, delta)-differentially private with respect to D(priv).

Note that our problem definition does not require us to give privacy guarantees on the D(pub). Public dataset is assumed to be public, and this is the assumption in all the previous works in DP + LLM space, and nothing specific to this work. See for example: “Large language models can be strong differentially private learners ICLR 2022, “Exploring the limits of differentially private deep learning with group-wise clipping ICLR 2023”, and most of the works cited in our paper.

Now if you are concerned that our framework is not DP with respect to D(priv), and we are happy to provide a full proof in the final version of the paper. We omitted the proof and only gave a sketch in Section 2 for two reasons: a) The privacy analysis of DPSGD is somewhat standard and is well known in the community b) Due to page limit. To give you some more confidence, a model trained using our framework is serving billions of queries per day in a top AI company, so we would not afford to make such a mistake. Having said that, we agree with you that providing a full proof helps to make our paper self contained. We will incorporate your feedback in the final version.

Let us give a brief explanation of why our overall framework is differentially private with respect to D(priv). First of all, finetuning stage of our framework uses DPSGD and thus it is (epsilon, delta)-DP with respect to D(priv). Another step in our algorithm that uses private data is training the classifier. Our classifier is simply a 124-million parameter pretrained GPT2 model. We train it using DPSGD on the following dataset: 1) samples from D(priv) labeled as positive 2) some random subset of public data labeled as negative. The classifier is trained to predict positive labels. We train the model using DPSGD. This is explained in Section 3.1

The proof of DPSGD (see for example Abadi et al 2016, Gopt et al 2021 ) says that weights of the classifier satisfies DP guarantees. This further implies that we can use this model to do data selection without any additional privacy loss due to the post processing theorem.

We split the privacy budget to account for both these steps. We allocate a small fraction of the privacy budget to train the classifier and the rest of the privacy budget is allocated to the fine-tuning stage. Finally, we use advanced composition to calculate the final privacy loss. This is also explained in Section 3.1 and in the experiments section.

We hope this alleviates your concern. We are happy to clarify if you have further questions.

Multiple reviewers (cqKu,mFxj) found our work novel, important, and correct. We are quite excited about this work as 1) it shows that private data can play a role in all the stages of the training pipeline, and opens up exciting new research directions 2) one can train small language models of size 40 million parameters that are as good significantly larger public models, which was an important open problem in the literature. Further, since our model is now being deployed in a large AI company and serves billions queries per day, it gives further evidence that our framework should generalize to real life work loads.

If you feel more confident about correctness now, please consider revising the score.

We thank the reviewer for careful reading of the paper and giving us valuable feedback. We appreciate your time and effort. Before we address your specific questions, we want to offer a different view of our results and seek your feedback. The review seems to evaluate the paper by directly comparing it to non-DP deep learning and concludes that technical novelty is limited. However, our main focus is on private language models and the role of private data. Even if we accept that all the ideas have been tried in non-private literature, which we do not completely agree with, it is not at all clear to us why these things should come together in the private world and lead to good DP models. To give an example, knowledge distillation works great in the non-private world, but we know it does not work in the DP world; see Mireshghallah et al. paper. Further, there is no evidence in the DP literature prior to our work to suggest that a 30 million parameter model can beat the next word prediction accuracy of a significantly larger public model and satisfy DP guarantees at the same time. Moreover, we believe that our work introduces several novel ideas to the DP LLM community, which other reviewers (mFxj,cqKu) seem to agree with and find it novel.

We list some novelty of our work with respect to DP Literature:

Almost all the recent results in DP literature suggest that one needs large pretrained models to achieve good performance in the private world. See “Exploring the limits of DP DL with group wise clipping” for example. One of the main open problems in differentially private deep learning is to bridge the utility-vs-privacy gap between large language models compared to small language models. As you can verify, other reviewers agree with this and find our solution novel. This is the first paper to show that small models can achieve comparable utility-vs-privacy tradeoffs, and we achieve SOTA of the results for DP performance for small models improving upon Mireshghallah et al. [58].

No prior work in DP literature has studied how one must do pretraining to enhance the transfer learning abilities of a model and its impact on the model size. Here we also show that better pretraining is more important for DP deep learning compared to non-private deep learning.

Our work shows how the quality of tokens can change pretraining scaling behavior. If one is not careful in pretraining data quality, then we need a larger model to achieve the same performance. On the other hand, restricting pretraining dataset to smaller but better tokens, one can train a smaller model. We are not aware of such a study in DP literature before.

We provide a general framework on how a small amount of private data can be used to improve pretraining, fine-tuning, and data selection. No prior work has explored utilizing private data at all stages of the training process.

In many real-world applications, the public models such as GPT or LLaMA cannot be trained (or fine-tuned) on private datasets due to privacy concerns. Hence, both improvements in performance and model efficiency are possible if one had access to the private data. Privacy enhancing technologies such as DP provide the infrastructure to safely unlock the high-quality data, thus simultaneously improving both performance and model efficiency.

DP as a tool for model compression and using private learning to improve efficiency vs accuracy tradeoff has never appeared in the DP deep learning literature. We anticipate (and very excited) that it is this aspect of private learning that will have a huge impact in practice. Indeed, a model trained using our framework is saving millions of $ of inference cost in a large organization as we wrote in Lines [85, 92]. It is a message worth spreading. Given a venue like ICLR, we anticipate that this work will be a harbinger of a new direction in privacy preserving deep learning.