Scaling Laws for Differentially Private Language Models

We appreciate the feedback, and thank you for sharing ideas for specific ways to improve the paper. We are glad that you found the paper well-written and thorough, but we respectfully disagree with some of your comments, and respond to the main critiques below.

[Motivation] There are several complementary reasons why we believe DP scaling laws for pretraining are well-motivated and of interest to the privacy community.

• Importance of pretraining privacy: For current frontier models, and pretraining specifically, there is increasing need for data privacy, e.g., due to risks of memorization in large models. This makes DP pretraining research relevant as pretraining data is often scraped from the web (and can contain sensitive personal information [1]). The need for privacy will further increase if companies start training on sensitive user data as they run out of public datasets to train on.

• Addressing utility challenges: Utility degradation due to DP actually serves to motivate our work. With enough data and compute, and well-tuned mechanisms, the performance degradation due to DP can be mitigated to a large degree [2]. We believe scaling laws will help in this mitigation.

• Broader Context: Privacy-preserving machine learning falls squarely within the ICML call for papers under the Trustworthy Machine Learning category, and the best paper from ICML 2024 highlighted DP pre-training as an important future direction [1]. Several large organizations have invested in this space already [2,3,4,5,6].

[1] Tramer et al., Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining

[2] De et al., Unlocking High-Accuracy Differentially Private Image Classification through Scale

[3] Xu et al., Federated Learning of Gboard Language Models with Differential Privacy

[4] Anil et al., Large-Scale Differentially Private BERT

[5] Pelikan et al., Federated Learning With Differential Privacy for End-to-End Speech Recognition

[6] https://www.microsoft.com/en-us/research/group/privacy-preserving-machine-learning-innovation/

[Encoder-decoder Models vs. Decoder-only models] Our choice to study BERT models in this research was motivated by previous research on pre-training with differential privacy [3]. We agree that our focus on BERT models raises questions about generality to other model architectures. We acknowledged this limitation in Appendix A. We further provide evidence in Section 3.7 that despite the differences in training setups with prior (non-private) scaling laws work, we are able to reproduce the main finding of Hoffman et al., i.e., that data and model size should be increased in equal proportion with increasing compute. This provides some evidence that our results should translate to other settings. This observation is also consistent with [7] who found qualitatively similar scaling between encoder-decoder and decoder-only architectures. See also our response to Reviewer qUEy.

[7] Yao et al., Towards Neural Scaling Laws for Time Series Foundation Models

[BertMega] BertMega is the only non-standard model config we used, to also explore a larger BERT model than the ready-made configs listed at https://github.com/google-research/bert. We will note this explicitly in the revision.

[1B-2B Models] This is a good idea, and something we discuss in the limitations section (Appendix A). Scaling to larger models introduces significant engineering challenges, most notably handling model parallelism. For such models, weights—and thus their corresponding gradients—are typically sharded across devices. The gradient clipping step in DP-SGD requires storing and synchronizing per-example gradients across layers and devices for clipping, which necessitates substantial changes to existing implementations. While some work has explored this in the context of LLM fine-tuning [8], these challenges remain for training LLMs from scratch. We will be sure to highlight this as an additional challenge to scale DP training to the billion-parameter scale (in addition to DP scaling laws) in the revision.

[8]: He et al., Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping

[Evals on downstream tasks] Thank you for raising this point. We note that performance on downstream tasks are generally too noisy for scaling law studies themselves, as a result, we follow the previous work on LLM scaling laws and focus on the crossentropy loss. We agree that beyond the scaling law studies, it would be valuable to show some downstream evaluations on large (maybe 1B-2B models) DP pre-trained models. We will highlight this as an important direction for future work in revision.

[Other suggestions] We will incorporate your additional comments and suggestions to improve the presentation of our figures and tables, and clarify the relevant parts of the text in revision. Thanks for the suggestions!

We thank the reviewer for the careful review and feedback, and we are glad you liked the paper. Below we respond to the main questions / critiques:

[Missing details] We will be sure to update our discussion around Figure 1 and Section 4.5 to clarify what is shown.

• Regarding Figure 1, we do not use a constant noise-batch-ratio here; instead it is varied through the privacy budget, batch size, iterations, and data budget (as shown in the diagram in Figure 2).
• In Section 4.5, we fix M and T and only consider how the noise batch ratio (sigma / B) changes with respect to the Privacy, Data, and Compute budget.

[Fixed batch size assumption] Our modeling assumption that the training loss primarily depends on the batch size through the noise batch ratio (Figure 2) is a limitation we acknowledge in Appendix A. However, this modeling assumption was necessary to make this work practically feasible, as adding yet another independent variable to consider greatly increases the number of experiments and accelerator hours needed to conduct this research.

As the reviewer points out, our ablations in Appendix C.3 and C.4 do study the effect of this variable in isolation, although we believe it will be interesting to revisit this in future work to better understand the phenomenon we observed and under what conditions it manifests.

[Fixed clipping + learning rate] We lean on findings from prior work to establish a reasonable default DP training setup. In particular, [1] advocates for using a small clipping threshold where all (or nearly all) gradients are clipped. We used a clipping threshold of 1, and checked that this resulted in > 90% of gradients being clipped. We also used gradient normalization as [1] also suggests, since it decouples the clipping parameter from the learning rate. While more careful tuning of this parameter could help somewhat, our primary goal in this work was to focus on variables that relate directly to the compute budget.

Regarding learning rate, we actually used three learning rates, which were chosen based on comprehensive ablations discussed in Appendix C.7 and Figure 12.

[1] De et al., Unlocking High-Accuracy Differentially Private Image Classification through Scale

We thank the reviewer for the careful review and feedback, and for recognizing the various strengths of this work. Below we respond some of the questions / criticisms:

[Masked Language Modeling vs. Other Tasks] The reviewer is correct to point out that our focus on BERT models raises questions about generality of the results with respect to other model architectures, like decoder-only models used in many modern GenAI applications. We acknowledged this limitation in Appendix A. We further provide evidence in Section 3.7 that despite the differences in training setups with prior (non-private) scaling laws work, we are able to reproduce the main finding of Hoffman et al., i.e., that data and model size should be increased in equal proportion with increasing compute. This provides evidence that our results should translate to other settings.

Our choice to study BERT models in this research was based on a few considerations:

• Established pre-determined model sizes and configurations (https://github.com/google-research/bert)
• Maturity of DP training support in distributed ML libraries (at the time this research started): We decided to use PaxML (https://github.com/google/paxml/) to conduct this research, since it supports large-scale training of transformer models coordinated across many accelerators, and supports per-example gradient clipping routines which are necessary to use DP-SGD that can run in distributed settings. This open source library does not feature an easily-forkable implementation of a standard decoder-only model and thus would require a large effort to re-engineer purely from the available primitives.

We believe that future research in this space should probably study (now) open-source decoder-only models like NanoGPT (https://github.com/karpathy/nanoGPT). We believe that the findings we showed are still useful and informative in their current form, however.

[Pretraining vs. Finetuning] The reviewer is correct to point out that with DP, finetuning has some advantages over pretraining, and would be interesting to study. We acknowledged this limitation in Appendix A, and agree with the reviewer that this would be an interesting direction for future work.

[Evaluation task diversity] This is a valid criticism, and something we will be sure to add to our Limitations section in revision, and highlight this as an important consideration for future work.

[Other weaknesses] We have updated our references in the uploaded revision to include a more expansive set of citations. If the reviewer is aware of any relevant citations that were missed, we are happy to consider and add these as well. We have fixed the other minor issues identified during review, including the references on line 15 and the notational errors. Thanks for the feedback!