LMO-DP: Accurately Fine-Tuning Language Models with Stronger Differential Privacy

Thanks a lot for the positive feedback and constructive comments. Please find our response to each comment as below.

Presentation/Typos/Inconsistencies

Thanks for these comments. We have fixed the typos/inconsistencies and added more clarifications. The new submission/manuscript will be updated soon.

Can you explain what you mean by LM geometry in more detail? I found it hard follow the LM Geometry section? What are the specific characteristics? Is $P_{\frac{i}{o}}$ simply the conditional probability $P[o|i]$? This seems to be a conventional language model. I'm failing to understand what aspect of this definition is geometric.

Thanks for this question. LM Geometry refers to the geometric structure underlying the Language Model (LM). Let's delve into its components:

• Parameter Space $\Omega$: this represents the configuration space of the LM. For instance, in a word embedding function, $\Omega$ could include parameters defining word vectors.

• Mapping Functions $\mathcal{F}$: these functions map model configurations within $\Omega$ to specific characteristics. For example, in a word embedding function $v(w)$, $\mathcal{F}$ will map words $w$ to vectors in $\mathbb{R}^d$. Thus, $\mathcal{F}$ captures the transformation from model configurations to observable features.

• Probability Distribution $\mathbb{P}_{i/o}$: this probability distribution characterizes the probabilistic transition from an input token $i$ to an output word $o$. In a standard language model, it indeed aligns with the conditional probability $P[o|i]$. However, the emphasis on "geometry" lies in the broader interpretation. It encapsulates the entire distribution of transitions, offering a geometric view of how the model navigates from one token to another probabilistically.

Specific Characteristics:

The specific characteristics captured by $\mathcal{F}$ might include the spatial arrangement of word vectors or the distribution of model weights. These characteristics contribute to the overall geometry, allowing us to analyze and quantify how the LM operates probabilistically.

We will add more clarifications to the new version and update it asap.

Why did the authors chose RDP accounting over tighter accountants e.g. PRV accountant? PLRV based accountant should be able to handle multiple different noise distributions? (for Gaussian) Would that provide an even better privacy utility trade off?

Thanks for this question. LMO-DP can use other accountants (orthogonal to them) as long as they can handle multiple different noise distributions (e.g., PLRV based accountant). RDP accounting is chosen for two reasons: (1) it can handle different noise distributions and provide a tight accountant, and (2) our Theorem 4.1 can prove that optimizing over RDP is equivalent to optimizing cross-entropy loss. Although PLRV may be able to provide even better privacy/utility trade off, LMO-DP may need non-trivial extensions to integrate it for efficiently optimizing the noise. We will add a discussion for this.

Can you explain figure 1 in more detail and how it shows that the mixture of distributions covers the whole space?

In probability theory, to our best knowledge, there is no universally accepted measure for quantifying the comprehensiveness of a subset of probability functions. Therefore, Algorithm 2 (Quantification of LMO Search Space) is employed to assess the comprehensiveness of this subset in comparison to a universally simulated space, introducing a novel quantification test. We utilize three distance metrics (both probabilistic and deterministic): KL divergence, $\ell_2$ and EMD metrics to measure the distance between these two spaces. Our results demonstrate that, for any given noise 'n', there exists an LMO noise that remains close to 'n'. Please note for the EMD metric, we experience a small divergence (scaled by $10^{-3}$) when the domain of noise increases which necessitates a smaller quantization rate to achieve even better results.

It would be interesting to learn more about the mixture of noise distribution? What type of noise is the main contribution?

Thanks for the question. It is a Laplace-based two-fold noise (the first-fold is the Laplace distribution while the second-fold is a mixture distribution). Specifically, the "inverse" of "scale parameter" of the Laplace distribution (1/b) is subject to the linear combination of Gamma, exponential and uniform distributions. In addition, we have added an ablation study to exhibit the contribution of noises (please see the supplementary document).

Privacy curve \delta(\epsilon) vs the PLRV

We will fix it. Thanks.

As stated above, I believe the main weakness of the paper is the overall presentation. It believe if the authors could improve the writeup the value to the community would significantly higher.

Thanks for the suggestion and encouraging feedback. We are working on improving the presentation/clarity of the work. The new version will be updated soon.

Thanks a lot for the insightful and constructive comments. Please find our response to each comment as below.

Recent works on tight privacy accounting for differential privacy.

Thanks for the suggestions (we will add a discussion and cite them). Their theories can also be applied to our proposed algorithm. We designed our LMO-DP method based on randomization which tries to find a smaller noise while meeting the same level of privacy guarantee. The contributions are orthogonal to these two works. If we integrate LMO-DP with these two works, we need to revise the constraints in Equation (2) and we can replace the RDP by those new accountants/methods with tighter bounds. New efficient and accurate solvers for finding the (sub)optimal noise would be desirable as well.

In section 3.3, the global optimization seems to be formulated for a fixed pair of neighboring datasets D and D′.

Thanks a lot for this good catch. Yes, our design involves all pairs of D and D’ (worst case protection). It was a typo in the constraints in Equation (2). We will add a supremum over choice of D and D’ in Equation (2) to represent all the neighboring datasets (and worst case). Our proof (Theorem 4.1 in Appendix B.1) and implementation (in Algorithm 3 - LMO-DP in the original submission and the code) are indeed over the worst case, i.e., $\sup_{\forall D,D'\in\mathcal{D}}$.

Writing

Thanks. We will fix them.

In equation (2), is there a particular reason for the range of \alpha to be 2 to 129? Is it possible that the best parameter is achieved outside of this range for some datasets and problems?

The range of $\alpha$ was selected from 2 to 32 in most existing works/codes. However, when computing the R'enyi privacy of the LMO mechanism within this $\alpha$ range, null values frequently arise. Expanding the upper bound of $\alpha$ can facilitate the computation of R'enyi privacy for the LMO-DP mechanism. Nevertheless, opting for larger $\alpha$ values may result in moments too tiny for effective processing of the floating point numbers. Consequently, there exists a tradeoff between the choice of $\alpha$ and potential privacy leakage. For our specific tasks, we have empirically settled on 128 or 129 as a compromise. For other tasks, the selection of the $\alpha$ range should be based on privacy requirements and practical constraints (e.g., processing the floating point numbers).

It is surprising to me that the noise level for \epsilon=2,3 is slightly larger than \epsilon=0.2. Could the authors explain this phenomenon?

Thanks for this question. The noise levels in our Figure 2 are different for $\epsilon=0.2$ and $\epsilon=2$. They look close to each other since different scales for y axes were applied in the subfigures. To show the dynamic of the LMO-DP mechanism, we replace Figure 2 with the statistical figure of sampled noise in the revised version (please see the supplementary document and the submission will be updated asap).

Thanks a lot for the positive feedback and constructive comments. Please find our response to each comment as below.

Algorithm 3 seems to be important, adding it to the paper would be nice.

We will move Algorithm 3 to the main text in the new version. Thanks for the suggestion.

There is an inherent tradeoff between privacy and utility, however the results seem to be the opposite, as in even for extremely small values of epsilon, the accuracy of model is pretty high, can the authors please explain this observation?

The privacy/utility tradeoff also exists in our results. Due to the benefit of our mechanism (small noise in a small privacy budget), the accuracy is already very high on a small privacy budget. Thus, sometimes the accuracy improvement is not observed to be significant (but accuracy is still increasing) while increasing the privacy budget. For instance, please see the privacy/utility trade offs in our results in Figure 3, Figure 4 and Table 3.

How would the results change if the models were trained from scratch instead of fine-tuning?

Yes, similar to DP-SGD, LMO-DP can be applied to training from scratch. We did fine-tuning in this work since this is popular in practice for language models (LMs). It can also be applied to other applications, even in different domains (e.g., vision tasks). Due to the greatly reduced noise, we conjecture that similar performance will hold in training or other learning tasks.

How can the proposed method be extended to vision tasks? or vision generative models? For instance NAR generative models solve a bert like optimization problem during training, is there a trivial way to extend this method?

Thanks for this question and the ideas for extension to other domains. It is very interesting to investigate new DP solutions (with better privacy/utility trade offs) for those tasks. For the traditional vision tasks, there may exist a trivial way to extend LMO-DP with the support of DP-SGD (though the LM Geometry should be revised since it is mainly designed for NLP tasks). More specifically, our proposed method can be extended to these vision tasks since the key component of our method is to search the less noise under the privacy constraints. Then, we still can have a similar framework to search less noise. However, for different domains (e.g., vision task), we may need to change the combination of mixture distributions and the loss functions and so on.

For the vision generative models, if DP works for their training/fine-tuning, the accuracy can be further improved by adapting this method with proper extensions. To our best knowledge, in the case of NAR generative models, we may also need to design a new DP framework (and then optimize the noise) for the training as it solves a BERT like optimization problem during training.

Thanks a lot for the insightful and constructive comments. Please find our response to Weaknesses (E)-(I) as below.

(E) Scale in Figure 2 and report useful statistics.

Thanks very much for this suggestion. We changed it to the log scale and reported the average reduction with more data samples (please see Figure A.1 in the supplementary file). The results for noise comparison look more clear (the average reduction rate is 95.13% for $\epsilon=0.3$, 92.19% for $\epsilon=0.7$, 87.71% for $\epsilon=2$, and 87.31% for $\epsilon=3$).

(F) The empirical details are lacking for both reproducibility and understanding of results. How large are the dataset sizes? How many classes are there?...

Thanks for this comment. The optimal weights $\alpha_1$, $\alpha_2$ and $\alpha_3$ for mixture distributions to generate the noise are in the range [0.1, 0.3]. After that, we use the full datasets and all the classes in the original datasets for fine-tuning (same as SOTA). SST-2 has more than 60k+ samples in the training set; QNLI has more than 100k+ samples; MNLI and QQP contain more than 350k but less than 400k samples for each dataset. SST-2, QNLI, and QQP include two classes each; MNLI includes three classes. The hyperparameters setting: batch size 2048, 6 epochs and sampling rate 2048/|D|. We will clarify it in the paper and the readme file for the code.

(G) Figure 5 is also extremely difficult to interpret...it looks like this model can be trained in <10 steps...

This shows the steps that each method needs to achieve the same accuracy. Specifically, we evaluate the steps that need to reach 48.51% accuracy and 41.83% accuracy for the BERT-base and RoBERTa-base on the MNLI-m dataset; the steps that need to reach 70.49% accuracy and 71.09% accuracy for the BERT-base and RoBERTa-base on the QQP dataset; the steps that need to reach 48.98% accuracy and 45.67% accuracy for the BERT-large and RoBERTa-large on the MNLI-m dataset; the steps that need to reach 63.18% accuracy and 67.84% accuracy for the BERT-large and RoBERTa-large on QQP dataset.

We cannot compare the steps to achieve the highest/converged accuracy since the baseline(s) cannot achieve such high accuracy (given the same small privacy budget $\epsilon$). Thus, we found the minimum of the highest accuracy for all the tasks on a specific dataset using similar-parameter models as the reference; then we count the steps to reach this reference.

For instance, LMO-DP needs 1146 steps on the MNLI-matched dataset and 192 steps on the SST-2 dataset (6 epochs) using the roberta-base model. The non-private training needs relatively less (but comparable) steps.

(H) Figure 4 and several tables indicate that LMO-DP with a full order magnitude less privacy cost can outperform DP-SGD. This is an outstanding feat...

Thanks for the suggestion. We will clarify this relationship between noise value and the language model performance in the revised version. Specifically, we propose a suboptimal solution for the cross-entropy optimization based solely on minimizing DP accountant, which helps us reduce the noise for gradients. In our revised version, we redraw Figure 2 to better visualize the comparison between the LMO-DP noise and Gaussian noise. With significantly reduced noise (e.g., high reduction rate), LMO-DP only needs a very small privacy cost to achieve high accuracy compared to DP-SGD.

(I) Though I am not very familiar with the work of Mohammady et al. (2020), their work requires specifying both the utility metric and the query (to be protected by DP). Theorem 4.1 currently only shows the loss/utility metric (cross-entropy) but does not define the query. This makes it seem as though the query is acutally releasing the loss and not the gradients which would lead to privacy leakage. Please clarify.

This is a good question. Yes, Mohammady et al. (2020), requires specifying the utility metric and the query. We define the cross-entropy loss as the utility metric, and the optimization for noise/randomization is formulated at the level of training/fine-tuning (by considering all the gradients with accounted DP leakage as the query). Note that the LMO space is defined based on the overall DP guarantee for the entire training/fine-tuning, which ensures the same DP guarantee as the DP-SGD on noisy gradients with the RDP accountant. In other words, noise (injected to the gradients) is optimally derived for the training/fine-tuning while ensuring the same overall DP guarantee as DP-SGD.

Different from Mohammady et al. (2020), we theoretically prove that optimizing the noise based on the cross-entropy loss/utility metric can be approximated to be equivalent to search over the LMO space (Theorem 4.1). Then, the optimization problem can be more efficiently solved. Note that optimizing the noise at the gradient level may also be feasible, but based on a different metric and different space for DP guarantee of the gradient, which may not be efficiently solved.

Thanks a lot for the insightful and constructive comments. Please find our response to each comment as below.

(*) Seems independent of LMs and can potentially have larger impact.

Thanks for this point. Yes, its applicability can easily extend to other training or fine-tuning scenarios with various adaptations. This is primarily due to two key characteristics of LMO-DP. First, it leverages the cross-entropy utility metric, which finds broad application in tasks across Language Models, Computer Vision, and Audio domains. Second, the universality established in Theorem 4.1 highlights a suboptimal solution based solely on minimizing Renyi-DP and optimization without imposing constraints.

Renyi-DP has also demonstrated its versatility through extensive applications in diverse fields, showcasing compelling adaptations for enhanced efficiency and accuracy. The noise introduced by LMO-DP can be seamlessly integrated with these approaches.

(A) No proofs or high-level description in the main text.

Thanks for the suggestion. We have restructured the core methodology section (to be updated soon).

(B) Algorithm 3

We will move it to the main text. Thanks.

(C) Terminology

Our methodology involves a grid search across 8 parameters, encompassing Gamma, exponential, and uniform distributions along with their corresponding weights, denoted as $a_1$, $a_2$, and $a_3$. We have refined the description for improved readability. “Others” refers to other LMO parameters, including weights of different randomization PDFs and PDF parameters. “Secondary optimization” refers to the optimization inside the predicate for the initial optimization in Equation (2): where R'enyi-DP privacy bound $\epsilon_\alpha$ is defined as $\epsilon(\delta) = \min_{\alpha > 1}...$. It seeks to minimize the discrepancy between $\epsilon_{LMO}(\delta)$ and $\epsilon$.

(D) Comparison with orthogonal methods.

Thanks for this comment. Yes, our LMO-DP framework is orthogonal to several existing works (we focus on minimizing the noise). For this reason, in most of our experiments, we primarily compare it with the Gaussian mechanism for fine-tuning language models on accuracy. Finally, we added a small group of experiments to compare LMO-DP with SOTA methods. Notice that, although those memory reduction (e.g., Ghost Clipping) and parameter efficiency methods improve the performance from different aspects, they are also commonly considered as SOTA methods. We can also consider such comparisons as showing the improvements on accuracy for such orthogonal methods.

Low memory in Table 1

Also, as clarified in Section 1, given such orthogonal contributions, our LMO-DP framework can boost the performance of those methods (e.g., Ghost Clipping) and inherit the benefits of other works (e.g., memory reduction). Thus, in our LMO-DP implementation (shown in the source code), we have two modes, including the one integrated with the Ghost Clipping with low memory and the same accuracy (please see line 1 in lmo-dp/experiments/private-transformers/commands_task1.sh, we can choose ghost or default for “--clipping_mode”). That is why we also give low memory to this work.

Faster Convergence

Yes, our faster convergence is based on empirical observation (we will add a note for empirical observation or remove that column in Table 1). The possible reason for such empirically faster convergence lies in the dynamics of LMO-DP noise. By introducing a significantly smaller perturbation, as illustrated in Fig 2, the incorporation of LMO-DP noise in DPSGD-based technologies intriguingly leads to faster convergence.

Specifically, as verified by our implementation results, LMO-DP noise for strong privacy guarantees (e.g., privacy budget $\epsilon < 0.2$) yields outcomes equivalent to those obtained with Gaussian and Laplacian noise when applied with larger budgets ($\epsilon > 2$), thanks to its noise function. The state-of-the-art in DP versus Convergence, exemplified by JIT [ICLR 21], has demonstrated that DP negatively impacts the convergence rate, especially for smaller epsilon values, leading to considerably longer convergence times of the SOTA methods.

Hence, LMO noise effectively addresses two challenges (accuracy and convergence) simultaneously, making it a solution that hits two birds with one stone.