Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning

Q1 Limited contribution: straightforward solution building upon existing method

A: While our work builds on existing methods, our main contribution is to provide a comprehensive understanding of the pros and cons of group privacy and user-wise DP-SGD through systematic evaluation. We tested two datasets, three models, various DP budgets, different examples per user, and multiple data selection methods.

Our proposed data selection strategies, such as random chunk selection that crosses sequence boundaries, are well-motivated. They aim to maximize the preservation of per-user information to achieve a better privacy-utility tradeoff and have proven to be effective, improving the Enron Email perplexity from 35.68 to 34.01 w/ ε=1.0.

Q2 Comparison w/ other DP approaches

e.g. Large Language Models Can Be Strong Differentially Private Learners. ICLR’22

A: Please note that the choice of DP fine-tuning recipe is orthogonal to the user-level DP algorithms, as the latter can be seamlessly combined with any DP fine-tuning pipelines. Our evaluation used the strong DP fine-tuning baseline from [1], and showed consistent results across models (GPT2 small, medium, large) and datasets (BookSum & Enron Email).

In response to the suggestion, we reran our evaluation using the pipeline from the referenced ICLR'22 work on Enron Email. Consistent with our previous findings, random chunk works best for user-wise DP-SGD, and longest sequence selection for group privacy. The performance of both mechanisms, using their respective best selection strategies, is shown below and aligns with our submission results.

Q3 Assuming user info available: might be missing in real applications

A: We study the standard notion of user-level DP (Liu'20, Levy'21, Ghazi'24), which is increasingly important [2] as example-level DP alone is insufficient to provide the same level of protection for users contributing multiple examples. While our current evaluation uses user identifiers to group examples due to the lack of dedicated public datasets for such evaluation, in real world settings, these methods do not require user identifiers as long as there is a mechanism to group examples based on user ownership.

[1] Yu et al. Differentially Private Fine-tuning of Language Models. ICLR’21

[2] Case et al. A distributed attribution and aggregation protocol.

Q1 Implementation of User-wise DP-SGD

The paper mentions that User-wise DP-SGD may require modifications to the data loading pipeline, which could be a barrier to adoption in practice.

A: Thanks for the comment. Please note that an efficient implementation of user-wise DP-SGD can be achieved with only minor changes to the data loading pipeline. Specifically, it entails:

• Modifying the per-example dataloader to a per-user dataloader (line 5 in Algorithm 2), which only requires implementing a customized dataloader with a few additional lines of code.
• Fetching training examples for each user and taking the gradient average per user on the fly during each gradient step (lines 6~10 in Algorithm 2).

Fortunately, since both steps can be implemented with minimal effort, user-wise DP-SGD can be easily adopted in practice.

Q2 Computational overhead

The computational overhead introduced by the differential privacy mechanisms is mentioned but not thoroughly analyzed, particularly for larger models or datasets.

A: Thanks for the comment. The main overhead of group privacy training methods is introduced by the per-example gradient clipping and noise addition (lines 11-13 in Algorithm 1). For the user-wise DP-SGD method, the primary overhead stems from the user-wise data sampling (line 5 in Algorithm 2), as well as per-user gradient clipping and noise addition (lines 8-9 in Algorithm 2).

To provide a more quantitative analysis, we present the following table showcasing the training time (in seconds) per step for different models and training methods with $k=10$ on the Enron Email dataset. The results are obtained using LoRA fine-tuning with rank being 32 and a batch size of 1024, averaged across 100 training steps, on a single NVIDIA A100-80G GPU.

As shown in the table, both methods introduce very little computational overhead (4%–9%) compared to non-private training for the GPT-2 model, regardless of the size.

Q1 Limited technical novelty

What is the challenge to adapt User-wise DP-SGD in LLM given there are many existing methods?

A: We outline three challenges in adapting existing methods and place our contributions in this context.

1. Lack of a systematic evaluation: Prior work (McMahan'18; Kairouz'21; Xu'23) has neither systematically studied user-level DP implementation choices nor their performance in LLMs. Many applications used naive group privacy (Amin'19; Levy'21) which leads to worse utility for smaller ε's. (see Sec.6).

• Our contribution: We systematically evaluated the two mainstream mechanisms for user-level DP in LLMs. We also empirically show the overly restrictive assumptions of Asi'23, an algorithm with a theoretically optimal excess rate.

2. Lack of reliable testbeds for user-level DP methods in LLM applications for effective comparison

• Our contribution: We derived two testbed datasets, Enron Email and BookSum, from existing non-DP-purpose datasets to simulate real-world use cases and benefit future research.

3. Overlooked data selection strategy when adapting user-level DP to LLMs

• Our contribution: We proposed and evaluated data selection strategies for both group privacy and user-wise DP-SGD, showing significant performance gap based on the strategy used - e.g., on Enron Email w/ ε=1.0, perplexity ranges from 34.01 for the best strategy to 35.68 for the worst (Table 4a).

Q2 Lack of baselines

e.g. Large Language Models Can Be Strong Differentially Private Learners. ICLR’22

A: Please note that the choice of DP fine-tuning recipe is orthogonal to the user-level DP algorithms, as the latter can be seamlessly combined with any DP fine-tuning pipelines. Our evaluation used the strong DP fine-tuning baseline from [1], and showed consistent results across models (GPT2 small, medium, large) and datasets (BookSum & Enron Email).

In response to the suggestion, we reran our evaluation using the pipeline from the referenced ICLR'22 work on Enron Email. Consistent with our previous findings, random chunk works best for user-wise DP-SGD, and longest sequence selection for group privacy. The performance of both mechanisms, using their respective best data selection strategies, is shown below and aligns with our submission results.

[1] Yu, et al. Differentially Private Fine-tuning of Language Models. ICLR’21