Privacy Preserving API Fine-tuning for LLMs

Thank you for the review! Please find our response to your comments below:

Regarding the privacy-preserving backpropagation. Although $\partial l / \partial h$ is protected, the server still has clean $\partial h/\partial \theta$. This still leaks information about the label. The 2-dimensional TSNE of $\partial h/\partial \theta$ of should be similar to $\partial l/\partial\theta$, because multiplying is only a linear operator.

For this concern, we agree that $\partial h/\partial \theta$ is similar to $\partial l/\partial\theta$. For this reason, we report $\partial l/\partial\theta$ in Figure 3 ( in addition to $\partial l / \partial h$ before that). There, we observe that $\partial l/\partial\theta$ is also obfuscated through private backpropagation. Please also note that while $\partial l/\partial\theta$ and $\partial h/\partial\theta$ are indeed linearly dependent, there is a distinct linear dependency for each sample and at each training step: as a result, simply trying to recover them with TSNE is unlikely to be fruitful.

If there is any specific experiment we can run to check your hypothesis, we are happy to discuss that.

Regarding the privacy-preserving forward. If the label can be predicted via only linear transformations of $h_i$, then it means the unaggregated $h_i$ leaks a lot of information about the label. E.g., the server can simply run some clustering algorithms, which will achieve good performance because $h_i$ are linearly separable w.r.t. the labels.

We split this concern into two statements:

1. If the label can be predicted via only linear transformation, then $h_i$ leaks information about the label

2. If 1. is true, the server can run clustering that will achieve good performance for separating labels

The first statement is true: if the label can be predicted via linear transformation, then $h_i$ leaks labels in the information-theoretic sense. However, in our case the label cannot be predicted with a linear transformation. The algorithm described in Section 3.3 explicitly trains $h_i$ so that no fixed linear transformation can recover labels from them. This idea is formulated and discussed in the first half of page 7.

Furthermore, even the label could be predicted linearly, this would not guarantee that a clustering algorithm would be able to recognize it. To illustrate this point, consider a uniform ball of data points with a radius of 1 that contains points with two labels. All points with label 1 are above a certain (e.g., random) separating hyperplane, and all points with label 0 are below that hyperplane. Therefore, the data is linearly separable. However, no clustering algorithm can distinguish between the correct hyperplane and any other hyperplane without knowing labels. Without labels, the data is a uniform ball that can be split equally well in any direction.

In preliminary experiments, we found that 2-cluster k-means has near-random performance for both P$^3$EFT and baseline methods. We will add these results in the revised version of the paper.

We thank the reviewer for their feedback; we do our best to address their concerns below. We also ask the reviewer for clarifications on several replies.

No clear security model.

We firmly believe that we have provided a detailed portrayal of the attacker model in section 3.1 (i.e., “honest, but curious” model with details); however, we are open to improving it further. If there are any specific aspects that require clarification, kindly provide further details so that we can address them in the revised version.

Equation 2 in Section 3.2 contains "two identical independent servers that offer backprop API."

Thank you for pointing out this oversight! We will update the submission to clarify that we consider two setups: one with a single server and the other with multiple servers. We will also specify which setup we are addressing in each specific part of the article.

In particular, Equation 2 is essentially similar to Part 2 in [REF1].

We thank the reviewer for drawing our attention to the technique of secret sharing, and we agree that it has similarities with Equation 2. However, we believe that there is still a significant difference between these methods. Please see the upcoming General Response for details.

Additionally, secret-shared backpropagation has already been solved in the early work [REF2].

We agree that the algorithm proposed in [REF2] does address the challenges of private forward and backward passes in the case of two servers. However, in our opinion, their solution may not be suitable for the scenario of fine-tuning models via APIs. This article uses MPC-friendly approximation of softmax, but trains the model from scratch. Nevertheless, the original model was pretrained with original softmax, and in order to fine-tune it using their method, it would be necessary to replace softmax with an MPC-friendly approximation. However, in general, replacing a layer does not guarantee equivalent results, and the authors of the paper did not specifically investigate this particular scenario of fine-tuning.

Furthermore, in their case, the algorithm always requires two servers, whereas our method, with slight modifications, operates effectively with just one server (second part of section 3.2).

In this sense, how about local differential privacy on labels or noisy label learning?

Thank you for pointing out this oversight. We fully agree that the paper will benefit from including a discussion and comparison of our method with techniques based on noisy labels and differential privacy.

Previous work has shown that gradients with respect to activations are a very vulnerable aspect of split learning in terms of privacy. Techniques such as noisy labels and differential privacy are used to overcome this very problem [1,2]. However, these techniques rely on using noisy gradients instead of the true gradients, which can lead to a decrease in final quality (see e.g. non-private baselines in [3], [4]).

In contrast to previous work, parameter-efficient fine-tuning does not require computation of gradients with respect to all parameters. We only need to update a very small subset of parameters, therefore it is needed for the client to know true gradients just for that subset. This allows us to compute true gradients in a completely private manner without decreasing final quality.

Following 1, with access to the full features of target domain, this work is also related to source-free domain adaptation. I understand the applied loss takes label in this work and thus should be more informative than UDA. It would be better if the necessity of using labels could be clarified.

The majority of language models have been pretrained either on a masked word prediction or causal language modeling objective. As a result, their pretrained activations are not suitable for classification tasks such as sentiment classification or predicting logical relations between statements. We can provide one of our experiments as evidence for this claim: as a baseline, we trained only the classification head without adapters, and the results were significantly worse (Figure 4). Thus, we believe that unsupervised learning may not be very effective for this task. If you know of any results that show otherwise, we would greatly appreciate it if you could point us to them so that we can compare our method with them.

There is not much referring to the “local layers” in Fig. 1. Are these layers learnable or fixed? Can you explain why it is rational to be learnable/fixed for clients in real scenarios?

These layers are learnable, they correspond to a standard learnable classification head for BERT fine-tuning. Running those computations takes negligible time compared to LLM and thus can be performed even by general purpose computers without dedicated accelerators (e.g. GPUs). Therefore, we believe that there is no need to freeze these layers as it would reduce the number of trainable layers and thus the overall model capacity. Additionally, to the best of our knowledge, training only the head is a standard solution for fine-tuning BERT for classification tasks.

When taking about fine-tuning APIs in the paragraph 3-4, I think some recent works are missing, especially from the privacy preserving motivation.

We appreciate your bringing these articles to our attention. We acknowledge that both of these methods are potentially much less vulnerable to potential attacks than our method, but they do have some drawbacks which we will discuss below.

From our perspective, the first article presents a promising approach for our problem setting, but there may be issues with its application due to the large number of forward and backward passes required to obtain gradient estimates, which increases compute cost.

The second article also proposes an elegant method that is fully private for the client by design. This method still requires running a distilled model on the client side. However, the authors explicitly state in their limitations that they used a distilled model size that was not smaller than one-third of the size of the original model. This is still incredibly challenging for clients due to the large memory and computational resources required for training. In contrast, our method only requires the client to perform forward and backward passes through the head layers, which poses no problem. Therefore, we believe that our method is potentially more scalable.

Thank you for a detailed review! In the following response, we aim to address your concerns. Overall, we believe that most of the weaknesses stated in the review are questions and we do our best to answer them below.

Description of the proposed method

The reviewer asks how LoRA adapters are initialized and how they are used in computations. Overall, we define the multi-party fine-tuning procedure in Section 3.1, which follows standard LoRA fine-tuning. Our algorithm is an extension of this procedure.

Before the training, adapter weights and head weights are randomly initialized on the client side.

During the training on each step:

1. adapters first transferred along with the inputs from client to server
2. server runs forward pass with given adapters
3. server computes activations from the given inputs and adapters and returns them to the client
4. client computes loss and real gradients w.r.t. to the received activations
5. client transfers noisy gradients (produced with method described in S3.2) to the server
6. server runs backward passes and obtains noisy gradients w.r.t to adapter weights
7. server transfers noisy gradients w.r.t. to adapter weights to the client
8. client computes real gradients w.r.t. adapter weights
9. client executes a step of an arbitrary optimization algorithm.

Overall, we agree that the paper can be improved by adding a dedicated description of the algorithm, and we will improve it in the nearest revision.

What is this task-specific loss function?

For SST2 and MRPC tasks use categorical cross-entropy. In general, this is the loss function used in the standard fine-tuning procedure for downstream tasks, as defined in the supplementary code of the GLUE benchmark paper.

How and which adapters parameters are updated?

Client updates all adapter parameters after each backpropagation step, which is the standard way of training LoRA adapters defined in Hu et al., (2021) [1]

There are no privacy guarantees provided, the privacy promise of this paper is ad-hoc and it is just based on increasing the number of servers

While we discuss the privacy of two-party fine-tuning scenario at the end of Section 3.2, we agree that the paper would benefit from making the claims more specific. We will rewrite them in the updated version.

assuming they do not collude but assuming that they have the same model

We would like to clarify that the second assumption (same model) is relatively common. In NLP, there are popular repositories of public models that are used for fine-tuning, e.g. [2] [3]. Most fine-tuning runs reported in NLP research, including most of the runs in the LoRA papers (see [1, 4]) use open-access models. When fine-tuning in this setup, every server can download openly available model weights prior to training, and the client benefits from the shared compute power of the servers. Thus, if the first assumption holds (there are non-colluding servers), then these servers can get the same model.

The observation listed as one of the main contributions … has been already demonstrated in existing works such as Li et al. (2022) even in a more generic way as opposed to simple binary classification tasks that are considered in this submission.

To the best of our knowledge, Li et al. (2022) also consider the problem of binary classification. Additionally, in the work of Li et al. (2022), the goal is to prevent label leakage through gradients, while our work also considers activations. Finally, we state our contribution as analyzing the privacy of PEFT algorithms (specifically) in practical fine-tuning setups, while Li et al (2022) studies full model fine-tuning. We agree that Li et al (2022) is a highly related work and we cite it as such. However, it does not overlap with our contributions.

a clear description of LoRA which is the main building block of the proposed framework is missing.

We agree and will include that description in the nearest revision.

Not clear what would be the novelty of the proposed privacy-preserving backpropagation in comparison to secret sharing in 2 party computation

We appreciate the reviewer's input regarding the technique of secret sharing and agree that it bears some similarities to our method of privacy-preserving backpropagation. However, we maintain that there are important distinctions between us and secret sharing from multi-party computation. Since another reviewer asked for the same comparison, we will discuss this in more detail in the upcoming General Response.

[1] https://arxiv.org/abs/2305.14314

[2] https://huggingface.co/

[3] https://pytorch.org/hub/

[4] https://arxiv.org/abs/2305.14314