Label Privacy in Split Learning for Large Models with Parameter-Efficient Training

We thank the reviewer for their feedback and address their concerns and question below.

For example, the last sentence in the abstract is hard to understand.

We thank the reviewer for their careful attention to our paper's writing. We agree that the original phrasing contained a logical contradiction. Our intention was to convey that P$^3$EFT successfully competes with existing methods and outperforms them.

To address this ambiguity, we propose the following reformulation: "We find that P$^3$EFT outperforms existing privacy-preserving methods in multi-party and two-party setups." We would welcome the reviewer's thoughts on this revised phrasing.

Additionally, we are open to any further specific suggestions the reviewer might have regarding the paper's writing quality and clarity.

Only classification datasets are used although generation task is also mentioned in the paper.

Indeed, while we mention generation in line 043, it specifically refers to image generation. But we acknowledge that the current phrasing might be ambiguous, and we will revise this statement in the updated version to provide better clarity.

Regarding text generation specifically, we agree that ensuring label privacy in such setups is of significant importance. However, this particular scenario presents its own unique challenges. For instance, text generation tasks utilize the same tokens for both inputs and labels, creating potential information leakage in both directions. Adapting our method to address these challenges would require additional approaches --- such as integration with input privacy methods --- which would extend beyond the scope of this paper. We believe these considerations require separate, focused study.

We've chosen to focus on classification as a specific, but important setup. This focus allows us to thoroughly examine key aspects of privacy-preserving techniques in split learning. We believe this provides valuable insights that can serve as a foundation for future research.

Aside from label privacy, I believe sample input privacy is also important.

We share the reviewer's emphasis on the critical nature of input privacy. In our work, however, we specifically explore label privacy as one significant component of private learning [1,2,3,4], while recognizing it is not the only one. Since our approach is orthogonal to most feature privacy methods, it can be effectively combined with these methods to provide comprehensive protection for both inputs and labels.

I expect that the authors chould provide a proper real-world setting in which only label information but not input features are sensitive and should be protected.

We provide a potential real-world example of such a scenario in lines 060-064. Specifically, we consider the case of social media user pages, where the data is publicly available, while the labels --- such as user behavior and click information --- are accessible only to the social network.

As I know, some closed-source model training API requires uploading the whole training dataset to the server. Is the proposed method able to preserve the privacy of label under this setting? If not, what might can be done to address this case?

Not in the exact setting you describe. Specifically, requiring clients to provide their exact dataset to the server would result in complete exposure of private data. Traditional approaches to address this privacy concern primarily rely on anonymization techniques, which include either the systematic removal of sensitive components from the training data [5] or the generation of synthetic data that approximates the original distribution while avoiding exact private records [6]. However, these methods have notable limitations, as both approaches typically lead to diminished model accuracy and cannot guarantee complete privacy preservation.

To overcome these limitations, we explicitly design the API protocol in Section 3 in such a way that does not require the client to share their training labels with the server.

[1] Li et al., Label leakage and protection in two-party split learning. ICLR 2022.

[2] Sun et al., Label leakage and protection from forward embedding in vertical federated learning. arXiv:2203.01451, 2022.

[3] Liu and Lyu., Clustering label inference attack against practical split learning. arXiv:2203.05222, 2022.

[4] Zou et al., Defending batch-level label inference and replacement attacks in vertical federated learning. IEEE Transactions on Big Data, 2022.

[5] Gardiner et al., Data Anonymization for Privacy-Preserving Large Language Model Fine-Tuning on Call Transcripts, Proceedings of the Workshop on Computational Approaches to Language Data Pseudonymization. 2024

[6] Xie et al., Differentially Private Synthetic Data via Foundation Model APIs 2: Text. arXiv:2403.01749, 2024.

We appreciate the reviewer's detailed feedback and address each weakness and question below.

establishing a theoretical privacy guarantee for P$^3$EFT would enhance understanding of privacy leakage and offer a more solid theoretical basis for fair comparisons with other label privacy-focused algorithms.

We appreciate the reviewer's emphasis on the importance of theoretical privacy guarantees. While we agree that such guarantees would strengthen our work, developing them for P³EFT presents unique challenges. Traditional differential privacy approaches typically rely on introducing randomness through noise addition to gradients [1] or label flipping [2], which serves as the foundation for their theoretical analysis.

In contrast, our method preserves privacy of the activations through a deterministic regularization approach. The absence of explicit randomization makes it challenging to apply standard differential privacy analysis techniques, which typically rely on comparing outcomes between neighboring datasets through probabilistic bounds.

Nevertheless, recognizing the importance of theoretical foundations, we have provided a theoretical analysis of a key component of P³EFT --- the private_backprop algorithm --- in Appendix C. While this analysis operates under slightly different assumptions, we believe it represents a meaningful first step toward building a theoretical framework for our approach.

We welcome suggestions from the reviewer on potential approaches to establishing theoretical privacy guarantees for regularization-based privacy mechanisms, as this could be a valuable direction for future research.

This paper addresses label privacy in PEFT over an API within an "honest but curious" attacker model, making it a unique but somewhat limited scenario in terms of generality. Analyzing cases where the server is not "honest" would provide a broader perspective, especially given the emphasis on worst-case privacy scenarios in the paper.

We appreciate the reviewer's suggestion about considering scenarios beyond the "honest but curious" server model. However, in our specific Split Learning setup, where the client computes the final loss using their private labels after receiving intermediate results from the server, we believe the "honest but curious" model is both practical and sufficient. This is because any deviation from honest behavior by the server would be immediately detectable by the client through observable training dynamics (e.g., unexpected increases in training loss).

Our focus on the "honest but curious" model aligns with the established literature in this domain [3,4,5], where this threat model has proven to be both realistic and meaningful for analyzing privacy concerns in split learning scenarios. To the best of our knowledge, there are no existing works analyzing "not honest" server behavior in our specific setup, likely due to the client's ability to detect such behavior through the training process. However, if the reviewer is aware of any relevant work in this direction, we would be very interested in considering it for future research.

Additionally, extending the framework to protect both input and label privacy would be a natural generalization, though it appears challenging to adapt P$^3$EFT for this purpose.

We fully agree with the reviewer that protecting inputs is also incredibly important. We plan to extend our framework to protect inputs in future work. Meanwhile, to protect both inputs and labels, P$^3$EFT can be combined with an existing input protection algorithm.

For instance, line 52 refers to private multi-party fine-tuning methods that support a narrow class of fine-tuning algorithms but does not specify which algorithms fall within this scope and which do not.

We agree with the reviewer that specifying the particular class of algorithms will improve the quality of the paper. Here we are referring to prompt-tuning, and we have clarified this in the revision.

Similarly, line 132 states that differential privacy upper bounds are loose and do not align with practical observations on real models, but it lacks a reference or supporting argument, leaving some statements unclear.

We acknowledge that statements like the one on line 132 should be better supported with appropriate references. This particular statement was primarily motivated by our empirical findings presented later in Section 4.2, but we agree that this connection should have been made more explicit in the text.

We have revised the formulation of this statement to make it clearer and better connected to our experimental results. If the reviewer has identified other parts of the paper that would benefit from similar clarification or additional references, we would greatly appreciate their suggestions.

Question 2

We thank the reviewer for their attention to detail. Indeed, there should be $\frac{1}{2}$. We have corrected the formula in the revision (line 264).

It could help to move Algorithm 2 (or a more concise version of it) into the main text from the Appendix to help make the final P$^3$EFT method clear.

We agree with the reviewer that including Algorithm 2 in the main paper would be beneficial for the coherence of the paper's presentation. We had originally omitted it due to space limitations. We have now moved Algorithm 2 along with its description from the Appendix to Section 3.4 (see lines 352-372 and 411-414).

The abstract mentions that P$^3$EFT is competitive to methods in both a multi-party and 2-party setting. As far as I can tell, the experiments in Section 4 are focused only on a 2-party setting? How does this process change or scale to a multi-party setting?

We thank the reviewer for this important clarification question. When we mentioned "multi-party" in the abstract, we were referring to scenarios discussed in Section 3.3 where either:

1. a client interacts with multiple servers at different stages of training through their APIs, or 
2. there is a single server but with multiple trusted execution environments required.

While our experiments in Section 4 focus on the basic two-party setting, our method naturally extends to these setups. However, we acknowledge that we should be more careful with the terminology, and we welcome the reviewer's suggestions on how to better characterize these scenarios.

Minor: Typo L485 algorothm → algorithm

We thank the reviewer for their attention to detail. We have corrected this typo.

[1] Hu et al. (2022), Lora: Low-rank adaptation of large language models. ICLR 2022.

[2] Wan et al. (2023), PSLF: Defending against label leakage in split learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023.

We are grateful for the reviewer's insightful feedback. We address their concerns and questions below.

Did you run any experiments varying the number of adapters and what effect did this have on utility and privacy?

We thank the reviewer for this question. We agree that understanding the effect of varying the number of adapters on utility and privacy is indeed important for better comprehension of the method's potential. To address this, we conducted additional experiments on SST2 and QNLI using DeBERTa. The results are presented in Table 8.

The results generally demonstrate, that increasing the number of adapters has minimal influence on the resulting privacy and accuracy. We have also evaluated the efficacy of our setup when utilizing a single set of adapters. Despite slightly reduced stability concerning the $\alpha$ (reguralization weight hyperparameter), this setup proved highly competitive, which opens promising direction for further research.

Further to this, how small are the adapters that are used? It would be useful to include some information about the overhead that is required for clients.

As specified in lines 504-505 for DeBERTa, we followed the setup from the original LoRA paper [1]. This setup used rank $r=8$. The same value was used for T5 (see lines 513-514). For LLaMA, as indicated in line 521, we used adapters with rank 16.

Overall, the total number of these parameters is less than 1% of all model parameters, so the overhead for clients is negligible.

To improve readability of the paper, we have added clarification about the DeBERTa adapter rank in line 505.

I think some of the Appendix results with the PSLF method could be added into the main paper

We agree with the reviewer that including PSLF results in the main paper would be beneficial for a clearer comparison between label DP approaches and P$^3$EFT. We have incorporated some of these results — specifically DeBERTa and Flan-T5 on SST2 — into the main paper in Table 4.

What does it mean in Table 6 to train with an $\varepsilon=0$ with PSLF?

While we acknowledge that from the formal definition of differential privacy, setting $\varepsilon=0$ might not be entirely rigorous (as some definitions require the privacy budget $\varepsilon$ to be a positive real number), it carries meaningful practical implications. Specifically, if we consider Randomized Response (equation (3) in the original PSLF Paper [2]), which forms the foundation of the PSLF framework, setting $\varepsilon=0$ means that for each training sample, the flipped label $\tilde{y}$ has an equal probability of belonging to any class.

Consequently, training with $\varepsilon=0$ is equivalent to training with random labels, corresponding to the setup with theoretical lower bound for privacy. We included these entries in the table to better illustrate the trend of results across the hyperparameter grid. We have added this explanation of the $\varepsilon=0$ case in Appendix B. We welcome any further questions or suggestions from the reviewer regarding this matter.

How many times are the experiments repeated over?

As stated in the main paper (line 499 of the revised version), we repeated the training procedure with 3 random seeds.

There is often a large amount of variance in the privacy leakage e.g., on QNLI in Table 1 and Table 3 which can make DC and P$^3$EFT seem quite comparable.

We acknowledge that the standard deviation for QNLI in Table 1 is indeed high. This occurred because training DeBERTa on QNLI proved to be somewhat unstable, with one of the three seeds showing a significant spike in privacy leakage at one point during training. However, regarding QNLI in Table 3, we respectfully disagree with the reviewer's assessment, as the standard deviation there is relatively small.

We would also like to note that while P$^3$EFT and DC might be comparable in some tasks, P$^3$EFT significantly surpasses DC in many cases. For instance, when training DeBERTa on MNLI using DC, we were unable to find hyperparameters that would allow the model to learn effectively without complete (~100%) privacy leakage. Similarly, when training DeBERTa on SST2 with DC, the privacy leakage exceeded 90, indicating that the adversary recovered almost all labels.

Furthermore, we would like to emphasize that, unlike DC, our P$^3$EFT method provides protection beyond just activation-based attacks. One of our work's main contributions is the private_backprop algorithm, which preserves gradient privacy. To maintain bidirectional privacy when using DC, it would need to be combined with another algorithm to prevent privacy leakage from gradients --- such as our private_backprop.

We thank the reviewer for their feedback and provide responses to their concerns hereafter.

The paper claims that backpropagation is conditionally linear in output gradients and attempts to decompose gradients to enhance privacy. However, based on my understanding, if decomposed gradients are backpropagated through earlier layers, how can this linearity be maintained?

We believe there has been a misunderstanding: the backprop API method does not update $\theta$, it only computes the gradients with respect to $\theta$ and returns them to the client (see lines 180-181). Instead, the client updates $\theta$, which is possible because $\theta$ are parameter-efficient adapters. Since parameters $\theta$ remain the same, the computational graph on the server --- which represents Jacobian --- also remains unchanged, which allows to backpropagate through it again (see lines 256-268).

The detailed sequence of gradient computation and parameter updates is described in Algorithm 2. Specifically, line 14 executes the private_backprop algorithm (during which the backprop API method is called multiple times). After the gradients are computed (and only then), the client updates the adapter set.

If $\theta$ were updated during each backprop call , linearity would indeed be violated. For this reason, we deliberately design backprop to be stateless, i.e. it merely computes gradients with respect to adapter parameters based on given gradients with respect to activations, without changing either the adapter weights or the weights of the original model.

The proposed approach requires the user to decompose gradients into separate parts and make multiple API calls. What is the associated computation and communication overhead?

As specified in Algorithm 1, a single private_backprop requires $m$ API calls, where $m$ is a hyperparameter chosen by the client. Each API call requires the same amount of computation (to compute backward) and communication (for the client to send obfuscated gradients to the server, and for the server to send gradients w.r.t. adapter weights to the client). To preserve label privacy, the value of $m$ must be at least $2$, meaning that one forward pass requires two backward passes. In this case, private_backprop introduces a computational overhead of approximately 1.5x compared to a regular training step (forward and 2 backwards instead of forward-backward).

Additionally, to my knowledge, I am not aware of any LLM companies that offer APIs for forward and backward passes in this way. Could the authors provide examples of such an application if available?

This is correct that no company specifically offers this exact interface for forward and backward. However:

1. companies do offer forward pass API (without the backward), e.g. see [1].
2. some experimental open-source API frameworks [2], offer both forward and backward pass API as described in line 154.

In summary, while we agree that companies currently typically don’t offer this exact API, we use a setup that is close to what they offer and they can, should they choose to, adjust their API to support our method efficiently — as demonstrated by the referenced libraries.

If the client is involved in parameter updates and can observe the training process, how is it ensured that the client cannot access private information during training?

We believe there has been some misunderstanding. In the setup discussed in the paper, we emphasize the privacy of labels, as indicated in lines 018-019, lines 060-061, etc. In this setup, the client is the only training participant who has access to private labels, as seen in lines 060-061, line 075, line 161, line 196, etc. Conversely, it is the server that does not have access to the labels.

If the reviewer has additional questions, we would be happy to provide clarification.

I don’t see a clear definition of what "leak" represents in the tables. My assumption is that it refers to the client’s prediction accuracy.

The notation "leak" represents a measure of privacy leakage, which is measured across 3 potential attacks that can be conducted by the server. A detailed description can be found in lines 472-485. We would also like to note that privacy leakage is not always measured in accuracy; for two main attacks --- Spectral attack and Norm attack --- following prior works, we measure classifier ROC AUC.

Additionally, to improve and facilitate the paper's readability, we will add a clear definition of what we label as "acc" and "leak" in the Tables.

If this is correct, with prediction accuracy over 60%, it doesn’t seem that label privacy is effectively preserved.

We kindly disagree with the reviewer on this matter. We would like to note that for an $n$-class classification task, a leak value of $1/n$ corresponds to a random guess. Thus, theoretically, the minimum leakage value for binary classification (as in SST-2 and QNLI) is 50%. Moreover, the pre-existing representations learned during pre-training contribute to an increase of this lower bound --- we measured this baseline in the "Without LoRAs" column. Therefore, we believe that a leakage value slightly above 60% represents a very minor degradation compared to the lower bound baseline (for instance, T5 on QNLI shows only a minimal increase from 58.7 to 63.0).

We would also like to specifically emphasize that our method offers better empirical guarantees than the baselines. Specifically:

1. As shown in Table 6, for PSLF we struggled to find a suitable hyperparameter $\varepsilon$ that would maintain both reasonable performance and privacy

2. DC performs noticeably better than PSLF but operates less stably than our P³EFT method, as evidenced in Table 1, where DC exhibits significantly higher leakage.

Can you provide some definitions to describe the metrics used to evaluate the approach?

As stated in line 472, to evaluate utility, we standardly measured accuracy on the target task. To assess privacy leakage, as described in lines 472-485, we measured vulnerability to 3 different attacks. The two main attacks --- Spectral Attack AUC and Norm Attack AUC --- were adopted from prior works [3],[4] correspondingly.

To make the description of our experimental setup more comprehensive and detailed, we will add a separate section in the appendix with a thorough description of these attacks.

[1] https://platform.openai.com/docs/guides/embeddings/

[2] https://petals.dev/

[3] Sun et al. (2022), Label leakage and protection from forward embedding in vertical federated learning. arXiv preprint arXiv:2203.01451, 2022.

[4] Li et al. (2022), Label leakage and protection in two-party split learning. In International Conference on Learning Representations, 2022

We thank the reviewer for acknowledging our response. As we've provided comprehensive answers to each weakness, we'd like to understand which specific concerns still warrant the unchanged score. We're ready to address any remaining issues.