Dear Reviewer NdAW,

We sincerely appreciate your comment. Hope our additional experiments and clarifications could address your concerns.

Q1. Privacy concerns of denoise model.

The training of denoise model relies solely on the public dataset, and thus the denoise model is irrelavant to the private data. We added the clarification at the end of Section 3.4. Furthermore, the input to the denoise model requires the knowledge of specific noise matrix, which is not accessible by the server.

Q2. The impact of knowing the noise level or not

We added theoretical (see Section 3.4) and empirical studies (see Section A.10) to assess the impact of noise level awareness. For theoretical analysis, we showed that the denoising ability is inherently in tension with the level of privacy protection and provided a lower bound on the MSE of denoised results. For empirical analysis, we run the experiment that deploys the denoise model on server side using only privatized token representations as input. The results show that SnD significantly outperforms the server-side denoise method by over 10% in almost all cases for MRPC task.

Q3. Empirical evaluation of privacy protection & different $\eta$s

Refering to general response to Q1, we added experiment considering the inference attacks in Section 4.3 and 4.4.3. We found that the three classes of models show different robustness exhibit distinct robustness against inference attacks, and thus the privacy budget varies across the models. The $\eta$ is selected that meets the following two criteria simultaneously: (a) the accuracy for embedding inversion attack is below 0.01, and (b) the accuracy for attribute inference is below 0.53 (the accuracy for random guessing is around 0.5).

Regarding the interpretation of $\eta$, it can be viewed as a parameter controlling the indistinguity between the distribution of the randomized mechnisms with two possible inputs $x$, $x'$. The log-likelihood ratio of observing any particular output $y$ given two inputs $x$, $x'$ is bounded by $\eta d(x, x')$, where $d(x, x')$ is the distance metric of the two inputs. In this paper, we adopted the Euclidean norm as the distance metric.

Q4. Typo errors and expression issues

Thanks for pointing out. We corrected the typos and used consistent $\eta$ in the paper.

Q5. Extension to training/finetuning

That’s a good question. While the privatization method described in our paper can be utilized during the training stage, directly applying the denoise model to the training/finetuning stage would pose a significant challenge. It would be another field of research to protect LLM training with local differential privacy while maintaining utility.

Q6. Limitations of $d_{\chi}$-differential privacy

The method proposed by "Mattern et al. - The Limits of Word Level Differential Privacy" couldn’t be directly applied to our framework, as it involves repharasing the sentence and thus is not compatible with the denoise model. Indeed, $d_{\chi}$-DP has been used frequently for text inputs in recent research [1]. Below we present some discussions on the weakness of the $d_{\chi}$-DP bound:

• Linear growth of privacy budget with token length. In the experiment, we limited the input length within 512 tokens. We also conducted inference attack on the privatized token representations to verify the privacy protection capability. Furthermore, in Section A.11, we show that our approach retains the performance even under extremely low privacy budgets, indicating that the user could set a very low privacy budge in case of possible leakage for long text inputs.
• Lack of syntactic changes. The syntactic properties of the original text could be hided when the privacy budget is set to a very low level.
• Limitation for usefulness of the text data on local DP. To preserve the utility of the subsequent output, we deploy a denoise model on user side to conduct error correction.

[1] Feyisetan, O., Balle, B., Drake, T., & Diethe, T. (2020, January). Privacy-and utility-preserving textual analysis via calibrated multivariate perturbations. In Proceedings of the 13th international conference on web search and data mining (pp. 178-186).

Dear Reviewer NdAW,

Thanks a lot for acknowledging and valuing our efforts. We sincerely appreciate your support in our work. We have updated the expression in Section A.10 and underscored the server's inability to infer private information.

Best regards,

Authors of Submission 5188

Dear Reviewer C4Af,

Thanks for your comments. Hope our modifications and answers below could address your concerns.

Q1. The overhead of proposed approach

We added experiment in Appendix A.9 to evaluate overhead of SnD. Refer to the general response to Q3.

Q2. Denoise model for varying $\eta$ levels

In Appendix A.6 we added the specification of training the denoise model. Simply speaking, we devide the $\eta$ in to three groups according to the correlation between the privatized and plain token representation tokens, and train a seperate denoise model for each group. During inference, the user could choose the corresponding denoise model based on their privacy requirement.

Q3. Baseline description & lack of baseline

Refering to the answer to Q2 in general response, we added two baselines in the experiments. We also present more illustration of the baseline methods in “prior works” and Appendix A.1. The accuracy performance for a vanilla model is presented in the case of $\eta=\infty$ in table 1, 2, and 3, where no noises are added and denoise model is not deployed on user side.

Q4. Privacy performance of the proposed method

Refering to general response to Q1, we added experiment considering the inference attacks in Section 4.3 and 4.4.3.

Q5. Interpretation of Cosine Similarity

The cosine similarity measures the similarity between the clean and denoised embeddings in the case of SnD (or privatized embedding in the case of text2text and tokEmbPriv). Clean embedding refers to the output embedding computed from the plain token embeddings, and denoised embedding refers to the output embedding derived from the privatized token embeddings and transformed by the denoise model. A higher cosine similarity means that the denoised embedding is more similar to the original clean embeddings and thus is prefered. We make the expression clearer in Section A.8.

Dear Reviewer C4Af,

As the discussion period comes to a conclusion, we are eager to understand if our response and revision have addressed your concerns. Your further insights would be valuable as we endeavor to enhance our submission in these final days. We sincerely appreciate your time and guidance.

Best Regards,

Authors of Submission 5188

Dear Reviewer C4Af,

In the latest version, we have added the description and implementation details for the baseline methods in Section A.6.5. As the deadline of ICLR rebuttal period is approaching, we look forward to hearing from you and would be pleased to address any remaining concerns that you may still have.

Best Regards,

Authors of Submission 5188

Dear Reviewer xFRn,

Thanks for your suggestions. We have enhanced our paper based on your comment, and hope that will address your concerns.

Q1. Testing of GPT2 on classification task

The autoregressive GPT model has been evaluated on classification tasks in recent research [1]. Furthermore, OpenAI has released the Embedding-as-a-Service (EaaS) for downstream tasks including classification [2], where GPT2 and 3 are provided as the underlying models. Therefore, it’s of pratical concerns to test the performance of GPT2 on the downstream tasks given by the paper. Meanwhile, we are conducting experiment of GPT2 on HellaSwag and lambada dataset, but we haven't completed it due to computer resource and time limit. We will add the results to the paper once the experiments are done.

Q2. Comparison to other baselines

We added the comparison of SnD with more privacy-preserving LLM inference methods. In Section 4.1 and 4.4.2, we compare the auc of SnD on downstream tasks with text2text and RAPT. Benefiting from the denoise model, our method results in AUC higher than the three baselines with an average over 10%.

In Section A.9, we evaluated the computation and communication overhead for two encryption-based method [3] [4], and found that our method results in thousands of speedup for the SOTA HE algorithms. The privacy comparison between PCFT and SnD can be found in Section A.1, where PCFT adopts the security definition in multiparty computation while SnD protect the data with LDP.

Q3. Difference between RAPT and SnD

In Section A.1, we illustrated the distinction between RAPT and our framework. Though RAPT employs a reconstruction head to improve the robustness against noises during prompt-tuning stage, the module only reconstruct the random tokens and doesn’t perform any direct denoise either on the privatized input or the corresponding representations/activations for the private token. Furthermore, the reconstruction head is discarded during the inference stage, rendering no denoise mechanism for LLM inference. In constrast, our work performs explicit local denoise for the output embedding on the user-side, which demonstrates better performance.

Q4. Privacy-preserving capabilities

Refer to the general response of Q1, we added experiment considering the inference attacks in Section 4.3 and 4.4.3. Such attack could be carried out by an eavesdropper or the curious server.

Q5. Privacy concerns for denoising layer

The training of denoise model relies only on the public dataset, and thus the denoise layer release no information about the private data. We added the clarification at the end of Section 3.4.

Q6. Inaccurate claims

We rewrited the claims and highlighted them in blue.

[1] Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training.

[2] https://platform.openai.com/docs/guides/embeddings/what-are-embeddings

[3] Liu, X., & Liu, Z. (2023). LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformers. arXiv preprint arXiv:2305.18396.

[4] Hao, M., Li, H., Chen, H., Xing, P., Xu, G., & Zhang, T. (2022). Iron: Private inference on transformers. Advances in Neural Information Processing Systems, 35, 15718-15731.

Dear Reviewer xFRn,

As the discussion period comes to a conclusion, we are eager to understand if our response and revision have addressed your concerns. Your further insights would be valuable as we endeavor to enhance our submission in these final days. We sincerely appreciate your time and guidance.

Best Regards,

Authors of Submission 5188

Dear Reviewer xFRn,

Thanks for your reply. I have uploaded a revised version of paper based on your suggestion. Hope the revision and answers below could address your concern.

Q1. Testing of GPT2 on classification task.

Our EaaS model works similar to OpenAI's solution. Instead of finetuning GPT2 on the classification task, we employ the LLM model to extract embeddings and use the embeddings as the feature score to train a separate classification model on the downstream task. For more details refer to A.5. We also mention this point in Table 7 of Section A.1.

Q2. Implementation details on how competitors were run

Thanks for the suggestion. We have added the implementation details in section A.6.1 and A.6.5. Noted that all DP methods utilizes the same $d_{\chi}$-privacy with Euclidean distance as metrics.

Q3. RAPT's reconstruction head

We understand that the reconstruction head help the model to better understand the privatized input. Since the reconstruction module works solely on decoding random tokens, we note that the precise mechanism by which the LLM learns to decode the privatized tokens remains unclear. We updated the expression to be more precise in Section A.1.

Q4. Discussion on the limitations of TokEmbPriv

Thanks for the suggestion. We included the discussions on the limitation of TokEmbPriv and Text2Text in Section 4.4.2. The lack of denoise mechanism and unbounded noise support in TokenEmbPriv might lead to its poorer performance in downstream task.

Dear Reviewer 4SyG,

Thanks for your comments. Hope the following response and our additional experiments could address your concerns.

Q1. Evidence to support the practicality of the proposed method.

We added experiment in Appendix A.9 to compare the computation and memory cost of user-side inference of SnD with the case where the whole model is downloaded. The computation time and memory cost saves by SnD ranges from, respectively, 78% to 96% and 65% to 94%, across the 10 models. The saving is more significant for models of large size, such as GPT2 Xlarge.

Q2. Evidence for the privacy benefits & consistence of privacy budget.

We added embedding inversion attack and attribute inference attacks to verify the privacy protection cabilities under different $\eta$s in Section 4.4.3 and 4.3. We found that the three classes of models show different robustness exhibit distinct robustness against inference attacks, and thus the privacy budget varies across the models. The $\eta$ is selected that meets the following two criteria simultaneously: (a) the accuracy for embedding inversion attack is below 0.01, and (b) the accuracy for attribute inference is below 0.53 (the accuracy for random guessing is around 0.5). Noted that we have updated the privacy budget for GPT2 to be 1, 50, 100. We also added evaluation for the performance of extremely low privacy level in Appendix A.11.

Q3. Shallow discussion of results.

We enhance the discussion in the Section 4. Compared with the vanilla model, Bert, T5, and GPT models yield average model losses of 4.31%, 4.48%, and 5.25%, respectively. In addition, SnD outperforms the baselines by an average of over 10% benefiting from the denoise model. More details are presented in our paper.

Q4. Typos

We have corrected the typos and conducted grammer check on the paper.

Dear Reviewer 4SyG,

As the discussion period comes to a conclusion, we are eager to understand if our response and revision have addressed your concerns. Your further insights would be valuable as we endeavor to enhance our submission in these final days. We sincerely appreciate your time and guidance.

Best Regards,

Authors of Submission 5188

Dear Reviewer 4SyG,

Thanks for your reply. We uploaded a revised version based on your suggestions. Hope our further clarification and revisions could address your concern.

Q1. How do you measure the computation costs and what is its unit in Table 12?

The computation costs are in second, which is indicated by the caption of Table 12.

Q2. Communication costs of encryption-based methods

The high communication cost is caused by the multiparty computation techniques adopted by the methods. Instead of only uploading the encrypted data to the server, the client and server transmit the secret shares as well as encrypted versions of intermediate values obtained from a series operations in the LLM. For more details refer to the secure computation protocols in [1] and [2]. I included this point in A.9.

Q3. Figure 5 confirms my concern that the computation costs of your method is high and very close to the computation costs of running the whole LLM locally on the user's device.

We are afraid that there might be some misunderstanding. It's important to note that our results reveal significant computation benifit of SnD compared with whole LLM inference. According to the Figure, SnD saves the computation cost by an average of over 85%, and in particular 95.3% for GPT2 Xlarge, compared with running the whole LLM locally. We also added more explanations in the caption of figure 5 to make it clearer.

Q4. The y-axis is in seconds I would not show it with the steps size of $10^{-2}$ to not be misleading please.

Could you make further illustration on this suggestions? Seems like there are some confusions on our side. We will strive to make the figure clearer based on your advice.

Q5. The description of your embedding inversion attack is missing

The description of embedding inversion attack is presented in Section 4.3. The server recover the tokens from the privatized token embedding based on the L_2 distance.

Q6. How strong is your attack? Is it an existing SOTA attack or is it something that you designed.

The attacks have been adopted in existing research to verify the effectiveness of SOTA methods [3][4]. We added the citation after each attack in Section 4.3.

Q7. Figure 2 misses labels for its x-axis and y-axis.

Thanks for pointing out. We added the labels in the latest version. The x-axis denotes the privacy budget and y-axis denotes attack accuracy.

Q8. It is not clear if the attack is performed on the embedding obtained by your method or not?

The attack is performed on the privatized token embedding obtained by our method. We included this point in Section 4.3.

Q9. Figure 2 misses a discussion.

The discussion for Figure 2 is presented in Section 4.4.3.

[1] Liu, X., & Liu, Z. (2023). LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformers. arXiv preprint arXiv:2305.18396.

[2] Hao, M., Li, H., Chen, H., Xing, P., Xu, G., & Zhang, T. (2022). Iron: Private inference on transformers. Advances in Neural Information Processing Systems, 35, 15718-15731.

[3] Li, Y., Tan, Z., & Liu, Y. (2023). Privacy-preserving prompt tuning for large language model services. arXiv preprint arXiv:2305.06212.

[4] Qu, C., Kong, W., Yang, L., Zhang, M., Bendersky, M., & Najork, M. (2021, October). Natural language understanding with privacy-preserving bert. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (pp. 1488-1497).