Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utility

Dear Reviewer,

We sincerely thank you for your valuable feedback and thoughtful acknowledgment of our work. Below, we present our responses to your comments and questions.

More evaluation on different PII.

We conducted an additional experiment to evaluate the protection of 281 users' email addresses in the aeslc training dataset. Using Levenshtein distance [1], we compared the predicted email addresses to the ground truth. As shown in the table, PPA successfully defends all users' email addresses against probing attacks while maintaining model performance comparable to other baseline defense methods. Detailed results can be found in Appendix K.

Discuss optimization strategies for PPA.

Thank you for raising this concern. Our initial research was conducted under the assumption of abundant resources, providing a controlled environment to validate our approach. We appreciate your suggestion to combine scalability and optimization strategies for PPA, which will be invaluable for future work.

We have also conducted an initial investigation into the scalability and optimization strategies for PPA. Our experiments involved combining PPA with efficient fine-tuning techniques, such as LoRA [2], using a rank of 16 and an alpha value of 32. As shown in the table below, applying LoRA to PPA produced promising results: after fine-tuning for three epochs, the risk score reduced to 1.0, and after four epochs, it further decreased to 0.0, all while maintaining comparable model performance. Although PPA with LoRA required four epochs, compared to just one epoch for full fine-tuning of PPA, it achieved the same defensive effectiveness.

These results demonstrate that PPA has potential for scalability. Furthermore, exploring additional optimization strategies could be a valuable direction for future work. We sincerely thank you for this valuable suggestion. Detailed results can be found in Appendix M.

Enhances the paper's presentation.

Thank you very much for your feedback. We will carefully consider your suggestions and incorporate them into our paper. We will also make the necessary adjustments to the figures.

Address Risk Score Table meaning.

Appreciate your query. We designed Table 1 to assess the relative risk of exposing users' physical addresses by category. For example, we consider a street address to be 60 times more sensitive than a country. Specifically, the risk value for a country is 0.005, while for a street it is 0.3. The ratio of street to country risk is calculated as 0.3 / 0.005 = 60. The Address Risk Score Table helps identify the exposure risk associated with users' physical addresses.

Reference:

[1] Po, Daw Khin. "Similarity based information retrieval using Levenshtein distance algorithm." Int. J. Adv. Sci. Res. Eng 6.04 (2020): 06-10.

[2] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).

Thank you once again for your valuable feedback and continued support for our work! We sincerely appreciate these constructive and meaningful discussions, as they contribute to making our work more thorough and robust. Should you have any further questions, we would be more than happy to address and discuss them!

Sincerely

Authors

Dear Reviewer,

We greatly value your thoughtful feedback and acknowledgment of different aspects of our work. We are thankful to have the chance to address your questions and concerns.

Novelty

Firstly, we thank you for your suggestions. We acknowledge that novelty is not our primary contribution; our approach is simple and practical, aimed at prioritizing more stable and reliable implementations for privacy protection. We are grateful for your suggestions to improve our memory implanting technique, and have made some attempts, such as integrated selective analysis into the memory implanting process. In a supplementary experiment, we modified the memory implanting component to focus on replacing the key element with a different token. For instance, in the example 'John Griffith's phone number is (713) 853-6247,' where the key element is '8', we selectively forgot '8' and replaced it with a different number at its position.

As shown in the table, we observe that the Modified Memory Implanting PPA provides the same protection for users' phone numbers and outperforms PPA in GPT-4o EmailScore by approximately 9.6%.

Address substitution presents challenges because addresses are highly contextually dependent. Replacing a key element in an address with an arbitrary token can impair the model’s understanding of the context. For example, substituting '_Su' in 'Jeffrey Dasovich address 101 California St. Suite 1950' disrupts the model’s comprehension of the address structure. Additionally, partial substitution may inadvertently expose parts of the user's address. Discussing how to customize selective analysis and memory implanting for different types of PII is a pertinent issue. Your suggestion to design memory implanting to optimize performance for various PII types is valuable. Given that our current priority is to ensure stable and reliable privacy protection, we continue to opt for PPA. We will prioritize your suggestions as key components of our future work. Thank you again for your invaluable feedback. Detailed results can be found in Appendix I.

Adding the Exposure metric, as discussed in [1]

Thank you for your suggestion. The exposure metric [1] is an effective method for detecting whether an model has memorized PII. We applied the exposure metric across all baseline methods for our evaluation. We calculate the exposure metric using all baseline methods. Calculating the exposure of PII is time-consuming; therefore, referencing Table 2 as outlined in The Secret Sharer [1], we calculated the exposure for 10 phone numbers. The average exposure across these numbers is presented in the table below.

As the table illustrates, PPA outperforms other baseline defense methods. Detailed results can be found in Appendix J.

Comparison with differential privacy-based methods

Thank you for your insightful feedback. We implemented Just Fine-Tune Twice (JFT) [2] and Differentially Private Decoding (DP Decoding) [3]. To evaluate these methods, we conducted probing attacks on both JFT and DP Decoding. Specifically, for DP Decoding, we tested various values of the lambda parameter ranging from 0.1 to 0.9 and selected a result where the utility score was worse than that of PPA. We observed that our PPA method outperformed both DP Decoding and JFT, achieving a lower risk score and a higher utility score. This superior performance can be attributed to the fact that DP Decoding applies a uniform distribution adjustment to next-token predictions, which lacks the necessary customization for scenarios involving PII. Detailed results and discussion can be found in Appendix E.

Reference:

[1] Carlini, Nicholas, et al. "The secret sharer: Evaluating and testing unintended memorization in neural networks." 28th USENIX security symposium (USENIX security 19). 2019.

[2] Shi, Weiyan, et al. "Just Fine-tune Twice: Selective Differential Privacy for Large Language Models." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

[3] Majmudar, Jimit, et al. "Differentially private decoding in large language models." arXiv preprint arXiv:2205.13621 (2022).

Thank you for your responses. Could you please provide more details about the parameter selection for the added DP experiment and explain why it is reasonable and comparable to the proposed method? The current description: "we tested various values of the lambda parameter ranging from 0.1 to 0.9 and selected a result where the utility score was worse than that of PPA" is a bit vague. Including this detailed information in the paper would also be beneficial.

Thank you for your thoughtful questions. Because DP Decoding [1] aims to protect users' PII in an already trained model, we added DP Decoding [1] into our experiment. DP Decoding employs a computationally lightweight perturbation mechanism at the decoding stage. The core principle of this method is illustrated by the equation q' = λq + (1 − λ)u, where q' represents the perturbed distribution—a linear interpolation between the original distribution q and the uniform distribution u. The parameter λ impacts the utility (the effectiveness of the generated output) and privacy (the protection of sensitive information). As λ varies from 0 to 1, a value closer to 0 enhances privacy but reduces utility, whereas a value closer to 1 improves utility at the expense of privacy. Therefore, we selected results for DP Decoding where λ ranged from (0.1, 0.3, 0.5, 0.7, 0.9), as illustrated in the table below.

In the table, we observe a trend across λ values of 0.1, 0.3, 0.5, 0.7, to 0.9, showing a progression from low to high risk scores and from low to high utility. We selected λ=0.3 as it achieved the best balance between utility and privacy protection among the tested values, offering better utility compared to λ=0.1, and improved defense capability compared to λ=0.5.

These results demonstrated that the utility score was lower than that of PPA and that PPA's defense capability was superior to that of DP Decoding, thereby illustrating that PPA outperforms DP Decoding. Detailed results can be found in Appendix E. (We apologize for the typo in our previous results. According to the rounding rules, λ=0.3 should be 4.8, not 4.7.)

Thank you for your time and effort. We hope our responses address your concerns. If you have any further questions, we will do our best to provide clarification. We will also incorporate all your suggestions and our revisions into the paper. Thank you very much for helping make our paper more complete.

Reference:

[1] Majmudar, Jimit, et al. "Differentially private decoding in large language models." arXiv preprint arXiv:2205.13621 (2022).

Dear Reviewer,

We deeply appreciate your insightful feedback and recognition of various aspects of our work. Our responses to your comments and questions are provided below.

Additional utility metrics.

Thank you for your valuable feedback. We provide some new evaluations on the model's performance metrics on both MMLU [1] and TruthfulQA [2]. As our research primarily focuses on the text generation capabilities of models, we had the models that have been protected by various defense methods respond to the MMLU and TruthfulQA questions directly. GPT-4o was then employed to rate these responses on a scale from 1 to 5, where 5 represents the best possible score and 1 the worst. Given the extensive volume of the MMLU dataset, and in order to manage computational costs efficiently, We selected 20 data points from each subtask to form a representative subset, totaling 1,140 data points. For each defense method, we calculate the mean score for comparative analysis between defense methods. We found that PPA achieves the highest MMLU and TruthfulQA score among all baseline defense methods. Detailed results can be found in Appendix F.

More detailed analysis between PPA and DEPN.

We truly appreciate your insightful feedback. Table 4 is based on the Fraud Email dataset, where the physical addresses are simpler compared to those in the Enron Email dataset. For example, an address in the Fraud Email dataset, "Brian Smith address London", whereas in the Enron Email dataset, "Jeffrey Dasovich address 101 California St. Suite 1950". This comparison highlights that PPA is better equipped to handle more complex physical addresses, demonstrating that PPA is a more generalizable method compared to DEPN.

How to verify that LLMs have learned PII.

The unlearning of specific PII is reasonable if the PII was indeed learned by the LLM. In fact, our ability to successfully attack the model indicates that the LLM has learned the PII. For example, our attacks achieving an exact match score demonstrate that the LLM has fully memorized it. Specifically, we found that with no defense on phone numbers, the exact match score reached 56.0.

More evaluation on different PII.

We conducted an additional experiment to evaluate the protection of 281 users' email addresses in the aeslc training dataset. Using Levenshtein distance [3], we compared the predicted email addresses to the ground truth. As shown in the table, PPA successfully defends all users' email addresses against probing attacks while maintaining model performance comparable to other baseline defense methods. These results demonstrate that PPA effectively safeguards users' PII. Detailed results can be found in Appendix K.

Reference:

[1] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).

[2] Lin, Stephanie, Jacob Hilton, and Owain Evans. "Truthfulqa: Measuring how models mimic human falsehoods." arXiv preprint arXiv:2109.07958 (2021).

[3] Po, Daw Khin. "Similarity based information retrieval using Levenshtein distance algorithm." Int. J. Adv. Sci. Res. Eng 6.04 (2020): 06-10.

Thank you once again for your insightful feedback and support for our work! We deeply appreciate these meaningful and constructive discussions, as they help make our work more thorough and robust. If you have any additional questions, we would be delighted to discuss and address them!

Sincerely,

Authors

Explain the relationship between Sensitivity Analysis and Unlearning + Memory Implanting

We sincerely appreciate your insightful feedback, and you are correct. The sensitivity analysis is expected to lessen the impact on the utility of the LLM. As shown in Table 6, the combination of unlearning and memory implanting results in poorer performance compared to PPA. However, it also demonstrates better defense capabilities.

The reason why phone numbers deviate from expectations, as shown in Table 5, is due to their simple format, such as (xxx) xxx-xxxx. Consequently, memory implanting more effectively restores utility, leading to the observed similarity in performance between Unlearning + Memory Implanting and Proactive Privacy Amnesia. Additionally, due to the simplicity of the phone number format, Sensitivity Analysis proves sufficient for defense.

Average proportion of the key element relative to the total length of the PII.

The proportions of phone numbers and physical addresses are 6.7% and 27.6%, respectively. Detailed results can be found in Appendix H.

We sincerely thank you for your thoughtful insights and support for our work! Engaging in discussions about these constructive and meaningful questions has been truly rewarding, as it helps make our work more comprehensive and robust. If you have any additional questions, we would be delighted to discuss and address them!

Best regards,

Authors

Dear Reviewer,

We sincerely appreciate your constructive feedback and recognition of various aspects of our work. Below, we provide responses to your comments and questions.

Compare with additional methods

Thank you for your insightful feedback. We implemented the Differentially Private Decoding (DP Decoding) in [1] and the Just Fine-Tune Twice (JFT) method in [3], building on the framework introduced in [2]. To evaluate these methods, we conducted probing attacks on both DP Decoding and JFT. Specifically, for DP Decoding, we tested various values of the lambda parameter ranging from 0.1 to 0.9 and selected the result that achieved the best balance between utility and privacy protection. We observed that our PPA method still outperformed both DP Decoding and JFT, achieving a lower risk score and a higher utility score. This superior performance can be attributed to the fact that DP Decoding applies a uniform distribution adjustment to next-token predictions, which lacks the necessary customization for scenarios involving PII. Detailed results can be found in Appendix E.

More evaluation on model utility.

Thank you for your valuable feedback. We provide some new evaluations on the model's performance metrics on both MMLU [4] and TruthfulQA [5]. As our research primarily focuses on the text generation capabilities of models, we had the models that have been protected by various defense methods respond to the MMLU and TruthfulQA questions directly. GPT-4o was then employed to rate these responses on a scale from 1 to 5, where 5 represents the best possible score and 1 the worst. Given the extensive volume of the MMLU dataset, and in order to manage computational costs efficiently, We selected 20 data points from each subtask to form a representative subset, totaling 1,140 data points. For each defense method, we calculate the mean score for comparative analysis between defense methods. We found that PPA achieves the highest MMLU and TruthfulQA score among all baseline defense methods. Detailed results can be found in Appendix F.

In our paper, we evaluate the model's utility not only on the same unlearning dataset but also using an email completion metric, which assesses the model's performance on additional truncated emails, with GPT-4o judging the completion email scores.

Assume Attacker has prior knowledge of the PII

Appreciate your query. We have implemented a more advanced attack scenario where the attacker possesses prior knowledge of the PII. This is, in fact, a more challenging scenario, and we are more than willing to explore it further. Specifically, we assume the attacker knows the information from the beginning of the PII up to a key element. For instance, in the case of "John Griffith phone number (713) 853-6247," the key element is "8". In this scenario, the attacker’s prompt would resemble: "The phone number of John Griffith is (713) 8".

As shown in the table, we observe that PPA achieves the best balance between defense capability and model performance. Detailed results can be found in Appendix G.

Reference:

[1] Majmudar, J., Dupuy, C., Peris, C., Smaili, S., Gupta, R., & Zemel, R. (2022). Differentially private decoding in large language models. arXiv preprint arXiv:2205.13621.

[2] Li, Xuechen, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. "Large Language Models Can Be Strong Differentially Private Learners." In International Conference on Learning Representations.

[3] Shi, W., Shea, R., Chen, S., Zhang, C., Jia, R., & Yu, Z. (2022). Just fine-tune twice: Selective differential privacy for large language models. arXiv preprint arXiv:2204.07667.

[4] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).

[5] Lin, Stephanie, Jacob Hilton, and Owain Evans. "Truthfulqa: Measuring how models mimic human falsehoods." arXiv preprint arXiv:2109.07958 (2021).

I appreciate the authors' significant effort during the rebuttal. I still have one question for the sensitivity analysis:

Furthermore, would unlearning the latter part, i.e., the tokens after the top-k, lead to better performance?

I'm curious if there's any difference between the cases: 1) unlearn only the top-k tokens; 2) unlearn some middle parts that span the k-th token; 3) unlearn some tokens from the k-the token. Why do the authors choose (1) in the paper?