A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage

”Regarding the concept of privacy”

Whether converting "23" to "early 20s" constitutes a privacy breach depends on context and potential harm (Shao et al., 2024). Our framework doesn't make this normative judgment - instead, it measures information persistence after sanitization. We find concerning pattern preservation even with aggressive sanitization: medical records maintain linked combinations of symptoms, age ranges, and conditions (Zhang et al., 2024), while chat data preserves writing styles and topic preferences. These persistent patterns enable re-identification through modern machine learning techniques.

Our results reveal fundamental limitations in current text privacy approaches, demonstrating the need for more sophisticated protection mechanisms that consider semantic-level information leakage. This is particularly crucial as organizations increasingly handle sensitive text data across healthcare, customer service, and other privacy-critical domains, and as private user data holds the key to unlocking new model capabilities.

Reference

Zhang, Z., Jia, M., Lee, H. P., Yao, B., Das, S., Lerner, A., ... & Li, T. (2024, May). “It's a Fair Game”, or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-26).

Shao, Y., Li, T., Shi, W., Liu, Y., & Yang, D. (2024) PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

We thank the reviewer for the helpful feedback and highlighting our strengths, including the comprehensive benchmarking of sanitization methods, straightforward pipeline, comprehensive experiments and ablations, and easy to follow write-up. We hope the explanations below address the reviewer’s remaining concerns and questions.

”My major concern is that the auxiliary data is highly contrived.”

Thank you for raising the concern about the lexical overlaps between the auxiliary data and original records. We agree that in realistic applications, auxiliary information can come in more nuanced or complex formats. However, in our experiments, we use the same auxiliary information to highlight differences between existing lexical-based and proposed semantic-based privacy metrics, showing that our semantic-based metric uncovers more leakage. However, if the auxiliary information is more nuanced—e.g. semantically similar to the original atoms—it becomes even harder for privacy metrics to detect, further demonstrating our point of these lexical methods providing a “false sense of privacy.”

While our auxiliary setup may appear contrived, privacy guarantees must account for worst-case scenarios (Dwork & Roth, 2014). Real-world privacy breaches like the Netflix Prize de-anonymization (Narayanan & Shmatikov, 2008) demonstrate how seemingly innocuous auxiliary information enables re-identification. Our ablation studies validate framework robustness across varying information settings: MedQA shows linking rates of 58-78% for LLM-based sanitization and 81-94% for PII removal, while WildChat maintains consistent rates of 56-62% across methods. This variation in success rates indicates our framework captures meaningful privacy risks rather than artificially inflated matches.

Reference

Narayanan, Arvind, and Vitaly Shmatikov. "Robust de-anonymization of large sparse datasets." In 2008 IEEE Symposium on Security and Privacy (sp 2008), pp. 111-125. IEEE, 2008.

Dwork, Cynthia, and Aaron Roth. "The algorithmic foundations of differential privacy." Foundations and Trends® in Theoretical Computer Science 9, no. 3–4 (2014): 211-407.

”For the claim that 'private information can persist in sanitized records at a semantic level,”

While differential privacy provides formal guarantees through $\epsilon$, its practical implications for language models remain unclear (Habernal, 2022). Different $\epsilon$ values have ambiguous meaning for text privacy - our work provides empirical quantification of these guarantees. With $\epsilon$=1024, we observe improved privacy scores (0.92 from 0.43) but significant degradation in both task performance (0.62 to 0.40) and text coherence (3.44 to 2.25). This aligns with recent findings showing that DP's theoretical guarantees may not translate directly to meaningful privacy protection in high-dimensional text data (Brown et al. 2022).

Reference

Brown, H., Lee, K., Mireshghallah, F., Shokri, R., & Tramèr, F. (2022, June). What does it mean for a language model to preserve privacy?. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency (pp. 2280-2292).

Habernal, Ivan. “When Differential Privacy Meets NLP: The Devil Is in the Detail.” In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 1522–28. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021. https://doi.org/10.18653/v1/2021.emnlp-main.114.

”It is not likely to scale this framework for a large number of overlapped atoms.”

Our framework demonstrates practical effectiveness using just 3 claims for meaningful privacy evaluation, contradicting concerns about scalability. This efficiency stems from our novel semantic matching approach (detailed in Section 3.2) which captures information leakage without requiring exhaustive claim combinations. The framework adapts naturally across domains - medical records separate into symptoms, history, and demographics (average 15.6 claims/document), while conversational data follows dialogue structure and topic boundaries (Wang et al., 2023).

Reference

Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., ... & Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.

“The numerical results rely on LLM output …”

We conducted thorough human evaluation showing strong agreement (0.93 Spearman Correlation) between annotators and LLM judgments, validating our approach. LLM-based evaluation is increasingly accepted in the research community (Chiang & Lee, 2023; Zheng et al., 2023), with recent work using similar approaches for code similarity assessment (Chon et al., 2024), text generation evaluation (Wang et al., 2023), and information extraction validation (Hsu et al., 2024). Our choice to query LLaMA three times is supported empirically by a range of prior works on self-consistency of LLM prompting (Wang et al., 2022). There is significant instruction-following inconsistency with single queries (where the agreement drops to 0.84 Spearman Correlation).

We would be happy to clarify any of these points further or provide additional details about specific aspects of our methodology.

We appreciate your detailed review and constructive feedback. We thank you for highlighting that we have an extensive set of experiments as well as our methodology. We hope the following response addresses your concerns.

“identifiers should be the privacy focus, and privacy leakage should refer specifically to identifier leakage.”

While identifier leakage is a necessary component for measuring privacy leakage, measuring it alone is not sufficient to ensure privacy. Modern privacy threats increasingly leverage semantic patterns and quasi-identifiers (Ganta et al., 2008). Moreover, real-world privacy breaches like the Netflix Prize de-anonymization (Narayanan & Shmatikov, 2008) demonstrate how de-identified information, which has no identifier leakage, enables the breach of privacy. It is therefore important to go beyond identifier leakage for a proper measurement of privacy.

Furthermore, we agree with the reviewer that the semantic information is heavily tied to the utility of the record; however, there is a long-standing tradeoff between privacy and utility, which is complicated by the fact that privacy is inherently context-dependent (Nissenbaum, 2004, Shao et al., 2024). Our work does not attempt to make normative judgments about what constitutes a privacy violation - rather, we provide a quantitative framework for measuring information persistence after sanitization. We aim to help disentangle the complex relationship between privacy and utility by providing a framework to measure and better understand these trade-offs. Our broader view of privacy is especially critical given the unprecedented scale and intimacy of user-LLM interactions.

References

Nissenbaum, H. (2004). Privacy as contextual integrity. Wash. L. Rev., 79, 119.

Shao, Y., Li, T., Shi, W., Liu, Y., & Yang, D. (2024) PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Narayanan, Arvind, and Vitaly Shmatikov. "Robust de-anonymization of large sparse datasets." In 2008 IEEE Symposium on Security and Privacy (sp 2008), pp. 111-125. IEEE, 2008.

“The technical novelty is relatively limited …”

“The findings are not particularly interesting…”

Our work addresses a critical gap: while privacy limitations are documented for structured data (Stadler et al., 2022), text data presents unique challenges that current methods fail to address. Major cloud providers and healthcare organizations continue to rely on simple PII detection and de-identification (Johnson et al., 2020), following outdated privacy models (Garcia et al., 2019). Our results quantify this failure - showing 94% information leakage with state-of-the-art PII removal methods - and provide compelling evidence that current approaches are fundamentally inadequate. We argue that our findings of textual sanitization methods falsely preserve privacy uncovers a fundamental issue beyond merely applying it to text data.

The urgency of this work is amplified by the scale of personal data sharing with LLMs (over 200M monthly ChatGPT users) and users' demonstrated tendency to share more intimate details with AI systems than human interlocutors (Zhang et al., 2024). This combination of increased disclosure and inadequate protection mechanisms creates significant privacy vulnerabilities that practitioners can no longer ignore.

References

Johnson, A. E., Bulgarelli, L., & Pollard, T. J. (2020, April). Deidentification of free-text medical records using pre-trained bidirectional transformers. In Proceedings of the ACM Conference on Health, Inference, and Learning (pp. 214-221).

Garcia, D. (2019). Privacy beyond the individual. Nature human behaviour, 3(2), 112-113.

Zhang, Z., Jia, M., Lee, H. P., Yao, B., Das, S., Lerner, A., ... & Li, T. (2024, May). “It's a Fair Game”, or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-26).

Stadler, Theresa, Bristena Oprisanu, and Carmela Troncoso. “Synthetic Data -- Anonymisation Groundhog Day.” arXiv, January 24, 2022. http://arxiv.org/abs/2011.07018.

“Did you find major differences between the two datasets in terms of atomizing claims in documents? It seems to me that this would be more structured in a medical Q&A dataset, as compared to LLM-user interactions.”

Thank you for this insightful question about dataset differences. We'll address this by examining three key aspects: (1) the structural patterns we found in each dataset type, (2) how these differences affected sanitization effectiveness, and (3) the practical implications for privacy protection.

(1) Structural Differences in Claims: In MedQA, we found highly structured patterns with consistent medical attributes - 89% of records contained patient age, 81% included specific symptoms, and 63% contained medical history information, with an average of 15.6 distinct medical claims per document. This structured nature made the atomization process more systematic - we could reliably separate claims about symptoms, medical history, and demographics. However, this revealed a key privacy challenge: even after sanitization, the semantic relationships between medical attributes remained intact, making re-identification possible through these linked attributes. This was particularly problematic due to the sparsity of specific age-symptom-history combinations in medical data - unique combinations of these attributes could often identify a single patient even when individually sanitized.

(2) Dataset-Specific Sanitization Effectiveness: The structural differences led to interesting patterns in sanitization effectiveness. For MedQA, while DP-based synthesis achieved strong privacy scores (0.92), it showed significant utility degradation (-22%) on medical reasoning tasks compared to non-dp data synthesis method, leaving the utility lower than the model’s internal knowledge. This sharp utility drop occurred because medical reasoning requires precise preservation of sparse, specialized attribute combinations - even small perturbations in the relationships between symptoms, age, and medical history can change the diagnostic implications. Identifier removal performed poorly (privacy score 0.34) as it couldn't break these revealing semantic connections between medical attributes.

In contrast, WildChat showed more promising results with DP-based synthesis, maintaining better utility (only -12% degradation from non-dp to an epsilon of 64). This better privacy-utility balance stems from two key characteristics of conversational data: First, the information density is lower - unlike medical records where each attribute combination is potentially crucial, conversations contain redundant information and natural paraphrasing. Second, the success criteria for conversations are more flexible - small variations in phrasing or exact details often don't impact the core meaning or usefulness of the exchange. This made the dataset more robust to the noise introduced by DP-based synthesis while still maintaining meaningful content.

(3) Practical Guidelines for Sanitization: Our findings challenge the common practice of relying on PII removal and scrubbing methods for text privacy, showing they provide a false sense of security. These insights are particularly timely as organizations increasingly handle sensitive text data across healthcare, customer service, and other domains. We thank you again for your thoughtful feedback and will incorporate the above discussion in the paper.

Talking about metrics: In Equation 1 you define a "privacy metric", I guess that is what is reported under the (why differently named?) "Semantic Similarity" column in Table 1. It is based on the "similarity metric" from Section 2.4, which has values between 1 and 3 -- How does it end up with values between 0 and 1 in Table 1?? I couldn't see any discussion on some form of normalization of these scores. The expected value of scores >= 1 in Eq. 1 would still result in a value >= 1, and not in [0,1). Please double-check how you actually compute the metrics. Try not to distribute information pertaining to one concept across the paper, but put it concisely into one place if possible. Also prefer consistent naming.

Thank you for identifying this inconsistency. The semantic similarity metric indeed originates from a 1-3 scale. We normalize these scores to the [0,1] range. We have consolidated all metric-related information, including this normalization step, in Section 2 to ensure clarity and completeness. Additionally, we have standardized the terminology throughout the paper to consistently refer to these metrics using the same names in both the methodology section and results discussion.

The effect of \epsilon appears surprisingly small to me, with only minimal changes across all metrics even when comparing \epsilon=3 and 1024. Can you explain this behavior?

Explained above.

It would be interesting to compare with a random baseline where the utility is determined from completely random texts -- to rule out that 0.4x task utility in the case of MedQA can already be achieved based on completely random input (say, if the dataset suffers from strong class imbalance and the classifier always just guesses the largest class, thus obtaining an overly optimistic accuracy).

Thank you for this suggestion. We have a related baseline that measures the language model's inherent knowledge bias. Instead of using random text input, we evaluate the model's performance when given only the question without any context or private information, achieving 0.44 accuracy.

L404: What exactly is the "linkage rate"? Please specify.

There are two stages in our pipeline: the linking stage using the linking method L, and the stage where we apply the similarity metric μ. The linkage rate measures the percentage of documents correctly matched with their corresponding auxiliary information using linking method L. For this metric, we only report sanitization methods that preserve a correspondence between original and sanitized documents.

L423: Contradicting statements: Here, you state the last three claims are used, previously in L296, you mentioned 3 randomly selected claims.

We use randomly selected claims in our experiments. We have fixed this inconsistency in the manuscript.

L443/Section 2.4: If you can switch the similarity metric µ also to ROGUE-L, please already state this as possible option in Section 2.4 where you introduce µ. Currently, you only say there that µ is determined by querying a language model.

Thank you. We have updated our paper to distinguish between the language model-based metric and the ROUGE-L based privacy metric.

Lastly, what are your thoughts on information that is both privacy-sensitive and utility-relevant, say, if one or more atomized claims are also strongly related to a downstream task? For instance, what if an atomized claim turns out to be "John likes baseball", and one of the WildChat categories is "baseball", too? Feel free to substitute "baseball" with something more delicate, such as "drinking alcohol". (Case A: If the baseball aspect is kept in the sanitized document, both µ and the chi^2 distance should be small, indicating poor privacy but good utility. Case B: If the baseball aspect was redacted, both µ and chi^2 should be larger, indicating better privacy but poorer utility.)

Answered above.

Additional considerations for related work

Thank you! We have added them in the paper.

L147-148: You state your "approach aggregates the auxiliary information into a single text chunk" -- Does that mean you combine (concatenate?) all "atomized claims" x^(i)_j across all j into the "aux info" {\tilde x}^(i)? (Wouldn't hurt to write this down more explicitly.) (Just found the info in L296/Section 3.3 that the attacker gets 3 random claims for the linkage phase. I find that a bit late, it would be better to mention it directly in Section 2.3 to avoid confusion/guessing on the side of the reader.)

Yes this is exactly what we meant. We’ll update it in the draft in the next version.

L156: Can you give a specific reason for querying only with claims that were not utilized in the linking phase?

Our metric seeks to measure the information gained from having access to released sanitized data. Therefore, when computing the final score, we ignore the auxiliary information.

Besides, how do I know whether a claim about an original document was used for linking?

We randomly select three claims from a given record in this study.

If all atomized claims are combined into the aux info (cf. previous question), and the aux info is used as query in the retriever, wouldn't this imply that all claims are already consumed in the linking phase?

For records containing fewer than three claims, we exclude them from the final privacy metric computation to maintain consistent evaluation conditions across the dataset.

L159: If I understood correctly, you define the "similarity metric" µ between the original and linked documents by querying a language model to judge the document similarity, where you assign values on a scale from 1 (for 'identical documents') to 3. I wonder if it would make more sense to start the scale at 0, since mathematically, a metric has the property of evaluating to 0 for identical inputs. (In your case, we would get "µ(x,x) > 0" instead of "µ(x,x) = 0".)

Yes, we normalize the score to 0-1 when reporting the numbers in the table. We’ll add it to the paper.

How do I know that the atomized claims, which are used to compute µ and hence to measure privacy preservation, are actually privacy-relevant, and not just some arbitrary, privacy-insensitive facts?

Answered above.

L313: I'm confused regarding the symbol "µ" seemingly used for multiple purposes. It is defined in Sec. 2.4, but here, you also use it for another metric induced by ROGUE-L scores.

We apologize for the ambiguity regarding µ. In this baseline, we use ROUGE-L as both the linking function L and the privacy metric µ to investigate privacy score using established text similarity metrics. This choice simulates a classical baseline approach where the same algorithmic method serves both purposes. We have revised the notation to distinguish between different implementations of the steps in the paper.

L317 (also L386): You state "zero-shot prompting achieves an accuracy of 0.44" for MedQA task utility, but why am I unable to find that result in Table 1 (for "No Sanitization", it says 0.69)?

We apologize for the unclear terminology. The 0.44 accuracy refers to our baseline measurement where the model receives only the question and multiple choice options, without any context. This represents the model's inherent knowledge. In contrast, the 0.69 accuracy under "No Sanitization" represents our upper bound, where the model receives complete, unmodified context. We have updated the manuscript to reflect this.

Calling the privacy metrics "Overlap" and "Similarity" is very confusing, since they actually mean the opposite (high lexical overlap and semantic similarity would indicate a high agreement between the two documents, but high scores in Table 1 mean good privacy). Name them lexical/semantic "distance" instead?

Thank you. We have revised the terminology accordingly.

The effect of \epsilon appears surprisingly small to me, with only minimal changes across all metrics even when comparing \epsilon=3 and 1024. Can you explain this behavior?

The fact that privacy preserving techniques with extremely large $\epsilon$ have a significant effect, and at the same time the value of $\epsilon$ has relatively small effect has been observed in other applications of differential privacy also. For example, it is widely known in the DP community that adding DP with very large $\epsilon$ significantly mitigates Membership Inference Attacks (MIA) as measured by standard MIA metrics. We believe that this is partly due to the fact that DP-SGD methods use clipping (even for very large $\epsilon$), which already significantly reduces the influence of any single sample for any $\epsilon$ values.

In addition, there is a significant difference between our threat model and the strong adversarial model assumed in DP. While DP provides worst-case privacy guarantees against an adversary who knows all but one record in the dataset, our threat model considers a substantially weaker adversary who only has access to partial information from a single record. Additionally, the DP-SGD training framework we adopted in the paper composes privacy costs from each optimization step, assuming the adversary can observe gradients throughout training. In contrast, our threat model only allows the adversary to access the final sanitized dataset, not the resulting model and let alone the training process. This further reduces the effective strength of the attack. This finding aligns with recent findings that demonstrate DP's effectiveness against membership inference attacks even with larger epsilon values (Lowy et al., 2024)

Reference

Lowy, A., Li, Z., Liu, J., Koike-Akino, T., Parsons, K., & Wang, Y. (2024). Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks?. arXiv preprint arXiv:2402.09540.

Below are our responses to your remaining questions:

Also, what would happen if you shifted the (apart from the ordering: somewhat arbitrarily) assigned scores for the "similarity metric" in Section 2.4 from {1,2,3} to {0,1,2} or to {1,10,100}?

We apologize for not explaining this clearly in the paper. The privacy score in our framework is normalized to a [0,1] range, where 1 represents complete privacy and 0 represents no privacy. Shifting the scoring scale from {1,2,3} to {0,1,2} would not affect our final results, as the normalization process preserves the relative distances between scores. However, using highly uneven spacing like {1,10,100} could affect the results by introducing non-linear weighting between different privacy levels. We maintained equal intervals in our scoring to ensure consistent sensitivity across all privacy levels.

L081, L099: Just to be sure: If I understood correctly, "claims" can refer to any sensitive or non-sensitive information in the original texts?

Yes, claims encompass any discrete piece of information from the original text, whether sensitive or non-sensitive.

L101: If (1) PII removal had 94% leakage (inferable claims), and (2) data synthesis (with or without DP?) has 9% lower leakage (i.e., 85% ?), why does (3) data synthesis without DP state 57% << 85% leakage?

We apologize for the confusion. The term "identifier removal" in line 101 refers to the broad category of all identifier removal methods, not just PII removal. Our results compare two main categories of data sanitization: identifier removal methods and data synthesis methods. The 9% improvement in privacy protection refers specifically to the difference between Dou et al.'s (2024) identifier removal method, which showed the best performance among removal techniques, and the data synthesis approach.

L146: Could you briefly motivate the use of the sparse BM25 retriever? Did you consider dense retrieval methods (say, using some form of text embeddings)? What are the benefits of BM25 vs. other sparse methods, or of sparse methods vs. dense methods, in particular for the linkage task at hand?

We initially implemented dense retrieval using state-of-the-art dense retriever models GritLM (Muennighoff et al., 2024), but we found that the BM25 sparse retriever performed on average 16% better than dense approach in the MedQA dataset, and they performed similarly in the WildChat dataset.

Reference

Muennighoff, Niklas, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. “Generative Representational Instruction Tuning.” arXiv, April 17, 2024.

Thank you for your thoughtful feedback. We appreciate your highlighting our strengths, including the critical contribution of a systematic evaluation of novel sanitization methods and that our writing is easy to follow. We hope the response below addresses your concerns and questions.

Lastly, what are your thoughts on information that is both privacy-sensitive and utility-relevant?

How do I know that the atomized claims, which are used to compute µ and hence to measure privacy preservation, are actually privacy-relevant, and not just some arbitrary, privacy-insensitive facts?

We are making an important first step towards making privacy metrics and definitions more relevant and practical. Existing privacy metrics, such as differential privacy, address data privacy, where a single token in the textual data is considered private. While this approach works well for structured data (Stadler et al., 2022), text data presents unique challenges that current methods fail to address. As a result, major cloud providers and healthcare organizations continue to rely on simple PII detection and de-identification (Johnson et al., 2020), following outdated privacy models (Garcia et al., 2019).

We take a first step towards addressing this gap by proposing, for the first time, an inferential privacy metric. Our results quantify the failure of existing approaches–showing 94% information leakage with state-of-the-art PII removal methods--and provide compelling evidence that current approaches are fundamentally inadequate.

Ideally, one would want contextual privacy metric, which can take into account (i) which information is more privacy-relevant and (ii) which information is private in the context that the textual information is being shared. These are extremely challenging questions that we believe are beyond the scope of this paper. Nevertheless, they represent exciting research directions to pursue, particularly given recent advances in LLMs. We have added this discussion to the limitation section.

References

Johnson, A. E., Bulgarelli, L., & Pollard, T. J. (2020, April). Deidentification of free-text medical records using pre-trained bidirectional transformers. In Proceedings of the ACM Conference on Health, Inference, and Learning (pp. 214-221).

Garcia, D. (2019). Privacy beyond the individual. Nature human behaviour, 3(2), 112-113.

Stadler, Theresa, Bristena Oprisanu, and Carmela Troncoso. “Synthetic Data -- Anonymisation Groundhog Day.” arXiv, January 24, 2022. http://arxiv.org/abs/2011.07018.

L323: I think the conclusion from a "disparity between lexical and semantic similarity" to "these techniques primarily modify and paraphrase text without effectively disrupting the underlying connected features and attributes" is made a bit prematurely: Both are entirely different measures, and even for "no sanitization", the lexical score is twice the semantic score.

We agree with the reviewer that direct numerical comparison between lexical and semantic scores may not be methodologically ideal, but we focus on how users interpret these privacy metrics. In practice, users often interpret these numbers as direct indicators of privacy protection levels. By showing both metrics, we provide a more complete picture that helps users avoid over-relying on any single measure when assessing privacy guarantees, which can later be used on privacy nutrition labels designed to help practitioners (Smart et al., 2024). This dual approach promotes a more nuanced understanding of actual privacy protection rather than depending on potentially misleading single metrics (Kelley et al., 2009).

References

Smart, M. A., Nanayakkara, P., Cummings, R., Kaptchuk, G., & Redmiles, E. (2024). Models matter: Setting accurate privacy expectations for local and central differential privacy. arXiv preprint arXiv:2408.08475.

Kelley, P. G., Bresee, J., Cranor, L. F., & Reeder, R. W. (2009, July). A" nutrition label" for privacy. In Proceedings of the 5th Symposium on Usable Privacy and Security (pp. 1-12)

Focus evaluation on a range of smaller epsilons

Thank you for the suggestions! We’ll look into adding more smaller epsilon comparisons. Here are our reasons for selecting the existing set of epsilon values:

1. In our experiments, we observe when \epsilon is 3, the model output is private, but the utility is quite low. In particular, the text produced is incoherent (please refer to results in Table 2). We therefore opted not to try lower values of \epsilon, which we would expect to increase privacy (which is already very high) but further decrease utility. Instead, we studied higher values of \epsilon in an attempt to improve utility. We observe that even at these higher values, DP can still protect privacy; this is consistent with recent studies that have also shown that higher values of \epsilon can still protect against membership inference attacks (Lowy et al., 2024; Ponomareva et al., 2022).

2. Our minimum value of \epsilon = 3 follows established practices in the literature, including Yu et al. (2021), Mehta et al. (2022) and Mattern et al. (2022). This value provides stronger privacy guarantees compared to the one evaluated in Yue et al. (2023), whose differential privacy sanitization method we adopted. This informed our decision to examine epsilon values above 3.

References

Yue, Xiang, Huseyin A. Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. "Synthetic text generation with differential privacy: A simple and practical recipe." arXiv preprint arXiv:2210.14348 (2022).

Mehta, Harsh, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. "Large scale transfer learning for differentially private image classification." arXiv preprint arXiv:2205.02973 (2022).

Mattern et al., "Differentially Private Language Models for Secure Data Sharing", EMNLP 2022

Yu, Da, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni et al. "Differentially private fine-tuning of language models." arXiv preprint arXiv:2110.06500 (2021).

Lowy, Andrew, Zhuohang Li, Jing Liu, Toshiaki Koike-Akino, Kieran Parsons, and Ye Wang. "Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks?." arXiv preprint arXiv:2402.09540 (2024).

Ponomareva, Natalia, Jasmijn Bastings, and Sergei Vassilvitskii. "Training text-to-text transformers with privacy guarantees." In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2182-2193. 2022.

where the drop in utility is more significant than the improvement in privacy…

Thank you for highlighting this issue. While DP methods achieve strong privacy protection on both MedQA and WildChat, the privacy gains differ specifically due to variations in the privacy protection of fine-tuning approaches. This difference stems from the threat models: in MedQA, we treat both questions and answers as public information, to evaluate the sanitization method's ability to generate context corresponding to correct choices. Conversely, for WildChat, we consider the entire conversation as private information. We hypothesize that this distinction in information availability directly affects the fine-tuning methods' ability to learn private information, explaining the observed differences in privacy gains across our experiments. We will update the manuscript to reflect the discussion, and improve the claim we are making.

I assume that you will add the additional explanations to the paper where they are required/helpful for readers to better/quicker follow the paper.

We thank you again for your constructive feedback, especially regarding clarity improvements. All suggested clarifications have been incorporated into the manuscript, with modifications highlighted in yellow for reference.