Language Models are Advanced Anonymizers

We thank the reviewer for their comprehensive feedback. We respond to the raised concerns below, numbered Q1-Q5. We have also uploaded a revised version of the paper, with updates highlighted in violet. We are happy to continue the discussion further if the reviewer has additional questions.

Q1: Does the simplicity of the final anonymization algorithm decrease the overall merit of the work?

No, we believe that even though our eventual anonymization method is conceptually straightforward from a purely technical perspective, it makes significant contributions in the field of text anonymization in several ways—especially so in casual communicative settings, such as online forums, where, as shown in [1] LLMs pose a severe novel threat to anonymity.

On the setting side, we are the first to provide an (inference-based) adversarial view on text anonymization, addressing the key limitation of prior stance on anonymization; namely that even entity-blanking-based anonymization still leaks private information in presence of an inferential adversary. This is a critical issue that is fundamental to current anonymization tools and evaluation. As also shown in [1], we demonstrate in our evaluation that on real-world data existing methods are clearly insufficient both against adversaries and in utility.

On a practical side, as reviewers agree, adversarial-feedback anonymization using LLMs provides a natural instantiation of this setting and achieves consistently better privacy protection and utility than commercial anonymization tools. As such, our strong empirical contributions could have an noticable impact on the state of anonymization currently in application.

Further, having such an elaborate analysis and extensive experimental evaluation (across 17 models, multiple datasets, and including a human study), sets a strong baseline for any potential follow-up work in this area.

As such, we believe that there is significant merit in (1) defining a practical and free form setting for text anonymization and (2) providing a method that outperforms current industry-standard anonymization across very extensive evaluations. Such a contribution is particularly timely as developments in LLMs have made such inferences much easier [1] up to the point where it can be potentially misused on a large scale [2], constituting a practical real-world threat.

Q4: Can you relate adversarial anonymization to DP? In which cases could there be joint applications?

We agree with the reviewer that DP's privacy guarantees are the gold standard in many settings. However, in practice, these mechanisms are often very difficult to apply to text (For instance as in [6] (When did Tesla move to New York City? -> Wave did Tesla It way Dru Tully breaking?)), especially when we want to keep the general utility and meaning of the text, such as in free form online settings considered in our work. This problem is exacerbated in the setting we are targeting, where people are individually contributing to a "database" and have to resort to local DP approaches. Further, potential future interactions of a user are unbounded, inhibiting rigorous and finite DP guarantees that hold also in the future. Finally, DP fundamentally requires randomness, which would mean that DP-based anonymization would not allow the user to interfere with the anonymized outputs. Our previous example illustrates well why this is a problem; imagine if a user wants to ask “When did Tesla move to New York City?” in a forum but once they hit the “post” button, the posted text appears as “Wave did Tesla It way Dru Tully breaking?”, without an option to edit. In contrast, our method enables more practical fine-grained interactions with the user at each round of anonymization.

Nonetheless, we believe that adversarial anonymization and DP can actually complement each other as they target different notions of privacy - in particular, we can imagine a setting in which the collected data text is first anonymized in order to protect individual attributes at the time of collection and, later, model training via DP-SGD provides privacy guarantees on that particular model instance itself. If the reviewer is interested in a more comprehensive overview of text anonymization and DP methods we can recommend [7] as a great starting ressource.

References
[1] Staab, Robin, et al. "Beyond memorization: Violating privacy via inference with large language models." ICLR 2024.
[2] Thomas Brewster. Chatgpt has been turned into a social media surveillance assistant, Nov 2023.
[3] Jin, Di, et al. "What disease does this patient have? a large-scale open domain question answering dataset from medical exams." Applied Sciences 11.14 (2021): 6421.
[4] DOL, 2023. URL https://www.dol.gov/general/ppii.
[5] European Union. (n.d.). What personal data is considered sensitive?. European Commission.https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/sensitive-data/what-personal-data-considered-sensitive_en.
[6] Yue, Xiang, et al. "Differential privacy for text analytics via natural text sanitization." arXiv preprint arXiv:2106.01221 (2021).
[7] Lison, Pierre, et al. "Anonymisation models for text data: State of the art, challenges and future directions." Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Q2: Have you considered measuring the utility via downstream performance? For which settings do you believe adversarial anonymization is useful here?

One of the key challenges with free-form text in our target online setting is that utility in many cases (particularly for users that write the texts) is defined via the coherence and expression of the text itself. Notably measuring the utility via some downstream task can actually hide a lot of utility-loss (as we detail further in Q4) that would be important to humans in this setting (such as readability). Nevertheless, if there is a clear downstream target one could also use this to get a practical indication of how much task-specific utility is retained. Prompted by the reviewer's question, we extended our evaluation in two ways: First, we evaluate also on the MedQA dataset [3] having a much more rigid text setting, with a clearly defined downstream MC-Accuracy metric, and secondly, we compute embeddings over all anonymized (and original) texts in PersonalReddit allowing us to quantify their similarity independent of potential downstream tasks.

On MedQA using GPT-4o as a downstream classifier we achieve $85.4\%$ baseline accuracy. After applying a single round of adversarial anonymization we reduce the number of adversarial age, location, and place-of-birth, predictions by $>50\%$ while also showing strong results other attributes like Sex ($>25\%$) and occupation, while still maintaining a downstream accuracy of $81.4\%$ (we expect some drop as in some cases this information can be quite relevant for predictions). This makes it competitive in utility with Azure ($81.5\%$) that works that works better on such reports than on free-form data (e.g., almost every text starts with “A XX-years old man/woman/baby has been …”), while slightly outperforming it on privacy. We present a full overview in App. H.1.

As a proxy for the retention of downstream utility on free-form text, we further also compute embeddings using the text-embedding-3-large model by OpenAI. As these embeddings are usable for all sorts of potential downstream tasks, they constitute a strong proxy for how well we might perform on arbitrary downstream tasks and quantify how close the anonymized text is to the original. We report cosine similarity between embedding vectors (1 being a perfect match). We find that, e.g., on Llama-3.1-70B-AA, we have a median cosine-similarity of $0.93$ after one round ($0.88$ after 2 and $0.84$ after 3). This is in stark contrast to the median of $0.24$ between $1000$ selected random comments in PersonalReddit and re-affirms that adversarial anonymization maintains a high level of utility.

We provide more details and an additional discussion on the above experiments in the newly added App. H.

Q3: How can this be turned into a more comprehensive tool? Does adversarial anonymization require me to pre-define all attributes that should be removed in the text? Is this feasible?

The reviewer raises an interesting question. In particular, we find that Adversarial anonymization only requires one to define the attributes that they want to protect, not the way they are expressed in text. As such, there could be no unintended privacy leakage in case the user lists everything to the algorithm that they consider “private”. In fact, these “attributes” may be even more complex aspects to protect, e.g., mental health information or descriptions of any specific information to hide. This is a natural improvement over how it is done in classical anonymizers (Presidio, Azure) that target the direct expression of a clearly scoped attribute in the text (e.g., a direct mention of a location but not clues that would allow a very certain inference). The interpretation of adversarial anonymization is much more aligned with regulatory requirements here (e.g., [4]). However, as pointed out by the reviewer, this still requires us to define the initial set of attributes. In a pragmatic sense, these can quite often be directly informed by existing legislation ([4] and [5]). Note that theoretically, as a first step, the adversarial LLM could be tasked to “infer everything and anything” from the given text, from which then the user could select which information they wish to remove from the snippet. The deeper investigation of the promise of such approaches is, we believe, an interesting avenue for future work.

We thank the reviewer for their comprehensive feedback. We respond to the raised concerns below, numbered Q1-Q4. We have also uploaded a revised version of the paper, with updates highlighted in violet. Further we thank the reviewer for catching typos which we adapted in the manuscript. We are happy to continue the discussion further if the reviewer has additional questions.

Q1: Does the simplicity of the final anonymization algorithm decrease the overall merit of the work?

No, we believe that even though our eventual anonymization method is conceptually straightforward from a purely technical perspective, it makes significant contributions in the field of text anonymization in several ways—especially so in casual communicative settings, such as online forums, where, as shown in [1] LLMs pose a severe novel threat to anonymity.

On the setting side, we are the first to provide an (inference-based) adversarial view on text anonymization, addressing the key limitation of prior stance on anonymization; namely that even entity-blanking-based anonymization still leaks private information in presence of an inferential adversary [1]. This is a critical issue that is fundamental to current anonymization tools and evaluation. As also shown in [1], we demonstrate in our evaluation that on real-world data existing methods are clearly insufficient both against adversaries and in utility.

On a practical side, as reviewers agree, adversarial-feedback anonymization using LLMs provides a natural instantiation of this setting and achieves consistently better privacy protection and utility than commercial anonymization tools. As such, our strong empirical contributions could have an noticable impact on the state of anonymization currently in application.

Further, having such an elaborate analysis and extensive experimental evaluation (across 17 models, multiple datasets, and including a human study), sets a strong baseline for any potential follow-up work in this area.

As such, we believe that there is significant merit in (1) defining a practical and free form setting for text anonymization and (2) providing a method that outperforms current industry-standard anonymization across very extensive evaluations. Such a contribution is particularly timely as developments in LLMs have made such inferences much easier [1] up to the point where it can be potentially misused on a large scale [2], constituting a practical real-world threat.

We thank the reviewer for their comprehensive feedback. We respond to the raised concerns below, numbered Q1-Q7. We have also uploaded a revised version of the paper, with updates highlighted in violet. We are happy to continue the discussion further if the reviewer has additional questions.

Q1: Does the remaining 41% accuracy after anonymization indicate failures of the anonymization?

We thank the reviewer for raising this important point. To clarify, no, it does not indicate the failure of anonymization, it is instead an artifact of highly uncertain predictions in later anonymization stages. First, we note that certain discrete features enable even a random adversary to achieve high-looking scores, e.g., random accuracy on sex would be ~50%. Further, to elaborate on the main reason behind the remaining accuray, note that in the reported joint adversarial accuracy of 41%, we count any prediction made by the adversary for any certainty level (1-5). In particular, this also accounts for cases when there was no clear evidence or basis for inference in the text and the model relied purely on inherent biases (i.e., certainty 1 and 2 - we provide qualitative examples in the newly added App. D.4).

Crucially, if we only account for predictions made with certainty $3$ or higher (following [1]) the resulting adversarial accuracy drops down to only $7.7$ (For $\geq 2$ we get $19\%$. Notably as we had shown in Fig. 4 this even furthers the advantage adversarial anonymization has over traditional techniques as it not only leads to lower accuracy in guessing but also a much lower certainty in the guesses.

Prompted by the reviewers question we have added an additional discussion on this to the updated manuscript in App. G. We are happy to expand on this in case the reviewer has further recommendations.

Q2: Why did you evaluate particularly on the online domain setting? How does adversarial anonymization transfer to other settings?

We primarily focused on the online domain as (1) it is a setting where real-world risks were shown in prior works [1,2] and (2) it is a setting which is uniquely challenging for existing methods in anonymization—to an extent where they are borderline unusable (and unpreferred, as shown in our human evaluation). In that sense we believe that providing a solution that works well here is a valuable contribution.

Prompted by the reviewer's question, we extended our evaluation in two ways (presenting new results in App. H): First, we evaluate also on the MedQA dataset [2] having a much more rigid/structured text setting, with a clearly defined downstream utility metric via multiple-choice accuracy. Second, we compute embeddings over all anonymized (and original) texts in PersonalReddit allowing us to quantify their similarity independent of potential downstream tasks.

On MedQA using GPT-4o as a downstream classifier we achieve $85.4\%$ baseline accuracy. After applying a single round of adversarial anonymization we reduce the number of adversarial age, location, and place-of-birth, predictions by $>50\%$ while also showing strong results other attributes like Sex ($>25\%$) and occupation, while still maintaining a downstream accuracy of $81.4\%$ (we expect some drop as in some cases this information can be quite relevant for predictions). This makes it competitive in utility with Azure ($81.5\%$) that works that works better on such reports than on free-form data (e.g., almost every text starts with “A XX-years old man/woman/baby has been …”), while slightly outperforming it on privacy. We present a full overview in App. H.1. As a proxy for the retention of downstream utility on free-form text, we further also compute embeddings using the text-embedding-3-large model by OpenAI. As these embeddings are usable for all sorts of potential downstream tasks, they constitute a strong proxy for how well we might perform on arbitrary downstream tasks and quantify how close the anonymized text is to the original. We report cosine similarity between embedding vectors (1 being a perfect match). We find that, e.g., on Llama-3.1-70B-AA, we have a median cosine-similarity of $0.93$ after one round ($0.88$ after 2 and $0.84$ after 3). This is in stark contrast to the median of $0.24$ between $1000$ selected random comments in PersonalReddit and re-affirms that adversarial anonymization maintains a high level of utility. We provide more details and an additional discussion on the above experiments in the newly added App. H.

I appreciate the authors' rebuttal, which has sufficiently addressed my concerns. I have increased my score as a result. After reading all the other discussion, I note that the primary criticism centers on the method's lack of novelty. While it is less satisfying that this work does not provide theoretical guarantees or propose a novel method to protect privacy, it presents a fresh and more practical perspective on privacy that is backed up with extensive experiments.