Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning

We sincerely thank Reviewer jBLe for the insightful suggestions and for recognizing the potential impact of our work on the field of unlearning. Below, we address the reviewer’s questions and suggestions.

Major comments & W1. Clarification in Introduction and Related Work.

We sincerely thank the reviewer for the thoughtful suggestion. We agree that further clarifying our contributions relative to prior work will help readers better situate our findings and we outline below the changes we have made in the Introduction and Related Work sections to address this point.

Revision to the Introduction: We have revised the paragraph beginning with “We identify a critical pitfall...” to more explicitly highlight the dual motivation behind our work. First, prior work [1,2] shows that unlearning difficulty varies significantly across samples in vision tasks. Second, LLM studies reveal highly non-uniform memorization patterns. Building on both insights, we first show that privacy risk in LLM unlearning varies across samples, and further demonstrate that high-risk points often coincide with identifiable minority groups. This connection adds a new dimension of societal relevance to existing technical findings.

Revision to the Related Work: We have updated the final paragraph of the Related Work section to more accurately describe [2] as estimating the unlearning difficulty of individual samples in vision tasks using an entanglement-based metric, and showing how variations in the memorization of image representations impact unlearning performance. Our study investigates how minority subgroups in LLMs fine-tuning systematically suffer from degraded unlearning efficacy, an effect that coincides with harder-to-unlearn examples. As a future direction, it would be interesting to explore the connection between such entanglement-based difficulty measures and our minority-aware findings in the LLM setting.

[1] Thudi et al. Gradients look alike: Sensitivity is often overestimated in DP-SGD.

[2] Zhao et al. What makes unlearning hard and what to do about it.

Missing clarification on evaluation repetitions.

We thank the reviewer for the careful comment. All components involving randomness were run with a fixed random seed (42), as noted in App. B.2 and in line with common practice. Due to the large scale of our experiments, we did not repeat runs with multiple seeds. We will clarify this in App. B.2 and note that using a broader range of seeds could further improve robustness in future work.

Could the authors compare the metric proposed in [2]?

We thank the reviewer for this insightful suggestion. We believe the reviewer is referring to the ToW metric proposed in [2]. While the original ToW metric is designed for vision tasks and based on prediction accuracy, its underlying spirit can be adapted to our LLM setting using privacy-related metrics. Specifically, we construct a ToW metric based on AUC differences between the unlearned model ($M_{\text{unlearn}}$) and a retrained model ($M_{\text{retrain}}$) across key dataset partitions. Our adapted ToW metric:

$\text{ToW} = \prod_{(D_i, D_j)} \left(1 - \left| \text{AUC}(M_{\text{retrain}}, D_i, D_j) - \text{AUC}(M_{\text{unlearn}}, D_i, D_j) \right| \right)$

where $(D_i, D_j) \in \{ (D_{\text{retrain}}, D_{\text{forget}}), (D_{\text{forget}}, D_{\text{test}}), (D_{\text{retain}}, D_{\text{test}})\}$. Higher ToW indicates lower privacy leakage.

We applied this metric to the ECHR dataset with GPT-2 under loss-based attack. As shown in the table, the canary and minority settings consistently yield lower ToW scores than the random setting, indicating greater privacy leakage. These results reinforce our main finding that minority points are harder to unlearn. In addition, we believe it would be interesting future work to further investigate how the idea behind ToW could be leveraged to help identify minority points in the LLM setting.

Typos.

We thank the reviewer for pointing out these typos and have corrected both in the revised version.

Summary: We sincerely thank Reviewer jBLe again for the thoughtful and constructive feedback. We would be happy to clarify any remaining concerns or engage in further discussion. We hope that our responses have addressed the key points raised, and we truly appreciate your openness to reconsidering the score upon clarification.

We greatly thank Reviewer Cub4 for reviewing our paper. Below, we address the questions and comments raised by Reviewer Cub4.

The paper fails to provide a rigorous definition of ‘minority’. The definition should use a mathematical model.

We thank the reviewer for this comment. As noted in Footnote 2 and detailed in Sections 4.1 and 4.2, our paper defines minority instances based on the population frequency of their corresponding identifiers (e.g., rare phone area codes or email domains). This definition is precise, grounded in observable statistics, and enables systematic evaluation.

We further validate it through controlled canary experiments, where replacing common identifiers with rare ones while keeping the rest of the input fixed leads to significantly higher privacy leakage. Similar trends are observed in real-world minority samples (“Minority Setting”).

While our definition is frequency-based, we welcome clarification on the type of mathematical model the reviewer envisions. Extending the framework to semantically defined minority groups may benefit from additional modeling, which we consider a promising future direction.

The minority points seem to conflate with outliers or hard-to-unlearn examples.

We thank the reviewer for raising this point and allowing us to clarify a potential misunderstanding. Our experiments demonstrate that the PII-population-defined minority samples consistently exhibit significantly higher privacy leakage across a wide range of unlearning methods. These results indicate that our defined minority points are systematically harder to unlearn in terms of privacy risk, as acknowledged by Reviewer jBLe. To be clear, by harder to unlearn, we refer specifically to their elevated vulnerability under privacy metrics. To prevent potential confusion, we will further refine the language in the paper.

The claim that Langevin unlearning performs better due to noise addition is unconvincing, as its performance instead can be attributed to its theoretical guarantees.

We thank the reviewer for their comment. There is an intrinsic connection between noise injection and the theoretical guarantees of Langevin unlearning. While it is true that Langevin dynamics enjoys privacy guarantees, these guarantees fundamentally rely on the analysis of Langevin Dynamic, how injected Gaussian noise affects the model’s trajectory over adjacent datasets, typically measured via Rényi divergence. In other words, the theoretical robustness of Langevin unlearning is itself a direct consequence of noise injection. We will revise the paper to make this connection more explicit.

Using PII to represent minority data points lacks justification—there’s no clear reason why PII should indicate minority status.

We thank the reviewer for this question. First, PII inherently contains sensitive personal information, making it a central focus in privacy-related research [1,2,3]. Moreover, using PII also enables us to clearly define minority points by leveraging population-level frequency (e.g., rare area codes or email domains), which we believe helps address the reviewer’s concern regarding the clarity of the minority definition. Finally, from an empirical perspective, we show that instances defined as minority under this framework consistently exhibit significantly higher privacy leakage across multiple unlearning methods.

[1] Lukas et al. Analyzing leakage of personally identifiable information in language models.

[2] Kim et al. Propile: Probing privacy leakage in large language models.

[3] Li et al. Llm-pbe: Assessing data privacy in large language models.

Other related references.

We thank the reviewer for sharing the relevant works [4, 5, 6]. These papers focus on corrective unlearning, which aims to address corrupted or harmful training data. As noted in these papers, their goals and assumptions differ from those of privacy-oriented unlearning, which is the focus of our work. To acknowledge their relevance, we have cited them in the introductory paragraph of Section 3 (Preliminaries), where we discuss the broader unlearning landscape.

While canary crafting has been explored in privacy research, it remains overlooked in evaluating unlearning methods for LLMs. Our work fills this gap by demonstrating that selecting high-risk samples, especially minority points, is key to uncovering underestimated privacy risks in current unlearning evaluations.

[4] Goel et al. Corrective machine unlearning.

[5] Li et al. Delta-Influence: Unlearning Poisons via Influence Functions.

[6] Schoepf et al. Potion: Towards poison unlearning.

Summary: We thank Reviewer Cub4 for the thoughtful and detailed feedback. We appreciate the opportunity to respond and are happy to clarify any remaining concerns. If our responses have addressed the points raised, we would be grateful if this could be reflected in the final evaluation.

We sincerely thank Reviewer WLjM for acknowledging the novelty and comprehensiveness of our minority-aware framework, as well as for supporting the acceptance of our paper. Below, we provide detailed responses to the questions and concerns raised.

Q3 & W3 & C3: Code for our paper.

We appreciate the reviewer’s emphasis on the importance of code availability for both reproducibility and broader dissemination of our work. Yes, in the supplementary material (provided as a ZIP file), we have included all the relevant code and provided a detailed README.md outlining the installation steps. After the paper is made public, we will further release our GitHub repository to ensure wider accessibility and community engagement.

W1 & C2 & Q2 : “The paper focuses only on PII data, leaving the applicability to other minority attributes uncertain.” & Could the authors apply their minority-aware evaluation framework to non-PII minority data, such as dialects or underrepresented social groups?

We thank the reviewer for this insightful and important question. We fully agree that clearly defining what constitutes a “minority” in different contexts is nontrivial, and that generalizing beyond PII is an open and meaningful research direction.

In this work, we chose to focus on PII for two key reasons. First, PII is inherently sensitive and has been a central focus in privacy-related studies [1,2,3], making it a practical and impactful setting for studying unlearning. Second, PII (such as phone number area codes or email domains) offers a well-defined population structure, which allows us to rigorously formulate and operationalize the notion of minority subgroups in a reproducible way.

That said, when non-PII attributes exhibit a clear format and can be associated with measurable population-level frequencies, our framework can be directly generalized to these settings. For attributes that are more semantically defined or lack an explicit population distribution, such as linguistic patterns, we acknowledge that identifying minority status is more challenging. Extending our framework to these settings is a promising direction for future work, and we would be excited to explore this further.

[1] Lukas et al. Analyzing leakage of personally identifiable information in language models. In S&P 2023.

[2] Kim et al. Propile: Probing privacy leakage in large language models. In NeurIPS 2023.

[3] Li et al. Llm-pbe: Assessing data privacy in large language models. In VLDB 2024.

W2: The paper does not explore the impact of different fine-tuning strategies, such as LoRA versus full fine-tuning.

We appreciate the reviewer’s suggestion. Exploring the impact of different fine-tuning strategies, such as LoRA versus full fine-tuning, on unlearning and privacy leakage is indeed an interesting direction. Our current experiments are designed to support our core claim that minority samples are disproportionately vulnerable to privacy leakage across multiple datasets, attack methods, and model scales. Due to computational constraints, especially for large-scale multi-GPU training, we have so far focused on the more resource-efficient LoRA-based fine-tuning strategy. While full fine-tuning on models like LLaMA-7B would be a valuable extension, we believe it is not essential to establish the key findings of our study. We view this as a promising avenue for future exploration and would be happy to investigate it as resources permit.

C1 & Q1: Have the authors considered testing larger models (e.g., 13B, 30B) to see if privacy risks scale with model size?

We thank the reviewer for the valuable question. Investigating how privacy risks scale with model size under our minority-aware framework is indeed an interesting and important direction. Prior work has demonstrated that both verbatim memorization and extractability tend to increase with model scale [4], which suggests that minority-group privacy risks may also become more severe in larger models. While our current experiments include both small (GPT-2) and mid-sized (LLaMa-2 7B) models to support our central claims, which align with model sizes used in current literature [1,5,6], computational constraints have limited our ability to run experiments on larger models such as 13B or 30B. We view this as a promising avenue for future work and plan to explore it as resources become available.

[4] Carlini et al. Quantifying memorization across neural language models. In ICLR 2023.

[5] Maini et al. Tofu: A task of fictitious unlearning for llms. In COLM 2024.

[6] Shi et al. Muse: Machine unlearning six-way evaluation for language models. In ICLR 2025.

Summary: We are grateful to Reviewer WLjM for the supportive and thoughtful review. We would be happy to provide further clarifications or engage in additional discussion. If our responses have helped resolve the raised concerns, we humbly hope this might be taken into account in the final score.