UnSTAR: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs

W1 Hallucination concerns for UnSTaR

This is a valid concern, and thank you for pointing it out. While it is true that intuitively constructing anti-samples involves generating misleading or incorrect answers (which might increase the potential for hallucinations), we address this issue as:

1. Extensive filtering of semantically divergent questions and near-correct incorrect answers ensures the relevance of the generated anti-samples. So, hallucinated content are discarded in the unlearning process

2. The iterative paraphrasing further uses hallucination positively to cater to a variety of ways the same question can be asked.

3. We fine-tune the model progressively using anti-samples for the forget set, and on samples for the retain set. This careful balance prevents overfitting on incorrect answers, which is a primary contributor to hallucination.

4. Our experiments show that this approach minimizes hallucination while effectively achieving the desired unlearning using the metric hallucination avoidance (Line 322).

W2 Experiment Settings, Evaluations, Datasets, Comparisons, Ablations

Experimental Setup and Hyperparameters: The experimental setup, dataset size, training settings, and hyperparameters for the unlearning baselines are consistent with the EMNLP paper [1], ensuring that our results are directly comparable.

Metrics: Yes these metrics are used in the existing LLM Unlearning (refer EMNLP paper [1])

Dataset: We conduct additional experiments on 2 datasets: Peter Parker and TOFU

Runtime and Performance Comparison with Existing Methods: We show the time cost comparison with 3 existing state-of-the-art methods in Table 1 (refer response to Reviewer DtLx). We also compare the ROUGE-L scores with the SNAP method across 3 datasets: Harry Potter, Peter Parker, and TOFU in Table 2 (refer response to Reviewer DtLx).

Ablation Study: We show the impact of N on the experimental results in Table 3. The results of fine-tuning over 10 iterations, where the number of generated paraphrased questions (N) increases with each iteration, and the model is fine-tuned for 10 epochs per iteration. As N grows, the model's accuracy steadily improves, reaching 100% by the 10th iteration. The number of retained samples after filtering increases over time, indicating better data quality generation, while the number of filtered samples decreases.

Table 3: Impact of N on the experimental results

[1] Liu Y, Zhang Y, Jaakkola T, Chang S. Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective. EMNLP 2024.

Q1 Typo Error

We apologize for the oversight. It refers to the citation:

Yao, Yuanshun, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv:2310.10683, 2023.

Q2 Sampling of r and Clarity on notation

In Eq. (1), the notation ( p_{\mathcal{M}}(a \mid q, \varphi) \) refers to the conditional probability of the answer ( a \), given the query $q$ and model parameters $\varphi$. The rationale $r$ is sampled from the model’s distribution ( p(r \mid q, \varphi) \), where the model first samples a latent rationale $r$ before predicting the answer $a$. This process follows a greedy approach, where the rationale $r$ is chosen to maximize the likelihood of predicting the correct (or unlearned) answer $a$, which is not necessarily the original answer but any answer that aligns with the unlearning process. This procedure is aligned with the STaR framework, but in our case, the correct answer refers to the unlearned answer from the forget set ( \mathcal{Q}_f \), as opposed to the original answer in the retain set. The model is trained to optimize the unlearning objective, ensuring that the model unlearns specific knowledge while still predicting answers consistent with the unlearned information.

Dear Reviewer,

Based on the above, could you please consider updating the rating of our paper?

Dear Reviewer,

We would like to briefly summarize our discussion and the current status:

1. Addressing hallucination concerns for UnSTaR -> DONE
2. Adding details in paper on experiment settings, implementation of baselines, and datasets sizes. -> DONE
3. Additional results on runtime and performance comparison with existing methods -> DONE
4. Additional Ablation Study -> DONE

Thank you for your reviews. We are waiting for the final rating of our paper. We would be grateful if you could raise the rating.

Thank you for the overall positive feedback. We are glad that our work's motivation and clarity resonated well, and we hope it inspires further exploration in this important area.

W1 On novelty of anti-sample generation

We agree that an iterative process is appealing, and we want to clarify that our anti-sample generation is indeed iterative, much like prompt engineering. Beyond the initial step of generating similar words, we employ STaR-inspired rationales to create misleading answers that target specific knowledge for unlearning. STaR is used for general reasoning tasks and doesn’t produce anti-samples on its own. Our work is first to propose an unlearning method based on the principles of STaR. Existing unlearning methods have not explored reasoning based unlearning. The novelty lies in our method’s ability to generate anti-samples with misleading rationales, ensuring not only dissociation but also maintaining the integrity of the retain set.

Additionally, extensive filtering ensures the relevance of the generated anti-samples, removing semantically divergent questions and near-correct incorrect answers. The iterative paraphrasing further refines the process, making our approach more nuanced and effective than traditional methods like refusal responses or random labels.

W2 Details of methodology

In response to your comment, prompt adjustment is not required for generating Semantically Divergent Questions and Near-Correct Incorrect Answers. The generation process itself, combined with extensive filtering, ensures that only the relevant anti-samples are retained.

When samples are too semantically close to the original, they are filtered out during this process. The remaining valid questions and answers, which pass the filtering criteria, are then used for fine-tuning. This approach ensures that the unlearning process remains effective and the generated anti-samples are sufficiently distinct from the original knowledge.

Another insightful and interesting point the reviewer says is whether there could be instances where no anti-sample can be generated. While we did not encounter such cases in our experiments, we acknowledge that LLMs might struggle to generate anti-samples, particularly when justifying a difficult incorrect answer. In such cases, a human expert can provide the necessary justification, which can then be fed back into the LLM to generate additional anti-samples, further strengthening the unlearning process.

In the context of Line 7 in Algorithm 1 of our paper, the phrase "do similar steps for retain set $\mathcal{Q}_r$" refers to applying the same general process to the retain set, but with a key difference: instead of generating anti-samples (incorrect answers), the model should focus on preserving and enhancing the correct knowledge from the retain set.

To elaborate

1. For the forget set $\mathcal{Q}_f$: The process involves generating anti-samples by paraphrasing questions and falsifying answers (e.g., $\bar{a}_i$ is a falsified or incorrect answer). The anti-samples are used to fine-tune the model so that it unlearns the knowledge in the forget set.

2. For the retain set $\mathcal{Q}_r$: The same procedure is followed, but here we do not falsify answers. Instead, the model is fine-tuned using correct answers from the retain set. This ensures that the knowledge in $\mathcal{Q}_r$ is preserved and reinforced during the training process, avoiding unlearning of essential information.

In summary, the phrase refers to repeating the same process (i.e., paraphrasing and selecting) for the retain set, with the critical distinction that fine-tuning occurs using correct answers (not falsified ones) to preserve the knowledge in the model.

W3 Approximation to RL policy learning

We understand your concern regarding the use of Reinforcement Learning (RL) terminology. It is not meant to imply that UnStar introduces a fundamentally novel RL method. Instead, it serves as a conceptual analogy to explain the iterative nature of the unlearning process.

W4 Additional Results

We show the time cost comparison with 3 existing state-of-the-art methods in Table 1. We also compare the ROUGE-L scores with SNAP method across 3 datasets: Harry Potter, Peter Parker, and TOFU in Table 2 (refer: Reviewer DtLx or refer: Official Comment by Authors)

Regarding implementation details, since we closely followed the settings of EMNLP paper [1] for both UnStar and the baselines, we did not rewrite them in the manuscript to avoid redundancy. For clarity, we will explicitly reference these settings in the appendix to ensure reproducibility.

[1] Liu, Yujian, et al. Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective. EMNLP 2024.

Ablation Study: We show ablation study in Table 1: Impact of N on the experimental results (refer: Reviewer fwuc or refer: Official Comment by Authors). We will add statistical error bars in the final version of the paper.

Dear Reviewer,

We would like to briefly summarize our discussion and the current status:

1. Details on novelty of anti-sample generation -> DONE
2. Specific details of method: continuous paraphrasing, and filtering semantically divergent questions and near-correct incorrect answers -> DONE
3. Elaboration of Algorithm 1 -> DONE
4. Additional results on more datasets -> DONE
5. Comparison with more unlearning methods -> DONE
6. Explanation of The Reinforcement Learning Style Policy Gradient Approximation -> DONE
7. Additional Ablation Study -> DONE
8. Details of dedicated customization of STaR to the unlearning problem -> DONE

Thank you for your reviews. We are waiting for the final rating of our paper. We would be grateful if you could raise the rating.

Thank you for your positive feedback on our concept selectivity and practical examples.

W1 About the novelty of the method...

STaR is used for general reasoning tasks and doesn’t produce anti-samples on its own. Our work is first to propose an unlearning method based on the principles of STaR. Existing unlearning methods have not explored reasoning based unlearning. The novelty lies in our method’s ability to generate anti-samples with misleading rationales, ensuring not only dissociation but also maintaining the integrity of the retain set. Unlike existing unlearning methods, which often rely on data deletion or gradient masking, our approach leverages the generation of anti-samples to target specific associations, which is both efficient and less disruptive to the model’s overall knowledge.

To emphasize our contributions, we will expand the discussion in the revised manuscript to highlight why the adaptation of STaR for unlearning is non-trivial, particularly in generating rationales that effectively counteract prior associations without degrading related knowledge.

Challenges in combining these methods:

1. LLMs hallucinate and generate semantically divergent paraphrasing of questions or near-correct incorrect answers. To address this, we employ additional filtering mechanisms to retain only the most relevant paraphrasings and truly incorrect anti-samples, ensuring high-quality input for fine-tuning.

2. Another challenge lies in the variety of ways the same question can be asked. If unlearning is not effectively performed and merely suppresses the response, the original answer might still come up with creative prompts. To mitigate this, our method generates a diverse range of paraphrasings. We demonstrate that the original answer is not revealed even after such varied questions.

W2 Regarding evaluation

Thank you for the suggestion of splitting performance by subgroups, we would like to highlight that our approach already integrates the contributions from Forget, Hard-retain, and General-retain sets using harmonic mean in the composite metrics. This provides a balanced, holistic view of UnStar’s performance across these different subgroups. For instance:

1. Unlearning Efficacy combines FQA with GPT privacy scores to capture the effectiveness of unlearning.

2. Model Utility aggregates HRQA and GRQA to measure the retention of relevant knowledge.

Harmonic means ensures that all subgroups are already considered, and further splitting might not add significant new insights. This is already explained in the paper Line 300-323.

Additional Results We show the time cost comparison with 3 existing state-of-the-art methods in Table 1. We also compare the ROUGE-L scores with SNAP method across 3 datasets: Harry Potter, Peter Parker, and TOFU in Table 2 (refer: Reviewer DtLx or refer: Official Comment by Authors)

The relevance of Figure 3 has been discussed in the Paper Line 483-485.

“Figure 3 illustrates the LLM’s unlearning efficacy as it progressively … improves its responses over time.”

W3 Failure cases of UnSTaR

You are correct in pointing out that UnStar ensures the model never answers with the original answer for any of the unlearned questions. This is true for the datasets we tested upon. However, other datasets might reveal difficult to unlearn questions. Specifically, certain deep-rooted or highly ingrained concepts can cause additional challenges. Forgetting a widely recognized concept like "dog," may require extensive iterations and a larger number of questions and justifications to fully eliminate every specific detail associated with it. Then, UnStar might struggle to achieve full unlearning, as it demands a significant amount of diverse anti-samples and paraphrasings. This challenge highlights the need for future work to refine the process and explore additional techniques for more difficult and deeply embedded knowledge.

W4 About using Membership Inference Attacks (MIA) for evaluation

MIA has been a critical aspect of evaluating unlearning in traditional neural networks based models. However, a recent COLM 2024 paper titled "Do Membership Inference Attacks Work on Large Language Models?" [1] demonstrates that MIA is not appropriate to assess unlearning in the context of LLMs. The paper concludes that MIA against LLMs is difficult for two key reasons:

1. Data Imprint: The pre-training process of LLMs, especially with large datasets and single-epoch training, does not leave a strong imprint of individual data points, making it harder to discern membership.

2. Similarity Between In and Out Members: The high similarity between in-domain and out-of-domain data, coupled with massive datasets, results in a fuzzy distinction between data members, complicating the effectiveness of MIAs.

[1] Duan M, Suri A, Mireshghallah N, Min S, Shi W, Zettlemoyer L, Tsvetkov Y, Choi Y, Evans D, Hajishirzi H. Do membership inference attacks work on large language models? COLM 2024

Dear Reviewer,

We would like to briefly summarize our discussion and the current status:

1. Clarification about novelty -> DONE
2. Challenges in combining unlearning with STaR -> DONE
3. Evaluation on more datasets -> DONE
4. Comparison with more unlearning methods -> DONE
5. Failure cases of UnSTaR -> DONE
6. Clarification about Membership Inference Attack -> DONE

Thank you for your reviews. We are waiting for the final rating of our paper. We would be grateful if you could raise the rating.

Dear Reviewer,

Given that we have addressed all your concerns, we kindly request your final rating of our paper. We are grateful for your persistent participation in rebuttal process.

We would also like to inform you that Reviewer DtLx has raised their rating to 8.

Thank you

Thank you for the reviews. We address each of your concerns below.

W1: Adhoc non-comparable evaluation methods

We use the same metrics as used in the existing LLM Unlearning work (refer EMNLP paper [5])

W2: Missing references to prior work

We will add the missing references.

W3 and Q1: Regarding the context, goals, and evaluation metrics of LLM Unlearning

Context of LLM Unlearning: Traditional unlearning literature, such as Bourtoule et al. (2021), primarily focuses on privacy guarantees for structured datasets like classification models. However, our work addresses unlearning in LLMs, which introduces distinct challenges and motivations beyond privacy as highlighted in Lines 033–044 of Introduction. We elaborate the same here:

(a) AI Safety & Legal Compliance: Adhering to privacy laws and copyright regulations

(b) Ethical Concerns: Eliminating harmful or biased data to ensure fair and responsible AI use.

(c) Epistemological Necessity: Removing obsolete or irrelevant information to maintain model accuracy and alignment with evolving requirements.

Unlike classical privacy-focused unlearning, our approach addresses the practical challenge of disentangling specific knowledge representations in LLMs, ensuring targeted associations are unlearned without compromising the model’s overall utility.

Metrics: We have followed the EMNLP paper [5] for evaluation metrics, which are widely accepted within the community. These metrics align with our goal of achieving fine-grained targeted unlearning by balancing two critical factors:

(a) Unlearning Efficacy: Capturing the selective removal of targeted knowledge.

(b) Model Utility: Maintaining performance on related and general knowledge.

Other factors are Response Quality, Hallucination Avoidance and Adversarial Robustness as mentioned in Line 315-341 of our paper.

About relation with Ununlearning [1] & different ways of questioning in [2]: Ununlearning, as described by Shumailov et al., highlights that unlearned knowledge can resurface through contextual interactions. To address this, our work evaluates the unlearning process not just by directly querying the model but also through multiple paraphrased ways of asking about the same knowledge. This ensures robustness against contextual variations. The specific evaluation protocol and examples are detailed in Appendix A.2, demonstrating our guarantees from this perspective.

About Membership Inference Attacks (MIAs) in [2]: MIA has been critical in evaluating unlearning in traditional neural networks based models. However, a recent COLM 2024 paper titled "Do Membership Inference Attacks Work on Large Language Models?" demonstrates that MIA is not appropriate to assess unlearning in the context of LLMs. The paper concludes that MIA against LLMs is difficult for two key reasons:

(a) Data Imprint: The pre-training process of LLMs, especially with large datasets and single-epoch training, does not leave a strong imprint of individual data points, making it harder to discern membership.

(b) Similarity Between In and Out Members: The high similarity between in-domain and out-of-domain data, coupled with massive datasets, results in a fuzzy distinction between data members, complicating the effectiveness of MIAs.

Q2 and Q3: About comparison with random labelling

In [4], the method leverages a reinforced model to replace expressions and generate alternative labels to approximate predictions of a model untrained on the target data. While this effectively erases original information, it focuses on token-level unlearning and uses generic counterparts for relabeling. In contrast, UnStar emphasizes semantically rich anti-samples with misleading rationales. These rationales are carefully designed to introduce conflicting contextual associations, going beyond token replacement. Our approach maintains semantic coherence, ensuring the fine-grained unlearning of specific associations while retaining related knowledge. This difference lies in our reliance on structured rationale generation inspired by STaR and our iterative paraphrasing strategy, which tackles adversarial rephrasings more robustly.

We apologize for the oversight about the broken citation. It refers to:

Yao, Yuanshun, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv:2310.10683, 2023.

[Q8] Not copied verbatim from [5]

The related work subsection is Not a direct copy of [5]. While it covers similar topics, the content is clearly rephrased and appropriately cited. This is a misassessment, as no verbatim copying has occurred.

Content in our paper:

LLM Unlearning. LLM unlearning has garnered significant research interest as a means to improve privacy, enhance safety, and mitigate bias in large language models Lu et al. (2022); Kassem et al. (2023); Wang et al. (2023); Patil et al. (2023); Huang et al. (2024); Yu et al. (2023); Wu et al. (2023); Zhang et al. (2024a); Liu et al. (2024b); Jia et al. (2019)). The predominant approach utilizes gradient ascent to maximize prediction loss on the data to be forgotten (Jang et al. (2022); Yao et al. (2023)). Other techniques involve training the model to produce alternative responses, such as “I don’t know” (Ishibashi & Shimodaira (2023)), random labels (?), or predictions based on perturbed inputs (Eldan & Russinovich (2023)). Additionally, recent studies have investigated task arithmetic by incorporating specific instructions or in-context examples (Thaker et al. (2024); Pawelczyk et al. (2024)). However, these methods often lack the granularity required for fine-tuned control over what specific information is forgotten, which is where our approach—utilizing anti-samples—proposes a more refined solution.

Content in [5]:

LLM unlearning has attracted wide research attention as a way to enhance privacy, safety, and mitigate bias in LLMs (Lu et al., 2022; Kassem et al., 2023; Wang et al., 2023; Yu et al., 2023; Wu et al., 2023; Patil et al., 2023; Zhang et al., 2023a; Liu et al., 2024b; Jia et al., 2024; Ji et al., 2024; Huang et al., 2024). The mainstream method employs gradient ascent to maximize prediction loss on forget data (Jang et al., 2023; Yao et al., 2024a). Other methods train the LLM to generate alternative responses such as ‘I don’t know’ (Ishibashi and Shimodaira, 2024), random labels (Yao et al., 2024b), or LLM’s predictions on perturbed inputs (Eldan and Russinovich, 2023). Recently, some works have also explored task arithmetic (Ilharco et al., 2023; Barbulescu and Triantafillou, 2024; Zhang et al., 2023c) and training-free methods for LLM unlearning by prepending specific instructions or in-context examples (Thaker et al., 2024; Pawelczyk et al., 2023). Unlike existing works, we study the new targeted unlearning setting, where few existing methods can satisfy all criteria, but our causal intervention framework remains competitive in all of them.

We would like to point out that there are multiple fundamentally incorrect reasons (W1, Q1, W3, Q5, Q7) in the review of this paper:

[W1] Misdirected Criticism of Evaluation Metrics:

W1: Adhoc non-comparable evaluation methods

We have used the same evaluation metrics as the EMNLP paper [1]. [1] Liu Y, Zhang Y, Jaakkola T, Chang S. Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective. EMNLP 2024.

[Q1 and W3] Unfounded Confusion Regarding the Unlearning Setting:

W3: I am confused with the base definition and the goal

It is suggested that this work be compared with privacy-focused unlearning, such as that by Bourtoule et al. (2021). However, our work targets fine-grained unlearning in LLMs aimed at removing harmful or biased associations while preserving general knowledge. This is a different setting that requires distinct methodologies and evaluation criteria.

[Q5] Misinterpretation of Table 2:

Q5: Table 2 shows how semantically close the responses are. Does that make the task easier? How to verify that it did in fact unlearn the related knowledge.

It is a misinterpretation that Table 2 shows semantically close responses, implying that this makes the task easier. In reality, Table 2 highlights the distinction between fine-grained unlearning and targeted unlearning, demonstrating that while responses may appear superficially similar, they differ in terms of the specific knowledge associations unlearned. This showcases the precision of our method in selectively removing targeted content.

[Q7] Flawed opinion that WHP+ and RWHP are incomparable with UNSTAR:

Q7: Given that the results are very similar to that of WHP+ and RWHP, yet the impact on the model is different I am left wondering if we are just comparing apples to oranges.

It is suggested that comparing our results to WHP+ and RWHP is like "comparing apples to oranges." However, these methods also target unlearning in LLMs. Comparing our fine-grained approach to these baselines is essential to demonstrate the relative efficacy of our method. Suggesting that comparisons between unlearning approaches are inherently invalid undermines the purpose of empirical evaluation in this research area.

It is critical to receive a response, as other reviewers are making their decisions based on this review, which we believe contains multiple fundamental inaccuracies. These issues could significantly impact the overall evaluation of our work. We are ready to provide further clarifications if needed and urge you to address this matter promptly. Your response is essential to ensure a fair and accurate assessment of this paper. Thank you.

Response continued in the next comment...

Dear Reviewer,

We hope this message finds you well. Having thoroughly addressed all your concerns in our previous responses, we were surprised not to receive any follow-up from you. We kindly request your feedback on our rebuttals, as it would greatly aid us in understanding and improving our work.

Additionally, some other reviewers are waiting for your reply to make their own final assessment. We have also updated the Related Work section based on your concern. Please review the updated PDF file of the paper.

Thank you for your time and consideration.

Dear Reviewer,

We would like to briefly summarize our discussion and the current status:

1. Clarification regarding the context and the setting. -> DONE
2. Comparison with previous works on random labelling -> DONE
3. Explanation on semantically close responses in Table 2 -> DONE
4. Impact of about the decoding strategy -> DONE
5. On use of evaluation metrics -> DONE
6. Validating comparison with WHP+ and RWHP -> DONE
7. Rephrasing section on related work -> DONE
8. Clarification about relation with Ununlearning [1] & different ways of questioning in [2] -> DONE

Thank you for your reviews. We are waiting for the final rating of our paper. We would be grateful if you could raise the rating.

Thank you for your positive feedback on the strengths of our approach. We are glad that you appreciated the nuanced design of UnSTAR, which focuses on dissociating specific concepts while maintaining overall model integrity.

W1: Time Cost comparison of our method with other unlearning methods

We show the time cost comparison with three existing state-of-the-art methods in Table 1. Our UnStar demonstrates superior efficiency in unlearning in comparison with existing state-of-the-art methods, with relatively low runtimes, even for larger fact sets across various datasets. The results highlight its capability to handle both fine-grained and targeted unlearning tasks effectively. In contrast, SNAP struggles with agglomerative clustering, often resulting in prolonged runtimes without clear termination. WAGLE and NPO show comparable performance to UnStar, but with slightly higher time costs, making UnStar a more efficient choice for such unlearning tasks.

Table 1: Unlearning time cost comparison of our UnStar with SNAP, WAGLE, and NPO across Harry Potter, Peter Parker, and TOFU datasets. (time in seconds)

α: SNAP struggles to generate a sufficient number of questions forming distinct clusters via agglomerative clustering, often resulting in prolonged runtimes without clear termination.

φ: Struggle to work for fine grained unlearning

$: Omitted: expected to align with 400-fact results.

We also compare the ROUGE-L scores for UnStar with SNAP across three datasets: Harry Potter, Peter Parker, and TOFU in Table 2. A lower ROUGE-L score indicates better performance, as it reflects a higher degree of overlap between the generated responses and the ground-truth answers. For the Harry Potter dataset, UnStar significantly outperforms SNAP with a much lower score of 0.02997 compared to 0.14752. Similarly, in the TOFU dataset, UnStar achieves a better score of 0.04507, while SNAP scores 0.11362. In the Peter Parker dataset, UnStar also performs better, with a score of 0.20611, compared to SNAP's 0.24044. Overall, UnStar consistently provides more accurate and concise responses across all three datasets, demonstrating superior performance in terms of ROUGE-L.

Table 2: Unlearning results comparison with SNAP method.

SNAP: Choi M, Rim D, Lee D, Choo J. SNAP: Unlearning Selective Knowledge in Large Language Models with Negative Instructions. arXiv preprint arXiv:2406.12329. 2024.

WAGLE: Jia, J., Liu, J., Zhang, Y., Ram, P., Baracaldo, N., & Liu, S. WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models. NeurIPS 2024.

NPO: Zhang R, Lin L, Bai Y, Mei S. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868. 2024.

TOFU: Maini P, Feng Z, Schwarzschild A, Lipton ZC, Kolter JZ. TOFU: A Task of Fictitious Unlearning for LLMs. ICLR 2024 Workshops

W2: It also lacks comparison to other representation-based unlearning algorithms such as RMU.

The RMU code is riddled with bugs which could not be resolved in the given short time. We conduct the experiments on SNAP, WAGLE and NPO methods and show the results and comparison in Table 1 and Table 2.

W3: Any reason to use harmonic mean to combine evaluation metrics?

We chose the harmonic mean to combine evaluation metrics as it equitably balances the objectives of maximizing unlearning and minimizing retain-set performance degradation. By penalizing large disparities between metrics, it offers a more robust and fair assessment compared to the arithmetic mean, which can overemphasize one metric in cases of imbalance. This approach is adopted in [1] as well.

[1] Liu Y, Zhang Y, Jaakkola T, Chang S. Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective. EMNLP 2024.

Dear Reviewer,

Based on the above, could you please consider updating the rating of our paper?

Although RMU has some issues, I have seen a fair amount of papers using RMU as a comparison benchmark and I think it should be beneficial to include them.

I think the paper is generally sound, but given methods of similar nature has been explored albeit from slight different angles [1], and pending the discussion with Reviewer 9vHW I tend to keep my score.

[1] Liu Y, Zhang Y, Jaakkola T, Chang S. Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective. EMNLP 2024.

Dear Reviewer,

We would like to briefly summarize our discussion and the current status:

1. Comparison of time cost with existing unlearning methods -> DONE
2. Comparison to other representation-based unlearning algorithms such as RMU -> DONE
3. Reason to use harmonic means to combine evaluation metrics? -> DONE
4. Clarification about the similarity between RWHP [1] and UnSTAR -> DONE

Thank you for your reviews. We are waiting for the final rating of our paper. We would be grateful if you could raise the rating.

Thank you for your response. I think the use of "anti-samples" is not novel in itself. WHP itself uses the similar notion of anti-samples by substituting key vocabs. I agree that the fine-grained reasoning unlearning it novel and differentiate the method from other methods. Although I think this fine-grained unlearning lacks an applicable real-world use case analogue. I appreciate the authors adding RMU as a comparison. I think it is an interesting investigation of LLM unlearning/reasoning. I will raise the score to 8 but lower my confidence score to 2.

We are grateful for your overall positive remarks and persistent participation in rebuttal process and for raising your rating to 8. This has greatly helped us refine our work. We are glad you found the comparison with RMU valuable.

On the novelty of anti-samples:

We appreciate the reviewer’s observation regarding the use of anti-samples in WHP. While WHP introduces anti-samples through token substitution and alternative labels, these are primarily surface-level interventions targeting token-level changes in text data. UNSTAR, on the other hand, pioneers a reasoning-guided approach to anti-sample generation. By leveraging frameworks like STaR, UNSTAR employs anti-samples as structured interventions that target the model’s reasoning pathways, enabling fine-grained unlearning while maintaining related reasoning capabilities.

This distinction marks a shift from task-specific token substitution to a more generalizable framework for reasoning-guided unlearning. To the best of our knowledge, this is the first work to integrate anti-samples with reasoning-based methodologies for fine-grained unlearning, thereby tapping into the unexplored potential of anti-samples as a systematic mechanism in large language models.

We acknowledge the prior contributions of WHP and will clarify this distinction in the revised manuscript to better highlight how UNSTAR builds on and extends the concept of anti-samples.

Fine-grained unlearning and real-world applicability:

Below are several practical applications that highlight the relevance and importance of fine-grained unlearning in real-world scenarios:

1. Compliance with Data Privacy Laws: Removing specific personal data (e.g., "Donald Trump visited the Pentagon") to ensure compliance with GDPR/CCPA, while preserving general knowledge about Trump (e.g., "Trump is the President of the United States"). A recent example involves Asian News International (ANI) suing OpenAI in the Delhi High Court, accusing it of using its content without permission. [1]

2. Updating Models with Changing Information: Correcting outdated information (e.g., "Beyonce’s latest album is Cowboy Carter and not Renaissance") while retaining accurate related facts (Beyonce was born in Texas) [2]

3. Fixing Individual Factual Mistakes: Correcting errors like "Einstein invented the telescope" while maintaining relevant knowledge about Einstein and telescopes.

4. Customized Personalization: Forgetting deprecated product information in a company-specific LLM while retaining broader organizational knowledge.

5. Unlearning Sensitive Information: Erasing private or sensitive details (e.g., user-specific medical diagnoses) without impacting general domain knowledge.

Lines 033–044 of Introduction mentions motivations to unlearn. We will expand to mention these analogues in the revised manuscript. We hope this helps in better understanding the applicability of our method.

[1] News, I.S. (2024) Indian startup news on linkedin: Available at: https://www.linkedin.com/posts/indianstartupnews_openai-chatgpt-ani-activity-7264872675250982912-fnWk/ (Accessed: 03 December 2024).

[2] Hase, Peter, et al. "Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?." CoRR (2024).

Thank you for a detailed review! We are happy to resolve your concerns.

Q1 “Regarding the algorithm…”

The structure of Algorithm 1 is indeed designed to ensure comprehensive coverage of potential dissociations between concepts. Within the same set Q*, while it is possible that multiple incorrect answers aˉ may be generated for the same question, our approach ensures that fine-tuning is performed using each unique tuple (q*,aˉ,r) only once to avoid overfitting. In cases where duplicates might occur, they are filtered to prevent redundant updates, thus optimizing the fine-tuning process. The average number of repetitions is empirically low, as our paraphrasing and anti-sample generation methods aim to maximize diversity.

Q2 Regarding the experiments…

We show the impact of N on the experimental results in Table 1. The results of fine-tuning over 10 iterations, where the number of generated paraphrased questions (N) increases with each iteration, and the model is fine-tuned for 10 epochs per iteration. As N grows, the model's accuracy steadily improves, reaching 100% by the 10th iteration. The number of retained samples after filtering increases over time, indicating better data quality generation, while the number of filtered samples decreases. More comparative and runtime results are also provided in the common response to all the Reviewers.

Table 1: Impact of N on the experimental results

Q3 Regarding the randomness…

This is a valid concern, especially given the inherent variability and occasional hallucinations in outputs from LLMs. To mitigate the impact of randomness, we employ a controlled seeding mechanism using mlx.core.random.seed during both the paraphrase and anti-sample generation processes. This ensures that while the outputs are diverse, they remain reproducible across runs, helping to stabilize the unlearning performance metrics.

Additionally, we recognize that LLMs can sometimes generate semantically divergent questions or near-correct incorrect answers due to hallucinations. To address this, our filtering mechanisms in UnSTar effectively remove such samples, ensuring that only high-quality, relevant data is used for fine-tuning.

Another insightful point the reviewer raises is whether there could be instances where no anti-sample can be generated. While we did not encounter such cases in our experiments, we acknowledge that LLMs might struggle to generate anti-samples, particularly when justifying a difficult incorrect answer. In such cases, a human expert can provide the necessary justification, which can then be fed back into the LLM to generate additional anti-samples, further strengthening the unlearning process. We will include these clarifications in the revised manuscript to provide a clearer understanding of our approach.

Q4 & Q5 Format issue and source code

We apologize for the oversight.

Citation for “random labels” (?) : Yao, Yuanshun, Xiaojun Xu, and Yang Liu. "Large language model unlearning." arXiv preprint arXiv:2310.10683, 2023.

Citation on Lines 356-357: Liu Y, Zhang Y, Jaakkola T, Chang S. Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective. EMNLP 2024.

We will correct this in the updated version.

We understand the importance of reproducibility and transparency. The source code of our work will be publicly available upon acceptance of the paper.

Dear Reviewer,

We would like to briefly summarize our discussion and the current status:

1. Expected average number of repetitions in anti-sample generation. -> DONE
2. Impact of number of paraphrased questions on experimental results. -> DONE
3. Handling challenge of uncontrollability in randomness of generation. -> DONE

Thank you for your reviews. We are waiting for the final rating of our paper. We would be grateful if you could raise the rating.