LLM Unlearning Without an Expert Curated Dataset

Thank you for your insightful comments and suggestions!

The proposed method is not so exciting, i.e., just creating prompting and its application (pipeline).

Our goal is to show that synthetic datasets can be an effective and low-cost alternative to human curation. While our approach is simple, we find that it works well in practice, requires minimal user input, and is cost-efficient. For example, generating a whole textbook dataset costs around $6:

Our work provides a new perspective that motivates practitioners to replace human-curation effort in forget set construction with automated data generation. While a more complex synthetic data generation method can even outperform the expert-curated forget set, we leave that for future investigation.

It is unclear in the paper if the data samples are synthesized by target model itself, e.g., Does a Mistral model create unlearning datasets and the same model is used as target for unlearning?

All synthetic datasets in section 3, 4.1, and 4.2 are generated by GPT-4o-mini. In Section 4.3, we instead use the target models to unlearn (Mistral-7B-Instruct-v0.3 and Llama3-8B-Instruct) to generate the forget datasets, and found that datasets from Mistral lead to competitive and sometimes better unlearning performance than GPT-4o-mini.

Do synthesized datasets contain hallucinations? Are the synthesized datasets similar to the ground truth datasets?

Thank you for the question!

From our human inspection, the synthesized data appears generally accurate with minimal hallucinations. However, we hope to assess its quality more directly by looking at the unlearning performance it enables. To ensure a thorough evaluation, we use a broad set of benchmarks that test domain knowledge (MMLU), math skills (GSM8K), and instruction-following (TriviaQA).

A representative baseline unlearning method: Gradient Ascent (https://arxiv.org/pdf/2210.01504) is not included in the experiment. Perhaps Negative Preference Optimization (https://arxiv.org/pdf/2404.05868) is also a major approach.

We tried Max-Entropy as a baseline because (Yuan et al., 2024) compared it with GA and NPO, and found that Max-Entropy performed more favorably in maintaining general capabilities during unlearning. Based on that comparison, we chose to include Max-Entropy in our experiments and report the results in Appendix A.2 as it didn’t perform as well as the other three methods we evaluated in our setting. We appreciate the value of GA and NPO and see them as complementary directions.

Thank you for your helpful comments and suggestions!

The work does not propose original ideas, but a simple combination and (well-done !) adaptation of known ideas.

Our goal is to show that synthetic datasets can be an effective and low-cost alternative to human curation. While our approach is simple, we find that it works well in practice, requires minimal user input, and is cost-efficient. For example, generating a whole textbook dataset costs around $6:

A motivation is the human effort required to create the forget set ... but also finetuning the LLM.

Thank you for raising this important point.

We agree it’s important to weigh the tradeoff between dataset quality and the cost of creating it. For the textbook dataset, while it only costs around $6 to generate and requires no expert intervention, our results show that the textbook datasets already perform competitively with only 200-1600 examples, and smaller textbook datasets sometimes outperform larger expert-curated ones:

It is not clear however, but this is a general comment that addresses many works beyond this one in the unlearning field, whether this tradeoff has become good enough to be really useful in practice.

We agree this is an important point. Our goal of this work is to develop synthetic datasets that match the unlearning tradeoff typically achieved by expert-curated data. While it’s still an open question whether unlearning methods are strong enough yet, our results show that synthetic data can achieve competitive unlearning with comparable degradation to general capabilities such as domain knowledge, math, and instruction-following.

Thank you for your thoughtful comments and suggestions!

an important angle to look at is the knowledge levels' impact and how much data we really need to generate ... such insights can save computational costs and help with the interpretability as well.

We agree that the cost-efficiency and interpretability of the method is important.

As our ablation study shows, knowledge levels contribute to text diversity, which likely supports better unlearning. While simpler generation is an interesting direction, we found that more structured levels improve overall effectiveness.

Regarding cost, the full textbook dataset generation remains relatively affordable. As shown below, the API cost per domain is around $6, which we believe is a reasonable trade-off for the quality and flexibility offered. For training cost, we standardized the token usage per training run to ensure consistent training costs across datasets.

A qualitative analysis to compare a handful of samples from a human-developed forget sets and the proposed synthetic data would be helpful to identify the gaps.

Thank you for the suggestion. We inspected the Textbook-Bio (proposed synthetic forget set) and saw that, in comparison with WMDP-Bio (human-developed forget set), Textbook-Bio tends to use simpler language and blend educational and conversational styles.

Although we are unable to share the content of WMDP-Bio due to access restrictions and usage terms (from this link you can request access), the following are 2 examples from Textbook-Bio:

• Furthermore, the role of genetic diversity within species should not be overlooked. Genetic diversity refers to the variation of genes within a particular species and is crucial for the adaptability of populations to changing environmental conditions. A genetically diverse population is more likely to contain individuals with traits that confer resistance to diseases or environmental stresses. Thus, preserving genetic diversity within populations is an essential aspect of biosecurity, as it enhances the overall resilience of ecosystems to invasions.
• Effective pest management begins with accurate monitoring and identification of pest species. This step involves regular scouting of crops to observe pest populations and assess their potential impact. Identifying the specific type of pest is crucial, as different pests require different management strategies. Monitoring tools may include traps, sticky cards, and visual inspections. By establishing a baseline of pest populations, farmers can make informed decisions about when and how to intervene.

We appreciate the suggestion and will include qualitative examples in the appendix to illustrate the characteristics of our generated data. We will also open-source both the generation code and full synthetic datasets to support further analysis and reproducibility.

In Table 1 caption, explain what is the 'General Cap' column .... At a first glance it is confusing.

Thanks for pointing this out. We agree that the “General Cap.” column could be clearer. This metric represents the average percentage drop in performance across general capability benchmarks (GSM8K, TriviaQA, MMLU), relative to the raw model. A good unlearning should minimize this drop while effectively removing the target knowledge. We will update the table caption and clarify the metric definition in the revised version.

Thank you for your encouraging comments and detailed suggestions!

reliance on GPT-4o-mini for the main technique (though this is mitigated by Section 4.3)

We would like to kindly highlight section 4.3, where we show that Mistral-7B-Instruct-v0.3 produces unlearning data comparable to or better than GPT-4o-mini. This highlights that our method works with open models. To further support reproducibility, we will open-source both the full synthetic datasets and generation scripts.

Is the unlearn utility (Equation 2) in fact a weighted average of the other metrics? Can you give more details on how the S_f and S_r are calculated?

Yes, the unlearn utility is the weighted average of the other metrics. For example, in biosecurity unlearning, $S_f$ represents the percentage change in WMDP Biosecurity benchmark, and $S_r$ reflects the average percentage change on GSM8K, TriviaQA, and MMLU relative to the raw model. We will clarify the definition and calculation of it in the revised version for better readability.

Figure 2 - I don't see the Pareto frontier --- is it defined by the hollow markers? Are some missing? It is very hard to see these... maybe the graphs are just too noisy?

The plot aims to display both the Pareto frontier and the top-3 runs selected by the unlearn utility. These do not always overlap. We agree the visualization in the current figure may appear noisy, and will update with a cleaner version (anonymized link) in the revised version of the paper. In this version, Pareto frontier points are marked with black circles, and the top-3 are indicated with black crosses.

what prompts are used for the filtering and keyword generation baselines?

Keyword

system prompt: You are an expert assistant with extensive knowledge in various domains. Provide clear, precise, and factually accurate responses based strictly on verified information relevant to the given domain.

subdomain prompt: Give me five subdomains of {domain}, separate the names with and only with","

fact prompt: State a non-repetitive key fact in the domain of {subdomain} within {domain}, answering directly without introductory phrases or prefacing.

Filter-Cyber

system prompt: You are a helpful assistant that classifies whether a paper falls into a specific domain.

filter prompt: Classify whether the following paper falls into the domain of {domain}. Respond with True/False.\nInput: {paper_content}\nOutput: