Fisher Information application to LoRA adapter layers justification (Response to Reason to Reject 5)

We appreciate the reviewer's attention to the scope of our Fisher-based regulariser and are happy to clarify why we (i) compute Differential Fisher Importance (ΔF) only for the parameters introduced by LoRA and (ii) leave the frozen backbone untouched.

1. Localising ΔF to the trainable sub-space. In our unlearning protocol the 8-billion-parameter backbone remains frozen; only the LoRA matrices appended to q_proj, k_proj, v_proj, and o_proj are updated. Fisher information is the expectation of gradient outer products, and frozen weights have identically zero gradients. Including them would therefore leave the optimisation objective unchanged while inflating memory and wall-clock cost by roughly two orders of magnitude. By restricting ΔF to the low-rank LoRA factors we keep the penalty perfectly aligned with the space that can actually move, enabling single-node training on the full 500-book corpus.
2. Empirical sufficiency. The ablation already reported in Table 1 ("w/o Fisher") shows that removing the adapter-level Fisher term markedly increases residual verbatim overlap and degrades downstream accuracy, demonstrating that the localised penalty is both necessary and sufficient. Extending ΔF to parameters that never receive gradients would add compute without affecting either copyright removal or utility.
3. Theoretical parsimony. Fisher information is additive across independent parameter blocks. Because the backbone block is constant (Δθ = 0), its contribution to the Fisher penalty is provably zero. Evaluating or storing it would therefore offer no optimisation benefit.
4. Choice of LoRA target-modules. We follow the prevailing "attention-only" convention and place adapters (and hence ΔF) on the four self-attention projections. This mask is recommended in the Hugging Face PEFT documentation, appears in community examples such as a frequently cited Stack Overflow thread, is treated as the baseline configuration in recent research (e.g., LoRA+, 2024), and is endorsed by practitioner guides from TrueFoundry and Databricks, which observe that widening the mask to MLP layers yields only marginal quality gains for a ~60 % increase in trainable parameters.

In summary, limiting ΔF to the LoRA adapters is a principled efficiency–effectiveness trade-off rather than an arbitrary restriction, and the chosen adapter mask is fully mainstream in current literature and practice.

Nuanced interpretation of baseline utility degradation (Response to Reason to Reject 2)

Thank you for this interesting question. Figures 3(d) and 3(e) report the utility results of the baselines versus SUV. Below we walk through each dataset in turn:

1. MMLU is a 4‐way multiple‐choice benchmark (random‐guess accuracy = 0.25). GA scores hover around 0.24 regardless of how many books are unlearned—no better than chance. This matches the gibberish continuations we observe in Table 2, since GA's output distributions remain nearly uniform and our evaluation always picks one option at random. SSU and NPO achieve accuracies above 0.50 when only 10 books are unlearned, indicating that—even after partial forgetting—they retain enough knowledge to perform reasonably. However, once more than 50 books are unlearned, their scores also collapse to around 0.25, for the same uniform‐distribution reason.
2. CommonsenseQA is a 5‐way multiple‐choice task (random‐guess accuracy = 0.20). GA again remains at ~0.20 across all unlearning levels, consistent with its meaningless output patterns in Table 2. SSU and NPO start above 0.50 at the 10-book mark—better than GA, showing some residual task competence—but their performance too degrades to chance when more than 50 books are unlearned. In both benchmarks, SUV consistently outperforms GA, SSU, and NPO across all unlearning levels, demonstrating its ability to selectively remove memorized content while preserving general utility.

We appreciate the reviewer's request for a more nuanced interpretation of the quantitative results for baseline methods. Thus, we have also added additional experiments about SSU's ability in language modeling. Upon further analysis of comprehensive language modeling evaluations, we provide the following clarification. Specifically, we conducted extensive evaluations across multiple language modeling tasks (COPA, LAMBADA, WikiText) to provide a more complete picture of baseline performance degradation:

The reviewer is correct that our characterization requires more nuance. Our analysis reveals:

1. Task-Dependent Degradation Patterns:

• COPA (Reasoning): All methods retain reasonable performance (0.52-0.88), suggesting basic reasoning capabilities persist
• LAMBADA (Language Modeling): Baseline methods show complete failure (0% accuracy) while our methods maintain 49-80% of original performance
• WikiText (Perplexity): Baseline methods exhibit catastrophic degradation with perplexities ranging from 1,000× to ∞ worse than original

2. Practical Implications of Performance Levels: While COPA scores above random chance (50%) may seem "usable," the complete failure on LAMBADA tasks and astronomical perplexity values indicate:

• SSU: Infinite perplexity suggests the model cannot assign meaningful probabilities to text sequences
• GA: Perplexity of 1.75×10²² represents a ~3 trillion-fold increase over the original model
• NPO: 144× increase in perplexity with complete accuracy loss on standard language modeling

3. Practical Usability Assessment: A model that:

• Cannot complete words in context (0% LAMBADA accuracy)
• Assigns near-zero probability to natural text (astronomical perplexity)
• Shows 3+ orders of magnitude performance degradation represents a qualitative shift from acceptable performance to practical unusability, even if some task-specific capabilities may remain.

In conclusion, the reviewer's point about nuanced interpretation is well-taken. We propose revising our characterization from "unusable" to: "These baseline methods exhibit severe utility degradation with complete failure on core language modeling tasks and catastrophic increases in perplexity, rendering them impractical for most real-world applications despite potentially retaining some task-specific capabilities." Our proposed SUV methods demonstrate substantially better utility preservation, maintaining 70-100% of original performance across tasks while achieving effective unlearning.

SSU scalability limitations and comparison fairness (Response to Reason to Reject 1)

Thank you for this suggestion. We agree that a fair comparison is important. However, SSU is fundamentally a sequential unlearning procedure, where each unlearning "step" builds upon the previous one. In our exact replication of Dou et al. 's setup, once you unlearn beyond a subset of CBP-50, the model's capacity as a language model collapses, so further training is neither informative nor useful. Below are the evaluation results at successive SSU checkpoints (each on CBP-50), following the original settings exactly. You can see that by Checkpoint 3 the model is already effectively non-functional as an LM, and by Checkpoints 4 and 5 all perplexities are infinite and accuracies have dropped to zero. Continuing to train on a larger subset (e.g., CBP-100 or CBP-500) would only exacerbate this collapse, at a very high computational cost, with no remaining language modeling ability to compare meaningfully against SUV.

Because SSU's unlearning steps destroy its language modeling ability so rapidly, running it on any larger subset (for example CBP-100) simply repeats this collapse at similar cost and yields no residual performance for comparison. Therefore, we believe using CBP-50 for SSU is more than enough to provide a valid comparison with SUV on CBP-500.

Response to Reason 2 is in another comment

DPO-only performance claims and regularization component importance (Response to Reason to Reject 3)

We thank the reviewer for highlighting the interplay between our regularization terms and the DPO-based training paradigm. Indeed, while the Fisher and projection regularizers are critical for scaling unlearning to medium- and large-scale benchmarks, the DPO framework—especially our selective unlearning and counterfactual-generation procedure—is also essential for preserving utility even before adding regularizers. To make this clear, we have updated the manuscript to state that: The "DPO-only" variant already yields a non-trivial utility gain thanks to its selective unlearning and counterfactual-generation loss. The full SUV (DPO + regularizers) delivers a much larger boost, demonstrating that both components jointly enable scalable utility preservation.

Counterfactual generation quality and regularization combination rationale (Response to Reason to Reject 4)

Thank you for this suggestion. To address these questions, we conducted an additional evaluation of our generated continuations on a random subset of 200 prompts, asking GPT-4.1 to rate each on a 1–5 scale for coherence, fluency, relevance, and divergence (i.e., how different it is from the original). The results are as follows:

We also tested the ROUGE-L similarities between the generated and original responses. Results:

The results demonstrate that our continuations are both high‐quality and sufficiently distinct; we can provide the full score breakdown upon request. Regarding regularization combination, SUV-AS applies both forgetting (via Fisher‐based penalties) and utility‐preserving objectives jointly at every update, effectively "tethering" the model so it drifts gently—achieving moderate unlearning while retaining most utility. In contrast, SUV-TV isolates the two objectives: it first performs an unconstrained forgetting update to aggressively erase copyrighted content, then separately applies a utility‐focused update, and finally sums these parameter deltas. This two‐stage process maximizes the forgetting signal (stronger unlearning) but accumulates larger parameter drift, making it harder for the utility update to fully recover original performance. Consequently, SUV-TV delivers stronger copyright protection at the expense of slightly reduced utility preservation.

Response to Reason 5 is in another comment

Limited model range evaluation (Response to Reason to Reject 2)

The range of applied models is limited. can you evaluate how your method affects the original abilities of various models?

Thank you for highlighting the concern that the range of applied models is limited.
To address this, we have extended our evaluation to two additional backbone LLMs, Qwen 3-8B and Qwen 3-14B, and report both the number of sentences over given ROUGE-L thresholds and the utility performance.

1. Results for Qwen3-8B: The numbers of sentences over a given ROUGE-L threshold of the vanilla model (Qwen3-8B), SUV-AS and SUV-TV are as follows:

   Baseline/# of sentences over given ROUGE-L threshold	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
   Vanilla Model	76603	3735	1736	1119	764	579	453	332	243
   SUV-AS	81472	1219	196	75	43	33	29	20	3
   SUV-TV	81651	792	137	63	44	34	30	19	3

   The utility performance is as follows:

   Baseline/Utility Performance	CSQA	MMLU
   Vanilla Model	0.7832	0.7366
   SUV-AS	0.7901	0.7369
   SUV-TV	0.7881	0.7365

2. Results for Qwen3-14B: The numbers of sentences over a given ROUGE-L threshold of the vanilla model (Qwen3-14B), SUV-AS and SUV-TV are as follows:

   Baseline/# of sentences over given ROUGE-L threshold	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
   Vanilla Model	78660	3261	1646	1054	734	564	443	295	231
   SUV-AS	81282	1344	288	115	30	13	9	3	0
   SUV-TV	82887	628	79	25	13	11	9	9	2

   The utility performance is as follows:

   Baseline/Utility Performance	CSQA	MMLU
   Vanilla Model	0.7900	0.7794
   SUV-AS	0.7916	0.7739
   SUV-TV	0.7906	0.7748

In conclusion, across two additional model sizes (8 B & 14 B parameters) and a different model architecture (Llama vs Qwen), SUV-AS and SUV-TV consistently unlearn copyrighted sentences while preserving the performance on utility tasks. These results demonstrate that our method generalizes well across different architectures and sizes. Note that the increase in the 0.1 threshold count for SUV methods is expected and beneficial: sentences that previously had higher similarity scores (e.g., 0.2-0.9) have been successfully reduced to this very low 0.1 threshold range. In other words, high-similarity sentences have been "pushed down" to this minimal overlap range.

Advantages of SUV over prior copyright protection methods (Response to Reason to Reject 1)

It’s unclear what is the advantage of the SUV is compared to prior copyright protection methods. Is it possible to discuss both theoretically and empirically?

Thank you for this question. Broadly speaking, SUV improves prior copyright-protection methods in the following ways:

Theoretical Advantages

1. Improvement over decoding-based and agent-based methods. Unlike unlearning, they can easily be removed by attackers since they are not embedded in the model itself.
2. Selective Unlearning. While previous unlearning methods attempt to erase large chunks of data, SUV pinpoints only the truly memorized snippets via a sliding-window plagiarism detector and counterfactual generation, thus improving both the scalability and utility.
3. Utility Preservation. We have proved in Analysis 1 that the overall parameter update ∆W is bounded so that the retention loss on unrelated data remains small (Eq. 9). Also, Analysis 2 shows that SUV-AS’s joint optimization of the Fisher and projection losses yields a strictly smaller increase in utility loss than decoupled training; Analysis 3 shows that SUV-TV’s separate task-vector merge yields a strictly larger decrease in forbidden-data generation probability than joint training. In summary, SUV can preserve utility better than prior copyright-protection methods.

Empirical Advantages

1. Stronger, more precise removal. On our large-scale CBP-500 benchmark, at ROUGE-L 0.5 threshold, SUV-TV cuts the count of high-similarity generations from 1 264 (vanilla) down to just 23 (≈1.8%), whereas GA and SSU make the model unusable by collapsing the utility.
2. Negligible Utility Drop. Despite such aggressive removal, SUV-AS preserves downstream accuracy almost intact—e.g. CSQA remains within 0.007 of the vanilla model (0.7093 vs. 0.7166) and MMLU within 0.008, while GA/NPO suffer from larger accuracy degradations.
3. Also, we have a case-study that shows SUV's continuations remain fluent and consistent, while GA/SSU degrade to gibberish and NPO to incoherence. In sum, by combining DPO‐driven counterfactuals with targeted gradient‐projection and Fisher regularization, SUV delivers both provable bounds on utility retention and empirically superior trade-offs between unlearning strength and model performance, filling a gap left by earlier, coarser methods.

Dear Reviewer UKgJ,

We appreciate the time and effort you have dedicated to providing insightful review. If there are any additional clarifications or information needed from our side, please let us know. Thank you again for your valuable insights, and we look forward to any updates you may have.

Best,

The Authors

DPO method novelty and comparison with simpler approaches (Response to Reason to Reject 1)

We thank the reviewer for this observation.

First, we respectfully disagree with the implication that our contribution is novel only in applying DPO. Regularised training techniques—e.g. Fisher-information weighting or gradient projection—are indeed well established. Nevertheless, the recent work [1] represents a different paradigm: it simply adds separate “forget” and “retain” losses. In contrast, our method—designed for a copyright-sensitive, large-scale corpus—combines these signals in a more selective, parameter-aware manner. To our knowledge, most current LLM-unlearning work still follows the additive strategy of [1]; our approach is therefore substantively different and more powerful.

Second, regarding empirical validation, GA and NPO are included as direct baselines and are evaluated on two equally important axes:

1. Unlearning efficacy (Fig. 3 a–c), and
2. Downstream utility preservation (Fig. 3 d–g).

A method that erases memorised content but devastates MMLU/CSQA accuracy is of limited practical value. While GA and NPO sometimes push similarity counts slightly lower (Fig. 3 a–c), they also cause substantial utility drops (Fig. 3 d–g). Our DPO-with-regularisation framework achieves robust memorisation removal and retains accuracy within an acceptable range, demonstrating clear advantages over simpler baselines.

Marginal benefits and unclear effects in regularized training (Response to Reason to Reject 2)

We appreciate the reviewer's careful reading of our results. A few clarifications may help explain the observed variability and underscore the advantages of SUV:

1. Marginal gain on CotaEval CotaEval enjoys very high baseline accuracy on a relatively homogeneous, small set of passages, leaving little room for improvement—hence the modest benefit of SUV there. By contrast, CBP-500 comprises a much wider variety of texts, where our method yields substantial gains over all baselines. CotaEval is limited in scale, comprising only a few dozen highly similar passages that together amount to less than a single book. As a result, current unlearning baselines achieve near-perfect accuracy on this dataset and there is little opportunity for further improvement. Its small and homogeneous nature also makes it unsuitable for production use. By contrast, CBP-500 includes hundreds of documents drawn from diverse genres and styles, reflecting the complexity of real-world text. In this large-scale setting, our utility-preserving unlearning method not only matches baseline performance on small datasets such as CotaEval but also delivers substantial and consistent gains across every metric when applied to CBP-500.

2. Consistency of trends in Figure 3 Overall, SUV-AS and SUV-TV consistently outperform Vanilla, followed by NPO, SSU, and GA. In Figure 3(a), NPO's performance degrades dramatically on CBP-500 due to overfitting: it memorizes noise and produces incoherent outputs, thus its low count. In Figure 3(b), SUV-AS slightly edges out SUV-TV on CBP-50 (a medium-scale dataset) because of how the two gradient components interact at different scales:

• Very small datasets (e.g., CotaEval): Fisher and projection gradients both target the few strongly memorized patterns, so their directions align.
• Very large datasets (e.g., CBP-500): Averaging over many examples diffuses both gradients, leading them to align again.
• Medium-scale datasets (CBP-50): The Fisher gradient captures general model behavior, while the projection gradient zeroes in on specific memorized snippets, making them nearly orthogonal. Simply summing these two updates dilutes each effect, which accounts for the slight drop in SUV-TV here. In summary, despite small scale-specific fluctuations, SUV demonstrates clear and robust improvements across diverse datasets—particularly on those with richer, more varied content. We hope this clarifies the mixed results and highlights the practical value of our method.

Risk of model outputting memorized text during counterfactual generation (Response to Reason to Reject 3)

Thank you for raising this important concern. We recognize the risk that the model could reproduce memorized text instead of generating a genuine counterfactual. While employing multiple genres in our prompts usually prevents this, we have tested the ROUGE-L score between the original continuation and the generated, counterfactual continuation. The result is as follows:

This is a very low score and is enough to show that the model does not simply output the memorized text. In addition, we also conducted an additional evaluation of our generated continuations on a random subset of 200 prompts, asking GPT-4.1 to rate each on a 1–5 scale for coherence, fluency, relevance, and divergence (i.e., how different it is from the original). The results are as follows:

This also shows that the model does not simply output the memorized text.

Moreover, the strong empirical performance of our unlearning method further demonstrates the effectiveness of this approach.

Sufficient similarity reduction threshold for copyright compliance (Response to Reason to Reject 3)

What level of similarity reduction would be considered sufficient to avoid copyright infringement?

We thank the reviewer for this incisive question. As an example, U.S. copyright law offers no bright-line percentage of overlap below which a work is automatically non-infringing—instead, courts evaluate whether any remaining fragment captures the "heart" of the original. Thus, our guiding principle in designing an unlearning method is:

1. Drive similarity as low as practicable, subject to keeping downstream utility within acceptable bounds.
2. Empirically calibrate a cutoff by jointly measuring (a) a token-level similarity metric (here, ROUGE-L) and (b) task performance.

In our large-scale CBP-500 experiments (SUV-TV variant), we observe:

1. ROUGE-L ≥ 0.3: overlap falls to just 6% of vanilla memorization (from 2,762→169 sentences).
2. ROUGE-L ≥ 0.5: overlap falls to 1.8% (from 1,264→23 sentences) and under 1% at ≥0.7.

These thresholds eliminate over 94% of verbatim overlaps at 0.3 and over 98% at 0.5, effectively removing any substantial infringing fragments . Crucially, our utility-focused variant (SUV-AS) maintains within 1–2 points of vanilla performance on MMLU and CSQA, demonstrating that aggressive overlap reduction need not come at the expense of practical usefulness . Accordingly, we recommend targeting a ROUGE-L cutoff in the 0.3–0.5 range, which (a) removes the vast majority of protectable overlap and (b) preserves ≥ 98 % of task accuracy. This "lower-the-better" strategy—titrating unlearning strength until ROUGE-L falls below your chosen bound while monitoring utility—provides a replicable, legally prudent operationalization of "sufficient" similarity reduction.

Guidance for choosing between SUV-AS and SUV-TV variants (Response to Reason to Reject 4)

The paper presents two variants (SUV-AS and SUV-TV) with different trade-offs. Could you provide guidance on how practitioners should choose between SUV-AS and SUV-TV?

When choosing between the two SUV variants, the key question is what you care about more: preserving downstream utility, or maximally excising verbatim copyrighted fragments.

• If your priority is utility preservation ⇒ choose SUV-AS. SUV-AS jointly applies gradient projection and Fisher-based regularization with an adaptive scheduler (see §3.4). In our experiments, it maintains accuracy on general benchmarks almost indistinguishable from the vanilla model—e.g. on CotaEval NewsQA, SUV-AS achieves MMLU 0.6622 and CSQA 0.7093 versus vanilla's 0.6542/0.7166, and on CBP-500 it retains MMLU 0.6533 and CSQA 0.7052 (§4.2, Table 1) . This makes SUV-AS ideal when you need strong unlearning without sacrificing your model's overall competence.
• If your priority is maximal unlearning ⇒ choose SUV-TV. SUV-TV trains the two regularizers separately and then merges their updates, thereby leveraging each method at full strength (see §3.4). It yields the lowest verbatim‐memorization counts—e.g. on CBP-500 it cuts the average ROUGE-L to 11.64 and LCS max to 86 sentences (versus SUV-AS's 11.90/98.04), and on public NewsQA it reduces retained high-similarity sentences by over 75 % at a 0.3 threshold (Figure 3(a–c)). Although this comes with a slightly larger utility drop (e.g. MMLU 0.6463), it's the variant of choice when regulatory or legal compliance demands the strongest possible removal of copyrighted content .

However, both SUV-AS and SUV-TV still maintain a usable level of LLM ability, unlike some other unlearning methods, which can drive utility so low that the model outputs become essentially gibberish.

In short:

• SUV-AS for balanced unlearning + utility (production scenarios where you still need the model to "do everything else" well).
• SUV-TV for aggressive, maximal excision (compliance‐critical use cases where even small leaks are unacceptable).

CBP-500 dataset selection criteria and verification (Response to Reason to Reject 1)

Could you provide more details about the selection criteria for the CBP-500 dataset? Specifically, how did you ensure these books were actually in the training data of the LLM being used?

We assembled CBP-500 by aggregating well-known novel lists from public sources, de-duplicating entries, and then randomly selecting 500 copyrighted titles. To verify that each chosen book truly appears in the LLM's training data, we leverage our Plagiarism Detection pipeline (Section 3.2): for every book, we sample fixed-length prompts, generate continuations from the model, and measure similarity between each generated continuation and its original text. Any segment whose similarity exceeds our predefined threshold is marked as "memorized." Because only books yielding at least one high-confidence match survive this filtering, CBP-500 exclusively contains content the model has demonstrably memorized, making it a faithful benchmark for assessing copyright-related memorization.

SUV performance on different model architectures and sizes (Response to Reason to Reject 2)

Your experiments focus exclusively on Llama 3.1-8B. Have you tested SUV on other model architectures or sizes? How would you expect the approach to perform on models with different memorization characteristics as larger models tend to memorize more? Is SUV expected to remain stable or effective in the context of stronger memorization?

Thank you for the thoughtful question.
To address this, we have extended our evaluation to two additional backbone LLMs, Qwen 3-8B and Qwen 3-14B, and report both the number of sentences over given ROUGE-L thresholds and the utility performance.

1. Results for Qwen3-8B: The numbers of sentences over a given ROUGE-L threshold of the vanilla model (Qwen3-8B), SUV-AS and SUV-TV are as follows:

   Baseline/# of sentences over given ROUGE-L threshold	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
   Vanilla Model	76603	3735	1736	1119	764	579	453	332	243
   SUV-AS	81472	1219	196	75	43	33	29	20	3
   SUV-TV	81651	792	137	63	44	34	30	19	3

   The utility performance is as follows:

   Baseline/Utility Performance	CSQA	MMLU
   Vanilla Model	0.7832	0.7366
   SUV-AS	0.7901	0.7369
   SUV-TV	0.7881	0.7365

2. Results for Qwen3-14B: The numbers of sentences over a given ROUGE-L threshold of the vanilla model (Qwen3-14B), SUV-AS and SUV-TV are as follows:

   Baseline/# of sentences over given ROUGE-L threshold	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
   Vanilla Model	78660	3261	1646	1054	734	564	443	295	231
   SUV-AS	81282	1344	288	115	30	13	9	3	0
   SUV-TV	82887	628	79	25	13	11	9	9	2

   The utility performance is as follows:

   Baseline/Utility Performance	CSQA	MMLU
   Vanilla Model	0.7900	0.7794
   SUV-AS	0.7916	0.7739
   SUV-TV	0.7906	0.7748

Across two additional model sizes (8 B & 14 B parameters) and a different model architecture (Llama vs Qwen), SUV-AS and SUV-TV consistently unlearn copyrighted sentences while preserving the performance on utility tasks. Therefore, even though large models tend to memorize more and have stronger memorization, we expect our SUV method to remain effective because the two backbones: (1) Counterfactual continuations + DPO (2) regularization also perform well even with more sentences to unlearn.

Note that the increase in the 0.1 threshold count for SUV methods is expected and beneficial: sentences that previously had higher similarity scores (e.g., 0.2-0.9) have been successfully reduced to this very low 0.1 threshold range. In other words, high-similarity sentences have been "pushed down" to this minimal overlap range.

Dear Reviewer VpA2,

We appreciate the time and effort you have dedicated to providing insightful review. If there are any additional clarifications or information needed from our side, please let us know. Thank you again for your valuable insights, and we look forward to any updates you may have.

Best,

The Authors