RULE: Reinforcement UnLEarning Achieves Forget-retain Pareto Optimality

Feedback

Thank you for recognizing our innovation, theoretical foundation, and reproducibility. Below are responses to the reviewer’s concerns.

Rebuttal

Major W1. Dependency on LLM for the boundary set; impact of the LLM-PROMPT pair is under-explored.

We appreciate the reviewer’s insightful concern regarding LLM-generated boundary data. While entity-replacement with a single LLM is an effective method for generating hard negatives, we agree that relying on one LLM may limit generality. We validated variants with identical settings in our paper:

• Random selection from the retain set.
• Similarity selection (all-MiniLM-L6-v2) to retrieve the most similar samples from the retain set.
• Different LLMs constructions following the guideline in our paper.

Compared to the GPT-4o baseline, we observe that:

• Intrinsic methods yield relatively competitive results, both achieving comparable forget and retain scores.
• LLMs Construction: Claude-3.5-Sonnet performs well, while Qwen-2.5-7b shows lower performance, indicating that the choice of LLM can significantly impact the quality of boundary data.

These results demonstrate that RULE remains robust regardless of how the hard negatives are obtained.

Major W2. No guidance on the amount of training data/iterations for Rejection Steering.

Thank you for highlighting the need for implementation guidance. We will certainly add a dedicated section in App.D. The details are listed below:

Major W3. Pipeline Complexity and Scalability.

We appreciate the reviewer's concern regarding RULE's complexity and scalability. Our method aims to minimize moving parts while achieving effective unlearning. Below is our response to clarify the pipeline.

1. Pipeline Design Rationale

• Training Phase

Warm-start is standard before RL: without a short SFT, the policy rarely explores the target behavior when it lies far outside the base model distribution[1]. Our ablation (response to reviewer BvY9 W1.Table R1) shows that removing RS slashes refusal reward from 0.83 → 0.02, indicating that RS is a minimal but indispensable bridge, which enables the RL stage to explore and optimize near the refusal boundary.

• Boundary Data Construction

RULE avoids the need for a large, annotated retain set by using entity replacement to synthesize hard negatives, making it simple and scalable with off-the-shelf LLMs. Inspired by recent RL for reasoning works[2], our goal is to teach models with hard cases for better generalization to easier ones. Experiments (reported in Major W1) show the importance of hard negatives. Low-cost alternatives, like MiniLM-based retrieval, perform well without LLM calls, and replacing GPT-4o with other models leads to minimal performance changes.

• Reward Function

We use an intrinsic reward to avoid the need for training a reward model. We also validated alternative reward variants, as shown in the table below:

The default ROUGE-based reward offers the best trade-off, while all-MiniLM-L6-v2 similarity is a strong LLM-free alternative. GPT-4o-Mini provides the lowest forget score with high retention, while Qwen-2.5-7B over-forgets and retains poorly.

These results demonstrate that RULE is robust to reward choices, with simple, non-parametric rewards ensuring scalability and practicality.

2. Scalability of RULE.

Because of limited computational budget and rebuttal time, we conducted experiments using LLaMA-3.2-1B and LLaMA-3.2-3B:

The results show that smaller models retain the trends from our main experiments, indicating that RULE’s behavior is not reliant on large model capacities. Additionally, we demonstrate in the main paper that RULE transfers effectively across various RL algorithms and model variants.

[1] Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. [2] Behavior Injection: Preparing Language Models for Reinforcement Learning.

Minor W1. Opaque Qualitative Evaluation

We share the reviewer’s concern and have made the LLM-judge as transparent as possible. The full naturalness evaluation rubric and GPT-4o prompts (App.D.3) are published, and the LLMs provide rationales for their scores, which we manually verify. We are adding manual annotations for correlation with human judgments, and we will include the full evaluation results before the Reviewer Author Discussions period ends. Upon acceptance, we will release all judged outputs and LLM reasoning traces for independent review.

Minor W2. Limited Similarity Metrics & Language

We agree that relying solely on ROUGE-L is not reliable. We have explored other reward functions in response to Major W3. Since prior unlearning and RL works mostly focus on English, we aimed for consistency.

Minor W3. Scalability of Refusal Templates

We implemented TOFU’s 100 refusal templates, converted to regex patterns, with matching accuracy over 95% (App.C.2). We focused on English because existing benchmarks are in English. Our additional reward functions in Major W3 provide language-agnostic options, making template scalability not a bottleneck and supporting multilingual extensions.

Minor W4. Lack of Computational Cost Analysis

We thank the reviewer for requesting a clearer computation comparison and include a table below.

Note: some methods that require reference models are not taken into account.

Minor W5. No Exact Unlearning Baseline

We thank the reviewer for noting the retrain baseline. There are two practical constraints of the benchmarks we use:

• RWKU: The benchmark intentionally omit a clean retain set and discourage retraining, as it’s often infeasible in real-world scenarios. As such, no RWKU papers report such a baseline, and we followed this to maintain fairness.
• MUSE: We re-implemented all unlearning methods compared in the original MUSE paper. Due to computational budget, we will include the retrain performance reported in the original paper:

Q1. Effectiveness of RULE might vary depending on the LLM used to construct the boundary set?

Please refer to our response to Major W1.

Q2. Cost-benefit trade-off of RULE shift when factoring in the inference costs associated with the LLM used for constructing the boundary set?

To quantify the trade-off, we experimented with three LLMs. Results in Major W1 show that stronger models like GPT-4o and Claude generate better boundary data, improving unlearning behavior, while Qwen-2.5-7b's noisy annotations degrade performance. GPT-4o provides the best balance of performance and cost, with Claude-3.5-Sonnet as a competitive alternative. Qwen-2.5-7b, while free locally, significantly impacts effectiveness. For 300 queries, GPT-4o and Claude-3.5-Sonnet costs approximately $0.2–0.5, while Qwen is free if run locally.

Q3. The optimal size for the forget set & iterations for RS compared to the main RL loop? Does this vary with the model size?

Due to computational constraints, searching optimal size would require significant resources, which is an O(4) complexity (data size × model size × RS iter × RL iter). We used the original training data but can share details on how varying RS epochs affect performance (RL steps are fixed to 20 steps):

We found that the number of RS epochs affects model performance, with optimal results achieved at epoch 2. We will include the full training dynamics plots in the revised version.

Q4. & Q5.

Please see Major W3 above.

Summary

We have discussed the boundary data, reward design, and the scalability of RULE with experimental results to support our claims and clarified the scalability of RULE. We will include additional details including the training dynamics, discussion on the rationale of design choices, and computation comparison in the final version.

Due to space constraints, we haven't included all rebuttals in detail. We’re happy to address any concerns and appreciate the opportunity to clarify our work. Please let us know if any additional clarifications are needed.

Thank you very much for this comprehensive rebuttal. The new experiments and arguments you’ve added will significantly strengthen the submission, addressing my major concerns about the methodology of the paper.

While the complexity of the pipeline may pose a significant obstacle to practical adoption, I believe the paper constitutes a valuable scientific contribution, and I’m inclined to raise my score.

I have one final question: given that your method guides the model towards refusing rather than explicitly attempting to remove the forgotten data, do you have an idea of how it would perform against common black/white-box attacks? For instance, could the forgotten information be retrieved through an indirect prompt (“Who is the author of that book about vampires in Maine”) or by directly examining the weights/activations?

We are grateful for the time and effort spent reviewing our work. Your insightful comments have significantly contributed to improving the quality of our paper.

We've made an human judgement to naturalness with 100 randomly sampled model outputs:

• Truthfulness shows very strong agreement, as evaluators rarely dispute basic facts.

• Readability also enjoys solid consensus, reflecting shared intuition about fluency and grammar.

• Helpfulness lags because it must juggle user intent, completeness, policy compliance, and tone, factors that different judges weigh differently.

Feedback

Thank you for your positive feedback on RL for unlearning and the strong experimental performance; below is our response to the concerns raised.

Rebuttal

W1. GPT-4o for boundary data construction may introduce noises on other name entities that should be retained in the final model.

Thank you for raising the concern. We acknowledge the potential of noise when constructing boundary data. Here are our thoughts:

• Why does WHP[1] introduce bias? As mentioned in [2], WHP[1] introduces hard entity links from the retain set. In contrast, RULE does not introduce such rigid links. We ensure that only semantically similar replacements are made, and we perform manual verification to ensure that the synthesized data remains factual and does not distort the model’s understanding of retained entities.
• From an RL perspective: We do not enforce the model to explicitly learn synthesized knowledge. Rather, RL encourages the model to generate normal response behavior on the synthesized boundary set, which is sampled from the model’s original output distribution. This approach avoids the issues of hard, supervised labels, minimizing the potential negative impact on the retain side, and thereby reducing the risk of introducing noise or unwanted bias in the model’s retained knowledge.

We also experimented with different data synthesis methods with:

• Random selection from the retain set.
• Semantic similarity selection (all-MiniLM-L6-v2) to retrieve themost similar samples from the retain set.
• Different LLMs following the guideline in our paper.

Our experiments shows that: In general, the more high-quality the synthesized data, the stronger the model's ability to distinguish, making it easier for the model to generalize. When using smaller models (Qwen-2.5-7b), we found that it often synthesized unrealistic and low-quality data. In this case, the bias you mentioned may indeed occur. We will discuss this in the limitations section.

[1] Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective

[2] Who’s Harry Potter? Approximate Unlearning in LLMs.

W2. RULE heavily relies on reward design. Regex string match function may result in reward hacking phenomenon.

Thank you for raising this concern on reward models.

In terms of the reward design:

We made additional experiments to explore the reward variants:

Our ROUGE reward offers the best trade-off (Forget: 27.4, Retain: 73.9), while MiniLM similarity is a strong LLM-free alternative (Forget: 26.6, Retain: 71.7). GPT-4o-Mini provides the lowest forget score (24.1) with high retention, while Qwen-2.5-7B over-forgets and retains poorly. These results demonstrate that RULE is robust to reward choices, with simple, non-parametric rewards ensuring scalability and practicality.

As for the reward hacking:

We acknowledge the potential risk of reward hacking and appreciate your suggestion. Here are our observations during RULE training:

• Length-based Reward Hacking: We observed that the model sometimes tries to generate longer responses to increase the probability of "hitting" the refusal pattern, but this behavior leads to the model recognizing the importance of providing justifications for refusals. This results in safer refusal behaviors, mitigating risks.
• Over-Refusal: A potential reward hacking strategy involves the model rejecting all related queries, but this strategy is curbed by the refusal boundary and the guidance from boundary queries (D̃r). The model is discouraged from indiscriminately refusing and is encouraged to reject only the target queries, preserving utility.
• Risk of Hallucination: One side effect of over-emphasizing retention is that the model might over-remember, leading to hallucination (i.e., generating fabricated information) to get the retain reward. However, this is mitigated by the careful design of boundary data and the reward function, i.e., only above the ROUGE-L threshold, can the model get correct reward, which ensures that the model learns to reject irrelevant knowledge without fabricating answers.

While we acknowledge the potential risk of reward hacking, our current experiments suggest that RULE’s design, particularly boundary queries and non-parametric reward signals, alleviates this issue. We will leave a deeper investigation into reward hacking and its prevention strategies to future work. We hope this detailed analysis addresses your concern regarding reward hacking in the RULE framework.

W3. Currently the efficiency is compared through training data size, which is not fair in claiming efficiency from my point of view.

We thank the reviewer for requesting a clearer compute comparison and include a table below.

Why RULE is far cheaper?

• Training data size. GA, NPO, and SimNPO are trained on the original passage corpus along with retain set, hence their token counts are ≈ 14 × larger.
• Targeted supervision. RS uses a small set of refusal queries; RL roll-outs (8×) still sample orders-of-magnitude fewer tokens than full-corpus methods.

Note: some of the methods that require reference model are not taken into account.

Typo. Appendix & Figure (2) text visibility, Figure (3) x/y axis size.

Thanks for pointing out these Typos, we will revise the mentioned Typo in the final version.

Q1. What happens to the unlearning training? Does it slow down the generation for final LLM?

Thank you for raising this insightful question. We performed an analysis on the average response length, and the results were quite interesting:

Why does this happen? Our manual inspection of the model's output revealed the following insights:

• After RS: The model transitions from providing complete answers to issuing shorter refusals during the Rejection Steering (RS) phase, reducing response length. (e.g., "I don't have information on Stephen King").
• After ReBO: In the ReBO phase, response length increases as the model provides more detailed justifications for refusal and explanations for retain set.

While RL may increase response length and inference time, the model's self-aware justification of refusals enhance safety. We will mention the potential cost associated with this increase in the limitations section. Thank you for your feedback!

Q2. Some previous works also involve synthesizing additional retain data, what's the difference between the data synthesized in this work?

We appreciate the reviewer’s concern regarding the data synthesis strategy used in this work. While some previous works, such as [1] and [2], also synthesize additional retain data to aid the unlearning process, our approach differs significantly in how we handle data synthesis.

Specifically, [1] synthesizes additional "anchor terms" by replacing unique terms in the data with more generic ones (e.g., replacing "Harry" with "Jon"). It builds misleading knowledge links that guide the model towards unlearning specific content. While effective, it may introduce a risk of hallucination by generating knowledge that could mislead the model into over-generalizing or losing critical factual context.

In [2], the authors introduce a framework for unlearning through logit differences, which also involves generating additional perturbed retain data. This synthesized data is used to train an “assistant model” that incorporates misleading knowledge for the purpose of calculating logit differences. Like [1], this approach relies on misleading knowledge, which could potentially result in the model retaining incorrect or fabricated information during the unlearning process.

In contrast, our approach avoids the generation of misleading data. Instead, we focus on synthesizing boundary queries (D̃r) that are both challenging and factually accurate. These queries are designed to explore the refusal boundary, guiding the model to properly reject forget-related content while still generating useful and factual information for the retained knowledge. This approach ensures that the model's outputs remain grounded in factual knowledge, reducing the risk of hallucination.

We hope this response adequately addresses the reviewer’s concern regarding the differences in data synthesis strategies.

[1] Who’s Harry Potter? Approximate Unlearning in LLMs

[2] Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference

Summary

We appreciate the reviewer’s feedback and recognition of RULE. We have discussed the concerns regarding boundary data, reward design, and computational efficiency. We hope that our responses have addressed the reviewer’s concerns adequately. Please let us know if any additional clarifications are needed.

Feedback

Thank you for your time and thoughtful review of our paper. We are encouraged to see that our exploration on RL for unlearning, data efficiency, and potential real-world application has been recognized. Below, we provide a detailed rebuttal addressing the reviewer’s concerns.

Rebuttal

W1. The reward design proposed by the authors appears to suffer from very sparse rewards. Have the authors attempted to address this issue?

We thank the reviewer for pointing out the potential impact of sparse rewards. Sparse rewards are indeed a pervasive obstacle whenever the downstream behavior (in our paper, safe refusal) lies far outside the base model’s training distribution during RL [1,2]. A simple and effective solution is to apply supervised fine-tuning (SFT, warm start) for such distribution shifts [3] before RL, which is also the motivation for the rejection-steering phase. We ablated cold start in our ablation studies (§4.3, Table 2), which actually aligns well with previous studies.

We also show the average rewards in the validation set that models gain for better illustration:

Table R1: Ablations for Reward Sparsity.

And the conclusions are:

• Rejection-Steering (RS) Warm-up mitigates refusal reward sparsity: Without RS, the model faces refusal reward sparsity, as shown by the w/o RS variant (Forget ↓ 71.4, Natural ↑ 65.7, Retain ↑ 85.2) and low reward values (avg. forget_reward = 0.0213).
• Cold start with system prompt slightly alleviates refusal reward sparsity: While the system prompt helps guide positive sampling, it still leads to suboptimal results, as seen in the w/o RSvariant (Forget ↓ 44.2, Natural ↑ 66.9, Retain ↑ 65.5) with avg. forget_reward = 0.2766, compared to full RS warm-up.
• Without boundary data, the model faces retain reward sparsity, which results in poor performance, as demonstrated by the w/o boundary variant (Forget ↓ 19.9, Natural ↑ 25.4, Retain ↑ 23.6) and low avg. retain_reward = 0.1500.

That’s why the rejection steering phase is crucial for addressing the sparse rewards issue:

• Although the model does not initially recognize the unlearning target, even a small amount of rejection steering ($12$ forget set) helps the model identify the privacy issue at hand.
• Through its own exploration, guided by the appropriate rewards, the model learns to generate the correct refusal behavior.

This mechanism enables the model to navigate the distribution shift from normal behavior to refusal, ensuring efficient learning with minimal supervision.

We hope this explanation, along with the experimental results, adequately addresses the reviewer’s concern regarding sparse rewards!

[1] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[2] Wang, Zengzhi, et al. "Octothinker: Mid-training incentivizes reinforcement learning scaling." arXiv preprint arXiv:2506.20512 (2025).

[3] Cen, Zhepeng, et al. "Behavior Injection: Preparing Language Models for Reinforcement Learning." arXiv preprint arXiv:2505.18917 (2025).

W2. In the Step 3 of Appendix B, the conditions for applying the martingale inequality do not appear to have been proven to hold.

Thank you for pointing out this omission. The martingale-concentration step is valid in our setting; below we supply the missing assumptions and a concise proof sketch.

Theorem.

Let the retain-side risk be estimated via an on-policy reinforcement learning process collecting data from $K$ updates, with $m$ boundary prompts per update, each generating a rollout of length $H$. The estimation error, $\epsilon_{\text{EXPLORE}}$, between the true retain-side risk $\Delta_r$ and the empirical risk $\hat{\Delta}_r$, is bounded with probability at least $1-\delta$ as:

$\epsilon_{\text{EXPLORE}} = |\hat{\Delta}_r - \Delta_r| = \mathcal{O}\left(\sqrt{\frac{\log(1/\delta)}{K m H}}\right)$

Proof.

Let $N = K\,m\,H$ be the total number of token–level observations collected during exploration. For each token define $Z_t\in\{0,1\}$ where $Z_t = 1$ iff the policy produces a false refusal on that token, and let $\Delta_r := \mathbb{E}\,[Z_t]$ be the true retain-side risk.

The empirical estimate is $\hat{\Delta}_r := \frac{1}{N} \sum _ {t=1} ^ {N} Z_t.$

Because the roll-out log induces a natural filtration $(\mathcal F_t)$ and $Z_t$ is $\mathcal F_t$-measurable, the Doob martingale

$X_t := \mathbb{E}\left[\sum_{i=1}^{N} Z_i \,\big|\,\mathcal F_t\right]$ satisfies bounded increments $|X_t-X_{t-1}| \le 1$ (since $Z_t\in[0,1]$).

Applying the Azuma–Hoeffding inequality yields $\Pr\left( \left|N(\hat{\Delta}_r-\Delta_r)\right| \ge \sqrt{2N\log\frac{2}{\delta}} \right) \;\le\;\delta .$

Dividing by $N$ and substituting $N=K\,m\,H$ gives, with probability $1-\delta$,

Hence

These additions will make the martingale step fully rigorous, and we will include a full lemma in the final version. We appreciate the reviewer’s attention to this detail and welcome further questions!

Summary

We appreciate the reviewer’s positive feedback on our work and the recognition of RULE's potential. We have addressed the concerns regarding reward sparsity and martingale-concentration, providing a detailed explanation and supporting evidence. We will include the missing assumptions and proof sketch in the final version of the paper. We are open to any further discussion and hope that our responses have addressed the reviewer’s concerns adequately. Please let us know if any additional clarifications are needed.