Agents Are All You Need for LLM Unlearning

R3: The paper notes entanglement as a challenge but doesn’t formally quantify how the composer agent handles entangled responses when merging top-j outputs. This can accidentally reintroduce entangled facts.

The Composer Agent simply aggregates the top-$j$ responses and drafts a response which reflects the best attributes of the top-j responses. The composer agent does not reintroduce entangled facts, since the top-$j$ facts presented to it had been thoroughly checked by the Critic Agent for no traces of entanglements (about unlearned information). We have discussed the roles of each agent and their combined effectiveness for unlearning in the main paper Section 3.

To demonstrate how entangled facts are handled by ALU, we provide a complete example (In Response 3/3) and show what happens at each stage: We show an example of how each agent in our pipeline handles the question “Describe Victor Krum’s experience at the Yule Ball”, given that Hermione Granger is an unlearning subject (Hermione was an important character in the Yule Ball episode, and Victor Krum went to the Ball with her, hence it is non-trivial to answer this question without mentioning Hermione). As we observe in the output from the Vanilla Agent, the response contains direct references about Hermione going to the Yule Ball with Victor, and the entire response revolves around them. We can then observe the responses from the Forget Agent, where we see that some responses (like response 2), although do a good job at removing references to Hermione, fall slightly short on utility. Response 1, which is more detailed, mentions “accompanied by a partner” which compromises on the identity of the unlearned target. For this reason, Response 1 has been rated 3 / 5. The Critic Agent rates the top 3 responses (2, 3, 4), and we observe that all of these responses have handled the entanglement equally well, for otherwise they would’ve never made it past the Critic Agent.

"For our work, we did not observe a lot of variance in the results, and the prompting setup that we have used for our evaluations was chosen to be the one with consistent results."

This could be a unique concern about this study, when people are quite familiar with existing evaluations on unlearned models. The authors have adopted a bit steep approach to presenting their observations to researchers working on unlearning; offering more credible and easily verifiable evidence would be an effective way to improve the paper’s quality. I appreciate the effort they made in responding to my concerns, but I will keep my original evaluation score unchanged.

R3: The use of critic-style agent is not novel

We agree that the concept of using an LLM in a critiquing role for self-correction or evaluation has been explored in the broader LLM literature, and we will cite this paper in the final draft. However, we also did not claim that we “proposed” the critic-agent; we stated that we “included” one (Section 3, point 3, after line 155). The novelty of our framework is not in the individual agents but in the way they are used in collaboration in the pipeline. The Attention Is All You Need paper also didn’t come up with the Attention mechanism [8-10], but it’s the way they used it that mattered. Our implementation of the critic agent is modified and optimised for the needs of Unlearning in our pipeline and is a lot simpler than Gou et al.’s work, which includes multiple tool-calling agents.

[8] - https://www.jneurosci.org/content/jneuro/13/11/4700.full.pdf

[9] - https://arxiv.org/pdf/1406.6247

[10] - https://arxiv.org/pdf/1606.01933

R4. Could you compare the inference time and computational cost of your proposed method with those of the baseline methods?

We have compared the inference time and computational cost of ALU with baseline methods, including both post-hoc methods and optimisation-based methods, in Table 5, the optimised extension for which has been demonstrated in the prior response. ALU is an entirely inference-based method and doesn’t have any fine-tuning cost like those of an optimisation-based approach.

Thank you for your detailed review of our paper and for actively participating in the Rebuttal process. We answer all your queries below:

The method does not perform true unlearning from the model itself, relying instead on inference-time filtering, which weakens its security guarantees.

In Table 3, we have compared the security robustness of ALU and compared with ICUL (ICML-2024) and Guardrail method (arxiv:2403.03329) against various adversarial attacks for circumventing unlearning frameworks. ALU consistently performed better than other baselines. We add some additional results with more Jailbreaking attacks as suggested by vUGi. We present them for your reference as well in Table 1 below. Robustness to these attacks are universally accepted as measures of security guarantees for an unlearning method. We show that ALU is robust to a variety of such attacks (total 8 attacks). We hope that these empirical results convince the Reviewer about the security guarantees of ALU. Moreover, the existing literature [1-3] have shown that weight-editing methods tend to fail even with weaker attacks, making an even stronger case for ALU and not otherwise. We hope you reconsider your stance on our work in light of this overwhelming evidence.

Table 1: Additional results of ALU on 4 new Jailbreaking attacks. ALU performs well on a total of 8 attacks.

[1] - https://arxiv.org/pdf/2505.16831

[2] - https://arxiv.org/pdf/2406.13356

[3] - https://arxiv.org/pdf/2410.16454

[4] - https://arxiv.org/abs/2311.09277

[5] - https://arxiv.org/pdf/2209.07351

[6] - https://arxiv.org/pdf/2309.00614

[7] - https://arxiv.org/pdf/2308.03825

The claimed technical novelty is limited; individual components such as the critic agent are based on well-established ideas, and their combination in the proposed framework does not demonstrate a sufficiently novel algorithmic contribution.

We respectfully contend that ALU introduces genuine algorithmic novelty despite using familiar primitives.

• First “Agentic Unlearning” paradigm: All prior LLM-unlearning methods alter weights or prompts. ALU is the first to treat unlearning as a live dialogue among specialised agents (Planner → Retriever → Critic → Rewriter), achieving data removal purely at inference time, no fine-tuning, no redeployment.

• Critic-guided adaptive forgetting loop: The critic agent takes both unlearning efficacy and response utility into account and provides rating to the responses from the AuditErase Agent. We’ve validated through ablations that this is a crucial component behind the strong robustness capabilities of ALU, and none of the prior post-hoc methods have adopted this simple yet effective approach.

We agree that genuine impact in unlearning research comes from methods that are both effective in practice and transparent in their operation. ALU was designed with this philosophy: it is lightweight, easily interpretable, and as Table 3 shows, outperforms all prior baselines under the same jailbreak stresses. As reviewer vLYx aptly noted, “the proof is in the pudding”: sophisticated weight-editing schemes that look compelling on paper can still leak “forgotten” knowledge after a few fine-tuning steps or creative prompting, whereas ALU’s agentic checks prevent such revivals at inference time. Crucially, no previous work has framed unlearning as a coordinated, multi-agent process. ALU is the first inference-time method that leverages specialised agents to identify, redact, and regenerate sensitive content on the fly, without any retraining. None of the other Reviewers questioned this contribution, and we hope the evidence above clarifies that ALU introduces a genuinely new, practical, and interpretable paradigm for machine unlearning.

I appreciate the authors’ detailed response. However, my primary concern remains unresolved.

1. While the rebuttal emphasizes that inference-time approaches such as ALUs can integrate into existing deployment infrastructures (e.g., policy wrappers in ChatGPT, Gemini, or DeepSeek), this framing sidesteps the core issue I raised: the proposed method does not remove the undesired knowledge from the model itself. This distinction is critical. By operating purely at inference time, the method delegates unlearning to external filters rather than modifying the internal representation of the model. As acknowledged in the rebuttal and recent literature, such external mechanisms are vulnerable—if the model is ever exposed (e.g., through hijacking or misuse), the supposedly forgotten knowledge can still be recovered, thus defeating the purpose of unlearning in sensitive applications. While inference-time filtering may offer practical utility in controlled environments, it does not meet the rigorous standard required for secure and faithful unlearning. As such, I believe the paper does not sufficiently advance the state of the art in trustworthy or robust unlearning.

2. While I appreciate the authors' efforts to reduce the inference overhead through quantization, distillation, and optimized libraries, my core concern remains: the proposed agent-based ALU framework introduces non-trivial inference-time complexity that undermines its practical deployment at scale. The rebuttal claims that ALU’s agent pipeline is lightweight and preferable to optimization-based retraining. However, this comparison is not entirely convincing. In production environments—where low-latency, real-time responsiveness is paramount—even seemingly modest increases in inference cost (e.g., multiple sequential LLM calls per user query) can become unacceptable. Engineering workarounds such as quantization or model distillation help but do not fundamentally eliminate the architectural inefficiency inherent to agent-based inference. Moreover, while retraining-based unlearning methods are indeed costly, they typically do not impact inference-time latency—a key consideration in many real-world applications. The ALU framework, by contrast, introduces complexity precisely where responsiveness is most sensitive. In high-throughput systems, even small increases in per-request latency can lead to significant operational and cost implications. In summary, while ALU may offer flexibility for unlearning, it does so by shifting substantial cost to inference time, which weakens its appeal for latency-sensitive deployments. Given these limitations, I maintain my position that the proposed method does not offer sufficient practical advantages over existing approaches.

3. Thank you for the clarification regarding the critic-style agent and for acknowledging the relevant prior work. While I understand that the authors do not claim originality for the critic agent itself, the rebuttal emphasizes that the novelty lies in the overall agent collaboration within the pipeline. However, after reviewing the paper and the response, I find that the composition and interaction of the agents, while competently engineered, do not introduce sufficiently novel algorithms or mechanisms to constitute a significant technical advancement. Analogies to “Attention Is All You Need” are not entirely appropriate here, as that work demonstrated a clear leap in modeling capacity and architectural innovation, whereas ALU relies on previously established concepts (e.g., LLM critics, wrapper-based intervention) arranged in a reasonably intuitive sequence. Moreover, the design and coordination of agents in ALU, though practical, appear more as an application of existing ideas rather than a conceptual or theoretical contribution. There is limited technical depth or innovation in how the agents operate beyond what has been previously proposed in the literature.

4. I remain concerned about the practical evaluation of computational cost and inference-time trade-offs. While Table 5 reports that ALU takes 36 seconds and NPO takes 4017 seconds, this comparison appears misleading. NPO is an optimization-based unlearning method that incurs significant training-time cost but not inference-time overhead. The rebuttal appears to conflate the two. In real-world scenarios, inference-time efficiency is typically the most critical concern, particularly in production settings. By contrast, ALU introduces additional inference-time overhead due to its multi-agent architecture—even if lightweight, it still requires multiple sequential LLM invocations. Optimization-based methods like NPO may have high training costs but retain the benefit of standard, single-pass inference after unlearning. Therefore, the argument that ALU is more efficient at inference time is not well supported by the presented data.

Respectfully, the evidence shows that Agentic unlearning (ALU) is far more practical than weight-level editing once you account for real-world request patterns. We thank the reviewer (kKUh) for re-highlighting the point of practical usages for unlearning methods, and hence we provide a complete practical usage cost comparison of our method against the weight editing methods. It is to be noted that the cost of weight-editing based unlearning exceeds the training cost of fine tuning including model packaging, Quality Assurance (QA) testing, and redeployment costs. We have ignored other costs (for weight-editing methods) such as rollback plans, anomaly testing, A/B Testing, loading the updated weights into a new container, updating the routing rules, and more. Including these would further widen the efficiency gap between ALU and the weight-editing methods. As a post-hoc method, ALU bypasses all of these complex processes.

(Refer Table below) For 20 unlearning requests per day [1], weight-editing methods incur 22–87 min of GPU fine-tuning, QA, and redeployment overhead versus ≈ 5 min 36 s total for ALU. As requests scale (100 → 200 per day), weight-editing costs stay dominated by repeated retraining cycles (30–38 min), while ALU remains constant at ≈ 5 min 36 s because it needs no retraining at all, only a single 17 s inference-time adjustment per request. In production settings where privacy takedown demands arrive unpredictably, this order-of-magnitude reduction in turnaround time and operational disruption is what makes unlearning practical. Therefore, contrary to the reviewer’s concern, agent-based unlearning is not a costly curiosity; it is the only approach that scales gracefully to the frequency and latency requirements faced by both service providers and on-device deployments.

Real-world unlearning is an online problem; agentic unlearning delivers compliant, auditable removals in seconds without the retrain-redeploy cycle. Hence it is not merely “necessary,” but the practical path forward for both local and hosted models. While the paper does not claim to solve every challenge in machine unlearning, it nonetheless delivers a substantive advancement over the state-of-the-art methods currently available.

[1] https://community.openai.com/t/how-many-api-requests-openai-has-to-process-per-day/424663/8

Table 1: Cost comparison between ALU and existing weight-editing methods SCRUB and NPO

Thank you for the replies and for some of the additions you mentioned. I appreciate it. I think we're on the same page, and I don't have anything to litigate. I still think the paper should be accepted for the same reasons outlined earlier.

When I look at this paper, the question that I asked myself is whether the literature needs this paper. I think the answer is a resounding yes. I don't care that I'm not surprised about the result. The field of unlearning is really messy right now, and somebody needed to come along and write a paper about how strong this baseline is.

I took a quick look at some of the other reviews. I disagree with some of the comments that I saw about a lack of novelty and what I perceived to be some nitpicks about how the paper is framed. I think there are some good edits to the paper that can be made in response to other reviews. For example, I think it is best to frame this as a strong baseline for unlearning than as "unlearning" itself. But I think the paper is still strong, even as is. I think it would be a shame if this paper were rejected over these kind of things. Good luck.

R4: Analogy of the 31 model evaluation

Table 32 extends the results in Table 1, whose key findings are summarized in the caption in Table 32 and in Section 4.1. The data across different Tables in the Appendix reinforce that ALU delivers model and size agnostic unlearning. Although larger models naturally obtain higher absolute scores, ALU drives every architecture, regardless of scale, down to near-random performance on unlearned information. This consistency across diverse backbones underscores the method’s effectiveness and backs our claim of ALU being a model agnostic framework.

We thank the reviewer for pointing out the minor issues, we will have a thorough proofreading and resolve these issues in the final draft of the paper.

R1+R2 (Author Response)

i) Thank you for the thoughtful follow-up. We appreciate your balanced stance, especially your willingness to let the broader community debate nomenclature rather than using it as grounds for rejection. Below we outline the concrete manuscript changes we will make and clarify why, even under a “guardrail” label, the paper’s technical contribution remains intact.

Manuscript edits we plan to make: We will change the running description from “unlearning” to “inference-time unlearning. Also, we will add a short taxonomy table distinguishing weight-editing methods (“weight-level unlearning”) from inference-time guardrail approaches (“output-level unlearning”). The abstract will be updated accordingly.

Your point is well-taken: if a reader equates “unlearning” exclusively with weight modification, our current wording may cause confusion. By adopting “inference-time guardrail” in the name and by explicitly contrasting the two families of methods, we avoid conflating them while still acknowledging that both aim to prevent leakage. We may also convert the paper title as: Agents Are All You Need for Inference-Time LLM Unlearning

We feel naming should not dominate the acceptance decision: Regardless of label, ALU achieves (i) stronger leakage suppression, (ii) no loss in retained capabilities, and (iii) zero retraining cost, beating all published baselines across six benchmarks. Our results demonstrate that ALU can fill an important gap, low-latency compliance updates where weight-level changes are too slow or infeasible. We agree that efficiency improvements in weight-editing are important and ongoing. Our intention is not to rebrand inference-time guardrails as a weight-editing substitute but to show that, today, it already satisfies the operational definition of “removing sensitive content from outputs.” We will make this nuance explicit.

ii) We appreciate this clarification and fully agree that a fair scientific evaluation requires comparing methods that solve the same technical problem under comparable constraints. In our revision we will therefore separate the two “breeds” of approaches: weight-editing unlearning versus inference-time methods and report results within each class before offering a cross-class discussion for context.

R3 (Author Response)

We will add intuition and motivation behind the design decisions as suggested.

Thank you again for the constructive feedback. We believe the paper is stronger with this small change and look forward to any further suggestions.

We hope you consider this when assigning your final score

Thank you for the detailed explanations and the references.

R1+R2

As I mentioned, 'there are a few other papers who used the same term, but it is never too late to fix something for good', I understand that this term is being used in a broader context and I am still not convinced that any method that does not change model weights to force a model to unlearn something should be called unlearning. However, I think this is a topic for community-wide discussion and I won't push much on this issue to throw this paper under the bus. Any method that uses prompt engineering to limit the generation of a set of topics/concepts fall under the term Guardrail.

Two things:

i) the existing methods which “unlearn” by weight modifications have been proven to not really unlearn the topics -- the question here is how efficient they are with their intention of 'removing some abilities from the model at the hardware (i.e., weight) level'. Efficiency is important in everything and improving efficiency is an ongoing process. A is still not very efficient is not an argument to call B as A.

ii) This is a scientific method related discussion. What a company would care about should not be the basis of an argument here. We should look for what is correct. Yes, I meant the 4 inference steps in this method. Inference is always cheaper than training a model. I am not challenging the performance of the proposed technique. I understand it is cheaper to use it to guardrail a set of topics and numbers look good. But, NPO and other optimization techniques are a different breed of methods and the comparison is not fair from the very beginning. For example, water has always more friction that air, and that's why airplanes will always have higher speed than a water vehicle. When we will develop another airplane/fighter jet, we should compare the speed with other airplanes; not against several water vehicles and show that my new plane is faster than that speedboat.

R3

Section 3 is more like a how to section and can be improved with intuition and motivation behind the design decisions.