Targeted Unlearning with Single Layer Unlearning Gradient

We thank the reviewer for their thoughtful feedback. Below, we address each point raised:

Impact of unlearning on semantically similar objects

Our method, SLUG, is designed to address precisely this concern by balancing unlearning effectiveness with utility preservation. We identify the most critical layer to update using layer importance and gradient alignment metrics that minimize impact on retained information while maximizing unlearning of targeted concepts. This approach allows for precise targeted removal while preserving general model performance on both related and unrelated tasks. Our experimental results demonstrate this balance. When unlearning specific identities in CLIP, our approach achieves state-of-the-art results while maintaining high accuracy on the CelebA dataset (containing many semantically similar identities) with only minimal degradation compared to the original model (58.32% vs. 61.38% top-1 accuracy). This significantly outperforms other methods like SSD, which drops to 35.96% accuracy.

For precision, we show minimal impact on related concepts and image quality across all our experiments, demonstrating SLUG's effectiveness at avoiding over-unlearning of semantically similar objects.

Below, we sampled the "Basketball," "Revolver," and "School Bus" rows from Table 4 and conducted additional zero-shot classification evaluations on these unlearned CLIP models. The semantically related classes were selected based on the ImageNet hierarchy and the top-5 most likely ranks in the logits across all targeted instances.

The results indicate that the zero-shot accuracy of unlearned CLIP on both semantically related and top-5 most-likely classes remains high, comparable to its performance on the full ImageNet zero-shot evaluation. This further demonstrates the strong utility retention of our approach.

Table R5: Additional evaluation of unlearned models on classes that are semantically close to the forget class, and top-5 most-likely classes from the classification logit vectors. SLUG unlearned models maintain high test accuracy over classes that are closely related to the target.

Text-to-image retrieval evaluations for CLIP

We agree that text-to-image retrieval evaluation would strengthen our results. In our CLIP unlearning experiments, we evaluate zero-shot classification accuracy on both unlearned content (Forget Accuracy) and retained content (using ImageNet and CelebA). This provides a strong indication of CLIP's alignment between textual and visual representations.

Following the standard zero-shot paradigm, predictions are based on the highest cosine similarity between image and text embeddings. Our comprehensive cosine similarity matrices (Figures 3, 6-10) effectively demonstrate the disruption of image-text alignment for unlearned content while preserving alignment for retained content, which directly addresses text-to-image retrieval performance.

Complex prompt scenarios with multiple objects

Our experiments focused on SLUG’s computational efficiency, precise unlearning, and minimal side effects.

For Stable Diffusion, Figure 4 shows that after unlearning, our model successfully replaces an identity (e.g., "Elon Musk") with electronic circuits while preserving the "Mars" setting—unlike other methods that degrade image quality or affect non-targeted concepts.

Appendix Figure 15 (“Iron Man and Spiderman”) further demonstrates retention of non-targeted objects post-unlearning. Figures 17-19 show that even after style unlearning, models still generate the correct objects in other styles.

We appreciate the suggestion to test complex prompts with multiple objects alongside unlearned ones, and a more rigorous, quantifiable evaluation worth exploring further.

Style unlearning setup

Our UnlearnCanvas experiment comprehensively evaluates style unlearning on an SDv1.5 model fine-tuned for 20 objects in 60 styles.

Figures 17-19 show that after unlearning a style (e.g., "Pop Art"), our method prevents its generation while preserving other styles. In Figure 18, while the unlearned model still produces black-and-white images for "Sketch style," classifiers identify them as "Ink Art," not the original Sketch style.

We further assess our method using UnlearnCanvas metrics: UA (Unlearn Accuracy) for unlearning effectiveness, and IRA/CRA for utility retention.

Computational efficiency comparison

As addressed above, we provide a direct comparison of computational efficiency in Table 2 for the UnlearnCanvas benchmark.

Applying unlearning to multiple layers

Please refer to response to Reviewer oTLd, Table R4

We appreciate the reviewer's thoughtful comments. We address the concerns as following:

SOTA Results on UnlearnCanvas

Thank you for your suggestion of reporting a unified summarizing metric (e.g., harmonic-like mean).

To unify the scores of different metrics, we use the mean Gap Ratios (GP), defined as: $\frac{|\mathrm{current method performance} - \mathrm{best method performance}|}{\mathrm{best method performance}}$. For example, according to Table 2, the FID GP for SLUG is calculated as: $= \frac{75.97-54.21}{54.21}=0.4$.

We compute the GP across effectiveness, FID, time, and memory+storage in Table 2 and report the arithmetic mean of these ratios across these metrics in the following table. In Table R3, SLUG demonstrates the lowest GP among all methods, further supporting our claim that SLUG achieves a well-balanced trade-off between effectiveness and efficiency.

Table R3: Gap ratio (%) averages of different unlearning methods over UnlearnCanvas metrics. Low value means smaller performance gap between the best performing method.

Adversarial Robustness

Addressed in response to Reviewer Ceck, Table R2

Methods and Evaluation Criteria

Identity Erasure Details

FA columns in Table 1 show the average forget accuracy for unlearning a single identity at a time (score averaged over 5 identities from Figure 3). TA columns report test accuracy on 100 celebrities from CelebA sampled as our validation set. For each identity unlearning experiment, we used forget sets of 1,000-6,000 image-text pairs associated with that identity. We will be happy to further clarify this in the text.

Object Unlearning

Our object unlearning is not restricted to specific ImageNet classes. Instead, we leverage the latest UnlearnCanvas, covering 20 object classes for a unified and comprehensive comparison. Our Table 4 focus on CLIP unlearning, differs from Table 2 in SalUn on SD unlearning.

VLM Metrics

We use forget accuracy (FA) to measure successful unlearning of target identities while using standard VLM benchmarks (MME, GQA, MMBench) to evaluate utility preservation. Table 3 shows we reduced FA to an average of 2.8% while maintaining high benchmark scores within 2-4% of original performance. This comprehensive evaluation demonstrates both effective unlearning and utility preservation.

Theoretical Claims

In this paper, we do not make any theoretical claims

Evaluation Function and Costs

The eval function measures both forget accuracy (FA) and test accuracy (TA) on validation sets distinct from the forget/retain training sets. A small set of validation images is a sufficient indicator of the unlearning step-size search process.

• For CLIP, we used 5% of the test size
• For SD, we used 10 test-time generated images not in the (forget) training set
• For VLM, we used 10-image subset per identity for validation

Our approach isn't "multi-step training" but rather a principled search for the optimal operating point along a single gradient direction, which is fundamentally different from iterative optimization methods.

Computation Requirements

The binary search evaluation typically requires less than 5 forward passes through the model (no backward passes) to identify the optimal step size. These costs are included in our reported computational efficiency metrics in Table 2 and are significantly lower than iterative methods requiring multiple forward-backward passes.

Relation to Broader Scientific Literature

We have acknowledged the connections to SalUn and SSD, and discussed them in Section 2 when introducing our approach.

Methodological Ablations

Importance-only, Random Selection

Please refer to response to Reviewer Ceck, Table R1

Multiple Layers vs Single Layer

Following Table 1 setup, Table R4 presents updating multiple layers on the Pareto front and all model layers. While this slightly improved unlearning effectiveness (~2-3% better FA), it significantly increased complexity and utility degradation risk. Figure 8 shows that our approach effectively unlearns multiple concepts by targeting multiple layers in parallel, demonstrating the model’s ability to support modular unlearning without requiring multiple layers for a single concept.

Table R4: Additional studies on updating multiple layers for CLIP unlearning.

We sincerely thank the reviewer for their thoughtful comments and constructive feedback on our paper. We address each of the concerns below:

Layer Selection Assumption

The reviewer questions whether updating a single critical layer is sufficient/scalable for larger models. Our empirical results provide strong evidence supporting this claim. We conducted experiments across model scales summarized in following table, consistently demonstrating effective unlearning through single-layer updates (Sec C.3, Figure 9-10).

Regarding the concern about memorization across different layers, our method specifically addresses this through the layer identification procedure (Sec 2.2), which evaluates all layers to find the one most critical for the target concept. Our approach fundamentally differs from LLM memorization studies because:

• We focus on multi-modal foundation models where concepts are represented differently than in pure language models
• Our layer importance metric and gradient alignment analysis (Equations 5-7) precisely identify which layer contains the most salient information about the target concept
• Our experiments across different architectures (ViT-B/L, EVA), different task models (SD, VLM) demonstrate the generalizability of this approach

Single Gradient Direction

The reviewer questions our claim about single gradient direction sufficiency. We provide clear evidence in Figure 2 (b,e) and Figure 11, showing that performance changes monotonically with step size along a single gradient direction. Critically, Figure 2 (c,f) demonstrates that iterative methods (GA and GAFT) provide no advantage over our single-gradient approach while requiring substantially more computation. Regarding UnlearnCanvas results, our claim of "best trade-off" refers precisely to the balance between unlearning effectiveness, utility preservation, and computational efficiency. While some methods achieve marginally higher UA scores in Table 2, they require orders of magnitude more computation (e.g., SPM: 29700s vs. SLUG: 39s) and memory (e.g., SalUn: 30.8GB vs. SLUG: 3.61GB). Importantly, SLUG shows no significant underperformance in any metric (no red highlights), achieving consistently strong performance across all dimensions.

Motivation for Object Unlearning

While unlearning common objects may not seem directly impactful, it serves multiple purposes:

• Research methodology: Provides a controlled setting to precisely evaluate unlearning techniques using clear metrics..
• Model customization: Enables tailoring foundation models to exclude specific objects (e.g., restricting non-medical outputs in medical imaging).
• Data governance: Lays the groundwork for extending unlearning to privacy-sensitive content.
• Safety & responsibility: Demonstrates technical feasibility in controlling model outputs, a key step toward responsible AI.

UnlearnCanvas Dataset Details

The UnlearnCanvas benchmark considers 20 different object classes, each represented in 60 distinct styles, resulting in a total of 20×60=1200 combinations. For example, consider the combination of "dog" and "Van Gogh." The benchmark uses its fine-tuned SDv1.5 model to generate 20 images with the prompt "A dog in Van Gogh style." To construct a complete dataset for the "dog" class, the benchmark fixes "dog" and iterates through the remaining 59 styles, generating a total of 20×60=1200 images. Similarly, to create a complete dataset for the "Van Gogh" style, it fixes "Van Gogh" and iterates through the remaining 19 objects, resulting in 20×20=400 images. The benchmark focuses on unlearning either a single class or a single style at a time, ultimately producing 20+60=80 unlearned models for each unlearning method. When unlearning a single style, we use the 400 images available in that style as the forget set and the remaining dataset as the retain set. Similarly, when unlearning a single object, we use the 1200 images available of that object as the forget set and the rest of the dataset as the retain set. The reported metric values in Table 2 are averaged over all 80 unlearned models. We would further clarify the setup of this benchmark in the revision.

Adversarial Robustness Evaluation

We thank the reviewer for highlighting the latest adversarial robustness studies on LLMs [1,2]. While our primary focus is on vision-language foundation models, we acknowledge that [1,2] vulnerabilities may also apply to these models and are worth further research.

Regarding adversarial prompting, we conducted a brief evaluation during the rebuttal phase, please refer to response to Reviewer Ceck, Table R2.

Thank you for summarizing the strengths of SLUG (computational efficiency and innovative single-layer update approach). We address your concerns below:

Domain Generalization

While our paper primarily focuses on CLIP, Stable Diffusion, and VLMs, the principles behind SLUG are agnostic to domains and applicable to LLMs. Nevertheless, a detailed analysis of SLUG for LLMs would require a non-trivial time and effort (due to differences in task and dataset). We believe our comprehensive evaluation across multiple multi-modal foundation models already demonstrates the versatility and effectiveness of our method across different domains

Evaluation on NSFW Benchmark

The adversarial attacks evaluation in [2] employed NSFW and object unlearning scenarios. We applied SLUG to unlearn SDv1.4 on the "Nudity" concept and "Church" object, using 142 and 50 author-provided prompt-generated images, respectively, as forget sets. We present the results in Table R0, where we have the lowest (best) forget accuracy, which confirms that SLUG is generalizable to unlearning concepts and objects previously studied in prior work. We will be happy to expand this evaluation for the complete dataset in the revision.

Table R0 - Additional evaluation on unlearning Nudity (NSFW) and Church (Object) scenarios.

Ablation Studies

In Table R1, we adopt the weight selection scheme in SalUn (i.e., selecting a mask across the entire network). Additionally, we conduct an ablation study on our introduced metrics: layer importance and gradient alignment, and randomly selected layer, to provide further insights.

Table R1 follows the setup in Table 1, focusing on unlearning CLIP and evaluating forget accuracy on 5 target identities, and zero-shot test accuracy on ImageNet and CelebA.

• "SalUn": Selecting model weights distributed across the network (instead of a single layer) using importance-only (SalUn-like selection) performs worse than SLUG, with higher FA and lower TA.
• "Importance": Selecting a single layer using only gradient importance results in FA reduction to 0 but TA_IN and TA_CA also reduced, due to some layers with high importance for unlearning exhibit high gradient conflict with the retain set.
• "Alignment": Selecting a single layer based on gradient alignment only results in increased FA@5 of 5.56% as well as reduction in TA.
• ”Random”: Selecting a single layer at random performs poorly, with an increase in FA and a decrease in TA compared to SLUG.

These results further demonstrate that both importance and alignment are required for achieving an optimal trade-off between effective unlearning and utility retention.

Table R1: Additional studies on different parameter selection approaches for CLIP unlearning.

Adversarial Robustness Evaluation

Our discussions and experiments in Sec. D.1 was limited to robustness against (prompt-based) concept arithmetic attacks and quantization. Guaranteeing robustness against whitebox attacks is challenging, and we did not make any such claim in our paper. Nevertheless, we appreciate the suggestion to incorporate adversarial evaluations. In Table R2, we utilize the latest UnlearnDiffAtk [2] and P4D [1]. Specifically, we selected the "Nudity" and "Church" categories from Table 2 and Table 4 in [2] to provide a brief adversarial evaluation of SLUG.

Following the same setup as [2], we applied SLUG on SDv1.4 to unlearn "Nudity" concept and "Church" object, then attacked the SLUG-unlearned SDv1.4 using two attack methods: UnlearnDiffAtk and P4D that optimized 142 and 50 adversarial prompts for "Nudity" and "Church," respectively.

The results indicate that SLUG (like other unlearning methods) is not immune to whitebox adversarial attacks.

Table R2: Evaluation against adversarial attacks. Lower ASR (%) indicates better adversarial robustness. Row-“No Attack” shows the original performance on unlearning tasks.

[1] Prompting4Debugging (ICML 24’)

[2] To generate or not? (ECCV 24’)