SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders

Thank you for the positive review of our work, we would like to explain and address the comments and remaining weaknesses with the help of additional tables and figures provided in the anonymized link Anonymous github:

The paper lacks a comprehensive ablation study on the intervention mechanism. Specifically, it is uncertain if the chosen negative multiplier is the optimal way to modify the SAE features or if alternative interventions, such as zeroing out the feature or dynamically set multiplier might yield better results.

Thank you for pointing out this omission. Our unlearning method has two main parameters: the number of features selected for unlearning (percentile) and the negative multiplier. We comprehensively analyze their impact in the additional results. As shown in Figure 10, our method is robust to these parameters, with a broad range of values—apart from extreme cases—yielding comparably high performance. Following the reviewer's suggestion, we evaluate the alternative intervention of zeroing out features (when multiplier = 0), observing that we need negative values of the multiplier in order to obtain satisfying unlearning results.

It remains unclear whether the removal of one concept (e.g., Husky) may inadvertently affect the generation of similar or neighboring concepts (e.g., Chihuahua). A deeper analysis of collateral damage on related concepts after unlearning would strengthen the study.

To provide a more comprehensive analysis of the unlearning effect on similar concepts, in Fig.9, we present how the unlearning of a specific object affects all of the other objects evaluated in the UnlearnCanvas benchmark. We can observe some cases where the unlearning of, for example, dogs can lead to the degradation of the quality of cats. This observation is consistent across evaluated methods as presented in appendix D.3 of the original UnlearningCanvas benchmark.

For practical deployment, it would be valuable to evaluate the method’s performance when a large number of concepts (e.g., all 50 concepts in the UnlearnCanvas benchmark) are removed simultaneously. It is unclear whether the method can retain its performance under such extreme scenarios.

Thank you for this excellent suggestion. We run additional experiments to evaluate such an extreme approach, where we simultaneously unlearn 49 out of 50 styles present in the UnlearnCanvas benchmark (leaving one style out to evaluate the quality of its preservation). For 3 randomly selected combinations, we observe almost no degradation in SAeUron performance. For unlearning of 49/50 styles simultaneously, we observe UA: 99.29%, IRA: 96.67%, and CRA: 95.00%. Those results highlight the unprecedented degree of unlearning precision with SAeUron.

The reported 10% computational overhead during inference is relatively high compared with alternative approaches such as ESD or SalUn. This increased overhead may limit the practicality of the method in time-sensitive applications.

Thank you for highlighting this important limitation of our approach. Your feedback prompted us to re-evaluate our implementation. Since the SAE is a small linear model, its forward pass adds only two matrix multiplications, which should be negligible compared to the diffusion UNet model. Upon further analysis, we discovered that the 10% computational overhead was primarily due to an inefficient implementation: we were redundantly recalculating the same feature selection for unlearning during each pass through the UNet model. By computing this information once and storing it in memory, we have now reduced the computational overhead to just 1.92%.

Thank you for noticing that our method is one of the few studies that have applied SAE to achieve state-of-the-art results in practical applications. To further strengthen this claim, in Tab.2 of additional results, we present how SAeUron can also be used in an even more practical use-case of nudity unlearning, where it also achieves state-of-the-art results.

We hope that our answers provided meaningful clarification to your questions and addressed all of the weaknesses. If true, we kindly ask the reviewer to consider raising their score.

Link to additional tables and figures: Anonymous github

Nudity evaluation

We thank the Reviewer for suggesting showcasing SAeUron's strength in the real-world use case of unlearning nudity. To do this, we evaluated our method on an established I2P benchmark consisting of 4703 inappropriate prompts. We train SAE on SD-v1.4 activations gathered from random 30K captions from COCO train 2014. Additionally, we add to the train set prompts "naked man" and "naked woman" to enable SAE to learn nudity-relevant features. SAE is trained on up.1.1 block, following all hyperparameters used for class unlearning in our submission. We use our score-based method to select features related to nudity, selecting features that strongly activate for "naked woman" or "naked man" prompts and not activating on a subset of COCO train captions.

Following other works, we employ the NudeNet detector for nudity detection, filtering out outputs with confidence less than 0.6. Additionally, we calculate FID and CLIPScore on 30k prompts from the COCO validation set to measure the model's overall quality when applying SAE. As shown in Tab 2, SAeUron achieves state-of-the-art performance in removing nudity while preserving the model's overall quality. This highlights the potential of our method in real-world applications.

Relation to [1]

Thank you for pointing out this interesting paper. We respectfully note that it has been submitted to arxiv after the conference deadline. Nonetheless, while both works intervene on SAE's latent space to remove undesirable concepts, there is a crucial difference in that [1] trains SAE on activations of the text encoder while we directly apply our approach to the diffusion UNet. Since both works utilize SAEs for similar use cases, we will gladly add this paper as concurrent work in the final version of our submission.

[1] Kim, Dahye, and Deepti Ghadiyaram. "Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable Generations."arxiv2025.

I believe that the two-phase approach is challenging in terms of training time for unlearning.

To showcase the efficiency of our approach, we measure the time needed to present the results of unlearning in Figs 11 and 12. We evaluate the scaling of the SAeUron approach using the training set of sizes: 100, 200, 500, 750, and 1000 images. We train our SAE for 5 epochs for each scenario, keeping the hyperparameters constant. Our approach achieves good unlearning results even in limited training data scenarios while being more efficient from all methods requiring fine-tuning.

Online (streamlined) unlearning is appealing but lacks a rigorous explanation for the preservation metrics.

In our main experiments, we stricktly follow the evaluation provided by the UnlearnCanvas benchmark where independently trained classifiers are provided to evaluate In-domain retain accuracy (IRA) that measures the quality of generation of other concepts when unlearning a particular one (e.g., 49 other styles when unlearning a single one), and cross-domain retain accuracy (CRA) which assesses the preservation quality in a different domain (e.g., in style unlearning, we calculate object classification accuracy). Additionaly, following instructions from the benchmark, we report the FID metric when generating all of the concepts except for the unlearned ones.

I believe that this paper needs to explore the correlation among multiple concepts and how stable performance is maintained.

We agree with the Reviewer that capturing concept-specific features doesn't directly imply precise unlearning. To measure the impact of unlearning on other concepts in Fig 9 we present accuracy on each of 20 classes from UnlearnCanvas benchmark during unlearning. For most classes, our method successfully removes the targeted class while preserving the accuracy of the remaining ones. Nonetheless, in some cases where classes are highly similar to each other (e.g., Dogs and Cats), removing one of them negatively impacts the quality of the other. This observation is consistent across evaluated methods as presented in appendix D.3 of the original UnlearningCanvas benchmark.

To study how well the stable performance of our method can be maintained, we run an additional experiment where we subsequently unlearn 49 out of 50 styles present in the UnlearnCanvas benchmark (leaving one style out to evaluate the quality of its preservation). We observe almost no degradation in SAeUron performance for three randomly selected combinations. For unlearning of 49/50 styles simultaneously, we observe UA: 99.29%, IRA: 96.67%, and CRA: 95.00%. Those results highlight the unprecedented degree of unlearning precision with SAeUron.

We hope our responses have clarified your questions and addressed any concerns. If so, we would appreciate your consideration in raising the score.

Link to additional tables and figures: Anonymous github

Unlearning of nudity

To highlight the potential of SAeUron in real-world applications, we extend our study to the evaluation with the I2P benchmark focusing on the real-world application of NSFW content removal (see reply to rev 8z9i for technical details). As presented in Table 2, SAeUron achieves the best score in nudity content removal while maintaining high-quality generations for the remaining concepts.

Lack of qualitative comparisons with baselines

In Fig 1 and Fig 2, we present a qualitative comparison with the best-performing contemporary techniques. In the first set of plots, we show how SAeUron can remove the Cartoon style while retaining the original objects and the remaining styles. This is not the case for the other approaches that often fail to generate more complex classes like statues or the sea. At the same time, our technique can unlearn the bear class without affecting the remaining objects or all of the evaluated styles.

SAeUron only achieves the best score in (IRA) and average performance of style unlearning. This does not fully support the claim that SAeUron achieves state-of-the-art results.

Thank you for pointing out the inaccuracy; we will rewrite the sentence in the final version of our paper to highlight that our approach yields state-of-the-art results for style unlearning while scoring the second-best score for objects. The evaluation metrics are designed in a contradictory way, so it is easy to improve on one metric at the expense of another. Therefore, we report the average performance across all three metrics to enable a fairer comparison among all evaluated methods.

Performance against adversarial attacks

As requested, we run additional comparisons with adversarial attacks using the 143 nudity prompts prvided by the authors of UnlearnDiffAtk as a benchmark. As presented in Tab 3, SAeUron outperforms even the methods highlighted by the reviewer as specifically designed for robustness against adversarial attacks.

Selection of the targetted block

The selection of the appropriate activations for applying SAE is a critical decision. To simplify this process, we introduce a straightforward methodology in which we ablate the diffusion blocks one by one to identify the one essential for generating the desired concepts, as detailed in Appendix C. Although this heuristic may not always pinpoint the optimal location for the SAE application, it allowed us to achieve high-quality results.

Choice of hyperparameters

As stated in Section 5.3.2, we tune hyperparameters in our validation set. Following the reviewer’s suggestion, we evaluated multiple values for the multiplier $\gamma$ and the number of features selected for unlearning (instead of percentile $\tau$, to make the analysis clearer). As shown in Figure 10 of additional results, our method is robust to these parameters, with a broad range of values (apart from extreme cases) yielding comparably high performance.

Method Efficiency Methods like SLD or Receler do not require additional training or require only lightweight training, whereas SAeUron involves learning whole autoencoder networks, which may introduce overhead?

We would like to point out that sparse autoencoders used by our method are simple linear models consisting of two matrices. Therefore, their training can be considered lightweight. To prove those claims, in Fig 11 and 12, we compare the unlearning time for different approaches with their scores. We evaluate SAeUron with different training set sizes, showing that our method is faster than almost all evaluated approaches while yielding higher performance as measured by the averaged score.

Generalization to OOD: What specific properties of SAeUron contribute to its robustness in OOD scenarios? Furthermore, how do other benchmark methods perform under the same experimental setup?

In this scenario, we train Sparse Autoencoder on the limited set of styles available in the UnlearnCanvas dataset. Then, we unlearn OOD styles not seen by the SAE during training. This could be understood as a variant of zero-shot unlearning where we select features to block only from those already available in SAE without any additional training. Since SAEs are trained in an unsupervised way, they learn an overcomplete set of features, activating also on samples not presented to the model during training. This is in contrast to other methods, where targets need to be explicitly provided during finetuning. As presented in Tab. 3, our method can still achieve 51% unlearning accuracy, with a limited drop in retention accuracy. To our knowledge, none of the evaluated methods could be directly used in such a scenario.

We hope our responses have clarified your questions and addressed any concerns. If so, we would appreciate your consideration in raising the score.

Thank you for the positive review of our work, below we would like to explain and address the comments and remaining weaknesses with the help of additional tables and figures provided in the anonymized link Anonymous github:

The weak point is that the distinctive idea between the proposed method versus existing ones is not clear stated.

Thank you for pointing out this limitation; we will emphasize the distinctiveness of our method when compared with existing solutions in the final version of our submission. Below, we briefly outline the key differences.

Most existing approaches to machine unlearning in diffusion models (e.g., EDiff, ESD, FMN, SalUn, SHS, SA, CA) follow a common principle: fine-tuning the pre-trained model to unlearn a specific concept while either constraining weight updates or replaying the remaining dataset to prevent degradation. In contrast, our method takes a fundamentally different approach. Instead of fine-tuning, we train a single Sparse Autoencoder (SAE) on the activations of the pre-trained model. We then identify sparse features associated with the unwanted concepts and block them to prevent their generation. To our knowledge, this is the first method to introduce such an approach.

Our distinctive idea brings several benefits compared to recent approaches:

• We demonstrate its effectiveness in unlearning styles and objects and further validate its practical utility by applying it to nudity unlearning (see Table 2 of additional results).
• Due to the inherent nature of SAEs, the features selected for unlearning are highly interpretable (see Figures 3–8), enhancing the transparency of our approach.
• Because our method blocks sparse and highly selective features, intervening on activations of a model, it is robust against adversarial attacks and can be used to unlearn multiple concepts at the same time (up to 49 out of 50 available styles, as presented in Table 4).

Those properties distinguish SAeUron from existing solutions in terms of capabilities.

When talking about unlearning, a key issue is whether the erasing affects other output, especially similar concepts or objects. The paper has mentioned retention accuracy in the appendix but did not clearly state the evaluation criteria and the meaning of the results. Please elaborate it.

We fully agree that retaining the remaining concepts is crucial for any unlearning method. This is why, in our studies, we employ the UnlearnCanvas benchmark designed to specifically measure not only the effectiveness of unlearning but also the retention accuracy. In particular, in Table 1, presenting the main results for different unlearning methods, we include the In-domain retain accuracy (IRA) that measures the quality of generation of other concepts when unlearning a particular one (e.g., 49 other styles when unlearning a single one), and cross-domain retain accuracy (CRA) which assesses the retention quality in a different domain (e.g., in style unlearning, we calculate object classification accuracy). As visible, our approach achieves state-of-the-art performance on those metrics in style unlearning and very high results for the object unlearning. To further strengthen our analysis of retention accuracy during unlearning, Fig 9 of additional results presents accuracy on each of the 20 classes from the UnlearnCanvas benchmark during unlearning. In general, our method successfully removes targeted classes while preserving the accuracy of the remaining ones. Nonetheless, in extreme cases where classes are highly similar to each other (e.g., Dogs and Cats), removing one of them negatively impacts the quality of the other. This observation is consistent across evaluated methods as presented in appendix D.3 of the original UnlearningCanvas benchmark.

To further evaluate the retention capabilities of our approach, we run an additional experiment with an extreme scenario where we unlearn 49 out of 50 styles present in the UnlearnCanvas benchmark (leaving one style out to evaluate the quality of its preservation). We observe almost no degradation in SAeUron performance for three randomly selected combinations. For unlearning of 49/50 styles simultaneously, we observe UA: 99.29%, IRA: 96.67%, and CRA: 95.00%. Those results highlight the unprecedented degree of unlearning precision with SAeUron.

We are happy to discuss any other issues or questions regarding our paper. If there are none, we would highly appreciate you considering raising the score.