SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models

W1: Equation 6 and Line 179: The mean value $\mu=\mathbb{E}_{c_0\sim R}\left[||\Delta _\text{erace} c_0||^2\right]$is used as a threshold, but it is not supported by a theoretical explanation. There should be several options as follows. An explanation or an ablation study would be desired.

We thank the reviewer for the insightful suggestion. We agree that comparing different strategies for determining $\mu$ in Eq. (6) is important and helps justify our design choice in IPF. Thus we conducted an ablation study comparing four variants: (1) Arithmetic mean (our default), (2) Geometric mean, (3) Median value, and (4) $\mu + \lambda\sigma$, where $\lambda \in \{ \pm 0.1, \pm 1, \pm 2 \}$.

First, we observe that the performance among the arithmetic mean, geometric mean, median, and $\mu \pm 0.1\sigma$ are relatively similar. This is because these strategies produce similar threshold values in practice. To further investigate this, we examine the prior shift distribution across the whole retain set and find that the distribution is relatively uniform without significant outliers. We attribute this to two reasons: (1) The CLIP feature space is relatively dense, and text embeddings rarely produce true outliers. (2) The retain set we used usually includes common, high-frequency concepts whose embeddings are more evenly distributed. Therefore, our choice of the arithmetic mean is motivated by its simplicity, being the most basic form without introducing extra hyperparameters.

Second, we find that the $\mu + \lambda\sigma$  formulation shows significant performance variation when $\lambda$ becomes large (e.g., $\pm 1, \pm 2$). This is because large $\lambda$ values greatly affect the threshold, which can either overly enlarge or overly shrink the retain set. As further discussed in our responses to reviewers WTi1, 1DqN, and zWMb (shown below), we perform additional ablations on retain set size and found that both excessively small and overly large retain sets hurt performance.

In summary, although our current threshold is empirically selected, the extensive evaluations confirm that it consistently achieves a strong trade-off between erasure and preservation quality.

---

W2: Equation 8: A standard normal distribution is used, but its variance can be changed. A reason for the choice or the following ablation study would be desired.

Thanks for your suggestions. Based on your comment, we conduct an ablation study on the scaling coefficient of the standard deviation in the Gaussian noise $\epsilon\sim\mathcal{N}(0, s ^2 I)$ used in Eq. (8).

From the results, we observe that changing the noise scale $s$ has limited impact on erasure performance (CS remains stable). However, prior preservation (FID) is sensitive to overly large or small values of $s$. Very small noise (e.g., $s=0.1$) lacks sufficient diversity, while large noise (e.g., $s \geq 3$) is more likely to introduce noisy and semantically-inconsistent concepts. Our default choice $s = 1.0$ achieves the considerable overall balance. We will include this ablation in the revised paper.

---

W3: Some recent papers (SAFREE, AdaVD, etc.) demonstrate that their method works with generative models other than Stable Diffusion v1.4. Showing that the proposed method works well with other models will make the paper more valuable.

Thanks for your advice. We agree that demonstrating the effectiveness of our method on other generative models is important for showcasing its generality. In the main paper, we have already provided qualitative results on a range of diffusion models, including DreamShaper, RealisticVision, SDXL, and even SD3 with the different DiT architecture (see Fig. 6). We will include more quantitative and qualitative results in the revised paper.

---

W4: This paper lacks some citations. The following papers should be introduced and compared (in the supplementary material, if it is not very new).

We thank the reviewer for the helpful suggestion. We agree that a more comprehensive discussion of related work is necessary for better positioning SPEED within the broader concept erasure literature.

Our main paper focuses on methods that directly modify model parameters under an editing-based manner for concept erasure, as this paradigm is more practical in white-box deployments (e.g., Stable Diffusion). Other methods based on external modules (e.g., gating module) or sampling-based optimization (e.g., attention swapping) can be easily bypassed in this white-box setting, which are less focused in our main paper.

In response, we categorize current concept erasure methods into three types: (1) Training-based methods (using gradient descent to optimize a training objective), (2) Editing-based methods (deriving parameter update with a closed-form solution), and (3) Sampling-based methods (intervening in the diffusion sampling process to suppress the generation of target semantics) as follows:

• Training-based methods: Beyond those mentioned in the main paper, several other approaches adopt different training strategies and objectives to achieve effective concept erasure. For instance, CPE introduces an additional non-linear module acting as a classifier within each cross-attention layer to filter semantics aligned with the target concept. Similarly, Receler adds a plug-in module after each cross-attention layer to erase the target concept, and adopts adversarial training to enhance robustness. However, both methods are fragile in white-box attack scenarios, where attackers have full access to model parameters. Since all parameters remain unchanged, it is easy to bypass the auxiliary modules. In contrast, methods such as RACE and AdvUnlearn fine-tune the parameters directly and incorporate adversarial training to enhance robustness. RACE treats the diffusion model itself as a classifier to generate adversarial prompts, while AdvUnlearn introduces a fast attack generation method to identify such prompts effectively. ACE proposes to inject the erasure guidance into both conditional and the unconditional noise prediction, enabling the model to effectively prevent the creation of erasure concepts during both editing and generation. AGE dynamically selects optimal target concepts tailored to each undesirable concept, minimizing unintended side effects.

  However, most training-based methods primarily focus on single-concept erasure and involve long training time (e.g., typically over 1,000× slower than our method in erasing a single concept), often requiring hundreds to thousands of iterations per target concept. This significantly limits their scalability and practicality in real-world applications, especially when erasing a large number of concepts within tight time constraints.

• Editing-based methods mainly involve TIME, UCE, and RECE and have been thoroughly discussed in our main paper. A contemporaneous work published in March 2025, GLoCE, injects a lightweight module into the diffusion model with low-rank matrices and a simple gate, determined only by several generation steps for concepts with closed-form solutions (more detained comparisons of GLoCE can be found in the response to Reviewer jEfu, W3). However, GLoCE requires approximately 11,500 seconds to erase 100 concepts, 2,300× slower than our method. Moreover, GLoCE computes a separate gate module for each target concept, which can be easily bypassed in open-source white-box scenarios, as the core model parameters remain unchanged.

• Sampling-based methods directly intervene in the diffusion sampling process during image generation to avoid generating target concept semantics. SLD modifies classifier-free guidance to steer generation away from the undesired concept. AdaVD and SAFREE leverage orthogonal projection techniques to eliminate the semantic influence of the target concept during inference. SAFREE performs orthogonal decomposition directly on the text embedding, which may lead to over-erasure and degradation of prior knowledge. Differently, AdaVD applies the decomposition in the value space of each cross-attention layer in the U-Net, and further introduces an adaptive shift mechanism to balance concept erasure and prior preservation. TraSCE guides the diffusion trajectory away from generating harmful content using a specific formulation of negative prompting.

  However, although sampling-based methods demonstrate promising performance without parameter modification, they still remain vulnerable to white-box attacks, where full model parameters can be exploited to bypass the erasure mechanism.

We will summarize this discussion in the revised version and provide a more detailed analysis, including full citations and comparisons, in the supplementary material to clarify the strengths and limitations of each paradigm.

Dear Reviewer Rrzc,

Thank you again for your thoughtful comments and your willingness to raise the rating during the discussion phase.

We wanted to kindly follow up as the discussion period is approaching its end. We have engaged in discussions with the other reviewers (WTi1, 1DqN, and zWMb), and they have expressed that our rebuttal successfully addressed their concerns and showed support for recommending our submission towards acceptance.

Given your positive comments and acknowledgment that your main concerns have been resolved, we would greatly appreciate it if you could kindly raise your score in the final justification, should you find it appropriate.

Thank you again for your time and contributions to the review process.

Best regards,

Authors of Paper 14342

W1: Please provide an Algorithm that includes the method details.

Thanks for the suggestion. We have provided a Markdown version of the algorithm with detailed method steps, and we will include it in the revised version of the paper.

Input: Model parameters $\mathbf{W}$, Erasure set $\mathbf{E}$, Retain set $\mathbf{R}$, Augmentation times $N_A$

Output: Refined retain set $\mathbf{R}_\text{refine}$

1. Initialize concept embeddings
   • Initialize $\mathbf{C}\_1$ and $\mathbf{C}_*$ from $\mathbf{E}$
   • Compute $\boldsymbol{\Delta}\_\text{erase}$ using Eq. 1
2. Filter the original retain set with IPF
   • Filter $\mathbf{R}$ to obtain $\mathbf{R}\_f$ via Eq. 2
3. Further augment and filter with DPA and IPF
   • Augment $\mathbf{R}\_f$ to get $\left(\mathbf{R}\_f\right)^{\text{aug}}$ using Eq. 3
   • Filter $\left(\mathbf{R}\_f\right)^{\text{aug}}$ to obtain $\left(\mathbf{R}\_f\right)^{\text{aug}}\_f$ via Eq. 2
4. Combine the final set
   • $\mathbf{R}_\text{refine} \leftarrow \mathbf{R}_f \cup \left(\mathbf{R}_f\right)^{\text{aug}}_f$

Return: $\mathbf{R}_\text{refine}$

---

W2: In the experiments, how many concepts are preserved when erasing, for instance, 100 concepts simultaneously? Since the number of retained concepts is constrained by the feature dimensionality in Eq. (4), what is the upper bound on the number of concepts that can be preserved?

The dimension of the null space is upper-bounded by the feature dimension $d$ of the model as shown in Eq. (4).

$$\dim(\text{Null}(\mathbf{C}_0)) = d - \text{rank}(\mathbf{C}_0 \mathbf{C}_0^\top)$$

Since the feature dimension of the generation model is usually fixed, we investigate extreme cases where the retain set size is greater than $d$. As shown in Fig. 2, we visualize the prior preservation when the retain set includes 20,000 $\gg d$ concepts and thus $\text{rank}(\mathbf{C}_0 \mathbf{C}_0^\top) = d$.

Following [A], we include singular vectors w.r.t. non-zero singular values to ensure sufficient degrees of freedom for concept erasure. However, this leads to an approximate null space and induces semantic degradation within the retain set Fig. 2 (row 1). To mitigate this problem, we propose Prior Knowledge Refinement, a structured strategy for refining the retain set to enable accurate null-space construction as shown in Fig. 2 (row 2). This includes Influence-based Prior Filtering (IPF) which focuses preservation efforts on the most vulnerable concepts, Directed Prior Augmentation (DPA) to construct a more flexible null space by augmenting the retain set, and Invariant Equality Constraints (IEC) to protect sampling invariants.

Therefore, our proposed method is specifically designed to preserve more concepts by refining the retain set, it yields significantly better prior preservation compared to baseline methods, even when the theoretical null-space capacity is nearly exhausted.

Moreover, recent diffusion models have significantly increased their feature dimensionality—for instance, from $d = 768$ in SD v1.4 to $d = 4096$ in SD3. This trend inherently improves the potential for null-space-based preservation. As demonstrated qualitatively in Fig. 6, our method shows reliable erasure and preservation performance on SD3, indicating our strong potential for future scalability.

[A] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models, ICLR25.

---

W3: In Table 1, the paper reports only CS for target concepts and FID for non-target concepts. Based on my understanding, both metrics can be informative for evaluating both erasure efficacy and prior preservation. Providing a more complete comparison that reports both CS and FID for both target and non-target concepts would offer a more comprehensive view of the erasure performance.

Thank you for the suggestion. Due to page limits, we reported the core metrics in the main paper. A complete comparison with both CS and FID for target and non-target concepts is provided in Appendix Tables 6 & 7. The results demonstrate that our method consistently achieves superior prior preservation, as indicated by higher CS and lower FID across the majority of non-target concepts.

---

W4: The authors should include more ablation studies on introduced hyperparameters, such as the threshold choice in IPF and the number of augmentations in DPA.

Thank you for the suggestion. We have ablated the effectiveness of our three proposed modules (IPF, DPA, and IEC) in Table 4. For the finer-grained ablation you mentioned, we conduct the following ablation studies:

1. IPF module: We ablate the different strengths of $\mu$ by multiplying $\alpha$, i.e., $\mathbf{R}\_{f}: \mathbf{R} \mapsto \{ \boldsymbol{c}\_0 \in \mathbf{R} \mid \| \boldsymbol{\Delta}_{\text{erase}} \boldsymbol{c}_0 \|^2 > \alpha * \mu \}$, where different $\alpha$ corresponds to different ratios of the retain set (e.g., $\alpha=0$ means no the whole retain set).

   As shown below, varying $\alpha$ impacts the trade-off between erasure (CS) and preservation (FID). A lower $\alpha$ includes more weakly affected concepts into the retain set, increasing its rank and overly shrinking the null space. As dicussed in Sec. 4.1 (line 164-169), this leads to worse erasure efficacy (higher CS) and poor preservation (higher FID). Conversely, a higher $\alpha$ yields better erasure performance due to fewer retain concepts, but still increases the FID because of non-comprehensive prior coverage. The best balance is observed at moderate thresholds ($\alpha=1$ in our setup). We will add this clarification to the revised version.

   $\alpha$	0.0	0.5	0.75	1.0 (Ours)	1.25	1.5
   retain set ratio	100%	96%	77%	46%	20%	9%
   CS ↓ (Erase)	27.16	27.09	26.90	26.29	25.90	25.73
   FID ↓ (Retain)	48.29	45.56	44.38	29.35	34.61	37.29

2. DPA module: We have included an ablation study of the DPA module in the Appendix (Fig. 10), analyzing the effects of the number of augmentations $N_A$ and augmentation rank $r$.

---

W5: While the scalability and efficiency of SPEED are well-demonstrated, some practical aspects (e.g., memory usage when erasing a very large number of concepts) could be discussed more explicitly to guide practitioners.

Thanks for your advice. We report both the GPU memory (GB) and editing time (s) in erasing 1, 10, and 100 concepts, respectively. Since our method is based on a closed-form solution paradigm, it does not require additional image generation or gradient backpropagation, resulting in both memory efficiency and time efficiency. This advantage becomes particularly prominent in multi-concept erasure scenarios, where no significant increase in memory or time consumption is observed.

W1: The sentence "we use the k-means algorith to select k centroids to reduce redundancy" in footnote 4 is a bit unclear. I would like to see more details.

Thanks for your feedback. We introduce Invariant Equality Constraints (IEC) module to explicitly protect invariants during image sampling by forcing $(\boldsymbol{\Delta} \mathbf{P}) \mathbf{C}_2 = \mathbf{0}$, where $\mathbf{C}_2$ is the stacked invariant embedding matrix. When processing the null-text embedding, the CLIP would encode it into 77 token embeddings: 1 [SOT] and 76 [EOT] tokens.

Although these [EOT] embeddings are semantically similar, they still exhibit numerical differences that unnecessarily increase the rank of $\mathbf{C}_2$, which increases the optimization in obtaining $\boldsymbol{\Delta}$, thereby limiting the degrees of freedom available for null-space optimization. To avoid this rank inflation while preserving the core semantics of the null-text, we apply k-means clustering to the 76 [EOT] embeddings and retain $k$ representative centroids (we set $k=3$). These centroids are then used to construct $\mathbf{C}_2$ in Eq. (11) as part of the IEC constraints.

We will revise the manuscript to make this rationale and implementation choice clearer.

---

W2: Lack of experiments or ablation studies on the impact of the retain set size (e.g., 10%, 20%, ..., 100%). Since a major motivation of SPEED is related to the size of the retain set, I believe it is important.

We thank the reviewer for highlighting the importance of retain set size in selecting influential priors. In response, we ablate different strengths of $\mu$ by multiplying $\alpha$, i.e., $\mathbf{R}\_{f}: \mathbf{R} \mapsto \{ \boldsymbol{c}\_0 \in \mathbf{R} \mid \| \boldsymbol{\Delta}_{\text{erase}} \boldsymbol{c}_0 \|^2 > \alpha * \mu \}$, where different $\alpha$ corresponds to different ratios of the retain set (e.g., $\alpha=0$ means the whole retain set).

As shown below, varying $\alpha$ impacts the trade-off between erasure (CS) and preservation (FID). A lower $\alpha$ includes more weakly affected concepts into the retain set, increasing its rank and overly shrinking the null space. As dicussed in Sec. 4.1 (line 164-169), this leads to worse erasure efficacy (higher CS) and poor preservation (higher FID). Conversely, a higher $\alpha$ yields better erasure performance due to fewer retain concepts, but still increases the FID because of non-comprehensive prior coverage. The best balance is observed at moderate thresholds ($\alpha=1$ in our setup). We will add this clarification to the revised version.

---

W3: Missing citations of related works. Including more related works would help improve the completeness of literature review and help readers better understand the development of this field.

We thank the reviewer for pointing out the missing citations. We agree that a more comprehensive discussion of recent related works will help better position SPEED within the broader concept erasure literature. We now incorporate an extended comparison with our work including the following:

• Receler [A] adds plug-in modules after each attention layer to learn a lightweight Eraser for concept erasing and adopts adversarial training to refrain the model from erased target concepts. However, since it only incorporates additional modules without modifying any model parameters. it can be easily bypassed in white-box settings. Instead, our work directly modifies model parameters and thus is more applicable in white-box settings.
• RACE [B] fine-tunes model parameters using adversarial prompts to identify and mitigate adversarial text embeddings, significantly reducing the Attack Success Rate. However, it focuses mainly on single-concept erasure due to its inferior prior preservation (multiple erasing would lead to mode collapse) and involves high training cost (e.g., 800× slower than SPEED).
• AdaVD [C] applies interventions during the sampling process, projecting value features onto orthogonal direction to target embeddings and meanwhile introduces adaptive shifting at each operation. While lightweight, this sampling-based paradigm can also be easily bypassed in white-box settings, as the model parameters are not modified.

We have also consolidated all reviewers’ suggestions on the related work improvement. Please see our full summary in the response to Reviewer Rrzc, W4. We will summarize this discussion in the revised version.

---

W4: The paper only compares SPEED with four baselines. It would be helpful to include additional recent methods at least in the main table, especially since their code is available.

Thanks for your comment. Since these methods (ESD, RACE, and Receler) do not focus on multi-concept erasure scenarios, we compare their performance in single-concept erasure (i.e., erasing Van Gogh). Our method consistently outperforms these baselines in balancing between erasure and preservation with the lowest FID. Notably, our method can achieve single-concept erasure with only 3.6s, which is 150 to 1500 times faster than these baselines. We will incorporate these results in the revised version.

---

W5: The ablation study is only conducted with single-concept erasure. Since some components are designed for multi-concept, it would be helpful to include ablation results on multi-concept settings to better understand the contribution of each component.

We thank the reviewer for the valuable suggestion. In response, we conduct an ablation study under the multi-concept erasure setting. The results are consistent with the single-concept ablation results in Table 4, further confirming the individual effectiveness of the proposed components (IEC, DPA, and IPF) in editing-based concept erasure. We will include this additional ablation in the revised paper.

---

W6: How does SPEED perform against adversarial prompts? Since robustness against attacks is an important issue in concept erasure tasks, it would be valuable to evaluate SPEED using adversarial methods (e.g., [D, E]).

Thanks for your advice. Due to the character limit, please refer to our response to Reviewer WTi1, comment W4.

---

W7: In experiments, the retain set is always chosen based on the target concept, which limits the flexibility. I just wonder is it possible to use a fixed, general-purpose retain set (e.g., MSCOCO) for any target concept?

We agree with the reviewer that using a fixed, general-purpose retain set (e.g., MSCOCO) would offer greater flexibility. The use of a targeted retain set is by default adopted from prior works (e.g., UCE, RECE), where the retain set is designed to maximize the preservation of semantically related concepts during each erasure task. This paradigm stems from our discussion in Sec. 4.1 where concept erasure inherently exhibits locality. This indicates that when erasing a specific concept (e.g., Van Gogh), some related concepts (e.g., Picasso and Monet) are more influenced than those general concepts (e.g., MSCOCO). To further validate this statement, we compare the erasure and preservation performance with both general and targeted retain sets below.

From the table, using a targeted retain set significantly improves the preservation of related concepts (e.g., Picasso, Monet), while achieving comparable performance to the general concepts from MSCOCO. This demonstrates that targeted retain sets are more effective at preserving semantically close concepts, and notably, even without explicitly including general concepts in the retain set, their general prior is remaining largely preserved.

Furthermore, since mainstream concept erasure scenarios typically involve specific domains such as IP instances, artistic styles, celebrity identities, and NSFW content, preparing a dedicated retain set for each category in advance is often sufficient to cover most practical use cases. Nevertheless, we agree that developing a more universal, general-purpose retain set is a promising direction, and we plan to explore this further in future work.

W1: This paper doesn’t explicitly explain the role of directed prior augmentation (DPA) using random generated noises. I understand that low-dimensional structure, as decomposed by SVD, consists of principal and meaningless embeddings. This procedure encourages to offer augmented concepts that align with the original concepts. However, I’m curious about the specific improvements in terms of image quality and concept erasure that DPA derives. In its current form, analyzing ablation studies alone doesn’t provide a clear understanding of DPA’s role. Visualizations would help readers grasp the justification of DPA.

We thank the reviewer for the insightful comment. Your understanding is correct. Since we cannot include visualizations in the rebuttal, we describe the effect of DPA in details. Overall, DPA mainly improves the consistency of retain concept images before and after erasure. For example, we visualize both the target (snoopy) and retain (mickey) concept when erasing Snoopy w/ and w/o DPA. (1) For retaining image quality (mickey), generations w/ DPA exhibit more consistent visual features with pre-trained model's generations, with better preservation of both subject (e.g., buttons on Mickey's clothes) and background (e.g., flowers and plants in the background). (2) For concept erasure (snoopy), generations w/ DPA demonstrate more successful erasure (e.g., Snoopy's iconic black ears and facial shape are more cleanly removed visually) compared to those w/o DPA. However, the effect on erasure is positive but less visually noticeable, as w/ and w/o DPA both show successful erasure of target semantics.

In summary, the visualization results align with our quantitative ablation in the main paper, DPA ensures that augmented concepts remain valid and consistent, which helps improve both erasure efficacy and prior preservation. We will add visualizations in the revised version.

---

Q1: In Eq 6, the authors mention that making influential priors depends on the hyperparameter $\mu$ . However, when I read the manuscript, I noticed that the authors did not explicitly state how the strength of $\mu$ affects the performance in multiple concept erasing. I believe that lowering $\mu$ could lead to ambiguity in augmenting the prior information.

We thank the reviewer for highlighting the importance of $\mu$ in selecting influential priors. In response, we ablate different strengths of $\mu$ by multiplying $\alpha$, i.e., $\mathbf{R}\_{f}: \mathbf{R} \mapsto \{ \boldsymbol{c}\_0 \in \mathbf{R} \mid \| \boldsymbol{\Delta}_{\text{erase}} \boldsymbol{c}_0 \|^2 > \alpha * \mu \}$, where different $\alpha$ corresponds to different ratios of the retain set (e.g., $\alpha=0$ means the whole retain set).

As shown below, varying $\alpha$ impacts the trade-off between erasure (CS) and preservation (FID). A lower $\alpha$ includes more weakly affected concepts into the retain set, increasing its rank and overly shrinking the null space. As dicussed in Sec. 4.1 (line 164-169), this leads to worse erasure efficacy (higher CS) and poor preservation (higher FID). Conversely, a higher $\alpha$ yields better erasure performance due to fewer retain concepts, but still increases the FID because of non-comprehensive prior coverage. The best balance is observed at moderate thresholds ($\alpha=1$ in our setup). We will add this clarification to the revised version.

---

Q2: I’m curious about extreme cases where low-dimensional structures fail to span the original spaces. These situations suggest that either the null spaces of retained sets become almost empty sets or the retained sets consist of multiple terms sharing similar or identical meanings. In such cases, I’d like to observe how the proposed method performs in extreme scenarios.

We appreciate the reviewer’s interest in our performance under extremely low-dimensional conditions. Indeed, the dimension of the null space is upper-bounded by the feature dimension $d$ of the model as shown in Eq. (4).

$$\dim(\text{Null}(\mathbf{C}_0)) = d - \text{rank}(\mathbf{C}_0 \mathbf{C}_0^\top)$$

Since the feature dimension of the generation model is usually fixed, we investigate extreme cases where the retain set size is greater than $d$. As shown in Fig. 2, we visualize the prior preservation when the retain set includes 20,000 $\gg d$ concepts and thus $\text{rank}(\mathbf{C}_0 \mathbf{C}_0^\top) = d$.

Following [A], we include singular vectors w.r.t. non-zero singular values to ensure sufficient degrees of freedom for concept erasure. However, this leads to an approximate null space and induces semantic degradation within the retain set Fig. 2 (row 1). To mitigate this problem, we propose Prior Knowledge Refinement, a structured strategy for refining the retain set to enable accurate null-space construction as shown in Fig. 2 (row 2). This includes Influence-based Prior Filtering (IPF) which focuses preservation efforts on the most vulnerable concepts, Directed Prior Augmentation (DPA) to construct a more flexible null space by augmenting the retain set, and Invariant Equality Constraints (IEC) to protect sampling invariants.

Therefore, our proposed method is specifically designed to alleviate such extreme cases by refining the retain set, it yields significantly better prior preservation compared to baseline methods even when the theoretical null-space capacity is nearly exhausted.

[A] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models, ICLR25.

---

Q3: I’m curious about the reason behind Eq. 22. I don’t think Eq. 22 is trivial without a specific condition. While I understand Eq. 33 works, Eq. 22 doesn’t hold unless a specific assumption is made. I believe the authors must specify a specific condition that doesn’t contradict the core hypothesis.

We thank the reviewer for the careful reading and for pointing this out. We agree that Eq. 22 is not self-evident and may appear unclear without further justification. We have provided the proof below to clarify its validity. We apologize for the confusion and will include this derivation explicitly in the revised paper to ensure clarity.

Proof.

The singular value decomposition (SVD) of $\mathbf{X}$ can be written as follows:

$$\mathbf{X} = \mathbf{U} \Sigma \mathbf{V}^T,$$

where $\mathbf{U}$ and $\mathbf{V}$ are orthogonal matrixs. Considering the Frobenius norm is invariant under orthogonal transformations, thus we have:

$$\|\mathbf{X}\mathbf{Y}\|_F = \|\mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^\top \mathbf{Y}\|_F = \|\boldsymbol{\Sigma} \mathbf{V}^\top \mathbf{Y}\|_F$$

Let $\mathbf{Z} = \mathbf{V}^\top \mathbf{Y}$ , then:

$$\|\mathbf{X}\mathbf{Y}\|_F = \|\boldsymbol{\Sigma} \mathbf{Z}\|_F$$

Recall that $\boldsymbol{\Sigma}$ is a diagonal matrix, so multiplying it scales each row (corresponding to each singular value):

$$\|\boldsymbol{\Sigma} \mathbf{Z}\|_F^2 = \sum\_{i=1}^r \sigma\_i^2 \|\mathbf{z}_i^\top\|^2$$

where $\mathbf{z}\_i^\top$is the $i$-th row of $\mathbf{Z}$. Noting that $\sigma\_i \geq \sigma\_{\min}$, we have:

$$\|\boldsymbol{\Sigma} \mathbf{Z}\|_F^2 = \sum\_{i=1}^r \sigma\_i^2 \|\mathbf{z}_i^\top\|^2 \geq \sigma\_{\min}^2 \sum\_{i=1}^r \|\mathbf{z}_i^\top\|^2 = \sigma\_{\min}^2 \|\mathbf{Z}\|_F^2$$

Thus, we obtain:

$$\|\mathbf{X}\mathbf{Y}\|_F = \|\boldsymbol{\Sigma} \mathbf{Z}\|_F \geq \sigma\_{\min} \|\mathbf{Z}\|_F = \sigma\_{\min} \|\mathbf{V}^\top \mathbf{Y}\|_F = \sigma\_{\min} \|\mathbf{Y}\|_F$$

i.e.,

$$\|\mathbf{X}\mathbf{Y}\|_F \geq \sigma\_{\min}(\mathbf{X}) \|\mathbf{Y}\|_F$$

---

Q4: In the recent concept erasing, the task of removing nudity has become crucial in preventing Not Sale for Work (NSFW) images. I was curious if this approach could effectively address an issue where adversarial users attempt to generate NSFW images [1,2]. In this paper, the authors addressed this issue by utilizing the I2P dataset. However, adversarial text prompts raise a concern, as recent defense methods still produce harmful content.

Thanks for your question. As discussed with Reviewer jEfu in W4, we compare our method w/ and w/o adversarial training/editing (AT) under both white- (UnlearnDiff) and black-box (MMA and Ring-A-Bell) adversarial attack benchmarks, and report Attack Success Rate (ASR). From the table, our method with adversarial training/editing (Ours w/ AT) exhibits effective performance against both black-box and white-box attacks on par with competitive baselines such as CPE, AdvUnlearn, and Receler, while maintaining efficient runtime (4.5s).

Note: ”-” indicates that the method cannot defend white-box attack as it doesn’t modify any model parameters

Current improvements with AT demonstrates the great potential of SPEED in resisting adversarial attacks. Due to the limited time during rebuttal, we could only include preliminary results. We will conduct a further study in the revised paper.

W1: There are a number of recent methods that can erase 100 concepts … I am very curious about other results such as erasing 100 artistic styles and retaining other celebrities while erasing 50-100 celebrities.

We thank the reviewer for pointing out that these recent works (CPE, GLoCE, and Six-CD) can also address multi-concept erasure. Specifically, CPE applies nonlinear residual gates to suppress target concepts; GLoCE introduces low-rank modules with simple gating for concept suppression; Six-CD aggregates multiple strategies for large-scale erasure evaluation.

Compared to these methods, SPEED introduces a null-space constrained method to directly calculate the model parameter update in a closed-form solution. This formulation allows erasing 100 concepts in 5 seconds with two advantages: (1) significantly higher efficiency (e.g., thousands of speedup) derived from the closed-form solution, and (2) applications in white-box scenarios where external modules can be bypassed. Instead, our method directly modifies model parameters. We will include more discussions of these recent methods in the revised paper.

To further support our scalability, we compare 100 artistic styles and 100 celebrities erasure with CPE and GLoCE (we exclude Six-CD as it only releases a benchmark). It can be seen that our method achieves comparable overall multi-concept erasure and preservation performance with these baselines. While our overall performance is slightly lower than CPE, our method offers two key advantages: time efficiency (nearly 10,000× faster) and white-box application (direct parameter editing without reliance on external modules).

---

W2: The proposed method exploits the null space / editing space via SVD … How was this work using SVD differently and using the null space differently?

Thanks for your comment. Our method uses null space with SVD to achieve prior-preserved concept erasure in an editing-based manner. To mitigate the optimization difficulty in null-space constrained editing, our main contribution lies in the proposed Prior Knowledge Refinement strategy with three complementary modules (IPF, DPA, and IEC) to construct an accurate and reliable null space. Our differences with the mentioned methods are discussed respectively as follows:

1. GLoCE is inspired by LEACE, exploring SVD-based subspace to remove concept-specific components from diffusion models. While both GLoCE and our method focus on concept erasure in diffusion models, the motivation and implementation are significantly different:

   • Motivation: GLoCE uses SVD to enable efficient erasure, addressing the overhead of full-rank LEACE projections. Specifically, it performs SVD on target and mapping embeddings to extract their principal components. Then, these principal components are used to compute an optimized low-rank LEACE projections for efficient concept erasure. In contrast, our method focuses on prior preservation. We perfor SVD on retain embeddings and extract the null space components to construct a subspace orthogonal to the retain concepts, which ensures that all edits lie in directions orthogonal to the retain concepts
   • Implementation: GLoCE performs SVD on the covariance of image embeddings, which requires image sampling and forward pass through the diffusion generator network for each concept. In contrast, our method performs SVD on text embedding tokens with only text inputs. This design avoids additional image sampling and only requires text embeddings for each concept within seconds, making it significantly more efficient than repeatedly forward-passing diffusion generators.

   Despite these differences, we think these two perspectives are complementary in practice: GLoCE enhances the concept erasure precision, while our method enables scalable, precise, and efficient editing with prior preservation over the retain set. We believe integrating insights from both directions could inspire future works.

2. RLACE, similar to GLoCE, uses SVD to identify the concept subspace to be erased and removes it via orthogonal projection, while our method uses SVD to define the null space of non-target concepts and constrains updates to lie entirely within it.

3. UNSC leverages null-space projection to map the training gradients onto the null space of the retain subspaces, thereby mitigating the over-unlearning problem in continual learning. In contrast, our method projects the parameter update onto the null space of the remaining concepts in a closed-form solution, without requiring any training. Moreover, we introduce three complementary modules to refine a more accurate and reliable null space, leading to significant improvements in prior preservation.

---

W3: It seems that the proposed method focuses on remaining concepts while not much focusing on erasing concepts and thus the proposed method does not seem to achieve SOTA performance in erasing.

Thank you for your comment. We would like to clarify that our method is not focused on preserving non-target concepts at the expense of erasure efficacy. Rather, it is designed to jointly optimize both objectives, addressing the well-known trade-off between erasure and preservation, particularly in multi-concept settings. In our work, we address this fundamental joint challenge by introducing a null-space constrained formulation combined with adaptive refinement of the retain set, explicitly emphasizing both erasure and preservation.

This is supported by our experimental results in Table 1 & 2, where we achieve SOTA prior preservation as well as competitive erasure performance. Though our CS of erasing concepts is not the lowest among all methods, it is already sufficiently low to indicate successful erasure. As shown in Fig. 4 & 5, the target concepts are clearly removed. This is further supported by Fig. 7 & 8 in the Appendix, where different visually successful erasures can result in a wide range of CS scores. For example, erasing Snoopy into a generic dog (ours) and meaningless concepts (RECE) yields only a 6-point drop and exceeding 10-point drop in CS, respectively, despite the Snoopy concept both being effectively erased in both cases.

---

W4: This work seems quite inefficient for NudeNet detection on I2P, which is one of the most important applications that this work also claimed.

We agree that NSFW concepts like nudity are important in concept erasure. Recent works (e.g., CPE, AdvUnlearn, and RACE) have shown strong results on this task, largely due to the use of adversarial training. In contrast, our method mainly focuses on scalability, precision, and efficiency of concept erasure without introducing additional nudity-specific training mechanisms. As shown in Table 8, our method has already achieved better performance against non-adversarial-training methods (e.g., MACE and UCE) for NudeNet detection on I2P.

To make a fair comparison with adversarial training-based methods, we extend SPEED with adversarial training/edting following RECE as discussed in Appendix E and conduct the below comparison:

With adversarial training/editing, our method exhibits significant improvement in I2P Total from 113 to 55, only inferior to CPE and AdvUnlearn. However, our method can defend white-box attack and achieve high efficiency by taking only 4.5s compared to them. Due to the limited rebuttal period, we provide this as preliminary evidence of our potential in adversarial settings, and we will include a more comprehensive study in the revised version.

---

W5: It is important to ensure that the concept erasing method is robust against adversarial attacks.

Thanks for your advice. Due to the character limit, please refer to our response to Reviewer WTi1, comment W4.

Dear Reviewer jEfu:

Thanks for your constructive comments. We would like to follow up to see if our response addresses your concerns or if you have any further questions. Thanks for your attention and best regards.

Faithfully,

Authors of Paper 14342

Dear Reviewer jEfu,

Thank you again for your valuable efforts and constructive advice in reviewing our paper. As the discussion period nears its end, we look forward to your feedback on our responses. We have made every effort to address all your concerns and are happy to clarify any points or discuss any remaining questions.

Best regards,

Authors of Paper 14342