Training-Free Safe Text Embedding Guidance for Text-to-Image Diffusion Models

We appreciate the reviewers’ constructive and encouraging comments. We have carefully considered all suggestions and provide our responses below.

W1: [GPU consumption]

We appreciate the reviewer’s observation regarding GPU consumption. As discussed in Appendix C.3 and D, the added inference cost of STG mainly stems from gradient computations required for updating the text embedding. To mitigate this overhead, we explore time- and memory-efficient techniques. Specifically, we apply half-precision (FP16) inference during sampling, which significantly reduces both runtime and GPU memory usage, while preserving the safety performance of our method, as shown in the below table. We extended our runtime and memory usage analysis (originally presented in Table 7 of the appendix) by including results with half-precision (FP16) inference. Although STG requires gradient computation during update steps, we observe that FP16 inference maintains nearly the same safety performance. This suggests that existing time- and memory-efficient techniques can be effectively leveraged to address the computational cost of STG in practice.

[Table D1: Sampling time and memory usage with FP16]

W2 & Q2: [Application of bias mitigation]

Thank you for the suggestion. STG can be extended to bias mitigation tasks if a suitable safety function can be defined to measure and discourage biased behavior. To explore this, we conduct a preliminary experiment targeting gender imbalance across occupations. Specifically, we use the prompt format ‘a photo of {occupation}’ and analyze the gender distribution of generated images from the Stable Diffusion v1.4 backbone. We observe that prompts such as ‘nurse’ resulted in 100% female images, while ‘farmer’ yielded 98% male images, indicating strong gender bias in the base model.

To mitigate this, we define a safety function as the negative squared difference between CLIP scores for ‘a photo of male {occupation}’ and ‘a photo of female {occupation}’. This formulation encourages the generated image to avoid being overly aligned with either gender-specific prompt, thereby guiding it toward a more neutral representation. As shown in the below table, the degree of bias mitigation can be controlled by adjusting the update scale hyperparameter $\rho$. The reported ratio corresponds to the proportion of male-presenting images, and values closer to 0.5 indicate that the generated images are achieving a more balanced gender distribution, demonstrating effective mitigation of occupational gender bias.

[Table D2: Gender distribution in generated images for occupation prompts]

That being said, we note that STG operates on a per-instance basis and does not explicitly enforce balance over the distribution of generated samples. As such, it may produce images that appear more neutral, but this does not necessarily indicate fair distribution-level behavior. We view this as a potential direction for future work, particularly in integrating group-level fairness constraints into our framework.

W3: [Incremental update on SDG]

While the proposed method may appear to be an incremental modification of SDG, shifting the guidance target from the image latent space to the text embedding space; our work provides a thorough theoretical and empirical justification for this change. Specifically, Section 4.3 with Theorem 1 analytically shows that updating the text embedding implicitly influences the latent space while preserving the underlying model likelihood. Section 4.4 further supports this claim with toy experiments, demonstrating that STG more faithfully maintains the original generative distribution compared to SDG, especially when the safety function is imperfect. This distinction is particularly important for tasks such as style removal or copyright mitigation, where it is critical to preserve the semantic content or visual fidelity of the original image while modifying only specific attributes.

In addition, although SDG itself is derived from the previous Universal Guidance framework, our work is the first to adapt and implement both SDG and STG for the safety domain, along with extensive evaluations across safety tasks. We believe this comprehensive adaptation and evaluation further contributes to the field.

W4 & Q1: [Multiple text embeddings]

Our method is compatible with recent generative models that utilize multiple text embeddings, such as FLUX, SDXL, and SD3. In these models, STG can be applied by updating all text embeddings in the same way. We conduct additional experiments using more recent diffusion models, including FLUX, SDXL, and SD3. For each model, we follow the default sampling and configuration settings provided in the diffusers library. In the case of FLUX, we use the guidance-distilled version.

The below table reports the results on the Ring-A-Bell (violence) benchmark for safe generation metrics, and COCO dataset for general generation quality. Note that for this table, the COCO evaluation is conducted using 1,000 images, which accounts for the scale difference in FID values.

[Table D3: Results on recent diffusion models: FLUX, SDXL, and SD3]

As shown in the table, the Ring-A-Bell prompts still induce harmful outputs even in the recent models, as indicated by the low DSR values for the base models. Applying STG improves safety (higher DSR), while maintaining similar image quality as measured by FID and CLIP score. These results demonstrate that STG generalizes well across diverse backbones, pipelines, and fast generation methods. Additionally, the trade-off between image quality and safety can be adjusted via the update threshold hyperparameter $\tau$, providing flexibility depending on application requirements.

Dear Reviewer 5G8U,

We sincerely appreciate your valuable feedback and constructive comments. We have tried to address your comments in our rebuttal, and we would be grateful to know whether our responses have resolved your concerns. If there are any remaining questions or points to discuss, we would be happy to discuss them further. Thank you again for your time and thoughtful review.

Best regards,
Authors

Thanks to the authors for the rebuttal. My primary concern in my review was the generalizability of the method to newer models, computational efficiency and the application of the method towards tasks such as bias mitigation overall. My questions have been addressed in the author rebuttal. Given that the other reviewers are also leaning towards the acceptance of the paper, and the supplementary experiments presented in the rebuttal, I will be adjusting my score towards positive. The authors are strongly encouraged to include these supplementary examples in the camera-ready version. Also I believe qualitative samples over models such as FLUX, SD3 and SDXL would also benefit the paper in a positive way.

We thank the reviewer for the helpful feedback. Our responses to the raised concerns are summarized below.

W1: [Reliance of the external classifier]

We agree with the reviewer’s concerns, and this limitation is explicitly acknowledged in Appendix D of the paper. However, as discussed in Sections 4.3 and 4.4, our theoretical analysis and the toy experiments demonstrate that STG could be effective when the safety function is not a perfect estimator of the true safety probability. Unlike data-space guidance methods, STG jointly considers the model likelihood and safety objective, which helps mitigate the risk of overfitting to potentially biased or imperfect proxy metrics.

Recent advances in pre-trained vision-language models, such as CLIP, provide a promising alternative. These models enable flexible zero-shot classification and can be leveraged to construct effective proxy safety functions. In our experiments, the violence and artist-style removal tasks utilized CLIP, even though CLIP was not specifically designed for those safety objectives.

Together, our theoretical and empirical findings suggest that STG could be robust to the safety functions and can effectively improve safety while preserving the model’s original generative capabilities. This supports the practical use of proxy classifiers, such as CLIP-based scores, within our framework.

W2: [Reduced efficiency]

We appreciate the reviewer’s observation regarding the efficiency. As discussed in Appendix C.3 and D, the added inference cost of STG mainly stems from gradient computations required for updating the text embedding. To mitigate this overhead, we explore time- and memory-efficient techniques. Specifically, we apply half-precision (FP16) inference during sampling, which significantly reduces both runtime and GPU memory usage, while preserving the safety performance of our method, as shown in the below table. We extended our runtime and memory usage analysis (originally presented in Table 7 of the appendix) by including results with half-precision (FP16) inference. Although STG requires gradient computation during update steps, we observe that FP16 inference maintains nearly the same safety performance. This suggests that existing time- and memory-efficient techniques can be effectively leveraged to address the computational cost of STG in practice.

[Table C1: Sampling time and memory usage with FP16]

Additionally, we would like to highlight an observation from Table 4 in the main text. By adjusting the update step ratio, under comparable sampling time, STG attains a similar defense success rate to SLD while achieving substantially higher safety performance than SAFREE. Also, it does so with better prior preservation than both methods. This demonstrates that STG offers flexible trade-offs between sampling speed, safety, and fidelity, depending on the desired deployment scenario.

W3 & Q2: [Generalization]

To evaluate the generalization ability of STG, we conducted experiments using the PixArt-alpha model, a recent diffusion model with a Transformer-based backbone, in combination with the DPM-Solver sampler. These results are reported in Table 6 and discussed in Appendix C.2.

Additionally, to further address concerns about scalability, we extend our experiments to include more recent diffusion models such as FLUX, SDXL, and SD3. For each model, we follow the default sampling and configuration settings provided in the diffusers library. In the case of FLUX, we use the guidance-distilled version.

The below table reports the results on the Ring-A-Bell (violence) benchmark for safe generation metrics, and COCO dataset for general generation quality. Note that for this table, the COCO evaluation is conducted using 1,000 images, which accounts for the scale difference in FID values.

[Table C2: Results on recent diffusion models: FLUX, SDXL, and SD3]

As shown in the table, the Ring-A-Bell prompts still induce harmful outputs even in the recent models, as indicated by the low DSR values for the base models. Applying STG improves safety (higher DSR), while maintaining similar image quality as measured by FID and CLIP score. These results demonstrate that STG generalizes well across diverse backbones, pipelines, and fast generation methods. Additionally, the trade-off between image quality and safety can be adjusted via the update threshold hyperparameter $\tau$, providing flexibility depending on application requirements.

We also investigate sensitivity of STG to different samplers by conducting experiments on the Stable Diffusion v1.4 backbone using the DDPM sampler, keeping all settings identical to those used with the DDIM sampler (as in Figure 3a for the Ring-A-Bell nudity task). Due to the rebuttal policy, we are only able to provide tabular results at this stage. For methods originally visualized as curves, we report three representative points based on DSR (the lowest, median, the highest values). We will include the full curves in the revised version of the paper.

[Table C3: Results with DDPM samplers]

The results show that STG consistently delivers better performance across different samplers, indicating that STG is robust to the sampler.

Across these diverse backbones and samplers, STG consistently demonstrates strong performance, highlighting its generality and robustness across a wide range of model architectures and sampling strategies.

Q1: [More categories]

We understand the reviewer’s question as referring to safety performance across a broader range of inappropriate categories (e.g., hate, harassment, etc.). To investigate this, we conduct experiments on the I2P benchmark, which includes prompts from seven categories used in the SLD paper: hate, harassment, violence, self-harm, sexual, shocking, and illegal activity.

Due to the large number of prompts in I2P benchmark, we adopt a fast diffusion model, specifically the Latent Consistency Model (LCM) (Luo et al., 2023) with 4 sampling steps to enable efficient sampling. For the safety function $g$, we use the CLIP score between each image and the fixed general inappropriate text prompt, as defined in the SLD paper. This unified safety function allows STG to be applied consistently across all seven categories. As commonly done in the I2P dataset, we compute the DSR using the combined Q16/NudeNet classifier and report the experimental results.

[Table C4: Safety performance (DSR) across diverse categories on the I2P benchmark using LCM + STG]

As shown in the table, STG demonstrates strong safety performance even when applied to modern diffusion models and across diverse safety categories.

(Luo et al., 2023) Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference

Thanks for your detailed rebuttal. My concerns are resolved and I keep my score.

We greatly appreciate the reviewers' insightful comments and provide our responses below.

W1 & W2: [Requirement of external classifier]

As the reviewer pointed out, Our method relies on an external classifier for the safety function. This is feasible for tasks with public classifiers (e.g., NudeNet for nudity) but limits generality where none exist. However, recent advances in pre-trained vision-language models, such as CLIP, provide a promising alternative. These models enable flexible zero-shot classification and can be leveraged to construct effective proxy safety functions. In our experiments, the violence and artist-style removal tasks utilized CLIP, even though CLIP was not specifically designed for those safety objectives.

This explanation of the safety function is briefly mentioned in Section 4.1 (lines 129-131), but we agree that the reliance on external classifiers and implications for generality should be explicitly stated as a limitation. We will revise the manuscript accordingly to include this discussion.

Additionally, as discussed in Section 4.3 and 4.4, our theoretical analysis and the toy experiments demonstrate that STG could be effective when the safety function is not a perfect estimator of the true safety probability. Unlike data-space guidance methods, STG jointly considers the model likelihood and safety objective. This allows STG to better preserve the original generative capabilities while improving safety, which supports the practical use of proxy classifiers (e.g., CLIP scores) within our framework.

W3 & Q2: [Experimental setup]

(Hyperparameter settings)

We provided the specific hyperparameter values of STG in Appendix B.1 (lines 717-735). As illustrated in our results (e.g., Figure 3 in the main text), there is a trade-off between the defense success rate and prior preservation. To reflect this, we report multiple variants across a range of hyperparameter values to illustrate this balance.

We set $\gamma$ based on sampling time constraints, as it directly determines the number of update steps during sampling. We usually set to $\gamma=0.6$. The threshold $\tau$ defines which samples are considered unsafe (and thus require updates), and is inherently tied to the range and semantics of the safety function $g$. For example, in the artist-style removal task, where $g$ is defined as the CLIP score difference between ‘art’ and a target artist, the decision boundary is expected near zero, so $\tau$ is chosen close to zero accordingly. The update scale $\rho$ controls the trade-off between safety and prior preservation, and can be adjusted flexibly at inference time depending on the desired balance. Since our method is training-free and modular, $\rho$ can be easily adapted for different use cases or safety definitions without any retraining overhead.

For baseline methods, their respective experimental configurations are provided in Appendix B.2 (lines 737-776), where we describe the setup for each method used in our comparisons.

(SLD variants)

We clarify that the main results (Figure 3 in the main text) include all three SLD variants: Max, Strong, Medium. These are represented by the green solid line, ordered from left to right as Max, Strong, Medium. While SLD-Medium yields the lowest safety performance among the three configurations, it achieves the best performance in terms of preserving the original generative capabilities, particularly in terms of prior preservation and general generation metrics such as FID and CLIP score on COCO. For this reason, we chose SLD-Medium as the representative configuration in Table 2, which focuses on general generation quality. We also report the results for SLD-Strong and SLD-Max in the below table:

[Table B1: SLD-Strong and SLD-Max results for Table 2 in the main paper (COCO evaluation)]

[Table B2: SLD-Strong and SLD-Max results for Table 3 in the main paper (artist-style removal)]

As expected, FID and CLIP score on the COCO deteriorate for SLD-Strong and SLD-Max. For the artist-style removal task, SLD-Max improves removal on the erased concepts but unintentionally affects unerased concepts.

(SAFREE reported values)

The discrepancy in reported DSR for SAFREE originates from different Ring-A-Bell benchmark versions. As noted in the second issue thread on SAFREE’s official Github repository, SAFREE used a custom set from RECE authors, which we used the official adversarial prompts (later released by the Ring-A-Bell authors) following DUO, as mentioned in Appendix B.1 (lines 687-688). A comparison of the two datasets reveals significant differences. The version used in SAFREE consists of human-readable prompts, while the official version we used contains highly adversarial prompts that are much harder to interpret. This distinction explains the observed discrepancy in performance.

We also evaluate STG on SAFREE’s version and reproduce their results; STG still shows superior performance. We will revise the manuscript to clearly specify datasets and configurations.

[Table B3: Results on Ring-A-Bell (nudity) used in SAFREE]

W4 & Q3: [Evaluation metric]

We appreciate the reviewer’s concern regarding potential metric bias in our evaluation protocol.

For the nudity task, we acknowledge that using the same classifier (NudeNet) for both generation guidance and evaluation may introduce bias. To address this, we additionally evaluate the generated images using an alternative metric: an open-source ViT-based NSFW image classifier provided by Falcons.ai. We present a comparison between the original DSR values obtained using the NudeNet detector and those computed using the Falconsai classifier for each point in Figure 3a. While there are some mismatches and variations between the two metrics, we observe that STG consistently achieved higher DSR at similar levels of Prior Preservation (PP). Furthermore, as shown in Table 2 of the main paper, STG also maintains general generation quality.

[Table B4: DSR results using Falconsai classifier]

Regarding violence, we agree with the reviewer’s concern about the reliability of GPT-4o as an evaluation tool. In our work, we followed the evaluation protocol introduced by DUO, which also used GPT-4o for safety assessment for violence.

To further validate this metric, we additionally evaluate our results using the Q16 classifier, an open-source model commonly used for detecting harmful content. Specifically, we collect pairs of safety assessment scores from both GPT-4o and the Q16 classifier across various model and hyperparameter configurations. Then, we compute the Pearson correlation between these scores, which resulted in a value of 0.943. This strong linear correlation suggests that GPT-4o provides safety evaluations that are highly consistent with those of an established classifier like Q16.

We will include the Q16-based evaluation curves and this discussion in the revised version.

Q1: [Configurations in Figure 3]

As mentioned in our response to W3, the SLD results in Figure 3 are represented by the green solid line, with the three variants, SLD-Max, Strong, and Medium, ordered from left to right.

For STG and SDG, the multiple points correspond to different hyperparameter settings, as detailed in Appendix B.1 (lines 726-732) for STG and Appendix B.2 (lines 758-762) for SDG. Specifically, for STG on the nudity task, the points represent different (ρ, τ) combinations: {(1.8, 0.01), (1.3, 0.01), (0.5, 0.01), (0.5, 0.03), (0.5, 0.2)}, plotted from left to right in Figure 3. For the violence task, τ is fixed at 0.05, and ρ is varied across {3, 2, 1, 0.5, 0.2, 0.1}, also in left-to-right order. For SDG, in the nudity task, τ is fixed at 0.01 and ρ is varied as {5, 1, 0.7, 0.5, 0.1}; for the violence task, τ is fixed at 0.1 and ρ is varied as {40, 15, 10, 5, 1}. Again, in both cases, the points correspond to the values listed from left to right.

For DUO, the blue line in Figure 3 corresponds to different values of the hyperparameter β, varied as {100, 250, 500, 1000, 2000}, in left-to-right order. These settings are described in Appendix B.2 (lines 768-770).

Note that SAFREE is shown as a single point in Figure 3. For this method, we adopted the default hyperparameter configuration provided in the official SAFREE code, as noted in Appendix B.2 (lines 742-744).

We will explicitly include these details in the revised version of the paper to ensure clarity and avoid confusion for readers.

Thank you for the detailed rebuttal and additional experiments. The clarifications on hyperparameter settings, baseline configurations, and dataset differences address my main concerns about the experimental setup. The added evaluations with alternative safety metric are particularly helpful and reveal the classifier dependency as an important limitation (nudenet/falcsonsai), which should be stated in the final version. Overall, the rebuttal strengthens the paper, and I encourage explicitly including the new evaluation results and the limitation discussion in the final version.

We sincerely appreciate the reviewers’ careful assessment and constructive feedback. Below, we provide detailed responses to each comment.

W1 & Q3: [Scalability]

Thank you for the insightful comment. We acknowledge that using Stable Diffusion v1.4 with DDIM sampler may be considered relatively outdated. We selected this setting primarily to ensure fair comparison with prior safety-focused works, many of which report results using this same configuration.

To address the concern regarding scalability, we conduct additional experiments using more recent diffusion models, including FLUX, SDXL, and SD3. For each model, we follow the default sampling and configuration settings provided in the diffusers library. In the case of FLUX, we use the guidance-distilled version.

The below table reports the results on the Ring-A-Bell (violence) benchmark for safe generation metrics, and COCO dataset for general generation quality. Note that for this table, the COCO evaluation is conducted using 1,000 images, which accounts for the scale difference in FID values.

[Table A1: Results on recent diffusion models: FLUX, SDXL, and SD3]

As shown in the table, the Ring-A-Bell prompts still induce harmful outputs even in the recent models, as indicated by the low DSR values for the base models. Applying STG improves safety (higher DSR), while maintaining similar image quality as measured by FID and CLIP score. These results demonstrate that STG generalizes well across diverse backbones, pipelines, and fast generation methods. Additionally, the trade-off between image quality and safety can be adjusted via the update threshold hyperparameter $\tau$, providing flexibility depending on application requirements.

W2 & Q5: [Increased sampling time]

We appreciate the reviewer’s observation regarding the increased sampling time. As discussed in Appendix C.3 and D, the added inference cost of STG mainly stems from gradient computations required for updating the text embeddings. To mitigate this overhead, we explore time- and memory-efficient techniques. Specifically, we apply half-precision (FP16) inference during sampling, which significantly reduces both runtime and GPU memory usage, while preserving the safety performance of our method, as shown in the below table. We extended our runtime and memory usage analysis (originally presented in Table 7 of the appendix) by including results with half-precision (FP16) inference. Although STG requires gradient computation during update steps, we observe that FP16 inference maintains nearly the same safety performance. This suggests that existing time- and memory-efficient techniques can be effectively leveraged to address the computational cost of STG in practice.

[Table A2: Sampling time and memory usage with FP16]

Additionally, we would like to highlight an observation from Table 4 in the main text. By adjusting the update step ratio, under comparable sampling time, STG attains a similar defense success rate to SLD while achieving substantially higher safety performance than SAFREE. Also, it does so with better prior preservation than both methods. This demonstrates that STG offers flexible trade-offs between sampling speed, safety, and fidelity, depending on the desired deployment scenario.

Q1: [Different sampler]

Thank you for the thoughtful question. To evaluate how sensitive STG is to the choice of sampler, we conduct additional experiments using the Stable Diffusion v1.4 backbone with the DDPM sampler, keeping all settings identical to those used with the DDIM sampler (as in Figure 3a for the Ring-A-Bell nudity task). Due to the rebuttal policy, we are only able to provide tabular results at this stage. For methods originally visualized as curves, we report three representative points based on DSR (the lowest, median, the highest values). We will include the full curves in the revised version of the paper.

[Table A3: Results with DDPM samplers]

The results show that STG consistently delivers better performance across different samplers, indicating that STG is robust to the sampler.

Q2: [Compatible with fast generation methods]

STG can be integrated with various fast generation methods. For example, in Table 6 of Appendix C.2, we demonstrate successful application of STG with the DPM-Solver sampler, which is a representative fast generation method of the sampler side, on the PixArt-alpha backbone.

Our method only requires access to the mean predicted images $\bar{x}_0$ at intermediate diffusion timesteps during sampling. Since most diffusion models and samplers, including LCMs, provide this, STG can generally be applied without architectural modifications.

We also conduct experiments with Latent Consistency Models on Ring-A-Bell (violence). We find that STG is compatible with this framework as well.

[Table A4: Results on the fast generation method: LCMs]

Q4: [Correction of directional error in table]

Thank you for pointing that out. The arrow direction in the PP column should be an upward arrow ($\uparrow$). PP (Prior Preservation) is a metric where higher values are better. We will make this correction in the revision.