RespoDiff: Dual-Module Bottleneck Transformation for Responsible & Faithful T2I Generation

We thank the reviewer for the valuable feedback and thoughtful comments. We address each of the concerns below:

W1. Clarity

We present RAM and SAM as general transformation modules to keep the framework flexible and extensible. While our main experiments use constant vectors for simplicity and strong empirical performance, our training framework and losses are architecture-agnostic. As shown in Section 7.6 of the supplementary, we ablate alternative architectures such as MLPs and convolutional layers, and find that constant transformations work best in our setting. In short, RespoDiff’s contributions do not rely on any specific parameterization. RAM and SAM can be replaced with richer, input-dependent modules if desired, and our ablations confirm the method remains effective under such changes. We will clarify this in the final version.

W2. Novelty

We appreciate the reviewer’s concern. While it is true that our implementation uses constant transformation vectors similar to SDisc, our contributions are conceptually and technically distinct in several key aspects:

• Score-Matching Objective: SDisc optimizes a reconstruction loss by generating target concept images, adding noise, and training a vector to denoise using the neutral prompt. In contrast, RespoDiff introduces a score-matching objective that explicitly models trajectory shifts between the neutral and target prompts using denoised latents. This provides more direct and structurally grounded supervision for learning responsible transformations. We highlight this difference in Line 104 of the main paper.
• Explicit semantic fidelity preservation: A core novelty of our work is the Semantic Alignment Module, which explicitly regularizes the generated trajectory to remain close to the original diffusion model. This is an issue not addressed in SDisc, which lacks any such semantic fidelity constraint.
• Modular training: We decouple concept steering (RAM) from semantic fidelity preservation (SAM) using dedicated score‑matching and semantic objectives with alternating optimization. This modular design is a key novelty of our approach and distinguishes it from SDisc’s single‑vector reconstruction setup.
• Empirical Performance: RespoDiff consistently outperforms SDisc on fairness and safety metrics (Tables 1–2) while maintaining strong alignment. Qualitatively (Fig. 6; Sec. 7.4.5), SDisc often overfits to narrow stereotypes (e.g., sad or elderly faces, unrealistic profession depictions), likely due to its noise-reconstruction objective. In contrast, RespoDiff preserves profession context while effectively steering toward target concepts.

We believe that the above elements constitute a distinct and novel framework relative to SDisc.

W3. Performance on SDXL

We would like to clarify that while the fairness gain with SDXL may appear less pronounced in absolute terms, it is still a substantial relative reduction (Deviation Ratio drops by ~64% from 0.72 to 0.26). This is a significant improvement considering that SDXL is a more complex and larger model. Additionally, the slight increase in FID (+0.95) is marginal and still within high-quality generation bounds, especially when weighed against the large fairness gains. Importantly, the alignment scores (CLIP and WinoAlign) remain virtually unchanged or even slightly improved. To further validate RespoDiff on SDXL, we now include safety experiments showing that it effectively removes inappropriate content while preserving strong image fidelity.

We also conduct additional experiments to investigate the effectiveness of our approach on a non-UNet architecture such as Flux. Due to resource and time constraints, we performed these experiments with Flux-mini. Since Flux uses transformer architectures without an explicit intermediate bottleneck space (as opposed to UNet architectures), we applied our modules directly to the text representations. Identifying a similarly interpretable latent space within transformers, as in UNet, required further exploration. We evaluated gender debiasing using this setup, and the results are summarized below.

The experimental results demonstrate that our approach can effectively enable responsible generation even on architectures like Flux. We will also provide the qualitative results for Flux in the final version of our paper. We believe this highlights the versatility of RespoDiff beyond UNet-based models. Overall, our work includes extensive evaluations across diverse model architectures, going beyond prior responsible generation methods, to support the robustness and generality of our approach.

Q1. Handling arbitrary user prompts

Robustness to other neutral prompts: To assess the robustness of our approach to the choice of neutral prompt, and in response to the reviewer’s feedback, we conducted an additional experiment. We retrained the man and woman modules using an alternative neutral prompt “a group of people”, and evaluated them on two out-of-distribution prompts: “a group of friends on a picnic” and “a couple of doctors.” This allows us to test whether our method remains effective when trained on more complex neutral formulations. Due to the limitations of CLIP in capturing fairness and alignment for complex prompts, we employed GPT-4o for evaluation. We generated 200 images, uniformly sampling male and female modules, and used GPT-4o to:

• Classify each image as male or female (to compute a deviation ratio).
• Rate alignment to the prompt on a 1–5 scale. Images scoring ≥4 were considered well-aligned.

Our results indicate that training with a different complex neutral prompt also yields balanced gender representations on arbitrary unseen queries such as ‘A group of friends in the picnic’, while preserving prompt alignment, suggesting that the approach generalizes to other generic neutral prompts. We will present the qualitative results that support our analysis in the final version of our paper.

Handling arbitrary user queries: As demonstrated in our experiments, our approach generalizes well across unseen contexts using a small set of general-purpose neutral prompts (e.g., “a person,” “a group of people”). To identify the neutral prompts, we envision training RespoDiff modules on a limited set of such prompts. At inference time, the system can select the most appropriate trained modules by computing similarity between the arbitrary user prompt and the available neutral prompts. For instance, a user prompt like “a group of friends on a picnic” would naturally align with a module trained on “a group of people.”

Q2. Intersectional fairness

In our intersectional fairness experiments (gender and race), we noticed some interference affecting gender more significantly than race. We believe this arises because we initially apply unit scales to both concepts. When composed, the concept with the larger effective residual can dominate, under‑steering the weaker one. To probe this, we varied per‑concept steering scales during composition (e.g., slightly >1 for gender and slightly <1 for race).

Evaluation of Gender fairness before and after Gender, Race composition

Evaluation of Race fairness before and after Gender, Race composition

As shown in the tables, light concept‑specific scaling rebalances the composition, restoring gender steering to near its single‑concept behavior without significantly changing alignment, while race fairness remains stable. We will include these results and qualitative examples in the final version. Additionally, learning the scaling factors dynamically to compose concepts is an interesting direction to explore in future work.

Q3. Sensitivity to $\lambda$

We now present results on the sensitivity of RespoDiff to the hyperparameter $\lambda$.

Empirically, increasing $\lambda$ keeps the trajectory closer to the base model, improving fidelity but weakening target‑concept steering, while decreasing $\lambda$ has the opposite effect. We therefore use $\lambda = 0.5$ that balances fairness and fidelity.

Dear Reviewer skB6,

We are reaching out to kindly initiate a discussion and request if you could take a moment to review our rebuttal and share any remaining concerns you might have. We have carefully addressed all comments raised by the reviewers, and the other three reviewers have responded very positively to our clarifications. If there is anything further we can clarify - beyond the concerns already discussed - we would be very glad to do so within the remaining author-reviewer discussion period. Thanks for your understanding.

Best, The Authors

We thank the reviewer for the constructive and thoughtful feedback. We will carefully revise the manuscript to address all writing-related concerns and improve overall clarity in the final version. Below, we respond to the remaining points in detail.

W1. Discussion of related work on diffusion models

Our primary focus is on enabling responsible (fair and safe) text-to-image generation rather than proposing changes to the diffusion process or diffusion models themselves. Accordingly, we believe Sec. 3 of the main paper provides sufficient background on diffusion models and includes the most relevant related work to support a clear understanding of our approach. However, if the reviewer feels any important references have been overlooked, we greatly appreciate the suggestions and will be happy to include them in the final version.

W2. Regarding Figure 1

We clarify that the trapezoid shapes for RAM and SAM were chosen only to visually indicate transformations within the modules, not dimensionality reduction. Similarly, the arrows and red arrows were intended to simplify the visualization and avoid excessive complexity in the figure. We agree, however, that the current labeling can cause confusion and we will address these labeling inconsistencies clearly in the final version of the paper. Also, we will add a figure that depicts the inference-time process in the final version.

W4/W8a. The choice of $t$

We believe there may be a misunderstanding regarding RespoDiff's training process. In each training iteration, we sample a timestep $t$ randomly and update the shared RAM and SAM modules using a score-matching objective. These modules are not learned per timestep; instead, they are trained to capture average statistics across the entire diffusion trajectory. This random $t$ training choice is stated explicitly in the paper (lines 174 and 201). Accordingly, the training intervention point is a randomly sampled $t$ at each iteration. At inference, the same learned RespoDiff modules are applied at every sampling step to ensure fair and safe generation. To further address the reviewer's concern regarding the choice of $t$, we conducted targeted ablations by restricting the training-time sampling range. Specifically, instead of sampling $t$ from all 1000 DDPM steps, we trained by sampling $t$ under two settings :

1. $t \in [600, 1000]$, representing noisier diffusion steps
2. $t \in [0, 350]$ , representing cleaner diffusion steps.

The results of these controlled experiments are reported below.

It can be observed that the noisier steps $[600, 1000]$ reduce deviation to a large extent but severely hurt alignment; cleaner steps improve alignment but hurt fairness. Sampling across the full range offers the most balanced trade-off and is used in our main experiments. We will include this in the final version of our paper.

W6/Q1. Relevance of results on SD1.4 and adaptation to Flux

Relevance of results on SD1.4 and additional safety results on SDXL : We conducted the majority of our experiments and comparisons using Stable Diffusion v1.4 (SD v1.4), primarily because prior responsible-generation methods (Li et al. 2024) have predominantly used SD v1-based models. To ensure fair and meaningful comparisons, we adhered to the same setup. However, we recognize the importance of evaluating our approach on newer models. Therefore, we provided core fairness results on SDXL as well, reported in Table 4 and Table 5 of the main paper. We also provide qualitative analysis on SDXL in Figure 4, 10 and 11 to support our quantitative results. Since the existing baseline methods do not yet support SDXL, we compared our approach against the default SDXL baseline. Our results clearly demonstrate that RespoDiff effectively mitigates gender and racial biases in SDXL without compromising image fidelity. Furthermore, to substantiate our claims regarding the capability of RespoDiff to handle responsible generation using SDXL, we have now also conducted safety experiments on SDXL and present the corresponding results below. From the table, it can be observed that RespoDiff successfully eliminates inappropriate content while maintaining competitive image fidelity on SDXL.

Adaptation to Flux: Following the reviewer's suggestion, we conducted additional experiments to investigate the effectiveness of our approach on Flux. Due to resource and time constraints, we performed these experiments with Flux-mini. Since Flux uses transformer architectures without an explicit intermediate bottleneck space (as opposed to UNet architectures), we applied our modules directly to the text representations. Identifying a similarly interpretable latent space within transformers as in UNet, required further exploration. We evaluated gender debiasing using this setup, and the results are summarized below.

The experimental results demonstrate that our approach can effectively enable responsible generation even on architectures like Flux. We will also provide the qualitative results for Flux in the final version of our paper. We believe this highlights the versatility of RespoDiff beyond UNet-based models. Overall, our work includes extensive evaluations across diverse model architectures, going beyond prior responsible generation methods, to support the robustness and generality of our approach.

W8b. Joint training of RAM and SAM

During experimentation, we found that jointly training RAM and SAM using our combined loss led to higher deviation ratios and weaker alignment compared to our proposed alternating strategy. We believe this is due to gradient interference where the two modules may inadvertently oppose each other’s objectives, particularly in the early stages of training. In contrast, alternating training separates the updates: RAM is optimized solely with the concept alignment loss, and SAM is updated in a separate step using the semantic loss, conditioned on RAM’s output. This decoupling enables each module to specialize more effectively, reduces interference, and results in more stable optimization. We also explore a related variant using a shared transformation in the supplementary material (Section 7.7, Table 15), where we again observe that alternating, modular updates lead to improved performance. We will include a direct comparison to joint training in the final version of the paper.

Q2. Regularization to keep RAM small

RAM serves as the steering component that shifts the latent representation toward the target responsible concept, while SAM acts as a regularizer that preserves alignment with the original model’s trajectory. We do not impose an explicit constraint to keep RAM small because (i) the required magnitude of transformation varies by concept and (ii) enforcing a small RAM norm could lead to under-steering, reducing the effectiveness of fairness or safety interventions. Instead, we regularize the composite transformation via the semantic loss (weighted by $\lambda$), which keeps the overall trajectory close to the original model.

Shared transformation (Same module for RAM and SAM) : We would like to clarify that the shared transformation design has been evaluated in our supplementary material (Section 7.7; Table 15). Specifically, training a single transformation module using the combined loss underperforms compared to our proposed two-module setup (RAM + SAM) across all metrics. We believe this is because concept steering and semantic alignment are orthogonal objectives. When both are optimized through a single shared transformation, their competing gradients interfere, preventing effective specialization and resulting in suboptimal performance on both fairness and fidelity. In contrast, our modular design separates responsibilities: RAM applies strong, concept-specific transformations, while SAM maintains semantic consistency by remaining close to the identity. This decoupling enables each module to specialize effectively, resulting in improved performance across both dimensions. Additionally, such control is especially valuable for real-world deployment and interpretability, where the ability to tune each component independently is highly desirable. Achieving this level of modularity and transparency is challenging with a shared transformation approach.

We thank the reviewer once again for their valuable feedback and hope that our responses have addressed all the concerns raised.

We thank the reviewer for the valuable feedback and thoughtful comments. We address each of the concerns below:

W1. Reliance on predefined, binary demographic concepts

Our categories are not strictly binary - for race, consistent with prior works (Li et al., 2024), we adopt three categories (Black, Asian, White) as noted in Line 150 of the main paper. For gender, we currently consider man/woman; we acknowledge that this does not capture non‑binary identities and explicitly note this limitation in Section 7.1. We chose these categories to align with prior works (Li et al. 2024) and ensure a fair comparison. Regarding predefined concepts, we view responsible generation as comprising two complementary stages: (i) identifying the relevant fairness/safety concepts, and (ii) mitigating bias or unsafe content given those concepts. Our work addresses (ii) mitigation. Consistent with prior mitigation approaches, we assume the fairness/safety concepts are known in advance, which we acknowledged in the limitations. To the best of our knowledge, this is a common assumption in prior fair/safe generation works (e.g., Li et al., 2024; Gandikota et al., 2024; Shen et al., 2024; Chuang et al., 2023). As noted in the main paper (Line 152), our categories and concepts are chosen to align with these works. That said, we believe that extending to unseen or emergent concepts is practical within the same framework since RespoDiff is modular. Each concept corresponds to a lightweight transformation. Hence, for any new concept, we can train an additional module without changing the backbone. Moreover, our approach can mitigate intersectional biases without additional training by composing existing transformations, as demonstrated in Section 7.4.9, which covers unseen combinations of known concepts and thus partially addresses “unseen” identities. However, we also agree that unseen/emergent concept discovery is important in practice and is orthogonal to our contribution. One approach would be to utilise a world‑model or LLM‑based procedure to propose new or emergent identities [C], after which RespoDiff can learn the corresponding transformations straightforwardly. We will make these points explicit in the main paper.

[C] D’Incà et al., OpenBias: Open-set Bias Detection in Text-to-Image Generative Models, CVPR 2024.

W2/Q3. Robustness to ambiguous prompts or domain shifts

Regarding domain shifts: Our modules are trained using general-purpose neutral prompts (“a person” for fairness, “a scene” for safety), but are evaluated on a wide variety of unseen prompts without any additional fine-tuning. For fairness, this includes profession-related prompts, while for safety, we use the I2P benchmark consisting of real-world, safety-critical queries, introducing a clear prompt distribution shift. Additionally, we assess image fidelity and alignment on the COCO-30K dataset, which spans diverse, non-human-centric domains. Across these varied settings, RespoDiff consistently improves fairness and safety metrics while preserving image quality (see Tables 1–3), demonstrating strong robustness to domain shifts. Moreover, our fairness evaluation is intentionally human-centric, as attributes such as gender and race are specifically defined for human domains.

Regarding ambiguous prompts that implicitly invoke bias: Yes, we explicitly evaluate RespoDiff on ambiguous prompts that implicitly invoke bias, as detailed in Section 7.4.7 of the supplementary. Specifically, we consider prompts such as "a photo of a successful doctor" or "an image of a successful teacher," since terms like "successful" are known to amplify implicit gender biases in generation (Gandikota et al., 2024). Section 7.4.1 provides a detailed discussion on our prompt construction process. As shown in the results in Section 7.4.7, RespoDiff effectively mitigates such biases without degrading semantic fidelity, even in these challenging prompt scenarios. Additionally, we emphasize that our default evaluation setting, which uses professions from the WinoBias set (e.g., "doctor," "CEO"), already tests the model's robustness to implicitly biased prompts. Such prompts commonly trigger stereotypical gendered images (e.g., predominantly male for "CEO" or "doctor"), and addressing these implicit biases is precisely the focus of our mitigation approach

W3/Q1. Assumption on the availability of a neutral prompt

For fairness in human subjects, we adopt the neutral prompt “a person,” which we find generalizes well across diverse human contexts. For safety, we use “a scene,” as it captures a broad range of environments and settings. As shown in Tables 1–3, both prompts effectively support generalization to unseen scenarios. To further assess the robustness of our approach to the choice of neutral prompt, and in response to the reviewer’s feedback, we conducted an additional experiment. We retrained the man and woman modules using an alternative neutral prompt “a group of people”, and evaluated them on two out-of-distribution prompts: “a group of friends on a picnic” and “a couple of doctors.” This allows us to test whether our method remains effective when trained on more complex neutral formulations. Due to the limitations of CLIP in capturing fairness and alignment for complex prompts, we employed GPT-4o for evaluation. We generated 200 images, uniformly sampling male and female modules, and utilised GPT-4o to:

• Classify each image as male or female (to compute a deviation ratio).

• Rate alignment to the prompt on a 1–5 scale. Images scoring ≥4 were considered well-aligned.

Our results indicate that training with a different complex neutral prompt also yields balanced gender representations on arbitrary unseen queries such as ‘A group of friends in the picnic’, while preserving prompt alignment, suggesting that the approach generalizes to other generic neutral prompts. We will present the qualitative results that support our analysis in the final version of our paper.

Strategy for automatically identifying or validating neutral prompts: As demonstrated in our experiments, our approach generalizes well across unseen contexts using a small set of general-purpose neutral prompts (e.g., “a person,” “a group of people”). To identify the neutral prompts, we envision training RespoDiff modules on a limited set of such prompts. At inference time, the system can select the most appropriate trained modules by computing similarity between the arbitrary user prompt and the available neutral prompts. For instance, a user prompt like “a group of friends on a picnic” would naturally align with a module trained on “a group of people.”

Q2. Handling prompts that contain multiple overlapping responsible concepts

RespoDiff can support prompts involving multiple responsible concepts by composing the corresponding modules at inference, without requiring additional training, as detailed in Section 7.4.9. In our intersectional experiments (gender and race), we noticed some interference affecting gender more significantly than race. We believe this arises because we initially apply unit scales to both concepts. When composed, the concept with the larger effective residual can dominate, under‑steering the weaker one. To probe this, we varied per‑concept steering scales during composition (e.g., slightly >1 for gender and slightly <1 for race).

Evaluation of Gender fairness before and after Gender, Race composition

Evaluation of Race fairness before and after Gender, Race composition

As shown in the tables, light concept‑specific scaling rebalances the composition, restoring gender steering to near its single‑concept behavior without significantly changing alignment, while race fairness remains stable. We will include these results and qualitative examples in the final version. Additionally, learning the scaling factors dynamically to compose concepts is an interesting direction to explore in future work.

We thank the reviewer once again for their valuable feedback and hope that our responses have addressed all the concerns raised.

We thank the reviewer for the valuable feedback and thoughtful comments. We address each of the concerns below:

W1/Q1. Shared Transformation and Sensitivity to $\lambda$

Shared Transformation: We would like to clarify that the shared transformation design has been evaluated in our supplementary material (Section 7.7; Table 15). Specifically, training a single transformation module using the combined loss underperforms compared to our proposed two-module setup (RAM + SAM) across all metrics. We believe this is because concept steering and semantic alignment are orthogonal objectives. When both are optimized through a single shared transformation, their competing gradients interfere, preventing effective specialization and resulting in suboptimal performance on both fairness and fidelity. In contrast, our modular design separates responsibilities: RAM applies strong, concept-specific transformations, while SAM maintains semantic consistency by remaining close to the identity. This decoupling enables each module to specialize effectively, resulting in improved performance across both dimensions. Additionally, such control is especially valuable for real-world deployment and interpretability, where the ability to tune each component independently is highly desirable. Achieving this level of modularity and transparency is challenging with a shared transformation approach.

Sensitivity to $\lambda$: We now present results on the sensitivity of RespoDiff to the hyperparameter $\lambda$.

Empirically, increasing $\lambda$ keeps the trajectory closer to the base model, improving fidelity but weakening target‑concept steering, while decreasing $\lambda$ has the opposite effect. We therefore use $\lambda =0.5$ as a knee point that balances fairness and fidelity.

W2. Comparison to LoRA

We now provide comparisons to LoRA-based approaches on the Doctor profession to test a plug‑and‑play alternative for gender debiasing. We trained two concept‑specific LoRAs (“male doctor” and “female doctor”) using SD v1.4 with the default HuggingFace training arguments (≈15k iterations per LoRA). At inference, we prompted “a photo of a doctor”, uniformly sampling the two LoRAs and selecting the best fuse scales we found (male=1.0, female=1.2).

It can be observed that the LoRA approach produced a much higher deviation ratio than RespoDiff, while alignment was comparable. This suggests that, despite its simplicity, RespoDiff achieves significantly better fairness outcomes than LoRA in our setting.

Scalability: While RespoDiff requires training (~1,500 steps) per concept per neutral prompt, this cost is incurred only once and is not tied to specific user prompts. For instance, modules trained using general prompts like “a person” (for fairness) or “a scene” (for safety) generalize effectively to diverse, unseen prompts without further finetuning, as demonstrated in our profession and I2P evaluations. Thus, we believe that the per‑concept cost is amortized over a wide prompt distribution.

Inference Overhead: Compared to LoRA, RespoDiff introduces negligible overhead at inference. Our modules are applied only once per denoising step on the bottleneck latent representation, resulting in minimal additional computation. In contrast, LoRA typically modifies multiple cross-attention layers throughout the U-Net, which introduces repeated low-rank matrix operations and increases both memory and latency. In this regard, RespoDiff offers a plug-and-play mechanism for responsible generation with significantly lower runtime burden.

Q2. Choice of training schedule

We chose the schedule based on loss curves and qualitative analysis. Both the concept and semantic losses converge, and visualisations stabilize by ~1,500 iterations; training much longer (e.g., ≥10,000 iterations) introduced visible artifacts and overfitting to certain visual types. Larger batch sizes slightly increased the concept loss, but the qualitative results were essentially unchanged. Accordingly, we adopt 1,500 iterations with a small batch, which yields stable visuals with converged losses.

W3/Q3. Qualitative results

Beyond Figs. 2–3 in the main paper, we provide extensive qualitative results in the supplementary: Sec. 7.8 (Figs. 8–9) presents SD v1.4 generations across a wide range of professions, Figs. 10–11 present corresponding results for SDXL, and Fig. 7 shows qualitative examples for safe generation. Across these diverse prompts, we believe RespoDiff reliably steers toward the intended target concept while maintaining visual fidelity. While we are unable to include additional qualitative analyses in the rebuttal phase due to the restriction on image submission, we appreciate the reviewer’s feedback and plan to incorporate further qualitative results, such as failure cases in the final version of the paper.

W4/Q4. FID for safety

We agree there is a trade‑off when enforcing safety. We believe this arises from the nature of safety‑oriented transformations and how FID is computed. Safety controls suppress potentially unsafe visual cues (e.g., weapon‑like shapes, exposed‑skin/texture patterns), even when such cues are faint or unintentional. Because these cues are entangled with broader scene/style attributes (lighting, background textures), the transformations can introduce distortions but systematic changes (extra coverage, local occlusions, background adjustments). These shifts move the generated distribution away from COCO’s statistics, which FID, measured in Inception feature space penalizes [A], even when semantic alignment remains high (consistent with our CLIP results in Table 3). By contrast, we believe that fairness‑oriented edits are typically localized identity adjustments (skin tone, facial cues) that preserve scene layout, lighting, and background statistics, so the induced distribution shift and thus the FID change is much smaller. This behavior aligns with findings by Jayasumana et al. [B], where they show that FID increases with high distortions while being less sensitive to low distortions. Even with semantic regularization, some safety cases unavoidably move farther from the base distribution to remove unsafe cues, while keeping semantics intact. Within these constraints, our results indicate that RespoDiff improves safety more than prior methods while avoiding large fidelity degradation, yielding a comparatively stronger safety fidelity balance.

[A] Jung et al. Internalized Biases in Fréchet Inception Distance NeurIPS 2021, DistShift Workshop

[B] Jayasumana et al., Rethinking FID: Towards a Better Evaluation Metric for Image Generation, CVPR, 2024.

W5/Q5. Extension of RespoDiff to unseen concept categories/concept discovery

We appreciate this point. We view responsible generation as comprising two complementary stages: (i) identifying the relevant fairness/safety concepts, and (ii) mitigating bias or unsafe content given those concepts. Our work addresses (ii) mitigation. Consistent with prior mitigation approaches, we assume the fairness/safety concepts are known in advance, which we acknowledged in the limitations. To the best of our knowledge, this is a common assumption in prior fair/safe generation works (e.g., Li et al., 2024; Gandikota et al., 2024; Shen et al., 2024; Chuang et al., 2023). As noted in the main paper (Line 152), our categories and concepts are chosen to align with these works. That said, we believe that extending to unseen or emergent concepts is practical within the same framework since RespoDiff is modular. Each concept corresponds to a lightweight transformation. Hence, for any new concept, we can train an additional module without changing the backbone. Moreover, our approach can mitigate intersectional biases without additional training by composing existing transformations, as demonstrated in Section 7.4.9, which covers unseen combinations of known concepts and thus partially addresses “unseen” identities. However, we also agree that unseen/emergent concept discovery is important in practice and is orthogonal to our contribution. One approach would be to utilise a world‑model or LLM‑based procedure to propose new or emergent identities [C], after which RespoDiff can learn the corresponding transformations straightforwardly. We will make these points explicit in the main paper.

[C] D’Incà et al.,OpenBias: Open-set Bias Detection in Text-to-Image Generative Models, CVPR 2024.

We thank the reviewer once again for their valuable feedback and hope that our responses have addressed all the concerns raised.