LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders

Thanks for your constructive comments, and we would like to make the following response.

Q1: The two-stage strategy assumes late-stage bias emergence, which may not generalize to all prompts or attributes.

A1: Thank you for your valuable question! We do not claim that bias always appears only in the later stages of generation for every prompt or attribute. Instead, our two-stage design builds on our theoretically supported finding that diffusion models first generate low-frequency information, then high-frequency details.

For attributes where bias relates to coarse structure, such as scene type or perspective, the debiased CLIP guidance works best early in the denoising process. We test this on artistic style and find that applying the debiased text encoder during the first quarter of the steps ($\frac{1}{4}T$) produces the best results. For attributes linked to fine appearance, such as gender or racial features, it is more effective in later steps.

Our strategy uses a tunable switch-over point, not a fixed “late” phase. Experiments show that this simple, attribute-aware schedule consistently reduces bias without harming image quality. We will clarify in the next version that the two-stage framework is flexible, not rigid.

Q2: Occasional visual artifacts.

A2: Thank you for your question. Visual artifacts are an inherent limitation of the diffusion backbone and can appear even when using the original base model. Since our work focuses on fairness rather than general image quality enhancement, we do not add extra modules designed to remove artifacts. LightFair does not introduce new failure cases or worsen existing ones. It is supported by FID and CLIP-T scores that remain the same or even improve compared to the baseline, as well as visual comparisons of many generated samples. We address this limitation clearly in the Limitation section.

Q3: Evaluation still relies on attribute classifiers and standard fairness metrics, which may themselves be biased. i.e., binary identities.

A3: We acknowledge that single attribute classifiers and individual fairness metrics are not perfect and may introduce their own biases. This is a common challenge across nearly all fairness evaluations in generative models. To reduce this risk, we follow the same evaluation protocol used in previous well-established studies [1,3].

We use two different classifiers for evaluation, as shown in Appendix Table 7 of the original submission. Bias-O uses the classifier from [2], and FD uses the one from [3,4]. Our LightFair achieves the best performance on both metrics, reducing the risk of bias from any single classifier.

For fairness evaluation, we report results on 3 fairness metrics and 7 quality metrics. In addition, we use GPT-4o for evaluation in Reviewer 5saT’s Q2, and conduct a user study in response to Q7. Although individual classifiers may carry their own biases, our LightFair consistently performs best across all evaluation methods, demonstrating its strong overall effectiveness.

In fact, we briefly discussed this issue in Appendix K.1 of the original submission. We will expand on this discussion in the next version. While we hope future work will develop unbiased evaluation metrics, this is beyond the scope of the current paper.

[1] Finetuning Text-to-Image Diffusion Models for Fairness.

[2] Facenet: A unified embedding for face recognition and clustering.

[3] Balancing Act: Distribution-Guided Debiasing in Diffusion Models.

[4] Fair Generative Modeling via Weak Supervision.

Q4: Sensitivity to switching timestep and generalization to non-attribute prompts.

A4: Thank you for your helpful suggestion! Figure 8 shows that setting $\tau$ anywhere between the final step $T$ and around $\frac{3}{4}T$ consistently achieves effective debiasing. This suggests that our method is not sensitive to the exact switching point. We choose $\tau = \frac{3}{4}T$ because it has the least impact on image quality. For clarity, we convert the images in Figure 8 into a table shown below.

Appendix  J.11 further shows that our LightFair performs equally well on prompts without attribute terms. In fact, even the extreme case $\tau = T$, where the debiased CLIP is applied at all steps, does not affect quality or semantic accuracy on such prompts. We will add more visual examples in the next version to better illustrate this robustness.

Q5: Have the authors validated the method on prompts involving non-binary or intersectional identities, and how does the method generalize?

A5: Yes, we have already evaluated LightFair on prompts that span binary, non-binary, and intersectional identities. Section 4.2 of the main paper presents results on gender (binary) and race (non-binary). Appendix J.6 adds results for age (binary), and Appendix J.5 explores the joint attribute gender $\times$ age, which represents an intersectional case. In addition, in response to Reviewer 5saT's Q2, we add new experiments that divide race into four and seven categories, using GPT‑4o as an independent evaluator.

The algorithm generalizes well. The only change needed when moving from binary to non-binary or multi-attribute settings is to redefine the protected group set $A$ in Equations (9)–(11). All other components stay the same. Our experiments confirm that the debiasing effect holds across these cases without reducing image quality.

Q6: Could the adaptive foreground extraction mechanism fail if the background contains semantically entangled features (e.g., uniforms)?

A6: Thank you for your helpful question! The adaptive foreground extraction module is designed to follow the highest-salience regions linked to the prompt’s main subject. Our prompts specify foreground-related content, such as “male doctor”, so the module continues to function even when background elements include attribute-related semantics, like uniforms.

To test a worst-case scenario, we generate 100 images using prompts that explicitly require uniforms in the background and examine the resulting attention maps. In 94 of them, the peak activations still focus on the foreground subject. When we apply LightFair to these images, only 10% extra training epochs are needed to reach the same debiasing level as with standard prompts. This suggests the method remains effective in practice.

We conclude that for more complex scenes, a slight increase in training epochs is enough to maintain performance. We will include these quantitative and visual results in the next version to document the module’s robustness.

Q7: Include user studies or perceptual evaluations to validate qualitative fairness and diversity from a human perspective.

A7: Thank you for your constructive suggestion! Following your suggestion, we conduct an additional user study to support the quantitative results with human judgment. We recruit 30 participants and show each of them images generated by four baselines (SD, FinetuneFD, FairMapping, BalancingAct) and our LightFair.

Participants rate perceived fairness, diversity, and image quality using a five-point Likert scale. The results are shown in the table below. LightFair achieves a mean fairness score of 4.3, compared to 3.8 for the best-performing baseline. It also receives the highest scores for diversity and image quality. Inter-rater agreement, measured by Fleiss' kappa, reaches 0.78.

These findings confirm that the improvements reported in Table 1 of the original submission are clearly perceived by human evaluators without reducing variety. We plan to conduct a broader user study and will include it in the next version.

Q8: Typo.

A8: Thank you for your careful review. We will thoroughly check the paper for typos and correct them in the next version.

Dear Reviewer KsrX,

Thank you once again for your insightful questions and your acceptance of our paper. Please feel free to reach out if you have any further questions or require any additional clarification. We would be glad to assist at any time.

Best regards,

Authors

We deeply appreciate your time and effort in providing such constructive feedback.

Q1: Extension to Diffusion Transformers (DiTs).

A1: Thank you for your hypothetical suggestion. LightFair only applies lightweight modifications to the text encoder and places no constraints on the denoising network architecture. This is one reason why our method generalizes easily to various diffusion models. Following your advice, we replaced the U-Net denoising network with a DiT-based architecture and conducted a preliminary evaluation. The results are shown in the table below. They indicate that our method remains effective even with a DiT-style backbone. We will include this extended experiment in the next version.

Q2: Lack of qualitative evidence for AFE.

A2: Thank you for your helpful suggestion! Stable Diffusion often generates contextual elements not mentioned in the prompt. For example, generating “a doctor” frequently includes a hospital background. Our debiasing-by-distance objective must focus only on the foreground semantics. The Adaptive Foreground Extraction (AFE) module addresses this by isolating the foreground, preventing the background from misleading the optimization.

Figure 4 in the original submission already compares AFE with the baseline. Without AFE, the model takes nearly twice as many training steps to align with the correct semantic center and produces lower-quality images. In contrast, AFE enables faster convergence and generates sharper, less biased outputs.

Due to NeurIPS policy, we cannot add new figures or external links during the rebuttal. However, we will include attention heat maps and timestep visualizations in the next version to highlight AFE’s qualitative benefits.

Thanks for your constructive comments, and we would like to make the following response.

Q1: Impact of Classifier Free Guidance on Bias.

A1: Thank you for your valuable question! We keep the classifier-free guidance (CFG) scale fixed at 7.5 in all experiments and visualizations, as it is the default setting for Stable Diffusion v1.5 and v2.1. Our study focuses on adjusting the text encoder to achieve debiasing in diffusion models. We do not modify other components or parameters; all intermediate steps use their default settings.

Further, we discuss CFG from three perspectives:

1. Necessity of CFG. The conditional score can be written as:

   $$\nabla_{x_t}\log p(x_t \mid y)=\nabla_{x_t}\log p(x_t)+s\bigl(\nabla_{x_t}\log p(x_t \mid y)-\nabla_{x_t}\log p(x_t)\bigr)$$

   where $s$ is the guidance scale. The unconditional term ensures image quality and diversity, while the second term steers generation toward the prompt. Removing the unconditional component harms generation quality, so using CFG is necessary.

2. Robustness across scales. Following the official documentation [1], we test CFG values from 7.0 to 8.5. The results (Bias-Q) are shown in the table below. LightFair consistently reduces bias at all settings. This confirms that our conclusions are not dependent on a specific guidance scale.

3. Relation to bias amplification. A higher guidance scale pushes samples more strongly toward the prompt, which can amplify pre-existing bias. This helps explain why diffusion steps increase the model's bias. Directly correcting the unconditional branch would require retraining on the full data distribution, which is costly. Instead, our method refines the conditional guidance, implicitly addressing both branches. This combined effect achieves bias reduction without sacrificing image quality.

[1] Stable Diffusion with Diffusers.

Q2: $\lambda$–bias trend unclear.

A2: Thank you for your valuable suggestion! We agree that a monotonic trend is expected when the regularization weight $\lambda$ varies within a meaningful range. In fact, Figure 7 already shows this trend once outliers are excluded.

In Figure 7(a), the sequence $\lambda_1=5\to 2\to 1\to0.5$ forms a smooth downward curve. The best Bias-O/Bias-Q balance is achieved at $\lambda_1=1$. In contrast, $\lambda_1=10$ and $0.1$ deviate from the main pattern due to over- and under-regularization. A similar trend appears in Figure 7(b). The sequence $\lambda_2=1\to 0.5\to 0.2\to0.1$ creates a consistent slope, while $\lambda_2=0.05$ and $0.01$ fall outside the stable range. We will clarify this in the next version.

Q3: Pictures are highly pixilated.

A3: We apologize for the distracting pixelation in the PDF. The original images are high-resolution ($512 \times  512$ or larger) and average around 2 MB each. Embedding them directly would have created an oversized file, exceeding the OpenReview’s limits and making the download difficult. To meet these constraints, we applied image compression, which unfortunately introduced visible artifacts. The original, uncompressed images remain clear. We will provide a link to view the high-quality images in the next version.

Q4: Some minor formatting issues.

A4: Thank you for pointing out these minor formatting issues. The “$859.52M \to 3.69M$” in Line 87 means that LightFair reduces the parameter from $859.52M$ (full fine-tuning) to just $3.69M$, while still improving performance. We will clarify this in the next version. The unintended leading whitespace will also be removed.

In addition to the questions mentioned above, we provide our perspectives on the reviewer's open-ended discussion.

Discussion: The definition of model fairness.

Answer: Thank you for the helpful discussion! While we fully agree that any definition of “fairness” must be interpreted within its application context, our work is grounded in Equalized Odds, a widely accepted machine learning group fairness criterion [1,2]. We extend this concept to diffusion models by introducing a complementary “Equalized Quality” metric, which captures perceptual image quality. We follow the same setting used in previous diffusion fairness works[3,4]. Although these choices do not resolve the broader societal debate on the meaning of fairness, they align our analysis with established practices and ensure fair comparison with existing work.

That said, the issue you raised is indeed important and urgent. As stated in Appendix L, achieving fairness requires joint efforts from researchers, policymakers, and practitioners. This remains one of our central goals. We will include a more detailed discussion of this topic in the future work section of the next version.

[1] Hardt M, Price E, Srebro N. Equality of opportunity in supervised learning. NeurIPS 2016.

[2] Ghassami A E, Khodadadian S, Kiyavash N. Fairness in supervised learning: An information theoretic approach. ISIT 2018.

[3] Shen X, Du C, Pang T, et al. Finetuning Text-to-Image Diffusion Models for Fairness. ICLR 2024.

[4] Parihar R, Bhat A, Basu A, et al. Balancing act: distribution-guided debiasing in diffusion models. CVPR 2024.

Thank you for your constructive comments. Below, we provide detailed responses to your specific comments:

Q1: Strong independence assumptions.

A1: Thank you for your valuable question! We respond on both empirical and theoretical grounds as follows:

1. From an empirical perspective, the assumption is a weak one specific to training. Although attributes and concepts are rarely independent in the real world, the images we use for training are generated by Stable Diffusion. By controlling the prompts, we can easily enforce independence between attributes and concepts. For example, in our experiments, we generate equal numbers of images for different attributes to compute semantic centroids. In this controlled setting, the condition $\mathbb{P}(a, c) = \mathbb{P}(a)\mathbb{P}(c)$ clearly holds.
2. From a theoretical perspective, we further relax Assumption E.3 to a softer condition: $1 - \epsilon \le \frac{\mathbb{P}(a \mid c)}{\mathbb{P}(a)} \le 1 + \epsilon.$ (Assumption E.4) We also prove that under this relaxed assumption, the induced probability error from the distance constraint is on the order of $O(\epsilon)$, where $\epsilon$ is a small constant. In our training data, $\epsilon$ is always less than 0.01.

Assumption E.4. For any attribute $a$ and concept $c$, there exists $\epsilon \in [0, 1)$ such that

$$1 - \epsilon \leq \frac{P(a \mid c)}{P(a)} \leq 1 + \epsilon, \quad \forall a \in A,\, c \in C.$$

When $\epsilon = 0$, this reduces to the original Assumption E.3.

Theorem. Under Assumptions E.1, E.2, and E.4, let $d_i = \left\| f(a_i, c) - f^t\big(P(\cdot, c)\big) \right\|,$ and let $\rho$ be the bandwidth of the Gaussian kernel. If Equalized Odds holds, i.e.,

$$\mathbb{P}(a_i \mid P(\cdot, c)) = \mathbb{P}(a_j \mid P(\cdot, c)) \quad \text{for any } a_i, a_j \in A,$$

then the corresponding embedding distances satisfy

$$\left| d_i^2 - d_j^2 \right| \le 2\rho^2 \cdot \left| \ln \frac{1+\epsilon}{1-\epsilon} \right| \approx 4\rho^2 \epsilon.$$

In particular, as $\epsilon \to 0$, the bound vanishes. Equalized Odds then implies exact equality in embedding distances, recovering the original theorem.

Proof. First, since the input of the diffusion denoising process is the prompt embedding produced by the text encoder, Equalized Odds can still be reformulated as

$$\mathbb{P}\bigl(a_i \mid f^{t}(P(\cdot,c))\bigr) = \mathbb{P}\bigl(a_j \mid f^{t}(P(\cdot,c))\bigr), \qquad\forall\,a_i,a_j\in A, \tag{17}$$

where $f^t(\cdot)$ denotes encoding by the CLIP text encoder, and $P(\cdot)$ is shorthand for $prompt(\cdot)$.

According to Assumption E.2, we have

$$\mathbb{P}\bigl(c \mid f^{t}(P(\cdot,c))\bigr)=1. \tag{18}$$

Under the weak‑independence Assumption E.4, for any $a_i,a_j\in A$

$$\frac{1-\epsilon}{1+\epsilon} \le \frac{\mathbb{P}(a_i,c \mid f^{t}(P(\cdot,c)))}{\mathbb{P}(a_j,c \mid f^{t}(P(\cdot,c)))} \le \frac{1+\epsilon}{1-\epsilon}. \tag{19}$$

Let $f(a,c)$ represent the joint encoding of the attribute $a$ and concept $c$. Because $f(a,c)$ captures both $a$ and $c$, inequality (19) can be rewritten as

$$\frac{1-\epsilon}{1+\epsilon} \le \frac{\mathbb{P}\bigl(f(a_i,c)\mid f^{t}(P(\cdot,c))\bigr)}{\mathbb{P}\bigl(f(a_j,c)\mid f^{t}(P(\cdot,c))\bigr)} \le \frac{1+\epsilon}{1-\epsilon}. \tag{20}$$

According to Assumption E.1 and [12], the conditional probability $\mathbb{P}\bigl(f(a_i,c)\mid f^{t}(P(\cdot,c))\bigr)$ can be modeled by a Gaussian distribution whose mean is $f^{t}(P(\cdot,c))$. Hence

$$\mathbb{P}\bigl(f(a_i,c)\mid f^{t}(P(\cdot,c))\bigr) =\frac {\exp\Bigl(-\tfrac{\lVert f(a_i,c)-f^{t}(P(\cdot,c))\rVert^{2}}{2\rho^{2}}\Bigr)} {\displaystyle\sum_{a_k\in A} \exp\Bigl(-\tfrac{\lVert f(a_k,c)-f^{t}(P(\cdot,c))\rVert^{2}}{2\rho^{2}}\Bigr)}, \tag{21}$$

where $\rho$ is a constant depending only on $f^{t}(P(\cdot,c))$.

Combining (20) and (21) gives

$$\frac{1-\epsilon}{1+\epsilon} \le \exp\Bigl(-\tfrac{\lVert f(a_i,c)-f^{t}(P(\cdot,c))\rVert^{2}-\lVert f(a_j,c)-f^{t}(P(\cdot,c))\rVert^{2}}{2\rho^{2}}\Bigr) \le \frac{1+\epsilon}{1-\epsilon}. \tag{22}$$

Taking natural logarithms and absolute values on (22) yields

$$\bigl|\lVert f(a_i,c)-f^{t}(P(\cdot,c))\rVert^{2}-\lVert f(a_j,c)-f^{t}(P(\cdot,c))\rVert^{2}\bigr| \le 2\rho^{2}\, \Bigl|\ln\frac{1+\epsilon}{1-\epsilon}\Bigr|. \tag{23}$$

Recalling the definition $d_i=\lVert f(a_i,c)-f^{t}(P(\cdot,c))\rVert$ and using $\ln\frac{1+\epsilon}{1-\epsilon}=2\epsilon+O(\epsilon^{3})$, we have

$$\left|d_i^{2}-d_j^{2}\right| \le 2\rho^{2}\,\left|\ln\frac{1+\epsilon}{1-\epsilon}\right| \approx 4\rho^{2}\epsilon. \tag{24}$$

As $\epsilon\to0$, the logarithmic term vanishes, so Equalized Odds enforces $d_i^{2}=d_j^{2}$, recovering the exact equality of embedding distances established under the stronger independence assumption.

Q2: Limited coverage of non-binary or continuous attributes. Current formulation handles categorical attributes only. Can the method be extended to continuous or multi-label attributes (e.g., non-binary gender, mixed race)?

A2: Our formulation already generalizes beyond simple binary categories. The main paper evaluates race using three classes. In response to your suggestion, we add new experiments that divide race into four (White, Asian, Black, and Indian) and seven (White, Middle Eastern, Latino Hispanic, East Asian, Southeast Asian, Black, and Indian) levels, using GPT‑4o as an independent evaluator. LightFair still performs strongly, confirming that the method scales well to finer attribute granularity. Appendix J.5 further presents results on the joint attribute of gender and race. This represents a true multi-label setting and also shows consistent improvements.

We follow the standard protocol in the fairness-in-diffusion literature, which focuses on attributes with reliable visual classifiers. However, we agree that fully continuous or user-defined attributes remain an open challenge.

Our distance constraint only requires redefining the protected group set and adjusting the centroids. In principle, it can support continuous attributes by sampling representative anchors along the spectrum or by optimizing against attribute regressors. We will explore these directions in future work and outline them in the next version.

Q3: Metric dependence on external classifiers.

A3: We acknowledge that single attribute classifiers are not perfect and may introduce their own biases. This is a common challenge across nearly all fairness evaluations in generative models. To reduce this risk, we follow the same evaluation protocol used in previous well-established studies [1,3].

We use two different classifiers for evaluation, as shown in Appendix Table 7 of the original submission. Bias-O uses the classifier from [2], and FD uses the one from [3,4]. Our LightFair achieves the best performance on both metrics, reducing the risk of bias from any single classifier.

In addition, we use GPT-4o for evaluation in Q2, and conduct a user study in response to Reviewer KsrX’s Q7. Although individual classifiers may carry their own biases, our LightFair consistently performs best across all four evaluation methods, demonstrating its strong overall effectiveness.

In fact, we briefly discussed this issue in Appendix K.1 of the original submission. We will expand on this discussion in the next version. While we hope future work will develop unbiased evaluation metrics, this is beyond the scope of the current paper.

[1] Finetuning Text-to-Image Diffusion Models for Fairness.

[2] Facenet: A unified embedding for face recognition and clustering.

[3] Balancing Act: Distribution-Guided Debiasing in Diffusion Models.

[4] Fair Generative Modeling via Weak Supervision.

Q4: Limited method innovation.

A4: We sincerely apologize if the description of our contributions caused any misunderstanding. Below, we clarify the differences between our LightFair and prior work:

1. Although the Collaborative Distance-Constrained Debiasing Strategy (CDDS) may appear similar to center-based regularizers, they differ in two key ways. First, our debiasing objective starts by requiring equalized probabilities across protected groups. Theorem 3.1 transforms this probabilistic condition into an equivalent distance constraint that is easier to optimize. CDDS is thus derived directly from theory, not adapted from existing center losses. Second, CDDS enables cooperative debiasing across modalities. It aligns the text-conditioned diffusion path with both visual and textual centroids. In contrast, center-based regularizers are usually added as a single loss term and do not consider cross-modal interactions.
2. Multi-stage sampling is only an intermediate step in our overall pipeline. Although it is mentioned in prior work, there are two key differences. First, previous multi-stage sampling methods typically generate low-resolution images, then upscale them to high resolution. These methods aim to improve sampling efficiency. In contrast, our approach uses different text encoders in the two stages. It is designed to enhance image quality during model correction. The motivation and method are fundamentally different. Second, prior methods are largely empirical and based on experimental intuition. We go further by providing a comprehensive theoretical and empirical analysis of the underlying mechanism.

In summary, our main contribution lies in lightweight debiasing of diffusion models from the perspective of the text encoder. All components are supported by both theoretical and experimental evidence. We believe our method offers a novel and valuable contribution to the fairness community.