FairGen: controlling fair generations in diffusion models via adaptive latent guidance

We thank the reviewer for the feedback and constructive comments. We have carefully considered each point raised and provide detailed responses below.

Q1: Writing Improvements

We appreciate the feedback on the writing and organization! We revised the text part and also refined the figure in our updated manuscript.

Q2 Clarification on Orthogonality (Line 218)

We apologize for the confusion regarding the term "orthogonality." In this context, orthogonality refers to the independence between the guidance for fairness and the original content of the prompt, which also aligns with the prior work [1]. Our method aims to adjust the generation process to mitigate bias without altering the semantic content specified by the user. We want to ensure that the fairness adjustments do not interfere with or distort the user's intended output. Therefore, the guidance vector (which is the difference of two vectors in the original space) would show "orthogonality" to the original vectors.

[1] Wang X, Chen T, Ge Q, et al. Orthogonal subspace learning for language model continual learning[J]. arXiv preprint arXiv:2310.14152, 2023.

Q3: HBE Dataset Split Proportions

The allocation of 40% training, 10% validation, and 50% testing in the HBE dataset is designed to prioritize evaluation robustness. Given that our method operates primarily during inference and does not require extensive training data (held for other methods), we allocated a larger portion of the data to testing to thoroughly assess the model's performance across diverse scenarios. This allows us to evaluate the generalization and fairness of the model more comprehensively.

Q4: Source of the Target Distribution in Figure 1

The target distribution in Figure 1 is determined based on fairness objectives and societal considerations. In the absence of a known correct ratio for attributes, we assume a uniform distribution (e.g., 50:50 split) as a starting point for fairness. This assumption aligns with principles of equal representation when specific demographic data is unavailable or inapplicable. We acknowledge that in real-world scenarios, the true distribution may differ. Our method is flexible and allows for the target distribution to be adjusted based on reliable data or policy requirements.

We thank the reviewer for the feedback and constructive comments. We have carefully considered each point raised and provide detailed responses below.

Q1: More Details and More clarifications on the Indicator Guidance Model

We apologize for the lack of clarity regarding the indicator guidance model. We expanded Section 3.2 to provide a detailed formulation of the indicator guidance model, to ensure that readers can fully understand and reproduce our method. In brief, each cluster represents the set of prompts regarding the same main objective (e.g., computer programmer) and the generation statistics is basically the number of generations for different attributes (males/females/white/nonwhite/....). According to these statistics, we can determine the target attribute for the current generation to achieve a certain target distribution. We use the same attribute detection module as test time (i.e, VQA model). We assume all this information are maintained in a global memory in the central server. We also would like to clarify that simialr to existing literature, we do not test such attribute-aware prompts in our benchmark. If we aim to consider such prompt, we can simply have an attribute detection module which enforces no attribute control given a positive detection result.

Q2: Discussion on Computational Costs

Thank you for pointing out the need to discuss computational costs. We added additional comparisons of runtime in Section 5.4. As a training-free method, FairGen induces no training computational costs. In the inference stage, although FairGen induces $1+k*|\mathcal{A}|$ noises estimates in each diffusion step, where $k$ is the number of bias types we want to control and $|\mathcal{A}|$ is the average number of sensitive attributes, the adaptive guidance is only enforced at a small portion of intermediate diffusion steps (details in Section 5.3.3). Additionally, the noise estimates for different attributes are independent and parallelized in the inference. Therefore, FairGen only leads to marginal runtime overhead compared to the baselines while mitigating the bias significantly.

Dear Reviewer kpaF,

As the end of the discussion period is approaching, we would like to gently remind you of our responses to your comments. We wonder whether your concerns have been addressed and appreciate any further questions or comments you might have.

Sincerely,

Authors of Submission 5559

We thank the reviewer for the feedback and constructive comments. We have carefully considered each point raised and provide detailed responses below.

Q1: Comparison to Baselines Beyond Diffusion Models

Thank you for highlighting the need for broader comparisons. While our focus is on text-to-image diffusion models due to their recent prominence and unique challenges, we agree that comparing FairGen with conventional fairness learning could provide valuable insights. Indeed, the finetuning-based method presents one conventional fairness learning algorithm which finetunes the model on a fair data distribution. Via detailed comparisons to it in Section 5.2, we validate that FairGen achieves more fair generations than this strong baseline.

Q2: Lack of Detailed Error Analysis

Thank you for the question! One failure mode of FairGen is that it sometimes induced figures with no target objectives (e.g., humans) and we also find that manipulation of multi-attribute is challenging, which is also analyzed in Section 5.2.

Q3: Discussion on Computational Costs

Thank you for pointing out the need to discuss computational costs. We added additional comparisons of runtime in Section 5.4. As a training-free method, FairGen induces no training computational costs. In the inference stage, although FairGen induces $1+2|\mathcal{A}|$ noises estimates in each diffusion step, where $|\mathcal{A}|$ is the number of sensitive attributes, the adaptive guidance is only enforced at a small portion of intermediate diffusion steps (details in Section 5.3.3). Additionally, the noise estimates for different attributes are independent and parallelized in the inference. Therefore, FairGen only leads to marginal runtime overhead compared to the baselines while mitigating the bias significantly.

Q4: Inclusion of Visualization Results

We agree that intermediate visualizations can enhance the understanding of our method's effectiveness. We add some visualization examples in Section 5.5.

Dear Reviewer 15oU,

As the end of the discussion period is approaching, we would like to gently remind you of our responses to your comments. We wonder whether your concerns have been addressed and appreciate any further questions or comments you might have.

Sincerely,

Authors of Submission 5559

We thank the reviewer for the feedback and constructive comments. We have carefully considered each point raised and provide detailed responses below.

Q1: Presentation Issue with Figure 1

We appreciate the reviewer highlighting the inconsistency in Figure 1. We acknowledge that the adaptive latent guidance module indeed takes the previous step's denoised latent result as an input, which was not clearly depicted in the original figure. We revised Figure 1 to accurately reflect all inputs to the adaptive latent guidance module, ensuring it includes the target attribute, input prompt, and the denoised latent result from the previous timestep.

Q2: Effectiveness of the DPO Component

Thank you for pointing out the minimal contribution of the DPO component in Table 6. We recognize that the gap gained by DPO appears minimal. However, the DPO component can consolidate the guidance to enhance the awareness of nuanced differences that may lead to effectiveness for target attribute control. We clarified that DPO is not an essential block of FairGen and more sophisticated cases may boost the benefit of DPO training.

Q3: Practical Significance Regarding Memory of Previous Generations

Thank you for the question! According our memory-based indicator guidance module in Section 3.3, we perform conditional generation based on past statistics. In other words, the target attribute is determined such that the generation distribution would be close to the target distribution. Even when the number of generations is small, such mechanism is still applicable. For example, if we aim for balanced gender distribution and the first generation is a male, then we will target a female generation in this case.

Q4: Novelty in Relation to Prior Work

We acknowledge the reference to the work "Safe Latent Diffusion" [1]. First, we would like to point out that [1] focus on unsafe content moderation, while FairGen aims for unbiased generation. Therefore, [1] cannot be viewed as a direct baseline for comparisons. Second, while both methods employ inference-time guidance strategies, our approach differs significantly in both methodology and application. The adaptive latent guidance module in our work is specifically designed to mitigate biases related to sensitive attributes by dynamically adjusting the latent space during inference. Moreover, our indicator guidance module introduces a novel mechanism that leverages historical generation data to inform the guidance direction, which is not present in [1]. This combination allows for a more targeted and effective bias mitigation strategy.

Dear Reviewer inXs,

As the end of the discussion period is approaching, we would like to gently remind you of our responses to your comments. We wonder whether your concerns have been addressed and appreciate any further questions or comments you might have.

Sincerely,

Authors of Submission 5559