Dear Reviewer iJBG,

We sincerely appreciate your time and effort in reviewing our work and providing valuable feedback! Below, we address your concerns point by point.

---

Weaknesses

1. Gendered/Racial Terminology

We deeply appreciate the reviewer's critical insight regarding our use terminology. This is indeed a fundamental concern that we take very seriously.

To clarify our approach: We adopt the exact binary "male" and "female" labels from the established text-to-image diffusion fairness benchmark [1] to ensure consistence. However, we fully acknowledge that these terms represent perceived gender presentation in generated images rather than gender identity, and this distinction is crucial.

We will add a disclaimer and modify our limitation section to acknowledge how binary classification can ignore gender-diverse individuals

---

2. Discrete Racial Categories

We acknowledge the reviewer's concern about the limitations and potential harms of discrete racial categorization. Our racial category is directly inherited from the benchmark [1]:

• WMELH (White, Middle Eastern, Latino/Hispanic)
• Asian (East Asian, Southeast Asian)
• Black
• Indian (South Asian)

We recognize several critical issues with this taxonomy:

• It reflects Western-centric constructions of race that may not be meaningful in other contexts
• It cannot represent mixed-race individuals

In our revision, we will:

• Add a note explaining we use these categories solely for benchmark compatibility
• Acknowledge that these categories are social constructs with limited validity

---

Questions

1. Human Annotators Checklist Error

We sincerely apologize for this oversight in our checklist. To provide complete transparency:

Our human evaluation involved only 3 adult participants who:

• Were recruited as volunteers and provided informed verbal consent to participate
• Were asked to: (a) provide occupation-based text prompts, and (b) rate generated images on a 1-5 scale for demographic diversity and image quality
• Were not exposed to any harmful or sensitive content (only AI-generated professional portraits)

We mistakenly marked this as "N/A" because we initially considered this minimal-risk evaluation of our model's outputs rather than formal human subjects research. We now recognize this was an error in judgment, and any human involvement should have been properly disclosed.

---

2. IRB Approval

We acknowledge the reviewer's valid concern about the lack of formal IRB review. Our reasoning at the time was:

• The evaluation involved minimal risk (3 participants only viewed and rated AI-generated images)
• The task was similar to routine user feedback on model outputs and no personal data was collected beyond ratings
• Participants were colleagues who volunteered with full knowledge of the research purpose

However, we now recognize that this reasoning was insufficient. We apologize for this oversight. We commit to adding explicit statements about informed consent and data handling.

---

3. Bias Amplification

This is an excellent point about cascading bias effects. Our investigation revealed that the language model was the primary source of observed bias. Our current results show:

• Linear probing of vision encoder embeddings achieves high accuracy in demographic classification, confirming these attributes are encoded
• However, when we compare token distributions between neutral and demographic-augmented prompts, the language model shows 99.80% alignment with its implicit biases
• After applying BPO to the language model alone, overall generation bias drops significantly

This suggests that while the vision encoder preserves demographic information, the language model's sampling process is where biased preferences manifest most strongly.

Nevertheless, we acknowledge potential interaction effects:

• The vision encoder's representation might "prime" certain token sequences
• Subtle biases in visual features could compound with language model biases
• Complete debiasing might require addressing both components simultaneously

We will expand our discussion to include these nuances and cite relevant work suggested.

---

Limitations

1. Policy Based on Human Preferences

We acknowledge that our assumption of uniform distribution as "fair" encodes specific value judgments. This approach:

• Aligns with common fairness metrics in literature (demographic parity)
• Matches the evaluation protocol of our benchmark [1]
• Received positive feedback in our (limited) human evaluation

However, we recognize this is just one possible operationalization of fairness:

• It may not suit all contexts
• Different communities might prefer proportional representation matching local demographics
• Some applications might require context-dependent fairness criteria

We will incorporate those paper suggested such as[2][3] and insights about the ethical implications.

---

2. Fine-Tuning on Balanced Data Issues

We acknowledge that balanced training data may not ensure absolute fairness. Issues might be:

• Unnatural associations: Equal representation might create unrealistic scenarios
• Downstream amplification: Balanced training doesn't prevent bias reemergence during deployment

Our approach partially addresses these through the balanced preference loss, which:

• Operates on generation probabilities rather than data frequencies
• Penalizes any systematic preference regardless of training distribution

This means we're optimizing for equal likelihood of generating different demographics. The model learns to be indifferent to demographic choices at the generation level.

However, we acknowledge residual risks might remain, and we will add discussion of these nuanced fairness considerations and cite relevant work suggested.

---

3. Synthetic Data and Distribution Shifts

We appreciate the reviewer's concern about potential artifacts from synthetic data. Our approach addresses these challenges through design and validation:

Why Synthetic Data is Necessary:

• Data Scarcity: No existing large-scale dataset provides occupational images balanced across gender × race intersections. Creating one would require photographing thousands of individuals across all demographic combinations—prohibitively expensive and prone to human annotator biases.
• Controlled Distribution: Synthetic generation enables perfect demographic balance (exactly one image per demographic per prompt), which is very challenging with real-world collection.
• Standard Practice: Latest U-MLLMs like Janus-Pro incorporate ~72M synthetic samples. We follow this established approach while adding targeted bias mitigation.

How We Mitigate Synthetic Artifacts

1. BPO Design: Our balanced preference loss operates on token-generation probabilities. This abstraction layer insulates the debiasing objective from FLUX-specific visual patterns—we optimize for equal likelihood of generating different demographics.
2. Enumeration Strategy: By explicitly generating each demographic variant, we prevent FLUX's implicit biases from skewing the training distribution. Even if FLUX renders certain groups with characteristic styles, our balanced pairing ensures equal representation. (see figure 8 and 9 under Appendix J)

Empirical Evidence of Robustness(see line 247)

• Cross-Domain: 564 stereotype prompts beyond occupations show consistent bias reduction
• Cross-Language: Chinese and French evaluations maintain effectiveness
• Human Validation: User studies confirm improved diversity with minimum quality degradation

While our approach proves effective, we recognize:

• FLUX's aesthetic choices may subtly influence generated styles
• Synthetic-to-real domain gap could affect deployment
• Unknown biases in FLUX's training data might propagate

We will add to our limitations section that investigating mixed synthetic-real training data could further improve robustness.

---

4. Additional Bias Sources

4a. Author Positionality: We acknowledge that our perspectives as researchers inevitably shape this work:

• Our choice to focus on gender and race (rather than age, disability, etc.)
• Our interpretation of "fairness" as demographic parity
• Our methods for evaluation and validation from prior work[1]

These choices reflect our academic training and the specific research context we operate within. Other communities might approach these questions entirely differently.

4b. Category Context and Implications: The choice of gender/race as primary axes of analysis:

• Following standard in bias analysis literature[1]
• May inadvertently suggest these are the only or most important biases
• Could ignore intersectional effects or other forms of discrimination

We will add explicit acknowledgment that our categories are contextual and fairness extends beyond the specific metrics we optimize.

---

Ethical Concerns: We want to assure that participants were treated ethically: they volunteered for the task and were not exposed to any harmful content or risk.

---

Paper Formatting: We appreciate the suggestion. In revised version, We will fully integrate Related Work and Limitations into main text.

---

Reference

[1] Shen, X., et al. (2024). Finetuning Text-to-Image Diffusion Models for Fairness. In Proceedings of the International Conference on Learning Representations (ICLR 2024).

[2] Berg, Hugo, et al. "A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning." arXiv preprint arXiv:2203.11933 (2022).

[3] Smith, Brandon, et al. "Balancing the picture: Debiasing vision-language datasets with synthetic contrast sets." arXiv preprint arXiv:2305.15407 (2023).

---

We hope these clarifications address your concerns. We are grateful for your valuable suggestions and welcome any further feedback!

The author

We sincerely thank the reviewer for raising this important point about IRB approval for human evaluation. Upon careful consideration and consultation with our institution's IRB office, we have made the decision to remove the tiny scale human evaluation component from our paper entirely.

We acknowledge that proper IRB approval should have been sought before conducting any evaluation involving human participants, regardless of scope or risk level. We take research ethics seriously and have learned from this experience.

Impact on Our Contributions: The removal of the human evaluation section does not affect our core technical contributions or conclusions. To put this in perspective: our human evaluation involved only 3 participants providing 3 prompts each (9 prompts total), while our primary evaluation uses hundreds of prompts from established benchmarks, including:

• 1,000 occupational prompts for training
• 50 occupational prompts for testing, 160 images per prompt
• 564 stereotype prompts across diverse domains (personal attributes, activities, healthcare, education)

Our method's effectiveness in reducing demographic bias is comprehensively demonstrated through:

1. Quantitative Metrics:

   • 72% reduction in gender bias (0.89 → 0.25)
   • 46% reduction in race bias (0.48 → 0.26)
   • Consistent improvements across all prompt categories

2. Technical Validation: Our component-wise analysis and ablation studies provide strong evidence for the effectiveness of our Balanced Preference Optimization (BPO) approach

3. Generalization Results: Cross-language evaluation (Chinese, French) and out-of-distribution prompts demonstrate robust bias reduction

The human evaluation was icing on the cake—a tiny supplementary component to our total evaluation. All conclusions in the paper are supported by our extensive computational experiments and automated metrics, which is from existing study in the fairness evaluation benchmarks[1].

Nevertheless, we sincerely apologize for this oversight. To provide complete transparency: We will revise the manuscript to remove all references to human evaluation while maintaining the comprehensive technical evaluation that forms the foundation of our contributions. Additionally, we are working with our IRB office to ensure full compliance with research ethics protocols for any future work involving human participants.

We hope this addresses the reviewer's ethical concerns while preserving the scientific rigor and contribution of our work. We are sincerely grateful for your guidance in helping us meet the highest ethical standards in our research!

---

Reference

[1] Shen et al. (2024). Finetuning Text-to-Image Diffusion Models for Fairness. ICLR 2024.

---

The author

Dear Reviewer,

We are very grateful for your positive reassessment and support! Your feedback has been invaluable to us. We completely agree with your concerns regarding the "Gendered/Racial Terminology" and are committed to making the necessary revisions in our final version.

Thank you for helping us improve our work!

The author

Dear Reviewer 3uNx,

We sincerely appreciate your time and effort in reviewing our work and providing valuable feedback! Below, we address your concerns point by point.

---

(W1) Limited evaluation to occupational prompts

We appreciate the reviewer's observation and want to clarify that our evaluation extends beyond occupational prompts. As mentioned briefly in line 247, we conduct extensive testing on 564 stereotype prompts from the latest bias benchmark[3], covering diverse domains:

• Personal Attributes: "A face of a kind person," "A face of an honest individual"
• Daily Activities: "A face of a person folding freshly washed clothes"
• Healthcare Contexts: "A face of an individual with diabetes"
• Educational Settings: "A face of a sophomore"

These non-occupational prompts constitute a substantial portion of our evaluation and demonstrate consistent bias reduction across contexts. We will add more details to better highlight these results and show that our method generalizes beyond occupations. We agree that occupational bias analysis often compares to real-world statistics—while we don't perform this comparison in this work, our broader evaluation validates the method's general applicability.

---

(W2) Limitation to uniform distribution target

This is an excellent point that touches on fundamental questions about fairness definitions! We acknowledge that uniform distribution is indeed one specific solution of fairness, not a universal solution.

Current Approach: Our BPO loss minimizes pairwise odds ratios to achieve demographic parity, which aligns with widely-used fairness metrics in the literature and our evaluation benchmark[1].

Extension to imbalanced Distributions: Our framework might be adapted to non-uniform targets by modifying the loss function. Instead of having |OR(y_di, y_dj) - 1| in our loss function, we could have |OR(y_di, y_dj) - τ_ij| where τ_ij represents the desired ratio between demographics di and dj. We will add discussion addressing:

1. The Gemini incident as a cautionary example of context-blind uniformity
2. The trade-off between fairness and contextual accuracy

---

(W3) Missing discussion on categorical limitations

We fully agree that our use of discrete demographic categories is a significant limitation that warrants thorough discussion. We will add more details in Limitations:

Gender Categories: We adopt binary labels ("male"/"female") from the established benchmark [1] to ensure comparability. However, we explicitly acknowledge:

• These represent perceived presentation in images, not gender identity
• This binary framework excludes non-binary and gender-diverse individuals

Race Categories: Our four-group taxonomy (WMELH, Asian, Black, Indian) directly follows [1]:

• WMELH: {White, Middle Eastern, Latino/Hispanic}
• Asian: {East Asian, Southeast Asian}
• Black: African and African diaspora
• Indian: South Asian

We acknowledge this categorization:

• Conflates distinct ethnicities (problematic as noted in [2])
• Reflects Western-centric racial constructs
• Cannot capture mixed-race individuals

We will emphasize that we use these categories solely for benchmark compatibility while recognizing their fundamental limitations in representing human diversity.

---

Minor Issues

Thanks for the suggestions!

Line 60: We will add a parenthetical explanation: "where 0 indicates perfect balance and 1 indicates complete skew toward one gender".

Line 463: We will correct this by Revising the sentence.

---

Response to Questions

Q1: Why only occupational prompts?

As clarified above under w1, we actually test on 564 diverse prompts beyond occupations. We will restructure our presentation to make this clearer. Occupations serve as our primary training domain because they're well-studied in bias literature, but our evaluation demonstrates broader applicability.

Q2: Skewed target distributions?

Yes, our method might be adapted. The balanced preference loss could be modified to target imbalanced distribution by replacing the uniform odds ratio target (OR=1) with custom ratios. Specifically:

L_custom = Σ log[1 + σ(log OR(y_di, y_dj) - log τ_ij) - 0.5]²

where τ_ij encodes the desired probability ratio p(di)/p(dj). We will add this in discussion.

Q3: Race label selection?

We inherit the exact categorization from the established fairness benchmark [1] to ensure consistence. We recognize the limitations of these categories (as discussed in W3). We will add a thorough discussion of these limitations and encourage future work to develop more nuanced demographic representations.

---

Reference

[1] Shen, X., et al. (2024). Finetuning Text-to-Image Diffusion Models for Fairness. In Proceedings of the International Conference on Learning Representations (ICLR 2024).

[2] Khan & Fu (2021): One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision.

[3] Xu, C., et al. (2025). MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models. In Proceedings of the International Conference on Learning Representations (ICLR 2025).

---

We sincerely thank the reviewer for these thoughtful critiques, which will substantially strengthen our paper's contribution and clarity.

The author

Dear Reviewer GpU1,

We sincerely appreciate your time and effort in reviewing our work and providing valuable feedback! Below, we address your concerns point by point.

---

1. Reliance on Synthetic Dataset from a Biased Model

We appreciate this insightful critique. Our approach specifically addresses these limitations:

Bias Mitigation at Multiple Levels:

1. Enumeration Strategy: By explicitly generating images for each demographic combination, we prevent FLUX's implicit biases from affecting the demographic distribution in our dataset. Even if FLUX generates stereotypical features for certain demographics, our balanced pairing ensures equal representation(See pipeline in Figure 8 under appendix J and samples in Figure 9).

2. Loss Function Design: Our BPO loss operates on token-generation probabilities. This means we're optimizing for equal likelihood of generating different demographics regardless of FLUX's visual biases. The model learns to be indifferent to demographic choices at the generation level.

3. Empirical Evidence of Robustness(see line 247)

   Cross-Domain: 564 stereotype prompts beyond occupations show consistent bias reduction

   As mentioned briefly in line 247, we conduct extensive testing on 564 stereotype prompts from the latest bias benchmark[1], covering diverse domains:

   • Personal Attributes: "A face of a kind person," "A face of an honest individual"
   • Daily Activities: "A face of a person folding freshly washed clothes"
   • Healthcare Contexts: "A face of an individual with diabetes"
   • Educational Settings: "A face of a sophomore"

   Cross-Language: Chinese and French evaluations maintain effectiveness.

   Human Validation: User studies confirm improved diversity with minimum quality degradation

---

2. Limited Scope of Demographic Attributes

We appreciate the reviewer's concern and acknowledged this in our limitation section in Appendix. Our method currently requires discrete categories. We adopt binary labels ("male"/"female") and category of race from the established benchmark [2] to ensure comparability. This indeed constrains its applicability:

Current Limitations:

• Binary gender categories exclude non-binary identities
• Discrete racial categories cannot capture mixed-race individuals or continuous ethnic variations
• Attributes like age, disability, and culture are not addressed

Challenges for Extension:

1. Continuous Attributes: For age, we could discretize into ranges (20-30, 30-40, etc.) but this loses granularity
2. High-Cardinality Attributes: For culture/nationality (200+ categories), the pairwise loss becomes computationally expensive: O(K²) comparisons.

Possible Solutions such as Group similar demographics hierarchically. We promise to add more discussing these details.

---

3. "Partial Alignment" Phenomenon and Levels of Abstraction

This is an excellent insight that enriches our understanding of the phenomenon.

• Understanding (I→T): Maps diverse visual inputs → single abstract concept
  • "Male professor" image → "professor" (many-to-one mapping)
  • Operates at high semantic level, abstracting away visual details
• Generation (T→I): Maps abstract concept → specific visual instance
  • "Professor" → specific image tokens (one-to-many mapping, different images)
  • Sample from learned distribution

Connection to Our Findings: This abstraction-level difference explains why I→T finetuning fails to debias generation:

1. During I→T training, the model learns to ignore demographic features (abstraction)
2. During T→I generation, the model must still select specific features, defaulting to the highest-probability (biased) mode
3. The "Reversal Curse" is thus a special case of this broader abstraction asymmetry

Implications:

• Debiasing needs to target the specific level where bias manifests
• Understanding tasks may require different debiasing strategies than generation tasks

Really appreciate the insights!

---

4. Choice of Data Synthesizer

We appreciate this insightful suggestion. Before we generated dataset, we did experiment and evaluate multiple image generators for creating our balanced dataset:

Generator Comparison:

1. FLUX.1-dev (Selected):
   • State-of-the-art at the time of our study
   • Superior demographic control - reliably generates requested gender/race attributes
   • Consistent high-quality outputs across most occupational categories(see samples in Figure 9)
   • Stable generation with minimum failure cases
2. Stable Diffusion v1.5 (Evaluated but not selected):
   • Less reliable demographic adherence - often ignored specific gender/race prompts
   • Variable quality, particularly for certain occupations
   • Failed to generate coherent images for some professional categories (e.g., produced low quality and wrong outputs for some occupations)

Our approach would work with sufficiently capable image generator that can reliably produce demographic variations. We will add these details. Thank you for the suggestion!

---

Reference

[1] Xu, C., et al. (2025). MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models. In Proceedings of the International Conference on Learning Representations (ICLR 2025).

[2] Shen, X., et al. (2024). Finetuning Text-to-Image Diffusion Models for Fairness. In Proceedings of the International Conference on Learning Representations (ICLR 2024).

---

We sincerely thank the reviewer for these thoughtful critiques, which will substantially strengthen our paper's contribution and clarity.

The author

Dear Reviewer BVqp,

We sincerely appreciate your time and effort in reviewing our work and providing valuable feedback! Below, we address your concerns point by point.

---

1. Template-based training prompts limitation

We appreciate the reviewer's concern about scalability. In fact, our approach is more diverse than it may initially appear. We promised to add more details in our pipeline figure 8 under Appendix J. We employ a multi-stage augmentation strategy:

1. Prompt Generation: Starting from 1,000 base templates, we use ChatGPT to generate semantically equivalent paraphrases, expanding our prompt set to over much more unique variations. For example, "an associate professor" is paraphrased into "university faculty member," "academic researcher," "college instructor," etc.
2. Semantic Diversity: This paraphrasing ensures our model encounters diverse linguistic expressions mapping to the same occupational concepts, preventing overfitting to specific phrasings and better simulating real-world prompt diversity.
3. Scalability: Our pipeline is scalable - new occupations or domains can be added and automatically expanded through the same paraphrasing process. We demonstrate generalization through our evaluation on 564 stereotype prompts (line 247) that were not seen during training. These stereotype prompts covering diverse domains:

• Personal Attributes: "A face of a kind person," "A face of an honest individual"
• Daily Activities: "A face of a person folding freshly washed clothes"
• Healthcare Contexts: "A face of an individual with diabetes"
• Educational Settings: "A face of a sophomore"

---

2. Synthetic FLUX data vs. real-world images

We acknowledge the reviewer's valid concern about synthetic data artifacts. Our approach specifically addresses these limitations:

1. Necessity of Synthetic Data: Based on our knowledge, there is no existing large scale dataset provides real-world occupational images balanced across both gender and race dimensions. Creating such a dataset manually would be prohibitively expensive and potentially introduce human annotator biases.
2. Widespread Practice: Synthetic data is standard in training unified MLLMs - For example, Janus-Pro uses ~72M synthetic samples for image generation. Our approach follows established practices while adding specific bias mitigation.
3. Empirical Validation: Our extensive evaluations demonstrate:
   • Consistent bias reduction across different test sets (stereotype prompts, cross-language)
   • Human evaluation confirming improved diversity with minimum degradation in quality
   • Generalization to out-of-distribution prompts not seen during training
4. Future Work: We will add to our limitations section that investigating mixed synthetic-real training data could further improve robustness.

---

3. Generalization of training data generation pipeline

Thanks for the suggestion! In figure 8 under appendix J, we will expand our discussion of the data generation pipeline's generalization capabilities:

1. Pipeline Robustness: Our approach explicitly controls for demographic attributes through augmented prompts ("Indian associate professor," "female construction worker"), ensuring coverage across all demographic intersections. This enumeration strategy is model-agnostic and can work with any sufficiently capable image generator.
2. Balanced Pairing Strategy: Each neutral prompt is paired with one image per demographic group, creating a balanced training distribution(see Figure 9 for some samples). This guarantee of balance is independent of the underlying generator's biases.
3. Generalization Evidence:
   • Cross-language evaluation (Chinese, French) shows consistent bias reduction
   • Performance on unseen stereotype prompts demonstrates learned fairness principles as mentioned before

---

4. Core difference between BPO and standard DPO

The fundamental distinction between our Balanced Preference Optimization (BPO) and standard Direct Preference Optimization (DPO) lies in objectives and mathematical formulations:

DPO:

• Objective: Maximize likelihood of human-preferred responses over rejected ones
• Formulation: Binary preference p(y_preferred > y_rejected | x)
• Use case: General alignment with human preferences

BPO:

• Objective: Enforce uniform generation probability across demographic groups
• Formulation: Minimize pairwise odds ratio deviations: min Σ[log(1 + σ(log OR(y_di, y_dj)) - 0.5)²]
• Use case: Designed for demographic fairness

Practical Difference: While DPO optimizes a single preference direction (good vs bad), BPO simultaneously balances pairwise preferences to achieve demographic parity. DPO would require labeling images as "preferred" or "rejected," which is subjective for preference. BPO directly optimizes for equal representation: the optimized model has no preference between generating demographic di or dj, achieving fairness without subjective preference labels.

This balancing addresses the unique challenges of fairness in generation, where the goal isn't to prefer one output over another, but to ensure no systematic preference exists across demographic groups.

---

We sincerely thank the reviewer for these thoughtful feedback, which will substantially strengthen our paper's contribution and clarity!

The author