Reliable Image Quality Evaluation and Mitigating Quality Bias in Generative Models

We thank the reviewer for the insightful comments and suggestions.

W1, Q1: Definition of Quality Bias, Clarifying Our Fairness Objective

The quality evaluation is based on the distance between representations of generated and reference images. We clarify that the notion of quality bias in our paper follows definition A, not B.

• A: The quality of generated images given different genders should be equal, i.e. $D(f(A_{gen}), f(A_{ref})) - D(f(B_{gen}), f(B_{ref}))$ as mentioned in Section 3.
• B: The quality shift of generated images given different genders from reference images given different genders should be equal, i.e.$D(f(A_{gen}), f(A_{gen+shift})) - D(f(B_{gen}), f(B_{gen+shift}))$

To measure the fairness A, FID is widely used where $f$ is InceptionV3, and $D$ is Frechet Distance. However, we argue that FID can fail to reliably capture this bias A. As illustrated in Figure 2, the an evaluation metric may incorrectly rate a poorly generated female nurse image as being of higher quality than a well-generated male image. This makes trustworthiness issue in quality evaluations and motivates our key research question (see Lines 56, 72, and 148): "Which evaluation metric is reliable for fairness in image quality?"

To answer this, we construct a controlled dataset with known quality shifts and examine how different image encoders respond across groups with equally degraded inputs. This setup is not a redefinition of quality bias, but a means to asseses whether a metric reflects the true underlying quality gap fairly across groups.

In summary:

• Our definition of quality bias is A, not B.
• We do not aim to ensure the same quality shift across groups as a fairness goal.
• Rather, we utilize controlled quality shifts as a diagnostic tool to assess the reliability of evaluation metrics in detecting quality bias.
• Quality shift is used only to find out reliable image encoder for evaluation by isolating the bias inherent in the encoder.

We hope this distinction clarifies the rationale behind our experimental setup.

W2: Constructing Dataset

While traditional image degradation techniques (noise addition, color jittering, or blurring) are commonly used in image quality research, they do not reflect the nature of degradation observed in generative models. Our objective is to simulate the quality failures specific to image generation, rather than conventional distortions.

Our controlled dataset construction is grounded in prior work. We follow the degradation protocols used in [1] and [2], both of which analyze generation failures that emerge from sampling dynamics (e.g., truncated denoising or reduced steps). These reflect real-world failure modes of diffusion models.

Moreover, [3] draw a clear distinction between traditional distortions and generative artifacts. They show that defect classifiers trained on conventional degradation types fail to generalize to generation-induced artifacts, further supporting our decision to simulate degradation through the generative process itself.

In summary, our degradation strategy:

• Accurately reflects generation-specific quality failure modes.
• Is aligned with prior literature focused on evaluating and analyzing diffusion model failures. This approach provides a more valid basis for assessing the fairness and reliability of evaluation metrics in generative models.

[1] A. Borji. Qualitative failures of image generation models and their application in detecting deepfakes, 2023.
[2] Kim et. al. Model-Agnostic Human Preference Inversion in Diffusion Models, 2024.
[3] Wang et al. CNN-generated images are surprisingly easy to spot... for now." CVPR. 2020.

W3: Prompt-Related Bias

We appreciate the reviewer’s concern regarding the possible mismatch between prompts and actual attributes in generated images. To address this, we adopt the generation pipeline proposed by [4], which was designed specifically to mitigate such inconsistencies.

As noted in Line 175, [4] employs a carefully controlled prompt structure with explicit attribute descriptions (e.g., gender, race) and performs post-processing using an inpainting model to correct any deviations. We directly reuse both the prompt format and pipline from [4], ensuring alignment between prompt and image.

Moreover, we manually reviewed all generated images in our dataset. No instances of attribute mismatch were observed. Although our current pipeline addresses this concern effectively, we agree that integrating automatic consistency scoring could provide additional rigor in future work.

[4] N. Lui, et. al. Leveraging diffusion perturbations for measuring fairness in computer vision. AAAI 2024.

W4: No Miscategorization Assumption

We respectfully clarify that our work does not rely on any miscategorization assumption, nor does our dataset contain such issues. Figure 3 is just a synthetic illustration (see Appendix A) designed solely to highlight the potential pitfalls of group-agnostic evaluation metrics like FID. The figure does not assume or imply any misclassification of group identity in real or generated samples. Rather, it demonstrates that even under idealized and controlled group separation, a group-agnostic score may yield misleading interpretations.

Thus, our method is not predicated on any miscategorization scenario, and the evaluation is built on valid, well-annotated data.

W5-1: On Reference Set for DQA-Guidance

We agree with the reviewer that it is reasonable to use a desired-quality dataset as the reference for guidance. However, using the FairFace dataset introduces two concerns:

• Quality Bias in Real Image: As acknowledged in the FairFace paper, the dataset contains quality imbalance, e.g., some photos captured by professional photographers, introducing group-level quality disparities.
• Limited Coverage: FairFace includes only facial images, whereas failures in generative models often occur beyond the face e.g., in limbs, which would not be captured by facial datasets alone.

We prioritize real datasets whenever feasible. As noted in Appendix G.1 (line 624), we use a real chest X-ray dataset for medical image generation. This dataset provides consistent quality across genders, with quality control ensured by human annotators.

W5-2: Different image quality assessment

Thank you for the suggestion. In Appendix I, we also assessed DQA for alternative image quality assessment (IQA) methods. These include VQA-based evaluations and general-purpose IQA models.

We summarize the alternative evaluation strategies below:

• VQA-based methods: Use visual question answering models (e.g., PaliGemma) to detect unrealistic artifacts by asking questions such as: "Is this image real or fake?"
• General IQA models: Evaluate perceptual quality using models like TOPIQ, which are trained to detect visual degradation (e.g., blur, noise, color issues).

These results indicate that DQA-Guidance does not compromise overall image quality, while effectively improving fairness across demographic groups even with other image quality assessment.

Q2: Clarifying Real Images and Terminology in Figure 4

Thank you for pointing out the potential confusion. The real images shown in Figure 4b are included as an illustrative example to help interpret how different image types are positioned in the embedding space. These were not used in any experiment. As noted, they were sourced from publicly available Google Image Search, and we will clarify this explicitly in the revised version.

Regarding the criteria for well- and poorly-generated images, we follow a consistent convention throughout the paper. The "reference" set refers to well-generated images (as described in Figure 5), generated with full sampling steps, lower noise, and a refinement stage. The "poor-generated" images (e.g., in T6) are created under a degraded setting with fewer diffusion steps, stronger noise, and no refinement module, resulting in visibly lower-quality outputs.

To address your question about the gender in the poor-quality set (T6): as discussed in W3, all images maintain their intended gender and profession. As shown in Figure 8, these images remain identifiable in terms of gender and profession but exhibit clear distortions in face, limbs, and background, which reduce overall perceptual quality.

We will revise the text and figure caption to ensure that the usage of "reference" and the nature of the real and generated images are unambiguous.

Q3: Scale of FD with DINO

Thank you for raising this point and for citing helpful references. The DINO model we used is DINOv1 [6] with ResNet-50, not DINOv2 [7] with ViT-L/14 as discussed in the cited papers. This difference explains the discrepancy in FD values, as the training objectives, regularization strategies, and architectures (CNN vs transformer) differ significantly.

Additionally, the references [8] do not make a general claim that FD values are always higher with DINO than with Inception. For example, [9] (Fig. 2) mentions that FD with DINOv2 is higher in their setup but attributes this to differences in sample count. Thus, direct comparison of absolute FD values across models, datasets, and evaluation settings is not meaningful, and we suggest interpreting FD within-model rather than across different architectures.

[6] Emerging properties in self-supervised vision transformers.
[7] Dinov2: Learning robust visual features without supervision.
[8] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.
[9] Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples.

Human Perception Analysis

We appreciate the reviewer’s comment regarding the lack of human evaluation.

In this work, we focused on designing a scalable and automated evaluation framework using DQA, which captures relative group-wise disparities in image quality without relying on costly and potentially subjective human annotation. Our motivation stems from the inherent difficulty of obtaining consistent and fair human evaluations across large-scale datasets and demographic conditions. Specifically, we have 120k samples, as mentioned in our paper:

250 images for each combination of profession, gender, and race, resulting in 20,000 images per scenario (10 professions, 2 genders, and 4 race),

where a scenario refers to each degradation type, T1 to T6.

We acknowledge the importance of grounding DQA in human perception and have noted this as a limitation in Section 5.4. The observed alignment between DQA values and perceptual differences across demographic groups suggests its potential as a reliable proxy. We consider incorporating structured human evaluations a valuable direction for future work to further reinforce the practical relevance of DQA.

Fairer but Less Powerful Model

Thank you for the thoughtful follow-up question.

We clarify that in our study, the generated images are controlled to have comparable quality across demographic groups. Therefore, the encoder is not "ignoring bias" in the input, because the input itself contains no systematic bias by design.

This controlled setup ensures that any disparity measured by DQA arises from the encoder’s behavior rather than differences in input quality. Our analysis shows that a higher DQA indicates group-specific sensitivity in the encoder's output, which reveals fairness issues. In contrast, a lower DQA reflects more equal treatment across demographic groups by the encoder, given that the input quality is controlled via synthetic degradations.

The concern of "fake fairness", where an encoder assigns the same embedding regardless of input quality, indicates a poor representation power. Such encoders might yield low DQA, but at the cost of ignoring all variation, not just group differences.

Hence, we emphasize that fairness (measured by DQA) and representation ability (like [3]) are parallel and complementary dimensions, not contradictory. Ideally, a good encoder should achieve both: low DQA (fairness) and high sensitivity to meaningful differences (representation), treating all the demographic groups' degradation equally.

[3] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.

We thank the reviewer for the insightful comments and suggestions.

W1: Clarification on Image Quality Definitions and Equivalence

• Thank you for raising this important point. We clarify that the terms "well-generated" and "poorly-generated" do not reflect human evaluation but are conceptual labels referring to controlled generation settings. Specifically, they correspond to predefined corruption levels (e.g., T1 and T6 in Figure 5), which are described in Section 4.2 and detailed in Appendix D. These terms are only used to explain the intuition behind quality variations, but all experiments use the explicit degradation labels (T1 through T6) rather than subjective descriptions.
• To ensure strict quality equivalence across demographic groups, we apply the same generation pipeline, prompt, and parameters with a fixed seed. Moreover, to maintain more robust consistency across demographic groups, we adopt an in-painting strategy, following [1], and detailed in Appendix D. We also follow the degradation protocols used in Borji [2] and Kim et al. [3], both of which analyze generation failures (e.g., truncated denoising or reduced steps). This setup is described in Section 4.2 and illustrated in Figure 8, with further configuration details in Appendix D. By generating 20,000 images, we minimize the influence of outlier generations and ensure robust group-level comparisons.
• The terms "comparable quality" and "consistent quality" both refer to images being generated under the same degradation type. We will clarify this language in the revision to avoid ambiguity and reinforce the operational definition used in our setup.

[1] Lui et al. Leveraging diffusion perturbations for measuring fairness in computer vision. AAAI 2024.
[2] Borji et al. Qualitative failures of image generation models and their application in detecting deepfakes. Image and Vision Computing, 2023.
[3] Kim et. al., Model-Agnostic Human Preference Inversion in Diffusion Models

W2: Rationale for DQA Term

We appreciate the reviewer’s detailed observations and the opportunity to clarify the rationale behind our DQA formulation.

The strength of the DQA metric lies in the joint use of the numerator and denominator. The numerator, $D(f(A_{gen}), f(A_{ref}))$ $-D(f(B_{gen}), f(B_{ref}))$, captures group-wise disparity and isolates potential encoder-induced bias under the assumption of equivalent quality. However, as noted by the reviewer in W2.1, when both group-specific embeddings (e.g., $f(A_{gen})$, $f(B_{gen})$) are similarly close to their respective references, the numerator will be near zero. This is expected behavior and indicates fairness under poor generation conditions if both groups are similarly affected.

On the other hand, the denominator $D(f(I_{gen}), f(I_{ref}))$ plays a critical role. This term captures the global distributional shift caused by the generation process. By normalizing the group disparity by this global degradation, DQA focuses not on raw absolute differences but on relative bias.

This design improves sensitivity in multiple ways, as requested in W2.2:

• The normalization is necessary because different encoders operate at different scales. Without it, a large numerator may reflect scale differences rather than fairness issues.
• When both groups are equally degraded (i.e., poor generations), and the encoder maps them similarly, the numerator is small and the denominator is large, resulting in low DQA. This is a desirable outcome, as there is no unfair treatment relative to the severity of the degradation.
• When image quality is high (i.e., small denominator), but the encoder introduces a significant disparity between the groups (i.e., large numerator), DQA becomes large. This flags unfairness that could otherwise be masked in absolute terms.
• When overall quality is poor (e.g., blur, distortion), but one group is consistently treated more favorably in the embedding space, DQA remains sensitive to such relative disparities, rather than dismissing them due to uniformly bad generation.

In this way, DQA provides a scale-aware measure that allows for fair comparison across varying quality levels and highlights relative inconsistency in evaluation, even when absolute degradation exists.

W3: Alternative approaches?

Thank you for suggesting this thoughtful direction. Our method employs an external encoder to compute DQA, which is then used to guide the diffusion model during the denoising process. This serves as a form of regularization in the noise sampling stage, encouraging the generation of outputs that are both visually plausible and fair in quality across demographic groups.

There do exist methods for quantifying bias in internal representations of generative models [4, 5]. However, these primarily focus on distributional bias. For example, ensuring that both genders appear with similar frequency in the generated outputs. Such approaches generally assess the probability or count of demographic categories rather than the visual fidelity of the outputs. In contrast, our focus is on quality bias in the generated images. To the best of our knowledge, this is the first attempt to mitigate such bias using a quality-aware fairness regularization.

Importantly, we believe debiasing internal representation would not lead to mitigation of quality bias because internal tokens or latent features are not suitable for directly measuring perceptual quality in image-level. The final output is heavily influenced by the stochastic decoding and sampling process, which introduces significant variability that cannot be captured by inspecting internal representations alone.

Therefore, we argue that using an external encoder is necessary to robustly quantify and correct quality discrepancies across groups.

[4] Jung et. al. A unified debiasing approach for vision-language models across modalities and tasks. NeurIPS 2024.
[5] Seth et. al. DEAR: Debiasing vision-language models with additive residuals. CVPR 2023.

W4: Limited Human Evaluation

We appreciate the reviewer’s comment regarding the lack of human evaluation.

In this work, we focused on designing a scalable and automated evaluation framework using DQA, which captures relative group-wise disparities in image quality without relying on costly and potentially subjective human annotation. Our motivation stems from the inherent difficulty of obtaining consistent and fair human evaluations across large-scale datasets and demographic conditions. DQA reflects relative differences in encoder activations across groups under controlled quality shifts, as described in Section 4.2 and Appendix D.

We acknowledge the importance of grounding DQA in human perception and have noted this as a limitation in Section 5.4. The observed alignment between DQA values and perceptual differences across demographic groups suggests its promise as a reliable proxy. We consider incorporating structured human evaluations a valuable direction for future work to further reinforce the practical relevance of DQA.

W5: Scope Constraints

We appreciate the reviewer’s observation. Our focus on gender, race, and profession stems from their well-established relevance in the fairness literature and the accessibility of existing works generating gender-profession datasets [1] enabling controlled evaluation. We believe this setup offers a meaningful and representative testbed for assessing group-based disparities in generative quality.

In fact, to demonstrate the generalizability of our method beyond common fairness benchmarks, we additionally applied DQA and DQA-Guidance to the medical domain, as detailed in Appendix F-H. The results show consistent behavior of our proposed metrics and guidance mechanism across this distinct and sensitive setting. This provides evidence that our framework is not limited to a specific set of attributes, and we plan to expand to other underexplored groups in future work.

[1] Leveraging diffusion perturbations for measuring fairness in computer vision. AAAI 2024.

We thank the reviewer for the insightful comments and suggestions.

W1: DQA Guidance for Different Diffusion Models

Thank you for raising this important point. While the main experiments used Stable Diffusion, we additionally applied DQA-Guidance to DeepFloyd, which consists of three stages: an initial generation stage followed by two refinement stages. Since it follows a denoising diffusion process, DQA-Guidance can be integrated without architectural changes.

The results demonstrate that DQA-Guidance improves both image quality and fairness in DeepFloyd, suggesting its adaptability across diffusion frameworks. Exploring additional models will be part of future work.

W2: Repetitive Contents

Thank you for pointing this out. We acknowledge that lines 96 to 101 in the methodology section repeat points made in the introduction. This repetition was intentional, as we are addressing a relatively unexplored challenge, the reliability of image quality evaluation metrics, and wanted to ensure that the motivation and problem setup are clearly communicated for readers who may need reinforcement of the key ideas.

That said, we agree that improving the flow and reducing redundancy can enhance clarity. In the revised version, we will refine the content to avoid unnecessary repetition while preserving the clarity of the problem statement.

W3: Consistent Notation

In Section 3.1, $D$ is used as a general notation for distance measures such as Fréchet Distance or MMD. We chose not to use this mathematical notation in Figure 3 because it would require additional definitions and might reduce clarity for readers unfamiliar with the formal setup. Instead, we focused on using accessible language in the figure for intuitive understanding, while maintaining precise and formal notation in Section 3.1. We will clarify this design choice in the revised version.

W4: On why previous works used all references

As noted in line 126, using a shared reference set $I_{\text{ref}} = A_{\text{ref}} \cup B_{\text{ref}}$ when computing group-specific distances can lead to misleading results. Nevertheless, this approach has been adopted in prior studies such as [41] and [44], likely due to its simplicity and compatibility with standard FID computation pipelines, which often assume a single reference distribution.

These works primarily focused on identifying the presence of demographic disparities in generated images and used FID as a convenient, widely accepted metric for quantifying quality. However, they did not explicitly account for the fact that reference distributions differ across groups. As a result, the fairness conclusions drawn from these evaluations may have been influenced by reference mismatch rather than actual quality differences.

Our work addresses this limitation by recommending group-specific reference sets to ensure a more valid comparison. We also introduce DQA to diagnose such inconsistencies in encoder-based evaluation and to guide more reliable measurement practices going forward.

W5-1: Clarifying Figure 4(b) and Use of t-SNE

In Figure 4(a), we include example images from both high-quality and low-quality generations for visual reference. However, the t-SNE in Figure 4(b) exclusively visualizes embeddings from the low-quality image set on purpose, as this condition better reveals how different encoders respond to degraded content. The t-SNE for high-quality images is separately included in Figure 7 in Appendix C, and we will clarify this in the main text.

Regarding the suggestion to visualize different encoders in Figure 4(b), we appreciate this insight and explored several options. However, because our visualization is class-conditional (in-class t-SNE, i.e., only images from the same class e.g., "nurse"), certain encoders produced clusters based on background, clothing, or color. As a result, the global embedding trend became less interpretable, even if local gender mis-embeddings were still present. We therefore chose to present the CLIP-based visualization in Figure 4(b), which offers the clearest global separation aligned with demographic attributes and best supports our discussion in Section 3.2.

We will add clarifications to the figure caption and appendix to avoid potential confusion.

W5-2: Using CLIP in DQA-Guidance

We ablated the effect of different encoders in the DQA-Guidance procedure by replacing DINO-RN50 with CLIP-ViT. This test confirms that DQA-Guidance is model-agnostic and can be applied with any image encoder, though the effectiveness varies depending on encoder reliability.

As shown below, DINO-RN50 yields stronger improvements in both quality and fairness. While our study highlights FID’s limitations in capturing fairness, we report FID-based results for reference. These results demonstrate that while DQA-Guidance is compatible with multiple encoders, selecting a more reliable encoder leads to stronger improvements in fairness.

W6: Details of Evaluation Dataset

We provide detailed information on the construction of the evaluation dataset in Appendix D, including the exact generation settings for each degradation type. The combinations of profession, gender, and race are also explicitly listed, and the dataset includes equal-sized sets for each combination to ensure balance.

In addition, we include visual examples in Figure 8 to illustrate the six degradation types used in the main experiments, and in Figure 10 for the medical image setting. These examples offer concrete guidance for reproducing the dataset. We will further highlight this information in the revised version to make it easier to locate.

W7, Q1: On Selection of $\lambda_1$ and $\lambda_2$

We confirm that both $\lambda_1$ and $\lambda_2$ are determined empirically, based on performance trends observed in Figure 9.

• $\lambda_1$ controls the strength of fairness guidance. It is selected to minimize quality discrepancy across groups, reflecting our primary goal of mitigating bias in evaluation.
• $\lambda_2$ serves as a regularization term to maintain overall generation quality. We select its value near the elbow point in Figure 9(b), where further increases begin to degrade quality more noticeably.

While we do not provide an analytical rule for choosing these values, the selection is not arbitrary. It is guided by consistent trends in the fairness–quality trade-off curves. In this sense, the choice is both evidence-based and repeatable.

Importantly, because our method is post-hoc and does not require training or fine-tuning the generation model, tuning $\lambda_1$ and $\lambda_2$ in practice is relatively lightweight. This makes the method easy to deploy in real-world scenarios without additional computational overhead. We will clarify this in the revised version.

W8: Clarifying Metric Directionality in Tables and Figures

Thank you for the helpful suggestion. In the main body of the paper, all reported metrics, image quality (e.g., MMD, FID), fairness (e.g., inter-group discrepancy), and reliability (DQA), are such that lower values are better.

The only exception is in Appendix J, where we report AUC for downstream tasks. In that case, higher is better. We will revise the captions and figure legends to make this directionality explicit in all cases.

Q2: On Eq.(1) and the Denominator in Failure Cases

We understand the concern regarding the potential influence of low-quality images on the DQA score, particularly when the denominator becomes large.

However, this behavior is actually desirable and aligned with the purpose of DQA. If both groups experience similar degradation (i.e., $D_A \approx D_B$), then the numerator will be small, and the resulting low DQA score correctly indicates reliable and fair treatment, even if the image quality itself is poor (high denominator). This means the encoder is consistently handling degradation across groups, which is exactly what we want to measure.

By contrast, if the image is low quality, but the encoder produces smaller distance values (D) for one group, the numerator becomes large relative to the denominator. In this case, DQA becomes high, which correctly flags unreliable or biased behavior under degradation.

Therefore, the normalization by the overall distance acts as a safeguard to ensure that fairness is not claimed at the cost of universally degraded output. We believe this design supports the interpretability and robustness of DQA, even in low-quality image regimes.

Extensive Experiments

We have conducted extensive experiments to validate the effectiveness of DQA as a reliability assessment for encoders, its validity, and the effectiveness of DQA-Guidance across different tasks (human images and medical images), supported by solid ablation studies.

• We generate 20,000 controlled samples and evaluate 13 encoders to identify factors affecting the reliability of image quality evaluation. The number of samples is comparable to that used in the literature (20k–25k), while the number of evaluation comparisons is significantly greater (ours: many; literature: typically 1–2) [1,2].
• DQA’s validity is supported by downstream tasks (Sec 4.5); datasets with lower DQA (using a fixed encoder) show improved fairness (Appendix B).
• The proposed mitigation strategy, DQA-Guidance, demonstrates consistent effectiveness (Table 1, 3, 4; Fig. 6, 9, 11, 12).

If the reviewer has any additional experimental suggestions, we are happy to conduct them.

[1] Rethinking FID: Towards a Better Evaluation Metric for Image Generation, CVPR 2024
[2] DEMYSTIFYING MMD GANS, ICLR 2018

Thank you for your efforts and response, and for mentioning that some of them will be fixed in the revised version.

The authors assert that selecting lambda values is feasible due to lightweight inference, which seems reasonable.

While recent work from 2024 [1] has conducted fewer evaluations than the authors’ studies, it focuses on innovation related to the FID metric. In contrast, the authors' work centers on addressing quality bias through DQA, which evaluates discrepancies in the quality of different demographic groups. This is why I raised the concern about the limited empirical evaluation. I would appreciate understanding the author's reasoning and justification behind this. Additionally, could you provide information about other works, besides [1, 2], 'in the literature' that have also used 20k-25k samples for evaluation?

We thank the reviewer for the insightful comments and suggestions.

W1: Title and Scope

Our work proposes a reliability measure for image quality evaluation with a focus on fairness. We will consider revising the title to "Assessing the Fairness Reliability of Image Quality Evaluation..." to better reflect this focus.

W2: Generalizability Beyond Demographics

The core idea of evaluating the reliability of image quality metrics extends beyond demographic fairness. While our main experiments focus on demographic attributes on facial images, we demonstrate generalizability by applying DQA to medical images (Chest X-rays) in Appendix F. More broadly, the proposed framework can be applied to any underrepresented group or condition in generative tasks, such as remote sensing or manufacturing.

W3: Comparison to the Paper

The referenced paper from reviewer shows that CLIP is affected by spurious correlations, mainly in classification tasks using high-quality images and filtering out low-quality images. In contrast, ours is a new perspective focusing on how image encoders behave with low-quality images across underrepresented groups, which leading to unreliable evaluations. DQA aims to quantify those reliability, which hasn't been explored in the literature.

W4: Paper Organization

Due to the page limit, we placed several experiments in the appendix, though they remain integral to the main narrative. We clarify their roles as follows:

• Appendix B and J demonstrate the impact of quality bias on downstream performance, supporting the motivation that fairness in generative quality affects fairness in classification.
• Appendix C provides additional evidence for Figure 4, reinforcing the claim that encoder bias becomes more pronounced for low-quality images.
• Appendix E offers a detailed ablation study that complements Table 1 and Figure 6 in the main paper.
• Appendices F, G, and H replicate the core experiments using medical images, demonstrating the generalizability of our method beyond general image.

We will consider moving key elements into the main paper to improve clarity.

W5: Systematic Variation in Synthetic Dataset

Here, we generate systematic variations images by varying one hyperparameter.

We vary the scale of Classifier-Free Guidance (CFG) ($s$), and the number of diffusion sampling step ($T$), respecively, which directly impact image quality. The gradual degradation shows the synthetic dataset is well controlled to generate intended poor quality.

W6: Computational Cost in time

We acknowledge the oversight in not including explicit timing details. The average generation time was 2.49 seconds per image for the baseline and 4.83 seconds per image with DQA-Guidance, reflecting the additional cost introduced by the guidance.

W7: On Controlled Dataset Design

We have considered using real-word dataset as reference, but found it unsuitable for controlled evaluation. It raises a question: Do the original images for two groups have the same quality? In practice, real images from different groups differ in background, lighting, and pose. This blurs whether embedding differences stem from encoder bias or image content.

Even though we utilize the real-world samples as reference and regard their corruption as quality degradation, the type of degradation is limited. The traditional image corruption such as jittering and blurring doesn't reflect the degradation in generative model [1]. Our synthetic setup avoids this issue by using shared seeds and controlled degradation.

As a special case, for medical images where real data can be assumed to have consistent quality, we do use real datasets as references, as shown in Appendix F and G.

[1] CNN-generated images are surprisingly easy to spot... for now. CVPR 2020.

Q1: Analyzing Figure 3

The reviewer's observation is correct. In Figure 3(a), both groups are shifted similarly from their references, so the measured distances should be comparable. However, using All Ref. incorrectly shows a larger shift for Group A, which is misleading. This becomes clearer in Figure 3(b), where Group B is farther from its reference, yet All Ref. still assigns a larger distance to Group A.

This demonstrates that All Ref. can produce unreliable results when group distributions are not symmetric. In contrast, group-specific references provide a more accurate assessment of each group’s distribution shift.

Q2: Regarding Denominator

Let the group-specific distance $Q_A$ and $Q_B$, and let the global quality on union set be $Q_{AB}$. The question is re-written: why $Q_{AB}$, not $Q_A+Q_B$ for denominator? While $Q_A+Q_B$ can be used potentailly, we intentionally chpose $Q_{AB}$ for global quality to capture cross-group interaction. Consider two scenarios: 1) $B_{gen}$ is embedded close to $A_{ref}$. 2) $B_{gen}$ is embdded far from $A_{ref}$, but in both cases, the distance to $B_{ref}$ remains the same. In these scenarios, $Q_B$ is identical, so $Q_A+Q_B$ distinguish between them. In contrast, $Q_{AB}$ reflects the differnece. Thus, while $Q_A+Q_B$ is not incorrect, $Q_{AB}$ better aligns with our philosophy of normalizing by global quality.

Q3: Clarification on $\lambda_1 = 0$

$\lambda_1 = 0$ case is already included in Table 1 as the “Baseline,” which corresponds to generation without DQA-Guidance. In contrast, all rows labeled “+ DQA-Guidance” use $\lambda_1 > 0$. The global quality metric is reported as Avg.MMD.

Q4, Q7: Without Denominator & Exccessive $\lambda_1$

The denominator in Eq(1) captures the overall distributional shift, while the numerator reflects group-specific disparity. This normalization is necessary because different encoders operate at different scales. Without it, a large numerator may reflect scale differences rather than fairness issues.

In Eq(4), using only the numerator as a guidance term could reduce group disparity but allow overall image quality to degrade. To prevent this, we adopt DQA as guidance, also add the denominator as a separate regularization term, controlled by $\lambda_2$. The reviewer's question would be

1. $\lambda_1 |Q_A-Q_B|/Q_{AB} +\lambda_2Q_{AB}$ <- Why this,
2. $\lambda_1 |Q_A-Q_B| +\lambda_2Q_{AB}$ <- Not this?

Case 2 depends on the balance between $\lambda_1$ and $\lambda_2$. However, ours (case1) offers adaptive behavior. When overall quality is poor, the first term ($|Q_A-Q_B|/Q_{AB}$) is relatively small and the second term ($Q_{AB}$) is large, increasing the weight on improving global quality. Conversely, when the overall quality is already high (small $Q_{AB}$), it can focus more on the first term, the group disparity.

Here we report the experimental result with numerator-only case.

However, setting a excessively high $\lambda_1$ overly emphasizes fairness by aggressively minimizing the DQA score (i.e., the numerator), which can unintentionally increase the denominator ignoring the impact of $\lambda_2$, resulting in degrading global image quality. This is consistent with the reviewer’s interpretation (also noted in line 254).

Q5, Q6: On the Source of Missing Fairness and Novelty of Our Contribution

The unfairness in image quality stems from biases in both the text encoder and the diffusion model's training data.

The references in line 35, Naik et al. show that even specific attributed prompts, Diffusion produces lower-quality images for underrepresented groups, suggesting that the text encoder struggles with rare concepts due to biased training data. Perera et al. demonstrate that diffusion models generate lower-quality images even when trained on balanced datasets by, as they struggle to learn balanced representations due to variations in data like lighting, hair, and makeup.

While we are not the first to highlight quality disparities across demographic groups, our work makes two novel contributions:

• We show that widely used evaluation metrics (e.g., FID) can mislead fairness assessments across groups. We introduce a diagnostic tool (DQA) to quantify metric reliability across demographic attributes (line 72).
• DQA-Guidance, a method to mitigate quality bias at the sampling stage of diffusion models without retraining (lines 73, 93, 279).

To the best of our knowledge, these aspects, evaluation metric reliability and mitigation at inference, have not been addressed in prior work.

Q8: Ablation study with different image encoder.

We ablate the effect of encoder in the DQA-Guidance by replacing DINO-RN50 with CLIP-ViT. The results demonstrate that while DQA-Guidance is compatible with any encoders, selecting a more reliable encoder leads to stronger improvements in fairness.

Q8-2: DQA score for different attributes?

DQA is not computed separately for each group. In Figure 5, DQA is computed for each attribute category (race or gender) by averaging over each degradation type (T2–T5), as mentioned in the caption.

Q9: Result with DINOv2 DQA

DINO is trained on the ImageNet 1K dataset, while DINOv2 uses the LVD-142M dataset. We conducted additional experiments with DINOv2 and BYOL-RN50, and the results align with our analysis:

• CNN-based models tend to be more reliable than ViT-based models
• Larger training datasets do not necessarily improve reliability

W4: Relocating Experiments

Thank you for the thoughtful follow-up. We agree that experiments cited in the introduction should not be confined solely to the appendix. We plan to revise the structure accordingly:

• Appendix B will be moved into the main body, potentially as a "Problem Definition" or "Motivation" section. This section shows how our framework extends beyond general image domains and motivates the importance of addressing quality bias in generative model.
• Appendix C will be merged with Figure 4 to present results from both well-generated and poorly-generated images in a single figure while maintaining clarity.
• Appendix E (Figure 9) supports the ablation analysis in Table 1 and Figure 6. We plan to merge the quantitative results into this figure and move it into the main paper.

Please let us know if you have suggestions beyond this proposed restructuring.

W7: Regarding Traditional Image Degradation

Thank you for your insightful follow-up and clarification. We acknowledge that our earlier mention of "jittering" was inaccurate, and we appreciate the opportunity to elaborate further.

We argue that traditional degradation (e.g., blur, JPEG) and generative degradation (e.g., insufficient denoising, early termination) are inherently different types of perturbations. This distinction is supported by two observations from [1]:

• In Figure 5 of [1], classifiers trained to detect generative artifacts fail to generalize to traditionally degraded images (e.g., blur, JPEG) unless proper augmentations are introduced. This indicates that generative and traditional degradations manifest differently in image space.
• The authors also note:

  “DeepFake images do not contain obvious artifacts. We note that DeepFake images have gone through various pre- and post-processing, where the synthesized face region is resized, blended, and compressed with MPEG.”

This suggests that generative degradations result in more subtle and complex artifacts that are not easily captured by traditional corruption.

However, we fully agree that using a more controlled degradation (e.g., applying symmetric traditional corruption across groups) would support a more systematic study of encoder robustness. While our current design is tailored to study degradation induced within the generative process (which reflects practical failure cases in text-to-image models), we see traditional degradation as a complementary axis to explore.

We will retain this direction as a promising future work, and we plan to include a note or preliminary analysis in the appendix of the revised version.

[1] CNN-generated images are surprisingly easy to spot... for now
[2] Exposing photo manipulation with inconsistent reflections

Q3: Experimental Results for $\lambda_1 = 0, \lambda_2 > 0$

Thank you for the clarification. We initially presented results with both coefficients at zero and ablation studies with either $\lambda_1=20$ or $\lambda_2=0$ fixed. Below, we provide new results for $\lambda_1=0$ and $\lambda_2> 0$:

The results show that using $\lambda_2$ alone improves overall performance, but combining both $\lambda_1$ and $\lambda_2$ yields the strongest improvement. We will update Table 1 in the revised version accordingly.

Q8-2: Individual Ranking

Thank you for the clarification. We now understand that you are requesting per-condition results rather than aggregated metrics.

In our study, the DQA metric has three axes of analysis: (1) the encoder, (2) the type of degradation (T2 to T6), and (3) the profession. For each target attribute (e.g., gender or race), we compute DQA for every (degradation type $\times$ profession) pair. This results in a detailed DQA matrix per encoder.

Below we present the full breakdown of Gender DQA scores for three representative encoders, BYOL-RN50, DINO2-ViT, and DINO-RN50, across degradation types and professions.

These granular results provide several key insights:

• T3 (fewer generation steps) and T4 (strong noise influence) often result in higher DQA values across encoders, in contrast to T5 (no refiner), which consistently shows lower DQA. This indicates that encoders are more vulnerable to demographic inconsistency under T3 and T4.
• Each encoder behaves differently across profession-attribute combinations. For instance:
  • DINO-RN50 handles the "pilot" profession fairly (low DQA) but shows more bias for "flight attendant".
  • In contrast, BYOL-RN50 treats "nurse" profession fairly, but performs poorly on "pilot".
• DINO2-ViT consistently shows high DQA, indicating it may have stronger demographic disparities despite its perceptual strength.

Overall, while the individual scores slightly deviate from aggregate trends, the relative ranking of encoders remains broadly consistent, supporting our main claims. These detailed results will be included in the revised appendix, and we will expand our discussion to reflect the profession- and degradation-specific insights.